Why do virtual hands still struggle to grasp objects?

The camera loses sight of the fingers

Camera-based hand tracking estimates joint positions from images. It already uses computer vision and machine learning; the displayed skeleton is an interpretation of what the cameras see. Fingers hiding one another, overlapping hands and rapid movements make that interpretation harder. Meta's documentation describes these limits, including confidence loss and occasions when the virtual hand disappears.

That matters particularly during manipulation. A hand can look convincing while moving through empty space, then become unstable as the fingers close. Smoothing the estimated joints may reduce jitter, but it also leaves a more basic question unanswered: is the person trying to hold the object, adjust their grip or let go?

Better tracking and better interaction are related, but they are different problems. The review by Mangalam and colleagues describes difficulties with collision, finger coordination and release. A fingertip reaching a surface does not by itself specify how an object should attach to the hand, move with it or become free again.

What can movement reveal about an intended action?

Imagine reaching towards a virtual cup. The direction of the reach suggests a target; the opening hand suggests a possible grasp. Each observation is ambiguous on its own. Together with the nearby objects, they give the software more evidence about what might happen next.

Point & Grasp, presented at CHI 2026, applies this reasoning to selecting distant objects. It combines pointing direction and grasp shape probabilistically. When several objects lie in a similar direction, the grasp gesture supplies another clue. The authors report better selection accuracy and speed than single-cue alternatives in their studies.

SIAgent goes further towards interpreting an action. It converts eye and hand motion into a description that a language model uses to infer intent, then uses an agent to execute the operation. Its authors report 97.2% intent recognition compared with 93.1% for gaze-plus-pinch across the evaluated tasks. The result concerns recognising and executing operations, rather than generating every finger movement during a grasp.

These papers use "intent" at different levels. Selecting a cup, grasping its handle and deciding where to put it are separate decisions. Recognising the object does not determine the contact points or the path the hand should follow.

Uncertain tracking can still produce useful input

TouchInsight offers a concrete example. It uses egocentric hand tracking to estimate when a finger touches a physical surface, which finger touched and where the contact occurred. Instead of treating the estimated location as exact, it represents a range of likely locations and uses contextual information to resolve the intended input.

In the authors' offline evaluation, mean touch-location error was 6.3 mm. In an online text-entry study, participants averaged 37 words per minute with a 2.9% uncorrected error rate. The text-entry result shows what the method supported in use, beyond its accuracy at locating individual touches.

The surface supplies a useful constraint: the finger is touching somewhere on it. Grasping an object in free space introduces more possibilities. The software has to reason about several moving contact points, the object's motion and whether the user intends to maintain the grasp.

The visible hand does not have to copy every joint

VR-HandNet generates hand motion from controller input and the spatial relationship between hand and object. A neural network predicts desired joint orientations; a physics simulation applies joint torques to produce the next pose. This lets the fingers respond to the object instead of simply replaying a measured skeleton.

ForceGrip, published at SIGGRAPH 2025, develops this idea around grip force. It trains a hand controller on generated scenarios with varied objects, wrist movements and trigger inputs. The resulting motion reflects the intended force supplied through the controller.

Both projects demonstrate that plausible finger motion can be generated from a smaller set of control inputs. The handheld controllers supply explicit signals through their triggers. A camera-based method has to infer a comparable instruction from potentially ambiguous movement.

An intended action still needs contact geometry

The Gaze2HOI preprint makes this distinction explicit. It separates intention inference, contact grounding and hand-object motion generation. Contact grounding describes where the hands and objects will meet, and where an object will receive support.

Suppose the predicted action is to put down the cup. That still leaves the destination, orientation, supporting surface and release timing to resolve. Gaze2HOI uses gaze to help disambiguate the intended action, then conditions the motion forecast on contact and placement information. It is forecasting research, published as a preprint in October 2026, rather than a demonstration of continuous interaction on a standalone headset.

This explains why a high intent-recognition score cannot stand in for a successful manipulation result. The software may identify the action correctly and still produce the wrong contact geometry or release the object too early.

A helpful correction can also feel like a loss of control

An independently animated hand creates a human problem as well as a geometry problem. If a person reaches past the cup, a plausible but mistaken prediction could make the virtual hand seize it. If they change their mind halfway through a reach, the displayed movement needs to follow that change.

Gonzalez and colleagues' reach-redirection work studies how a virtual hand can be shifted during a reach to guide the real hand towards a physical location. Their controller generates the redirection under constraints that include perceptual thresholds. It provides a way to study deliberate differences between real and displayed motion.

The relevant question is how much deviation a person accepts while still feeling in control. A grasp that looks correct to an observer may feel wrong to the person performing it. Task success and the user's sense of agency therefore describe different aspects of the interaction.

Headset hardware changes what transfers

SIAgent and Gaze2HOI use eye gaze. Quest 3 and Quest 3S lack eye tracking in Meta's device comparison. Removing gaze removes evidence that those methods use to distinguish between possible actions. Their results cannot simply be carried over to a hand-only implementation.

Timing is another constraint. Meta lists hand tracking at 30 Hz by default and 60 Hz in Fast Motion Mode, with a jitter trade-off. A Quest 3 display can refresh at up to 120 Hz. The display may draw several frames between observations, but those extra frames contain no new camera evidence about what the person is doing.

Inference and animation add their own processing time. Warburton and colleagues' motion-to-photon measurement work shows how physical and displayed movements can be recorded together to measure the delay. That complete path matters when judging responsiveness, especially when a user interrupts an action already in progress.

What remains unresolved?

Selection accuracy, touch-location error and plausible animation measure different things. A method can perform well on one and struggle with another. Comparing these papers is most useful for understanding which part of the interaction each method solves and what inputs it needs.

The unresolved combination is continuous manipulation that infers actions from imperfect hand observations, maintains contact through occlusion and responds promptly when the person changes direction or releases. The cited work does not establish that whole combination on a standalone headset.

A useful comparison would keep the tracking input fixed and examine what intent inference adds beyond smoothing and contact correction. Wrong grasps, unwanted releases and recovery after a change of mind would be as informative as successful pickups. That is where a convincing animation has to demonstrate that it also makes the interaction easier for the person using it.

Sources

Selected from research dossier LI-VR-001, reviewed 8 November 2026. Measurements are those reported by the cited research teams. Gaze2HOI is a preprint; the examples involving a cup are illustrative.

1. Meta, hand-tracking limitations

   Official documentation on tracking volume, occlusion and confidence loss.

2. Wang, Gu and Lu, SIAgent

   2026. Intent recognition from eye and hand motion, followed by agent execution.

3. Luo and colleagues, Point & Grasp

   CHI 2026. Probabilistic integration of pointing and grasp cues for distant selection.

4. Streli and colleagues, TouchInsight

   UIST 2024. Uncertainty-aware touch inference from egocentric hand tracking.

5. Han and colleagues, VR-HandNet

   IEEE TVCG, 2024. Learned, physically plausible virtual-hand manipulation.

6. Han and colleagues, ForceGrip

   SIGGRAPH 2025. Learned hand motion reflecting intended grip force through controller input.

7. Ma and colleagues, Gaze2HOI

   October 2026 preprint, revised 3 November. Intention, contact grounding and hand-object motion forecasting.

8. Gonzalez and colleagues, reach redirection

   CHI 2022. Model predictive control of visual hand redirection within perceptual constraints.

9. Mangalam and colleagues, precision hand-object interaction

   Virtual Reality, 2024. Contact, collision and release problems in precise manipulation.

10. Meta, Hands Technology

    Official documentation, updated August 2026. Camera-based tracking and its update-rate trade-offs.

11. Meta, device comparison

    Official documentation, updated October 2026. Display and eye-tracking capabilities by device.

12. Warburton and colleagues, motion-to-photon latency

    Behavior Research Methods, 2023. Measuring delay through the complete VR sensing and display path.