How to use 3D gaze to capture human attention for robot training
Cite as: Argus Science (2026). The Missing Half of the Demonstration: How to use 3D gaze to capture human attention for robot training. Eye Tracking in the Age of Physical AI, Part 2. Zenodo. https://doi.org/10.5281/zenodo.23579181
Robot demonstrations record what the demonstrator's hands did. They seldom record where she looked, yet her eyes hold the context of each action, such as which part she was working with and what she checked before she moved. A policy that learns from the hands alone has to guess that context. With a few hundred demonstrations or fewer, it can guess wrong and reach for a cable tie that happened to sit beside the workpiece.
Many robot policies weigh the parts of a camera image with the same attention mechanism language models use to weigh words. A language model tuned it on trillions of words. A robot policy has a few hundred demonstrations. Studies have used human gaze to show policies where to attend, and report better robustness to clutter, transfer to new object positions and faster training.
For gaze to serve training, it has to arrive in the robot's world model, tied to the coordinates the robot's cameras and arm share and to the clock of the demonstration. Gaze from eye-tracking glasses sits in the image of a camera that moves with the head, and the devices that put gaze on the robot's own images change how people look.
Argus Science ETVision with the ET3Space add-on records the demonstrator's gaze in the coordinates of the work cell while she works with her own hands. Each look lands on a named surface, in step with the robot's own records, and gaze from forty demonstrators forms one dataset. This paper explains why that matters for robot training and how 3D gaze makes it possible.
A robot demonstration is recorded in great detail. Joint angles, gripper state, forces and camera images are logged many times a second, and from them a policy learns to map what the cameras saw to what the arm did.
Suppose a workpiece sat beside a brightly colored cable tie in most of the demonstrations. In the data the two appear together, and either one predicts a good grasp. Nothing in the record tells the policy which one the demonstrator was working with. If the policy settles on the cable tie, the error shows up only when someone moves it and the robot reaches for it. Imitation learning calls this causal confusion. De Haan and colleagues illustrate it with a driving policy that learns to brake when a dashboard light comes on, because in its data the light comes on whenever the brake is pressed [de Haan et al. 2019].
The person giving the demonstration knew which object mattered. Her eyes went to the workpiece, to where it had to go and to whatever might be in the way, usually a fraction of a second before her hand moved [Land 2006]. Part 1 of this series called such a sequence of looks a person's reading of the scene. In a demonstration, that reading is the context of each action, and it is the half the recording leaves out.
A language model predicts the next word by weighing the words that came before it. The mechanism is called attention, and the model learned to use it from trillions of words of text.
Many robot policies are built on the same kind of model, and when they choose an action they weigh the parts of a camera image in the same way. But a manipulation task is often taught with a few hundred demonstrations or fewer, and from so few a policy can settle on the cable tie as easily as on the workpiece. More demonstrations help, and each one costs an operator's time.
Gaze can supply that context more directly. It records which parts of the scene the demonstrator attended to, one look at a time, and researchers have brought it into several parts of the training problem.
Where to attend. Adding gaze to the training objective improved behavioral cloning by 95% on average across twenty Atari games [Saran et al. 2021]. In vision-language-action models for manipulation, steering the model's attention toward predicted human gaze improved results by 4 to 12%, reached the same performance in fewer training steps, and gave attention patterns that resembled human strategies [Pani and Yang 2026].
How much to compute. Look, Focus, Act lets a vision transformer process about 20 image patches around the gaze point, where a uniform grid would use 324, and it trained about seven times faster [Chuang et al. 2025].
How far a skill carries. GazeBot builds each motion around the 3D point the operator looked at. When objects started outside the demonstrated range, it kept success rates between 75 and 100%, while a standard imitation method fell to 40% or lower [Takizawa et al. 2025b].
Where one step ends. Because the eyes move to the next object before the hand does, transitions between fixations have been used to divide long demonstrations into reusable sub-tasks [Takizawa et al. 2025a].
What comes next. The most recent work has the policy predict where a person would look before it acts, and it handled changed object positions, objects and scenes better than the π0.5 model it was compared with [Li et al. 2026].
In most of these methods the robot needs no gaze once it is deployed. Some policies learn to predict gaze from their own cameras, and they learn it from gaze recorded during collection [Chuang et al. 2025; Li et al. 2026]. The eye tracker belongs to data collection, where the gaze it records serves as ground truth for these models.
A language model reads its context in the same text it learns from. A robot finds the context of a demonstration in the demonstrator's eyes, and it can learn from that context only if the context arrives in the robot's world model, the description of the scene the robot learns and acts in. Part 1 of this series set out why gaze has to enter that model to be of use to a machine.
In training, the requirement takes a specific shape. A policy learns to connect what its cameras saw with what its arm did, and both live in coordinates the cameras and the arm share. Each look has to sit in those coordinates, so that it can be related to the camera images and the arm's motion of the same moment. It also has to sit on the demonstration's clock, because the eyes run ahead of the hands and the order of looks carries the structure of the task.
The term world model also covers learned world models, which predict how a scene will change when the robot acts and have become a central part of robot learning [Hou et al. 2026]. They too describe the scene in the robot's own terms.
Large human-demonstration datasets are increasingly recorded with head-worn cameras while people work with their own hands; one recent policy was pretrained on more than 20,000 hours of such video [Zheng et al. 2026]. Eye-tracking glasses fit this way of collecting, but they record gaze in the moving frame of the head camera. Before anyone can say which object was looked at, each look has to be matched to the scene. In one study of demonstration devices, labeling the task objects in a single participant's recordings took about six hours [Ishida et al. 2025].
Teleoperation headsets take another path. They show the demonstrator the robot's own camera images, so gaze lands on the images the policy trains on. Much of the research above used them. The device, however, shapes the gaze. The same study compared six ways of giving demonstrations and found that the more a device imitated the robot, the more people watched the gripper and the less they looked at the task objects. Gaze from people working with their own hands gave the best policies [Ishida et al. 2025].
Depth adds a further difficulty. GazeBot, for example, needs a 3D point and looks up the depth at the gaze pixel in a stereo image [Takizawa et al. 2025b]. On a bench, objects overlap along the line of sight, and a near part and a far one can lie on the same ray from the eye. A single line of sight cannot separate them. Two eyes give two lines of sight, and the point where they meet carries a depth measured from the eyes themselves (Figure 1).
Finally, a training set pools many demonstrators and many sessions, and their gaze has to share one frame before it can be combined.
Together these difficulties help explain a remark in a recent study of gaze for vision-language-action models. Its authors note that robotic datasets rarely include eye-tracking labels, so they generated gaze with a network and described their results as “an initial bound on the benefits achievable with real eye-tracking” [Pani and Yang 2026].
Eye tracking has met these questions before, in cockpits, simulators and control rooms, where the question is which instrument a person checked, when and for how long. Argus Science built ETVision with the ET3Space add-on for those settings, and a demonstration cell is a similar place, with a bench, fixtures and bins, and a person working among them.
ET3Space builds a lightweight digital twin of the cell and of the demonstrator's attention within it. ETVision measures both eyes, a motion-capture system tracks the head, and ET3Space finds where each line of gaze meets the surfaces in the model of the cell. Part 1 and the technical article “3D Gaze in the Real World” describe how this works [Argus Science 2026]. While the demonstration runs, each look gets an address in the cell, made of a named surface and a point on it, and the two eyes add a depth along the line of sight. In this form, human attention becomes a spatial signal that a machine can compute with (Figure 2).
The record fits the way demonstrations are collected and used. The demonstrator works with her own hands, the condition that produced the most useful gaze in the device study. Her gaze arrives in the cell's coordinates, which can be related to the robot's base and cameras with the kind of calibration robotics teams already perform.
ET3Space can also show a demonstration from outside. With a stationary camera fixed in the cell, it draws the point of gaze into that camera's image and exports the projected point as data. The result is a third-person view of the demonstrator at work, with her attention drawn on the scene, in the kind of fixed external view many manipulation policies take as input. The same view makes a recorded demonstration easy to follow for people who were not wearing the glasses, such as the engineers who review a dataset before training.
The record also lines up with the rest of the demonstration. Gaze, head pose and the surface hit travel on one time-synchronized stream, and event values from the robot controller, such as an episode number, can be written into it, so each look can be matched to the joint, camera and force data of the same moment.
Every demonstrator and session is measured against the same named surfaces, so recordings from forty people form one dataset, and a session can be re-analyzed later with different surface definitions without being collected again. Attention on each surface is scored as the task runs (Figure 3). Since gaze shifts with the device and even with a single instruction [Ishida et al. 2025], the live scores show during collection where demonstrators' attention is going.
The model of the cell suits workspaces whose geometry stays put, as a demonstration cell usually does. For parts that are picked up and moved, AI LMOT (Live Multiple Object Tracking), the live object recognition in ETVision, names the object being looked at as it happens, and it can be trained on a team's own parts.
Robot learning has advanced by recording more of what people do. Gaze measured in the robot's world model adds a record of what they attended to while they did it. The studies so far suggest that policies can learn from it, and where it helps most is still an open question.
Learned world models raise a further question. They are trained on recorded experience to predict what is likely to happen next, and in a demonstration gaze reaches an object before the hand does. As far as we know, no one has yet tested whether that early signal helps world models learn. A test would need gaze measured in three dimensions, in the scene's own coordinates and on the demonstration's clock.
ETVision with ET3Space is a mature product, used in defense, aviation, medicine and research, including high-consequence settings such as cockpits and operating rooms. It is available today for teams that want to study what human attention can add to robot training. For ETVision and ET3Space product and purchasing inquiries, contact sales@argusscience.com.
A demonstrator's gaze shows which parts of the scene she attended to while she acted. Researchers have used recorded gaze to teach policies where to attend, to build motions around the 3D point a person looked at, to divide long tasks into steps and to predict intent before an action. Reported gains include better robustness to distracting objects, better transfer to object positions outside the demonstrated range and faster training [Saran et al. 2021; Takizawa et al. 2025a, 2025b; Chuang et al. 2025; Li et al. 2026].
When an irrelevant object appears beside the right one in most demonstrations, a policy can learn to rely on it, a problem known as causal confusion [de Haan et al. 2019]. More varied demonstrations help. Another approach uses the demonstrator's gaze as a training signal, so that the policy learns to weight the parts of the image a person attended to. A gaze-based loss improved behavioral cloning by 95% on average across twenty Atari games [Saran et al. 2021], and gaze-guided attention improved vision-language-action models for manipulation by 4 to 12% [Pani and Yang 2026].
Studies report that it can. Look, Focus, Act used gaze to choose about 20 image patches for a vision transformer to process, where a uniform grid would use 324, and it trained about seven times faster [Chuang et al. 2025]. Steering a vision-language-action model's attention toward human gaze reached the same performance in fewer training steps [Pani and Yang 2026].
Rarely. The authors of a 2026 study that added gaze to vision-language-action models note that robotic datasets rarely include eye-tracking labels. They generated gaze with a network and described their results as an initial bound on what real eye tracking could deliver [Pani and Yang 2026].
Usually not. In most published methods, gaze shapes training and nothing gaze-related runs on the deployed robot, or the policy learns to predict gaze from its own camera images [Saran et al. 2021; Pani and Yang 2026; Li et al. 2026]. Those gaze-prediction models are trained on gaze recorded from people during data collection, so measured gaze serves as their ground truth [Chuang et al. 2025; Li et al. 2026].
Argus Science ETVision with the ET3Space add-on records gaze in the fixed coordinates of a workspace, such as a robot demonstration cell. ETVision measures both eyes, a motion-capture system tracks the head, and ET3Space reports which defined surface each line of gaze meets and where on it, while the task runs. Because the cell's frame can be related to the robot's, each look becomes a position the robot's world model can hold.
One way is for the demonstrator to wear Argus Science ETVision while working with her own hands, with the ET3Space add-on and a motion-capture system such as NDI, Advanced Realtime Tracking, OptiTrack, Vicon or Qualisys tracking the head. ET3Space places each look on a named surface in the cell's coordinates, on a time-synchronized stream that can be lined up with the robot's logs. Processing runs on the local system, with no cloud service in the path.
Gaze measured in the cell's coordinates can be projected into the image of any calibrated camera whose position in the cell is known. ET3Space does this for a stationary camera fixed in the cell, drawing the point of gaze on defined surfaces into the camera's image and exporting the projected point as data. Fixed external views of this kind are common inputs to manipulation policies.
ET3Space sends gaze, head pose and the surface hit on one time-synchronized stream, live over the network and recorded to file. Event values from another system, such as an episode number from the robot controller, can be written into the stream, and TTL sync pulses are available with the EEG TTL Sync add-on. A free SDK includes samples in Python, C++, C# and MATLAB, with Lab Streaming Layer support.
Yes, when every recording is measured against the same model of the workspace. ET3Space measures each demonstrator and session against the same named surfaces in the same coordinate frame, so recordings from forty people form one dataset, and a session can be re-analyzed later with different surface definitions without being collected again.
Eye Tracking in the Age of Physical AI examines how measured human gaze can contribute to machines that learn, act, and work with people. Part 1, The Missing Signal, sets out the signal and how it reaches a machine. This second paper examines its use in robot training. Later papers take the signal into other settings, and the series will grow as the work does.
Technical review by Guanglin Liu, Co-Founder of Argus Science. Argus Science develops eye-tracking technology, algorithms, and instruments in Massachusetts, USA. Its technical lineage runs back more than forty years, to a photoelectric-based eye tracking device invented by two MIT professors and manufactured by a company called Space Sciences. Space Sciences eventually became Applied Science Laboratories (ASL), one of the companies that created the modern eye tracking field. ASL instruments spent the next decades in cockpits, operating rooms, laboratories, and stadiums around the world. Argus Science was founded by people who came from ASL and continue to pursue the same goal: take the frontier of the technology and make it work in the field, not just the lab.
How ETVision and ET3Space are used to record gaze for robot training and embodied AI.
Gaze in the cell's coordinates on named surfaces, with a motion capture system for head pose.
Tell us about the cell and the data pipeline. We will show what a gaze record from ETVision and ET3Space looks like.
Contact us