How to use 3D gaze to capture human attention for robot training
A robot policy can reproduce every demonstration it was shown and still reach for the wrong object once a cable tie on the bench is moved. The demonstrator did not make that mistake, and her eyes suggest why: to the workpiece, to the gripper as it closed, to the drop point. Most demonstration datasets keep the motion of her hands and leave out where she looked. That gaze record is the missing half of the demonstration.
A demonstration records what the demonstrator did; her gaze traces how she read the scene against the goal of the task, which gives each action its context. A policy that receives the actions alone has to infer that context, and with limited data it may infer one that fits the demonstrations and fails when the scene changes.
Early studies suggest gaze can help. It has been used to help make policies more robust to distracting objects, to carry skills to object positions that were not demonstrated, and to divide long tasks into steps. Measured gaze is still rare in robot datasets, in part because it has been hard to capture well: teleoperation headsets can change where people look, and eye-tracking glasses leave gaze in the image of a camera that moves with the head.
For a robot to learn from gaze, it helps for gaze to arrive in the robot's own world model of the work cell: in its coordinates, attached to the objects it knows, and on the same clock as the rest of the demonstration. Argus Science ETVision with the ET3Space add-on is built to record it that way, while the demonstrator works with her own hands and eyes. Robot learning approaches the problem of understanding scenes in many ways. We write as specialists in measuring gaze, about a signal the early evidence suggests is worth recording.
A demonstration dataset is thorough about the body. Joint angles, gripper state, forces, and camera images are logged many times a second, and a policy learns to map what the cameras saw to what the arm did.
It records much less of how the demonstrator did it. Before her hand moves, her eyes have already gone to the part she will take, to where it has to go, to what might be in the way; they typically reach an object a fraction of a second before her hand acts on it [Land 2006]. Those looks reflect her understanding of how she stands in relation to the scene: held against the goal of the task, and updated as one step is done and the next begins. The gaze record that carries this trace is the missing half of the demonstration.
Part 1 of this series drew a comparison with language models, which learn what a word means from its context: the words around it and how they relate. An action in a demonstration has a context too. The same grasp means something different depending on what the demonstrator was relating it to: the part she meant to take, the fixture she was lining it up with, the step that came before. Demonstrations are being collected at growing scale; one recent study pretrained a policy on more than 20,000 hours of human demonstration video [Zheng et al. 2026]. As usually recorded, they keep the actions and much less of that context. Gaze offers a way to measure part of it.
A policy then has to infer the context from the actions, and the inference can go wrong. Suppose a workpiece sat next to a brightly colored cable tie in most demonstrations. In the data, both may predict a successful grasp equally well, and the actions alone may not show which one the demonstrator was using. Imitation learning calls this causal confusion. In a well-known example, a driving policy learned to brake when a brake indicator on the dashboard lit up, because in the demonstrations the two always went together [de Haan et al. 2019]. More varied demonstrations help, and they are paid for in demonstration hours [Takizawa et al. 2025b].
Meanwhile gaze, which reflects much of this context, is produced in every demonstration at no cost to the task, and it is seldom kept.
Recorded with the demonstration, gaze can give a learning system a measured trace of the context behind each action, the demonstrator's relation to the scene at that step, so that less of it may need to be guessed. The question is what this particular signal can add. Early studies have used it in three ways, which correspond to three aspects of the demonstrator's understanding: which things matter at a given step, where they are, and what comes next.
Which things matter. Gaze indicates which objects the demonstrator relied on and which she largely ignored. Used as an extra training signal, it has helped make manipulation policies more robust to irrelevant objects [Kim et al. 2020] and has been aimed directly at causal confusion [Banayeeanzade et al. 2025]. In games and driving, it improved behavioral cloning by 95% across twenty Atari games [Saran et al. 2021] and reduced steering error by 23.5% in a driving simulator [Chen et al. 2019]. Added to vision-language-action models for manipulation, it improved results by 4 to 12% across benchmarks [Pani and Yang 2026]. In several of these methods, gaze is needed during training and not when the robot runs.
Where they are. Gaze can also locate the relevant thing in space. GazeBot turns the demonstrator's gaze into a 3D point and builds each motion around it, so that learned skills carry over to object positions and gripper poses outside the demonstrated range [Takizawa et al. 2025b]. Look, Focus, Act concentrates a policy's visual processing around where the operator looked, using about 20 image patches where a uniform grid used 324. It trains several times faster and holds up better against distractors [Chuang et al. 2025].
What comes next. Because the eyes move ahead of the hands, the sequence of looks can also mark the structure of a task: where one step ends and the next begins. Transitions between fixations have been used to divide long demonstrations into reusable sub-tasks [Takizawa et al. 2025a].
Most of these studies use one aspect at a time, and mostly as a point in an image. The full sequence of looks may carry more: how the demonstrator related one part to another, when she checked and checked again, how her understanding shifted as the task went on. How much of that learning systems can use is an open question, and an interesting one.
Two limits apply. Eye tracking measures overt visual attention, which in hands-on work moves closely with attention itself, though not perfectly. And a fixation is evidence about what the demonstrator was using, not a label for why she acted. The studies above treat gaze as guidance for learning, and their results should be read that way.
The authors of the vision-language-action study note that robotic datasets rarely include human eye-tracking data. They generated gaze with a prediction network instead, and called their results an initial bound on what real eye tracking could deliver [Pani and Yang 2026]. Part of the shortage likely comes down to capture. The two common routes each involve a compromise.
Through the robot's eyes. The demonstrator teleoperates in a headset that shows the robot's cameras, with eye tracking built in. Gaze lands on the images the policy trains on, which is one reason much of the work above was done this way. But the device can change the gaze. A study comparing demonstration devices found that the more a device imitates the robot's body or view, the more the demonstrator watches the end-effector instead of the task objects. Policies trained on gaze from the most robot-like device did worst, in some cases below a policy trained without gaze. Gaze from people working with their own hands did best [Ishida et al. 2025].
Through the demonstrator's own eyes. The demonstrator wears eye-tracking glasses and works with her own hands, which is increasingly how large human-demonstration datasets are collected [Zheng et al. 2026]. The gaze is natural, but it lands in the image of a camera that moves with the head, so the same pixel can point at different things from one moment to the next. Turning it into “she looked at the fixture” takes work. Manual annotation took about six hours per participant in the device study [Ishida et al. 2025], and automatic mapping by image matching becomes unstable under occlusion and fast head motion [Zuo et al. 2026].
Both routes can also struggle with depth. On a bench, objects overlap along the line of sight. A depth camera reports the distance to whatever surface it sees at the gaze pixel, which is not the same as the point where the two eyes converged (Figure 1), and the two can differ most at edges and overlaps, where the distinction matters. Both routes make it harder for gaze to enter the robot's world model: one can alter what it was meant to record, and the other records it in a frame the robot does not use.
Table 1. How demonstrations capture gaze today, and the alternative
| Route | Natural gaze? | In the robot's frame? | Depth from |
|---|---|---|---|
| Screen-based (games, driving simulators) | Yes | Yes, for single screen; No, for multi screens. | Not applicable |
| Headset teleoperation through robot cameras | Changed by the device | Yes | A depth camera, if any |
| Eye-tracking glasses | Yes | No, a moving head-camera image | A depth camera, if any |
| Measured in the cell's coordinates | Yes | Yes, once the cell is registered to the robot | Both eyes and the cell model, solved as geometry |
A robot typically plans and acts from an internal model of its surroundings, increasingly called its world model. In a demonstration cell, part of that model describes where the bench, fixtures, bins, and parts are, in a frame the robot can compute in. Part 1 argued that gaze is most useful to a machine when it enters that world model. For a demonstration cell, that means measuring natural gaze directly in the cell's coordinates, then relating the cell's frame to the robot's once, with the same kind of calibration a robotics team already performs between its cameras and its arm.
From the measurement side, a training-ready gaze record would ideally have five properties. The demonstrator looks as she naturally would, with her own hands and eyes. Each fixation has a position in the cell, with depth from geometry rather than from an image lookup. It is attached to a named surface or object at the moment of the glance. It shares a clock with joints, cameras and forces. And it is calibrated for each demonstrator.
There are several established ways to place gaze on real objects [Argus Science 2026]. Markers fixed in the workspace, as one assembly dataset used [Grand et al. 2025], reconstruction from the head camera's video, and live object recognition each meet some of these properties and trade off others. A tracked head combined with a model of the cell can meet them together, at the cost of a head tracker. That cost is often already paid. Many robotics laboratories run optical motion capture, and a public hand-object dataset was recorded in a capture volume that tracked the headset, the hands and every object [Banerjee et al. 2024].
Tracking the glasses is the first step. Markers on an eye tracker, synchronized with motion capture, give a gaze ray and a head pose. A model of the cell, the intersection of each ray with its surfaces, the name of what was hit and a running count of attention on each object are a further layer (Figure 2). That layer turns a stream of rays into a record a training pipeline can use more directly.
ETVision with the ET3Space add-on was built for settings much like a demonstration cell: cockpit simulators, flight training, and control rooms, where surfaces stay put and the question is which one a person checked, when and for how long. Applied to a demonstration, it records eye data as part of the cell's model and while the demonstration runs.
The demonstrator wears the glasses, over her own prescription glasses if she needs them, and works with her own hands. ETVision measures both eyes independently at 180 Hz, and a motion-capture system, often one the laboratory already runs, tracks her head. ET3Space combines the two, computes each line of gaze in the cell's fixed coordinates, identifies the surface it lands on, and reports where on that surface. Each glance becomes a position in the cell rather than a dot on a video, in the same frame for every demonstrator, trial, and session.
The cell's fixed parts, such as the bench, bins, fixtures, and drop zones, are defined once as named surfaces, with a few clicks each on the scene image. ET3Space, including this way of setting up the space, is patent pending. Every demonstrator is then measured against the same named surfaces, so recordings from many people combine into one dataset, and the way different people read the same cell can be compared.
ET3Space turns eye tracking into a spatial signal a machine can compute with. ET3Space scores attention on every defined surface as the task runs, and draws it as bar plots that update live (Figure 3).
The method suits fixed geometry, such as a bench, a station, or a cell set up the same way each day, and it needs a head tracker. For gathering video across thousands of homes, where there are no stable surfaces to define, live object recognition on the glasses (AI LMOT, included in the standard version of ETVision) is likely the better instrument.
Robot learning has advanced by recording more of what people do. The demonstrations it learns from are thorough about the body and, as usually collected, thinner on the context that gave each movement its meaning. Gaze offers a measurable trace of that context. Recorded in the cell's frame and on the demonstration's clock, it can sit beside the actions it helps explain. Early studies suggest a policy can learn from it. How much, and in which tasks, is a question the field has just begun to ask.
We do not suggest how that question should be answered. Which objects deserve a surface of their own, whose gaze is worth learning from, and how much of the record should enter a training corpus are for the teams who train policies, who are better placed than we are to judge where gaze adds most. Recorded eye data is also a record of a person, and who holds it and what may be inferred from it deserves a written answer before collection starts.
Language models learned what words mean by meeting them in context. Demonstrations may come to play a similar role for robot learning, and gaze can supply part of the context their actions have so far been recorded without. Recording both halves of each demonstration, the actions and the gaze that places them in context, is one way to make it a richer teacher.
A later paper in this series turns to a setting where attention has to be read live and failure is most expensive: one pilot directing several uncrewed aircraft.
For ETVision and ET3Space product and purchasing inquiries, contact sales@argusscience.com.
Rarely. In a 2026 study that added gaze to vision-language-action models, the authors note the shortage and generate gaze with a prediction network instead [Pani and Yang 2026]. Most earlier studies recorded gaze on a screen, or through a headset showing the robot's cameras.
One way is to measure gaze in the coordinates of the work cell. A wearable binocular eye tracker gives each eye's line of sight, a motion-capture system gives the head's position and orientation, and a model of the cell gives the line of sight something to land on. ET3Space combines the three and reports, while the demonstration runs, the convergence point and the surface or object looked at, on one clock. Registering the cell's frame to the robot's frame once makes those positions usable in the robot's world model.
A robot's world model is its internal representation of its surroundings, which it uses to plan and act; much current research builds world models that also predict how a scene will change. For training, gaze helps most when it is already in the robot's terms: positions in the cell's frame registered to the robot's, the names of the objects the model represents, and timestamps shared with the demonstration. ET3Space supplies that record. How it is used in a policy or a world model is for the team training it to decide.
Not necessarily. In several methods cited here, gaze shapes training and nothing gaze-related is needed at deployment [Saran et al. 2021; Pani and Yang 2026]. In others, a model trained on recorded gaze predicts where to look, and the robot runs that model on its own camera images [Kim et al. 2020; Chuang et al. 2025]. Live human gaze during operation, used to tell a robot what a person wants, is a separate use [Zuo et al. 2026].
2D gaze is a point on the image of a head-mounted camera. A depth camera adds the distance to the surface at that point, which is not the same as where the eyes converged; the two can disagree at object edges and overlaps. 3D gaze measures both eyes in space and solves where their lines of sight meet. In ETVision that solution is a geometric computation, repeatable and checkable against the recorded eye video. With ET3Space, gaze also resolves into room coordinates and surface coordinates for each defined surface, twenty in the standard configuration and more if needed: a digital twin of the workspace, with the same coordinate frames for every participant.
You can, and it gives you a gaze ray and a head pose. Turning that into attention on named objects still needs a model of the cell, intersection of the ray with its surfaces, accounting per surface, and synchronization with the rest of the rig. ET3Space provides that layer as a product, with head pose from NDI, Advanced Realtime Tracking, Vicon, Qualisys, OptiTrack, and other systems.
Into any camera whose position in the cell's frame is known. ET3Space does this for a stationary camera and exports the projected point. For a camera on the robot, three conditions apply: the camera is calibrated, its pose is known at each moment if it moves, and the gazed point is visible to it rather than hidden behind something.
Gaze direction, eye positions and vergence, the convergence point, head pose, the surface hit and its coordinates, pupil diameter, and attention measures travel on one time-synchronized stream, live over the network and recorded to file. TTL pulses (with the EEG TTL Sync add-on), an external recording trigger and event values sent from another system, such as an episode number from the robot controller, line it up with the robot's logs. A free SDK includes samples in Python, C++, C#, MATLAB, and Lab Streaming Layer.
Automatic calibration takes about five seconds. The ETVision frame fits over the demonstrator's own prescription glasses, so no one has to be excluded for wearing glasses. The battery supports more than four hours of recording.
No. Recording, gaze computation, object recognition, and analysis run on the local system, with no cloud service or account in the path.
Yes. The StimTrac module in ETAnalysis maps head-mounted gaze onto a stationary screen recorded in sync, so a demonstrator who teleoperates from a monitor can be measured with the same glasses.
Eye Tracking in the Age of Physical AI examines how measured human gaze can contribute to machines that learn, act, and work with people. Part 1, The Missing Signal, sets out the signal and how it reaches a machine. This second paper examines its use in robot training. Later papers take the signal into other settings, and the series will grow as the work does.
Technical review by Guanglin Liu, Co-Founder of Argus Science. Argus Science develops eye-tracking technology, algorithms, and instruments in Massachusetts, USA. Its technical lineage runs back more than forty years, to a photoelectric-based eye tracking device invented by two MIT professors and manufactured by a company called Space Sciences. Space Sciences eventually became Applied Science Laboratories (ASL), one of the companies that created the modern eye tracking field. ASL instruments spent the next decades in cockpits, operating rooms, laboratories, and stadiums around the world. Argus Science was founded by people who came from ASL and continue to pursue the same goal: take the frontier of the technology and make it work in the field, not just the lab.
How ETVision and ET3Space are used to record gaze for robot training and embodied AI.
Gaze in the cell's coordinates on named surfaces, with a motion capture system for head pose.
Tell us about the cell and the data pipeline. We will show what a gaze record from ETVision and ET3Space looks like.
Contact us