Eye tracking where the reader is a machine
In skilled work, the hand follows the eyes. Before a part goes into a fixture, the eyes have already checked the opening, the edge, and the gauge. Those looks are how a person reads the scene, and the reading is updated with every glance. It guides the movements that follow, but little of it shows in the finished work.
That reading is the missing signal of this series: the visual relationship unfolding between a person and the scene. Language models learned context from text, where people had written their understanding down; in physical work, that understanding is seldom written anywhere. Eye tracking, the most direct behavioral measure of human attention available, can record much of that reading as the work unfolds.
Physical AI may find that record useful wherever machines are shaped by people: when they learn from human demonstrations, when they share a task with people, and when they work under a person's direction. Research in each of these areas has begun to show what measured gaze can add.
There is a catch. Eye-tracking data has traditionally been read by people, who look at a dot on head-camera video and supply the meaning themselves. In Physical AI, the reader is a machine, and a machine cannot easily understand what a dot on a video image means. It needs the person's gaze inside its own world model, the internal model of the scene it works from: in its coordinates, attached to the things it knows, on its clock, and computed where the work happens.
Argus Science ETVision with the ET3Space add-on is built for that step. It creates a lightweight digital twin of the workspace, places measured gaze inside it, and reports which surface a person is looking at, and where on it, while the task runs. This paper sets out the signal and how it reaches a machine. Part 2 follows it into robot training, and later papers in the series take it into other settings.
Consider a person fitting a component into a fixture. Before the hand moves, the eyes are already at work: into the opening, along the edge of the part, back to the opening, across to the gauge on the bench.
Each look on its own is small. Together they express something larger: the person's working understanding of the scene. Which parts matter to the task, how they relate to one another, what needs checking now, and what can wait. That understanding is not fixed. It is revised look by look as the task unfolds, and it shapes each decision about what to do next.
Studies of everyday tasks show how closely the eyes follow this understanding. In making tea or a sandwich, people look almost exclusively at objects the task involves, and rarely at the many others in view [Hayhoe and Ballard 2005]. Gaze typically arrives at an object about 0.6 seconds before the hand begins to act on it, and moves on to the next object before the action is complete [Land 2006]. The eyes are reading ahead of the hands, and each glance revises the reading that guides the next movement.
That reading is the missing signal considered here: the visual relationship unfolding between a person and the scene. The relationship is easy to miss because it leaves little behind. A finished assembly shows that the part went in. It does not show the clearance checked on the way, the edge compared against the opening, or the gauge consulted before the next step. A scene camera records the workspace. A motion-capture system records the movement. Neither, on its own, records the sequence of looks. This is one reason it is so often absent from the data used to design and train machines, even when every other channel is recorded with care.
Eye tracking can measure it. Eye-tracking research commonly treats gaze as the measure of visual attention, and in hands-on work the two move closely together. Recorded over time and mapped to the environment, gaze shows which surfaces a person attended to, where on them, in what order, and for how long. It also captures looks that produce no immediate action, such as a check of a clearance or a return to an instrument. What a particular look means still depends on the task: looking at a component may mean selecting it, inspecting it, or confirming that it is out of the way.
Language models offer a useful comparison. A language model interprets each word through its context: the words around it, how they relate, and how the meaning shifts as a passage unfolds. Models learned to do this from text, and text was a rich teacher in part because people had already written their understanding into it. The relationships between ideas were on the page.
Physical tasks have a context too: the scene as a person reads it. That context is seldom written down. The video and motion logs of a demonstration show what the hands did with which objects; the understanding that guided them largely stays with the person. Measured gaze offers a way to record it: a trace of how attention moved through the scene as the work was done.
Physical AI systems perceive and act in physical environments, and many of them are shaped by people. They learn from human demonstrations, share tasks with human partners, or work under human direction. In each case the system can receive detailed records of what people did. It seldom receives this trace of how they read the scene while doing it. Where a task depends on that reading, the machine may have to infer it or work without it. The three uses below are three expressions of the same opportunity.
A demonstration connects what a person saw with what they did. Video, hand trajectories and robot states describe the movement in detail, but leave open which visual information guided it. A background object may appear in every successful demonstration without contributing to the skill, and a learning system can mistake it for a cue.
Gaze adds the demonstrator's reading of the scene alongside the movement. In a manipulation study, Kim and colleagues used human gaze to improve a robot's robustness to irrelevant objects. GazeBot combined gaze information with motion bottlenecks to improve the reuse of learned skills in changed settings [Kim et al. 2020; Takizawa et al. 2025]. These results do not show that every fixation matters to the task. They do give a reason to record gaze alongside demonstrations. Part 2 of this series, The Missing Half of the Demonstration, examines this use in detail.
In a shared task, a machine and a person see the same objects. The machine also needs to know which one the person has in mind and what they may do next. A request such as “pass me that one” leaves the target open, and gaze can often help settle it before the words are finished.
In a collaborative sandwich-making task, Huang and colleagues found that a customer's gaze pattern predicted the requested ingredient correctly 76% of the time, about 1.8 seconds before the spoken request. More recently, GAMMA combined wearable gaze tracking with a vision-language model to select and parameterize robot manipulation skills from the surrounding scene [Huang et al. 2015; Tay et al. 2026]. Looking and commanding remain different events: gaze can help a machine identify a likely target, while the task logic determines whether to prepare, ask, or act.
Some machines work under a person's direction: autonomous vehicles, aircraft, or robots that a person oversees and commands, often while doing other work. The machines record their own states, what they displayed, and what the operator entered. They usually hold no comparable record of the person: what the operator has examined and what has gone unseen, whether the operator is near overload, or whether the operator's trust in the autonomy matches how it is performing.
Laboratory work suggests that gaze can fill parts of that gap. In a simulated task in which operators supervised five unmanned aircraft, Breslow and colleagues combined eye-fixation measures with the time available to respond. Their model predicted, in real time, when an operator would fail to prevent vehicle damage, and cues based on those predictions cut the damage rate by half relative to a condition without cues [Breslow et al. 2014]. In an automated supervisory task, fixation and saccade measures helped distinguish over-trust, calibrated trust and under-trust in the automation [Wang et al. 2024]. In a simulated flight study, a controller compared where the pilot was looking with where its own attention model said a competent pilot should look, and took over when the two diverged; the collision rate fell from 46% to 23% [Yin et al. 2023].
The studies are small, and a fixation does not prove that a warning was understood. Together, they point to machines that know something about the attention of the person directing them, and can act on it. Where machines already share control with people at scale, attention has entered regulation: since July 2026, every new car sold in the European Union must carry a system that follows the driver's gaze and head movements and warns of distraction [EU Regulation 2019/2144]. A later paper in this series takes these questions into the cockpit.
A machine works from an internal model of the scene around it, often called its world model, on which it plans and acts. For the person's reading of the scene to help, it is usually not enough for gaze to sit beside the machine's data as a separate recording of the same scene. Gaze is most useful when it enters the machine's world model itself: expressed in the model's coordinates, attached to the objects the model contains, and kept on the model's clock. The person's attention then becomes one more live element of the model the machine already works from. Wherever the person looks, the machine can hold the same point, on the same object, at the same instant.
A head-mounted eye tracker commonly displays a gaze point on scene-camera video. For a person reviewing the recording, that is often enough: the reviewer supplies the rest, recognizing the fixture, noticing the head turn, judging what the glance meant. A machine does not do this on its own. The camera moves with the head, so two looks at the same fixture can have different image coordinates, and the same image coordinate can refer to different objects after a head turn. When the reader of eye-tracking data is a machine, the spatial meaning needs to arrive with the data.
In the terms of the language-model comparison, a gaze point on head-camera video is a position on a page with no word attached. A spatial model can supply the word. Define the relevant surfaces and objects, give them positions and boundaries, and express gaze in the same coordinate frame. A measurement can then identify the fixture and a position on it. Repeated looks remain associated with that fixture as the person moves.
This is what a digital twin contributes here: a structured representation of the physical environment, with measured gaze mapped into it (Figure 1).
Once this model is registered to the machine's coordinate frame, the application can connect the person's gaze with its own scene and task records. A learning system can relate a glance to the next movement. A collaborative system can relate it to a candidate target. A system working under a person's direction can relate it to the display, vehicle, or event that the person has, or has not, examined.
The model supplies a common spatial reference for those connections. Reliable timestamps preserve their order. Together, they make gaze part of the machine's world model rather than a record beside it.
ETVision measures binocular gaze. ET3Space builds the digital twin described above and performs the spatial calculations that place that gaze on its surfaces. Other approaches can cover parts of this chain; together, ETVision and ET3Space provide the measurement and mapping layers as one existing product [Argus Science 2026].
Building such a model can otherwise require measuring surface positions and entering geometry by hand. ET3Space lets the operator define a surface with a few clicks on the scene-camera image. It uses the imagery and tracked pose to place that surface in world coordinates, with a name and boundaries (Figure 2).
This is a central part of the technology. A world coordinate for gaze becomes useful when there is a model against which to interpret it. ET3Space provides both the means to create that model and the calculations that use it during the session.
Twenty surfaces or objects come in the standard configuration, and more can be added. When the geometry remains unchanged, it provides a consistent reference across people and sessions. Measurements accumulate against the same named surfaces, making repeated recordings easier to group and compare. ET3Space, including this way of building the environment model, is patent pending.
ETVision measures both eyes independently at 180 Hz. Calibration relates the eye measurements to gaze direction. A motion-capture system tracks the position and orientation of the head in six degrees of freedom.
ET3Space combines these measurements to express the gaze ray in world coordinates (Figure 3). It then computes the line of gaze in fixed room coordinates, identifies the surface gaze lands on, and reports exactly where on that surface. The result is not just a point-of-gaze dot on a video. It is a 3D gaze with an address in the room, in the same coordinate frame for every participant, every trial, and every session. Line-of-gaze is specified in the room coordinate frame. Point-of-gaze is specified both as a point on a surface coordinate frame and a point with respect to the room coordinate frame.
One distinction matters when selecting an eye tracker. Combining eye tracking with motion capture leaves you with a gaze ray and a head pose. The environment model, surface intersection, and attention record are further steps, and ET3Space provides them within the product.
ET3Space streams the record over the network and writes it to file as the task runs: gaze direction, eye position and vergence, head pose, and the surface identity and intersection coordinates, with measures such as fixation duration and time to first fixation calculated for each modeled surface. Argus provides an SDK and sample code for receiving data and controlling recording. A robot-training dataset can associate the measurements with demonstrations; a live system can use them as inputs to its own logic. The gaze geometry is solved by geometric computation, so the same inputs always give the same answer, and each result can be traced back to its measurements.
For most of its history, eye tracking has answered questions people ask about people: what a reader noticed, where a driver looked, how an expert's scan differs from a novice's. The answers were read by researchers, who brought the context themselves. Physical AI adds a second reader. Machines that learn from people, work beside them, or act under their direction can receive the same measurement, provided it arrives inside their world model, in real time.
That changes what the measurement can be. It becomes more than a record reviewed after the work: it can become part of the context a machine works from while the work is under way. How much Physical AI gains from that context will be shown task by task, and much of that work is still ahead. Measuring the signal, and placing it where a machine can read it, is already possible.
For ETVision and ET3Space product and purchasing inquiries, contact sales@argusscience.com.
It is the visual relationship between a person and the scene: the person's working understanding of which parts matter, how they relate and what needs attention next, as expressed in where they look. Scene cameras record the workspace and motion capture device records movement, but neither, on its own, records this relationship. Because it shapes how a task is done rather than what the task produces, it is easy to leave out of the data used to design and train machines. Eye tracking can measure it.
In large part because of how gaze has been captured. Gaze from wearable eye trackers is usually expressed in the image of a camera that moves with the head, and relating it to the scene takes manual or additional processing. Researchers working on gaze for robot policies note that robotic datasets rarely include human eye-tracking data [Pani and Yang 2026]. Large collections of human demonstration video, such as the more than 20,000 hours used to pretrain one recent robot policy, record hand and wrist motion without gaze [Zheng et al. 2026]. Part 2 of this series looks at capture in detail.
In Physical AI, a world model is a machine's internal representation of its surroundings, which it uses to plan and act; much current research builds world models that also predict how a scene will change. Gaze becomes usable by such a machine once it is placed inside that model rather than kept as a separate recording. ET3Space supplies a lightweight digital twin of the workspace with gaze mapped into it, which can be registered to the machine's own frame. It complements the machine's world model rather than replacing it.
Eye tracking is the most direct behavioral measure of visual attention available, and eye-tracking research commonly uses the two terms together. Attention can shift briefly without the eyes moving, but in hands-on tasks the eyes and visual attention usually move together, with gaze arriving at an object shortly before the hand acts on it.
No. The comparison with language models in this paper concerns context: how understanding depends on relationships across a whole scene or passage. Human gaze is measured from a person's eyes; an attention map is computed inside a model. The two have different origins, but they can be related. Researchers have used human gaze to guide or regularize the visual attention of robot policies [Pani and Yang 2026].
It adds a measured record of human attention to the scene, action and system-state data that Physical AI already uses. Research points to three areas: learning from human demonstrations, resolving targets and anticipating actions in human-robot collaboration, and autonomous systems that work under a person's direction.
For decades the usual reader of eye-tracking data was a person reviewing video, who could recognize the object under a gaze point and allow for head movement. A machine generally cannot. For a learning or autonomy system to use gaze, the data needs to arrive inside the machine's world model: in its coordinates, attached to named surfaces or objects, with timestamps that line up with the machine's other records, and often computed on site.
A 3D eye model describes the eye and can help estimate its orientation. A 3D gaze result describes the eyes' positions and lines of sight in space. ETVision measures both eyes independently and calculates binocular vergence and a convergence point. ET3Space adds the mapping of gaze into the environment's coordinate frame.
A compatible head-tracking system and a model of the relevant environment. Supported motion-capture systems include NDI, Advanced Realtime Tracking, Vicon, Qualisys, OptiTrack, and others. Reflectors on the ETVision frame allow the motion-capture system to measure head pose. Surfaces are defined through the scene-image workflow. A suitable starting point is a workspace with stable geometry, such as a demonstration bench, a simulator, an instrument panel, or a control station.
ET3Space streams its record over the network and records it to file, and Argus provides an SDK and sample code for receiving data and controlling recording. A robot-training dataset can associate the measurements with demonstrations; a live system can use them as inputs to its own logic. Connecting a particular robot or simulator requires agreement on coordinate frames, timing and data fields. The gaze system supplies the spatial measurement; the application determines how to use it.
The surface model depends on the geometry remaining valid. If an object moves, its original modeled location no longer describes it. For changing visual targets, Argus provides AI LMOT and Live Areas of Interest, which identify and follow objects in the scene images. Object recognition in an image should not be confused with maintaining an updated 3D model of the object.
Yes. The Stationary Scene Camera option provides a synchronized outside view. With spatial calibration, gaze that lands on a defined surface can be projected into that camera's image, so a reviewer can see the person's actions and gaze together from a viewpoint in the room.
Yes. ETVision fits over the wearer's prescription glasses. This is useful when collecting data from many different participants.
No. It is an eye-tracking and spatial-mapping tool that can supply data to training or live systems. The studies cited here demonstrate uses of gaze in their own experimental systems; they are not evaluations of ETVision or ET3Space. Performance in a particular robot application depends on how the data is used.
Eye Tracking in the Age of Physical AI examines how measured human gaze can contribute to machines that learn, act, and work with people. This first paper sets out the signal and how it reaches a machine. Part 2, The Missing Half of the Demonstration, examines its use in robot training. Later papers take the signal into other settings, and the series will grow as the work does.
Technical review by Guanglin Liu, Co-Founder of Argus Science. Argus Science develops eye-tracking technology, algorithms, and instruments in Massachusetts, USA. Its technical lineage runs back more than forty years, to a photoelectric-based eye tracking device invented by two MIT professors and manufactured by a company called Space Sciences. Space Sciences eventually became Applied Science Laboratories (ASL), one of the companies that created the modern eye tracking field. ASL instruments spent the next decades in cockpits, operating rooms, laboratories, and stadiums around the world. Argus Science was founded by people who came from ASL and continue to pursue the same goal: take the frontier of the technology and make it work in the field, not just the lab.
Part 2 of the series: how to use 3D gaze to capture human attention for robot training.
How gaze is placed on real objects in a room, and what a world model has to contain for that to work.
The product described here: gaze in room coordinates on real objects, with a motion capture system for head pose.
Describe the workspace and the system that will use the data. We will scope a configuration for it.
Contact us