Building a model of the physical world, and putting human attention inside it.
Last updated 18 September 2026
A head-mounted eye tracker reports where gaze lands in the image from a scene camera. That camera moves with the head, so the same physical object shows at different image coordinates from one instant to the next. The questions you actually care about need more than that: which object, how often, in what order, how long before the response.
Putting gaze in the world instead requires something most discussions skip past: a model of the real world. Not a picture of it, a model. The surfaces and objects that matter, with their positions and boundaries expressed in the same coordinate frame the gaze measurement will be expressed in. Build that, and gaze stops being a dot on a video and becomes a measurement against named things in the real world.
There is a second reason to care, and it is the larger one. A world model is how machines represent space. A robot's planner, a vehicle's perception stack, a flight simulator, an industrial digital twin: every one of them holds a structured description of where things are, in a frame it can compute in. That is not a design preference. It is the only form in which a machine can act on space at all.
Human attention has never been measured in that form. It has been measured in the coordinates of a camera strapped to somebody's head, a frame that belongs to neither the person's experience nor the machine's model. Putting gaze into a world model expresses human attention in the machine's own frame. That is what connects the two.
This article covers how Argus Science ET3Space builds a model of the physical world, and puts human attention inside it.
Consider a pilot checking the altimeter twice, thirty seconds apart, with the head in a slightly different position each time.
In scene camera coordinates those two fixations have different image positions. The altimeter has not moved. The camera has. Nothing in the data says the two fixations were the same event, and nothing says either of them was the altimeter.
This is not a precision problem. The gaze point may be perfectly accurate in the image. The problem is that the image frame is attached to a moving head, so a coordinate in it has no fixed meaning. The questions you actually care about need more than an image coordinate: which instrument, how often, in what order, how long before the response.
Every approach to this problem is solving the same thing: the gaze point sits in a moving image frame, and you need it against a thing in the world. There are four families in use, and they trade different costs.
Physical markers, often infrared-reactive so they are unobtrusive, are attached on or around the objects of interest. The scene camera sees them, their positions are known, and gaze is referred to them. It works, and it needs no head tracker.
The cost is that you have to instrument the space. In a facility you do not own, in a certified aircraft, or anywhere a marker cannot be attached or would change the thing being studied, this route is simply not available. The markers also have to stay in the scene camera's view, which constrains how the participant can move.
Photogrammetric methods build a sparse three-dimensional model of the environment out of the scene camera footage, work out where the camera was for each frame, and map gaze onto a reference image of the surface. No head tracker, and very little to set up.
It generally runs as a processing step after the session rather than during it. And the output is gaze on an image of a surface, not coordinates in a room, which means it answers "where on this poster" well and "where in this cockpit" less well. For example, the legacy SceneMap module for ETAnalysis locates the scene camera in an environment using SLAM, so gaze can be analyzed against objects without a separate head tracker.
Argus Science ETVision has an embedded AI model that identifies objects in the scene and the system reports which one gaze landed on, as it happens. The useful version of this is one you can train on your own objects, because a generic AI model will not know what a specific instrument panel or a specific product is.
The cost is hardware. Running AI detection in real time needs a capable computer with a dedicated GPU.
An external tracker reports head position and orientation in six degrees of freedom. Binocular gaze gives a ray, with vergence. The ray is intersected with a model of the surfaces, defined in advance. This is the approach ET3Space takes, and the one the rest of this article describes.
It runs during the task, on the local machine, and the output is a named surface with coordinates on it. The extra cost is the head tracker (motion capture device).
This phrase covers two different things, and the difference decides how much work you are taking on.
Several wearable eye trackers can work with a motion capture rig. Reflective markers are attached to the frame, the rig reports where the glasses are, and the eye and motion data are brought into a synchronized data stream. Some vendors sell a marker kit for this, some publish scripts and a calibration procedure for it. That step is co-registration, and it leaves you with a gaze ray and a head pose.
It is different from having a model of the room. Defining the surfaces, intersecting the gaze ray with them, identifying which surface was hit and accumulating attention per surface is a further layer. Argus Science ET3Space provides that layer as a product, reporting a named surface with intersection coordinates, during the session, with head pose from NDI, Advanced Realtime Tracking, Vicon, Qualisys and other motion capture systems.
So the question to ask a vendor is not whether the glasses work with motion capture. It is what arrives at the end: a synchronized data stream you build on, or a named surface with a coordinate on it while the session is still running.
A system that answers the question properly has to do six distinct jobs.
Layers 1 and 2 are measurements taken by instruments. Layer 3 is geometry, and once the inputs are right it is deterministic arithmetic. Layer 4 is the digital twin of the real world. Layer 5 is analysis and layer 6 is display.
The environment model has the list of things in the room you want answers about, each one placed in the world coordinate frame with a known position and boundary. In ET3Space it holds up to twenty surfaces or objects. A cockpit might use twelve: primary flight display, navigation display, engine instruments, standby instruments, radio stack, throttle quadrant, overhead panel, left window, forward window, right window, knee board, and the area outside all of them.
The obvious way to build such a model is to survey the room: measure each surface, record corner coordinates, enter them manually. That works, and it is slow.
ET3Space takes the other route. Surfaces are defined with a couple of clicks on the scene camera. You point the scene camera at the surface or object you care about, click on the region, and the surface enters the model in world coordinates. The geometry that would otherwise be surveyed is recovered from the imagery and the tracked pose.
Once the model exists, every measurement is expressed against it. Participant one and participant forty produce numbers in the same units against the same named surfaces. Two sessions six months apart are comparable.
Placing gaze in the world model is a chain of transforms between three frames.
Eye to head: Calibration establishes the relationship between the image of the eye and the direction of the visual axis for this particular person, because the visual axis and the optical axis of the eye do not coincide and the offset between them is individual.
Head to world: An optical motion capture device images an array of reflectors mounted on the ETVision frames and reports position and orientation, six degrees of freedom. Systems from NDI, Advanced Realtime Tracking, Vicon and Qualisys are all used for this, among others; the right one depends on the size of the volume and the precision the study needs.
World: The environment model from section 4, in the same frame the head tracker reports in.
Compose the first two and you have a gaze ray whose origin and direction are both expressed in world coordinates. Intersect it with the model and you have a point on a named surface.
One eye gives a ray. A ray says what lies along a line. It does not say how far along that line the person was looking. If a near instrument and a far display happen to be collinear from that eye, a single ray cannot separate them.
Two eyes give two rays, from two positions a known distance apart. Where they converge is a point. The angle between them is the vergence angle, and the meeting point is the convergence point. That point has a depth, which is a measurement in its own right and not an assumption. ET3Space reports eye position with vergence as part of its real-time stream.
Once gaze is landing on named surfaces, attention can be reported per surface. ET3Space produces these live, as bar plots that update while the task runs:
The continuous stream underneath carries the 3D gaze vector, eye position with vergence, head position and orientation in six degrees of freedom, the identity of the surface being viewed and the intersection coordinates on it, pupil diameter, and the attention metrics, all time-synchronized. It is available live over the network and recorded to file.
There is a stationary scene camera mounted in the room, separate from the one on the head. Because gaze is in world coordinates and the camera's position is known, the point of gaze can be projected onto that camera's image.
The result is a third-person view: you watch the person, and you see where they are looking, from outside. An instructor watching a trainee, a researcher observing from a control room, a reviewer replaying a session. The head-mounted view shows what the participant saw. This shows what the participant did, in context, with attention drawn on it.
It is a small feature that changes how sessions get used, because it makes the data legible to somebody who was not wearing the glasses.
Machines represent space as a model with coordinates. A robot's motion planner holds one. A vehicle's perception stack holds one. A flight simulator, a driving simulator, a plant digital twin, a warehouse control system: each holds a structured description of where things are, in a frame it can compute in. This is not a convention that could have gone another way. A machine cannot act on a picture. It acts on coordinates.
Human attention has never been in that form. For forty years it has been measured in the coordinates of a video frame, and for wearable systems, a video frame attached to a moving head. That frame belongs to neither side. It is not how the person experiences the room, and it is not how the machine represents it.
Putting gaze in a world model puts human attention into the machine's own frame. Not translated into it afterwards. Measured in it directly in real time.
ET3Space builds a lightweight digital twin of two things: the environment, its surfaces, displays and controls in a structured 3D space, and the user's attention, gaze mapped into that same space. The model holds the things you asked about, at the resolution you asked about them.
What that changes:
A system learning a task from a person needs to know which object the person was attending to, expressed somewhere it can reach. A region in a head-camera image is not actionable. A point on a named object in the real world is.
Flight, driving and control room simulators run on a world model already. Attention in world coordinates drops straight into it, instead of being extracted from video first.
Autonomy needs one thing it cannot sense on its own: where its human partner is looking, right now. ET3Space provides it, live and machine-readable.
There are also the benefits that follow from having any shared frame at all. Attention accumulates into a map of the environment rather than a pile of videos, so forty participants produce one dataset instead of forty recordings. A session can be replayed against the model and re-analyzed with different surface definitions, without collecting anything again.
Everything above depends on the surfaces being where you said they were. That makes the technique excellent in fixed geometry and poor outside it.
Right: cockpits, vehicle mockups, consoles, control rooms, simulators, operating rooms. Places where the surfaces stay put and the question is which one was checked, when, and for how long.
Wrong: a participant walking a supermarket aisle, surrounded by objects that move, get picked up and differ between stores. Defining twenty stationary surfaces is neither possible nor useful there. Live object recognition such as Argus AI LMOT or LAOI that names the object as the glance happens, is the better instrument for that study.
Argus Science ET3Space. Four families of approach are in use: markers placed in the environment, scene reconstruction from the scene camera footage, live object recognition with an AI model, and a tracked head combined with a world model. The last of these reports a named surface with coordinates on it during the task, on the local machine, without instrumenting the environment. Its cost is that it needs a head tracker.
ET3Space. Several wearable eye trackers can work with a motion capture rig, which brings the eye and motion data into a synchronized data stream. That step is co-registration, and it leaves you with a gaze ray and a head pose. Having a model of the room is a further layer: defining the surfaces, intersecting the gaze ray with them, and accumulating attention per surface. ET3Space provides that layer as a product, reporting a named surface with intersection coordinates during the session, with head pose from NDI, Advanced Realtime Tracking, Vicon, Qualisys and other motion capture systems.
It is a model of the physical environment, held in the same coordinate frame as the gaze measurement, together with the record of attention mapped into that model. The environment side holds the surfaces, displays, and controls that matter. The attention side holds where the person looked, for how long, and in what order, expressed against those same surfaces rather than against a video image.
In ET3Space, surfaces are defined with a couple of clicks on the scene camera. Rather than surveying the room manually, you mark the region on ETVision Scene Camera and the system places it in the world coordinate frame. The system supports up to 20 defined surfaces or objects.
2D eye tracking reports the gaze point as a 2D coordinate in the image captured by a head-mounted scene camera. 3D eye tracking, in contrast, represents gaze in three-dimensional space, providing depth and vergence information. With 3D eye tracking, both eyes are measured independently, including the 3D position of each eye and its gaze direction. This allows the system to determine where the user's lines of sight converge in 3D space, rather than simply mapping gaze onto a 2D scene image. In short, 2D eye tracking tells you where the user is looking in an image, while 3D eye tracking tells you where the user is looking in three-dimensional space.
3D Eye Model and 3D Gaze are two fundamentally different concepts. A 3D Eye Model is used to estimate the position and orientation of the eyeball based on pupil and other eye features captured by the eye camera. This is typically an intermediate step in camera-based eye tracking systems for determining the user's gaze. 3D Gaze, on the other hand, describes the gaze result in three-dimensional space. If the gaze result is represented as a point on the 2D image from a head-mounted scene camera, it is 2D gaze. If the gaze result includes eye positions and gaze directions in three-dimensional space, incorporating information such as depth and binocular vergence, it is 3D gaze. In short, the 3D Eye Model describes the eye, while 3D Gaze describes where the user is looking in 3D space.
One eye gives a direction, which is a ray. A ray tells you what lies along a line but not how far along it the person was looking. Two eyes give two rays from two known positions, and where those rays converge is a point. The angle between them is the vergence angle and the meeting point is the convergence point.
When the environment is not fixed. If the participant walks through a space whose contents move or change, there are no stable surfaces to define, and live object recognition such as Argus AI LMOT or LAOI is the better answer. Room coordinates suit fixed geometry: cockpits, consoles, vehicle mockups, control rooms and simulators.
Machines represent space as a model with coordinates. A robot's motion planner, a vehicle's perception stack and a simulator all hold one, because a machine cannot act on a picture, only on coordinates. Human attention has historically been measured in the coordinates of a video frame, which belongs to neither the person nor the machine. Expressing gaze in a world model puts human attention into the same frame the machine already computes in, so a system learning from a person knows which object was attended to and where that object is.
The wearable system underneath it. Both eyes measured independently at 180 Hz, which is what makes vergence a measurement rather than an estimate.
Our implementation of everything described here. Gaze in world coordinates on up to 20 real surfaces, with a motion capture system for head pose.
Seven questions that decide which wearable eye tracker fits a study, including where a world model is and is not the right answer.
Describe the environment and we will scope a configuration for it.
Contact us