Eye tracking for any Unity VR scene

The Meta Quest 3 has no eye tracker, so I built one.

problem

VR training can show what a trainee did, but not what they looked at. The Quest 3 has no eye tracker, and it is unclear which gaze metrics are reliable enough to trust.

solution

A low-cost infrared camera placed under the headset lens films the eye. A Python tracker on the Mac fits the pupil outline in about 10 ms per frame, even when the pupil appears half bright and half dark from the LEDs, and streams gaze to the headset over Wi-Fi. In Unity, a 9-dot calibration maps pupil position to gaze direction and reports its own error.

I first built a VR workbench for testing virtual tools: shaping clay, breaking rock with a hammer, sawing wood. The obvious next question was where people look while they use it. Do they watch the tool, the material or their hand?

The hardware

The camera module has no IR-cut filter, so it sees the eye in infrared. Mounting it under the lens and aiming it before the glue set took more attempts than the code. A camera that looks up from too far below sees only a flat sliver of pupil.

What the data shows

Calibration fits to 0.8°, but an independent 25-point grid shows about 5° in the centre, and the error grows as the headset slips on the face. Precision stays under 1°, so the gaze is stable but offset. At that accuracy, material blocks and the workbench took 78 % of fixation time and tools only 6 %, because a tool handle is smaller than the error. Eye and head direction differed by more than 10° in 63 % of samples, so head tracking alone would often name the wrong object.

What I take from it

The most interesting moment was a rule break: hitting the marble with a hammer. The gaze landed next to the marble, so this setup cannot say whether I looked at it. That is the kind of question a training team would ask, and it shows how much accuracy you need before trusting the answer. I wrote this up in the repository as a blueprint of which gaze metrics a setup like this can support.

Solo build: hardware, pupil tracking, Unity toolkit and analysis. One user, so the numbers are examples of what the pipeline can measure, not findings about people. Next steps are gaze from headsets with built-in eye tracking through OpenXR, and comparing VR with the same task at a real workstation.