11 / 12 · Research · 2024
Face Tracking with HMDs
Feasibility study on tracking faces in real time from a headset user’s view.
- Erasmus+ research internship
- Feasibility study
- Role
- Researcher
- Where
- University of Salerno
- Year
- 2024
Illustration
How it works
Screen 01
What the headset sees
Varjo XR3 passthrough with faces boxed in real time; dlib, OpenCV and VGG Face compared.
Presenting? ← → walk through this page and on to the next project.
Built with
Every layer of the stack, and what it does in this project.
- L1Hardware
Passthrough view of the people around the user.
- L2Engines
Where the tracking had to sync.
- L3Vision
Three approaches compared.
- L4Language
OpenCV detection on the Python side.
What I did
Targeted a real gap: almost all existing work tracks the HMD user’s own face; tracking the people around the user was largely unexplored.
Tested three approaches: dlib failed on hardware incompatibility; OpenCV detected faces on the Python side but hit performance and compatibility problems when synced with Unity/Unreal; VGG Face was strongest on paper but hit the same integration issues.
Concluded it wasn’t feasible then: the Varjo SDK didn’t support face tracking directly, expert resources were limited, and EU research and data constraints applied.
Authored the project’s technical report.
Team & thanks · Thanks to Prof. Alessandro Naddeo, Dr. Carlotta Fontana, PhD researcher Alfonso Magliano and the whole team.
Where to next?