Researchers use eye reflections to create 3D models of the worlds people see
Sci-fact is weirder than science fiction and now we can see people's eye reflections and turn those reflections using NERF into 3D models. Insane tech!
Key takeaways
- Imagine being able to reconstruct a 3D scene from the reflections in a person's eyes.
- The human eye, apart from its primary function of vision, also holds a wealth of information about the world around us.
- By imaging a person's eyes, one can collect multiple views of a scene outside the camera's direct line of sight through the reflections in the eyes1.
Cite or link to this article
Griffin, M. (2023) 'Researchers use eye reflections to create 3D models of the worlds people see', 311 Institute, 18 June. Available at: https://www.311institute.com/researchers-use-eye-reflections-to-create-3d-models-of-the-worlds-people-see/ (Accessed: 1 October 2026).
The human eye, apart from its primary function of vision, also holds a wealth of information about the world around us. If we look at someone else's eyes, the light reflected from the cornea can provide a snapshot of what they are seeing. This is the fundamental idea behind Alzayer's work. By imaging a person's eyes, one can collect multiple views of a scene outside the camera's direct line of sight through the reflections in the eyes1.
Insane Tech
However, such a task is not without its challenges. Accurately estimating eye poses and differentiating the intertwined appearance of the iris and the scene reflections present significant obstacles. Alzayer and his team address these challenges with a novel approach that refines the cornea poses, the radiance field depicting the scene, and the observer's eye iris texture. They propose a simple regularization prior on the iris texture pattern to enhance the reconstruction quality1.
The team repurposes Neural Radiance Fields (NeRF) for training on eye images, incorporating two crucial components: texture decomposition, which leverages a simple radial prior to facilitate separating the iris texture from the overall radiance field, and eye pose refinement, which enhances the accuracy of pose estimation despite the challenges presented by the small size of eyes1.
This research is an extension of the concept of catadioptric imaging, which uses a combination of lenses and mirrors for image capturing. The human eye, in this case, serves as an external curved mirror, which is a creative realization of an accidental catadioptric imaging system1.
The study's advancements extend the current capabilities of 3D scene reconstruction through neural rendering to handle partially corrupted image observations obtained from eye reflections. This opens up new possibilities for research and development in the broader area of accidental imaging, revealing and capturing 3D scenes beyond the visible line of sight1.
As the research suggests, capturing both eyeballs could indeed result in even higher volumetric fidelity, further pushing the boundaries of what we thought was possible.
FAQ
Why does this matter?
Sci-fact is weirder than science fiction and now we can see people's eye reflections and turn those reflections using NERF into 3D models. Insane tech!
About the author
Matthew Griffin Founder, 311 Institute
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath."
Read full bio
Matthew Griffin is a multi-award winning Futurist and expert in Disruption and Innovation, Geopolitics, Leadership, and Technology, who NASA have described as a "walking encyclopaedia of the future" and a "futurist Polymath." 15-time best selling author of the "Codex of the Future" series, Matthew is the Founder and Futurist in Chief of the 311 Institute, a global Futures and Deep Futures advisory firm working with royal households, world leaders, G7, G20, and G77 governments, NGOs, and multi-national mid and mega cap firms to help them explore, shape, and lead the next 50 years of business and society.
An award-winning YouTube creator with over a million followers, with an unrivalled global reach and impact, Matthew is a highly sought-after international keynote speaker, lecturer, and mentor who collaborates with global leaders through the United Nations Alliance of Civilizations (UNAOC) and United Nations General Assembly (UNGA) to shape pivotal initiatives such as the UN’s AI for Humanity program, the United Nations Conference of the Parties (UN COP), and the World Economic Forum in Davos.
As the former Global Head of Cloud, National Security, and Enterprise Sales for companies including Atos, Dell-EMC, and IBM, Matthew has a proven track record of building multi-billion dollar business units and turning failing divisions into market leaders. His ability to identify, analyse, and communicate the implications of hundreds of emerging technologies and trends is unparalleled, and his insights are trusted by many of the world’s most respected organisations, including ABB, Accenture, Adidas, AON, ARM, BCG, Centrica, Citi, Coca-Cola, Dentons, Deloitte, Dow Jones, EY, Google, KPMG, Lego, Legal & General, LinkedIn, Microsoft, PepsiCo, Qualcomm, RWE, Samsung, Siemens AG and Siemens Energy, T-Mobile, UBS, VISA, Walmart, Workday, Worldpay and many others.
Regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, Matthews mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.
What future do you need to see?
Choose one to get started on virtual worlds and the future of your organisation.
Sources and further reading
- 2306.09348 arxiv-vanity.com
Source: first published by the 311 Institute on 18 June 2023. Cite as: Griffin, M. (2023). Researchers use eye reflections to create 3D models of the worlds people see. 311 Institute. https://www.311institute.com/researchers-use-eye-reflections-to-create-3d-models-of-the-worlds-people-see/
You are welcome to quote this article with credit and a link to the original.