VeriSight
A mixed-reality assistant that helps people living with schizophrenia tell hallucinations from reality.
- Year
- 2025
- Reading
- 04 min
- Figures
- 08
VeriSight is a mixed-reality assistant for Meta Quest 3 that helps people living with schizophrenia tell what’s real. Press a button and every real person in view is outlined in blue; hold a trigger and ask a Gemini-powered voice assistant about what you’re looking at. It won Best Overall at ImmerseGT 2025.
- Role
- XR developer
- Team
- Yiming “Danny” Huang
- Lorenzo Ametrano
- Ming Jin Yong
- Justin Zhu
- Disciplines
- Mixed reality
- Computer vision
- Voice UI
- Healthcare
- Tools
- Meta Quest 3 / Unity / C# / Unity Sentis / Passthrough Camera API / Gemini Live / Python / WebSockets
- Recognition
- Winner, Best Overall — ImmerseGT 2025
VeriSight is a mixed-reality assistant that helps people distinguish hallucinations from reality through real-time visual and voice-based scene analysis.
Built for headsets like Meta Quest, it combines AI vision models, spatial awareness and a conversational interface to detect inconsistencies in the user’s environment — and gently guide them toward what’s real.
Millions face daily uncertainty because of schizophrenia. These are the numbers our pitch was built on.
80%
of schizophrenia patients experience chronic delusions
69%
lack adequate support — no caregivers, service animals or real-time grounding tools
24M
people affected by schizophrenia worldwide
When reality blurs, fear follows.
Hallucinations — visual, auditory or tactile — can be terrifying and disorienting. For people living with schizophrenia, PTSD, or even temporary psychosis triggered by trauma or substance use, it’s often hard to tell what’s real and what’s not.
Existing care leaves gaps. Antipsychotic medication isn’t adaptive and under-addresses “negative symptoms”. CBT for psychosis is expensive and centralised, so it doesn’t scale. VR-based therapy is intrusive, costly and not something you use day to day.
So instead of treating AR solely as a tool for productivity or entertainment, we asked a different question.
What if extended reality could serve as a cognitive anchor? Something that sits quietly in your periphery — but steps in when your senses betray you.”
Two outside checks, rebuilt in one headset.
The only way for someone with schizophrenia to know whether they are hallucinating is to be sceptical of what they see — or to have an external force tell them.
Service dogs can be tasked to “greet” someone their owner suspects is a hallucination. Another trick is to open a phone’s camera and point it at the thing in question.
VeriSight is a bridge between those two external forces, built inside mixed reality.
Two channels for checking what’s real.
Channel 1 — human detection. With the click of a button, VeriSight senses the space and projects a blue outline over every person it detects. No outline could mean that what you’re seeing isn’t actually there.
Channel 2 — AI scene awareness. Hold down the trigger, ask any question about what you’re looking at, and hear an answer in seconds.
“Hey Veri, how many people are in the room?” — “There is one person in front of you. You are in your living room.”
Fig. 02Channel 1 in motion: scanning a conference room.
Fig. 03The demo menu: press A to sense your environment, hold the right trigger to ask Gemini a question.
Spatial and verbal, at once.
I set up the project on the Oculus Passthrough API, designed the front-end interaction that displays real-time object detection from Unity Sentis, and tested the Gemini Live backend through our own Python server — so VeriSight takes verbal and spatial input at the same time.
How it works
- 01
People, not objects
We modified the Unity Sentis agent to detect only humans, out of the 80 object classes its model was trained on.
- 02
Anchored outlines
Each detected person’s position, combined with the y-axis of the boundary floor, places a visual on top of them. The A button on the right controller runs the scan.
- 03
Faint on purpose
Running the model on the headset caused performance issues that sometimes misaligned the overlays, so the outline is drawn faintly — not disruptive, and forgiving when it’s slightly off.
- 04
Camera and voice in
Passthrough image data and the voice recording are pulled from the Quest and sent to a server holding the Gemini API key. The model runs on the prompt and the photo together.
- 05
Speech back out
The response goes through Gemini’s speech-to-speech model and plays back in the headset — look anywhere and ask anything about it, much like Meta Ray-Bans.
Fourteen hours, one firmware update.
Unity had released its Passthrough APIs only a couple of weeks earlier, with almost no documentation. Navigating it was like walking into a minefield blindfolded — we learned the tool and its limits as we went.
Getting camera data off the headset and onto a server was its own fight. Even once our C# script talked to the API, nothing arrived on the server side. It took us 14 hours to realise the test headset wasn’t on the latest Horizon firmware. After the update, the pipeline worked as intended.
Ironically, the model still sometimes hallucinates things that aren’t in the room. We haven’t fixed that one yet.
From headset to glasses.
We designed VeriSight knowing that the Quest’s mixed-reality capabilities will one day fit into something as compact as glasses. Then it could be part of daily life — a non-intrusive, unmedicated treatment option for schizophrenia symptoms.
The team is passionate about this problem space and is discussing continued development as new Sentis models are released.
