I Replaced $3,000 Eye-Trackers with a Laptop Webcam: Building a Hands-Free Spatial Communicator with SmartSpectra

This is a submission for the Presage SmartSpectra SDK Challenge

What I Built

Traditional Augmentative and Alternative Communication (AAC) systems for people with severe motor and speech impairments (such as ALS, high-level quadriplegia, or locked-in syndrome) force an impossible trade-off: spend $3,000+ on specialized infrared eye-tracking hardware, or endure exhausting dwell-time interfaces where staring at a button for 1,000 ms just to register a single click leads to rapid visual fatigue and constant accidental selections (the classic "Midas Touch" problem).

Worse, existing gaze communicators suffer from two real-world bedside failures that nobody talks about:

1. Attention Theft: The screen holds the user's eyes hostage. If they glance up to make eye contact with a loved one or try to vocalize, the cursor misfires and types gibberish.
2. One-Sided Visibility: Caregivers and family members standing across the room can't read a laptop screen facing the patient.

NeuroScribe Studio turns any standard consumer webcam into a responsive, spatial assistive communication console by decoupling targeting from activation, introducing conversational standby gating, and syncing messages in real time to a Caregiver Partner HUD (/partner).

Core Capabilities

• 3D Spatial Gaze & Head-Pose Targeting: High-contrast, multi-level hierarchical communication boards (Physical Needs, Comfort & Position, People & Care, Quick Dialogue, Expressions, Emergency) driven by smoothed iris vectors and head pitch/yaw.
• Temporal Blink Discrimination: A client-side state machine separates natural involuntary blinks (< 250 ms, ignored) from deliberate selection blinks (350–850 ms, confirms card) and extended closures (> 1,200 ms, undo/backspace).
• Attentive Standby Lockout: Automatically freezes card inputs the instant face.talking is detected or the user tilts their head up (pitch > 14°) to make natural eye contact with a caregiver.
• Interactive 5-Second Calibration: Dynamically normalizes iris bounding thresholds (minX, maxX, minY, maxY) and blink timing windows to adapt to different seating angles, wheelchair heights, and eyelid control speeds.
• Cross-Device Caregiver Mirror (/partner): Real-time WebSocket sync broadcasts composed phrases, spoken audio chimes, and haptic vibration alerts (navigator.vibrate) to a caregiver's smartphone across the room—complete with two-way acknowledgment back to the patient.
• Human-in-the-Loop Speech Synthesis: Messages are never spoken automatically; users review their constructed sentence and confirm [Speak Aloud] to trigger the browser's native Web Speech API.

Demo

What Happens in the Demo:

1. Consent & Privacy Gate: Explicit pre-capture authorization explaining local in-memory processing and zero telemetry.
2. Hands-Free Spatial Navigation: Navigating the 3D spatial wave-cards via eye gaze and selecting phrases with deliberate blinks—with hands completely off the keyboard.
3. Attentive Standby Lockout: Looking up and speaking freezes the grid so natural eye contact with a caregiver never causes accidental selections.
4. Caregiver Mirror Sync (/partner): Confirming Speak Aloud broadcasts the message in real time to a second device with audio and haptic feedback.

Code

• Public GitHub Repository: https://github.com/Akulbanxal/Neuroscribe

How It Responds to Human Signals

Rather than treating SmartSpectra as a passive telemetry display, NeuroScribe Studio orchestrates four live Presage human signals in real time to power a hands-free, zero-fatigue assistive communication loop:

1. Iris Vectors & 478 Facial Landmarks (faceMetrics) → Spatial Card Targeting & 3D Parallax:

   • Normalized iris coordinates ([0, 1]) and head pose (pitch and yaw) are streamed from the local SmartSpectra Node SDK daemon (@smartspectra/node-sdk) at 30 FPS over WebSockets (ws://localhost:8080).
   • To eliminate raw webcam jitter, the frontend applies an Exponential Moving Average (EMA) filter (α = 0.28–0.35) and maps the smoothed gaze vector onto calibrated 3×2 spatial wave-cards with deadzones, dynamically tilting the 3D glass deck in response to head orientation.

2. Binary Blink Detection (face.blinking) → Temporal Selection & Undo State Machine:

   • Instead of exhausting dwell-time timers (staring at a button for 1,000+ ms), NeuroScribe measures the exact duration of continuous face.blinking.detected windows on the client side:
     • Natural involuntary blinks (< 250 ms): Ignored completely to prevent accidental clicks.
     • Deliberate selection blinks (350–850 ms): Fills a live neon progress bar on the targeted card and appends the phrase to the sentence builder.
     • Extended eye closure (> 1,200 ms): Triggers an instant Undo / Backspace action.

3. Speech Articulation Detection (face.talking) & Head Pitch → Attentive Standby Lockout:

   • The moment face.talking.detected flips to true—or when the user lifts their chin (pitch > 14°) to make natural eye contact with a bedside caregiver—the UI immediately enters Attentive Standby, freezing all card selections and dimming the grid so real-world conversation never triggers false inputs.

4. Respiratory Waveform (breathing.upperTrace) & Facial Expressions (faceMetrics.expressions) → Live HUD & Ambient Feedback:

   • Real-time upper-chest respiratory excursions drive the rhythmic breathing sparkline in the system status bar, while dominant facial expression probabilities subtly adapt the ambient spatial lighting.

SDK Integration Architecture

• Backend Bridge (server.mjs): Initializes SmartSpectraSDK locally via C++ FFI with requestedMetrics: [...faceMetrics, ...breathingMetrics] and enableTelemetry: false, normalizing all frame timestamps to epoch microseconds (Date.now() * 1000) to guarantee continuous, stall-free streaming.
• Frontend Consumer (useBiometrics.ts): Ingests the 30 FPS WebSocket JSON stream, applies EMA coordinate smoothing, runs the temporal blink discriminator, and synchronizes confirmed phrases to both the Web Speech API (speechSynthesis) and the cross-device Caregiver Mirror (/partner).

Prize Categories

I am entering NeuroScribe Studio for Best Thoughtful Real-World Use (Primary) as well as Best Overall Human-Sensing Experience.

Best Thoughtful Real-World Use

• The Real-World Need: Millions of individuals living with severe motor and speech impairments—such as ALS, high-level spinal cord injuries, cerebral palsy, or brainstem stroke—lose the ability to speak or use touchscreens. Existing Augmentative and Alternative Communication (AAC) setups either cost $3,000+ for dedicated infrared hardware or rely on exhausting dwell-time timers (staring at a button for 1,000+ ms to click) that cause rapid ocular fatigue and constant accidental inputs.
• Who It’s For: Non-speaking or motor-impaired individuals who need a reliable, zero-cost way to express urgent needs, comfort adjustments, and everyday thoughts using only a standard laptop webcam—alongside their bedside caregivers and family members who need immediate visibility across the room.
• How It Was Designed for Real-World Usability:
  • Zero-Fatigue Temporal Blink Activation: Decouples eye-gaze targeting from intentional blink confirmation (350–850 ms), ignoring involuntary natural blinks (< 250 ms) and mapping extended closures (> 1,200 ms) to Undo.
  • Attentive Standby Lockout: Automatically freezes card inputs when face.talking is detected or when the user tilts their head up (pitch > 14°) to make natural eye contact with a loved one—eliminating accidental selections during real-world conversation.
  • 5-Second Adaptive Calibration & High-Contrast Spatial Cards: Large, unmistakable 3×2 wave-cards with customizable blink thresholds and seating calibration accommodate varying wheelchair angles and eyelid control speeds.
  • Cross-Device Caregiver Mirror (/partner): Broadcasts composed phrases, spoken audio chimes, and haptic vibrations (navigator.vibrate) to a caregiver's smartphone in real time, complete with two-way acknowledgment back to the patient.

Consent and Data Handling

NeuroScribe Studio was architected from the ground up with strict privacy-first guardrails and full compliance with the challenge rules:

• Explicit Pre-Capture Consent: Camera access and WebSocket biometric streaming remain completely disabled at launch until the user explicitly reviews and accepts the on-screen Camera Consent & Privacy Notice Modal (with a one-click option to revoke/pause capture at any time).
• 100% Local In-Memory Processing (Zero Storage / Zero Sharing): Video frames are captured locally by the Node.js daemon (@smartspectra/node-sdk), processed strictly in volatile RAM, and immediately discarded. Zero video frames, facial images, landmark recordings, or biometric logs are ever saved to disk, persisted in databases, or transmitted to external servers.
• Retention & Deletion: Because all processing occurs ephemerally in volatile memory with zero disk persistence, data retention is 0 seconds—nothing is stored, so no residual user data ever remains after closing the session.
• SDK Telemetry Disabled: The SmartSpectra SDK instance is explicitly initialized with enableTelemetry: false, ensuring no diagnostic session telemetry or usage metrics leave the local machine.
• No Third-Party Cloud AI Services: All gaze smoothing, blink classification, sentence assembly, and text-to-speech synthesis (window.speechSynthesis) run 100% locally on the user's device without calling external AI APIs.
• Non-Medical & Non-Diagnostic Disclosure: NeuroScribe Studio is strictly an assistive human-computer interaction (HCI) communication interface. It makes no medical, diagnostic, or treatment claims, performs no clinical decision-making, and does not enable covert or non-consensual monitoring.

Agent Session

Built solo using React 19, TypeScript, Vite, Tailwind CSS, and @smartspectra/node-sdk, with AI-assisted architectural iteration and UI prototyping.