JANSAATHI AI
Multilingual healthcare triage kiosk fusing regional voice intake, live ESP32-S3 vitals, and lab OCR into a doctor-verified priority dossier.
- Problem, approach and result explained
- Expert reviewed· 6 reviews
—
Complete multimodal patient intake & 30s vitals lock
—
Indian languages supported for voice & form triage
—
Deterministic red-flag safety lock before doctor review
Overview
The problem
Walk into any Primary Health Center (PHC), rural dispensary, or campus medical clinic across India during morning OPD hours, and you see the exact same bottleneck: over a hundred patients waiting outside a single consultation room while one overwhelmed Medical Officer has less than two minutes per patient. Patients arrive describing symptoms in regional languages—like Odia, Hindi, Bengali, or Telugu—carrying crumpled paper prescriptions or handwritten lab reports, with zero baseline vitals taken beforehand. When an asymptomatic hypoxic patient with 88%\text{ SpO}_2 or someone quietly developing sepsis sits at the back of that queue, they wait just as long as someone who came in for a routine skin rash. We set out to build JanSaathi AI: a human-in-the-loop multimodal intake kiosk and clinical review station designed specifically for institutional and public health facilities across India. Our goal was simple and strict: never replace the doctor, never diagnose, and never prescribe—instead, capture regional speech, live physical vitals, and existing medical documents at the front desk, organize everything into a structured, urgency-ranked triage note, and hand it to a qualified Medical Officer for fast, safe review.
The approach
We built a hardware-coupled, dual-mode triage system running across an ESP32-S3 N16R8 biometric node and a full-stack FastAPI + React/TypeScript clinical workstation.
- The Physical Kiosk Node (ESP32-S3 N16R8): Instead of relying on expensive proprietary medical carts, we wired five low-cost sensors onto a single ESP32-S3 board communicating over USB CDC serial at 921,600 baud:
- An INMP441 I2S digital microphone (GPIO 4, 5, 6) so patients speak directly into the kiosk unit rather than a laptop mic.
- A MAX30102 pulse oximeter (GPIO 8, 9, 10) for heart rate and \text{SpO}_2.
- A DS18B20 1-Wire probe (GPIO 7 with a 4.7\text{ k}\Omega pull-up) for body temperature.
- An AD8232 single-lead ECG sensor (GPIO 1, 2, 3) streaming a 50 Hz waveform with leads-off detection.
- A 0.96" I2C OLED display (0x3C sharing GPIO 8/9) plus an onboard WS2812 RGB NeoPixel (GPIO 48) to show patients a live 30-second sensor countdown and status feedback.
[DROP CLIP HERE: Hardware Kiosk Close-Up — Finger placed on MAX30102, 0.96" OLED counting down from 30s, and live 60 FPS ECG waveform drawing on the web screen]
- Dual-Mode Intake ("Ananya" Voice Assistant + Live Synchronized Form): We designed a 6-step wizard supporting 11+ Indian languages. Patients can speak naturally in their own dialect—transcribed via Sarvam AI / Gemini 2.5 Flash and answered in a warm, empathetic female nurse voice—while a manual form on the right stays synchronized in real time. If a patient wants to edit a single detail at the end ("Change my age to 28"), selective delta-patching updates just that field without restarting the session.
[DROP SCREENSHOT HERE: Patient Portal Dual-Mode Screen — Language selector & Step 2 split-view showing conversational chat on the left syncing live into the clean white-orange-black manual form on the right]
- The Hardest Architectural Decision — Hybrid ML + Deterministic Safety Guardrails: Early on, we had to decide how patient urgency should be calculated. Pure rule-based thresholds miss subtle multi-factor risks (like an elderly patient with borderline 93%\text{ SpO}_2, elevated heart rate, and chest tightness), while letting an LLM or pure ML model decide acuity alone risks hallucinating or under-triaging an emergency. We solved this by pairing a LightGBM + Clinical NLP (TF-IDF) model trained on 100,000 Emergency Severity Index (ESI) records with hard deterministic physiological guardrails—if \text{SpO}_2 < 91%, \text{Temp} \ge 39.4^\circ\text{C}, or acute red-flag symptoms appear, the system locks the case to CODE RED regardless of what any model predicts.
- What We Deliberately Chose Not to Do: We strictly forbade the AI from generating disease diagnoses, differential probabilities, or medicine recommendations—even when a patient directly asks "Can I take medicine for this?" Every output remains strictly advisory and reviewer-facing.
The result
When a patient completes intake, the system locks in their 30-second averaged vitals, assigns a priority tier (CODE RED, CODE YELLOW, or CODE GREEN), issues a unique Patient ID and tracking passcode, and generates a downloadable ₹0.00 Government OPD Token & Triage Slip PDF.
- Intake Speed & Structuring: Compared to unstructured manual OPD registration—where a Medical Officer spends 2–3 minutes just asking basic history, checking vitals, and reading loose lab slips—JanSaathi completes multilingual demographic capture, targeted follow-up questioning, report OCR extraction, and a 30-second multi-sensor vitals lock in under 90 seconds before the patient ever steps into the doctor's room.
- Clinician Review & Escalation: On the Doctor Review Station, cases automatically sort by clinical urgency rather than arrival order. The doctor sees the 30-second averaged vitals, chronological symptom summary, abnormal lab biomarkers highlighted side-by-side against the uploaded report photo, a missing-information checklist, one-click triage override with an immutable audit log, and a one-click District Headquarter Hospital (DHH) / Medical College Referral Slip generator.
Reflection
- What We Would Do Differently: In our initial build, we coupled the ESP32-S3 telemetry too tightly to cloud speech APIs and hit real-world latency and °F/°C unit-conversion mismatches between the firmware and backend triage rules. If we started over, we would standardize all internal sensor payloads to strict SI units (°C, bpm, %) from day one and run local voice activity detection (VAD) directly on the ESP32-S3 before streaming audio frames.
- What Is Unfinished or Known-Broken: Right now, our WhatsApp status notification pipeline runs as an institutional webhook simulator rather than a live Meta Business API integration, and offline clinic mode still relies on browser WebSerial rather than a local edge-hosted whisper/LLM container when facility internet drops completely.
- What We Learned: Building for Indian public healthcare taught us that reliability and clinical restraint matter more than flashy AI autonomy. Doctors don't want an AI trying to play physician; they want clean, verified vitals, abnormal lab flags pulled out of messy paper reports, and ironclad safety boundaries so no critical patient ever gets lost in the crowd.
Submitted to The Multimodal Triage Assistant
- Your idea in one line
JANSAATHI AI is an AI-powered frontline clinical triage and medical hardware telemetry workstation that empowers community health workers (ASHA/ANM) and rural clinics to capture multimodal patient symptoms (vernacular voice, paper lab OCR) and stream real-time physiological vitals (Lead-I ECG, SpO2 PPG) into a physician-verified telemedicine diagnostic dossier.
- What will be working by the finale?
Plug-and-Play Medical Hardware Telemetry Stream: Real-time browser ingestion via Web Serial API (115200 baud) connecting an ESP32 microcontroller with a MAX30102 pulse oximeter (SpO2 & arterial plethysmography) and AD8232 sensor (Lead-I ECG at 100 Hz), with active lead-off detection. Standard 15-Second Diagnostic Rhythm Strip & Electronic Calipers: Live 25 mm/s, 10 mm/mV dual-trace display (Lead-I ECG + PPG) with real-time electrophysiological calipers calculating Mean R-R, PR interval, QRS width, Bazett-corrected QTc, and Perfusion Index (PI). Multimodal Clinical Intake (Voice, OCR, Text): Speech-to-text in Indian vernacular languages (Hindi, Tamil, Telugu, Bengali, Marathi, etc.) paired with Gemini Vision OCR for paper diagnostic lab reports and handwritten prescriptions. Hybrid Clinical Decision Support & Emergency Red-Flag Triage: Dual-engine safety architecture combining Google Gemini 2.5 Flash (structured clinical narrative extraction) with deterministic clinical safety rules that flag immediate life threats (acute coronary syndrome, respiratory distress, shock). Physician Tele-Triage Workstation & Printable IEC 60601-2-25 Diagnostic Dossier: Doctor dashboard with prioritised queues, case review, clinical notes, and instant one-click generation of high-resolution, print-ready PDF clinical reports. Built-in Living Physiological Simulator: Fallback real-time simulation displaying respiratory sinus arrhythmia (RSA) and pulsatile dynamics for uninterrupted demonstrations even without physical USB hardware.
- Tech stack and hardware you plan to use
Hardware & Sensors: ESP32 DevKit V1 (Dual-core Xtensa 32-bit LX6 @ 240 MHz). AD8232 Single-Lead ECG Front-End with 3-lead Ag/AgCl disposable snap electrodes (Lead-I RA/LA/RL placement). MAX30102 High-Sensitivity Pulse Oximeter & Heart-Rate Sensor (I2C interface with optical AC filtering and ambient light cancellation). Native Web Serial API (navigator.serial) via USB-UART bridge (CP2102/CH340)—zero driver installation needed. Frontend & Visual Architecture: React 19, TypeScript 5.8, Vite, Tailwind CSS, and modern Claymorphic clinical design tokens. Custom HTML5 Canvas rendering engine delivering 60 fps continuous waveform scrolling across standard IEC millimeter medical grids. AI Models & Backend: Google Gemini 2.5 Flash (via @google/genai SDK) for structured clinical extraction, vernacular translation, and dynamic follow-up questioning. Gemini Live Multimodal API (WebSockets) for real-time conversational clinical interview intake. Supabase (PostgreSQL, Row-Level Security, Realtime subscriptions, and Storage for reports). Validation & Quality Assurance: Vitest test suite with 21 unit and integration tests verifying packet parsing, deterministic emergency triage rules, and schema conformity. Items still needed / optional stretch: Handheld 3D-printed enclosure for the sensor probe wand, and an optional Bluetooth Low Energy (BLE) module for wireless operation.
- What have you built or tested so far?
Working Hardware Firmware: Written and tested in Arduino C++ ( jansaathi_esp32_vitals.ino ). Features optical AC tracking, 25-sample noise debouncing, and dual-channel streaming at 100 Hz (ECG) and 50 Hz (PPG). Zero-Driver Web Serial Gateway: Direct browser connectivity ( HardwareGateway.ts ) that decodes JSON/Key-Value telemetry and handles sensor disconnection and COM port busy states. High-Resolution 15-Second ECG & PPG Waveform Graph: Custom HTML5 Canvas engine ( EcgWaveformGraph.tsx ) supporting 5.0s zoom (25 mm/s standard), 1.0x/2.0x/Auto-fit gain, and real-time electronic calipers for R-R, PR, QRS, QTc, and PI. Active Physiological Simulation Engine: Realistic respiratory sinus arrhythmia (RSA) causing resting heart rate to undulate naturally between 72–76 bpm over a 4.0-second breathing cycle with synchronized Pulse Transit Time (~180ms delay). Automated Test Suite: 21 passing automated tests in Vitest covering serial stream parsing, deterministic emergency flags, and Gemini schema conformity.
- Pitch deck (PDF)
- Pitch deck (PDF)PDF
- GitHub repository
- github.com/b2bhau570/jansaathi-ai
- What's working now?
1. What is Working: • Multilingual Voice Intake & Biometrics: Autonomous patient triage in Odia, Hindi, English, and 21+ Indic languages using Sarvam AI Saaras STT, Gemini 2.5 Flash conversational orchestration, and Neural Studio TTS. Voice captures patient demographics, height/weight with automatic BMI calculation, and chief symptoms while ambient clinic BGM plays during AI thinking turns. • ESP32-S3 Medical IoT Kiosk: Live 921600-baud WebSerial telemetry streaming 50 Hz AD8232 continuous ECG waveforms, MAX30102 pulse oximetry (HR bpm and SpO2 %), DS18B20 digital temperature (°F/°C), and 30-second countdown with live readings on a local SSD1306 0.96" OLED display. • Automated 30-Second Vitals Averaging: Computes rolling arithmetic means across 30 seconds of sensor contact, eliminating false single-point sensor spikes. • Smart Camera Document OCR: Real-time paper contour detection, 4-point perspective rectification, and Gemini Multimodal OCR extracting lab test values and prescriptions. • ML Clinical Triage: Trained LightGBM classifier categorizing patients into 4 color-coded clinical priorities (Emergency, Urgent, Elevated, Routine) alongside ICD-10 differential diagnosis hints. • Doctor Clinical Command Center: Attending physician workspace with QR-code & 6-digit OTP patient access, real-time ECG strip charts, structured English summaries, voice guidance recording, and digital prescription fulfillment.
- GitHub repository
- github.com/b2bhau570/JanSaathi_Web
- Deck or report (PDF)
- Deck or report (PDF)PDF
AI tools used
Used Gemini for multimodal prescription/lab report OCR extraction, multilingual Indian voice intake structuring, and architectural debugging.