Jeevia
Multilingual triage assistant for Indian clinics: patients check in by voice, reports are read offline, and clinical rules (not AI) decide who’s seen first.
- Video of it working
- Problem, approach and result explained
- Public source code
- Expert reviewed· 6 reviews
100%
Lab values read exactly right on scanned reports (synthetic held-out set)
0 of 3,425
Wrong lab values left unflagged (synthetic held-out set)
22
Indian languages on staff screens
Walkthrough
Overview
The problem
Picture a primary health centre in Odisha on a Monday morning. A hundred and fifty people are waiting, and there’s one doctor and two nurses. Patients speak Odia, Hindi, Santali or Bengali, and some can’t read. Many bring a plastic bag of old lab reports and a doctor’s handwritten slip. Whoever is loudest, or happens to be first in the line, gets seen first. A child with a fever and a convulsion can end up waiting behind a routine check-up.
That’s the gap we went after. Indian government and institutional clinics see far more patients than doctors can talk to properly. Three things make it worse:
Language: the patient and the staff often don’t share one. Paper: the information that matters is buried in reports and handwriting. No fair, visible way to decide who goes first.
We set out to build a triage assistant that:
lets every patient explain their problem in their own language; turns that, plus their reports, into a clear note for the doctor; puts the sickest person at the top of the queue.
From the start we were clear on what it must not do: it doesn’t diagnose or prescribe, and AI never decides who’s urgent.
The approach
We built Jeevia, which covers three moments in a visit.
- Check-in. The patient checks in on a waiting-room tablet, on their own phone by scanning a QR poster, or with a nurse.
They choose their language. We never guess it from the region. They speak, type or tap their symptoms, then hear them read back and confirm. Speech is converted to text by AI4Bharat’s IndicConformer running on the facility laptop, and a second engine double-checks what it heard. Lab report photos are read offline by two OCR engines, PP-OCR and docTR.
[Image: screenshot of check-in, “Get your token”]
-
Prioritisation. A fixed rules engine sets urgency and shows the exact reason: Critical, Semi-urgent or Routine. The rules follow the AIIMS triage protocol, IMCI for children under five, and maternal red flags.
-
Review and hand-off.
Nurses add vitals at the bedside. If a danger value is confirmed, the rules raise the urgency themselves. Nurses can alert the doctor. The alert stays until the doctor acknowledges it, and escalates if nobody does. Doctors confirm or edit the note and can refer the patient with a QR summary. Every view and every change is written to a tamper-evident audit log.
The hardest decision: AI doesn’t decide urgency. It would have been easy, and impressive in a demo, to let a language model read the symptoms and assign a priority. We chose not to:
A model can be confidently wrong. In triage, a confidently wrong “routine” can kill someone.
So AI only helps where a human checks the result: hearing speech, translating, reading reports and summarising. Urgency comes only from rules a clinician can read and audit. Only a doctor can change it, and only with a written reason.
Other things we chose not to do:
No guessing values. If the two OCR engines disagree on a lab value, we don’t pick one quietly. The reviewer is asked to check it, and the rules use the more worrying reading in the meantime. No cloud by default. Speech, translation and OCR all run on one laptop with no GPU, because many clinics have poor internet. Online services are used only when the patient has agreed to AI help. That includes Reducto, which reads handwritten prescriptions. No real patient data. Everything in the project is synthetic or from public datasets.
The result
The full flow works end to end: patient check-in, doctor queue, nurse vitals and alert, doctor confirmation and referral. [Video: 4-minute walkthrough]
Every number below comes from a script in our repo, run on a laptop with an i5 processor and 16 GB of RAM, no GPU.
What we measured Data Result Lab values read exactly right 3,425 values on synthetic Indian lab reports, never seen while tuning Scans 100%, phone photos 99.6%, poor photos 82% (and a retake is asked for on about half of those) Wrong values that went unflagged Same set 0 of 3,425 Lab values on real reports Eka Care’s doctor-labelled Indian lab reports, 163 scored 95.4% of the values we read were exactly right Handwritten prescriptions 500 public prescriptions (bilingual Bangladesh set), read by Reducto 58.7% of medicines found (1,217 of 2,073). The nurse confirms every one. Speech recognition, Odia Public FLEURS clips 21% word error rate, 5.5% character error rate, offline on a CPU
Against a baseline:
Two OCR engines against one: on the same held-out reports, unflagged wrong values fell from 10 to 3. Adding arithmetic checks across related values (for example, the parts of a blood count must add up) brought the newer test set to 0. Before and after tuning: our first version read only 70.5% of values right on phone photos. Fixing the problems we found took that to 99.6%. Offline readers on handwriting: on an earlier public set of handwritten prescriptions, the offline OCR engines correctly named fewer than 5% of medicines. That’s why we moved to an online reader, and only with the patient’s consent.
How we measured:
Synthetic reports are generated with their true values recorded, so every value can be checked exactly. For real lab reports we used the Eka Care validation set and its doctor-written answers. Scoring is plain number comparison, with no AI judge. For prescriptions, we compared against each dataset’s own answers. Those answers contain misspellings, so 58.7% understates how well Reducto actually reads.
Reflection
What’s unfinished or known-broken
Silent errors on real lab reports aren’t at zero. About 7 values were wrong without a flag. The main cause: numbers printed inside units, such as the “3” in “10³/µL” or “3rd generation”, were read as the result. We know the fix but haven’t shipped it. Wrong strengths on some automatic prescription picks. In about 6 of 78 cases, the system picked a medicine automatically at the wrong strength, for example 0.5 mg read as 2 mg. Until that’s fixed, those picks have to be checked by the nurse like everything else. Reminder calls don’t reach Indian numbers. The free trial accounts we used (Twilio and Vonage) can’t call India. SMS reminders do work. The knowledge-base triage engine and the guided intake conversation are on a branch, not yet merged into the main app. No real patients yet. Nothing has been tested with real patients or real clinic audio.
What we’d do differently
Test on real-world data from week one. Our synthetic lab reports said 99.6%, and the first real set said about 95%. That gap taught us more than any synthetic test did. Measure safety errors before accuracy. “Wrong and not flagged” turned out to be the number that matters, not “percent right”.
What we learned
In healthcare, a system that knows when it’s unsure is worth more than one that’s slightly more accurate. Showing every value next to the crop of the report it came from cost us extra work. It’s also what makes a nurse able to trust the screen. Honest numbers, with the dataset and the method beside them, are more convincing than big ones.
“AI helps. Rules decide. Clinicians confirm.”
Submitted to The Multimodal Triage Assistant
- Your idea in one line
Jeevia listens to a patient in their own language, reads their reports and photos, and gives the nurse or doctor a checked triage note before the consultation. Urgency is set by published medical rules, never by the AI, and a qualified person confirms every note.
- What will be working by the finale?
Everything shown at the mid-eval, fully working and rehearsed end to end: A patient registers at a kiosk or on their phone and gives consent; a family member can answer for them. They describe their problem by tapping, typing or speaking in any of India’s 22 scheduled languages. Lab reports and prescriptions are photographed or uploaded and read automatically. Each value is checked and linked back to the image it came from. Every patient gets a RED, YELLOW or GREEN urgency tag. Each tag shows the exact rule and source behind it (AIIMS Triage Protocol, WHO IMCI and others), and the queue sorts the sickest patient to the top. A short note for the doctor, with what’s missing and what each role should ask next. Nothing is final until a nurse or doctor confirms it, and a health worker cannot sign off a RED case alone. Referral packets, a QR summary for the patient, and a FHIR export for health records. The same app adapts to a PHC, hospital, clinic, health camp, college campus or factory clinic, each with its own staff roles and questions. It keeps working offline at a health camp and syncs when the network returns. By the finale we also plan to add review time measured with a clinician, native-speaker checks of the Odia, Hindi and Kannada wording, and a recorded set of team voice samples for speech accuracy.
- Tech stack and hardware you plan to use
Frontend: Next.js (an installable web app that works offline), Tailwind Backend: Python FastAPI, a rules engine of 167 cited rules, PostgreSQL (Neon) or SQLite offline, Cloudinary for files AI models (open source, all running offline on the laptop): IndicConformer for speech in 22 languages IndicTrans2 for translation RapidOCR and docTR for reading reports Qwen3-4B to write the note summary Qwen3-VL-4B for photos YuNet to blur faces MMS-TTS for spoken prompts Optional online helper, only with the patient’s consent: Sarvam AI for a second speech check and voice. Hosting: Vercel (website) and Render (API). Hardware: one ordinary 16 GB laptop runs every model with no internet; a GPU is optional. Patients can use any phone or tablet as a kiosk.
- What have you built or tested so far?
The full prototype is built and live at ps-03-bput.vercel.app. Code: github.com/Jyoti0525/PS03_BPUT- What we’ve tested, all on synthetic data: All 167 urgency rules are tested to fire when they should and stay quiet when they shouldn’t. On a 300-patient test set, no patient who should have been RED came out GREEN. The colour stayed the same when only the language or the patient’s sex changed. Every one of 101 unsafe AI outputs we tried was blocked, such as naming a disease or suggesting a medicine, and no correct sentence was blocked. Speech accuracy is measured in 21 languages; for example, Hindi has a 10.7% word error rate. Report values are read 99.3% exactly from scans and phone photos. 1,151 automated tests in total. We also report what isn’t done yet: review time with a real clinician hasn’t been measured, and live phone calls need a paid telephony account.
- Pitch deck (PDF)
- Pitch deck (PDF)PDF
- What's working now?
Working: Patients check in on a kiosk, their phone or with a nurse, in their own language, by voice, typing or taps, and confirm what was heard. Fixed clinical rules (AIIMS, IMCI, maternal red flags) set urgency with a visible reason; AI never decides. Lab reports are read offline by two OCR engines, and disagreements are flagged. Nurses add vitals and alert the doctor, and unanswered alerts escalate. Doctors confirm notes, change urgency only with a written reason, and refer by QR. Consent, encryption and a tamper-evident audit log are in place. Partly working: Handwritten prescriptions are read by Reducto (58.7% of medicines found on 500 public prescriptions), and a nurse confirms each one. The knowledge-base engine is on a branch, not yet merged. Not done: Reminder calls to Indian numbers (blocked by the trial accounts), and testing with real patients.
- Demo video
- youtu.be/9d9X0-HRVcg?si=Uc7Pqs4m0h2_ZcCa
- GitHub repository
- github.com/Jyoti0525/PS03_BPUT-.git
- Deck or report (PDF)
- Deck or report (PDF)PDF
AI tools used
We used Claude as a coding assistant to write and review code, run evaluation scripts, and draft documentation. Every clinical rule, design decision and reported number was checked by the team and reproduced with scripts in the repo.