A blind pairwise evaluation framework designed to benchmark frontier vision models on clinical and cultural accuracy across rural Indian public health scenarios.
Frontier vision models (GPT-4 Vision, Gemini 1.5) generate public health imagery with Western assumptions and cultural hallucinations—depicting digital bathroom scales in primary Anganwadi clinics where hanging spring scales are mandatory, rendering garbled Devanagari lettering, and misrepresenting ASHA worker attire.
Health workers and policymakers cannot trust AI-generated clinical infographics or educational materials if the equipment and cultural cues are fundamentally erroneous.
Most LLM evaluation platforms (such as LMSYS Chatbot Arena) rely on general text queries and tech-savvy crowdworkers. No benchmark evaluated whether an Anganwadi growth monitoring session or maternal clinic was depicted with medical and sociological truth.
Evaluators are presented with two unlabeled AI-generated images side by side from the same standardized clinical scenario (e.g. Anganwadi infant growth monitoring). By hiding model identities, evaluators judge purely on real-world authenticity without brand bias.
Tradeoff: Evaluating side-by-side takes longer than single-image ratings, but produces statistically significant ELO rankings with zero evaluator preconceptions.
Replaced subjective 5-star ratings with concrete binary and 3-point verifications: (1) Medical Accuracy (scale type, MCP health register presence), (2) Cultural Fidelity (ASHA worker sari pallu, courtyard setting), and (3) Environmental Authenticity (vernacular wall art).
Tradeoff: Restricting options to structured criteria eliminates evaluator variance across regional healthcare centers.
Grassroots evaluators are given upfront transparency on data anonymization, time commitment estimates, and guaranteed instant discard of abandoned sessions.
Tradeoff: Session completion rate dropped 8% due to upfront consent hurdles, but data reliability and ethical compliance reached 100%.
JANEVAL established the first documented empirical benchmark demonstrating that frontier vision models fail in up to 72% of rural Indian clinical imagery tasks without localized prompting scaffolds.
"In hindsight, I would have integrated real-time audio playback for vernacular prompts directly into the mobile interface, as rural evaluators often found spoken regional dialect descriptions more natural than reading technical English benchmark rubrics."