Chatbot Doctors Without Regulators

Short summary

Studies in Rwanda and Pakistan showed LLMs outperforming local clinicians in diagnostic tasks, sparking excitement about AI in underserved healthcare. But an Oxford study published days later revealed that when lay users interacted with the same models, accuracy collapsed to 34.5% — worse than unaided search. The gap between benchmark performance and real-world outcomes raises uncomfortable questions about deploying unregulated AI medical chatbots without oversight.

• •LLMs beat local clinicians in Rwanda/Pakistan diagnostic studies
• •Oxford study showed accuracy collapsed when lay users interacted with same models
• •Same condition produced contradictory advice — highlighting communication breakdown and regulatory gaps

Generated with AI, which can make mistakes.