Last updated: October 4, 2026
Typing symptoms into an app or chatbot before deciding whether to seek medical care is becoming increasingly common. These tools can suggest possible conditions and estimate how urgently someone may need care, but their accuracy varies considerably. The research paints a more mixed picture than the marketing often suggests.
This article reviews published studies that tested symptom checkers and one general-purpose language model on standardized clinical cases. It does not recommend any product. Instead, it examines what the evidence shows, where the results are useful, and where important limitations remain. It is part of our series on AI in medicine at Helathora.
What Is an AI Symptom Checker?
A symptom checker is a website or app that asks about your symptoms, age, sex and medical history, then offers possible explanations or advice. It usually does one or both of these jobs:
- Diagnosis (differential diagnosis): a list of conditions that could fit the information you entered. This is not a medical diagnosis, because the tool has no physical examination, lab results or imaging.
- Triage: an estimate of how urgently you need care, such as emergency care, a same-day visit, a routine appointment or self-care.
General-purpose chatbots have also been tested on these tasks. Most were not built or validated as medical devices, so they should not be treated as equivalent to a clinically validated symptom checker.
How We Reviewed the Evidence
This review is based on published research, not our own testing of individual apps. We looked for studies that:
- tested several symptom checkers or an AI language model;
- used standardized clinical cases (“vignettes”);
- reported measurable diagnostic or triage results;
- explained how a correct answer was defined.
Each study used different tools, cases and scoring, so the percentages below are not scores from one common test. They also describe the versions tested at the time, and many products have changed since.
Key Studies at a Glance
| Study | What was evaluated | Main findings |
|---|---|---|
| Semigran et al., BMJ (2015) | 23 online symptom checkers, 45 vignettes | Correct diagnosis listed first in 34% of evaluations and within the top 20 in 58%. Appropriate triage in 57%. |
| Hill et al., Medical Journal of Australia (2020) | 36 symptom checkers, 48 vignettes | Correct diagnosis first in 36%, within the top three in 52%, within the top ten in 58%. Appropriate triage in 49%. |
| Levine et al., The Lancet Digital Health (2024) | GPT-3 vs. physicians and lay people, 48 vignettes | Correct diagnosis in GPT-3’s top three in 88% of cases. Triage correct in 70%, versus 91% for physicians. |
These figures show the shape of the research. They do not rank products against each other.
Diagnostic Accuracy: Useful Clues, Not a Diagnosis
In the 2015 BMJ study, the correct diagnosis appeared first in about one-third of the evaluations. It appeared somewhere in the top 20 in 58%. The Australian study reported similar numbers: of the 36 tools tested, 27 produced diagnosis lists, and among those the correct condition came first in 36% of tests and within the top ten in 58%.
There is a gap between a tool that includes the right condition in a list and a tool that can reliably tell you what you have. Real diagnosis draws on a detailed history, a physical examination, tests and clinical judgment. A symptom checker is better seen as a source of information than as a substitute for that process.
Triage Accuracy: Cautious, and Uneven
A tool does not need to name the exact disease to give useful triage advice. It does need to recognize when symptoms could be serious. In the 2015 study, appropriate triage advice was given in 57% of evaluations, but performance depended on the scenario:
Appropriate triage advice by scenario type (2015 BMJ study)
The authors described the tools as generally risk-averse: they often advised seeking care when self-care would have been reasonable. The Australian study reported a comparable pattern, with better results for emergency and urgent cases than for non-urgent and self-care ones.
This creates a trade-off. Cautious advice may reduce the risk of falsely reassuring someone with a serious condition, but it can also lead to unnecessary worry, medical visits and use of emergency services.
What About General-Purpose AI Models?
Unlike most traditional checkers, language models can read free-text descriptions and hold a conversation, which is why researchers began testing them on medical reasoning. Levine and colleagues compared GPT-3 with practicing physicians and members of the public on 48 validated vignettes:
| Group | Correct diagnosis in top three | Correct triage |
|---|---|---|
| GPT-3 | 88% | 70% |
| Physicians | 96% | 91% |
| Lay participants | 54% | 74% |
In this study, GPT-3 performed better on the diagnostic task than on triage. Its triage accuracy was also lower than that reported for the physicians in the same evaluation. The results should be read with three caveats:
- It is an older model. The study tested GPT-3, not the chatbots people use today, so it says little about their accuracy.
- The cases were clean. Vignettes are complete and carefully written. Real users leave out symptoms, describe things inaccurately, misuse medical terms, forget medications or change their answers mid-conversation.
- Models change. A result for one version does not carry over automatically to the next.
What Does “Accuracy” Actually Mean?
The same output can count as right or wrong depending on the metric. Imagine a tool that returns: 1. migraine, 2. tension headache, 3. sinusitis, 4. meningitis, 5. dehydration. If the reference answer is migraine, the tool scores as correct on a first-diagnosis metric. If the reference answer were sinusitis, it would fail a first-diagnosis test but pass a top-three test. That is why papers report several measures, and why comparing a top-three figure from one study with a first-result figure from another is misleading.
Dark green: counts on a first-diagnosis test. Light green: also counts on a top-three test. Grey: counts on neither. This is the hypothetical example from the text, not study data.
Why Results Differ So Much
| Factor | Why it matters |
|---|---|
| Condition coverage | A tool cannot suggest conditions outside its design or knowledge base. |
| Quality of user input | Missing or wrong answers change the output. |
| Study design | Different cases and definitions of “correct” produce different numbers. |
| Scoring method | First-result, top-three and top-ten accuracy answer different questions. |
| Population | Performance may vary by age, background and health status. |
| Software or model version | Updates can change how a system responds. |
Limitations of the Evidence
- Most studies use simulated cases. Vignettes let researchers give every system identical information, but they are not real consultations.
- The technology moves fast. Some tested products no longer exist and others have been rebuilt.
- Tools work differently. Some use structured algorithms and others use language models, so treating them as one technology can mislead.
- Populations differ. Results from one country or health system may not apply elsewhere.
- Independent evidence is thin. High-quality studies comparing modern AI tools in realistic clinical settings are still scarce.
- Real-world performance is uncertain. Results from standardized vignettes may not reflect how a symptom checker performs when users provide incomplete, ambiguous or inaccurate information.
How to Use AI Symptom Checkers More Safely
- Treat the output as information. A list of conditions does not mean you have any of them.
- Never use it to rule out an emergency. Severe breathing difficulty, chest pain, sudden weakness or neurological symptoms, and heavy bleeding need urgent care.
- Give complete information. Include when symptoms started, how they have changed, medications and existing conditions.
- Consider your context. Age, pregnancy, chronic illness and recent procedures change how symptoms should be read.
- See a professional if symptoms persist, worsen or worry you.
Frequently Asked Questions
How accurate are AI symptom checkers?
It depends on the tool and the task. In the studies reviewed here, the correct diagnosis was often not the first result, and triage accuracy varied widely.
Can a symptom checker replace a doctor?
The evidence does not support that. A doctor can combine examination findings, tests, imaging, records and context, which a symptom checker generally cannot reproduce.
Are chatbots more accurate than traditional symptom checkers?
There is not enough evidence for a general answer. The GPT-3 study showed strong results on its diagnostic cases and weaker triage than physicians, but other systems, models and scenarios can differ.
Does a high top-three accuracy mean an AI can diagnose me?
No. It means the correct answer appeared among the first three suggestions under the study’s conditions. It does not mean the tool can establish a diagnosis for an individual patient.
Why are symptom checkers often overly cautious?
Many are designed to avoid missing serious conditions. This can lead them to recommend medical evaluation when self-care would be reasonable, a pattern reported in both the 2015 and 2020 studies.
Conclusion
Related Reading on Helathora
Sources
- Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ. 2015;351:h3480. doi:10.1136/bmj.h3480. BMJ | PubMed
- Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Med J Aust. 2020;212(11):514-519. doi:10.5694/mja2.50600. PubMed
- Levine DM, Tuwani R, Kompa B, et al. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study. Lancet Digit Health. 2024;6(8):e555-e561. doi:10.1016/S2589-7500(24)00097-9. PubMed