Key points
- On 1,953 questions from MedMCQA, a research dataset built from India's NEET PG exam, MedGemma 4B chose the dataset's answer 56% of the time, against 25% for random guessing.
- On 300 questions from CMB, a dataset built from China's postgraduate entrance exam in clinical medicine, it chose the dataset's answer 49% of the time, against about 24% for random guessing.
- When only the order of the options was changed, it gave a different answer to about three in ten questions on one set and more than a third on the other. It did better than Google's general Gemma 3 model of the same size on both.
Open medical AI models are now small and free enough that a health department, a non-profit or a start-up could try building them into a tool. Before going further, a reasonable first question is how such a model does on the kind of questions doctors answer. I wanted to see what that looks like as someone working in technology and health, using only published question banks with their own answer keys, so that nothing depended on my own judgement of the answers.
This is not a clinical evaluation, and it says nothing about whether the model is safe to use with patients. It is a simple benchmark run.
The model and the questions
MedGemma is a family of open models released by Google in 2025 for medical text and images. I used the smallest, MedGemma 4B, which runs on the free graphics processor in Google Colab. Google describes MedGemma as a starting point for developers, not a finished clinical tool.
The questions came from two published, peer-reviewed research datasets.
- MedMCQA (Pal and colleagues, 2022) is built from Indian medical entrance questions. Its authors describe its development set as questions from NEET PG, the exam for entry to postgraduate medical training. I used every single-answer question in that set, leaving out dental questions and 16 questions that refer to an image the dataset does not include. That gave 1,953 questions across 19 subjects, each with four options.
- CMB (Wang and colleagues, 2024) is a Chinese medical benchmark. Its authors describe one section as China’s national postgraduate entrance exam in clinical medicine. I used the single-answer questions from that section, which gave 300 questions, each with four or five options, asked in Chinese.
Both datasets were compiled by their authors from public sources. The exam boards do not publish official question banks, so these are research datasets built from past entrance questions, and they may include some practice questions and occasional errors in the answer key.
How the test worked
The model saw only the question and its options and was asked for the letter of its answer. It never saw the answer key. Its letter was then compared with the dataset’s answer key automatically. The settings were fixed so that the same question always gets the same answer, and the downloaded questions were checked against a fixed list before the run.
I ran two extra checks on all 300 Chinese questions and on a fixed random 300 of the Indian ones. The Chinese section has only 300 single-answer questions, so all of them were used, and the Indian subset was drawn to match its size. The first asked the same questions again with the options in a different order. The second asked the same questions to Gemma 3 4B, Google’s general model of the same size, which has not been trained specifically for medicine.
What it got right
India (MedMCQA, NEET PG). MedGemma chose the dataset’s answer for 1,098 of 1,953 questions, or 56% (95% confidence interval 54% to 58%). Random guessing would give 25%.
Every subject’s score was above the 25% expected from guessing, although for small subjects such as psychiatry (10 questions) the difference is within chance. Among subjects with at least 20 questions, it ranged from 68% in pharmacology and 63% in pathology to 44% in anatomy and 39% in forensic medicine. Smaller subjects have wide uncertainty, so small differences between them should not be read too closely.
MedGemma 4B on 1,953 NEET PG questions from the MedMCQA dataset, by subject. Chart: Hussain MS.
China (CMB, postgraduate entrance, clinical medicine). MedGemma chose the dataset’s answer for 147 of 300 questions, or 49% (95% confidence interval 43% to 55%). With a mix of four and five options, random guessing would give about 24%.
The two results should not be compared with each other. The questions are in different languages, have different numbers of options and come from datasets built in different ways.
When only the order of the options changes
A good answer should not depend on whether the right option is labelled A or C. So the same questions were asked again with the options rotated, so that every correct answer moved to a different letter.
On the 300 Indian questions, MedGemma gave the same underlying answer 70% of the time, so about three in ten answers changed. Its score on that subset went from 58% to 52%.
On the 300 Chinese questions, it gave the same underlying answer 63% of the time, so more than a third of answers changed. Its score went from 49% to 43%.
For anyone building a tool on a model like this, that is worth knowing. A small change in how a question is presented, with no change to its content, can change the answer.
Compared with a general model
On the same 300 Indian questions, MedGemma scored 58% and the general Gemma 3 4B scored 45%. MedGemma alone was right on 56 questions and Gemma alone on 16, a difference unlikely to be due to chance.
On the 300 Chinese questions, MedGemma scored 49% and Gemma 3 4B scored 42%. MedGemma alone was right on 48 questions and Gemma alone on 27, again a difference unlikely to be due to chance.
So on these benchmarks, the medical training appears to help.
What this does and does not show
Exam-style questions measure recall of textbook knowledge. They do not show whether a model gives safe and accurate advice in free text, which is what a health worker would actually see. They also say nothing about how a model behaves inside a real tool, with real users and real data.
There are other limits too. These datasets are public, so the model may have seen some of the questions during training. This was one small model, at one setting. And the datasets carry their own small error rates.
Reporting and evaluation frameworks for AI in health each fit a different stage, and a benchmark run like this sits at the very beginning.
| Stage | What it is | Framework |
|---|---|---|
| Offline testing of a language model, like this one | Questions, answers and scoring, no patients | TRIPOD-LLM |
| First use with real patients, at small scale | Early clinical evaluation of decision support | DECIDE-AI |
| A randomised trial of an AI tool | Comparing outcomes with and without it | CONSORT-AI |
| Governance across all stages | Ethics, accountability and oversight | WHO guidance on large multi-modal models (2024) |
In India, BODH (Benchmarking Open Data Platform for Health AI), developed by IIT Kanpur with the National Health Authority, is designed to provide a structured way to test health AI before it is used at scale. A benchmark run like this is a small, informal version of the same first step.
The data
The full results are available to download. For each question, the file gives the benchmark, the question’s ID in its dataset, the subject, the dataset’s answer key and the answers given by MedGemma, by MedGemma with the options reordered, and by Gemma 3 4B. It does not include the question text, which can be found in the original datasets using the IDs. The notebook and run files are available on request.
Views are my own.
Sources
- Google. MedGemma model card (2025)
- Pal A., Umapathi L.K., Sankarasubbu M. MedMCQA, a large-scale multi-subject multi-choice dataset for medical domain question answering. 2022
- Wang X. et al. CMB, a comprehensive medical benchmark in Chinese. NAACL 2024
- Gallifant J. et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nature Medicine, 2025.
- Vasey B. et al. DECIDE-AI. Nature Medicine, 2022. Liu X. et al. CONSORT-AI. Nature Medicine, 2020.
- World Health Organization. Ethics and governance of artificial intelligence for health, guidance on large multi-modal models. 2024.