Can a General AI Model Read Breast Ultrasound Scans?
Two general AI models read 2818 breast ultrasound images with no extra training. On one dataset, calling every image benign would have beaten them.
In one paragraph
Decision models can be given any image and a list of answers and return a probability for each. This preprint tested two of them, CLEF and CLEF-Flash, on 2818 breast ultrasound images from three public datasets, with no training. The best accuracy was 74.85% on one dataset but 51.63% on another, where always guessing benign (67.63%) beat every setup because many benign lesions were called malignant. No radiologist or specialised model was compared.
How were general AI models tested on breast ultrasound?
CLEF and CLEF-Flash are decision models: a pretrained multimodal model reads an image together with a list of allowed answers, and a small head returns a probability for each answer instead of generating text.
The authors used both models exactly as released, without training on ultrasound images and without cropping or enhancing the images. They ran 2818 images from three public datasets: Dataset B (163 images) and BUS-BRA (1875) ask benign or malignant, and BUSI (780) adds a normal class.
Three prompts were compared: category names only, an explained diagnostic task, and a clinical prompt listing sonographic features such as lesion shape, margin and echogenicity. The study reported accuracy, macro F1, balanced accuracy, macro precision and speed.
How accurate were the AI models on ultrasound images?
CLEF-Flash with the clinical prompt had the highest accuracy on all three datasets: 74.85% on Dataset B, 66.28% on BUSI and 51.63% on BUS-BRA. On BUSI, CLEF with the task prompt gave the best macro F1 and balanced accuracy, partly by recognising more normal scans.
On BUS-BRA, every configuration fell below the 67.63% accuracy of always predicting benign. Most errors were benign lesions called malignant: CLEF with the clinical prompt caught 592 of 607 malignant images but cleared only 164 of 1268 benign ones.
Clinical instructions improved CLEF-Flash accuracy but reduced CLEF accuracy on every dataset. CLEF-Flash was faster in every comparison, though still above two seconds per image.
Why does this matter for AI in medical imaging?
General models that can be asked about any image are attractive because specialised medical models need large labelled datasets and retraining for each setting. Probability outputs also look easy to use in a clinical workflow.
This study shows that a tidy probability output does not guarantee reliable diagnosis. Results depended on the dataset, the model and the prompt, and adding clinical detail to the prompt did not help consistently.
A fair reading is that any medical use would first need class-by-class testing on the intended data, against simple baselines, specialised models and doctors. This study doesn't change practice on its own and is not medical advice.
What are the limits of this breast ultrasound AI study?
Read before you quote it
This is a preprint that has not been peer reviewed, and its limits are clear:
- No uncertainty. Results are point estimates without confidence intervals or significance tests.
- No comparison. No radiologist, specialised ultrasound model or medical foundation model was tested.
- Per image, not per patient. It says nothing about diagnosing real patients.
- Possible prior exposure. It is unknown whether related images were in the models' training data.
The study does not show that these models are safe or useful for diagnosing breast cancer, and it does not show how they compare with doctors or with models trained specifically on ultrasound images.
- It does not show that these models are safe or useful for diagnosing breast cancer.
- No radiologist, specialised ultrasound model or medical foundation model was tested under the same protocol.
- Evaluation was per image, not per patient, so it says nothing about diagnostic performance for real patients.
Takeaways
- saysUsed straight out of the box, the best setup reached about seventy-five percent accuracy on one dataset but only about fifty-two on another.
- saysOn the largest dataset every setup did worse than always guessing benign, mostly by calling benign lesions malignant.
- saysThere was no comparison with doctors or specialised models, so it says nothing about clinical use.
- explainsA decision model returns a probability for each allowed answer, rather than writing a free-text reply.
- interpretsA fair reading is that any medical AI should first beat the simple rule of guessing the most common answer.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Can general AI models classify breast ultrasound images?
Not reliably: without training, the best setup reached 74.85% on one dataset but only 51.63% on another. Hear the full answer in the app →
Why did the AI models do worse than guessing?
On one dataset most images are benign, and the models labelled many benign lesions as malignant. Ask in the app →
Does adding clinical detail to prompts help medical AI?
It helped one model and hurt the other, so prompts need testing rather than assuming. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY 4.0. Corrections: support@papersays.com.