Can an AI Learn When to Say 'I Don't Know'?
Chatbots guess confidently even when they don't know. This study compares two ways to teach a model when to hold back: retrain it, or add a separate checker.
In one paragraph
Language models often answer confidently even when wrong. This preprint scored answers so that mistakes cost points and saying I don't know cost nothing, then tried two fixes on Qwen3-14B. A separately trained answer selector did best on TriviaQA: coverage rose from 20.2% to 30.9% and the error rate among answers fell from 23.0% to 14.5%. Retraining with reinforcement learning gave no consistent gain, and on NQ-Open the selector still scored below always abstaining.
How did researchers teach a language model to abstain?
The task was short factual questions. With each question the model received a confidence threshold, such as 0.85: a correct answer earns 1 minus the threshold, a mistake loses the threshold, and answering IDK scores zero. Answering therefore pays off only when the chance of being right exceeds the threshold.
The authors compared two routes on the open model Qwen3-14B. Route one retrained the model with reinforcement learning, in four variants, three seeds each. Route two kept the model fixed and trained a separate selector that predicts whether a candidate answer is correct; the answer is shown only if its calibrated probability exceeds the threshold.
Tests used 2,000 TriviaQA and 500 NQ-Open questions at thresholds 0.65 and 0.85, which were not used in training, with exact-match grading and a partial human review of 150 responses.
Did the AI get better at knowing when it doesn't know?
On TriviaQA, the selector beat the original model's calibrated confidence filter. Coverage rose from 20.2% to 30.9%, and the error rate among answered questions fell from 23.0% to 14.5%. Mean utility rose from 0.02418 to 0.04248, and the gain was positive for each of the three seeds.
The reinforcement learning variants did not establish consistent gains: every comparison's interval included zero.
On NQ-Open, the selector improved on the filter but its utility stayed below the zero of always abstaining, so calibration learned on one dataset did not transfer well. Better confidence rankings alone were not enough to make answering worthwhile there.
Why does abstention matter for AI answers?
An assistant that guesses confidently can mislead people, while one that refuses everything is useless. Useful abstention means answering when likely right and holding back otherwise.
This study suggests a practical route: keep the model as it is and add a trained checker that decides which answers to show, rather than retraining the whole model. In these experiments the selector was also much faster to train than the reinforcement learning runs.
A fair reading is that answer selection is a promising tool, but each deployment would need to check it on its own kinds of questions.
What are the limits of this abstention study?
Read before you quote it
This is a preprint that has not been peer reviewed, and its limits are clear:
- Narrow scope. One model family, short English factual questions and three training seeds.
- Strict grading. In a review of 150 responses, 40 marked as errors by exact matching were judged correct.
- Partial review. The audit was AI-assisted, with one annotator checking labels proposed by ChatGPT.
- Weak transfer. Calibration learned on TriviaQA did not carry over well to NQ-Open.
The study does not show that reinforcement learning cannot teach abstention, only that the tested recipes were inconclusive. It also does not show reliable abstention across datasets or in open-ended conversation.
- It does not show that reinforcement learning cannot teach abstention; the tested recipes were simply inconclusive.
- It does not show reliable abstention across datasets or in open-ended chat.
- Only one model family, short English factual questions, and three training seeds were tested.
- Automatic exact-match grading rejected some valid answers: in a review of 150 responses, 40 marked as errors were judged correct.
- The human review was partial and AI-assisted, with one annotator checking labels proposed by ChatGPT.
- Calibration learned on TriviaQA did not transfer well to NQ-Open.
Takeaways
- saysOn TriviaQA, a trained answer selector answered more questions, coverage up from twenty point two to thirty point nine percent, with fewer errors, down from twenty-three to fourteen point five percent.
- saysRetraining the model itself with reinforcement learning gave no consistent gain.
- saysOn a second dataset, even the selector scored below simply always saying I don't know.
- explainsCoverage is the share of questions a system answers; selective risk is how often those answers are wrong.
- interpretsA fair reading is that adding a trained checker is a simpler way to cut confident mistakes than retraining the model, at least on trivia.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
Can language models learn to say I don't know?
Partly: a separate trained answer checker helped on TriviaQA, but reliable abstention across datasets remains unsolved. Hear the full answer in the app →
Does reinforcement learning teach a model to abstain?
In this study the tested reinforcement learning recipes gave no consistent gain. Ask in the app →
Why do chatbots guess instead of admitting uncertainty?
When mistakes and abstentions score the same, any chance of being right makes guessing worthwhile. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Paper licence: CC BY 4.0. Corrections: support@papersays.com.