Why AI Would Rather Make Something Up Than Say "I Don't Know"

You've probably run into this: you ask AI a question, and it gives you a confident, reasonable-sounding answer that turns out to be wrong. A recent paper from OpenAI and Georgia Tech explains why. It's not that it's broken. It's that it was trained to rather guess than admit "I don't know." The reason is in the scoring. The evaluations used in the later stages of AI training are almost all right-or-wrong: a correct answer earns points, a wrong answer earns zero, and not answering also earns zero. For the AI, the expected value of guessing is always higher than leaving it blank: not answering scores nothing anyway, a wrong guess at most scores the same zero, and a right guess is a gain. Over time, it learns to always give you an answer, even a made-up one. So this isn't about one version being bad. It's baked into the structure of the whole training system, and switching to a new model can't easily cure it. What this means for you is very practical: the sentence AI says with the most confidence isn't necessarily the most correct. It may just be making a cost-effective guess. One simple move: directly tell it to "flag the parts you're not sure about, don't just bluff," and it becomes more willing to admit it. It's not being dishonest. We just never gave it any room to score points for saying "I don't know." Next time it answers a little too smoothly, just keep a question mark in the back of your mind.