You paste a logic puzzle into ChatGPT, get a crisp answer back, and feel a little smarter. A new study says that feeling can become a problem.
Researchers found that people using ChatGPT on LSAT-style logic questions scored higher than people working alone. Yet they also became worse at judging how well they did.
The headline number is the gap. AI users typically guessed they got about 17 out of 20 correct. Their real performance was closer to 13. That four-question difference is an "illusion of competence," confidence rising faster than accuracy.
That mismatch matters because self-checking is part of competence. In school, it exposes what you still do not understand. At work, it prevents a polished answer from turning into a fragile decision.
The paper, published in Computers in Human Behavior, reports two studies with hundreds of participants. Study 1 included 246 people and showed a performance lift alongside a larger confidence lift. Study 2 included 452 people and added financial incentives to encourage honest self-evaluation, and the overestimation remained.
One of the more unsettling findings is how evenly overconfidence spread. Instead of following the familiar pattern where low performers overestimate more than high performers, AI use flattened the curve. People across skill levels misjudged themselves in similar ways.
So why does this happen? Many participants treated the model like a single-step answer machine, paste the prompt, accept the output, move on. That workflow removes the friction where doubt and error-detection usually live.
If you use AI to learn, the best defense is a routine that forces calibration. Do a first attempt without help, even if it is messy. Then ask the model to critique your steps, list assumptions, and name what evidence would change the answer.
Quick self-check routine for AI-assisted work:
- Write your answer first, even a rough one.
- Ask for assumptions and edge cases, then test at least one.
- Ask what would change the conclusion.
- Explain the reasoning back in your own words, without looking.
If you are building fundamentals, well-designed learning paths can help you structure practice in a way that makes gaps visible, including learning artificial intelligence. If you want a more course-style on-ramp, this intro to AI course is a straightforward starting point.
It also helps to remember that AI confidence effects are shaped by interfaces and ecosystems, not just the model’s raw capability. If you want a broader example of how platform integrations can change user behavior and expectations, see Hackr’s coverage of the “write-only handshake” OpenAI and Apple Music story.
Read the study to get a deeper look.
