You Can Just Ask, But Not About Its Own Confidence
You don’t need to study computer science to work out what these tools can do. You can just ask them. That instinct is right, and it’s how most people who use AI well got started. One popular version of it does not work, though, and it happens to be the version that gets recommended most.
Asking what it can do
Opening with “I’m new to this, what can you help me with, and what are you bad at” is a reasonable first move. You’ll get a readable tour, usually including some honest-sounding limitations.
One caveat on the answer. What you get describes a generic assistant, not your account. Whether you have web search, file uploads, image generation, memory or connected apps depends on your plan, your region and sometimes on what your workplace has turned off, and the model often does not know which of those are live in your session.
So the better move skips the question. Ask it to do the thing. Paste the PDF and see whether it reads it. Ask something that changed this week and see whether a link comes back that you can open.
The self-check that doesn’t work
Here is advice you’ll find in a lot of beginner guides, including an earlier version of this page: after you get an answer, ask the model to review it and tell you which parts it’s confident about and which might be wrong.
As a check on whether the answer is true, that doesn’t do what it looks like it does.
OpenAI published a paper on this in September 2025. Hallucinations, in their words, are “plausible but false statements generated by language models”, and they persist because “standard training and evaluation procedures reward guessing over acknowledging uncertainty”. A system shaped to guess rather than to say it doesn’t know isn’t going to become reliable about its own uncertainty on a second pass. You get another piece of text from the same process, and now it has a confidence label attached.
The paper’s opening demonstration is the part worth keeping. Asked for one author’s birthday, with the instruction to answer only if it knew, a state-of-the-art open-source model was tried three separate times and returned three different wrong dates: 03-07, 15-06 and 01-01. It was confident each time, and the confidence was not a signal you could have used.
To be fair to the discarded advice, it isn’t useless, it’s misapplied. Asking a model which parts of its answer are shakiest is a decent way to find out which claims are worth checking. It’s a triage step. It is not the check itself.
What to do instead
The same paper hands you two tests, and both cost nothing.
Same model, separate chats. Ask the same factual question in two or three new conversations without referring to the earlier ones. Three different answers means at least two are invented, and you found that without knowing the right answer yourself. This is the birthday demonstration.
Different models. Stronger, and it’s what the paper’s other example actually shows: asked for the same dissertation title, ChatGPT, DeepSeek and Llama each returned a different wrong answer. Two tools disagreeing is a louder signal than one tool wobbling, and most people have access to at least two.
Agreement is not verification. Matching answers tell you the models are being consistent, not that they are right. A mistake that is common in the training data comes back the same way every time. Researchers working on this kind of detection say so explicitly: the method targets answers that are wrong and arbitrary, and does not cover a model that is confidently and systematically wrong.
Check the chats really are separate. If memory is switched on, or you are inside a project or a custom assistant, later conversations can see the earlier ones and you are no longer getting independent answers. A temporary chat avoids this.
Two habits do most of the remaining work. The first is to ask where a claim comes from and then open the link, because a model can produce a citation that looks perfect and points at a paper that does not exist, or at a real paper that says something different; the click is the test, not the presence of a link, and if the page opens you still have to find the sentence that supports the claim rather than assuming it’s in there somewhere. The second is to switch web search on for anything current. Prices, dates, laws, product features and anything that changed recently belong in a search-backed answer, and if your tool doesn’t offer one, that is itself worth knowing before you rely on it.
The one self-question that helps
There is a question worth asking the model about itself, and it works because it’s about the conversation and not about the world.
“What would you need to know from me to answer this better?”
That asks what is missing from the context you gave it, which is in front of it, not a fact it might be inventing. In my testing the replies are usually mundane: which country you’re in, what the output is for, how much the reader already knows, what constraints you left out. Supplying two of them and asking again tends to improve the answer more than rewording the original question does.
Curiosity still works
None of this argues against experimenting. The range is genuinely wide: explaining a contract clause in plain language, turning a messy list into a table, drafting the awkward email, working through a concept you half remember from school, planning a trip around the constraints you actually have.
One rule covers most of it. The more specific and checkable a claim is, the more it is worth verifying, because specificity is what these systems produce most fluently and where a wrong answer is hardest to notice and most expensive. A rough explanation of how mortgages work is usually fine. A named court case, a dated figure or a precise rate is something to confirm elsewhere before you use it.
In a nutshell
Asking the tool about itself is a good way in, as long as you remember it’s describing a generic assistant and not your account, and that the fastest test is to ask it to do the thing. Don’t rely on it grading its own confidence: by OpenAI’s own account these systems are trained in a way that rewards guessing over admitting uncertainty. Ask the same question in separate chats, or better, ask a second tool, and treat disagreement as a red flag. Agreement is not proof, since a common error repeats consistently. Then ask for the source and open it.
OpenAI, Why language models hallucinate, 5 September 2025, for the definition and the training-incentive argument. The underlying paper, Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, Why Language Models Hallucinate, for both demonstrations, read at the source: the birthday case is one model tried three times, and the dissertation case in Table 1 is three different models answering once each. The blog summary compresses those into one, which is why the paper is the better read. On the limits of consistency checking: Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn and Yarin Gal, Detecting hallucinations in large language models using semantic entropy, Nature, 2024, whose authors state that their method does not address cases where models are confidently wrong for systematic reasons. OpenAI has a commercial interest in how this problem is framed, which is a reason to read the paper alongside the blog post rather than instead of it.
If you want a structured place to start, the everyday guide goes through the jobs worth trying first.
