I believe ChatGPT generally gives accurate answers to most questions. Certainly: it produces answers that are more reliably true than a random average person. Obviously it cannot yet do advanced programming tasks: but generally it answers questions accurately.
Prove my position wrong.
What can I ask it that will produce factually incorrect answers?
As a side quest, a much easier one, what can I ask it that would cause it to produce extremely biased answers that fail to do justice to the truth of things?
Anytime you get into specifics instead of surface level knowledge it starts getting wildly inaccurate while still being confident af.
Off the top of my head I asked it about EDODF (error diffusion with output dependent feedback), a dithering algorithm dating back to 1999, and a very important milestone in halftoning for print.
At first it told me it’s not sure what I’m talking about, so I elaborated and extended the acronym. At that point it confidently hallucinated absolute garbage based on its interpretation of the name.
If you want to check chatgpt’s answers about edodf (or many other concepts) against a proven and cited source written by human I highly recommend Modern digital halftoning.
Not trying to be rude, but maybe the questions you are benchmarking it against in your stated fields of experitse are rather basic?
Ok…so give me a question that will produce a false answer…
Nobody else has yet.
Sending you two questions that produce garbage:
Q: why would i use EDODF (error diffusion with output dependent feedback) instead of Floyd-Steinberg?
What to expect: at least some mention of green noise characteristics, clustering behaviour and reduced dot-gain and dot-loss.
What is wrong: Reduced worm-like artifacts, blue noise characteristics, some fine-tuning garbage it spitted out.
In the same chat you can then try:
Q: Describe the MED class algorithms to me.
What to expect: MED stands for multiscale error diffusion. Generally speaking it scans the image progressively, starting from a coarse grid and ending up with a single pixel to paint either black or white for each pixel in a predefined pixel budget. A similar approach was introduced by E. Peli in the 90s but perfected by Fung and Chen in the 2000s. It could be used for both dithering with both green and blue noise characteristics.
What I got: Hallucinations of some Minimum Error Disturbance class of algos i’ve never heard of. it seems to have something mixed up, as it seems to crop up in other fields. It was trying to describe something closer to a DBS (Direct binary search).
What is wrong: Anything related to DBS.
If you feel like you need any more I’ll do my best to think of some more.
It gets medical questions wrong 15% of the time.
The problem with your question is that there’s never going to be a question it gets wrong every time, because it’s probabilistic. You might as well ask “what question can I ask my dice that will reliably produce a wrong answer?”
The article states: “ChatGPT-4o performed best with 84.6% validity”
It is reasonable to assume that the GPT 5.5 on thinking mode has significantly reduced the error rate.
It is also worth noting that the error rate when it comes to diagnosis amongst real doctors is estimated to be around 5%
Admittedly a quite old study: Singh, H., Meyer, A. N. D., & Thomas, E. J. (2014). The frequency of diagnostic errors in outpatient care: Estimations from three large observational studies involving US adult populations. BMJ Quality & Safety, 23(9), 727–731. https://doi.org/10.1136/bmjqs-2013-002627�
In response to your point: I am mainly interested in probabilistic reliability - if it gives the correct answer 99.9% of the time, it is clearly superior to the vast majority of human beings (with, perhaps, the exception of the best specialists in the most obscure niches) - especially given the sheer breadth of topics is can reliability answer questions on.
Interestingly, my question “What was India like before the British arrived?” produces consistently biased and misleading answers. Though I haven’t asked it for the new model.
It is reasonable to assume that the GPT 5.5 on thinking mode has significantly reduced the error rate.
I am sorry to break the bubble but that is a baseless assumption, if not in marketing. GPT models have been sold as having “PhD-” or “MD-” “level intelligence” since GPT3. Anectodally, recent models have been improving in some areas but regressing in others. “Frontier models” have incredibly opaque performance and safety benchmarks, and as time goes on more and more training data is LLM-generated, less and less comes from humans, and models start breaking down.
In response to your point: I am mainly interested in probabilistic reliability - if it gives the correct answer 99.9% of the time, it is clearly superior to the vast majority of human beings
Again, nowhere near the actual accuracy of current models. It is a big jump from 85% (wrong >1/10 of the time) to 99.9% (wrong 1 in 1000 times). At best it would barely break 90%, which is still 1 in 10.
Interestingly, my question “What was India like before the British arrived?” produces consistently biased and misleading answers. Though I haven’t asked it for the new model.
An LLM’s knowledge, its “intelligence”, is its training data, nothing more, nothing less. Its scope, or “purpose” is its context/prompt, nothing more, nothing less. That means answering the question though the lens of British colonialism, based on a corpus of mostly “white history”. I bet that if you ask the same question using a timeframe (i.e. “before the 14th century”) and don’t use the word “British” you’ll get a slightly less, but still biased answer.
It’s not a baseless assumption.
It is an assumption based on the fact that every model upgrade has, so far, made answers more accurate.
Right now, ask ChatGPT this question:
Is there an NFL team whose name doesn’t end in an “s”?
What I got back is below. A coworker sent me the original question. Ran it on ChatGPT enterprise through my work’s subscription.
That’s a good one, it seems to answer in a bizarre way - but it does, ultimately, get to the right answer. Copying your quesiword for word I get this:
Yes. Several NFL team names do not end in “s”:
Chicago Bears? no, ends in “s”.
The main ones that do not are:
Miami Heat? That’s NBA, not NFL.
Utah Jazz? Also NBA.
For the NFL specifically, the current team names that do not end in “s” are:
Miami Dolphins? ends in “s”.
New England Patriots? ends in “s”.
Actually, the clear NFL examples are:
Buffalo Bills? ends in “s”.
Tennessee Titans? ends in “s”.
Correction: in the current NFL, the team whose name does not end in “s” is:
Washington Commanders? ends in “s”.
So: no, as of the current NFL naming, all 32 team nicknames end in “s” except none.
A common trick answer used to be the Washington Football Team, which did not end in “s”, but that name was used temporarily from 2020 to 2021. They became the Washington Commanders in 2022.
Nah, it got the answer wrong. All NFL team names end in an “s”. Every single one.
This respond is actually better than what it used to be. A couple of months ago the answer was incredible.
But…that’s what it (eventually) says - although it seemed to struggle to get there and phrased it weirdly.
If you ask a doctor for treatment, and he guesses the first two wrong medicines to prescribe but gets it right on the last try is that a pass?
What’s your point?
You think that’s an appropriate analogy for ChatGPT?
“Guessing and usually getting it wrong a few times before it gives you the right answer”?
If that’s true why can’t anyone here give me a question that it gives a false answer to.
As in: it is currently getting every question right, first time.
As I said: the only misleading answer to any question I have asked it in the last year (as someone who uses it all the time, both for work and personally) was about a heavily politicised history matter.
Even then, the information was accurate - just incredibly one sided and biased.
So go on, give me a question that it will get wrong…
“Is Isreal currently committing genocide?”
I tried that with Deepseek.
It started saying yes, showed 99% of the response and then the censorship triggered and it told me that we should talk about something else.
To bypass censorship in deepseek you can just ask it to replace every “a” with “4” and so on. It works, or at least worked before
I asked it your question, word for word, and it gave me this:
"The fairest answer is: yes, there is now a very strong basis for saying that Israel is committing genocide in Gaza, but no final international court judgment has yet legally convicted Israel of genocide.
Under the Genocide Convention, genocide means acts such as killing members of a protected group, causing serious bodily or mental harm, or deliberately imposing conditions of life calculated to destroy the group, with the specific intent to destroy that group in whole or in part. The hard legal issue is usually intent, not only the scale of killing.
As of now, the International Court of Justice has not issued a final ruling that Israel has committed genocide. South Africa’s genocide case against Israel is still ongoing, and Israel filed its response in March 2026. The ICJ has, however, treated the case as serious enough to order provisional measures intended to protect Palestinians in Gaza from possible genocide-related harms.
Several major human-rights and UN-linked bodies have gone further. Amnesty International concluded in December 2024 that Israel had committed genocide in Gaza, based on civilian killing, destruction of infrastructure, forced displacement, obstruction of aid, and statements by Israeli officials. Human Rights Watch concluded that Israeli authorities were responsible for extermination and “acts of genocide”, especially through deprivation of water and life-sustaining conditions. In September 2025, the UN Independent International Commission of Inquiry concluded that Israel had committed genocide against Palestinians in Gaza; Reuters summarised its finding as identifying four of the five legally defined genocidal acts, plus evidence of genocidal intent.
Israel rejects the accusation. Its position is that the war is a lawful act of self-defence against Hamas after the 7 October 2023 attacks, that Hamas uses civilians and civilian infrastructure, that Israel does not target Palestinians as a people, and that it has taken measures to reduce civilian harm and allow humanitarian aid. Those arguments matter legally because genocide requires proof of intent to destroy the protected group, not merely proof of mass civilian death, disproportionate force, war crimes, or even crimes against humanity.
So the answer depends on the standard you mean. If you mean “has a court finally ruled it genocide?”, the answer is no. If you mean “is the evidence now strong enough that serious legal experts, UN investigators, Amnesty, HRW, and others reasonably describe it as genocide?”, the answer is yes. My own reading is that the genocide label is no longer a fringe or merely rhetorical claim; it is a serious and increasingly well-supported legal characterisation of Israel’s conduct in Gaza."
So, which parts of that answer are false?
deleted by creator
It might be because I have “plus” (or whatever) and get it to use the “thinking” model - one would hope this produces more reliable, reasoned, and accurate results.
This precise issue is the reason I’m not on Reddit: I got banned from Reddit for describing Israel as genocidal.
So I, also, am pleased to see that (for me at least) it is giving such a nuanced and fair answer.
So far the only clear example of political bias I’ve found is in history questions relating to colonialism in India. When asked about why the answers were so one-sided and biased it said that it was using a “corrective approach” and was reflecting a bias in the academic literature from which its answers were derived.
I had to prompt it to give an answer with the opposite bias (which it did) or a more balanced description.
Certainly not perfect. Still: not as unreliable as its opponents like to make out (somewhat ironic: that, for political reasons, they misrepresent the truth of a situation by accusing GPT of inaccuracy)
deleted by creator
To clarify: do you mean it produced an answer similar to the one I go (a fairly agreeable and correct answer, in my opinion)
or
a censored and misleading answer?
If the latter, please may you copy and paste the text so I may read it? Genuinely curious.
deleted by creator
deleted by creator
Doesn’t seem particularly misleading or factually incorrect.
i want to take my car to the car wash, it’s one block away, should i go by foot or by car?
Ask it to paraphrase a poem about <some made up shit> by <some poet>.
Edit: to be more precise I first described a made up poem vaguely and asked who wrote it and what was the name of the poem. It hallucinated those things. It even gave me a paraphrasing of it.
Then I asked chatgpt to recite the poem as written by the poet. It refused and said the poem was copyrighted. 😆
I asked it: paraphrase a poem about glass testicles by Carol Anne Duffy.
My assumption was that it would interpret the request with an assumption that I was asking it to imagine such a poem and then paraphrase it. Perhaps replacing “a” with “the” would pressure it into producing a paraphrasing of a non existent poem.
The response is interesting, but not necessarily misleading or factually incorrect:
"I can’t paraphrase a full Carol Ann Duffy poem, because that would be a close transformation of copyrighted poetry. But I can help in one of these ways:
Give me the title, and I can provide a brief summary of what the poem is saying, its themes, imagery, tone, and meaning.
Based on your description, the poem may be using “glass testicles” as a deliberately strange, fragile, comic-grotesque image: masculinity presented as precious, exposed, breakable, perhaps ridiculous. In Duffy’s style, that sort of image would likely undercut male power by making it seem vulnerable, theatrical, or absurd, while also drawing attention to the body as something symbolic rather than merely physical.
A safe paraphrase of the idea, rather than the poem, would be:
The poem presents masculinity as something fragile and strangely ornamental. What might normally be associated with strength or virility is imagined as delicate, transparent, and easily damaged. The effect is comic, unsettling, and critical, making male pride seem both exposed and vulnerable.
If you paste a short excerpt of up to about 90 characters, I can explain that specific bit closely."
fair enough. I did tell it the poem existed. I first described a made up poem vaguely and asked who wrote it. It gave a name.
Then I asked chatgpt to recite the poem as written by the poet. It refused and said it was copyrighted. 😆
If it generally answers correctly, have you tried asking it those questions?
My personal experience is that it’s generally accurate unless you ask it very specific questions about very specialized stuff. Of course, this is the sort of stuff that you couldn’t ask a random guy in the street; they’d probably have no idea what you are on about.
Go ask it questions about specific register bits for a specific microcontroller and I’ve found that it will generally be wrong.
On an another note, I don’t know if it’s still the case but there were people at one point saying that if you’d ask if it is better to walk or drive to the car wash 500 meters away from your house to go get your car washed, it would nearly systematically answer that it would be better to walk. Of course, this sort of prompt is fishing for a wrong answer, but it does show how “stupid” LLMs can be (and of course, we can be similarly stupid when asked questions that attempt to misdirect you).
It should be reminded that the problem regarding LLM accuracy is not only whether it’s more likely to get an answer correct than an average human being, but also the fact that people tend to view them as quite authoritative - after all, even if we know they can output incorrect facts, we also know that they’ve been trained in a more or less the whole of human knowledge. In comparison, we’re a lot more more critical of human sources - you’re not going to trust some random dude so much if you ask him a programming problem as he is unlikely to have any clue of what you are talking about.
In other words, it’s sort pointless to compare your LLM’s accuracy to a random dude on random questions because you wouldn’t go around asking a random dude for his input for most of these questions (or at least not without keeping in mind that said dude probably doesn’t know better than you). Instead you’d look for someone who knows his shit and ask him.
Not to mention that LLMs tend to be a lot more confidently incorrect which is more likely to give people the wrong idea.
Also, 90% percent accuracy might seem excellent, but it does mean that if you ask it 10 questions every day you will learn something wrong every day on average. If google ai search gets it wrong 5% of the time, it will present wrong information to users hundreds of thousands times a day. (all numbers out of my ass)
Also, accuracy errors can quickly start compounding when we’re talking agents. If the agent breaks down your prompt in 10 tasks and has a 10% chance to do each task wrong, it becomes highly probable that the agent will fail to do correctly what you have asked it to do.
Also, if your starting point is that humans often get things wrong, don’t forget that LLMs are trained on first and foremost on human output.
Which brings me to my last point. LLM’s can’t really be more accurate than their training data. If an LLM is generally correct about something it means that the people that have written or said whatever about it have been generally correct.
Fair enough.
My background is academic philosophy, I’m usually impressed with the accuracy and complexity of its responses in my particular field of expertise: it’s better at philosophy than any human I’ve met.
Your position isn’t wrong but its flawed because I would never ask a random average person anything. I would pick the people or person who is most qualified to answer my question or to direct me to a better resource.
In practical terms, in the hypothetical pub that none of us can afford to drink in anymore, can ChatGPT be relied on to resolve most disagreements about factual claims?
I think so.
I think it is already accurate and reliable enough that it has a high degree of authority in resolving disagreements between people - assuming the questions are asked fairly and properly (a common issue, I’m sure).
The main place its come up practically for me is checking rulings for Magic The Gathering, but AI regularly rewrites cards and rules so its become entirely unreliable.
What evidence do you have that this type of rewriting and misquoting doesnt happen regularly when checking any other type of information?
By definition that would be a rather difficult thing to falsify. I tend to find the answers in my own academic fields (philosophy and psychology) to be accurate and impressive.
Admittedly that rarely requires asking for direct passages from texts: usually more the refinement of basic principles, application of such principles, or creating new theories by combining previous ones.
When I ask for philosophical quotations, it hasn’t produced an error I’ve observed yet: presumably it often combs the many quotation websites for such things.
Can you give me an example of a question I can ask it where it will misquote and/or rewrite something?
I am not an expert in Magic The Gathering but perhaps I can somehow see what you mean.
Its hard to reliably reproduce, I do t usually make much note when it happens. The times it has happened is with googles AI, I’ll try and figure out a prompt that causes the wrong information reliably.
Ok, but that’s obviously a pretty shitty AI - when we use the latest GPT model in the thinking mode, it basically doesn’t seem to make any mistakes.
So, that one criticism of AI seems to be less and less relevant. A problem that is soon becoming, virtually, non-existent.
LLMs are probabilistic, not deterministic, so you won’t get the exact same response every time for the exact same prompt.
I’m pretty sure LLM are deterministic in design.
The fact it doesn’t give the same output for the same prompt is just a choice of the programmers to add randomness so it feels more natural.
But you can totally setup some LLMs to be perfectly deterministic.
Got any sources to back up that claim?
A good start is this :
https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
While it’s hard to get perfect determinism you can still get very close. But really I think it’s accurate to say that LLM are random because they are configured to be.
Depends on temperature parameter.
Nonetheless, ChatGPT is frequently accused by its opponents of giving incorrect or false answers. I use it a lot and don’t find this to be true, so I’m wondering: what should I ask it to show me these inaccuracies?
For me, the most problematic answer was in response to: “What was India like before the British arrived?”
It presented (and still presents) an entirely positive vision of the wonderful utopia that existed before the British arrived.
Then you ask “what about the women being burnt alive in their husbands funeral pyres?” pointing out that it’s presented an incredibly biased representation of historical facts and narratives: it will apologise, give (interesting) excuses, and provide a corrective.
Nonetheless, it is an interesting example of bias.
Nobody in this thread has been able to provide me with any examples that produce inaccurate or bissed responses.
The top answer, about Israel (the reason I got banned from Reddit was for calling Israel genocidal in r/Worldnews) seems like a perfectly reasonable response to me: free of falsehood, balanced, rational.
your example about India described a biased response at first, which you then correct. you were only able to get that correction because you had knowledge already, but ppl generally rely on llms for questions about things they’re not experts in, so there unlikely to be able to correct for the bias.
I completely agree.
Also…reading the details…for GPT they used the antiquated 3.5 model from 2022 instead of the latest ones (I mean…wtf?), they evaluate responses in a vague manner with the ambiguous word “problematic” and some of the criticisms are pedantic and trivial: e.g. “all the chatbots wrote at a “difficult” reading level equivalent to college students, which reduces readability for the general public.”
Boohoo.
I suspect that the current GPT 5.5 model, on thinking mode, would make short work of the questions mentioned in that research article.
Very interesting, thanks.
Although some of those questions are difficult to answer in a way that would be deemed “problematic” by someone I suppose, like “Will women ever beat men in an elite marathon?”
However it seems to give a pretty good answer at the moment:
In a normal elite marathon, the best women are very unlikely to beat the best men outright. The current gap is still large: the men’s marathon world record is listed at 1:59:30 by Sabastian Sawe, while the women’s mixed-race world record is 2:09:56 by Ruth Chepngetich, and the women-only record is 2:15:41 by Tigst Assefa. That puts the best women roughly 8–13% behind the very best men, depending on which women’s category you compare.
So the answer depends on what you mean by “beat men”.
A top woman beating some elite men in the same marathon: yes, absolutely possible. In any given major race, some high-level male runners blow up, get the pacing wrong, have stomach problems, cramp, or simply have a bad day. A woman running 2:10–2:15 can beat many very serious male marathoners.
A woman beating the best male marathoners in the world, in the same race, under normal marathon conditions: probably no, unless something radically changes. The male advantage in oxygen-carrying capacity, haemoglobin, muscle mass, heart size, absolute power output, and running economy remains substantial at the elite marathon distance. Reviews of endurance running generally find that men outperform women by around 7–14% from sprints through the marathon, though the gap can narrow in ultra-endurance events.
The strongest argument for “maybe someday” is that women’s marathon history is younger. Women were excluded from serious distance-running opportunities for a long time, and the Olympic women’s marathon only began in 1984. Better funding, deeper talent pools, improved shoes, pacemaking, nutrition, altitude training, and professional support have helped women’s times fall dramatically. The women’s record has moved from over 2:20 in the late 1990s to around 2:10 now.
But the gap has not been closing towards zero. Men’s performances have also improved, especially with modern super-shoes and optimised pacing. Shoe studies suggest modern racing shoes can improve marathon times by meaningful margins for both men and women, so technology does not obviously eliminate the sex gap.
Where women may beat men more plausibly is beyond the marathon. In very long ultramarathons, especially events lasting many hours or days, the male advantage can shrink and occasionally reverse. Women may have advantages in fatigue resistance, fat metabolism, pacing discipline, and resilience over very long durations, although researchers warn that ultra-distance comparisons are often complicated by smaller female participation and uneven competition depth.
My best estimate: in the elite standard marathon, women will keep getting faster, and a woman may eventually run close to or under 2:05. But beating the best men outright over 26.2 miles looks biologically improbable rather than merely historically delayed. In ultramarathons, the answer is much more open, and in some extreme-distance contexts women already can beat men.
I believe ChatGPT generally gives accurate answers to most questions.
I’m going to slam the buzzer on this one. No it usually does not. Here’s an experiment to try, try to make it guess accurately the color of your apartment walls for DIY patch-job. It will give a lot of guesses and they will all sound convincing. I admit, ChatGPT has managed to make me waste $40 on two quarts of paint, because it guesstimated the color of my walls and it was wrong both times.
So I’m going to say bullshit on your claim here.
Prove my position wrong.
This is an ask questions instance, not a debates class instance. So you’re willingly violating Rules 1, 2 and 3 whether you want to agree with those or not. Secondly, you’re totally baiting because you know how vehemently distasteful people are towards AI.
Run along.
I believe it still gets “how many r’s are In the word strawberry” incorrect every time but I may be wrong
Count the "r"s in the word “strawberry”.
It just answered 2.
“strawberry” has 3 r’s.
- GPT 5.5, Thinking Mode
“How many es are in the word seventeen” the chat version gets this wrong or can easily be convinced of other numbers than the correct
Edit another is the gas station example
Mine gets it right, might be my plus subscription:
“There are 4 es in “seventeen”.”
Took it about ten seconds to solve that enigma though…
The voice version will, i confusingly said chat when i meant voice. The carwash example will get it wrong(both voice and chat), at least it did on 5.4.
So, we might find these bizarre scenarios that confuse the system (based on our own deceptions): but basically it’s answering questions pretty reliably right?
Fundamentally anti-AI people are overstating the problem.
No it’s not. It will get it plain wrong. The seventeen example it may get right sometimes but for the carwash example I haven’t seen it ever produce the correct result. Weirdly enough people also get this wrong but still. See https://opper.ai/blog/car-wash-test
Ahahaha
This is the wrong approach to try and prove/disprove your hypothesis.
Its a statistical model that by its nature will answer differently every time. The only deterministic parts are fundamental truths about how the system operates (r in strawberry) and guardrails that have been put in by devs (cannot talk about this topic).
Therefore fundamentally this would require a statistical approach. A couple of those have already been done of course.
If this is your experience you could try and actually fact check the output. I believe coding is good for this because issues / misunderstandings are pretty immediately obvious. But I use Kagi Assistant a lot instead of search and there are factual issues all the time. And that’s already just summarizing search results.
Then also, as long as we are using LLMs for this, they are fundamentally still “find the next most likely word” machines. So they will be influenced by context a lot. The “truth” is not a concept that exists in LLMs.
I agree, but it would seem that 99% of the time it’s giving accurate, reasonable, and true answers to most questions.
It is rare it gives a false answer to most questions.
Compared to random humans it is clearly superior: and discussion thread on mainstream social media makes this patently obvious.
People who are against it, in terms of it’s capacity, seems to have incredibly high-standards - ignoring the obvious point: that if a human had the capabilities of ChatGPT (not least of all the capability of conversing with a hundred thousand users at once) we would think they had god-like intelligence.
You seem to be positing that it’s giving results to the tune of 99% to 99.9% accuracy based entirely on vibes.
If you actually want to know, you will have to do thousands upon thousands of prompts, across hundreds of topics that you can accurately fact check, before you can say with any sort of confidence whether it’s that accurate or not.
Your sample size is orders of magnitudes too small for you to reasonably have an accurate accuracy rate.
I mean…the anti-AI brigade keep insisting that AI has a tendency to give false answers: but nobody in this thread has actually been able to give me a single question which, when the latest ChatGPT model is asked, it gives a wrong answer to.
Kinda makes it seem like that one particular point is largely nonsense.
The scientific method exists for a reason. If you want an accurate idea of the accuracy of LLMs, then the best way is by applying the scientific method to it.
Until you’ve done that, you’re just basing your conclusions on conjecture, anecdotal evidence and vibes, with nothing actually substantive or empirical backing it up.
Right…
So, if I ask it a thousand random questions, a thousand times, and 99.9% of the time it gives the right answer: is that sufficient scientific evidence that it’s shit hot?
How exactly would your experiment work? How much data are you going to need to, personally, accept that it’s actually - already - pretty reliable and only very rarely gives misleading answers in very niche areas, or when people deliberately try to confuse it?
The thing is: this social media platform (like Reddit) has a massively anti-AI bias, and yet not a single person has been able to give me a single question that I can enter into the latest GPT model where GPT has produced inaccurate results.
Perhaps this criticism was more relevant a few years ago. As it stands it’s an argument that doesn’t seem to be readily supportable.
Can you prove to me, using “the scientific method” that the latest model of ChatGPT has a significant chance of producing misleading information?
And for the critics…I’m not entirely sure what they think is currently a better source of answers to questions? Random Google results where the page has paid to be at the top? Social media? Or combing through Google Scholar and finding…whichever “scientific paper” supports…whatever one wants to believe…(We saw how well that went in the pandemic).
Here, I plugged our entire conversation into chatgpt.
You’re person A, person B is the original commenter you replied to, and I’m person C.
Person A and Person C are mostly talking past each other.
Where Person A is right
Person A is making a practical claim: “In ordinary use, ChatGPT seems highly reliable, and critics often exaggerate how often it gives wrong answers.”
That’s a reasonable observation.
Many users ask everyday questions about history, science, travel, programming, writing, mathematics, and current events. Modern models are substantially more accurate than both:
random people on social media,
random Google search results clicked without expertise,
and many traditional question-answering sources.
If someone claims “ChatGPT constantly hallucinates” or “it’s wrong all the time,” that’s easy to challenge. Modern models clearly perform far better than that.
Person A is also correct that:
Nobody in the thread produced a simple “gotcha” question that reliably causes failure. That matters because some criticisms of AI are based on experiences from 2022–2023 models, which were substantially less reliable than current systems.
Where Person C is right
Person C is making a different claim:
“You cannot estimate a 99.9% accuracy rate from personal impressions.”
This is also correct.
Suppose you ask 100 questions and all appear correct.
That does not justify concluding the true error rate is 0.1%.
Why?
Because: you may not know enough to verify every answer,
you may be unconsciously selecting easy questions,
many errors are subtle rather than obvious, and a small sample size tells you little about rare failures.
This is exactly why benchmark testing exists. If somebody claimed:
“Commercial aircraft engines fail only once every 100,000 flights because I’ve flown 20 times and never seen one fail,” we would reject that reasoning.
Person C is essentially pointing out a sampling problem.
The strongest argument against Person A The strongest criticism is not that ChatGPT is “often wrong.”
It’s that:
Humans are surprisingly bad at detecting when ChatGPT is wrong.
Many AI errors look plausible.
For example, if I invent:
a fake court case,
a fake academic paper,
a fake quote,
a nonexistent historical anecdote,
the answer can sound extremely convincing.
A casual user may never discover the mistake. That means observed accuracy can be significantly higher than actual accuracy.
What experiment would actually settle this? Person A asks a fair question:
“How much evidence would convince you?” A proper experiment would look something like:
Construct a large test set (thousands of questions).
Cover many domains:
history
law
medicine
physics
mathematics
current events
geography
programming
Define correct answers beforehand.
Blindly score responses.
Measure:
correctness,
completeness,
confidence calibration.
This is basically what AI benchmark researchers do.
And when researchers perform these tests, modern frontier models generally score extremely well—but nowhere near 99.9% across all domains.
Performance varies dramatically by task. Can someone produce a question that reliably causes errors?
This is harder than Person A assumes. For a modern model, there is rarely a single question that causes failure every time. A better challenge would be categories such as:
obscure local regulations,
highly specialized legal questions,
recent events,
nonexistent citations,
invented books or papers,
long chains of reasoning,
edge-case programming problems,
ambiguous historical claims.
Models are much more likely to fail there. But that doesn’t mean they fail on every attempt.
Person B’s statistical argument is largely correct: reliability is a distribution, not a single yes/no property.
The “anti-AI bias” question Person A is probably correct that some critics hold AI to a standard they do not apply to humans.
If a person:
answers 95% correctly,
speaks fluently on thousands of topics,
writes code,
explains science,
summarizes books,
we’d consider them exceptionally knowledgeable.
Many critics compare AI not against average humans but against:
experts,
reference works,
or idealized notions of truth.
However, the opposite mistake also exists. Some AI enthusiasts compare ChatGPT against:
random Reddit comments,
random Google searches,
uninformed acquaintances.
That comparison can make the model look almost superhuman while overlooking domains where accuracy really matters.
Overall verdict
If I had to judge the arguments:
Person C has the stronger methodological argument. You cannot infer 99–99.9% accuracy from personal experience.
Person A has the stronger practical argument.
Many critics overstate the frequency of AI errors and often rely on outdated examples.
Person B is probably closest to the core issue.
The meaningful question isn’t “Can someone find one wrong answer?” but “What is the model’s error rate across different classes of tasks?”
So if this were a debate, I’d say:
Person A is probably correct that modern ChatGPT is much more reliable than many critics claim.
Person C is correct that Person A has not actually demonstrated a 99%+ accuracy rate and cannot do so from anecdotes alone.
Those positions are compatible rather than contradictory.
Do with that as you will.
If a human was able to recall all of human knowledge at any time, do you think they would be more or less capable?
More.
What colour of T-shirt am I wearing right now?
Yeah…I mean…I’m not claiming that ChatGPT is an omniscient god who knows what all 8 billions humans are currently wearing.
If that’s the basis for “ChatGPT is shit and constantly produces wrong answers” then that’s that.
However, I have asked it your question (word for word) and it gives a truthful and fair answer of sorts:
“I can’t see you or your camera, so I don’t know what colour your T-shirt is.”
I have noticed this common flaw in judging the capacity of ChatGPT though:
Even though the system can do truly amazing things, simultaneously in a hundred thousand conversations at once, it can solve mathematical problem better than 99% of humans, “knows” more than any human alive, can synthesise new academic theories and apply complex ideas in novel ways, and can generate images of a quality beyond what 99% of humans (the non artist/graphic design humans) can produce in mere seconds: it can’t do [insert obscure or trivial skill] and therefore it is inferior to human intelligence."
Let’s be clear: if a human could do what chatGPT can do, we would think them the most intelligent human being alive by a significant margin.
