ChatGPT and Google Gemini are terrible at summarizing news, according to a
new study

Date:
Tue, 11 Feb 2025 12:00:05 +0000

Description:
A new study finds AI chatbots more often than not inaccurately summarize news.

FULL STORY

A new study from the BBC has found that four of the world's most popular AI chatbots including ChatGPT are inaccurately summarizing news stories.

The BBC asked ChatGPT , Copilot, Gemini , and Perplexity to summarize 100
news stories from the news outlet and then rated each answer to determine
just how accurate the AI responses were.

The study found that "51% of all AI answers to questions about the news were judged to have significant issues of some form." and "19% of AI answers which cited BBC content introduced factual errors, such as incorrect factual statements, numbers and dates."

The study showcases multiple examples of inaccuracies that showcased
differing information to the news it was summarizing. The examples note that "Gemini incorrectly said the NHS did not recommend vaping as an aid to quit smoking" and "ChatGPT and Copilot said Rishi Sunak and Nicola Sturgeon were still in office even after they had left."

Inaccuracies aside, there's another crucial finding. The report found that AI "struggled to differentiate between opinion and fact, editorialised, and
often failed to include essential context."

While these results are unsurprising considering how often we see issues with news summarization tools at the moment, including Apple Intelligence's
mix-ups that have led Apple to temporarily remove the feature in iOS 18.3 , it's a good reminder not to believe everything you read from AI.

Are you surprised?

From the study, the BBC concludes that "Microsoft's Copilot and Google's
Gemini had more significant issues than OpenAI's ChatGPT and Perplexity,"

While this research doesn't necessarily give us much more info, it validates the skepticism towards AI summary tools and emphasizes just how important it
is to take information from AI chatbots with a pinch of salt. AI is
developing rapidly and large language models (LLMs) are released almost
weekly at the moment, so it's to be expected that mistakes will happen. That said, from my personal testing I've found inaccuracies and hallucinations to
be less frequent now in software like ChatGPT than it was just a few months ago.

Sam Altman said in a blog post yesterday that AI is progressing faster than Moores law and that means we'll continue to see constant improvements to software and how it interacts with the world around it. For now, however,
it's probably best not to trust AI for your daily news.

======================================================================
Link to news story: https://www.techradar.com/computing/artificial-intelligence/chatgpt-and-google -gemini-are-terrible-at-summarizing-news-according-to-a-new-study

$$
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)

ChatGPT and Google Gemini are terrible at summarizing news, according to a new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.


--- AfterShock/Android 1.7.5
* Origin: South of Heaven - Chaos rampant, an age of distrust (1:154/30.1)

ChatGPT and Google Gemini are terrible at summarizing news, according new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.

Are you not buying into the hype of AI?

I haven't seen it do anything too awesome yet. Gemini has been a little handy for getting examples of things, but I haven't seen it save any lives yet, or take anyones job.

I have a difficult time believing anything until I see it, and I haven't seen AI yet.

--- Mystic BBS v1.12 A49 2023/04/30 (Windows/64)
* Origin: JoesBBS.Com, Telnet:23 SSH:22 HTTP:80 (1:342/201)

ChatGPT and Google Gemini are terrible at summarizing news, according to a >MP> new study

No one should be shocked by this. So called "AI" is just your phone's autocorrect on steroids.

No one sensible should be shocked. How many people do you know who would likely take AI's word for it, so long as the misinformation it provides confirms their beliefs?


* SLMR 2.1a * If it's Tourist Season, howcum we can't shoot 'em, Pa?
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)

I haven't seen it do anything too awesome yet. Gemini has been a little handy for getting examples of things, but I haven't seen it save any lives yet, or take anyones job.

Not yet.

I have a difficult time believing anything until I see it, and I haven't seen AI yet.

If you done any google (or bing or...) searches at all, you've seen it and probably not known it.

While recently, Google has started doing a better job of labeling their own
AI search responses as "AI generated," are you sure that other information you have read in past, or other search results it presents from other sites, are also not AI generated?

It was recently uncovered that some news sites were using AI to summarize
their news articles with very mixed to downright libelous results. Someone
got sued over it.


* SLMR 2.1a * Been there, done that, got the T-shirt.
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)

If you done any google (or bing or...) searches at all, you've seen it
and probably not known it.

While recently, Google has started doing a better job of labeling their own AI search responses as "AI generated," are you sure that other information you have read in past, or other search results it presents from other sites, are also not AI generated?

I've seen it on the internet, but I just haven't seen it serve any useful purpose. (News isn't that useful, swear filters aren't that useful, customer service chatbots aren't that useful.)

--- Mystic BBS v1.12 A49 2023/04/30 (Windows/64)
* Origin: JoesBBS.Com, Telnet:23 SSH:22 HTTP:80 (1:342/201)

If you done any google (or bing or...) searches at all, you've seen it and probably not known it.

While recently, Google has started doing a better job of labeling their own AI search responses as "AI generated," are you sure that other information you have read in past, or other search results it presents from other sites, are also not AI generated?

I've seen it on the internet, but I just haven't seen it serve any useful purpose. (News isn't that useful, swear filters aren't that useful, customer service chatbots aren't that useful.)

Well, that is part of the problem. It is not very useful yet and, as a
result, can lead people into doing things they should not, especially if
they do not realize the information they received is from AI and not a good source.


* SLMR 2.1a * Shin n. device for finding furniture in the dark.
--- SBBSecho 3.20-Linux
* Origin: capitolcityonline.net * Telnet/SSH:2022/HTTP (1:2320/105)