Google's AI Overviews Faces Criticism for High Error Rate
Testing suggests Google's AI Overviews tells millions of lies per hour
Ars Technica
Image: Ars Technica
Google's AI Overviews, powered by the Gemini model, has been found to provide incorrect answers 10% of the time, leading to potentially millions of false responses daily. A recent analysis revealed its accuracy improved to 90% after updates, but significant errors remain, raising concerns about its reliability.
- 01AI Overviews has a 10% error rate, equating to hundreds of thousands of inaccuracies every minute.
- 02The accuracy of AI Overviews improved from 85% to 90% after the Gemini 3 update.
- 03The analysis was conducted by The New York Times in collaboration with AI startup Oumi.
- 04Examples of inaccuracies include wrong dates and misleading claims about organizations.
- 05The SimpleQA evaluation tool was used to assess the AI's factuality.
Advertisement
In-Article Ad
Google's AI Overviews, which debuted in 2024, has been scrutinized for its accuracy in providing information. A recent analysis by The New York Times, aided by the AI startup Oumi, found that while the AI's accuracy has improved to 90% following the Gemini 3 update, this still means it generates hundreds of thousands of incorrect answers every minute. The SimpleQA evaluation, which tests AI models with over 4,000 verifiable questions, revealed that 1 in 10 answers from AI Overviews is wrong, leading to concerns about the reliability of information users receive. The report highlighted specific instances where the AI provided incorrect information, such as conflicting dates regarding Bob Marley’s former home and erroneous statements about the Classical Music Hall of Fame. These findings underscore the challenges Google faces in ensuring the accuracy of its AI-driven search tools.
Advertisement
In-Article Ad
The inaccuracies in AI Overviews can mislead users, affecting their access to reliable information.
Advertisement
In-Article Ad
Reader Poll
Do you trust AI-generated information from search engines?
Connecting to poll...
More about Google
Read the original article
Visit the source for the complete story.




