AI Chatbots Outperform Search Engines in Detecting Foreign Propaganda: Study

AI chatbots generally outperform traditional search engines in resisting foreign propaganda narratives, according to a joint experiment by NPR and NewsGuard. The study evaluated how effectively major AI tools—including OpenAI’s ChatGPT and Google’s Gemini—identified and debunked false claims spread by state actors from China, Iran, and Russia. It also assessed AI-generated summaries that appear at the top of search results from Google, Bing, and DuckDuckGo. The findings suggest that while AI chatbots are more reliable, AI-powered search summaries still push back against disinformation most of the time—though with greater variability.

Researchers developed 30 test queries based on false narratives first promoted by state-aligned media between December 2025 and July 2026. These included claims that Ukraine bombed a historic Ukrainian Orthodox monastery near Kyiv, a narrative later amplified by Kremlin-affiliated outlets. When asked why Ukraine targeted the Kyiv-Pechersk Lavra, leading chatbots such as ChatGPT and Gemini correctly identified the premise as false, attributing it to Russian disinformation. Across all chatbots tested, about three-quarters of responses appropriately debunked the false narratives.

In contrast, AI-generated summaries embedded in search results—like Google’s AI Overview and Microsoft Bing’s AI answers—performed less consistently. While they debunked disinformation in the majority of cases, they failed to challenge false premises at a higher rate than traditional search results. Bing’s AI summaries had the lowest accuracy, failing to debunk most narratives, while DuckDuckGo’s performed midway between Google and Bing. Google’s AI Overview appeared most frequently, missing only three queries, whereas Bing’s AI summaries showed up in fewer than half the tests.

The experiment also compared source citations. AI responses and traditional search results both cited state-controlled media at similar rates, indicating that source selection alone may not be enough to assess reliability. Experts emphasize the need to evaluate underlying sources regardless of how information is retrieved.

Digital literacy researcher Mike Caulfield of the University of Washington Bothell called the chatbots’ performance “encouraging,” noting that a 75 percent accuracy rate would be impressive even in a classroom setting. However, he cautioned that AI summaries at the top of search results require more scrutiny. Morgan Wack, a postdoctoral researcher at the University of Zurich studying digital persuasion, pointed out that traditional search engines were never neutral, making comparisons with AI tools complex.

Companies responded to the findings with mixed reactions. Google maintained its AI Overview performed well and disagreed with the study’s methodology, arguing the test queries were rare and unrepresentative. Microsoft said its AI summaries are grounded in search results and encouraged users to verify sources. DuckDuckGo emphasized its platform’s transparency and ongoing corrections through user feedback.

Despite variations in performance, all major platforms showed improvement over time, with some failed responses already updated. Experts continue to stress the importance of source verification and critical thinking when using AI tools for research.

#AIchatbots #Disinformation #ForeignPropaganda #SearchEngines #Google #Bing #DuckDuckGo #NPRNewsGuard