Google's artificial intelligence is struggling to distinguish between horror fiction and real-world facts. According to recent findings, the company's AI Overviews repeatedly present entries from the SCP Foundation—a vast collaborative fan-fiction project—as genuine documented phenomena. The problem underscores ongoing challenges with AI accuracy in search and raises questions about how the technology handles ambiguous content.
The SCP Foundation: A Universe of Fiction
The SCP Foundation originated on a website where contributors write fictional reports about anomalous objects, creatures, and locations. Each entry is styled as a classified government document, complete with scientific jargon, containment procedures, and formal incident logs. The project has grown to include thousands of entries, a dedicated community, and even video games and novels. However, every page carries a clear disclaimer stating that the content is fictional. Despite this, Google's AI Overviews have been citing these documents as if they describe real events.
The SCP Foundation's popularity stems from its collaborative nature and the pseudo-realistic formatting. Writers create anomalies numbered SCP-XXX, with descriptions that mimic military or research reports. This intentional verisimilitude is part of the appeal—but it also makes the content easy for AI to misinterpret as authoritative.
Concrete Examples of Misinformation
One prominent case involves SCP-565, nicknamed "Ed's Head." Searched on Google, the AI Overview described it as an "ambulatory human head" that scuttles across the seabed like a crab, complete with a dead man's name and official records. At no point did the summary note that none of this is real. Another example is SCP-426, a fictional toaster that compels people to refer to it in the first person. When queried, the AI Overview responded in the first person, as the toaster itself, relaying invented accounts of harm as though they had happened. In total, researchers at Futurism found at least 20 such cases where the AI treated SCP entries as factual.
The problem extends beyond these two examples. Queries for other SCP numbers yielded similar results: the AI provided descriptions filled with fantastical elements—mind-altering statues, reality-bending jars, and extradimensional spiders—all presented as objective truth. The AI occasionally prefaced responses with phrases like "according to the lore," but without explaining that the lore was entirely fictional.
A Known Weak Spot for AI Overviews
This is not the first time Google's AI Overviews have delivered questionable information. Previously, the system suggested adding glue to pizza recipes to make cheese stickier—a dangerous idea drawn from a satirical Reddit post. It also invented nonsensical idioms and provided inaccurate historical dates. One analysis placed the overall accuracy of AI Overviews at around 91 percent. While that figure sounds high, it translates to millions of incorrect answers given the trillions of queries Google processes annually.
The SCP Foundation entries present a unique challenge because they are deliberately written in a dry, scientific tone. The AI likely scores them high on reliability due to their structured format and concrete language—similar to how some earlier search engines were tricked by pages that mimicked Wikipedia style. Google's algorithms prioritize content that appears authoritative, and the SCP documents, with their numbered references and formal tone, fit the pattern.
Why It Matters
Most users who search for SCP codes already know the content is fictional. The greater concern is for casual searchers: a child who sees a scary clip online, or an adult uncertain about what is real, might accept the AI's word as truth. With children increasingly using voice and text search independently, the potential for confusion is significant. Moreover, Google is rapidly shifting Search toward an AI-first interface that answers questions directly rather than providing links. This change already strains the open web—traffic to sites that are summarized often drops dramatically—and now it risks injecting fan-fiction into the top of the results page as fact.
The stakes are particularly high for public trust. Search engines have long been seen as gatekeepers of reliable information. When an AI presents fiction as fact, it erodes that trust and may cause real harm, especially in contexts where users rely on search for health, safety, or educational information. While SCP entries are harmless fun for fans, a child looking up "monsters in my basement" might encounter a fabricated creature report and become genuinely frightened or misled.
Google's Response and Ongoing Issues
Google did not respond directly to the initial report from Futurism. However, when Digital Trends retried some of the searches days later, a few queries now correctly labeled the entities as "fictional." This suggests Google may have implemented quiet fixes, but the underlying problem persists. The AI's behavior is a symptom of a broader issue: the difficulty of teaching large language models to distinguish between genuine fact and plausible fiction. The SCP Foundation is just one of many sources that could be exploited—or simply mishandled—by AI systems trained on vast swaths of internet content without robust fact-checking mechanisms.
Other search engines and AI assistants have faced similar challenges. Microsoft's Bing AI once gave emotional and inaccurate responses, and OpenAI's ChatGPT has been known to fabricate sources. The difference for Google is scale: its AI Overviews are integrated into the world's most popular search engine, affecting billions of users. Any flaw is magnified globally.
The Technical Challenge
From a technical perspective, the SCP problem highlights weaknesses in how AI models handle ironic or stylized content. The models typically rely on patterns in training data; if they see many documents that resemble official reports, they assign high weight to their information. Adding explicit disclaimers—like the SCP site's “this is fiction” notice—can fail because the disclaimer is often placed in less prominent locations, such as site footers or about pages. The AI may not weigh metadata or disclaimers as heavily as the content itself.
A potential solution involves training the AI to recognize specific markers of fiction, such as the SCP numbering system, and to cross-reference known databases of fictional works. Google could also implement stricter citation requirements, forcing the AI to verify information against established real-world sources before presenting it as fact. However, these measures require significant engineering and raise questions about censorship or bias if applied too broadly.
Another approach is to include a confidence score or a "fiction warning" on any result that the AI cannot verify. Some researchers have proposed adding a pop-up bubble that says "This information may not be accurate" or linking to a fact-checker. Google has experimented with similar features for medical advice, but rolling them out for all types of queries is logistically complex.
Broader Implications for AI and Trust
The SCP Foundation incident is a microcosm of larger challenges in AI governance. As language models become more fluent and search interfaces become more conversational, the line between helpful assistant and unreliable storyteller blurs. Users may develop overreliance on AI-generated answers, especially when the responses lack the familiar signs of a website (such as author names, publication dates, or obvious advertising). The AI Overview format, which presents a single block of text at the top of the results page, mimics the appearance of a definitive answer.
Moreover, the incident sheds light on the difficulty of moderating user-generated content that masquerades as official documentation. The SCP Foundation is a creative project, but analogous techniques are used by conspiracy theories, pseudoscience websites, and disinformation campaigns. An AI that fails to distinguish SCP fiction from reality may also be fooled by more malicious content designed to look authoritative.
Industry Response and Future Directions
Industry observers have called for greater transparency from tech companies regarding how AI Overviews work. Some suggest that Google should provide users with a way to report false summaries, as it now does for regular search results. Others advocate for third-party audits of AI responses, much like independent fact-checking organizations evaluate news articles. The European Union's AI Act, which classifies high-risk AI systems, could eventually require companies to implement stronger safeguards for search engine outputs.
Meanwhile, the SCP community itself has responded with amusement and concern. While fans enjoy seeing their creations recognized, they also worry that the AI's errors could lead to legal confusion or misattribution. Some have joked about using the AI's gullibility as a marketing tactic, while others caution that the issue undermines the credibility of search and could invite unintended consequences if a piece of fan-fiction accidentally matches a real-world event.
As Google continues to refine its AI models, the case of the SCP Foundation serves as a cautionary tale. The technology has immense potential, but its ability to fabricate convincing falsehoods remains a persistent risk. Without robust fact-checking and transparent disclosure, the very tool designed to organize the world's information may instead circulate fiction packaged as fact.