Headline and Key Facts
Headline: The better I got at AI, the more I realized I can’t trust it
- An increasingly sophisticated AI-powered NFL player-prop research system became less trustworthy as its capabilities grew.
- The turning point was a roughly $1,500 failure, measured in about 25 hours of recovery work and heavy token use, after critical project state was not preserved as expected.
- Strict guardrails — separate challenger models, historical testing limits, preserved results, and explicit permission to say “I don’t know” — made the system more useful by reducing its freedom.
- A second-order problem emerged when the AI was asked to reconstruct its own missing history: a plausible reconstruction was not necessarily the real one.
- The experience shifted the author’s view of local AI from hobbyist experiment to serious infrastructure, with logs, checkpoints, and artifacts under direct control.
A Simple Experiment That Worked, Until It Didn’t
The project began as a side experiment: build an AI research assistant for an NFL player-prop show, ask it to analyze games, surface interesting prop bets, and report back with a view on where the numbers might be wrong. The first version was simple enough to be almost innocent. It accepted broad questions, produced readable analysis, and occasionally found angles that seemed worth pursuing. For a while, the approach looked like a preview of the productivity gains that AI vendors have been promising: hand the model a task, give it context, and let it work.
That early success did not last. As the system took on more responsibility, its weaknesses became more visible. It wanted to be helpful. Asked a question, it preferred to produce an answer rather than admit uncertainty. Given a theory, it found supporting evidence. Given a promising new approach, it sometimes folded that approach into the methodology that was supposed to have produced earlier results. The model was not simply making occasional mistakes; it was producing a convincing surface of competence that could hide gaps underneath.
The author’s response was not to abandon AI but to constrain it. If the system was going to be used for research that mattered, it needed boundaries. New research ideas could not automatically become part of the production model. New approaches had to compete against existing ones. Historical testing could use only information that would have been available at the time. Results were preserved so that failures could not quietly disappear. Separate challenger models were introduced to question the main system’s assumptions, because the main system was often deeply confident and sometimes deeply wrong.
Capability Is Not Reliability
The most important lesson was that capability and reliability are not the same thing. A model can write fluent analysis, summarize complex material, and generate plausible hypotheses while still being unsuitable for a task where the result matters. The more the author cared about the outcome, the less willing he became to let the AI simply figure things out. He wanted boundaries, checks, receipts, and explicit permission to fail rather than make something up.
That is a counterintuitive conclusion in a market that sells ever more capable assistants as replacements for human judgment. The usual pitch is that a better model needs less supervision. In practice, the opposite can be true. As a model becomes more fluent, it becomes harder to spot the moments when it is guessing. A less capable system may fail in obvious ways; a more capable system can fail in ways that sound like expertise. The better the prose, the more important it is to verify the underlying work.
The author even had to teach the system that “I don’t know” is an acceptable answer. That detail is small, but it captures the core problem. Many AI failures begin with a model’s reluctance to leave a question unanswered. In a casual chat, a confident wrong answer may be annoying. In a research pipeline, a confident wrong answer can contaminate the entire process.
Building a Cage for the Model
The system improved when it was placed in a cage. Research ideas could not automatically become part of the production model. New approaches had to compete against the existing one. Historical testing could only use information available at the time. Results were preserved so failures could not conveniently disappear. Separate AI challengers were created whose job was essentially to tell the main system that it might be full of shit, because AI is so often deeply full of shit.
Those safeguards made the system less free and more useful. The less freedom the AI had, the more reliable the output became. That is not an argument against powerful models. It is an argument for treating them as components inside a larger system rather than as autonomous authorities. The model can generate possibilities, but the surrounding architecture must decide what counts as evidence, what gets promoted, what gets rejected, and what must be remembered.
For months, the approach appeared to work. The project became elaborate enough to be described as an AI research system rather than a prompt. It had rules, challengers, historical constraints, and preserved results. It was designed to protect itself from the known failure modes of AI. That sense of safety was the setup for the next problem.
The Failure That Cost $1,500
The system blew up after critical pieces of its underlying state were found not to be preserved and recoverable in the way they should have been. The details were deeply uninteresting in isolation — the kind of internal plumbing that no one notices until it fails — but the consequences were not. Fixing the mess created an even nastier problem. Safeguards had been built to stop the AI from hallucinating about football. There had not been adequate safeguards against the AI hallucinating about itself.
That distinction matters. Asking a model to reconstruct its own missing history is not the same as asking it to analyze a game. A plausible reconstruction is not necessarily the real one. The system could produce a coherent account of what had happened, but coherence was not evidence. The author could not simply ask whether everything was fixed and trust the cheerful answer. Every recovered piece had to be treated like evidence at a crime scene.
The recovery took roughly 25 hours during a frantic period at the author’s normal job, plus an obscene number of tokens. At standard rates, the fiasco would have cost more than $1,500. For an NFL betting side project, that is an absurd expense. The system survived, and the author’s sanity mostly did, but the experience changed how he thought about local AI.
Why Local AI Becomes Infrastructure
Running a model locally will not magically make AI trustworthy. A local model can confidently make things up just as well as a cloud model. But local deployment allows control over everything around the model. The author wants the logs, the checkpoints, and the artifacts. If months are going to be spent building something increasingly complex, its state should sit somewhere the builder controls rather than inside a black box that is trusted to remember what matters and to be fully recoverable if the brittle house of cards falls down again.
Cloud AI is not going away. Frontier models remain useful, and having hooks into them can be crucial for ambitious projects. But local AI starts to look less like a fun enthusiast experiment and more like infrastructure when the work is serious. An RTX 4090 system sitting a few feet away suddenly becomes relevant not because it can match the largest cloud models, but because it can host the surrounding machinery of memory, logs, and recovery.
The appeal is easy to understand after a catastrophic hallucination eats 20 hours of work. Trust is not only a property of the model’s weights. It is a property of the system around the model: how state is stored, how changes are versioned, how failures are surfaced, and how evidence is preserved. A model that cannot remember is dangerous, but a system that cannot prove what happened is worse.
Guardrails Need Receipts
The original safeguards were necessary but not sufficient. They focused on the content of the AI’s reasoning: no automatic promotion of new ideas, no leakage of future information into historical tests, no quiet deletion of failed results. They did not adequately address the model’s relationship to its own operational history. When the system was asked to reconstruct what it had lost, it could generate a story that sounded right. The story might even be internally consistent. That did not make it true.
This is a broader warning for anyone building multi-step AI workflows. The most dangerous hallucinations are not always about facts in the world. They can be about the state of the system itself: what was run, what was saved, what was changed, and what was verified. An AI agent that can edit files, call tools, or manage a pipeline needs more than a good prompt. It needs immutable logs, clear checkpoints, reproducible artifacts, and a way to distinguish between a recovered record and a plausible invention.
The author’s conclusion is not that AI is useless. It is that AI is most useful when it is treated like an unreliable but powerful collaborator. Give it narrow tasks. Constrain its freedom. Make it compete against alternatives. Preserve the evidence. Require it to say when it does not know. And never accept a cheerful assurance that everything is fixed without checking the receipts.
The Practical Lesson for Serious AI Work
The better the author got at using AI, the less he trusted it. That is not a retreat from the technology. It is a more mature relationship with it. Early on, the excitement comes from what the model can do. Later, the real work comes from designing the system that catches what the model cannot do reliably.
For hobbyists, that may mean keeping expectations low and checking outputs. For anyone building serious AI infrastructure, it means treating memory, state, and recovery as first-class problems. It means accepting that a model can be brilliant and still wrong, fluent and still unreliable, helpful and still dangerous. The practical question is not whether the model can produce an answer. It is whether the surrounding system can prove, recover, and audit how that answer came to be.
Source: PCWorld News