The AI assurance trap: When agents generate their own evidence

The AI assurance trap: When agents generate their own evidence

For millennia, the definition of “work” was defined by the grip of a hand on a tool. When a craftsman held a hammer, the control was absolute. The feedback loop was instant – physics, muscle, result. Then came industrialization. We built machines that amplified force but abstracted control. The worker no longer held the tool; they pulled a lever. Yet, we maintained a safety net with an “industrial expert.” If the machine broke, a master mechanic understood every gear and piston. We had regulations, safety stops and a clear understanding of cause and effect. As we moved into the age of software, that control began to drift. We built systems of logic so complex that security breaches became inevitable. Companies paid the price in data and dollars, but we still operated in a world of “deterministic logic.” Code was written by humans, read by humans and (mostly) understood by humans. Then came the first wave of AI. We started with “white box” models, then moved to opaque “black boxes.” Now, we have breached the final wall. We have moved from models that predict to AI agents that act . The reality check: It’s not just engineers anymore I have observed a concerning pattern recently — both at technical Ideathons I’ve attended and in conversations with peers across the industry. We are witnessing a massive shift in the fundamental nature of engineering. In many of these events, 80% to 90% of the projects are based on AI agents. The energy is undeniable. Agents are the “new car” that every executive and developer wants in their driveway. But what stands out is who is building them. It isn’t just engineers new to AI. I saw participants from completely different backgrounds – business analysts, QA testers and non-technical domain experts – suddenly developing complex AI agents. These systems appeared to rival the work of seasoned software architects. They were building fast and also building cheap. But they weren’t actually building . This is the illusion of competence. It is not that the creators were using tools to work faster; it is that the large language model (LLM) had become the architect. The human operator had been demoted to a mere runtime environment — a pair of hands typing what the model suggested. The trap of circular validation The counterargument is often: “If the AI writes the code, we still have people to review it, test it and ask the AI to explain it.” This is a dangerous comfort, especially in regulated industries like finance or healthcare where I operate. In many modern workflows, the operator asks the agent to write the function, asks the agent to write the unit tests and asks the agent to document the logic. This creates a closed loop of circular validation. In a regulated industry, we don’t just need to know what the code does (explanation); we need to know why it is the correct architectural choice (intent). When we rely on an agent to build the system and then ask the agent to explain it, we feel safe, but we have lost the “first principles” understanding. When a subtle, non-obvious failure happens — such as a bias in the data pipeline that the LLM doesn’t recognize as a bias — the operator cannot spot it because they didn’t build the actual base model of the system. They are just “operators” of a black box. The solution: Evidence-based engineering We cannot ban AI, nor should we. But we must fundamentally change the contract between the Human and the agent. We must move from “generation” to “justification,” adopting rigorous standards like those found in the MLOps Manual or internal governance frameworks. To regain control, we must implement three pillars of evidence-based engineering: 1. The “citation mandate” In a regulated environment, code that works is not enough; we need to know why it works. We must configure our AI workflows to demand evidence. If an engineer asks an agent to design a microservice architecture or write a complex SQL query, the agent must be required to provide valid links, official documentation or internal wiki citations that support its decision. The rule: No code is accepted into the codebase unless the engineer has verified the primary source linked by the agent. The effect: The AI finds the knowledge, but the engineer consumes it. This forces the human back into the loop of learning. 2. Independence through mastery We often use AI explanations to pass a compliance check. This is backward. The engineer must use the AI’s explanation to build their own base model. The goal of using an AI agent should be mastery. If an agent teaches you how to solve a specific concurrency bug today, you should understand the ‘why’ so deeply that you own the knowledge. If you are using AI to solve the same problem twice without learning, you aren’t engineering; you are copy-pasting. 3. The “signature” standard We need to reinstate the culture of “signing off.” When an engineer commits code generated by an LLM, they are essentially signing a legal document stating: “I understand every line of this. I know the edge cases. I own the failure.” We must stop accepting “The AI hallucinated” as a valid root cause for outages, especially in a sensitive industry. “I don’t know, the AI did it” is not an excuse; it is a lawsuit. The root cause is always: “The engineer failed to verify.” Cognitive scaffolding vs. offloading Ultimately, this comes down to how we view the tool. There are two ways to use AI: Cognitive offloading: You let the AI do the thinking, so you don’t have to. You save energy, but your skills atrophy and you become dependent. Cognitive scaffolding: You use the AI to support your thinking, allowing you to reach a higher level of understanding than you could alone – but you are still doing the climbing. We need to encourage “teacher prompts.” Instead of asking, “Write this code,” we should ask, “Propose a solution, cite the design patterns used and explain the trade-offs compared to Alternative B.” Conclusion LLMs are powerful engines. But in the hands of a passive operator, they are dangerous. In the hands of an engineer who demands evidence, citations and understanding, they are the greatest learning tool ever invented.