Working in software, you learn about the term technical debt: after enough time, unchecked bad patterns build up and a piece of software starts to accumulate debt. Debt that slows people down. In the world of AI, I think a new kind of debt is starting to form. Comprehension debt, and just like the technical kind, if left unchecked it leads to negative returns on the productivity it’s promising.

At first its simple

When you pull up a window to chat with AI you usually have a pretty clear picture of what you’re trying to get at. “Why does a whiteboard marker erase permanent marker on the whiteboard?” That’s a straightforward question. You have the context on what’s being asked since you’re the one asking it, and whatever comes out, you’ll most likely have enough understanding from high school science to reason about the response.

As AI has evolved, we’ve shifted from simple questions to actions, and then to iterations on our actions. We build agentic systems now that take AI from guessing the next letter to building complex applications. The problem I’m seeing is that while with smaller, simpler problems it’s pretty easy to reason about what’s spit out, as the agentic systems we build get more and more complex we’re going to start to struggle with why certain decisions are being made. Or worse, we won’t even bother to check.

Going full auto

That lack of checking, just letting the agentic behavior go full “auto,” is amazing to know is possible. It also makes it a lot harder to really claim understanding, or to make a clear judgment call, when the decisions the system is making sit past the edge of your comprehension.

This is the comprehension debt. As we move faster and faster and let machines do more and more, every time we opt to let the system ride without checking, we’re choosing to add debt to our own understanding. The alternative is actually reading what got decided, and letting that little bit of friction do its work. The friction is what keeps you connected to the decision.

We don’t need to know everything, but we need to know enough

Now, I’m not arguing we have to hold on to all of it. There’s the famous toaster project, where Thomas Thwaites tried to build a toaster from scratch, mining his own ore, and after nine months ended up with something that worked for about five seconds. I have no idea how a toaster is made, and I don’t need to. Technology is bound to make information more and more abstracted.

I think the difference is that with a toaster, someone knows. Someone has the system. We built a whole industry around toaster making, with standards and inspectors and warranties attached to it. The understanding is out there, it just isn’t in my head, and that’s fine. With agentic systems we don’t have any of that yet. No industry, no inspectors, no warranty. Which means for now, the only person in a position to catch it is the person running it. And maybe that’s what we need to build.

So as we work more and more with agentic systems, we can’t just treat them as black boxes. We have to treat them like systems, and in that system we should be building ways for us to peek in and understand. Understand the decision, understand the reasoning. You don’t need the full picture, just the key points, explained well enough for your mind to wrap around.

What if the system taught us?

There’s a talk I listened to recently by Geoffrey Litt that I thought was brilliant . The gist of it was that we should be pushing AI to teach us. And with vibe coding being as easy as it is, why not have AI vibe code the lessons for us, test us on them, and put the burden of our understanding onto the system itself.

That really unlocked my perception of what a website could even be. It’s no longer just about putting something online for people to look at. They really are just another form of rich documentation, interactive, a tool for learning. We’ve just gotten to the point where it’s cheap enough to generate one automatically. Which might be the closest thing to an inspector we’ve got.