What I Built This Week That Broke, and Why That's the Point

I built something this week that did not work. Not in the way failure usually stays quiet in consulting, buried in a client call with a promise to iterate. This was a tool meant for a training company we work with, designed to help their instructors surface common questions from past cohorts so they could anticipate what the next group would ask. The premise was solid. The execution was half-baked. And I shipped it anyway.

I want to walk you through what happened, not because the story is particularly elegant, but because the way it broke and how we responded is the actual lesson. Most people learn about AI by watching polished case studies or reading about tools that worked perfectly on the first try. That is not how any of this works in reality. The real learning happens when you build something, it fails in a specific way, and you have to figure out why.

The idea started simple. Our client's instructors were spending time at the start of each cohort reviewing recordings and notes from the previous cohort to anticipate common confusion points. That is good teaching practice. It is also time-consuming. We thought an AI tool could help compress that work. Instead of watching hours of video and scrolling through chat logs, an instructor could feed those materials into a system and ask it to surface the ten most-asked questions, the moments where learners seemed most confused, the topics that took longest to explain. Then the instructor could design the next cohort's curriculum around those patterns.

On paper, it made sense. So we built a workflow. We set up a document ingestion pipeline, we trained the system on a sample of their cohort materials, we created a simple interface where an instructor could upload materials and get back a ranked list of questions. We tested it internally. It produced outputs. It looked reasonable. So we told the client it was ready.

It was not ready.

The first instructor to use it noticed something almost immediately. The tool was surfacing questions that had appeared maybe once or twice across all the cohort materials, treating them as high-priority. Meanwhile, it was burying questions that came up repeatedly, questions that had clearly been sticking points for half the cohort. The rankings were backwards. The system was optimizing for novelty instead of frequency, flagging outliers as important while smoothing over the actual patterns.

When I dug into why this happened, the reason was both obvious and instructive. I had tuned the system to avoid returning duplicate results. The logic was meant to be helpful, don't return the same question five times. But in trying to eliminate redundancy, we had accidentally trained it to penalize frequency. A question that appeared in five different contexts got deprioritized because the system saw it as "the same question repeated." A question that appeared once in a unique framing got boosted because it looked fresh.

This is a perfect example of how AI systems optimize for what you measure, not what you want. I measured deduplication. The tool delivered deduplication. I did not measure whether the instructor's actual need, find the patterns that matter most, was being served. So it was not.

Here is what matters about this failure, though. It happened fast. It was caught immediately. And it was caught by the person who actually uses the tool, not by me testing it in a vacuum.

That last part is critical. If I had kept iterating in the testing environment, keeping the tool private until it was "perfect," I probably would have missed this problem entirely. Why? Because I would have kept testing it against the criteria I thought mattered. Deduplication would have looked clean. The outputs would have ranked nicely. The system would have looked sound. But it would have failed the moment it met the real world.

Instead, by shipping early and letting the actual user catch the break, we discovered the misalignment between what I built and what was actually needed. That is not a bug in the process. That is the process working correctly.

So we rebuilt the ranking logic. We changed the optimization from "avoid duplicates" to "rank by frequency across all contexts." We added a second layer that allowed an instructor to drill into why the tool thought a question was important, what specific patterns it had found that elevated it. And we pushed it back out with that transparency built in.

The second version worked better. Not perfectly. But it actually served the work.

Now, I want to translate this into something useful for you, because the lesson here is not specific to our training company client or to this particular tool. The principle applies whether you are building something yourself or evaluating an AI system someone else built.

If you are an owner thinking about bringing AI into your business, you have probably heard a lot about the importance of getting the right tool, finding the right vendor, asking the right questions before you buy. That advice is not wrong. But it misses the bigger point. The tool is not the hard part. Understanding what actually matters for your work is the hard part. And you cannot do that without using the tool in real situations.

There is a trap that many owners fall into when they start with AI. They want it to be right before they deploy it. They want to vet it thoroughly, test it against scenarios, make sure it will work. All of that is reasonable instinct. But it is also the fastest way to end up with an AI system that does not actually work for your business, because you will have designed it around assumptions instead of real use.

The alternative is to think of AI the way you would think about hiring a new team member. You would not hire someone, lock them in a training room for three months, never let them actually do the work, and only release them when they were perfect. That would be absurd. You would bring them in, train them, put them in front of actual work, watch what happens, and course-correct. You would learn what they are good at and what they are not. You would find the gaps between what you expected and what the reality is. Then you would adjust.

That is how you should approach AI in your business.

This means a few practical things. First, you need to be willing to deploy something that is not perfect. Not recklessly. Not untested. But ready. Ready means it does the core job without completely falling apart. It does not mean it is optimized. It does not mean you have thought of every edge case. It means you can put it in front of the people who actually do the work and learn from what happens.

Second, you need to be explicit about what you are measuring. In our case, I thought I was solving for deduplication. But the actual goal was frequency and importance. That gap is what created the failure. So before you deploy an AI tool, decide what success actually looks like for your business. Not what the vendor says it should look like. What does success mean in your context? If you are using AI to handle customer intake, is success speed? Accuracy? Coverage of all the questions a customer might have? Those are very different things, and an AI system optimized for one might fail at another.

Third, you need someone using the tool who can tell you when it is wrong. This is not a vendor relationship. This is feedback. The worst outcomes I have seen with AI happen when an owner buys a tool, deploys it, and never actually watches what it is doing. They trust that it is working because it is running. You need at least one person who is skeptical enough, or close enough to the work, to say "that does not match what we actually need."

Fourth, plan for iteration, not perfection. If you approach AI as a install-and-forget tool, you are going to be disappointed. If you approach it as something you will tune and adjust based on what you learn, you will get better results over time. That is not a weakness in AI. That is how learning works. You try something, it tells you something about reality, you adjust, you try again.

The build-in-public approach forces all of this. Because you cannot hide. The tool fails in front of the person who uses it. You have to see the failure and respond. You cannot convince yourself it is working when the evidence is right there that it is not. That pressure, that visibility, is actually the accelerant for better outcomes.

So if you are thinking about bringing AI into your business, or if you have started but are not getting the results you hoped for, ask yourself these questions. Am I being clear about what success actually looks like for this work? Am I deploying something that is ready but not perfect, or am I waiting for perfect? Do I have someone close to the work who can tell me when something is wrong? And when I find out it is wrong, am I treating that as a failure of the tool or as useful information about what I actually need?

The tool that broke this week was not a failure. It was the first version of something that works. And it only became that because we were willing to let it break in front of the person who uses it, in real conditions, where the actual gap between what I designed and what was needed could show up.

That is the only way to build AI that actually serves your work.