What to Expect from your AI Project

Introduction

You’ve signed off on an AI project. The proposal made sense, the team seems capable, and now you’re waiting for updates. Then the first status call happens, and it doesn’t sound like the software projects you’ve managed before. There’s a percentage instead of a checklist. There’s a phrase like “the model got it wrong on the first pass, so we’re changing the prompt.” That’s not a red flag. It might be exactly what a healthy AI project sounds like.

In this article, you’ll learn how to tell normal uncertainty from actual warning signs, what a milestone or demo should show you to prove the project is on track, and what “done” actually means for a system that continues to evolve after launch.

Why AI Projects Feel Different

Traditional software is built to a spec. You define what a button should do, an engineer builds it, and it either does its job or it doesn’t. An AI system is built around an acceptable error rate instead, and that changes what progress looks like from the outside.

We saw this on a project where a client needed event listings, dates, and prices pulled from hundreds of different partner websites. Every site was structured differently, so there was no single rule that could reliably find “the ticket price” across all of them. The first attempt got the information wrong roughly a third to nearly half the time. That wasn’t a bug we could fix once and move on from. It was the expected starting point for a system that improves through iteration: try, evaluate, correct, and try again. We wrote up the engineering side of this project in Behind an AI-Driven Web Scraping System opens a new window .)

If you’re used to a team telling you a feature is built and working, hearing that a system is “wrong on the first try about a third of the time, and that’s fine” can sound like an excuse. It isn’t. It’s a different kind of project, with a different definition of progress.

Normal Uncertainty vs a Warning Sign

Once you accept that an AI system won’t be perfect from the start, the real question becomes how to tell a project that’s on track from one that’s at risk. The presence of mistakes alone doesn’t tell you much. What matters is whether someone is watching for them, measuring them, and can explain what happens next.

For the scraping project, incorrect results were expected, but were never invisible. A person reviewed what the system pulled, flagged what was wrong, and the system tried again with that feedback, for a few rounds, before anything was treated as finished. That happened for a few rounds before anything was treated as finished. The review loop is what made those early errors manageable rather than alarming.

Normal uncertainty sounds like this: “The system got it wrong in these specific cases. Here’s what we changed, and here’s the error rate now.” You get a number, an explanation, and a next step.

A warning sign sounds different. It’s vague reassurance without a number: “It’s working pretty well.” It’s an error rate that stays outside the agreed target without a clear explanation or plan. Or it’s a team that can’t show you what “wrong” looked like the last time it happened, which raises questions about how closely the results are being reviewed.

If you can’t get a straight answer to “What’s the error rate right now, and how has it changed since the last check-in?” that’s worth paying attention to, regardless of how confident the team sounds.

What a Milestone or Demo Should Actually Show You

A working demo is not the same as evidence that a project is on track. A single successful run may just be an easy case. What tells you more is a demo that shows how the system handles its mistakes. What it got wrong, what happened next, and how its performance compares with the last check-in.

Teams building AI systems responsibly track this directly. On the evaluation side of our own work, that means building a labeled dataset of real examples, scoring outputs against it, and using those scores to find where a system is still weak before it ships, the same feedback loop we walked through in Traces to Insights: Evaluating LLM Apps opens a new window . A milestone review is your chance to ask for a version of that same evidence, in plain terms.

At a checkpoint, it’s fair to ask for three things: the current error rate on a representative sample, one or two specific examples of what the system got wrong recently, and what changed as a result. If a team can walk you through those three things without hedging, the project is likely in good shape. If a demo only shows you the cases where the system worked, ask to see one where it didn’t, and how that was handled. That tells you much more about whether the team understands the system’s limitations and has a process for improving them.

What “Done” Means for an AI System

Traditional software gets built against a spec it can pass: a feature ships, it passes QA, and it generally behaves the same way until someone deliberately changes it. It still needs maintenance over time, but its behavior is relatively predictable. An AI system is different. Its performance can shift as the inputs, users, data, or models around it change. Expecting it to hold still the way a traditional feature does is where a lot of buyer frustration starts.

We laid out the underlying test for when an AI system is worth building at all in When Does Your Business Need an AI Agent? opens a new window : the task has to be too varied to hand-code, common enough that manual work doesn’t scale, and forgiving enough that mistakes are caught before they cause unacceptable consequences. That last condition is also what “done” rests on here. Not that mistakes have stopped, but that you know how often they happen, can catch the ones that matter, and know what happens next.

In practice, done means the team has agreed with you on an acceptable level of performance, has a way to measure it on an ongoing basis, and has a plan for what happens when performance drifts. This includes not just at launch, but the following month and the month after that. A project that meets its agreed target and has monitoring in place to detect when it falls outside that target is done in the way that matters.

A project that never had an agreed target to hit was never going to feel finished, no matter how good the demo looks.

Conclusion

Buying an AI project means asking different questions than buying traditional software. Instead of “is it working,” ask “what’s the error rate, and is it improving?” Instead of “is it done,” ask “is the error rate acceptable, and is someone watching it?” None of this means lowering your standards. It means measuring the right thing: not whether the system is ever wrong, but whether wrong gets caught, explained, and corrected.

Not sure what to ask for at your next AI project milestone? Talk to our team opens a new window . We build and monitor AI systems in production, and we can help you set clear performance targets, review the right evidence at each milestone, and plan for what happens after launch.