We build these systems for a living, and even we don’t reach for an agent by default. In this article, you’ll learn what an AI agent actually does differently from a chatbot, what that looks like on a real project, and how to tell if your business has a task that’s actually a good fit for one.

What an AI Agent Actually Does

Most of what gets called “AI” in a business tool today is a single request and a single response: you ask a question, a model answers it, done. A chatbot on a website works this way. So does a tool that summarizes a document or drafts an email for a person to review. One step, and a person decides what happens next.

An agent is built around a loop instead of a single step. It takes on a task, works out what it needs to do first, does it, looks at the result, and decides what to do next, on its own, for as many steps as the task takes. If a step doesn’t work, it doesn’t just fail, it can recognize that and try something different. The “agent” part isn’t the model getting smarter, it’s the system giving that model a task, tools to act with, and room to keep going without someone approving every single move.

That last part, not needing a person at every step, is also exactly why an agent takes more work to get right than a single request does. Every decision it makes on its own is a decision that can go wrong on its own too.

What This Looks Like in Practice

Here’s what this looked like when we built one for real. A client needed event listings, dates, and prices pulled from hundreds of different partner websites into one searchable list. Every site is built differently, so there’s no single rule that finds “the event title” or “the ticket price” across all of them, and writing custom logic for each site by hand wasn’t realistic at that scale.

So we built a system where the AI works out where that information lives on each site and tries it. Its first guess is wrong roughly a third to nearly half the time, sites are messier than they look. What makes this workable is a loop, not a smarter guess: a person checks what the AI pulled, flags what’s wrong, and the AI tries again with that feedback, for a few rounds, before anything is treated as finished.

That’s the shape of an agent doing real work: not one clever guess, but a loop that catches and fixes its own mistakes, with a person checking in along the way instead of approving every single step. (If you want the engineering details behind this one, we wrote those up in our Behind an AI-Driven Web Scraping System post.)

When an Agent Is the Right Call

An agent earns its complexity when a task is too varied to write one rule for, happens often enough that a person doing it by hand doesn’t scale, and getting it wrong occasionally is something you can catch and fix rather than something that quietly costs you money or a customer. The project above fits all three: hundreds of different sites, too many to hand-code one at a time, and a wrong guess just gets flagged and retried instead of going out the door.

That’s a real bar to clear. Not every business task meets it, and we’ve written elsewhere about how to tell whether AI is worth pointing at a task at all before you get as far as choosing a specific tool like an agent.

The honest answer to “should we use an agent for this” is the same as it is for most tools: it depends on the task, not on how often the word shows up in a pitch deck this year.

Conclusion

An agent isn’t a smarter chatbot, it’s a system that takes a task, works through it in steps, and catches its own mistakes along the way, with a person checking in rather than approving every move. On a real project, that looked like hundreds of different websites and a loop that got things wrong on the first try roughly a third to nearly half the time, then corrected itself before anything shipped. Whether it’s worth building one for your business comes down to the task in front of you, not the word itself.

None of that makes an agent free to run. It still needs someone watching how it performs, and someone willing to say a simpler tool would do the job better, when that’s true.

Not sure if your business has a task that’s actually a fit for an agent? Talk to our AI Agents team and we’ll help you figure out if it’s worth building one.

On August 29, 2026, OpenAI told Cursor, one of the most widely used AI coding tools, that Cursor’s access to OpenAI models ends on November 12. There was no outage, no price change, and no wrongdoing on Cursor’s side. Cursor had just been bought by SpaceX, and OpenAI decided it could not trust the new owner to honor its terms. Cursor’s CEO wrote that they had “trusted their platform to be neutral infrastructure for our business.”

That sentence is why this post exists. Cursor is a good example, and this post is not really about Cursor. It is about what it means to run a business on tools whose provider can decide, for reasons that have nothing to do with you, that you can’t use them anymore. We ran the exercise on ourselves at OmbuLabs: where AI actually sits in the business, which company is underneath each tool, what stops if that company says no, and which of those gaps are worth paying to close. What follows is that exercise.

What happened to Cursor, and what didn’t

The timeline is short. SpaceX agreed to buy Cursor in June for about $60 billion in stock, and the deal closed on August 14. Fifteen days later, OpenAI announced that it would stop serving its models to Cursor on November 12, giving Cursor roughly ten weeks and the option to pick an earlier date. OpenAI’s stated reason was earlier disputes with Elon Musk’s companies, including the admission that xAI had trained on OpenAI outputs.

The way Cursor used those models did not change. Its owner did, and OpenAI made a decision about that owner. OpenAI’s help center lists three ways to keep using its models in Cursor after that, your own API key, OpenAI’s Codex extension, or a gateway such as Amazon Bedrock or Azure, and all three mean the thing you were paying Cursor for is now something you set up and pay for yourself.

Cursor’s CEO, Michael Truell, said OpenAI models accounted for about 5% of Cursor’s traffic. Most of its users were already on Anthropic’s Claude or Google’s models, both of which keep working, and Anthropic said publicly it would add capacity for Claude in Cursor. Cursor will be fine.

It is worth being honest about why. Cursor did not spread its usage across model providers as a continuity strategy. It offered several models because its users wanted the choice, and those users drifted toward Claude because they preferred it for coding. The resilience was real, but nobody planned it, and that is the part you can’t count on for your own business.

The other ways a provider says no

Cursor is not the first coding tool this has happened to, and OpenAI is not the only provider that does it. In June 2025, while OpenAI was reportedly buying Windsurf, Anthropic cut off nearly all of Windsurf’s direct access to Claude models with, in Windsurf’s words, less than five days of notice. Windsurf said it had wanted to pay for the full capacity. It is the same shape as Cursor, with a different provider doing the cutting and about a tenth of the notice.

Acquisitions are one trigger, and geography is another. In June 2024, OpenAI emailed developers that it would block API traffic from countries it doesn’t support starting July 9, about two weeks out. In September 2025, Anthropic updated its terms to refuse service to any company more than 50% owned by entities headquartered in places like China, wherever that company actually operates. A business in Singapore or Germany with the wrong parent company lost access without moving anywhere.

Retirements are the quiet version. On January 29, 2026, OpenAI announced it would remove GPT-4o and three other models from ChatGPT on February 13, noting that only 0.1% of users still picked GPT-4o on a given day. Business and Enterprise plans kept it inside custom GPTs until April 3. If your team built a workflow around one model’s particular behavior, a retirement breaks it as thoroughly as a ban does, just more politely.

Across all of these, the notice ranged from ten weeks to five days, none of the affected customers had done anything wrong, and both of the providers most businesses use today, including the one we use most, have done it at least once.

The inventory

The exercise itself is not complicated, and a spreadsheet is enough. Four columns: the tool, the company whose model runs underneath it, who in the business uses it, and what stops if it goes away. Here is what a first pass might look like for a small accounting firm, made up for the example:

The first column is easy, and the third and fourth are mostly a matter of asking around. The second column is the one people skip, because for most tools the provider underneath is nowhere on the invoice. Each of those tools is somebody else’s product, calling a model provider you never chose, and it is not unusual for several of them to be calling the same one, which means a single “no” can take out three rows at once.

Finding out is usually possible. Most software vendors publish a list of subprocessors, the other companies they send your data to, because privacy regulations require it, and the model provider tends to appear there. Failing that, the vendor’s documentation or a direct question to your account manager will do. It is the same question we put on the list for safety reasons in The Guardrail Question to Ask Any AI Vendor, asked this time for continuity reasons.

The one answer to be wary of is “we use multiple providers.” It is a good sign, but ask which provider handles the feature you actually depend on. Cursor also used multiple providers, and its OpenAI users still lost their model.

What breaks, and what a fallback costs

With the fourth column filled in, most rows sort themselves into one of three piles. Some are an annoyance: people lose a shortcut and go back to doing the thing the way they did it eighteen months ago. Some slow a team down in a way that shows up in the numbers, a support queue that grows or a sales team that stops logging calls. A few stop something that makes money or keeps you compliant, like a chatbot that is the only after-hours channel or a document pipeline a client deadline depends on. Be strict about which pile each row belongs in, because the temptation is to put everything in the third one, and that is how you end up paying for fallbacks nobody will use.

For the first pile, do nothing. Diversifying has a cost, and for a tool people would grumble about losing and then get on with their day, the cost is not worth it. For the second pile, the cheapest insurance is usually not a second provider. It is making sure the things that live inside the tool also live somewhere else: the prompts and instructions people have refined over months, the kind of shared knowledge Fiona described in AI for Small Business Starts With People, the document templates, the custom assistants someone built, exported into a folder you control. A workaround like bringing your own API key gives you the model back, but whatever was built inside the tool around it, saved conversations, custom assistants, months of refined instructions, does not move with you.

For the third pile, a real second option is justified, and it is usually cheaper than it sounds: a second seat with another provider for the two or three people who depend on the tool most, tested once a quarter so someone knows it works, and a written version of the manual process, kept current. What is expensive is running two providers for everything at once, because the outputs differ and someone has to check both. We would not recommend that for a business without an engineering team, and even for us it only makes sense on a few rows.

Our own inventory

We are a software company, so our list looks different from the accounting firm’s, but the columns are the same.

The biggest row is Claude Code, from Anthropic. Most of our engineers use it daily, and our blog writing process, the one we described in AI Assistant for Our Blog Writing Process, is a set of Claude Code skills on top of a server we built. If Anthropic said no tomorrow, engineering would slow down noticeably for a few weeks and the blog workflow would have to be rebuilt for a different agent. That puts it in the second pile, and we have decided to accept the dependency. The parts we own, the server and the skill files, are plain code and plain text that other agents can read, so what we would actually lose is the agent itself and the habits built around it, and no second provider gives us those back. What we do instead is keep the skills and the server portable and avoid building on features only Claude Code has.

The same server uses Voyage AI for embeddings, the numbers that make the search over our posts work, as we covered in A Blog MCP Server with FastMCP and pgvector. Voyage was bought by MongoDB in February 2025, before we chose it, so that row already carries an ownership change. If it went away, re-embedding a few hundred posts with another provider is a script and an afternoon. That is the first pile.

The newsletter assistant we wrote about in 2025 ran on OpenAI models. It has since become a marketing assistant, folded into the same platform as the blog server, and it runs on Anthropic models now. The row moved from one provider to the other without anyone treating it as a provider decision, and our list is more concentrated on Anthropic than it was a year ago.

Two smaller rows point elsewhere. Our Rails Superhero Card Generator, the small app that turns a photo and a few skills into a superhero card, uses Google’s Gemini image models. A report generation tool we use now and then runs on OpenAI models. Both would be missed for about as long as it takes to notice, and neither gets a fallback.

The rows we expected to be the hardest to fill, the AI features inside the software we pay for, turned out to be mostly empty. GitHub, Slack, and Jira all ship AI features, and we barely use them. Our AI usage is our own tooling and Anthropic’s Claude products, which makes our inventory shorter than most and more concentrated than most. Client work stays off this list, for their sake rather than ours.

Conclusion

Cursor lost a provider it had trusted to be neutral, and it came out fine because its users had already spread themselves across three providers, for reasons that had nothing to do with resilience. The other cases we looked at came with notice periods from ten weeks to five days, both of the big providers were on the giving end at least once, and none of the customers involved had done anything to deserve it.

The exercise that follows from that is a four-column inventory, with the provider column being the one worth the effort, a sort into three piles, and a fallback only for the rows in the third pile. For our own list, the result was one accepted dependency, several rows that need nothing, and a concentration on a single provider that is higher than it was a year ago and that we are choosing to live with for now.

The caveat is that an inventory is a snapshot. Ours changed in under a year without anyone deciding to change it, as one tool became another and switched providers on the way. Providers change their terms faster than anyone updates a spreadsheet, so the useful habit is a short review whenever something new gets rolled out, or whenever a headline like Cursor’s shows up.

Not sure which provider sits under the tools your team uses every day, or which ones would actually hurt to lose? We can help you find out, so let’s talk.

In this post, we’ll walk through the core evaluation loop, how to build an evaluation dataset from production traces, and how to use evaluation signals to debug and improve your application.

Introduction

In the previous post, we explored the foundations of LLM observability and tracing. We discussed how traces give you visibility into your LLM application by capturing the full sequence of events across every request, from the initial prompt all the way through retrieval steps, tool calls, and model responses.

Evaluation is the process of systematically measuring how well your AI system performs across the full pipeline, from the intermediate steps like retrieval and tool use through to the final response. Rather than relying on intuition, anecdotal feedback, or reactive debugging, evaluation provides a structured way to determine whether your application is performing as intended. It transforms quality from a subjective judgment into something that can be measured, tracked, and improved over time.

Evaluating LLM applications comes down to a set of questions that traces help you answer: whether the model’s response was accurate, whether it actually solved the user’s problem, whether the retrieved context was relevant to the query, and whether the latest prompt change moved output quality in the right direction or quietly made things worse.

The cost of skipping evaluation

Before diving into how evaluation works, it’s worth being honest about what happens without it.

Without a systematic way to measure quality, improving an LLM application becomes largely guesswork. You fix one issue and introduce another. Your prompts grow longer and more complicated as you attempt to patch edge cases one by one. Performance tuning is speculative at best, there is no way to be sure if a change made things better or just different. And at scale, cost control becomes nearly impossible when you have no visibility into where your tokens are actually going.

A good evaluation system changes this. It makes debugging targeted, optimization data-driven, and improvement continuous.

What evaluation actually means for LLM apps

Unlike most software testing, where the expected output is deterministic, evaluating LLM applications means contending with open-ended, context-sensitive outputs that cannot be reduced to a simple pass/fail.

At a high level, LLM evaluation usually focuses on three dimensions: quality, behavior, and cost efficiency.

Quality

Quality measures whether the application completes the task successfully. Is the answer accurate? Is it grounded in the right context? Does it handle uncertainty appropriately? Does it fail gracefully when it cannot answer? Quality evaluation helps define what an acceptable output looks like and what kinds of errors matter most for your application.

Behavior

Behavior measures whether the system acts the way you intended across different scenarios. This includes consistency, instruction following, tone, refusal behavior, tool usage, and edge case handling. A model may produce a high-quality answer in one case but behave unpredictably when the input changes slightly. Behavior-focused evaluation helps identify those inconsistencies before they become production issues.

Cost efficiency

Cost efficiency measures how much work the system does to produce a useful result. In LLM applications, cost is not limited to the final model call. Tokens may be spent during retrieval, tool selection, planning, summarization, intermediate reasoning steps, and retries. When evaluation is connected to tracing, you can inspect token usage trace by trace and span by span, making it easier to see where cost is accumulating and whether that cost is justified by better outcomes.

The organizing philosophy that ties these together is sometimes referred to as Eval Driven Development (EDD). Before making changes to prompts, models, retrieval pipelines, or agent workflows, you define what success looks like and create datasets that represent the tasks your system needs to perform.

The most useful evaluation datasets are usually task-specific. Off-the-shelf benchmarks can be helpful for understanding general model capabilities, but they rarely capture the details of your application’s domain, users, constraints, or edge cases. A support chatbot, a code review assistant, and a document analysis workflow all need different evaluations because they fail in different ways.

Once those evaluations exist, they become a feedback loop for development. In this sense, evaluation is not a one-time quality gate. It is the foundation for continuously improving an LLM application. Tracing helps you see what happened inside the system. Evaluation helps you decide whether that behavior was correct, useful, and worth the cost.

The core evaluation loop: humans define, models scale

Before you can evaluate at scale, you need a foundation of human judgment. This is where traces become essential in providing domain experts the raw material to work with.

The process works in four stages:

Define. Domain experts manually review a diverse sample of traces and establish what “good” looks like. This means writing explicit success criteria and rubrics that express what the model should and shouldn’t do in specific situations.

Label. Those same experts evaluate a dataset of real traces, marking each one as Pass or Fail accompanied by a short critique explaining their reasoning. A binary system is intentionally simpler than a 1 to 5 scale. LLM judges diverge from human scores, and the disagreement typically worsens as the scoring scale grows more granular, which is why binary classification gives you a cleaner starting signal.

Scale. Once you have a labeled dataset with human critiques, you can use a powerful LLM as a judge to replicate that judgment at scale. The human critiques feed directly into the evaluator prompt as few-shot examples, grounding the model’s assessments in real human reasoning. Use the most capable model you can afford here because this task requires advanced reasoning ability. Before you trust the output, measure precision and recall between the model’s grades and the human labels, and iterate on the evaluator prompt until agreement is comfortably high (in our experience, often in the 80–90% range).

Calibrate. Human evaluation doesn’t stop once the model is running. Periodically, reviewers sample a portion of the model’s grades to check for drift. If the model-based evaluator starts diverging from human judgment, which happens for many reasons, you catch it early and retune.

This loop where humans define the criteria and models run the evals at scale is the engine that makes continuous improvement possible without drowning your team in manual review.

Building your evaluation dataset from traces

Your evaluation system is only as good as the data it runs on. This is one of the clearest arguments for investing in tracing early: production traces become the raw material for your evaluation dataset.

A common mistake when building evaluation systems is waiting until you have a perfect dataset before getting started. In practice, most teams begin with limited data and gradually improve their evaluations as their application matures.

Start synthetic if you have to. If you’re early in development and don’t yet have a body of production traces, bootstrapping with synthetic test cases or open-source data is a reasonable starting point. But treat this as temporary scaffolding, not a foundation.

Migrate to production data as soon as possible. Once your application is serving real users and collecting traces, those traces become one of the most valuable assets for improving quality. Every trace represents a real interaction, complete with user intent, contextual information, model decisions, tool calls, and outcomes. Rather than guessing how users might interact with your system, production traces show you exactly how they do.

Label strategically. Many teams immediately gravitate toward numerical scoring systems such as rating outputs on a scale from one to ten. While these scores appear more precise, they often introduce significant disagreement between reviewers. Even expert evaluators frequently differ by several points when assigning scores to the same response, making it difficult to establish reliable thresholds and compare results over time.

For this reason, it is often better to begin with binary classification tasks. Instead of asking whether a response deserves a score of seven or eight, ask simpler questions: was the response relevant to the user’s request? Did the answer remain consistent with the provided context? Was the retrieved information used correctly? Did the system follow the expected behavior?

Consistency and relevance are particularly effective starting points because they capture two of the most common failure modes in LLM applications while remaining straightforward for both humans and automated evaluators to assess.

When you’re starting out, review as much trace data as you can. Over time, as your dataset grows and your model stabilizes, you can sample more selectively. Three sampling strategies are worth knowing. Failure-driven sampling prioritizes traces the model flagged as low-confidence or that received poor scores, which is where your gaps are. Precision-focused sampling selects instances the model predicted as positive with high confidence, to catch false positives slipping through. Calibration sampling targets borderline cases where the model was uncertain, since those examples sharpen the evaluator’s judgment on the edges.

Evaluation signals: from automated checks to human review

Not all evaluation signals are equal, and not all of them require the same level of effort. A practical evaluation system layers them.

Automated and heuristic checks

Automated and heuristic checks are your first line of defense. These evaluations are inexpensive, highly scalable, and deterministic. They work particularly well when the success criteria can be expressed as objective rules.

Examples include:

• Output format validation
• Structured JSON schema checks
• Length requirements
• Toxicity detection
• PII detection
• Keyword matching
• Regular expression validation
• Safety and compliance rules

For example, if an application is expected to return valid JSON, a simple parser can immediately determine whether the output passes or fails. If a support chatbot must include a ticket reference number, a regular expression can verify its presence.

These checks are often insufficient for measuring overall response quality, but they are valuable guardrails that catch obvious failures before more sophisticated evaluation methods are needed.

LLM-as-judge

As applications become more complex, many evaluation questions require subjective judgment: is the answer relevant? Did the model correctly use the retrieved context? Was the explanation helpful? These questions are difficult to encode as deterministic rules, which is why LLM-as-judge systems have become increasingly popular.

In this approach, another language model evaluates the output and determines whether it meets specific criteria. Because LLM judges can reason about language and context, they can often approximate the decisions a human reviewer would make while operating at a much larger scale.

This makes them particularly useful for relevance evaluation, faithfulness and grounding checks, instruction-following verification, consistency evaluation, and comparative testing between model versions.

However, LLM judges are not perfect.

They can exhibit biases, overfit to particular response styles, or disagree with human reviewers. Different judge models may also produce different outcomes for the same example.

For this reason, LLM judges should be calibrated against human-labeled datasets whenever possible. Human reviewers establish the ground truth, and the judge is optimized to approximate those decisions at scale. As more human annotations become available, the judge can be continuously validated and improved.

The goal is not to replace humans entirely, but to extend human judgment to a volume that would otherwise be impossible to review manually.

Human review

Human review remains irreplaceable for certain tasks. Experienced domain experts can often identify subtle issues that automated systems and LLM judges miss entirely. They understand business context, user expectations, regulatory requirements, and nuanced quality concerns that are difficult to formalize.

Human review is particularly valuable when:

• Defining evaluation criteria for a new application
• Investigating novel failure modes
• Reviewing high-impact decisions
• Evaluating subjective qualities such as usefulness or clarity
• Measuring alignment with business requirements
• Creating high-quality training and evaluation datasets

In many organizations, subject matter experts become a critical part of the evaluation workflow. Their annotations help establish ground truth, identify emerging problems, and provide the feedback needed to continuously improve prompts, retrieval systems, and agent behaviors.

This is often where traces become especially valuable. Rather than reviewing isolated outputs, experts can inspect the entire execution path, including retrieved documents, tool invocations, intermediate steps, and model decisions.

Sometimes the most valuable evaluation question is simply: “Would I be comfortable shipping this response to a customer?”

No automated metric can fully replace that judgment.

Beyond quality scores, your traces expose a rich set of signals worth monitoring. A well-instrumented trace surfaces application latency at the span level, so slow LLM invocations are identified precisely rather than buried in an end-to-end average. Token consumption is tracked per call, making it straightforward to spot inefficient prompts before they become a cost problem. In RAG pipelines, retrieved documents are recorded along with their relevance scores and rankings, giving you visibility into retrieval quality independently from generation quality. The prompt templates used at each step are captured for prompt-level debugging, alongside the LLM parameters (temperature, top-p, and other settings). Tool calls are recorded with their full function signatures, input parameters, and outcomes, so agentic behavior is inspectable rather than opaque. Finally, runtime exceptions such as rate limit errors, timeouts, and other failures are captured at the span level.

Using traces to evaluate, debug, and improve

This is where the value of tracing becomes most tangible. Once your evaluation system is running and scores are flowing back into your trace data, you can use that combined view to systematically improve your application across three dimensions.

Offline evaluation is your safety net before deployment. Before shipping a new prompt, a model change, or a new retrieval strategy, you run your evaluation dataset against the new version and compare scores to the baseline. Only ship if things improved or at least didn’t regress.

Online evaluation runs continuously against live production traces. Automated evaluators score incoming requests in real time, giving you a live quality signal that catches degradation as it happens rather than after users have already noticed. When scores drop, you have the traces to investigate immediately.

When something goes wrong, traces give you a precise map of what happened. Instead of guessing which part of your pipeline failed, you can follow the span tree to the exact step where things broke down:

When a workflow produces bad output, the trace shows exactly where things went sideways. Prompt failures are visible in the raw model response. Tool errors are captured as span events. Retrieval issues show up in the exact documents retrieved and their relevance scores and orchestration bugs such as loops, unexpected branching, and bottlenecks in multi-step agents become visible as anomalous span patterns.

Root cause analysis that once required reproducing issues locally now starts with filtering traces by low evaluation scores and reading the span tree.

With evaluation scores linked to individual traces, optimization becomes targeted rather than speculative. Span-level data makes several improvement workflows more concrete and systematic. When traces are scored and a prompt is underperforming, you can filter down to the worst performers, read through the critiques, identify the recurring pattern, update the prompt, and re-run evals.

Latency tuning benefits from the same granularity. Rather than guessing from an end-to-end average, you can see exactly which component is slow and address it directly, whether the culprit turns out to be a retrieval step, a particular model, or an inefficient tool call. Token usage data works similarly for cost. Seeing consumption broken down at the span level reveals which parts of your application are more expensive. Linking that spend to specific workflows or user segments takes it a step further, replacing a high-level aggregate bill with a per-feature breakdown that engineering and product teams can actually reason about together.

Conclusion

Tracing gives you visibility into what happened while evaluation gives you the understanding to act on it. Together, they form the feedback loop that turns an LLM application from something you ship and hope for the best into something you can measure, debug, and systematically improve.

The path forward is straightforward in principle, even if the details take work: instrument your application, collect traces, build a labeled evaluation dataset from real production data, establish a human-in-the-loop grading process, scale it with model-based evaluation, and run it continuously. Each iteration makes your evals sharper, your prompts better, and your application more reliable.

In our experience, teams whose LLM applications hold up over time tend not to be distinguished by model quality. What sets them apart is having feedback loops that tell them when things are working and when they’re not.

Working on an LLM application and not sure where to start with evaluation? Let’s talk!

That post was about deciding what to build. This one is about building it. We’ll go through the ingestion pipeline that gets our posts into a database, the embeddings that make the corpus searchable by meaning, the MCP server that exposes all of it, and the plugin that packages the workflows on top.

The stack

The pipeline and the server are one Python application, deployed to a single Heroku dyno:

• FastAPI for the web application
• FastMCP for the MCP server
• SQLAlchemy and Postgres for storage
• pgvector for the embedding column and similarity queries
• Voyage for the embeddings themselves

Semantic search is a part of the work here, I won’t cover the details of embeddings, cosine distance, and why use cosine instead of inner product or Euclidean distance in this post. The Semantic Search with Sequel and pgvector covers all of that. The stack there is Ruby, but the concepts are the same.

Getting the data in

The blogs are the source of truth, not the database. Both of them are Jekyll sites, which means every post is a markdown file with YAML front matter sitting in a GitHub repository. The pipeline’s job is to mirror that into something queryable.

Posts and their content are separate tables. The content table is the one that carries the embedding:

class PostContent(Base):
    __tablename__ = "post_content"

    id: Mapped[UUID] = mapped_column(primary_key=True, default=uuid4)
    post_id: Mapped[UUID] = mapped_column(
        ForeignKey("posts.id", ondelete="CASCADE"),
        nullable=False,
        unique=True,
    )
    content: Mapped[str] = mapped_column(nullable=False)
    embedding: Mapped[Sequence[float]] = mapped_column(VECTOR(1024), nullable=False)

VECTOR(1024) comes from pgvector.sqlalchemy, and 1024 is the dimension of the Voyage model we use. Splitting content off from metadata keeps the posts table small enough to query comfortably, since most of the tools only need metadata.

The posts table itself is mostly ordinary columns, with two constraints worth pointing at:

class Post(Base):
    __tablename__ = "posts"

    id: Mapped[UUID] = mapped_column(primary_key=True, default=uuid4)
    source_id: Mapped[UUID] = mapped_column(ForeignKey("sources.id", ondelete="CASCADE"), nullable=False)
    file_path: Mapped[str] = mapped_column(String(255), nullable=False)
    title: Mapped[str] = mapped_column(String(100), nullable=False)
    published_at: Mapped[datetime] = mapped_column(DateTime, nullable=False, index=True)
    keywords: Mapped[list[str]] = mapped_column(MutableList.as_mutable(ARRAY(String)), nullable=False)
    author_id: Mapped[int | None] = mapped_column(ForeignKey("authors.id", ondelete="CASCADE"), nullable=True)
    author_ids: Mapped[list[int] | None] = mapped_column(MutableList.as_mutable(ARRAY(Integer)), nullable=True)

    __table_args__ = (
        UniqueConstraint("source_id", "file_path", name="uq_posts_source_file_path"),
        CheckConstraint("author_id IS NOT NULL OR author_ids IS NOT NULL", name="ck_author_id_or_author_ids"),
    )

That listing is trimmed, the real model has a few more columns. The unique constraint is on source plus file path rather than file path alone, because two blogs can quite happily have a post with the same filename. The check constraint encodes an editorial rule: a post has either one author or a list of them, never neither. title is String(100) for the same reason, because that fits the length limit our blogs enforce comfortably.

Front matter parsing uses the python-frontmatter library, and the pipeline normalizes the metadata schema where the two blogs disagree, so every consumer downstream sees one interface and doesn’t need to worry about which blog it’s looking at:

Re-reading four hundred posts on every sync would be wasteful, so the pipeline works from commit SHAs instead. It stores the SHA it last processed, asks GitHub what changed between then and now, and acts on the file statuses:

added = {file.filename for file in post_files if file.status == GitHubFileStatus.ADDED}
removed = {file.filename for file in post_files if file.status == GitHubFileStatus.REMOVED}
updated = {file.filename for file in post_files if file.status == GitHubFileStatus.MODIFIED}

if added:
    added_files = [
        self._github_service.get_contents(file_name=filename, ref=current_sha)
        for filename in added
    ]
    self._add_posts(added_files=added_files)

if removed:
    self._repos.post_repo.bulk_delete(removed)

if updated:
    updates = self._updated_posts(
        current_sha=current_sha,
        last_processed_sha=last_processed_sha,
        updated=updated,
    )
    self._repos.post_repo.bulk_update(updates)

When there is no last processed SHA, which is to say on the very first run, it walks the whole repository instead. Updates are handled by fetching both versions of the file, diffing the parsed front matter key by key, and only writing what actually changed:

changes = {
    key: value
    for key, value in current_post_metadata.items()
    if previous_content_metadata.get(key) != value
}
if changes:
    post_update.metadata = changes

if current_post.content != previous_content.content:
    post_update.content = current_post.content

That last check is what keeps the embedding bill down. If a post’s body is untouched, there is no reason to re-embed it, so a typo fix in the front matter costs nothing.

Voyage embeddings

Anthropic doesn’t offer its own embedding models, it recommends evaluating embedding vendors to decide on the best one for your use case. Voyage AI features prominently and its models are documented as working particularly well with Claude’s retrieval systems, so that is the provider we used.

Embedding a post is a single call, and the text we embed is the title glued to the body:

embeddings = self._voyage_service.embed_many(
    text=[
        f"{post['metadata']['title']} - {post['content']}"
        for post in new_posts
    ],
)

The title goes in because it is often the most concentrated description of what a post is about, and dropping it loses information that the body sometimes never states outright.

The service wrapping Voyage is small, and the important part to pay attention to here is the type on input_type:

EmbeddingInputType = Literal["document", "query"]

BATCH_SIZE: int = 50


class VoyageService:
    def embed_one(
        self,
        text: str,
        input_type: EmbeddingInputType = "document",
    ) -> Sequence[float]:
        embeddings = self.client.embed(
            texts=[text],
            model=settings.EMBEDDING_MODEL,
            input_type=input_type,
        )
        return embeddings.embeddings[0]

    def embed_many(
        self,
        text: Sequence[str],
        input_type: EmbeddingInputType = "document",
    ) -> Sequence[Sequence[float]]:
        embeddings = []
        for batch in batched(text, BATCH_SIZE):
            embeddings.extend(self._embed_batch(batch=batch, input_type=input_type))
        return embeddings

Voyage’s models take an input_type parameter, and passing the right one is not optional if you want good results. The model embeds a short question differently from the way it embeds a long document, because those two things are asymmetric in practice: somebody typing “zero-downtime Rails upgrades” is describing what they want to find, not providing a sample of it.

That asymmetry is why we ended up with two search tools rather than one:

def search_posts(
    query: str,
    source: BlogSources | None = None,
    limit: int = _DEFAULT_SEARCH_HITS,
) -> list[SearchHit]:
    """Semantic search over published blog posts.

    query should be a short natural-language phrase describing a topic (e.g. "zero-downtime Rails upgrades").
    Omit source to search both blogs. Results are ordered by similarity score (higher = closer) and include a
    snippet; use get_post with a hit's file_path for full content.
    """
    embedding = _voyage.embed_one(text=query, input_type="query")
    with session_scope() as db:
        return PostQueryRepo(db).similar_posts(
            embedding=embedding,
            source=source,
            limit=limit,
        )


def find_related_posts(
    text: str,
    source: BlogSources | None = None,
    exclude_file_path: str | None = None,
    limit: int = _DEFAULT_SEARCH_HITS,
) -> list[SearchHit]:
    """Find published posts most similar to a full draft or document.

    Pass the entire draft (or a section) as text, unlike search_posts, which expects a short query. Use this to find
    cross-reference and internal-link candidates for a post being written or reviewed. Set exclude_file_path when the
    draft is a revision of an existing post so it doesn't match itself.
    """
    embedding = _voyage.embed_one(
        text=text,
        input_type="document",
    )
    with session_scope() as db:
        return PostQueryRepo(db).similar_posts(
            embedding=embedding,
            source=source,
            exclude_file_path=exclude_file_path,
            limit=limit,
        )

The two functions do almost the same thing and differ in three places: what they call their first parameter, the input_type they pass, and whether they can exclude a file path. search_posts takes a phrase and embeds it as a query. find_related_posts takes an entire draft and embeds it as a document, which is what you want when the question is “what have we already published that this piece should link to”.

exclude_file_path earns its place the first time you point the tool at a revision of a post that is already published. Without it the top hit is the post itself, which is correct and completely useless.

Both paths land on the same query. pgvector gives us a cosine distance operator through SQLAlchemy, so the ranking is a single order_by:

def similar_posts(
    self,
    embedding: Sequence[float],
    source: BlogSources | None = None,
    exclude_file_path: str | None = None,
    limit: int = 10,
) -> list[SearchHit]:
    distance = PostContent.embedding.cosine_distance(embedding).label("distance")
    query = (
        self._db.query(Post, PostContent, distance)
        .join(PostContent, PostContent.post_id == Post.id)
        .join(Source, Post.source_id == Source.id)
    )
    if source:
        query = query.filter(Source.repository_name == source)
    if exclude_file_path:
        query = query.filter(Post.file_path != exclude_file_path)

    rows = query.order_by(distance).limit(limit).all()

Distance is what the database returns, and similarity is what a reader wants, so the hits get built with the subtraction done for them:

SearchHit(
    **self._to_summary(post=row.Post, names=names).model_dump(),
    snippet=row.Post.description or row.PostContent.content[:300],
    score=round(1 - row.distance, 4),
)

The description or content[:300] fallback exists because our older posts predate us treating the description field as required, so a good few of them have nothing in it.

There is a cheap test for whether any of this is wired up correctly, and it is worth running before trusting the tool: paste a paragraph from a post you have already published and search for it. The post it came from should come back first. If it doesn’t, something in the chain is wrong, and it is much easier to find out that way than by squinting at similarity scores and wondering whether 0.62 is a good number.

Building the MCP server

With the data in place, the server is almost boring. FastMCP takes a name, a set of instructions, and an auth provider:

mcp = FastMCP(
    name=settings.PROJECT_NAME,
    instructions=_INSTRUCTIONS,
    auth=_build_auth(),
)

Those instructions are sent to the client, so they are the first thing a model learns about the server. They are worth writing carefully. Ours says what the two blogs are and what each one covers, because a model that knows FastRuby.io is the Rails blog and OmbuLabs.ai is the AI blog will pick the right source filter without being told every time.

Access is Google OAuth with the consent screen set to internal, which means our Workspace organization is the access list and there is no permission code of our own to maintain.

Tools are plain functions, registered one call at a time:

# Posts tools
mcp.tool(posts.list_posts)
mcp.tool(posts.get_post)

# Search tools
mcp.tool(search.search_posts)
mcp.tool(search.find_related_posts)

# Stats tools
mcp.tool(stats.get_content_stats)

# Style tools
mcp.tool(style.check_style)

# Validation tools
mcp.tool(validation.validate_post_metadata)

Resources are registered the same way, with a URI and a MIME type, and the ones with {source} in them are templates that take the blog as a parameter:

mcp.resource("blog://sources", mime_type="application/json")(resources.sources)
mcp.resource("blog://{source}/style-guide", mime_type="text/markdown")(resources.style_guide)
mcp.resource("blog://{source}/brand", mime_type="text/markdown")(resources.brand)
mcp.resource("blog://{source}/frontmatter-template", mime_type="text/markdown")(resources.frontmatter_template)
mcp.resource("blog://{source}/taxonomy", mime_type="application/json")(resources.taxonomy)
mcp.resource("blog://{source}/authors", mime_type="application/json")(resources.authors)
mcp.resource("blog://writing-principles", mime_type="text/markdown")(resources.writing_principles)

Both listings are trimmed to the blog-post surface.

The division between the two is the one design decision in here that actually mattered. Tools are things a skill can do, and resources are documents a skill can read. A skill asking “what is our house style” should get a document, not a function call that returns a paragraph at a time.

Serving it is three statements:

mcp_app = mcp.http_app(path="/mcp", stateless_http=True)

app = FastAPI(
    title=settings.PROJECT_NAME,
    lifespan=mcp_app.lifespan,
)

app.mount("", app=mcp_app)

Passing mcp_app.lifespan to FastAPI is easy to miss and the server will not work without it, because that is what starts and stops the MCP session manager alongside the app.

stateless_http=True is there because of a problem we ran into rather than a preference. Streamable HTTP used to keep session state on the server and hand the client a session ID. Our dynos restart daily, and every deploy restarts them too. After each restart, every connected client was holding a session ID the server had never heard of, and got a perfectly spec-compliant 404 for its trouble. The only fix on the client side is to reconnect.

The protocol has since moved the same way. The 2026-07-28 revision, published two days before this post, retires sessions and the Mcp-Session-Id header entirely, along with the initialize handshake. What was a workaround for us is now simply how the transport works, so if you are reading this some time later, treat stateless as the only path rather than as a choice.

Docstrings and type hints

FastMCP builds each tool’s advertised description from the function’s docstring, and its input schema from the type hints. That has a consequence worth sitting with: the docstring is not developer documentation. It is the interface, and its reader is a model deciding which tool to call.

Here is get_post, which is about as small as a tool gets:

def get_post(source: BlogSources, file_path: str) -> PostDetail:
    """Fetch a single blog post with its full markdown content.

    file_path is the identifier returned by list_posts. Errors if no post matches in the given source.
    """

One sentence saying what it returns, one saying where the identifier comes from, one saying what happens when it fails. That middle sentence is doing the real work. file_path is a meaningless string on its own, and without being told, a model will try to guess one. Naming the tool that produces it turns two tools into a sequence.

The same trick runs the other way in list_posts, whose docstring ends by pointing at get_post:

def list_posts(
    source: BlogSources | None = None,
    category: str | None = None,
    content_type: str | None = None,
    author: str | None = None,
    keyword: str | None = None,
    published_after: datetime | None = None,
    published_before: datetime | None = None,
    limit: int = _DEFAULT_LIMIT,
    offset: int = 0,
) -> list[PostSummary]:
    """List blog post summaries, newest first.

    Returns metadata only, no post content or body. Includes the post's url (link) that lands directly in the post's
    page in the blog.

    Filter by source (omit to search both blogs), category or content_type (exact names), author (username), keyword
    (exact match), and/or a published_after/published_before date range. Page with limit/offset. Use get_post with a
    returned file_path to fetch full post content.
    """

“Returns metadata only, no post content or body” is there to stop a model calling list_posts and then complaining that it cannot see the text. “Exact names” is there because category and content type are not free text, and a model that guesses AI instead of artificial-intelligence gets nothing back.

The type hints do the rest. That signature is what the client actually receives:

{
  "properties": {
    "source": {
      "anyOf": [{"enum": ["ombulabs.com", "fastruby.io"], "type": "string"}, {"type": "null"}],
      "default": null
    },
    "category": {"anyOf": [{"type": "string"}, {"type": "null"}], "default": null},
    "limit": {"default": 20, "type": "integer"},
    "offset": {"default": 0, "type": "integer"}
  },
  "type": "object"
}

The BlogSources enum becomes a JSON schema enum, which means the two valid blog identifiers are in front of the model rather than in a document it has to remember to read. That schema is trimmed, the real one has every filter in it.

Going back to the two search tools from earlier, this is why their docstrings look the way they do. search_posts says “query should be a short natural-language phrase describing a topic” and gives an example. find_related_posts says “pass the entire draft (or a section) as text, unlike search_posts, which expects a short query”. Each one names the other, because the failure mode is not a model that cannot use either tool, it is a model that reaches for the wrong one and gets mediocre results without ever knowing why.

When tools got picked incorrectly during testing, the fix was almost always in the description. Writing text whose audience is a model takes some getting used to, but the description is usually the cheapest thing to change.

Not every tool needs to be interesting, either. Two of ours contain no model, no embeddings, and no AI buzzwords of any kind:

mcp.tool(style.check_style)
mcp.tool(validation.validate_post_metadata)

check_style is a handful of compiled regular expressions run over a draft. validate_post_metadata is a series of if statements and a few database lookups. Neither would raise an eyebrow as a standalone script, and that is rather the point: an MCP tool does not have to be anything more than a function you already had. What the server adds is a runtime, and that turned out to matter more than expected. A script has to run somewhere, which means everyone using it needs the right language installed and the right version of the file. A tool on the server runs identically for everyone, the rules ship in the same deploy as the code that enforces them, and nobody is ever checking a draft against last month’s version.

The other thing it adds is a result rather than an opinion. Ask a model whether a draft uses banned constructions and you get a judgment, which is fine when judgment is what you need and unhelpful when it isn’t. A regex either matches or it doesn’t. validate_post_metadata goes further and checks categories, content types and author usernames against what is in the database right now, so a draft claiming a category we retired last year fails, and fails with the reason. No prompt can do that, however well written.

Testing with the Inspector

The MCP Inspector makes a server easy to test. It connects the way a client would, lists the tools and resources you have registered, and lets you call them by hand and read exactly what comes back. FastMCP ships a wrapper around it, so there is nothing to wire up separately. It runs the Inspector through npx, so you need Node available:

fastmcp dev inspector app/mcp/server.py:mcp

This gives you a client attached to your server, and three things are worth checking every time.

The first is whether the tools appear with the descriptions you meant. This is the only place you see your docstrings the way a model sees them, and reading them there rather than in the source is a different experience. Ours got shorter and more specific after the first look.

The second is the input schema. Enums should be enums, defaults should be the defaults you intended, and optional parameters should be genuinely optional. A BlogSources | None = None hint that quietly became a required string is not something you notice from the Python side.

The third is the output. Make sure the tool returns what you expect it to, and in the right shape. This is what a tool actually returns:

class PostSummary(BaseModel):
    source: BlogSources
    title: str
    description: str | None = None
    file_path: str
    authors: list[str]
    category: str
    keywords: list[str]
    published_at: UTCDatetime
    content_type: str
    link: str


class SearchHit(PostSummary):
    snippet: str
    score: float
    """Cosine similarity; higher = more similar."""

There are no foreign keys in there. category is a name, not a category_id. authors is a list of display names, not a list of integers pointing at another table. link is a URL the model can hand straight to the reader, assembled on the way out. All of that costs a join and saves the model from either asking a second question or, worse, guessing.

The Inspector is a mirror though, not a gate. It will happily accept some things a real client will not, so the last step is always to install the thing where it is actually going to be used and try a real task. Ours was straightforward enough: point it at a published post, ask for related articles, and see whether the answers are ones a person would have given.

The plugin

The server holds what is true about the blogs. The plugin holds how we work, and it contains no code at all. A skill is a markdown file with front matter:

---
name: blog-reviewer
description: Review a blog post draft for FastRuby.io or OmbuLabs.ai before the author opens a PR. Verifies claims (factual accuracy, overstatement, stale time-bound statements), then style and brand compliance including AI-drift patterns, then front matter validity and cross-references. Use when the user asks to review a blog post, check a draft, fact-check a post, or invokes /blog-reviewer, optionally with a file path.
---

That description is long deliberately. It is what Claude Code matches against when deciding whether the skill is relevant, so it names the artifact, both blogs, what the skill actually does and the phrasings someone might reach for. Check out our how to write a Claude Code skill post for more detail.

The body of the file is where the interesting constraint lives. Everything in a SKILL.md is loaded the moment the skill triggers, and it stays loaded. Context is finite and it is shared with the thing the author is actually working on. So SKILL.md does as little as possible. It says what the phases are, in what order, and where the detail for each one lives, for example:

Each reference file gets read when its pass starts and not before. The protocol for checking claims is detailed, and none of that detail is useful while the style pass is running, so it is not in the room. The router is enough to know what to do next and where to look it up. That split pays off twice. Context stays available for the work rather than the instructions, and any phase can be rewritten by editing one file, without touching the skill that calls it.

One rule in every skill matters more than the rest, and it is the seam between the two halves of this post:

Rules are fetched, not remembered. Read blog://{source}/style-guide, blog://{source}/brand, and blog://{source}/frontmatter-template before the style and mechanics passes.

No skill contains a style rule. They fetch the resource. Change the style guide on the server and every skill picks it up on its next run, with no plugin release and nothing for anyone to install.

The plugin also ships its own .mcp.json pointing at the server, which is what turns installation into one step instead of a setup guide.

{
  "mcpServers": {
    "blog": {
      "type": "http",
      "url": "https://mcp-server.com/mcp"
    }
  }
}

Distribution is a private GitHub repository acting as its own marketplace, so access to the repo is access to the plugin and updates ride on git push.

Conclusion

Building an AI-powered solution to solve an operational workflow problem can be quite easy. None of this is fancy or unusual, the stack is as common and lean as it gets, and the plugin is just text files. Only two decisions really shaped it: what belongs on the server versus in the skills, and what gets checked by code versus by judgment. Both of those come down to knowing which parts of your own process are stable enough to encode.

If you build something like this, the part that will need looking after is not the part you would expect. The skills are easy to change and the server is small. The pipeline is what quietly decides whether any of it is useful, because a corpus that stops being current takes the tools down with it without ever throwing an error. Everything downstream trusts that data, so the ingestion side is where the attention belongs once the interesting work is done.

Need help deciding the best way to leverage your data for productivity gains? Let’s talk!

Most of that is wrong, and the loudest claim, that every Claude output is already watermarked, happens to be the easiest one to check. So we checked. On August 13, we generated images with Claude on two different surfaces, the desktop app and the API, and inspected every file for the provenance metadata Anthropic described. None of it is there yet.

So let’s take a look at what Anthropic actually committed to, what our own testing found, and why the four scariest claims circulating about the text watermark don’t hold up.

What we could actually verify

The first half of Anthropic’s announcement is something anyone can verify. Provenance metadata based on the C2PA standard (the Coalition for Content Provenance and Authenticity) is designed to be read: there’s a free command-line tool, c2patool, and a web-based verifier, and a marked file will show you its full manifest, digitally signed, including which software produced it. We first ran the tool against a known-marked sample image to confirm it reads manifests correctly, and it does.

Then we generated images with Claude and checked those. We used both of Anthropic’s current top models, Claude Opus 5 and Claude Fable 5, on two different surfaces: the desktop app’s image feature, downloading the files directly (screenshots strip metadata, so we avoided them), and the API, where Claude creates image files through its code execution tool. Every check came back the same way:

$ c2patool claude-image.png
Error: No claim found

No manifest, no signature, nothing to read. As of August 13, 2026, none of the Claude-generated files we produced carried the provenance metadata the announcement describes.

The fine print explains why. Anthropic’s own article, How Claude marks AI-generated content, says that “Claude models launched on or after August 2, 2026 will support machine-readable marking at launch.” There haven’t been new models launched since that date, so that category is, for the moment, empty. For everything already released, the same page says the law includes a transition period and that Anthropic is “working to add marking support for those models as well,” with no timeline attached. That can change any week, and we’d expect it to. But it means the announcement is a commitment plus work in progress, not a switch that has been flipped, and any advice that starts from “your Claude output is already watermarked” is starting from a claim that isn’t true at this moment in time.

What Anthropic actually committed to

The commitment covers two mechanisms that behave nothing alike, and most of the confused commentary comes from blending them together.

For text, Anthropic says it will weave an imperceptible watermark directly into what Claude writes. In their words: “You won’t see it, and it doesn’t change the meaning, quality, or readability.” There are no marks to spot and nothing added to the text, the signal lives in the writing itself, in a way we’ll unpack in the first myth below. Because it’s part of the words, it travels wherever the words are copied.

For files, the mechanism is entirely different: digitally signed metadata following the C2PA standard, attached to generated files in formats like .png, .jpg, and .svg. One clarification worth making, because a lot of coverage gets it wrong: Anthropic has no image-generation model. Claude doesn’t produce images the way DALL-E or Midjourney do. The images it makes are files it creates, drawings it renders in the app, or charts produced by code it writes. So the C2PA half is file provenance, a signed record that says “Claude produced this file, here’s whether it was altered since,” not a watermark baked into the pixels of an AI-generated picture. That distinction matters later, when we get to what survives editing.

The scope is genuinely broad. Anthropic says marking will apply to output from supported models “wherever Claude is offered, worldwide,” across its own products and through cloud partners, not just for EU users. The reason is the one we covered in our Article 50 checklist: the EU AI Act requires providers to mark generative output in a machine-readable format, and Anthropic signed the Article 50(2) Code of Practice on Transparency of AI-Generated Content as both a model provider and a system provider. Shipping one global mechanism is simpler than maintaining an EU-only one, so worldwide output gets the EU treatment.

Myth 1: it’s hidden characters you can strip out

Advice circulating on this one ranges from “paste the text through a plain-text editor” to scripts that promise to delete the watermark by removing invisible Unicode characters. All of it rests on the same misunderstanding of what a text watermark is.

Some earlier tools did mark text with invisible characters: zero-width spaces, special joiners, lookalike letters from other alphabets. Those marks are trivially findable with a character inspector and trivially strippable, which is exactly why nobody serious builds watermarks that way anymore. Anthropic hasn’t published its exact scheme yet, but everything in its description, imperceptible, surviving copy and paste, detectable only in aggregate, matches how modern statistical text watermarks work, the approach documented publicly in Google DeepMind’s SynthID-Text paper.

The plain-language version: as a model writes, it constantly chooses between words that would all be fine, “choose” versus “pick” versus “select.” A statistical watermark nudges those choices in a pattern that looks completely natural in any single sentence but becomes measurable across enough text, to a verifier that knows what pattern to test for. The watermark isn’t hidden in the text as an extra ingredient. It is the text.

Two things follow. Character-stripping tools do nothing, because there is nothing to strip. Anything selling you watermark removal on that basis is selling you a placebo. And copying the text anywhere, email, Word, your CMS, carries the watermark with it, because the words themselves are the signal.

Myth 2: anyone can now detect Claude text

A watermark that lives in word-choice patterns can only be read by a verifier that knows which pattern to test for. That knowledge sits with whoever generated the watermark, in this case Anthropic. A statistical watermark is not a public label, it’s closer to a signature that only the signer can check for.

And right now, nobody can check for it, including Anthropic’s own customers. There is no detection tool. Anthropic says it is “working to enable users and other third parties to detect Claude’s embedded watermarks,” with details promised in forthcoming technical documentation, and as of this writing that documentation hasn’t shipped. So the fear of a client quietly scanning your deliverables for Claude’s watermark fails twice over: the marking isn’t being applied yet, and the instrument to read it doesn’t exist.

What about the AI detectors that already exist online? Those are a different thing entirely. Public detectors don’t read watermarks, they guess, from surface features of the writing, whether text looks machine-generated. Their track record is poor in both directions: they miss AI text that’s been lightly edited, and they flag human writing as AI, a problem documented most starkly for non-native English speakers. A detector verdict on your copy, or a client’s copy, tells you very little today, and it will still be guesswork after Anthropic’s watermark goes live, because reading the actual watermark will require Anthropic’s verification mechanism, not a third-party guessing tool.

Myth 3: it identifies you

The version of this myth we’ve seen most often goes: the watermark ties the content to your Anthropic account, so anyone who checks it learns who made it, or even what you asked for. Neither half matches what these mechanisms are for.

Start with what the law actually demands, because that’s what shaped the design. Article 50 requires generative output to be marked as artificially generated. The obligation is about what kind of thing this is, AI-generated content, not whose it is. Nothing in Anthropic’s announcement or documentation describes encoding user identity, and the text watermark as described is a model-level pattern: the same kind of signal in everyone’s Claude output, saying “a model produced this,” not a per-customer serial number. Anthropic hasn’t published the technical specification yet, so we’d stop short of absolute guarantees, but “it tells clients it was you” is not a feature in any published description.

The C2PA side is easier to check, because manifests are designed to be readable. A manifest records which software produced the file, a digital signature, a timestamp, and what operations were performed. An author name appears only if the creator’s tool explicitly wrote one in. The sample manifest we inspected named its creator because its test author chose to embed that. There’s no covert channel to worry about here: whatever a Claude-generated file ends up disclosing, you’ll be able to read it yourself with the same free tool we used, and so will your client.

It’s also worth remembering where documents actually leak identity today: Word’s author metadata, tracked changes, and file properties reveal far more about who touched a document than anything in this announcement.

Myth 4: editing removes it, or nothing does

This one circulates in both directions: “any edit kills the watermark” and “it survives everything, don’t bother.” The honest answer is in between, and it’s different for text than for files.

For text, the only robustness information that exists is Anthropic’s own, so we’ll label it as such: their documentation says watermarks “may persist through some editing” but can be lost when text is “heavily edited, paraphrased, translated, or mixed” with other content. That is consistent with how a statistical watermark has to behave. The signal accumulates across many small word choices, so a light copyedit leaves most of it intact, while a heavy rewrite replaces the very choices that carried it. Where the threshold sits, nobody outside Anthropic can currently measure, because there’s no verifier to measure with. When the detection tooling ships, we intend to test it and publish what we find.

For files, the picture is close to the opposite, and it follows from the mechanism. C2PA metadata is attached to the file, not woven through the content. Send the file as an attachment or post it for download, and the manifest travels along, signed and intact. But take a screenshot of the image, re-export it through a tool that doesn’t speak C2PA, or run it through a platform that strips metadata on upload, and the manifest is simply gone. The standard is built to be tamper-evident, not removal-proof: editing a file with C2PA-aware tools leaves a recorded trail, while a missing manifest proves nothing at all about where a file came from.

So the text watermark lives in the content and dies by rewriting. The file record lives on the container and dies by re-rendering. Anyone selling you a workflow that promises either “watermark-free output” or “tamper-proof provenance” today is ahead of what the mechanisms support.

What this means for your business

The practical reading of all this is calmer than the headlines, but it does ask something of you.

Nothing about your existing Claude-assisted content changed this month. It isn’t watermarked, nothing can scan it, and no client is about to trace it. What did change is the direction of travel: marking is coming, worldwide, at the vendor level, and the window before it arrives is the right time to decide your disclosure posture rather than have it decided for you. If AI-assisted drafting is part of how you produce client work, the durable move is to say so on your own terms, in your engagement letters or your content process, before a marked file or a headline prompts the question. A client who hears your AI policy from you is a client; a client who thinks they caught you is a problem, and after marking goes live, “we never mentioned it” stops being a neutral choice.

Two narrower points. First, vendor marking doesn’t discharge your own obligations. Article 50 puts duties on deployers, visible labeling of deepfake-style media, disclosure for AI-generated text on matters of public interest, that exist independently of whatever Anthropic embeds machine-readably in the output. The checklist half of this is in our Article 50 post. Second, what your AI vendors embed in their output now belongs on the list of questions you ask them, alongside data handling and guardrails: which outputs get marked, when, and what the verification story is.

Conclusion

The announcement is real, the scope is genuinely worldwide, and the direction is clear. But as of the day we tested, none of it has reached Claude’s actual output: the files we generated on two surfaces carried no provenance metadata, no model that marks at launch exists yet, and the tool that would read a text watermark hasn’t shipped. Against that baseline, the four claims we walked through all fall: the watermark isn’t strippable characters, public detectors can’t read it, nothing published suggests it names you, and editing sits between the two extremes people claim, differently for text than for files.

We could verify the file half ourselves, and we did. The text half can’t be verified by anyone outside Anthropic yet, which is worth remembering whenever someone states confidently what the text watermark does. When the detection documentation ships and marking goes live, we’ll re-run all of this and publish what actually survives.

Everything here reflects what was published and testable on August 13, 2026, and this area is moving quickly, so check against Anthropic’s current documentation before relying on the details.

Not sure what your AI-assisted content will carry, or what to tell clients before it does? Let’s talk.

A few weeks ago, we sat down with a potential client for a project kickoff conversation. The goal was to find out how we could help them integrate AI into their company. Their whole team joined the call, about 15 people, and within the first ten minutes, it was clear everyone had a different idea of what “using AI” meant to them.

One person wanted an easier way to schedule meetings. Another wanted AI to summarize call notes. A few wanted to search through years of client history. Some of what people described, honestly, was closer to plain automation than a need for artificial intelligence. More than a few people in the room were hesitant to bring AI anywhere near client data.

We see versions of this conversation often. Getting a small team to actually use AI well has less to do with picking the right tool and more to do with getting everyone talking about it the same way, and rolling it out based on collective priorities. In this article, we’ll look at what actually gets in the way of AI adoption, and what it takes to get everyone on the same page.

Adoption Rarely Follows a Clean Plan

We too are a small company, and when ChatGPT first became available, people on our team started experimenting with it individually, mostly for writing emails, Slack messages, and other everyday communication. Some team members also found it especially useful for polishing blog posts before publishing. It wasn’t long before our Operations team confirmed that using AI to help write blog posts was an approved use case, as long as it was being used responsibly. That became one of our first shared, agreed-upon use cases for AI.

Around that time, I was pairing with our founder while he was doing quality assurance on blog posts, and I noticed he was using AI a little differently. He wasn’t using it to generate a draft, but to review one from a few different angles before it went out. It was a small moment, but it stuck with me. There were more interesting ways to use AI than what most of us were doing, and at that point they weren’t something the rest of the team really knew about. I realized that while it may seem obvious to the person who is using AI in that way, it may not be obvious to everyone else. From then on I was always thinking about how knowledge sharing would be one of our greatest tools for incorporating AI.

On the engineering side, adoption moved on its own track. A few engineers asked for GitHub Copilot, while others kept writing code the way they always had. As tools like Claude Code showed up, a handful of people started experimenting with those too, on their own schedule.

I gave an internal presentation around this time on using AI responsibly, mostly focused on avoiding sharing sensitive client data, but I also mentioned some of the ways I had seen our team using it during various pair sessions. The discussion that followed surfaced a handful of practices and considerations that a lot of us, myself included, hadn’t heard yet. The landscape was moving fast enough that even people paying close attention were catching up in real time.

From there, more people started weaving AI into their day to day work, each in their own way. We eventually settled on a smaller set of approved tools, company managed OpenAI and Claude accounts, with clear guidance that client work needed to stay inside those approved accounts. Often we don’t use AI at all when it comes to client projects. It depends on our agreements with each client. We also started building AI into our own internal tools, and not long after, into projects for clients directly.

One thing that stuck with me from a client meeting was hearing our founder describe our approach to accidental AI mistakes, like someone unintentionally sharing information they shouldn’t have, as “tell, don’t blame.” The point wasn’t to pretend mistakes wouldn’t happen. It was to make sure people felt safe enough to flag them the moment they did, so that we could prevent any further damage if necessary.

As our AI work matured, we started taking on client projects focused specifically on building AI solutions. And even with all of that progress, our team still spans a real range. Some engineers use AI constantly, some are much more cautious about it or opt out entirely, and plenty of people fall somewhere in between.

Building Alignment on Purpose

There isn’t a single clean framework that gets a team from “everyone has a different idea of what AI means” to “the whole team is using it exactly the same.” Treating alignment as something you build on purpose, rather than something that happens automatically once you pick a tool, seems to matter more than the tool itself.

A few things make that process go more smoothly, based on what we’ve seen work both internally and with clients.

Start by getting people to describe what “using AI” means to them before you talk about specific tools. In that meeting with fifteen people, nobody had actually said out loud what they meant by “using AI,” and that gap is what created the disagreement. A short conversation early on can surface those differences before they turn into confusion later. Of course it is possible that different members of the team use it for different use cases, but it’s important that everyone has the shared knowledge of what’s going on.

Make room for people to ask questions and admit mistakes without getting defensive about it. Our “tell, don’t blame” approach to accidental AI mishaps exists because people are much more likely to flag a problem, like an AI tool touching data it shouldn’t have, if they aren’t worried about getting in trouble for it.

Give people a shared vocabulary and a couple of sanctioned starting points. Blog writing became one of our first agreed-upon use cases for AI because it was clear, low risk, and something people could point to. Having somewhere concrete to start made individual experimentation easier to build on, instead of leaving everyone to figure things out entirely on their own.

Alignment also means talking honestly about what AI shouldn’t do to the way people show up at work. If someone drafts a client email with AI, we expect it to still sound like the person sending it, and like the company as a whole. We try to hold each other accountable for that, catching it when something feels flattened or off voice, the same way we’d catch a typo.

That kind of accountability only works if people feel safe raising harder questions too, like whether AI is going to change what their job looks like, or replace parts of it outright. We don’t have a tidy answer to that. What we do have is a habit of talking about it directly instead of letting the question sit unspoken in the room.

Treat AI as a Team Capability

One of the biggest differences we’ve noticed between teams that successfully adopt AI and those that struggle is that the successful teams don’t leave it entirely up to individuals to figure things out.

One engineering team we work with introduced AI gradually, one workflow at a time. They didn’t expect everyone to suddenly change how they worked overnight. Instead, they rolled out new practices deliberately, made sure everyone understood how to use them, talked openly about where AI was helpful and where it wasn’t, and reinforced that AI suggestions still required human judgment.

Just as importantly, everyone was expected to use the same process. There wasn’t one developer embracing AI while another ignored it entirely. By making AI part of the team’s workflow instead of an individual preference, they created consistency. That consistency made it easier to share what was working, improve their processes over time, and establish common expectations around quality.

What stood out wasn’t the specific AI tools they were using. It was the culture they built around them. AI wasn’t treated as a shortcut. It was treated as another skill the team was developing together.

Start Building Organizational Knowledge Early

Another pattern we’ve noticed is that the earlier a team starts building shared AI knowledge, the more valuable that knowledge becomes over time.

Many people think about AI adoption as choosing the right model or purchasing the right subscription. In reality, a large part of the value comes from everything your team learns after that decision. Prompts get refined, workflows improve, and internal documentation grows as people share what they’ve learned. People discover better ways to solve problems and share those discoveries with one another.

Over time, that knowledge becomes part of the organization itself. New employees can build on it instead of starting from scratch. Teams begin improving existing workflows instead of reinventing them.

The longer a team waits to start that process, the wider the gap becomes. It isn’t just a gap in familiarity with AI tools. It’s a gap in organizational knowledge. Teams that start earlier have had more time to develop shared practices, build internal context, and improve the systems that support their work.

That doesn’t mean every company needs to move as quickly as possible or adopt every new AI tool that comes along. It does mean there is value in starting small, learning together, and letting that knowledge compound over time.

Conclusion

One thing we’ve learned over the past few years is that AI adoption is never really finished.

Even after we settled on a smaller set of approved tools, new practices and considerations keep surfacing. The tools continue to evolve, new capabilities appear every few months, and teams continue discovering better ways to use them.

The small companies getting the most value from AI aren’t necessarily the ones using the newest models or the most sophisticated tools. They’re the ones creating shared expectations, documenting what works, improving their processes together, and treating AI as something the organization learns, not just something individual employees experiment with.

Think back to that fifteen-person kickoff call. The disagreement in the room wasn’t really about AI at all. It was a team that had never stopped to build the shared vocabulary and expectations we’ve been describing throughout this article. That’s the gap worth closing before you pick a single tool. And once that tool or set of tools is picked, make sure that everyone on the team knows how to knowledge share and is onboard with where you are going.

None of this replaces the work of figuring out which AI problems are actually worth solving for your business. We’ve written about that separately. But it’s hard to get any use case off the ground if your team isn’t talking about AI the same way to begin with.

The tools will keep changing. The teams that build a culture of learning, knowledge sharing, and thoughtful experimentation will be in the best position to adapt right alongside them.

Have you started introducing AI into your organization, or are you still figuring out where to begin? If you need help, let’s talk.

That gap is worth taking seriously, because the mechanism most vendors mean when they say “guardrails” has a specific, documented failure mode. In this post, we’ll look at what that mechanism usually is, what recent research shows about how it fails, and the questions worth asking any AI vendor before “we have guardrails” is enough of an answer.

What “we have guardrails” usually means

Ask a vendor what “guardrails” means, and you’ll usually get a version of the same answer: the AI’s actions get reviewed by something else before or after they happen. Sometimes that’s a human in the loop. Increasingly, especially for an agent handling requests continuously, it’s another AI model watching the first one and flagging anything that looks wrong.

What that second model is actually watching is the part vendors rarely spell out. In many implementations, the reviewer isn’t looking at what the AI did. It’s looking at what the AI said it was doing, the explanation or reasoning it wrote out for itself before or alongside the action. That’s a convenient thing to review: it reads like a sentence, and a human or a second model can skim it for a verdict without inspecting every downstream effect. It’s also a much easier thing to fake, because the model producing that explanation is the same model whose motives the guardrail exists to check in the first place.

The guardrail can be lied to

That gap, between what the AI did and what it said it was doing, is exactly what a new paper, “A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense”, went after. The researchers attacked the AI’s own explanation of its actions while leaving the actions themselves untouched, then measured whether a monitoring layer built to catch bad behavior, specifically an AI gaming its own reward signal instead of doing what was actually intended, still caught it.

Before the attack, the monitor caught the bad behavior in around 95% of cases. After, detection fell to under 11%, and the drop held up across different kinds of monitors and different underlying AI models, not just one setup. The researchers describe the 95% figure as a “false average”: it’s an accurate number across situations where a monitor has other signals available, but in the one situation where the explanation is the only signal, the number that actually applies is under 11%, not 95%.

Worth noting: this is one study, published within the last week, and the authors haven’t released code to reproduce it. Treat the specific numbers as evidence of a real failure mode, not as a settled, precisely-calibrated result.

Splitting a request across sessions

The reasoning-trace study looked at a single interaction. A second paper, “SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks”, published the same week, looked at what happens across a longer back-and-forth, and it matters just as much for anyone asking how access is managed at the level of an individual AI agent, not just the account.

The researchers organized jailbreak attempts, conversations engineered to talk a model into doing something it’s supposed to refuse, by how the adversarial intent gets distributed: laid out in one request, or spread piece by piece across several messages. What predicted success wasn’t how long the conversation ran, it was how the objective was organized. One category stood out: splitting the goal across multiple sessions rather than one continuous conversation defeats any safety check that resets its state at the start of each session. That’s not an edge case. Treating each request or each session as a clean slate is the easiest way to implement a check, and it’s also the version this research shows falling apart under exactly this kind of pressure.

If a vendor tells you access is “managed,” the useful follow-up is whether that management carries any memory across sessions, or whether it’s re-deciding from scratch every time a new conversation starts.

When benign interactions compound

The session-splitting finding assumes someone is deliberately spreading out an attack. A third paper, “Benign Alone, Harmful Together: Exploiting Experience Composition in Self-Evolving LLM Agents”, shows the same failure mode can happen without anyone trying to defeat anything on purpose.

The setup: an AI agent with persistent memory, meaning it remembers past interactions and carries that context forward rather than starting fresh each time. The researchers built a sequence of interactions where each one, judged individually, looked entirely reasonable, nothing that would trip a review at the moment it happened. Composed together over time, though, that sequence produced an unsafe outcome. A safety check that reviews each new piece of information as it comes in, on its own, has no way to catch this by design, because there’s never a single moment where anything looks wrong.

This is the sharpest version of the same underlying problem: a guardrail built to inspect one thing at a time, one action, one session, one message, is answering a narrower question than “is this system safe,” and a vendor who can’t describe how they check for the version that unfolds over time hasn’t actually answered it.

OpenAI’s own disclosure

Everything so far comes from research. On August 4, 2026, OpenAI published “Third-party cyber evaluations involving OpenAI models”, something less abstract: two recent third-party evaluations where a model reached past the boundary the evaluation was supposed to enforce, and neither one required the model to defeat a hardened sandbox.

In one, a misconfiguration left the test environment connected to the public internet, and the model found a real website that happened to share a name with its fictional CTF target and used it, credentials included. OpenAI is explicit that this wasn’t a sandbox escape or a zero-day, the boundary had simply never been closed. In the other, the model reused a leaked GitHub token and tunneled a local service out to the internet. OpenAI attributes part of the cause to the evaluation itself never telling the agent that open internet access was off-limits. Neither is the dramatic version of this story, a model breaking out of a hardened container. It’s the less comfortable version: the access was already there, and the model used it.

If the company building the model can have this happen inside its own evaluations, “the vendor sandboxes it” isn’t enough of an answer on its own either. A boundary that exists on paper, or as an instruction the model is expected to follow, isn’t the same as one enforced by default: scoped access, no open network egress unless explicitly allowed, not just a model that’s merely expected to stay inside the lines.

What to ask before you trust a guardrail

None of this means every guardrail is worthless, or that monitoring has no place in a safety architecture. It means “we have guardrails” describes a category, not a specific implementation, and the difference between implementations is exactly what the last few sections covered. (If you’re still earlier in the process, deciding how AI tools fit into your organization’s data handling at all, we’ve covered that separately in “Safely Leveraging AI: Privacy and Security”.) Once you’re at the vendor conversation specifically, a short list of questions gets at the difference faster than a marketing page will:

• Is the check independent of the model it’s checking? A monitor built from the same model family being reviewed, or one that only sees the model’s own explanation of itself, is checking a much softer signal than one that inspects the model’s actual outputs, tool calls, or side effects independently.
• Does it look at what the AI actually did, not just what it said it was doing? Ask specifically whether the review has access to an audit log of real actions (files touched, requests made, data moved) versus a summary the model generated about itself.
• Does it survive a long or split-up conversation? A check that resets at the start of every session or every request is checking a narrower slice of the interaction than a check that carries context across the whole thing.
• How is access managed at the level of an individual AI agent, not just at the account level? An account-level permission doesn’t tell you what a specific automated agent acting under that account can actually reach.
• Which models and model providers are actually involved, and what safety assurances exist with each of them? A vendor’s own guardrails sit on top of whatever the underlying model provider offers (or doesn’t), and that’s a second, separate set of commitments worth asking about directly.

These aren’t gotcha questions. They’re the specific version of “is this safe” that the research above shows actually matters, and a vendor with a real answer to each one has done meaningfully different work than a vendor with a slide that says “guardrails” on it.

Conclusion

“We have guardrails” is a real answer, just not a complete one. That phrase usually means a review layer that watches an AI’s explanation of itself rather than its actions, and three pieces of research from the same week show that explanation can be manipulated, that safety checks resetting per session can be walked around by splitting a request across a longer conversation, and that even individually harmless interactions can add up to an unsafe outcome in a system with memory. OpenAI’s own disclosure that two of its recent evaluations relied on boundaries that were assumed rather than enforced, and got crossed as a result, is the reminder that none of this is theoretical.

All of this research is new, some of it published within the last week, and none of it means monitoring is worthless or that every vendor claiming guardrails is overselling. It means “guardrails” describes a category with a wide range of actual implementations, and the difference matters enough to ask about before you sign a contract, not after.

That’s a different question from the one we covered in our EU AI Act Article 50 piece: that one is about what you’re required to disclose to users. This one is about whether the mechanism keeping the system safe actually holds up, regardless of what you disclose.

Evaluating an AI vendor and not sure which questions actually matter? Let’s talk.

That catches a lot of businesses off guard, because most of the coverage of the AI Act so far has focused on high-risk systems: hiring tools, credit scoring, biometric surveillance. Those obligations are real, but they’re not due for a while yet, more on that below. The transparency tier is different: it applies now, and it applies broadly, to providers and deployers whose systems are placed on the EU market or whose output reaches EU users, so a US business with EU clients or site visitors is in scope too.

In this post, we’ll cover who’s actually in scope of Article 50’s transparency rules, the concrete steps you need to take, and what’s genuinely still years away.

Who’s actually in scope

Most of what got written about the EU AI Act in 2024 and 2025 focused on its risk tiers: unacceptable risk (banned outright), high-risk (strict conformity requirements), and everything else. If you weren’t building a hiring algorithm or a credit-scoring model, it was easy to conclude the Act didn’t apply to you.

Article 50 doesn’t work off that classification. It’s a separate, horizontal layer of obligations that applies based on what an AI system does, not how risky regulators judged it to be. A system that interacts directly with people, like a chatbot or a virtual agent, is in scope. A system that generates or manipulates content, text, image, audio, or video, is in scope, whether or not a person reviews the output before it goes out. A system that performs emotion recognition or biometric categorization is in scope. None of these is classified as high-risk under the Act’s own tiers, and all three are covered by Article 50 regardless.

A support chatbot on your website is the first category. AI-drafted marketing copy or blog content is the second. Neither reads as “high-risk AI” to most SMB owners, which is exactly why this tier catches people who assumed they were in the clear.

Scope isn’t limited to companies established in the EU, either. The obligations reach any provider or deployer whose AI system is placed on the EU market, or whose output is used by people in the EU. A US-based business with EU customers interacting with its chatbot, or an EU audience reading its AI-drafted content, is in scope on that basis alone.

What Article 50 requires

Per the Commission’s guidelines on Article 50, the obligations break down by what your system does and who you are in the transaction: provider (the business that builds or supplies the AI system) or deployer (the business that uses it). For most SMBs reading this, you’re a deployer: you didn’t build the chatbot or the generative model, you’re running one someone else built.

In practice, that means:

If your chatbot already says “AI Assistant” in its name or interface, you likely already satisfy the first row. If your marketing team publishes AI-drafted blog posts or social copy without a human edit pass, the fourth row is the one to look at first, it’s also the easiest to miss, since “matters of public interest” is broader than it sounds and covers ordinary commentary, not just news reporting. (If AI-drafted marketing copy is new territory for your team, we’ve also covered the privacy and security side of adopting AI tools, which is a separate concern from Article 50 but tends to come up around the same time.)

For the marking and labeling rows, you don’t have to invent a visual system from scratch. The Commission publishes a set of standard EU icons for labeling AI-generated content: a basic icon for AI involvement in deepfake or published text, a “fully AI-generated” icon for wholly synthetic content, and a “partially AI-modified” icon for human content that’s been altered with AI. Using these specific icons is optional, but whatever marking you use has to meet the same bar: it needs to be clearly perceivable at the point someone first encounters the content, embedded directly in it rather than living in a separate disclaimer page, and it has to survive the content being reshared or downloaded. The icons themselves are free, available as SVG or PNG, in black or white with optional transparency, which makes them a reasonable default if you don’t want to design your own.

For the other two rows, disclosing a chatbot and notifying people subject to emotion recognition or biometric categorization, the guidelines don’t prescribe an exact format. A clear statement in the interface satisfies the chatbot disclosure; there’s no required certification or specific wording.

There’s one exception worth knowing about if you were already running a generative AI system before August 2, 2026: providers get a four-month transitional period to adapt existing practices to the new marking requirements. The obligation itself isn’t delayed, the grace period only covers systems that were already on the market. If you’re deploying something new now, it doesn’t apply to you.

The cheap compliance route

You don’t have to work out how to satisfy Article 50 from scratch. The European Commission published a Code of Practice on Transparency of AI-generated Content specifically to give providers and deployers a pre-approved way to comply: one section covers marking and detecting AI-generated or manipulated content (the provider side), the other covers labeling deepfakes and AI-generated text (the deployer side, which is the one most SMBs will use).

By the end of July 2026, around 190 companies and organizations had signed it. Signing means you can point to the Code’s measures as your compliance approach, rather than building your own justification for a regulator. It also buys predictability: signatories get the same treatment across every EU member state, instead of a patchwork of national interpretations, and get a seat in the taskforces that shape how the Code evolves.

You don’t have to sign it. But if you don’t, the burden shifts to you: you have to independently demonstrate that whatever alternative measures you built are adequate, and that gets assessed case by case by whichever member state’s market surveillance authority is looking at you. For an SMB without in-house legal counsel, adopting the Code’s existing measures is very likely the cheaper path, both in setup cost and in what happens if a regulator ever asks.

What’s not due yet

The AI Act’s high-risk tier obligations, the ones for hiring algorithms, credit scoring, and similar systems under Annex III and Annex I, got a real timeline change this year. The Digital Omnibus (Regulation (EU) 2026/1744, in force since July 27, 2026) pushed the Annex III deadline (AI systems classified as high-risk under Article 6(2)) from August 2, 2026 to December 2, 2027. It pushed the Annex I deadline (product-embedded high-risk systems under Article 6(1)) from August 2, 2027 to August 2, 2028.

If your business isn’t in one of those high-risk categories, this section doesn’t change anything for you, Article 50 still applies now, as covered above. But it’s worth knowing about for two reasons. First, if a consultant or a client asks you about AI Act “deadlines” broadly, they may be thinking of these dates, not Article 50’s. Second, a lot of what’s published online about the AI Act’s timeline hasn’t caught up with the Omnibus yet.

We checked one of the more commonly cited trackers, artificialintelligenceact.eu’s implementation timeline, while writing this. As of this writing, it states: “2 August 2027: Application: Article 6(1) and the corresponding obligations in the Regulation start to apply,” with no mention of the Omnibus or Regulation (EU) 2026/1744 anywhere on the page. That’s the pre-Omnibus date. The actual date is now a year later, August 2, 2028. If you’re relying on a bookmarked timeline page instead of checking the regulation directly, you may be working from a schedule that’s already out of date.

Penalties and enforcement

Enforcement sits mainly with national market surveillance authorities in each member state; the AI Office’s own enforcement role is narrower, reaching only systems built on general-purpose AI models where the same company is both provider and model-maker, or systems folded into a DSA-designated very large online platform or search engine. As of August 2, 2026, both have the authority to act on Article 50 violations, not just issue guidance.

The headline number is up to €15 million or 3% of global annual turnover, whichever is higher, for non-compliance with the transparency obligations. EU institutions, bodies, and agencies face a separate, lower ceiling of €750,000.

If you’re an SME or a small mid-cap company, including a startup, the Act caps your fine at whichever of the two figures is lower, the percentage or the fixed amount, not whichever is higher, as applies to larger companies. In practice, that makes 3% of turnover the real ceiling for most small businesses, not €15 million. The obligations still apply in full either way, this only changes what a violation can cost you.

Conclusion

This should give you a very good idea of who’s actually in scope of Article 50 (probably you, if you’re running a chatbot or generating AI content), the five things the transparency tier actually requires, the Code of Practice as the lower-effort way to demonstrate compliance, and the higher-risk obligations that are still years away regardless of what an old bookmark might say.

None of this is legal advice, and guidance in this area is still evolving even as enforcement starts. If your business touches any of the five categories above, the practical next step is to check your own systems against them directly, rather than assuming a chatbot or a content workflow is too small to count. And if you’re still deciding where AI belongs in your business in the first place, that’s a question worth answering before a compliance one, we’ve written about finding the right problems to solve with AI too.

Not sure whether your chatbot or your AI-generated content already meets Article 50? Let’s talk.

Like a lot of teams, we started leaning on AI to help our authors get ideas onto the page. Posts get written faster and people get past their writer’s block quicker. It comes with its downsides though. Asked to fill in a gap, it will make an assumption. Asked to make a paragraph read better, it will rewrite a carefully hedged sentence into a cleaner one that means something slightly different, or expand it into a claim that can’t be sustained. So AI helped our authors move faster, but it created a good deal more work for the reviewers. It didn’t remove a bottleneck, it moved it downstream.

To help with this, I built an assistant for our blog writing process. In this article, I’ll talk through why I built it, how I decided what it should be, and what it has done for our writing process so far.

Solving the AI problem with more AI is a bit counterintuitive, but some battles we are not going to win, so it’s better to make it work for us than to try to prevent it from being used. The useful thing to build here was a properly constrained version of the tool people were already using. It has two pieces: an MCP (Model Context Protocol) server that knows our blogs: what we have published and how we write, and a Claude Code plugin the team uses to suggest topics, plan a post, draft it and review it before a pull request is ever opened. The whole thing took about ten hours to put together, which is most of the point. It was cheap, it was easy, and so far it looks like it’s working. AI solutions don’t have to be complex, and not everything needs to be a custom agent.

The problem to solve

We are committed to the quality of our content, and reviews are thorough and done by experts on the topic. Naturally, for some topics, we have fewer reviewers than authors. This means what looked like a productivity gain driven by AI was in fact a productivity decrease. Reviewers received more posts, saw more things to flag, posts saw more rounds of reviews, reviews took longer. That came back around to the authors too: more feedback to work through and more back and forth before a post could ship.

Not only that, but the style of writing started to shift. AI introduced its own way of writing and organizing information, and most reviewers won’t flag every little instance of this happening. It feels like a nitpick, and at the end the content reads just fine. This is a problem though. Our content stopped sounding like us, like our voice.

On top of that, with more than 400 articles published across OmbuLabs.ai and FastRuby.io, it’s very hard for one person to keep track of everything we’ve published. So cross-references got missed, redundant content got shipped, and opportunities for creating a series or deepening a topic never surfaced.

Why an assistant

We’re no stranger to automation. In fact, we’ve built a few internal tools on that basis: an LLM assistant that drafts the article summaries for the FastRuby.io newsletter, Pecas, which flags problems in our time entries, and another one that handles our quarterly maintenance reports. We believe that humans should spend time on the things that need a human brain, and automation should take care of the boring bits.

This problem was no different. The accuracy issues were eating reviewer time and the style issues were slipping through, and both were landing on the same people who were already the bottleneck. What we needed to do was:

• Cut down the time our reviewers were spending on each post;
• Improve the quality of drafts before they ever got to review;
• Do it in a way that let our authors keep using AI wherever they find it helpful, because they’re going to either way, and because it does help.

None of that is a writing problem, so a tool that writes better wasn’t going to fix it. What we needed was a tool that knows things: what we’ve already published, what our house style actually is (rather than what we’d say it is if someone asked), and which sentences in a draft are making claims that somebody needs to check.

Deciding what to build

Knowing what we needed didn’t tell me what to build. There are a lot of different ways to solve the problem, and when Large Language Models (LLMs) are involved, there are a lot of fancy, complex ways that sound appealing (but ultimately aren’t worth it). I explored a few different possibilities:

A reviewer running in CI

We already have a tool like this that checks for grammar issues and typos. Plugging something into that would be a viable option.

This was quickly discarded though. It isn’t interactive enough, it’d need to focus on the review aspect and catch issues after they are there. Also, to make sure issues wouldn’t slip through, the goal was to make the AI reviewer very nitpicky. This would result in a lot of comments in the PR that nobody has the time to go through and hurt the conversation that can happen between two people in the PR itself. In other words, it makes things worse for both author and reviewer.

A well-crafted prompt

Write it once, share it with the team. Simple enough, cheap, and it’s where the whole thing starts, how you prompt the LLM.

However, the issue we have at hand is dynamic in nature, and the shared prompt would be static. It’d need to have categories, existing keywords, and references hard-coded into it. All that is stale the minute a new keyword is added, a category is added, or references shift with time.

It also has no concept of related posts or per-author style and expertise, so it doesn’t do much to help find a topic, flesh out an idea, or even review a post by a particular author. A lot of maintenance for a subpar result.

A custom agent

We hear this a lot. I have this problem, I think AI can help solve it, I need an agent. Agents are dynamic, can tackle complex tasks, can call tools and access external data, perfect for this. This would have, in fact, been a valid solution. But an agent needs a place to live, it needs maintenance, and it needs a significant amount of upfront work to work as intended.

Our workflows aren’t complex, the amount of data involved in this is small, and it really just needs to make the tool people are already using better. Anthropic built us a perfectly good agent, we just needed to make it a little bit more specialized. It needs data, not custom pipelines, workflows, and complex actions.

What we built

The last point settled the solution: let’s build something within Claude Code that extends its knowledge of our domain without relying on people telling it stuff. There are several ways to do that, and two of them fit what I needed: an MCP server for giving it access to our data, and a plugin for packaging up the workflows and shipping them to the team. So that’s what I built, a plugin and an MCP server.

Nothing in life is as easy as it sounds though. In order for the MCP to be effective, we needed the data it’d be exposing to be structured and available. As happens in most of these cases, it wasn’t.

So the first thing I built was a very simple data pipeline. It pulls the markdown files from both blogs’ GitHub repositories, reads the block of metadata at the top of each one, and loads the posts into a Postgres database along with their categories, content types, keywords and authors. It also creates an embedding for each post, a numeric summary of what that post is about (stored with pgvector, a Postgres extension built for this kind of data). Two posts that cover similar ground end up with similar numbers, which is what lets the assistant find related posts by meaning rather than by matching keywords.

On top of that database sits the MCP server, which exposes tools, meaning things the assistant can do, and resources, meaning documents it can read. The taxonomy resource is generated live from the database, so the list of valid categories, content types and keywords is whatever we are actually using today rather than something written down once and left to go stale. This makes it much more useful than just asking Claude to do a web search against our blogs. It allows it to see all available categories, top keywords, find the posts closest to a draft, pull stats on the content, and so on. It also exposes tools like style checks, looks for common antipatterns, and tools that validate the metadata at the top of a post.

The plugin complements it by holding the workflows. It determines the steps to follow to suggest a topic, plan a post, draft one, and then review it. It works the same for every author, it pulls from one source of truth, but it still has author-specific context available to it.

The split matters. Anything that is true about the blogs lives on the server, so when a style rule changes or a new pattern to look for needs to be added, the change applies to everyone at once. And when something in how we work changes, or we want to improve a workflow, the change applies to the plugin through a pull request like any other, and everyone gets it at the same time.

For an author, this means they can get better, more relevant topic suggestions, higher quality drafts, and less back and forth on reviews as issues get flagged early. Human reviewers are not replaced, we still need an expert to review the post and catch anything the tool can’t. It just helps reduce the number of things left for them to catch, and how much slips through.

Apart from the metadata validation, none of what the automated reviewer flags is a hard rule. Issues flagged in review can be ignored and the post will move forward all the same. The author still decides.

What it cost

The whole point here was simplicity. A quick but effective solution for a problem that’s been bugging us for months.

Those ten hours were a flight from Atlanta to São Paulo (so it’s not like I had anything better to do anyway), and this was enough because of everything I left out. There’s no custom interface, the interface is Anthropic’s problem. There’s no user management, because access to the repository is access to the plugin. There’s no orchestration layer, nothing coordinating a chain of steps, because the skills are markdown files that describe what to do and in what order (we’ve written about how to write a Claude Code skill before). Nearly every part of this that would have taken real time to build is a part I decided I didn’t need.

On the other side of the ledger, we expect this to cut the time spent reviewing a post, and the back and forth that comes with it, by something in the region of thirty to forty percent. The early results have been better than that, but we’re being careful with them: it’s a small number of posts so far, and those posts had already been through part of our usual process before the tooling saw them, so the comparison flatters the tool. We’re treating the early numbers as encouraging rather than settled. Even at the conservative end, the time being saved belongs to the people who are hardest to free up, and ten hours is not a lot to spend to get some of it back.

Conclusion

There’s nothing especially sophisticated about this assistant. It’s a data pipeline, a database, an MCP server that exposes what’s in it, and a handful of markdown files describing how we work. Ten hours and around $2,000 worth of time, against the weeks and the tens of thousands a custom agent would have cost to build and keep running. Most of the effort went into working out what those pieces needed to do, and that part had very little to do with AI.

One thing worth being explicit about though is that this worked because our writing process was already well-established. We knew what the steps were, who did them and where they got stuck. Automating a workflow you don’t understand very well tends to produce a tool nobody uses or, worse, one that makes the confusion happen faster. If you’re looking at your own process and wondering where AI fits into it, that’s the part to get right first. We go deeper into that in Finding the Right Problems to Solve with AI.

Got a workflow where AI would obviously help, but you’re not sure what to actually build? Let’s talk!

With today’s image generation models, it might seem like this should be straightforward. A user describes what they want, an image generation model creates a design, and you’re done.

In practice, building an application that consistently produces useful product designs is much more involved. The image generation model is only one part of the solution. The real challenge is designing the application around the model so that it has the information and context it needs to generate useful results.

There is no single way to build these kinds of systems. Every project presents different challenges, different sources of domain knowledge, and different technical constraints. Throughout development, we found ourselves making a series of decisions. Where should different types of information live? Should it become part of a prompt? Should it be retrieved from a knowledge base? Should it be provided through a tool or reference image? Which image generation model best supports the workflow we want to build?

In this post, we’ll walk through how we approached those decisions. We’ll look at the system that transforms a user’s request into a design that better reflects a company’s products, expertise, and constraints.

The Illusion of “Easy” AI Design

To make this concrete, let’s imagine a company called SneakerCompany. They are a direct-to-consumer brand that wants to let customers design their own sneakers using AI.

A customer enters a request like:

“A modern sneaker with a premium feel, neutral tones, and subtle texture.”

An image model generates a sneaker, and you’re done, right?

If the goal is simply to create an interesting image, maybe.

If the goal is to generate a product that could actually be manufactured, that single request needs to become something much richer.

That richness includes a material for the upper (leather, knit, suede), a call on whether the sole is minimal or performance focused, and a concrete definition of what “premium” means for this company. It also has to reflect which materials are approved, which combinations can actually be manufactured, and which finishes align with the brand.

This is the gap between AI-generated images and real product design. Success depends less on the image model itself and more on the system that communicates with the model to generate the right result.

Capture Domain Knowledge

Once we understood the problem we were trying to solve, the next step was understanding what information an application would need.

That knowledge rarely comes from a single source. Depending on the product, it may come from designers, engineers, manufacturing teams, product managers, brand specialists, existing documentation, product catalogs, or years of institutional knowledge.

Each source contributes a different piece of the puzzle.

Some of the questions we started with included things like:

• How do experts describe the product when speaking about it?
• What terminology do they use?
• Which combinations should never be generated?
• What materials, colors, or finishes are approved?
• What information exists in a visual form?
• What do users commonly ask for, and how was the company interpreting that before?

As we worked through these questions, we realized that not all knowledge is created equal.

Some information rarely changes, like brand guidelines or materials that should never be used.

Some information is highly visual, such as stitching patterns, sole designs, or shoelace styles.

Other information changes regularly, such as product catalogs, trends or historical customer designs.

Understanding the different kinds of knowledge available became just as important as collecting the knowledge itself.

Designing the System

Once we understood the information available to us, the next challenge was deciding how the application should use it.

There is no single architecture for AI-powered design applications. Every piece of information raises a different design decision.

While iterating with our clients we began coming up with these kinds of questions for ourselves:

• Should a business rule, like a banned sole material, become part of the instructions sent with every request?
• Should thousands of historical sneaker designs be retrieved from a database when needed?
• Should curated images of approved laces and stitching patterns be provided directly to the image generation model as reference images?
• Should the application call a tool to retrieve information that changes frequently, like current shoe trends?

These decisions weren’t made in isolation. They were influenced by both the type of information we were working with and the capabilities of the image generation model. As we refined the system, we found ourselves iterating between the architecture and the model. Sometimes a new piece of domain knowledge changed how we wanted to build the application. Other times, the capabilities of the model changed what was possible.

That kind of back-and-forth iteration is only possible if you can see what’s actually happening inside the system, which is why we built tracing in as early as possible. AI applications often have many moving parts, including retrieved context, reference images, tool calls, prompts, and model responses. Having visibility into each step made it much easier to understand why a particular design succeeded or failed, and let us iterate on the system with much greater confidence.

For example, we found that company-wide rules, such as sneaker materials that should never appear in generated products, were a natural fit for the system prompt because they should apply to every request.

On the other hand, large collections of historical product designs were too extensive to include directly in a prompt. Instead, they were better suited for retrieval or tool calls that could surface the most relevant examples based on the user’s request.

Highly visual concepts presented another challenge. Curated reference images of approved laces, materials, or stitching patterns often produced more reliable results than text descriptions alone. In those cases, the workflow was designed to provide both text and images to the model.

As we gathered more domain knowledge, we were continually asking the same question:

What’s the best way for the application to use this information?

That question shaped nearly every architecture design decision we made.

Selecting an Image Generation Model

As we designed the system, we quickly realized that many of our architectural decisions depended on the capabilities of the image generation model itself. Choosing a model wasn’t simply a matter of finding the one that produced the nicest images. We needed to understand what each model could do, how those capabilities fit into our workflow, and whether the tradeoffs made sense for the application we were building.

One example came from a project where we wanted to generate four different design concepts for the user on the very first iteration. This immediately raised the question of whether the model could generate multiple images in a single request, and whether doing so together would be more efficient than issuing four separate requests, given the tradeoffs each approach carried for latency and cost.

Cost became an important consideration because image generation pricing varies significantly between models. A workflow that generates four images for every user interaction can become expensive very quickly, making it important to balance image quality with the overall cost of running the application. We keep the client updated on these costs and how they could change over time as we are going through the project.

Input capabilities also played a major role in our thought process. Some models only accept text, while others can work with one or more reference images. As we discussed in the previous section, we found that certain types of domain knowledge, such as approved materials, stitching patterns, or historical product designs, were often communicated more effectively through images than text alone. That meant we needed a model capable of accepting image inputs as part of the generation process.

Editing capabilities were another important factor. In many design applications, the first image is only the beginning of the conversation. Users often want to refine an existing design by changing the color, replacing a material, or adjusting part of the product while leaving everything else unchanged. If the workflow depends on iterating from previous designs, the model needs to support image editing in addition to generating images from scratch.

Rather than selecting a model based on benchmarks alone, we found it more useful to evaluate models against realistic product design workflows. Image quality was only one consideration. We also compared cost, latency, support for reference images, editing capabilities, and how well each model handled the kinds of inputs our application would generate. The architecture, the available domain knowledge, and the desired user experience all influenced which model was ultimately the best fit.

Building and Refining the System

Designing the architecture and selecting an image generation model wasn’t the end of the process. Building AI-powered applications is inherently iterative, and many of our early assumptions changed as we gained a better understanding of the domain and observed how the system behaved in practice.

As we experimented with different approaches, we continually refined how information was presented to the models. Small changes to the instructions sent with each request, retrieved context, reference images, or tool outputs could have a significant impact on the quality of the generated designs. Rather than thinking about prompt engineering as a single activity, we found it more useful to think about how the entire system prepared information for the model.

Tracing played an important role throughout this process. Because image generation applications involve many moving parts, including prompts, retrieved knowledge, tool calls, reference images, and model responses, it can be difficult to understand why a particular result was successful or where something went wrong. Building tracing into the application from the beginning gave us visibility into every step of the workflow and made it much easier to evaluate architectural changes as the system evolved.

Evaluation became equally important. Instead of only evaluating the final image, we looked at the entire workflow: whether the correct reference images were retrieved, whether business rules were applied correctly, and whether the generated design reflected the user’s original request. Looking at the complete system helped us identify opportunities for improvement that would have been difficult to find by evaluating the model alone.

Putting It All Together

Let’s return to our SneakerCompany example.

A customer enters the request:

“A modern sneaker with a premium feel, neutral tones, and subtle texture.”

Rather than sending that request directly to the image generation model, the application begins gathering the additional information needed to produce a useful result.

It applies company-wide business rules, retrieves similar historical sneaker designs, selects curated reference images for approved materials and laces, and incorporates any other relevant context gathered during the design process. The application then combines all of this information into the final request that is sent to the image generation model.

The model uses that enriched context to generate design concepts that better reflect the company’s products, design language, and manufacturing constraints rather than relying solely on the user’s original request and a model’s breadth of knowledge.

Although every application will differ, the overall workflow often follows a similar pattern.

The exact implementation will vary depending on the product and the domain, but the underlying principle remains the same. The application is responsible for transforming a simple user request into the information the image generation model needs to produce useful, realistic, and brand-appropriate designs.

Conclusion

One of the most interesting parts of working with generative AI for physical products is realizing that success rarely comes from the model alone. The real value comes from understanding the people already doing the work, capturing the knowledge they rely on every day, and building systems that translate that expertise into something scalable and usable.

As image generation models continue to evolve, the biggest opportunities won’t simply come from generating better images. They’ll come from designing better systems around those models, systems that combine domain expertise, business rules, reference material, and user requests into a workflow capable of producing reliable and useful results.

Have an idea for how AI can enhance your product? Our team at OmbuLabs.ai can help. Let’s chat.