Introduction
When we onboard a new event source, the system looks at a few sample events on the site and works out where each field lives on the page: title, description, date, time, price, image, venue, and ticket link. An admin then reviews the extracted samples before the source goes live. Sometimes a value is wrong. The extraction grabbed footer text instead of the event title, or pulled in a paragraph of extra text around a price, or found nothing at all.
The obvious fix is to let the admin edit the value and save it. That fixes the one event on the screen and nothing else, because the next event from the same site is still extracted by the same wrong rule. An edit is far more useful when we read it as information about the rule: for this page, this is what the extraction should have returned. If the system can reproduce the admin’s value on its own, the rule is fixed for every event from that source, not only the ones the admin touched.
We covered how the LLM generates the extraction rules, and gave a first look at the correction loop, in Behind an AI-Driven Web Scraping System . In this article, you will learn how we compare an admin’s edits against a fresh extraction, how we find where the corrected value lives on the page, and how we stop a failed correction from changing the stored rules.
Edits as an Answer Key
When the admin submits a review, we first check whether anything changed. If every value was left alone, the review counts as an approval and the source is validated without any extraction or LLM work. If something changed, the source moves to a reprocessing state and the rest happens in a background job, so the admin gets an immediate response instead of waiting on the screen.
What we keep from the review is small: for each sample URL, only the fields the admin edited, with the values they typed. That is the answer key, and the rest of the work is about making the extraction rules reproduce it. The loop starts by running the current rules over the pages we saved during onboarding and comparing the output with the answer key. If everything already matches, we are done and no LLM call happens. If not, we ask the LLM for a revised set of rules, extract again, compare again, and repeat up to three times.
Two properties of the loop matter for the rest of this article. First, it works only on the pages saved during onboarding. We never fetch new pages while correcting, so a correction can only be as good as the samples the admin was looking at. Second, each attempt starts from the previous attempt’s rules, and nothing is written while the loop runs. The new rules and the new sample events are saved only once every edit has been reproduced.
Let’s look at a simplified version of the loop in Python-style pseudocode:
MAX_ATTEMPTS = 3
def resolve(rules, pages, answer_key):
mismatches = compare(extract(rules, pages), answer_key)
if not mismatches:
return rules
for attempt in range(MAX_ATTEMPTS):
events = all_corrected_events if attempt == 0 else mismatches
proposed = ask_llm(rules, events, pages)
rules = apply_anchors(proposed, mismatches, pages)
mismatches = compare(extract(rules, pages), answer_key)
if not mismatches:
return rules
raise Exhausted(mismatches)
The interesting decisions are inside the helpers, so let’s take them one at a time. compare decides what counts as wrong, which is the next section, and apply_anchors pins the admin’s value to a place on the page, which comes after that. The events line is worth a closer look. On the first attempt, we show the LLM every event the admin corrected, including the ones the current rules already get right. If it only saw the failures, it could fix them with a selector that happens to break events that were fine, so we give it the already-correct examples as well and the revised rules have to keep reproducing them. Later attempts send only the events that are still wrong, because every event we include brings its page HTML along and the prompt grows quickly.
Deciding What Counts as Wrong
Before the LLM revises anything, the loop has to decide which values are wrong, and plain string equality is a poor way to do that. Admins copy values from the page and paste them into a form, so the text they type can differ from what we extracted in ways nobody would call a real difference: a non-breaking space where the page had a regular one, an invisible zero-width character, an en dash where the extraction had a hyphen. If we treated those as mismatches, the loop would spend LLM calls chasing differences that do not exist.
The first step is a normalization pass applied to both values before comparing them:
def normalize(value):
value = unicodedata.normalize("NFKC", value)
value = re.sub("[\u200b\u200c\u200d\ufeff]", "", value)
value = re.sub("[\u2013\u2014\u2212]", "-", value)
return re.sub(r"\s+", " ", value).strip()
NFKC folds compatibility characters, like the non-breaking space, into their ordinary forms. The next line strips zero-width characters, the one after maps the en dash, em dash, and minus sign to a plain hyphen, and the last collapses runs of whitespace. If the two normalized values are equal, the field matches.
Normalization does not cover everything. $10 - $25 and $10-$25 are different strings after it, because collapsing whitespace does not decide whether the spaces around a dash matter, but they carry the same information. For pairs like that we make one small LLM call, a judge, that sees the field name and both values (each capped at 2,000 characters) and answers whether they say the same thing. The call is made only when both values are non-empty and still differ after normalization. If either side is empty, the field is a mismatch without asking anyone:
def equivalent(field, extracted, expected):
a, b = normalize(extracted), normalize(expected)
if a == b:
return True
if not a or not b:
return False
return judge(field, extracted, expected)
The judge’s prompt changes the bar by field. For a price, the extracted value matches if it contains every price the admin listed, even if it also contains others, since the selector may have captured a whole pricing table. For a description, paraphrasing is fine but missing essential facts is not. For identifier-style fields such as the title, venue, date, and links, it asks for near-exact equivalence and accepts only differences in formatting.
The judge fails closed: if the LLM call errors, the pair counts as a mismatch. That means a judge outage can make a correct value look wrong and cost us a wasted attempt. Judges have limits of their own, which we cover in Traces to Insights: Evaluating LLM Apps .
Diagnosing Before Retrying
Once we know which fields are wrong, we work out why before asking the LLM to fix anything. Sending it “this value is wrong, try again” gives it very little to go on, while “you captured the right text plus extra text around it” points it at the actual problem. The failure types will look familiar if you read the earlier post , where we described them as extraction failures. What is new here is that the system assigns them mechanically, with plain string logic and no LLM call, by comparing what we extracted with what the admin typed:
def diagnose(field, extracted, expected, row):
extracted, expected = normalize(extracted), normalize(expected)
if extracted and expected:
if expected in extracted:
return "over_scoped"
if extracted in expected:
return "under_scoped"
if extracted and equals_another_url_field(extracted, row):
return "field_collision" if is_url(field) else "extracted_is_url"
if is_text(field) and extracted and expected:
return "unrelated_content"
if is_text(field) and expected and not extracted:
return "selector_matched_nothing"
return None
The checks run in order and the first match wins. If the expected value sits inside the extracted one, the selector matched a container with extra text around the target, which we call over-scoped. If it is the other way around, the selector matched only a fragment, so it is under-scoped. Next comes the case where the extracted value is exactly the value of another URL field on the same row, which means a text field captured a link instead of readable text, or that two link fields captured the same URL. Two text values with nothing in common are labeled unrelated content, as when a selector matched the footer address instead of the event title. When the extraction is empty but the admin supplied a value, the selector matched nothing on this page.
We also diagnose whole pages. If every content field on a page came back empty, we flag the page itself as unmatched, because that usually means its layout differs from the pages the rules were written against.
Each label goes to the LLM next to the extracted and expected values, and the prompt explains what the label means and what to do about it. For unrelated content, it says that tightening or widening the selector will not help, and that the model should find the element that holds the expected text, scoped to the main content area instead of page-wide regions like the footer. For an unmatched page, it asks the model to compare that page’s HTML with the pages that worked, and if the layouts differ, to widen the selectors or add a fallback rule for the same field. The diagnosis also decides whether we can do something better than ask the LLM, which is the next section.
Anchoring the Corrected Value
A diagnosis tells us what is wrong with a selector, but the admin’s value tells us where the right one is. The admin typed the correct text, so that text exists somewhere on the page we saved, and we can search for it instead of hoping the LLM finds it.
The search looks through the text on the page we saved, using the same normalization as the comparison. Admins often paste several elements into one value, like a venue name and an address separated by a pipe, so we also search for each part on its own, starting with the longest. A piece that appears in exactly one place is the best match, since it is the least ambiguous. If the text is split across several inline elements, or the admin’s version drifts a little from the page (“@ 9pm” versus “@ 9 pm”), we fall back to the smallest element that contains the start of the value.
Once we have the element, we turn it into a selector by starting from the nearest ancestor with a class or a stable id and walking down to the element, something like div.entry-content p strong. We skip ids that look generated, because they only exist on that one page. Then we check the result: the selector’s first match on the page has to be the element where we found the text. If it is not, we try a more specific selector, and if none works, we discard it.
An anchor goes to the LLM as a hint, and the prompt tells the model to use it directly or only narrow it further. A model can ignore a hint, so after it responds we apply the anchors ourselves:
def apply_anchors(proposed, mismatches, pages):
for field, anchors in find_anchors(mismatches, pages).items():
selector = ", ".join(anchors.selectors)
if anchors.replace:
proposed = replace_mapping(proposed, field, selector)
else:
proposed = extend_mapping(proposed, field, selector)
return proposed
Only three diagnoses are eligible, and only for text fields: over-scoped, unrelated content, and matched nothing. For unrelated content and over-scoped, the anchor replaces the selector, because the old one was pointing at the wrong place. For matched nothing, we extend the selector instead, writing it as old selector, anchor, so the pages the old selector already handles keep working. If anchors from several pages disagree, because they use different templates, they are joined into one selector list, and if any of them calls for a replacement, the whole field is replaced. Either way the result becomes a plain CSS read, and if the LLM proposed no rule at all for the field, we add one.
Failure Handling
If mismatches remain after the third attempt, the loop stops. We do not try anything else, like a fourth attempt or a different prompt. The run is marked as failed, and the reason counts how many fields across how many events still do not match. If some events extracted nothing at all, the reason adds that their pages likely use a different layout. This is the same rule we described in Case for AI powered Data Pipelines : if a correction does not produce a working extraction, onboarding fails rather than letting a bad result through.
Because nothing is written while the loop runs, a failure leaves the rules and the sample events exactly as they were before the correction. A success writes everything in one update: 2026-10-28 20:02:21
The failure page shows the reason, followed by one table per event with a row for each field that still does not match: what we extracted, what the admin expected, and a likely cause taken from the diagnosis. For a page where nothing was extracted, the table says its layout probably differs from the other sample pages. From there, the admin can pick different sample events and onboard the source again. These per-field details exist only when the loop runs out of attempts. Other failures, like an LLM request error or missing saved pages, mark the run as failed with the error message and no table.
Conclusion
In this article, we talked about how we use an admin’s edits as an answer key instead of just saving the edited values. We normalize both sides before comparing them and ask a small LLM judge only about the differences that remain. We diagnose each wrong field with plain string logic, so the LLM gets a reason along with the failure. We search the saved page for the admin’s value and pin the field to that location, so a wrong guess from the LLM cannot undo it. The loop gives up after three attempts, and a failed attempt changes nothing.
The main caveat is the one we mentioned earlier: the loop can only fix what the saved pages can show. If the sample pages the admin reviewed do not represent the rest of a site, a correction can match every edit and still miss other layouts.
Need help building an AI agent that works in production? Let’s talk .