Introduction
A few months ago, we were tasked with building a platform that aggregates events across an entire city, concerts, gallery openings, museum exhibitions, comedy shows, community gatherings, and makes them searchable in real time.
On the surface it sounds straightforward: scrape some websites, store the data, build a search index. In practice it is anything but. Every event organiser publishes their calendar differently. One venue lists events in a structured JSON-LD block. Another buries them in a hand-coded HTML table. A third loads them dynamically from a proprietary ticketing API. Dates are formatted as “Sat Jun 20”, “June 20th at 7pm”, “Through August 16”, and “Ongoing.” Prices appear as “$15”, “Free with admission”, “$10–$25 sliding scale”, and sometimes not at all. Venues give you “847 N 3rd St” or “Ardmore, PA” or the full mailing address including the phone number concatenated into the same field.
We had to onboard hundreds of partners, each with a different website structure. We had to parse every field into a consistent schema. We had to classify what was actually an event versus what was a call-to-action, a membership prompt, or a press release. We had to enrich every event with categories and tags so users could find it. And we had to do all of this continuously, as events were added, updated, and removed.
Doing this with traditional engineering, scrapers, hand-written parsers, rule-based classifiers, would have required a large team and would have broken constantly as partner websites changed. Instead, we made a deliberate architectural decision: use AI for every task where the input is variable and the output needs to be structured. Use deterministic code everywhere else.
Here is what that looks like in practice, and why we think it is the right model for this kind of problem: high-variability inputs that need language understanding and interpretation.
1. Faster Time to Value: Onboarding a Partner in Minutes, Not Days
The first challenge was onboarding. Before a single event could appear in our platform, we had to understand each partner’s website well enough to reliably extract events from it, every time the pipeline ran.
With a traditional approach, a developer would inspect the HTML, write a custom scraper, test it against edge cases, and maintain it over time. For a platform with hundreds of partners, that is weeks of engineering work upfront and a permanent maintenance burden.
We replaced this with an automated onboarding pipeline backed by Claude.
When a new partner is added, the system fetches their events listing page and passes the raw HTML to an LLM with one job: figure out the CSS selector that identifies event cards on this page, whether the page paginates, and whether events link out to detail pages or can be extracted inline. The output is a validated ScrapeConfig, a structured object. If it passes a deterministic gate (the selector must match at least as many cards as there are known sample events), onboarding proceeds. If not, the system escalates to a tool-calling agent.
The agent is a bounded loop: it has access to DOM-querying tools, count_selector, dom_excerpt, query_matches, find_repeating_blocks, capture_network, and a maximum budget of 40 tool calls and 7 minutes. It explores the HTML interactively, validates its hypotheses against the real page, and when it finds a selector that works, submits it through a hard acceptance gate. The gate is deterministic: the LLM proposes, code verifies.
The result: adding a new partner, including scraping configuration, field mapping, and data cleaning, is largely automated. What previously took a developer a day of work now takes minutes of compute time and a few minutes of human review.
2. Reduced Engineering Costs: Parsing at Scale without Custom Code
Once events are extracted from a partner’s site, the raw field values are messy. Every partner formats dates differently. Prices come in dozens of forms. Addresses range from fully structured to completely freeform. Venue names sometimes include the full address, sometimes a phone number, sometimes nothing at all.
The traditional solution is a library of regex patterns and parsing rules, one per field type, tuned against the actual data from each partner. It is brittle, it needs constant maintenance, and it fails on formats it has never seen.
We replaced it with LLM-powered parsing.
Every raw date string is passed to Claude with a single prompt: parse this into a start date, an end date, a date type (single day, range, through, ongoing, upcoming), and whether it has a specific start time. The output is a Pydantic-validated object, not a string, not a dict, a typed structured object. The same pattern applies to prices (minimum, maximum, is_free flag) and addresses (street, city, state, structured separately).
These calls run in parallel batches of 20 to 30 items. The LLM handles every format variation without any code, “Sat Jun 20”, “June 20th, 2026 at 7 PM EDT”, “Through August 16”, “Ongoing” all parse correctly. Adding a new partner with a novel date format requires no code change.
The key engineering decision that makes this production-safe: every LLM call returns a structured output, not free text. The AI generates structure; code validates it. If the model returns something that fails validation, it is retried or flagged, never silently passed through.
3. Fewer Errors, Better Data Quality: AI as a Filter and a Feedback Loop
Two of the most valuable uses of AI in this system are ones that prevent bad data from ever reaching the database.
The first is event classification. When a scraper runs on a partner’s events page, it picks up everything that matches the card selector, including things that are not events. Membership prompts. Newsletter sign-ups. “Follow us on Instagram” cards. Press releases. Grant announcements. Each one of these would appear as a fake event in the feed if not caught.
We run every extracted title and description through a classifier before anything is written to storage. Claude reads the title and description and returns a single boolean: is this an event or not? The classifier is prompted with specific examples of what to drop, CTAs, promos, news items, leadership announcements, and has been tuned over hundreds of partner onboardings. Bad data is dropped early in the pipeline, before it reaches storage.
The second is the correction workflow. When an admin reviews an onboarding result and finds that a field is being extracted incorrectly, the title is picking up extra text, the date is wrong, the venue name contains the full address, they flag it. The system takes those corrections, encodes them as expected values, and passes them back to Claude with the original HTML. Claude re-generates the extraction config with the corrections incorporated.
This means the system learns from human feedback without any manual rule-writing. The LLM absorbs the correction and produces a config that handles the edge case. The key framing here: we do not trust AI to be right the first time, and a feedback loop is not a guarantee either. If a correction does not produce a working extraction, onboarding fails rather than letting a bad result through.
4. Move at the Speed of the Business: Search-Ready in Seconds
Once an event passes through the pipeline, it needs to be immediately findable. That means categorised and indexed for search, without any manual curation.
Every new or changed event is automatically enriched by two parallel LLM tasks.
Categorisation: Claude assigns each event to one or more active categories, chosen from a constrained taxonomy. The LLM cannot invent a category that does not exist in the system, that constraint keeps the classifier predictable, though a misclassification within the existing set can still happen.
Keyword enrichment: Claude generates a set of search keywords for each event, terms a user might type to find it that do not appear verbatim in the title or description. These keywords power the keyword-matching side of a hybrid search index, alongside the title and description.
Once enriched, the event is ready for search, keyword and semantic alike, with no editorial step in between.
The entire path from new event on a partner’s website to appearing in a user’s search results is automated and runs continuously. No manual curation, no editorial queue.
5. Scalability Without Headcount Growth: One Team, City-Scale Data
The cumulative effect of these decisions is that a small engineering team can operate at a scale that would otherwise require a much larger organisation.
Consider what the alternative looks like. Each new partner requires a developer to inspect their HTML, write a scraper, write field parsers, handle edge cases, and maintain all of it as websites change. That is a day of work per partner, minimum, and an ongoing maintenance cost. Categorisation requires editorial staff or a complex rules engine. Search requires a search team.
With this architecture, the engineering team’s job shifts from operating the pipeline by hand to building and maintaining the systems that let AI operate within it, the validation gates, the observability, the correction workflows. Adding a new partner is a product decision, not a multi-day engineering project, and enrichment runs without an editorial queue. That does not mean the team disappears, the pipeline, the agent loop, and the taxonomy still need engineers watching them. What changes is that headcount stops scaling linearly with partner count.
The LLM handles the full lifecycle: discovery (what selectors to use), parsing (what the data means), classification (is this real), enrichment (what category is this), and retrieval (what does this query mean). Engineers write the scaffolding, the validation gates, the orchestration, and the observability. AI handles the variable, judgment-intensive work.
6. Risks and How We Mitigate Them
None of this comes without tradeoffs. Here is an honest account of the risks we encountered and how we addressed them.
Risk: Hallucination and Silent Errors
The most dangerous failure mode in an AI-powered pipeline is not an obvious error, it is a plausible-looking wrong answer that passes through undetected. A CSS selector that matches the right number of elements but the wrong ones. A date parsed as 2027 when it should be 2026. A venue identified as being in the wrong city entirely.
Mitigation: Every LLM call in this system returns a structured output, not free text. If the output does not conform to the schema, it fails validation and is retried or flagged. Classification is taxonomy-constrained, the LLM can only choose from active categories. The ExtractionAgent’s submissions go through a hard deterministic gate before being accepted. And the correction workflow means human-spotted errors feed back into the system.
The honest answer: some errors do slip through. The pipeline is designed to surface them, through admin review, correction workflows, and observability tooling, rather than pretend they cannot happen.
Risk: Cost at Scale
Parsing runs on every event on every crawl cycle. Enrichment runs on every new or changed event. If these tasks are not designed carefully, the API cost scales linearly with data volume.
Mitigations used: Batch processing (20–30 items per LLM call instead of one call per item). Change detection, events are only re-enriched when their content actually changes, not on every pipeline run. Fast path attempts before escalating to the more expensive agent loop. Most onboardings never need the full ExtractionAgent, the cheaper structured prediction call is sufficient.
Risk: Data Privacy and Compliance
Sending data to a third-party API raises obvious questions about what data is being sent.
In this system, the answer is clean: the LLM receives publicly scraped event data, the same HTML and text that any browser would see. No user data, no PII, no client-proprietary information enters the AI calls. The boundary between what is AI-processed and what is not is explicit and enforced at the architecture level.
Risk: Inconsistent / Non-Deterministic Output
For a data pipeline that runs continuously, this means the same event could be categorized differently on different runs.
Mitigations: Taxonomy constraints (the output space is finite and bounded). Structured output with schema validation (invalid outputs are rejected and retried). The change detection layer means re-enrichment only runs when content changes, limiting re-classification churn. And for the most critical task, the scrape config, the acceptance gate is deterministic: the LLM’s proposal is tested against real HTML, and the result is binary.
Conclusion
The pattern across all of these use cases is consistent: AI as a proposal engine, deterministic code as the validator.
The LLM is not making decisions. It is generating candidates, selectors, parsed values, categories, keywords, that are then verified by code before being committed to the pipeline. This distinction is what makes AI usable in a production data pipeline rather than just a prototype.
The value is real. Onboarding that would take days takes minutes. Parsing that would require code per partner requires none. Enrichment that would require editorial staff is automatic. A small team can operate city-scale data infrastructure.
The risks are also real. Hallucinations happen. Costs accumulate. Outputs drift. The answer is not to avoid AI, it is to build the scaffolding that catches failures, constrains outputs, and keeps humans in the loop where it matters.
Have questions about how we built this? We’d be happy to talk.