The Extraction Layer That Cut My LLM Token Bill by 96%

I have been building a content pipeline where automated agents read the web — pull down a competitor page, work out what it covers, and report back on what is missing. It worked on the first try, which should have been the warning sign. The API bill climbed steeply, responses took several seconds to start streaming, and the agents kept citing URLs that did not exist anywhere on the page they had supposedly just read.

All three symptoms had one cause: I was passing raw page HTML straight into the model. The fix was not a better prompt or a bigger context window. It was an extraction layer in front of the model, and it turned out to be the single highest-leverage piece of plumbing in the whole system.

Two different jobs, both called extraction

The word "extractor" gets used for two unrelated tasks, and conflating them is why this area is more confusing than it should be. The first job is pulling links out of text — you have a sitemap, a search-results dump, a newsletter, or a page of documentation, and you want the URLs it contains as a clean list. That is what a URL extractor does: messy text in, one URL per line out.

The second job is fetching one of those URLs and reducing the page to its readable content — stripping the navigation, the cookie banner, the inlined stylesheets, the analytics snippets, and the eight variants of the same Open Graph tag, leaving the article. Both stages are necessary and they fail in completely different ways. The first decides what your agent reads. The second decides what it costs.

What raw HTML actually costs

On a typical article page, the content you want is under 10% of the document. Measured across the competitor pages my own pipeline reads, the raw HTML ran 40,000 to 80,000 tokens each — roughly 150 to 350 kilobytes of markup, before counting the images and script bundles the browser would fetch separately. The same articles, once reduced to plain markdown, came to 1,500 to 3,000 tokens. That is a 93 to 96% reduction in input tokens carrying the same information.

The arithmetic scales the way you would expect: pages per day, times average tokens per page, divided by a million, times whatever your provider charges per million input tokens. At ten thousand pages a day, the difference between a 50,000-token average and a 2,000-token average is the difference between 500 million and 20 million tokens daily. Input pricing varies by more than an order of magnitude between models and changes often enough that quoting a rate here would be actively misleading — but the ratio holds regardless of which model you are billed by, and the ratio is the part worth designing around.

There is a latency win that is easier to miss. Time to first token is dominated by the prefill phase, and prefill scales with prompt length. Cutting a prompt by 25 times cuts the wait before the first character streams back. In an agent loop that makes several calls in sequence, that saving compounds at every hop.

Context dilution is the expensive half

Cost you can budget for. The quality problem is worse. Boilerplate lowers the signal-to-noise ratio of the entire prompt, and in a retrieval system it does real damage further downstream: chunk a raw HTML page and you end up with vectors that encode navigation structure and cookie-consent copy rather than meaning. Retrieval then confidently returns the chunk whose markup happened to be closest in the embedding space.

The hallucinated citations came from the same place. Buried in the markup of every page are URLs that have nothing to do with the content — analytics endpoints, CDN paths, preconnect hints, schema identifiers. Given a prompt where 90% of the tokens are markup containing dozens of such URLs, a model asked to cite its sources will sometimes produce one of them. It is not really a hallucination; the URL was right there in the context I gave it. I had asked the model to find the signal in a haystack I built myself.

The delimiter bug that corrupted every URL

Stage one looks like a five-minute regex job, and I have the commit message proving otherwise. The URL extractor on this site matched http and https links, plus www-prefixed URLs written without a scheme, and stopped at whitespace, closing parens, and closing brackets — so it handled sentence punctuation and markdown links correctly. Its delimiter character class was [^\s)>\]], which excludes whitespace, a closing paren, a greater-than, and a closing bracket.

Then I pasted this site's own sitemap into it. Every single entry came back as https://devtoolstack.io/tool/extract-urls</loc — the URL with a fragment of markup welded to the end. The class excluded the greater-than character but not the less-than, so a URL sitting inside a tag ran straight past its own closing tag and only stopped at the next greater-than it found. Quoted attributes broke the same way: the namespace declaration xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" yielded a URL with a trailing double quote still attached, because the class did not exclude quotes either.

The fix needed two character classes rather than one. The body of a URL is [^\s<>"'`)\]] — everything except whitespace, angle brackets, quotes, a backtick, and closing brackets. The final character uses a stricter class that additionally refuses to end on .,;:!? so that sentence punctuation does not get glued onto the link. An XML sitemap, a quoted href, a markdown link, a URL in backticks, and a URL wrapped in parentheses all extract cleanly now, and none of the cases that already worked regressed.

The uncomfortable part is why this survived so long. The test suite for these tools was thorough and entirely green. It had simply never fed an extractor any markup — and markup is the single most common thing anyone pastes into a URL extractor. The bug was not subtle, it was untested. The real fix was adding markup as an input class to the suite, along with an invariant that an extracted value must never retain an angle bracket or a quote, then confirming that check failed against the old pattern before trusting the new one.

Why bare domains are deliberately out of scope

One limitation is intentional. The extractor does not match bare domains like example.com written with no scheme and no www prefix, and it never will. Matching those reliably requires a real list of valid top-level domains, because without one you also match config.json, version strings like 2.5.1, and the word Node.js. Every naive attempt trades a false negative you can see for a swarm of false positives you cannot. Requiring http, https, or www is the honest boundary, and stating it on the page is better than a tool that quietly guesses wrong.

Reducing the page itself

For stage two, do not write your own. Separating article text from boilerplate is a genuinely hard problem, and the maintained libraries encode years of accumulated heuristics about how real pages are structured. Most tutorials still reach for newspaper3k; its last release was in 2020 and it has known installation problems on current Python versions. Trafilatura is the better default — actively maintained, stronger on boilerplate removal, and able to emit markdown directly, which is already the shape you want a prompt in.

The setting that mattered most was preferring precision over recall. When output feeds a model, dropping a sidebar you might have wanted costs you nothing, while keeping a cookie banner costs tokens and accuracy at the same time. Bias every ambiguous call toward discarding. It is also worth extracting the page metadata — title, author, publication date — as separate fields rather than leaving them embedded in the text, because a model reasoning about whether a source is current does much better with an explicit date field than with one buried in a byline.

Deduplicate before fetching, not after. Sitemaps and search-result dumps are full of URLs that differ only by a trailing slash, a tracking parameter, or a fragment, and every duplicate you do not catch is a paid round trip and a wasted rate-limit slot. Normalising with a URL parser to compare hosts and paths while ignoring the query string caught more duplicates than a plain string comparison did.

Four things that cost me time

Pages rendered entirely in JavaScript return nothing useful to a static fetch — you get an empty shell and no error. A headless browser solves it but is expensive enough that running it by default would eat the savings, so it belongs behind a check: if static extraction returned less than some minimum length, escalate to the browser, otherwise do not.

Cache aggressively, keyed on the normalised URL, with a sensible expiry. Agent loops revisit the same handful of pages constantly, and re-fetching costs money, adds latency, and raises your odds of getting rate-limited by a site that was being perfectly reasonable about it.

Respect robots.txt and terms of service. Python ships a robots parser in its standard library, so there is no excuse for skipping the check. Set a User-Agent that identifies you and provides a way to get in touch, and rate-limit per host rather than globally.

Extraction failures are silent by default, which is the trap that bit hardest. Paywalls and JavaScript shells return an empty string or a stub rather than raising, so without a minimum-length assertion after every extraction you will eventually hand a model an empty context and get a fluent, confident answer about nothing at all. Assert on the output length and fail loudly.

What ties this together

Extraction reads like plumbing you bolt on once the interesting parts work. It is not. It determines what your model costs to run, how fast it responds, and whether the answers it produces are traceable to anything real. A 25-fold reduction in prompt size and a measurable drop in invented citations came out of two regex character classes and one well-chosen library — no model change, no prompt engineering, no larger context window.

The wider lesson is the same one that keeps surfacing in this codebase: a green test suite tells you the cases you thought of are handled. It says nothing whatsoever about the case that matters most, which is the thing a normal person actually pastes in on a normal Tuesday. For a URL extractor, that thing was a sitemap, and it had been broken the whole time.