How we measure what gets read aloud
We built a content-extraction benchmark over a thousand real web pages, and the first thing it did was tell us our previous benchmark had been lying to us. Here is the method, the result, and the two places we got it wrong — including one we only found because a colleague squinted at a page and said "that doesn't look right."
Lullula reads web pages aloud. Before it can read anything it has to decide what on the page is worth reading — the article, not the navigation, the cookie banner, the "you may also like" rail or the four-item colour-theme picker. Every reader-mode tool solves some version of this. The difference for us is that a listener cannot skim.
When a visual reader grabs the wrong block, your eye crosses it in half a second and you forget it happened. When a listening tool grabs it, you sit through every word. The same extraction error that is a shrug in a reader pane is ninety seconds of a legal disclaimer in your headphones. That asymmetry is why we care about the tail of this problem rather than the average of it.
The benchmark that lied
We had a hundred pages with model-judged transcripts, and three days of rule changes measured against them. The score climbed from 0.860 to 0.929. Every rule was justified by a page it fixed.
Then we captured forty-eight new pages, tuned nothing, and scored them. The same rules read 0.688 — below both reference engines. Not a small regression: a reversal. The rules had been fitted to a hundred pages, and a hundred pages is a small enough target to hit by accident.
A single-corpus gain is fit, not capability, until a second corpus moves with it. The climb and the reversal landed on the same day — the cheapest possible version of that lesson, and we still nearly shipped the number.
Three rounds of real fixes later that held-out corpus reads 0.838. But we had to read four of those forty-eight pages to diagnose the fixes, and a held-out set survives exactly one look — after that it is a tuning corpus wearing a disguise. A fresh slice of 44 read 0.865 blind. We wanted one big enough that no handful of pages could swing it, and enough of it left unread that we could not spend it by accident.
What a score means
Every number here is an F1 against a gold transcript: a judge reads the page's visible lines and marks the ones a listener wants — the headline, the body, the captions — and drops the rest. Scoring an engine is then a comparison of two pieces of text. Recall is the share of the gold's characters the engine produced; precision is the share of what it produced that is gold; F1 is their harmonic mean, which is near zero if either is. An engine cannot buy a good score by dumping the whole page, or by emitting one safe sentence.
They are weighted by characters, not lines, because a listener's time is spent in characters. Missing a 600-word section costs three hundred times what missing a two-word caption costs, and a line-counting metric would call those the same mistake. What the numbers feel like, using pages from this corpus:
| F1 | What you would hear | The page |
|---|---|---|
| 0.99 | The article, and nothing else | 3,005 characters against 3,008 of gold; one stray line in 37 |
| 0.89 | The whole article, then about a minute of furniture | recall 1.00, precision 0.80 — 750 extra characters of author bio, a trivia quiz and a cast list |
| 0.55 | The whole article, and twice as much again in junk | recall 1.00, precision 0.38 — a 2½-minute read becomes seven |
| 0.06 | A fragment, buried | 902 characters emitted where the article is 185 |
Why you cannot judge a thousand pages
Gold is the expensive part. Those line lists run about 9 KB across roughly 180 lines a page, so judging all 1,011 is a few million tokens before a single score exists — and that is one pass, where the hard pages want a stronger judge and a second look. Not impossible. Enough to ask whether every page needs judging at all.
Running the engines, though, needs no gold. So we ran three — Lullula, Mozilla Readability and Chromium's DOM Distiller — over all 1,011 pages and let the pages sort themselves by whether the engines agreed with each other:
| Stratum | Pages | Share |
|---|---|---|
| All three agree | 473 | 47% |
| Everything else | 347 | 34% |
| Only Distiller differs | 91 | 9% |
| Only Readability differs | 67 | 7% |
| Only Lullula differs | 33 | 3% |
Strata are assigned by pairwise character-weighted F1 between the engines' own outputs, at a 0.85 threshold. "Everything else" is the residual — most often two engines agreeing and one not.
That sorting is only useful if it predicts correctness. On the fully judged 100-page corpus, all twenty pages where exactly one engine deviates have the deviating engine as the wrong one. But that corpus cannot test the case we care about most: it contains no page where Lullula alone deviates, because our rules were tuned on it. The check stays one-sided until the thousand, where those 33 pages are judged in full — and they score 0.533 for us against 0.734 and 0.759 for the other two. The rule holds in the direction that costs us: when our engine stands alone, our engine is wrong.
So we judged a sample of 199, allocated by variance — a stratum everyone agrees on is pinned down by a handful of pages, a scattered one is not — and the corpus figure is a stratified estimate with a standard error. Validated before we trusted it: treat the 100-page corpus as a population, draw 40-page samples, and the estimator recovers its true 0.9331 to a mean absolute error of 0.008.
The result
The paired margins — each engine against ours on the same page, where most of the difficulty cancels — are +0.043 ±0.020 over Readability and +0.030 ±0.029 over DOM Distiller. Two intervals can overlap while the paired difference between them still excludes zero, and when the two disagree the paired comparison is the one to read. The first is a win. The second has a lower bound of 0.001: a win by one thousandth of a margin of error.
Read blind, before we fixed anything the corpus exposed, Lullula scored 0.853 ±0.029 — ahead of Readability and statistically tied with Distiller. Our tuned corpus had been suggesting margins of +0.08 and +0.11. Those were fit.
And the number we find most useful is not a mean at all. On 47% of pages all three engines agree to within 0.85 F1 of each other — substantially the same text, though only 1.6% match once case, punctuation and line breaks are stripped, and exactly one page in 1,011 matches byte for byte. On that half of the web nobody has a measurable advantage. What separates extractors is how often they fall over:
| Engine | Badly wrong (F1 < 0.5) | Returned nothing | Returned the page |
|---|---|---|---|
| Lullula | 8.8% ±3.0 | 15 | 62 |
| Readability | 11.6% ±3.4 | 19 | 83 |
| DOM Distiller | 11.6% ±3.7 | 34 | 57 |
The first column is estimated from the judged sample; the last two are exact counts. All three are over the same 972 clean pages. The intervals overlap, so read that column as about one page in eleven against about one page in nine, and not as a precise ranking.
Three things it found
A cookie banner was winning the page
Fifty-eight of the 1,011 roots were a consent dialog. A consent notice is long, dense,
link-free prose — exactly the shape a density score rewards — while the article it covers
is shorter and broken up by images and subheadings. Our fix leads with
data-nosnippet rather than a list of vendor names: it is a documented
attribute meaning "do not use this text", publishers put it on consent banners and never
on the article they want read, and it survives the next consent vendor.
Readability gives up on short articles
Its default character threshold is 500. If an extraction comes in under that, Readability calls the parse failed and retries with its filters progressively disabled. On a page whose article genuinely is that short, the last attempt hands back the page. We found it on an aggregator whose stories run 200–400 characters: it returns the nav, and then the site's colour-theme picker, before the story. Any site with short posts — link blogs, changelogs, release notes — sits under that floor.
Chrome's Reading mode behaves like Readability
Chrome's reader renders in a browser side panel, where no automation can read it, so we collected it by hand on nine pages and scored it the same way. It matched Readability within 0.01 on six of the nine — twice emitting output within five characters of it — and was never materially worse. The differences run one way: Chrome recovers where Readability's threshold bottoms out to nothing. Nine hand-collected pages cannot establish that one is built from the other, but for anyone comparing extractors they behave as one baseline rather than two.
Where we were wrong
We had a comparison page with twelve cases on it. Three are gone, and finding out why was the most useful hour of this project.
A colleague looked at one case — a certificate authority's blog post where both reference engines appeared to drown in 40,000 characters of cookie-declaration table — and said it did not look like the main content of that site. The title was right; the post was real. But the numbers were ours, not the web's.
Our captured fixture rendered 81,407 visible characters. The live page has 8,989. The
consent tool's declaration panel carries a "hide" class in both the live DOM and our
fixture — identical markup — but the CSS rule that acts on that class lives in a
stylesheet absent from document.styleSheets. Our capture copies
<style> elements by their text content, and a sheet built through the
CSSOM has none. The rule vanishes, and on replay everything it hid becomes visible.
Re-capturing reproduced it to the byte.
So those engines were drowning in our capture, not in Chrome. Withdrawing the demonstrations was the easy half. The harder half is that the same 39 fixtures were sitting inside our headline estimate, and the corruption pushes in one direction — it inflates hidden text that the reference engines swallow and our consent rule refuses. Taking all 39 out moves our estimate by 0.001 and theirs up by 0.012 and 0.009, cutting the margin over DOM Distiller from +0.040 to +0.030. Every number in this post is the version with them removed. We would rather publish the smaller margin than the one our own bug paid for.
The second one is smaller and more embarrassing. The regression test we wrote for the
cookie fix passed with the fix removed. We had built the test page with semantic
<main> and <article> wrappers, and those beat the
banner on their own — the bug never reproduced. The real pages have hashed class names on
plain divs. Rebuilt that way, the test fails without the fix and passes with
it. It is the second fixture in this benchmark's history to test nothing on the first
attempt, so ablating every new test is now written down as a rule.
What we will not claim
The margin over DOM Distiller is +0.030 ±0.029. That will not support a superlative, and we will not print one.
The gold is model-judged, not hand-judged. Two models scoring the same pages agree at 0.985 — a ceiling on the instrument, not a detail about judges. This benchmark cannot resolve differences between extractors much finer than that, which is worth remembering before anyone reads meaning into a 0.005 gap, including us.
And 19% of the corpus is judged. The other 784 pages are represented by weighting, not by inspection. That is a real estimate with a real error bar, and it is not the same thing as having looked.
Method. 1,011 pages from 819 domains in 30 languages, harvested from the
Tranco top list via each site's own feed or sitemap — 6,454 domains walked, of which 18%
published one we could use — captured as self-contained fixtures and scored offline.
Reference engines run at their defaults on the same markup. DOM Distiller is
chromium/dom-distiller-dist, the library Chrome's old Reader Mode was built
on, and not what Chrome ships today. Reported figures exclude the 39 fixtures affected by
the capture bug; including them reads 0.880 / 0.824 / 0.841.