Skip to content

The same page, the same article

Nika 0.119.0 made article extraction give one text per page in every run, fixed pages it misread, and made the crawl read robots.txt per RFC 9309.

A workflow that reads a news story or a blog post often wants the article, not the whole page. The menu, the cookie banner, the share buttons and the related links are noise for a model that has to summarize, and noise in a file you keep. Nika's nika:fetch tool, the same built-in whose retries v0.118.6 made effect-safe, has a mode for that. The extraction-modes page of the language's standard library shows it like this:

fragment
invoke:
  tool: "nika:fetch"
  args:
    url: "https://news.example.com/2026/05/22/headline"
    mode: article

mode: article returns the main body of the page as Markdown. The same page describes how the reference engine gets there: a cascade of three extractors. First a rule pass that targets the page's content container, then a Readability pass for pages with poor markup, then a text-density floor for pages built from generic blocks. The earlier stage wins when it yields a substantial body.

Before v0.119.0, that mode had a problem no single run would show. Run it twice on the same page and it could return two different articles.

Why one page gave two articles #

The 0.119.0 notes explain the cause. The Readability stage scores candidate blocks of a page and keeps the best one. When two candidates tied, it broke the tie by the iteration order of a hash map, and that order changes from one process to the next. So, in the notes' words, "one page could yield two different articles from one run to the next". For one page of a benchmark set they record "40 identical runs through the door → 30/10"; for another, "three variants".

The fix keeps candidates in the order they were first scored, which is what the original readability.js does. The engine pins a revision of the dom_smoothie library that behaves this way "until upstream releases it". Over the whole development set of the benchmark, the notes report that "20 runs over the 1497-page dev split now agree byte for byte", with the quality score unchanged.

This kind of defect matters more than its size. A step whose output changes between runs makes every step after it harder to test. A summary changes because the article changed, not because the model did. A comparison of two runs blames the model for a difference the reader caused. A fixed input is the first thing a test needs.

Pages it read wrong #

The same release fixed several ways the cascade misread a page. Each one is described in the 0.119.0 notes.

  • Empty shells. Some pages build their text with JavaScript and ship the whole body inside a <noscript> fallback. The notes name a forum shape. The parser kept that fallback as raw text, so the article came back empty. The cascade now reads the fallback, but only when all three stages found nothing in the served page and nothing of the served page would be lost. A page that extracted cleanly is never replaced by its own "enable JavaScript" notice.
  • Teasers. A semantic container such as main or article could win with a short teaser or byline while the real body sat elsewhere. A zone now has to carry at least 5% of the page's visible text after pruning. Below that, the cascade lets Readability score the whole page.
  • Mastheads. A thin result whose every word already appears in the page's own title or meta description is the page's title bar, not its body. article now reports an empty result instead of serving it. A later fix in the same release keeps short bodies that a page repeats in its meta description, so a real news brief is not mistaken for that furniture.
  • Search boxes. A page that merely has a search field in its header is no longer classified as a page of search results when it has real content.
  • SVG and loose lists. A depth guard protects the parser from documents nested deep enough to be an attack. It counted an element as open until it met a matching close tag, so a page of SVG shapes, or a long list written without </li>, climbed thousands of levels and was refused. The guard now follows the HTML rules for self-closing and implied end tags. The notes name two real pages of the benchmark set that went "from a refusal to an extraction".

The numbers, with their scope #

The changelog publishes its measurements, so they deserve their exact scope. They come from one benchmark the notes call WCXB: a development split of 1,497 pages, and a frozen test split of 511 pages "run once on the final candidate". The notes say the pages were scored with "the benchmark's own scorer" and report a word-level F1. As a general definition, F1 is a score between 0 and 1 that is high only when an output contains most of the reference and little else; the notes do not define it further.

For three fixes measured together (the richer reconstruction that overrides a thin pick, the masthead veto and the search-box rule), the notes report "dev F1 0.7991 → 0.8124, test F1 0.8322 → 0.8504". Other entries of the same release report their own steps on the development split only. The <noscript> change, for example, moved forum pages from a word F1 of 0.487 to 0.628.

These figures describe that benchmark and that scorer. They do not predict how article reads your sites.

The crawl reads robots.txt by the RFC #

nika:fetch can also crawl. With traverse: it walks a site from one page, on the same origin, up to a page limit you set, and by default it honors the site's robots.txt. RFC 9309 says how a crawler must read that file, including when fetching it fails. Before 0.119.0, Nika read it too generously. The notes say the crawl "used to read every non-2xx and every error as allow-all and to govern the landed host with the seed's rules."

Now a 4xx answer on robots.txt means the file is unavailable, and the crawl may proceed. A 5xx answer or a transport failure means the file is unreachable, which RFC 9309 treats as a complete disallow: the crawl refuses loudly before it spends a page, and marks the refusal as transient. When the first page redirects to another origin, the crawl reads that origin's robots.txt before it follows any link. The standard library's nika:fetch page was updated the same day to state the rule.

Two smaller crawl fixes rode along. Each crawled page now passes the same depth check as every other HTML mode. And link discovery reads every link on a page, instead of stopping at the 30 shown in each page's preview.

Why it matters #

A workflow is a file you can review, check and run again. That promise is only as good as its least predictable step. When reading a web page is the first step, a first step that gives a different answer each time makes the rest of the run hard to trust and hard to debug.

The robots fix is the same idea applied to the other side. A crawl should behave the way the site asked, by rules the site's owner can read, including when their server fails.

What this change does not do #

  • Determinism here means the same served page gives the same article in every process. If a site serves different HTML on the next visit, the article will differ too.
  • The tie fix depends on a pinned library revision, kept until upstream ships the same behavior.
  • article can now return an empty result on purpose, for shells and mastheads. A workflow that reads it should expect that case.
  • The <noscript> path helps only pages that ship their text in such a fallback.
  • The F1 figures are measurements on one benchmark, with its own scorer. They say nothing about other sites.
  • A crawl that meets an unreachable robots.txt now stops. That is the RFC's reading, and it can make a crawl refuse where it used to run.

Read the v0.119.0 release, the extraction modes and the nika:fetch reference in the specification.