Core tools updates (Heritrix and pywb) and pre-crawl assessment tools

The IIPC Technical Speaker Series (TSS) facilitates knowledge sharing and fosters conversations and collaborations among IIPC members around web archiving technical work. In this webinar, moderated by Laura Wrubel (Stanford University Libraries), Alex Osborne (National Library of Australia) will provide an update on two core tools (Heritrix and pywb) and introduce two pre-crawl assessment tools developed by László Tóth (National Library of Luxembourg) and Thomas Smedebøl (Royal Danish Library).

UPDATE ON THE NEW RELEASES AND FEATURES IN HERITRIX AND PYWB

An update on Heritrix and pywb covering changes in releases since the April 2026 General Assembly, including improvements to browser-assisted crawling and Groovy job configs in Heritrix, client-side replay in pywb and other notable changes.

Alex Osborne is Web Archiving Technical Lead at the National Library of Australia, where he has worked on the development, architecture, and operation of the Australian Web Archive, and digital library preservation storage and web infrastructure. He is the lead maintainer of Heritrix, helps maintain pywb, and created OutbackCDX and jwarc.

“EXISTENTIAL” WEB ARCHIVING TOOLS

BROWSER-OR-NOT

Browser-or-not is a program that scans one (or more) URLs and tries to decide whether to harvest them with a browser-based crawler. To make this decision, the program runs several tests by crawling the URL with and without an embedded headless browser. This talk will introduce the key features of the tool alongside examples from the BnL’s testing.

László Tóth is a software engineer involved in developing web-archiving tools at the National Library of Luxembourg (BnL). This includes web crawling, ingest workflows, and playback. Previously, he worked as a developer for a European media company, specializing in software concerned with broadcasting, media, and post-production. László holds an MSc in Advanced Computing Science from the University of East Anglia (United Kingdom). Outside of software development, he is mainly interested in mathematics and classical music.

Links: https://github.com/natliblux/browser-or-not

THE CRAWLER CHOICE

The Crawler Choice extension is an experimental, curator-facing proof of concept for site-level pre-crawl assessment. It provides an informed recommendation on whether a website as a whole is better suited for Heritrix or Browsertrix before the first crawl.

Rather than assessing a single page, it builds a stratified sample intended to represent the site’s structural diversity across page types and depth bands. Sampling is based on the current page and sitemap data, with crawling as a fallback.

The sampled pages are used as evidence for site-level behaviour: whether browser execution or interaction reveals archive-relevant content that a conventional crawler may miss, including client-side rendering, dynamic loading, SPA navigation, fetch/XHR activity, infinite scroll, load-more patterns, and media playback.

The extension also uses Chrome’s built-in Gemini Nano as a local secondary assessment layer. It interprets the collected technical evidence, helps identify patterns that may be difficult to capture with fixed heuristics alone, and contributes to the explanation of the recommendation. The model does not make the crawler choice independently; observable technical signals remain the primary basis for the decision.

Heritrix is the default unless there is positive evidence for browser-based crawling. Strong signals can trigger stop-early handling, avoiding unnecessary analysis of the remaining sample. The result is an explainable site-level decision aid for curators, not a replacement for curator judgment or crawl QA.

Links: https://github.com/netarchivesuite/browsertrix-custom-behaviors/raw/refs/heads/main/crawler-choice-extension-v1.3.zip

Thomas Smedebøl has worked with the collection and preservation of digital cultural heritage at the Royal Danish Library since 2021, primarily overseeing four annual harvests of all Danish websites for the Netarchive. Thomas has a broad background in user engagement, teaching, and the development of digital library services aimed at children.

MODERATOR

Laura Wrubel is a software developer on the Engineering & Design Team at Stanford University Libraries, where she supports the Libraries’ digital repository and web archiving infrastructure.