BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//WordPress - MECv7.37.0//EN
X-ORIGINAL-URL:https://netpreserve.org/
X-WR-CALNAME:IIPC
X-WR-CALDESC:International Internet Preservation Consortium
X-WR-TIMEZONE:UTC
BEGIN:VTIMEZONE
TZID:UTC
X-LIC-LOCATION:UTC
BEGIN:STANDARD
TZOFFSETFROM:+0000
TZOFFSETTO:+0000
TZNAME:UTC
DTSTART:20261023T211640
END:STANDARD
END:VTIMEZONE
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-PUBLISHED-TTL:PT1H
X-MS-OLK-FORCEINSPECTOROPEN:TRUE
BEGIN:VEVENT
CLASS:PUBLIC
UID:MEC-71c5957954fed812a697ac98da7905ec@netpreserve.org
DTSTART;TZID=UTC:20261014T120000
DTEND;TZID=UTC:20261014T131500
DTSTAMP:20260929T203122Z
CREATED:20260929
LAST-MODIFIED:20261001
PRIORITY:5
SEQUENCE:1
TRANSP:OPAQUE
SUMMARY:Core tools updates (Heritrix and pywb) and pre-crawl assessment tools
DESCRIPTION:The IIPC Technical Speaker Series (TSS) facilitates knowledge sharing and fosters conversations and collaborations among IIPC members around web archiving technical work. In this webinar, moderated by Laura Wrubel (Stanford University Libraries), Alex Osborne (National Library of Australia) will provide an update on two core tools (Heritrix and pywb) and introduce two pre-crawl assessment tools developed by László Tóth (National Library of Luxembourg) and Thomas Smedebøl (Royal Danish Library).\nUPDATE ON THE NEW RELEASES AND FEATURES IN HERITRIX AND PYWB\nAn update on Heritrix and pywb covering changes in releases since the April 2026 General Assembly, including improvements to browser-assisted crawling and Groovy job configs in Heritrix, client-side replay in pywb and other notable changes.\nAlex Osborne is Web Archiving Technical Lead at the National Library of Australia, where he has worked on the development, architecture, and operation of the Australian Web Archive, and digital library preservation storage and web infrastructure. He is the lead maintainer of Heritrix, helps maintain pywb, and created OutbackCDX and jwarc.\n“EXISTENTIAL” WEB ARCHIVING TOOLS\nBROWSER-OR-NOT\nBrowser-or-not is a program that scans one (or more) URLs and tries to decide whether to harvest them with a browser-based crawler. To make this decision, the program runs several tests by crawling the URL with and without an embedded headless browser. This talk will introduce the key features of the tool alongside examples from the BnL’s testing.\nLászló Tóth is a software engineer involved in developing web-archiving tools at the National Library of Luxembourg (BnL). This includes web crawling, ingest workflows, and playback. Previously, he worked as a developer for a European media company, specializing in software concerned with broadcasting, media, and post-production. László holds an MSc in Advanced Computing Science from the University of East Anglia (United Kingdom). Outside of software development, he is mainly interested in mathematics and classical music.\nLinks: https://github.com/natliblux/browser-or-not\nTHE CRAWLER CHOICE\nThe Crawler Choice extension is an experimental, curator-facing proof of concept for site-level pre-crawl assessment. It provides an informed recommendation on whether a website as a whole is better suited for Heritrix or Browsertrix before the first crawl.\nRather than assessing a single page, it builds a stratified sample intended to represent the site’s structural diversity across page types and depth bands. Sampling is based on the current page and sitemap data, with crawling as a fallback.\nThe sampled pages are used as evidence for site-level behaviour: whether browser execution or interaction reveals archive-relevant content that a conventional crawler may miss, including client-side rendering, dynamic loading, SPA navigation, fetch/XHR activity, infinite scroll, load-more patterns, and media playback.\nThe extension also uses Chrome’s built-in Gemini Nano as a local secondary assessment layer. It interprets the collected technical evidence, helps identify patterns that may be difficult to capture with fixed heuristics alone, and contributes to the explanation of the recommendation. The model does not make the crawler choice independently; observable technical signals remain the primary basis for the decision.\nHeritrix is the default unless there is positive evidence for browser-based crawling. Strong signals can trigger stop-early handling, avoiding unnecessary analysis of the remaining sample. The result is an explainable site-level decision aid for curators, not a replacement for curator judgment or crawl QA.\nLinks: https://github.com/netarchivesuite/browsertrix-custom-behaviors/raw/refs/heads/main/crawler-choice-extension-v1.3.zip\nThomas Smedebøl has worked with the collection and preservation of digital cultural heritage at the Royal Danish Library since 2021, primarily overseeing four annual harvests of all Danish websites for the Netarchive. Thomas has a broad background in user engagement, teaching, and the development of digital library services aimed at children.\nMODERATOR\nLaura Wrubel is a software developer on the Engineering & Design Team at Stanford University Libraries, where she supports the Libraries’ digital repository and web archiving infrastructure.\n \n
URL:https://netpreserve.org/event/iipc-tss-core-tools-updates-and-pre-crawl-assessment-tools/
CATEGORIES:Member Event,TSS,Webinar
ATTACH;FMTTYPE=image/png:https://netpreserve.org/resources/TSS-webinar-logo.png
END:VEVENT
END:VCALENDAR