Regex Club Live: Collaborative Exploration of Crawl Logs and Crawler Traps

This Regex Club Live event is for IIPC members only.

Crawler traps are a familiar issue for the web archiving community, leading to crawls capturing erroneous or useless material, creating an unnecessary environmental and financial burden, and potentially missing in-scope content when crawls are force-finished. But dealing with crawler traps requires specialist knowledge which, at the UK Government Web Archive, has been built up informally over time, and as such is unevenly distributed throughout the team. Over the last few years, we have sought to address this with a regular Crawl Log Review workshop (informally known as ‘Regex Club’) where problematic crawls are discussed, their logs analysed, and appropriate reject regex patterns are identified and tested. This has become a forum for knowledge sharing and capacity building within our team.

For Regex Club Live we invite you to join one of our Crawl Log Review sessions, where we will work through real examples of problematic crawls, and collaboratively come up with proposed solutions. We will share our process and tools for identifying problematic crawls, preparing crawl logs for review, the review process itself, how we implement and check our solutions, and how we record our outcomes for future re-use. As crawler traps are a shared problem for our community, we look forward to learning more about them together.

Speakers

The UK Government Web Archive Team

About The UK Government Web Archive

The UK Government Web Archive, part of The National Archives (UK), preserves and makes accessible UK central government information published on the web. The Web Archive includes videos, tweets, images and websites dating from 1996 to the present day. Our initial work on crawler traps was presented as a poster at IIPC WAC 2024.