The Web Curator Tool relaunch 15:30-16:30

Ben O’Brien, National Library of the Netherlands
Hanna Koppelaar, Koninklijke Bibliotheek - National Library of the Netherlands

This tutorial will highlight the new features of the Web Curator Tool (WCT), added from January 2018 onwards through collaboration between the National Library of New Zealand (NLNZ) and the Koninklijke Bibliotheek - National Library of the Netherlands (KB-NL). One of the themes from the collaboration has been to future proof the WCT. This involves learning the lessons from the previous development and recognising the advancements and trends occurring in the web archiving community. The objective is to get the WCT to a platform where it can keep pace with the requirements of archiving the modern web. The first step in that process was decoupling the integration with the old Heritrix 1.x web crawler, and allowing the WCT to harvest using the more modern Heritrix 3.x version. A proof of concept for this change was successfully developed and deployed by the NLNZ, and has been the basis for a joint development work plan. While it will primarily be a demonstration, the tutorial is intended to be an interactive session with the audience and a showcase of how to work collaboratively on opposite sides of the world.

Target Audience:
The target audience for this tutorial is existing WCT users and entry-level organisations/institutions wanting to start web archiving. Some interest from the wider web archiving community would also be expected. We regularly find through the WCT support channels, people and institutions that are new to web archiving wanting to try the WCT. It is often viewed as having a low technical barrier to general use, which we believe is important in bringing in new participants to web archiving. Even after 10 plus years, the WCT still contributes as one of the most common, open-source enterprise solutions for web archiving.
Participant numbers estimated at 10-15.

Background:
In 2006 the NLNZ and the British Library developed the WCT, a collaborative open-source software project conducted under the auspices of the IIPC. The WCT managed the web harvesting workflow, from selecting, scoping and scheduling crawls, through to harvesting, quality assurance and archiving to a preservation repository. The NLNZ has used the WCT for its selective web archiving programme since January 2007. However, the software had fallen into a period of neglect, with mounting technical debt: most notably its tight integration with an out-dated version of the Heritrix web crawler. While the WCT is still used day-to-day in various institutions, it had essentially reached its end-of-life as it has fallen further and further behind the requirements for harvesting the modern web. The community of users have echoed these sentiments over the last few years.

During 2016/17 the NLNZ conducted a review of the WCT and how it fulfils business requirements, and compared the WCT to alternative software/services. The NLNZ concluded that the WCT was still the closest solution to meeting its requirements - provided the necessary upgrades could be made to it, namely a change to use the modern Heritrix 3 web crawler. Through a series of fortunate conversations the NLNZ discovered that another WCT user, the National Library of the Netherlands, was going through a similar review process and had reached the same conclusions. This led to collaborative development between the two institutions to uplift the WCT technically and functionally to be a fit for purpose tool within these institutions’ respective web archiving programmes.

Program:

Introduction
• An overview of the WCT
• A brief re-cap of the state of the WCT prior to development.
• Motivations for keeping the WCT and developing it further.

Demonstration
• Run through a basic setup of the WCT, demonstrating the improvements we have made to this process. This is important because while the application itself was reasonably user friendly, the installation and setup was less than straight forward, even for a system administrator. This was often a pain point for new users, who wanted to get started with web archiving but lacked the technical knowledge to setup WCT.
• Demonstrate new WCT functionality and other improvements, with a particular focus on crawling with Heritrix 3 (and any other crawl tools - pending development).

Next Steps
• Discuss future direction of WCT development, and current NLNZ-KB-NL work plan.
• Any further questions.