Overview

Run your own full stack SolrWayback

Thomas Egense & Toke Eskildsen

The Royal Danish Library

Description:

This workshop will

1. Explain the ecosystem for SolrWayback 4 (https://github.com/netarchivesuite/solrwayback)
2. Perform a walkthrough of installing and running the SolrWayback bundle. Participants are invited to mirror the process on their own computer and there will be time for solving installation problems
3. Leave participants with a fully working stack for index, discovery and playback of WARC files
4. End with open discussion of SolrWayback configuration and features.

Prerequisites:

• Participants should have a Linux, Mac or Windows computer with Java 8 or Java 11 installed.
• Downloading the latest Release from https://github.com/netarchivesuite/solrwayback beforehand is recommended: the releases can be found at the right side of the GitHub page.
• Having institutional WARC files available is a plus, but sample files can be downloaded from https://archive.org/download/testWARCfiles

Target audience:

Researchers with medium knowledge of web archiving and tools for exploring web archives. Basic technical knowledge of starting a program from the command line is required; the SolrWayback bundle is designed for easy deployment.

Background:

SolrWayback 4 (https://github.com/netarchivesuite/solrwayback) is a major rewrite with a strong focus on improving usability. It provides real time full text search, discovery, statistics extraction & visualisation, data export and playback of webarchive material. SolrWayback uses Solr (https://solr.apache.org/) as the underlying search engine. The index is populated using Web Archive Discovery (https://github.com/ukwa/webarchive-discovery). The full stack is open source and freely available. A live demo is available at http://webadmin.oszk.hu/solrwayback/

During the conference there will be focused support for SolrWayback in a dedicated Slack channel by Thomas Egense and Toke Eskildsen.

Web archive discovery systems and scaling

Thomas Egense & Toke Eskildsen

The Royal Danish Library

Description:

This workshop will

1. Present challenges for building and maintaining a web archive scale discovery system
2. Explain concrete strategies for running Solr at different scales (same strategies should work for Elacticsearch)
3. Provide a forum for sharing experiences and problems with the scale of web archives. Bring your own challenges and we will solve them together!

Prerequisites:

• An interest in the scaling of web archive discovery systems

Background:

The Royal Danish Library has been providing full text search and discovery for the Danish Netarchive for several years, lately using SolrWayback. The archive contains 33 billion records, which are all indexed and available online. Solr is used as the underlying search engine and scaling has been both a design criteria and an ongoing challenge.

Indexing (using Web Archive Discovery (https://github.com/ukwa/webarchive-discovery) and searching (using Solr (https://solr.apache.org/) each have their own issues which can easily compound to larger problems: Setups that works well at a certain size is no guarantee for a working system at 10× that size.

The General Assembly (GA), hosted online by the National Library of Luxembourg was held on Monday, 14 June 2021. GA 2021, followed by the Web Archiving Conference (WAC) on 15-16 June, and the 4th RESAW conference, organised by Luxembourg Centre for Contemporary and Digital History (C²DH) / University of Luxembourg on 17-18 June, form a Web Archiving Week.

Agenda [CEST]

MAIN MEETING chaired by IIPC Vice-Chair Kristinn Sigurðsson (Europe / Asia / Australia / New Zealand) 

09:45-10:00
Login, introduction to Remo & welcome from the Organising Committee

10:00-10:10
Vice-Chair Address: Kristinn Sigurðsson (National and Unviersity Library of Iceland)

10:10-10:30
SPO Report: Olga Holownia, IIPC Senior Program Officer

10:30-10:45
Strategic Plan 2021-2025: Hansueli Locher (Swiss National Library)

10:45-10:55
Strategic Plan 2021-2025: Q&A

10:55-11:00 Short break

11:00-11:05
IIPC Financial Report: Sylvain Bélanger, IIPC Treasurer (Library and Archives Canada)

11:05-11:20 Updates from Working Groups (WGs)

Content Development WG, Nicola Bingham, British Library
Training WG, Maria Ryan, National Library of Ireland
Research WG, Ben O’Brien, National Library of New Zealand

11:20-11:45 Member updates

Németh Márton, National Széchényi Library, Hungary
Friedel Geeraert, KBR – Royal Library of Belgium
Maria Ryan, National Library of Ireland
Yves Maurer, National Library of Luxembourg
Kristinn Sigurðsson, National and Unviersity Library of Iceland

12:00-13:20 Networking and members’ meetings
13:20-13:30 Networking and members’ meetings: summary

13:30-15:30 Break

15:30-15:40 Recap of the earlier networking and members’ meetings
15:40-16:55 Networking and members’ meetings

MAIN MEETING chaired by IIPC Chair Abbie Grotke (Europe / America)
16:55-17:00 Welcome from the Organising Committee

17:00-17:10
Chair Address: Abbie Grotke (Library of Congress)

17:10-17:30
SPO Report: Olga Holownia, IIPC Senior Program Officer

17:30-17:45
Strategic Plan 2021-2025: Ian Cooke (British Library)

17:45-17:55
Strategic Plan 2021-2025: Q&A

17:55-18:00
IIPC Financial Report: Kristinn Sigurðsson

18:00-18:05 Short break

18:05-18:20 Updates from Working Groups (WGs)

Content Development WG, Alex Thurman, Columbia University Library
Training WG, Lauren Baker, Library of Congress
Research WG, Olga Holownia, IIPC

18:20-18:45 Member updates

Daniel Gomes, Arquivo.pt
Mark Phillips, University of North Texas Libraries
Grace Thomas, Library of Congress
Ian Cooke, British Library
Jefferson Bailey, Internet Archive

18:45-19:00 AOB & wrap-up

Call for nominations

Election Calendar

14 June – 14 September 2021: Nomination period. IIPC Designated Representatives are invited to nominate their organisation by sending an email including a statement of up to 200 words to the IIPC SPO.

15 September 2021: Nominees statements are published on the Netpreserve Blog and Members mailing list. Nominees are encouraged to campaign through their own networks.

15 September – 15 October 2021: Members are invited to vote online. Each organisation votes only once for all nominated seats. The vote is cast by the Designated Representative.

18 October 2021: The results of the vote are announced on the Netpreserve blog and Members mailing list.

1 January 2022: The newly elected SC members start their term on 1 January.

Timeline
Proposals due: 15 September 2021
Projects start: 1 January 2022
Final report due: by 31 December 2022

• Information about the process & application forms
• Awarded projects
• Contact: projects[at]iipc.simplelists.com

ABSTRACTS – FULL SCHEDULE

CEST 

08:45-09:00 Login / Intro to Remo

09:00-09:15 Conference opening & welcome speech
Claude D. Conter, Director of the National Library of Luxembourg

09:15-10:15 Session 1: INVITED PANEL: DOCUMENTING COVID-19
Chairs: Ben Els & Olga Holownia

Panelists: Nicola Bingham, Helge Holzmann, Paul Koerbin, Gillian Lee, Ivy Lee & Yasuyo Takamine

10:15-10:30 Break / Networking

10:30-11:00 Session 2: ANALYZING ARCHIVES Q&A

* Youssef Eldakar, Mohammed Elfarargy & Ben O’Brien: LinkGate for web archive visualization
* Eld Zierau & Per Møldrup-Dalum: Making web collections for research sustainable & reusable – possibilities and challenges experienced

11:00-11:15 Break / Networking

11:15-12:00 Session 3: PANEL: SUPPORTING RESEARCH USE OF WEB ARCHIVES: A ‘LABS’ APPROACH
Chairs: Sally Chambers & Olga Holownia

Panelists: Anders Klindt Myrvoll, Katrine Hoffman Gasser, Andrew Jackson, Julie M. Birkholz, Marie Carlin, Vladimir Tybin and Tim Sherratt

12:00-12:30 Break / Networking

ABSTRACTS – FULL SCHEDULE

CEST 

14:30-14:45 Login / Intro to Remo

14:45-15:00 Opening remarks

15:00 Session 4:  TRAINING, TEACHING, LEARNING Q&A

Chair: Maria Ryan

* Abbie Grotke, Brenda Ford & Lauren Baker: Please, Come to our <online> Office Hours!”: Lessons Learned from Web Archiving Training at the Library of Congress
* Samantha Abrams: Teaching web archiving to new and established information professionals

15:30-16:00 Break / Networking

16:00-16:30 Session 5: MULTI-LINGUAL & COLLABORATIVE ARCHIVING

Chair: Sally Chambers

* Ivo Branco, Ricardo Basílio, Daniel Gomes: A transnational and cross-lingual crawl of the European Parliamentary Elections 2019

* Silvia Sevilla: Web archiving in a multilingual environment: an EU experience

16:30-16:50 Short break

16:50-17:20 Session 6: ARCHIVING COMMUNITIES

Chair: Chase Dooley

* Hélène Brousseau: Contemporary art knowledge: a community-based approach to web archiving
* Ian Milligan, Samantha Fritz, Nick Ruest & Jimmy Lin: Building community through Archives Unleashed datathons: lessons learned

17:20-17:25 Short break

17:25-18:10 Session 7: NATIONAL DOMAINS
Chair: Lori Donovan

* Jefferson Bailey: The Whole Earth Web Archive
* Kees Teszelszky: Side fûn: mapping the Frisian web domain in the Netherlands
* Moisés Rockembach: Brazilian elections web archive

18:10 Break / Networking

ABSTRACTS – FULL SCHEDULE

CEST 

20:30-21:00 Login / floor plan

21:00-21:30 Session 8: MODELING, PACKAGING, RE-USING

Chair: Kristinn Sigurðsson

* Sawood Alam, Michele Weigle, Michael Nelson, Martin Klein & Herbert Van de Sompel: Readying Web Archives to Consume and Leverage Web Bundles
* Yinlin Chen: Analyzing WARC on serverless computing working

21:30-22:45 Neworking 

21:45-22:30 Session 9: QUALITY ASSURANCE & CONTROL

Chair: Jefferson Bailey

* Brenda Reyes Ayala, James Sun, Jennifer McDevitt & Xiaohui Liu: Detecting quality problems in archived website using image similarity
* Patricia Klambauer & Tom J. Smyth: The black hole of quality control: toward a framework for managing QC effort to ensure value
* Jeffrey Van Der Hoeven, Ben O’Brien, Hanna Koppelaar, Trienka Rohrbach, Andrea Goethals & Steve Knight: Improving the quality of web harvests using Web Curator Tool

22:30-22:45 Break / Networking

22:45-23:30 Session 10: ARCHIVING FRAMEWORKS & TOOLS

Chair: Ben O’Brien

* Martin Klein & Herbert Van de Sompel: Memento Tracer – an innovative approach towards balancing web archiving at scale and quality
* Ilya Kreymer & Humbert Hardy: Not gone in a Flash! Developing a Flash-capable remote browser emulation system
* Mat Kelly: WASAPIfying private web archiving tools for persistence and collaboration

23:30 END OF DAY 1

ABSTRACTS – FULL SCHEDULE

CEST 

09:00-09:15 Login / Intro to Remo

09:15-09:30 Welcome to Day 2 of the Web Archiving Conference

09:30-10:00 Session 11: CAPTURING & UNLOCKING SOCIAL MEDIA
Chair: Anne Helmond

* Claire Newing & Phil Clegg: The UK Government Social Media Archive – now more comprehensive, more user friendly and searchable for the first time
* Eveline Vlassenroot, Friedel Geeraert, Sally Chambers, Peter Mechant, Fien Messens & Julie M. Birkholz: Unlocking web and social media archives for humanities research: a critical reflection

10:00-10:15 Break / Networking

10:15-10:45 Session 12: LEGAL ASPECTS

Chair: Nicola Bingham

* Libby Cass: The Australian Web Archive – the legal, technical, and ethical challenges
* Jason Webber: Making the most of legal deposit in the UK Web Archive

10:45-11:00 Break / Networking

11:00-12:00 Session 13: POSTERS & DROP-IN TALKS

* Haykel Boukadida, Jerome Thievre, Thomas Drugeon: The WebMedia browsing solution
* Helena Byrne: Ten years of archiving the Olympic movement
* Bethany Aylward: Capturing social movements: Web archiving needs of activist collections in Yorkshire

12:00-12:30 Break / Networking

ABSTRACTS – FULL SCHEDULE

CEST 

14:45-14:55 Login / Intro to Remo

14:55-15:00 Opening remarks / welcome to day 2

15:00-15:45 Session 14: NON-TRADITIONAL ARCHIVES Q&A

Chair: Sawood Alam

* Anne Helmond & Fernando van der Vlist: Platform and app histories: Assessing source availability in web archives and app repositories
* Jimmy Fournier: End to end integration of NFB’s web interactive works archive with WebRecorder
* Michael Kurzmeier: Website defacements: finding hacktivism in web archives

15:45-16:00 Break / Networking

16:00-16:45 Session 15: ARCHIVE PROFILING, SUMMARIZATION

Chair: Yves Maurer

* Sawood Alam, Michele Weigle, Michael Nelson, & Daniel Gomes: Summarize your Archival holdings with MementoMap
* Shawn Jones, Martin Klein, Michael Nelson & Michele Weigle: MementoEmbed and Raintale for web archive storytelling
* Maya Bangerter, Kai Jauslin & Barbara Signori: Interactive Collage of Websites: A deep dive into the Web Archive Switzerland

16:45-16:50 Short break

16:50-17:20 Session 16: RESEARCH INTO ARCHIVES Q&A
Chair: Samantha Abrams

*Katie Mackinnon: Ethical approaches to researching youth cultures in historical web archives
*Samantha Fritz: Accessible web archives: Rethinking and designing usable infrastructure for sustainable research platforms

17:20-17:40 Networking break

17:40-18:40 Session 17: POSTERS & DROP-IN TALKS

Chair: Martin Klein

* Yoo Young Lee, Marina Bokovay: Starting Small but Dreaming Big – A beginner’s journey to capture a local event through web archiving activities
* Shawn Jones, Martin Klein, Michael Nelson, Michele Weigle: Off-Topic Memento Toolkit to Identify Topical Outliers in Web Archive Collections
* Angela Beking, Russell White: Collaborate to Capture: New Approaches to Social Media Archiving at Library and Archives Canada
* Sandra Craig: Website Archiving at the Legislative Assembly of Ontario: overview, challenges, partnerships
* Rebekah Glendinning: Using Browser Developer Tools in QA: A Novice’s Toolkit

18:40-19:00 Break / Networking

Photo by Vera Davidova on Unsplash

In a world free of Covid-19, the IIPC, BnL and C2DH would have had the pleasure of welcoming all participants of the Web Archiving Conference and RESAW conference in Luxembourg.
While it wasn’t possible for us to host these events at the National Library and University of Luxembourg, we still wanted to include a joint panel in both programs, reflecting the work of Luxembourg institutions, researchers and journalists.

Covid-19 in Luxembourg:

As a follow-up to the first “Luxembourg Covid-19 collections” round table organised in September 2020, this edition focuses on born-digital collections.
After more than one year of ups and downs surrounding the pandemic, the panelists are presenting the progress of their projects in regards to:
– What is the collection’s contribution to Luxembourg born-digital heritage?
– What was the starting point and scope of the collection?
– What were the successes and limits along the way?
– By what means can we promote and maintain these collections and make their contents more accessible?

The panel:

Frédéric Clavert – C2DH, University of Luxembourg
Ben Els – National Library of Luxembourg
Marie-Laure Rolland – Reporter.lu
Silvia Sevilla – Publications Office of the EU
Tizian Zumthurm – C2DH, University of Luxembourg

Chaired by Yves Maurer – National Library of Luxembourg

As a joint  event between the IIPC Web Archiving Conference and the 4th RESAW conference, this session is free of charge and will be streamed live via Webex.

Covidmemory.lu by C2DH

ABSTRACTS – FULL SCHEDULE

CEST 

20:50-21:00 Login / floor plan

21:00-21:45 Session 18: ACCESS, PROMOTION, POLICY

Chair: Abbie Grotke

* Ben Els: Drawing attention with special collections
* Nicola Bingham: Policies and processes for ingesting WebRecorder WARCS into the UK Web Archive
* Jefferson Bailey: Perpetual access to open scholarship through web archiving

21:45-22:00 Break / Networking

22:00 Session 19: INVITED PANEL: TRUST IN WEB ARCHIVES
Chairs: Martin Klein & Yves Maurer

Panelists: Ada Lerner, Michael L. Nelson, Clare Stanton & Takuya Watanabe

23:00-23:15 CLOSING REMARKS: Abbie Grotke, IIPC Chair