Your web archives are your everything archives 14:00-14:30

Jefferson Bailey, Internet Archive

As the web continues to consume all media — from publishing to video to music — large-scale web harvests are collecting a rich corpus of news, government publications, creative works, scholarly research, and other materials that formerly had their own defined dissemination and consumption frameworks. Large-scale harvesting efforts, however, are intentionally designed for breadth, scale, speed, and “content agnostic” collecting methods that treat different types of works similarly. Relatedly, the URL-centric nature of web collecting and access can limit how we curate and enable discovery of the materials within our web archives. Web archives themselves, as they grow, are evermore an agglomeration of diverse material whose only shared trait is publication via the web. How can we increase our knowledge of what is contained within the web collections we are building? And how can we gain a better understanding of what we have collected in order to inform improved description, curation, access and collection strategies?

This presentation will detail a number of both in-production and research and development projects by Internet Archive and international partners aimed at building strategies, tools, and systems for identifying, improving, and enhancing discovery of specific collections within large-scale web collections. It will outline new methods to situate and enhance the valuable content already collected in general web collections and implement automated systems to ensure future materials are well-collected and, when possible, are associated with the appropriate context and metadata. This work spans search, data mining, identifier association, integration with publishers, registries, and creator communities, machine learning, and technology and partnership development. The talk will outline potential approaches for moving from a concept of “archives of the web” to one of “archives from the web.”