{"id":1409,"date":"2019-05-28T11:42:55","date_gmt":"2019-05-28T11:42:55","guid":{"rendered":"http:\/\/netpreserve.org\/ga2019\/?post_type=mp-event&#038;p=1409"},"modified":"2019-06-03T06:15:21","modified_gmt":"2019-06-03T06:15:21","slug":"workshop-julien-nioche-introduction-to-web-crawling-with-stormcrawler","status":"publish","type":"mp-event","link":"https:\/\/netpreserve.org\/ga2019\/timetable\/event\/workshop-julien-nioche-introduction-to-web-crawling-with-stormcrawler\/","title":{"rendered":"Workshop: Introduction to web crawling with StormCrawler"},"content":{"rendered":"\n<h4 class=\"wp-block-heading\">JULIEN NIOCHE<\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><em>CameraForensics<\/em><\/p>\n\n\n\n<h5 class=\"wp-block-heading\">Introduction to web crawling with StormCrawler (and Elasticsearch)<\/h5>\n\n\n\n<p class=\"wp-block-paragraph\">In this workshop, we will explore StormCrawler a collection of resources for building low-latency, large scale web crawlers on Apache Storm. After a short introduction to Apache Storm and an overview of what Storm-Crawler provides, we\u2019ll put it to use for a simple crawl before moving on to the deployed mode of Storm.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the second part of the session, we will introduce metrics and index documents with Elasticsearch and Kibana and dive into data extraction. Finally, we\u2019ll cover recursive crawls and scalability. This course will be hands-on: attendees will run the code on their own machines.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Agenda<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We will cover the following topics:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Introduction to web crawling<\/li><li>Apache Storm: architecture and concepts<\/li><li>StormCrawler: basic building blocks<\/li><li>How to use the archetype<\/li><li>Building &amp; configuring<\/li><li>URLFilters, ParseFilters<\/li><li>Simple recursive crawls<\/li><li>How to debug?<\/li><li>Distributed mode: UI, logs, metrics<\/li><li>Elasticsearch resources<\/li><li>WARC module<\/li><li>Q&amp;As<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Audience<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This course will suit Java developers with an interest in big data, stream processing, web crawling and archiving. It will provide a practical introduction to both Apache Storm and Elasticsearch as well of course as StormCrawler and will not require advanced programming skills.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Prerequisites<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Attendees should bring their own laptop with Apache Maven and Java 8 or above installed. The examples and instructions will be conducted on a Linux distribution and using Eclipe IDE. Ideally, students should look at the Apache Storm and StormCrawler documentation and think about particular websites or crawl scenarios that they might be interested in.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>JULIEN NIOCHE CameraForensics Introduction to web crawling with StormCrawler (and Elasticsearch) In this workshop, we will explore StormCrawler a collection of resources for building low-latency, large scale web crawlers on Apache Storm. After a short introduction to Apache Storm and<\/p>\n","protected":false},"author":5,"featured_media":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","mp-event_category":[13],"mp-event_tag":[],"class_list":["post-1409","mp-event","type-mp-event","status-publish","hentry","mp-event_category-thursday","mp-event-item"],"_links":{"self":[{"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/mp-event\/1409","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/mp-event"}],"about":[{"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/types\/mp-event"}],"author":[{"embeddable":true,"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/comments?post=1409"}],"wp:attachment":[{"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/media?parent=1409"}],"wp:term":[{"taxonomy":"mp-event_category","embeddable":true,"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/mp-event_category?post=1409"},{"taxonomy":"mp-event_tag","embeddable":true,"href":"https:\/\/netpreserve.org\/ga2019\/wp-json\/wp\/v2\/mp-event_tag?post=1409"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}