Using the Archives Unleashed Cloud Derivative Files

One of the main features of the Archives Unleashed Cloud is the creation of derivative files. These are datasets that provide information about your web archives and can be used by researchers in place of the original WARC files.

Once analytics have been run on a collection, as discussed in the documentation, you will be able to view and access a series of four derivative files.

Gephi

Network

Domains

Full Text

Learning Guides

The following guides walk through how to use Cloud derivative files and the Archives Unleashed Cloud Notebooks for further analysis.

We are excited to introduce our Archives Unleashed Jupyter Notebooks, a prototype method for working with the derivatives generated by the Archives Unleashed Cloud. They allow you to interactively explore and filter the domain count information, extracted full text, and network visualization data generated by the Cloud.

Tutorial

Getting Started with the Archives Unleashed Cloud Jupyter Notebooks: This tutorial provides a basic walkthrough for integrating data from the Archives Unleashed Cloud with our Cloud Notebooks.

When using the web as a historical source, the ability to see the way websites link to each other can be invaluable. However, using network analysis in historical research can also be a daunting prospect.

Tutorial	An Introduction to Gephi: This tutorial provides a basic "click by click" approach to using Gephi. Network Graphing Archived Websites With Gephi: This tutorial provides an intermediate-level exploration of using Gephi to answer historical questions with a web archive file.
File Output	`.gexf` and `.graphml`
Files Generated	Gephi file, which can be loaded into Gephi. It will have basic characteristics already computed and a basic layout. Raw Network file, which can also be loaded into Gephi. You will have to use that network program to lay it out yourself.

The full text of a web collection can be invaluable for a researcher! Archives Unleashed Cloud users will have access to the full text file. In it, each website within the web archive collection will have its full text presented on one line, along with information around when it was crawled, the name of the domain, and the full URL of the content.

Tutorial	Filtering the Full-Text Derivative File: This tutorial explores the use of the "grep" command line tool to filter out dates, domains, and keywords from plain text. Text Analysis Part One: Beyond the Keyword Search: Using AntConc: This tutorial explores how you can explore text within a web archive using the AntConc tool. Text Analysis Part Two: Sentiment Analysis With the Natural Language Toolkit: This tutorial explores how you can calculate the positivity or negativity (in an emotional sense) of web archive text.
File Output	`.txt`
Files Generated	A text file is generated which can be used in various other programs and software.
Additional Resources	Working with lots of text can be challenging. Fortunately, there are lots of resources on the web to help teach you the basics of text analysis. Programming Historian: The Programming Historian has many lessons, including a specific section on distant reading. Text Analysis Portal for Research: TAPoR, or the Text Analysis Portal for Research, provides a curated collection of tools that can be used to explore text. Digital Humanities Guide at University of Ottawa: This is a great guide that highlights several approaches to text analysis in the field. Voyant Tools: This is more of a tool than an introductory resource, but it's a gentle yet powerful introduction to the powers of text analysis.

Sometimes you might just want to know what is inside a web archive! The web is a complex beast: a small web archive that aims to collect just one page might end up capturing embedded widgets and content from other websites just to make something play. Alternatively, a crawl might have been corrupted and begun to find material that's far out of scope.

Tutorial	Making Sense of the Domains Count
File Output	`.txt`
Files Generated	A text file containing the frequency count of domains captured within your web archive.

How to Cite Archives Unleashed

If you've found the Archives Unleashed Toolkit or Cloud helpful in your research, please consider using the citation below in your publications. Your citations help to further the recognition of using open-source tools for scientific inquiry, assists in growing the web archiving community, and acknowledges the efforts of contributors to this project.

Archives Unleashed Project. (2019). Archives Unleashed Toolkit (Version 0.17.0). Apache License, Version 2.0.

Using the Archives Unleashed Cloud Derivative Files

Gephi

Network

Domains

Full Text

Learning Guides

Archives Unleashed Cloud Notebooks

Gephi & Raw Network Files

Plain Text Files

Domains File

How to Cite Archives Unleashed