No Description
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
mark f beasley 51ff3adab2 Announce list: fix variable usage in models, send truthy value from frontend (#663) 2 weeks ago
frontend Announce list: fix variable usage in models, send truthy value from frontend (#663) 2 weeks ago
nginx changes from prod: 7 months ago
redis use /usr/bin/env in shebangs (#403) 1 year ago
webrecorder Announce list: fix variable usage in models, send truthy value from frontend (#663) 2 weeks ago
.gitignore Collections overview redesign 4 months ago
.gitmodules start using public-web-archives repo: (#363) 1 year ago
.travis.yml Build: Travis OSX update (#661) 2 weeks ago
LICENSE add Apache LICENSE and NOTICE 3 years ago
NOTICE add Apache LICENSE and NOTICE 3 years ago
README.md README: readme edits from @despens 3 months ago
appveyor.yml Cookie Check: Relax Refer Check (#632) 3 months ago
docker-compose.yml Remove redis port bindings for improved security (was: bind Redis to loopback interface to avoid open Redis port on host machine (#653)) 2 weeks ago
init-default.sh use /usr/bin/env in shebangs (#403) 1 year ago
install-browsers.sh README and Doc Fixes (#599) 4 months ago
migrating-4.0.md Updated README and added migration instructions (#598) 4 months ago
recreate.sh migration script fixes: (#597) 4 months ago

README.md

Webrecorder Project

Web archiving for All!

Webrecorder provides an integrated platform for creating high-fidelity, ISO-compliant web archives in a user-friendly interface, providing access to archived content, and sharing collections.

This repository represents the hosted service running on https://webrecorder.io/, which can also be deployed locally using Docker

This README refers to the 4.x version of Webrecorder, released in June, 2018.

The release included a significant UX redesign, including new curatorial features, and an architecture redesign splitting the project into an API-based backend and a React-powered frontend architecture.

Frequently asked questions

  • If you have any questions about how to use Webrecorder, please see our User Guide.

  • If you have a question about your account on the hosted service (webrecorder.io), please contact us via email at support@webrecorder.io

  • If you have a previous Webrecorder installation (version 3.x), see Migration Info for instructions on how to migrate to the latest version.

Using the Webrecorder Platform

Webrecorder and related tools are designed to make web archiving more portable and decentralized, as well as to serve users and developers with a broad range of skill levels and requirements. Here are a few ways that Webrecorder can be used (starting with what probably requires the least technical expertise).

1. Hosted Service

Using our hosted version of Webrecorder at https://webrecorder.io/, users can sign up for a free account and create their own personal collections of web archives. Captures web content will be available online, either publicly or only privately, under each user account, and can be downloaded by the account owner at any time. Downloaded web archives are available as WARC files. (WARC is the ISO standard file format for web archives.) The hosted service can also be used anonymously and the captured content can be downloaded at the end of a temporary session.

2. Offline Browsing

We also provide a OSX/Windows/Linux Electron application, Webrecorder Player that can browse WARC created by Webrecorder (and other web archiving tools) locally on the desktop.

3. Preconfigured Deployment

To deploy the full version of Webrecorder with Ansible on a Linux machine, the Webrecorder Deploy workbook can be used to install this repository, configure nginx and other dependencies, such as SSL (via Lets Encrypt). The workbook is used for the https://webrecorder.io/ deployment.

4. Full Webrecorder Local Deployment

The Webrecorder system in this repository can be deployed directly by following the instructions below. Webrecorder runs entirely in Docker and also requires Docker Compose.

5. Standalone Python Wayback (pywb) Deployment

Finally, for users interested in the core “replay system” and very basic recording capabilities, deploying pywb could also make sense. Webrecorder is built on top of pywb (Python Wayback/Python Web Archive Toolkit), and the core recording and replay functionality is provided by pywb as a standalone Python library. pywb comes with a Docker image as well.

pywb can be used to deploy your own web archive access service. See the full pywb reference manual for further information on using and deploying pywb.

Running Locally

Webrecorder can be run on any system that has Docker and Docker Compose installed. To install manually, clone

  1. git clone https://github.com/webrecorder/webrecorder

  2. cd webrecorder; bash init-default.sh.

  3. docker-compose build

  4. docker-compose up -d

(The init-default.sh is a convenience script that copies wr_sample.envwr.env and creates keys for session encryption.)

Point your browser to http://localhost:8089/ to access the locally running Webrecorder instance.

(Note: you may see a maintenance message briefly while Webrecorder is starting up for the first time. Refresh the page after a few seconds to see the Webrecorder home page).

Installing Remote Browsers

Remote Browsers are standard browsers like Google Chrome and Mozilla Firefox, encapsulated in Docker containers. This feature allows Webrecorder to directly use fixed versions of browsers for capturing and accessing web archives, with a more direct connection to the live web and web archives. Remote browsers in many cases can improve the quality of web archives during capture and access. They can be “remote controlled” by users and are launched as needed, and use the same amount of computing and memory resources as they would when just running as regular desktop apps.

Remote Browsers are optional, and can be installed as needed.

Remote Browsers are just Docker images which start with oldweb-today/, and are part of oldweb-today organization on GitHub. Installing the browsers can be as simple as running docker pull on each image.

To install all of the officially supported browsers, run install-browsers.sh

For example, to install a version of Chrome 60 and Firefox 49, run:

docker pull oldwebtoday/chrome:60
docker pull oldwebtoday/firefox:49

Configuration

Webrecorder reads its configuration from two files: wr.env, and less-commonly changed system settings in wr.yaml.

The wr.env file contains numerous deployment-specific customization options. In particular, the following options may be useful:

Host Names

By default, Webrecorder assumes its running on localhost or a single domain, but on different ports for application (the Webrecorder user interface) and content (material rendered from web archives). This is a security feature preventing archived web sites accessing and possibly changing Webrecorder’s user interface, and other unwanted interactions.

To run Webrecorder on different domains, the APP_HOST and CONTENT_HOST environment variables should be set.

For best results, the two domains should be two subdomains, both with https enabled.

The SCHEME env var should also be set to SCHEME=https when deploying via https.

Storage

Webrecorder uses the ./data/ directory for local storage, or an external backend, currently supporting S3.

The DEFAULT_STORAGE option in wr.env configures storage options, which can be DEFAULT_STORAGE=local or DEFAULT_STORAGE=s3

Webrecorder uses a temporary storage directory for data while it is actively being captured, and temporary collections. Data is moved into the ‘permanent’ storage when the capturing process is completed or a temporary collection is imported into a user account.

The temporary storage directory is: WARCS_DIR=./data/warcs.

The permanent storage directory is either STORAGE_DIR=./data/storage or local storage.

When using s3, the value of STORAGE_DIR is ignored and data gets placed into S3_ROOT which is an s3:// bucket URL.

Additional s3 auth environment settings must also be set in wr.env or externally.

All data related to Webrecorder that is not web archive data (WARC and CDXJ) is stored in the Redis instancem, which persists data to ./data/dump.rdb. (See Webrecorder Architecture below.)

Email

Webrecorder can send confirmation and password recovery emails. By default, a local SMTP server is run in Docker, but can be configured to use a remote server by changing the environment variables EMAIL_SMTP_URL and EMAIL_SMTP_SENDER.

Frontend Options

The react frontend includes a number of additional options useful for debugging. Setting NODE_ENV=development will switch react to development mode with hot reloading on port 8096.

Administration tool

The script admin.py provides easy low level management of users. Adding, modifying, or removing users can be done via the command line.

To interactively create a user:

docker exec -it webrecorder_app_1 python -m webrecorder.admin -c

or programmatically add users by supplying the appropriate positional values:

docker exec -it webrecorder_app_1  python -m webrecorder.admin \
                -c <email> <username> <passwd> <role> '<full name>'

Other arguments:

  • -m modify a user
  • -d delete a user
  • -i create and send a new invite
  • -l list invited users
  • -b send backlogged invites

See docker exec -it webrecorder_app_1 python -m webrecorder.admin --help for full details.

Restarting Webrecorder

When making changes to the Webrecorder backend app, running

docker-compose kill app; docker-compose up -d app

will stop and restart the container.

To integrate changes to the frontend app, either set NODE_ENV=development and utilize hot reloading. If you’re running production (NODE_ENV=production), run

docker-compose kill frontend; docker-compose up -d frontend

To fully recreate Webrecorder, deleting old containers (but not the data!) use the ./recreate.sh script.

Webrecorder Architecture

This repository contains the Docker Compose setup for Webrecorder, and is the exact system deployed on https://webrecorder.io/. The full setup consists of the following components:

  • /app - The Webrecorder Backend system includes the API, recording and WARC access layers, split into 3 containers:
    • app -- The API and data model and rewriting system are found in this container.
    • recorder -- The WARC writer is found in this container.
    • warcserver -- The WARC loading and lookup is found in this container.

The backend containers run different tools from pywb, the core web archive replay toolkit library.

  • /frontend - A React-based frontend application, running in Node.js. The frontend is a modern interface for Webrecorder and uses the backend api. All user access goes through frontend (after nginx).

  • /nginx - A custom nginx deployment to provide routing and caching.

  • redis - A Redis instance that stores all of the Webrecorder state (other than WARC and CDXJ).

  • dat-share - An experimental component for sharing collections via the Dat protocol

  • shepherd - An instance of OldWebToday Browser Shepherd for managing remote browsers.

  • mailserver - A simple SMTP mail server for sending user account management mail

Dependencies

Webrecorder is built using both Python (for backend) and Node.js (for frontend) using a variety of Python and Node open source libraries.

Webrecorder relies on a few separate repositories in this organization:

The remote browser system uses https://github.com/oldweb-today/ repositories, including:

Contact

Webrecorder is a project of Rhizome, developed with generous support from the Andrew W. Mellon Foundation.

For more info on using Webrecorder, you can consult our user guide at: https://guide.webrecorder.io/

For any general questions/concerns regarding the project or https://webrecorder.io/ you can:

License

Webrecorder is Licensed under the Apache 2.0 License. See NOTICE and LICENSE for details.