Heritrix

Heritrix is a web crawler designed for web archiving. It was written by the Internet Archive. It is free software license and written in Java. The main interface is accessible using a web browser, and there is a command-line tool that can optionally be used to initiate crawls.