websphinx

The entire set of pages is preceded by a "prolog" and followed by an "epilog", which are constant strings of HTML.

To write a crawler, extend this class and override shouldVisit () and visit() to create your own crawler.

To use a crawler:

1. Initialize the crawler by calling setRoot() (or one of its variants) and setting other crawl parameters.
2. Register any classifiers you need with addClassifier().
3. Connect event listeners to monitor the crawler, such as websphinx.EventLog, websphinx.workbench.WebGraph, or websphinx.workbench.Statistics.
4. Call run() to start the crawler.

The default LinkTransformer simply converts all links to absolute URLs.

On the entire page, this classifier sets the following labels:

• root: page is the root page of a Web site.