| java.lang.Object com.torunski.crawler.core.AbstractCrawler com.torunski.crawler.MultiThreadedCrawler
MultiThreadedCrawler | public class MultiThreadedCrawler extends AbstractCrawler (Code) | | Project: Smart & Simple Web Crawler
Crawls through the web with a lot of threads.
author: Lars Torunski version: $Revision: 1.8 $ |
Constructor Summary | |
public | MultiThreadedCrawler() Default constructor for 5 loading threads and 2 parsing threads. | public | MultiThreadedCrawler(int maxLoadThreads, int maxParseThreads) Creates a multi threaded crawler which delegates the load and parse tasks to different threads. |
Method Summary | |
public void | start(String server, String start) Starts the crawling process in a multi threaded enviroment. | public void | start() Starts the crawling process in a multi threaded enviroment. |
MultiThreadedCrawler | public MultiThreadedCrawler()(Code) | | Default constructor for 5 loading threads and 2 parsing threads.
|
MultiThreadedCrawler | public MultiThreadedCrawler(int maxLoadThreads, int maxParseThreads)(Code) | | Creates a multi threaded crawler which delegates the load and parse tasks to different threads.
Per RFC 2616 sec 8.1.4 the maximum number of connections allowed per host is 2.
See Also: org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.DEFAULT_MAX_TOTAL_CONNECTIONS Parameters: maxLoadThreads - maximum of threads for loading the content Parameters: maxParseThreads - maximum of threads for parsing the download |
start | public void start(String server, String start)(Code) | | Starts the crawling process in a multi threaded enviroment.
|
|
|