| org.archive.crawler.framework.Processor
 org.archive.crawler.fetcher.FetchHTTP
| Inner Class :class PostRestore implements Runnable | |
| Method Summary | |
| protected void | addResponseContent(HttpMethod method, CrawlURI curi)
This method populates curi with response status and
content type. | | protected boolean | checkMidfetchAbort(CrawlURI curi, HttpRecorderMethod method, HttpConnection conn)
| | protected void | cleanupHttp()
Perform any final cleanup related to the HttpClient instance. | | protected void | configureHttp()
| | protected HostConfiguration | configureMethod(CrawlURI curi, HttpMethod method)
Configure the HttpMethod setting options and headers. | | public void | crawlCheckpoint(File checkpointDir)
| | public void | crawlEnded(String sExitMessage)
| | public void | crawlEnding(String sExitMessage)
| | public void | crawlPaused(String statusMessage)
| | public void | crawlPausing(String statusMessage)
| | public void | crawlResuming(String statusMessage)
| | public void | crawlStarted(String message)
| | protected void | doAbort(CrawlURI curi, HttpMethod method, String annotation)
| | public void | finalTasks()
| | protected Object | getAttributeEither(CrawlURI curi, String key)
Get a value either from inside the CrawlURI instance, or from
settings (module attributes). | | protected AuthScheme | getAuthScheme(HttpMethod method, CrawlURI curi)
Parameters:
method - Method that got a 401.
Parameters:
curi - CrawlURI that got a 401. | | protected HttpClient | getHttp()
| | protected DecideRule | getMidfetchRule(Object o)
| | protected void | handle401(HttpMethod method, CrawlURI curi)
Server is looking for basic/digest auth credentials (RFC2617). | | public void | initialTasks()
| | protected void | innerProcess(CrawlURI curi)
| | protected void | listUsedFiles(List<String> list)
| | public void | loadCookies(String cookiesFile)
Load cookies from a file before the first fetch.
The file is a text file in the Netscape's 'cookies.txt' file format.
Example entry of cookies.txt file:
www.archive.org FALSE / FALSE 1074567117 details-visit texts-cralond
Each line has 7 tab-separated fields:
1. | | public void | loadCookies()
Load cookies from the file specified in the order file.
The file is a text file in the Netscape's 'cookies.txt' file format.
Example entry of cookies.txt file:
www.archive.org FALSE / FALSE 1074567117 details-visit texts-cralond
Each line has 7 tab-separated fields:
1. | | public String | report()
| | public void | saveCookies()
Saves cookies to the file specified in the order file. | | public void | saveCookies(String saveCookiesFile)
Saves cookies to a file. | | protected void | setConditionalGetHeader(CrawlURI curi, HttpMethod method, String setting, String sourceHeader, String targetHeader)
Set the given conditional-GET header, if the setting is enabled and
a suitable value is available in the URI history. | | protected void | setSizes(CrawlURI curi, HttpRecorder rec)
|
| ATTR_ACCEPT_HEADERS | | final public static String ATTR_ACCEPT_HEADERS(Code) | | |
| ATTR_BDB_COOKIES | | final public static String ATTR_BDB_COOKIES(Code) | | |
| ATTR_DEFAULT_ENCODING | | final public static String ATTR_DEFAULT_ENCODING(Code) | | |
| ATTR_DIGEST_ALGORITHM | | final public static String ATTR_DIGEST_ALGORITHM(Code) | | |
| ATTR_DIGEST_CONTENT | | final public static String ATTR_DIGEST_CONTENT(Code) | | |
| ATTR_FETCH_BANDWIDTH_MAX | | final public static String ATTR_FETCH_BANDWIDTH_MAX(Code) | | |
| ATTR_HTTP_PROXY_HOST | | final public static String ATTR_HTTP_PROXY_HOST(Code) | | |
| ATTR_HTTP_PROXY_PORT | | final public static String ATTR_HTTP_PROXY_PORT(Code) | | |
| ATTR_IGNORE_COOKIES | | final public static String ATTR_IGNORE_COOKIES(Code) | | |
| ATTR_LOAD_COOKIES | | final public static String ATTR_LOAD_COOKIES(Code) | | |
| ATTR_LOCAL_ADDRESS | | final public static String ATTR_LOCAL_ADDRESS(Code) | | |
| ATTR_MAX_LENGTH_BYTES | | final public static String ATTR_MAX_LENGTH_BYTES(Code) | | |
| ATTR_MIDFETCH_DECIDE_RULES | | final public static String ATTR_MIDFETCH_DECIDE_RULES(Code) | | Rules to apply mid-fetch, just after receipt of the response
headers before we start to download body.
|
| ATTR_SAVE_COOKIES | | final public static String ATTR_SAVE_COOKIES(Code) | | |
| ATTR_SEND_CONNECTION_CLOSE | | final public static String ATTR_SEND_CONNECTION_CLOSE(Code) | | |
| ATTR_SEND_IF_MODIFIED_SINCE | | final public static String ATTR_SEND_IF_MODIFIED_SINCE(Code) | | |
| ATTR_SEND_IF_NONE_MATCH | | final public static String ATTR_SEND_IF_NONE_MATCH(Code) | | |
| ATTR_SEND_RANGE | | final public static String ATTR_SEND_RANGE(Code) | | |
| ATTR_SEND_REFERER | | final public static String ATTR_SEND_REFERER(Code) | | |
| ATTR_SOTIMEOUT_MS | | final public static String ATTR_SOTIMEOUT_MS(Code) | | |
| ATTR_TIMEOUT_SECONDS | | final public static String ATTR_TIMEOUT_SECONDS(Code) | | |
| ATTR_TRUST | | final public static String ATTR_TRUST(Code) | | SSL trust level setting attribute name.
|
| COOKIEDB_NAME | | final public static String COOKIEDB_NAME(Code) | | Name of cookie BDB Database
|
| DEFAULT_DIGEST_ALGORITHM | | final public static String DEFAULT_DIGEST_ALGORITHM(Code) | | Default algorithm to use for message disgesting.
|
| DEFAULT_DIGEST_CONTENT | | static Boolean DEFAULT_DIGEST_CONTENT(Code) | | Default whether to perform on-the-fly digest hashing of content-bodies.
|
| DIGEST_ALGORITHMS | | public static String[] DIGEST_ALGORITHMS(Code) | | |
| SHA1 | | final public static String SHA1(Code) | | The different digest algorithms to choose between,
SHA-1 or MD-5 at the moment.
|
| cookieDb | | protected Database cookieDb(Code) | | Database backing cookie map, if using BDB
|
| FetchHTTP | | public FetchHTTP(String name)(Code) | | Constructor.
Parameters:
name - Name of this processor. |
| addResponseContent | | protected void addResponseContent(HttpMethod method, CrawlURI curi)(Code) | | This method populates curi with response status and
content type.
Parameters:
curi - CrawlURI to populate.
Parameters:
method - Method to get response status and headers from. |
| cleanupHttp | | protected void cleanupHttp()(Code) | | Perform any final cleanup related to the HttpClient instance.
|
| configureMethod | | protected HostConfiguration configureMethod(CrawlURI curi, HttpMethod method)(Code) | | Configure the HttpMethod setting options and headers.
Parameters:
curi - CrawlURI from which we pull configuration.
Parameters:
method - The Method to configure. |
| crawlCheckpoint | | public void crawlCheckpoint(File checkpointDir)(Code) | | |
| crawlEnding | | public void crawlEnding(String sExitMessage)(Code) | | |
| crawlPaused | | public void crawlPaused(String statusMessage)(Code) | | |
| crawlPausing | | public void crawlPausing(String statusMessage)(Code) | | |
| crawlResuming | | public void crawlResuming(String statusMessage)(Code) | | |
| finalTasks | | public void finalTasks()(Code) | | |
| getAttributeEither | | protected Object getAttributeEither(CrawlURI curi, String key)(Code) | | Get a value either from inside the CrawlURI instance, or from
settings (module attributes).
Parameters:
curi - CrawlURI to consult
Parameters:
key - key to lookup value from either CrawlURI (preferred) or settings |
| getAuthScheme | | protected AuthScheme getAuthScheme(HttpMethod method, CrawlURI curi)(Code) | |
Parameters:
method - Method that got a 401.
Parameters:
curi - CrawlURI that got a 401. Returns first wholesome authscheme found else null. |
| getHttp | | protected HttpClient getHttp()(Code) | | Returns the http instance. |
| handle401 | | protected void handle401(HttpMethod method, CrawlURI curi)(Code) | | Server is looking for basic/digest auth credentials (RFC2617). If we have
any, put them into the CrawlURI and have it come around again. Presence
of the credential serves as flag to frontier to requeue promptly. If we
already tried this domain and still got a 401, then our credentials are
bad. Remove them and let this curi die.
Parameters:
method - Method that got a 401.
Parameters:
curi - CrawlURI that got a 401. |
| initialTasks | | public void initialTasks()(Code) | | |
| loadCookies | | public void loadCookies(String cookiesFile)(Code) | | Load cookies from a file before the first fetch.
The file is a text file in the Netscape's 'cookies.txt' file format.
Example entry of cookies.txt file:
www.archive.org FALSE / FALSE 1074567117 details-visit texts-cralond
Each line has 7 tab-separated fields:
1. DOMAIN: The domain that created and have access to the cookie
value.
2. FLAG: A TRUE or FALSE value indicating if hosts within the given
domain can access the cookie value.
3. PATH: The path within the domain that the cookie value is valid
for.
4. SECURE: A TRUE or FALSE value indicating if to use a secure
connection to access the cookie value.
5. EXPIRATION: The expiration time of the cookie value (unix style.)
6. NAME: The name of the cookie value
7. VALUE: The cookie value
Parameters:
cookiesFile - file in the Netscape's 'cookies.txt' format. |
| loadCookies | | public void loadCookies()(Code) | | Load cookies from the file specified in the order file.
The file is a text file in the Netscape's 'cookies.txt' file format.
Example entry of cookies.txt file:
www.archive.org FALSE / FALSE 1074567117 details-visit texts-cralond
Each line has 7 tab-separated fields:
1. DOMAIN: The domain that created and have access to the cookie
value.
2. FLAG: A TRUE or FALSE value indicating if hosts within the given
domain can access the cookie value.
3. PATH: The path within the domain that the cookie value is valid
for.
4. SECURE: A TRUE or FALSE value indicating if to use a secure
connection to access the cookie value.
5. EXPIRATION: The expiration time of the cookie value (unix style.)
6. NAME: The name of the cookie value
7. VALUE: The cookie value
|
| saveCookies | | public void saveCookies()(Code) | | Saves cookies to the file specified in the order file.
Output file is in the Netscape 'cookies.txt' format.
|
| saveCookies | | public void saveCookies(String saveCookiesFile)(Code) | | Saves cookies to a file.
Output file is in the Netscape 'cookies.txt' format.
Parameters:
saveCookiesFile - output file. |
| setConditionalGetHeader | | protected void setConditionalGetHeader(CrawlURI curi, HttpMethod method, String setting, String sourceHeader, String targetHeader)(Code) | | Set the given conditional-GET header, if the setting is enabled and
a suitable value is available in the URI history.
Parameters:
curi - source CrawlURI
Parameters:
method - HTTP operation pending
Parameters:
setting - true/false enablement setting name to consult
Parameters:
sourceHeader - header to consult in URI history
Parameters:
targetHeader - header to set if possible |
| setSizes | | protected void setSizes(CrawlURI curi, HttpRecorder rec)(Code) | | Update CrawlURI internal sizes based on current transaction (and
in the case of 304s, history)
Parameters:
curi - CrawlURI
Parameters:
rec - HttpRecorder |
|
|