| java.lang.Object
 org.archive.crawler.datamodel.RobotsExclusionPolicy
| RobotsExclusionPolicy | | public class RobotsExclusionPolicy implements Serializable(Code) | | RobotsExclusionPolicy represents the actual policy adopted with
respect to a specific remote server, usually constructed from
consulting the robots.txt, if any, the server provided.
(The similarly named RobotsHonoringPolicy, on the other hand,
describes the strategy used by the crawler to determine to what
extent it respects exclusion rules.)
The expiration of policies after a suitable amount of time has
elapsed since last fetch is handled outside this class, in
CrawlServer itself.
author:
gojomo |
| RobotsExclusionPolicy | | public RobotsExclusionPolicy(int type)(Code) | | |
|
|