| A step, late in the processing of a CrawlURI, for updating the per-host
information that may have been affected by the fetch. This will initially
be robots and ip address info; it could include other per-host stats that
would affect the crawl (like total pages visited at the site) as well.
author: gojomo version: $Date: 2006-09-25 20:19:54 +0000 (Mon, 25 Sep 2006) $, $Revision: 4654 $ |