Java Doc for WriterPoolProcessor.java in  » Web-Crawler » heritrix » org » archive » crawler » framework » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation 1.  6.0 JDK Core 2.  6.0 JDK Modules 3.  6.0 JDK Modules com.sun 4.  6.0 JDK Modules com.sun.java 5.  6.0 JDK Modules sun 6.  6.0 JDK Platform 7.  Ajax 8.  Apache Harmony Java SE 9.  Aspect oriented 10.  Authentication Authorization 11.  Blogger System 12.  Build 13.  Byte Code 14.  Cache 15.  Chart 16.  Chat 17.  Code Analyzer 18.  Collaboration 19.  Content Management System 20.  Database Client 21.  Database DBMS 22.  Database JDBC Connection Pool 23.  Database ORM 24.  Development 25.  EJB Server geronimo 26.  EJB Server GlassFish 27.  EJB Server JBoss 4.2.1 28.  EJB Server resin 3.1.5 29.  ERP CRM Financial 30.  ESB 31.  Forum 32.  GIS 33.  Graphic Library 34.  Groupware 35.  HTML Parser 36.  IDE 37.  IDE Eclipse 38.  IDE Netbeans 39.  Installer 40.  Internationalization Localization 41.  Inversion of Control 42.  Issue Tracking 43.  J2EE 44.  JBoss 45.  JMS 46.  JMX 47.  Library 48.  Mail Clients 49.  Net 50.  Parser 51.  PDF 52.  Portal 53.  Profiler 54.  Project Management 55.  Report 56.  RSS RDF 57.  Rule Engine 58.  Science 59.  Scripting 60.  Search Engine 61.  Security 62.  Sevlet Container 63.  Source Control 64.  Swing Library 65.  Template Engine 66.  Test Coverage 67.  Testing 68.  UML 69.  Web Crawler 70.  Web Framework 71.  Web Mail 72.  Web Server 73.  Web Services 74.  Web Services apache cxf 2.0.1 75.  Web Services AXIS2 76.  Wiki Engine 77.  Workflow Engines 78.  XML 79.  XML UI Java Java Tutorial Java Open Source Jar File Download Java Articles Java Products Java by API Photoshop Tutorials Maya Tutorials Flash Tutorials 3ds-Max Tutorials Illustrator Tutorials GIMP Tutorials C# / C Sharp C# / CSharp Tutorial C# / CSharp Open Source ASP.Net ASP.NET Tutorial JavaScript DHTML JavaScript Tutorial JavaScript Reference HTML / CSS HTML CSS Reference C / ANSI-C C Tutorial C++ C++ Tutorial Ruby PHP Python Python Tutorial Python Open Source SQL Server / T-SQL SQL Server / T-SQL Tutorial Oracle PL / SQL Oracle PL/SQL Tutorial PostgreSQL SQL / MySQL MySQL Tutorial VB.Net VB.Net Tutorial Flash / Flex / ActionScript VBA / Excel / Access / Word XML XML Tutorial Microsoft Office PowerPoint 2007 Tutorial Microsoft Office Excel 2007 Tutorial Microsoft Office Word 2007 Tutorial

Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.framework   Source Cross Reference   Class Diagram   Java Document (Java Doc)   org.archive.crawler.framework.Processor    org.archive.crawler.framework.WriterPoolProcessor All known Subclasses:   org.archive.crawler.writer.ExperimentalWARCWriterProcessor,  org.archive.crawler.writer.ExperimentalV10WARCWriterProcessor,  org.archive.crawler.writer.ARCWriterProcessor, WriterPoolProcessor abstract public class WriterPoolProcessor extends Processor implements CoreAttributeConstants,CrawlStatusListener,FetchStatusCodes(Code) Abstract implementation of a file pool processor. Subclass to implement for a particular WriterPoolMember instance. author:    Parker Thompson author:    stack Field Summary final protected static  String ANNOTATION_UNWRITTEN      final public static  String ATTR_COMPRESS      Key to use asking settings for file compression value. final public static  String ATTR_MAX_BYTES_WRITTEN      Key for the maximum bytes to write attribute. final public static  String ATTR_MAX_SIZE_BYTES      Key to use asking settings for file max size value. final public static  String ATTR_PATH      Key to use asking settings for arc path value. final public static  String ATTR_POOL_MAX_ACTIVE      Key to get maximum pool size. final public static  String ATTR_POOL_MAX_WAIT      Key to get maximum wait on pool object before we give up and throw IOException. final public static  String ATTR_PREFIX      Key to use asking settings for file prefix value. final public static  String ATTR_SKIP_IDENTICAL_DIGESTS      final public static  String ATTR_SUFFIX      Key to use asking settings for file suffix value. final public static  boolean DEFAULT_COMPRESS      Default as to whether we do compression of files. Constructor Summary public   WriterPoolProcessor(String name)      public   WriterPoolProcessor(String name, String description)      Method Summary protected synchronized  List<String> cacheMetadata()      protected  void checkBytesWritten()      protected  void checkpointRecover()      Called out of WriterPoolProcessor.initialTasks() when recovering a checkpoint. public  void crawlCheckpoint(File checkpointDir)      public  void crawlEnded(String sExitMessage)      public  void crawlEnding(String sExitMessage)      public  void crawlPaused(String statusMessage)      public  void crawlPausing(String statusMessage)      public  void crawlResuming(String statusMessage)      public  void crawlStarted(String message)      public  Object getAttributeUnchecked(String name)      Version of getAttributes that catches and logs exceptions and returns null if failure to fetch the attribute. Parameters:   name - Attribute name. protected  String getCheckpointStateFile()      protected  String[] getDefaultPath()      protected  String getFirstrecordBody(File orderFile)      Write the arc metadata body content. Its based on the order xml file but into this base we'll add other info such as machine ip. Parameters:   orderFile - Order file. protected  String getFirstrecordStylesheet()      protected  String getHostAddress(CrawlURI curi)      Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line). public  long getMaxSize()      Max size we want files to be (bytes). Default is ARCConstants.DEFAULT_MAX_ARC_FILE_SIZE. public  long getMaxToWrite()      public synchronized  List<String> getMetadata()      Return list of metadatas to add to first arc file metadata record. Default is to stylesheet the order file. public  List<File> getOutputDirs()      protected  WriterPool getPool()      public  int getPoolMaximumActive()      public  int getPoolMaximumWait()      public  String getPrefix()      protected  AtomicInteger getSerialNo()      public  String getSuffix()      protected  long getTotalBytesWritten()      public synchronized  void initialTasks()      abstract protected  void innerProcess(CrawlURI curi)      Writes a CrawlURI and its associated data to store file. public  boolean isCompressed()      protected  int loadCheckpointSerialNumber()      protected  void saveCheckpointSerialNumber(File checkpointDir, int serialNo)      protected  void setPool(WriterPool pool)      protected  void setTotalBytesWritten(long totalBytesWritten)      abstract protected  void setupPool(AtomicInteger serialNo)      Set up pool of files. protected  boolean shouldWrite(CrawlURI curi)      Whether the given CrawlURI should be written to archive files. Field Detail ANNOTATION_UNWRITTEN final protected static String ANNOTATION_UNWRITTEN(Code) CrawlURI annotation indicating no record was written ATTR_COMPRESS final public static String ATTR_COMPRESS(Code) Key to use asking settings for file compression value. ATTR_MAX_BYTES_WRITTEN final public static String ATTR_MAX_BYTES_WRITTEN(Code) Key for the maximum bytes to write attribute. ATTR_MAX_SIZE_BYTES final public static String ATTR_MAX_SIZE_BYTES(Code) Key to use asking settings for file max size value. ATTR_PATH final public static String ATTR_PATH(Code) Key to use asking settings for arc path value. ATTR_POOL_MAX_ACTIVE final public static String ATTR_POOL_MAX_ACTIVE(Code) Key to get maximum pool size. This key is for maximum files active in the pool. ATTR_POOL_MAX_WAIT final public static String ATTR_POOL_MAX_WAIT(Code) Key to get maximum wait on pool object before we give up and throw IOException. ATTR_PREFIX final public static String ATTR_PREFIX(Code) Key to use asking settings for file prefix value. ATTR_SKIP_IDENTICAL_DIGESTS final public static String ATTR_SKIP_IDENTICAL_DIGESTS(Code) Key for whether to skip writing records of content-digest repeats ATTR_SUFFIX final public static String ATTR_SUFFIX(Code) Key to use asking settings for file suffix value. DEFAULT_COMPRESS final public static boolean DEFAULT_COMPRESS(Code) Default as to whether we do compression of files. Constructor Detail WriterPoolProcessor public WriterPoolProcessor(String name)(Code) Parameters:   name - Name of this processor. WriterPoolProcessor public WriterPoolProcessor(String name, String description)(Code) Parameters:   name - Name of this processor. Parameters:   description - Description for this processor. Method Detail cacheMetadata protected synchronized List<String> cacheMetadata()(Code) checkBytesWritten protected void checkBytesWritten()(Code) checkpointRecover protected void checkpointRecover()(Code) Called out of WriterPoolProcessor.initialTasks() when recovering a checkpoint. Restore state. crawlCheckpoint public void crawlCheckpoint(File checkpointDir) throws IOException(Code) crawlEnded public void crawlEnded(String sExitMessage)(Code) crawlEnding public void crawlEnding(String sExitMessage)(Code) crawlPaused public void crawlPaused(String statusMessage)(Code) crawlPausing public void crawlPausing(String statusMessage)(Code) crawlResuming public void crawlResuming(String statusMessage)(Code) crawlStarted public void crawlStarted(String message)(Code) getAttributeUnchecked public Object getAttributeUnchecked(String name)(Code) Version of getAttributes that catches and logs exceptions and returns null if failure to fetch the attribute. Parameters:   name - Attribute name. Attribute or null. getCheckpointStateFile protected String getCheckpointStateFile()(Code) getDefaultPath protected String[] getDefaultPath()(Code) getFirstrecordBody protected String getFirstrecordBody(File orderFile)(Code) Write the arc metadata body content. Its based on the order xml file but into this base we'll add other info such as machine ip. Parameters:   orderFile - Order file. String that holds the arc metaheader body. getFirstrecordStylesheet protected String getFirstrecordStylesheet()(Code) getHostAddress protected String getHostAddress(CrawlURI curi)(Code) Return IP address of given URI suitable for recording (as in a classic ARC 5-field header line). Parameters:   curi - CrawlURI String of IP address getMaxSize public long getMaxSize()(Code) Max size we want files to be (bytes). Default is ARCConstants.DEFAULT_MAX_ARC_FILE_SIZE. Note that ARC files will usually be bigger than maxSize; they'll be maxSize + length to next boundary. ARC maximum size. getMaxToWrite public long getMaxToWrite()(Code) getMetadata public synchronized List<String> getMetadata()(Code) Return list of metadatas to add to first arc file metadata record. Default is to stylesheet the order file. To specify stylesheet, override WriterPoolProcessor.getFirstrecordStylesheet() . Get xml files from settingshandler. Currently order file is the only xml file. We're NOT adding seeds to meta data. List of strings and/or files to add to arc file as metadata ornull. getOutputDirs public List<File> getOutputDirs()(Code) getPool protected WriterPool getPool()(Code) getPoolMaximumActive public int getPoolMaximumActive()(Code) Returns the poolMaximumActive. getPoolMaximumWait public int getPoolMaximumWait()(Code) Returns the poolMaximumWait. getPrefix public String getPrefix()(Code) getSerialNo protected AtomicInteger getSerialNo()(Code) getSuffix public String getSuffix()(Code) getTotalBytesWritten protected long getTotalBytesWritten()(Code) initialTasks public synchronized void initialTasks()(Code) innerProcess abstract protected void innerProcess(CrawlURI curi)(Code) Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https. Parameters:   curi - CrawlURI to process. isCompressed public boolean isCompressed()(Code) loadCheckpointSerialNumber protected int loadCheckpointSerialNumber()(Code) Serial number from checkpoint state file or if unreadable, -1(Client should check for -1). saveCheckpointSerialNumber protected void saveCheckpointSerialNumber(File checkpointDir, int serialNo) throws IOException(Code) setPool protected void setPool(WriterPool pool)(Code) setTotalBytesWritten protected void setTotalBytesWritten(long totalBytesWritten)(Code) setupPool abstract protected void setupPool(AtomicInteger serialNo)(Code) Set up pool of files. shouldWrite protected boolean shouldWrite(CrawlURI curi)(Code) Whether the given CrawlURI should be written to archive files. Annotates CrawlURI with a reason for any negative answer. Parameters:   curi - CrawlURI true if URI should be written; false otherwise Fields inherited from org.archive.crawler.framework.Processor final public static String ATTR_DECIDE_RULES(Code)(Java Doc) final public static String ATTR_ENABLED(Code)(Java Doc) protected String attrDecideRules(Code)(Java Doc) Methods inherited from org.archive.crawler.framework.Processor protected void checkForInterrupt() throws InterruptedException(Code)(Java Doc) protected void finalTasks()(Code)(Java Doc) public CrawlController getController()(Code)(Java Doc) protected DecideRule getDecideRule(Object o)(Code)(Java Doc) public Processor getDefaultNextProcessor(CrawlURI curi)(Code)(Java Doc) protected void initialTasks()(Code)(Java Doc) protected void innerProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc) protected void innerRejectProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc) protected boolean isContentToProcess(CrawlURI curi)(Code)(Java Doc) protected boolean isExpectedMimeType(String contentType, String expectedPrefix)(Code)(Java Doc) protected boolean isHttpTransactionContentToProcess(CrawlURI curi)(Code)(Java Doc) public void kickUpdate()(Code)(Java Doc) final public void process(CrawlURI curi) throws InterruptedException(Code)(Java Doc) public String report()(Code)(Java Doc) protected boolean rulesAccept(Object o)(Code)(Java Doc) protected boolean rulesAccept(DecideRule rule, Object o)(Code)(Java Doc) public void setDefaultNextProcessor(Processor nextProcessor)(Code)(Java Doc) public Processor spawn(int serialNum)(Code)(Java Doc)