Java Doc for Kw3WriterProcessor.java in  » Web-Crawler » heritrix » org » archive » crawler » writer » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation 1.  6.0 JDK Core 2.  6.0 JDK Modules 3.  6.0 JDK Modules com.sun 4.  6.0 JDK Modules com.sun.java 5.  6.0 JDK Modules sun 6.  6.0 JDK Platform 7.  Ajax 8.  Apache Harmony Java SE 9.  Aspect oriented 10.  Authentication Authorization 11.  Blogger System 12.  Build 13.  Byte Code 14.  Cache 15.  Chart 16.  Chat 17.  Code Analyzer 18.  Collaboration 19.  Content Management System 20.  Database Client 21.  Database DBMS 22.  Database JDBC Connection Pool 23.  Database ORM 24.  Development 25.  EJB Server geronimo 26.  EJB Server GlassFish 27.  EJB Server JBoss 4.2.1 28.  EJB Server resin 3.1.5 29.  ERP CRM Financial 30.  ESB 31.  Forum 32.  GIS 33.  Graphic Library 34.  Groupware 35.  HTML Parser 36.  IDE 37.  IDE Eclipse 38.  IDE Netbeans 39.  Installer 40.  Internationalization Localization 41.  Inversion of Control 42.  Issue Tracking 43.  J2EE 44.  JBoss 45.  JMS 46.  JMX 47.  Library 48.  Mail Clients 49.  Net 50.  Parser 51.  PDF 52.  Portal 53.  Profiler 54.  Project Management 55.  Report 56.  RSS RDF 57.  Rule Engine 58.  Science 59.  Scripting 60.  Search Engine 61.  Security 62.  Sevlet Container 63.  Source Control 64.  Swing Library 65.  Template Engine 66.  Test Coverage 67.  Testing 68.  UML 69.  Web Crawler 70.  Web Framework 71.  Web Mail 72.  Web Server 73.  Web Services 74.  Web Services apache cxf 2.0.1 75.  Web Services AXIS2 76.  Wiki Engine 77.  Workflow Engines 78.  XML 79.  XML UI Java Java Tutorial Java Open Source Jar File Download Java Articles Java Products Java by API Photoshop Tutorials Maya Tutorials Flash Tutorials 3ds-Max Tutorials Illustrator Tutorials GIMP Tutorials C# / C Sharp C# / CSharp Tutorial C# / CSharp Open Source ASP.Net ASP.NET Tutorial JavaScript DHTML JavaScript Tutorial JavaScript Reference HTML / CSS HTML CSS Reference C / ANSI-C C Tutorial C++ C++ Tutorial Ruby PHP Python Python Tutorial Python Open Source SQL Server / T-SQL SQL Server / T-SQL Tutorial Oracle PL / SQL Oracle PL/SQL Tutorial PostgreSQL SQL / MySQL MySQL Tutorial VB.Net VB.Net Tutorial Flash / Flex / ActionScript VBA / Excel / Access / Word XML XML Tutorial Microsoft Office PowerPoint 2007 Tutorial Microsoft Office Excel 2007 Tutorial Microsoft Office Word 2007 Tutorial

Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.writer   Source Cross Reference   Class Diagram   Java Document (Java Doc)   org.archive.crawler.framework.Processor    org.archive.crawler.writer.Kw3WriterProcessor Kw3WriterProcessor public class Kw3WriterProcessor extends Processor implements CoreAttributeConstants,Kw3Constants(Code) Processor module that writes the results of successful fetches to files on disk. These files are MIME-files of the type used by the Swedish National Library's Kulturarw3 web harvesting [http://www.kb.se/kw3/]. Each URI gets written to its own file and has a path consisting of: A dir named with the first two chars of the website's md5. A dir named after the website. 'current' - a dir indicating that this is the directory being written to by the ongoing crawl. A file on the format . Example: '/53/www.kb.se/current/6879ad79c0ccf886ee8ca55d80e5d6a1.1169211837' The MIME-file itself consists of three parts: 1. ArchiveInfo - Metadata about the file and its content. 2. Header - The HTTP response header. 3. Content - The HTTP response content, plus content-type. author:    oskar Field Summary final public static  String ATTR_CHMOD      Key to use asking settings if chmod should be execuated . final public static  String ATTR_CHMOD_VALUE      Key to use asking settings for the new chmod value. final public static  String ATTR_COLLECTION      Key for the collection attribute. final public static  String ATTR_HARVESTER      Key for the harvester attribute. final public static  String ATTR_MAX_BYTES_WRITTEN      Key for the maximum ARC bytes to write attribute. final public static  String ATTR_MAX_SIZE_BYTES      Key to use asking settings for max size value. final public static  String ATTR_PATH      Key to use asking settings for arc path value. final public static  String DEFAULT_CHMOD_VALUE      Default value for permissions. final public static  String DEFAULT_COLLECTION_VALUE      Default value for collection. final public static  String DEFAULT_HARVESTER_VALUE      Default value for harvester. final public static  int DEFAULT_MAX_FILE_SIZE      Default max file size. Constructor Summary public   Kw3WriterProcessor(String name)      Method Summary protected  OutputStream initOutputStream(CrawlURI curi)      protected  void initialTasks()      protected  void innerProcess(CrawlURI curi)      protected  void writeArchiveInfoPart(String boundary, CrawlURI curi, ReplayInputStream ris, OutputStream out)      protected  void writeContentPart(String boundary, CrawlURI curi, ReplayInputStream ris, OutputStream out)      protected  void writeHeaderPart(String boundary, ReplayInputStream ris, OutputStream out)      protected  void writeMimeFile(CrawlURI curi)      Field Detail ATTR_CHMOD final public static String ATTR_CHMOD(Code) Key to use asking settings if chmod should be execuated . ATTR_CHMOD_VALUE final public static String ATTR_CHMOD_VALUE(Code) Key to use asking settings for the new chmod value. ATTR_COLLECTION final public static String ATTR_COLLECTION(Code) Key for the collection attribute. ATTR_HARVESTER final public static String ATTR_HARVESTER(Code) Key for the harvester attribute. ATTR_MAX_BYTES_WRITTEN final public static String ATTR_MAX_BYTES_WRITTEN(Code) Key for the maximum ARC bytes to write attribute. ATTR_MAX_SIZE_BYTES final public static String ATTR_MAX_SIZE_BYTES(Code) Key to use asking settings for max size value. ATTR_PATH final public static String ATTR_PATH(Code) Key to use asking settings for arc path value. DEFAULT_CHMOD_VALUE final public static String DEFAULT_CHMOD_VALUE(Code) Default value for permissions. DEFAULT_COLLECTION_VALUE final public static String DEFAULT_COLLECTION_VALUE(Code) Default value for collection. DEFAULT_HARVESTER_VALUE final public static String DEFAULT_HARVESTER_VALUE(Code) Default value for harvester. DEFAULT_MAX_FILE_SIZE final public static int DEFAULT_MAX_FILE_SIZE(Code) Default max file size. Constructor Detail Kw3WriterProcessor public Kw3WriterProcessor(String name)(Code) Parameters:   name - Name of this processor. Method Detail initOutputStream protected OutputStream initOutputStream(CrawlURI curi) throws IOException(Code) initialTasks protected void initialTasks()(Code) innerProcess protected void innerProcess(CrawlURI curi)(Code) writeArchiveInfoPart protected void writeArchiveInfoPart(String boundary, CrawlURI curi, ReplayInputStream ris, OutputStream out) throws IOException(Code) writeContentPart protected void writeContentPart(String boundary, CrawlURI curi, ReplayInputStream ris, OutputStream out) throws IOException(Code) writeHeaderPart protected void writeHeaderPart(String boundary, ReplayInputStream ris, OutputStream out) throws IOException(Code) writeMimeFile protected void writeMimeFile(CrawlURI curi) throws IOException(Code) Fields inherited from org.archive.crawler.framework.Processor final public static String ATTR_DECIDE_RULES(Code)(Java Doc) final public static String ATTR_ENABLED(Code)(Java Doc) protected String attrDecideRules(Code)(Java Doc) Methods inherited from org.archive.crawler.framework.Processor protected void checkForInterrupt() throws InterruptedException(Code)(Java Doc) protected void finalTasks()(Code)(Java Doc) public CrawlController getController()(Code)(Java Doc) protected DecideRule getDecideRule(Object o)(Code)(Java Doc) public Processor getDefaultNextProcessor(CrawlURI curi)(Code)(Java Doc) protected void initialTasks()(Code)(Java Doc) protected void innerProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc) protected void innerRejectProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc) protected boolean isContentToProcess(CrawlURI curi)(Code)(Java Doc) protected boolean isExpectedMimeType(String contentType, String expectedPrefix)(Code)(Java Doc) protected boolean isHttpTransactionContentToProcess(CrawlURI curi)(Code)(Java Doc) public void kickUpdate()(Code)(Java Doc) final public void process(CrawlURI curi) throws InterruptedException(Code)(Java Doc) public String report()(Code)(Java Doc) protected boolean rulesAccept(Object o)(Code)(Java Doc) protected boolean rulesAccept(DecideRule rule, Object o)(Code)(Java Doc) public void setDefaultNextProcessor(Processor nextProcessor)(Code)(Java Doc) public Processor spawn(int serialNum)(Code)(Java Doc)