Java Doc for ExperimentalWARCWriterProcessor.java in  » Web-Crawler » heritrix » org » archive » crawler » writer » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation 1.  6.0 JDK Core 2.  6.0 JDK Modules 3.  6.0 JDK Modules com.sun 4.  6.0 JDK Modules com.sun.java 5.  6.0 JDK Modules sun 6.  6.0 JDK Platform 7.  Ajax 8.  Apache Harmony Java SE 9.  Aspect oriented 10.  Authentication Authorization 11.  Blogger System 12.  Build 13.  Byte Code 14.  Cache 15.  Chart 16.  Chat 17.  Code Analyzer 18.  Collaboration 19.  Content Management System 20.  Database Client 21.  Database DBMS 22.  Database JDBC Connection Pool 23.  Database ORM 24.  Development 25.  EJB Server geronimo 26.  EJB Server GlassFish 27.  EJB Server JBoss 4.2.1 28.  EJB Server resin 3.1.5 29.  ERP CRM Financial 30.  ESB 31.  Forum 32.  GIS 33.  Graphic Library 34.  Groupware 35.  HTML Parser 36.  IDE 37.  IDE Eclipse 38.  IDE Netbeans 39.  Installer 40.  Internationalization Localization 41.  Inversion of Control 42.  Issue Tracking 43.  J2EE 44.  JBoss 45.  JMS 46.  JMX 47.  Library 48.  Mail Clients 49.  Net 50.  Parser 51.  PDF 52.  Portal 53.  Profiler 54.  Project Management 55.  Report 56.  RSS RDF 57.  Rule Engine 58.  Science 59.  Scripting 60.  Search Engine 61.  Security 62.  Sevlet Container 63.  Source Control 64.  Swing Library 65.  Template Engine 66.  Test Coverage 67.  Testing 68.  UML 69.  Web Crawler 70.  Web Framework 71.  Web Mail 72.  Web Server 73.  Web Services 74.  Web Services apache cxf 2.0.1 75.  Web Services AXIS2 76.  Wiki Engine 77.  Workflow Engines 78.  XML 79.  XML UI Java Java Tutorial Java Open Source Jar File Download Java Articles Java Products Java by API Photoshop Tutorials Maya Tutorials Flash Tutorials 3ds-Max Tutorials Illustrator Tutorials GIMP Tutorials C# / C Sharp C# / CSharp Tutorial C# / CSharp Open Source ASP.Net ASP.NET Tutorial JavaScript DHTML JavaScript Tutorial JavaScript Reference HTML / CSS HTML CSS Reference C / ANSI-C C Tutorial C++ C++ Tutorial Ruby PHP Python Python Tutorial Python Open Source SQL Server / T-SQL SQL Server / T-SQL Tutorial Oracle PL / SQL Oracle PL/SQL Tutorial PostgreSQL SQL / MySQL MySQL Tutorial VB.Net VB.Net Tutorial Flash / Flex / ActionScript VBA / Excel / Access / Word XML XML Tutorial Microsoft Office PowerPoint 2007 Tutorial Microsoft Office Excel 2007 Tutorial Microsoft Office Word 2007 Tutorial

Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.writer   Source Cross Reference   Class Diagram   Java Document (Java Doc)   org.archive.crawler.framework.WriterPoolProcessor    org.archive.crawler.writer.ExperimentalWARCWriterProcessor ExperimentalWARCWriterProcessor public class ExperimentalWARCWriterProcessor extends WriterPoolProcessor implements CoreAttributeConstants,CrawlStatusListener,WriterPoolSettings,FetchStatusCodes,WARCConstants(Code) Experimental WARCWriterProcessor. Goes against the pending release of 0.12 of the WARC specification, the "Marcel Marceau" release. See latest revision for current state. The 0.10 WARC implemenation has been moved to ExperimentalV10WARCWriterProcessor . TODO: Remove ANVLRecord. Rename NameValue or use RFC822 (commons-httpclient?) or find something else. author:    stack Field Summary final public static  String ATTR_WRITE_METADATA      final public static  String ATTR_WRITE_REQUESTS      final public static  String ATTR_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS      final public static  String ATTR_WRITE_REVISIT_FOR_NOT_MODIFIED      Constructor Summary public   ExperimentalWARCWriterProcessor(String name)      Method Summary protected  String[] getDefaultPath()      protected  String getFirstrecordStylesheet()      protected  URI getRecordID()      protected  void innerProcess(CrawlURI curi)      Writes a CrawlURI and its associated data to store file. protected  URI qualifyRecordID(URI base, String key, String value)      protected  void saveHeader(String origName, HttpMethodBase method, ANVLRecord headers, String newName)      protected  void setupPool(AtomicInteger serialNo)      protected  void write(String lowerCaseScheme, CrawlURI curi)      protected  URI writeMetadata(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields)      protected  URI writeRequest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields)      protected  URI writeResponse(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields)      protected  URI writeRevisitDigest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields)      protected  URI writeRevisitNotModified(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields)      Field Detail ATTR_WRITE_METADATA final public static String ATTR_WRITE_METADATA(Code) Key for whether to write 'metadata' type records where possible ATTR_WRITE_REQUESTS final public static String ATTR_WRITE_REQUESTS(Code) Key for whether to write 'request' type records where possible ATTR_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS final public static String ATTR_WRITE_REVISIT_FOR_IDENTICAL_DIGESTS(Code) Key for whether to write 'revisit' type records when consecutive identical digest ATTR_WRITE_REVISIT_FOR_NOT_MODIFIED final public static String ATTR_WRITE_REVISIT_FOR_NOT_MODIFIED(Code) Key for whether to write 'revisit' type records for server "304 not modified" responses Constructor Detail ExperimentalWARCWriterProcessor public ExperimentalWARCWriterProcessor(String name)(Code) Parameters:   name - Name of this writer. Method Detail getDefaultPath protected String[] getDefaultPath()(Code) getFirstrecordStylesheet protected String getFirstrecordStylesheet()(Code) getRecordID protected URI getRecordID() throws IOException(Code) innerProcess protected void innerProcess(CrawlURI curi)(Code) Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https. Parameters:   curi - CrawlURI to process. qualifyRecordID protected URI qualifyRecordID(URI base, String key, String value) throws IOException(Code) saveHeader protected void saveHeader(String origName, HttpMethodBase method, ANVLRecord headers, String newName)(Code) Save a header from the given HTTP operation into the provider headers under a new name Parameters:   origName - header name to get if present Parameters:   method - http operation containing headers setupPool protected void setupPool(AtomicInteger serialNo)(Code) write protected void write(String lowerCaseScheme, CrawlURI curi) throws IOException(Code) writeMetadata protected URI writeMetadata(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code) writeRequest protected URI writeRequest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code) writeResponse protected URI writeResponse(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code) writeRevisitDigest protected URI writeRevisitDigest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code) writeRevisitNotModified protected URI writeRevisitNotModified(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code) Fields inherited from org.archive.crawler.framework.WriterPoolProcessor final protected static String ANNOTATION_UNWRITTEN(Code)(Java Doc) final public static String ATTR_COMPRESS(Code)(Java Doc) final public static String ATTR_MAX_BYTES_WRITTEN(Code)(Java Doc) final public static String ATTR_MAX_SIZE_BYTES(Code)(Java Doc) final public static String ATTR_PATH(Code)(Java Doc) final public static String ATTR_POOL_MAX_ACTIVE(Code)(Java Doc) final public static String ATTR_POOL_MAX_WAIT(Code)(Java Doc) final public static String ATTR_PREFIX(Code)(Java Doc) final public static String ATTR_SKIP_IDENTICAL_DIGESTS(Code)(Java Doc) final public static String ATTR_SUFFIX(Code)(Java Doc) final public static boolean DEFAULT_COMPRESS(Code)(Java Doc) Methods inherited from org.archive.crawler.framework.WriterPoolProcessor protected synchronized List<String> cacheMetadata()(Code)(Java Doc) protected void checkBytesWritten()(Code)(Java Doc) protected void checkpointRecover()(Code)(Java Doc) public void crawlCheckpoint(File checkpointDir) throws IOException(Code)(Java Doc) public void crawlEnded(String sExitMessage)(Code)(Java Doc) public void crawlEnding(String sExitMessage)(Code)(Java Doc) public void crawlPaused(String statusMessage)(Code)(Java Doc) public void crawlPausing(String statusMessage)(Code)(Java Doc) public void crawlResuming(String statusMessage)(Code)(Java Doc) public void crawlStarted(String message)(Code)(Java Doc) public Object getAttributeUnchecked(String name)(Code)(Java Doc) protected String getCheckpointStateFile()(Code)(Java Doc) protected String[] getDefaultPath()(Code)(Java Doc) protected String getFirstrecordBody(File orderFile)(Code)(Java Doc) protected String getFirstrecordStylesheet()(Code)(Java Doc) protected String getHostAddress(CrawlURI curi)(Code)(Java Doc) public long getMaxSize()(Code)(Java Doc) public long getMaxToWrite()(Code)(Java Doc) public synchronized List<String> getMetadata()(Code)(Java Doc) public List<File> getOutputDirs()(Code)(Java Doc) protected WriterPool getPool()(Code)(Java Doc) public int getPoolMaximumActive()(Code)(Java Doc) public int getPoolMaximumWait()(Code)(Java Doc) public String getPrefix()(Code)(Java Doc) protected AtomicInteger getSerialNo()(Code)(Java Doc) public String getSuffix()(Code)(Java Doc) protected long getTotalBytesWritten()(Code)(Java Doc) public synchronized void initialTasks()(Code)(Java Doc) abstract protected void innerProcess(CrawlURI curi)(Code)(Java Doc) public boolean isCompressed()(Code)(Java Doc) protected int loadCheckpointSerialNumber()(Code)(Java Doc) protected void saveCheckpointSerialNumber(File checkpointDir, int serialNo) throws IOException(Code)(Java Doc) protected void setPool(WriterPool pool)(Code)(Java Doc) protected void setTotalBytesWritten(long totalBytesWritten)(Code)(Java Doc) abstract protected void setupPool(AtomicInteger serialNo)(Code)(Java Doc) protected boolean shouldWrite(CrawlURI curi)(Code)(Java Doc)