Java Doc for ExperimentalV10WARCWriterProcessor.java in  » Web-Crawler » heritrix » org » archive » crawler » writer » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation
1. 6.0 JDK Core
2. 6.0 JDK Modules
3. 6.0 JDK Modules com.sun
4. 6.0 JDK Modules com.sun.java
5. 6.0 JDK Modules sun
6. 6.0 JDK Platform
7. Ajax
8. Apache Harmony Java SE
9. Aspect oriented
10. Authentication Authorization
11. Blogger System
12. Build
13. Byte Code
14. Cache
15. Chart
16. Chat
17. Code Analyzer
18. Collaboration
19. Content Management System
20. Database Client
21. Database DBMS
22. Database JDBC Connection Pool
23. Database ORM
24. Development
25. EJB Server geronimo
26. EJB Server GlassFish
27. EJB Server JBoss 4.2.1
28. EJB Server resin 3.1.5
29. ERP CRM Financial
30. ESB
31. Forum
32. GIS
33. Graphic Library
34. Groupware
35. HTML Parser
36. IDE
37. IDE Eclipse
38. IDE Netbeans
39. Installer
40. Internationalization Localization
41. Inversion of Control
42. Issue Tracking
43. J2EE
44. JBoss
45. JMS
46. JMX
47. Library
48. Mail Clients
49. Net
50. Parser
51. PDF
52. Portal
53. Profiler
54. Project Management
55. Report
56. RSS RDF
57. Rule Engine
58. Science
59. Scripting
60. Search Engine
61. Security
62. Sevlet Container
63. Source Control
64. Swing Library
65. Template Engine
66. Test Coverage
67. Testing
68. UML
69. Web Crawler
70. Web Framework
71. Web Mail
72. Web Server
73. Web Services
74. Web Services apache cxf 2.0.1
75. Web Services AXIS2
76. Wiki Engine
77. Workflow Engines
78. XML
79. XML UI
Java
Java Tutorial
Java Open Source
Jar File Download
Java Articles
Java Products
Java by API
Photoshop Tutorials
Maya Tutorials
Flash Tutorials
3ds-Max Tutorials
Illustrator Tutorials
GIMP Tutorials
C# / C Sharp
C# / CSharp Tutorial
C# / CSharp Open Source
ASP.Net
ASP.NET Tutorial
JavaScript DHTML
JavaScript Tutorial
JavaScript Reference
HTML / CSS
HTML CSS Reference
C / ANSI-C
C Tutorial
C++
C++ Tutorial
Ruby
PHP
Python
Python Tutorial
Python Open Source
SQL Server / T-SQL
SQL Server / T-SQL Tutorial
Oracle PL / SQL
Oracle PL/SQL Tutorial
PostgreSQL
SQL / MySQL
MySQL Tutorial
VB.Net
VB.Net Tutorial
Flash / Flex / ActionScript
VBA / Excel / Access / Word
XML
XML Tutorial
Microsoft Office PowerPoint 2007 Tutorial
Microsoft Office Excel 2007 Tutorial
Microsoft Office Word 2007 Tutorial
Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.writer 
Source Cross Reference  Class Diagram Java Document (Java Doc) 


org.archive.crawler.framework.WriterPoolProcessor
   org.archive.crawler.writer.ExperimentalV10WARCWriterProcessor

ExperimentalV10WARCWriterProcessor
public class ExperimentalV10WARCWriterProcessor extends WriterPoolProcessor implements CoreAttributeConstants,CrawlStatusListener,WriterPoolSettings,FetchStatusCodes,WARCConstants(Code)
Experimental WARCWriterProcessor. Implements 0.10 version of the WARC Specification since superceded by version 0.12.
author:
   stackExperimentalWARCWriter



Constructor Summary
public  ExperimentalV10WARCWriterProcessor(String name)
    

Method Summary
protected  String[]getDefaultPath()
    
public  ListgetMetadata()
    
protected  URIgetRecordID()
    
protected  voidinnerProcess(CrawlURI curi)
     Writes a CrawlURI and its associated data to store file.
protected  URIqualifyRecordID(URI base, String key, String value)
    
protected  voidsetupPool(AtomicInteger serialNo)
    
protected  voidwrite(String lowerCaseScheme, CrawlURI curi)
    
protected  URIwriteMetadata(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields)
    
protected  URIwriteRequest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields)
    
protected  URIwriteResponse(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields)
    


Constructor Detail
ExperimentalV10WARCWriterProcessor
public ExperimentalV10WARCWriterProcessor(String name)(Code)

Parameters:
  name - Name of this writer.




Method Detail
getDefaultPath
protected String[] getDefaultPath()(Code)



getMetadata
public List getMetadata()(Code)



getRecordID
protected URI getRecordID() throws IOException(Code)



innerProcess
protected void innerProcess(CrawlURI curi)(Code)
Writes a CrawlURI and its associated data to store file. Currently this method understands the following uri types: dns, http, and https.
Parameters:
  curi - CrawlURI to process.



qualifyRecordID
protected URI qualifyRecordID(URI base, String key, String value) throws IOException(Code)



setupPool
protected void setupPool(AtomicInteger serialNo)(Code)



write
protected void write(String lowerCaseScheme, CrawlURI curi) throws IOException(Code)



writeMetadata
protected URI writeMetadata(ExperimentalWARCWriter w, String timestamp, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code)



writeRequest
protected URI writeRequest(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code)



writeResponse
protected URI writeResponse(ExperimentalWARCWriter w, String timestamp, String mimetype, URI baseid, CrawlURI curi, ANVLRecord namedFields) throws IOException(Code)



Fields inherited from org.archive.crawler.framework.WriterPoolProcessor
final protected static String ANNOTATION_UNWRITTEN(Code)(Java Doc)
final public static String ATTR_COMPRESS(Code)(Java Doc)
final public static String ATTR_MAX_BYTES_WRITTEN(Code)(Java Doc)
final public static String ATTR_MAX_SIZE_BYTES(Code)(Java Doc)
final public static String ATTR_PATH(Code)(Java Doc)
final public static String ATTR_POOL_MAX_ACTIVE(Code)(Java Doc)
final public static String ATTR_POOL_MAX_WAIT(Code)(Java Doc)
final public static String ATTR_PREFIX(Code)(Java Doc)
final public static String ATTR_SKIP_IDENTICAL_DIGESTS(Code)(Java Doc)
final public static String ATTR_SUFFIX(Code)(Java Doc)
final public static boolean DEFAULT_COMPRESS(Code)(Java Doc)

Methods inherited from org.archive.crawler.framework.WriterPoolProcessor
protected synchronized List<String> cacheMetadata()(Code)(Java Doc)
protected void checkBytesWritten()(Code)(Java Doc)
protected void checkpointRecover()(Code)(Java Doc)
public void crawlCheckpoint(File checkpointDir) throws IOException(Code)(Java Doc)
public void crawlEnded(String sExitMessage)(Code)(Java Doc)
public void crawlEnding(String sExitMessage)(Code)(Java Doc)
public void crawlPaused(String statusMessage)(Code)(Java Doc)
public void crawlPausing(String statusMessage)(Code)(Java Doc)
public void crawlResuming(String statusMessage)(Code)(Java Doc)
public void crawlStarted(String message)(Code)(Java Doc)
public Object getAttributeUnchecked(String name)(Code)(Java Doc)
protected String getCheckpointStateFile()(Code)(Java Doc)
protected String[] getDefaultPath()(Code)(Java Doc)
protected String getFirstrecordBody(File orderFile)(Code)(Java Doc)
protected String getFirstrecordStylesheet()(Code)(Java Doc)
protected String getHostAddress(CrawlURI curi)(Code)(Java Doc)
public long getMaxSize()(Code)(Java Doc)
public long getMaxToWrite()(Code)(Java Doc)
public synchronized List<String> getMetadata()(Code)(Java Doc)
public List<File> getOutputDirs()(Code)(Java Doc)
protected WriterPool getPool()(Code)(Java Doc)
public int getPoolMaximumActive()(Code)(Java Doc)
public int getPoolMaximumWait()(Code)(Java Doc)
public String getPrefix()(Code)(Java Doc)
protected AtomicInteger getSerialNo()(Code)(Java Doc)
public String getSuffix()(Code)(Java Doc)
protected long getTotalBytesWritten()(Code)(Java Doc)
public synchronized void initialTasks()(Code)(Java Doc)
abstract protected void innerProcess(CrawlURI curi)(Code)(Java Doc)
public boolean isCompressed()(Code)(Java Doc)
protected int loadCheckpointSerialNumber()(Code)(Java Doc)
protected void saveCheckpointSerialNumber(File checkpointDir, int serialNo) throws IOException(Code)(Java Doc)
protected void setPool(WriterPool pool)(Code)(Java Doc)
protected void setTotalBytesWritten(long totalBytesWritten)(Code)(Java Doc)
abstract protected void setupPool(AtomicInteger serialNo)(Code)(Java Doc)
protected boolean shouldWrite(CrawlURI curi)(Code)(Java Doc)

www.java2java.com | Contact Us
Copyright 2009 - 12 Demo Source and Support. All rights reserved.
All other trademarks are property of their respective owners.