Java Doc for MirrorWriterProcessor.java in » Web-Crawler » heritrix » org » archive » crawler » writer » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation

1.	6.0 JDK Core
2.	6.0 JDK Modules
3.	6.0 JDK Modules com.sun
4.	6.0 JDK Modules com.sun.java
5.	6.0 JDK Modules sun
6.	6.0 JDK Platform
7.	Ajax
8.	Apache Harmony Java SE
9.	Aspect oriented
10.	Authentication Authorization
11.	Blogger System
12.	Build
13.	Byte Code
14.	Cache
15.	Chart
16.	Chat
17.	Code Analyzer
18.	Collaboration
19.	Content Management System
20.	Database Client
21.	Database DBMS
22.	Database JDBC Connection Pool
23.	Database ORM
24.	Development
25.	EJB Server geronimo
26.	EJB Server GlassFish
27.	EJB Server JBoss 4.2.1
28.	EJB Server resin 3.1.5
29.	ERP CRM Financial
30.	ESB
31.	Forum
32.	GIS
33.	Graphic Library
34.	Groupware
35.	HTML Parser
36.	IDE
37.	IDE Eclipse
38.	IDE Netbeans
39.	Installer
40.	Internationalization Localization
41.	Inversion of Control
42.	Issue Tracking
43.	J2EE
44.	JBoss
45.	JMS
46.	JMX
47.	Library
48.	Mail Clients
49.	Net
50.	Parser
51.	PDF
52.	Portal
53.	Profiler
54.	Project Management
55.	Report
56.	RSS RDF
57.	Rule Engine
58.	Science
59.	Scripting
60.	Search Engine
61.	Security
62.	Sevlet Container
63.	Source Control
64.	Swing Library
65.	Template Engine
66.	Test Coverage
67.	Testing
68.	UML
69.	Web Crawler
70.	Web Framework
71.	Web Mail
72.	Web Server
73.	Web Services
74.	Web Services apache cxf 2.0.1
75.	Web Services AXIS2
76.	Wiki Engine
77.	Workflow Engines
78.	XML
79.	XML UI

Java

Java Tutorial

Illustrator Tutorials

GIMP Tutorials

C# / C Sharp

C# / CSharp Tutorial

C# / CSharp Open Source

SQL Server / T-SQL Tutorial

Oracle PL / SQL

Oracle PL/SQL Tutorial

Flash / Flex / ActionScript

VBA / Excel / Access / Word

XML

XML Tutorial

Microsoft Office PowerPoint 2007 Tutorial

Microsoft Office Excel 2007 Tutorial

Microsoft Office Word 2007 Tutorial

Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.writer

Source Cross Reference

Class Diagram

Java Document (Java Doc)

org.archive.crawler.framework .Processor

org.archive.crawler.writer .MirrorWriterProcessor

MirrorWriterProcessor

public class MirrorWriterProcessor extends Processor implements CoreAttributeConstants(Code)

Processor module that writes the results of successful fetches to files on disk. Writes contents of one URI to one file on disk. The files are arranged in a directory hierarchy based on the URI paths. In that sense they mirror the file hierarchy that might exist on the servers.

There are a number of issues involved:

URIs can have arbitrary length, but file systems have length constraints.
URIs can contain characters that file systems prohibit.
URI paths are case-sensitive, but some file systems are case-insensitive.

This class tries very hard to map each URI into a file system path that obeys all file system constraints and yet reasonably represents the original URI.

There would normally be a single instance of this class per Heritrix instance. This class is thread-safe; any number of threads can be in its innerProcess method at once. However, conflicts can still arise in the file system. For example, if several threads try to create the same directory at the same time, only one can win. Therefore, there should be at most one access to a server at a given time.
author:
Howard Lee Gayle

Inner Class :abstract class PathSegment

Inner Class :class DirSegment extends PathSegment

Inner Class :class EndSegment extends PathSegment

Inner Class :class LumpyString

Inner Class :class URIToFileReturn

Field Summary
final public static String	ATTR_CASE_SENSITIVE Key to use asking settings for case sensitive option.
final public static String	ATTR_CHAR_MAP Key to use asking settings for character map.
final public static String	ATTR_CONTENT_TYPE_MAP Key to use asking settings for content type map.
final public static String	ATTR_DIRECTORY_FILE Key to use asking settings for directory file.
final public static String	ATTR_DOT_BEGIN Key to use asking settings for dot begin replacement.
final public static String	ATTR_DOT_END Key to use asking settings for dot end replacement.
final public static String	ATTR_HOST_DIRECTORY Key to use asking settings for host directory option.
final public static String	ATTR_HOST_MAP Key to use asking settings for host map.
final public static String	ATTR_MAX_PATH_LEN Key to use asking settings for maximum file system path length.
final public static String	ATTR_MAX_SEG_LEN Key to use asking settings for maximum file system path segment length.
final public static String	ATTR_PATH Key to use asking settings for base directory path value.
final public static String	ATTR_PORT_DIRECTORY Key to use asking settings for port directory option.
final public static String	ATTR_SUFFIX_AT_END Key to use asking settings for suffix at end option.
final public static String	ATTR_TOO_LONG_DIRECTORY Key to use asking settings for too-long directory.
final public static String	ATTR_UNDERSCORE_SET Key to use asking settings for underscore set.

Constructor Summary
public	MirrorWriterProcessor(String name)

Method Summary
protected void	innerProcess(CrawlURI curi)

Field Detail

ATTR_CASE_SENSITIVE
final public static String ATTR_CASE_SENSITIVE(Code)
	Key to use asking settings for case sensitive option.

ATTR_CHAR_MAP
final public static String ATTR_CHAR_MAP(Code)
	Key to use asking settings for character map.

ATTR_CONTENT_TYPE_MAP
final public static String ATTR_CONTENT_TYPE_MAP(Code)
	Key to use asking settings for content type map.

ATTR_DIRECTORY_FILE
final public static String ATTR_DIRECTORY_FILE(Code)
	Key to use asking settings for directory file.

ATTR_DOT_BEGIN
final public static String ATTR_DOT_BEGIN(Code)
	Key to use asking settings for dot begin replacement.

ATTR_DOT_END
final public static String ATTR_DOT_END(Code)
	Key to use asking settings for dot end replacement.

ATTR_HOST_DIRECTORY
final public static String ATTR_HOST_DIRECTORY(Code)
	Key to use asking settings for host directory option.

ATTR_HOST_MAP
final public static String ATTR_HOST_MAP(Code)
	Key to use asking settings for host map.

ATTR_MAX_PATH_LEN
final public static String ATTR_MAX_PATH_LEN(Code)
	Key to use asking settings for maximum file system path length.

ATTR_MAX_SEG_LEN
final public static String ATTR_MAX_SEG_LEN(Code)
	Key to use asking settings for maximum file system path segment length.

ATTR_PATH
final public static String ATTR_PATH(Code)
	Key to use asking settings for base directory path value.

ATTR_PORT_DIRECTORY
final public static String ATTR_PORT_DIRECTORY(Code)
	Key to use asking settings for port directory option.

ATTR_SUFFIX_AT_END
final public static String ATTR_SUFFIX_AT_END(Code)
	Key to use asking settings for suffix at end option.

ATTR_TOO_LONG_DIRECTORY
final public static String ATTR_TOO_LONG_DIRECTORY(Code)
	Key to use asking settings for too-long directory.

ATTR_UNDERSCORE_SET
final public static String ATTR_UNDERSCORE_SET(Code)
	Key to use asking settings for underscore set.

Constructor Detail

MirrorWriterProcessor
public MirrorWriterProcessor(String name)(Code)
	Parameters: name - Name of this processor.

Method Detail

innerProcess
protected void innerProcess(CrawlURI curi)(Code)

Fields inherited from org.archive.crawler.framework.Processor

final public static String ATTR_DECIDE_RULES(Code)(Java Doc)
final public static String ATTR_ENABLED(Code)(Java Doc)
protected String attrDecideRules(Code)(Java Doc)

Methods inherited from org.archive.crawler.framework.Processor

protected void checkForInterrupt() throws InterruptedException(Code)(Java Doc)
protected void finalTasks()(Code)(Java Doc)
public CrawlController getController()(Code)(Java Doc)
protected DecideRule getDecideRule(Object o)(Code)(Java Doc)
public Processor getDefaultNextProcessor(CrawlURI curi)(Code)(Java Doc)
protected void initialTasks()(Code)(Java Doc)
protected void innerProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc)
protected void innerRejectProcess(CrawlURI curi) throws InterruptedException(Code)(Java Doc)
protected boolean isContentToProcess(CrawlURI curi)(Code)(Java Doc)
protected boolean isExpectedMimeType(String contentType, String expectedPrefix)(Code)(Java Doc)
protected boolean isHttpTransactionContentToProcess(CrawlURI curi)(Code)(Java Doc)
public void kickUpdate()(Code)(Java Doc)
final public void process(CrawlURI curi) throws InterruptedException(Code)(Java Doc)
public String report()(Code)(Java Doc)
protected boolean rulesAccept(Object o)(Code)(Java Doc)
protected boolean rulesAccept(DecideRule rule, Object o)(Code)(Java Doc)
public void setDefaultNextProcessor(Processor nextProcessor)(Code)(Java Doc)
public Processor spawn(int serialNum)(Code)(Java Doc)

www.java2java.com | Contact Us

All other trademarks are property of their respective owners.