Java Doc for ExtractorHTML.java in  » Web-Crawler » heritrix » org » archive » crawler » extractor » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation 1.  6.0 JDK Core 2.  6.0 JDK Modules 3.  6.0 JDK Modules com.sun 4.  6.0 JDK Modules com.sun.java 5.  6.0 JDK Modules sun 6.  6.0 JDK Platform 7.  Ajax 8.  Apache Harmony Java SE 9.  Aspect oriented 10.  Authentication Authorization 11.  Blogger System 12.  Build 13.  Byte Code 14.  Cache 15.  Chart 16.  Chat 17.  Code Analyzer 18.  Collaboration 19.  Content Management System 20.  Database Client 21.  Database DBMS 22.  Database JDBC Connection Pool 23.  Database ORM 24.  Development 25.  EJB Server geronimo 26.  EJB Server GlassFish 27.  EJB Server JBoss 4.2.1 28.  EJB Server resin 3.1.5 29.  ERP CRM Financial 30.  ESB 31.  Forum 32.  GIS 33.  Graphic Library 34.  Groupware 35.  HTML Parser 36.  IDE 37.  IDE Eclipse 38.  IDE Netbeans 39.  Installer 40.  Internationalization Localization 41.  Inversion of Control 42.  Issue Tracking 43.  J2EE 44.  JBoss 45.  JMS 46.  JMX 47.  Library 48.  Mail Clients 49.  Net 50.  Parser 51.  PDF 52.  Portal 53.  Profiler 54.  Project Management 55.  Report 56.  RSS RDF 57.  Rule Engine 58.  Science 59.  Scripting 60.  Search Engine 61.  Security 62.  Sevlet Container 63.  Source Control 64.  Swing Library 65.  Template Engine 66.  Test Coverage 67.  Testing 68.  UML 69.  Web Crawler 70.  Web Framework 71.  Web Mail 72.  Web Server 73.  Web Services 74.  Web Services apache cxf 2.0.1 75.  Web Services AXIS2 76.  Wiki Engine 77.  Workflow Engines 78.  XML 79.  XML UI Java Java Tutorial Java Open Source Jar File Download Java Articles Java Products Java by API Photoshop Tutorials Maya Tutorials Flash Tutorials 3ds-Max Tutorials Illustrator Tutorials GIMP Tutorials C# / C Sharp C# / CSharp Tutorial C# / CSharp Open Source ASP.Net ASP.NET Tutorial JavaScript DHTML JavaScript Tutorial JavaScript Reference HTML / CSS HTML CSS Reference C / ANSI-C C Tutorial C++ C++ Tutorial Ruby PHP Python Python Tutorial Python Open Source SQL Server / T-SQL SQL Server / T-SQL Tutorial Oracle PL / SQL Oracle PL/SQL Tutorial PostgreSQL SQL / MySQL MySQL Tutorial VB.Net VB.Net Tutorial Flash / Flex / ActionScript VBA / Excel / Access / Word XML XML Tutorial Microsoft Office PowerPoint 2007 Tutorial Microsoft Office Excel 2007 Tutorial Microsoft Office Word 2007 Tutorial

Java Source Code / Java Documentation » Web Crawler » heritrix » org.archive.crawler.extractor   Source Cross Reference   Class Diagram   Java Document (Java Doc)   org.archive.crawler.extractor.Extractor    org.archive.crawler.extractor.ExtractorHTML All known Subclasses:   org.archive.crawler.extractor.AggressiveExtractorHTML,  org.archive.crawler.extractor.JerichoExtractorHTML, ExtractorHTML public class ExtractorHTML extends Extractor implements CoreAttributeConstants(Code) Basic link-extraction, from an HTML content-body, using regular expressions. author:    gojomo Field Summary final static  String APPLET      final public static  String ATTR_EXTRACT_JAVASCRIPT      final public static  String ATTR_IGNORE_FORM_ACTION_URLS      final public static  String ATTR_IGNORE_UNEXPECTED_HTML      final public static  String ATTR_OVERLY_EAGER_LINK_DETECTION      final public static  String ATTR_TREAT_FRAMES_AS_EMBED_LINKS      final static  String BASE      final static  String CLASSEXT      final static  String EACH_ATTRIBUTE_EXTRACTOR      final static  String FRAME      final static  String IFRAME      final static  String JAVASCRIPT      final static  String LIKELY_URI_PATH      final static  String LINK      final static  int MAX_ATTR_VAL_LENGTH      final static  String NON_HTML_PATH_EXTENSION      final static  String RELEVANT_TAG_EXTRACTOR      final static  String WHITESPACE      protected  long numberOfCURIsHandled      protected  long numberOfLinksExtracted      Constructor Summary public   ExtractorHTML(String name)      public   ExtractorHTML(String name, String description)      Method Summary public  void extract(CrawlURI curi)       void extract(CrawlURI curi, CharSequence cs)      Run extractor. This method is package visible to ease testing. Parameters:   curi - CrawlURI we're processing. Parameters:   cs - Sequence from underlying ReplayCharSequence. protected  boolean isHtmlExpectedHere(CrawlURI curi)      Test whether this HTML is so unexpected (eg in place of a GIF URI) that it shouldn't be scanned for links. Parameters:   curi - CrawlURI to examine. final protected  void processEmbed(CrawlURI curi, CharSequence value, CharSequence context)      protected  void processEmbed(CrawlURI curi, CharSequence value, CharSequence context, char hopType)      protected  void processGeneralTag(CrawlURI curi, CharSequence element, CharSequence cs)      protected  void processLink(CrawlURI curi, CharSequence value, CharSequence context)      Handle generic HREF cases. protected  boolean processMeta(CrawlURI curi, CharSequence cs)      Process metadata tags. Parameters:   curi - CrawlURI we're processing. Parameters:   cs - Sequence from underlying ReplayCharSequence. protected  void processScript(CrawlURI curi, CharSequence sequence, int endOfOpenTag)      protected  void processScriptCode(CrawlURI curi, CharSequence cs)      Extract the (java)script source in the given CharSequence. protected  void processStyle(CrawlURI curi, CharSequence sequence, int endOfOpenTag)      Process style text. Parameters:   curi - CrawlURI we're processing. Parameters:   sequence - Sequence from underlying ReplayCharSequence. public  String report()      Field Detail APPLET final static String APPLET(Code) ATTR_EXTRACT_JAVASCRIPT final public static String ATTR_EXTRACT_JAVASCRIPT(Code) whether to try finding links in Javscript; default true ATTR_IGNORE_FORM_ACTION_URLS final public static String ATTR_IGNORE_FORM_ACTION_URLS(Code) ATTR_IGNORE_UNEXPECTED_HTML final public static String ATTR_IGNORE_UNEXPECTED_HTML(Code) ATTR_OVERLY_EAGER_LINK_DETECTION final public static String ATTR_OVERLY_EAGER_LINK_DETECTION(Code) ATTR_TREAT_FRAMES_AS_EMBED_LINKS final public static String ATTR_TREAT_FRAMES_AS_EMBED_LINKS(Code) BASE final static String BASE(Code) CLASSEXT final static String CLASSEXT(Code) EACH_ATTRIBUTE_EXTRACTOR final static String EACH_ATTRIBUTE_EXTRACTOR(Code) FRAME final static String FRAME(Code) IFRAME final static String IFRAME(Code) JAVASCRIPT final static String JAVASCRIPT(Code) LIKELY_URI_PATH final static String LIKELY_URI_PATH(Code) LINK final static String LINK(Code) MAX_ATTR_VAL_LENGTH final static int MAX_ATTR_VAL_LENGTH(Code) NON_HTML_PATH_EXTENSION final static String NON_HTML_PATH_EXTENSION(Code) RELEVANT_TAG_EXTRACTOR final static String RELEVANT_TAG_EXTRACTOR(Code) WHITESPACE final static String WHITESPACE(Code) numberOfCURIsHandled protected long numberOfCURIsHandled(Code) numberOfLinksExtracted protected long numberOfLinksExtracted(Code) Constructor Detail ExtractorHTML public ExtractorHTML(String name)(Code) ExtractorHTML public ExtractorHTML(String name, String description)(Code) Method Detail extract public void extract(CrawlURI curi)(Code) extract void extract(CrawlURI curi, CharSequence cs)(Code) Run extractor. This method is package visible to ease testing. Parameters:   curi - CrawlURI we're processing. Parameters:   cs - Sequence from underlying ReplayCharSequence. Thisis TRANSIENT data. Make a copy if you want the data to live outsideof this extractors' lifetime. isHtmlExpectedHere protected boolean isHtmlExpectedHere(CrawlURI curi) throws URIException(Code) Test whether this HTML is so unexpected (eg in place of a GIF URI) that it shouldn't be scanned for links. Parameters:   curi - CrawlURI to examine. True if HTML is acceptable/expected here throws:   URIException - processEmbed final protected void processEmbed(CrawlURI curi, CharSequence value, CharSequence context)(Code) processEmbed protected void processEmbed(CrawlURI curi, CharSequence value, CharSequence context, char hopType)(Code) processGeneralTag protected void processGeneralTag(CrawlURI curi, CharSequence element, CharSequence cs)(Code) processLink protected void processLink(CrawlURI curi, CharSequence value, CharSequence context)(Code) Handle generic HREF cases. Parameters:   curi - Parameters:   value - Parameters:   context - processMeta protected boolean processMeta(CrawlURI curi, CharSequence cs)(Code) Process metadata tags. Parameters:   curi - CrawlURI we're processing. Parameters:   cs - Sequence from underlying ReplayCharSequence. Thisis TRANSIENT data. Make a copy if you want the data to live outsideof this extractors' lifetime. True robots exclusion metatag. processScript protected void processScript(CrawlURI curi, CharSequence sequence, int endOfOpenTag)(Code) processScriptCode protected void processScriptCode(CrawlURI curi, CharSequence cs)(Code) Extract the (java)script source in the given CharSequence. Parameters:   curi - source CrawlURI Parameters:   cs - CharSequence of javascript code processStyle protected void processStyle(CrawlURI curi, CharSequence sequence, int endOfOpenTag)(Code) Process style text. Parameters:   curi - CrawlURI we're processing. Parameters:   sequence - Sequence from underlying ReplayCharSequence. Thisis TRANSIENT data. Make a copy if you want the data to live outsideof this extractors' lifetime. Parameters:   endOfOpenTag - report public String report()(Code) Methods inherited from org.archive.crawler.extractor.Extractor abstract protected void extract(CrawlURI curi)(Code)(Java Doc) public void innerProcess(CrawlURI curi)(Code)(Java Doc)