Java Doc for BulletParser.java in  » Search-Engine » mg4j » it » unimi » dsi » mg4j » util » parser » Java Source Code / Java DocumentationJava Source Code and Java Documentation

The bullet parser has been written with two specific goals in mind: web crawling and targeted data extraction from massive web data sets. To be usable in such environments, a parser must obey a number of restrictions:

• it should avoid excessive object creation (which, for instance, forbids a significant usage of Java strings);
• it should tolerate invalid syntax and recover reasonably; in fact, it should never throw exceptions;
• it should perform actual parsing only on a settable feature subset: there is no reason to parse the attributes of a P element while searching for links;
• it should parse HTML as a regular language, and leave context-free properties (e.g., stack maintenance and repair) to suitably designed callbacks.

Thus, in fact the bullet parser is not a parser. It is a bunch of spaghetti code that analyses a stream of characters pretending that it is an (X)HTML document. It has a very defensive attitude against the stream character it is parsing, but at the same time it is forgiving with all typical (X)HTML mistakes.

The bullet parser is officially StringFree™. MutableStrings are used for internal processing, and Java strings are used only to return attribute values. All internal maps are from fastutil, which helps to accelerate further the parsing process.

HTML data

The bullet parser uses attributes and methods of it.unimi.dsi.mg4j.util.parser.HTMLFactory , it.unimi.dsi.mg4j.util.parser.Element , it.unimi.dsi.mg4j.util.parser.Attribute and it.unimi.dsi.mg4j.util.parser.Entity . Thus, for instance, whenever an element is to be passed around it is one of the shared objects contained in it.unimi.dsi.mg4j.util.parser.Element (e.g., it.unimi.dsi.mg4j.util.parser.Element.BODY ).

Callbacks

The result of the parsing process is the invocation of a callback. The of the bullet parser remembers closely SAX2, but it has some additional methods targeted at (X)HTML, such as Callback.cdata(it.unimi.dsi.mg4j.util.parser.Elementchar[]intint) , which returns characters found in a CDATA section (e.g., a stylesheet).

Each callback must configure the parser, by requesting to perform the analysis and the callbacks it requires. A callback that wants to extract and tokenise text, for instance, will certainly require BulletParser.parseText(boolean) parseText(true) , but not BulletParser.parseTags(boolean) parseTags(true) . On the other hand, a callback wishing to extract links will require to certain attribute types.

A more precise description follows.

Writing callbacks

The first important issue is what has to be required to the parser. A newly created parser does not invoke any callback. It is up to every callback to add features so that it can do its job. Remember that since many callbacks can be , you must always add features, never remove them, and moreover your callbacks must be ready to be invoked with features they did not request (e.g., attribute types added by another callback).

The following parse features may be configured; most of them are just boolean features, a.k.a. flags: unless otherwise specified, by default all flags are set to false (e.g., by the default the parser will not parse tags):

• tags ( BulletParser.parseTags(boolean) method): whether tags should be parsed;
• attributes ( BulletParser.parseAttributes(boolean) and BulletParser.parseAttribute(Attribute) methods) : whether attributes should be parsed (of course, setting this flag is useless if you are not parsing tags); note that setting this flag will just activate the attribute parsing feature, but you must also every attribute whose value you want to obtain.
• text ( BulletParser.parseText(boolean) method): whether text should be parsed; if this flag is set, the parser will call the it.unimi.dsi.parser.callback.Callback.characters(char[]intintboolean) method for every text chunk found.
• CDATA sections ( BulletParser.parseCDATA(boolean) method): whether CDATA sections (stylesheets & scripts) should be parsed; if this flag is set, the parser will call the Callback.cdata(Elementchar[]intint) method for every CDATA section found.

Invoking the parser

After , you just call BulletParser.parse(char[],int,int) .

This method will search for the end of an entity starting at the given offset (the offset must correspond to the ampersand).

Real-world HTML pages often contain hundreds of misplaced ampersands, due to the unfortunate idea of using the ampersand as query separator (please use the comma in new code!).

This method will modify the mutable string s so that all legal occurrences of entities are replaced by the corresponding character.
Parameters:
  s - a mutable string whose entities will be replaced by the corresponding characters.
Parameters:
  entity - a support mutable string used by BulletParser.scanEntity(char[],int,int,boolean,MutableString).
Parameters:
  loose - a parameter that will be passed to BulletParser.scanEntity(char[],int,int,boolean,MutableString).

This method will search for the end of an entity starting at the given offset (the offset must correspond to the ampersand).

Real-world HTML pages often contain hundreds of misplaced ampersands, due to the unfortunate idea of using the ampersand as query separator (please use the comma in new code!). All such ampersand should be specified as &. If named entities are delimited using a transition from alphabetical to non-alphabetical characters, we can easily get false positives. If the parameter loose is false, named entities can be delimited only by whitespace or by a comma.
Parameters:
  a - a character array containing the entity.
Parameters:
  offset - the offset at which the entity starts (the offset must point at the ampersand).
Parameters:
  length - an upper bound to the maximum returned position.
Parameters:
  loose - if true, named entities can be terminated by any non-alphabetical character (instead of whitespace or comma).
Parameters:
  entity - a support mutable string used to query ParsingFactory.getEntity(MutableString). the position of the last character of the entity, or -1 if no entity was found.