Java Doc for Tokenizer.java in  » Parser » JTopas » de » susebox » jtopas » Java Source Code / Java DocumentationJava Source Code and Java Documentation

The interface Tokenizer contains setup methods, parse operations and other getter and setter methods for a tokenizer. A tokenizer splits a stream of input data into various units like whitespaces, comments, keywords etc. These units are the tokens that are reflected in the Token class of the de.susebox.jtopas package.

A Tokenizer is configured using a TokenizerProperties object that contains declarations for whitespaces, separators, comments, keywords, special sequences and patterns. It is designed to enable a common approach for parsing texts like program code, annotated documents like HTML and so on.

To detect links in an HTML document, a tokenizer would be invoked like that (see StandardTokenizerProperties and StandardTokenizer for the classes mentioned here):

 Vector               links     = new Vector();
 FileReader           reader    = new FileReader("index.html");
 TokenizerProperties  props     = new StandardTokenizerProperties();
 Tokenizer            tokenizer = new StandardTokenizer();
 Token                token;
 props.setParseFlags(Tokenizer.F_NO_CASE);
 props.setSeparators("=");
 props.addString("\"", "\"", "\\");
 props.addBlockComment(">", "<");
 props.addKeyword("HREF");
 tokenizer.setTokenizerProperties(props);
 tokenizer.setSource(new ReaderSource(reader));
 try {
 while (tokenizer.hasMoreToken()) {
 token = tokenizer.nextToken();
 if (token.getType() == Token.KEYWORD) {
 tokenizer.nextToken();               // should be the '=' character
 links.addElement(tokenizer.next());
 }
 }
 } finally {
 tokenizer.close();
 reader.close();
 }
 

To extract the contents of a HTML file, one would write:

 StringBuffer         contents  = new StringBuffer(4096);
 FileReader           reader    = new FileReader("index.html");
 TokenizerProperties  props     = new StandardTokenizerProperties();
 Tokenizer            tokenizer = new StandardTokenizer();
 Token                token;
 props.setParseFlags(Tokenizer.F_NO_CASE);
 props.addBlockComment(">", "<");
 props.addBlockComment(">HEAD<", ">/HEAD<");
 props.addBlockComment(">!--;", "--<");
 tokenizer.setTokenizerProperties(props);
 tokenizer.setSource(new ReaderSource(reader));
 try {
 while (tokenizer.hasMoreToken()) {
 token = tokenizer.nextToken();
 if (token.getType() != Token.BLOCK_COMMENT) {
 contents.append(token.getToken());
 }
 }
 } finally {
 tokenizer.close();
 reader.close();
 }
 

Parsing (tokenizing) is done on a well defined priority scheme. See Tokenizer.nextToken for details.

NOTE: if a character sequence is registered for two categories of tokenizer properties (e.g. as a line comments starting sequence as well as a special sequence), the category with the highest priority wins (e.g. if the metioned sequence is found, it is interpreted as a line comment).

The tokenizer interface is clearly designed for "readable" data, say ASCII- or UNICODE data. Parsing binary data has other characteristics that do not necessarily fit in a scheme of comments, keywords, strings, identifiers and operators.

Note that the interface has no methods that handle stream data sources. This is left to the implementations that may have quite different data sources, e. g. java.io.InputStreamReader , database queries, string arrays etc. The interface TokenizerSource serves as an abstraction of such widely varying data sources.

The Tokenizer interface partly replaces the older de.susebox.java.util.Tokenizer interface which is deprecated.

• TokenizerProperties.F_RETURN_WHITESPACES and its sub-flags
• TokenizerProperties.F_TOKEN_POS_ONLY

• TokenizerProperties.F_KEEP_DATA
• TokenizerProperties.F_COUNT_LINES

• Carriage Return (ASCII 13, '\r'). This EOL is used on Apple Macintosh
• Linefeed (ASCII 10, '\n'). This is the UNIX EOL character.
• Carriage Return + Linefeed ("\r\n"). This is used on MS Windows systems.

1. Check for an end-of-file condition. If there is such a condition then return it.
2. Try to collect a sequence of whitespaces. If such a sequence can be found return if the flag F_RETURN_WHITESPACES is set, or skip these whitespaces.
3. Check the next characters against all known pattern. A pattern is usually a regular expression that is used by java.util.regex.Pattern . But implementations of de.susebox.jtopas.spi.PatternHandler may use other pattern syntaxes. Note that pattern are not recognized within "normal" text (see below for a more precise description).
4. Check the next characters against all known line and block comments. If a line or block comment starting sequence matches, return if the flag F_RETURN_WHITESPACES is set, or skip the comment. If comments are returned they include their starting and ending sequences (newline in case of a line comment).
5. Check the next characters against all known string starting sequences. If a string begin could be identified return the string until and including the closing sequence.
6. Check the next characters against all known special sequences. Especially, find the longest possible match. If a special sequence could be identified then return it.
7. Check for ordinary separators. If one could be found return it.
8. Check the next characters against all known keywords. If a keyword could be identified then return it.
9. Return the text portion until the next whitespace, comment, special sequence or separator. Note that pattern are not recognized within "normal" text. A pattern match has therefore always a whitespace, comment, special sequence, separator or another pattern match in front of it or starts at position 0 of the data.

 Token token1 = tokenizer.nextToken();
 tokenizer.setReadPositionAbsolute(tokenizer.getReadPosition() - token1.getLength());
 Token token2 = tokenizer.nextToken();
 assert(token1.equals(token2));
 

• de.susebox.jtopas.spi.WhitespaceHandler ,
• de.susebox.jtopas.spi.SeparatorHandler ,
• de.susebox.jtopas.spi.SequenceHandler ,
• de.susebox.jtopas.spi.KeywordHandler and
• de.susebox.jtopas.spi.PatternHandler