Java Doc for TokenizerProperties.java in  » Parser » JTopas » de » susebox » jtopas » Java Source Code / Java DocumentationJava Source Code and Java Documentation

The interface TokenizerProperties declares constants and methods to maintain the characteristics of a Tokenizer , e. g. comments, keywords and special sequences. A TokenizerProperties implementation instance can be used by one or more Tokenizer instances.

A TokenizerProperties object that is used to parse Java or C code including line and column information, would be setup like this:

 TokenizerProperties props = new StandardTokenizerProperties();
 props.setParseFlags(Flags.F_COUNT_LINES);
 props.addLineComment("//");
 props.addBlockComment(TokenizerProperties.DEFAULT_BLOCK_COMMENT_START,
 TokenizerProperties.DEFAULT_BLOCK_COMMENT_END);
 props.addString("\"", "\"", "\\");
 props.addString("'", "'", "\\");
 props.addSpecialSequence(">>");
 props.addSpecialSequence("<<");
 props.addSpecialSequence("++");
 props.addSpecialSequence("--");
 ...
 props.addKeyword("class");
 props.addKeyword("if");
 props.addKeyword("then");
 props.addKeyword("while");
 props.addKeyword("do");
 ...
 

Beside the dedicated method groups for comments, strings, keywords and special sequences (e.g. TokenizerProperties.addKeyword , TokenizerProperties.removeKeyword and TokenizerProperties.keywordExists ) there is a set of generic methods working with TokenizerProperty objects:

• TokenizerProperties.addProperty ,
• TokenizerProperties.removeProperty and
• TokenizerProperties.propertyExists .

When adding a property through one of the add... calls without the flags parameter, the currently active flags of the TokenizerProperties instance will be used for the property.

This interface is separated from the Tokenizer interface mainly to distinguish between the more static information, the actual source of data and the tokenizing process. Especially in multithreaded environments where multible instances of "equal" tokenizers are run at the same time, it saves both memory resources and setup effort to have one TokenizerProperties instance for all Tokenizer instances.

Beside its function as a store for lexical element descriptions, this interface also provides an event mechanism to notify all interested objects about changes (additions, modifications and removals) of such lexical element descriptions.

This interface partly replaces the older de.susebox.java.util.Tokenizer interface which is deprecated.