de.susebox.jtopas

Base class for Tokenizer implementations.

The class AbstractTokenizerProperties provides the skeleton for implementations of the TokenizerProperties interface.

Extension of the TokenizerSource interface and the java.lang.CharSequence interface that was introduced in J2SE 1.4.

• Globally for a TokenizerProperties object: The setting affects all Tokenizer instances that share this TokenizerProperties object as well as the TokenizerProperty objects registered in this TokenizerProperties that do haven't set the flag locally.
• Separately for a Tokenizer (see Tokenizer.changeParseFlags : A single Tokenizer will behave differently to the setting in the used TokenizerProperties object, but still follow the setting for a single TokenizerProperty object.

This is the test suite for the java package tree.

This implementation of the TokenizerSource interface uses the JDK java.io.Reader class to realize the requested functionality.

This is the mainstream Tokenizer .

The class StandardTokenizerProperties provides a simple implementation of the TokenizerProperties interface for use in most situations.

Note that this class takes advantage of JTopas features that use Java 1.4 or higher.

The class contains a number of test cases that are supposed to be difficult to handle for a Tokenizer , e.g.

This unit test checks the embedded-tokenizer feature of the class StandardTokenizer . With this technique it is possible to parse multipart documents like HTML with embedded CSS and script parts, Java and javadoc comments etc.

This test suite works with a test configuration file.

This test suite generates a huge file with a common mix of comments, special sequences, keywords, separators etc.

This class tests Tokenizer implementations in a multithreaded environment.

This class tests the implementations of TokenizerProperties .

The class contains a number of test cases related to the pattern matching facility in a Tokenizer .

This test suite works with a test configuration file.

This class tests the input data access of a Tokenizer and the setting of the read position.

This test suite checks memory usage, registration end other things related to cleanup operations of a Tokenizer .

This class tests the behaviour of a Tokenizer regarding the flags set in the backing TokenizerProperties and the registered TokenizerProperty objects.

This class tests the implementations of TokenizerProperties .

The class contains a number of test cases that are supposed to be difficult to handle for a Tokenizer , e.g.

Instances of this class are created by the classes implementing the Tokenizer interface.

The interface Tokenizer contains setup methods, parse operations and other getter and setter methods for a tokenizer.

Wrapper exception for all the problems that may occur while parsing.

The interface TokenizerProperties declares constants and methods to maintain the characteristics of a Tokenizer , e.

This class is mainly used by TokenizerProperties implementations to return java.util.Iterator objects of their various properties (keywords, special sequences etc.).

The class TokenizerPropertyEvent describes changes in TokenizerProperties objects.

The interface TokenizerPropertyListener is implemented by classes that are interested in changes in TokenizerProperties objects.

This interface describes the data source for a Tokenizer .