| de.susebox.jtopas.Flags
| Flags | | public interface Flags (Code) | | The interface defines flags that are used by various classes during tokenizing.
A flag can be set in three ways:
See Also: TokenizerProperties
author:
Heiko Blau |
| Field Summary | |
| final public static short | F_ALLOW_NESTED_COMMENTS
Nested block comments are normally not allowed. | | final public static short | F_CASE
General compare operations are case-sensitive, that means 'A' equals 'A'
but not 'a'. | | final public static short | F_COUNT_LINES
Tells a
Tokenizer to count lines and columns. | | final public static short | F_FREE_PATTERN
Treat pattern the same way as whitespaces, separators or special sequences.
Pattern of this type are recognized anywhere outside comments and strings.
They terminate normal token. | | final public static short | F_KEEP_DATA
Set this flag to let a
Tokenizer buffer all data. | | final public static short | F_NO_CASE
When this flag is set globally for a
TokenizerProperties instance
(see
Flags.setParseFlags , input data is generally treated case-insensitive.
Specific properties may still be treated case-sensitive. | | final public static short | F_RETURN_BLOCK_COMMENTS
Return block comments. | | final public static short | F_RETURN_IMAGE_PARTS
By setting this flag for a
TokenizerProperties instance, a
Tokenizer or for a single property, a tokenizer returns not only
the token images but also image parts (see
Token.getImageParts ). | | final public static short | F_RETURN_LINE_COMMENTS
Return line comments. | | final public static short | F_RETURN_SIMPLE_WHITESPACES
Return simple whitespaces. | | final public static short | F_RETURN_WHITESPACES
In many cases, parsers are not interested in whitespaces. | | final public static short | F_SINGLE_LINE_STRING
Per default, strings are all characters between and including a pair of
string start and end sequences, regardless if there are line separators in
between. | | final public static short | F_TOKEN_POS_ONLY
For performance and memory reasons, this flag is used to avoid copy operations
for every token. |
| F_ALLOW_NESTED_COMMENTS | | final public static short F_ALLOW_NESTED_COMMENTS(Code) | | Nested block comments are normally not allowed. This flag changes the
default behaviour.
Implementation note: The flag should be applicable for both
TokenizerProperties and
TokenizerProperty instances. It should not to be used
dynamically (as in versions of JTopas prior to 0.8).
|
| F_CASE | | final public static short F_CASE(Code) | | General compare operations are case-sensitive, that means 'A' equals 'A'
but not 'a'. It is not nessecary to set this flag, since case-sensitive
comparison is the default.
The flag was mainly used in conjunction with
Flags.F_NO_CASE . If
F_NO_CASE is set via
TokenizerProperties.setParseFlags ,
F_CASE can be used for single properties where case-sensitivity
is nessecary inspite of the global case-insensitivity.
If neither F_CASE nor F_NO_CASE is set, F_CASE
is assumed. If both flags are set, F_CASE takes preceedence.
Implementation note: The flag should be applicable for both
TokenizerProperties and
TokenizerProperty instances. It should not to be used
dynamically (
Tokenizer.changeParseFlags ).
TokenizerProperties TokenizerProperty(int, java.lang.String[], java.lang.Object, int, int) |
| F_COUNT_LINES | | final public static short F_COUNT_LINES(Code) | | Tells a
Tokenizer to count lines and columns. The tokenizer may use
java.lang.System.getProperty ("line.separator") to
obtain the end-of-line sequence or accept different line separator sequences
for a better portability: single carriage return (Mac OS), single line feed
(Unix), combination of carriage return and line feed (Windows OS).
Usually, the end-of-line characters '\r' and '\n' are whitespaces. If they
are also part of one or more special sequences or pattern, it is
NOT guaranteed that the line counting mechanism of a
Tokenizer implementation finds these occurences. This is in order to
maintain a good performance, since otherwise there would be a potential huge
amount of unsuccessfull newline scans in these tokens. Consider defining
special sequences for '\r', '\n' and '\r\n' alone and remove them from the
whitespace set, if You cannot live with the described limitation.
Implementation note: The flag should be applicable for
TokenizerProperties and
Tokenizer objects, but not for single
TokenizerProperty
instances. It could also be a dynamic flag that can be switched on and off
during runtime of a tokenizer, although it is generally set before parsing
starts.
|
| F_FREE_PATTERN | | final public static short F_FREE_PATTERN(Code) | | Treat pattern the same way as whitespaces, separators or special sequences.
Pattern of this type are recognized anywhere outside comments and strings.
They terminate normal token. In fact, strings and comments could be
described as free pattern.
Without this flag, pattern are treated in the same way as normal token.
They are preceeded and followed by whitespaces, separators or special sequences.
Implementation note: The flag should be applicable for both
TokenizerProperties and
TokenizerProperty instances. It should not to be used
dynamically.
|
| F_KEEP_DATA | | final public static short F_KEEP_DATA(Code) | | Set this flag to let a
Tokenizer buffer all data. Usually, a tokenizer
will apply a strategie to allocate only a reasonable amount of memory.
Implementation note: The flag should be applicable for
TokenizerProperties and
Tokenizer objects, but not for single
TokenizerProperty
instances. It could also be a dynamic flag that can be switched on and off
during runtime of a tokenizer (
Tokenizer.changeParseFlags ), although
it is generally set before parsing starts.
|
| F_RETURN_BLOCK_COMMENTS | | final public static short F_RETURN_BLOCK_COMMENTS(Code) | | Return block comments. The flag is part of the composite mask
Flags.F_RETURN_WHITESPACES .
Implementation note: The flag should be applicable for TokenizerProperties,
Tokenizer and for single
TokenizerProperty instances. It is
also a dynamic flag that can be switched on and off during runtime of a
tokenizer (Note:: Flags for a single
TokenizerProperty take precedence over other settings).
|
| F_RETURN_LINE_COMMENTS | | final public static short F_RETURN_LINE_COMMENTS(Code) | | Return line comments. The flag is part of the composite mask
Flags.F_RETURN_WHITESPACES .
Implementation note: The flag should be applicable for TokenizerProperties,
Tokenizer and for single
TokenizerProperty instances. It is
also a dynamic flag that can be switched on and off during runtime of a
tokenizer (Note:: Flags for a single
TokenizerProperty take precedence over other settings).
|
| F_SINGLE_LINE_STRING | | final public static short F_SINGLE_LINE_STRING(Code) | | Per default, strings are all characters between and including a pair of
string start and end sequences, regardless if there are line separators in
between. This flag changes that behaviour for the TokenizerProperties
instance in general or for a single string property.
Implementation note: The flag should be applicable for both TokenizerProperties
and
TokenizerProperty instances. It should not to be used
dynamically.
|
| F_TOKEN_POS_ONLY | | final public static short F_TOKEN_POS_ONLY(Code) | | For performance and memory reasons, this flag is used to avoid copy operations
for every token. The token image itself is not returned in a
Token instance, only its position and length in the input stream.
Implementation note: The flag should be applicable for
TokenizerProperties ,
and
TokenizerProperty instances. It should also be a dynamic flag
that can be switched on and off during runtime using
Tokenizer.changeParseFlags .
|
|
|