| java.lang.Object
 xtc.tree.Visitor
xtc.parser.GrammarVisitor
xtc.parser.Tokenizer
| Tokenizer | | public class Tokenizer extends GrammarVisitor (Code) | | Visitor to recognize token-level productions.
This visitor recognizes the boundary between hierarchical and
lexical syntax:
This visitor recognizes all lexical productions. A production
is lexical if it is text-only or if it is void and only references
other lexical productions (if any). As a result, a lexical
production may not contain parser actions, semantic actions that
reference
CodeGenerator.VALUE , or bindings to
CodeGenerator.VALUE .
This visitor traverses the grammar, starting with its public
productions. When encountering a lexical production it does not
traverse into the production; rather, if the lexical production
also consumes the input, this visitor marks the production as
token-level.
This visitor ensures that all void productions are correctly
annotated with the
Properties.INPUT property, indicating
whether they consume the input.
This visitor does not change a token-level
production's type, so that later parser generator phases can still
distinguish between productions that used to be text-only and that
used to be void. It does however, remove any
Properties.TEXT_ONLY property.
This visitor assumes that the entire grammar is contained in a
single module and that text-only productions have been marked as
such. It may perform faster if the grammar has been annotated with
its real root.
See Also: TextTester
See Also: RootFinder
author:
Robert Grimm
version:
$Revision: 1.11 $ |
| Inner Class :public static class Tester extends Visitor | |
| Tokenizer | | public Tokenizer(Runtime runtime, Analyzer analyzer)(Code) | | Create a new tokenizer.
Parameters:
runtime - The runtime.
Parameters:
analyzer - The analyzer utility. |
| markToken | | public static void markToken(Production p, boolean verbose)(Code) | | Mark the specified production as token-level. This method sets
the specified production's
Properties.TOKEN property and
removes any
Properties.TEXT_ONLY property. It does,
however, not adjust the production's type to
Token.
Parameters:
p - The production. |
|
|