| org.netbeans.api.lexer |
org.netbeans.api.lexer
The entrance point into Lexer API is
{@link org.netbeans.api.lexer.TokenHierarchy} class with
its static methods that provide its instance for the given input source.
{@link org.netbeans.api.lexer.TokenHierarchy#tokenSequence()}
allows to walk through the tokens that were created for the given input.
{@link org.netbeans.api.lexer.Token} models a single token recognized in the input.
It carries a token identification {@link org.netbeans.api.lexer.TokenId} returned by
{@link org.netbeans.api.lexer.Token#id()}
and token text represented as {@link java.lang.CharSequence}
returned by {@link org.netbeans.api.lexer.Token#text()}.
{@link org.netbeans.api.lexer.Language} represents
all the token ids for a single language.
Each token id may belong to one or more token categories that allow to better operate
tokens of the same type (e.g. keywords or operators).
|
| Java Source File Name | Type | Comment |
| CustomTokenClassTest.java | Class | |
| InputAttributes.java | Class | Supplementary information about particular input
that may be used to influence the lexer's operation.
For example there may be a version of the language
to be used when lexing the input. |
| InputAttributesTest.java | Class | Test several simple lexer impls. |
| Language.java | Class | Language describes a set of token ids
that comprise the given language. |
| LanguagePath.java | Class | Language path describes a complete embedding
of the languages starting from the root (top-level) language
till the most embedded language.
Language path consists of one root language
and zero or more embedded languages.
E.g. |
| LanguagePathTest.java | Class | |
| LanguageTest.java | Class | Test of Language class. |
| PartType.java | enum | Whether
Token represents a complete token
or just a part of a complete token.
A complete token may consist of one start token, zero or more middle tokens
and zero or more end tokens (there may be incomplete token
at the end of input e.g. |
| Token.java | Class | Token describes a lexical element of input text. |
| TokenChange.java | Class | Token change describes modification on one level of a token hierarchy. |
| TokenHierarchy.java | Class | Token hierarchy represents a given input source as a browsable hierarchy of tokens. |
| TokenHierarchyEvent.java | Class | Description of the changes made in a token hierarchy. |
| TokenHierarchyEventType.java | enum | Token hierarchy event type determines the reason
why token hierarchy modification described by
TokenHierarchyEvent happened. |
| TokenHierarchyListener.java | Interface | Listener for chagnes in the token hierarchy. |
| TokenHierarchyTest.java | Class | Test methods of token sequence. |
| TokenId.java | Interface | Identifier of a token (could also be called a token-type).
It is not a token, because it does not contain
the text (also called image) of the token.
Token ids are typically defined as enums by the following pattern:
public enum JavaTokenId implements TokenId {
ERROR(null, "error"),
IDENTIFIER(null, "identifier"),
ABSTRACT("abstract", "keyword"),
...
SEMICOLON(";", "separator"),
...
private final String fixedText; // Used by lexer for production of flyweight tokens
private final String primaryCategory;
JavaTokenId(String fixedText, String primaryCategory) {
this.fixedText = fixedText;
this.primaryCategory = primaryCategory;
}
public String fixedText() {
return fixedText;
}
public String primaryCategory() {
return primaryCategory;
}
}
Token ids can also be generated (e.g. |
| TokenSequence.java | Class | Token sequence allows to iterate between tokens
of a token hierarchy.
Token sequence for top-level language of a token hierarchy
may be obtained by
TokenHierarchy.tokenSequence .
Use of token sequence is a two-step operation:
-
Position token sequence before token that should first be retrieved
(or behind desired token when iterating backwards).
One of the following ways may be used:
|
| TokenSequenceTest.java | Class | Test methods of token sequence. |
| TokenUtilities.java | Class | Various utility methods related to token text. |