| java.lang.Object
 org.antlr.runtime.BaseRecognizer
org.antlr.runtime.Lexer
All known Subclasses: org.antlr.codegen.ActionTranslatorLexer, org.antlr.tool.ActionAnalysisLexer,
| Lexer | | abstract public class Lexer extends BaseRecognizer implements TokenSource(Code) | | A lexer is recognizer that draws input symbols from a character stream.
lexer grammars result in a subclass of this object. A Lexer object
uses simplified match() and error recovery mechanisms in the interest
of speed.
|
| Field Summary | |
| protected int | channel
| | protected CharStream | input
| | protected String | text
You can set the text for the current token to override what is in
the input char buffer. | | protected Token | token
The goal of all lexer rules/methods is to create a token object.
This is an instance variable as multiple rules may collaborate to
create a single token. | | protected int | tokenStartCharIndex
What character index in the stream did the current token start at?
Needed, for example, to get the text for current token. | | protected int | tokenStartCharPositionInLine
| | protected int | tokenStartLine
| | protected int | type
|
| channel | | protected int channel(Code) | | The channel number for the current token
|
| input | | protected CharStream input(Code) | | Where is the lexer drawing characters from?
|
| text | | protected String text(Code) | | You can set the text for the current token to override what is in
the input char buffer. Use setText() or can set this instance var.
|
| token | | protected Token token(Code) | | The goal of all lexer rules/methods is to create a token object.
This is an instance variable as multiple rules may collaborate to
create a single token. nextToken will return this object after
matching lexer rule(s). If you subclass to allow multiple token
emissions, then set this to the last token to be matched or
something nonnull so that the auto token emit mechanism will not
emit another token.
|
| tokenStartCharIndex | | protected int tokenStartCharIndex(Code) | | What character index in the stream did the current token start at?
Needed, for example, to get the text for current token. Set at
the start of nextToken.
|
| tokenStartCharPositionInLine | | protected int tokenStartCharPositionInLine(Code) | | The character position of first character within the line
|
| tokenStartLine | | protected int tokenStartLine(Code) | | The line on which the first character of the token resides
|
| type | | protected int type(Code) | | The token type for the current token
|
| emit | | public void emit(Token token)(Code) | | Currently does not support multiple emits per nextToken invocation
for efficiency reasons. Subclass and override this method and
nextToken (to push tokens into a list and pull from that list rather
than a single variable as this implementation does).
|
| emit | | public Token emit()(Code) | | The standard method called to automatically emit a token at the
outermost lexical rule. The token object should point into the
char buffer start..stop. If there is a text override in 'text',
use that to set the token's text. Override this method to emit
custom Token objects.
|
| getCharErrorDisplay | | public String getCharErrorDisplay(int c)(Code) | | |
| getCharIndex | | public int getCharIndex()(Code) | | What is the index of the current character of lookahead?
|
| getCharPositionInLine | | public int getCharPositionInLine()(Code) | | |
| getLine | | public int getLine()(Code) | | |
| getText | | public String getText()(Code) | | Return the text matched so far for the current token or any
text override.
|
| mTokens | | abstract public void mTokens() throws RecognitionException(Code) | | This is the lexer entry point that sets instance var 'token'
|
| matchAny | | public void matchAny()(Code) | | |
| nextToken | | public Token nextToken()(Code) | | Return a token from this source; i.e., match a token on the char
stream.
|
| recover | | public void recover(RecognitionException re)(Code) | | Lexers can normally match any char in it's vocabulary after matching
a token, so do the easy thing and just kill a character and hope
it all works out. You can instead use the rule invocation stack
to do sophisticated error recovery if you are in a fragment rule.
|
| reset | | public void reset()(Code) | | |
| setCharStream | | public void setCharStream(CharStream input)(Code) | | Set the char stream and reset the lexer
|
| setText | | public void setText(String text)(Code) | | Set the complete text of this token; it wipes any previous
changes to the text.
|
| skip | | public void skip()(Code) | | Instruct the lexer to skip creating a token for current lexer rule
and look for another token. nextToken() knows to keep looking when
a lexer rule finishes with token set to SKIP_TOKEN. Recall that
if token==null at end of any token rule, it creates one for you
and emits it.
|
| traceIn | | public void traceIn(String ruleName, int ruleIndex)(Code) | | |
| traceOut | | public void traceOut(String ruleName, int ruleIndex)(Code) | | |
| Methods inherited from org.antlr.runtime.BaseRecognizer | public boolean alreadyParsedRule(IntStream input, int ruleIndex)(Code)(Java Doc)
public void beginResync()(Code)(Java Doc)
protected BitSet combineFollows(boolean exact)(Code)(Java Doc)
protected BitSet computeContextSensitiveRuleFOLLOW()(Code)(Java Doc)
protected BitSet computeErrorRecoverySet()(Code)(Java Doc)
public void consumeUntil(IntStream input, int tokenType)(Code)(Java Doc)
public void consumeUntil(IntStream input, BitSet set)(Code)(Java Doc)
public void displayRecognitionError(String[] tokenNames, RecognitionException e)(Code)(Java Doc)
public void emitErrorMessage(String msg)(Code)(Java Doc)
public void endResync()(Code)(Java Doc)
public int getBacktrackingLevel()(Code)(Java Doc)
public String getErrorHeader(RecognitionException e)(Code)(Java Doc)
public String getErrorMessage(RecognitionException e, String[] tokenNames)(Code)(Java Doc)
public String getGrammarFileName()(Code)(Java Doc)
public List getRuleInvocationStack()(Code)(Java Doc)
public static List getRuleInvocationStack(Throwable e, String recognizerClassName)(Code)(Java Doc)
public int getRuleMemoization(int ruleIndex, int ruleStartIndex)(Code)(Java Doc)
public int getRuleMemoizationCacheSize()(Code)(Java Doc)
public String getTokenErrorDisplay(Token t)(Code)(Java Doc)
public String[] getTokenNames()(Code)(Java Doc)
public void match(IntStream input, int ttype, BitSet follow) throws RecognitionException(Code)(Java Doc)
public void matchAny(IntStream input)(Code)(Java Doc)
public void memoize(IntStream input, int ruleIndex, int ruleStartIndex)(Code)(Java Doc)
protected void mismatch(IntStream input, int ttype, BitSet follow) throws RecognitionException(Code)(Java Doc)
protected void pushFollow(BitSet fset)(Code)(Java Doc)
public void recover(IntStream input, RecognitionException re)(Code)(Java Doc)
protected boolean recoverFromMismatchedElement(IntStream input, RecognitionException e, BitSet follow)(Code)(Java Doc)
public void recoverFromMismatchedSet(IntStream input, RecognitionException e, BitSet follow) throws RecognitionException(Code)(Java Doc)
public void recoverFromMismatchedToken(IntStream input, RecognitionException e, int ttype, BitSet follow) throws RecognitionException(Code)(Java Doc)
public void reportError(RecognitionException e)(Code)(Java Doc)
public void reset()(Code)(Java Doc)
public List toStrings(List tokens)(Code)(Java Doc)
public void traceIn(String ruleName, int ruleIndex, Object inputSymbol)(Code)(Java Doc)
public void traceOut(String ruleName, int ruleIndex, Object inputSymbol)(Code)(Java Doc)
|
|
|