| java.lang.Object
 fri.patterns.interpreter.parsergenerator.lexer.LexerImpl
| LexerImpl | | public class LexerImpl implements Lexer,StrategyFactoryMethod,Serializable(Code) | | This Lexer must be created using LexerBuilder. It knows token and ignored terminals.
To get this Lexer working the setTerminals() call must be called at least once.
When using the Lexer standalone, the client must do this, else the Parser will
call that method.
This lexer can be reused, but it can not be loaded with other syntaxes after it
has been built for one.
author:
(c) 2002, Fritz Ritzberger |
| Constructor Summary | |
| public | LexerImpl(List ignoredSymbols, Map charConsumers)
Creates a Lexer from token- and ignored symbols, and a map of character consumers (built by LexerBuilder).
Parameters:
ignoredSymbols - list of Strings containing ignored symbols to scan. | | protected | LexerImpl()
Do-nothing constructor for subclasses (currently unused). |
| Method Summary | |
| public void | addTokenListener(Lexer.TokenListener tokenListener)
Implements Lexer. | | public void | clear()
Implements Lexer: Does nothing as no states are stored. | | protected Token | createToken(String tokenIdentifier, ResultTree result, LexerSemantic lexerSemantic)
Token factory method. | | protected Token | createToken(String tokenIdentifier, String text, Token.Range range)
Token factory method. | | public void | dump(PrintStream out)
Outputs current and previous line, with line numbers. | | public int | getColumn()
Returns the position within the current line, 0-n. | | public int | getLine()
Returns the number of the current line, 1-n. | | public String | getLineText()
Returns the current line, as far as read. | | public Token | getNextToken(LexerSemantic lexerSemantic)
This is an optional functionality of Lexer. | | public Token | getNextToken(Map expectedTokenSymbols)
Implements Lexer: returns the next token from input, or EPSILON when no more input.
This is called by the Parser to get the next syntax token from input.
When returned token.symbol is null, no input could be recognized (ERROR).
Parameters:
expectedTokenSymbols - contains the expected String token symbols (in keys),can be null when no Parser drives this Lexer. | | public int | getOffset()
Returns the offset read so far from input. | | public boolean | lex(LexerSemantic lexerSemantic)
This is an optional functionality of Lexer. | | protected void | loopResultTree(ResultTree result, LexerSemantic lexerSemantic)
After top-down lexing this method is called to dispatch all results. | | public Strategy | newStrategy()
Implements StrategyFactoryMethod. | | public void | removeTokenListener(Lexer.TokenListener tokenListener)
Implements Lexer. | | public void | setCompeteForLongestInput(boolean competeForLongestInput)
When false, the sort order (significance) of scan items without fixed start character decide what token is returned. | | public void | setDebug(boolean debug)
Implements Lexer: Set debug on to output information about scanned tokens. | | public void | setInput(Object text)
Implements Lexer: set the input to be scanned. | | public void | setTerminals(List terminals)
Implements Lexer: Parser call to pass all tokens symbols (all enclosed in `backquote`) and literals ("xyz"). |
| LexerImpl | | public LexerImpl(List ignoredSymbols, Map charConsumers)(Code) | | Creates a Lexer from token- and ignored symbols, and a map of character consumers (built by LexerBuilder).
Parameters:
ignoredSymbols - list of Strings containing ignored symbols to scan. These are NOT enclosed in `backquotes` like tokens.
Parameters:
charConsumers - map with key = nonterminal and value = Consumer. |
| LexerImpl | | protected LexerImpl()(Code) | | Do-nothing constructor for subclasses (currently unused).
|
| addTokenListener | | public void addTokenListener(Lexer.TokenListener tokenListener)(Code) | | Implements Lexer. Adds the passed token listener to listener list.
|
| clear | | public void clear()(Code) | | Implements Lexer: Does nothing as no states are stored. This Lexer can not be loaded with new syntaxes.
|
| createToken | | protected Token createToken(String tokenIdentifier, ResultTree result, LexerSemantic lexerSemantic)(Code) | | Token factory method. Can be overridden to access the lexing ResultTree. Delegates to createToken(tokenIdentifier, text, range).
|
| createToken | | protected Token createToken(String tokenIdentifier, String text, Token.Range range)(Code) | | Token factory method. Can be overridden to convert token.text to some Java object.
|
| dump | | public void dump(PrintStream out)(Code) | | Outputs current and previous line, with line numbers. Call this on ERROR.
|
| getColumn | | public int getColumn()(Code) | | Returns the position within the current line, 0-n.
|
| getLine | | public int getLine()(Code) | | Returns the number of the current line, 1-n.
|
| getLineText | | public String getLineText()(Code) | | Returns the current line, as far as read.
|
| getNextToken | | public Token getNextToken(LexerSemantic lexerSemantic) throws IOException(Code) | | This is an optional functionality of Lexer. It is NOT called by the Parser.
It can be used for heuristic reading from an input (not knowing if there is more input
after the token was read).
The passed LexerSemantic will receive every matched rule (top-down) together with
its ResultTree. See lex() for details.
Parameters:
lexerSemantic - the LexerSemantic to be called with every evaluated Rule and its lexing ResultTree. a Token with a terminal symbol and its instance text, or a Token with null symbol for error. |
| getNextToken | | public Token getNextToken(Map expectedTokenSymbols) throws IOException(Code) | | Implements Lexer: returns the next token from input, or EPSILON when no more input.
This is called by the Parser to get the next syntax token from input.
When returned token.symbol is null, no input could be recognized (ERROR).
Parameters:
expectedTokenSymbols - contains the expected String token symbols (in keys),can be null when no Parser drives this Lexer. a Token with a terminal symbol and its instance text, or a Token with null symbol for error. |
| getOffset | | public int getOffset()(Code) | | Returns the offset read so far from input. This is an absolute offset, including newlines.
|
| lex | | public boolean lex(LexerSemantic lexerSemantic) throws IOException(Code) | | This is an optional functionality of Lexer. It is NOT called by the Parser.
It can be used to run a standalone Lexer with a LexerSemantic, processing a ready-scanned
syntax tree. Other than with Parser Semantic no value stack is available for LexerSemantic,
and all input will have been read when LexerSemantic is called with the built syntax tree.
The passed LexerSemantic will receive every matched rule (top-down) together with
its results ResultTree, containing the range within input.
ResultTree can be converted to text by calling resultTree.toString().
This method evaluates the input using end-of-input like a parser, that means it returns
false if the input was either syntactically incorrect or EOF was not received when all rules
have been evaluated.
MIND: This method does not call any TokenListener, as the LexerSemantic is expected to
dispatch results!
Parameters:
lexerSemantic - the LexerSemantic to be called with every evaluated Rule and its lexing ResultTree. true when lexer succeeded (input was syntactically ok), else false. |
| loopResultTree | | protected void loopResultTree(ResultTree result, LexerSemantic lexerSemantic)(Code) | | After top-down lexing this method is called to dispatch all results. Can be overridden to change dispatch logic.
This method calls itself recursively with all result tree children. Nonterminals starting with "_" are ignored
by default, as this marks artificial rules.
Parameters:
result - lexer result, returns text on getText().
Parameters:
semantic - semantic that dispatches the lexer results. a Token with the range and return of the Semantic call for this Rule/ResultTree. |
| newStrategy | | public Strategy newStrategy()(Code) | | Implements StrategyFactoryMethod. To be overridden to create a derived Strategy implementation.
|
| removeTokenListener | | public void removeTokenListener(Lexer.TokenListener tokenListener)(Code) | | Implements Lexer. Removes the passed token listener from listener list.
|
| setCompeteForLongestInput | | public void setCompeteForLongestInput(boolean competeForLongestInput)(Code) | | When false, the sort order (significance) of scan items without fixed start character decide what token is returned.
When true (default), the scan item (without fixed start character) that scnas longest wins.
|
| setDebug | | public void setDebug(boolean debug)(Code) | | Implements Lexer: Set debug on to output information about scanned tokens.
|
| setInput | | public void setInput(Object text) throws IOException(Code) | | Implements Lexer: set the input to be scanned. If text is InputStream, no Reader
will be used (characters will not be converted).
Parameters:
text - text to scan, as String, StringBuffer, File, InputStream, Reader. |
| setTerminals | | public void setTerminals(List terminals)(Code) | | Implements Lexer: Parser call to pass all tokens symbols (all enclosed in `backquote`) and literals ("xyz").
Parameters:
terminals - List of String containing "literals" and `lexertokens`. |
|
|