| java.lang.Object
 org.antlr.runtime.CommonTokenStream
All known Subclasses: org.antlr.runtime.TokenRewriteStream,
| CommonTokenStream | | public class CommonTokenStream implements TokenStream(Code) | | The most common stream of tokens is one where every token is buffered up
and tokens are prefiltered for a certain channel (the parser will only
see these tokens and cannot change the filter channel number during the
parse).
TODO: how to access the full token stream? How to track all tokens matched per rule?
|
| Method Summary | |
| public int | LA(int i)
| | protected Token | LB(int k)
| | public Token | LT(int k)
Get the ith token from the current position 1..n where k=1 is the
first symbol of lookahead. | | public void | consume()
Move the input pointer to the next incoming token. | | public void | discardOffChannelTokens(boolean discardOffChannelTokens)
| | public void | discardTokenType(int ttype)
| | protected void | fillBuffer()
Load all tokens from the token source and put in tokens. | | public Token | get(int i)
Return absolute token i; ignore which channel the tokens are on;
that is, count all tokens not just on-channel tokens. | | public TokenSource | getTokenSource()
| | public List | getTokens()
| | public List | getTokens(int start, int stop)
| | public List | getTokens(int start, int stop, BitSet types)
Given a start and stop index, return a List of all tokens in
the token type BitSet. | | public List | getTokens(int start, int stop, List types)
| | public List | getTokens(int start, int stop, int ttype)
| | public int | index()
| | public int | mark()
| | public void | release(int marker)
| | public void | rewind(int marker)
| | public void | rewind()
| | public void | seek(int index)
| | public void | setTokenSource(TokenSource tokenSource)
Reset this token stream by setting its token source. | | public void | setTokenTypeChannel(int ttype, int channel)
A simple filter mechanism whereby you can tell this token stream
to force all tokens of type ttype to be on channel. | | public int | size()
| | protected int | skipOffTokenChannels(int i)
Given a starting index, return the index of the first on-channel
token. | | protected int | skipOffTokenChannelsReverse(int i)
| | public String | toString()
| | public String | toString(int start, int stop)
| | public String | toString(Token start, Token stop)
|
| channel | | protected int channel(Code) | | Skip tokens on any channel but this one; this is how we skip whitespace...
|
| channelOverrideMap | | protected Map channelOverrideMap(Code) | | Map to override some Tokens' channel numbers
|
| discardOffChannelTokens | | protected boolean discardOffChannelTokens(Code) | | By default, track all incoming tokens
|
| discardSet | | protected Set discardSet(Code) | | Set; discard any tokens with this type
|
| lastMarker | | protected int lastMarker(Code) | | Track the last mark() call result value for use in rewind().
|
| p | | protected int p(Code) | | The index into the tokens list of the current token (next token
to consume). p==-1 indicates that the tokens list is empty
|
| tokens | | protected List tokens(Code) | | Record every single token pulled from the source so we can reproduce
chunks of it later.
|
| CommonTokenStream | | public CommonTokenStream()(Code) | | |
| CommonTokenStream | | public CommonTokenStream(TokenSource tokenSource, int channel)(Code) | | |
| LB | | protected Token LB(int k)(Code) | | Look backwards k tokens on-channel tokens
|
| LT | | public Token LT(int k)(Code) | | Get the ith token from the current position 1..n where k=1 is the
first symbol of lookahead.
|
| consume | | public void consume()(Code) | | Move the input pointer to the next incoming token. The stream
must become active with LT(1) available. consume() simply
moves the input pointer so that LT(1) points at the next
input symbol. Consume at least one token.
Walk past any token not on the channel the parser is listening to.
|
| discardOffChannelTokens | | public void discardOffChannelTokens(boolean discardOffChannelTokens)(Code) | | |
| discardTokenType | | public void discardTokenType(int ttype)(Code) | | |
| fillBuffer | | protected void fillBuffer()(Code) | | Load all tokens from the token source and put in tokens.
This is done upon first LT request because you might want to
set some token type / channel overrides before filling buffer.
|
| get | | public Token get(int i)(Code) | | Return absolute token i; ignore which channel the tokens are on;
that is, count all tokens not just on-channel tokens.
|
| getTokens | | public List getTokens(int start, int stop)(Code) | | |
| getTokens | | public List getTokens(int start, int stop, BitSet types)(Code) | | Given a start and stop index, return a List of all tokens in
the token type BitSet. Return null if no tokens were found. This
method looks at both on and off channel tokens.
|
| getTokens | | public List getTokens(int start, int stop, int ttype)(Code) | | |
| release | | public void release(int marker)(Code) | | |
| rewind | | public void rewind(int marker)(Code) | | |
| rewind | | public void rewind()(Code) | | |
| seek | | public void seek(int index)(Code) | | |
| setTokenSource | | public void setTokenSource(TokenSource tokenSource)(Code) | | Reset this token stream by setting its token source.
|
| setTokenTypeChannel | | public void setTokenTypeChannel(int ttype, int channel)(Code) | | A simple filter mechanism whereby you can tell this token stream
to force all tokens of type ttype to be on channel. For example,
when interpreting, we cannot exec actions so we need to tell
the stream to force all WS and NEWLINE to be a different, ignored
channel.
|
| skipOffTokenChannels | | protected int skipOffTokenChannels(int i)(Code) | | Given a starting index, return the index of the first on-channel
token.
|
| skipOffTokenChannelsReverse | | protected int skipOffTokenChannelsReverse(int i)(Code) | | |
|
|