| java.lang.Object
 org.antlr.runtime.CommonTokenStream
org.antlr.runtime.TokenRewriteStream
| TokenRewriteStream | | public class TokenRewriteStream extends CommonTokenStream (Code) | | Useful for dumping out the input stream after doing some
augmentation or other manipulations.
You can insert stuff, replace, and delete chunks. Note that the
operations are done lazily--only if you convert the buffer to a
String. This is very efficient because you are not moving data around
all the time. As the buffer of tokens is converted to strings, the
toString() method(s) check to see if there is an operation at the
current index. If so, the operation is done and then normal String
rendering continues on the buffer. This is like having multiple Turing
machine instruction streams (programs) operating on a single input tape. :)
Since the operations are done lazily at toString-time, operations do not
screw up the token index values. That is, an insert operation at token
index i does not change the index values for tokens i+1..n-1.
Because operations never actually alter the buffer, you may always get
the original token stream back without undoing anything. Since
the instructions are queued up, you can easily simulate transactions and
roll back any changes if there is an error just by removing instructions.
For example,
CharStream input = new ANTLRFileStream("input");
TLexer lex = new TLexer(input);
TokenRewriteStream tokens = new TokenRewriteStream(lex);
T parser = new T(tokens);
parser.startRule();
Then in the rules, you can execute
Token t,u;
...
input.insertAfter(t, "text to put after t");}
input.insertAfter(u, "text after u");}
System.out.println(tokens.toString());
Actually, you have to cast the 'input' to a TokenRewriteStream. :(
You can also have multiple "instruction streams" and get multiple
rewrites from a single pass over the input. Just name the instruction
streams and use that name again when printing the buffer. This could be
useful for generating a C file and also its header file--all from the
same buffer:
tokens.insertAfter("pass1", t, "text to put after t");}
tokens.insertAfter("pass2", u, "text after u");}
System.out.println(tokens.toString("pass1"));
System.out.println(tokens.toString("pass2"));
If you don't use named rewrite streams, a "default" stream is used as
the first example shows.
|
| Inner Class :static class RewriteOperation | |
| Inner Class :static class InsertBeforeOp extends RewriteOperation | |
| Inner Class :static class ReplaceOp extends RewriteOperation | |
| Inner Class :static class DeleteOp extends ReplaceOp | |
| Method Summary | |
| protected void | addToSortedRewriteList(RewriteOperation op)
If op.index > lastRewriteTokenIndexes, just add to the end. | | protected void | addToSortedRewriteList(String programName, RewriteOperation op)
Add an instruction to the rewrite instruction list ordered by
the instruction number (use a binary search for efficiency).
The list is ordered so that toString() can be done efficiently.
When there are multiple instructions at the same index, the instructions
must be ordered to ensure proper behavior. | | public void | delete(int index)
| | public void | delete(int from, int to)
| | public void | delete(Token indexT)
| | public void | delete(Token from, Token to)
| | public void | delete(String programName, int from, int to)
| | public void | delete(String programName, Token from, Token to)
| | public void | deleteProgram()
| | public void | deleteProgram(String programName)
| | public int | getLastRewriteTokenIndex()
| | protected int | getLastRewriteTokenIndex(String programName)
| | protected List | getProgram(String name)
| | protected void | init()
| | public void | insertAfter(Token t, Object text)
| | public void | insertAfter(int index, Object text)
| | public void | insertAfter(String programName, Token t, Object text)
| | public void | insertAfter(String programName, int index, Object text)
| | public void | insertBefore(Token t, Object text)
| | public void | insertBefore(int index, Object text)
| | public void | insertBefore(String programName, Token t, Object text)
| | public void | insertBefore(String programName, int index, Object text)
| | public void | replace(int index, Object text)
| | public void | replace(int from, int to, Object text)
| | public void | replace(Token indexT, Object text)
| | public void | replace(Token from, Token to, Object text)
| | public void | replace(String programName, int from, int to, Object text)
| | public void | replace(String programName, Token from, Token to, Object text)
| | public void | rollback(int instructionIndex)
| | public void | rollback(String programName, int instructionIndex)
Rollback the instruction stream for a program so that
the indicated instruction (via instructionIndex) is no
longer in the stream. | | protected void | setLastRewriteTokenIndex(String programName, int i)
| | public String | toDebugString()
| | public String | toDebugString(int start, int end)
| | public String | toOriginalString()
| | public String | toOriginalString(int start, int end)
| | public String | toString()
| | public String | toString(String programName)
| | public String | toString(int start, int end)
| | public String | toString(String programName, int start, int end)
|
| DEFAULT_PROGRAM_NAME | | final public static String DEFAULT_PROGRAM_NAME(Code) | | |
| MIN_TOKEN_INDEX | | final public static int MIN_TOKEN_INDEX(Code) | | |
| PROGRAM_INIT_SIZE | | final public static int PROGRAM_INIT_SIZE(Code) | | |
| lastRewriteTokenIndexes | | protected Map lastRewriteTokenIndexes(Code) | | Map String (program name) -> Integer index
|
| programs | | protected Map programs(Code) | | You may have multiple, named streams of rewrite operations.
I'm calling these things "programs."
Maps String (name) -> rewrite (List)
|
| TokenRewriteStream | | public TokenRewriteStream()(Code) | | |
| TokenRewriteStream | | public TokenRewriteStream(TokenSource tokenSource, int channel)(Code) | | |
| addToSortedRewriteList | | protected void addToSortedRewriteList(RewriteOperation op)(Code) | | If op.index > lastRewriteTokenIndexes, just add to the end.
Otherwise, do linear
|
| addToSortedRewriteList | | protected void addToSortedRewriteList(String programName, RewriteOperation op)(Code) | | Add an instruction to the rewrite instruction list ordered by
the instruction number (use a binary search for efficiency).
The list is ordered so that toString() can be done efficiently.
When there are multiple instructions at the same index, the instructions
must be ordered to ensure proper behavior. For example, a delete at
index i must kill any replace operation at i. Insert-before operations
must come before any replace / delete instructions. If there are
multiple insert instructions for a single index, they are done in
reverse insertion order so that "insert foo" then "insert bar" yields
"foobar" in front rather than "barfoo". This is convenient because
I can insert new InsertOp instructions at the index returned by
the binary search. A ReplaceOp kills any previous replace op. Since
delete is the same as replace with null text, i can check for
ReplaceOp and cover DeleteOp at same time. :)
|
| delete | | public void delete(int index)(Code) | | |
| delete | | public void delete(int from, int to)(Code) | | |
| delete | | public void delete(String programName, int from, int to)(Code) | | |
| deleteProgram | | public void deleteProgram()(Code) | | |
| deleteProgram | | public void deleteProgram(String programName)(Code) | | Reset the program so that no instructions exist
|
| getLastRewriteTokenIndex | | public int getLastRewriteTokenIndex()(Code) | | |
| getLastRewriteTokenIndex | | protected int getLastRewriteTokenIndex(String programName)(Code) | | |
| init | | protected void init()(Code) | | |
| insertAfter | | public void insertAfter(int index, Object text)(Code) | | |
| insertBefore | | public void insertBefore(int index, Object text)(Code) | | |
| replace | | public void replace(int from, int to, Object text)(Code) | | |
| rollback | | public void rollback(int instructionIndex)(Code) | | |
| rollback | | public void rollback(String programName, int instructionIndex)(Code) | | Rollback the instruction stream for a program so that
the indicated instruction (via instructionIndex) is no
longer in the stream. UNTESTED!
|
| setLastRewriteTokenIndex | | protected void setLastRewriteTokenIndex(String programName, int i)(Code) | | |
| toDebugString | | public String toDebugString(int start, int end)(Code) | | |
| toOriginalString | | public String toOriginalString(int start, int end)(Code) | | |
| Methods inherited from org.antlr.runtime.CommonTokenStream | public int LA(int i)(Code)(Java Doc)
protected Token LB(int k)(Code)(Java Doc)
public Token LT(int k)(Code)(Java Doc)
public void consume()(Code)(Java Doc)
public void discardOffChannelTokens(boolean discardOffChannelTokens)(Code)(Java Doc)
public void discardTokenType(int ttype)(Code)(Java Doc)
protected void fillBuffer()(Code)(Java Doc)
public Token get(int i)(Code)(Java Doc)
public TokenSource getTokenSource()(Code)(Java Doc)
public List getTokens()(Code)(Java Doc)
public List getTokens(int start, int stop)(Code)(Java Doc)
public List getTokens(int start, int stop, BitSet types)(Code)(Java Doc)
public List getTokens(int start, int stop, List types)(Code)(Java Doc)
public List getTokens(int start, int stop, int ttype)(Code)(Java Doc)
public int index()(Code)(Java Doc)
public int mark()(Code)(Java Doc)
public void release(int marker)(Code)(Java Doc)
public void rewind(int marker)(Code)(Java Doc)
public void rewind()(Code)(Java Doc)
public void seek(int index)(Code)(Java Doc)
public void setTokenSource(TokenSource tokenSource)(Code)(Java Doc)
public void setTokenTypeChannel(int ttype, int channel)(Code)(Java Doc)
public int size()(Code)(Java Doc)
protected int skipOffTokenChannels(int i)(Code)(Java Doc)
protected int skipOffTokenChannelsReverse(int i)(Code)(Java Doc)
public String toString()(Code)(Java Doc)
public String toString(int start, int stop)(Code)(Java Doc)
public String toString(Token start, Token stop)(Code)(Java Doc)
|
|
|