Java Doc for TokenSequence.java in  » IDE-Netbeans » lexer » org » netbeans » api » lexer » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Use of token sequence is a two-step operation:

1. Position token sequence before token that should first be retrieved (or behind desired token when iterating backwards).
   One of the following ways may be used:
   • TokenSequence.move(int) positions TS before token that either starts at the given offset or "contains" it.
   • TokenSequence.moveIndex(int) positions TS before n-th token in the underlying token list.
   • TokenSequence.moveStart() positions TS before the first token.
   • TokenSequence.moveEnd() positions TS behind the last token.
   • Do nothing - TS is positioned before the first token automatically by default.
   Token sequence will always be positioned between tokens when using one of the operations above ( TokenSequence.token() will return null to signal between-tokens location).
   
2. Start iterating through the tokens in forward/backward direction by using TokenSequence.moveNext() or TokenSequence.movePrevious() .
   If moveNext() or movePrevious() returned true then TS is positioned over a concrete token retrievable by TokenSequence.token() .
   Its offset can be retrieved by TokenSequence.offset() .

An example of forward iteration through the tokens:

 TokenSequence ts = tokenHierarchy.tokenSequence();
 // Possible positioning by ts.move(offset) or ts.moveIndex(index)
 while (ts.moveNext()) {
 Token t = ts.token();
 if (t.id() == ...) { ... }
 if (TokenUtilities.equals(t.text(), "mytext")) { ... }
 if (ts.offset() == ...) { ... }
 }
 

This object should be used by a single thread only. For token hierarchies over mutable input sources the obtaining and using of the token sequence needs to be done under a read-lock of the input source.

Initially or after TokenSequence.move(int) or TokenSequence.moveIndex(int) token sequence is positioned between tokens:

 Token[0]   Token[1]   ...

 +----------+-----+----------------+--------------+------
 | Token[0] | ...

 Token[0]   ...

The next token may not necessarily start at the offset where the previous token ends (there may be gaps between tokens caused by token filtering).

The previous token may not necessarily end at the offset where the previous token started (there may be gaps between tokens caused by token filtering).

This method may be handy if the token instance is referenced in a standalone way (e.g.

 TokenSequence ts = tokenHierarchy.tokenSequence();
 // Possible positioning by ts.move(offset) or ts.moveIndex(index)
 while (ts.moveNext()) {
 Token t = ts.token();
 if (t.id() == ...) { ...

<!-- HTML comment start<% System.out.println("Hello"); %>still in HTML comment --<

Initially or after TokenSequence.move(int) or TokenSequence.moveIndex(int) token sequence is positioned between tokens:

 Token[0]   Token[1]   ...   Token[n]
 ^          ^                ^
 Index: 0          1                n
 

After use of TokenSequence.moveNext() or TokenSequence.movePrevious() the token sequence is positioned over one of the actual tokens:

 Token[0]   Token[1]   ...   Token[n]
 ^          ^                ^
 Index:      0          1                n
 

 +----------+-----+----------------+--------------+------
 | Token[0] | ... | Token[index-1] | Token[index] | ...
 | "public" | ... | "static"       | "int"        | ...
 +----------+-----+----------------+--------------+------
 ^                ^                ^
 Index: 0             index-1           index
 Offset:                                  ---^ (if offset points to 'i','n' or 't')
 

Subsequent TokenSequence.moveNext() or TokenSequence.movePrevious() is needed to fetch a concrete token.
If the offset is too big then the token sequence will be positioned behind the last token.

If token filtering is used there may be gaps that are not covered by any tokens and if the offset is contained in such gap then the token sequence will be positioned before the token that follows the gap.

 Token[0]   ...   Token[index-1]   Token[index] ...
 ^                ^                ^
 Index: 0             index-1           index
 

Subsequent TokenSequence.moveNext() or TokenSequence.movePrevious() is needed to fetch a concrete token in the desired direction.
Subsequent TokenSequence.moveNext() will position TS over Token[index] (or TokenSequence.movePrevious() will position TS over Token[index-1]) so that TokenSequence.token() != null.
Parameters:
  index - index of the token to which this sequenceshould be positioned.
If index >= TokenSequence.tokenCount()then the TS will be positioned to TokenSequence.tokenCount().
If index < 0 then the TS will be positioned to index 0. difference between requested index and the index to which TSis really set.
throws:
  ConcurrentModificationException - if this token sequenceis no longer valid because of an underlying mutable input source modification.

The next token may not necessarily start at the offset where the previous token ends (there may be gaps between tokens caused by token filtering). TokenSequence.offset() should be used for offset retrieval.

The previous token may not necessarily end at the offset where the previous token started (there may be gaps between tokens caused by token filtering). TokenSequence.offset() should be used for offset retrieval.

This method may be handy if the token instance is referenced in a standalone way (e.g. in an expression node of a parse tree) and it's necessary to get the appropriate offset from the token itself later when a token sequence will not be available.

 TokenSequence ts = tokenHierarchy.tokenSequence();
 // Possible positioning by ts.move(offset) or ts.moveIndex(index)
 while (ts.moveNext()) {
 Token t = ts.token();
 if (t.id() == ...) { ... }
 if (TokenUtilities.equals(t.text(), "mytext")) { ... }
 if (ts.offset() == ...) { ... }
 }
 

The lifetime of the returned token instance may be limited for mutable inputs. The token instance should not be held across the input source modifications.