| java.lang.Object
 net.sf.saxon.expr.Tokenizer
| Tokenizer | | final public class Tokenizer (Code) | | Tokenizer for expressions and inputs.
This code was originally derived from James Clark's xt, though it has been greatly modified since.
See copyright notice at end of file.
|
| BARE_NAME_STATE | | final public static int BARE_NAME_STATE(Code) | | State in which a name is NOT to be merged with what comes next, for example "("
|
| DEFAULT_STATE | | final public static int DEFAULT_STATE(Code) | | Initial default state of the Tokenizer
|
| OPERATOR_STATE | | final public static int OPERATOR_STATE(Code) | | State in which the next thing to be read is an operator
|
| SEQUENCE_TYPE_STATE | | final public static int SEQUENCE_TYPE_STATE(Code) | | State in which the next thing to be read is a SequenceType
|
| currentToken | | public int currentToken(Code) | | The number identifying the most recently read token
|
| currentTokenStartOffset | | public int currentTokenStartOffset(Code) | | The position in the input expression where the current token starts
|
| currentTokenValue | | public String currentTokenValue(Code) | | The string value of the most recently read token
|
| inputOffset | | public int inputOffset(Code) | | The current position within the input string
|
| startLineNumber | | public int startLineNumber(Code) | | The starting line number (for XPath in XSLT, the line number in the stylesheet)
|
| getColumnNumber | | public int getColumnNumber()(Code) | | Get the column number of the current token
|
| getColumnNumber | | public int getColumnNumber(int offset)(Code) | | |
| getLineAndColumn | | public long getLineAndColumn(int offset)(Code) | | Get the line and column number corresponding to a given offset in the input expression,
as a long value with the line number in the top half
and the column number in the lower half
the line and column number, packed together |
| getLineNumber | | public int getLineNumber()(Code) | | Get the line number of the current token
|
| getLineNumber | | public int getLineNumber(int offset)(Code) | | |
| getState | | public int getState()(Code) | | |
| lookAhead | | public void lookAhead() throws StaticError(Code) | | Look ahead by one token. This method does the real tokenization work.
The method is normally called internally, but the XQuery parser also
calls it to resume normal tokenization after dealing with pseudo-XML
syntax.
exception:
net.sf.saxon.trans.StaticError - if a lexical error occurs |
| next | | public void next() throws StaticError(Code) | | Get the next token from the input expression. The type of token is returned in the
currentToken variable, the string value of the token in currentTokenValue.
exception:
net.sf.saxon.trans.StaticError - if a lexical error is detected |
| nextChar | | public char nextChar() throws StringIndexOutOfBoundsException(Code) | | Read next character directly. Used by the XQuery parser when parsing pseudo-XML syntax
the next character from the input
throws:
StringIndexOutOfBoundsException - if an attempt is made to read beyondthe end of the string. This will only occur in the event of a syntax error in theinput. |
| recentText | | public String recentText()(Code) | | Get the most recently read text (for use in an error message)
|
| setState | | public void setState(int state)(Code) | | |
| tokenize | | public void tokenize(String input, int start, int end, int lineNumber) throws StaticError(Code) | | Prepare a string for tokenization.
The actual tokens are obtained by calls on next()
Parameters:
input - the string to be tokenized
Parameters:
start - start point within the string
Parameters:
end - end point within the string (last character not read):-1 means end of string
exception:
net.sf.saxon.trans.StaticError - if a lexical error occurs, e.g. unmatchedstring quotes |
| treatCurrentAsOperator | | public void treatCurrentAsOperator()(Code) | | Force the current token to be treated as an operator if possible
|
| unreadChar | | public void unreadChar()(Code) | | Step back one character. If this steps back to a previous line, adjust the line number.
|
|
|