Java Doc for StringTokenizer.java in  » Template-Engine » ostermillerutils » com » Ostermiller » util » Java Source Code / Java DocumentationJava Source Code and Java Documentation

The tokenization method is much simpler than the one used by the StreamTokenizer class. The StringTokenizer methods do not distinguish among identifiers, numbers, and quoted strings, nor do they recognize and skip comments.

The set of delimiters (the characters that separate tokens) may be specified either at creation time or on a per-token basis.

There are two kinds of delimiters: token delimiters and non-token delimiters. A token is either one token delimiter character, or a maximal sequence of consecutive characters that are not delimiters.

A StringTokenizer object internally maintains a current position within the string to be tokenized. Some operations advance this current position past the characters processed.

The implementation is not thread safe; if a StringTokenizer object is intended to be used in multiple threads, an appropriate wrapper must be provided.

The following is one example of the use of the tokenizer. It also demonstrates the usefulness of having both token and non-token delimiters in one StringTokenizer.

The code:

String s = "  (   aaa \t  * (b+c1 ))";
StringTokenizer tokenizer = new StringTokenizer(s, " \t\n\r\f", "()+*");
while (tokenizer.hasMoreTokens()) {
    System.out.println(tokenizer.nextToken());
};

prints the following output:

(
aaa

(
b
+
c1
)
)

Compatibility with java.util.StringTokenizer

In the original version of java.util.StringTokenizer, the method nextToken() left the current position after the returned token, and the method hasMoreTokens() moved (as a side effect) the current position before the beginning of the next token. Thus, the code:

String s = "x=a,b,c";
java.util.StringTokenizer tokenizer = new java.util.StringTokenizer(s,"=");
System.out.println(tokenizer.nextToken());
while (tokenizer.hasMoreTokens()) {
    System.out.println(tokenizer.nextToken(","));
};

prints the following output:

x
a
b
c

The Java SDK 1.3 implementation removed the undesired side effect of hasMoreTokens method: now, it does not advance current position. However, after these changes the output of the above code was:

x
=a
b
c

and there was no good way to produce a second token without "=".

To solve the problem, this implementation introduces a new method skipDelimiters(). To produce the original output, the above code should be modified as:

String s = "x=a,b,c";
StringTokenizer tokenizer = new StringTokenizer(s,"=");
System.out.println(tokenizer.nextToken());
tokenizer.skipDelimiters();
while (tokenizer.hasMoreTokens()) {
    System.out.println(tokenizer.nextToken(","));
};

First, the sets of token and non-token delimiters are changed to be the tokenDelims and nontokenDelims, respectively. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter sets remains the used ones after this call.
Parameters:
  nontokenDelims - the new set of non-token delimiters.
Parameters:
  tokenDelims - the new set of token delimiters.

First, the sets of token and non-token delimiters are changed to be the tokenDelims and nontokenDelims, respectively; and whether or not to return empty tokens is set. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter set remains the one used for this call and empty tokens are returned in the future as they are in this call.
Parameters:
  nontokenDelims - the new set of non-token delimiters.
Parameters:
  tokenDelims - the new set of token delimiters.
Parameters:
  returnEmptyTokens - true if empty tokens may be returned; false otherwise.

Is equivalent to:

• If the second parameter is false -- nextToken(delimiters, null)
• If the second parameter is true -- nextToken(null, delimiters)

Parameters:
  delims - the new set of token or non-token delimiters.
Parameters:
  delimsAreTokens - flag indicating whether the first parameter specifies token ornon-token delimiters: false -- the first parameterspecifies non-token delimiters, the set of token delimiters isempty; true -- the first parameter specifies tokendelimiters, the set of non-token delimiters is empty.

Is equivalent to nextToken(delimiters, null).
Parameters:
  nontokenDelims - the new set of non-token delimiters (the set oftoken delimiters will be empty).

Empty tokens occur when two delimiters are next to each other or a delimiter occurs at the beginning or end of a string.

true if and only if is found that an empty token should be returned or if empty token was the last thing returned.

If returnEmptyTokens in false, then this variable will always be false.
since:
   ostermillerutils 1.00.00

Represents the position at which we should start looking for the next token(the position of the character immediately following the end of the last token, or 0 to start), or -1 if the entire string has been examined.
since:
   ostermillerutils 1.00.00

The current position is set at the beginning of the string.
Parameters:
  text - a string to be parsed.
Parameters:
  nontokenDelims - the non-token delimiters, i.e. the delimiters that only separatetokens and are not returned as separate tokens.
Parameters:
  tokenDelims - the token delimiters, i.e. delimiters that both separate tokens,and are themselves returned as tokens.
throws:
  NullPointerException - if text is null.
since:
   ostermillerutils 1.00.00

Empty tokens are tokens that are between consecutive delimiters.

It is a primary constructor (i.e. all other constructors are defined in terms of it.)

The current position is set at the beginning of the string.
Parameters:
  text - a string to be parsed.
Parameters:
  nontokenDelims - the non-token delimiters, i.e. the delimiters that only separatetokens and are not returned as separate tokens.
Parameters:
  tokenDelims - the token delimiters, i.e. delimiters that both separate tokens,and are themselves returned as tokens.
Parameters:
  returnEmptyTokens - true if empty tokens may be returned; false otherwise.
throws:
  NullPointerException - if text is null.
since:
   ostermillerutils 1.00.00

Is equivalent to:

• If the third parameter is false -- StringTokenizer(text, delimiters, null)
• If the third parameter is true -- StringTokenizer(text, null, delimiters)

Is equivalent to StringTokenizer(text,nontokenDelims, null).
Parameters:
  text - a string to be parsed.
Parameters:
  nontokenDelims - the non-token delimiters.
throws:
  NullPointerException - if text is null.
since:
   ostermillerutils 1.00.00

Is equivalent to StringTokenizer(text, " \t\n\r\f", null);
Parameters:
  text - a string to be parsed.
throws:
  NullPointerException - if text is null.
since:
   ostermillerutils 1.00.00

The current position is not changed. true if and only if there is at least one token in thestring after the current position; false otherwise.
since:
   ostermillerutils 1.00.00

The current position is set after the token returned. the next token from this string tokenizer.
throws:
  NoSuchElementException - if there are no more tokens in this tokenizer's string.
since:
   ostermillerutils 1.00.00

First, the sets of token and non-token delimiters are changed to be the tokenDelims and nontokenDelims, respectively. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter sets remains the used ones after this call.
Parameters:
  nontokenDelims - the new set of non-token delimiters.
Parameters:
  tokenDelims - the new set of token delimiters. the next token, after switching to the new delimiter set.
throws:
  NoSuchElementException - if there are no more tokens in this tokenizer's string.
See Also:   StringTokenizer.nextToken()
since:
   ostermillerutils 1.00.00

First, the sets of token and non-token delimiters are changed to be the tokenDelims and nontokenDelims, respectively; and whether or not to return empty tokens is set. Then the next token (with respect to new delimiters) in the string after the current position is returned.

The current position is set after the token returned.

The new delimiter set remains the one used for this call and empty tokens are returned in the future as they are in this call.
Parameters:
  nontokenDelims - the new set of non-token delimiters.
Parameters:
  tokenDelims - the new set of token delimiters.
Parameters:
  returnEmptyTokens - true if empty tokens may be returned; false otherwise. the next token, after switching to the new delimiter set.
throws:
  NoSuchElementException - if there are no more tokens in this tokenizer's string.
See Also:   StringTokenizer.nextToken()
since:
   ostermillerutils 1.00.00

Is equivalent to:

• If the second parameter is false -- nextToken(delimiters, null)
• If the second parameter is true -- nextToken(null, delimiters)

Parameters:
  delims - the new set of token or non-token delimiters.
Parameters:
  delimsAreTokens - flag indicating whether the first parameter specifies token ornon-token delimiters: false -- the first parameterspecifies non-token delimiters, the set of token delimiters isempty; true -- the first parameter specifies tokendelimiters, the set of non-token delimiters is empty. the next token, after switching to the new delimiter set.
throws:
  NoSuchElementException - if there are no more tokens in this tokenizer's string.
See Also:   StringTokenizer.nextToken(String,String)
since:
   ostermillerutils 1.00.00

Is equivalent to nextToken(delimiters, null).
Parameters:
  nontokenDelims - the new set of non-token delimiters (the set oftoken delimiters will be empty). the next token, after switching to the new delimiter set.
throws:
  NoSuchElementException - if there are no more tokens in thistokenizer's string.
See Also:   StringTokenizer.nextToken(String,String)
since:
   ostermillerutils 1.00.00

Empty tokens occur when two delimiters are next to each other or a delimiter occurs at the beginning or end of a string. If empty tokens are set to be returned, and a comma is the non token delimiter, the following table shows how many tokens are in each string.

String		Number of tokens
"one,two"		2 - normal case with no empty tokens.
"one,,three"		3 including the empty token in the middle.
"one,"		2 including the empty token at the end.
",two"		2 including the empty token at the beginning.
","		2 including the empty tokens at the beginning and the ends.
""		1 - all strings will have at least one token if empty tokens are returned.

Parameters:
  returnEmptyTokens - true iff empty tokens should be returned.
since:
   ostermillerutils 1.00.00

This is useful when for StringTokenizer re-use so that new string tokenizers do not have to be created for each string you want to tokenizer.

The string will be tokenized from the beginning of the string.
Parameters:
  text - a string to be parsed.
throws:
  NullPointerException - if text is null.
since:
   ostermillerutils 1.00.00

This method skips non-token delimiters but does not skip token delimiters.

This method is useful when switching to the new delimiter sets (see the second example in the class comment.) true if there are more tokens, false otherwise.
since:
   ostermillerutils 1.00.00