The tokenization method is much simpler than the one used by the StreamTokenizer class.
StreamTokenizer