Java Doc for Tokenizer.java in  » ERP-CRM-Financial » SourceTap-CRM » org » ofbiz » rules » parse » tokens » Java Source Code / Java DocumentationJava Source Code and Java Documentation

Java Source Code / Java Documentation
1. 6.0 JDK Core
2. 6.0 JDK Modules
3. 6.0 JDK Modules com.sun
4. 6.0 JDK Modules com.sun.java
5. 6.0 JDK Modules sun
6. 6.0 JDK Platform
7. Ajax
8. Apache Harmony Java SE
9. Aspect oriented
10. Authentication Authorization
11. Blogger System
12. Build
13. Byte Code
14. Cache
15. Chart
16. Chat
17. Code Analyzer
18. Collaboration
19. Content Management System
20. Database Client
21. Database DBMS
22. Database JDBC Connection Pool
23. Database ORM
24. Development
25. EJB Server geronimo
26. EJB Server GlassFish
27. EJB Server JBoss 4.2.1
28. EJB Server resin 3.1.5
29. ERP CRM Financial
30. ESB
31. Forum
32. GIS
33. Graphic Library
34. Groupware
35. HTML Parser
36. IDE
37. IDE Eclipse
38. IDE Netbeans
39. Installer
40. Internationalization Localization
41. Inversion of Control
42. Issue Tracking
43. J2EE
44. JBoss
45. JMS
46. JMX
47. Library
48. Mail Clients
49. Net
50. Parser
51. PDF
52. Portal
53. Profiler
54. Project Management
55. Report
56. RSS RDF
57. Rule Engine
58. Science
59. Scripting
60. Search Engine
61. Security
62. Sevlet Container
63. Source Control
64. Swing Library
65. Template Engine
66. Test Coverage
67. Testing
68. UML
69. Web Crawler
70. Web Framework
71. Web Mail
72. Web Server
73. Web Services
74. Web Services apache cxf 2.0.1
75. Web Services AXIS2
76. Wiki Engine
77. Workflow Engines
78. XML
79. XML UI
Java
Java Tutorial
Java Open Source
Jar File Download
Java Articles
Java Products
Java by API
Photoshop Tutorials
Maya Tutorials
Flash Tutorials
3ds-Max Tutorials
Illustrator Tutorials
GIMP Tutorials
C# / C Sharp
C# / CSharp Tutorial
C# / CSharp Open Source
ASP.Net
ASP.NET Tutorial
JavaScript DHTML
JavaScript Tutorial
JavaScript Reference
HTML / CSS
HTML CSS Reference
C / ANSI-C
C Tutorial
C++
C++ Tutorial
Ruby
PHP
Python
Python Tutorial
Python Open Source
SQL Server / T-SQL
SQL Server / T-SQL Tutorial
Oracle PL / SQL
Oracle PL/SQL Tutorial
PostgreSQL
SQL / MySQL
MySQL Tutorial
VB.Net
VB.Net Tutorial
Flash / Flex / ActionScript
VBA / Excel / Access / Word
XML
XML Tutorial
Microsoft Office PowerPoint 2007 Tutorial
Microsoft Office Excel 2007 Tutorial
Microsoft Office Word 2007 Tutorial
Java Source Code / Java Documentation » ERP CRM Financial » SourceTap CRM » org.ofbiz.rules.parse.tokens 
Source Cross Reference  Class Diagram Java Document (Java Doc) 


java.lang.Object
   org.ofbiz.rules.parse.tokens.Tokenizer

Tokenizer
public class Tokenizer (Code)

Title: Tokenizer

Description: None

Copyright (c) 1999 Steven J. Metsker.

Copyright (c) 2001 The Open For Business Project - www.ofbiz.org

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
A tokenizer divides a string into tokens. This class is highly customizable with regard to exactly how this division occurs, but it also has defaults that are suitable for many languages. This class assumes that the character values read from the string lie in the range 0-255. For example, the Unicode value of a capital A is 65, so System.out.println((char)65); prints out a capital A.

The behavior of a tokenizer depends on its character state table. This table is an array of 256 TokenizerState states. The state table decides which state to enter upon reading a character from the input string.

For example, by default, upon reading an 'A', a tokenizer will enter a "word" state. This means the tokenizer will ask a WordState object to consume the 'A', along with the characters after the 'A' that form a word. The state's responsibility is to consume characters and return a complete token.

The default table sets a SymbolState for every character from 0 to 255, and then overrides this with:

 From    To     State
 0     ' '    whitespaceState
 'a'    'z'    wordState
 'A'    'Z'    wordState
 160     255    wordState
 '0'    '9'    numberState
 '-'    '-'    numberState
 '.'    '.'    numberState
 '"'    '"'    quoteState
 '\''   '\''    quoteState
 '/'    '/'    slashState
 
In addition to allowing modification of the state table, this class makes each of the states above available. Some of these states are customizable. For example, wordState allows customization of what characters can be part of a word, after the first character.
author:
   Steven J. Metsker
version:
   1.0


Field Summary
final protected static  intDEFAULT_SYMBOL_MAX
    
protected  TokenizerState[]characterState
    
protected  NumberStatenumberState
    
protected  QuoteStatequoteState
    
protected  PushbackReaderreader
    
protected  SlashStateslashState
    
protected  SymbolStatesymbolState
    
protected  WhitespaceStatewhitespaceState
    
protected  WordStatewordState
    

Constructor Summary
public  Tokenizer()
     Constructs a tokenizer with a default state table (as described in the class comment).
public  Tokenizer(String s)
     Constructs a tokenizer to read from the supplied string.

Method Summary
public  PushbackReadergetReader()
     Return the reader this tokenizer will read from.
public  TokennextToken()
     Return the next token.
public  NumberStatenumberState()
     Return the state this tokenizer uses to build numbers.
public  QuoteStatequoteState()
     Return the state this tokenizer uses to build quoted strings.
public  voidsetCharacterState(int from, int to, TokenizerState state)
     Change the state the tokenizer will enter upon reading any character between "from" and "to".
public  voidsetReader(PushbackReader r)
     Set the reader to read from.
public  voidsetString(String s)
     Set the string to read from.
public  voidsetString(String s, int symbolMax)
     Set the string to read from.
public  SlashStateslashState()
     Return the state this tokenizer uses to recognize (and ignore) comments.
public  SymbolStatesymbolState()
     Return the state this tokenizer uses to recognize symbols.
public  WhitespaceStatewhitespaceState()
     Return the state this tokenizer uses to recognize (and ignore) whitespace.
public  WordStatewordState()
     Return the state this tokenizer uses to build words.

Field Detail
DEFAULT_SYMBOL_MAX
final protected static int DEFAULT_SYMBOL_MAX(Code)
The number of characters that might be in a symbol;



characterState
protected TokenizerState[] characterState(Code)
The state lookup table



numberState
protected NumberState numberState(Code)
The default states that actually consume text and produce a token



quoteState
protected QuoteState quoteState(Code)



reader
protected PushbackReader reader(Code)
The reader to read characters from



slashState
protected SlashState slashState(Code)



symbolState
protected SymbolState symbolState(Code)



whitespaceState
protected WhitespaceState whitespaceState(Code)



wordState
protected WordState wordState(Code)




Constructor Detail
Tokenizer
public Tokenizer()(Code)
Constructs a tokenizer with a default state table (as described in the class comment). a tokenizer



Tokenizer
public Tokenizer(String s)(Code)
Constructs a tokenizer to read from the supplied string.
Parameters:
  String - the string to read from




Method Detail
getReader
public PushbackReader getReader()(Code)
Return the reader this tokenizer will read from. the reader this tokenizer will read from



nextToken
public Token nextToken() throws IOException(Code)
Return the next token. the next token.
exception:
  IOException - if there is any problem reading



numberState
public NumberState numberState()(Code)
Return the state this tokenizer uses to build numbers. the state this tokenizer uses to build numbers



quoteState
public QuoteState quoteState()(Code)
Return the state this tokenizer uses to build quoted strings. the state this tokenizer uses to build quotedstrings



setCharacterState
public void setCharacterState(int from, int to, TokenizerState state)(Code)
Change the state the tokenizer will enter upon reading any character between "from" and "to".
Parameters:
  from - the "from" character
Parameters:
  to - the "to" character
Parameters:
  TokenizerState - the state to enter upon reading acharacter between "from" and "to"



setReader
public void setReader(PushbackReader r)(Code)
Set the reader to read from.
Parameters:
  PushbackReader - the reader to read from



setString
public void setString(String s)(Code)
Set the string to read from.
Parameters:
  String - the string to read from



setString
public void setString(String s, int symbolMax)(Code)
Set the string to read from.
Parameters:
  String - the string to read from int the maximum length of a symbol, whichestablishes the size of pushback bufferwe need



slashState
public SlashState slashState()(Code)
Return the state this tokenizer uses to recognize (and ignore) comments. the state this tokenizer uses to recognize(and ignore) comments



symbolState
public SymbolState symbolState()(Code)
Return the state this tokenizer uses to recognize symbols. the state this tokenizer uses to recognizesymbols



whitespaceState
public WhitespaceState whitespaceState()(Code)
Return the state this tokenizer uses to recognize (and ignore) whitespace. the state this tokenizer uses to recognizewhitespace



wordState
public WordState wordState()(Code)
Return the state this tokenizer uses to build words. the state this tokenizer uses to build words



Methods inherited from java.lang.Object
native protected Object clone() throws CloneNotSupportedException(Code)(Java Doc)
public boolean equals(Object obj)(Code)(Java Doc)
protected void finalize() throws Throwable(Code)(Java Doc)
final native public Class getClass()(Code)(Java Doc)
native public int hashCode()(Code)(Java Doc)
final native public void notify()(Code)(Java Doc)
final native public void notifyAll()(Code)(Java Doc)
public String toString()(Code)(Java Doc)
final native public void wait(long timeout) throws InterruptedException(Code)(Java Doc)
final public void wait(long timeout, int nanos) throws InterruptedException(Code)(Java Doc)
final public void wait() throws InterruptedException(Code)(Java Doc)

www.java2java.com | Contact Us
Copyright 2009 - 12 Demo Source and Support. All rights reserved.
All other trademarks are property of their respective owners.