| StandardLexerRules |
| abstract public class StandardLexerRules (Code) |
| Standard lexer rules are building blocks for lexers dealing with text input.
This class resolves nonterminals enclosed in `backquotes` within an EBNF,
e.g. `cstylecomment`.
Furthermore it provides methods to retrieve sets of rules describing certain standard
scan items like `number` or `identifier`. The resulting arrays can be built together
by SyntaxUtil.catenizeRules(...).
This class provides rules for comments with an arbitrary start character or start/end sequence:
- getCustomOneLineCommentRules(String startChar)
and
- getCustomMultiLineCommentRules(String startSeq, String endSeq)
.
Example (CStyleCommentStrip):
String [][] rules = {
{ Token.TOKEN, "others" }, // define what we want to receive
{ Token.TOKEN, "`stringdef`" }, // need this rule as string definitions could contain comments
{ Token.IGNORED, "`cstylecomment`" },
{ "others", "others", "other" },
{ "others", "other" },
{ "other", "`char`", Token.BUTNOT, "`cstylecomment`", Token.BUTNOT, "`stringdef`" },
};
Syntax syntax = new Syntax(rules);
SyntaxSeparation separation = new SyntaxSeparation(syntax);
LexerBuilder builder = new LexerBuilder(separation.getLexerSyntax(), separation.getIgnoredSymbols());
Lexer lexer = builder.getLexer();
TODO: Refactor this class and make smaller units with better names.
See Also: fri.patterns.interpreter.parsergenerator.lexer.LexerBuilder
author:
(c) 2002, Fritz Ritzberger |
| Field Summary |
|
| final public static String[][] | chardefRules
Rules describing C/Java-like character definitions: 'c', '\r', '\007'. |
| final public static String[][] | digitRules
Numerical rules for binary and octal digits. |
| final public static String[][] | lexerSyntax
Premade lexer syntax used to scan textual EBNF-like syntax specifications. |
| final public static String[][] | newlinesRules
Rules describing one or more newlines. |
| final public static String[][] | numberRules
Numerical rules for numbers within sourcecode: number ::= integer | float. |
| final public static String[][] | whitespaceRules
Rules describing whitespace: newlines and spaces, minimum one. |
| final public static String[][] | xmlCharRules
XML Char definitions of W3C. |
| final public static String[][] | xmlCombinigAndExtenderRules
XML CombiningChar and XML Extender definitions of W3C. |
| Method Summary |
|
| final public static String[][] | catenizeRules(String[][][] arrays)
Catenizes some rule sets to one rule set. |
| final public static String[][] | getBinDigitsRules()
Rules for binary number chars. |
| final public static String[][] | getCStyleCommentRules()
Rules to scan C-style slash-star and slash-slash comments. |
| final public static String[][] | getCommentRules()
Rules to scan C-style slash-star and slash-slash AND shell-style # comments. |
| final public static String[][] | getCustomMultiLineCommentRules(String nonterminalName, String startSeq, String endSeq)
Returns rules for a custom comment (like C-style "/*", but with passed start and end sequence).
Parameters:
nonterminalName - name of comment to be used within syntax, e.g. |
| final public static String[][] | getCustomOneLineCommentRules(String nonterminalName, String startChar)
Returns rules for a custom comment (like C-style "//", but with passed start sequence).
Parameters:
nonterminalName - name of comment to be used within syntax, e.g. |
| final public static String[][] | getFloatRules()
Rules for float number chars. |
| final public static String[][] | getHexDigitRules()
Rules to scan one hexdigit. |
| final public static String[][] | getHexDigitsRules()
Rules to scan hexdigits that form a number, starting "0x" not included. |
| final public static String[][] | getIntegerRules()
Rules for integer number chars. |
| final public static String[][] | getNewlineRules()
Rules to scan one platform independent newline. |
| final public static String[][] | getNewlinesRules()
Rules to scan one platform independent newline. |
| final public static String[][] | getNumberRules()
Rules for general number chars (integer, float). |
| final public static String[][] | getOctDigitsRules()
Rules for octal number chars. |
| final public static String[][] | getQuantifierRules()
Rules to read quantifiers "*+?" within EBNF syntax specifications. |
| final public static String[][] | getRulerefRules()
Rules to read a `lexerrule` within EBNF syntax specifications. |
| final public static String[][] | getShellStyleCommentRules()
Rules to scan # shell-style comments. |
| final public static String[][] | getSpaceRules()
Rules to scan one space. |
| final public static String[][] | getSpacesRules()
Rules to scan spaces. |
| final public static String[][] | getUnicodeBNFChardefRules()
Rules to scan BNF-like 'c'haracterdefinitions. |
| final public static String[][] | getUnicodeCharRules()
Rules to scan one UNICODE character: 0x0 .. |
| final public static String[][] | getUnicodeChardefRules()
Rules to scan C/Java-like 'c'haracterdefinitions: '\377', 'A', '\n'. |
| final public static String[][] | getUnicodeCombiningCharRules()
Rules for XML combining chars. |
| final public static String[][] | getUnicodeDigitRules()
Rules to scan one digit. |
| final public static String[][] | getUnicodeDigitsRules()
Rules to scan digits. |
| final public static String[][] | getUnicodeExtenderCharRules()
Rules for XML extender chars. |
| final public static String[][] | getUnicodeIdentifierRules()
Rules to scan identifiers that start with letter and continue with letter or digit or '_'. |
| final public static String[][] | getUnicodeLetterRules()
Rules to scan one letter. |
| final public static String[][] | getUnicodeStringdefRules()
Rules to scan "stringdefinitions" that can contain backslash as masking character. |
| final public static String[][] | getUnicodeXmlCharRules()
Rules for XML combining chars. |
| final public static String[][] | getWhitespaceRules()
Rules to scan one space or newline. |
| final public static String[][] | getWhitespacesRules()
Rules to scan spaces or newlines. |
| public static void | printRules(String[][] syntax)
Print a grammar to System.out. |
| public static String[][] | rulesForIdentifier(String id)
Returns e.g. |
