A TokenizerFactory breaks up a stream of characters
into tokens.
TokenizerFactories are registered for FieldType s with the
IndexSchema through the schema.xml file.
Example schema.xml entry to register a TokenizerFactory
implementation to tokenize fields of type "cool"
<fieldtype name="cool" class="solr.TextField">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
...
A single instance of any registered TokenizerFactory is created
via the default constructor and is reused for each FieldType.
author: yonik version: $Id: TokenizerFactory.java 472574 2006-11-08 18:25:52Z yonik $ |