lucene中的Token, TokenStream, Tokenizer, Analyzer
2013-06-03 16:00
369 查看
Token: 如果一个字段被token化,这表示它经过了一个可将内容转化为tokens串的分析程序。 Token是建立索引的基本单位,表示每个被编入索引的字符。 在token化的过程中,分析程序会在使用任何转换逻辑(例如去掉 "a” 或 "the" 这类停用词,执行词干搜寻,将无大小写区分的所有文字转换成小写等)的同时,抽取应被编入索引的文本内容。由于和字段相关的内容减少到只剩核心元素,因此,索引作为与某个字段相关的文字内容,它的规模也被缩小了。只有被token化的字段也将被编入索引的情况下才有意义。 对Akamai.com来说,“标题”被token化,所以Lucene不用搜寻如 "a" 或 "the" 这类单词。
[java] view plaincopy
public final class Token {
String termText; // the text of the term
int startOffset; // start in source text
int endOffset; // end in source text
String type = "word"; // lexical type
private int positionIncrement = 1;
public Token(String text, int start, int end)
public Token(String text, int start, int end, String typ)
public void setPositionIncrement(int positionIncrement)
public int getPositionIncrement() { return positionIncrement; }
public final String termText() { return termText; }
public final int startOffset() { return startOffset; }
public void setStartOffset(int givenStartOffset)
public final int endOffset() { return endOffset; }
public void setEndOffset(int givenEndOffset)
public final String type() { return type; }
public String toString()
}
TokenStream是用来走访Token的iterator(迭代器)
[java] view plaincopy
public abstract class TokenStream {
public abstract Token next() throws IOException;
public void close() throws IOException {}
}
Tokenizer继承自TokenStream,其输入为Reader
[java] view plaincopy
public abstract class Tokenizer extends TokenStream {
protected Reader input;
protected Tokenizer() {}
protected Tokenizer(Reader input) {
this.input = input;
}
public void close() throws IOException {
input.close();
}
}
TokenFilter继承自TokenStream,其作用是用来完成对TokenStream的过滤操作,譬如
去StopWords,将Token变为小写等
[java] view plaincopy
public abstract class TokenFilter extends TokenStream {
protected TokenStream input;
protected TokenFilter() {}
protected TokenFilter(TokenStream input) {
this.input = input;
}
public void close() throws IOException {
input.close();
}
}
Analyzer就是一个TokenStream工厂
[java] view plaincopy
public abstract class Analyzer {
public TokenStream tokenStream(String fieldName, Reader reader){
return tokenStream(reader);
}
public TokenStream tokenStream(Reader reader){
return tokenStream(null, reader);
}
}
Analyzer的作用就是把文本分解为便于Lucene处理的token,好比人体的肠道,负责把食物分解成易于吸收的小块。
Query query = new QueryParser(Version.LUCENE_36,"text",analyzer).parse(nameen.trim());
这个是lucene内部先对nameen进行分词,比如按空格之内的,然后分词之后的结果任意一个被包含于text字段里面的内容,那么该条记录就被选中
例如:
[java] view plaincopy
public final class Token {
String termText; // the text of the term
int startOffset; // start in source text
int endOffset; // end in source text
String type = "word"; // lexical type
private int positionIncrement = 1;
public Token(String text, int start, int end)
public Token(String text, int start, int end, String typ)
public void setPositionIncrement(int positionIncrement)
public int getPositionIncrement() { return positionIncrement; }
public final String termText() { return termText; }
public final int startOffset() { return startOffset; }
public void setStartOffset(int givenStartOffset)
public final int endOffset() { return endOffset; }
public void setEndOffset(int givenEndOffset)
public final String type() { return type; }
public String toString()
}
TokenStream是用来走访Token的iterator(迭代器)
[java] view plaincopy
public abstract class TokenStream {
public abstract Token next() throws IOException;
public void close() throws IOException {}
}
Tokenizer继承自TokenStream,其输入为Reader
[java] view plaincopy
public abstract class Tokenizer extends TokenStream {
protected Reader input;
protected Tokenizer() {}
protected Tokenizer(Reader input) {
this.input = input;
}
public void close() throws IOException {
input.close();
}
}
TokenFilter继承自TokenStream,其作用是用来完成对TokenStream的过滤操作,譬如
去StopWords,将Token变为小写等
[java] view plaincopy
public abstract class TokenFilter extends TokenStream {
protected TokenStream input;
protected TokenFilter() {}
protected TokenFilter(TokenStream input) {
this.input = input;
}
public void close() throws IOException {
input.close();
}
}
Analyzer就是一个TokenStream工厂
[java] view plaincopy
public abstract class Analyzer {
public TokenStream tokenStream(String fieldName, Reader reader){
return tokenStream(reader);
}
public TokenStream tokenStream(Reader reader){
return tokenStream(null, reader);
}
}
Analyzer的作用就是把文本分解为便于Lucene处理的token,好比人体的肠道,负责把食物分解成易于吸收的小块。
Query query = new QueryParser(Version.LUCENE_36,"text",analyzer).parse(nameen.trim());
这个是lucene内部先对nameen进行分词,比如按空格之内的,然后分词之后的结果任意一个被包含于text字段里面的内容,那么该条记录就被选中
例如:
Hybrid safflower (Carthamus tinctorius) oil
分词为:+(text:hybrid text:safflower text:carthamus text:tinctorius text:oil) -id:246281
相关文章推荐
- lucene中的Token, TokenStream, Tokenizer, Analyzer
- Lucene分词实现---Analyzer、TokenStream(Token、Tokenizer、Tokenfilter)
- lucene中的Token, TokenStream, Tokenizer, Analyzer
- lucene中的Token, TokenStream, Tokenizer, Analyzer
- Lucene分词实现---Analyzer、TokenStream(Token、Tokenizer、Tokenfilter)
- Lucene中TokenStream,Tokenizer,TokenFilter,TokenStreamComponents与Analyzer
- Lucene中TokenStream,Tokenizer,TokenFilter,TokenStreamComponents与Analyzer
- lucene分词器中的Analyzer,TokenStream, Tokenizer, TokenFilter
- lucene 的分析器(analyzer)与分词器(tokenizer)和过滤器(tokenfilter)
- Lucene中TokenStream,Tokenizer,TokenFilter,TokenStreamComponents与Analyzer
- Lucene--TokenStream(TokenFilter、Tokenizer)
- 基于Lucene3.5.0怎样从TokenStream获得Token
- 基于Lucene3.5.0怎样从TokenStream获得Token
- Lucene.net(4.8.0) 学习问题记录二: 分词器Analyzer中的TokenStream和AttributeSource
- Lucene源码解析--Analyzer之Tokenizer
- 基于Lucene3.5.0怎样从TokenStream获得Token
- Lucene 3.0.0 的TokenStream与Analyzer
- 配置Paoding分词时出现的错误:java.lang.AbstractMethodError: org.apache.lucene.analysis.TokenStream.incrementToken()
- org.apache.lucene.analysis.TokenStream.incrementToken()Z
- Lucene源码解析--Analyzer之Tokenizer