org.apache.solr.analysis
Class RemoveDuplicatesTokenFilter

java.lang.Object
  org.apache.lucene.analysis.TokenStream
      org.apache.solr.analysis.BufferedTokenStream
          org.apache.solr.analysis.RemoveDuplicatesTokenFilter

public class RemoveDuplicatesTokenFilter
extends BufferedTokenStream

A TokenFilter which filters out Tokens at the same position and Term text as the previous token in the stream.

Constructor Summary

RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)

Method Summary

protected org.apache.lucene.analysis.Token

process(org.apache.lucene.analysis.Token t) Process a token.

Methods inherited from class org.apache.solr.analysis.BufferedTokenStream

next, output, peek, pushBack, read, write

Methods inherited from class org.apache.lucene.analysis.TokenStream

close, reset

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RemoveDuplicatesTokenFilter

public RemoveDuplicatesTokenFilter(org.apache.lucene.analysis.TokenStream input)

Method Detail

process

protected org.apache.lucene.analysis.Token process(org.apache.lucene.analysis.Token t)
                                            throws java.io.IOException

Description copied from class: BufferedTokenStream

Process a token. Subclasses may read more tokens from the input stream, write more tokens to the output stream, or simply return the next token to be output. Subclasses may return null if the token is to be dropped. If a subclass writes tokens to the output stream and returns a non-null Token, the returned Token is considered to be at the head of the token output stream.

Specified by:

process in class BufferedTokenStream

Throws:

java.io.IOException