Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
 PREV PACKAGE   NEXT PACKAGE FRAMES    NO FRAMES    

Package org.apache.lucene.search

Code to search indices.

See:
          Description

Interface Summary
FieldCache Expert: Maintains caches of term values.
FieldCache.ByteParser Interface to parse bytes from document fields.
FieldCache.DoubleParser Interface to parse doubles from document fields.
FieldCache.FloatParser Interface to parse floats from document fields.
FieldCache.IntParser Interface to parse ints from document fields.
FieldCache.LongParser Interface to parse long from document fields.
FieldCache.Parser Marker interface as super-interface to all parsers.
FieldCache.ShortParser Interface to parse shorts from document fields.
NRTManager.WaitingListener NRTManager invokes this interface to notify it when a caller is waiting for a specific generation searcher to be visible.
RMIRemoteSearchable Deprecated. This package (all of contrib/remote) will be removed in 4.0.
Searchable Deprecated. In 4.0 this interface is removed/absorbed into IndexSearcher
SearcherLifetimeManager.Pruner See SearcherLifetimeManager.prune(org.apache.lucene.search.SearcherLifetimeManager.Pruner).
 

Class Summary
AssertingIndexSearcher Helper class that adds some extra checks to ensure correct usage of IndexSearcher and Weight.
BooleanClause A clause in a BooleanQuery.
BooleanFilter A container Filter that allows Boolean composition of Filters.
BooleanQuery A Query that matches documents matching boolean combinations of other queries, e.g.
BoostingQuery The BoostingQuery class can be used to effectively demote results that match a given query.
CachingCollector Caches all docs, and optionally also scores, coming from a search, and is then able to replay them to another collector.
CachingSpanFilter Wraps another SpanFilter's result and caches it.
CachingWrapperFilter Wraps another filter's result and caches it.
CachingWrapperFilterHelper A unit test helper class to test when the filter is getting cached and when it is not.
ChainedFilter Allows multiple Filters to be chained.
CheckHits Utility class for asserting expected hits in tests.
CheckHits.ExplanationAsserter Asserts that the score explanation for every document matching a query corresponds with the true score.
CheckHits.ExplanationAssertingSearcher an IndexSearcher that implicitly checks hte explanation of every match whenever it executes a search.
CheckHits.SetCollector Just collects document ids into a set.
Collector Expert: Collectors are primarily meant to be used to gather raw results from a search, and implement sorting or custom result filtering, collation, etc.
ComplexExplanation Expert: Describes the score computation for document and query, and can distinguish a match independent of a positive value.
ConstantScoreQuery A query that wraps another query or a filter and simply returns a constant score equal to the query boost for every document that matches the filter or query.
DefaultSimilarity Expert: Default scoring implementation.
DisjunctionMaxQuery A query that generates the union of documents produced by its subqueries, and that scores each document with the maximum score for that document as produced by any subquery, plus a tie breaking increment for any additional matching subqueries.
DocIdSet A DocIdSet contains a set of doc ids.
DocIdSetIterator This abstract class defines methods to iterate over a set of non-decreasing doc ids.
DuplicateFilter  
Explanation Expert: Describes the score computation for document and query.
Explanation.IDFExplanation Small Util class used to pass both an idf factor as well as an explanation for that factor.
FieldCache.CacheEntry EXPERT: A unique Identifier/Description for each item in the FieldCache.
FieldCache.CreationPlaceholder  
FieldCache.StringIndex Expert: Stores term text values and document ordering data.
FieldCacheDocIdSet Base class for DocIdSet to be used with FieldCache.
FieldCacheRangeFilter<T> A range filter built on top of a cached single term field (in FieldCache).
FieldCacheTermsFilter A Filter that only accepts documents whose single term value in the specified field is contained in the provided set of allowed terms.
FieldComparator<T> Expert: a FieldComparator compares hits so as to determine their sort order when collecting the top results with TopFieldCollector.
FieldComparator.ByteComparator Parses field's values as byte (using FieldCache.getBytes(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.DocComparator Sorts by ascending docID
FieldComparator.DoubleComparator Parses field's values as double (using FieldCache.getDoubles(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.FloatComparator Parses field's values as float (using FieldCache.getFloats(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.IntComparator Parses field's values as int (using FieldCache.getInts(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.LongComparator Parses field's values as long (using FieldCache.getLongs(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.NumericComparator<T extends Number>  
FieldComparator.RelevanceComparator Sorts by descending relevance.
FieldComparator.ShortComparator Parses field's values as short (using FieldCache.getShorts(org.apache.lucene.index.IndexReader, java.lang.String) and sorts by ascending value
FieldComparator.StringComparatorLocale Sorts by a field's value using the Collator for a given Locale.
FieldComparator.StringOrdValComparator Sorts by field's natural String sort order, using ordinals.
FieldComparator.StringValComparator Sorts by field's natural String sort order.
FieldComparatorSource Provides a FieldComparator for custom field sorting.
FieldDoc Expert: A ScoreDoc which also contains information about how to sort the referenced document.
FieldValueFilter A Filter that accepts all documents that have one or more values in a given field.
FieldValueHitQueue<T extends FieldValueHitQueue.Entry> Expert: A hit queue for sorting by hits by terms in more than one field.
FieldValueHitQueue.Entry  
Filter Abstract base class for restricting which documents may be returned during searching.
FilterClause A Filter that wrapped with an indication of how that filter is used when composed with another filter.
FilteredDocIdSet Abstract decorator class for a DocIdSet implementation that provides on-demand filtering/validation mechanism on a given DocIdSet.
FilteredDocIdSetIterator Abstract decorator class of a DocIdSetIterator implementation that provides on-demand filter/validation mechanism on an underlying DocIdSetIterator.
FilteredQuery A query that applies a filter to the results of another query.
FilteredTermEnum Abstract class for enumerating a subset of all terms.
FilterManager Deprecated. used by remote package which is deprecated as well.
FuzzyLikeThisQuery Fuzzifies ALL terms provided as strings and then picks the best n differentiating terms.
FuzzyQuery Implements the fuzzy search query.
FuzzyTermEnum Subclass of FilteredTermEnum for enumerating all terms that are similar to the specified filter term.
IndexSearcher Implements search over a single IndexReader.
MatchAllDocsQuery A query that matches all documents.
MultiCollector A Collector which allows running a search with several Collectors.
MultiPhraseQuery MultiPhraseQuery is a generalized version of PhraseQuery, with an added method MultiPhraseQuery.add(Term[]).
MultiSearcher Deprecated. If you are using MultiSearcher over IndexSearchers, please use MultiReader instead; this class does not properly handle certain kinds of queries (see LUCENE-2756).
MultiTermQuery An abstract Query that matches documents containing a subset of terms provided by a FilteredTermEnum enumeration.
MultiTermQuery.ConstantScoreAutoRewrite A rewrite method that tries to pick the best constant-score rewrite method based on term and document counts from the query.
MultiTermQuery.RewriteMethod Abstract class that defines how the query is rewritten.
MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewrite A rewrite method that first translates each term into BooleanClause.Occur.SHOULD clause in a BooleanQuery, but the scores are only computed as the boost.
MultiTermQuery.TopTermsScoringBooleanQueryRewrite A rewrite method that first translates each term into BooleanClause.Occur.SHOULD clause in a BooleanQuery, and keeps the scores as computed by the query.
MultiTermQueryWrapperFilter<Q extends MultiTermQuery> A wrapper for MultiTermQuery, that exposes its functionality as a Filter.
NGramPhraseQuery This is a PhraseQuery which is optimized for n-gram phrase query.
NRTManager Utility class to manage sharing near-real-time searchers across multiple searching thread.
NRTManager.TrackingIndexWriter Class that tracks changes to a delegated IndexWriter.
NRTManagerReopenThread Utility class that runs a reopen thread to periodically reopen the NRT searchers in the provided NRTManager.
NumericRangeFilter<T extends Number> A Filter that only accepts numeric values within a specified range.
NumericRangeQuery<T extends Number> A Query that matches numeric values within a specified range.
ParallelMultiSearcher Deprecated. Please pass an ExecutorService to IndexSearcher, instead.
PhraseQuery A Query that matches documents containing a particular sequence of terms.
PositiveScoresOnlyCollector A Collector implementation which wraps another Collector and makes sure only documents with scores > 0 are collected.
PrefixFilter A Filter that restricts search results to values that have a matching prefix in a given field.
PrefixQuery A Query that matches documents containing terms with a specified prefix.
PrefixTermEnum Subclass of FilteredTermEnum for enumerating all terms that match the specified prefix filter term.
Query The abstract base class for queries.
QueryTermVector  
QueryUtils Utility class for sanity-checking queries.
QueryWrapperFilter Constrains search results to only match those which also match a provided query.
ReferenceManager<G> Utility class to safely share instances of a certain type across multiple threads, while periodically refreshing them.
RemoteCachingWrapperFilter Deprecated. This package (all of contrib/remote) will be removed in 4.0.
RemoteSearchable Deprecated. This package (all of contrib/remote) will be removed in 4.0.
ScoreCachingWrappingScorer A Scorer which wraps another scorer and caches the score of the current document.
ScoreDoc Holds one hit in TopDocs.
Scorer Expert: Common scoring functionality for different types of queries.
Scorer.ScorerVisitor<P extends Query,C extends Query,S extends Scorer> A callback to gather information from a scorer and its sub-scorers.
ScoringRewrite<Q extends Query>  
SearchEquivalenceTestBase Simple base class for checking search equivalence.
Searcher Deprecated. In 4.0 this abstract class is removed/absorbed into IndexSearcher
SearcherFactory Factory class used by SearcherManager and NRTManager to create new IndexSearchers.
SearcherLifetimeManager Keeps track of current plus old IndexSearchers, closing the old ones once they have timed out.
SearcherLifetimeManager.PruneByAge Simple pruner that drops any searcher older by more than the specified seconds, than the newest searcher.
SearcherManager Utility class to safely share IndexSearcher instances across multiple threads, while periodically reopening.
Similarity Expert: Scoring API.
SimilarityDelegator Deprecated. this class will be removed in 4.0.
SingleTermEnum Subclass of FilteredTermEnum for enumerating a single term.
Sort Encapsulates sort criteria for returned hits.
SortField Stores information about how to sort documents by terms in an individual field.
SpanFilter Abstract base class providing a mechanism to restrict searches to a subset of an index and also maintains and returns position information.
SpanFilterResult The results of a SpanQueryFilter.
SpanFilterResult.PositionInfo  
SpanFilterResult.StartEnd  
SpanQueryFilter Constrains search results to only match those which also match a provided query.
TermQuery A Query that matches documents containing a term.
TermRangeFilter A Filter that restricts search results to a range of term values in a given field.
TermRangeQuery A Query that matches documents within an range of terms.
TermRangeTermEnum Subclass of FilteredTermEnum for enumerating all terms that match the specified range parameters.
TermsFilter Constructs a filter for docs matching any of the terms added to this class.
TimeLimitingCollector The TimeLimitingCollector is used to timeout search requests that take longer than the maximum allowed search time limit.
TimeLimitingCollector.TimerThread  
TopDocs Represents hits returned by Searcher.search(Query,Filter,int) and Searcher.search(Query,int).
TopDocsCollector<T extends ScoreDoc> A base class for all collectors that return a TopDocs output.
TopFieldCollector A Collector that sorts by SortField using FieldComparators.
TopFieldDocs Represents hits returned by Searcher.search(Query,Filter,int,Sort).
TopScoreDocCollector A Collector implementation that collects the top-scoring hits, returning them as a TopDocs.
TopTermsRewrite<Q extends Query> Base rewrite method for collecting only the top terms via a priority queue.
TotalHitCountCollector Just counts the total number of hits.
Weight Expert: Calculate query weights and build query scorers.
WildcardQuery Implements the wildcard search query.
WildcardTermEnum Subclass of FilteredTermEnum for enumerating all terms that match the specified wildcard filter term.
 

Enum Summary
BooleanClause.Occur Specifies how clauses are to occur in matching documents.
CachingWrapperFilter.DeletesMode Expert: Specifies how new deletions against a reopened reader should be handled.
 

Exception Summary
BooleanQuery.TooManyClauses Thrown when an attempt is made to add more than BooleanQuery.getMaxClauseCount() clauses.
TimeLimitingCollector.TimeExceededException Thrown when elapsed search time exceeds allowed search time.
 

Package org.apache.lucene.search Description

Code to search indices.

Table Of Contents

  1. Search Basics
  2. The Query Classes
  3. Changing the Scoring

Search

Search over indices. Applications usually call Searcher.search(Query,int) or Searcher.search(Query,Filter,int).

Query Classes

TermQuery

Of the various implementations of Query, the TermQuery is the easiest to understand and the most often used in applications. A TermQuery matches all the documents that contain the specified Term, which is a word that occurs in a certain Field. Thus, a TermQuery identifies and scores all Documents that have a Field with the specified string in it. Constructing a TermQuery is as simple as: 

        TermQuery tq = new TermQuery(new Term("fieldName", "term"));
In this example, the Query identifies all Documents that have the Field named "fieldName" containing the word "term".

BooleanQuery

Things start to get interesting when one combines multiple TermQuery instances into a BooleanQuery. A BooleanQuery contains multiple BooleanClauses, where each clause contains a sub-query (Query instance) and an operator (from BooleanClause.Occur) describing how that sub-query is combined with the other clauses:

  1. SHOULD — Use this operator when a clause can occur in the result set, but is not required. If a query is made up of all SHOULD clauses, then every document in the result set matches at least one of these clauses.

  2. MUST — Use this operator when a clause is required to occur in the result set. Every document in the result set will match all such clauses.

  3. MUST NOT — Use this operator when a clause must not occur in the result set. No document in the result set will match any such clauses.

Boolean queries are constructed by adding two or more BooleanClause instances. If too many clauses are added, a TooManyClauses exception will be thrown during searching. This most often occurs when a Query is rewritten into a BooleanQuery with many TermQuery clauses, for example by WildcardQuery. The default setting for the maximum number of clauses 1024, but this can be changed via the static method setMaxClauseCount in BooleanQuery.

Phrases

Another common search is to find documents containing certain phrases. This is handled two different ways:

  1. PhraseQuery — Matches a sequence of Terms. PhraseQuery uses a slop factor to determine how many positions may occur between any two terms in the phrase and still be considered a match.

  2. SpanNearQuery — Matches a sequence of other SpanQuery instances. SpanNearQuery allows for much more complicated phrase queries since it is constructed from other SpanQuery instances, instead of only TermQuery instances.

TermRangeQuery

The TermRangeQuery matches all documents that occur in the exclusive range of a lower Term and an upper Term. according to String.compareTo(String). It is not intended for numerical ranges, use NumericRangeQuery instead. For example, one could find all documents that have terms beginning with the letters a through c. This type of Query is frequently used to find documents that occur in a specific date range.

NumericRangeQuery

The NumericRangeQuery matches all documents that occur in a numeric range. For NumericRangeQuery to work, you must index the values using a special NumericField.

PrefixQuery, WildcardQuery

While the PrefixQuery has a different implementation, it is essentially a special case of the WildcardQuery. The PrefixQuery allows an application to identify all documents with terms that begin with a certain string. The WildcardQuery generalizes this by allowing for the use of * (matches 0 or more characters) and ? (matches exactly one character) wildcards. Note that the WildcardQuery can be quite slow. Also note that WildcardQuery should not start with * and ?, as these are extremely slow. To remove this protection and allow a wildcard at the beginning of a term, see method setAllowLeadingWildcard in QueryParser.

FuzzyQuery

A FuzzyQuery matches documents that contain terms similar to the specified term. Similarity is determined using Levenshtein (edit) distance. This type of query can be useful when accounting for spelling variations in the collection.

Changing Similarity

Chances are DefaultSimilarity is sufficient for all your searching needs. However, in some applications it may be necessary to customize your Similarity implementation. For instance, some applications do not need to distinguish between shorter and longer documents (see a "fair" similarity).

To change Similarity, one must do so for both indexing and searching, and the changes must happen before either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it just isn't well-defined what is going to happen.

To make this change, implement your own Similarity (likely you'll want to simply subclass DefaultSimilarity) and then use the new class by calling IndexWriter.setSimilarity before indexing and Searcher.setSimilarity before searching.

If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at Overriding Similarity. In summary, here are a few use cases:

  1. SweetSpotSimilarity — SweetSpotSimilarity gives small increases as the frequency increases a small amount and then greater increases when you hit the "sweet spot", i.e. where you think the frequency of terms is more significant.

  2. Overriding tf — In some applications, it doesn't matter what the score of a document is as long as a matching term occurs. In these cases people have overridden Similarity to return 1 from the tf() method.

  3. Changing Length Normalization — By overriding lengthNorm, it is possible to discount how the length of a field contributes to a score. In DefaultSimilarity, lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be 1 / (numTerms in field), all fields will be treated "fairly".

In general, Chris Hostetter sums it up best in saying (from the Lucene users's mailing list):
[One would override the Similarity in] ... any situation where you know more about your data then just that it's "text" is a situation where it *might* make sense to to override your Similarity method.

Changing Scoring — Expert Level

Changing scoring is an expert level task, so tread carefully and be prepared to share your code if you want help.

With the warning out of the way, it is possible to change a lot more than just the Similarity when it comes to scoring in Lucene. Lucene's scoring is a complex mechanism that is grounded by three main classes:

  1. Query — The abstract object representation of the user's information need.
  2. Weight — The internal interface representation of the user's Query, so that Query objects may be reused.
  3. Scorer — An abstract class containing common functionality for scoring. Provides both scoring and explanation capabilities.
Details on each of these classes, and their children, can be found in the subsections below.

The Query Class

In some sense, the Query class is where it all begins. Without a Query, there would be nothing to score. Furthermore, the Query class is the catalyst for the other scoring classes as it is often responsible for creating them or coordinating the functionality between them. The Query class has several methods that are important for derived classes:

  1. createWeight(Searcher searcher) — A Weight is the internal representation of the Query, so each Query implementation must provide an implementation of Weight. See the subsection on The Weight Interface below for details on implementing the Weight interface.
  2. rewrite(IndexReader reader) — Rewrites queries into primitive queries. Primitive queries are: TermQuery, BooleanQuery, and other queries that implement Query.html#createWeight(Searcher searcher)

The Weight Interface

The Weight interface provides an internal representation of the Query so that it can be reused. Any Searcher dependent state should be stored in the Weight implementation, not in the Query class. The interface defines six methods that must be implemented:

  1. Weight#getQuery() — Pointer to the Query that this Weight represents.
  2. Weight#getValue() — The weight for this Query. For example, the TermQuery.TermWeight value is equal to the idf^2 * boost * queryNorm
  3. Weight#sumOfSquaredWeights() — The sum of squared weights. For TermQuery, this is (idf * boost)^2
  4. Weight#normalize(float) — Determine the query normalization factor. The query normalization may allow for comparing scores between queries.
  5. Weight#scorer(IndexReader, boolean, boolean) — Construct a new Scorer for this Weight. See The Scorer Class below for help defining a Scorer. As the name implies, the Scorer is responsible for doing the actual scoring of documents given the Query.
  6. Weight#explain(Searcher, IndexReader, int) — Provide a means for explaining why a given document was scored the way it was.

The Scorer Class

The Scorer abstract class provides common scoring functionality for all Scorer implementations and is the heart of the Lucene scoring process. The Scorer defines the following abstract (some of them are not yet abstract, but will be in future versions and should be considered as such now) methods which must be implemented (some of them inherited from DocIdSetIterator ):

  1. DocIdSetIterator#nextDoc() — Advances to the next document that matches this Query, returning true if and only if there is another document that matches.
  2. DocIdSetIterator#docID() — Returns the id of the Document that contains the match. It is not valid until next() has been called at least once.
  3. Scorer#score(Collector) — Scores and collects all matching documents using the given Collector.
  4. Scorer#score() — Return the score of the current document. This value can be determined in any appropriate way for an application. For instance, the TermScorer returns the tf * Weight.getValue() * fieldNorm.
  5. DocIdSetIterator#advance(int) — Skip ahead in the document matches to the document whose id is greater than or equal to the passed in value. In many instances, advance can be implemented more efficiently than simply looping through all the matching documents until the target document is identified.

Why would I want to add my own Query?

In a nutshell, you want to add your own custom Query implementation when you think that Lucene's aren't appropriate for the task that you want to do. You might be doing some cutting edge research or you need more information back out of Lucene (similar to Doug adding SpanQuery functionality).

Overview   Package  Class  Use  Tree  Deprecated  Index  Help 
 PREV PACKAGE   NEXT PACKAGE FRAMES    NO FRAMES