How to Get All Documents in Lucene

Introduction

If you're new to Lucene, you may be wondering how to fetch all documents in your index. Querying for a specific term is straightforward, but retrieving everything requires a different approach.

There are two ways: using MatchAllDocsQuery or using *:* with QueryParser.

Using MatchAllDocsQuery

Lucene provides a special query class that matches every document in the index:

Query query = new MatchAllDocsQuery();

As the name suggests, this query has no filter — it returns all indexed documents.

Using QueryParser with *:*

The other approach is to use QueryParser with the special syntax *:*. This tells Lucene to match all terms in all fields, which is equivalent to fetching every document:

QueryParser queryParser = new QueryParser("title", analyzer);
Query query = queryParser.parse("*:*");

Full Working Example

Here's a complete example that indexes 100 documents and then fetches them all using both methods, with pagination:

public class GetAllDocs {

  public void getAllDocsWithMatchAllDocsQuery(String indexPath)
      throws IOException, ParseException {
    IndexReader indexReader = DirectoryReader.open(
      FSDirectory.open(Paths.get(indexPath)));
    IndexSearcher indexSearcher = new IndexSearcher(indexReader);

    Query query = new MatchAllDocsQuery();
    searchAndPrintResults(indexSearcher, query);
  }

  public void getAllDocsWithQueryParser(String indexPath)
      throws IOException, ParseException {
    IndexReader indexReader = DirectoryReader.open(
      FSDirectory.open(Paths.get(indexPath)));
    IndexSearcher indexSearcher = new IndexSearcher(indexReader);
    Analyzer analyzer = new StandardAnalyzer();
    QueryParser queryParser = new QueryParser("title", analyzer);

    Query query = queryParser.parse("*:*");
    searchAndPrintResults(indexSearcher, query);
  }

  private void searchAndPrintResults(IndexSearcher indexSearcher, Query query)
      throws IOException {
    TopDocs topDocs = indexSearcher.search(query, 10);
    System.out.println(String.format("Found %d hits.", topDocs.totalHits.value));

    while (topDocs.scoreDocs.length != 0) {
      ScoreDoc[] results = topDocs.scoreDocs;
      for (ScoreDoc scoreDoc : results) {
        Document doc = indexSearcher.doc(scoreDoc.doc);
        System.out.println(String.format("Found: %s", doc.get("title")));
      }
      // Fetch next page
      ScoreDoc lastDoc = results[results.length - 1];
      topDocs = indexSearcher.searchAfter(lastDoc, query, 10);
    }
  }

  public void index(String indexPath) throws IOException {
    Directory directory = FSDirectory.open(Paths.get(indexPath));
    Analyzer analyzer = new StandardAnalyzer();
    IndexWriterConfig config = new IndexWriterConfig(analyzer);
    config.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
    IndexWriter indexWriter = new IndexWriter(directory, config);

    for (int i = 1; i <= 100; i++) {
      Document doc = new Document();
      doc.add(new TextField("title", "This is document " + i, Field.Store.YES));
      indexWriter.addDocument(doc);
    }
    indexWriter.close();
  }

  public static void main(String[] args) throws IOException, ParseException {
    String indexPath = "index";
    GetAllDocs example = new GetAllDocs();
    example.index(indexPath);

    System.out.println("--- QueryParser method ---");
    example.getAllDocsWithQueryParser(indexPath);

    System.out.println("--- MatchAllDocsQuery method ---");
    example.getAllDocsWithMatchAllDocsQuery(indexPath);
  }
}

Note the pagination pattern: we use searchAfter(lastDoc, query, 10) to fetch the next page instead of fetching all documents at once. This is memory-efficient and works for arbitrarily large indexes.

Conclusion

We've seen both ways to fetch all documents from a Lucene index. MatchAllDocsQuery is the more direct and idiomatic approach. QueryParser with *:* is useful when your query is constructed as a string. Either way, pair it with searchAfter for efficient pagination.