How I Built a Search Engine for Wikipedia

Introduction

The best way to learn a new technology is to get your hands dirty. When I was learning search engineering, I found myself increasingly curious about how Google and Amazon search actually work. I read information retrieval theory, moved on to Apache Lucene and Solr — but I wasn't confident I was learning real skills until I decided to build something from scratch.

That's how wikisearchengine.com was born: a full-text search engine over all of English Wikipedia.

Tech Stack at a Glance

• Backend: Java 11, Spring Boot, Apache Solr 8
• Frontend: HTML, JavaScript, jQuery, Bootstrap 5
• Infrastructure: DigitalOcean droplets and block storage
• Tools: Git, GitHub, Maven
• Libraries: WikiClean, IndexWikipedia

Getting the Wikipedia Data

Wikipedia makes its complete data available for download at the wiki dumps website. The data is available in both database and XML format. I went with XML to avoid running a database.

The file I chose was the English Wikipedia article pages dump — roughly 20 GB compressed, containing approximately 11.5 million articles.

Parsing the XML Dump

After downloading the data, I needed to extract indexable text. I searched for a Java library and found a project called IndexWikipedia on GitHub. It uses a parser from Lucene's benchmarks library that can iteratively extract article title, body, page ID, and date from the dump.

The key advantage: it's built on the SAX XML parser, so it never loads the full 20 GB dump into memory. It streams through the file, making it viable on modest hardware.

Removing Wikipedia Markup

Raw text extracted from the XML dump is in Wikipedia's markup language — full of [[links]], {{templates}}, and other formatting that would corrupt the search index.

I used WikiClean to strip this. The API is beautifully simple:

WikiClean cleaner = new WikiClean();
String plainText = cleaner.clean(rawWikiMarkup);

That single call strips all markup and returns clean plain text ready for indexing.

Choosing Between Lucene and Solr

I had three options: Apache Lucene, Apache Solr, and Elasticsearch. I ruled out Elasticsearch early since I wanted to get better at the technology I was using in my day job — Apache Solr.

A colleague wisely steered me away from bare-bones Lucene: "You'll spend most of your time writing plumbing code, not learning search." Solr gives you that plumbing out of the box.

Solr also has another advantage over Lucene for this use case: it exposes an HTTP interface, handles distributed indexing, and manages performance automatically on top of Lucene's core.

Indexing in Solr

I used SolrJ — the official Java client — rather than issuing raw HTTP requests. The indexing steps were:

1. Create a Solr collection
2. Batch-index documents (individual document updates are too slow at this scale)

Each Wikipedia article became a Solr document with four fields:

• id — the Wikipedia page ID, used to construct links back to Wikipedia
• title — the article title
• body — the full article text
• date — the article's last modified date

Selecting the Right Field Types

Field type selection is critical in Solr. The wrong type and search breaks silently.

• id → String — exact match, no tokenization needed
• title → Text — full-text search, partial word matching
• body → Text — same as title
• date → DateField — enables date range filtering

I used Solr's dynamic fields to avoid modifying the schema: adding _t suffix for text fields (title_t, body_t), _s for string (id), and _dt for date (date_dt). Solr auto-maps these to the correct types.

Creating the Search API

I built a Spring Boot REST service sitting between Solr and the frontend. Two reasons for this:

1. Exposing Solr directly to the web would allow anyone to modify or delete indexed data
2. A backend service lets me write relevancy tuning logic without touching the frontend

The search logic:

• Search the query in both title and body fields
• Boost title matches by 20×, reduce body boost to 0.5 — a match in the title is 40× more important than in the body
• Paginated results via page and resultsPerPage parameters, capped at 20 results per page to prevent abuse

The Frontend

I deliberately kept the frontend simple. No heavy framework — just Bootstrap, jQuery, and custom CSS. Clean UI, fast to build, easy to maintain. The goal was to make it easy for anyone to test the search engine, not to showcase frontend engineering.

Deployment

I chose DigitalOcean over AWS because I needed at least 2 GB RAM and 100 GB disk cheaply. I created a droplet with 2 GB RAM and 50 GB SSD, plus 50 GB additional block storage for the index.

Maven's assembly plugin created deployment ZIPs for both the indexer and searcher modules. Simple startup scripts handled restarts. I then pointed the domain wikisearchengine.com (purchased on GoDaddy for a few dollars) to the server IP — and within 15 minutes the app was live.

Conclusion

Building this from scratch taught me more about search engineering than any book or course could. Parsing a 20 GB XML dump, tuning field types in Solr, and shipping a live API that returns results across 11.5 million articles — it's the kind of project that sticks.

I plan to add features in the future: better relevancy tuning, NLP-based query understanding, and maybe some machine learning to better interpret user intent. If you want to see it in action, visit wikisearchengine.com or check out the source on GitHub.