Background
Search engines are among the most widely used software systems, and I wanted to understand how one works end to end by
building it myself. My goals were to learn an unfamiliar technology stack, design a system larger than anything I had
built before, and deliver something people could actually use. My research drew on books and articles on information
retrieval, particularly
Introduction to Information Retrieval
by Manning, Raghavan and Schütze (Cambridge University Press, 2008). Production search engines combine dozens of
specialized technologies, so I scoped the project to the foundational architecture that early search engines were built
on.
Architecture
The system is a .NET solution of three projects: a shared data-access library (SearchEngine.Data), a web crawler
(SearchEngine.WebCrawler) and a search website (SearchEngine.SearchWebsite). The crawler and the website run as
independent processes with no dependency on each other. Both reference the data library, which encapsulates the Entity
Framework Core model and all database access, so persistence logic is written once and shared. This separation of
concerns keeps each component focused and lets the crawler and the website run and be deployed independently.
The use-case diagram I designed for the project.
Crawling and Indexing
The crawler runs a configurable number of concurrent, asynchronous workers. Each worker repeats this pipeline:
- Scheduling: takes the next URL from the crawl frontier, which groups queued URLs by domain. The worker chooses
between high-priority and regular domains with equal probability, so manually curated, reliable domains are crawled
more often and are better represented in the index. The URL is dequeued immediately to prevent duplicate crawls.
- Fetching and parsing: downloads the page and parses its HTML with Html Agility Pack.
- Metadata extraction: extracts the title and the meta description, falling back to the first paragraph, and adds
the page to the document index.
- Link extraction: resolves relative links to absolute URLs and canonicalizes them, so that
https://google.com
and https://google.com/ resolve to the same page. New URLs are deduplicated against the index and the frontier,
then enqueued with their domain.
- Term weighting: concurrently with the next crawl, stems each word on the page with the Porter stemmer, so
"engine" and "engines" map to the same term, and weights it by its position (such as a heading, a paragraph or bold
text) and its frequency.
The term weights are stored in an inverted index, a table that maps each stemmed term to the pages it appears on and its
weight on each. This lets the search website look up candidate pages by term instead of scanning every page.
Query Processing
Each query arrives as an HTTP GET request handled by ASP.NET Core. The query is normalized with the same stemmer used
at indexing time, matching pages are retrieved from the inverted index, and the results are ranked by their aggregated
term weights. The results page reports the total number of matches and the server processing time, and paginates the
results.
Challenges
I deliberately chose Entity Framework Core and LINQ over the hand-written SQL I was used to, adopting a code-first
approach in which the database schema is generated from the C# domain model. The crawler was the most demanding
component, requiring substantial code and extensive debugging, which I worked through with documentation and community
resources such as Stack Overflow.
Project Details| Web crawler | Concurrent asynchronous workers with domain-based priority scheduling, URL canonicalization and deduplication. |
|---|
| Database | A SQL Server schema of six tables (the crawl frontier, domains, the document index, document metadata, terms and the term-to-document inverted index), generated code-first by Entity Framework Core. |
|---|
| Search website | An ASP.NET Core application that stems each query with the Porter algorithm and ranks matching documents by their aggregated term weights. |
|---|
| Scope | Full-text search over webpages in any language. Images, videos, news, PDF files and spam filtering are out of scope. |
|---|
| Technologies | C#, .NET Core 3.1, ASP.NET Core 3.1, Entity Framework Core 3.1, LINQ, SQL Server 2019, Html Agility Pack, Porter Stemmer, CSS. |
|---|
| Deployment | Amazon Web Services (AWS), with Amazon EC2 hosting the crawler and the website, and Amazon RDS hosting the database. |
|---|
The search website across multiple devices.