Introduction

Information retrieval begins with a simple human situation: someone needs information, but the needed information is hidden inside a larger collection.

A student remembers that a lecturer mentioned an assignment extension, but cannot remember whether the message was in email, a class forum, or a group chat. A journalist searches an archive for earlier reports about a public figure. A software engineer looks through issue trackers, documentation, and commit messages to understand why a service failed last month. A patient searches a hospital portal for instructions sent after an appointment. In each case, the problem is not merely that data exists. The problem is that useful information must be found, selected, and presented from among many possible items.

This is the central concern of information retrieval, often abbreviated as IR. Information retrieval is the study and engineering of systems that help users find material relevant to an information need from a collection of information objects. Standard IR textbooks describe this field as concerned with representing, storing, organizing, and accessing information items so that users can obtain information relevant to them (Baeza-Yates and Ribeiro-Neto, 2011; Manning, Raghavan, and Schütze, 2008). The word retrieval does not mean that the system already knows the one perfect answer and simply fetches it. More often, the system must estimate which documents are likely to be useful.

A document in information retrieval is any retrievable unit. It may be a web page, email message, chat post, PDF file, product description, source-code comment, forum thread, video transcript, audio segment, image caption, or database record treated as searchable text. A collection is the set of documents being searched. A query is the expression the user gives to the system, such as meeting notes budget, how to reset router, or messages from Lina about internship. An information need is the underlying problem in the user's mind. The query is only an imperfect expression of that need.

This distinction between query and need is one of the first ideas to learn carefully. Suppose a student types:

java memory leak

The student might mean memory leaks in the Java programming language. Another user might mean memory problems in an application named Java, or even something unrelated to programming if the collection contains documents about Java island. The query string is short, but the possible meanings are many. Information retrieval systems must work in this uncertain space between short human expressions and large document collections.

Why retrieval is not just ordinary data lookup

At first, information retrieval may look like database search. A database system can answer questions such as:

SELECT * FROM orders WHERE order_id = 19384;

This is a precise request over structured data. The field order_id has a defined meaning, and the condition order_id = 19384 is exact. If the record exists, the database returns it. If it does not, the database returns nothing.

Information retrieval is usually less exact. A user may ask for:

best explanation of inverted index

There may be many useful documents, none of them using exactly those words. One document may say “a search engine stores a mapping from terms to postings lists.” Another may say “inverted files support efficient text search.” A good retrieval system should recognize that these documents may be relevant even though they do not exactly match the query phrase. This is why IR often uses ranking: instead of returning an unordered set of exact matches, the system orders documents by estimated usefulness.

This does not mean databases and information retrieval are unrelated. Modern systems often combine both. An email search system may retrieve messages using text ranking, while also filtering by sender, recipient, date, mailbox, and access permission. A digital library may rank papers by textual relevance while restricting results to a publication year or discipline. The practical boundary between database systems and IR systems is therefore not a wall; it is a difference in emphasis. Databases traditionally emphasize exact, structured queries, while IR emphasizes uncertain, language-based matching and relevance-ranked results (Croft, Metzler, and Strohman, 2010; Manning, Raghavan, and Schütze, 2008).

Relevance: the guiding idea

The most important word in this book is relevance. A document is relevant when it helps satisfy the user's information need. This sounds simple, but it is subtle.

Consider the query:

vaccine appointment child fever after

A page titled “General history of vaccines” may contain the word “vaccine,” but it may not help a parent who wants to know what to do after a child develops a fever following an appointment. A short clinic instruction page may be more relevant, even if it contains fewer query words. Relevance depends not only on words but also on task, context, user knowledge, time, trust, and intended action. Information science has long treated relevance as a central and complex concept rather than a simple property of a document alone (Saracevic, 1975).

A useful retrieval system must therefore ask a practical question:

Given what the user typed, what documents are most likely to help?

The system rarely observes the true information need directly. It observes signals: the query words, the user's location in an interface, previous interactions, document text, metadata, links, timestamps, clicks, and sometimes explicit feedback. Retrieval is the disciplined use of these signals to estimate relevance.

This is why the field is both technical and human-centered. The technical side builds indexes, ranking functions, evaluation metrics, and scalable architectures. The human side studies how people express needs, reformulate queries, scan result lists, judge usefulness, and search over time. Belkin's idea of an “anomalous state of knowledge” is useful here: people often search because they know something is missing or unclear in their understanding, but they may not yet be able to state exactly what would resolve it (Belkin, 1980).

Communication systems as retrieval environments

This book gives special attention to communication and computing. Communication systems are full of retrieval problems.

In email, a user might search:

invoice April from Nabila

The system must understand text content, sender metadata, dates, attachments, and perhaps thread structure. In a workplace chat system, a user might search:

what did we decide about authentication?

The answer may be spread across several messages, with one message asking a question, another proposing a solution, and a later one confirming the decision. In a forum, useful information may appear in replies rather than in the first post. In a video meeting archive, the relevant document might not be a written message at all, but a speech transcript generated from audio.

Communication data has special properties. It is often conversational, time-ordered, socially situated, and permission-sensitive. A message may make sense only because of previous messages. A phrase such as “that version is approved” is hard to interpret unless the system knows what “that version” refers to. Search in communication systems therefore requires more than matching isolated words. It often involves context, meaning the surrounding information needed to interpret a message correctly.

For example, imagine these chat messages:

10:04 Maya: I uploaded the revised slides.
10:06 Arif: Looks good. Use this one for Friday.
10:07 Maya: Great, I’ll send it to the team.

If a user later searches for approved Friday slides, the most relevant evidence may be the second message, even though it does not contain the word “slides.” The meaning comes from the thread. Retrieval in communication systems must often connect messages, participants, timestamps, attachments, and references across turns.

Computing systems also depend on retrieval

Information retrieval is also central to computing systems more broadly. Search engines are the most visible example, but IR appears in many less obvious places: code search, log search, documentation search, application-store search, customer-support systems, recommender systems, enterprise knowledge bases, and retrieval-augmented artificial intelligence systems.

A software developer searching logs might type:

timeout payment service after deploy

A useful system should find not only exact matches but also related log entries, incident reports, deployment notes, and code changes. A customer-support agent searching a knowledge base may need the best troubleshooting article, not every article containing the word “error.” A large language model connected to a document store may need retrieval to supply trusted passages before generating an answer. In all these cases, retrieval is an infrastructure for making stored information usable.

This point matters for undergraduate computing students: IR is not only a topic for web search companies. It is a general discipline for building systems that connect people, tasks, and stored information.

The basic retrieval pipeline

Although modern retrieval systems can be complex, many of them share a common structure.

First, the system prepares documents. This may involve extracting text from files, splitting text into tokens, normalizing words, identifying fields such as title and author, and storing metadata such as timestamps or senders. A token is a unit of text used for processing, often a word or word-like sequence. For example, the sentence:

Search systems rank documents.

might be tokenized into:

search, systems, rank, documents

Second, the system builds an index. An index is a data structure that helps the system find candidate documents quickly. The most important classical structure is the inverted index, which maps terms to lists of documents that contain them. For example, a tiny index might record that the term retrieval occurs in documents 2, 5, and 9. Without an index, the system might need to scan every document for every query, which is usually too slow for large collections. Inverted indexes are a foundational technique in search systems (Manning, Raghavan, and Schütze, 2008).

Third, the system processes the query. It may tokenize the query, correct spelling, expand terms, apply filters, or interpret operators such as quotation marks.

Fourth, the system scores and ranks candidate documents. A ranking function assigns a score to each candidate document for a given query. Classical ranking functions use signals such as term frequency, document length, and term rarity. For instance, a document containing inverted index several times may be more likely to discuss that topic than a document containing the phrase once in passing. However, a very long document naturally contains many words, so ranking methods often adjust for document length.

Fifth, the system presents results. Presentation is not trivial. Titles, snippets, highlighted terms, facets, dates, authors, and grouping can strongly affect whether users can judge results efficiently. A good system does not only compute a ranking; it helps users understand and act on that ranking.

Finally, the system is evaluated. An IR system should be tested, not merely admired. Evaluation asks whether the system retrieves relevant documents, avoids irrelevant ones, ranks useful results highly, responds quickly, respects access control, and serves users fairly. Measures such as precision, recall, mean average precision, and normalized discounted cumulative gain will be introduced later in the book. These measures give us disciplined ways to compare retrieval methods.

A small example

Suppose we have four documents in a class discussion archive:

D1: The exam will cover indexing, Boolean retrieval, and ranking.
D2: Please upload your group project slides by Friday.
D3: The inverted index maps terms to postings lists.
D4: Ranking functions estimate which documents are useful for a query.

A student searches:

index ranking

An exact system might find documents containing exactly index and ranking. It may return D1 if stemming or normalization treats indexing as related to index, and it may return D4 because it contains ranking. It may also return D3 because inverted index is clearly relevant, even though D3 does not contain ranking.

A ranked retrieval system would try to order the results. D1 might rank highly because it discusses both indexing and ranking as exam topics. D3 might be useful for the indexing part of the need. D4 might be useful for the ranking part. The best order depends on what the student actually needs. If the student is preparing for an exam, D1 may be most useful. If the student wants to understand data structures, D3 may be better. If the student wants scoring methods, D4 may be better.

This example shows why retrieval is not just word matching. It involves representation, uncertainty, ranking, and user purpose.

What this book will build

The chapters that follow develop information retrieval step by step.

We begin with the objects of retrieval: documents, queries, collections, users, information needs, and relevance. These concepts may look ordinary, but careful definitions prevent confusion later. We then study text processing and representation, because retrieval systems cannot search raw human language directly without making design choices. Tokenization, normalization, stop words, stemming, lemmatization, n-grams, metadata, and document fields all influence what the system can find.

Next we study indexing. The inverted index is one of the great practical ideas of computing: simple in principle, powerful in scale. With it, search systems can avoid scanning every document for every query. We then study Boolean retrieval and exact matching, where queries are treated as logical conditions. Boolean retrieval remains important in legal search, enterprise search, library systems, and expert search interfaces, even though many everyday users expect ranked results.

After exact matching, we move to ranked retrieval. The vector space model and TF-IDF show how documents and queries can be represented numerically, and how similarity can be computed. Probabilistic retrieval and BM25 then show how ranking can be grounded in assumptions about relevance, term frequency, and document length. BM25 remains an important classical ranking function in practical search systems and is widely discussed in IR literature (Manning, Raghavan, and Schütze, 2008; Croft, Metzler, and Strohman, 2010).

Evaluation comes next because retrieval engineering without measurement easily becomes guesswork. We will learn how test collections, relevance judgments, and ranking metrics allow systems to be compared. Then we turn to users and interfaces: query suggestions, snippets, faceted navigation, relevance feedback, and exploratory search.

The middle chapters apply retrieval to web search, communication platforms, and multimedia. Web search introduces crawling, link analysis, duplicate detection, spam resistance, and freshness. Communication retrieval introduces email, chat, forums, social media, call transcripts, collaborative platforms, and knowledge bases. Multimedia retrieval extends the same retrieval principles to images, audio, video, captions, transcripts, and cross-modal search.

The later chapters introduce machine learning, neural retrieval, semantic search, retrieval-augmented generation, scalable architecture, and responsible deployment. These topics are modern, but they rest on the same foundations: representation, indexing, matching, ranking, evaluation, and user need. Learning the foundations first will make modern systems easier to understand rather than mysterious.

How to think while reading

As you read, keep returning to three questions.

First, what is being represented? A document, a query, a user, a conversation, an image, a passage, or a whole collection may each need a different representation.

Second, what is being optimized? A system might optimize exact matching, relevance ranking, response time, user satisfaction, freshness, fairness, privacy, or some combination of these. No retrieval system optimizes everything equally.

Third, what evidence is being used? Words are evidence. Metadata is evidence. Links are evidence. Clicks are evidence, though imperfect and biased. Conversation structure is evidence. Embeddings are evidence. A retrieval system is a machine for turning evidence into ranked access to information.

A strong student of information retrieval learns to ask not only “What algorithm is used?” but also “What assumptions does this algorithm make, and when might those assumptions fail?”

The promise and responsibility of retrieval

Information retrieval systems shape what people see. They influence which documents are read, which answers are trusted, which memories are recoverable, and which voices are visible. This gives retrieval enormous practical value, but also real responsibility.

A search system for private messages must respect access control and privacy. A ranking system for news or health information must be careful about misinformation and source quality. A recommender or personalized search system may help users find what they like, but may also narrow what they encounter. A workplace knowledge search tool may improve productivity, but it may also expose sensitive information if designed poorly.

For this reason, this book treats retrieval as both a technical discipline and a social technology. The goal is not only to build systems that return results quickly. The goal is to build systems that help people find useful information reliably, responsibly, and with an honest understanding of their limitations.

Information retrieval is one of the quiet foundations of contemporary communication and computing. Every time a person searches a mailbox, scans a result page, asks a system to find a document, or relies on retrieved context for an answer, IR is at work. The rest of this book will show how that work is done.

References

Baeza-Yates, R., and Ribeiro-Neto, B. (2011). Modern Information Retrieval: The Concepts and Technology behind Search (2nd ed.). Addison-Wesley.

Belkin, N. J. (1980). Anomalous states of knowledge as a basis for information retrieval. Canadian Journal of Information Science, 5, 133–143.

Croft, W. B., Metzler, D., and Strohman, T. (2010). Search Engines: Information Retrieval in Practice. Addison-Wesley.

Manning, C. D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Journal of the American Society for Information Science, 26(6), 321–343.