Introduction

Modern neural networks are systems for learning functions from data. A function is a rule that maps an input to an output: an image to a label, a sentence prefix to a next word, a molecular graph to a predicted property, or a user query to a structured answer. A neural network is a particular family of parameterized functions, usually written

\[ f_\theta : \mathcal{X} \rightarrow \mathcal{Y}, \]

where \(\mathcal{X}\) is the input space, \(\mathcal{Y}\) is the output space, and \(\theta\) denotes the parameters: numerical values, such as weights and biases, that are adjusted during learning.

This simple notation hides a major scientific and engineering achievement. The field began with idealized mathematical models of neurons, such as the threshold logic units studied by McCulloch and Pitts, and later the perceptron proposed by Rosenblatt as a trainable pattern-recognition model (McCulloch & Pitts, 1943; Rosenblatt, 1958). Today, neural networks include convolutional systems for images, recurrent and attention-based systems for sequences, diffusion models for generation, and large language models trained on vast text corpora. The same basic question runs through all of them:

How can a machine improve its behavior by adjusting numerical parameters in response to data?

This book develops that question from first principles.

Why neural networks matter

A traditional program is usually written by a human as explicit rules. For example, a programmer might write rules for sorting numbers, computing taxes, or parsing a simple file format. But many important tasks resist hand-written rules. It is hard to write exact instructions for recognizing a cat in a photograph, translating informal speech, summarizing a legal document, or generating a plausible continuation of a paragraph.

Neural networks approach such tasks differently. Instead of specifying every rule manually, we choose a flexible family of functions and let data shape the parameters. A neural network for image classification, for example, may receive many image–label pairs:

\[ (\text{image of dog}, \text{dog}), \quad (\text{image of cat}, \text{cat}), \quad (\text{image of car}, \text{car}). \]

During training, the network predicts labels, measures its errors, and updates its parameters so that future predictions improve. The update is typically guided by gradient descent, an optimization method that changes parameters in the direction that locally reduces a chosen error measure. The modern form of efficient gradient-based learning in multilayer networks is closely associated with the backpropagation algorithm, whose importance was demonstrated in the 1980s by Rumelhart, Hinton, and Williams (Rumelhart et al., 1986).

The central idea is not that the machine “understands” in a human way merely because it has many parameters. The central idea is more precise: a neural network can learn useful internal representations—numerical encodings of patterns in data—that support prediction, classification, generation, or control. Deep learning became especially powerful because multilayer networks can learn hierarchies of representations, a point emphasized in modern surveys of the field (LeCun et al., 2015).

For example, in a vision model, early layers may respond to local edges or color contrasts, later layers may combine these into textures and object parts, and final layers may support classification. In a language model, early computations may encode token identities and local syntactic patterns, while later computations may represent longer-range dependencies, topic information, or task-relevant abstractions. These descriptions are simplified; real networks do not always separate knowledge into clean human-interpretable layers. But the example captures the essential concept of representation learning: instead of relying entirely on features designed by humans, the model learns features from data.

The basic learning loop

Almost every neural-network training procedure has four ingredients.

First, there is a model, a parameterized function \(f_\theta\). For a simple binary classifier, the model might take a vector \(x \in \mathbb{R}^d\) and output a number between 0 and 1 interpreted as the estimated probability of class 1. For a language model, the input may be a sequence of tokens, and the output may be a probability distribution over the next token.

Second, there is data. In supervised learning, data come with target outputs, such as labels or real-valued measurements. In self-supervised learning, the targets are derived from the data itself. For example, a language model can be trained to predict the next token in a text sequence. No human needs to label each next-token target separately, because the next token is already present in the original text.

Third, there is a loss function. A loss function assigns a numerical penalty to bad predictions. If the model predicts a high probability for the correct answer, the loss is small. If it predicts a low probability for the correct answer, the loss is large. In language modeling, a common loss is the negative log-likelihood of the correct next token. If the correct token is \(y\) and the model assigns it probability \(p_\theta(y \mid x)\), the loss for that example is

\[ -\log p_\theta(y \mid x). \]

This expression has a useful interpretation: it strongly penalizes assigning very small probability to the observed correct token.

Fourth, there is an optimizer, an algorithm that adjusts \(\theta\) to reduce the loss. The simplest conceptual optimizer is gradient descent. If \(L(\theta)\) is the loss as a function of parameters, gradient descent updates parameters by

\[ \theta \leftarrow \theta - \eta \nabla_\theta L(\theta), \]

where \(\nabla_\theta L(\theta)\) is the gradient and \(\eta > 0\) is the learning rate. The gradient points in the direction of steepest local increase of the loss, so subtracting it moves the parameters in a direction of local decrease, provided the step is not too large.

This loop—predict, measure error, compute gradients, update parameters—is the engine of much of modern neural-network training.

A first example: a single artificial neuron

The simplest artificial neuron takes input numbers \(x_1, x_2, \ldots, x_d\), multiplies them by weights \(w_1, w_2, \ldots, w_d\), adds a bias \(b\), and passes the result through an activation function \(\phi\):

\[ h = \phi\left(\sum_{i=1}^{d} w_i x_i + b\right). \]

The weighted sum

\[ z = \sum_{i=1}^{d} w_i x_i + b \]

is an affine function of the input. The activation function \(\phi\) introduces nonlinearity. Without nonlinear activation functions, stacking many layers would still collapse into a single affine transformation, which would severely limit the kinds of input–output relationships the network could represent.

As a concrete example, suppose \(x_1\) represents the number of suspicious login attempts from an account and \(x_2\) represents whether the login comes from an unfamiliar location. A simple neuron might compute

\[ z = 0.8x_1 + 2.0x_2 - 3.0. \]

If \(z\) is large, the model may output a high probability of suspicious activity; if \(z\) is small, the model may output a low probability. This is already a useful classifier, but it can only draw a linear decision boundary in the input space. Many real tasks require more complex boundaries, which motivates multilayer networks.

Chapter 1 begins with this simple neuron because it contains the seeds of the whole book: parameters, activations, decision boundaries, differentiability, and expressivity. The rest of the book repeatedly returns to these ideas at larger scales.

From neurons to deep networks

A layer is a collection of neurons computed in parallel. A deep neural network is a composition of many layers. If the first layer computes \(h_1 = f_1(x)\), the second computes \(h_2 = f_2(h_1)\), and so on, then the full network is

\[ f(x) = f_k(f_{k-1}(\cdots f_2(f_1(x)) \cdots)). \]

This composition is crucial. Each layer transforms the representation produced by the previous layer. The input may begin as pixels, tokens, or numerical measurements, but internal layers transform it into learned coordinate systems that are often better suited for the task.

For example, in language modeling, the raw input might be a sequence of token identifiers:

\[ [\text{The}, \text{cat}, \text{sat}, \text{on}, \text{the}] \]

The model maps these discrete tokens into continuous vectors called embeddings. An embedding is a learned numerical representation. The transformer then repeatedly updates these vectors using attention and feedforward computations until the final representation is used to predict the next token, such as “mat.” Transformer architectures made self-attention a central mechanism for sequence modeling and became the basis for many modern language and multimodal systems (Vaswani et al., 2017).

A large language model is a language model with many parameters, trained on large text corpora, usually with an objective related to predicting missing or next tokens. GPT-3, for example, demonstrated that very large autoregressive transformer language models can perform many tasks from natural-language prompts with few or no task-specific examples in the prompt (Brown et al., 2020). Such results helped motivate the broader concept of foundation models: models trained at scale on broad data and adaptable to many downstream tasks (Bommasani et al., 2021).

This book treats large language models not as magic, but as a sequence of technical ideas layered on top of one another: tokenization, embeddings, attention, optimization, scale, data, evaluation, alignment, retrieval, tools, and safety.

What “modern” means in this book

The word modern does not mean that older ideas are obsolete. Many core ideas in modern neural networks are decades old: artificial neurons, gradient-based learning, convolution, recurrence, attention-like mechanisms, probabilistic modeling, and regularization. What is modern is the combination of these ideas with large datasets, accelerator hardware, scalable optimization, improved architectures, and extensive empirical evaluation.

Modern neural networks are also distinguished by their size and generality. Earlier machine-learning systems were often built for one task at a time. A spam classifier classified spam; a speech recognizer recognized speech; an image classifier labeled images. Modern foundation models are often trained once at great cost and then adapted to many tasks. This shift raises new scientific questions: Why does scale improve performance in some regimes? How should model size, dataset size, and compute be balanced? Empirical scaling-law studies have investigated predictable relationships among model size, data, compute, and loss, while later work showed that compute-optimal training may require more training data and smaller models than some earlier scaling choices suggested (Kaplan et al., 2020; Hoffmann et al., 2022).

Modern systems also raise new engineering and social questions. How can inference be made efficient enough for real use? How should a model retrieve external knowledge instead of relying only on parameters? How can it use tools safely? How should we evaluate models when benchmarks may be contaminated, saturated, or misaligned with real-world needs? How should systems be aligned with human preferences and constraints? Reinforcement learning from human feedback became a prominent method for adapting language models to follow instructions more helpfully, though it also introduces its own modeling assumptions and failure modes (Ouyang et al., 2022).

These questions are not separate from the mathematics. They grow from it. A model’s safety properties, generalization behavior, and failure modes are shaped by its objective, data, architecture, optimization, and deployment context.

How this book is organized

The book begins with the computational neuron and expands outward.

Chapters 1 and 2 build the mathematical core: linear models, activations, multilayer networks, learned representations, and approximation. These chapters answer the question: what kind of functions can neural networks represent?

Chapters 3 through 5 study training. They explain loss functions, backpropagation, stochastic optimization, initialization, normalization, residual connections, and training dynamics. These chapters answer the question: how do large networks become trainable in practice?

Chapters 6 through 8 introduce major architectures: convolutional networks for spatial data, recurrent models for sequences, and transformers for attention-based computation. These chapters answer the question: how does architecture encode assumptions about data?

Chapters 9 through 13 move into modern generative and foundation-model training. They cover probabilistic modeling, variational autoencoders, flows, autoregressive models, diffusion models, self-supervised learning, large language models, scaling laws, instruction tuning, alignment, and preference optimization. These chapters answer the question: how do models learn from broad data and become useful interactive systems?

Chapters 14 through 16 place neural networks inside larger systems: retrieval, tools, agents, efficient training and inference, and multimodal learning. These chapters answer the question: how do neural models become practical components of real applications?

Chapters 17 through 20 examine robustness, interpretability, evaluation, safety, ethics, societal impact, and frontier research. These chapters answer the question: what do we still not understand, and how should powerful neural systems be studied responsibly?

The conclusion then synthesizes the path forward.

What to expect mathematically

This is a graduate-level book, so it uses mathematics as a working language. You will see vectors, matrices, probability distributions, derivatives, gradients, optimization objectives, and asymptotic complexity. The goal is not mathematical ornament. The goal is precision.

For example, saying “the model becomes more confident” is often too vague. We may instead say that the entropy of a predicted probability distribution decreases, or that the maximum predicted probability increases, or that the logit gap between two candidate tokens grows. These are different claims. A central habit in this book is to replace vague descriptions with measurable statements.

At the same time, the book develops notation gradually. When a symbol appears, it will be defined before it is used heavily. When an equation matters, we will ask what each part means and what assumptions make the equation valid. When an empirical claim appears, we will distinguish the observed result from the interpretation.

For instance, if a benchmark score improves after a model is scaled up, that does not automatically prove that the model has acquired a human-like cognitive ability. It may indicate better pattern recognition, broader memorized coverage, improved in-context adaptation, benchmark leakage, or a mixture of factors. Scientific validity requires careful evaluation, ablation, uncertainty estimates, and awareness of alternative explanations.

A guiding example: next-token prediction

One of the most important ideas in modern AI is that a simple training objective can produce surprisingly broad behavior when applied at scale. Consider next-token prediction. A text sequence is divided into tokens:

\[ x_1, x_2, \ldots, x_T. \]

A causal language model is trained to estimate

\[ p_\theta(x_t \mid x_1, x_2, \ldots, x_{t-1}), \]

the probability of the next token given the previous tokens. The training objective encourages the model to assign high probability to the actual next token in the training text.

At first, this may sound narrow. Predicting the next token seems less ambitious than answering questions, writing code, translating languages, or solving mathematical problems. But many tasks can be expressed as text continuation. If the prompt is

Translate to French: “The book is on the table.”

then a useful continuation is

Le livre est sur la table.

If the prompt is

Q: What is the derivative of \(x^2\)? A:

then a useful continuation is

\(2x\).

This does not mean that next-token prediction is equivalent to reasoning in a philosophical or human sense. It means that the statistical task of predicting continuations can train a model to internalize many patterns that are useful for tasks represented in language. Understanding both the power and the limitations of this fact is one of the main goals of the book.

The attitude of the book

This book aims to be technically serious without being mystical. Neural networks are not just curve-fitting machines in a dismissive sense, nor are they minds in a simple human sense. They are parameterized computational systems trained by optimization on data. Their behavior depends on architecture, objective, data distribution, scale, and deployment environment.

A good researcher or engineer must hold several truths at once.

Neural networks can generalize impressively, but they can also fail under distribution shift.

They can store and manipulate rich statistical structure, but they can also hallucinate unsupported claims.

They can learn representations that transfer across tasks, but they can also encode bias, private information, or spurious correlations.

They can be evaluated with benchmarks, but benchmarks can be incomplete, contaminated, or misleading.

They can be made more useful through instruction tuning and preference optimization, but alignment is not solved merely by making outputs more pleasant.

The purpose of this book is to give you the conceptual and mathematical tools to reason about these systems clearly. By the end, you should be able to read current research papers more critically, implement core models more confidently, diagnose training and evaluation problems more precisely, and understand why frontier neural-network research remains scientifically open.

We begin, therefore, with the smallest useful unit: the computational neuron.

References

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., … Liang, P. (2021). On the Opportunities and Risks of Foundation Models. arXiv:2108.07258.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901.

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. de Las, Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., … Sifre, L. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521, 436–444.

McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5, 115–133.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.

Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386–408.

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.