Deep Learning 3 - Neural Language Models

Welcome back - pull up a chair. In Deep Learning 2, we built a multilayer perceptron that could learn a useful hidden representation instead of relying on features chosen entirely by hand. Today we will put that idea to work on language.

The question sounds simple: given a few words, what word is likely to come next? Answering it carefully will take us through probability, maximum likelihood, autoregressive models, n-grams, neural networks, embedding geometry, and GloVe. The common thread is representation. A word can be treated as an isolated symbol, or it can be represented by a learned vector that shares statistical information with related words. That choice changes what the model can generalize.

Why Model Language Probabilistically?

A speech-recognition motivation

Suppose an acoustic system hears a signal and must infer the sentence that produced it. A generative formulation separates the problem into two pieces:

• an observation model , which says how likely sentence is to produce signal ;
• a language prior , which says how plausible the sentence is before hearing the audio.

Bayes’ rule combines them:

The denominator normalizes the distribution but does not depend on the particular candidate once is fixed. For decoding, we can therefore compare candidates using

This explains why a language model matters even when the input is not text. Two sentences may sound similar, but a useful prior should prefer a coherent sentence over an unlikely word sequence.

Key idea: a language model assigns a probability to a sequence. That probability can act as a prior, a scoring rule, or the basis for generating the next token.

Maximum likelihood and log likelihood

Assume the training corpus contains independent sentences . Maximum likelihood chooses parameters that make the observed corpus as probable as possible:

Products of many probabilities become extremely small, so we work with the logarithm. Since the logarithm is monotone, it does not change the maximizing parameters:

Equivalently, training minimizes the negative log likelihood. Once targets are represented as one-hot vectors, that objective is exactly the cross-entropy loss used for multiclass classification.

From Sentence Probability to Next-Word Prediction

The chain rule of probability

A sentence is a sequence . The chain rule factorizes its joint probability without making any approximation:

The notation means all words before position . A model of complete sentences has therefore become a collection of next-word prediction problems.

This factorization is autoregressive: the model predicts each word from earlier words, and a generated word becomes part of the context for future predictions. During training we normally provide the true previous words; during generation the model conditions on its own earlier outputs.

A finite-context Markov assumption

Conditioning on the entire history is expensive. A classical simplification assumes that the next word depends only on the previous words:

For , the model looks at the previous three words. This is often called a finite-memory or Markov assumption. It makes the problem manageable, but it also prevents the model from using information farther back than its context window.

Do not mix up the two steps: the chain-rule factorization is exact; replacing the full history by a fixed-length context is an approximation.

N-Gram Language Models

Estimating probabilities with counts

An n-gram is a sequence of consecutive words. If the context contains two words and the target is the third, we have a trigram model:

The name counts the target as well as the context: two context words plus one predicted word form a 3-gram, not a 2-gram.

This estimator is attractive because it is transparent. Every probability comes from a frequency table. But that table grows rapidly. With vocabulary size and context length , a direct conditional table has on the order of entries. Increasing the context by one word multiplies the space by another factor of .

Sparsity is more serious than storage

Even a huge corpus contains only a tiny fraction of all grammatically possible n-grams. An unseen n-gram receives probability zero under the naive empirical estimator. Since sentence probability is a product, one zero factor makes the entire sentence probability zero.

Traditional repairs include:

• shortening the context, which reduces sparsity but discards information;
• smoothing the counts so unseen events receive some probability mass;
• interpolating or backing off across models with different context lengths.

These techniques help, but a deeper problem remains. A count table treats every phrase as a separate case. Evidence about “the cat sat” does not automatically help with “the dog sat,” even though cat and dog play similar roles.

From Localist to Distributed Representations

What does localist mean?

In a one-hot representation, each word owns one coordinate. If the vocabulary has size , word is represented by a vector containing one 1 and zeros. Different words are orthogonal:

This representation preserves identity but says nothing about similarity. “Cat” is just as far from “dog” as it is from “parliament.” A conditional probability table is localist in the same sense: information about a word or context is stored in its own isolated row or column.

A distributed representation uses many coordinates jointly. A word might participate in several latent properties - semantic topic, syntactic role, plurality, animacy, or something that has no clean human label. Information about one word is spread across dimensions, and each dimension is reused by many words.

Why sharing changes generalization

Suppose a model has observed:

The cat rested in the garden on Friday.

It should learn something useful for:

The dog slept in the yard on Monday.

An n-gram table mostly sees different symbols. A distributed model can place cat near dog, rested near slept, garden near yard, and Friday near Monday. Training on one sentence can then affect predictions in nearby contexts.

Key idea: distributed representations do not merely compress the vocabulary. They create a geometry in which statistical evidence can be shared between related words.

The Neural Language Model

The supervised learning view

With a context of words, neural language modeling is a multiclass prediction problem:

• input: ;
• target: the next word ;
• output: a probability distribution over the vocabulary;
• loss: multiclass cross-entropy.

If is the one-hot target and is the predicted softmax distribution, the negative log likelihood of a sentence is

Because is one-hot, only the log probability of the correct next word contributes at each position.

An embedding lookup is a linear layer

Let be an embedding matrix. Multiplying it by a one-hot word vector selects one column:

The columns of are tied across all positions and examples, so the same word always retrieves the same learned vector. This operation is often implemented as a lookup table, but mathematically it is a linear layer applied to a one-hot input.

For a context of words, the model retrieves embeddings and concatenates them:

An MLP transforms this context, and a final softmax produces the next-word distribution:

A classic neural language model may also include a direct or skip-layer connection from the input embeddings to the output logits. The nonlinear path learns interactions among context words; the direct path lets simpler predictive relationships bypass the hidden layer.

Compared with an n-gram table, the dependence on context length is linear rather than exponential. The model still pays for a large vocabulary in the embedding matrix and output softmax, but it no longer needs a separate parameter for every possible context.

The Geometry of Word Embeddings

Dot products, distance, and cosine similarity

Once words are vectors, we can compare them geometrically. Two common measures are the dot product

and Euclidean distance

Expanding the squared distance gives

If both vectors have unit norm, then

For normalized vectors, ranking by smallest Euclidean distance is therefore equivalent to ranking by largest dot product. The normalized dot product is cosine similarity:

High-dimensional intuition and 2-D plots

Embeddings often have dozens or hundreds of dimensions. In high dimensions, most randomly oriented vectors are nearly orthogonal, and most points are far apart. A vector can nevertheless be close to several words along different semantic directions - a useful possibility that two dimensions cannot faithfully display.

Methods such as t-SNE map embeddings into two dimensions for visualization. They are useful for revealing local clusters, but the picture is not the original geometry. A two-dimensional projection cannot preserve every pairwise distance from a high-dimensional space, so apparent neighbors or gaps should not be treated as definitive evidence.

Learning Embeddings with GloVe

The distributional hypothesis

Training a full neural language model can be expensive, especially when its softmax must score every word. If the main goal is a useful word representation, we can learn directly from global co-occurrence statistics.

The guiding principle is the distributional hypothesis: words appearing in similar contexts tend to have similar meanings - often summarized as “judge a word by the company it keeps.”

Construct a co-occurrence matrix , where counts how often word appears near word within a chosen context window. A basic low-rank model writes

with and . Row of is a -dimensional representation of word , and

Minimizing

resembles a low-rank matrix factorization such as PCA. But applying ordinary least squares directly creates two problems:

1. is enormous and sparse, so iterating over every zero is wasteful.
2. Word counts are heavy-tailed, so a few extremely common words dominate squared error.

The GloVe objective

GloVe fits log co-occurrence counts and gives each observed pair a controlled weight:

$$
J
=\sum_{i,j}f(x_{ij})
\left(
\mathbf r_i^T\widetilde{\mathbf r}j
+b_i+\widetilde b_j
-\log x{ij}
\right)^2,
$$

where

The biases absorb overall word frequencies. Taking compresses the huge dynamic range of counts. The weighting function prevents rare pairs from receiving the same confidence as frequent pairs while capping the influence of very common pairs. Since , the objective only needs nonzero entries, which is a major computational saving for a sparse matrix.

GloVe and a neural language model use different training signals. The neural model learns to predict the next word from ordered local context; GloVe factorizes aggregated, usually unordered co-occurrence statistics. Both can produce a useful embedding geometry because both force words with related contexts to share parameters.

Word Analogies and What They Mean

Some semantic relations appear approximately as directions in embedding space. For example,

To answer “Paris is to France as London is to what?”, one searches for a word vector close to

This arithmetic works when a relation is represented consistently across several word pairs. It is evidence that distributed representations can organize more than simple topical similarity.

But analogies are not logical rules. Their quality depends on the corpus, context definition, objective, dimensionality, and similarity measure. Embeddings also inherit social and historical biases from their training data. A visually appealing analogy should therefore be treated as a diagnostic of learned geometry, not as proof that the model understands the relation in a human sense.

One Conceptual Map

The models in this blog answer related questions with increasingly reusable representations:

Model	What is stored?	Main strength	Main limitation
N-gram	Counts for specific contexts	Transparent empirical probabilities	Exponential table and severe sparsity
Neural language model	Embeddings plus a predictive network	Shares evidence and models nonlinear context effects	Large output softmax; fixed context here
GloVe	Low-dimensional factors of global co-occurrence	Efficient standalone word representations	Loses word order and sentence-level context

The route from counts to vectors is the central lesson. A one-hot vector tells us which word we have. An embedding also gives the learning system a notion of which words can support one another statistically. The coordinates do not need names; what matters is that the geometry helps the model predict.

This is also where the next technical question appears. We have described the computation from embeddings to loss, but how does the error update the output layer, hidden layer, and embedding table together? The answer is backpropagation: repeated application of the chain rule through the computational graph.