Deep Learning 1 - Single-Layer Models

Welcome - pull up a chair. A single-layer model is the smallest useful laboratory for deep learning. There is nowhere for the mathematics to hide: we can see exactly how an input becomes a prediction, how a loss turns an error into a number, and how a gradient tells every parameter which way to move.

We will begin with regression, where the target is a number. Then we will ask whether a richer feature map actually generalizes beyond the training set. Finally, we will reuse the same linear score for binary and multiclass classification. That progression matters: linear regression, logistic regression, and softmax regression look like separate algorithms only until we identify their shared structure.

Single-Layer Models

A single-layer model maps an input directly to an output without any hidden layers. Linear regression is the simplest example: it combines the input features linearly and adds a bias term.

Linear Regression

For an input vector , the prediction is

where is the weight vector and is the bias. Although the model is simple, it introduces the main ingredients that also appear in neural networks: parameters, predictions, a loss function, gradients, and an optimization rule.

Loss Function and Cost Function

For one training example with prediction and target , use the squared-error loss

The factor makes the derivative cleaner. For a dataset of examples, the average cost is

The loss measures the error on one example, while the cost aggregates the losses over the entire dataset.

Vectorized Form

Stack the input examples as rows of the design matrix

Let be the all-ones vector. The predictions for all examples can then be computed at once:

The cost function becomes

Vectorization avoids writing a separate prediction equation for every example and allows efficient matrix operations.

Partial Derivatives

For a function of multiple variables, a partial derivative measures the change with respect to one variable while keeping the others fixed. For example,

For the linear regression model

the local derivatives are

Chain Rule

The loss depends on a parameter through the prediction . The chain rule therefore gives

and

These equations show that the gradient is the prediction error multiplied by the sensitivity of the prediction to the parameter.

Gradients over the Dataset

After averaging over all examples,

and

In vector form,

Gradient Descent

Gradient descent repeatedly moves the parameters in the direction opposite to the gradient. For each weight,

and the bias is updated by

The vector update is

Here is the learning rate and is typically small, such as or . A learning rate that is too large can overshoot the minimum, while one that is too small leads to slow convergence.

For linear regression, a direct closed-form solution is possible, but it can require a matrix inversion with roughly cost. Gradient-based optimization is often more practical when the number of features is large, and it generalizes to models for which no closed-form solution exists.

Polynomial Regression and Feature Maps

A model can be nonlinear in the original input while remaining linear in its parameters. For a scalar input , define the polynomial feature map

Then polynomial regression can be written as

The feature map changes the representation of the input. Once the transformed features are constructed, the same linear regression loss, gradient calculations, and optimization procedure can be used.

The model is nonlinear as a function of the original input , but still linear in the learned parameters . That distinction is important. We can make the input representation more expressive without changing the optimization machinery.

Generalization

Training fit is not the final goal

Suppose we observe noisy samples from a smooth curve. A constant model cannot follow the trend and a degree-one line may still miss important curvature. A degree-three polynomial might capture the broad pattern. A degree-nine polynomial may pass almost exactly through every training point while oscillating violently between them.

These are two different failure modes:

• Underfitting: the hypothesis class is too restricted to explain even the training data.
• Overfitting: the model adapts to accidental details of the training sample and performs poorly on new data.

The aim is therefore not

but rather

This ability is generalization. A small training loss is evidence that optimization worked on the data we supplied; it is not, by itself, evidence that the learned relationship will survive outside that sample.

Parameters and hyperparameters

The coefficients are parameters: gradient descent or a direct solver can learn them from the training set. The polynomial degree is a hyperparameter: it chooses the family of functions in which training takes place. Other familiar hyperparameters include the learning rate, regularization strength, number of hidden units, and number of layers.

If we choose using the same training loss used to fit the coefficients, a sufficiently flexible model will usually look best. We therefore split the available data by role:

1. The training set determines model parameters.
2. The validation set compares hyperparameter choices and training decisions.
3. The test set is held back for a final, less biased estimate of performance.

The validation set is part of model development. Repeatedly adjusting a model after looking at test performance gradually turns the test set into another validation set.

Key idea: optimization asks, “Which parameters fit this training objective?” Generalization asks, “Did we choose a model and training procedure whose success transfers to data we did not fit?”

Binary Classification

From a score to a decision

In binary classification the target is discrete, usually . We call the positive class and the negative class. A linear classifier first computes a score

A hard classifier could threshold this score:

The boundary is a hyperplane. Points on one side receive class 1 and points on the other receive class 0. This makes a linear classifier geometrically simple, but the step function is discontinuous. Its derivative is zero almost everywhere and undefined at the jump, so classification accuracy based on this hard decision does not give gradient descent a useful direction.

Logistic regression

Instead of optimizing the thresholded decision, logistic regression passes the logit through the sigmoid

The model is

Because , we interpret as the model’s estimate of . The logit itself remains unbounded. Large positive logits map close to 1, large negative logits map close to 0, and maps to .

The name logistic regression is historical. Despite “regression,” its usual purpose here is classification. It is also called a log-linear model because the log-odds are linear:

Cross-entropy loss

For a Bernoulli target, binary cross-entropy is

When , only remains; when , only remains. A confident correct prediction has loss near zero. A confident wrong prediction has very large loss. Predicting when is punished much more than predicting , which matches the idea that unjustified confidence should be expensive.

Cross-entropy is not an arbitrary smooth replacement for accuracy. It is the negative log-likelihood of a Bernoulli model and a proper scoring rule: in expectation, the best reported probability is the true conditional probability.

The sigmoid and cross-entropy simplify beautifully. Since

the chain rule gives

The same “prediction minus target” term that appeared in linear regression reappears at the logit. The weight gradient is

This is one reason activation and loss are usually treated as a matched pair.

Multiclass Classification

One-of- targets and a matrix of weights

For classes, represent class as a one-hot vector

with the th entry equal to 1. Each class needs its own linear score. Arrange the class weight vectors as the rows of and the biases in :

The components of are logits. They are relative scores, not probabilities: they may be negative and need not sum to one.

Softmax

Softmax converts the logits into a probability vector:

$$
y_k=\operatorname{softmax}(\mathbf z)k
=\frac{e^{z_k}}{\sum{r=1}^{K}e^{z_r}}.
$$

Every is positive and . When one logit greatly exceeds the others, its probability approaches one, so softmax behaves like a differentiable “soft argmax.” Adding the same constant to every logit changes nothing:

For numerical stability, implementations subtract before exponentiating. This preserves the output while preventing unnecessarily large exponentials.

For two classes, softmax depends only on the logit difference and reduces to a sigmoid. If ,

Binary logistic regression is therefore the two-class special case of multiclass logistic regression.

Softmax cross-entropy

With a one-hot target, multiclass cross-entropy is

Only the correct class contributes directly because only one target entry is 1. Combining softmax and cross-entropy again yields a compact derivative:

Consequently,

The outer product has shape , exactly matching . The correct-class row is adjusted to raise its logit; the other rows are adjusted according to their predicted probabilities.

One Pattern Behind Three Models

The three single-layer models now line up:

Task	Linear score	Activation	Loss	Logit-level error
Scalar regression		Identity	Squared error
Binary classification		Sigmoid	Binary cross-entropy
Multiclass classification		Softmax	Multiclass cross-entropy

Each model first forms an affine score, then uses an output transformation appropriate to the target, then compares the prediction with the target using a compatible loss. Gradient descent propagates the resulting error into the parameters.

This shared pattern is the bridge to neural networks. A multilayer network will replace the hand-designed feature map with features learned by earlier layers. The final layer can still be the same familiar linear, sigmoid, or softmax output model.