Logistic regression

Level UniversityDifficulty ★★★★★Application⌖ Open in the map

What is it?

Binary classification with $p(y = 1\mid x) = \sigma(w\cdot x + b)$ , trained by maximum likelihood (cross-entropy). Convex, so gradient descent finds the global optimum — a single neuron, and the bridge to neural networks.

Formulas

\nabla_w L = \frac1N\sum_i\big(\sigma(w\cdot x_i + b) - y_i\big)\,x_i

Why does it matter?

Fraud scores, click-through prediction and medical risk models are still logistic regressions; its gradient is the simplest instance of backpropagation.

The mathematics behind it

Maximum likelihood estimation★★★★★fundamental
Logistic regression is the MLE of a Bernoulli model with $p = \sigma(w\cdot x + b)$ .

Where is it used?

Computing topics reachable from here, through the chain of ideas that leads to them:

ℒ AI and machine learning

Neural networks★★★★★
Neural networks→Backpropagation★★★★★
Neural networks→Loss landscape★★★★★
Neural networks→Backpropagation→Deep learning★★★★★
Neural networks→Loss landscape→Second-order (Hessian-based) optimization★★★★★
Neural networks→Backpropagation→Deep learning→Convolutional networks (CNNs)★★★★★
+2

What depends on it

Neural networks

This page has the essentials. A fuller treatment (intuition, formal definition, worked example) is on the way.