How Networks Learn: Loss Functions and Gradient Descent
A neural network starts out knowing nothing. Its weights are random, so its first guesses are basically nonsense. "Training" is the process that turns that random mess into something useful. In this lesson you will see the whole engine that makes learning happen: a loss function that measures exactly how wrong the network is with a single number, and gradient descent, the step-by-step method that nudges every weight in the direction that makes that number smaller. No calculus symbols — just clear pictures and real arithmetic you can check by hand.
The one big idea
Learning = adjusting the weights so predictions become less wrong. That is the entire story. Everything else — loss functions, gradients, learning rates, Adam — is just the machinery that answers two questions over and over: "How wrong am I right now?" and "Which way should I nudge each weight to be a little less wrong?" A network trains by asking those two questions millions of times and taking a tiny step each time.
- Explain what a loss function is and why a network needs one to learn.
- Compute Mean Squared Error (MSE) for a set of predictions by hand.
- Explain in plain words why cross-entropy punishes confident wrong answers so harshly, and compare two cases numerically.
- Picture the loss landscape and describe why training means finding the lowest valley.
- Explain gradient descent and read the update rule
w = w - lr × gradientterm by term. - Hand-trace a full descent and watch a weight converge to the right answer.
- Explain why the learning rate is the single most important dial, and show numerically what happens when it is too small or too large.
- Describe local minima, plateaus and saddle points — and why huge networks still train fine.
- Recognise the common variants: batch, stochastic, mini-batch, momentum and Adam.