The Perceptron and Activation Functions
Every giant neural network you have ever heard of — the one that recognises faces, the one that powers a chatbot — is built from millions of copies of one tiny idea invented in 1958: the perceptron, a single artificial neuron that can learn.
In this lesson we build a perceptron by hand, watch it learn an AND gate one number at a time, discover the famous problem that nearly killed the whole field (XOR), and then meet the activation functions — the little "decision curves" that give neural networks their real power.
- Explain what a perceptron is: inputs, weights, a bias, and an activation.
- Compute a perceptron’s output by hand and fill in a full truth table.
- Build an AND gate and turn it into an OR gate by changing one number.
- Follow the perceptron learning rule as it nudges the weights toward the right answer.
- Explain why XOR cannot be solved by one perceptron (“not linearly separable”).
- Explain why activation functions must be non-linear, and describe step, sigmoid, tanh, ReLU, leaky ReLU and softmax — and when to use each.
In 1958 an American psychologist named Frank Rosenblatt built the perceptron. It was inspired by a brain cell (a neuron), but it is really just a small piece of arithmetic. A perceptron takes in some numbers, gives each one an importance score, adds everything up, and then makes a yes/no decision. That is the whole machine.
Let us name the four parts precisely, because we will use these words for the rest of the course:
- Inputs (we write them $x_1, x_2, \dots$): the facts coming in. For a spam detector an input might be “does the email contain the word FREE?” (1 = yes, 0 = no).