Computational Neuroscience

In 1958 Frank Rosenblatt described the perceptron, a network of threshold units with a rule for learning. After each wrong answer, the weights of the active inputs are nudged towards the right answer. He proved that if some setting of the weights solves a classification, the rule will find it, and built a machine, wired to a camera of 400 light sensors, that learned to tell simple shapes apart. Newspapers promised machines that would soon walk, talk and think.

In 1969 Marvin Minsky and Seymour Papert published Perceptrons, which proved that a single layer of trainable units cannot compute some simple functions. Multilayer networks could, but nobody knew how to train them. Rosenblatt died in 1971, and research on neural networks waned.

Meanwhile David Marr in Cambridge and then at MIT built theories tied to real circuits, first of the cerebellum and then of vision. His lasting contribution was a way of thinking. A process in the brain must be understood at three levels: what problem it solves, what representation and procedure solve it, and how the hardware carries that out. Knowing every neuron is not enough without the first level.

A McCulloch–Pitts unit, in the simplified form usually taught today, takes inputs $x_1, x_2, \ldots$ that are each 0 or 1, multiplies each by a weight $w_i$, and fires, giving output 1, if

$$w_1 x_1 + w_2 x_2 + \cdots \geq \theta ,$$

where $\theta$ is its threshold. With two inputs of weight 1, a threshold of 2 makes an AND gate, since both inputs must be active. A threshold of 1 makes an OR gate. A single input of weight $-1$ with threshold 0 makes a NOT gate: it fires when the input is 0, because $0 \geq 0$, and is silent when it is 1, because $-1 < 0$.

$x_1$	$x_2$	AND ($\theta = 2$)	OR ($\theta = 1$)	XOR
0	0	0	0	0
0	1	0	1	1
1	0	0	1	1
1	1	1	1	0

The last column, exclusive or, fires when exactly one input is active. No single unit can compute it. The unit must be silent for input $(0,0)$, so $0 < \theta$. It must fire for $(1,0)$ and $(0,1)$, so $w_1 \geq \theta$ and $w_2 \geq \theta$. Adding these gives $w_1 + w_2 \geq 2\theta$, which is greater than $\theta$ because $\theta$ is positive. So the unit would also fire for $(1,1)$, which is wrong. A computer search over every integer weight and threshold from $-5$ to $5$ finds none that works, as the argument says it must.

Two layers solve it. Feed $x_1$ and $x_2$ to an OR unit and an AND unit, then feed those to a third unit with weights $+1$ from OR and $-2$ from AND, and threshold 1. For $(1,0)$ the third unit receives $1 - 0 = 1$ and fires. For $(1,1)$ it receives $1 - 2 = -1$ and stays silent. This is the whole of Minsky and Papert's objection and its answer in miniature: one layer cannot, two layers can, and the difficulty was finding the weights of the hidden layer by learning rather than by hand.

In 1982 John Hopfield, a physicist, gave a network of symmetrically connected units an energy that can only fall as the units update. Memories stored by a Hebbian rule become valleys in this energy, and a network started from a fragment slides into the nearest whole memory, the same mathematics as a magnet in statistical mechanics. A network of 1,000 units can hold about 138 random memories before they blur together. In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams showed that backpropagation, passing errors backwards through the layers, trains the hidden units that Minsky and Papert had found lacking. Deep learning grew from this.

Models also needed real circuits. In 1986 John White and Sydney Brenner published the complete wiring of the worm C. elegans, and in 2024 the FlyWire consortium mapped all 140,000 neurons of a fruit fly's brain. Even with the worm's wiring known for decades, its behaviour cannot yet be predicted from it, because the diagram does not say how strong each synapse is or what signals it uses. How the brain itself solves the problem that backpropagation solves for machines, deciding which synapses to change after a mistake, is still unknown.