Engineering Philosophy: Geoffrey Hinton, Conviction Over Fashion

Key Takeaways

• Conviction over fashion. Hinton bet his whole career on connectionism – learning from data in brain-like networks – through two AI winters when symbolic AI was the consensus and neural nets were dismissed as a dead end. He was right because he refused to follow the field.
• The brain is the existence proof. His argument was never a theorem. It was biological: a network of simple units obviously learns intelligence, because one is reading this sentence. Intuition and plausibility carried him where the formalism did not yet exist.
• He is honest about attribution. The 1986 backpropagation paper made the idea practical and famous, but Hinton co-authored it and credits David Rumelhart with the core idea; the underlying math predates them both. He popularized backprop – he did not solely invent it.
• The bet paid off, then frightened him. AlexNet (2012) ended the winters; a Turing Award (2018) and a Nobel Prize in Physics (2024) followed. In 2023 he left Google to warn that the thing he was right about may be dangerous.

The Principle

“I console myself with the normal excuse: if I hadn’t done it, somebody else would have.” – Geoffrey Hinton, on his life’s work, 2023¹

That line is about regret, but it reveals the principle underneath the whole career: Hinton believed the result was inevitable – that brain-like learning machines were not one option among many but the way intelligence actually gets built, and that someone was always going to prove it. For most of forty years almost no one else agreed. The field had decided that intelligence would be engineered the way a compiler is engineered: explicit symbols, hand-written rules, logic all the way down. Neural networks – loose webs of simple units that learn by adjusting connection strengths – were treated as a discredited 1960s curiosity. Funding dried up. Twice.

Hinton bet against the consensus anyway, and he did it for an unfashionable reason: he trusted biology over mathematics. The argument was never a clean theorem. It was an existence proof you are running right now. A brain is a network of slow, noisy, simple units, and it learns to see, speak, and reason without anyone writing the rules. So a sufficiently large network of artificial units, trained on enough data, should be able to do the same – not because the math guaranteed it, but because the only working example of intelligence we have is built exactly that way. That is conviction over fashion: holding a position through two winters because you believe the biology, not because the formalism or the funding agrees with you yet.

The discipline this demands is rare and uncomfortable. It means tolerating being wrong-looking for decades. It means preferring an intuition you cannot yet prove to a proof of something you suspect is beside the point. And it means that when the bet finally pays off – when the unfashionable approach turns out to be the whole game – you have earned the standing to say the hardest thing of all, which Hinton did in 2023: that you may have been too right, and that the thing you fought for might need to be feared.

Context

Geoffrey Everest Hinton was born December 6, 1947, in London.² The names are not incidental. He is the great-great-grandson of George Boole – the logician whose Boolean algebra is the foundation of every digital circuit – and of Mary Everest Boole, a mathematics educator; “Everest” is the same family that gave the mountain its name.² A man descended from the inventor of symbolic logic spent his life arguing that intelligence is not symbolic logic. The irony is exact, and he has noted it himself.

He took a Bachelor of Arts in experimental psychology from King’s College, Cambridge, in 1970 – psychology, not computer science, which matters: he came to machines through the study of how minds actually work.² He then did a PhD in artificial intelligence at the University of Edinburgh, completed in 1978, supervised by Christopher Longuet-Higgins.² This was already a quiet act of defiance. Longuet-Higgins had himself moved away from neural-network ideas toward symbolic AI, which was the rising orthodoxy; Hinton dug in on the connectionist side and stayed there.

What followed were the AI winters – stretches in the 1970s and again in the late 1980s and 1990s when neural networks were considered a failed program, funding evaporated, and serious researchers were advised to work on something else. Hinton carried the connectionist flag through all of it, moving through Carnegie Mellon and, from 1987, the University of Toronto, which became his long home.² He was not waiting out a fashion. He was betting that the fashion was wrong.

The Work

Backpropagation and Learning Representations (1986)

The single paper most associated with Hinton is “Learning representations by back-propagating errors,” published in Nature in 1986 with David Rumelhart and Ronald Williams.³ The problem it addressed is the central problem of the whole field: a network with hidden layers can in principle represent rich structure, but how do you train it – how do you decide, given an error at the output, how much each internal connection deep in the network was to blame? Backpropagation answers this by sending the error signal backward through the layers, using the chain rule of calculus to assign each weight its share of responsibility, then nudging every weight to reduce the error. The paper’s deeper claim, in its title, is that a network trained this way learns useful internal representations on its own – it discovers features nobody told it to look for.

Here honesty about attribution matters, because the popular story over-credits Hinton. He did not invent backpropagation. Reverse-mode automatic differentiation – the mathematical engine underneath it – was described by Seppo Linnainmaa in 1970, and Paul Werbos proposed using it to train neural networks in his 1974 PhD thesis.⁴ What the 1986 paper did was make the idea land: it demonstrated clearly that backprop let multi-layer networks learn internal representations, and it convinced a skeptical research community that training deep networks was actually practical.⁴ Hinton himself is scrupulous about this. In a 2018 interview he said plainly, “David Rumelhart came up with the basic idea of backpropagation, so it’s his invention.”⁴ The accurate statement is that Hinton co-developed and popularized backpropagation – he did not solely invent it. That he insists on this, when the field would happily hand him sole credit, is the same honesty that runs through this whole series, the difference between taste as a vibe you assert and a technical system you can actually defend.

Boltzmann Machines and the Energy-Based Idea

Before backprop, and alongside it, Hinton was chasing a different and stranger idea: that you could borrow the mathematics of statistical physics to build a learning machine. With Terry Sejnowski and David Ackley, around 1983-1985, he developed the Boltzmann machine.⁵ The lineage runs directly through John Hopfield, who in 1982 had shown that a network of simple units could store memories as low-energy states in a landscape. Picture a surface of hills and valleys; each stored pattern is a valley. Show the network a corrupted or partial version of a pattern, and it “rolls downhill” – flipping units to lower the total energy – until it settles into the nearest valley and recovers the complete memory. Memory as physics: recall with no address and no search.⁶

The Boltzmann machine took Hopfield’s energy landscape and made it generative and stochastic – it does not just settle into stored states, it can learn the statistical structure of a whole class of data and then generate new examples of it.⁶ This is the work the 2024 Nobel committee singled out: Hinton “used the Hopfield network as the foundation for a new network that uses a different method: the Boltzmann machine,” which “can learn to recognise characteristic elements in a given type of data.”⁷ The widget below is the Hopfield-style core of that idea – the associative-memory landscape that both men were ultimately honored for.

This is the purest expression of Hinton’s principle. Nobody could prove, in 1985, that energy-based stochastic networks were the road to machine intelligence. The justification was that it resembled how a physical system – and plausibly a brain – settles into stable states. He followed the physics and the biology, not a guarantee.

Deep Belief Nets to AlexNet: The Breakthrough (2012)

The bet stayed unpopular until two moments tipped it. The first was 2006, when Hinton and collaborators showed how to train deep belief networks layer by layer – a practical recipe for training networks deeper than anyone had managed, and the spark usually credited with reviving the term “deep learning.” The second was the one everyone remembers.

In 2012, Hinton’s graduate students Alex Krizhevsky and Ilya Sutskever built a deep convolutional neural network – AlexNet – and entered it in the ImageNet Large Scale Visual Recognition Challenge, the field’s hardest image-classification benchmark.⁸ It did not win narrowly. AlexNet posted a top-5 error rate of 15.3%, more than 10 percentage points ahead of the runner-up, in a contest where the previous year’s gains had been measured in fractions of a percent.⁸ As the ACM later put it, Hinton and his students “almost halved the error rate for object recognition and reshaped the computer vision field.”⁹ The winters ended in an afternoon. Within months, the trio formed a company, DNNresearch, and in 2013 Google acquired it – and the AlexNet code with it – bringing Hinton into Google Brain, where he stayed until 2023.² The thing he had been ridiculed for believing was, suddenly, the only thing the entire industry wanted to build.

Leaving Google and the Risk Turn – and the Honors

In May 2023, Hinton left Google. Not over a dispute, and not to join a rival – but, as he told the New York Times, so he could “freely speak out about the risks of AI” without it reflecting on his employer.¹ The man who spent half a century insisting these machines could be made intelligent had concluded they were becoming intelligent faster than expected, and that the consequences – misinformation, autonomous systems, the difficulty of stopping “bad actors” – were genuinely dangerous. It is one of the rare cases of a founder of a field turning to caution it, at the peak of being vindicated, rather than cashing the vindication in.¹⁰

The vindication is formal. In 2019, the ACM gave Hinton, Yoshua Bengio, and Yann LeCun – the “Godfathers of Deep Learning” – the 2018 A.M. Turing Award, computing’s highest honor, “for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.”⁹ Then, in 2024, Hinton shared the Nobel Prize in Physics with John Hopfield “for foundational discoveries and inventions that enable machine learning with artificial neural networks” – a physics prize, awarded for the statistical-mechanics roots of the Boltzmann machine, to a man with a psychology degree.⁷ Turing and Nobel are not usually held by the same person. Hinton holds both, for the same unfashionable bet.

The Method

The method is consistent across fifty years: hold a conviction longer than is comfortable, justify it by the brain rather than the proof, and stay honest about what is yours.

Bet on the biology, not the consensus. The field said intelligence was symbolic. Hinton said it was learned, distributed, and brain-like – and held that line through two winters when the funding and the fashion both said he was wrong. Conviction is only worth anything when it costs you something to keep it.²

Use the existence proof. When the math is not there yet, reason from the one working system you have. A brain learns without hand-written rules, so a large enough network trained on enough data should too. Plausibility before formalism; the proof can come later.⁷

Borrow from physics when logic stalls. Energy landscapes, stochastic settling, statistical mechanics – Hinton reached across disciplines for the Boltzmann machine because the tools he needed did not exist inside AI. The Nobel was, literally, physics rewarding that raid.⁶⁷

Train the people, not just the network. AlexNet was built by his students. Sutskever went on to OpenAI; the 2012 result is inseparable from the lab Hinton ran. The work propagates through people as much as papers.⁸

Claim only what is yours. He popularized backpropagation and says so – and credits Rumelhart with the idea and acknowledges the math that predates them both. The willingness to refuse credit you’d be handed is part of the craft, not separate from it.⁴

Be willing to fear your own result. The final move – leaving Google to warn about the danger of the thing you were right about – is the method turned on itself: follow the evidence even when it indicts your life’s work.¹¹⁰

Influence Chain

Who Shaped Him

George and Mary Everest Boole. Literally his ancestry, and a useful foil: the family that founded symbolic logic produced the man who argued intelligence is not symbolic. The lineage gave him both the mathematical seriousness and the thing to rebel against. (Formative influence)

Donald Hebb and the neuroscience of learning. Hinton’s whole stance – that learning is the adjustment of connection strengths between simple units – descends from Hebbian “cells that fire together wire together” thinking. He came through experimental psychology, and the brain was always the model. (Formative influence)

John Hopfield and statistical physics. Hopfield’s 1982 energy-based network was the direct foundation Hinton built the Boltzmann machine on. The two would share a Nobel for it forty-two years later. (Direct influence)

Who He Shaped

Ilya Sutskever and Alex Krizhevsky. His students built AlexNet; one of them went on to co-found OpenAI. The 2012 breakthrough that ended the AI winters came out of Hinton’s lab, in his students’ hands.

Yoshua Bengio and Yann LeCun. His co-laureates and fellow “godfathers” – a small, stubborn community that kept neural networks alive together and then accepted the Turing Award together.

Essentially every modern AI system. Backpropagation trains nearly every neural network running today; the “deep learning” he revived is the substrate under large language models, image generators, and the agent stacks I build on now.

The Throughline

Hinton is the root of the deep-learning branch of this series, and the clearest line runs forward to Andrej Karpathy. Karpathy’s whole “Software 2.0” reframe – that a neural network is a program compiled from data rather than written by hand – only makes sense in a world where Hinton’s bet already paid off; Karpathy even sat in on Hinton’s classes at Toronto as an undergraduate, absorbing the gospel before it was orthodoxy. Where Karpathy insists you build every layer from scratch to trust the stack, Hinton supplied the layer itself: backpropagation is the thing Karpathy implements by hand to understand it. The other end of the axis is John Carmack, who after a career in graphics turned to AGI research from the engineering side – the systems-and-performance approach to the same destination Hinton reached from biology and physics. Two routes to one mountain: Hinton trusted the brain; Carmack trusts the machine. (Series bridge)

What I Take From This

The lesson I keep is that being right early looks identical, for a long time, to being wrong. Hinton spent two winters holding a position the whole field had written off, and the only thing separating his conviction from mere stubbornness was that it was anchored to an existence proof – the brain works this way, so this can work. That distinction is the one I try to hold onto. Conviction without an anchor is ego; conviction anchored to evidence you can point at, held through the years when the fashion says you’re a fool, is how the actual breakthroughs happen. It is the same reason I treat quality as the only variable and not a thing to be traded against the prevailing pressure to ship – the standard doesn’t move because the room disagrees.

The harder lesson is the 2023 turn. Hinton won – Turing, Nobel, the entire industry built on his bet – and then used that standing to say the thing nobody wanted from him: that he may have been too right, and that the result might be dangerous. In a field of agents that now write code faster than anyone can read it, that is the example I find most relevant. It is not enough to follow the evidence to a result; you have to keep following it past the result, even when it indicts the work you’re proudest of. That is the evidence gate applied to your own legacy – the willingness to fear your own conclusion when the data turns. The biology was the existence proof for the upside. Hinton is now insisting we look just as honestly at the downside.

FAQ

What is Geoffrey Hinton’s engineering philosophy?

Conviction over fashion. Hinton bet his career on connectionism – intelligence as learning in brain-like networks of simple units – and held that position through two AI winters when symbolic AI was the consensus and neural nets were dismissed.² His justification was biological rather than mathematical: the brain is a network of simple units that learns without hand-written rules, so a large enough artificial network trained on enough data should be able to learn too.⁷ He trusted that existence proof over the prevailing formalism and the prevailing funding, and was eventually vindicated by AlexNet in 2012.⁸

Did Geoffrey Hinton invent backpropagation?

No – he co-developed and popularized it, which is an important distinction. The widely cited 1986 Nature paper “Learning representations by back-propagating errors,” by David Rumelhart, Hinton, and Ronald Williams, convinced the research community that training multi-layer networks with backpropagation was practical.³ But the underlying mathematics predates them: Seppo Linnainmaa described reverse-mode automatic differentiation in 1970, and Paul Werbos proposed applying it to neural networks in his 1974 thesis.⁴ Hinton himself credits Rumelhart with the core idea, saying “David Rumelhart came up with the basic idea of backpropagation, so it’s his invention.”⁴

Why did Geoffrey Hinton win both the Turing Award and the Nobel Prize?

He received the 2018 ACM A.M. Turing Award (announced 2019), shared with Yoshua Bengio and Yann LeCun, “for conceptual and engineering breakthroughs that have made deep neural networks a critical component of computing.”⁹ He then shared the 2024 Nobel Prize in Physics with John Hopfield “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”⁷ The Nobel specifically recognized the statistical-physics roots of his Boltzmann machine, built on Hopfield’s energy-based network.⁷ Holding both top honors in computing and physics for a single line of work is exceptionally rare.

Why did Geoffrey Hinton leave Google in 2023?

He left in May 2023 to speak freely about the risks of AI without it reflecting on his employer, telling the New York Times he wanted to “freely speak out about the risks of AI.”¹ After a lifetime arguing these systems could be made intelligent, he had concluded they were advancing faster than expected and posed real dangers – misinformation, autonomous systems, and bad actors. He summarized his ambivalence with the line, “I console myself with the normal excuse: if I hadn’t done it, somebody else would have.”¹¹⁰

Sources