Engineering Philosophy: Kyunghyun Cho

Key Takeaways

• He gave recurrent networks a learned memory gate. As first author of the 2014 paper “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation,” Kyunghyun Cho introduced the Gated Recurrent Unit (GRU) – a simplified gated cell that lets a network learn how much of its past to keep versus overwrite at each step, with fewer parameters than the LSTM it rivals.¹³
• He helped put attention into machine translation. With Dzmitry Bahdanau (first author) and Yoshua Bengio, Cho co-authored “Neural Machine Translation by Jointly Learning to Align and Translate” (2014), which let a decoder softly align to any part of the source instead of squeezing the whole sentence through one fixed-length vector. That mechanism is the direct ancestor of the Transformer and every modern large language model.²⁴
• He is a leading voice for open, reproducible science. Cho has been an active proponent of open reviewing and careful empiricism – honest baselines, skepticism toward one’s own results, and research published in the open – documented through his work on peer review and his use of open platforms like OpenReview.⁵⁶
• From Aalto to NYU to drug design. Born in South Korea in 1985, he took his doctorate at Aalto University, did a postdoc with Yoshua Bengio at Université de Montréal, joined NYU in 2015, and co-founded Prescient Design – now part of Genentech – applying machine learning to antibody design.¹⁷

The Principle

“One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols.” – Kyunghyun Cho et al., describing the RNN Encoder-Decoder that his later work would learn to outgrow¹

The oldest instinct in engineering is to design the control structure yourself. You decide which inputs matter, how long the system should remember a value, where to look in a buffer. You encode those decisions in the architecture, and the system inherits your guesses. Cho’s body of work runs the other direction. His defining contributions – the GRU’s gates and attention’s soft alignment – share one move: stop hard-coding the control structure, and let the model learn it. Which past state to keep, which input to attend to – those become parameters the network tunes from data rather than rules the designer fixes by hand.¹²

The GRU is the cleanest case. A plain recurrent network carries a hidden state forward and, at every step, blends the old state with the new input in a fixed way. That fixed blend is a designer’s guess, and it is usually wrong: important signals from early in a sequence get diluted into noise within a few steps. Cho’s gated unit replaces the guess with a learned gate – a small sigmoid-valued control that decides, per step and per dimension, how much of the old memory to retain versus how much to overwrite with the new input.³ The network learns when to hold on and when to let go. Nobody hand-codes the memory policy; the data writes it.

Attention is the same idea applied to looking instead of remembering. And there is a second half to Cho’s principle that keeps the first half honest – a scientist’s discipline. The way you find out whether a learned mechanism actually helps is not enthusiasm; it is a fair fight against a strong baseline, results you are skeptical of until they hold up, and methods published openly so others can reproduce them.⁵⁶ Let the model learn its own control structure – then verify, honestly and in the open, that it really worked. The first half is where the power comes from. The second half is what makes the power trustworthy.

Context

Kyunghyun Cho was born in South Korea in 1985.¹ He took his bachelor’s in computer science at KAIST in 2009, then moved to Finland for graduate study at Aalto University, earning a master’s in machine learning and data mining in 2011 and his Doctor of Science in 2014, supervised by Professor Juha Karhunen (with Tapani Raiko and Alexander Ilin).¹⁷ Finland is an unlikely cradle for one of deep learning’s pivotal years, but the timing is the point: he finished his doctorate exactly as the field was about to turn.

From Aalto he went to Montreal for a postdoctoral fellowship with Yoshua Bengio at the Université de Montréal – the lab that would become MILA, then the densest concentration of deep-learning talent in the world.¹ The two papers that anchor this essay – the GRU and the attention-based NMT model – both came out of that 2014 window, with Bengio as a co-author on each.¹² It is worth pausing on how fast that was: a researcher arrives as a postdoc and, within roughly a year, is first author on the paper that introduces the GRU and a co-author on the paper that introduces attention to translation. Both ideas now sit underneath a large fraction of working AI.

In 2015 he joined New York University, where he is now a professor of computer science and data science at the Courant Institute and the Center for Data Science, the Glen de Vries Professor of Health Statistics, and co-director of NYU’s Global AI Frontier Lab alongside Yann LeCun.⁷ He spent 2017-2020 as a research scientist at Facebook AI Research, and – the chapter that shows his principle reaching past language – in early 2021 he co-founded Prescient Design, a lab-in-the-loop antibody-design startup acquired by Genentech later that year, where he led frontier research applying generative machine learning to drug design.⁷⁸ Throughout, he has been a vocal advocate for open and reproducible science.⁵⁶

The Work

The GRU: a memory the network learns to gate

To feel why gating matters, watch a recurrent network try to carry one important value across a long, noisy sequence. The network reads one step at a time, holding a single hidden “memory” that it updates at every step. Early on, a value worth keeping appears; everything after it is noise. A plain RNN blends old and new in a fixed proportion, so the early value bleeds away within a few steps – by the end, it is gone. The whole problem is that the keep-versus-overwrite decision was hard-coded. The widget below makes that tangible: a single gate decides, at each step, how much of the old memory survives. Slide it toward “keep” and the early value reaches the end intact; slide it toward “overwrite” and it washes out almost immediately.

That is exactly the lever Cho’s GRU adds, except the network learns the gate instead of you setting it. Introduced in his 2014 RNN Encoder-Decoder paper, the GRU uses two learned gates: an update gate that controls how much of the previous hidden state to carry forward versus replace with new information, and a reset gate that decides how much of the past should influence the new candidate state.³ Both are sigmoid-valued – smooth knobs between 0 and 1 – which means the whole thing is differentiable and trainable by gradient descent. The network discovers, from data, when to hold a value for a long time and when to flush it.³

The design is consciously economical. It arrived alongside Hochreiter and Schmidhuber’s older LSTM, which solves the same vanishing-gradient problem with more machinery – a separate cell state and an extra output gate. The GRU “lacks a context vector or output gate, resulting in fewer parameters than LSTM,” yet performs comparably across speech recognition, music modeling, and language tasks.³ That is the engineering taste worth naming: the simpler unit that captures the essential idea – learned gating – without the parts you cannot prove you need. For years the GRU and LSTM were the two default recurrent cells in every sequence-modeling toolkit.

Attention: soft alignment, and the end of the fixed-vector bottleneck

The RNN Encoder-Decoder had a structural flaw, and Cho’s own paper named it: the encoder crushes an entire input sentence into “a fixed-length vector representation,” and the decoder must reconstruct the whole translation from that single vector.¹ For a short sentence that is fine. For a long one it is a catastrophe – you are forcing a paragraph through a keyhole, and recurrent networks tend to over-weight the words nearest the end while the beginning fades.⁴ The bottleneck is the fixed-length vector itself.

The fix, in “Neural Machine Translation by Jointly Learning to Align and Translate” (2014/2015) – Dzmitry Bahdanau as first author, with Cho and Bengio – was to remove the bottleneck entirely.² Instead of compressing the source into one vector, the model keeps a representation of every source word and lets the decoder, at each output step, compute a set of weights over all of them – a learned soft alignment that says “to produce this word, look mostly here, a little there.”⁴ The decoder gets direct access to any part of the input rather than reaching it only through a single squeezed state.⁴ Crucially, the alignment is not hand-specified by a linguist; it is learned jointly with translation, end to end. The model decides for itself where to look – the same principle as the GRU’s gate, now applied to attention instead of memory.

The line from there to today is short and load-bearing. Soft alignment generalized into self-attention, and in 2017 the Transformer’s “Attention Is All You Need” formalized scaled dot-product attention and threw the recurrence away entirely – a design that became the foundation for BERT, T5, GPT, and the whole generation of large language models.⁴ Modern LLMs are, in a real sense, attention scaled up and stripped of the RNN around it. The mechanism Cho helped introduce to translation in 2014 is the one your chatbot runs on. (The attribution matters: Cho is first author on the GRU paper and a co-author with Bahdanau and Bengio on the attention paper – he did not introduce attention alone.)¹²

Open and reproducible science

The second principle is quieter and, in a hype-prone field, harder. Cho has been a consistent advocate for open, reproducible science – research conducted and reviewed in the open, with honest baselines and a working skepticism toward one’s own results.⁵⁶ He has used and championed open platforms like OpenReview, and has contributed research on peer review itself, studying questions like whether authors’ own assessments of their papers can inform the review process.⁵⁶ The throughline is that a result is not a result until it survives scrutiny that the author did not control.

Why it matters technically: deep learning is unusually easy to fool yourself in. A new architecture will almost always beat a weak baseline, a lucky seed, or an under-tuned competitor – and the literature is littered with “improvements” that evaporate when someone gives the old method a fair fight. The discipline Cho models is to assume your own gain is an artifact until proven otherwise, to tune the baseline as hard as you tune your method, and to publish openly enough that someone can prove you wrong. It is the empirical conscience that keeps the “let the model learn it” instinct from curdling into “the model learned it because I wanted it to.”

Machine learning for the sciences: Prescient Design

The clearest sign that Cho’s principle is general, not language-specific, is where he took it next: drug design. In early 2021 he co-founded Prescient Design, acquired by Genentech that year, to build a “lab-in-the-loop” platform for designing therapeutic antibodies.⁷⁸ The loop couples generative ML models, multi-task property predictors, and active-learning selection with actual wet-lab experiments in an iterative cycle – the models propose antibody variants, the lab tests them, and the results retrain the models.⁸ Applied across clinically relevant targets, the system designed and tested over a thousand variants and engineered antibodies with dramatically stronger binding than the starting leads.⁸ The structure is the same one that runs through everything else he has built: don’t hand-design the answer, build a system that learns it from evidence – and close the loop with real measurements so the learning stays honest.

The Method

Read across the GRU, attention, the open-science advocacy, and the antibody work, and the same commitments recur. Cho’s method is less a slogan than a set of standing habits.

Make the control structure learnable. The defining move is to take a decision you would normally hard-code – how long to remember, where to look – and turn it into parameters the model tunes from data. The GRU’s gate and attention’s alignment are the same instinct in two domains. The lesson transfers far past sequence models: when you find yourself guessing a threshold, a weighting, or a routing rule, ask whether the system could learn it better than you can guess it.¹²

Prefer the simpler mechanism that captures the essential idea. The GRU keeps learned gating and drops the LSTM’s cell state and output gate, matching its performance with fewer parameters.³ That is minimum worthy product at the level of a neural cell – the smallest unit that still carries the load-bearing idea, the same economy of means that runs through Sophie Wilson’s ARM.

Name the bottleneck before you remove it. Cho’s own encoder-decoder paper named the fixed-length-vector limitation; the attention paper removed it.¹⁴ The discipline is to state the structural flaw precisely first – the paragraph through a keyhole – because a sharply named bottleneck is half-solved. It is the same instinct as Fei-Fei Li’s decision to attack the data, not the model: find the real constraint, not the convenient one.

Give the baseline a fair fight. Honest empiricism means tuning the method you are trying to beat as hard as the one you are proposing, and treating your own gain as suspect until it survives that test. This is the evidence gate made into a research practice – “it improved” is not evidence; “it improved against a strong, openly reproducible baseline” is.⁵⁶

Work in the open. Open review, open platforms, reproducible results. The value of a method is not what its author claims but what survives independent scrutiny – which is why publishing openly enough to be checked is part of the engineering, not a courtesy after it. It is quality is the only variable applied to science itself: the only thing that counts is whether the result is real.⁵⁶

Influence Chain

Who Shaped Him

Yoshua Bengio and the MILA group. Cho’s pivotal year happened as a postdoc in Bengio’s Montreal lab, and Bengio co-authored both the GRU and the attention papers.¹² MILA was the environment where the recurrent-network and machine-translation ideas of the early 2010s were being pushed hardest, and Cho’s work is inseparable from it. (Direct influence)

Juha Karhunen and the Aalto school. His doctoral training at Aalto under Juha Karhunen, with Tapani Raiko and Alexander Ilin, grounded him in the unsupervised-learning and neural-network tradition before the Montreal years gave it a translation target.¹⁷ (Formative influence)

The deep-learning lineage. The vanishing-gradient problem the GRU addresses, and the recurrent and convolutional architectures it sits among, descend from Geoffrey Hinton’s and Yann LeCun’s decades of work establishing that deep networks can be trained at all. Cho now co-directs an NYU lab with LeCun.⁷ (Formative influence)

Who He Shaped

Modern sequence modeling. For years the GRU was one of the two default recurrent cells – the simpler half of the GRU/LSTM choice that every NLP, speech, and time-series practitioner made by reflex.³

The Transformer and the LLM era. Attention, introduced to translation in the paper Cho co-authored, generalized into the self-attention that the Transformer is built from – and the Transformer is what every modern large language model runs on.⁴ The mechanism is one of the most consequential in the field’s history.

Machine learning for the sciences. Through Prescient Design and Genentech, Cho helped push generative ML into therapeutic antibody design, an argument that the “learn it from evidence, close the loop with measurement” pattern belongs in biology as much as in language.⁸

The Throughline

Cho is the mechanism hinge of this series’ deep-learning branch – the bridge from foundations to the LLM era. Geoffrey Hinton and Yann LeCun established that deep networks can learn; Fei-Fei Li supplied the data they learned on. Cho’s gating and attention are the architectural step that turned those foundations into something that could handle language at scale – and his attention work flows directly into the Transformer that Andrej Karpathy would later teach a generation to build and that powers the modern LLM. Where Hinton says the learning machine works and Li says here is the world to learn from, Cho says: let the model learn its own control structure – what to remember, what to look at – and then prove, in the open, that it actually did. The first clause is the line to the Transformer; the second is the empirical conscience that keeps it honest – the same rigor Fei-Fei Li brought to benchmarks. (Series bridge)

What I Take From This

The lesson I keep from Cho is to stop hard-coding the decisions a system could learn. My instinct, like most engineers’, is to encode my own judgment – this threshold, that weighting, this routing rule – because my judgment feels like the trustworthy part. The GRU and attention are both arguments that the opposite often ages better: the model, given the right mechanism and enough data, will learn a memory policy or an alignment far better than I would guess one. So when I catch myself tuning a magic constant or hand-writing a branch that decides what matters, I now ask whether I am encoding a decision the system should be learning instead. The skill is not making the choice; it is building the structure that lets the choice be learned.

The second lesson is the discipline that keeps the first one from lying to me. Letting a system learn its own behavior is intoxicating precisely because it so easily looks like it worked – the demo passes, the metric ticks up, and I want to believe. Cho’s open-science ethic is the antidote: assume your own result is an artifact until it survives a fair fight against a strong baseline, and publish it openly enough that someone can prove you wrong. That is exactly why I treat the evidence gate as non-negotiable – “it improved” is a feeling, “it improved against a tuned baseline, reproducibly” is evidence. Build systems that learn their own control structure, then hold them to a standard of proof you did not get to set. The power and the honesty are two halves of one practice.

FAQ

What is the GRU (Gated Recurrent Unit)?

The GRU is a type of recurrent neural network cell that Kyunghyun Cho introduced as first author in the 2014 paper “Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.”¹ It uses learned gates – an update gate and a reset gate, both sigmoid-valued between 0 and 1 – to control how much of the previous hidden state to keep versus overwrite with new input at each step. This lets the network learn what to remember over long sequences, solving the vanishing-gradient problem. It performs comparably to the LSTM while using fewer parameters, because it omits the LSTM’s separate cell state and output gate.³

What did Kyunghyun Cho contribute to attention and neural machine translation?

Cho is a co-author – with Dzmitry Bahdanau (first author) and Yoshua Bengio – of “Neural Machine Translation by Jointly Learning to Align and Translate” (2014), the paper that introduced the attention mechanism to neural machine translation.² Instead of compressing a source sentence into one fixed-length vector, the model keeps a representation of every source word and lets the decoder learn a soft alignment over all of them at each output step. That mechanism generalized into the self-attention of the Transformer in 2017, which is the foundation of modern large language models.⁴ Separately, Cho was first author on the 2014 paper that introduced the GRU.¹

What encoder-decoder bottleneck did attention solve?

The original RNN Encoder-Decoder, described in Cho’s 2014 paper, encodes an entire input sentence into “a fixed-length vector representation,” from which the decoder must reconstruct the whole output.¹ For long sentences this single vector is a bottleneck – a paragraph forced through a keyhole – and recurrent networks tend to lose information from the start of the sequence.⁴ Attention removed the bottleneck by keeping a representation of every source position and letting the decoder attend directly to any of them, with learned weights, rather than relying on one compressed state.⁴

What is Kyunghyun Cho doing now?

Cho is a professor of computer science and data science at NYU’s Courant Institute and Center for Data Science, the Glen de Vries Professor of Health Statistics, and co-director of NYU’s Global AI Frontier Lab with Yann LeCun.⁷ He co-founded Prescient Design in 2021 – acquired by Genentech that year – where he led frontier research applying generative machine learning to “lab-in-the-loop” therapeutic antibody design, coupling ML models with wet-lab experiments in an iterative optimization loop.⁷⁸ He remains an active advocate for open and reproducible science.⁵⁶

Sources