How AI Learned to Read a Whole Sentence at Once
In 2017, a Google team dropped word-by-word reading from machine translation. Their attention-only design beat the best systems and trained in days.
In one paragraph
Translation AI used to read a sentence one word at a time, which made training slow and hard to split across machines. This paper introduced the Transformer, a design built only on attention, in which every word is compared with every other word at once. On English-to-German it scored 28.4 BLEU, more than 2 points above the best earlier results, ensembles included, and on English-to-French it set a single-model record of 41.8 after 3.5 days on eight GPUs. The evidence comes from standard benchmarks, tested on text only.
What did the Attention Is All You Need paper propose?
Before this paper, the best translation systems were built on recurrent networks, which process a sentence one word after another and carry a running summary along. That order makes them slow to train, because each step must wait for the one before, and it makes links between distant words harder to learn.
The authors, a team at Google, replaced that design with the Transformer. It keeps the usual encoder and decoder, a part that reads the input and a part that writes the output, but builds both only from attention and simple feed-forward layers. In self-attention, each word is compared with every other word in the sentence to decide which ones matter for its meaning. Several of these comparisons, called heads, run side by side: 8 heads in each of 6 stacked layers. Word order is added as a position signal, since nothing is read in sequence.
They trained the models on standard translation data, about 4.5M English-German sentence pairs and 36M English-French sentences, on one machine with 8 P100 GPUs.
How much better was the Transformer at translation?
On the WMT 2014 English-to-German test, the big Transformer scored 28.4 BLEU, an automatic measure of how closely a machine translation matches human translations. That was more than 2 BLEU above the best previous results, including ensembles, where several models vote together.
On English-to-French, it reached 41.8, a new best for a single model, after training for 3.5 days on eight GPUs, a small fraction of the training cost of the best earlier models. A smaller base version trained in about 12 hours and still beat every previously published model on English-to-German.
The design choices mattered. With a single attention head, quality was 0.9 BLEU worse than the best setting, and it also dropped with too many heads. On a different task, English grammar parsing, a 4-layer Transformer reached 91.3 F1 using only the standard training set and 92.7 with extra data, ahead of almost all earlier systems, without task-specific tuning.
Why does the Transformer paper matter?
The main practical gain is speed. Because no step has to wait for the previous word, the work on a sentence can be spread across many chips at the same time. That is why the authors could train a better translator in days on a single machine.
The paper also shows the idea is not tied to one job: the same design did well at grammar parsing with almost no changes. Its attention patterns can be inspected, and the authors report heads that seem to follow sentence structure, such as linking a pronoun to the word it refers to.
A fair reading is that faster training made it practical to build ever larger models of this kind. The design later became the base of today's large language AIs, but that came after this paper and is not something the paper itself shows.
What are the limits of the Transformer paper?
Read before you quote it
The authors are open about several limits:
- Long inputs. Comparing every word with every other word grows quickly with length. For very long sequences they only suggest restricting attention to nearby words, as future work.
- Text only. The tests cover two translation tasks and one parsing task. Images, audio and video are named as future work.
- Automatic scores. BLEU and F1 are automatic measures on fixed test sets, and the settings were chosen on development data by the same team.
- Hand-picked examples. The pictures of attention heads are a few examples, so the claim that heads learn sentence structure is suggestive.
The paper does not show how the design copes with very long documents, or whether it works beyond text, such as on images or sound.
- It does not show how the design copes with very long documents.
- It does not show results beyond text, such as images or sound.
- The cost of self-attention grows with the square of the sequence length; for very long inputs the authors only propose restricting attention to a neighbourhood, as future work.
- Results cover two translation tasks and one parsing task; images, audio and video are named as future work.
- BLEU and F1 are automatic scores on fixed test sets, and the settings were chosen on development data by the same team.
- The attention pictures are a few hand-picked examples, so the claim that heads learn sentence structure is suggestive.
Takeaways
- saysReading the whole sentence at once beat word-by-word reading by more than two points.
- saysBecause it works in parallel, it cost far less to train.
- saysIt was only tested on sentences of text, so long documents, images and sound came later.
- explainsSelf-attention means each word in a sentence is compared with every other word to decide which ones matter for its meaning.
- interpretsA fair reading is that the speed of training mattered as much as the higher score, because it made much larger models practical.
Ask the paper
In PaperSays you can ask this paper anything, by text or by saying “Hey Aiden”. Three questions listeners asked:
What is the Transformer?
A translation model built only on attention, with no word-by-word reading, so every word is compared with every other word at once. Hear the full answer in the app →
Why did the Transformer train faster?
Without word-by-word reading, the work on a sentence can be spread across many chips at the same time. Ask in the app →
Was the Transformer tested beyond text?
No. It was tested on translation and grammar parsing; images, audio and video were left for future work. Ask in the app →
Briefing reviewed by an editor before publication. The paper belongs to its authors; this page summarises it and links to the original. Reviewed by Editor. Paper licence: arXiv non-exclusive distribution licence 1.0. Corrections: support@papersays.com.