Equalang
Equalang Notes

70 Years of Machine Translation: From Symbols to Meaning

Equalang

In brief

From dictionaries and word counts to word vectors, Transformers, and LLMs: how machine translation evolved over 70 years to understand human language.

Keywords: machine translation; Transformer; large language models; neural networks; attention mechanism; simultaneous interpretation

One Second to Reshuffle a Sentence, 70+ Years in the Making

Type a Chinese sentence into a translator: 「我昨天在北京见了他。」

In under a second, the English translation pops up on screen: "I saw him in Beijing yesterday."

Look closely at the two sentences and you'll notice something we take for granted that is anything but simple: the machine doesn't just swap each Chinese word for an English one in place. The words near the start of the Chinese sentence—「昨天」 ("yesterday") and 「在北京」 ("in Beijing")—move all the way to the end in English, while the action at the end—「见了他」 ("saw him")—moves up to the front. If you draw lines connecting matching words between the two sentences, they cross in mid-air like a web.

Chinese words 「我 昨天 在 北京 见了 他」 (labeled "Chinese") above English "I saw him in Beijing yesterday", with lines between matching words crossing
When translating between Chinese and English, words aren't just swapped in place—the word order is reshuffled

Behind that single second after you hit Enter lies a puzzle humanity has worked on for more than 70 years: how do you get a machine that only recognizes numbers to turn one language into another?

Looking back over those 70-plus years, machine translation wasn't simply a matter of building ever-bigger dictionaries. Instead, it climbed a step-by-step ladder toward understanding language—from rigidly swapping symbols, to grasping word meanings, to understanding a whole sentence, and finally to reading the context of an entire passage.

This article also has a 9-minute animated video version: https://www.youtube.com/watch?v=0XmNvD5ng3M

Step 1: Why Treating Foreign Languages as Ciphers Hit a Wall

The story begins in 1947. On March 4 of that year, the American mathematician Warren Weaver wrote a letter to his friend Norbert Wiener. In it, he shared a hunch: whenever he looked at an article in Russian, a thought crossed his mind—"This is really written in English, but it has been coded in some strange symbols."

To Weaver, translation was at heart a code-breaking puzzle. Find the rules mapping symbols between two languages, crack the cipher, and the English underneath would naturally emerge. In July 1949, he put this idea into a memorandum for his colleagues under a one-word title: Translation.

The idea was soon put onto a real computer. On January 7, 1954, IBM and Georgetown University teamed up for the first public demonstration of machine translation in history. An operator who knew no Russian sat at the machine and typed more than 60 Russian sentences onto punch cards; clicking away, the computer automatically translated all of them into English.

How did that machine translate? It worked much like looking up words in a dictionary that came with an instruction manual:

  • Number lookup: First, it looked up each incoming Russian word in a punch-card dictionary to find its identification number and its possible English translations.
  • Rule check: Next, it checked the nearby words and followed human-written grammar rules to do three things—pick the right English word from the candidates, reorder inverted words, or drop extra words not needed in English.
Three boxes side by side showing the 1954 Georgetown–IBM translation rules: "Word choice" (Russian word to English A / English B, "Decide from nearby words"), "Reorder" (B A, "Swap positions"), and "Delete" ("of" struck out, "Drop extras"), tagged "Illustrative"
Three types of rules in the 1954 Georgetown–IBM demo: word choice, reorder, and delete (illustrative)

The demonstration caused a sensation. Researchers at the time were wildly optimistic, convinced that if they kept going down this road, machine translation would be completely solved within three to five years.

What the audience didn't know was that the machine's dictionary and grammar rules were tiny—just 250 entries in total (including stems and endings) and a mere 6 grammar rules. Even the 60-plus Russian sentences used on stage had been carefully handpicked in advance.

Once machine translation stepped out of the lab to face real-world text, three years passed, then five, then ten, and the expected breakthrough never arrived. In 1966, the Automatic Language Processing Advisory Committee (ALPAC), organized by the US government, delivered a sobering report: a decade of research had failed to meet expectations. Funding was slashed, plunging machine translation into a long winter.

Despite the funding drought, determined researchers pressed on, refining the 1954 approach in ever greater detail. Dictionaries were expanded word by word, and grammar rules were written one by one, all taught to the machine by hand—an approach known as rule-based machine translation.

In specialized settings, this method saw real use. Starting in 1981, for instance, Canada's METEO system translated weather forecasts between English and French. It worked because weather forecasts use a restricted language, repeating the same few fixed patterns day after day with a narrow vocabulary and predictable sentence structures.

In everyday speech, however, rules fell apart. Daily language is far messier, and the biggest headache of all is polysemy—one word having many meanings.

Take the common Chinese character 「打」 (dǎ). If a dictionary rigidly assigns one meaning to each word—say, tying 「打」 to the English word hit—then translating 「我打车去公司」 ("I take a taxi to the office") word for word turns into a joke (illustrative). In real language, that same character 「打」 needs a completely different English word depending on the phrase it appears in:

  • 打电话 (dǎ diànhuà, make a phone call) is call;
  • 打车 (dǎchē, hail a cab) is take a taxi;
  • 打折 (dǎzhé, offer a discount) is discount;
  • 打伞 (dǎsǎn, hold up an umbrella) is hold an umbrella.

That single character has so many uses that translating it properly required humans to write a separate decision rule for every combination. Each new phrasing needed a new rule—and neither the rules nor the exceptions in language ever ended. Teaching a machine grammar one rule at a time was like building a tower out of playing cards: as the exception cards piled higher and higher, sooner or later they knocked the whole structure down.

Skipping Grammar for Recipes and Word Counts: How Statistical Translation Got Smart

If human-written rules could never cover everything, was there another way?

In 1984, the Japanese researcher Makoto Nagao proposed example-based machine translation: instead of making the machine memorize grammar rules, let it translate by following existing examples.

It's like adapting a recipe in the kitchen. Suppose you already have an old recipe—say, you know that 「我昨天在北京见了他」 translates to "I saw him in Beijing yesterday." Now a new dish comes along: 「我昨天在上海见了她」 ("I saw her in Shanghai yesterday"). The machine simply finds the closest old sentence in its database of examples, spots the only two differences (「北京」 Beijing → 「上海」 Shanghai, and 「他」 him → 「她」 her), copies the old translation, and swaps those two ingredients to produce the new translation (illustrative).

Around 1990, researchers at IBM's Thomas J. Watson Research Center went even further: they skipped grammar and rules altogether, fed the machine massive amounts of human-translated bilingual sentence pairs, and let it count which words kept appearing together.

Imagine stacks of paired Chinese and English sentences dropping into the machine. The machine doesn't know Chinese characters or English words, but it can do arithmetic. Every time 「苹果」 appears in a Chinese sentence and "apple" appears in its English counterpart, the machine adds one to the counter linking that pair of words.

「苹果」 and "apple" linked by a line, with "In the same sentence pair: 8,743 times" below and an "Illustrative" tag
「苹果」 linked to "apple", appearing in the same sentence pair 8,743 times (illustrative)

Across tens of thousands of sentence pairs, the more often 「苹果」 and "apple" appear together, the higher the probability the machine calculates, and the surer it becomes that the two symbols mean the same thing. It's like seeing the foreign label apple next to a red fruit at a fruit stand again and again without knowing the language—see it enough times, and you naturally get what it means.

This is statistical machine translation. From that point on, translation shifted from looking up rules to calculating probabilities.

Counting bilingual word pairs alone, however, isn't enough to assemble a full sentence. For example, the Chinese word 「他」 (tā) can correspond to either the subject pronoun he or the object pronoun him in English. Word-pair probabilities for 「他」 alone can't tell the machine which one to pick.

So researchers gave the machine a second tool: counting which words usually follow which in pure English sentences. Suppose the machine has already translated "I saw ＿". Checking its ledger of English word counts, it finds that saw is followed by him far more often than by he, giving him a higher score (illustrative). A model that looks at the past few words to guess the next word is called a language model.

Put those two pieces together, and the statistical machine translation pipeline is complete:

  • Translation model (word-pair lookup): Split the original sentence into small phrases and look them up in probability tables to find candidate English words (for instance, pulling up both he and him as candidates for 「他」);
  • Language model (picking the smoothest match): Check how nearby words usually fit together in English and pick the smoothest option from the candidates (keeping him and crossing out he after saw) to assemble the final sentence (illustrative).

When Google Translate launched in 2006, it was powered by this statistical method.

Yet statistical machine translation, like rule-based translation before it, was still trapped at the level of symbols. To the system, every word was just an isolated ID number. For example, cat（「猫」）might be ID #4127 and dog（「狗」）ID #9853 (numbers illustrative), with no connection between the two. The machine had no way to see that cat and dog are close in meaning, and nothing it learned about translating cat could be applied to dog. Whenever it ran into a phrase combination that had rarely appeared in its training text, its counts came up short and it was stumped.

To take the next step, machines had to stop treating words as cold, isolated ID numbers—they needed to grasp word meaning.

From Word Vectors to Attention: How Machines Learned to Read a Whole Sentence

A turning point came in 2013, when word2vec popularized word vectors—an idea that Yoshua Bengio and colleagues had proposed as early as 2000.

Word vectors did something clever: instead of handing each word an isolated ID tag, they turned each word into a long strip of numbers—typically several hundred numbers long.

You can think of those hundreds of numbers as a detailed "trait sheet" for every word: the first box scores whether it's an animal, the second whether it's edible, the third whether it's a city, with a score for each trait (note: in practice, individual dimensions don't have neat human-readable meanings; they are abstract features summarized by the AI itself).

Where do these numbers come from? Humans don't fill them in; instead, the machine takes massive fill-in-the-blank tests. Give it a sentence like "I have a pet ＿", hide the last word, and have it guess what goes in the blank (illustrative). Across thousands upon thousands of sentences, cat and dog keep fitting into similar blanks. As the machine repeatedly tunes its numbers to guess the missing words, the hundreds of numbers for cat and dog get tuned closer and closer together.

"cat", "dog", and "Beijing" each shown as a long strip of colored number cells: cat and dog are marked "Close 0.89", Beijing is marked "Apart 0.00", with a note "Similarity: 1 = identical, 0 = unrelated" and an "Illustrative" tag
"cat", "dog", and "Beijing" each become a long strip of numbers—closer meanings get closer numbers (illustrative)

The result is remarkable: the closer two words are in meaning, the more alike their number strips look; unrelated words like cat and Beijing end up with numbers that stay far apart. Because similar words have similar numbers, what the machine learns on cat finally carries over to dog. Even more importantly, once words are turned into continuous numbers, the machine can use those numbers to compute the meaning of a whole sentence.

In 2014, machine translation based on artificial neural networks (neural machine translation, or NMT) arrived. Instead of chopping sentences into pieces to look up in tables, it worked like a human interpreter:

  1. Read the whole sentence first (encoder): Read the entire Chinese sentence from start to finish and compress its meaning into a numerical "memory";
  2. Speak from memory (decoder): Take that memory and speak the English translation word by word, folding each spoken word back into the memory and carrying it forward to decide what to say next.

At the time, the structure responsible for building this memory and speaking the translation was a recurrent neural network (such as a Long Short-Term Memory network, or LSTM). It read a sentence like passengers filing through a security checkpoint one by one: as each word came in, it was folded into the memory from the previous step and passed along to the next.

Before long, though, this approach hit a bottleneck: the memory had a fixed size. Whether a sentence had five or six words or several dozen, all of it had to be squeezed into the same fixed-size memory box. In long sentences—just like trying to memorize a long string of phone numbers in one go—by the time the network reached the end, the words from the beginning had faded out and were mostly forgotten.

How do you keep a machine from forgetting the start of a long sentence? That same year, researchers introduced the attention mechanism.

Attention works much like a student doing a reading comprehension test: as the machine writes each English word of the translation, instead of relying solely on that fixed-size memory box, it glances back at the original sentence, focusing on the words that match best. When writing "I saw him in Beijing yesterday," for instance, it looks primarily at 「见了」 when writing saw, and glances all the way back to 「昨天」 near the start of the Chinese sentence when writing yesterday. Long-distance reordering of words—especially verbs—became one of neural machine translation's greatest strengths.

Chinese words 「我 昨天 在 北京 见了 他」 on top and English "I saw him in Beijing yesterday" below, with glowing lines from each English word to the Chinese words and the yesterday–昨天 line glowing brightest
Each English word in "I saw him in Beijing yesterday" reaches back to the Chinese sentence with rays of light—the more relevant, the brighter (attention mechanism, illustrative)

In 2015, Baidu launched the first large-scale neural machine translation system built with a recurrent neural network and attention; in 2016, Google Translate switched over to the same architecture. According to Google's 2016 report, neural machine translation cut translation errors by about 60% on average compared with the phrase-based statistical system it had used for a decade.

Still, recurrent neural networks had a built-in flaw they couldn't shake: words had to be read one by one in line. Because each step's memory had to wait for the previous step to finish computing before it could be passed along, later words had to wait in line while earlier words were processed. Even during training, when the entire source text and translation were already sitting in front of the machine, it couldn't scan the page all at once—making training as slow as a game of telephone.

Ditching the Queue and Grabbing Megaphones: How Transformers and Large Language Models Unlocked Context

In 2017, eight Google researchers published a paper centered on machine translation experiments titled Attention Is All You Need. The message in the title was blunt: drop the recurrent neural network; attention alone is enough.

The new architecture proposed in that translation paper was the now-famous Transformer. Not only did it reshape machine translation, it also became the foundation for most modern large language models, including BERT (2018), the GPT series (from 2018 onward), and ChatGPT (late 2022). According to a 2025 tally by Nature, the paper has become one of the ten most-cited scientific papers of the 21st century.

How does a Transformer differ from a recurrent neural network? Think of it in terms of passing a message down a line:

  • A recurrent neural network is like a telephone game: A row of people stand in line, and a whispered message can only pass from the first word to the second, and then to the third. Not only does the message fade out over long distances, everyone further down the line has to stand around waiting until their turn arrives.
  • A Transformer gives every word a megaphone: Every word in the line shouts at the exact same moment, and every word on the floor hears all the others loud and clear, whether near or far.

If everyone shouts at once, doesn't it turn into chaos? No—because each word also listens selectively, paying the most attention to whichever words in the sentence are most relevant to it in order to fix its exact meaning in context. This mechanism of words looking at one another inside the same sentence and pooling information by relevance is called self-attention.

Look at how self-attention resolves a word with multiple meanings. Take the Chinese word 「苹果」 ("apple" / "Apple") in two different sentences:

  • In 「苹果发布了新手机」 ("Apple released a new phone"), 「苹果」 looks across the sentence with its megaphone, sees that the last word, 「手机」 ("phone"), is most relevant to it, listens mostly to "phone", and tells right away that it refers to the tech company—so it translates as capitalized Apple (illustrative).
  • In 「我吃了一个苹果」 ("I ate an apple"), 「苹果」 listens most to the verb 「吃」 ("ate") instead, immediately knows it means the fruit, and translates as lowercase apple (illustrative).

![In the sentence 「苹果 发布 了 新 手机」, arcs connect 「苹果」 to every word, with 「手机」 marked "Most relevant", alongside notes "Here 苹果 = company" and "Translate as Apple (capitalized: company)", tagged "Illustrative"](/blog-media/machine-translation-history/fig62-en@2x.webp "In 「苹果发布了新手机」, 「苹果」 links to every word, with 「手机」 ("phone") most relevant (illustrative)")

Beyond seeing connections across a whole sentence, the Transformer brought a massive engineering advantage: no queuing. During training, all the words on an entire page can be computed in parallel at once, sending training speeds soaring. And once computation got fast enough, researchers could finally train models on massive amounts of text that would have been unthinkable before.

Later, researchers built large language models (LLMs) on the Transformer. Large language models weren't built specifically for translation; they were designed for general tasks, with translation as just one of many things they can do along the way. And during pre-training, the core skill they practice over and over is just one thing: guessing the next word.

Sound familiar? The n-gram language model in statistical machine translation did the exact same thing: from past words, guess the next. For example, after "Yesterday in Beijing I saw ＿", the machine calculates that the most likely next word is him; if you change the verb to "ate ＿", the most likely next words all become foods (illustrative). In math, this is called conditional probability—just like finishing someone's sentence in everyday conversation, what comes next depends on what was said before.

If both are just guessing the next word, what makes a large language model fundamentally different from the old statistical language models? Two key changes:

  1. How far back it can read: Old statistical language models had a tiny window and could only read a few words before the blank, whereas a large language model can read a whole passage (GPT-3's context window, for example, could hold 2,048 tokens). Suppose a passage reads: 「上周我去北京出差，昨天晚上吃了 ＿」 ("Last week I went to Beijing on a business trip, and last night I ate ＿"). An old language model could only frame the last couple of words ("last night I ate"), with no idea where the speaker was; a large language model takes in the whole passage, sees "Beijing" earlier on, and guesses that the next word is much more likely to be 「烤鸭」 ("roast duck") (illustrative).
  2. Whether it can generalize: Old language models relied on raw phrase counts—if the training text contained 「吃了 烤鸭」 ("ate roast duck") but never 「吃了 烤鹅」 ("ate roast goose"), they stumbled on "roast goose" for lack of data. Large language models inherit neural networks' ability to turn words into continuous numbers: because the number strips for "roast duck" and "roast goose" are very close, even if "ate roast goose" is rare, the model can generalize and get it right (illustrative).

Still, that leaves a puzzling question: if a model spends all day practicing how to guess the next word, and was never explicitly trained as a translator, how did it suddenly learn to translate?

A 2023 study uncovered a major clue: when researchers combed through the training text of a single large language model (PaLM), they found more than 30 million existing bilingual translation pairs across at least 44 languages tucked inside the web pages and documents scraped from the internet. In other words, humans had already slipped vast amounts of translation material into the everyday web text the model read. So when you type a translation request followed by a Chinese sentence, the most sensible continuation the model guesses word by word is the English translation (illustrative).

That is how GPT-3, released in 2020, was trained: by reading hundreds of billions of words so it could continue almost any text—whether writing code, summarizing articles, or translating. Behind GPT-3 was a bold bet: bigger models that read more text perform better.

"Big" here refers to the number of adjustable numbers inside the model—its parameters. From 2018 to 2020, parameter counts skyrocketed:

  • GPT-1 (2018): 117 million parameters;
  • GPT-2 (2019): 1.5 billion parameters—up about 13-fold;
  • GPT-3 (2020): 175 billion parameters—up another 117-fold;
  • For ChatGPT (launched in November 2022, initially based on a fine-tuned GPT-3.5) and GPT-4 (2023), OpenAI stopped disclosing exact parameter counts (media reports put GPT-4 at around a trillion parameters, unconfirmed).

Amid this surge in scale, researchers identified scaling laws: as you scale up parameters, training data, and compute in tandem, model performance rises predictably along a power law—plotted on a log scale, every tenfold increase in resources lifts performance by a steady step along a straight line that can be calculated in advance.

After ChatGPT launched on November 30, 2022, translation became a conversation anyone could refine on the fly. With older translators, you had to take whatever output you were given; now, if a first draft doesn't strike the right note, you can simply ask to change the tone. Ask it to translate 「明天的会改到下午三点」, and it first gives a plain "Tomorrow's meeting is moved to 3 pm." Follow up with "This is for a client, make it more formal," and it immediately rewrites the line in polished business English: "Please note that tomorrow's meeting has been rescheduled to 3:00 p.m." (illustrative).

Even more importantly, because a large language model reads long stretches of context, translation finally moved past literal word-for-word meaning. Take the simple three-character Chinese sentence 「他走了」: on its own, it can literally mean "he left," or it can be a euphemism for "he passed away." If the preceding paragraph has been talking about a hospital and an illness, a large language model can tell from context that it's a euphemism rather than translating it literally (illustrative).

Before large language models, even when neural machine translation got the meaning right, the output often felt stiff and "machine-translated," clinging tightly to the structure of the original words. A polite everyday office phrase like 「这个方案我们还要再研究一下」, for example, would often come out stiffly as "This plan we still need to research a little." (illustrative). A large language model, by contrast, produces natural English: "We need to look into this plan a bit more." (illustrative). By the time of the 2023 Conference on Machine Translation evaluations (WMT23), top machine translations were already on a par with human translations.

Beyond Text: How End-to-End Simultaneous Interpretation Decides When to Speak

Having tackled written text, machines are now taking on an even harder challenge: simultaneous interpretation—translating out loud while a speaker is still talking on stage.

Early speech translation systems used a three-step cascade approach:

  1. Speech recognition (ASR): Turn the incoming speech into source-language text;
  2. Machine translation (MT): Translate the source text into target-language text;
  3. Speech synthesis (TTS): Read the translated text aloud in a synthesized voice.

This three-step relay had two glaring pain points. First, delays and snowballing errors: each step had to wait for the previous step to finish, so the three delays stacked up; worse, if speech recognition misheard a word in step one, that mistake snowballed through translation and speech synthesis. Second, lost emotion: a speaker's tone, emphasis, and emotion live in the sound waves, and once speech is flattened into plain text in the middle step, those vocal cues vanish—leaving the final readout flat and robotic.

To skip the text middleman, researchers turned to end-to-end models: a single model that translates directly from speech in one language to speech in another. With no three-step split, errors from an earlier stage don't snowball into the next, and decoding only once makes responses faster. Google's Translatotron in 2019 was one such direct speech-to-speech model, and it could even keep the speaker's own voice when speaking the translation—though it still had to wait until the speaker finished a full sentence before speaking, falling short of true simultaneous interpretation.

For true simultaneous interpretation, the hardest question is when to speak.

That brings us right back to the problem from the opening of this article: languages like Chinese and English put words in different orders. When a speaker has only uttered the first half of a sentence, the crucial verb may still be waiting in the second half. The longer the machine waits, the more complete the information it hears and the more accurate its translation—but listeners waiting in their headsets will grow restless. Conversely, if it blurts out words the instant it hears them, it can easily mistranslate before hearing how the sentence ends. A good simultaneous system has to walk a tightrope between waiting and speaking: say, translating the first half once it has enough meaning, then attaching the rest once it's spoken (illustrative).

By 2025, newer simultaneous interpretation models adopted full-duplex architectures capable of listening and speaking at the same time. Under the hood, such a system is still a language model guessing what comes next step by step—except now it isn't just guessing which word comes next, but deciding at every moment whether to keep listening or speak now (illustrative).

In 2025, research teams reported that end-to-end full-duplex simultaneous interpretation models can translate live in the speaker's own voice while cutting average voice-cloning translation latency from nearly 10 seconds in earlier approaches down to about 3 seconds—a reduction of roughly 70%.

Four Steps Over 70 Years: To Translate Well, First Understand

Pull the timeline back to 1947 and plot these 70-plus years of exploration on a chart, and three major leaps stand out clearly: statistical machine translation around 1990, neural machine translation in 2014 (cutting errors by about 60% in Google's 2016 tests), and large language models in the 2020s (reaching near-human quality in 2023 evaluations).

Chart titled "Ever deeper understanding" plotting Quality vs. Year from "1947 Decipher" to "2025 Live interp.", highlighting three leaps including "Leap", "Leap: 2016 Google ~60% fewer errors", and "Leap: 2023 evals near-human", tagged "Illustrative"
70-plus years of machine translation and its three major leaps toward deeper understanding (illustrative)

At first, people thought translation was simply a code-breaking puzzle: match the symbols between two languages and you're done. Only after hitting wall after wall did it become clear that rigid grammar rules can never cover every exception in human language, and statistical word-pair counts still miss actual meaning.

Every step forward in machine translation has meant reading one layer deeper into human language:

  • Step 1: Symbols (1954–2013). Whether through human-written dictionaries and grammar rules or statistical word-pair counts across massive bilingual texts, words to the machine were just isolated symbols being matched to other symbols.
  • Step 2: Word meaning (2013). Word vectors turned each word into hundreds of continuous numbers, letting machines calculate how close words are in meaning for the first time and apply what they learned on one word to similar ones.
  • Step 3: Sentences (2014–2017). From recurrent neural networks and attention to the Transformer, machines learned to compress a whole sentence into memory and use self-attention to let every word listen to the rest of the sentence, fixing each word's meaning and order from the sentence as a whole.
  • Step 4: Context (2020–present). Large language models widened the view from a single sentence to a full passage or conversation, combining prior context, tone requests, and even a speaker's voice to deliver fitting translations.

Just like that simple opening line—「我昨天在北京见了他」 ("I saw him in Beijing yesterday")—even moving a few words to the right places takes understanding the sentence first. Because translation was never just about swapping symbols; translation is about understanding.

References