hmm

GT now gets 55% accuracy on English to Arabic. Human agreement on human translations is 60%. After this point they have no standard by which to measure their progress

2016-09-27: Getting amazingly close to human level performance. it’s interesting that for all languages, the gap between human and perfect translation is much much larger than between human and machine.

Neural Machine Translation: Much better translation quality
Full technical report (23 exciting pages of bedtime reading)

Research blog post

I’m very excited to announce that our new neural machine translation system closes the quality gap between the existing Google Translate production system and human quality translations by 58% to 87% for a variety of different language pairs (see table below, from the technical report we published today). This work has been a close collaboration between the Google Brain team and the Google Translate team.

Thanks to lots of hard engineering work and the computational efficiency of our Tensor Processing Units (see report), we are also rolling these benefits out to users of Google Translate, starting today with Mandarin to English as the first language pair live in production that uses this new system. We’ll be rolling out many more language pairs over the coming weeks.

This highlights the success of neural models at more accurately capturing the complexities of real human language, and is a powerful demonstration of the research our group has been doing on language understanding.

2016-11-15: Nice behind the scenes article on the recent translation breakthrough.

With this update, Google Translate is improving more in a single leap than we’ve seen in the last 10 years combined.

3 overlapping stories converge in Google Translate’s successful metamorphosis to A.I. — a technical story, an institutional story and a story about the evolution of ideas. The technical story is about 1 team on 1 product at 1 company, and the process by which they refined, tested and introduced a brand-new version of an old product in only about a quarter of the time anyone, themselves included, might reasonably have expected. The institutional story is about the employees of a small but influential artificial-intelligence group within that company, and the process by which their intuitive faith in some old, unproven and broadly unpalatable notions about computing upended every other company within a large radius. The story of ideas is about the cognitive scientists, psychologists and wayward engineers who long toiled in obscurity, and the process by which their ostensibly irrational convictions ultimately inspired a paradigm shift in our understanding not only of technology but also, in theory, of consciousness itself.

2023-07-08: Akkadian translation, with modest BLEU scores.

In its transliteration to English test, the AI model scored 37.47. In its cuneiform to English test, it scored 36.52. Both scores were above their target baseline and in the range of a high-quality translation. The model was able to reproduce the nuances of each test sentence’s genre. The AI model works best when it is translating short- to medium-length sentences. It also does better with more formulaic genres, like royal decrees and administrative records, than literary genres such as myths, hymns, and prophecies. With more training on a larger dataset, they aim to improve its accuracy. “100s of 100s of clay tablets inscribed in the cuneiform script document the political, social, economic, and scientific history of ancient Mesopotamia. Yet, most of these documents remain untranslated and inaccessible due to their sheer number and limited quantity of experts able to read them”