Machine Translation Engines vs. Generative AI: Differences in Translation Performance

Note: This blog post was originally written in Japanese for our Japanese website. We use our machine translation platforms to translate and make automatic corrections, and then partially edit to fit the content in English. The original Japanese post can be found here.

In this post, we’ll examine the difference in translation performance between still-red-hot generative AI and machine translation engines (also known as AI translation engines).

In recent years, the accuracy of machine translation engines has improved dramatically, and generative AI tools such as ChatGPT, Gemini, and Claude have also emerged. While these technologies are benefiting translation work, many of you may be wondering, ”Which is better for translation, generative AI or a machine translation engine?” or ”Is there a difference in translation accuracy?”

In this post, from the perspective of a sales team at a translation agency, we will compare the current landscape and explore the differences between machine translation engines and the translation capabilities of generative AI. Toward the end, we also touch on what Kawamura International sees as the current optimal solution, so we hope you will read through to the end. Please note that, in this post, generative AI refers specifically to ChatGPT.

What are the differences between machine translation and ChatGPT in terms of translation performance?

Both support multiple languages, and for everyday communication or simple business exchanges, they can often be used as-is with only minor edits or sometimes with none at all. Because you don’t need to translate from scratch, they can reduce not only turnaround time but also costs.

With that in mind, let’s take a look at the differences in translation performance between machine translation engines and ChatGPT. Here, we will focus on four points: translation accuracy, writing style, processing speed, and cost-effectiveness.

Translation accuracy

In terms of translation accuracy, it’s fair to say that both are highly accurate. However, depending on a document’s level of specialization and the required accuracy and quality, machine translation has the advantage of having been trained specifically for translation. For this, let’s look at this from three perspectives: 1.) Specialization, 2.) Accuracy, 3.) Reproducibility.

1. Specialization
ChatGPT typically outputs answers intended for a general audience without forcing the use of technical or specialized terminology. However, by specifying terminology and the field in a prompt (instructions or commands given to ChatGPT), it can accommodate this to some extent. That said, it takes some practice to use prompts effectively.On the other hand, there are already machine translation engines specialized for various fields such as patents and finance, and these engines can also be customized. Therefore, when it comes to translating in specific domains, machine translation engines can be said to have an advantage over ChatGPT.

2. Accuracy
ChatGPT is designed to generate natural-sounding text, and because it has been trained to respond in the form of answers to questions, it returns fluent writing.
On the other hand, machine translation engines basically work in a similar way, but because they are trained specifically for translation, machine translation engines can be said to be superior in terms of accuracy.

3. Reproducibility
ChatGPT can generate different translations even when you enter the same text, meaning it has low reproducibility. As a result, translations may lack consistency. This may not be an issue for a one-off translation, but when translating text you use repeatedly or passages with similar content, inconsistencies can become noticeable, so care is needed.
On the other hand, until a machine translation engine is updated, the same sentence will produce the same translation. That said, machine translation can sometimes generate very different translations if the wording of the source text differs even slightly. Even so, for documents with many repeated expressions or stock phrases, machine translation is arguably more practical than ChatGPT.

Writing style

Next, regarding style: as mentioned briefly earlier, ChatGPT is a conversational AI model that specializes in generating natural-sounding text. With ChatGPT, you can also flexibly control the overall style of a text by using prompts that specify the desired style in advance.

Machine translation engines, on the other hand, tend to produce a relatively formal, stiff style, depending on the engine. In this respect, if you can use prompts effectively, ChatGPT will likely allow more flexible output tailored to the specific use case.

Processing Speed

Next, let’s look at processing speed. In this respect, machine translation engines have the edge.

ChatGPT is said to have become faster with each new model (GPT-4o at the time of writing), but when it comes to translation, it still cannot be considered particularly fast. This means not only that the translation itself can take a long time, but also that, once you factor in the time needed to consider how to enter the prompts mentioned earlier, it can take a fair amount of time overall.

On the other hand, machine translation engines can handle a small amount of text instantly, and even high-volume content can be processed in tens of seconds to a few minutes. When we tested the processing speed with our platform LDX hub,* ChatGPT required approximately 2 to 8 times longer than machine translation. (The more characters, the longer it takes, so the time varies depending on the number of characters.)

*LDX hub is a platform with a rich set of APIs provided by Kawamura International to solve challenges in language services.

Cost-effectiveness

Finally, while this is not a matter of technical performance, let’s look at cost-effectiveness. In this respect as well, as with processing speed, machine translation engines come out on top.

Based on our in-house research, ChatGPT currently tends to be somewhat more expensive than MT and has the disadvantage that it is difficult to predict exactly how much it will cost. One reason is that ChatGPT is billed on a pay-as-you-go basis, with costs determined by the amount of text processed, but the unit used to measure that amount is a ”token.” At the time of writing, pricing is set as ”$X per 1,000 tokens,” but the unit price ($X) varies depending on the ChatGPT model used. In addition, both input and output are billable, so translating a high-volume document means you end up processing a correspondingly large amount of text.

On the other hand, some machine translation engines use a pay-as-you-go model, but even then, pricing is usually based on the number of characters, which keeps things simple. Many also offer flat-rate plans that allow unlimited use regardless of character count. To give you one example, our XMAT (TransMAT) is a flat-rate service that currently lets you use eight machine translation engines with no character limit.

So far, we’ve looked at the translation performance of machine translation engines and ChatGPT from four perspectives — accuracy, style, speed, and cost. What did you think? In terms of translation performance and processing speed for large volumes of text, machine translation engines appear to be more convenient. However, for tasks such as drafting, summarizing, proofreading, and translating short passages, generative AI can also be used very effectively.

The optimal solution for using machine translation

However, this does not mean that human translators’ work will disappear anytime soon. Naturally, for highly specialized texts or projects that require high quality, human translation comes out on top. And while machine translation accuracy is improving, it has not fundamentally solved the underlying issues that lead to errors such as mistranslations and omissions. Therefore, when using machine translation, human revision (post-editing) is essential, and those who already use machine translation may be revising the translations themselves.

That said, it is also true that many people simply do not have time to correct machine translation errors. To meet those needs, we offer a service called Hybrid Post-editing, which combines the best parts of machine translation and generative AI with a final touch of human translators (post-editing). This is a service that leverages machine translation output while having various errors corrected by generative AI with humans signing it off. Since this is a new service, more information will be coming out on our website in the near future. In the meantime, if you’re eager to learn more, feel free to reach out to us any time!

Kawamura International’s services

With XMAT or our other platforms, you can receive high-quality and secure machine translation without worrying about the transparency of the cost. At Kawamura International, we do more than just provide machine translation services. We also offer a range of suggestions to help make your translation operations run more smoothly. If you are facing translation challenges at the moment or have any questions about our language services, please feel free to contact us.