Note: This blog post was originally written in Japanese for our Japanese website. We used our machine translation platforms to translate it and post-edit the content in English.
A glossary is an essential resource for ensuring that technical terms, proper nouns, and other words and phrases are translated into standardized equivalents in the target language (for the importance of glossaries, see this post). In addition, in machine translation, using a bilingual glossary can enforce the specified translations, making it a valuable resource for improving translation accuracy.
The basic method for creating a glossary is quite simple. First, extract terms from the target documents and then assign appropriate translations to them. That’s it. However, what if there is an enormous number of target documents, and you also need to find corresponding translations from an equally vast amount of previously translated documents? It would require an immense amount of time and effort.
To automate this task, methods for term extraction using tools were devised; however, with conventional approaches, it was extremely difficult to extract high-precision terms from a large pool of candidates, and extraction from parallel texts in particular was far from practical. With the advent of generative artificial intelligence, or generative AI, however, this process can now be carried out efficiently and accurately.
Conventional terminology extraction tools used methods that automatically extract frequently occurring terms from large volumes of documents by applying morphological analysis and other techniques. While this approach could automatically extract candidate terms from many documents, the results contained a lot of noise (i.e., unnecessary term candidates), and it could not handle the extraction of bilingual term pairs. As a result, a great deal of time and effort was ultimately required for sorting them out manually.
By leveraging today’s much-discussed generative AI, it has become possible to extract terms in a specified domain with high accuracy. Because generative AI can flexibly follow detailed prompts about which terms to extract, it can efficiently retrieve only the necessary terms. Furthermore, extracting bilingual term pairs is now possible, enabling bilingual terminology extraction as well.
However, generative AI isn't perfect. Even with carefully tuned prompts, we still encountered some noise and missed terms. Therefore, we devised further measures to improve accuracy.
First, we handled term extraction in the source text and the process of aligning it with the translation as separate tasks. We also fine-tuned AI models for each, improving the accuracy of each task. Fine-tuning is the process of training a generative AI on specific data to further improve processing accuracy. For example, by training it on the patterns of terms contained in particular documents, we were able to reduce unnecessary term candidates and select only those suitable for the purpose. Similarly, for bilingual term extraction, we improved accuracy by training on data in which the translations are correctly aligned.
Terminology extraction process using generative AI
By leveraging a fine-tuned generative AI, the noise that occurred with conventional methods has been significantly reduced, enabling the extraction of only carefully selected terms. In addition, the extraction of bilingual aligned data can be performed smoothly, increasing consistency in the glossary. We're continuing to validate this approach with additional sample data to further improve its accuracy.
AI-powered terminology extraction is a practical answer for companies stalled by large document volumes. It can revive glossary-building efforts that were previously abandoned as too costly or slow.
At Kawamura International, we not only offer glossary extraction using generative AI, but also provide a wide variety of services that leverage AI translation, including the machine translation engine Kawamura NMT powered by NICT. We can make a wide range of proposals tailored to your security requirements, objectives, fields, and intended user groups. Reach out to talk through what would work best for your documents.