The 2025 International e-Assessment Awards | cApStAn | London, UK | June 9th – 11th
If you are attending the 2025 eAA conference in London, United Kingdom, we shall be delighted to have you in attendance at the session where cApStAn’s Founding Partner Steve Dept (he/him/his) will talk and explain about the capabilities and limitations of artificial intelligence in producing multiple language versions of assessment items.
Neural Machine Translation (NMT) is a mature technology and large language models (LLMs) are becoming increasingly accurate for a large number of language pairs. For 25 years, we have heard that translation will soon become commoditised as an inherent feature of almost any application. For the past 3 years, generative artificial intelligence (gen AI) has been used to translate and also to generate content directly in multiple languages. This has considerably boosted the expectations in the testing industry. At the same time, there is consensus that, for sensitive or high stakes content, human expertise in the loop (or at the core, depending on the type of workflow) is required. Unsurprisingly, hybrid workflows have emerged: AI-driven localisation workflows with human revision, reconciliation of a human translation and a machine translation, AI-driven translation quality estimation, machine translation and post-editing, to name but a few, have brought non-trivial efficiencies and a new approach to risk management.
This presentation will use an existing use case to outline the capabilities and limitations of artificial intelligence in producing multiple language versions of assessment items. It will also help determine at what points in the workflows the human intervention adds the most value. For 25 years, we have worked on maximising comparability across language versions of major international large-scale assessments (ILSAs). The work has evolved from validation of human team translations to building APIs and connectors to leverage state-of-the-art translation technology.
In this session; he shall set the stage with a brief history of translation technology in assessment—from early rule-based systems and computer-assisted translation (CAT) tools to statistical machine translation, neural machine translation, and finally today’s large language models.
Through concrete examples, he’ll show that (i) accurate translations do not necessarily imply functional equivalence; (ii) no-one has been able to teach AI how to purposefully deviate from the original when this is necessary to obtain the desired effect in the target culture; and (iii) culture-driven perception shifts are more challenging to address than language-driven meaning shifts.
Attendees will learn how:
• Straightforward translations can change item difficulty
• Fluent AI-generated text can mask critical semantic errors
• Content profiling enables different ad hoc workflows for different components of the content
The presentation will showcase a modern workflow that leverages AI efficiently while maintaining assessment integrity. This hybrid approach includes:
1. Using trained neural machine translation engines with domain-specific terminology
2. Applying AI quality estimation (AIQE) to identify confidence levels in translated segments
3. Strategic human post-editing focused on high-risk or low-confidence content
4. Validating translations through psychometric analysis of pilot results
The presentation will conclude with insights into the future trajectory of AI translation in assessment.
Attendees will leave with a practical framework for evaluating when and how to incorporate AI translation into their assessment processes, understanding both the significant efficiency gains possible and the critical limitations that require continued human oversight. This balanced approach will help assessment professionals make informed decisions that maintain validity, reliability, and fairness in multilingual assessment environments whilst taking advantage of emerging technologies.
To know more, contact us at hermes@capstan.be or bizdev@capstan.be