How to avoid the hidden threats to translation quality in computer-based assessments
by Steve Dept, cApStAn CEO
A majority of test providers have migrated from pencil-and-paper assessments to computer-based assessments in the first two decades of this century, and this has driven a surge in sophisticated testing platforms. These offer a broad range of advantages: from enhanced testing experience for respondents to adaptive testing, from end-to-end test security to reports delivered in real time, and from log data on respondent behaviour to remote proctoring, everything is faster, better, and cheaper. The logistics are simplified, the content is interactive, and even open-ended responses can be marked online. Onward!
In some of these computer-based assessments environments, however, scant attention was given to multilingual testing. Most testing platforms do, of course, accommodate multiple languages. The engineers have implemented and tested displays in kanji, hiragana and katakana for Japan; they have their act together for right-to-left and bidirectionality, so tests can be administered in Arabic, Farsi, Hebrew or Urdu. They can accommodate Georgian and Armenian and Devanagari script, as well as traditional and simplified Chinese. Well done. What many stakeholders do not seem aware of, is that there is a wide gap between accommodating multiple languages on a testing platform and implementing best practice in test translation and adaptation. Without state-of-the-art translation technology, ensuring translation quality becomes a tedious, work-intensive and error-prone process.
Let me reproduce the gist of a dialogue that I seem to have again and again with stakeholders of the testing industry:
“So, Steve, who uses translation technology then?
– All reputable language service providers do. Almost all translation professionals do.
– And why can’t they just continue to do so in computer-based assessments platforms?
– Because the platforms don’t offer any translation technology. Zilch. No well-built terminology bases, no translation memory management, no style guides, no good spell-checking function, no tag editors. Translation technology has evolved for over thirty years to reach its current state. One can’t just recreate that in a testing platform.
– So, Steve, tell me: what can be done about it?
– One can export well-formed XML files and let language professionals use the tools of their choice to ensure translation quality, and then import the files seamlessly into the platform.
– has that been done yet?
– Sure. And, potentially, each platform could do that. Put the platform engineer in a room with a good translation technologist, and they’ll work it out. This conversation needs to take place long before any test translation project begins, though.”
See, in the narrative of the switch from pencil-and-paper tests to computer-based assessments, the starting point has often been to reproduce in a computer-based environment what had previously been successful for pencil-and-paper instruments. In this process, workarounds were identified to circumvent obstacles and fixes were created to patch up flaws. Only basic functionalities of word processors were emulated and, failing better methods, abundant use was made of manually copy-pasting content from one mode to the other. Translation workflows were designed and managed by test platform managers who mostly had insufficient knowledge of the technology available in the language industry.
The testing organisations—who are the clients of the platform specialists—wish to have control over each and every aspect of the computer-based assessment, including translation quality. We have seen organisations invest several hundred thousand euros (or dollars) in integrating primitive translation tools and workflows into their testing environment rather than consider letting linguists use existing computer-assisted translation tools outside the platform. Professional translators regard this as sub-optimal: they will find the consistency checks inadequate, will be unhappy if they can’t create and maintain translation memories as they translate, they will miss the tools in which they have become expert users over the years.
In a nutshell: we have observed that, with present-day computer-based assessment, translation quality has at times regressed, not improved. All-in-one platforms with limited translation technology are at odds with the technology-driven need to professionalise translation and adaptation in testing environments. If modern testing platforms deny professional translators the opportunity to harness the power of mature translation technology, they are putting validity, reliability and cross-language comparability of tests at risk. Failing to devote enough attention to the translation, adaptation and localization processes from the outset actually generates a decrease in translation quality and consistency of translated versions of test instruments.
If you’d like to find out more about how to bridge the gap between accommodating multiple languages on a testing platform and implementing best practice in test translation, do email us your queries at firstname.lastname@example.org and we’ll get back to you as soon as we can.
Check out our blog for snippets of information about and around the linguistics field in general, about testing and assessment, and musings (occasionally even insights) on AI, ML, NMT, DL and NLP (artificial intelligence, machine learning, neural machine translation, deep learning and natural language processing).
Photo credit: Shutterstock