Psychometric tests often have high stakes: how to address potential biases and other challenges when adapting them in multiple languages

by Pisana Ferrari – cApStAn Ambassador to the Global Village

Psychometric tests in technology-rich environments, adaptive psychological tests, social and emotional skills, and profiling tests have gone multinational, thereby increasing the need for translation and adaptation into multiple languages. The International Test Commission guidelines for translating and adapting tests are our “guiding light” and our twenty years of experience in test translation have resulted in a vast knowledge base of linguistic features known to drive or affect psychometric characteristics of test items. This knowledge base is the underpinning of the linguistic quality assurance services we provide to talent management organizations before, during and after test translation.

Projects we have recently worked on in this field include transadaptation of personality tests, simulation exercises and cognitive tests into several languages, whereby the tests were designed to assess candidates for hiring and promoting them at higher manager roles. In addition, we were asked to verify the translation of a consolidated assessment report, which was an integration of the results of various psychometric tests (cognitive + non cognitive) for the shortlisted candidates. This report provided insights to hiring managers, and helped them make decisions.

Different types of psychometric tests require different adaptation approaches

Pre-hire profiling and personality tests can be questionnaires with anchoring vignettes and different response scales, e.g. intensity, frequency or agreement. They can be pairs of statements, for which the candidate needs to select which of the two statements is most like him or her. And there is a variety of other tailored psychological assessment methods. All are sensitive to semantic overlaps. Intentional deviations from the source may be necessary to achieve construct invariance. cApStAn linguists know how to deal with the subtleties of test adaptation in psychometric testing and are trained to document their choices. Whenever possible, in simulation exercises and situational judgement tests, we interact with test owners/test authors to find out to what extent the use of jargon is intentional and to discuss the range of adaptation permitted in distracting materials. We also take into account the potential impact of differences in professional environment and work culture. In verbal and numerical reasoning tests it is crucial to focus on linguistic features that drive or influence the level of difficulty. These include the use of passive voice, the proportional length of key and distractors in MCQ, the grammatical match between question and response options: that is why we draw up item-by-item translation and adaptation notes for translators and reviewers. All cApStAn linguists follow our training sessions in translating and reviewing test materials.

Potential biases when translating psychometric tests for different cultures and languages

Tests earmarked for translation may contain idioms and colloquialisms that cannot be rendered adequately in target languages. There can be potential ambiguities (wording that can be interpreted in more than one way), cultural issues (text elements may be difficult to adapt to a particular culture or language group) or register adjustments to make (text elements may not be suitable for the target population, too formal/too informal). Ultimately, test adaptation is about ensuring that translation does not put respondents from a particular target group at a disadvantage or at an advantage. Fairness is key to comparability. Take the question “What kind of a doctor is a dentist”? In some languages the answer may be in the name, giving an unfair advantage. In German “Zahnarzt” literally means “tooth doctor”…

Before: write a test suitable for translation to avoid issues later down the line

Time spent optimising the source version “upstream” drives quality of translated texts and saves time “downstream”. The cApStAn translatability assessment process checks each segment and uses a set of standardised translatability categories to report potential issues in a centralised monitoring tool. Occasionally, alternative text is proposed — but without loss of meaning. A translatability assessment will validate the assumption that a source questionnaire is ready for translation into the intended target languages. It will ensure that it is possible for multiple language versions of a document to meet the highest standards of linguistic, cultural and functional equivalence. cApStAn’s translatability assessment methodology is being applied in small and large multilingual projects, in both the private and the public sector. You can see our translatability assessment process diagram in this article.

After: makes sure the translated test is fair and comparable in all languages 

Once a test is translated, an independent quality check of the translation is always desirable in order to maximise comparability. Translation verification, an important component of linguistic quality assurance, should compare source and target versions segment by segment, check compliance with translation and adaptation notes, with a focus on semantic equivalence. At cApStAn we use a set of standardised intervention categories to report issues detected by the verifiers. The intervention categories include missing or added information, adaptation issues, matches and patterns, inconsistencies, wording issues, register, grammar, syntax, mistranslation, words left in source language, as well as format and layout.

Check out one of our case studies on translation verification of a psychometric test

Towards a collaborative model where psychometricians and linguists work together

At cApStAn we believe the time is ripe for the testing industry to challenge the traditional interaction between testing organisations and language service providers: replace the client/vendor consecutive model with a collaborative model in which test developers and linguists join forces to shape, streamline and promote good practice. Embedding translation in the item development phase will effectively contribute to maximising cross-language comparability. It allows psychometricians to take more language parameters into account and linguists more psychometric features.

Conclusions

Translating psychometric tests requires high quality translation by an experienced language service provider in order to ensure the highest level of linguistic, cultural and functional equivalence. John Kleeman, founder of Questionmark, one of the world’s leading online assessment platforms, who honoured us with a guest article for our blog, says that assessments provide huge benefits to organizations, to individuals and to society. For organizations, good quality tests allow them to choose the best candidates to recruit, promote or develop based on merit. For individuals, assessments allow bright people to show their merit, even if they are distant or disadvantaged. For society it makes the allocation of people more efficient, allowing skilled people to flow to areas where they are needed and helping people learn new skills. High quality translation of tests and exams really matters to our world, he concludes.