The translation and adaptation journey of a TIMSS grade 8 science item

by Pisana Ferrari – cApStAn Ambassador to the Global Village

The IEA’s Trends in International Mathematics and Science Study (TIMSS) is a series of international assessments of the mathematics and science knowledge of students around the world, administered every four years. TIMSS 2019 was conducted in 64 countries and 8 benchmarking systems. The outcomes of these studies inform education policies, and the most stringent quality assurance standards apply, from item development to data analysis. At a recent live webinar David Ebbs, Senior Researcher at the IEA, and cApStAn’s CEO Steve Dept, came together to present the translation and adaptation journey of a TIMSS grade 8 science item. cApStAn has been working with IEA to ensure comparability across language versions and across countries since 2001, and has collaborated with David Ebbs on TIMSS since 2005. Translation verification is one of cApStAn’s key linguistic quality services, and its methodology is applied in many high-stakes projects, including international large scale assessments, surveys, polls and talent management assessments.

A collaborative approach for the entire life cycle of an item, from its development to LQC

At the IEA item development is a collaborative process which involves constant interaction between the national teams, subject matter experts, reviewers, the IEA and the International Study Center. It includes a content development stage, pilots, revisions, and final approval of items for inclusion in the field trial. Countries can then begin the translation, adaptation and verification phase. Verifiers provide a fresh-eye perspective: they can diagnose and report potential issues that may lead to meaning shifts (language-driven) and perception shifts (culture-driven), which can affect the comparability of data across languages and suggest corrective action. The cApStAn methodology for translation verification includes segment by segment checks against pre-defined “intervention categories” which include missing or added information, adaptation issues, mistranslations, inconsistencies, register, grammar, syntax, words left in source language, format, layout. All interventions are documented in a central monitoring tool; the categories are linked to formulas embedded in the worksheets to generate metrics. The national teams have the final say, as they are responsible for the national translation. Together with the IEA and cApStAn they work together towards a common goal, that of ensuring that all the different language versions of the test meet the highest standards of linguistic, cultural and functional equivalence. Working closely together and documenting each step of the translation, adaptation and verification history provides a wealth of useful information to which researchers can revert to when an item performs in an unexpected way.

Check the webinar recording at this link to learn more about

• the history of translation quality management in IEA surveys
• details of the item development process at the IEA
• a concrete example of the life cycle of an e-TIMSS item
• adaptations to local usage and context that may be necessary in order to avoid putting respondents at an advantage or a disadvantage
• details of the translation verification process
• how translation verification differs from review and proofreading

Would you like your translated/adapted assessments or questionnaires to be verified by cApStAn? An independent quality check is always desirable in order to ensure that the translated test or questionnaire is fair and comparable in all languages. Request a quote here.