ESRA 2025 Conference | cApStAn | Utrecht, The Netherlands | 14th July to 18th July
While cApStAn’s founder Steve Dept is more of a field practitioner than a researcher, he has collected a wealth of empirical data in his work on survey questionnaires. He has learned a lot from sharing the data with researchers and academics, as they identified patterns and contributed to the conceptual frameworks that underpin cApStAn’s work. This form of cross-pollination is what has brought Steve and colleagues to the ESRA conferences. The first time he presented at ESRA was in Prague, in July 2007, about adaptation management and documentation in OECD/PISA’s contextual questionnaires. Eighteen years later, after having attended most ESRA conferences and contributed to maximising cross-linguistic, cross-cultural and cross-regional comparability in data collection instruments of over 60 European and international surveys, the evolution of technology has brought us to a crossroads. Translation technology has been used since the 1990s, machine translation and post-editing since the beginning of this century, neural machine translation became widely used as of 2017 and the chatbot-like interface of generative AI went mainstream in November 2022.
Are we at the dawn of a new era? Or do we just have new tools to do what we did earlier, but with greater efficiency, accuracy and consistency? When keeping reliability and validity as our lodestar, should we closely adhere to existing best practices in survey questionnaire translation and customise the new tools so that they can accommodate these? Or have the new tools rendered some components of this best practice obsolete?
The team translation approach to survey translation, for example, was developed at a time when data collection instruments were pencil-and-paper questionnaires. The TRAP-D design (Translation, Review, Adjudication, pre-testing and Documentation) is an often-cited methodology for survey translation and adaptation, and it has been widely recognised as best practice by the scientific community. Meanwhile, questionnaires have become digital and are delivered on computers, tablets, and smart devices. Multiple modes are being used, and respondent-facing materials are no longer exclusively conveyed by trained interviewers. Self-completion has become widespread.
New advances in machine learning, machine translation and large language models (LLMs) can bring efficiency gains but require an expertise that is not always available to survey methodologists. It has become possible to design workflows in which repetitive actions are automated and in which the input of professional linguists and subject matter experts (SMEs) adds value at very specific points in the translation management systems. This requires a level of professionalisation in survey translation workflows, discernment in using AI purposefully, and, inevitably, pre-testing of adapted versions of the survey questionnaires.
Steve looks forward to discussing these issues at ESRA 2025 in Utrecht with peers, survey methodologists, researchers and data scientists.
Do not hesitate to reach out to us on hermes@capstan.be if you want to know more about Steve’s session or TRAP-D approach.