Challenges in ensuring cross-cultural equivalence of scales in multilingual surveys

by Pisana Ferrari – cApStAn Ambassador to the Global Village

“Public opinion in a changing world” was the theme of the 2018 annual conference of WAPOR, the World Association for Public Opinion Research, which took place in Marrakech from June 27-30. Attending on behalf of cApStAn LQC was a great experience in terms of the insights and discussions around topical issues and the networking opportunities it offered with experts in the field from academia, research institutions, public sector bodies, market research companies. Over 170 presentations, spread over three days, addressed complex and sensitive issues such as trust in the media, citizen engagement and civic competence, the effects of populism on public opinion polls, the influence of social media on elections, the challenges in surveying displaced populations, in testing attitudes towards migrants and conducting polls in conflict areas, to name but a few. We were honoured to be given the opportunity to present in the course of a session on challenges in comparative surveys on the value of “Using a Searchable Repository of Scale Translations in Multilingual Surveys”. The paper was co-authored by Steve Dept and Elica Krajčeva from the cApStAn Brussels headquarters.

To quote scholars Barbara Byrne and Fons van de Vijver (2010), the goal of a translation of scales in a cross-cultural comparative survey is to produce an instrument that “measures the same construct(s) in exactly the same way” across all language versions. We know of course that word-by-word translation of all the formal or semantic elements of a scale from one language to another does not necessarily lead to functional equivalence. By functional equivalence we mean that the target version “works” or “functions” in the same way as the source. Different types of linguistic and cultural aspects come into play, e.g. when maintaining semantic distance between points on the scale. Once it is established that the construct can be measured in different cultures, it remains a challenge to strike the best possible balance between accuracy and fluency, because one needs to take into account possible measurement effects such as social desirability or acquiescence, but also linguistic and cultural effects such as differences in conversational norms.

Empirical evidence of the impact of linguistic and cultural parameters on the reliability and validity of translated or adapted scales exists. At WAPOR we presented a work-in-progress that consists of a searchable repository (developed in-house) of translations of widely used response scales. These are linked to language/item interactions observed during the analysis of results, whenever such documentation is available. New survey questions can be automatically matched with this repository, with a view to flagging known problems, and possibly prevent their replication. In this way, a number of challenges and biases related to questionnaire scales that are already known can be identified in the earliest stages of survey design, using a combination of human expertise and automated recognition of previously observed patterns. This identification process can be embedded in questionnaire development.

We believe that multidisciplinary discussions at different stages of questionnaire development, supported by a well-maintained repository of existing issues, such as the one we are working on, can lead to informed decisions. Systematic documentation and robust knowledge management will enable researchers to follow-up on any issue that may show at data collection or analysis of results.

Photo by Kees Kortmulder on Unsplash