Diversity, Equity, Inclusion and Bias Reduction: DEI-BR Analysis and Adjustment of survey questions
Awareness of diversity, equity, inclusion and bias reduction (DEI-BR) in surveys is growing, becoming increasingly indispensable as it ensures the reliability of any data collection instrument. cApStAn is a language service provider with a 20+ year experience in partnering with organizations who administer surveys in multiple languages and cultures.
Our initial contribution to DEI-BR started with our “Translatability Assessment” process (TA), which aims to identify and resolve potential translation and adaptation issues in questionnaires well before the translation process begins. These include sensitive content, relevance for all segments of the target audience, portability of constructs, translatability, ambiguity, gender mainstreaming, racialized stereotypes, and potential cultural bias. Over time our TA process was progressively adapted to include DEI-BR evaluation by linguists, with input from subject matter experts.
Project Scope
Our client (anonymous due to confidentiality agreement) approached us for two survey questions which they were fielding and for which they were finding similar results in several of the language versions they were administering them in. However, the objective was to get at two different experiences with these questions.
Challenges
There was an issue with distinguishing these two concepts, in other words that respondents were not “getting them” separately in other languages as they were in English. In English there are concepts that contain a range of definitions (this is not uncommon in other languages as well of course). So, it looked like there was a semantic problem due to the overlap of concepts in the source (English).
cApStAn’s approach
The translations of the questions had already been done, and, had we done a “Translatability Assessment “(TA) on the English source, prior to the translation, we could potentially have picked this issue up. Our approach consisted in:
- A check on the English source version to see if the problem originated there, as opposed to in the translations.
- Review of the translations.
- Addressing targeted questions to the linguists doing the review about the range of meaning of each of the two concepts in their language as compared to English to get to the nuances, e.g. What do these words mean in your language? What other words might cover these concepts better?
Outcomes of the research
The findings of our research indicated the two concepts were difficult or impossible to distinguish in some languages. Our proposal was to make the distinction clearer or drop one of the two questions.
Conclusions
Diversity, Equity and Inclusion adjustments in surveys are a real challenge, and the more people you aim to include the bigger the challenge. The challenge is exponentially bigger when adding multilingual and multicultural dimensions.
Have a process in place to mitigate possible problems.
Include representatives of the sample that is being targeted in initial preparations of developing the survey questionnaire to ensure cultural appropriateness and thus fairness and equity.
Select some sample items or sample questions and request a free DEI analysis at hermes@capstan.be or contact us via our Contact form for more details.
See also, from our blog