Do Trained Interviewers Follow Their Script? Reports from Zedland

by Steve Dept, cApStAn CEO

In multilingual surveys, numerous factors may contribute to — or compromise — data comparability across countries, across cultures and across languages. Upstream quality assurance work may include cognitive labs, piloting and a translatability assessment. A sophisticated team translation design can be used to minimize item bias due to language-driven meaning shifts and culture-driven perception shifts. The results of the pre-test can be analyzed carefully, and corrective action can be taken. Further downstream, a variety of control steps can and should be implemented to ascertain quality and reliability of the data.

This post reports surprising findings from spot checks on a household survey administered in multiple countries. These were requested by a reputable polling organization upon observing that a number of interview recordings from Zedland seemed unusually short. No need to google Zedland: it is a fictitious country name, used here because the focus is on interviewer behavior, which is not a national characteristic. In Zedland, the majority of the population speaks Zedish, which is not an Indo-European language. That is why the polling organization asked cApStAn whether reports on interview recordings in Zedish could be produced at short notice. Our Zedish linguist would not have to transcribe or back translate the interviews — which would have been too costly — but would listen to them carefully and report on the extent to which the interviewer followed the script, whether s/he asked all questions, whether s/he read out all response options and at what speed, whether s/he probed when the respondent seemed unsure.

Our linguist — who had previously been commissioned to verify and validate the Zedish version of the survey questionnaire —was shocked: the Zedish interviewers deviated from the script and interviewer guidelines in every possible way: they offered responses themselves; modified questions; skipped questions or response options (presumably when too embarrassed to ask) and allowed a second person to enter the interview.

A brief selection of findings

In a question about social acceptance of a given sexual inclination, the interviewer replied to the question herself and went on to the next question. One could wager that all interviews by this interviewer will have the same response…

In a question about marital status for which the interviewer is supposed to read out five different response options, the wording was “simplified” to “You are not married, are you?” The respondent replied: “no”.

In a frequency question about religious assiduity outside holy places, the interviewer changed the wording from “sharing one’s faith” to “imposing one’s religion, like Jehovah’s Witnesses do”, and changed the question format from a frequency scale to a yes/no question. One wonders how the responses are coded in such cases.

In another interview, the respondent left after answering the first half of the questions. The interviewer apologized for keeping him for so long. When the respondent was about to leave, the interviewer was heard saying “sure, you can go, your aunt will answer the rest of the questions”. Enter the aunt as a second respondent. She answered only a few questions, and the end of the questionnaire was skipped. Then our linguist picked up a really rich comment: “I can actually do it faster than that. I had a respondent, a very smart guy, he would answer my questions right away, without thinking, and we made it in 18 minutes, but they discarded the interview, can you imagine that? They said it was too short. Such a shame!” And the aunt replied “Oh! What a shame! They should have complimented you, because you are so quick.”

The polling organization said that they found the short reports on the interview recordings from Zedland “most helpful” and did not comment further. They will not have been impressed with the level of professionalism of some of the Zedish interviewers.

How can cApStAn help

As an independent linguistic quality control agency, cApStAn can intervene at multiple stages of your multilingual projects: from source optimization to training translation teams, from documenting adaptations to validating questionnaire translations, from analyzing language-induced item bias to managing translation memories across survey cycles. And we can produce informative reports on your interview recordings at short notice: no matter how much effort has been put into obtaining equivalent translations, if the interviewer deviates from the script, comparability is threatened.