Public opinion research in times of the COVID-19 pandemic: highlights from WAPOR 2021
by Pisana Ferrari – cApStAn Ambassador to the Global Village
The COVID-19 pandemic has taken millions of lives worldwide, led to job losses, reduced incomes, affected student learning and caused severe disruption in many different fields of human activity. The survey and polling industry is no exception. The pandemic has affected how data is collected, modes of delivery, mode effects, recruitment of respondents, and training of interviewers, to name but a few. Changes had to be made, and quickly: this involved an enormous amount of work, creativity in finding new solutions, and skills in leveraging the technology. With COVID-19 continuing to be a worldwide problem, this year’s conference of the World Association for Public Opinion Research (WAPOR) was again held virtually, from 2 to 5 November. The four-day conference, fittingly titled “Speaking Truth to Power. Public Opinion in Times of Crisis” featured a wide-ranging program, an impressive (as always) panel of speakers and excellent presentations. Almost 200 participants from all over the world gathered together for this important event to share experiences and discuss key learnings. In all there were 142 presentations spread out over 41 sessions, including the keynote and training sessions. Looking ahead to the future Marita Carballo, former President of WAPOR, speaking at one of the wrap up sessions, said that thanks to the pandemic pollsters learned a lot about new methods, reviewed and compared with past ones, tried mixed modes and shared best practices. This in turn means that when the pandemic is over they will be more prepared to decide on the most effective ways of polling, whether and when to return to traditional methods or evolve to new ones. The report that follows summarizes some of the presentations about approaches used to ensure continuity of research during the crisis and about emerging trends in the industry.
Experimenting with different modes of delivery
The COVID-19 pandemic radically changed how data for polls and surveys is collected. One of the first effects was to rule out face-to-face interviews (F2F) for obvious reasons of safety for respondents and interviewers. F2F interviews are widely used in qualitative research and are perceived by some scholars to be the “golden standard”. Depending on the countries and organisations, there was a large shift to computer assisted telephone interviews (CATI) and to mixed mode delivery (web-based, e-mail, etc). Each of the new methods adopted did not come without its challenges and consequences.
In Africa F2F has been the preferred mode for public opinion, market, and media research for decades. In a presentation titled “Opinion Polling Across Borders in Africa in 2020: Transitioning from F2F to CATI” the speaker explained that D3 Systems, and its partners in the Netherlands (GDCC), Kenya (Infinite Insight) and Senegal (Africa54) designed and tested a CATI-based solution to the problem, resulting in nationally representative public opinion polls of the telephone-owning population in Kenya and Senegal countries. The key challenges, said the researchers, were to (a) engage the field teams in a safe way, fully compliant with the regulatory environment, (b) align those teams with the linguistic and cultural diversity of the African countries where they lived, and (c) provide clients with the expected levels of quality control and reporting that companies would expect from a best-in-class call-center.
F2F has been the standard mode of delivery for surveys in many Western countries as well, e.g. for the US General Social Survey (GSS), conducted every two years, which collects data on the attitudes and opinions of the general public. The GSS is one of the most influential studies in the social sciences in the U.S. The GSS 2020 data collection had been set to launch on April 6th in F2F mode but was changed to mixed-mode web and telephone. This involved adapting the questionnaire content for web self-administration, refining it for device optimization, and redesigning respondent materials and mailings. The organisers were pleased with the results: the 2020 GSS re-design lead to 40% more completed surveys than originally targeted. Adapting the GSS for mail-push-to-web methodology was a labor-intensive but effective strategy, they concluded (“The 2020 General Social Survey: Adapting to change during the COVID-19 pandemic Switch to web based”).
Switching to online surveys was not a viable option for all. In India, for example, this was not possible, as a vast proportion of the population is still uneducated, said Yashwant Deshmukh, from the CVoter Foundation, during a session titled “Recent Challenges for Cross Cultural Research: Reflections on the International Social Survey Programme”. This means that online responses are totally biased towards urban, high education, better incomes and males. Hence, what works best in India as an alternative to the face-to-face surveys is CATI, but poll organisers were faced with the challenge (as indeed all those who made this change, we add) that interviewers could not come to the office to work. Decentralized CATI was the solution adopted. This in turn involves other issues, including the need to adequately train and monitor interviewers, which is more challenging when done remotely.
Time constraints related to the pandemic had serious consequences for some surveys. In May 2020 the Central Asia Barometer (CAB), a multi-country survey of voting aged adults (18+), officially converted from F2F to a telephone-based methodology. The survey was due to be administered in five countries (Turkmenistan, Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan), but Turkmenistan had to be left out as it did not manage to make the switch in time. As mobile penetration rates in each of these countries was close to 100%, compared to very low fixed line penetration, the organisers were confident that a mobile-only study would provide sufficient coverage of each country’s population. Response rates were high, the demographic profile indicates that the survey was able to reach traditionally difficult groups in F2F environments—the working population and labor migrants—and that there was a good geographic distribution, by both region and urbanicity, in each country (“Central Asia Barometer: A Case Study in COVID-driven Mode Conversion”).
In South Africa F2F polling was maintained, despite the pandemic, for the South African Social Attitudes Survey (SASAS) series. Round 17 of the SASAS series was in the field when it was interrupted by COVID-19: F2F interviews were resumed and conducted in between COVID-19 waves, based on regulatory status and ethics approval by the HSRC, the Human Sciences Research Council (HSRC), South Africa’s statutory research agency, and its Research Ethics Committee (REC). Benjamin Roberts, speaking at the session on “Collecting social survey data in South Africa: Adapting to the COVID-19 pandemic” described this as “riding the waves”. The organizers carefully weighed the risks to interviewers and participants, as well as the potential reputational risk to funders and to the HSRC. For other surveys pollsters in South Africa experimented with online panels, mixed-method studies and other forms of innovation.
In some cases the local political situation also played a role, said the researchers presenting at a session on “Challenges and lessons of fielding an ISSP module using a mixed-mode web and telephone probability-based panel in Chile”. In Chile, the effects of the COVID-19 in 2020 were compounded with those of the social upheavals of 2019, when the government had set night-time curfews and targeted lockdowns at the municipality level, among other policies. Roughly 60% of the population was on lockdown in July 2020; it reached 95% in April 2021. Chile has a strong reliance on face-to-face (F2F) surveys, even for electoral polls. During the pandemic there was a switch to a mixed-mode web and telephone for one of the ISSP, International Social Survey Programme, modules. Respondents received incentives conditional on response. The organisers concluded that mixed-mode helps dealing with respondents lacking literacy skills for web mode: low-educated respondents can be reached by phone. It is important to find the right mix. (1)
Novel approaches to the recruiting of respondents
We found two presentations particularly interesting in terms of novel ways to recruit respondents. The first example was about recruiting health care professionals (HCPs) and was conducted by GESIS and presented in a session title “Recruiting Health Professionals for a Web Survey via Ad Targeting on Social Network Sites: Lessons Learned and First Results”. HCPs are hard-to-reach respondents in the best of circumstances but all the more so during the pandemic. Using social media platforms to recruit specific groups or hard-to-reach respondents is a fairly new method. The speaker mentioned recent examples around the recruitment of migrants (Pötzschke & Weiß 2020) and LGBTQ individuals (Kühne 2020). The Facebook and Instagram ads targeted users who were (self) described as associated with the health care industry, as being interested in health, or a part of the total population (i.e. the “control” group). The data was collected in April/May 2021. In total 21,554 respondents accessed the first survey page, and 3,129 respondents completed the survey. The results add to the sparse empirical evidence and provide recommendations regarding this relatively new approach, say the researchers.
In the second example, text messages were used to collect data for the most recent National Election Study in Greece, in the context of a research project for the University of Thessaloniki. SMSs were sent to randomly generated mobile phone numbers, in three steps: a pre-notification, an invitation, with a link to the web survey, and (where necessary) a reminder. The findings indicate that although some groups (e.g. older people, people of lower education levels) were under-represented, this method has many advantages: over emails (recipients may ignore or even delete them along with the unwanted messages), and over calls: (bad timing, call answered by someone else), and should be considered (at least) as one of the modes. A total of 26,052 mobile phone numbers received an invitation, and the total number of cleaned and fully completed cases was 1,538. Considering all mobile phone numbers as eligible, the response rate was estimated at: 1,538/26,052 = 6% (“Recruiting participants for a probability based web survey sample via text messages (SMS)”)
Challenges in the remote training of interviewers
In-person interviewer training allows trainers to directly observe interviewers learning and practicing techniques related to survey administration. Virtual training can provide a safe, socially distant alternative but comes with some tradeoffs, Nina Thornton from Westat, speaking at a session titled “Virtual Interviewer Training in an International Large-Scale Assessment”. The international large scale assessment in question is the Program for the International Assessment of Adult Competencies (PIAAC), a cyclical, large-scale study that was developed by the OECD. The presentation discussed the lessons learned from virtual training that took place in four countries: Germany, Italy, Spain, and the United States. Challenges included having to adapt training scripts, building team rapport with interviewers and observing trainees to assess their performance in remote, and troubleshooting technical issues. On the plus side, there were savings in costs (travel, accommodation, other logistics considerations) and virtual attendance reduced issues that manifest during in-person training (e.g. audio, visualizing slides). Trainees were engaged and attentive, thanks to (virtual) breakout rooms trainee interaction was facilitated, and chats were effective in collecting questions or for clarifications.
New sources of information for public opinion
Social media and text messages were analysed by researchers at the University of Jaume I of Castelló, Spain, as potential new sources of information on public opinion, for a study titled “Social Media and Mobile Instant Messaging Services: The influence of gender, age, education and incomes on social and political participation”. To do this, an online survey was carried out on 1,202 people residing in Spain. The sample was stratified according to four sociodemographic variables: gender, age, educational level and income. The results indicated that users prioritize the “social” uses of social media and mobile instant messaging services. Political participation in these platforms of the people surveyed was low. Older men (65+) were found to use these platforms more frequently for political purposes, such as receiving and sharing information about political events, or disseminating political comments written by others. These findings, however, are in contrast with those obtained by previous studies, say the presenters, so that further research will be necessary.
Dion Deng, from the Hong Kong Baptist University, analysed the use of the Tik Tok social media platform to test his hypothesis that Tik Tok “cultivates” materialism in young people. For his study, titled “TikTok’s cultivating effect on youths’ perception of materialism” he recruited 748 under-25 TikTok users, using the online network sampling technique. The findings suggest that there is a positive correlation between youths’ perception of materialism and their frequency of using TikTok. Deng says this study is a pioneer in applying the “Cultivation Theory” (developed in the 60s and related to the influence of TV on viewers) to TikTok, and other social media, and can contribute to both the theoretical and the empirical understanding of media’s cultivation effect in a user-generated media (UGM) context.
Conclusions
At one of the final panel discussions on Day 4 titled “What lessons can be drawn from past experience to ensure that, in the wake of COVID-19, public opinion research is accessible and actionable by decision makers?”, one of the panelists, Michelle Harrison, from Kantar Public, said that when the pandemic struck, governments needed rapid insight, to inform their first response to the health impacts and the associated economic crisis of COVID-19. But, beyond the initial moments of crisis, she added, as global governments work to address the health, social and economic outcomes of the pandemic, representation across society, and rich, granular evidence, will be critical to policy making. How will this evidence be collected? What will public opinion research look like in future? Joseph Asunka, from Afrobarometer, pointed out that the pandemic brought about many challenges, but also opportunities. Afrobarometer, for example, expanded its methodological repertoire in ways that will allow it to more rapidly respond to urgent needs for public input and feedback. The pandemic has also pushed it to expand its outreach toolkit, including conducting regular webinars that may actually reach wider audiences than in the past. The pandemic has been a gamechanger in this respect for all players in the field. Whether the industry will return to its traditional ways of polling post-COVID-19, or embrace new, alternative methods, learnings from the pandemic will be key in informing any future research strategies.
Footnotes
1) The problem of polls in Chile was also the object of the sobering opening keynote by Marta Lagos, founder of the Latinobarometer and a leading Latam and world expert in the field, who reminded the audience of how challenging it is, aside from the pandemic issues, to conduct polls in “imperfectly consolidated democracies” such as Chile. Manipulation in producing and publishing polls, initiatives to restrict, regulate and ban polls, lack of official standards for polling organizations, inaccuracy of polls coverage in the news, are just some of the issues faced in such countries.
See also our blog article on New technologies are shaping the way surveys are done
Photo credit: Background photo by Fusion Medical Animation at Unsplash, graphic elaboaration by Graphillus