22.03.2021

A Tale of Joint Resolution of Seemingly Insuperable Challenges

by Dr Paulína Koršňáková in collaboration with Elica Krajceva from cApStAn

The work on large-scale assessments in education is similar to working on other projects. One just needs to imagine a rather complex endeavour that includes global coverage, multiple languages, and rigid timelines. Add a pinch of non-expert governance that has the power to steer operations at any point, usually without much consideration of the impact and consequences of decisions taken.

This specific international project involved a questionnaire survey, designed to collect valuable data from teachers and school principals. In order to serve the survey’s aims (informing policy review and development in the participating countries), comparability across participating educational systems was a necessity.

This implied that translated versions were expected:

(i) to convey the meaning of the international source version of the questionnaires;

(ii) to maintain the link to questionnaires formerly used at a national level;

(iii) to maintain any links between the teachers’ and the school principals’ questionnaires; and (iv) as a bonus challenge, the questionnaires had to take into account differences among the multiple levels in which schools are organized.

Note that, while IEA saw to it that a healthy portion of the questions were the same across all administered teacher questionnaires, a non-negligible number of adaptations had to be implemented, too. Neither the extent of the questionnaires (in number of words), nor the scope of required changes were anticipated or budgeted. It should come as no surprise that the quality assurance procedures were conceived as a check point focussing on key aspects of the instruments, aligning the different versions of questionnaires across school levels and across language versions.

Usually, one needs to pay attention to the size of the instruments, so respondents can complete the questionnaire swiftly and do not drop out. Thankfully, we found out that teachers are very kind respondents who like to share their opinion, so they coped well with the inordinate length of the questionnaires.

The less time you have, the more important is to get everything right the first time. Well, last minute additions to the instruments as well as late discoveries of ambiguous wording made it very complicated.

In the end, every target was attained. This was only possible because we all played with open cards, communicating problems as they surfaced, discussing the most suitable way out.

While IEA was trying to get organised with the post-last-minute notices and changes to the international source versions of the questionnaires all the way to layout verification stage, the verification team spared no effort to disentangle the wondrous knot of links we had in this project: links between the principal and teacher questionnaires, between the different school level options, between the past survey, the field test and the main data collection instruments.

The challenge was also high for the verifiers: cApStAn team went the proverbial extra mile to provide seemingly simple instructions for a highly complex task:

– The necessary background information shared with linguists was reduced to what was required to get the gist of the exercise.

– All checks that project managers could perform after the verifiers’ work were removed from the instruction document.

– And verifiers stretched their capacity to work their mind around all the possible effects a tiny change would trigger. Almost like the butterfly effect. Their work was heroic and often beyond the call of duty. The lists of checks that were in place would make any connoisseur of this type of operation dizzy!

To add delight to the exercise, IEA and cApStAn were exchanging pdf documents in all scripts and directions: at the time there was no online system that could help us to deal with the instruments in one go (or at least to garner real-time information on the state of art). We were sentenced to deal with a flurry of pdf, rtf and/or Word files that did not look alike on all operating systems.

When looked at from the outside and with the distance of time, rather than insuperable challenges we see the overall quality and consistency achieved in spite of the mission impossible. Maybe it also takes a grain of folly, but good old stories deserve some. Right?

P.S. Of course, there were some lessons learned and there would be some changes to the approach used at the time. The major one, and actually the only one that we could be in charge of, would be to make two or three waves of checks rather than to attempt repeating the tsunami gently outlined above.

Paulína Koršňáková from IEA

About Dr Paulína Koršňáková

As Senior Research and Liaison Adviser for IEA, Dr Paulína Koršňáková is responsible for supporting and developing existing memberships and institutional partnerships within IEA’s mission of researching education to improve learning. Prior to her various assignments at IEA since 2010, she led implementation and reporting for various international large-scale assessments at a national level in the Slovak Republic. Dr Koršňáková is an experienced teacher, lecturer and researcher and represents Slovakia on the European Educational Research Association (EERA) Council.

Paula on LinkedIn

Paula on Twitter @Paula_IEA

Paulína Koršňáková IEA

About IEA

The International Association for the Evaluation of Educational Achievement (IEA) is an independent, international cooperative of national research institutions and governmental research agencies. It conducts large-scale comparative studies of educational achievement and other aspects of education, with the aim of gaining in-depth understanding of the effects of policies and practices within and across systems of education.

IEA studies measure student achievement in subjects such as mathematics and science (TIMSS), reading (PIRLS), civic and citizenship education (ICCS), and computer and information literacy (ICILS). IEA provides external knowledge to the European Commission, as well as research services and expertise for partner organizations including the OECD and the World Bank. UNESCO recognizes IEA studies as providing a solid evidence base for researchers, educators and policymakers interested in monitoring progress towards Sustainable Development Goals.

IEA on Twitter @iea_education