What is the role of evidence in policymaking?

Responsible researcher: Viviane Pires Ribeiro

Paper title: When evidence is not taken for granted: the use and perception of “evidence” in the Czech Republic ministries

Authors: Arnošt Veselý, František Ochrana, Martin Nekola

Intervention Location: Czech Republic

Sample Size: 1351 questionnaires

Major theme: Economic Policy and Governance

Variable of Main Interest: Evidence

Type of Intervention : Analysis of the role of evidence in policy formulation

Methodology: Large-N survey and in-depth interviews

The role of evidence in policymaking is one of the most researched topics in public policy and public administration. Yet surprisingly little research has been done on how public officials actually use evidence in everyday life practice. Veselý, Ochrana, Nekola (2018) addresses this gap. Using a large-N survey of Czech ministerial officials and in-depth interviews with them, the authors explore what is understood under the term "evidence", what kind of evidence is used and preferred by civil servants and what.

Assessment Context

The rise of “evidence-based policy” (EBP) discourse is one of the most visible trends in public administration and public policy in the last decade. Although EBP is a well-recognized movement, its definitions vary from a very narrow view (a particular methodology for producing a specific form of evidence), to a broader and more comprehensive view of what it represents. Although many governments and professionals have been very enthusiastic, EBP has also been criticized and challenged for a number of reasons, for example, for assuming that research evidence can provide objective answers – and ultimately resolve inherently political questions.

Even with all these criticisms, many initiatives and projects on “bridging the gap between policy and research” have been carried out especially in Anglo-Saxon countries. In these countries, the EBP discourse is deeply rooted in public administration practices. Consequently, in countries that have been heavily influenced by this discourse, it seems difficult to challenge the basic claim that evidence should be used in policymaking. The use of evidence – whatever that term means – is taken for granted.

The Czech language has the word “evidence”, but with a different meaning than English. In Czech, evidence is derived from the verb "evidovat", which means "to record". Thus, the Czech word “Evidence” is generally associated with other types of activities, such as Evidence obyvatel (population registry) or elektronická Evidence tržeb (electronic sales record). Thus, the word evidence is generally understood as a “record”, that is, centrally collected and stored information.

Few people in public administration and even in academia are familiar with the concept of EBP. Occasionally, however, the concept is mentioned. Given the fact that evidence in the Czech language is associated with “to record” (which is quite different from its original meaning in English), PBE has often been translated as politika založená na důkazech. This literally means “evidence-based policy.” This is, of course, a rather narrow interpretation of evidence, as “proof” only includes evidence that is unquestionable and gives definitive answers. To avoid this terminological confusion, some authors use the term “poznatky” instead of “evidence”, as is the case of Veselý, Ochrana, Nekola (2018). This concept is the broadest and most neutral of all related concepts. It can be translated as “knowledge”, or more precisely as knowledge created through the process of cognition. The concept of poznatky has a slight connotation with research (research knowledge), but is not necessarily associated with research.

Intervention Details

The study carried out by Veselý, Ochrana, Nekola (2018) explores the understanding and use of evidence in the ministries of the Czech Republic as central institutions of public administration. This country represents an interesting case in the study of the use of evidence in policy formulation. The production and use of policy-relevant information has a long tradition in the Czech Republic, and Czech social science has always had a strong practical orientation. Furthermore, the demand for science-based policy-related knowledge has led to the establishment of different types of research institutions that serve directly ministries. Although both their number and impact declined after 1989, the emphasis on relevant and practical social science prevailed. At the same time, however, the idea and discourse of EBP never seriously entered the public administration and policymaking debate. Few people in public administration are familiar with the concept of EBP. There have been no initiatives or projects to incorporate more evidence into policymaking and decision-making in public administration, and the use of research evidence in policymaking appears to be quite limited. Furthermore, the basic concept of “proof” is hardly translatable into the Czech language.

Thus, Veselý, Ochrana, Nekola (2018) ask some traditional questions about the use of evidence in politics in the context of ministries in the Czech Republic. Specifically, we asked the following questions: What is considered evidence by public officials? What type of evidence is used? What type of evidence is considered important? When and how is evidence used in the policy-making process? In doing so, the authors draw on four strands of theories that focus on different aspects of the use of evidence. Discussing the extent to which these theories, most of which were developed in a context with a strong EBP discourse, can be applied in a context where EBP is almost unknown.

Methodology Details

To answer the study questions, Veselý, Ochrana, Nekola (2018) used quantitative and qualitative data. As for quantitative data, the team conducted a large-N survey of policy bureaucrats in ministries in the Czech Republic between April and July 2013. Step by step, 11 ministries agreed to participate in the survey. In seven ministries, data was collected through face-to-face interviews: interviewers met with interviewees, asked a series of pre-defined standard questions, and recorded their responses on a paper form (CAPI) or in a computer application ( PAPI). For two ministries that preferred to participate without the involvement of interviewers, data were collected by administration in the form of online questionnaires (CAWI). In one ministry, a combination of CAPI and CAWI was implemented. Respondents were randomly selected from each ministerial sample frame. After several waves of random sampling, all individuals in each sampling frame were invited to participate. Thus, what was supposed to be a random sampling ended up being a census. A total of 1351 completed questionnaires were obtained and the response rate was 29.4%.

Qualitative data comes from in-depth interviews conducted by the authors who specifically focused on the use of evidence. These interviews were carried out from April 2016 to May 2017. In total, there were 23 interviewees from different Czech ministries. Four researchers, members of the research team, conducted the interviews following the common interview guide. Interview topics concerned a description of the interviewee's strategic work, description of their use of scientific or other knowledge, their experience with the use and status of evidence in the broader context of their ministry and the state administration. The semi-structured interviews with experts lasted around an hour and a half or two hours, were recorded and field notes were prepared. The records were transcribed in full, and the interviews were coded and analyzed using thematic analysis focusing on the patterns identified regarding the use of knowledge in various contexts and phases of the political process.

Results

The use of evidence in the Czech Republic differs in many ways from what is reported in countries with a strong EBP discourse. Public officials report no commitment to using evidence in the sense of “what works” (although they are not opposed to it). More precisely: they do not demonstrate commitment to scientific evidence, but to the evidence of “administrative data”. Generally speaking, they are not interested in “evidence” but in “evidence” that can be used to legitimize political objectives and that supports negotiated consent on policy.

Veselý, Ochrana, Nekola (2018) argue that it is impossible to determine exactly to what extent these differences are caused by different governance in the Czech Republic and to what extent they are caused by the low impact of the EBP movement in the country. It is likely that these two are linked to each other. All interviewees refer to the relative political instability and frequent changes in the general scenario of public policies, which has two consequences for their work. First, they perceive time pressure, meaning they need to act very quickly. Second, they report the (repeated) experience of policies and strategies not being realized because of personal changes in ministry leadership. Time pressure leads public servants to find and prepare documents that are easy to understand. Respondents who worked on a policy document that was left unfinished, abandoned or even removed from the Government's decision-making process because of the change of position of the minister or his deputy report lower motivation for any complex work and data analysis. It is then combined with a low internal pressure (in a given ministry) to use evidence in the policy process maintained only slowly changing the overall culture of the ministries.

Many points in common were found with results from other countries. Most notably, public officials use different types of evidence and apply their own criteria about what counts as evidence and what is “useful evidence.” That is, public officials use very diverse sources of evidence, but evidence produced by other officials (or for officials) is used more intensively. In general, all four strands of theories seem to be useful in generating hypotheses about the use of evidence also in the context without EBP.

Public Policy Lessons

Although public servants and academics do not differ so profoundly in their training, they do differ in their speech and vocabulary, and also in terms of the standards by which they judge the merit of evidence. Thus, the study carried out by Veselý, Ochrana, Nekola (2018) suggests an important role of experts or “knowledge intermediaries” who translate the world of science into the world of practice. However, more detailed research is needed to explore exactly who knowledge brokers are and how they work.

Furthermore, the analysis also suggests that the recent movement towards adding context into the analysis of the use of evidence is very promising. The use and understanding of evidence is strongly influenced by the everyday practices of public servants, as well as their immediate social environment. The use of evidence cannot be separated from the entire political process in which it is inserted. Public servants want research that “helps them in their work”. And what should help them is influenced by the stage of the policy and their expectations. Combined with the recent emphasis on type of knowledge and evidence, this could be a very promising avenue for future research.

References

VESELÝ, Arnošt; OCHRANA, František; NEKOLA, Martin. When evidence is not taken for granted: the use and perception of “evidence” in the Czech Republic ministries. Network of Institutes and Schools of Public Administration in Central and Eastern Europe. The NISPAcee Journal of Public Administration and Policy , vol. 11, no. 2, p. 219-234, 2018.