How to respond to the challenges faced by evidence-based policies?

Paper Title: Is it time to give up on evidence-based policy? Four answers

Responsible researcher: Viviane Pires Ribeiro

Authors: Richard D French

Intervention Location: Global

Sample Size: 400 publications

Major theme: Economic Policy and Governance

Variable of Main Interest: Evidence-based policies

Type of Intervention: Literature review on the political potential of evidence-based Policies

Methodology: Qualitative systematic review of the literature

Given that there is a contradiction or ambiguity in the local and global defense of Evidence-Based Policies (EBP), French (2019) carries out a qualitative systematic review of the literature on the political potential of EBP. The study concludes that, to respond to the challenges facing EBPs, researchers must develop a more realistic understanding of the working environment in which ministers and senior officials operate, reject naive but prevailing assumptions about the level of analytical rationality in the government and recognize that sustained engagement with policymakers may not be compatible with academic career advancement.

Assessment Context

Is the evidence-based policy movement a sign that major improvements in policymaking are there for the taking, should governments and researchers make the necessary efforts? Are governments currently neglecting evidence that would provide valuable support for significantly improved policies? No one doubts that where research usefully addresses public problems, it should be exploited to the fullest extent possible. Not everyone agrees (a) that research 'usefully addresses' public problems, (b) that 'the greatest extent practicable' exceeds the current extent to which research is exploited in policymaking, and therefore (c) that efforts should be dedicated to obtaining a better supply of better used evidence?

French (2019) attempts to answer these questions through a qualitative systematic review of the literature on the political potential of EBP. Thus, the author defines evidence as the product of research: organized knowledge produced in accordance with the standards of relevant academic disciplines; and defines policy as the position or approach adopted by public authorities – governments, agencies, school boards, military, police – regarding problems or opportunities that are perceived to affect the public welfare.

Intervention Details

French (2019) performs a qualitative systematic review of the literature on the political potential of evidence-based policies. In this sense, the author emphasizes that the literature searching for qualitative systematic reviews must present the following characteristics:

a) Identify the main “schools of thought” in a given area, being alert to the identification of variants, minority and dissident views.

b) Research within a wide range of disciplines in order to bring different points of view to bear on the topic in question.

c) Use complementary electronic and manual search techniques to ensure that materials are not lost due to indexing inadequacies or selective coverage of databases.

However, it is important to note that, just as scientific articles do not provide an account of the research process, but rather of its results, so too do policy accounts provide any sense of the process of arriving at that policy, with its many reversals, 'irrationalities' and contingencies. The political rationales that accompany policy announcements should never be confused with accurate accounts of underlying processes or motivations. The point is that whatever the “educated” layman decides to assume about public policymaking, there is no substitute for reading the literature on the subject, also known as “the evidence.” Much of this literature argues the partiality or unfeasibility of the rationalist model cherished by many EBP proponents. Likewise, it has been difficult to identify research successes in resolving public policy challenges.

The potential of EBP is therefore contested and controversial. Thus, French (2019) describes four major schools of thought about EBP – the Reinforce , Reform , Reinvent and Reject .

Methodology Details

The PAIS bibliographic database was consulted for books and peer-reviewed academic articles with the term “evidence-based policy” in their titles. This generated 132 references, which were complemented by the bibliography of policy-relevant references. Articles that addressed issues using the expression as an indicator of legitimacy, but that did not explicitly address the practice of EBP, were discarded from the review. This left several dozen potential references, which were read and from which manual note searches yielded several more relevant references. A total of nearly 400 relevant books, book chapters, conference papers, and articles from a variety of disciplinary traditions were ultimately identified and reviewed by French (2019).

Results

Based on a systematic analysis of almost 400 publications, French (2019) identifies four contrasting perspectives on evidence-based policy. A first school of thought advocates strengthening demands that governments pay more attention to research. A second perspective advocates reforming relationships between researchers and policymakers. A third emphasizes the need to reinvent the formal procedures that govern the generation and use of evidence. The fourth rejects the possibility that research can simultaneously meet disciplinary standards and meaningfully meet the needs of policymakers.

There is a contradiction or ambiguity in the local and global defense of EBP. It seems that most of this defense betrays a conviction of the obvious merits of the idea, such that it itself is innocent of much evidence. For the scientifically trained and for social scientists who take their “science” seriously, the idea that policy can benefit significantly from much greater use of research results appears to be an idea that sells itself. There appears to be no need to read the literature on EBP or public policy making.

Sir Peter Gluckman (2017), the doyen of global scientific consultancy, repeatedly warns his audience that scientists must beware of arrogance. It is notable how often similar themes – modesty, humility – emerge in studies of EBP and related issues. The review found more than a dozen such warnings.

This situation – a wide variety of views on the viability and imminence of EBP – is unlikely to change anytime soon. As long as the socialization associated with doctoral work in the empirical disciplines remains as it is, there will always be sympathy for rationalist assumptions and bewilderment that they no longer seem to have application to the world of practice.

Those at the Reform and Reinvent who have devoted time and significant effort to EBP mostly share the conclusion that 'The search for evidence-informed policy and practice will be a long and arduous journey', and the latter school could even endorse the statement of Pawson, despite his own commitment to EBP, 'There is no evidence-based policy. Evidence is the weak point of the political world', while elsewhere in the academic and foundation world, in the Reinforce , optimism about EBP remains.

Public Policy Lessons

French (2019) argues that those who wish to address the challenges facing evidence-based policy in a fully informed way need to (i) develop a realistic understanding of the working environment of ministers and senior civil servants, i.e. one that goes beyond the ritual lamentation over their inexplicable failure to respond to research and expertise; (ii) address the naive decision theory, grossly overestimating analytical rationality – widely spread among academics, researchers and editors – through the cognitive psychology of decision making; (iii) given the main conclusion in the EBP literature that “direct and sustained relationships between researchers and policymakers are the ideal method for promoting the use of research in policymaking”, assess the extent to which this is compatible with the 20th century university career requirements and how their prospects can be enhanced; and finally, in light of the above, (iv) compare the surprising variety of prescriptions from the Reform and Reinvent and attempt to reach a realistic consensus.

References

FRENCH, Richard D. Is it time to give up on evidence-based policy? Four answers. Policy & Politics , vol. 47, no. 1, p. 151-168, 2019.