The call for evidence-informed decisions makes strange bedfellows of researchers and politicians. Researchers and political decision-makers live in different worlds, respond to different norms, and speak different languages.
Appeals for evidence-informed decisions received momentum when, in a fin de siecle effort “to get better government—for a better Britain,” the Blair government embarked upon efforts to modernize government, calling, among other things, for “better use of evidence and research in policy making and better focus on policies that will deliver long-term goals.” These calls for evidence-informed decisions required a bridging of the chasm.
In Britain, these bridges were built and, by 2002, researchers and policy-makers were living beneath the same roof. The Blair cabinet office had become home to a Strategy Unit, designed to provide strategy and policy advice to the Prime Minister by placing “strong emphasis on analytical rigour and an evidence-based approach to developing strategy.” What kind of a relationship would politicians and researchers have and, most important, could such a relationship last, given the historically dismissive attitude toward research held by policy makers?
In his seminal and lengthy 1999 volume, Governing from the Centre: The Concentration of Power in Canadian Politics, Donald Savoie makes the case that, in the last century, governments in Canada, the United Kingdom, and the United States “…would move to strengthen their centre to promote greater policy and program coordination, to generate policy advice, and to promote better management practices in government operations.” It is revealing that the role of research is of such little consequence in the aforementioned central government processes that it absent from the book’s 27 page index that addresses topics from Aboriginal affairs to youth employment.
Ben Levin, a Canada Research Chair at the University of Toronto who served as Deputy Minister in both Ontario and Manitoba, tells a story that reveals the distance between the worlds occupied by politicians and researchers. Levin informed a Minister with whom he worked that a course of action under consideration was supported by research to which the Minister replied, “You know, Ben, I don’t believe that research votes in my constituency.”
Levin’s anecdote makes clear that the primary audience for the decisions made by politicians is the electorate—or, to be specific, the electorate upon which the politician depends for support. On the other hand, the primary audience for the work of researchers is other researchers in their field of study, and secondarily the agencies that fund their research and the institutions that employ them. The wider audience for the work of politicians is the various publics (constituents, stakeholders, clients, and/or the media) affected by the decisions that are made.
Sceptics doubt that is it possible to bridge the chasm that divides decision-makers and researchers. Although the worlds in which decision-makers and researchers live are both tightly rational, the evidence and methods that govern the public policy decision-making process are, as Levin’s anecdote makes clear, quite different for those governing research.
Evidence-based decision-making relies on systematic research that has been replicated over time and in different settings. Eminence-based decision-making relies on expert advice.
Although dismissive of research at that moment, the Minister, had he been asked, would have said that he pays attention to evidence. In this case the Minister’s reply to Levin points to one source of evidence politicians consult in arriving at a decision. The approval of one’s constituency is typically the paramount source of evidence, but it is not the sole source. Others include: media; lobbyists and advocates; personal, anecdotal experience; ideological and values commitments; and expert advice—what is sometimes referred to as eminence-based advice.
Stefan Wolter, Director of the Swiss Coordination Centre for Research in Education, a body co-funded by the federal and cantonal levels of government to ensure the harmonization of education research across Switzerland, makes the distinction between evidence-based and eminence-based decision-making. Evidence-based decision-making relies on systematic research that has been replicated over time and in different settings. Eminence-based decision-making relies on expert advice. The challenge of the latter is that it is impossible for the decision-makers to know what part of an expert’s advice, if any, was research-based. According to Wolter, eminence-based “knowledge” is personal judgement masquerading as evidence.
Like Wolter, Phil Davies, who who worked in the UK cabinet secretariat as deputy director of the Government Chief Social Researcher’s Office in the Prime Minister’s Strategy Unit contrasts evidence-informed approaches with opinion-based policy. He notes that “opinion-based policy, which relies heavily on either the selective use of evidence (e.g. on single studies irrespective of quality) or on the untested views of individuals or groups, [is] often inspired by ideological standpoints, prejudices, or speculative conjecture.”
Both decision-makers and researchers are risk averse. In fact, managing risk is a key concern among decision-makers because visible decision failures undermine their credibility and legitimacy, threatening the longevity of their authority—especially if they are elected officials. Neither the visibility nor the consequences of risks faced by researchers are as great as those facing decision-makers, especially political decision-makers.
Researchers manage risks—primarily threats to the validity of their research—but they are better able to manage the visibility of such risks since the primary medium for the communication of their research is typically blind, peerreviewed publications. Consequences are also more manageable for researchers. They are typically offered the opportunity to revise their work before it is disseminated. The primary medium of dissemination is publications of limited circulation. There are professional consequences of poorly conducted investigations for researchers, but they are—given the system of review—less likely to occur and typically less cataclysmic than the public humiliation or loss of position that decision-makers face.
Decision-makers will take risks, but they carefully calculate the likelihood of failure. My own experience provides an illustration of risk management from the point of view of researcher and decision-maker. I was an academic seconded to serve as Deputy Minister who had, among other things, responsibility for the implementation of a major technological change. The project faced challenges. For example, the large corporation with which government had contracted had fallen behind in its commitments and had not tackled the most difficult installations.
One of the Ministers to whom I was responsible thought it desirable that some of the work that the large corporation had been commissioned to perform should be redirected to small, inexperienced local suppliers. The action— something permitted under the terms of the government’s contract—would build capacity among local suppliers to do such work in the future, a form of local economic development. The Minister in question called a meeting late one afternoon to discuss his idea and to seek advice. I approached the situation from the standpoint that my training as a researcher afforded. Never known for my reticence, when advice was called for I recommended against the direction that the Minister favoured. I reasoned that— given the delays and the difficulty in completing the work faced by a large, experienced corporation and the disparity between it and the small, inexperienced, local suppliers—it was highly likely that government would not be able to meet its commitment to a timely roll-out within budget. The Minister was not persuaded, and decided in favour of his plan to provide the work to the local suppliers and develop the local economies of the communities in which the suppliers were located. As we departed the Minister’s office, we were reminded that, despite the change in plans the Minister had made, the timelines and budgetary restrictions were to be met.
As it turned out, the Minister’s reasoning was correct. The local suppliers used the government contract to borrow money to buy the material they needed and acquired the expertise to do the work. Because they were more familiar with the local settings in which the work was performed, they were able to complete the work faster than the larger corporation. The project was completed on time and under budget, and the government garnered considerable support, although not enough to secure re-election!
This experience and others like it point to another of the many differences between research and policy-oriented decision-making. After considering the evidence of importance to him, the Minister arrived at a conclusion that he hoped could be achieved easily and would produce wide benefit to the community. Attentive to nuance and circumstance, researchers want to qualify their conclusions.
The Minister was typical of decision-makers; they seek resolution of conflict and elimination of problems as rapidly as might be achieved. Researchers do not. The pace of research is designed to engender thoughtful and careful reflection, while the pace of decision-making is more pressing and resolution-oriented. Decision-makers also seek results, but they are less concerned with specific mechanism so long as the anticipated outcomes can be achieved; they seek answers and impact. Though they, too, may want their immediate investigation to provide a solution to the problem that animated it, in the long run researchers want their work to beget more. If not more problems, certainly more research.
It is not inevitable that research will improve decision-making. Research evidence will not inform decisions if the need for a decision is urgent or if the decision-maker has an established position on the matter. If the matter is one that has attracted significant visibility and is highly divisive, the decision-maker might call for research evidence as a way of deferring decision-making— but the strategy is unlikely to work if the process of gathering such evidence will itself keep the issue’s visibility high.
The payoffs for decision-makers are the benefits or improvements brought about by the decisions made. These include problem reduction or managing the conflicts or the visibility of the conflicts that the allocation of scarce values engenders. The payoffs for researchers include the recognition that their work receives from their peers and the translation of peer approval into support for further research, tenure and promotion.
Politicians walk a fine line between seeking recognition for good work and avoiding visibility for making decisions that will be contentious. Researchers, on the other hand, seek visibility for themselves and for their research—albeit among audiences where their visibility is more easily controlled. Notwithstanding the substantive contribution that research makes to the creation of new knowledge, research, like politics, is also a form of self-promotion.
While a marriage of research and decision-making is unlikely, long-term liaisons are more frequent than one-night stands. Decision-makers—especially those occupying public office—are increasingly attentive to what the research has to say, using research evidence—along with a variety of other forms of evidence—to inform the decisions they make. Unless we are prepared to substitute a technocracy for a democracy, we should not demand that public policy decisions be determined by research evidence alone. AM
Charles Ungerleider is on leave from the University of British Columbia where he is a professor of the sociology of education to work as Director of Research and Knowledge Mobilization with the Canadian Council on Learning. An earlier version of this article was presented to 2008 Congress of the Humanities and Social Sciences in Vancouver, British Columbia.
