“We’re doomed” she said.
“She” is a dean at another university. Her name might easily be added to those of a long list of prognosticators predicting doom and gloom for universities. Reasons for their pessimism come readily to mind: funding cutbacks, large—sometimes enormous—pension deficits, costs escalating more rapidly than revenues, demographic trends reducing the numbers of students available to enroll, increasing competition from the colleges, and technological developments that are challenging traditional modes of course delivery.
But the dean in question wasn’t commenting on these issues, at least not directly. Her angst was a response to a particular way of dealing with them—to her university’s decision to undertake a program prioritization process commonly known by its acronym, PPP.
The most influential advocate of this approach is Robert C. Dickeson. Despite its dry title, his book, Prioritizing Academic Programs and Services, has become something of a bestseller. An impressive number of universities in the United States and now in Canada are adopting it as a how-to manual for PPP. The heart of Dickeson’s process is a review of all programs that aims to rank them in a way that enables a university to decide what programs it should cut, consolidate, or complement as it attempts to put its financial house in order.
Variants of PPP all share the notion that universities should review and rank their programs– academic and non-academic—in order to pick and choose among them. In Ontario at least, the decision to undertake PPP has sometimes been touted as proof that that an institution is willing to face up to the issues of the day.
We have a different view of PPP. Not because we think that difficult decisions about the cutting and consolidation of university programs can be avoided. It would be Pollyannaish to think that the gathering storm will soon abate and allow universities to return to heady days without the current budgetary pressures. In such a context, PPP can be useful, primarily because it can provide a university with important data that can inform the difficult decisions it must make. That said, PPP is frequently carried out in a way that damages morale (sometimes severely), creates conflict, and makes it more, not less, difficult to make key decisions.
At a time when PPP is emerging as the latest in a series of management movements that have captivated a generation of university leaders—Total Quality Management (TQM), Continuous Improvement, Lean Management, and so on—we want to share some doubts about the move in this direction. In doing so, we hope to make the headlong rush to PPP more circumspect.
One of the problems with PPP is its cost. We are not thinking of the sums that universities spend on the professional consultants and assistants who direct and carry out their PPP analyses (though these sums can be considerable). Our concern is the extent to which PPP reduces a university’s ability to focus on its central responsibilities of teaching and research. PPP is a very complex process and the work that it requires—establishing a PPP methodology and applying it to all the programs (hundreds in a small university, thousands in a large one)—requires the attention of a small army of people who must devote an inordinate number of hours to this task. Especially in a time of constrained resources, one must ask whether this is the best way to use the limited resources a university has at its disposal.
To fully appreciate what PPP requires, one must consider some of the complications that make the application of PPP difficult and time consuming.
To take only a few examples, Dickeson’s process suggests that one assess individual programs by measuring “the quality of program outcomes,” “the productivity of the program,” “the overall essentiality of the program,” and its potential as uncovered by an “opportunity analysis.” As long as one speaks in generalities, this sounds fine. Who could deny that a university should support “quality” programs which offer a “productive” route to positive “opportunities”?
The problem is that it is not at all clear how general criteria of this sort should be translated into useful criteria that can be used to assess a particular university with a particular history in a particular locale. Inevitably, such attempts to translate from the general to the specific, in a way that makes sense for a particular university, generate debate which is frequently contentious. There is no simple and obvious way to turn generalities about quality, productivity, and opportunity into the concrete, easily quantifiable metrics which are needed to create what PPP is supposed to generate: measurements of the success of different programs that allow meaningful comparisons and accurate rankings.
In view of this, the attempt to apply PPP is not a simple process. To begin with, it requires an enormous effort to ensure that the criteria it applies are relevant to the local situation. Creating the appropriate metrics becomes a sensitive and consuming task for deans, department heads, senior administrators, managers, faculty members, Senators, and members of the Board of Governors who are directly involved in the PPP process.
And this is only one step. Assuming metrics can be established, the intricacies of applying them must then be addressed. In judging a particular program, especially in a time of budget crisis, costs are an obvious consideration. However, costs cannot be fairly considered apart from the revenues a program generates. So, PPP requires an analysis of the costs and revenues associated with every program. This is a useful exercise, but a complex one which raises many questions about the way that costs and revenues are measured. To take one example: how does one assign the revenues generated from individual students? Should they be credited to the faculty or department where the students have their majors, or to the faculties who host the courses that they take, or in what proportion to each?
Assuming that these kinds of details can be worked through, a university employing PPP must establish a bureaucracy to secure the necessary data, apply the different metrics to programs, calculate PPP scores, and rank programs. This requires the coordinated effort of many—an effort that inevitably diverts an institution’s attention away from its core missions of teaching, research, and service.
And this is only the beginning of the costs that come with PPP. The most significant is the disruptive nature of the PPP exercise itself, which cannot be measured in dollars or time spent. The disruptive consequences are rooted in the attitudes of programs and departments, which see PPP for what it really is—a process designed to decide whether they are winners or losers in an ongoing battle for shrinking resources. With such high stakes and uncertain results (if the results were obvious ahead of time, there would be no reason for the exercise), PPP is an unsettling process for many programs and the people who work within them.
In the world of international business, one might compare the way PPP operates to the ‘forced ranking’ appraisal system once popularized by General Electric. It appraised employees; placed them into categories like ‘top 20 per cent’, ‘middle 70 per cent’, etc.; and then terminated the ‘bottom 10 per cent.’ The severity of the process has earned the forced ranking system the label ‘rank and yank.’ It is still in use in some corporations, but controversially. John Hollon of TLNT.com, described it as “an arbitrary, formula-heavy performance system that’s obsessed with cutting people down instead of helping to build them up.”
It is not hard to imagine the disruptive effects of rank and yank on the morale and well-being of individual employees. PPP is rank and yank at the program level. It exacerbates the morale issues in a university by pitting programs against one another. Some departments win because others lose. The whole point is to separate the winners from the losers, to separate the wheat from the chaff. In the process, PPP encourages a view of that tends to favour cutting programs as a way to deal with budget problems instead of considering other possibilities like reconceptualization, the identification of solvable problems, and re-organization.
There are situations outside of universities where rankings for scarce resources cannot be avoided. Triage is an obvious example. In a situation in which one does not have as many livers as patients in need of a liver transplant, one must find a way to rank the candidates to determine the greatest need. In situations in which this cannot be avoided, a standard must be used. For PPP, the problem is the perception of the metrics used, which are much more controversial and usually applied without any broad consensus on the question of whether they are appropriate or not.
Consider the metrics for research productivity. Many contrary metrics (focusing on publications, citations, ‘impact,’ ranked journal articles, etc.) have been proposed. The size of research grants is one simple measure popular with universities and governments. Certainly, this is a good indicator of success in research competitions. It has the additional advantage of being a relatively simple measure that is easily calculated—the dollars awarded are easy to count and compare. The simplicity of this approach, however, is a problem as much as it is an asset. It is too simple to equate research quality with research income. For one thing, the latter is a function, not merely of quality, but also of the cost of the research in question. Sampling the Arctic seas to catalogue the ways in which different species react to global warming is an expensive endeavour which requires large research grants. Solving a conundrum in pure mathematics may require little funding. This is not because the latter requires less talent or has less long-term impact, but because it requires little infrastructure, needing only the time required to allow a spectacular thinker to think through the problem.
And this is only the start of the questions that may be raised about the use of research grants as a measure of research productivity. Are all research dollars the same? Should a dollar from industry count as much as a dollar from a tri-council grant? What about the bulk of university-based research—which is funded, not by grants, but by the time tenured professors devote to research activities as part (typically understood to be 40 per cent) of their workload? Isn’t it possible that research projects in this category might have a transformational, even revolutionary, impact on their disciplines (and the world)?
What does a simple tallying of research dollars say about the relative value of pure and applied research?
How does one apply such a metric to programs dedicated to creative writing, the dramatic arts, theology, or history? The real impact of research is not seen in the short term but in the long term, making attempts to assess impact—an attempt to predict the future—an inherently hazardous endeavour. Jorge E. Hirsch, the inventor of the h-factor, has conceded in a December 2009 issue of The Chronicle of Higher Education that “no bibliometric measure will do a good job picking up every novel, nonmainstream research until it becomes mainstream.”
In any university, one can find influential professors whose work has not been supported by research grants. Between different kinds of programs, grant amounts vary enormously. Using these grants as a measure of research success is, in most universities, a constant source of debate and controversy. This is a familiar situation in university settings around the world. It is easier for administrations and administrators to recognize these complexities outside of a rigid process like PPP, which favours fixed comparators and a one size fits all set of metrics.
Despite these many questions, research productivity is one of the simplest issues raised by attempts to assess the value and success of a university’s programs. Much more complex issues are raised by other measures that are typically used to judge a program’s value—costs and revenues generated, student success, ‘essentiality,’ the opportunities for future development, contribution to the identity and success of an institution, and so on.
The challenges inherent in program metrics are compounded by the different directions in which the different metrics often point. Research productivity, however it is measured, may say one thing about a program; cost effectiveness, another; and the quality of undergraduate teaching yet another. When all is said and done, this may be inevitable; universities are eclectic places in which different programs have different goals, best measured by different constellations of metrics.
In light of all these complexities, and with so much riding on the outcome of PPP, it should come as no surprise that programs and departments tend to favour those metrics that place them higher in the rankings. This is a problem given that frameworks are usually established by committees in which influential committee members, or groups of committee members, may (even unintentionally) emphasize measures that favour the kinds of programs with which they are involved. As with all formulaic approaches to quality, clever departments will find ways of gaming the framework, so that their scores improve without substantive improvements in the quality of their programs. One worries that new initiatives that require radically new ways of thinking about teaching and research—just the kind of initiatives an institution may need at times of great flux and upheaval—are likely to be disadvantaged by metrics that are established by PPP committees composed of representatives from the traditional silos.
What PPP exercises are in need of is a set of metrics that can measure the value of a university’s programs. There may be cases where this is possible, but in most cases it is difficult to compare different kinds of programs, especially as most PPP exercises rank academic and non-academic programs together. This raises thorny issues. Are the activities of a university’s publishing house more or less valuable than intercollegiate athletics? How can one pick between an academic program with a long and celebrated tradition and one which aims to embrace the future? Is a large program with many students more or less valuable than a small program with elite admissions? Should an institution favour programs relevant to the local community over those that boast international renown?
The answer to such questions is that there is no answer. There is no way to rank, in an uncontroversial and unproblematic way, the myriad activities and programs contained within a university. In the case of academic programs, professors in each and every department are more than capable to justify what they do when asked.
In the worst cases, this is the reason why PPP tends to produce bitter suspicion, anger, and frustration rather than consensus. Rather than solve the thorny problems that it is supposed to address, all of the ambiguities and complications, and the insecurity and fear generated by PPP can inspire widespread opposition to change, making crucial, unavoidable decisions more, rather than less, difficult to make. One does not make change easier by whipping a university into the kind of uproar that can break out as different programs fight with one another over their places in a PPP ranking.
Those who advocate for PPP may concede that it is difficult, but argue that universities have no choice but to pursue it resolutely to face the challenges that confront them today. Such arguments are a false dilemma. We are not in a situation in which one must do PPP or nothing. Thinking otherwise equivocates the need to make difficult decisions with the use of one particular methodology for doing so.
It must be said that some of the data PPP collects can be useful. In making difficult decisions about resources, it is important to know the cost of a program. The quality of research and teaching in a program is relevant in an attempt to assess it. But a university with a transparent budget does not need a PPP exercise to discover the costs and revenues associated with particular programs. A PPP ranking exercise is not needed to provide such data. In the case of academic programs, systems of review established at the provincial level already mandate the rigorous review of all programs, usually involving the input of external reviewers. A PPP exercise is not needed as an additional form of quality assurance.
We wonder whether something is amiss if heads, deans, directors and others do not already know where the weak programs are located in a university. That knowledge, combined with broader institutional strategic objectives, is the best basis for decisions about what to cut, consolidate, and complement. When more information is required to make decisions, this can be accomplished without the bureaucracy, the contention, and the cost that accompanies PPP. Our own experience suggests that changes are best accomplished in a nuanced way that attempts to address the particular rather than general issues for institutions, students, faculty and staff.
In the case of academic departments, it is hard to see what is to be gained by putting a department on trial when it has a superior research record, first-class teaching, bountiful enrolment, and a solid bottom line. In the case of departments on the other end of the spectrum, what is needed is not a ranking exercise (and the embarrassment, uncertainty, and frustration that accompanies it) but focused intervention to decide the best way forward.
Despite our skepticism, we cannot deny that the fiscal challenges at universities these days have made PPP almost de rigueur. Certainly the process has a great deal of allure for administrators and their universities. Often it is announced with much fanfare. We do not think that everything that PPP does is wrongheaded—honest and transparent data will always be the best prerequisite for good decision making. But we do not think a PPP exercise is needed to determine that a program is strong and deserving support, or weak and in need of cuts, reorganization, or some form of consolidation.
Instead of helping universities sort through the difficult decisions, many attempts at PPP seem to produce conflict and resistance to more productive ways of dealing with the difficult questions that universities simply cannot avoid. It makes it harder, not easier, to take on the serious challenges we face. AM
Leo Groarke is Provost and Vice-President Academic at the University of Windsor. Beverley Hamilton is the Research and Projects Officer in the Provost’s Office at the University of Windsor.
