Richardson, R. and Martinez, M., Policy and Performance in American Higher Education: An Examination of Cases Across State Systems (Baltimore, Maryland: The John Hopkins University Press, 2009).
In ministry boardrooms and university classrooms around the world, policy makers and higher education researchers are grappling with questions trying to understand better the characteristics of a successful higher education system. Experienced researchers in the field of higher education governance, Richardson and Martinez have attempted to answer these questions by reviewing selected US state policy landscapes. They view a successful system as having high levels of preparation, participation, and completion rates in colleges and universities, and they aim to determine what government activities support these priorities. This book goes beyond examining what policies work and seeks to broaden the discussion by trying to extrapolate certain ‘rules’ that shape successful higher education strategies. By comparatively analyzing five American higher education systems, the authors attempt to divine what elements of governance are correlated with performance results.
This is a unique approach to higher education policy, as governance research is commonly conducted as case studies of specific policies in specific jurisdictions. A comparative analysis of systems of higher education is a challenging task, to say the least. As any Canadian policy maker knows, each provincial higher education landscape is unique and, therefore, strategies and policies are difficult to isolate for comparison. Much beyond apples and oranges, comparing jurisdictions is more like comparing weeping willows to pine trees. While there are common elements – a trunk, branches and some form of foliage — the landscape, size, and seeds are very different. Compounding these inherent challenges, the authors also acknowledge, there are only some features that policy makers and institutions can be held accountable for and that are within their power to change.
While this book focuses on American jurisdictions, the themes and topics that emerge are present in any higher education system, including those in Canadian jurisdictions. In this research, each of the chosen cases (New Mexico, California, South Dakota, New York, and New Jersey) has vastly different population sizes, demographics, legislative involvement, government bodies, and other actors, let alone history, system design or institutional differentiation. However, each state is dealing with similar issues that are dominating global higher education conversations: access, funding, accountability, privatization, centralization/decentralization, data collection, and transfer and mobility. These issues are also at the heart of policy discussions across Canada.
To simplify the comparison of these very different systems, Richardson and Martinez develop ‘rules in use’. The ‘rules in use’ are described as written or unwritten constraints that shape interaction by the ‘shifting array of actors’, market forces, regulation, and institutional autonomy that influence policy decisions. If able to determine the ‘rules in use’, the chaos of five varied systems, they say, should be able to be assessed systematically.
Providing structure to the research, the first two chapters of the book outline the approach to public policy, the guiding conceptual framework, the ‘action areas’ and ‘rules in use’. Chapter One clearly outlines the premise and position of the book. Acknowledging they are entering into uncharted territory in attempting comparative case studies in public policy research, the authors precisely lay out their framework, lead readers through their methodology and highlight how it serves as an appropriate means to evaluate the complex world of higher education policy. For researchers this background is a useful example of a clearly defined research methodology that can be applied to any jurisdiction for any public policy question.
Richardson and Martinez chose to use a modified model of Ostrom’s ‘Institutional Analysis and Development’ (1999) as the framework to assess policy decision-making. This model allows for the ‘multiple levels of analysis’, which support the necessary examination of various actors and bodies of influences while determining and maintaining focused attention on key variables. The authors then proceed to determine the common ‘action areas’ that demonstrate how the ‘Rules’ are operationalized in decision making. ‘Action areas’ include: planning, program review and approval, information, academic preparation, student financial assistance, tuition and operational support, capital support, and economic development initiatives.
Chapter Two defines the ‘rules in use’ and links them to performance outcomes and state ‘effort’. The authors identify six major categories as ‘rules in use’ that provide key information about the system: system design, information, access and achievement, fiscal policies, and research and development. By assessing the states on each category and associated subcategories, the authors briefly identify how each state is doing in broad relation to each other. Issues of state ‘effort’ are then presented, where ‘effort’ considered direct funding to the system, either per capita or per $1000 of personal income, and can be either low or high.
Chapter Two also determines the performance outcomes that are used to measure the success of the systems. They include indicators of preparation, participation and completion and various sub-indicators associated with each. Preparation performance is determined by five indicators: high school completion rates, K-12 student achievement on national math assessments, 8th grade proficiency at math and reading, and percentage of 7th-12th graders taught by teachers with a major in the subject. Participation performance is assessed by the ‘chance for college’ by age 19, participation rate by low-income students, age 18-24 enrollment rate, and age 25-49 part-time enrolment rate. Completion is determined by retention rates of first year students into the following year for both second- and fourth- year programmes, graduation rates (three years for associate degree students and six years for bachelor degree students), and by the number of bachelor degrees awarded per 100 students.
It is surprising that the authors chose not to include ‘quality’ as an indicator of performance outcomes. Recent Canadian provincial strategic documents and action plans show that there is broad agreement on the importance of categories of participation, accountability, system design – but they also include quality (Rae, 2005; Plant, 2007; Alberta Advanced Education 2006; Government of New Brunswick, 2008; Government of Newfoundland and Labrador, 2005). Thus, while each province addresses issues, as presented by Richardson and Martin, each also acknowledges that there can be a trade-off between increasing access and providing quality in expanding systems and have addressed this by targeting ‘quality’.
Admittedly, quality is challenging define, and perhaps the authors were unable to determine appropriate indicators. However, it was not explored as an option. Some literature suggests that certain indicators can be used as a proxy for quality. For example, student satisfaction or engagement surveys capture issues such as ‘overall experience’. The value of these surveys has been accepted by the Ontario government and institutions, and all Ontario universities currently implement the National Survey on Student Engagement (NSSE). Similarly, administrative data can provide information on class size or faculty-student ratio, which may be an indicator of quality.
In avoiding the issue of quality, the authors, it seems (or perhaps the American conversation), may be more focused on the question of input and output: ‘How to get bums in seats and out the door with a credential’ rather than ‘How do we provide the best education possible to the most students possible’. This latter question is at the forefront of Canadian policy decisions.
Chapters three to eight apply the well-designed structure of the ‘rules in use’ and subcategories to case studies in order to organize the array of actors, activities, strategies and policies. The individual case-study chapters begin by describing the context of each state, including the political, economic, and social features that have an impact on the higher education landscape. They then turn to examine two levels of power: the state government and the institutions. Each of these levels of policy making is discussed and reviewed, and the ‘rules in use’ are applied to each.
The depth and detail of the case studies are impressive. With only a few pages of text, the authors succinctly inform the reader of the vast landscape, primary actors, relationships between actors, formal and informal structures, and standing policies and strategies. There were certain elements that I was particularly drawn to that have particular relevance for the Canadian context, such as engaging new immigrants. Unquestionably, the authors could not explore in depth all issues, therefore one criticism is that stronger reference notes would have allowed readers to follow up specific areas of interest.
Because the information was organized as individual case studies, it was challenging to reflect comparatively on how each of the states differs within the ‘rules of use’. This made it difficult to reflect on how Ontario or other Canadian jurisdictions might compare. I held judgment on this matter, hoping that the conclusion would provide a strong summary of each of the state strategies employed within the ‘rules in use’ and how each was performing. Having devoted so much attention to the fine details of each chapter, I was looking forward to broad inferences and theoretical musings of these accomplished academics. Unfortunately, this was provided in a very short few pages that I felt could have been heartily expanded upon to help distill the essence of each ‘rule’.
The concluding chapter did provide brief statements on what ‘rules in use’ were associated with predictors of high performance. Within system design, high performing states generally have coordinating boards and incorporate private sector institutions in their planning. Leadership in the form of state defined goals, priorities, and accountability arrangements were strong predictors of undergraduate performance, while the use of incentives has a negative correlation. Access and achievement successes were associated with the use of need-based student aid policies, access programmes for disadvantaged groups, and the measurement of learning outcomes. Fiscal policies that supported the funding of private institutions were one of the strongest predictors of performance success. Indicators of research and development were not found to have a correlation with performance in preparation, participation, or completion.
One conclusion that came out clearly was that high ‘effort’ (or high funding) is not tied to higher levels of performance. Certain activities, primarily ones associated with high performance outcomes, suggest that successful strategies are not necessarily the most expensive. For example, state-wide coordination was highly correlated with success, but required low ‘effort’ from the state. This is strong evidence for jurisdictions wanting to alter or improve certain elements within their system to look at their higher education landscape rather than investing in piecemeal or ad hoc strategies.
Without explicit discussion from the authors, it appears states with ‘top-down’ governance structures were the ones most able to achieve performance success. For example, ‘statewide coordination and regulation’ and ‘use of steering and market forces’ had high correlation with success. On the other hand, having ‘self-governing institutes’ in the system was negatively correlated with performance success. Considering this, I return to my earlier thought that quality was not included as a measure of success. It is quite possible that indicators of quality would be more strongly correlated with a more autonomous university or a less market driven system. Overlooking quality as a measure may be a fatal flaw in this book as, arguably, its absence may skew the validity of the findings.
To apply Richardson and Martinez’s framework to the Ontario case, it seems our jurisdiction has many of the key ‘rules in use’ in place. There has been strong policy direction stemming from the Reaching Higher report (Ontario, 2005). This key document was developed in collaboration with stakeholders and recognized developing strategic priorities through the establishment of the Higher Education Quality Council of Ontario. Leadership through accountability was a predictor of success as a ‘rule in use’, and Ontario’s accountability framework has been strengthened through the multi-year accountability agreements (MYAA’s). The Reaching Higher plan also supported students through attention to underrepresented groups and student financial assistance that focuses on supporting low-income students through the Student Access Guarantee. Not all of the ‘rules in use’ are applicable to our jurisdiction. Ontario does not have a large private sector for higher education and, therefore, it isn’t possible to assess how well the sector is being incorporated into the system planning. As a snapshot however, this example shows that it is possible to apply roughly the ‘rules’ to a jurisdiction for comparison.
Overall, this ambitious book achieved its goal of comparatively analyzing state higher education policy and strategies to assess key ‘rules in use’ that support preparation, performance, and completion rates. The book was strong in two distinct manners. The information provided in the case studies was brief enough to provide a snapshot of the higher education policy environment in each state, yet rich in complexity and nuance to provide a full description. Second, as an exercise of determining what ‘rules of use’ support high performance, the authors successfully developed the framework and applied it to the cases to determine key elements of thriving higher education systems.
As a book, I recommend it for the bookshelf of any higher education governance researcher as a good example of how to develop and apply a comparative framework. For policy makers, this book is an encyclopedia of research-based policy recommendations and examples of how strategies are being implemented in other jurisdictions. Particularly in Ontario, as the 2005 Reaching Higher reaches its sunset phase, researchers and policy makers could benefit from understanding ‘rules’ that emerged from this book when planning the next phase of our higher education system.
Mary Catharine Lennon is a PhD candidate in Theory and Policy Studies in Higher Education at OISE/UT, and a research analyst at the Higher Education Quality Council of Ontario.
