The UK’s new metrics-based teaching evaluation framework is methodologically and politically flawed. What will this mean for the country’s universities and faculty?
In 2017, a new higher-education assessment system—known as the Teaching Excellence Framework (TEF) —was launched in the United Kingdom. Based heavily on metrics, the TEF seeks to “recognize and reward excellence in teaching and learning, and help inform prospective students’ choices for higher education.” However, the TEF is both methodologically and politically flawed. It is time for the UK’s faculty unions to put forward alternative approaches for valuing and supporting teaching in higher education.
It is time for the UK’s faculty unions to put forward alternative approaches for valuing and supporting teaching in higher education.
From REF to TEF
The UK has often led the way in developing performance-based assessment systems in higher education. For example, in the mid-1980s, it was the first country to bring in a nationwide research evaluation process, which became known as the Research Assessment Exercise (RAE). Thirty years later, the RAE has morphed into the Research Excellence Framework (REF). As with its predecessor, the REF is based on a peer review process but also uses key metrics on research income, citations, and a qualitative assessment of research impact. The new TEF is a conscious attempt to ape the language and logic of the REF, including the link between awards and additional fee income. But the methodology is different: the TEF relies largely on institutional metrics, combined with a short written submission from universities, which is then evaluated by an “expert panel” and individually benchmarked against the types of students who attend each institution.
The origins of the TEF lie in the UK Conservative Party’s 2015 manifesto commitment to “introduce a framework to recognize universities offering the highest teaching quality.” This proposal was based on a perception that official “accountability” mechanisms and commercial university rankings were too focused on research outcomes and were therefore of limited value to potential students in choosing where and what to study. Government ministers had also picked up on a widely shared view that university teaching lacked the same status as research (ironically, due largely to the REF) and so believed that the answer lay in a REF for teaching. The Conservative’s surprise electoral victory in 2015 meant that the policy had to be implemented and, following a year-long technical consultation exercise, the TEF came into being.
Key elements of the TEF
In essence, the TEF is an official process for measuring the undergraduate student experience in higher-education institutions. The TEF panel—comprising academics, students, and employers—considers evidence from a set of metrics using national data as well as a written statement submitted by the institution. The metrics cover retention rates, student satisfaction, and employment outcomes. These data are then benchmarked to take account of differences in students’ characteristics, entry qualifications, and subjects studied.
Therefore, unlike other rankings and evaluations, the TEF provides a judgment of relative, rather than absolute, performance through its data-benchmarking process. This means that elite, well-funded, “research-intensive” universities are not compared directly with newer “access-oriented” institutions on key indicators such as dropout rates. In some ways, this makes the TEF a fairer measure of performance than commercial university rankings are, but it also makes it more difficult to demonstrate student outcomes. Ironically, the new process also makes it much harder for the government to claim that the TEF will aid student choice, especially because the TEF is currently an institutional award that tells students nothing about subject-level provision.
Participation in the TEF is also a voluntary process, and consequences vary for the different nations of the UK. This reflects the increasingly different funding and regulatory systems within the country. For example, in England, participation in the TEF is linked to an ability to increase tuition fees, whereas in Scotland, which currently has no tuition fees for Scottish students, participation in the TEF is linked to potential reputational advantage. Given the absence of additional financial benefits, it is no surprise that the majority of Scottish universities opted out of the latest TEF.
In June, the TEF panel published the results of its first assessment. Using an Olympics-style classification system (gold, silver, and bronze), half of the participants were awarded silver, 26 per cent obtained a gold award, and 24 per cent received a bronze. In terms of institutional classifications, the results defy easy schematization. For example, a number of access-oriented universities achieved the highest award, while a few research-intensive universities were awarded a bronze. In fact, a disproportionate amount of press attention focused on the fact that the prestigious London School of Economics achieved the lowest grade, although the overall results were less disruptive of traditional university hierarchies than many predicted.
Faculty voices
Irrespective of the results, the UK’s University and College Union (UCU) has consistently opposed the TEF, both on methodological grounds and because of its potential impact on institutions, staff, and students. This opposition is based on a number of factors. First, the core metrics—student satisfaction expressed through the National Student Survey (NSS), retention rates, and graduate outcomes—are flawed for the purposes of assessing teaching quality. To a significant extent, these metrics are influenced by external factors such as social background, gender, and, in terms of jobs, the region in which the university or college is located. Above all, they are poor proxies for measuring teaching excellence; indeed, even the chair of the TEF panel admitted that this was the case for student satisfaction scores.
Rather than focusing on improving teaching practice per se, universities are likely to concentrate on targeting better survey results, higher completion rates, and graduate outcomes. That is the nature of metrics and quantitative measures when they end up becoming targets. But this, in turn, can have detrimental consequences for the composition of the student body. For example, some institutions are already talking about increasing their student-entry requirements and cutting student numbers on specific courses in a bid to reduce their dropout rates. Another serious concern is that universities may seek to improve their rating on graduate outcomes by altering their subject mix, such as moving away from creative arts courses, which score lower on short-term labour market outcomes.
As a faculty union, the UCU knows that the TEF has already been cited as a reason for job cuts by some universities, and we are concerned that other institutions may follow suit. There is also a legitimate concern that, alongside the REF, the TEF will lead to a further fragmentation of academic roles into teaching-only and research-only positions.
Despite the preoccupation with the choice of metrics, the most controversial part of the TEF is its link to tuition fee increases. England already has the highest public tuition fees in the industrialized world and the TEF allows institutions in England to increase these further (by the rate of inflation). The government’s ultimate objective is to allow for fee differentiation on the basis of TEF results, although these plans have been delayed until 2020 at the earliest. However, the extent to which the government will be able to deliver on this fee-differentiation agenda remains open to question. The Labour Party’s pledge to abolish tuition fees during the 2016 general election was instrumental in increasing its popularity amongst younger voters. As a result, fees have become an increasingly toxic issue for the Conservative Party.
Reactions from the sector
Because of the TEF’s role in increasing tuition fee levels, the National Union of Students (NUS), along with the UCU, has also opposed key aspects of the TEF. In an explicit attempt to undermine the process, it urged its local student unions to support a boycott of the NSS. The UCU supported this initiative and it had some effect at universities such as Manchester, Bristol, and Cambridge, where NSS returns dropped below the required threshold. Partly for this reason, the government has said that future TEFs will rely less heavily on NSS data than other metrics, although this decision also reflects pressure from the larger research-intensive universities, who tend to score poorly on student satisfaction.
Unfortunately, the response from university leaders has been largely one of self-interest. Although critical of aspects of the TEF, they were willing to go along with the system as a means of increasing institutional revenue (via higher fees). Those universities who struck gold were the first to pop open the champagne, while those who secured bronze were more likely to appeal the decision or dismiss the exercise as “meaningless.” These responses reflect one of the structural problems of the UK higher-education sector: namely, the division of the sector into competing groups who lobby on behalf of their own type of institution (e.g., research-intensive, business-focused, access-oriented, etc.), sometimes at the expense of the system as a whole. In many ways, both the TEF and the REF reflect and reinforce these divisions within the higher-education sector.
The shift to a more market-oriented model was also evident in the passing of a new Higher Education and Research Act. The legislation establishes a new regulatory body in England—the Office for Students—that will be responsible for overseeing future developments of the TEF and encouraging greater competition between providers. For example, the new law makes it easier for private providers to obtain degree-awarding powers and the university title, and access the student-loan system. In the interests of ensuring academic quality and protecting the public purse, the UCU has major concerns about the further entry of private, for-profit providers into the English system. The quicker and easier it is to become a university and award degrees, the more vulnerable the higher-education sector becomes to for-profit organizations pursuing financial gain rather than providing high-quality educational experiences. The UCU has consistently warned that the UK government is opening the floodgates to a repetition of US-type scandals involving for-profit providers. As a union, we will continue to call for additional regulation for these types of risky providers.
Next steps
What is the specific future for the TEF in this new regulatory environment? First, after a cursory lessons-learned exercise, the government announced that it will be making “no changes to the overall structure or methodology of the TEF.” However, in future exercises there will be a reduced role for benchmarking and student surveys in the evaluation process—changes that are likely to benefit research-intensive universities. New supplementary metrics on both graduate earnings and grade inflation have also been proposed by the government. The former reinforces the controversial link between the TEF and labour market outcomes, while the latter seems more of a populist response to media concerns about the growing proportion of students achieving top degree classifications. Despite their different origins, both proposals highlight the politicized nature of metrics in higher-education policy-making.
Second, the government remains committed to introducing a subject-level TEF exercise in the future. A series of pilots will begin later this year. Although no ratings will be attached to the pilots, the whole process is likely to be complex and contentious. If introduced as part of a full subject-based TEF, the impact on faculty will be considerable, particularly in terms of increased workload and student expectations. Whatever the outcomes, the UCU will continue to challenge the use of the TEF as a crude performance-management tool.
Support and recognition for teaching
Finally, despite the obvious flaws with the TEF, there is a need for a proper discussion about how universities can better support and recognize teaching in higher education. There is a key role for faculty unions in this process and an opportunity to build broader alliances with student organizations. The UCU has long argued that faculty working conditions are effectively student learning conditions. In addition to campaigns on public funding, better job security; reduced workloads; access to appropriate training, support, and professional development; and fairer promotion criteria need to be at the heart of the debate on good-quality teaching.
