It has been two years since the Ontario Confederation of University Faculty Associations published the Report of the OCUFA Student Questionnaires on Courses and Teaching Working Group. The working group, which included experts on methodology, research ethics, and human rights, engaged in a comprehensive examination of studies on student questionnaires on courses and teaching (SQCTs), sometimes referred to as student evaluations of teaching (SETs). The group’s goal was to address mounting concerns over the use of SQCTs in key academic personnel decisions, including hiring, tenure review, promotion, and performance assessment processes.

The working group’s findings affirm the groundbreaking arbitration award handed to the Ryerson Faculty Association in June 2018. The Report and the Ryerson case converge on two important points: 1) SET data do not generate valid measures of teaching effectiveness; 2) the use of such data to assess performance in teaching amounts to a discriminatory practice that consistently places faculty members of equity-seeking groups in a disadvantaged position. In support of his decision, Kaplan notes that “numerous factors, especially personal characteristics—and this is just a partial list—such as race, gender, accent, age and ‘attractiveness’ skew SET results. It is almost impossible to adjust for bias and stereotypes.” OCUFA’s Report goes further, stating that “the summative use of student questionnaires does not meet the bar set by the [Ontario Human Rights] Code, and that their use in deciding matters of pay, appointments, and career progress is a form of systemic discrimination.”

Yet arbitration ruling is not case law. Kaplan’s arbitration award is only legally binding on the parties involved—in this case, Ryerson University and the Ryerson Faculty Association. Likewise, OCUFA’s Report on Student Questionnaires on Courses and Teaching is not an enforceable policy document. Unless there is a case involving the use of SETs for summative purpose before the Human Rights Tribunal of Ontario that is ruled in favour of the applicant on the grounds stated in the Report, the document is incapable of generating binding changes in institutions that continue to use SET data for summative purposes. Against these limitations, it is reasonable to ask whether the arbitration award and the Report constitute the necessary rationale for dismantling the pivotal role played by SETs in hiring, tenure, promotion, and performance assessment decisions. Surely, the latest round of commitments made by universities to address systemic racism on campus ought to include an examination of policies that categorically disadvantage faculty members because of their race and/or ethnicity, among other prohibited grounds of discrimination.

OCUFA’s Report is clear that data generated by institutional SQCTs lack the validity, legality, and research ethics to support their use as an assessment tool of faculty members for summative purpose. If a performance assessment tool with comparable problems were used on students, they would most certainly fight back and no university administrators would allow the tool to be used. Yet, the pernicious effects of SETs seem to be largely unknown or of little concern among faculty. For many, SETs are accepted as simply a part of the job. Accordingly, the use of SET data as a valid, if imperfect, performance indicator remains firmly entrenched in the academy, resulting in an unsustainable situation in which not even faculty associations have a joint position on the matter among themselves.

Following in the footsteps of OCUFA’s Report, the Ryerson arbitration, and the notable decision in 2018 by the University of Southern California to halt the use of SETs on discriminatory grounds, this article argues that SETs should be suspended as a summative performance assessment tool across universities and colleges. It is both ethically and academically unsustainable to let SET data adjudicate what constitutes effective teaching and its correlation, which is substantive learning.

To begin, while SETs are well-studied, the literature overlooks a couple of anomalies that involve the assessor, that is, the student. They are as follows:

1. Student-driven institutional challenge—Academic misconduct

Student academic misconduct occurs regularly and, if not caught, allows students to complete assignments and obtain their desired grades through dishonest means. Sometimes this involves plagiarism from external sources or copying among peers. Sometimes this involves hiring others to write customized assignments or to take exams on the student’s behalf. The first major Canadian study to gauge the extent of academic misconduct was done between 2002 and 2003. It involved 11 higher education institutions across five provinces. Among the students surveyed (14,913 undergraduates and 1,318 graduates), 18 per cent of undergraduate and 9 per cent of graduate students admitted to having cheated on a test and 53 per cent of undergraduate and 35 per cent of graduate students admitted to having cheated on written work. These findings are consistent with many similar studies in the U.S. The scale of the problem, as suggested by these data, begs the question of the integrity of student-as-assessor who fills out a SET questionnaire. Yet, there is no questionnaire that I know of that systematically flags students who have engaged in academic misconduct.

2. School teachers are not assessed by students

The Ontario Ministry of Education has a 108-page long manual that provides the protocol and guidelines for assessing teachers of all grades. The current manual was released in 2010. In my research, I learned that the Ministry had wanted to incorporate feedback from students and parents as evidence for assessment. However, this proposal was strongly rejected by teachers on grounds that such feedback would be biased, subjective, and unprofessional. If a student who graduates from a high school in June is not deemed by the Ministry as an appropriate assessor for teachers, how is it that the same student is suddenly capable of assessing a university professor six months later?

On the surface, these two anomalies can be lumped in with other methodological considerations that call into question the validity of SET data for assessing teaching effectiveness. However, I believe that these two overlooked considerations should stand on their own as reminders that there is something fundamentally absurd about treating SETs as a professional accreditation enterprise. The very idea that students constitute a credible source of validation for who we are as university teachers deeply permeates our constructed identities as academics in the contemporary world.

Beginning with Bill Reading’s groundbreaking insight into the role of universities during the early years of globalization, which he articulated in The University in Ruins, the corporatization of universities and attendant consumerism culture are now well-researched grounds in which to root critical perspectives on universities. The concern over SETs being no more than a consumer satisfaction matrix is indicative of the predatory nature of advanced capitalism. In this matrix, teaching becomes a service and university teachers are service providers charged with the task of training the next generation of entrepreneurs, professionals, and wage-earners by turning learning into the acquisition of marketable skills.

However, the fact of the matter is that SETs predate globalization and even the student movement of the 1960s. SETs have beginnings that trace back to early twentieth-century America. In 1917, the German academic and thinker Max Weber—known best in the Anglo-American academic world as a key founder of sociology—pointed to a fundamental tension in the job of university professors, based in part on observations drawn from his trip to American campuses. In a famous lecture delivered at the University of Munich, “Scholarship (Wissenschaft) as a Vocation (more commonly known as “Science as a Vocation”), Weber said that to be a successful academic is a matter of “chance.” This is because the job expects its holder to be “both a good scholar and a good teacher” when the “two qualities are by no means the same.” Added to this fundamental tension is the mixed message that we get from universities today—the all-too-familiar claim that teaching and research are on par and should inform each other. Yet, all universities have some stand-alone office of research headed by a high-profile vice-president, while teaching development typically receives far less prestige and attention.

Returning to Weber’s observation of how the two activities require different qualities, I believe that one way to capture the contrast is to understand teaching as an interactive social activity that requires the teacher to impart intellectual clarity to students. Research, on the other hand, is a creative activity that requires the researcher to forge ahead and not be afraid to go it alone when needed. Stated differently, being an isolated teacher is an oxymoron, but being an isolated researcher is both conceivable and sometimes celebrated as genius at work.

The reality for most faculty members is that we are average. We work to the best of our ability to be good teachers and good researchers. Yet, realistically, many of us will end our academic careers without a teaching award and with a number of publications that have not been regarded as groundbreaking, as if the latter should be the norm rather than the exception. Indeed, as Weber noted in 1917, the trend is for more and more specialized research. These days, everyone is looking for a niche as a way to stand out when novelty and originality are not necessarily the same. Meanwhile, the Tri-Council’s push for knowledge mobilization, which treats research outcomes as concrete products with applicability and marketability, discourages research that does not promise immediate returns.

When we step back to think about how we end up in academia, most of us would say it is our passion for the pursuit of knowledge. As students, we were keen learners who wanted to know more and we aspired to become good researchers. Along the way, we found that we were also teachers. Yet, teaching is not a simple addition to an academic career. Rather, teaching is integral to the pursuit of knowledge—enabling it to be a sustainable collective human project across time and space. In this sense, teaching is a concrete way for us to close the loop and give back to society what we owe as learners and as researchers. Teaching is the public face of our rather secluded life as researchers. Moreover, in this public space, we have a captured audience that does not challenge our expertise in ways that we experience in our respective research communities. To be immersed in this receptive audience is tremendously reassuring. Better yet, having its immediate approval makes this rather impossible job more palatable.

To sum up, our identity as academics in today’s universities places students in the awkward position of validating who we are, impacting both how we are judged by others and how we judge ourselves. This fundamentally differs from how patients relate to their doctors or clients to their lawyers. Similar to university teachers, doctors and lawyers have a captured audience that taps into their respective expertise. But unlike university teachers, doctors and lawyers do not look to patient or client evaluations to affirm their abilities, qualifications, or professional merit. Doctors and lawyers do not need endless research to point to the obvious, namely, that what they do in their professional capacity is best guided by their independently acquired expertise rather than what patients and clients say is best. Otherwise, the integrity of their job could be easily compromised and their expertise denigrated. Accordingly, dismantling a dubious system of accreditation, which is what the summative use of SETs represents, is something that we should do as members of the academic community. By getting rid of a practice that is found to sustain systemic discrimination against members of minority and equity-seeking groups, the academic community will be taking an important step toward correcting its own injustices. Not only will this buttress our respect for each other as colleagues, it will earn respect from society more broadly.

Theresa Lee is an associate professor in the Department of Political Science at the University of Guelph. She is the author of Report on Teaching Evaluation Mechanism (2010) and in 2012-14 Chaired the Provost’s Working Group on SETs at the University of Guelph.