In 1974, Robert M. Pirsig wrote a book entitled “Zen and the Art of Motorcycle Maintenance”, within which he provided a philosophical argument for the primacy of quality. Quality is described as a metaphysical underpinning of , well, just about anything; an underpinning from which other characteristics can be derived. I read this book for the first time about 2 years ago, after I had become passionate about using technology to better support the development of meta-cognitive skills in students. I was completely amazed how well Pirsig’s conceptions fit with the sorts of assignments I had been promoting as powerful and much needed. In some sense, his depiction of the primacy of quality made explicit an assumption that was at the core of much of what I was doing.
Before I jump into a discussion of quality, and quality-based discriminations, let me briefly jump back on a soapbox I often find myself upon. As I view it, the initial role of universities was to help develop scholars but, somewhere along the way, we’ve lost much of that and instead are primarily developing databases. What’s the difference? Databases are repositories of information whereas scholars are consumers and transformers of information. That is, scholars possess cognitive skills that allow them to think critically or creatively, to be self-reflective and open to revision, and to communicate the results of their thoughts in clear and efficient prose or speech. When we view our jobs primarily as providing content to our students, we are building databases. When we give our students practice with the sorts of meta-cognitive skills described above, we are building scholars. In my assessment, we do too much off the former, and not nearly enough of the later.
I believe that our current state of affairs is due to conceptions, and misconceptions, about logistic barriers to teaching meta-cognition. That is, virtually every educator will agree that we should be asking our students to think more, and to communicate their thoughts more, but they think this can only really be done via essay-type assignments, assignments which are both time and resource consuming. Thus we instead turn to the sorts of assignments that one can use a Scantron to mark, assignments that almost invariably focus on content rather than cognition.
However, when one begins from the primacy of quality, other options are available. When these options are implemented in effective technologies, logistic barriers can be minimized, and sometimes eliminated completely. So what am I suggesting? If you’ve ever been around young children you know of their love of a simple 3 letter word; why? It seems that no matter what parent says, the child replies with “why?” This can be frustrating. Why? Well, because it requires us to justify every decision, and that requires us to both think about the basis of the decision, and to explain that basis in a way the child can understand. Hmmm, meta-cognitive thought and communication all at once- fascinating! All that from 3 simple letters.
Assignments Focused on Quality-Based Discrimination
So here then is an example of the sort of assignment I have in mind, described in general terms. Students are exposed to a number of exemplars of some sort of content. Perhaps three poems, or three examples of computer code, or three photographs depicting shade and light. They are then asked to rank order these exemplars in terms of quality, perhaps generally defined, or perhaps in accord to some more specific definition. Given the primacy of quality, most students find this a natural thing to do. They feel they can sense the core quality at some intrinsic level. Next, and most critically, they are asked to justify their rankings; that is, we ask them why they have ranked the exemplars as they have. This is where the learning really comes it. Call it experiential, call it constructivist; irrespective of the label, students are being asked to figure out for themselves why one exemplar seems to be of a higher quality than another.
Even at this basic level an assignment of this type requires students to contrast, analyze, evaluate and reflect. These are not only all core aspects of critical thought, but they are also the sorts of skills that are often difficult to encourage through multiple choice testing. Thus they provide a fantastic compliment to the sorts of assessments we typically use to test for knowledge of content.
There are two other positive attributes that make quality-based discrimination assignments very powerful within educational contexts. The first is flexibility with respect to the specifics of what the students are asked to compare. This allows these assignments to be conducted in virtually any learning context with students at virtually any level. For example, preschoolers could be asked to taste three different cookies, to then rate them from best to worst, and to justify their ratings as well as they are able. At another extreme, brain surgeons in training could be provided with three distinct courses of action with respect to some medical history, or rocket scientists could be asked to evaluate three different ignition systems with some specific mission in mind. Yup, from preschoolers to brain surgeons and rocket scientists!
Another really nice aspect of this flexibility attribute is the ability to enhance engagement by meeting the student half way with respect to their interests. Imagine a group of students who are generally not interested in a typical academic curriculum, but who were very interested in, say, popular music. Those students could be asked to compare three music videos in terms of the artistic aspects of the video, or they could be asked to compare three songs in terms of lyrical content. It may well be the case that these students would not only perform such assignments, but they might also actually enjoy the opportunity to engage deep thought in these contexts.
This brings us to the second positive attribute of assignments that promote quality-based discriminations; they can be utilized across a wide range of contexts. Students might perform such an assignment in the context of poetry in their English course, then perform a structurally similar assignment in the context of Psychology in another course, etc. Thus they can gain repeated experience utilizing the same cognitive skills throughout their experiences in higher education and, in fact, throughout their entire educational career across a wide range of contexts.
This repetition across contexts is really critical. Cognitive skills are like any other skill in the sense that they develop through practice. If we want to become expert guitar players we must commit to a lot of practice. If, instead, we get a little practice here, and a little there, and maybe even a lot at one time, but then only dribs and drabs, we simply will never achieve a level of true proficiency. We need relatively constant practice, and practice across specifics of the learning context are especially powerful. So if we want to produce students who are expert at meta-cognitive processes the same point applies; we must give them regular and systematic practice.
Sometimes I feel this point is lost, at least within the higher education community. For example, we want to teach our students how to communicate effectively. One solution some universities accept is to include one course within their program that teaches and reinforces these skills in detail. This may be viewed as the “writing course”, the one course wherein sufficient resources are placed in hopes students will learn all they need to know about writing in 12 weeks. This sort of massed learning is not nearly as strong as repeated practice throughout the entire program.
Adding Peer Assessment Into the Mix
While assignments focused on quality-based discrimination are powerful in their own rite, one can really enhance the impact on the development of meta-cognitive processes by combining them with peer assessment. In a sense, peer-assessment makes the assignments more personal which can be explicitly used to enhance self-reflection. In addition, peer-assessment both enhances the sense of the classroom as a community while also breaking down the perceived distinction between students and teachers.
So now imagine the following assignment. Students are first asked to perform some assignment; perhaps it is a written composition, perhaps they are asked to take photos that use light and shadow in interesting ways, perhaps they are asked to create their own music videos to popular songs. Once again, the content of the assignment remains quite flexible. After students submit their assignment they are then presented with the assignments submitted by a subset of their peers, and they are asked to rate and comment on them just as they did in the quality-based discriminations described earlier. This time, however, their own assignment is presented along with that of their peers, and they may even be asked to rate and comment on it as they comment on the work of their peers.
This sort of assignment exercises all the same cognitive processes as the previously described assignments focused on quality-based discriminations, but it also does much more. Students are given a very clear sense of how their work compares to that of at least a subset of their peers. This is an extremely powerful learning signal, and one that we typical do not provide our students with. What’s more, if they are asked to also rate and comment on their own piece within this context we are strongly supporting self-reflect and self-critical processes, the sort of processes that must be the cornerstone to improvement.
In fact, we can enhance this relation between self-reflection and improvement even more if we structure the assignments in a formative manner. That is, students could be asked to rate and comment on their peers’ work, but with the intent of helping their peers to improve their “draft” assignments for a final submission. Now we have students helping students, which is great for building community within the classroom while also emphasizing self-reflection as a basis for improvement.
Within this formative process we can also do ramp things up even more. Specifically, when I do these assignments with my class I tell them the following: throughout your life you will be asked to “perform” in various ways, and you will subsequently receive feedback on your performance. Some of the feedback will be valuable, but some will be misguided. It’s up to you to figure out the difference between the two, and to incorporate the useful feedback while ignoring the useless. Does this sound familiar? It is quality-based discrimination once again, but this time it is in the context of what others think of your work, so it includes a strong self-reflective element.
So by adding peer assessment into the mix we are strengthening self-knowledge via self-reflection that occurs at various points in the process. In fact, one could make self-reflection even more powerful by basing part of the mark on a student’s accuracy of self-reflection. When they mark themselves, how accurately do they do so? By basing part of a final mark on this accuracy one can insure students are motivated to self-reflect at a deep analytic level.
The Role of Technology
It is likely the case that many of you are reading this and thinking something along the lines of “sure Joordens, that all sounds really fantastic, but how does one logistically manage this entire process?” I am a firm believer that technology can play a critical role in education, but that role is to enable, not to be a pedagogical end in and of itself. So we now have laid out what I at least view as an extremely rich pedagogical experience, and one that might fill a gap that exists in our current education system. Can we use technology to enable a logistically reasonable way of supporting this experience?
The answer is yes. I recently did an assignment just like the one described, and I did it in my Introductory Psychology class, a class that numbers 1500 students! Did it take some extra resources? Yes, if one wishes to use the full formative process it does require some additional resources. For our assignment we required 200 additional TA hours. Great value considering that TAs did not just grade the final submission, but they also graded the process as well. Specifically, 40% of the final grade was based on the quality of the final submission, 40% was based on the quality of the comments that a student provided to their peers, and 20% was based on how well they revised their work in light of the comments they received. The signal sent to students was clear: we care as much about the thought and effort you put into the process as we do about your own final product.
How did we do it? Well, this is where this paper begins to feel like an advertisement and I begin to feel like a salesman. This is unavoidable I’m afraid. You see, it is possible to do these assignments because of an internet-based program called peerScholar, a program created by myself and my current Ph.D. student Dwayne Pare. We didn’t just create this program, we’ve also done and continue to do considerable research examining its efficacy and extending its use to all levels of the educational system.
The peerScholar system can by used for both summative or formative assignments. When used summatively, the process ends when students give each other ratings and comments. We have shown that when five or more undergraduate ratings are averaged, they result in a grade that is as reliable as the mark provided by a single expert TA. This version is not pedagogically as rich as the formative process, but it has the feature of requiring only minimal resources to use. Thus, in courses that currently have few if any assignments that ask students to perform open ended assignments to develop meta-cognitive skills, summative version of the peerScholar process can be employed with only a minimal addition in support hours.
The possibility of using the system in this summative way means that a given curriculum can indeed be littered with assignments that focus on quality-based discrimination in the context of peer assessment. A combination of summative and formative assignments would really enhance a program, giving students rich and regular practice of their meta-cognitive skills. Cognition and content in almost a zen-like balance, that’s the educational approach I would like to see spread.
Steve Joordens is a professor of Psychology at the University of Toronto Scarborough.
Photo by cdrummbks on Flickr
