Evaluations of tertiary teaching measure different factors
A major report that examined the relationship between the current national and local evaluations of teaching in Australia’s tertiary education system has found that they both measure different factors. The report provides valuable information on evaluation systems that play an important role in the measurement and accountability requirements of today’s tertiary institutions.
Currently teaching performance within Australia’s 39 public universities and 2 private tertiary universities are evaluated using both local instruments called teaching quality indicators (TQIs) and a national instrument called the Course Experience Questionnaire (CEQ). The retention and dismissal of academic staff is partly based on the results of institutional TQIs while university funding is partly allocated on the basis of CEQ results. TQIs can take the form of questionnaires given to students at the end of semester and each university across Australia has their own form. The CEQ gathers data on the teaching and learning experiences of graduates of all Australian Universities. This national data set is analyzed and reported by a national body and organised by field of study.
Considering the importance of these teaching evaluations it is somewhat surprising that an analysis that examined the relationship between both the CEQ and the TQIs had never previously been conducted before. This report, Measuring student experience: relationships between teaching quality instruments and the course experience questionnaire, is a long-term collaboration between the Department of Economics and the Teaching and Learning Unit at the University of Melbourne. Its authors Associate Professors Joe Hirschberg , Jenny Lye (Economics), Martin Davies (Teaching and Learning Unit) and Carol Johnston received a large Australian Learning and Teaching grant ($212,722), and a LTPF grant ($96,000), enabling them to examine this issue. Firstly they sampled the extensive literature that describes influences on responses to TQIs. From this survey, they found that there are many factors which influence students’ responses, other than a specific lecturer’s performance in class. It was also discovered that the phenomena of leniency or non-leniency in marking in the subject was also elemental in their ultimate perception of their course of study.
When conducting this analysis the authors were mindful that firstly, there is no consistent TQI for all institutions. Most of these surveys have been assembled using questions with different origins that have varied over time and that are individual to each institution. Secondly, even if the TQIs were constant and consistent across all institutions, it would be necessary to account for any other factors, such as the year level and the field of study, that may influence the specific subject TQIs and the overall course evaluation.
Additional research by the authors included collecting examples of the TQIs from all higher education institutions in Australia, designing a composite survey to elicit responses from Commerce students at four participating institutions (Flinders, Melbourne, Tasmania, and Wollongong) and matching the CEQ responses from students who completed degrees at the University of Melbourne and the students’ experiences in order to determine the degree to which their responses could have been anticipated.
This work allowed the authors to draw the following insightful conclusions
- In most cases, the items on the TQI and the CEQ are measuring different factors.
- Older students, those that receive higher marks than their peers, and those who study full-time in their last year, are more likely to rate a course more highly.
- Students who take more than one course, who are in subjects with higher than average enrolments, which received lower than average TQI ratings for the same subject taught at other times, are less likely to rate their course highly.
- The phenomena of leniency or non-leniency in marking in the subject was also elemental in their ultimate perception of their course of study
Commenting on the report, one of its authors, Martin Davies, Associate Professor in Higher Education in the Teaching and Learning Unit, hoped that it would contribute to a deeper understanding of the teritary teaching evaluation,’ this report is likely to be of use to higher education decision-making and local policy making within universities’.