Introduction
In 2014, the University at Buffalo (UB) embarked on a mission to revolutionize its approach to evaluating teaching effectiveness by introducing the Universal Student Evaluation of Teaching (UBSET). In doing so, UB sought to measure if eleven particular questions (six course-oriented and five instructor-based) derived from the 32-item Student Evaluation of Educational Quality (SEEQ), could reduce test fatigue while maintaining fruitful results.
Creating the UBSET
The UBSET emerged as a result of collaborative efforts involving measurement experts and faculty members from diverse university units. Adapted from the Student Evaluation of Educational Quality (SEEQ), the UBSET strategically narrowed down 11 crucial items to create a more concise yet effective instrument. The primary goal was to ensure comprehensive coverage of teaching effectiveness while alleviating potential test fatigue among students.
The report aims to construct a validity argument for the UBSET, evaluating its effectiveness through several sources of validity, including test content, response processes, internal structure, and relations to other variables. Using these four sources, this evaluation measures and discusses insights into what aspects are functioning well in the set while also identifying areas that may benefit from improvement.
Major Findings
The study explored the possibility of underrepresentation in its sample through interviews and additional questions, revealing that a majority of faculty and students found the UBSET evaluation questions to be appropriate. The report also notes feedback provided for potential improvements, emphasizing the need for evaluating faculty’s content knowledge, flexibility, and the ability to create a positive classroom environment.
Outcomes of the Study
In its current form, the UBSET demonstrates strong evaluation metrics of teaching effectiveness, with minimal suggestions for enhancements. Analysis of response processes indicates student understand all but one questions of the set, contributing to a unidimensional measure. The report highlights remedial actions taken, such as addressing wording issues, to enhance the UBSET’s effectiveness continually.
Addressing Existing Challenges
The report identifies persisting challenges, including a potentially confusing item related to workload and students’ reluctance to give low grades, leading to inflated faculty scores. The need for a nuanced approach to assess faculty effort, possibly through alternate wording or a rubric, is discussed. The study emphasizes the importance of ongoing data collection and analysis to refine the UBSET further.
Continual Improvement
The validity argument for the UBSET is viewed as a dynamic process, illustrating how data-driven insights lead to enhancements. Examples include resolving the Andrich threshold gap through the inclusion of a neutral response and rectifying wording issues in specific items. The report emphasizes the institution’s commitment to continual improvement and the importance of soliciting feedback from both faculty and students.
Conclusion
The UBSET represents a significant leap forward in evaluating teaching effectiveness at the University at Buffalo and its influence elsewhere. This analysis demonstrates their commitment to excellence, with a focus on refining the instrument based on data-driven insights. The report serves as a testament to the university’s dedication to providing a strong and meaningful evaluation tool for faculty, ensuring an ongoing culture of improvement and accountability.
For further information on the study, please contact ubcei@buffalo.edu