Peer Evaluation: The Original Comprehensive Assessment
When using teams in education, instructors often use peer evaluations and self-evaluations to assess how effectively each team member contributes to the team. The Comprehensive Assessment of Team Member Effectiveness (CATME) was developed for this purpose. This web-based instrument collects data on team-member effectiveness in five areas research has shown to be important.
Contributing to the Team’s Work
Interacting with Teammates
Keeping the Team on Track
Having Relevant KSAs
(Knowledge, Skills, and Abilities)
The CATME Peer Evaluation instrument is a behaviorally anchored rating scale, which describes behaviors typical of various levels of performance in each of the five categories. Raters select the category of behaviors that most closely matches the behavior of each student on their team (including themselves). The CATME website shows the instrument and allows faculty and students to practice using the system by rating fictitious team members.
Understand What is Happening in Student Teams
The system alerts faculty to exceptional conditions, which are rating patterns that warrant attention.
- Low—a student who rates him/herself as ineffective and who also receives “ineffective” ratings by teammates.
- Overconfident—a student rated as “ineffective” by teammates but rates him/herself as much more effective.
- High—a student who is rated as highly effective according to both teammate and self ratings.
- Underconfident—a student rated as highly effective by teammates but who under-rates her/himself.
- Manipulator—a student who rates him/herself as highly effective and who rates teammates as ineffective in disagreement with teammates. Such a student may be trying to influence the distribution of grades unfairly.
- Conflict—a team in which there is considerable disagreement among the various raters about the effectiveness of an individual student.
- Clique—a team in which cliques appear to have formed. The ratings show that subsets of the team rate members of their subset high and members of other subsets low.
Some of these conditions have more than one explanation. A student flagged as a “manipulator” might actually have performed a disproportionately large amount of the work on the project even though they worked to engage their teammates in the process. Thus, an instructor’s involvement and judgment are critical when exceptional conditions are flagged. Though the formal study of these exceptions has not been completed, instructors using the system have reported that both the clique and conflict conditions have accurately provided early warnings of those conditions.
The CATME Online Interface
The CATME website is a secure interface for collecting data on team-member effectiveness and reporting different views of the data to faculty and students. The CATME system has a number of convenient features—the ability to upload student and team data from files generated by Excel; support for multi-section courses and teaching assistants; the ability to edit teams, reset surveys, send email reminders, and track survey completion. The system also allows students to make comments for instructors to read and can compute grade adjustments based on how the ratings patterns compare with faculty-specified criteria.
Instructors can request an account at www.CATME.org or by clicking the button in the top right corner of this website. The process of defining a class and setting up teams is wizard-based, but a tutorial is available. Several typical screen shots are captured here:
The first of five rating categories: Contributing to the Team’s Work
The wizard-based interface for class creation in CATME is typical of other setup screens.
Faculty summary results (raw data available)
Student results: self, team, average
CATME Peer Evaluation is part of the CATME SMARTER Teamwork system, which includes other team-support tools. The CATME online interface was developed by Deer Run Associates. This material is based upon work supported by NSF Awards 0243254 and 0817403.
Development and Validity Testing for CATME Peer Evaluation Assessment
Ohland, M. W., Loughry, M. L., Woehr, D. J., Bullard, L. G., Felder, R. M., Finelli, C. J., Layton, R. A., Pomeranz, H. R., & Schmucker, D. G. (2012). The comprehensive assessment of team member effectiveness: Development of a behaviorally anchored rating scale for self and peer evaluation. Academy of Management Learning & Education, 11, 609-630. This paper won the 2013 Maryellen Weimer Scholarly Work on Teaching and Learning Award.