Do Surveys Make for Good Evaluation Schemes?

Thursday, June 14th, 2012

In my opinion, no, given what we know about bias. Typically, surveys of performance are pretty predictable. Students who do well in school give better reviews to teachers than students who do poorly. Often, attitudes are shaped by a hole host of variables unrelated to teachers. These kinds of things don’t tell us a lot about performance because they are difficult to filter. This account from the Connecticut Mirror provides a chart on the make up of teacher evaluations in Connecticut. I’m assuming the 45% Student Performance will come from testing.

I just don’t understand why smart people can’t come at evaluation schema in more practical ways, for example, evaluating teacher performance by the degree to which their students learn over time. After arithmetic, sit on the deck built by students and if it works, the teacher and the students did just fine. And someone got a deck out of the deal, too. But learning by doing is not a priority in testing cultures.

Consider this example of a quote from the Gates Foundation that supplies a high degree of confidence in survey feedback

He [John Lucsak] pointed to a recent report from the Bill & Melinda Gates Foundation’s Measures of Effective Teaching Project, that calls surveys “value-added measures [that] do seem to convey information about a teacher’s impact.”

Kinda, sorta, maybe. The use of the term value-added here is suspect. It would do you well to read the report linked to from the article and read the student survey questions. Note that I went through some of the bios of Education First advocates and was somewhat surprised at the qualifications of the experts.

I would also assume that the unions will be blamed again for getting in the way. They have good reasons to dislike this kind of performance model.

One of the problems all of these advocates, for or against whatever reforms, is they have to work within is the ecology of the current system, which works on a shoe string and goes severely under funded given the system’s size. It’s like a pond full of frogs but there’s not a lot of rain falling out of the heavens.

2 responses to “Do Surveys Make for Good Evaluation Schemes?”

  1. Katie Bentley says:

    As someone who has never taught but has done a fair amount of studying, I find that teacher surveys are exasperating from an entirely different standpoint. On several occasions a survey was the only recourse when I was not happy with professors—after being ignored by department heads that were too close to the other professors and a dean of students who was coasting until her retirement. In the last year of my BFA I had a few standout teachers that were clearly not prepared to do the task before them. For the rest of my time as a student, I approached teacher evaluations with the syllabus and course description in hand. I asked myself “Did I learn enough of what I was promised I would learn for the amount of effort that I put into this course?” In almost every one of the cases of poor teachers I received an A for the course, but I would have failed my professors. It was not a vendetta—it was the reality of unpreparedness, thoughtlessness, inability to answer questions, and so on. (And I would put out there that most students, even poor students, know when a teacher is prepared—or not.)

    Another point is that teachers clearly sway the type of responses that students give. Is the survey done before or after class? How much time is given? Can you do it privately—for example using systems like Blackboard Vista or Moodle to submit it electronically? Time and privacy can be all the difference for some students when it comes to answering questions thoughtfully or going down the line and responding with a C the for the length of the survey. Can teacher evaluations be made a course requirement? Perhaps a student should not collect their final grade until their teacher evaluations have been reviewed. I once had a teacher who told his class immediately before we filled the survey out that the surveys were used to determine bonuses. On another occasion, a professor that I had for multiple courses several semesters in a row told me he was able to distinguish which students had filled out surveys based on handwriting and language patterns.

    The survey process is flawed, but students must have recourse for bad teachers! Moving away from cold, inhuman, inaccurate surveys toward more personal exit interviews would be ideal. Perhaps a third party, trained in body language. Unfortunately for Connecticut, the state will be swayed by cost and whatever Diane Ullman has in mind that isn’t costly or complicated. Strange, since education is usually costly and complicated all on its own.

  2. Steve says:

    Good and fair points. I especially like that you would have failed poor teachers who gave you good marks.