Showing posts with label value-added ranking of teachers. Show all posts

Saturday, June 14, 2014

The guesstimate that struck down California’s teacher tenure laws

Fuzzy Math

The guesstimate that struck down California’s teacher tenure laws.
By Jordan Weissmann
Slate
June 2014

This week Los Angeles Superior Court Judge Rolf Treu handed the education reform movement a stunning legal victory, when he struck down California’s teacher tenure laws for discriminating against poor and minority students. The statutes made it so onerous to fire bad teachers, he wrote, that they all but guaranteed needy kids would be stuck in classrooms with incompetent instructors—rendering the laws unconstitutional.

As evidence, Treu cited a statistic that sounded damning: According to a state witness, between 1 and 3 percent of California’s teachers could be considered “grossly ineffective.” Here was the passage:

There is also no dispute that there are a significant number of grossly ineffective teachers currently active in California classrooms. Dr. Berliner, an expert called by State Defendants, testified that 1 to 3% of teachers in California are grossly ineffective. Given that that the evidence showed roughly 275,000 active teachers in this state, the extrapolated number of grossly ineffective teachers ranges from 2,750 to 8,250. Considering the effect of grossly ineffective teachers on students … it therefore cannot be gainsaid that the number of grossly ineffective teachers has a direct, real, appreciable, and negative impact on a significant number of California students, now and well into the future for as long as said teachers hold their positions.

This seemed like a fairly important piece of the decision—if you’re going to argue in court that a state law is dooming children to second-rate educations, you ought to be able to quantify the problem. Politically, it also seemed liked a pretty awful indictment of the state government if officials knew for certain that so many useless teachers were lounging around California’s classrooms. But where did this number come from?

Nowhere, it turns out. It’s made up. Or a “guesstimate,” as David Berliner, the expert witness Treu quoted, explained to me when I called him on Wednesday. It’s not based on any specific data, or any rigorous research about California schools in particular. “I pulled that out of the air,” says Berliner, an emeritus professor of education at Arizona State University. “There’s no data on that. That’s just a ballpark estimate, based on my visiting lots and lots of classrooms.” He also never used the words “grossly ineffective.”

The phrase appears to have been Treu’s shorthand to describe teachers whose students consistently perform poorly on standardized tests. But Berliner is a well-known critic of using student test scores—or “value-added models,” in the parlance of education experts—to measure teaching skills. In part, that’s because research suggests that teachers don’t really control much of how their pupils perform on exams; according to the American Statistical Association, they influence anywhere between 1 percent and 14 percent of the variation in students’ scores. As result, teachers often don’t deliver the same results year after year.

Still, if you look at enough data, there are always a few teachers who consistently underperform on test results. “There’s an occasional teacher who shows up really good a few years in a row,” Berliner said. “There are a few who show up really bad.” And that’s where the now-infamous statistic comes in. During a deposition, Berliner told me, the plaintiffs’ lawyers asked how many teachers deliver low test scores year after year. He didn’t have a hard number, so he said 1 percent to 3 percent, which he thought sounded suitably small. That led to the following exchange with a plaintiffs’ lawyer during cross-examination. (Berliner emailed me part of the transcript.)

Lawyer: Dr. Berliner, over four years value-added models should be able to identify the very good teachers, right?

Berliner: They should.

Lawyer: And over four years value-added models should be able to identify the very bad teachers, right?

Berliner: They should.

Lawyer: That is because there is a small percentage of teachers who consistently have strong negative effects on student outcomes no matter what classroom and school compositions they deal with, right?

Berliner: That appears to be the case.

Lawyer: And it would be reasonable to estimate that 1 to 3 percent of teachers fall in that category, right?

Berliner: Correct.

Berliner seems to have let something important get lost in translation on the stand. Because, as he said to me, he doesn’t necessarily believe that low test scores qualify somebody as a bad teacher. They might do other things well in the classroom that don’t show up on an exam, like teach social skills, or inspire their students to love reading or math. And while he has observed teachers he didn’t particularly like, or thought could use more training, he’s never encountered one he would consider “grossly ineffective.”

“In hundreds of classrooms, I have never seen a ‘grossly ineffective’ teacher,” he told me. “I don’t know anybody who knows what that means.”

I asked Stuart Biegel, a law professor and education expert at UCLA, whether he thought that the odd origins of the 1–3 percent figure might undermine Treu’s decision on appeal. Biegel, who represented the winning plaintiffs in one of the key cases Treu cited, said it might. But he thought that the decision’s “poor legal reasoning” and “shaky policy analysis” would be bigger problems. “If 97 to 99 percent of California teachers are effective, you don’t take away basic, hard-won rights from everybody. You focus on strengthening the process for addressing the teachers who are not effective, through strong professional development programs, and, if necessary, a procedure that makes it easier to let go of ineffective teachers,” he wrote to me in an email.

To me, the tale of this particular statistic is a good example of why the education reform battle doesn’t really belong in the courts, at least not yet. We haven’t come to an accord on what makes a “good” or “bad” teacher, or how many of either are out there—even if one judge is convinced we have.

Monday, December 27, 2010

Do kids, parents or even administrators have a clue about which teachers are really teaching?

Do kids, parents or even administrators have a clue about which teachers are really teaching?
Not necessarily.

Popularity hinges on many things. For example, studies have shown that people tend to highly rate those who speak with confidence--and with words that are hard to understand! It's unbelievable. The less they understand, the higher they rate the speaker! They also rate more highly people who are good-looking. Strangely enough, bullies tend to be more popular than non-bullies. The ability to rate teachers effectively hinges on non-subjective measures.

"...[S]ome of my best teachers have the absolute worst scores,” she said, adding that she had based her assessment of those teachers on “classroom observations, talking to the children and the number of parents begging me to put their kids in their classes.”

“So if you have a teacher consistently in the top 10 percent,” he said, “the chances are she is doing something right, and a teacher in the bottom 10 percent needs some attention. Everything in between, you really know nothing.”

Hurdles Emerge in Rising Effort to Rate Teachers
By SHARON OTTERMAN
New York Times
December 26, 2010

...“If I thought they gave accurate information, I would take them more seriously,” the principal of P.S. 321, Elizabeth Phillips, said about the rankings. “But some of my best teachers have the absolute worst scores,” she said, adding that she had based her assessment of those teachers on “classroom observations, talking to the children and the number of parents begging me to put their kids in their classes.”

...New York City began ranking teachers in the 2007-8 school year as part of a pilot project intended to improve classroom instruction. The project, which cost $1.3 million, with an additional $2.3 million budgeted over the next 18 months, was expanded in the 2008-9 school year to give rankings to more than 12,000 fourth- through eighth-grade teachers...

In support of the model, Douglas Staiger, an economics professor at Dartmouth College, cites research showing that if a teacher receives a high-performing score one year, there is a modest likelihood that he or she will receive a high-performing score the following year. The correlation is about 0.3, he said, with 1 being perfect, and 0 being no correlation. This means that about one-third of teachers ranked in the top 25 percent would appear among the top quarter of teachers the next year.

While that year-to-year link may seem low, in the budding and messy exercise of trying to quantify what makes students learn, it is one of the strongest predictors of future student performance, along with the reduction of class size. That means that, on average, students placed for a year with a high-value-added teacher will do better than those placed with a low-value-added teacher. Dr. Staiger placed the improvement at about three percentile points on a typical standardized test.

“This information is useful but has to be used with caution,” he said. “It’s that middle ground. It’s not useless, but it’s not perfect.”

Yet a promising correlation for groups of teachers on the average may be of little help to the individual teacher, who faces, at least for the near future, a notable chance of being misjudged by the ranking system, particularly when it is based on only a few years of scores. One national study published in July by Mathematica Policy Research, conducted for the Department of Education, found that with one year of data, a teacher was likely to be misclassified 35 percent of the time. With three years of data, the error rate was 25 percent. With 10 years of data, the error rate dropped to 12 percent. The city has four years of data.

...“So if you have a teacher consistently in the top 10 percent,” he said, “the chances are she is doing something right, and a teacher in the bottom 10 percent needs some attention. Everything in between, you really know nothing.”

San Diego Education Report Blog

Links