Accountability for you, but not for me, seems to be a common policy in schools I've worked in. Now it's been exposed in the state superintendent's office in Indiana.
"Christel’s ninth- and 10th-grade students got brutally low scores on the English and math tests, with only 70 percent passing English and just a third passing math...
"Bennett’s staff swung loyally and swiftly into action; within a day they had found what they called a “loophole” in the state law that allowed them to change Christel’s grade."
The Case of the Missing Zeroes
An astonishing act of statistical chutzpah in the Indiana schools’ grade-changing scandal.
By Jordan Ellenberg
Slate.com
Aug. 2, 2013
Florida Education Commissioner Tony Bennett resigned Thursday amid claims that, in his former position as superintendent of public instruction in Indiana, he manipulated the state’s system for evaluating school performance.
Bennett, a Republican who created an A-to-F grading protocol for Indiana schools as a way to promote educational accountability, is accused of raising the mark for a school operated by a major GOP donor. Bennett calls that charge, which arose from emails among Bennett’s staff obtained by the AP, “malicious and unfounded” and “frankly so off base.” He offered a different explanation for why the grades for Indianapolis charter school Christel House Academy—whose mark soared from a C to an A—and 12 other schools were changed at the last minute. According to Bennett, he was just correcting a simple math mistake.
Bennett’s explanation is perfectly mathematically reasonable, and it would get him off the hook. The only problem is that the story he’s telling appears to be totally false.
Bennett told AEI’s Rick Hess, “As we were looking at the grades we were giving our schools, we realized that state law created an unfair penalty for schools that didn't have 11th and 12th grades. Statewide, there were 13 schools in question had unusual grade configurations. The data for grades 11 and 12 came in as zero. When we caught it, we fixed it.”
Bennett’s stated rationale makes sense. Here’s an analogy. You’re teaching a course with three exams, and each student’s overall exam grade for the class is computed as an average of her three individual exams
(1/3) * (exam 1) + (1/3) * (exam 2) + (1/3) * (exam 3)
But what happens if a student misses exam 3, with a justified absence, and the test can’t be made up? Then we have what’s called a “missing data” problem—we have to infer something about the student’s performance without the full complement of data we have for everybody else. One natural approach is to compute that student’s overall exam grade as the average of the exams she did take:
(1/2) * (exam 1) + (1/2) * (exam 2).
What you shouldn’t do is give the student a zero for the exam and average it into her grade.
Christel House had started out as a middle school and was adding one high school grade each year; for the 2011-12 school year, it served students through the 10th grade. High schools, according to Indiana statute, were to be graded on four metrics, averaged like so:
30% * (English scores) + 30% * (algebra scores) + 30% * (graduation rate) + 10% * (college and career readiness score)
But, as Bennett said in an official statement, “Christel House only served students in grades K-10, thus the graduation rate and college and career readiness measures could not be calculated because the school did not serve grades 11 and 12.” So it faced a missing data problem, like the student who had to miss an exam.
Indeed, it wouldn’t be fair to count those scores as zero! The obvious fix, just as with the student, is to grade such a school on the two scores that it does have:
50% * (English scores) + 50% * (Algebra scores)
Does that mean Bennett really did get railroaded, and he was just fixing an obvious error?
No—because the “fixed” version of the grade was what Indiana was already using before Bennett started tinkering with the gears. When you dig into the numbers, the story about the unfair zeroes looks like a complete fabrication.
Christel’s ninth- and 10th-grade students got brutally low scores on the English and math tests, with only 70 percent passing English and just a third passing math. Here’s how Jon Gubera, then the Education Department’s chief accountability officer, described Christel’s performance in one of the emails released by the AP:
“OK, here is their breakdown ...
They served grades K-10 in 2011-12 so they are a combined school but do not have any graduates. So their grade is a combination of the standard E/MS model and the HS 9&10 model which only counts ECA proficiency.
E/MS results: 3.00 on E/LA (no growth bonuses) and 4.00 on math (bottom 25% bonus) = 3.50 points (B)
HS results: 2.00 on E/LA (70% pass rate) and 0.00 on math (33% pass rate) = 1.00 points (D)
Final Combined results: E/MS 3.50 x .76 (76% of school is in grades 3.8) = 2.66 + HS 1.00 x .24 (24% of school is in high school) = .24. Thus overall grade is 2.66 + .24 = 2.9 (C).
Bottom line: their terrible 10th grade Algebra I results (33% passing) was the principal factor in earning a C grade."
Christel’s grade 3–8 scores came to a 3.50, or a B, and their high school scores, thanks to the algebra fiasco, were a 1.00, or a D. The school’s score is then an average of the grade 3–8 scores and the 9–10 scores, weighted according to the proportion of students in each group.
Where are the zeroes for the graduation rates and readiness score that got averaged into Christel’s score—the “unfair penalty” that Bennett claims he fixed? They’re not there, because that’s the part that Bennett, as far as I can tell, simply made up.
Gubera’s computation wasn’t satisfactory for Bennett, because it didn’t give Christel an A, and—judging by the emails obtained by the AP—he had already told Christel administrators they were getting an A. No mere numbers were going to make a liar out of him!
What’s more, Bennett wrote in one of the emails, the low grade for Christel meant “legislative leadership as well as our critics of A-F are going to use this against us to undo our accountability metrics through legislation.” He went on: “If you can’t tell, I am more than a little miffed about this. I hope we come to the meeting today with solutions and not excuses and/or explanations for me to wiggle myself out of the repeated lies I have told over the past 6 months.”
Bennett’s staff swung loyally and swiftly into action; within a day they had found what they called a “loophole” in the state law that allowed them to change Christel’s grade. The statute clearly states that high schools without 12th-graders get a score made up half of the English score and half of the algebra score. What’s more, the law said schools that combined high school grades and lower grades should use a weighted average of the elementary/middle school measures and the four high school measures.
Here’s where Bennett’s team found the loophole big enough to drive a charter school through. A normal person would do exactly what Chief Accountability Officer Jon Gubera did—give Christel the weighted average of its elementary/middle school score, according to the rules for elementary/middle schools, and its high school score, according to the rules for high schools.
But Bennett had a better idea. Christel was, technically speaking, not a high school, so the statutory formula for the high school grades didn’t apply. But it also didn’t have all four high school measures, so, he argued, the rules for combined schools didn’t apply either. There were just 13 schools in the state that had both middle school and high school grades but no seniors. For these schools, Bennett reasoned, the Indiana education poobahs should have a free hand to set the grades however they pleased. You can guess what happened next: Bennett ruled that the ninth- and 10th-graders in these schools didn’t count at all. So it was that the offending algebra grades vanished in a puff of bureaucratic smoke. (Anne Hyslop at Ed Money Watch has an even more detailed accounting of the process, if you like watching sausage get made.)
This was an act of astonishing statistical chutzpah. Suppose the syllabus for my math class said that the final grade would be determined by averaging the homework grade and the exam grade, and that the exam grade was itself the average of the grades on the three tests I gave. Now imagine a student gets a B on the homework, gets a D-minus on the first two tests, and misses the third. She then comes to me and says, “Professor, your syllabus says the exam component of the grade is the average of my grade on the three tests—but I only took two tests, so that line of the syllabus doesn’t apply to my special case, and the only fair thing is to drop the entire exam component and give me a B for the course.”
I would laugh her out of the office. Or maybe suggest that she apply for a job as a state superintendent of instruction.
The saddest part is that I’m guessing Bennett sincerely felt he was doing the right thing. In his mind, he knew Christel was a great school, so if the scores said otherwise, the scores had to be wrong. In this respect, ironically, he ends up echoing his policy opponents, adopting the position that a mechanistic testing and scoring procedure can’t be allowed to override firsthand knowledge about teachers and schools.
Saying this out loud wasn’t an option, so any test scores that seemed to indicate learning problems at Christel had to be eliminated from the spreadsheet with extreme prejudice. (I attempted to reach both Tony Bennett and Jon Gubera and have not received a reply from either man. If I do hear back, I will add an update to this story.)
Bennett told AEI’s Hess, “I'm a track and field guy. I run, I try to keep my weight down at about 190. Christel has been a track-performing school for a number of years. If I get on the scale one day, am doing everything the same, and am still wearing my same clothes and they fit, and the scale suddenly reads 215, I am going to question what's going on.”
But Christel wasn’t doing everything the same. It didn’t have a 10th grade before, and then it did, and whatever it was doing to teach those new 10th-graders math, it didn’t lead them to pass algebra.
Bennett could, and should, have faced up to that fact, given the school the C it earned, and delivered them some honest tough love: “You’re still doing well at the stuff you’ve always done well, but you obviously haven’t succeeded at the new stuff you’re trying to do. We believe you can do it, but until you do, your grade’s going to suffer.” Isn’t that what accountability in education means, if it still means anything at all?
No comments:
Post a Comment