Years ago, I taught Intro to Statistics college classes. I used to look for misuses of statistics to share with my class, and my radar is still out for them. Last night, a post on Twitter led me to a prime example in Education Week: “Early Achievers Losing Ground, Study Says.”
Here’s basically what the article says: They did a study of kids scoring in the 90th percentile or above in math or reading in third grade, then checked how many of those kids were in the 90th percentile or above in eighth grade. Then they drew some wild conclusions from the results, apparently not understanding how percentiles work.
Here are the results:
“Tracking the individual scores of nearly 82,000 students on the Measures of Academic Progress, a computerized adaptive test, the study found, for example, that of the students who scored at the 90th percentile or above in math as 3rd graders, only 57.3 percent scored as well by the time they were 8th graders. The MAP test was developed by the Northwest Evaluation Association, a nonprofit group based in Portland, Ore. As an adaptive test, its difficulty is adjusted to the student’s performance.
“Analysis of MAP scores in the study also found that of more than 43,000 6th graders who scored in the top tenth on the reading test, only 52.4 percent were scoring as well as 10th graders.”
Okay, those results are given clearly. But I claim that they by no means allow you to conclude that we are not serving gifted kids adequately.
Suppose you have a group of 100 students. At the start of a month, you test them on their basketball skills and rank them. The top ten students probably have some exposure to basketball, and there are some in the bottom ranks who have never touched a basketball before. Now give all of them a month’s intense basketball training. Is it reasonable to assume the top ten students will not change? What if some kids who never touched a basketball before end up incredibly talented? And that’s not to say the original top ten students aren’t still great at basketball or haven’t been served. It’s just that they aren’t still necessarily the ten best. If some of them went down in rank, someone else by definition moved up.
Because that’s how percentiles work.
You could use this same study to say, “Hooray! Our school system is doing great! Some kids who were not in the top tenth compared with others have now risen to that level!”
Look at this paragraph:
“‘Is helping kids at the bottom improve hurting kids at the top?’ he said, acknowledging that bringing up that point as a topic of discussion can be difficult, but arguing that it’s necessary. ‘Let’s be honest about the trade-offs. It doesn’t make you a bad person or a racist.'”
Let’s change his question slightly. “Does helping kids at the bottom get into the top ten percent mean that some of the kids who were in the top ten percent before now have a lower ranking?” Umm, yes. That’s a matter of math, not sociology.
The article actually says:
“The new study also found that while some high-achieving students faltered, other students developed into high performers as they got older, although those students were likely to have scored between the 50th and 80th percentiles in the first place. In addition, many of the initially high-achieving students whose test scores fell below the 90th percentile after a few years didn’t fall far. Many scored in the 70th percentile or higher years later.”
Did this actually surprise them? That’s the definition of percentiles. It’s just how you rank compared with the other students. If some at the top go down, that means others have gone up. In fact, maybe now EVERYONE is achieving really well. It doesn’t say anything at all about the top kids doing worse.
The following paragraph made me laugh out loud:
“The study, “Do High Flyers Maintain Their Altitude?,” builds on a previous report from Fordham that suggests nationwide policies aimed at making schools more accountable for improving low-performing students’ achievement are hurting the brightest students. That 2008 report found that from 2000 to 2007, achievement for students who were the highest performers on the National Assessment of Educational Progress was flat, while the lowest-performing students improved dramatically.”
If you’re measuring achievement by percentiles, how can you expect the scores for the highest performers to be anything but flat or dropping? What did you want? For them to reach the hundred-and-fifth percentile? Are they going to start giving extra credit on the SAT so the top scorers can improve?
Take my son as an example. When he was five years old, he learned that he could prolong bedtime indefinitely with the magic words, “Just one more math problem, Mommy, please?” I absolutely did not have the power to resist those words. So he learned to multiply before he entered first grade, and could multiply two-digit numbers in his head.
No surprise, he was tested in first grade in the 99.9th percentile in math. Other kids didn’t have such crazy parents. Now that he’s a senior in high school, if he’s not still in the 99.9th percentile in math, can I say the schools haven’t served him well? Nonsense! Other bright kids have had a chance to catch up. So he still does well, but not necessarily at the tip-tip-top. (And do standardized tests even really provide valid tests for the very top students? That’s a whole other question.) Less than the 99.9th percentile does NOT mean he’s achieving at a lower level in math than he was in first grade!
Here’s another statement made in the article that seemed a completely invalid conclusion from the study:
“NCLB’s emphasis on getting all students to reach proficiency on math and reading tests may have a negative effect on high-achieving students, he suggested, especially when combined with other policies such as those that encourage more students, regardless of their academic records, to take Advanced Placement courses. Teachers working with students with a mix of abilities, he said, may not be able to cover as much material or in as much depth as they might if a majority of students in a class are high-performing.”
Excuse me? In the first place, eighth grade scores don’t have anything at all to do with Advanced Placement courses. But let’s think about it more deeply. Couldn’t you just as well cheer that kids who were at lower percentiles are now breaking into the top?
Again, let’s look at an example. When I was in high school, many years ago, not too many students took AP Calculus. I did, and it gave me a nice big advantage on standardized tests and math competitions. Other students just as bright as me may not have had the opportunity to learn as much, so they may not have scored as well. For my sons, taking AP Calculus is much more common, so it’s not going to give them as much of an advantage. Does this mean they’re not as smart as I was? I certainly don’t think so!
With the study based on percentiles, all it’s really saying is that the group at the top has changed from third grade to eighth grade. The overall level may be much higher; we don’t know. All we know is that the ranking has changed. There are many, many factors that may have gone into this change of rankings. Maybe we’re not serving the gifted well. But maybe we’re doing a really great job with the late bloomers. There’s no way to tell by comparing percentiles. If some go up, others will go down, even if they are achieving just as well as ever.
I was ranting about this to my son, enjoying someone listening who understood what I was saying. He came up with another good example. It’s like baseball. Years ago, there were many outstanding performers who had much higher batting averages than anyone else. But over the years, everyone has gotten better, so all the batting averages are higher, and great performers don’t stand out quite as much — because the overall level has gone up. He’s currently taking AP Statistics, and he said, “The average has gone up, but the standard deviation has gotten smaller.” (That’s my boy!)
What would we conclude if the study had gone differently? What if we found that the exact same kids in the top tenth in third grade were also in the top tenth in eighth grade? That would not necessarily mean we were serving those children well. I think you could make a stronger case that we were being elitist and providing the best resources to those who bloomed early. We were deciding who was smart early and teaching them the most. It might support the idea that some kids are born “gifted,” and you can’t change that with teaching.
Now, don’t get me wrong — I believe strongly in differentiated gifted education. I think you should make advanced classes available to those who are ready for them.
I’m just saying be careful what conclusions you draw from statistics. Are they really saying what you claim they are saying? Look at the facts from several different angles.
Remember, when you’re working with rankings or percentiles, for some to go up, others absolutely must go down. Everybody can’t be above average. But the average can go up. And that is not something percentiles will ever show.
Mathematics 1001
