ACT, Inc., has identified the ACT scores at which 75% of students pass introductory level college courses and 50% earn a B or better. The graph above lists each ACT subject with its resulting "benchmark" score.
The dark bar shows the percent of Kentucky juniors who reached that score in 2008, when 100% took the ACT.
The light comparison bar shows results for ACT's last nationally representative sample of seniors. States vary in how many students participate, making the sample the closest we can get to seeing results that are not tilted by an overrepresentation of students who are already planning to attend college.
Sources: Kentucky juniors here, representative sample from Technical Manual here.
Susan,
ReplyDeleteI have a lot of questions about this table.
For one thing, comparing results across 13 years over a period when ACT scores rose both nationally and in Kentucky is highly problematic. How do you think this sort of comparison would look if we had a 2008 national representative sample?
Also, I am concerned about demographic issues. For example, I note with considerable concern that the ACT Tech Manual says in Table 4.2 that the 1995 norming sample was 55 percent female and 45 percent male. I don’t think the actual high school senior class makeup across the nation was that imbalanced in 1995, but so far I have been unable to check that. Incredibly, it looks like NCES stopped collecting enrollment data by sex after the mid-1980’s. I haven’t been able to find either enrollment breakouts by sex for the 1995 period in either the Digests of Education Statistics or the NCES Table Builder Web tool. That’s incredible. However, the latest Nonacademic Data Report from KDE shows that the racial difference in Kentucky for 1995 was only a point or two, something very different from the ACT sample proportions.
There is enrollment data available by race, however, and the minority makeup of the 1995 ACT sample looks very different from Kentucky’s racial demographics. That complicates fair interpretation of your data even more in ways similar to the problems of comparing NAEP data across states, which I have discussed in this blog and the Bluegrass Blog before.
On a technical note, would you please explain in more detail how you got your numbers? I can’t reproduce them using Table 4.10 in the ACT Technical Manual you link to and Table 2.1 in “ACT State Test Profile Report, Spring 2008 ACT-Tested Juniors, Kentucky.” I must be missing something.
Richard,
ReplyDeleteAre you saying that ACT did not use a credible sample in its norming study?
Here's an answer on the technical note.
ReplyDeleteFor the 1995 Norming Study, Table 4.10's columns show the percent at or below each ACT scale score.
For English, 56% scored 17 or below. That leaves 44% scoring 18 or above in English, as shown in my graph above. I used the same method for the other three benchmarks.
For 2008 Kentucky juniors, Table 2.1 shows percent at or below each score, so the same method would have worked. Figure 1.1 saved me the effort: my graph uses the numbers it reports as "Percent of ACT-Tested Students Ready for College-Level Coursework."
Susan,
ReplyDeleteYou asked, “Are you saying that ACT did not use a credible sample in its norming study?”
I can’t tell without 12th grade enrollment by sex information. So far, I cannot find that information in NCES documents or the Common Core of Data Explorer tool.
Male versus female dropouts would have to be incredibly imbalanced to make the ACT norming sample a good one. Notice that while the ACT sample is weighted 55% to 45% favoring females, that the same Table 4.2 in the ACT Technical Manual shows the US population is 49% female and 51% male for the same time frame. If the ACT sample is valid, this implies that male dropout rates had to be much higher than female rates – so high that some very serious questions about bias in the schools would be immediately raised.
BTW, thanks for explaining what you did to calculate the numbers in your figure. That makes sense. Still, what I mostly get from this is that our kids today do about as well as the rest of the country did 13 years ago. That doesn’t seem very revealing.