Version 1 and Version 2 both show 100 students entering ninth grade.
In Version 1, 40 of the students are gone by eleventh grade, either as official dropouts or as students the school only knows are no longer there. In Version 2, only 20 are gone.
Now look at proficiency on eleventh grade testing. 15 of the remaining 60 students in Version 1 get there, or 25 percent. 20 of the remaining 80 percent get there in Version 2, also 25 percent.
And yet, I hope it's obvious that Version 2 is noticeably better. More students are still in school, and more students are proficient or distinguished.
I've added a wrinkle by showing that of the students who are not proficient, many more are in the apprentice category in Version 2: that's another way results can be better while percent proficient and above is unchanged.
The main idea I want to share, though, is that if a school simultaneously increases the number of students reaching proficiency and the number of students staying in school, the percent proficient could stay exactly the same. The report could be "no progress," when the reality was better results for an important number of students.
Ans the main thing I wish I could figure out is: what approach to the data would allow us to see if that, indeed, is a factor in the sluggish improvement of high school achievement defined in percentage terms?