Attachment No. 5
The SAT II Tests Cannot be Used to Establish "Grade Equivalent" SAT II Scores Cannot be Determined Using the Angoff Method
The Board of Regents contracted with ETS to create a "passing score" for the SAT II tests, to be used as an admissions criterion for homeschoolers applying to the University System of Georgia. The intention, as described by Dr. Fullerton to the Georgia House Education Committee on July 29, 1997, was that the passing scores would correspond or be equivalent to an average grade of "C" in the college preparatory curriculum. The ETS report, however, carefully avoids claiming that ETS has determined a "grade equivalent" SAT II score. Rather, ETS has been very careful to describe their results as a recommended "passing score". While subtle, the distinction is quite important--the professional staff at ETS know that they cannot determine a valid, verifiable "grade equivalent" score.
When viewed as "grade equivalent" scores--which is the way that Dr. Fullerton has described them--the ETS recommendations are not technically valid.
The Angoff method, as used in this case, addresses two groups of students, one a group of graduates of accredited high schools in Georgia who are "minimally qualified" for admission to the University System of Georgia, and the other the group of all students who actually have taken the SAT II subject tests. The data used for the analysis presented here comes from two sources:
To understand the basic capabilities of these two groups of students, consider how the two groups perform on the SAT I test--widely accepted as the most reliable single-factor predictor of college success. As defined by the Board of Regents in the instructions to the panelists , the group of "minimally qualified" students score 430 on the verbal part of the SAT I and 400 on the math part, for a composite score of 830. An 830 composite corresponds to a 19th percentile score, which means that for a single student scoring an 830, only 19% of all test takers score lower, and 81% of all test takers scored higher. For the group of students who actually have taken the SAT II subject tests, the College Board provides data on how they performed on the SAT I test. For students who actually have taken at least one SAT II test, the median composite score was 1195, or roughly the 79th percentile of all SAT I test takers. For a student scoring 1195 on the SAT I, only 21% of test takers scored better. In other words, there are 60 percentile points separating the imagined "minimally qualified" student, and the average student actually taking at least one SAT II test. As Dr. Fullerton stated to the House Education Committee, "Only very, very good students take the SAT II tests." To appreciate just how large this gap is, if a student took the SAT I and achieved an 830 composite, and then took it again and achieved an 1180 composite, ETS would automatically investigate the second score as potentially fraudulent.
Keeping in mind the fundamental differences in the ability of these two groups of students, and the fact that the "minimally qualified" student achieved only a "C" average in the college prep curriculum (they might have scored below a C in some courses), consider the ETS recommended passing scores. (For some tests the report lists two scores but not how to reconcile them, so an analysis is given for each score.)
It does not appear rational to suggest that a minimally qualified student as defined by the Board of Regents could outperform almost half the "very, very good students" who actually take the SAT II in American history and 57% of those who actually take the SAT II in Physics. The same observation could be made for the recommendations for English writing and Chemistry.
We understand, from Dr. Fullerton's presentation to the House Education Committee, that the Board of Regents staff recognizes the recommended passing scores are likely to be unachievable for large numbers of students, so they have decided to adjust the passing scores downward. They have made a decision to adjust down by one standard error. As Dr. Fullerton stated to the Committee on July 29, the Board has no plans to validate any of these recommendations using actual test performance data.
Even if it were possible to establish a "grade equivalent" score, the Board has implemented its policy for homeschoolers on a "minimum" rather than "average" basis. In other words, while graduates of accredited programs need achieve only an average grade at or above the standard, homeschoolers would have to achieve every grade at or above the standard. Thus, homeschoolers are treated differently, and the standard for them is more stringent.
The Board appears to have established an arbitrary standard, with no intention that it be validated. It is not fair to homeschooling children. There has been no role for the homeschooling community in the process of establishing the policy, homeschoolers have not been informed about the policy in a timely way, nor have they been given any meaningful way to present arguments opposing the policy or suggesting improvements. Dr. Fullerton informed HEIR that the Board of Regents does not hold public hearings, and it does not confer with special interest groups. Despite having been in contact with Dr. Fullerton's office on an almost weekly basis since late March, his statements to the Committee on July 29 regarding the adjustment to the ETS recommendations, and the change in requirements for the language requirement were a complete surprise.