The ETS report1 is based on two sources of data and a method of analysis that ETS refers to as the Angoff Method, after the ETS researcher who developed the method2. The Angoff Method involves collecting opinion data from a panel of "experts" and then combining the opinion data with standardized test statistical data in order to determine an "equivalent" score for some group of potential test takers. In this study, a separate panel was convened for each of six SAT II subject tests: writing, American history and social studies (one test), mathematics IC, biology, chemistry, and physics. The makeup of the group of potential test takers, who are the subjects of this study, were students meeting the minimum Board of Regents requirements for admission to the University System of Georgia.
The panelists (all GA high school teachers) were asked to provide three types of opinion data:
After analysis, ETS provided, for each subject test, the following:
Central to this method for establishing the "passing score" is the concept of a "minimally knowledgeable student". In its report ETS describes this concept:
"The minimally knowledgeable student is defined as the student with the minimum achievement level expected of a traditional student enrolling in the Georgia Board of Regents' college system. Specifically, enrolling students are expected to
For each study, the panelists were asked to conceptualize a subgroup of students who had just met the minimum requirements. The panelists were asked to make their item judgements based on this minimally knowledgeable subgroup."
Thus, the passing scores are ones that "could have been achieved" by the group of "minimally knowledgeable students" envisioned by the panelists, i.e., students with "C" grades in the subjects, and a 430 SAT-V and a 400 SAT-M.
After some training exercises, the panelists were given a subject test, and for each item asked to estimate the percentage of students in their imagined group of "minimally knowledgeable students" who would have given a correct answer. For some subject areas, the panelists evaluated only one test; in some cases they evaluated two. Based on the responses from the panelists, ETS then computed an "equivalent" score for each panelist, and averaged the "equivalent" scores to obtain the passing score.
Table 1 summarizes the ETS analysis for each of the six SAT II subject tests evaluated in the study. Panelist equivalent lists the "equivalent" scores produced by each panelist (except for writing, where the panelist equivalent listing is actually the raw score). Panelist SEM is the standard error of measurement estimated for the sample of panelist "equivalent" scores. Percentile is a value taken from Admission Staff Handbook for the SAT Program: 1996-97 (an ETS Publication), and it indicates, based on ETS data, where the recommended passing score would have been for students taking the SAT II tests in 1996-97. In other words, the listed percentage of students would have scored at that level or below on the corresponding SAT II.
| Writing | 550 550 |
-15 to 97 (raw scores) -15 to 99 (raw scores) |
33.7 32.7 |
42 42 |
| American History | 590 590 |
540, 550, 560, 570, 580, 590, 600, 610, 620, 630 530, 540, 550, 590, 590, 600, 600, 600, 620, 620 |
30.0 27.6 |
49 49 |
| Math IC | 550 510 |
490, 500, 510, 510, 520, 540, 540, 540, 560, 560, 570, 590, 600, 600, 630 470, 470, 470, 480, 490, 490, 500, 510, 510, 510, 510, 530, 550, 560, 570 |
33.8 33.2 |
39 25 |
| Biology | 510 460 |
450, 460, 510, 520, 570, 580 360, 410, 440, 500, 520, 540 |
27.6 29.1 |
20 9 |
| Chemistry | 570 | 510, 520, 530, 540, 570, 580, 600, 600, 660 | 29.5 |
36 |
| Physics | 650 | 620, 630, 630, 650, 670, 680, 700 | 29.3 |
57 |
While the passing scores for writing and American history were consistent between the two forms assessed, the scores for math IC and biology demonstrated quite significant variations. It should be noted that the reason only one test form was evaluated for each of chemistry and physics is that the panels were essentially the same for the two subjects. It would not be unreasonable to assume that the variability across forms observed for math and biology would also have been observed for chemistry and physics, if more than one form had been assessed.
This report contributed by Leon McGinnis - 7/30/97
| About HEIR ||| Home ||| Contents ||| News ||| Opinion |