Probability of Selecting a Representative Teacher Sample

• In Event: Feasibility of Teacher-Centric Analyses Using NAEP Data

Abstract

The purpose of this section is to determine whether all fourth grade teachers were sampled in each NAEP-sampled school with sufficient probability to assume a representative sample. If all teachers in each school were sampled, then NAEP contains a nationally representative sample of fourth grade teachers by virtue of its representative sample of schools. Analyses of teacher data can then be generalized to the population of all U.S. fourth grade teachers using sampling weights.
The NAEP assessments are not designed to sample teachers. According to NAEP technical documentation, teacher data are not from a representative sample of teachers. Therefore, variables derived from the teacher questionnaires are analyzed and reported only at the student level. If, however, we can demonstrate that all teachers were sampled in each sampled school, then NAEP would be similar to other NCES surveys which contain a nationally representative sample of teachers such as SASS or ECLS-K:98.
Typically for NCES surveys the percentage of sampled teachers is calculated in reference to the frame list of teachers working in each school. This was the approach taken in ECLS-K:98, which required each sampled school to provide a comprehensive list of kindergarten teachers. Unfortunately, NAEP does not gather frame data on teachers. NAEP does, however, gather information on the fourth grade student-teacher ratio. When this ratio is combined with the total number of fourth grade students (available from CCD), it is possible to estimate the total number of fourth grade teachers in each school. This number can then be used to calculate two further statistics of interest for each school: the percentage of teachers who were sampled in a given school, and the probability that all teachers in that school were sampled. As shown in Table 1, on average there were 3.4 teachers sampled per school. This number is similar to the average number of all fourth grade teachers per school (3.6). The discrepancy at the national level between the number of fourth grade teachers in NAEP and in CCD (0.2) is likely due to the imprecision involved in estimating the total number of teachers using CCD. In spite of this discrepancy, the number of sampled teachers per school and the number of all teachers per school are quite close. This proximity provides evidence that NAEP tended to sample all fourth grade teachers in each school.
Combining the estimates from Table 1 with the numbers from Table 2, it is also possible to calculate the probability that NAEP sampled all fourth grade teachers in each school. Given the relatively low number of fourth grade teachers in sampled schools (about 3.4 on average) and the relatively high number of sampled students per school (23 on average) the probability that any given teacher not included in the sample in a school is close to zero (assuming that the fourth grade teacher/student ratio is relatively constant within the school). The corresponding probability that all teachers in each school were sampled is close to 100% for all categories shown in Tables 1 and 2.

Author

• Austin Lasseter, American Institutes for Research