The surveys were Monitoring the Future (MTF), the National Household Survey on Drug Abuse (NHSDA), and the Youth Risk Behavior Survey (YRBS). The most basic differences in the surveys are in the three target populations being studied. MTF covers students in eighth grade (as well as tenth and twelfth). NHSDA covers teenagers who are not in a traditional high school setting as well as those who are. YRBS covers students in ninth and eleventh grades (as well as tenth and twelfth). If a researcher attempts to compare the three surveys without making any assumptions, he has only three options available for structuring an analysis. The first is to restrict the analysis to only students in two grades in 47 states. The second is to expand the analysis to include students in tabulations who are not surveyed in all surveys. The third is to expand the analysis to include students in all surveys within all age groups and all grades and circumstances, but include controls in the analysis for all the differences. Coverage denotes how well the sampling frames used represent the population being targeted. Each survey uses different frames to represent the different populations. A systematic comparison of the coverage of the surveys is needed, but one in which coverage issues are divided into listing/canvassing issues versus nonresponse issues. The sample designs for the three surveys differ radically from one another. The surveys are designed for multiple purposes. Weighting procedures also differ between each survey, and differ in substantial ways because of the differences in the sample designs. Each survey differs in the way it deals with weights produced at several levels. The recommendations are that a comprehensive coverage study should be conducted for each survey, and raw data should be tallied for the key items to be compared and cross-classified. However, any changes that would tend to bring estimates closer between the surveys may hurt the overall validity or reliability of each survey.
– Cowan, Charles D.