SLOP, CRAP, and the NewsMax Condi Poll
I just clicked through an ad on a Washington Post e-newsletter to participate in a NewsMax online poll about the 2008 Presidential election and a possible bid by Condi Rice. There are two serious problems with this poll: it is a self-selected sample and to participate in the poll, you must provide an email address to subscribe to NewsMax’s news alerts.
Here is the intro text to the poll:
NewsMax.com, one of America’s leading online news services, is conducting an urgent national online poll.
NewsMax will provide the results of this poll to major media, Congress, the president and key members of his administration. Additionally, NewsMax’s results will be shared with every major radio talk show host in America.
NewsMax reports have been cited by national major media, including Fox News, CNN and MSNBC.
Don’t miss this opportunity to let your voice be heard! Many media outlets and national leaders are interested in your opinion. Hundreds of media outlets have reported on NewsMax’s online polls. Your vote does count!
NewsMax is conducting one of the first online polls on the 2008 race.
The media and others want to know your opinion. Vote today!
Now, let’s talk about what this poll is worth. First and foremost, the sample for this poll is self-selected. Self-selected opinion polls, or SLOP, have been consistently demonstrated in scholarly research to produce very flawed results. Respondents tend to be more extreme in their views and overall, do not represent the population at all. And without being representative of the population, the survey results cannot be used to say anything about the population.
The American Association of Public Opinion Researchers, which is the pinnacle organization among opinion researchers, has outlined in its Best Practices section that survey samples have to represent the population to be a scientifically valid measure of the population’s opinions:
Select samples that well represent the population to be studied.
A replicable or repeatable plan is developed to randomly choose a sample capable of meeting the survey’s goals. Sampling should be designed to guard against unplanned selectiveness. A survey’s intent is not to describe the particular individuals who, by chance, are part of the sample, but rather to obtain a composite profile of the population. In a bona fide survey, the sample is not selected haphazardly or only from persons who volunteer to participate. It is scientifically chosen so that each person in the population will have a measurable chance of selection. This way, the results can be reliably projected from the sample to the larger population with known levels of certainty/precision.
Critical elements in an exemplary survey are: (a) to ensure that the right population is indeed being sampled (to address the questions of interest); and (b) to locate (or “cover”) all members of the population being studied so they have a chance to be sampled. The quality of the list of such members (the “sampling frame”)whether it is up-to-date and complete is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed. Where a particular sample frame is suspected to provide incomplete or inadequate coverage of the population of interest, multiple frames should be used.
Virtually all surveys taken seriously by social scientists, policy makers, and the informed media use some form of random or probability sampling, the methods of which are well grounded in statistical theory and the theory of probability. Reliable and efficient estimates of needed statistics can be made by surveying a carefully constructed sample of a population, provided that a large proportion of the sample members give the requested information. The latter requires that careful and explicit estimates of potential non response bias and sample representativeness be developed.
SLOP samples do not satisfy this criteria, at all. This alone makes the results of NewsMax’s poll invalid.
But it gets worse. After filling out the survey, the last two questions REQUIRE that you provide your email address and ZIP code. It further states that, “I understand that as a voter in this poll I will be signed up for FREE breaking news alerts.”
So in addition to being a SLOP sample, anyone (including myself) who doesn’t want to subscribe to NewsMax’s alerts will not submit their answers. This further undermines the representativeness of the sample.
And the kicker is, aside from the obvious slant in the ad on the Post newsletter that lead with a picture of Condi Rice, there is no indication that NewsMax is an ultra-conservative media website.
So when you hear the conservative talk radio hosts reporting on the results of this poll that the sample is SLOP and the results are CRAP.