Week 5

[MUSIC PLAYING]

Funding for this program is provided by Annenberg Learning.

[MUSIC PLAYING]

[MUSIC PLAYING]

[MUSIC PLAYING]

Hi, I'm Pardis Sabeti. And this is Against All Odds, where we make statistics count.

There's no denying that Americans have a lot of opinions. Whether its their rating of this season's blockbuster, satisfaction with a particular food or product, or whether they approve or disapprove of a piece of legislation, it seems like we're always hearing about polling results. But with all these polls out there, how can you tell which ones are accurate? How do pollsters survey a population as large and diverse as that of the United States and wind up with a complete and unbiased picture of attitudes on a particular topic?

Every poll aims to reflect the opinions of a specific group of people. Sometimes, the group of interest is all registered voters, or it could be people in a particular age group, or perhaps households with a certain income. In statistics, a group of all individuals who share something in common is called a population.

The characteristic of a population that we're interested in is called a parameter. For instance, if we wanted to know the average cholesterol level of all adult males in the United States, our population of interest is American men, and the parameter is the mean cholesterol count. Of course, it would be impossible to find out the cholesterol level of every man in America. We usually don't know the true value of a parameter, since we can't examine the entire population.

We can, however, estimate an unknown parameter by taking a sample. A sample is a small slice taken from the entire population. If we calculate the average cholesterol level from a sample of American men, we identified a statistic. A statistic is a number calculated from a sample. And if our sample is representative of the whole population, we can infer to the statistic is representative of the parameter.

In our example, we assumed that the mean cholesterol count in our sample is representative of the mean cholesterol count of all men in the United States. These two numbers can often get confused, so mind your P's and S's. Parameters are for populations, and statistics are for samples.

--for the Granite State Poll, from the University of New Hampshire's Survey Center.

This month, the university is conducting a confidential study of politics and public opinion in--

Do you live in New Hampshire all year round?

Sir, are you on vacation?

What is their religious preference? Are you currently employed full time?

The pollsters at the University of New Hampshire Survey Center conduct everything from academic research surveys to political polls.

We want to make sure that you really stick to the script. It's very important that you get all of the words in there, exactly as they're written.

They know they can't contact every person within a population, so they're expert at taking samples, selecting a smaller number of people to represent the attitudes and opinions of the whole population. For their polls to be accurate, it's crucial that the sample they take is truly random.

If we're doing a public opinion survey, we use random digit dialing. And what we do is we work with organizations that actually can randomly generate samples of all of the working telephone exchanges and blocks in the state. So we start with a random sample of households.

There's even more to it, though. Because when we call a household, we're calling the house, we're not calling an individual within the house. But we can't talk to the house, we need to talk to an individual person.

I also have to ask, just to randomize the survey-- so could you please tell me of the adults aged 18 or older that live in your household, including yourself, who has had the most recent birthday?

That's a way of another stage of that sampling to go from the household, down to an individual level for the sample.

If a pollster were to only survey a group that was convenient, say just their friends or family, or only

collected data from people who volunteered to participate, like in an online or call in survey, that could create an unrepresentative sample, and therefore produce biased results. The same issue arises when a simple random sample draws from a list that excludes a portion of the population. This happened in 1936, when a Literary Digest poll predicted Alf Landon would be the next US president. Their prediction was way off, since Franklin Roosevelt won the election with 62% of the vote.

Why was their poll so wrong? Well, it turns out Literary Digest drew their sample from lists of car and telephone owners, items that at the time were much more expensive, and therefore indicative of wealth than they are today. The poll had effectively omitted the largely pro-Roosevelt poor from their survey, causing bias in favor of Landon. The whole population would have been registered likely voters, not just those who owned cars or phones.

Getting a representative sample is the cornerstone of accurate sampling. But just as important is carefully designing the survey itself. The questions must address the issue of interest, while trying to avoid confusion as much as possible.

There is an art to doing it. And a lot of it is common sense. Use simple words. Don't ask people about things that they're not likely to know about. Don't ask long questions that require explanations. Don't provide the respondent with information that you expect them to know.

Reputable pollsters are on guard against the problem of what they call non-attitudes, when someone might have no knowledge of the issue they are being asked about, but feel pressured to give some kind of answer. Experienced surveyors work to avoid additional polling pitfalls that can influence the responses they receive.

Simple things, changing the order in which you read responses to people, can change how people answer a question. Changing the order of questions, changing the wording of questions slightly.

You can see how thinking through a good survey is crucial to the process.

Most people, though, tend to think the data collection is the difficult part, because technically, it's probably the most daunting if you were going to try to do it yourself. But for us, that's pretty straightforward. The hardest part in survey research is understanding what are the research needs and the data needs of the client. And then at the end, after we've collected all of the data, we analyze the data and present in a way in which I believe, and my staff believe, accurately reflects the answers to the questions.

Remember that in a simple random sample, each individual in a population has an equal chance of

being selected. This can be hard to achieve in a real life survey, though, since it could be nearly impossible to get a complete list that includes every single member of a large population to draw from.

Another way of ensuring a representative sample is by doing a multistage sample. In this type of sampling, statisticians first begin with a random selection of large groups, and then go on to take smaller, random samples within these groups. For example, the survey center might begin with a random sample of counties across the state of New Hampshire. Then they would take a random sample of towns within those counties. Finally, they would select random households within the random towns.

The problem with multistage sampling is that it could leave out groups of interest merely by chance. To solve this problem, we move on to a third type of sample design, a stratified random sample. In this type of sample, the entire population is divided into groups with similar characteristics, or strata.

If we take our New Hampshire example again, we might this time divide the state into strata by census tract type-- rural, suburban, and urban. Then we would randomly select tracts from each stratum. By including randomly chosen representatives from our three different strata, we make sure not to overlook residents from any of the population density types.

Polling can help us and our government representatives understand the issues that Americans really care about.

Politicians want to make sure that they're doing what it is the public wants, more or less, so they can stay in office. But also, that's their job. The role of government is to try to satisfy public opinion, try to give the public what they want, as much as possible. And you really need accurate information to help you make those decisions.

So next time you get a phone call during dinner, maybe consider taking a moment to answer a few questions. For Against All Odds, I'm Pardis Sabeti. See you next time.

[MUSIC PLAYING]

Funding for this program is provided by Annenberg Learning.