Read Description Carefully
alikh86
random_controlled_experiments__statistical_studies__polls.pptxRandom Controlled Experiments & Statistical Studies & Polls
& Observational Studies, & Mill’s Methods
Statistical Studies & Surveys
See link under “Week 8” to “Statistical Studies & Surveys”, which shows how these are designed, and how to evaluate them. You are expected to read this article, and I will focus on some main points on research design, and the characteristics necessary to a sample.
RESEARCH DESIGN OF STATISTICAL RESEARCH:
CHARACTERISTIC OF INTEREST: the question being addressed, or in what the researcher is interested: EXAMPLE: What are the most popular television programs in the U.S.?
TARGET POPULATION: the population about which the researcher wants to make a generalization, to which the ultimate data from the research is going to be applied. EXAMPLE: Residents of the United States who watch television
SAMPLE: a portion of the target population who will be directly studied, or polled, and from which inductive generalizations will be made to the target population. Therefore, it is important that the sample represent all the important characteristics of that target population, and be big enough to do so. EXAMPLE: At least 1,000 randomly selected Americans who watch television.
Notice how many people is considered minimum necessary in order to make inductive generalizations that have any strength. See next slides.
Large enough sample size needed to yield reliable results.
See link under “Week 8” on Statistical Studies & Surveys”.
P. 123: “Small studies are less reliable than large studies. The typical behavioral study involves about 70 subjects. Owing to statistical error, studies routinely fail to confirm relationships that are known to be true for larger populations….Everyone knows that girls at age eighteen are taller than they are at age fourteen. In samples of only 70 subjects, this lawful relationship will be validated statistically less than half the time! Common sense dictates that findings should count more if they are based on 10,000 subjects than if they are based on 100.” [Frank Sulloway]
The above warning holds, despite what the text says on p. 126:
“1. What is the sample size? For national public opinion polls, it is a general principle that at least 1,000 randomly selected individuals who are representative of the target population will give reliable results.
When a research study involves carefully supervised testing and training of subjects or expensive materials…a much smaller sample might be optimal. For example, it is unreasonable and undesirable to have hundreds of subjects test an experimental drug that may be helpful in treating a particular disease but my have significant side effects.”
MY COMMENT ON ABOVE PARAGRAPH: Mathematically or statistically, the conclusions of any kind of study, including research studies, that have small samples, can never be definitive, regardless of the reasons that the study was kept small. At best, the conclusions are preliminary or suggestive, awaiting the possibility (and the justification of funding) of a larger scale study.
Larger scale studies can reverse the conclusions of smaller scale studies and anecdotes from clinical practices of doctors, despite the preliminary value of small scale studies, and anecdotes from doctors in generating hypotheses for larger studies. AN EXAMPLE: Many small scale studies and anecdotes from doctors supported the idea that hormone therapy for women past menopause (after they stopped menstruating and thus, stopped producing estrogen), helped not only with symptoms called “hot flashes”, but also helped prevent a variety of cancers and heart disease. However, a large scale study that was eventually done, reversed these findings. In fact, hormone therapy may contribute to certain cancers and heart disease, rather than preventing them.
More on sample size & margin of error
See the link under “Week 6” to “Correlation & Margin of Error”, the chart on “Sample size and Sampling Error” on p. 508.
This chart shows how, as the sample size increases in number, the “margin of error” decreases in number. Thus, a sample of only 100 people has a margin of error of plus or minus 11 percentage points, while the margin of error for a sample of 4,000 people has a margin of error of only plus or minus 2 percentage points.
What does this mean? If we polled 100 people on their views of the Affordable Health Care Act (or Obamacare), and 48% stated that they approved it, and 52% said they disapproved of it, we would have consider a span of 22 percentage points (plus 11 or minus 11 points) on either side of these figures. Thus, 48% approval could be as low as 37% or as high as 59%, and 52% disapproval could be as low as 41% or as high as 63%. In other words, we cannot reliably predict if more people approve or disapprove of the Affordable Health Care Act. The same problem could occur with a sample of 4,000 people, if the results of the poll are very close, even though the margin of error is much less, but it is more likely that the poll will be reliable. Thus, the span for 4,000 people is 4 percentage points (plus 2 or minus 2 points). Therefore, 48% approval could be as low as 46% and as high as 50%, while 52% disapproval could be as low as 48% and as high as 54%. The possibility remains that the ratings are in dead heat.
Because all samples have margin of errors, regardless of how small, there is “wedge of doubt” or uncertainty in any studies that rely on samples.
An attempt to interview the entire target population does not necessarily make the results certain. For example, how reliable is the United States census? Some groups of people, such as undocumented workers, might be underrepresented in the census, & there is nothing to insure that people answer all questions accurately. For ex., a late 20th century census reported that thousands of people claimed to be born in the 1800’s which was unlikely.
Representativeness or randomness of samples
See link under “Week 8” on Statistical Studies and Surveys.
“2. Is the sample representative in all significant characteristics and in the proportion of those characteristics? For example, if 10 percent of a state’s voters are age 65 or older, are 10 percent of the sample voters in this age range? If the sample is not representative, then the study is considered biased.”
“3. Have all significant characteristics been considered? Sometimes it is hard to know exactly which factors about the target population are significant. Does the sex and age of the target matter? What about ethnicity and educational level?”
MY COMMENTS ON THE ABOVE PARAGRAPHS: Representativeness of a sample can be achieved in two ways:
Manually, by trying to match the characteristics of those in the sample with those in the target population, which presents the problem mentioned in #3 above.
OR by randomly generating a sample, which means that supposedly everyone in the target population has an equal chance of being selected for the sample. This is supposed to insure that the sample automatically turn out representative. Today, randomness is usually computer generated, but, in the past, the following methods might be used: choose every third (or sixth or tenth, etc.) person in the telephone book or on some similar list. Juries have often been chosen this way.
Deliberately over-representing a group in a sample
Some polls will deliberately over-represent a group in its sample in order to insure that they have enough people in that group in the sample in order to avoid a large margin of error for that group.
For example, if we were doing a poll on attitudes towards terrorists in the United States, and we decided that one of the significant characteristics we should consider in the sample of people is their religion, we might try to increase the proportion of Muslims, even Jews and Hindus and Buddhists, etc. in our sample beyond their actual proportion in the target population. Why? Because Muslims only represent approximately 1 percent of the U.S. population, as is a similar case (low proportions) of some of the other religions I mention. If we have a sample of 1000 people, 1 percent of that sample would only be 10 people, which would be a very small group with a large margin of error. We can then either increase our sample to 10,000 or we can deliberately decide to include more Muslims in our sample.
If a survey or poll deliberately over-represents a group in a sample, this should reported in its explanation of how the poll was done.
More on how polls are done & evaluating them
See also the link under “Week 8” on Statistical Studies and Surveys.
Other factors to consider in evaluating how reliable a poll or statistical study is:
Are the questions biased? What is the order of the questions asked? (Did a poll ask about birth control before abortion or vice versa?)
What language is used in the poll? For instance, several years ago two samples of Americans were polled on their views on gays in the military. For one sample, the language “homosexuals” was used. For the other sample, the language “gays” was used. The approval rating was significantly higher (see later slides on statistical significance) for “gays” in the military, than for “homosexuals” in the military, for reasons unknown.
What is the credibility of the polling organization or research institution? Even credible organizations, such as “Gallup, Harris, and Roper”, and Pew, might have unreliable results if the response rate to questions was very low, or there were other problems in collecting the data. For example, rape statistics have often been unreliable because some people do not report rapes to the police. Sometimes different agencies, such as the FBI and Department of Justice, collect their data on crimes in different ways. It is important to find out how the United Nations collects its data, because it might make a difference if they rely on individual countries to report the data (in which case some countries, especially in conflict situations) might not be in a position to reliably collect data, or if the United Nations’ own agencies collected the data. Usually how data is collected, or how a poll was designed, including whether it was conducted by phone (and how many phones were landline and how many were cell phones), can be found online.
Is the survey biased because of the vested interest of the company that paid for it?
Date of the study. How recent is the study?
“Statistical studies cannot always be taken at face value.” p. 356, text
“Don’t assume that statistics are `facts’ or that their use results in a true proposition [statement] or strong inductive argument.” [p. 357, text]
Not all arguments that use statistics are inductive: Example of a deductive argument that uses statistics: (p. 352 text)
P1. 90% of the time, the Buffalo Creek banjo player plays “Hook and Line”.
P2. Anyone who plays “Hook and Line” is playing an African-American tune.
C: Therefore, the Buffalo Creek banjo player plays an African-American tune at least 90% of the time.
Compare the above to this inductive argument that uses statistics:
P1. 65% of the fiddle players polled in Kentucky said they like to play solo as well as with a band.
C: Therefore, 65% of all fiddle players in Kentucky like to play solo as well as with a band.
The gears shift in the inductive argument: “We went from a polled group in Kentucky to those across the state. That’s not the case in deductive arguments – even though we were using percentages there. The deductive argument has no such shift” [p. 352] because all the information in the conclusion is implicit in the premises. (See the power point presentations on “Deduction and Induction”.
Randomized Controlled Experiments
Randomized controlled experiments are the “gold standard” for determining cause-and-effect relationships, because we try to manipulate the factors or variables under laboratory conditions as compared to a base group, where factors are held constant, in order to test hypotheses as to what the cause might be.
Many experiments cannot be performed in a lab for a variety of reasons.
We cannot actually bring the moon into the lab in order to test how it acts, and we cannot eliminate it as a factor to see what happens if there is no moon.
We consider it unethical to perform certain lab experiments. For example, we do not deliberately have people smoke cigarettes so that we can see what happens to them. (Although the original cigarette research deliberately had beagles inhale “cigarettes” in the first stage of the research.)
Lastly, some experiments are better done in the field, because we do not know if, for example, monkeys will act differently in the field than they do when caged in a lab.
However, even in such observational studies, where we examine monkeys in the field, or smokers who are already smoking, we often use “control groups”, which act as substitutes for “replaying the history” of the individual. In other words, since we cannot observe what might have happened to any one individual if they acted differently (the problem of counterfactuals), we substitute a group of people who does act differently, and, use that group as a base of comparison, by which to infer what would have happened if the individual who smokes had not smoked.
Research Design of A Random Controlled Experiment
See the electronic reserves article, “Becoming a Critical Thinker”, which you are expected to read. I will focus on the elements of the design, and some of the criteria needed to evaluate research, especially those shared with statistical studies (on samples), alternative explanations, and statistical significance.
A good research design includes the following:
A question to answer: (characteristic of interest)
A hypothesis – this is a reasoned speculation “about what will be discovered from the research”, and, in this way, a type of opinion plays a role in science. Science can be said to be “hypothesis-driven” rather than simply “data driven”, especially because not every hypothesis is directly derived from the data that already exists.
A sample that, in addition to being randomly selected and representative of the target population, is divided into 2 groups:
A CONTROL GROUP: A group of subjects from the sample who get no treatment or a placebo (sugar pill).
AN EXPERIMENTAL GROUP: A group of subjects from the sample who are exposed to a special treatment called a variable; for example, this group would be given the drug to assess its effects in comparison with similar people who are not given the drug or who are given a placebo.
Data: The observations made by the researcher as he or she completes the study.
Conclusions: After the study is carried out, the researcher compiles the data and draws conclusions; the researcher interprets [involves inferences] the meaning and significance of the data. “In addition, researchers carefully consider the implications of the findings, which means that they will speculate about further research that can be done to answer questions related to the study.” [p. 162]. This is another way that reasoned speculation plays a role in science.
Criteria For Evaluating Research Findings – pp. 163 – 169 in “Becoming A Critical Thinker”
How large was the sample? (See previous slides) A sample that is too small leads to the fallacy (an error in reasoning) of hasty generalization.
Is the study reliable? (The sample must be representative. If it is not, this leads to the fallacy of biased statistics.)
“Are there alternative explanations for the findings? Have all the important factors of the data been considered? There might be explanations for findings in research other than the ones given by the researchers.”
Was a BLIND study conducted? In a blind study the participants are not told whether they belong to the experimental group or the control group. For example, they will not know whether they received the medication being tested or a placebo (sugar pill), so their expectations should not play a role in their reports as to how they feel.
Was a DOUBLE-BLIND study conducted? In a double-blind study, neither the experimenter nor the participants know which is the control and which is the experimental group. For example, the experimenters do not which group is receiving the real medication and which group is receiving the placebo. That way expectations of experimenters should not bias their reports of what they observe.
Criteria for Evaluating Research Studies (Continued)
Are the results statistically significant? “When a finding is labeled statistically significant, it is probable that the reported effect will occur again in similar circumstances.” [p. 167, “Becoming a Critical Thinker”]
Statistical significance depends, in part, on the size of the sample group, and, in part, the difference between the results in the experimental group and the control group.
As the sample size increases, the difference between the frequencies of lung cancer in smokers and in non-smokers can be less in order to be statistically significant.
Another example: the differences between the frequencies of improved health from those who take the medication and of improved health from those who only take a placebo must be very great if the sample size is very small, in order for the results to be statistically significant.
Although many students learn that statistical significance alone “proves” a cause-and-effect relationship, this is not so. Firstly, correlations can show statistical significance, and yet not be meaningful, especially with big data collected on the internet. [See the link under Week 6, “Correlation, Causation, & Confusion”.] Secondly, causal relationships are not, strictly speaking, “proven” or “demonstrated”, but are “supported” by the data or evidence.
Have other researchers been able to duplicate the results? This is one reason studies are published in peer review journals – in order to have them duplicated or replicated, and not because they have been proven true. Duplication helps confirm or deny that the reported effect occur again in similar circumstances. Variations on the research can test reasoned speculations made in the conclusions.
Criteria for Evaluating Research Studies (continued)
Does the researcher claim that the study proves more than it was designed to prove? This is often a problem with media reports of studies, which might exaggerate the findings, or treat correlations in the research as causes, even when the research does not claim to discover causes.
Has the research been done by a respected institution? “Be careful in your judgments, however, if research from one reliable source contradicts research from another reliable source;…” This is why it is helpful to do “literature reviews” to find out what other similar studies have been done.
Are the researchers biased? “Even if the research organization is well respected, it may have a vested interest in the outcome of the study.” All research is funded, and conflicts of interest can arise depending upon who funds the research.
Date of study: How recent is the study?
Mill’s Methods
Mill’s Methods are named after John Stuart Mill, who treated them as ways to discover cause-and-effect relationships. Actually, they only uncover correlations, but they can be useful, especially in combination with each other.
The reading assignments on “Mill’s Methods” for Week 8 are recommended, not required.
Mill delineated 5 methods, but we are only concerned with the first 3 methods.
METHOD OF AGREEMENT: Find what factor all similarly situated people or things, sharing the same effects, have in common. For example, what do all the people who show up at the hospital emergency room on the same with the same symptoms of extreme diarrhea and vomiting have in common? Perhaps they all ate at the same restaurant and perhaps they all ate lettuce at that restaurant. This is a correlation that suggests the possibility of food poisoning from contaminated lettuce as the cause of the symptoms. We could also test the lettuce for the presence of bacteria known to cause food poisoning, given we know what bacteria cause food poisoning. This was a problem for AIDS, before we knew about the HIV virus.
METHOD OF DIFFERENCE: It is not enough to find factors in common among those sharing the same effects, because we also have to explain why other similarly situated people did not display the same effects. For example, what if several hundred other people ate at the restaurant where those who had diarrhea and vomiting ate, but these other people showed no symptoms. What explains this difference? Perhaps, the other people ate no lettuce. Perhaps, if they did eat lettuce, it was from a different crate of lettuce that was not contaminated.
JOINT METHOD OF AGREEMENT AND DIFFERENCE combines the two above methods, in a way that is similar to a controlled experiment.
WARNING ABOUT THE ABOVE METHODS: They are only as useful as we are able to discover factors that are actually relevant. Most factors do not have signs, “I am relevant, look at me”. Thus, the joke: the alcoholic who uses Mill’s Method of Agreement, by saying he drinks scotch and water, rum and water, vodka and water, and thereby, discovers that water is the factor that is the cause of his problems.
