Experimental Design
Question: does the new flu vaccine work better than the old one?
Explanatory variable: The type of flu vaccine (old/new)
Response variable: Whether the subject got flu or not
Question: does working memory training improves one’s IQ?
Explanatory variable: Group (memory training/control)
Response variable: Improvement in the IQ test score
Does the explanatory variable
influence
the response variable
in the population?
Is there an association
between the explanatory variable
and the response variable
in our sample?
Is the association causal?
Does the association occur in the whole population, or only in the sample?
Is the sample representative?
How likely would it be to have the association in the sample if there was no association in the population?
Could there be some other explanation for the results?
This is called confounding
Confounding variables, or confounders
A study in the 1970’s found an association between the percentage of paved roads in a county and the number of diagnosed cancer cases per resident in the county. The sample consisted of a large number of randomly selected counties in the US.
Repetition:
Include a large number of diverse subjects, with wildly varying values of the confounding variables, to see if the effect you are looking for is present regardless of the values of the confounders. (Sample size)
Repeat the whole study again with a different set of subjects. It is likely that the confounders will have different values, so if the result is similar, it makes it seem more likely that the effect is due to the treatment. (Replication)
Control:
Fix the value of some potentially confounding variables, so that they do not vary and therefore cannot influence the results in different ways. (controlling the variables)
Include a control group with the confounding variables still present, but with no treatment, to see if the effect will show even without the treatment.
Randomization:
Randomize the values of the confounding variables that cannot be controlled. That way the influence of the confounders will be reduced to a random “noise”, which can be analyzed mathematically.
Select the sample randomly, so that even the variables that are inherently present in the subjects themselves will be randomized.
Special techniques for some special cases of confounding variables: blinding, double blinding, binning, …
Are the researchers doing the study actually able to use these techniques?
Do they have enough control over the variables to be able to either control or randomize them?
A doctor at a university affiliated hospital wants to do a study that compares two different procedures that are both designed to alleviate certain condition.
After securing all the necessary permissions, she randomly divides her patients into two groups, and performs one procedure on the patients in group 1, and the other procedure on the patients in group 2. She then compares the results.
A researcher at a College of Health and Human Services at certain university wants to do a study that compares two different medical procedures that are both designed to alleviate certain condition. They contact doctors at 200 different hospitals that perform these procedures, and ask the doctors to report the results of the procedures to them.
Experiment: The researchers have control over all or most of the possible confounding variables, and are able to use some techniques to minimize confounding. When comparing multiple treatments, they can decide which subject gets which treatment.
Observational study: The researchers do not have control over the possible confounding variables. They cannot decide how the subjects will be divided into groups. These decisions are either not made at all, or someone else makes them, for reasons other that the study itself.
In order to eliminate confounding, one needs to have a lot of control.
That makes it hard to have a large representative sample.
Can the study establish a causal relationship between the treatment and the effect?
Can the result of the study be generalized to the whole population?