population: the larger universe in which you’re interested; all possible individuals to which you wish your conclusions to apply.
Sample: a representative subset selected from the population
Since the sample will be used to draw conclusions about the entire population, it must be representative of the entire population. That is, it should reflect as closely as possible the relevant characteristics of the population under consideration.
To try to insure that you have a representative sample, you should use some type of probability sampling methods. Probability methods:
· use some random methods to select units for the sample
· eliminate unintentional selection bias
· guarantee that you can apply inferential statistical techniques.
Simple Random Sampling: a sampling procedure for which each possible sample of a given size is equally likely to be the one obtained. The sampling is done without replacement (e.g., once an experimental unit has been chosen for the sample, it’s not eligible to be chosen again).
Example:
Suppose that our population is the top 5 state officials in Oklahoma:
Governor (G)
Lieutenant Governor (L)
Secretary of State (S)
Attorney General (A)
Treasurer (T)
We would like to select a sample of size 2 from this population. Let’s list all possible samples of size 2:
G,L G,A L,S L,T S,T G,S G,T L,A S,A A,T
In order to choose a simple random sample, we need a sampling method that will insure that each pair is equally likely to be selected. For example, you could write each pair on a slip of paper; place the pieces of paper in a box; and draw out one of the pieces of paper to be used as the sample. Each sample is equally likely to be selected with the probability of selection being 1/10th.
So, how would you select a simple random sample from a larger population
where it might not be practical to enumerate all possible samples of a
particular size?
Selecting a Simple Random Sample:
1. Label – Assign a numerical label to every experimental unit in the
population.
2. Use random digits to select labels for the sample at random.
You can do this by using a random number table or a random number generator
(e.g., M-lab).
Example:
Suppose that we want to know the bacteria levels in vats that we use to make ice cream. There are a total of 50 vats (our population). We can only sample 5 of the 50 vats. So, we need to select a simple random sample of size 5.
Using a random number table:
1. Label the vats from 1 to 50.
2. Use random digits to select the sample.
Turn to Table F in your textbook. Select a starting point.
For this example, we’ll start with the 5th row and 4th column. You’ll
see the following list of numbers:
… 37431 09723 63529 93977 95218 96074 42138 …
Since our largest possible number is 50 (a 2-digit number), we could just use the last 2 digits of each 5-digit number above. We’ll ignore the numbers 77 and 74 since they’re too large for use in our sample. That leaves us with the following numbers for our sample:
31 23 29 18 38
So, select vats 31, 23, 29, 18, and 38, and test the bacteria levels in each of those vats.
Alternatively, you could use the random number generator in M-lab:
To start M-lab, type the following at the UNIX prompt:
>> add st370_info
>> matlab
>> load st370data
These statements will load MATLAB and make the M-lab sample functions available.
To select a sample of size 5 from a population of 50, use the following commands:
>> rand(‘state’, 145)
The second argument can be any integer. This statement
simply sets the seed for the random number generator.
>> sample(50,5)
The first argument here is the population size. The second
argument is the desired sample size.
If you submit these statements, you will get the following output:
17 43 37 49 1
You would then select vats 17, 43, 37, 49, and 1, and you would test
the bacteria levels in those 5 vats.
Observational Studies vs. Designed Experiments
In an observational study, researchers simply observe characteristics and take measurements (e.g., a sample survey).
In a designed experiment, researchers impose treatments and controls
and them observe characteristics and take measurements. The goal
is to see how individuals respond to some stimulus.
Experimental Design
Experimental units: subjects for the experiment
Treatment: a specific condition which is applied to an experimental unit
If there is more than one treatment to be applied to the experimental units, we usually refer to the treatments as factors. The different values or settings of a factor are called levels.
The researcher can control or set the levels of a factor. His
goal is to determine whether different levels of the factor can explain
some kind of reaction in the experimental units. The variable measured
to judge the reaction is called the response.
Example:
An alloy was manufactured using one of 2 methods (Method 1 and Method 2). Samples of the alloy were then sent to one of two labs (Lab A and Lab B) which analyzed the composition of the samples to determine the percent of iron in the alloy.
Factors: manufacturing
method, laboratory
Levels: Manufacturing
method has 2 levels – Method 1 & Method 2
Laboratory has 2 levels – Lab 1 & Lab 2
Response: percent of iron in the alloy
Example:
An electric mixer is used to mix pancake batter. You’re interested in measuring the smoothness of the batter. You use two different speed settings (slow and fast) of the mixer to see which one gives a smoother batter. You also decide to test several different brands of mix. You choose two major brands (Aunt Jemimah and Hungry Jack) and two store brands (Food Lion and Kroger).
Factors: speed setting,
brand
Levels: Speed
setting has 2 levels – slow & fast
Brand has 4 levels – Aunt Jemimah, Hungry Jack, Food Lion, Kroger
Response: smoothness of the batter
Basic Principles of Experimental Design
1. Randomization
2. Replication
Randomization: experimental units must be randomly assigned to the treatments
Example:
Suppose that you work for a company that makes paint. You believe that your paint dries faster than another brand. In order to prove this, you take 8 plywood boards and paint them. You paint 4 with your paint and 4 with the other brand. You then measure the time for each board to dry.
The treatment or factor for this experiment is brand of paint. You need to randomly assign 4 boards to your brand of paint and 4 boards to the other brand of paint.
You could draw a sample of size 4 from a population of size 8. You would then assign the 4 boards that you selected to your brand of paint and assign the remaining 4 boards to the other brand of paint. However, this method only works for the case when your treatment has exactly 2 levels.
There is an alternative method that will work for a treatment with more than 2 levels.
1. Number the boards from 1 to 8.
2. Use a random number generator to get a random ordering (permutation)
of the integers 1 to 8. The first 4 numbers will be assigned to treatment
level 1 (in this case, your brand of paint) and the second 4 numbers will
be assigned to treatment level 2 (the other brand of paint).
To implement this method using M-lab, you will need to submit the following code:
>> rand(‘state’,292) The second
argument can be any integer. This statement
simply sets the seed for the random number generator.
>> sample(8)
The argument here is the total number of experimental
units that you have available.
These statements generate the following output:
3 7 8 5 1 2 6 4
So, boards 3, 7, 8, and 5 will be painted with your paint, and boards 1, 2, 6, and 4 will be painted with the other brand of paint.
Now, suppose that you had 12 boards and 3 brands of paint. To randomly assign the 12 boards to the different brands of paint, you would issue the following commands in M-lab:
>> rand(‘state’,478)
>> sample(12)
These statements generate the following output:
5 9 8 12 3 11 4 7 10 1 6 2
So, boards 5, 9, 8, and 12 will be painted with paint 1; boards 3, 11,
4, and 6 will be painted with paint 2; and, boards 10, 1, 6, and 2 will
be painted with paint 3.
Replication: assigning more than 1 experimental unit to each treatment.
The idea behind replication is that more data equals more information.
If you use only 1 experimental unit per factor/level combination, then
a single unusual observation can strongly affect your conclusions.
With more than 1 observation per factor/level combination, it’s easier
to distinguish real differences between treatments from differences that
might be due to unusual experimental units.
Common Experimental Designs
· Completely Randomized Design
· Randomized Complete Block Design
· Factorial Design
Completely Randomized Design
· Simplest type of experimental design
· Experimental units are randomly assigned to each treatment
Example:
Suppose that we have four different gasoline additives, and we want to compare the increase in gas mileage when they are used. We have the resources to conduct 20 trials using the same vehicle repeatedly. Think of each trial as an experimental unit, and use a completely randomized design to conduct the experiment. That is, randomly assign each of the 20 trials to one of the 4 additives.
NOTE: It’s extremely important to use randomization. Differences between the experimental units (in this case, the trials) may exist whether or not you recognize them before you conduct the experiment. For example, in this experiment, what if the trials are conducted repeatedly with little time in between? If the vehicle’s gas mileage decrease as the oil gets dirtier, we’ll probably notice a trend of decreasing gas mileage across the 20 trials. If we had assigned additive 1 to trials 1-5, additive 2 to trials 6-10, etc., then we would introduce bias into the experiment. Using randomization removes the possibility of bias.
So, suppose that we carry out the experiment using a completely randomized
design. The data that you collect might look like this:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Average: |
|
|
|
|
So, additive 2 appears to yield a slightly higher gas mileage.
Based on this experiment, have we actually shown that using an additive
increases gas mileage? NO
We’ve only compared the four additives. To test whether or not
using an additive yields an increase in gas mileage, you must also test
the case where no additive is used. That is, the experiment needs
a control.
Control: “no treatment” – placebo
Now, suppose that we have 2 cars – a Ford and a Chevy, and suppose that we run the experiment by using additives 1 and 2 in the Ford and additives 3 and 4 in the Chevy. In this case, it will be very difficult to say whether, for example, additive 1 gives better gas mileage than additive 4. Differences in gas mileage may be due to the additives. However, they may also be due to differences between the 2 makes of car. This phenomenon is called confounding.
Confounding: the effects of different variables on the response cannot be distinguished from each other.
Blocking is one way to remove the effects of confounding from an experiment.
Randomized Complete Block Design
Example:
Suppose that we have iron ore mined from 3 different locations. We would like to test 5 new types of acid. Some measure of reactivity will be used when the acid is introduced to the ore. We would like to know which acid is most volatile. The iron ore from each location is found to have different impurities that may affect the outcome of our experiment. Since we have a limited amount of ore from each location, we need to use the ore from all three locations. Therefore, we will not have identical ore samples.
We can’t consider location as a treatment since we can’t randomly assign ore samples with different impurities to different locations. Therefore, we will keep the ore samples separated into their three location groups. Next, within each location, we will randomly assign each of the ore samples to one of the 5 acid types (e.g., the randomization will take place within the blocks). So, location will be our blocking variable and acid type will be our treatment.
The ore samples for this experiment might be assigned as follows:
Acid 1 Acid 2 Acid3 Acid 4 Acid 5
Location 1 4
5 2
3 1
Location 2 5
4 1
3 2
Location 3 3
4 1
5 2
Our main focus will be on the differences between the acids.
Within each block, the experimental units are homogeneous, while the experimental units from 2 different blocks may have inherent differences. This design controls for differences in blocks because all of the outcomes within a block are affected in the same way. And, each treatment level is applied within each block. So, differences between locations tend to cancel out.
Advantages of Blocking
· Helps control for the effecting of “lurking” variables
· Provides a way to add replication
· Allows the results of the experiment to be applied to a larger
population
Factorial Design
In a factorial design, there is more than one factor or treatment that we wish to examine.
Example:
Suppose that you are interested in baking biscuits, and your goal is to bake the fluffiest biscuit. You could measure “fluffiness” by measuring the height of the biscuit. You can control 2 factors when baking the biscuits – baking time and baking temperature. You decide to try 3 different baking times – 8, 11, and 14 minutes. And, you decide to try 3 different oven temperatures – 350, 375, and 400 degrees.
A full factorial design will look at all possible factor level combinations. For this example, there are a total of 9 possible treatment combinations.
In general, if you have 2 factors – Factor A at a levels and
Factor B at b levels, then there will be a x b possible
treatment combinations.
Analysis of Variance
So, how do we analyze data from a designed experiment? We use Analysis of Variance (ANOVA).
The goal of an ANOVA is to determine whether or not there is a significant difference between the means of the different treatment groups. So, why is it called analysis of variance?
We’re really asking the question:
Are the differences between the sample means really due to differences between the treatment groups OR are the differences simply due to variation within the groups?
Example:
o o ooo
o
Group 1: -----+-----+-----+-----+-----+-----+-----+-----
group 1 mean = 20
o o o
o o o
Group 2: -----+-----+-----+-----+-----+-----+-----+-----
group 2 mean = 25
0 10 20
30 40 50
60
For this situation, there is no real difference between the 2 group
means due to the large variability within the groups. However, also
consider the following situation where there is the same absolute difference
between the group means.
o
oo
ooo
Group 1: -----+-----+-----+-----+-----+-----+-----+-----
group 1 mean = 20
o
oo
oo o
Group 2: -----+-----+-----+-----+-----+-----+-----+-----
group 2 mean = 25
0 10 20
30 40 50
60
For this situation, there is a real separation between the 2 group means. There is small variability within the groups and large variability between the groups.
So, we’re effectively comparing the variability within each treatment group to the variability between the treatment groups. If we large separations (variability) between the groups and small variability within the groups, then we can conclude that there is a real difference between the treatment means.
See ANOVA example using the jet data set in M-lab 3.
