Statics

Week 2 Lecture 1

Examining Differences

Last week we collected clues and evidence to help us answer our case question about males and females getting equal pay for equal work. As we looked at the clues presented by the salary and comp-ratio measures of pay, things got a bit confusing with results that did not see to be consistent.

We found, among other things, that the male and female compa-ratios were fairly close together with the female mean being slightly larger. The salary analysis showed a different view; here we noticed that the averages were apparently quite different with the males earning more on average. Contradictory findings such as this are not all that uncommon when examining data in the “real world.” One issue that we could not fully address last week was how meaningful were the differences? That is, would a different sample have results that might be completely different, or can we be fairly sure that the observed differences are real and show up in the population as well?

This week, our journey takes us to ways to explore differences, and how significant these differences are. Just as clues in mysteries are not all equally useful, not all differences are equally important; and one of the best things statistics will do for us is tell us what differences we should pay attention to and what we can safely ignore.

Side note; this is a skill that many managers could benefit from. Not all differences in performances from one period to another are caused by intentional employee actions, some are due to random variations that employees have no control over. Knowing which differences to react to would make managers much more effective.

Differences

In general, when measuring related or similar things, we have two types of differences and need to determine which we are working with. Before looking at that we should examine why do we even have differences on things that are the “same?” One reason is that variation occurs. Variation is simply the differences in things that occur in this world. If we can measure things with enough detail, everything we do in life has variation over time. When we get up in the morning, how long it takes to get to work, how effective we are at doing the same thing over and over, etc. Except for physical constants, we can say that things differ and we need to recognize this.

This suggests that any population that we are interested in will consist of things that are slightly different, even if the population contains only one “thing.” Males are not all the same, neither are females. Manufactured parts differ in key measurements; this is the reason we have quality control checking to make sure the differences are not too large. So, even if we measure everything in our population we will have a mean that is accompanied by a standard deviation (or range). Managers and professionals need to manage this variation, whether it is quantitative (such as salary paid for similar work) or even qualitative (such as interpersonal interactions with customers).

The second reason that we are so concerned with differences is that we rarely have all the evidence, or all the possible measures of what we are looking for. Having this would mean we have access to the entire population (everything we are interested in); rarely is this the case. Generally, all decisions, analysis, research, etc. is done with samples, a selected subset of the population. And, with any sample we are not going have all the information needed, obviously; but we also know that each sample we take is going to differ a bit. (Remember, variation is everywhere, including in the consistency of sample values.) If you are not sure of this, try flipping a coin 10 times for 10 trials, do you expect or get the exact same number of heads for each trial? Variation!

Since we are making decisions using samples, we have even more variation to consider than simply that with the population we are looking at. Each sample will be slightly different from its population and from others taken from the same population. How do we make informed decisions with all this variation and our not being able to know the “real” values of the measures we are using? The answer lies with statistical tests. They can use the observed variation to provide results that let us make decisions with a known chance of being wrong! Most managers hope to be right just over 50% of the time, a statistical decision can be correct 95% or more of the time! Quite the improvement.

The use of samples brings us to a distinction in summary statistics, between descriptive and inferential statistics. With one minor exception (discussed shortly), these two appear to be the same: means, standard deviations, etc. However, one very important distinction exists in how we use these. Descriptive statistics, as we saw last week, describes a data set. But, that is all they do. We cannot use them to make claims or inferences about any other larger group.

Making inferences or judgements about a larger population is the role of inferential statistics and statistical tests. So, what makes descriptive statistics sound enough to become inferential statistics? The group they were taken from! If we have a sample that is randomly selected from the population (meaning that each member has the same chance of being selected at the start), then we have our best chance of having a sample that accurately reflects the population, and we can use the statistics developed from that sample to make inferences back to the population. (How we develop a randomly selected sample is more of a research course issue, and we will not go into these details. You are welcome to search the web for approaches.)

If we are not working with a random sample, then our descriptive statistics apply only to the group they are developed for. For example, asking all of our friends their opinion of Facebook only tells us what our friends feel – we cannot say that their opinions reflect all Facebook users, all Facebook users that fall in the age range of our friends, or any other group. Our friends are not a randomly selected group of Facebook users, so they may not be typical; and, if not typical users, cannot be considered to reflect the typical users.

If our sample is random, then we know (or strongly suspect) a few things. First, the sample is unlikely to contain both the smallest and largest value that exists in the larger population – so, an estimate of the population variation is likely to be too small if based on the sample. This is corrected by using a sample standard deviation formula rather than a population

formula. We will look at what this means specifically in the other lectures this week; but Excel will do this for us easily.

Second, we know that our summary statistics are not the same as the population’s parameter values. We are dealing with some (generally small) errors. This is where the new statistics student often begins to be uncomfortable. How can we make good judgements if our information is wrong? This is a reasonable question, and one that we, as data detectives, need to be comfortable with.

The first part of the answer falls with the design of the sample, by selecting the right sample size (how many are in the sample), we can control the relative size of the likely error. For example, we can design a sample where the estimated error for our average salary is about plus or minus $1,000. Does knowing that our estimates could be $1000 off change our view of the data? If the female average was a thousand dollars more and the male salary was a thousand dollars less, would you really change your opinion about them being different? Probably not with the difference we see in our salary values (around 38K versus 52K). If the averages were closer together, this error range might impact our conclusions.

Note, this error range is often called the margin of error. We see this most often in opinion polls. For example, if a poll said that the percent of Americans who favored Federal Government support for victims of natural disasters (hurricanes, floods, etc.) was 65% with a margin of error of +/- 3%; we would say that the true proportion was somewhat between 62% to 68%, clearly a majority of the population. Where the margin of error becomes important to know is when results are closer together, such as when support is 52% in favor versus 48% opposed, with a margin of error of 3%. This means the actual support could be as low as 49% or as high as 55%; meaning the results are generally too close to make a solid decision that the issue is supported by a majority, the proverbial “too close to call.”

The second part of answering the question of how do we make good decisions introduces the tools we will be looking at this week – decision making statistical tests that focus on examining the size of observed differences to see if they are “meaningful” or not. The neat part of these tools is we do not need to know what the sampling error was, as the techniques will automatically include this impact into our results!

The statistical tools we will be looking at for the next couple of weeks all “work” due to a couple of assumptions about the population. First, the data needs to be at the interval or ratio level; the differences between sequential values needs to be constant (such as in temperature or money). Additionally, the data is assumed to come from a population that is normally distributed, the normal curve shape that we briefly looked at last week. Note that many statisticians feel that minor deviations from these strict assumptions will not significantly impact the outcomes of the tests.

The tools for this week and next use the same basic logic. If we take a lot of samples from the population and graph the mean for all of them, we will get a normal curve (even if the population is not exactly normal) distribution called the sampling distribution of the mean. Makes sense as we are using sample means. This distribution has an overall, or grand, mean

equal to that of the population. The standard deviation equals the standard deviation of the population divided by the square root of the population. (Let’s take this on faith for now, trust me you do not want to see the math behind proving these. But if you do, I invite you to look it up on the web.) Now, knowing – in theory – what the mean values will be from population samples, we can look at how any given sample differs from what we think the population mean is. This difference can be translated into what is essentially a z-score (although the specific measure will vary depending upon the test we are using) that we looked at last week. With this statistic, we can determine how likely (the probability of) getting a difference as large or larger than we have purely by chance (sampling error from the actual population value) alone.

If we have a small likelihood of getting this large of a difference, we say that our difference is too large to have been purely a sampling error, and we say a real difference exists or that the population mean the sample came from is not the same as we thought.

That is the basic logic of statistical testing. Of course, the actual process is a bit more structured, but the logic holds: if the probability of getting our result is small (for example 4% or 0.04), we say the difference is significant. If the probability is large (for example 37% or 0.37), then we say there is not enough evidence to say the difference is anything but a simple sampling difference from the actual population result.

The tools we will be adding to our bag of tricks this week will allow us to examine differences between data sets. One set of tools, called the t-test, looks at means to see if the observed difference is significant or merely a chance difference due mostly to sampling error rather than a true difference in the population. Knowing if means differ is a critical issue in examining groups and making decisions.

The other tool – the F-test for variance, does the same for the data variation between groups. Often ignored in “real life,” the consistency within groups is an important characteristic in understanding whether groups having similar means can be said to be similar or not. For example, if a group of English majors all took two classes together, one math and one English, would you expect the grade distributions to be similar, or would you expect one to show a larger range (or variation) than the other?

We will see throughout the class that consistency and differences are key elements to understanding what the data is hiding from us, or trying to tell us – depending on how you look at it. In either case, as detectives our job is to ferret out the information we need to answer our questions.

Hypothesis Testing

The process or standard approach that we will use is called the hypothesis testing procedure. It consists of six steps; the first 4 set up the problem and how we will make our decisions (and are done before we do anything with the actual data), the fifth step involves the analysis (done with Excel), and the final and sixth step focuses on interpreting the result.

The hypothesis testing procedure is a standardized decision-making process that ensures we make our decisions on whether things are different or not based on the data, and not some

other factors. Many times, our results are more conservative than individual managerial judgements – that is, a statistical decision will call fewer things significantly different than the judgement calls of managers. This is, at times, frustrating for managers who want to show that things have changed. At other times, it is a benefit such as if we are hoping that things – such as error rates – have not changed.

While a lot of statistical texts have slightly different versions of the hypothesis testing procedure (fewer or more steps), they are essentially the same – and are a spinoff of the scientific method. For this class, we will use the following six steps:

1. State the null and alternate hypothesis 2. Select a level of significance 3. Identify the statistical test to use 4. State the decision rule. Steps 1 – 4 are done before we examine the data 5. Perform the analysis 6. Interpret the result.

Step 1

A hypothesis is a claim about an outcome. It comes in two forms. The first is the null hypothesis – sometimes called the testable hypothesis, as it is the claim we perform all of our statistical tests on. It is termed the “Null” hypothesis, shown as Ho, as it basically says “no difference exists. Even if we want to test for a difference, such as males and females having a different average compa-ratio; in statistics, we test to see if they do not.

Why? It is easier to show that something differs from a fixed point than it is to show that the difference is meaningful – I mean how can we focus on “different?” What does “different” mean? So, we go with testing no difference. The key rule about developing a null hypothesis is that it always contains an equal claim, this could be equal (=), equal to or less than (<=), or equal to or more than (=>).

Here are some examples:

Ex 1: Question: Is the female compa-ratio mean = 1.0?

Ho: Female compa-ratio mean = 1.0.

Ex 2: Q: is the female compa-ratio mean = the male compa-ratio mean?

Ho: Female compa-ratio mean = Male compa-ratio mean.

Ex. 3: Q: Is the female compa-ratio more than the male compa-ratio? Note that this question does not contain a equal condition. In this case, the null is the opposite of what the question asks:

Ho: Female compa-ratio <= Male compa-ratio.

A null hypothesis is always coupled with an alternate hypothesis. The alternate is the opposite claim as the null. The alternate hypothesis is shown as Ha. Between the two

claims, all possible outcomes must be covered. So, for our three examples, the complete step 1 (state the null and alternate hypothesis statements) would look like:

Ex 1: Question: Is the female compa-ratio mean = 1.0?

Ho: Female compa-ratio mean = 1.0.

Ha: Female compa-ratio mean =/= (not equal to) 1.0

Ex 2: Q: is the female compa-ratio mean = the male compa-ratio mean?

Ho: Female compa-ratio mean = Male compa-ratio mean.

Ha: Female compa-ratio mean =/= Male compa-ration mean.

Ex. 3: Q: Is the female compa-ratio more than the male compa-ratio?

Ho: Female compa-ratio <= Male compa-ratio

Ha: Female compa-ratio > Male compa-ratio. (Note that in this case, the alternate hypothesis is the question being asked, but the null is what we always use as the test hypothesis.)

When developing the null and alternate hypothesis,

1. Look at the question being asked. 2. If the wording implies an equality could exist (equal to, at least, no more than,

etc.), we have a null hypothesis and we write it exactly as the question asks. 3. If the wording does not suggest an equality (less than, more than, etc.), it

refers to the alternate hypothesis. Write the alternate first. 4. Then, for whichever hypothesis statement you wrote, develop the other to

contain all of the other cases. An = null should have a =/= alternate, an => null should have a < alternate; a <= null should have a > alternate, and vice versa.

5. The order the variables are listed in each hypothesis must be the same – if we list males first in the null, we need to list males first in the alternate. This minimizes confusion in interpreting results.

Note: the hypothesis statements are claims about the population parameters/values based on the sample results. So, if we have; for example; two sample means that are obviously different, we can still say that the means are equal – we are talking about the population means.

If you look at the examples, you can notice two distinct kinds of null hypothesis statements. One has only an equal sign in it, while the other contains an equal sign and an inequality sign (<=, but it could be =>). These two types correspond to two different research questions and test results. If we are only interested in whether something is equal or not, such as if the male average salary equals the female average salary, we do not really care which is greater – just if they could be the same in the population or not. This is considered a two-tail test, as either of two conditions would cause us to reject the null. For our equal salary question,

if we find that male mean is > (greater than) the female mean or if the male mean is < (less than) the female mean.

The other condition we might be interested in, and we need a reason to select this approach, occurs when we want to specifically know if one mean exceeds the other. In this situation, we care about the direction of the difference. For example, only if the male mean is greater than the female mean or if the male mean is less than the female mean.

Step 2

The level of significance is another concept that is critical in statistics, but is often not used in typical business decisions. One senior manager told the author that their role was to ensure that the “boss’ decisions were right 50% +1 of the time rather than 50% -1.” This suggests that the level of confidence that the right decisions are being made is around 50%. In statistics, this would be completely unacceptable.

A typically statistical test has a level of confidence that the right decision is being made is about 95%, with a typical range from 90 to 99%. This is done with our chosen level of significance. For this class, we will always use the most common level of 5%, or more technically alpha = 0.05. This means we will live with a 5% chance of saying a difference is significant when we really have only a chance sampling error.

Remember, no decision that does not involve all the possible information that can be collected will ever have no possibility of being wrong. So, saying we are 95% sure we made the right call is great. Marketing studies often will use an alpha of .10, meaning that are 90% sure when they say the marketing campaign worked. Medical studies will often use an alpha of 0.01 or even 0.001, meaning they are 99% or even 99.9% sure that the difference is real and not a chance sampling error.

Step 3

Choosing the statistical test and test statistic depends upon the data we have and the question we are asking. For this week, we will be using compa-ratio data in the examples and salary data in the homework – both are continuous and at least interval level data. The questions we will look at this week will focus on seeing if there is a difference in the average pay between males and females in the population, based on our sample results. After all, if we cannot find a difference in our sample, should we even be working on the question?

In the quality improvement world, one of the strategies for looking for and improving performance of a process is to first look at and reduce the variation in the data – after all, if the data has a lot of variation, we cannot really trust the mean to be very reflective of the entire data set.

Our first statistical test is called the F-test. It is used when we have at least interval level data and we are interested in determining if the variances of two groups are significantly different or if the observed difference is merely chance sampling error. The test statistic for this is the F.

Once we know if the variances are the same or not, we can move to looking for differences between the group means. This is done with the T-test, and the t-statistic. Details on these two tests will be given later; for now, we just need to know what we are looking at and what we will be using.

Step 4

One of the rules in researching questions is that our decision rule – how we are going to make our decision once the analysis is done – should be stated upfront and before we even get to the data. This helps ensure that our decision is data driven rather than being made by emotional factors to get the outcome we want rather than the outcome that fits the data. (Much like making our detective go after the suspect that did the crime rather than the one he does not like and wants to arrest.)

The decision rule for our class is very simple, and will always be the same:

Reject the null hypothesis if the p-value is less than our alpha of .05. (Note: this would be the same as saying that if the p-value is not less than 0.05, we would fail to reject the null hypothesis.)

We introduced the p-value last week, it is the probability of our outcome being as large or larger than we have by pure chance alone. The further from the actual mean a sample mean is, the less chance we have of getting a value that differs from the mean that that much or more; the closer to the actual mean, the greater our chance would be of getting that difference or more purely by sampling error.

Our decision rule ties our criteria for significance of the outcome – the step 2 choice of alpha – with the results that the statistical tests will provide (and, the Excel tests will give us the p-values for us to use in making the decisions).

These four steps define our analysis, and are done before we do any analysis of the data.

Step 5

Once we know how we will analyze and interpret the results, it is time to get our sample data, and set it up for input into an Excel statistical function. Some examples of how this data input works will be discussed in the third lecture for this week.

What is constant about this step is the need to:

1. Look at the appropriate p-value (and it is indicated in the test outputs, as we will see below).

2. Compare the p-value with our value for alpha (0.05). 3. Make a decision – if the test p-value is less than (<) 0.05, we will reject the null

hypothesis. If the test p-value is more than or equal (=>) 0.05, we will fail to reject the null hypothesis.

Rejecting the null hypothesis means that we feel the alternate hypothesis is the more accurate statement about the populations we are testing. This is the same for all of our statistical tests.

Step 6

Once we have made our decision to reject or fail to reject the null hypothesis, we need to close the loop, and go back and answer our original question. We need to take the statistical result or rejecting or failing to reject the null and turn it into an “English” answer to the question. Doing so depends on how the original question lead to the hypothesis statements. Examples of this follow.

Lectures 2 and 3 will show how to use this process in conjunction with Excel and the F and T tests. For now, focus on the logic of setting up the testing instructions.

If you have any questions on this material, please ask your instructor.

After finishing with this lecture, please go to the first discussion for the week, and engage in a discussion with others in the class over the first couple of days before reading the second lecture.