Unit VI discussion board

Malkta

CH6.docx

Home >Business & Finance homework help >Operations Management homework help >Unit VI discussion board

6 Quantitative Research Design

Chapter outline

· The structure of experimental research

· Experimental and quasi-experimental research design

· Generalizing from samples to populations

· Designing valid and reliable research instruments

Keywords

· Experimental research

· Research questions

· Hypotheses

· Dependent variables

· Independent variables

· Descriptive statistics

· Inferential statistics

· Experimental design

· Quasi-experimental design

· Sampling

· Validity

· Reliability

Icon Key

Read

Explore

Define

Apply

Watch

Build

Practise

Discover

Author video

Chapter objectives

After reading this chapter you will be able to:

· Describe the experimental and quasi-experimental research approaches.

· Formulate appropriate questions and hypotheses.

· Identify populations and samples.

· Describe the principles of research tool design.

A research design is the overarching plan for the collection, measurement and analysis of data. Typically, a research design will describe the purpose of the study and the kinds of questions being addressed, the techniques to be used for collecting data, approaches to selecting samples and how the data are going to be analysed.

Define: A priori

We saw in Chapter 2 that experimental research methodology usually involves truth-seeking (as opposed to perspective- or opinion-seeking) and may often involve the use of quantitative methods for analysis. It tends, therefore, to utilize a deductive approach to research design, that is the use of a priori questions or hypotheses that the research will test. These often flow from sets of issues and questions arising from the researcher’s engagement with a relevant body of literature, such as marketing, knowledge management or supply chain logistics. The intention of experimental research is the production of results that are objective, valid and replicable (by the original researcher, or by others). In terms of epistemology, then, experimental research falls firmly into the objectivist camp, and is influenced by positivistic theoretical perspectives. It takes, for example, some of the principles of research design (such as the use of experimental and control groups) from the natural sciences. However, given the discredited status of positivism, advocates of the experimental approach are now likely to make more cautious and modest claims for the veracity and status of their research results.

In an organizational context, research might stem not from issues prompted by a body of literature, but from a real, live problem the researcher is asked to solve. The initial focus, then, is the problem itself (rising absenteeism, communication bottlenecks, data security, etc.), but the researcher will probably soon have to access both the academic literature (including technical and institutional sources) and also grey literature such as internal organizational documents and reports. Chapter 3 showed how the researcher journeys through a process of refinement, whereby the territory covered by the research literature becomes increasingly focused. But this is not just a question of narrowing the research. The core issues that emerge from the literature gradually build into significant sets of themes, or concerns that link to, and help to specify, the research questions and the research design for solving them.

Note that many of the issues discussed in this chapter (for example, the generation of research questions, the identification of samples from populations and issues of validity and reliability ) are also discussed in many of the chapters that follow – even those associated with more qualitative designs.

The structure of experimental research

The experimental research design process, put simply, comprises two steps: the planning stage and the operational stage (see Figure 6.1 ). At the planning stage, the main issue or research question may be posed and the relevant literature and theories investigated. From these it should be possible (if the issue is capable of being researched) to formulate research hypotheses. The dependent variables (the subject of the research) and independent variables (variables that effect the dependent variable) are identified and made explicit, after which we move into the operational stage. After the experiment has been conducted, the analysis stage may involve the use of both descriptive and inferential statistics (described in Chapter 24 ). From the analysis it then becomes possible to either accept or reject the hypothesis. A formal document or presentation is then prepared to report the results. Let us look at each of these stages in more detail.

Figure 6.1 Stages in the planning and operation of an experimental and quasi-experimental research project

Source: Adapted from Keppel, Saufley and Tokunaga (1992). Reprinted by kind permission of Macmillan

Identifying the issue or questions of interest

We saw in Chapter 3 that some of the criteria that make up a ‘good’ research topic include the availability of resources and access to sponsors and other people who may be able to help in the research. Sometimes a research issue may arise from your reading of a body of literature. In a workplace setting, issues or questions spring up as a result of real problems that require a solution, or as a result of a pilot study prior to the implementation of a research project.

Reviewing relevant literature and theories

As we saw in Chapter 2 , the experimental approach to research is often deductive, so once an area or issue has been chosen for research, the next stage is to identify and delve into a relevant body of literature. Chapter 5 illustrated some of the sources where you might find the literature you need. Early on in your research, you should try to identify the theories that are relevant to addressing your topic, and also what kinds of research methods have been used to address the subject. The literature search will also identify who are the dominant and influential writers in the field. Having looked at the literature, you may decide that the scale of the subject is too large (particularly in terms of your own tight timescales), or that the investigation you were considering has already been done. However, you may also see that previous investigations have been flawed, or that there are gaps in the research that are worth filling. For example, you may become aware of emerging technology-based learning theories, but notice that there have been few studies of their application within the realm of social media (in which you have personal experience). This could be your niche, your experience in the area giving you a head start.

Developing questions and hypotheses

Research questions and hypotheses are merely the configuration of issues into a transparent and measurable formulation. The way in which research questions are stated, their focus and the kinds of data they seek are strongly connected to the philosophy and research paradigm of the researcher (recall Chapter 2 ). As Wield (2002) also cautions, writing questions and hypotheses is not necessarily a linear process. Even after they have been formulated, either further reading of the literature, or surprises at the piloting or data gathering stages, can force the researcher to amend or even rewrite them. Let us look at research questions and hypotheses in more detail.

Constructing research questions

The ways in which we formulate key questions can sometimes drive us down unfruitful paths, even when the underlying concerns that motivate our questions are genuine and important. It might help if we reflect for a moment on the genuine concerns that drive us to ask the questions we ask (Sarasvathy, 2004). As Alford (1998) points out, research questions are not the same as problems. Problems, themes and concerns may be allocated to you by a sponsor, or may emerge from your engagement with a relevant body of literature. Alford, however, asserts that, in contrast to a problem, a research question comprises two elements: firstly, a connection to a theoretical framework; secondly, a sentence in which every word counts and which ends (not surprisingly) with a question mark. Questions also describe potential relationships between and among variables that are to be tested. Blumberg et al. (2005) similarly distinguish between what they call dilemmas and research questions. A dilemma is a signal that all is not well – for example, falling sales, higher staff absenteeism or higher borrowing costs. The key is knowing how to turn statements of dilemmas into tight research questions. Table 6.1 offers some examples.

Read: Formulating research questions

Table 6.1

It is clear from Table 6.1 that each dilemma is addressed by at least one question that explores the relationships between two variables. Kerlinger and Lee (2000) argue that a good research question:

· Expresses a relationship between variables (for example, company image and sales levels).

· Is stated in unambiguous terms in a question format.

But, as Black (2001) states, a question could meet both of Kerlinger and Lee’s criteria and still be invalid, because it may be virtually impossible to operationally define some of its variables. What, for example, do we mean by ‘digital technologies’ (in the above example), and how would we define them in ways that could be measured? As Hedrick et al. (1993) argue, researchers may have to receive sets of questions from research sponsors, and these may be posed by non-technical people in non-technical language. The researcher’s first step, then, is to re-phrase the questions into a form that is both researchable and acceptable to the client. Research questions can be classified into four major categories:

· Descriptive (‘What is happening?’, ‘Which methods are being used?’).

· Normative (‘What is happening compared to what should happen?’). The standards against which the outcomes are evaluated could include legal requirements, professional standards or programme objectives.

· Correlative (‘What is the relationship, and the strength of this relationship, between variable X and Y?’). Note that this establishes a relationship, but it does not imply a cause.

· Impact (‘What impact does a change in X have on Y?’). In contrast to correlation studies, impact questions do try to establish a causal relationship between variables.

Table 6.2 provides some examples of research questions for each of these categories.

It is often useful to take a research question and to break it down into subordinate questions. These are highly specific and assist in answering the question to which they are attached. Taking the first question in Table 6.2 , we might devise a set of subordinate questions such as:

· How common is drug misuse among male and female employees?

· How does drug misuse compare across different departments?

· Has drug misuse increased or decreased over the past five years?

This is also a useful exercise because subordinate questions can provide a stage between the original objective and the kinds of detailed questions needed for research tools such as questionnaires and interview or observation schedules. Case Study 6.1 provides an illustration of how research questions often have to be revised and refined before they become sufficiently focused and usable.

Table 6.2

Source: Adapted from Hedrick et al., 1993

Case Study 6.1 Getting those research questions right

A researcher, working for a voluntary association giving advice to the public, is concerned that most of those seeking the bureau’s help are white, with very few clients coming from the ethnic minority population. She receives a small grant from the bureau’s trustees to carry out a research project. She formulates her research questions as follows:

Research questions

1. To produce a report detailing the research. To check if the bureau is conforming to its organizational aims and objectives and if not how it can improve the delivery of services.

2. To increase awareness of the needs of ethnic minority clients and potential clients of the bureau among staff and to inform the organization of staff training needs.

3. To use this as a starting point for further work to be carried out by volunteers at the bureau.

Take a look at these research questions. What is wrong with them? Well, to be honest, quite a lot. Question 1 is not really a question but an output. This is what will be produced through the research. Questions 2 and 3 are aims or ambitions. What are listed as research questions do not deserve the description. They may result from the research but are not objectives, since there is nothing here that can be measured.

After some thought, the researcher arrives at the following list of questions.

1. What are the needs of ethnic minority groups in the district?

2. What access to information about the bureau do they have?

3. Do those that access the information implement its contents effectively?

4. Is there a relationship between the quality of information given, and ethnic minority trust in the bureau?

5. What degree of awareness should bureau staff have (in relation to their organizational service levels) about the needs of ethnic minority groups?

Activity 6.1

Examine the final set of questions in Case Study 6.1 . Which of these research questions belongs to the descriptive, normative, correlative or impact categories?

Suggested answers are provided at the end of the chapter.

Research questions are formulated as part of many research studies, whether perspective- seeking or truth-seeking, although not necessarily at the same stage of the research. In perspective-seeking studies, for example, questions may emerge as part of the data gathering exercise. For truth-seeking research, including experimental and quasi-experimental research, they are usually formulated at the beginning of the research process. But while perspective-seeking research usually relies just on research questions, truth-seeking approaches usually go further and require the formulation of a hypothesis.

Employability Skill 6.1 Setting objectives that are achievable

Coming up with a sufficiently focused research question will help you to develop the key employability skill of setting achievable objectives. Your research question must have an achievable objective. Breaking your overall question down into sub-questions will help you to decide whether or not you are being too ambitious in your aims, and whether you should refine your question further.

Constructing hypotheses

Research questions are usually broad in nature, and may lend themselves to a number of answers, but a hypothesis is capable of being tested and is predictive. For example, ‘How is trust promoted in organizations?’ is a research question and not a hypothesis. To convert the question into a hypothesis we might conjecture that: ‘Emotionally intelligent leadership promotes trust’. Kerlinger and Lee (2000) suggest that a hypothesis is a speculative statement of the relation between two or more variables. Good hypotheses, then, should contain a statement containing two or more variables that are capable of measurement. Measurement, however, can only occur if the variables contained in the hypothesis can be operationally defined (see next section). Certainly, in the above hypothesis, the two variables ‘emotionally intelligent’ and ‘trust’ can each be operationally defined, compared through a research study, and the statement either accepted or rejected.

Define: Hypothesis

In formulating a hypothesis, care should be taken to avoid what Kerlinger and Lee (2000) describe as value questions, for example those that contain words such as ‘should’, ‘ought’ or ‘better than’. Similarly, the statement ‘The implementation of the new information technology system has led to poor results’ is also a value statement because of the use of the word ‘poor’ – what, exactly, is meant by this? A better approach would be to state the results in measurable terms such as ‘reduced output’, ‘lower staff satisfaction’, or ‘computer error’. It is useful to reflect that negative findings are sometimes just as important as positive ones since they can highlight new lines of investigation.

Activity 6.2

Examine each of the following statements and decide which (if any) make valid hypotheses.

1. Using external coaches leads to disappointing levels of employee commitment.

2. What are the major causes of intranet failure?

3. The introduction of a Six Sigma process will increase levels of customer satisfaction.

Suggested answers are provided at the end of the chapter.

Operationally defining variables

One of the problems in formulating research questions and hypotheses is that they tend to be somewhat generalized and vague. Before research tools can be drawn up, it is important to operationally define key variables so it is quite clear what is being measured. Kerlinger and Lee (2000) define an operational definition as something that gives meaning to a construct or a variable by setting out the activities or ‘operations’ that are necessary to measure it. Classifying operational definitions can sometimes be quite challenging. For example, our research question might be: What factors provide the key drivers for ensuring business success in the medium term? As it stands, the question is far too vague to provide a basis for measurement. Returning to the question, we need to operationally define what we mean by ‘business success’: is it output, profitability, cost control or perhaps a combination of all of these? Similarly, what is meant by ‘medium term’? Is it one year, two years, ten years? Going through the process of producing operational definitions allows us the opportunity to rethink some of our assumptions and may even encourage us to rewrite our original research question or questions. Note the loops back to previous stages in Figure 6.1 .

Identifying independent and dependent variables

Scientific research aims to identify why conditions or events occur. These causes are called independent variables and the resulting effects, dependent variables. A variable is a property that can take different values. Thus, the focus of research might be the introduction of a new performance-related pay system (independent variable) which is designed to lead to greater output (dependent variable). But as Black (2001) warns, relationships between variables may be ones of association , but this does not necessarily imply causality: that is, that changes in one variable lead to changes in another. For example, after the introduction of performance-related pay, output may rise, but this increase may have been caused by completely different factors (for example, better weather or victory by the local football team, each of which might boost morale and hence output).

Explore: Independent variables

Indeed, independent variables may act upon dependent variables only indirectly via intervening variables . Thus, someone may undertake high-calibre professional training hoping that this will eventually lead to a higher income level. But in practice, the professional training (independent variable) acts upon income level (dependent variable) via its effects on the person’s job prospects (intervening variable, as illustrated in Figure 6.2 ). In addition to this, Figure 6.2 also shows other relationships. For example, it is conceivable that, having achieved a higher level of income, some people may then want to (and be able to afford) more professional training.

Watch: Variables in research design

In experiments, it is the independent variable that is manipulated to see the effect. So, using the above example of performance-related pay, we might introduce such a scheme into a company and observe the effect on output. But, as has already been suggested, there may be other factors at work that might influence such changes in output. These are termed extraneous variables and must be ‘controlled for’: that is, the study designed in such a way that the impact of extraneous variables does not enter the calculations.

Figure 6.2 Illustration of the relationship between dependent, independent and intervening variables

There are various ways of controlling for extraneous variables. One is through elimination. So, using our example of performance-related pay, if the study were concerned about the possible influence of current status or grade, we would only choose people from a certain grade for the study. Another way of controlling extraneous variables is through randomization. If randomization is achieved, then it is probable that the experimental groups are equal in terms of all variables. It should be noted, of course, that complete randomization is difficult to achieve in practice. Say, for example, that we know that male and female workers are exactly equally represented in the workforce. If we were to take a random sample of 100 workers, we might expect to finish with 50 men and 50 women. In practice, we often end up with slight variations such as 48 men and 52 women. If gender constitutes the independent variable of interest to the study, we might want to ensure that the groups are equally represented, and randomly select male workers until the numbers reached 50 and likewise for female workers (see stratified random sampling , p. 228).

Conducting the study

Here begins the operational stage of the research, the success of which depends not only on how the data are gathered, but on how well the study has been planned. While the research strategy (experimental) has been selected, there are still a variety of research designs at the researcher’s disposal (see experimental and quasi-experimental research design, next) and these have to be selected with care.

Using descriptive and inferential statistics

The data are analysed using a variety of statistical methods, all of which should have been selected at the planning stage. Descriptive statistics are used to describe or summarize a set of data, while inferential statistics are used to make inferences from the sample chosen to a larger population (see Chapter 24 ).

Accepting or rejecting hypotheses

As we saw in Chapter 2 , it is impossible to ‘prove’ that any theory is right. All theories are provisional and tentative (until disproved). However, the weight of evidence must be sufficient that a hypothesis can be accepted as proved. As we will see in Chapter 24 , experimental design makes use of inferential statistics and probability to calculate the risk involved in accepting the hypothesis as true (when it is in fact false) and rejecting the hypothesis as false (when it is in fact true).

Preparing the formal report

Particularly when a study has been sponsored or commissioned, the researcher will need to prepare and deliver some kind of formal presentation of the findings. At this stage the focus will be on:

· Why the study was conducted.

· What research questions and hypotheses were evaluated.

· How these were turned into a research design (with sufficient detail that the experiment could be replicated).

· What differences were observed between the hypotheses and the results.

· What conclusions can be drawn and whether they support or contradict the hypothesis and existing theories.

In a more organizational and less academic setting, the formal report will tend to focus on the rationale for the study, the kinds of questions being posed, the results, and what findings, if any, can be implemented. Writing the research report is covered in detail in Chapter 27 . For projects that have received research funding, sponsors usually want to go beyond the report and to be provided with information on how the results of the project will be disseminated.

Experimental and quasi-experimental research design

The basis of true experimental design is that the researcher has control over the experiment, that is who, what, when, where and how the experiment is to be conducted. This particularly includes control over the ‘who’ of the experiment – that is, subjects are assigned to conditions randomly. So, for example, a local authority might seek to measure whether a refuse recycling programme was effective, or not. Hence, it might run the campaign in several randomly selected areas, but not in others. Where any of the elements of control are either weak or lacking, the study is said to be a quasi-experiment. Often, in organizational settings, for example, for practical purposes it is only possible to use pre-existing groups. Hence, it is only possible to select subjects from these groups rather than randomly assign them (as in a true experimental study). Another important difference is that while in experiments we can manipulate variables, in quasi-experimental studies we can only observe categories of subjects. So, we could consider the differences between two groups to be the independent variable but we would not be manipulating this variable. So, taking our recycling issue mentioned above, we would collect data on recycling indicators across the local authority, and then seek to discover what independent variables might impact on different recycling rates – for example, social class, ethnic group, etc.

One of the strengths of experimental design is that randomization improves the control over threats to internal validity . In other words, if the experimental intervention (treatment) does lead to a change in the dependent variable, there is some justification for believing that this has been caused by the treatment itself, and not just by the effect of some extraneous variable. Yet it should not be assumed that random assignment is the goal of all experimental studies. As Hedrick et al. (1993) point out, using an experimental group also means using a control group who do not receive the intervention. Even if the treatment does not prove to be effective, it usually comes with more resources. The control group will be denied these, and for a long period if it is a longitudinal study. For example, in the recycling example, above, those in the control group would not receive any potential benefits of the recycling campaign. This of course can be rectified if they are presented with the campaign benefits but after the study is over. However, this would still be after any benefits enjoyed by those in the experimental group – an institutional review board might not approve such a study because of these time lags.

Define: Experimental design

One of the strengths of a quasi-experimental design is that it is about as near as one can get to an experimental design, so it can support causal inferences. In the words of Hedrick et al. (1993), it provides ‘a mechanism for chipping away at the uncertainty surrounding the existence of a specific causal relationship’ (1993: 62). Quasi-experimental designs are best used when:

· Randomization is too expensive, unfeasible to attempt or impossible to monitor closely.

· There are difficulties, including ethical considerations, in withholding the treatment.

· The study is retrospective and the programme being studied is already under way.

According to McBurney and White (2009), generally, experimental designs are usually considered superior to quasi-experimental (and quasi-experimental to non-experimental). However, it may not always be possible to replicate social, organizational or behavioural conditions in a laboratory setting. Therefore, observation in a field setting, say, might be preferable to an experiment because the advantage of realism outweighs the loss of control. The broad differences between experimental, quasi-experimental and non-experimental studies are summarized in Table 6.3 , and an example of a quasi-experimental design provided in Case Study 6.2 .

Image 6.1 A coaching session

Table 6.3

Case Study 6.2 A quasi-experimental design

Leonard-Cross (2010) reports on a research study conducted in a large public sector organization, employing over 3,000 staff in 12 geographical locations. The organization had implemented an accredited coach training programme, offering those in management-level posts the opportunity to undertake a coaching qualification and then coach fellow employees. The study sought to evaluate the impact of the programme on those who had received coaching. To do this, a quasi-experimental design was adopted with participants in the survey either in a coached or non-coached group (the latter randomly selected). The researcher had no control over group allocation since membership of the coached group depended on whether participants had taken part in the coaching programme over the last two years – hence the quasi-experimental nature of the design. The non-coached staff (control group) were matched to the coached staff based on geographical location and job type and were randomly selected by contacts in each geographical location who had no additional knowledge of the research. The study found that participants that had received developmental coaching (N = 61) had higher levels of self-efficacy than the control group of participants (N = 57) who had not received coaching.

Source: Leonard-Cross, 2010

Activity 6.3

Taking Case Study 6.2 , explain:

1. Why this is a quasi-experimental rather than an experimental study.

2. Why the non-coaching (control) group were matched against the coached group based on geographical location and job type.

Suggested answers are provided at the end of the chapter.

Let us take a look at a number of research designs, starting with frequently used (but faulty designs) and then some sound designs.

Faulty designs to avoid

Design 1: Non-experimental with intact group

In this design, an intact group is taken and attempts made to discover why changes in an independent variable occurred. There is no attempt made here to manipulate any independent variables – hence the design is non-experimental (see Table 6.4 ). Say that a voluntary organization analyses its charitable donation patterns over the past three years by geographic region. The dependent variable is the level of charitable donations for each region. The independent variable is not manipulated but is imagined. In other words, researchers would conduct a study that would try to find explanations for any regional differences, perhaps using documentary evidence. Clearly, the problem here is providing convincing evidence of causation – that a particular independent variable caused the changes in the dependent variable.

In their influential work, Campbell and Stanley (1963) describe designs that are devoid of a control group as being of almost no scientific value. This is not to say that they are completely worthless. Each design might reveal some interesting evidence of value to an organization, but they are worthless in the sense that it would be a mistake to draw firm conclusions from them.

Table 6.4

Design 2: Post-test only with non-equivalent control groups

In this type of design, a treatment is given to one group (the experimental group), but not to another (the control). Both groups are then given a post-test to see if the treatment has been effective (see Table 6.5 ). Unfortunately, subjects have not been randomly allocated between the experimental and control groups, so it is impossible to say that the two groups are equivalent. If, say, the experimental group performs better in the test, it is not possible to rule out the possibility that this was because the subjects in this group were more able or better motivated. Say, for example, that in a training setting, one group of participants is given coaching to improve their interpersonal skills, but a control group does not receive the coaching. Both take a post-test but the control group does better! This may be because there was no random allocation of subjects (both groups were taken intact) and it so happens that there are more able participants in the control group (or some had received coaching in the past).

Table 6.5

Design 3: One group, pre-test/post-test

In Design 3, a group is measured on the dependent variable by a pre-test , an independent variable is introduced, and the dependent variable measured by a post-test (see Table 6.6 ). So, an organization could measure staff attitudes towards racial tolerance, introduce a race-awareness programme, and measure staff attitudes once the programme was completed. Any change in attitudes would be measured by changes in scores between the two tests.

Table 6.6

This design is an improvement on Design 1 as it appears that any changes in attitude could be attributed to the impact of the treatment – the attitude training. Unfortunately, as Campbell and Stanley (1963) point out, there are other factors that could have affected the post-test score. These can impact on the experiment’s internal validity, that is the extent to which we can be sure that experimental treatments did make a difference to the independent variable(s). Such factors include:

· Maturation effects: people learn over time, which might affect scores on both mental ability and attitude, or they may grow more fatigued over time, which may also affect their post-test scores.

· Measurement procedures: the pre-test itself might have made the subjects more sensitive to race issues and influenced their responses on the post-test. Both controversial and memory issues are prone to be influenced in this way.

· Instrumentation: in which changes, say, in the observers or scorers used to assess the test results may affect the scores obtained.

· Experimental mortality : or the differential loss of respondents from one group compared to the other, for example through absence, sickness or resignations.

· Extraneous variables might influence the results, particularly if there is a large time gap between the pre-test and post-test.

Some sound designs

McBurney and White (2009) state that there is no such thing as a perfect experiment. Nevertheless, there are two elements of design that provide some control over threats to validity and which form the basis of all sound experimental designs: (a) the existence of a control group or a control condition; (b) the random allocation of subjects to groups. Some of the principles of random assignment are explained in the following Web link.

Watch: Random assignment

Go Online 6.1

Watch the following video clip to understand what a random assignment is. The URL for the video clip can be accessed via the companion website:

· http://www.youtube.com/watch?v=V_GIjFw6RZE

Design 4: Experimental group with control

In this design, subjects are randomly assigned to each of the experimental and control groups, which means that, at least theoretically, all independent variables are controlled (see Table 6.7 ). Hence, again using our racial tolerance example, the study would randomly assign groups of people to both the experimental and control groups.

The experimental group would receive the treatment (the race-awareness training) while the control group would not receive the training. Notice that any extraneous variables, such as the effects of the pre-test on attitudes, would be controlled for, since the impact should be the same on both the experimental and control groups. If the training has been genuinely successful, then the improvements in test scores for the experimental group should exceed those for the control group.

Image 6.2 An experimental group receiving training and a control group

Table 6.7

Design 5: Quasi-experimental design with non-equivalent control

Recall that one of the features of quasi-experimental designs is that it is not possible for the researcher to control the assignment of subjects to conditions, and s/he will often have to take groups that are intact (see Table 6.8 ). For example, studies of professional development will often have to use training groups that already exist. A typical feature of quasi-experiments is where we have an experimental and a control group, but subjects have not been randomly allocated to either of the two groups.

Table 6.8

The use of a control group makes this design superior to Designs 1, 2 and 3, since at least the impact of extraneous variables is controlled for, but not as reliable as Design 4. If steps can be taken to improve the equivalence between the two groups then this will improve the validity of the study. Matching, for example, will help in this direction. Here, steps are taken to match subjects between groups against significant variables such as age, sex, income, etc. If matching is not possible, then at least both groups should be chosen from the same population. So, for example, if we are investigating the impact of an incentives package on job performance, we would want to match the experimental (incentives) group and control (non-incentives) group against key variables such as age, work role and seniority.

One of the challenges of using a non-equivalent control group design is in the analysis of the results. McBurney and White (2009) distinguish between desired result patterns and those that it is impossible to interpret. In pattern A ( Figure 6.3 ), for example, both the experimental and control groups exhibit the same performance in a pre-test, but only the experimental group improves its performance in the post-test. Although the experimental and control groups are not equivalent, their performances can be compared because their behaviour was the same at the beginning. A similar situation pertains for pattern B – the experimental group performed worse than the control group in the pre-test but improved in the post-test, with the control showing no improvement. It would be difficult to find a reason as to why this process had occurred by chance alone.

Pattern C, however, is much harder to interpret. Although it is true that the performance of the experimental group has improved, the lack of improvement by the control group may be due to the ceiling effect – they began by being better than the experimental group and it may not be possible to improve on this level of performance. Hence, it cannot be deduced that the improvement in the experimental group was due to the treatment. In pattern D the performance of both the experimental and control groups has improved, with the experimental group improving to a higher level. At first sight this might appear to be a significant result but a claim for this would be mistaken since both groups have improved their performance by the same proportion.

Figure 6.3 Interpretable and uninterpretable patterns of results in a non-equivalent control group design with pre-test and post-test

Source: Republished with permission of South-Western College Publishing, a division of Cengage Learning, from McBurney, D.H. and White, T.L. (2009) Research Methods, 8th edn. Belmont, CA: Wadsworth; permission conveyed through Copyright Clearance Center, Inc.

Design 6: Developmental designs

Like interrupted time-series designs, developmental designs involve measurement across time and, again, do not involve the use of control groups. One kind of developmental design is the use of a cross-sectional study, which looks at a phenomenon at a particular period of time. For example, a cross-sectional design might study the determinants of accidents in an organization. A survey might be used to calculate an average number of days lost in accidents per employee. The next stage of the survey might examine accident rates by age group, gender, occupational role and seniority. One of the advantages of cross-sectional design is that it can reveal associations among variables (age, gender, etc.). But what it cannot do is reveal causation. To achieve this, we would have to turn to a longitudinal study, taking a series of samples over time. The problem here, however, is that it may be difficult to gain access to the same set of people over a long period. Indeed, even different sets of researchers may have to be employed.

Design 7: Factorial designs

The designs we have considered so far have involved manipulation or change in one independent variable. Sometimes, however, it becomes necessary to investigate the impact of changes in two or more variables. One reason for this could be that there is more than one alternative hypothesis to confirm or reject. Another reason might be to explore relationships and interactions between variables. Here we use a factorial design which allows us to look at all possible combinations of selected values.

The simplest form is where we have two variables, each of which has two values or levels. Hence, it is known as a two-by-two (2 × 2) factorial design. In Figure 6.4 , for example, the two variables are light and heat, each of which has two levels (cold/hot and dull/bright). Hence, we have four possible combinations, as illustrated. We could conduct an experiment to see which combination of factors gives rise to the most attentiveness (measured, say, by production levels, or on a self-assessment questionnaire) in a workplace. We might find, for example, that dull light combined with both heat and cold leads to low levels of attentiveness, as do bright/hot conditions; but the interaction of brightness with cold temperatures keeps all workers ‘on their toes’!

Figure 6.4 A 2 × 2 factorial design showing all possible combinations of factors

Generalizing from samples to populations

A typical human trait is to make generalizations from limited experience or information. For example, we may ask members of staff what they think of the new company environmentally friendly transport policy. We may infer that this could be the opinion throughout the organization, the entire workforce constituting what in research terms is known as the population. A population can be defined as the total number of possible units or elements that are included in the study. If it is not possible to evaluate the entire population (because of its large size or a lack of research resources), then we might select a sample of employees for evaluation. According to Fink, ‘A good sample is a miniature of the population – just like it, only smaller’ (2002a: 1).

Define: Population

Top Tip 6.1

The word ‘population’ can often cause some confusion. When we use this word in research methods we do not usually mean the population of a country. In research, a population refers to a group that has something in common – for example, Glasgow human resource managers, Berlin bar owners or Parisian journalists.

The process of selecting samples

A sample will be chosen by a researcher on the basis that it is a representative sample of the population as a whole, that is the sample’s main characteristics are similar or identical to those of the population. Samples are selected from a sampling frame , that is a list of the population elements (see Figure 6.5 ). Notice that, while every attempt will be made to select a sampling frame that provides details of the entire population, practical circumstances may make the sampling frame incomplete. For example, the population may comprise all people working in airport security over a weekend, but the human resources records may have missed out some staff by mistake, whilst new starters have not even been entered onto the database yet. The research sample itself might be less than the sampling frame just because using all sampling frame records is too expensive. But having established the sampling frame and how many people we are going to use, how do we choose them?

Read: Representative samples

Most methods utilized to achieve representative samples depend, in some way, on the process of random assignment. Random probability sampling is the selecting of a random sample such that each member of the population has an equal chance of being selected. Clearly, this can present practical problems. Can we, for example, acquire a full list of company employees from which to draw the sample (the sampling frame)? But as Black (2001) warns, even after taking a random sample, there remains a finite possibility that it may not be representative of the population after all. The chances of this happening are reduced if the study can be replicated: that is, other random samples are used and studied. Nevertheless, the chances of a sample being representative are higher through random selection than if the sample is purposive (specifically selected by the researcher).

Figure 6.5 Relationship between the population, sampling frame and sample

Of course, we may not always want to attain completely random samples. Again using the simple example of gender, a factory workforce of 100 people might comprise 90 women and 10 men. A random sample of 25 people might give us 23 women and 2 men. Clearly, if gender is the independent variable, a sample of 2 men would probably be of little value to the study. In this case, we might use stratified random sampling by deciding to randomly sample female workers until 15 are picked and follow the same strategy but oversample for men until we have a sample of 10. Let us look at some of the approaches to achieving representativeness in samples.

What size sample should we use?

The first stage is to determine the actual size of the sample needed. Before doing this, we need to decide on the size of the confidence interval . This is the range of figures between which the population parameter is expected to lie. Say we set the confidence interval at 4 per cent, and 45 per cent of the population pick a particular answer. This means that we are saying that we are confident that between 41 per cent (45 – 4) and 49 per cent (45 + 4) of the entire population would have picked that answer. We also decide on a confidence level, usually of either 95 per cent or 99 per cent. This states the probability of including the population mean within the confidence interval. This is chosen before working out the confidence interval. In many studies, a confidence level of 95 per cent is often deemed sufficient. In medical research, a level of 99 per cent is usually taken because of the need to be highly confident of estimates. Experimenting with Activity 6.4 should make this clearer.

Define: Confidence intervals

Go Online 6.2

To calculate the size of sample you need from a given size of population click on the following link: http://www.surveysystem.com/sscalc.htm

Explore: Sample size calculator

Selecting random samples

Having estimated the size of sample you need, you can now go about randomly selecting it. As we have seen, randomization is the process of assigning subjects to experimental and control groups such that the subjects have an equal chance of being assigned to either group. The process of random selection can be accomplished either by using the appropriate statistical table (see Table 6.9 ) or by using a special computer program (see Activity 6.4 ).

Say you have acquired a list of 1,000 of the company’s staff from which you want to randomly select 50 as your research sample. First, ascribe a number to each staff member on the list. Then, using a pencil, close your eyes and point to part of the table. If you happen to select, say, 707, the top number of the third column ( Table 6.9 ), take the first two numbers, 70, and work down your list of random numbers in the table to the 70th. Hence, your first number is 799. Then, using the last digit from 707 and the first digit of the next three-digit figure, 872, you get 78. Select the 78th position down the list, which gives you 343. Go back to the number 872 and choose the last two digits of that number, 72, and take the 72nd number from the table, etc. Repeat this process until 50 names have been selected. Now take a look at the Web randomizer ( Activity 6.4 ) – you may find it easier!

Table 6.9

Source: Adapted from Black, 2001

Activity 6.4

Your sample comprises 100 people from whom you want to randomly select 10 as your sample. All people are allocated a number from 1 to 100. You now want to produce a set of 10 random numbers ranging from 1 to 100. In your Web browser, go the following address:

· http://www.randomizer.org

Click on [Randomizer] then respond as follows to the questions presented:

· How many sets of numbers do you want to generate? = 1

· How many numbers per set? = 10

· Number range = 1 to 100

· Do you wish each number in a set to remain unique? = Yes

· Do you wish to sort your outputted numbers (from least to greatest?) = Yes

Click on [Randomize Now!]

You should see a set of 10 random numbers arranged in a row.

Types of random sample

In an ideal world, you would have sufficient time and resources to choose completely random samples. In the real world, due to practical constraints, you may have to choose other types of sampling techniques. In quantitative research, random samples are usually preferable to non-random. Given the importance of sampling in research design (both quantitative and qualitative designs), Chapter 9 , Sampling Strategies in Business, is entirely devoted to this theme.

Top Tip 6.2

Research students often agonize about the need to select a random sample. Indeed, even when using non-random samples, they can become tempted to make claims that the sample was somehow randomly selected. This is misguided for a number of reasons. Firstly, these kinds of studies (especially when undertaken for the purpose of writing a thesis or dissertation), for practical purposes, often work with fairly modest sample sizes, meaning that the ability to generalize is limited. Secondly, when working with such modest samples, it is the quality (representativeness) of the sample that becomes more important rather than the size.

Generalizing from samples to populations

One of the objectives of experimental research is to achieve a situation where the results of a study using a sample can be generalized. According to Kerlinger and Lee (2000), generalizing means that the results of a study can be applied to other subjects, groups or conditions. Generalizing means that the fruits of research can have a broader application than merely being limited to a small group. For example, say that researchers evaluated a staff development programme in which staff were taught to adopt new health and safety practices to reduce accident rates. If the study showed that scores for the trained group were significantly better than for a control group, then the results might be of relevance to other health and safety policy makers. On the other hand, just because a study does not find results that are capable of generalization does not mean they have no relevance. A small case study, for example, may produce findings that are interesting and possibly indicative of trends worthy of replication by further research. And from a perspective-seeking view they may be seen as valid in their own right. The important point is that you should not make firm or exaggerated claims on the basis of small, unrepresentative samples.

Build: Sampling in business

Employability Skill 6.2 Understanding the strengths and weaknesses of selected sampling design

When businesses make decisions, they want to be sure that the data they use are trustworthy and can be relied on. If sampling is involved (which it often is), then the type of sampling strategy used and its strengths and weaknesses need to be understood and taken into account in interpreting data.

Designing valid and reliable research instruments

We have looked, so far, at some of the general principles of research design, including the use of experimental and control groups and the selection of representative samples so that results can be generalized to a larger population. However, for defensible statistical inferences to be made on the basis of the data, any research tools used (such as questionnaires, interview schedules and observation schedules) must be internally valid and reliable. To achieve external validity, such instruments must be designed in such a way that generalizations can be made from the analysis of the sample data to the population as a whole.

Watch: Using quantitative methods

This section deals with some of the general principles of validity and reliability, but these important issues are taken up in more detail when describing the design of specific data collection tools in later chapters.

Principles of validity

To ensure validity, a research instrument must measure what it was intended to measure. This may sound like an obvious statement, but many novice researchers make the mistake of asking spurious questions in a misguided attempt to collect as much data as possible – just in case some data may be needed at the analysis stage! For example, a bank survey might seek to measure customer attitudes towards the counter services it provides, but the data gathering instrument might (erroneously) stray into asking about their attitudes to new financial products. This might be important, but not relevant to the study itself. In discussing validity, McBurney and White (2009) pose the interesting analogy of using a measurement of hat size to determine intelligence. You could measure someone’s hat size, say, every hour and always come up with the same result. The test, then, is reliable. However, it is not valid, because hat size has nothing to do with what is being measured.

In Figure 6.6 we can see that only part of the research instrument covers the subject areas that have been operationally defined. Some operationally defined subjects have not been addressed by the instrument (Zone of Neglect), while other parts of the instrument cover issues of no direct relevance to the research study at all (Zone of Invalidity). To achieve validity, the research instrument subject area and operationally defined subject areas must exactly match (Zone of Validity).

The issue of validity, however, is much more complex than this. The central question around validity is whether a measure of a concept really measures that concept – does it measure what it claims to measure? So, for example, do IQ tests really measure intelligence? Do formal examinations measure academic ability? At a basic level, validity can be defined as eight types: face , internal , external , criterion , construct , content , predictive and statistical validity . We will look at each in turn.

Figure 6.6 Relationship between research instrument and operationally defined subject areas and the issue of validity

Face validity

When developing a new research instrument (such as a questionnaire), it is vital that it is able to demonstrate at least face validity otherwise all is lost. Face validity means that the instrument at least appears to measure what it was designed to measure. But how do we demonstrate such face validity? For a start, it is up to the researchers to study their own instrument and critically evaluate what they have produced. Because they are so ‘close’ to their own work, the next step is to get other people to comment, particularly if they are subject experts in relation to the concept being measured. However, as McBurney and White (2009) warn, face validity is not an end in itself. A test may have a high or low degree of validity regardless of whether it has face validity or not.

Define: Face validity

Top Tip 6.3

In the event that you do not have ready access to relevant subject matter experts, the next best step is to ask friends or colleagues to evaluate the instrument. Make it clear to them what the instrument is meant to measure and that you want a critical appraisal.

Internal validity

Internal validity refers to correlation questions (cause and effect) and to the extent to which causal conclusions can be drawn. If we take, for example, an evaluation of the impact of a product promotion campaign, one group receives the promotional material (the experimental group) while one does not (the control group). Possible confounding variables are controlled for, by trying to make sure that participants in each group are of similar ages and educational attainment. Internal validity (the impact of the campaign) may be helped by testing only those who are willing to participate in the experiment. But this reduces the completely random nature of the experimental group and hence the external validity of the study (see next).

Define: Interval validity

External validity

This is the extent to which it is possible to generalize from the relationships found in the data within the sample’s experimental subjects to a larger population or setting (Cook and Campbell, 1979). Clearly, this is important in experimental and quasi-experimental studies where sampling is required and where the potential for generalizing findings is often an issue. As Robson (2002) points out, the argument for generalization can be made by either direct demonstration or by making a case. The problem of generalizing from a study is that cynics can argue that its results are of relevance only to its particular setting. Direct demonstration, then, involves carrying out further studies involving different participants and in different settings. If the findings can be replicated (often through a series of demonstrations), then the argument for generalizing becomes stronger. Making a case simply involves the construction of a reasoned argument that the findings can be generalized. So, this would set out to show that the group(s) being studied, or the setting or period, share certain essential characteristics with other groups, settings or periods (Campbell and Stanley, 1963).

Define: External validity

Criterion validity

This is where we compare how people have answered a new measure of a concept, with existing, widely accepted measures of a concept. If answers on the new and established measures are highly correlated, then it is usually assumed that the new measure possesses criterion validity. However, as de Vaus (2002) suggests, a low correlation may simply mean that the old measure was invalid. Furthermore, many concepts have no well-established measures against which to check the new measure. Hence, Oppenheim (1992) is probably correct to state that good criterion measures are notoriously hard to find.

Define: Criterion validity

Construct validity

Construct validity is concerned with the measurement of abstract concepts and traits, such as ability, anxiety, attitude, knowledge, etc., and is concerned with whether the indicators capture the expected relationships among the concepts being researched (Cook and Campbell, 1979). As we saw above, each of these traits has to be operationally defined before it can be measured. Taking each trait, the researcher proceeds to elaborate on all of the characteristics that make up that trait. For example, if we use the construct ‘confidence’ within a particular research context this might be defined as:

· The ability to make quick decisions.

· Sticking with personal decisions once these are made.

· Strong interpersonal skills.

Define: Construct validity

You might reflect here that, in fleshing out traits to this level of detail, it is only a relatively short step to the creation of a research instrument like a questionnaire. While a test that has construct validity should measure what it intends to measure, it is equally important that it should not measure theoretically unrelated constructs (McBurney and White, 2009). So, for example, a test designed to measure attitudes to change should not contain items that seek to measure, say, extraversion.

Content validity

Content validity is associated with validating the content of a test or examination. Since it is important to create a match between what is taught and what is tested, this might include comparing the content and cognitive level of an achievement test with the original specifications in a syllabus. Let us take the case of a computer company that provides a training programme in fault finding and rectification for those retail companies that sell its products. After a two-day training programme, participants are given a 50-question multiple-choice test. The computer company will want to ensure that the content of the test is matched with the content of the training programme so that the entire syllabus is covered, and only issues that have been taught are assessed. Equally, it will want to assure itself that it has delivered the training programme at a level so that attendees learn the skills of problem solving. The assessment, then, will also have to be at this problem-solving level (rather than, say, merely applying rules, or recalling facts) for the test to be valid.

Define: Content validity

Predictive validity

This shows how well a test can forecast a future trait such as job performance or attainment. It is no use if a test for identifying ‘talent’ in an organization has both construct and content validity but fails to identify, say, those who are likely to be ‘high performers’ in a key work role.

Define: Predictive validity

Statistical validity

This is the extent to which a study has made use of the appropriate design and statistical methods that will allow it to detect the effects that are present.

Principles of reliability

According to Black (1999) reliability is an indication of consistency between two measures of the same thing. These measures could be:

· Two separate instruments.

· Two similar halves of an instrument (for example, two halves of a questionnaire).

· The same instrument applied on two occasions.

· The same instrument administered by two different people.

If we were to take another sort of measuring device, a ruler, how sure could we be that it is always a reliable measure? If it is made of metal, does it expand in extreme heat and therefore give different readings on hot and cold days? Alternatively, we might use it on two different days with similar temperatures, but do we mark off the measurement of a line on a piece of paper with the same degree of care and accuracy? For a research tool to be reliable we would expect it to give us the same results when something was measured yesterday and today (providing the underlying trait(s) being measured has not changed). Similarly, we would expect any differences found in traits between two different people to be based on real differences between the individuals and not be due to inconsistencies in the measuring instrument. Reliability is never perfect and so is measured as a correlation coefficient . In the social and business sciences it is rarely above 0.90. If a research instrument is unreliable, it cannot be valid. Like validity, there are several ways of measuring reliability. Black (2001) describes five of them.

Define: Stability coefficient

Stability

This measures the scores achieved on the same test on two different occasions. Any difference is called subject error . For example, a survey of employee attitudes towards their workplace may yield different results if taken on a Monday than on a Friday. To avoid this, the survey should be taken at a more neutral time of the week.

Equivalence

Another way of testing the reliability of an instrument is by comparing the responses of a set of subjects with responses made by the same set of subjects on another instrument (preferably on the same day). This procedure is useful for evaluating the equivalence of a new test compared to an existing one.

Internal reliability

This measures the extent to which a test or questionnaire is homogenous. In other words, it seeks to measure the extent to which the items on the instrument ‘hang together’ (Pallant, 2013; Sekaran and Bougie, 2013). Are the individual scale items measuring the same construct? Internal reliability is measured by Cronbach’s alpha test, which calculates the average of all split-half reliability coefficients. An alpha coefficient varies between 1 (perfect internal reliability) and 0 (no internal reliability). As a rule of thumb a figure of 0.7 or above is deemed acceptable. However, as Pallant (2013) warns, Cronbach’s alpha results are quite sensitive to the number of items on a scale. For short scales (with items fewer than 10) it can be quite common to find Cronbach values as low as 0.5.

Inter-judge reliability

Inter-judge reliability compares the consistency of observations when more than one person is judging. An example would be where two people judge the performance of a member of an organization’s marketing staff in selling a product over the telephone to the public. The reliability of the observation is provided by the degree to which the views (scores) of each judge correlate. Observer error can be reduced by using a high degree of structure to the research through the use of a structured observation schedule or questionnaire.

Intra-judge reliability

Where a large amount of data has been collected by a researcher over time the consistency of observations or scores can be checked by taking a sample set of observations or scores and repeating them. A further problem, and often a significant one, is bias on the part of respondents. It is quite common, for example, for respondents to provide a response they think the researcher is seeking. Particularly if the researcher is seen to be representing ‘management’, respondents may be reluctant to provide honest answers if these are critical of the organization. Even assurances of confidentiality may not be enough to encourage complete honesty.

Explore: Getting truth in surveys

Top Tip 6.4

A useful way of measuring inter-judge reliability is through use of the Kappa score (recall the calculation in Chapter 5 ), which compares the level of agreement between two people against what might have been predicted by chance.

Activity 6.5

An organization representing the interests of small businesses in London plans to conduct a survey to measure business optimism, normally a guide to future investment intentions and economic growth. The aims of the survey are to: (a) measure the current level of business optimism and compare this with levels over the last three years; (b) measure the causes of this level of optimism; (c) establish links between levels of optimism and business intentions, such as hiring new employees and investment plans.

There are insufficient financial resources to send the questionnaire to all businesses in London so you must select a sample.

1. What is the population for this research?

2. What is the sampling frame?

3. What kind of sample will you select? Justify your choice.

4. Identify dependent and independent variables.

5. Produce an appropriate research design.

6. Using the aims outlined above, construct a valid and reliable research instrument.

Suggested answers are provided at the end of the chapter.

Summary

· The structure of experimental research generally comprises two stages: the planning stage and the operational stage.

· Experimental research begins from a priori questions or hypotheses that the research is designed to test. Research questions should express a relationship between variables. A hypothesis is predictive and capable of being tested.

· Dependent variables are what experimental research designs are meant to affect through the manipulation of one or more independent variables.

· In a true experimental design the researcher has control over the experiment: who, what, when, where and how the experiment is to be conducted. This includes control over the who of the experiment – that is, subjects are assigned to conditions randomly.

· Where any of these elements of control are either weak or lacking, the study is said to be a quasi-experiment.

· In true experiments, it is possible to assign subjects to conditions, whereas in quasi-experiments subjects are selected from previously existing groups.

· Research instruments need to be both valid and reliable. Validity means that an instrument measures what it is intended to measure. Reliability means that an instrument is consistent in this measurement.

Review questions

1. The use of control groups is essential in quantitative research designs. Do you agree with this view?

2. Studies that make use of descriptive (but not inferential) statistics are of limited value. Discuss.

3. Pre-test/post-test quantitative designs have been criticized. What practical steps can be taken to address the limitations of such a design if it is the only one available?

4. Should generalization always be the goal of quantitative research?

Research Action 6.1: Designing a quantitative project

If you have decided on a quantitative approach, take the following steps, referring to the detailed checklist at the end to ensure you have not overlooked anything:

1. Revisit your ideas for possible research questions. Is it possible to convert these into hypotheses containing two or more variables that can be measured and operationally defined? Break down your questions into subordinate questions which will help to determine the research tools you use (for example, questionnaires/interviews).

2. Revisit your planned statistical methods. Are they descriptive or inferential?

3. Decide between experimental and quasi-experimental research design and write down reasons for your choice.

4. Decide on your experimental and control group.

5. Think carefully about your sampling; choose your sampling frame and the size of your sample and consider whether it is going to be possible to generalize from your sample.

6. Think about your selected research tools in light of validity and reliability.

Your Research Project Checklist

Next Steps In Your Research Project

Push your project forward with a host of resources available to you online:

· Watch videos to build your understanding of important concepts

· Read journal articles to deepen your knowledge and reinforce your learning of key topics

· Discover case study examples that help you to gain insight into real research in the business world

Watch: Designing an experiment

Read: Video game methods

Discover: Quantitative methods in HRM

If you are using the interactive eBook, just click on the icons in the margin to access each resource.

Alternatively, go to: https://study.sagepub.com/graybusiness2e