Getting Acquainted with Research Designs

GETTING ACQUAINTED WITH RESEARCH DESIGNS Research Designs

Slide 1 Transcript

Welcome to Module 3 on research and design. In addition to the overview of research methods at the course beginning, along with understanding how variables influence a research plan, and how to obtain data through sampling, this module provides a bit more information about how to approach the structure for a research design, categories and methods to conduct the inquiry.

Methodological Approaches to Research

Quantitative Approach

Qualitative Approach

Isolating a known variable to study

Identifying variables in complex situations

Obtaining numerical evidence

Predict future outcomes

Obtaining descriptions and explanations of categories Understand and interpret outcomes

ObjectiveObjective QuantitativeQuantitative QualitativeQualitative

Purpose of Research

Predict, validate, confirm

Explain, interpret, build

Nature of Research Process

Known variables, objective

Holistic, flexible, emergent

Data and Collection

Numerical, pop. sampling

Textual, non-std observations

Analysis and Meaning

Statistical, deductive Themes, inductive

Presenting Findings

Numbers, statistics, formal

Words, narratives, personal

Slide 3 Transcript Early in the planning phase, the researcher must determine how to approach the effort. Sometimes, it will be a continuation of earlier work and the methodology will be apparent. Other times, it will depend on resources and time available. The most critical component, though, is the research problem – that is, what is it the researcher is seeking to learn more about. This will drive the choice of methodology. Data and methodology are closely connected, since the type of data determines the information gained and the nature of interpretations possible. Attempting to isolate a variable or interaction suggests a quantitative methodology expressed numerically, whereas discoveries about large and complex situations might favor a qualitative approach. A mixed approach combines the two, although one method typically will be the principal structure. So, to compare the two primary methods to see which is probably best to employ, five elements could be assessed. First, what is the purpose of the study? If it is to predict or validate something, perhaps an established theory, a quantitative approach might be best. If the purpose is to describe or explore, as in creating or building a theory, the qualitative approach may be best. Second, what is the nature of the research process? If it is focused with known variables, objective and relatively context-free with pre-planned methods, then a quantitative direction would seem to be indicated. If a holistic, context-bound, emergent situation with unknown variables is key, then a qualitative approach would fit the need. Third, consider the data. As noted several times before, if numerical data, representative sampling, and standardized instruments will be used, a quantitative approach fits best. If data are unstructured and textual, in loosely structured settings using a small sample, the qualitative approach would serve well. Fourth, data analysis to determine meaning must be considered. Statistical analysis with a deductive process is clearly a quantitative methodology. If data are used to find themes and categories, a subjective and more inductive qualitative process would be indicated. Finally, how will the data be presented? If in numbers, statistics, and in formal scientific style, a quantitative method is better. Where data are given in narratives and a more personal voice, the qualitative method fits best. A mixed methodology approach contains parts of both quantitative and qualitative methods, arranged in several possible configurations and will be covered in another module more completely. One thing about the mixed approach is that there are trade offs in some areas like validity and reliability.

Categories of Research Design

Differ by degree of control by researcher

Generally Grouped in three categories

Experimental (greatest control)

Quasi-Experimental (less control)

Nonexperimental (least control) Note: Also called ex post facto — data already exist

Research Designs Quasi-Experimental

Experimental

Nonexperimental

Used to determine the causes of behavior that can

explain why it occurs

Used to identify the relationship betwen preexisting variables

Used to describe variables and predict the relationship

between variables

Slide 5 Transcript If you consult several sources or textbooks on research methodology, you may find there are various ways to categorize how research methodologies may be grouped. For this course, one of the more prominent versions is employed. The categories used for this course, described in Chapter 6 of the Privitera text, include experimental, quasi-experimental, and non-experimental designs as shown here from Figure 6.1. The key aspect that differentiates the design methods centers around the level of control by the researcher or what is found in the research setting. The experimental design has the greatest degree of control exercised by selection of participants (randomly is a requirement), how the independent variables are manipulated, and how comparison to criteria or other groups is conducted. The objective is to explain cause and effect relationships. Quasi-experimental design has many similarities to the experimental approach, but the researcher has less control over conditions and participant experiences. For instance, the researcher may not have a control group for comparison, participants were not selected randomly, or pre-existing variables or factors are present that the researcher is unable to control. In this sense, then, the quasi-experimental design will examine relationships between or among variables that are known. Non-experimental designs (sometimes referred to as ex post facto designs) have the least control by a researcher and, essentially, are used to identify the variables involved, and relationships among them, which may not have been identified previously. In many non-experimental or ex post facto designs, observations are made in their natural settings without much manipulation by the researcher.

Experimental Designs

Required Random selection Manipulate independent variable(s)

May limit potential experiences of participants

Single-case designs

Within-subjects designs

Experimental

Between-subject designsABA reversal designs Multiple baseline designs

Changing-criterion designs

Between-subjects designs Within-subjects designs

Mixed designs

Comparison group

Only design capable of demonstrating cause and effect

Isolating effects of variables requires lab/controlled settings

Factorial designs

Results may differ in natural environment Increases internal reliability

Types of designs Pre-test/Post-test Solomon Four-group Post-test Only Control-Group Within-subjects

Slide 7 Transcript So, once the method and design categories are chosen, it is necessary to select a particular design to accomplish the study. A quick visual reference is shown here in Figure 6.2, which notes several variations of experimental designs. The primary distinctions, though, are that (1) any experiment must use random selection, (2) the researcher must be able to manipulate one or more independent variables, and (3) there must be a comparison or control group. As noted earlier, the experimental designs are the only ones capable of demonstrating cause and effect. Because researchers are manipulating variables and operating in a controlled environment to limit intervening variables, some of the other potential results will not be observed. Also, because the controlled conditions are not what would be found in natural settings outside the laboratory, the results may differ. However, limiting outside influences aids in achieving a higher degree of internal reliability when the experiment is replicated. Some of the particular designs used with an experimental method might be a pre-test, post-test control group design where the control group is isolated from any influences of the independent variable, as is just observed using the same evaluation devices before and after the independent variable has been introduced to the experimental group. Often, experimental designs are found in basic research efforts to systematically investigate an issue. This, in turn, helps to strengthen theories or related assumptions. Some of the typical experimental designs are shown, e.g., the pre-test post-test version you have probably seen before. Some of the other examples are for applying more than one independent variable.

Quasi-Experimental Designs

What the researcher DOES NOT control are confounding variables, so the researcher cannot rule out alternative explanations.

One-group designs Time-series designs

Quasi-Experimental

Developmental designs

Post-test only design Pre-test-Post-test design

No random selection and/or using a comparison group

No guarantee groups are similar in every respect but may be with respect to the dependent variable

Nonequivalent control group designs

Types of designs Time-Series Alternating Treatments Multiple-baseline Single Group

Basic design Interrupted design

Control-Series design

Longitudinal Cross-sectional

Cohort-sequential

Post-test only design Pre-test-Post-test design

Post-test Only

Slide 9 Transcript Again, we can use the figure developed by Privitera in the textbook. Quasi-experimental designs resemble experimental designs except that they do not adhere to all the required elements for true experiments. For instance, the quasi designation refers to when the researcher has only partial control of the independent variable. Most pre-existing characteristics, e.g., cannot be controlled appropriately during truly random selection. However, this is also the strength of a quasi-experimental design, which is that the researcher can intentionally select particular variables or conditions that must be present in all participants. So, if random section is not used, or if there is no comparison or control group, the research will not meet the requirements of a true experiment and cause and effect cannot be established reliably. When random selection is not used there can be no guarantee that the groups are alike, so validity reliability are compromised. Still, the results, shown in the dependent variable, can be informative, particular when they are similar among the groups. Some of the design types might include observations over time, for which there are several variations, using only a post-test, and so on.

Nonexperimental Designs

Least degree of control over variables by researcher

Quantitative

Nonexperimental

Qualitative

Unobtrusive measures may be used (journals, photos)

Cannot conclusively demonstrate cause and effect

Include most of the qualitative approaches for research

Correleational Naturalistic Survey Phenomenology

Ethnography Case study

E.g., ex-post facto (pre-existing) data or information

Natural settings and interactions

Existing data

Content analysis Archival research

Meta-analysis

Slide 11 Transcript As mentioned, non-experimental designs will allow the least control over variables. For instance, if a study is done in the natural setting, little control is possible. Or, if assessing data or information that already exists. The term “unobtrusive measures” may be used when describing non-experiments because the researcher is just observing events as they occur, rather than manipulating factors to produce something observed. Like the quasi-experimental designs, the non-experimental ones cannot demonstrate cause and effect reliably. Most studies employing a qualitative approach will fall into the non-experimental category. In later modules, the procedures for all three categories of research design introduced in this module will be covered in detail.

Relevance of Hypothesis or Research Statement

Avoid “is it, isn’t it” and “does it, doesn’t it” questions

Probably wise to limit scope of inquiry

Focus is more like “degree or extent to which…”

Should consider degree of control possible

Might consider involvement of IRBs

Clarify the delimitations

Slide 13 Transcript Having surveyed the various research design categories, it may make more sense now that the initial research question or hypothesis may have shifted focus or direction. Almost certainly, you should have abandoned question like “is it or is it not whatever” or “does or does it not something.” These inquiries require a huge degree of criterion proof for determining a definitive answer. Also, most research experts would tell you they are not proper research questions. So, you may be moving more toward determining the extent to which something occurs, or the degree to which something may happen. For a graduate capstone project or thesis, the scope of inquiry should be somewhat limited. Almost certainly you will not be able to exert the control required for a true experiment, unless you have access to facilities and resources to accomplish that. Also, if you are gathering data involving human subjects, approval from the Institutional Review Board is required. In addition, if you are an active military service member, you probably need to obtain similar approval through your branch of service. Companies also have restrictions, and you may need to get approval from the lawyers – good luck on that. Back to choosing a research design, in considering the research question or hypothesis, much applies to the delimitations of your research, and to the scope you intend to cover. These considerations involve how you intend to establish validity and reliability, if appropriate, for your design.

Measures

Limits data for comparison

Instruments have set formats for reporting results

Expressed as statistics or narratives

Physical measures have boundaries

Compared with point of orientation

Scales

Nominal (name) Ordinal (rank order)

Ratio (multiples and fractions) Interval (equal units)

Slide 15 Transcript

First, measurement is limiting data, so they can be interpreted and, usually, compared with something, like a standard. The approaches and methods for research designs all yield some kind of result. As such, they must be described or expressed. As we have seen, this can be with numerical measures, like statistics, or by narratives and categories. Where instrumentation is used, the formats will be established. Measures we are most familiar with are more likely going to be used in experimental and quasi-experimental designs. The choice of measure includes boundaries, e.g., maximum and minimum limits. Some are these are physical, like light, sound, or vibration, and might be measured using scales. Other measures may be more insubstantial, like opinions, feelings, or learning. For these, we use a different type of instrumentation like interviews or questionnaires. In research, measures are compared with a point of orientation or standard, like norms, averages, or goodness of fit. For statistical procedures, nearly all measures will fall one of the scale categories. These are nominal, which means “name;” ordinal which allows a rank order (hence ordinal) and is used to compare data; interval which has equal units of measurement and an arbitrary zero point like a mean or standard deviation; and ratio which is similar to interval but has a point of absolute zero, like weight. The short version, to summarize, is nominal means one object different from another, ordinal means one object bigger than another, interval means one object is so many units more than another, and ration is one object is so many times as big.

Types of Validity

Four Types of Validity Approaches to Determine Validity

Table of specifications

Lack of Universal Agreement Specific to Situation

Content representative

Construct correlation

Criterion not observed directly

Face on the surface Multitrait-multimethod

Panel of experts

Slide 17 Transcript A variable being measured must be observable and replicable to be valid. When referring to validity of a measuring instrument, four types are used. Face validity is the extent to which, on the surface, an instrument looks like it is measuring a characteristic. Content validity says what is sampled represents the entire domain. Criterion validity is about correlation with another measure and is essential to predict something. And construct validity refers to something that cannot be observed directly but is assumed to exist – like a pattern. In research, one of the issues surrounding validity of measurements is that there often is not universal agreement about the purpose for which it is used. So, the type of validity is specific to the situation. Some of the examples for addressing this situation might be to use a table of specifications that givers samples of domain content, or a multitrait-multimethod approach that correlates two different measures with the same characteristic or asking for a judgment by a panel of experts about the characteristic in question. So, to determine validity for a variable, the instrument of measurement must be valid.

Threats to Validity

Threats to internal validity (relationships between variables)

Threats to external validity (generalizability)

History Interaction effect

Man prefers to believe what he prefers to be true.

— Francis Bacon

"

"

Maturation Statistical regression Selection Mortality Testing Instrumentation Compensatory rivalry Resentful demoralization

Interaction effects of selection bias Reactive effects of experimental arrangements Multiple-treatment interference

Slide 19 Transcript As Francis Bacon, one of the original scientific thinkers wrote once, “Man prefers to believe what he prefers to be true.” So, research designs are efforts to see more clearly what the results are and what they mean. Are variations in the dependent variable attributable to other causes? Can we confidently conclude that changes in the independent variable caused observed changes in the dependent variable? If a study has low internal validity, then we must conclude there is little or no evidence of causality. Some threats may be related to other variables, e.g., extraneous or confounding ones. Internal validity threatens to compromise our confidence in saying a relationship exists between independent and dependent variables. Some types of threats include history, which is when some unanticipated event occurs while the experiment is in progress and affects the dependent variable. For maturation, the changes occur as the result of normal developmental processes. When measurement of the dependent variable is not reliable, scores will regress to the mean. As noted earlier, if selection of groups is not equivalent, validity may be threatened. When participants leave the groups during the data collection period, it is termed mortality, speaking of the sample integrity. A pre-test might sensitize a participant in unexpected ways in their performance on a post-test. If changes occur in the way the dependent variable is assessed, a measurement threat may be present. The term compensatory rivalry refers to treatments where desirable goods or services received by one group become known to other groups. Or the reverse, where participants in one group learn they received something less desirable and are resentful. Threats to external validity compromise our confidence is stating results can apply to other groups, for instance. An interaction effect happens when, e.g., the pretest interacts with the treatment and causes some effect that will not generalize to the untested population. There also can be an interaction effect if there is selection bias due to nonrandom selection. The threat of reactive effects of experimental arrangements would occur in situations where the Hawthorne effect occurs when participants know they are involved in something novel. Multiple treatment interference happens when the same participants receive two or more administrations, as in a repeated measures design, and there may be a carryover effect between treatments. Well, this concludes our overview about research designs. Thank you for listening, and that wraps up this part of the module, so have a great week.