Literature paper LJ

Updated July 2021 2

Becoming an

Intelligent Consumer of Research

(Journal Articles, Masters Theses, Doctoral Dissertations)

A Study Guide and Textbook for PED 598

(Can be used in PED 511, PED 514, and other Kinesiology Program Courses)

Dr. Terry Conkle

Alabama A. & M. University

Kinesiology Graduate Program Coordinator

Reading, Comprehending, and Critiquing Research Reports/Studies/Articles

Introduction

Becoming an intelligent consumer of research is essential in graduate and professional schools, because graduates of such programs will be (or should be ) consumers of research throughout their respective careers. Using their knowledge of research and research methods to analyze or evaluate research reports that have been published as articles will be vital. And, if a student pursues further advanced education, they will most certainly need to use such knowledge. Furthermore, in many cases, students are expected to produce research studies of their own. Thus, the initial goal for students embarking on a quest for greater knowledge is to set the goal of becoming a proficient reader and consumer of research.

If a student becomes equipped with the skill to read and analyze research, they will be positioned to make informed judgements or opinions about how they perform their professional duties. and responsibilities. Research results often are used to initiate or justify changes in all areas of education, kinesiology, recreation, fitness, and the broad sport sciences (and most other professions for that matter). Maintaining knowledge and skills based on research trends provides a basis on which to evaluate alternatives to best practices (e.g., physical therapy, among many professions).

Researchers take many forms. They may be scientists missing and testing chemicals in white lab coats. It could be business executives looking at charts and graphs, or it could be pollsters seeking to learn the political preferences of a populace. Possibly, it is government representatives tabulating information or data from the national census. Or, it may be a high school student looking through mounds of books in a library trying to do a “term paper” on Shakespeare or The American Revolutionary War for Senior English. However, those activities are not necessarily research. Simply mixing chemicals is not research, using the scientific method to find a new and better compound of chemicals for a better medication would be. Those business-people looking at several illustrations may be simply trying to get a visual of their sales history. The high school student is simply trying to compile known facts to support a thesis statement that (s)he hopes will earn a good grade and graduate from high school. Something is research only if previously separate concepts, facts, materials, etc. are pulled together to shine new light on a problem (or question) that requires a solution or answer.

Research is the application of scientific methods to the investigation of problems. Research in the end is searching for “knowledge” and “Truth” – in the academic, scholarly and scientific world, we “know” nothing unless or until a preponderance of research findings demonstrate something as being “Fact(ual)” or “Truth.” The scientific method or process involves looking at relationships between 2 or more variables, and includes:

Observing or becoming aware of a problem

Defining or stating the problem formally and specifically, or asking a question that needs answering

Determining possible solutions to the problem (or forming a “Hypothesis” and its “Null Hypothesis”)

Conducting a study to test the hypothesis

Drawing conclusions about the study’s results

There are several other terms with which a reader of research should know, to get started. Most will be discussed later in this primer.

Hypothesis - a specific, clear, and testable proposition or predictive statement about the possible outcome of a scientific research study based on a particular property of a population, such as presumed differences between groups on a particular variable or relationships between variables. To sum it up, a hypothesis is a projected tentative answer to a research question.

Null Hypothesis - is a type of hypothesis used in statistics that proposes that there is no difference between certain characteristics of

a population (or data-generating process). H0 is the commonly accepted fact; it is the opposite of the alternate hypothesis. Researchers work to reject (nullify) the null hypothesis. Researchers develop an alternate hypothesis, one that they think explains a phenomenon, and then work to reject the null hypothesis. The word “null” in this context means that it’s a commonly accepted fact that researchers work to nullify. It doesn’t mean that the statement is null itself!

Variable – something that can assume different values (quantitatively or qualitatively) in a given situation. A variable represents

a measurable attribute or trait that changes or varies across the study whether comparing results between multiple groups, multiple people or even when involving a single person in a study conducted over time. There are six common types. In all there are six basic variable types: dependent, independent, intervening, moderator, controlled and extraneous variables.

Independent Variables (IV) and Dependent Variables (DV)

In general, experiments purposefully change one variable, which is the IV. But a variable that changes in direct response to the independent variable is the dependent variable. If study is testing whether changing the position of an ice cube affects its ability to melt, the change in an ice cube's position represents the IV. The result of whether the ice cube melts or not is the DV.

Intervening and Moderator Variables

Intervening variables link the independent and dependent variables, but as abstract processes, they are not directly observable during the experiment. For example, if studying the use of a specific teaching technique for its effectiveness, the technique represents the IV, while the completion of the technique's objectives by the study participants represents the DV, while the actual processes used internally by the students to learn the subject matter represents the intervening variables. By modifying the effect of the intervening variables -- the unseen processes -- moderator variables influence the relationship between the IV and DV. Researchers measure moderator variables and take them into consideration during the experiment.

Constant or Controllable Variable

Sometimes certain characteristics of the objects under scrutiny are deliberately left unchanged. These are known as constant or controlled variables. In the ice cube experiment, one constant or controllable variable could be the size and shape of the cube. By keeping the ice cubes' sizes and shapes the same, it's easier to measure the differences between the cubes as they melt after shifting their positions, as they all started out as the same size.

Two additional types of variable are Extraneous and Confounding.

Extraneous Variables

A well-designed experiment eliminates as many unmeasured extraneous variables as possible. This makes it easier to observe the relationship between the independent and dependent variables. These extraneous variables, also known as unforeseen factors, can affect the interpretation of experimental results. Lurking variables, as a subset of extraneous variables represent the unforeseen factors in the experiment.

Confounding Variables are another type of lurking variable, which can render the results of an experiment useless or invalid. Sometimes a confounding variable could be a variable not previously considered. Not being aware of the confounding variable’s influence skews the experimental results. For example, say the surface chosen to conduct the ice-cube experiment was on a salted road, but the experimenters did not realize the salt was there and sprinkled unevenly, causing some ice cubes to melt faster. Because the salt affected the experiment's results, it's both a lurking variable and a confounding variable.

When considering all the variables associated with research, it all can be daunting and overwhelming to both experienced and novice research consumers. To help simplify matters, researchers should precisely define key terms associated with their study, to help themselves and, so readers of their report (article, thesis dissertation, etc.) know what is being studied. To ensure that the task of observation in research is manageable and thorough, there should be operational definitions for the variables involved.

Operational Definition - a definition that assigns meaning to a variable by stating what is the observable behavior, or what operations will represent the variable. It explains how some abstract, ambiguous, or general idea will be defined in a certain setting. One way of doing this is to explicitly state how the concept will be measured. Some examples (among myriad) of terms that need operational definitions are: Achievement, Intelligence, Time-on-Task, Athlete, etc.

An operational definition permits both precise observation and meaningful communication about what is (was, will be) observed. Any two people (or especially researchers) may deeply disagree concerning what a concept is or means, but when this is stated up-front before study begins and then again in the research report, all misunderstanding or disagreement is prevented.

Explanation – one goal of science is to find the level of association between variables and explaining it. Explaining associations is different from merely reporting facts. We need to see how they are related. For example, is there an association or relationship between student height and intelligence? Or, is there an association between socio-economic-status in adulthood and years of schooling? It is more than a matter of them being related or not related, but a matter of explaining exactly to what degree and what variations there may be since each variable can be observed quantitatively.

Prediction – this goal of science is based on explanation of the relationship or association between variables. Prediction involves make the best possible guess regarding how a given set of variables will be associated or related, based on what is known about the past and current relationship of the specific variables. In such situations, there must be existing knowledge of an association before speculating that the information concerning one variable can be used to predict the quantity or quality of a second variable.

For example, can elementary school grades be used to predict high school achievement? Can high school grades (GPA) be used to predict success in college? Can undergraduate GPA and GRE scores (used on a sliding scale) be used to predict achievement in graduate or professional school? Can the combined scores (data) from team tryouts predict success in the interscholastic sport setting? Or, can “NFL Combine” aggregate data predict success in the NFL? Researchers start with the known association(s). The strength of the association(s) between variables can be measured and utilized to predict future success based on what is known.

Often researchers are not content with finding associations among variables. They are curious as to whether they can control the relationship between phenomena. Control is the ability direct or influence a variable to bring about change in a related variable.

When researchers purposely manipulate or change the values of a variable they want to observe what happens to a 2nd related variable in the same setting.

Independent Variable – the manipulated, altered, changed variable in a research study.

Dependent Variable – the variable that changes in response to a manipulated variable (if there is a true cause-and-effect relationship between them). If there is no cause-and-effect, then there will be no change in the DV due to alterations in the IV.

Experimental studies are used to determine cause-and-effect. Typically, these are studies looking for differences between group means on some variable interest.

However, for example, it may be a study comparing Medicine A, Medicine B, and Medicine C to determine which is most effective for curing patients with COVID-19. The DV is patients cured, the IV is medicine (3 different pharmaceutical agents administered, 1 per group – the changing or manipulated variable). They may all cure it, but which one does it faster? Or, which one has the highest survival rate after 30, 45, 60, 90 days following treatment, and is there a statistically significant difference? Once a drug is found to work, it is possibly tested by gender, race/ethnicity, age-groups, etc. as the manipulated (demographic) variable, seeking significant differences between those categories of subjects.

Research Reports (also known as: “Original Research,” or “Research Article”) – gives a background for and history of a study, including what the researcher(s) wanted to find out (seeking solutions or answers to a problem) and why it seemed worth discovering, how they gathered the information, and what they thought it all meant (IMRaD format).

What is in a research report?

· They typically contain a clear statement of the “research question” or “research problem” (or maybe a “hypothesis” or hypotheses) that the investigator(s) addressed and that guided key decisions about the method of inquiry throughout the study. The question/problem is usually defined BEFORE the study began, and its source analyzed and developed.

· They situate or identify the purpose of a study (as much as possible), and the research question(s) stated about the existing body of knowledge regarding the problem.

· They commonly explain theoretical assumptions with which the research question(s) and ultimate data were framed (and understood), and upon which analysis and conclusions were based.

· They describe data collection procedures that were planned in before (or possibly modified during) the study’s progression.

· They provide details concerning the observations and recording of data, with accuracy and precision, that are appropriate to the demands of the research question(s).

· They demonstrate that the data quality was vital during the study, which is confirmed by providing information regarding reliability and validity of measurement procedures (or about other qualitative indexes related to type of research involved).

· They discuss how data were organized and specify the means of analysis.

· Results of the data analysis are explicitly linked to the research question(s) or problem(s).

Research is a fundamental aspect of education, if not life. Education focuses on informing people and broadening their knowledge of basic or essential information. This is called being literate or overcoming illiteracy. In terms of research, and learning to read it with understanding, informed consumers of research are known as having “Research Literacy.” Ultimate “Research Literacy” is, having an ability to read and understand research and being able to do it critically (by analyzing) and evaluating the quality of the report (someone who can do that with skill is a step beyond being research literate). In summary, an original scholarly or scientific research article (report) will have the following components:

· Title

· Names of authors in a specific order that should not be changed when referencing, and the institution they represent

· Abstract (sometimes articles may not have this; but, most do in contemporary journals)

· Introduction (sometimes this is not labeled because it is understood to come first, between the Title and Methods Section

· Methods (sometimes labeled as the Methodology or Procedures Section)

· Results (sometimes labeled as the Findings Section)

· Discussion (sometimes labeled as Conclusions

· References (all sources cited within the article’s text, alphabetical order by the lead authors’ last names)

I = Introduction

M = Methods

R = Results

and

D = Discussion

If the article reads like a “Term Paper,” and does not strictly follow such a format, then the article is may be a published Literature Review. If the article has an opening page “Header” that includes wording similar to what follows, it is likely a Meta-Analysis Research Report: “Meta-Analysis,” “Critical Review,” or “Systematic Review.” These ARE Research Reports of a special nature. If in doubt, ask your professor in a timely manner!

The Quantitative Approach to Research

This type of research follows the Scientific Method

· Making careful observations (from existing literature)

· Making predictions from those observations (research hypotheses, research questions, problem statements, etc.)

· Testing the predictions (data collection and analysis)

· Using data findings/results to support or modify the predictions (drawing conclusions of numerical or statistical fact)

IF/When there are adequate details in a research report, a study is deemed replicable. Research replication and extension broadens what is known about an issue and leads to increasing “the body of knowledge” (truth, or what is known).

Statistics are:

· A useful and meaningful tool for describing data sets

· A useful tool for testing data

· A powerful tool used to model the nature and operation of the empirical world

Types of Statistics in Quantitative Research

Descriptive Statistics

The simplest use of statistics is to gather and organize data in a meaningful way. This is done by creating distributions. Distributions are assemblies of related measurements that are combined to get a group picture – mean, median, mode, range, standard deviation (e.g., height weight, shoe size, 40 yd., sprint time, IQ, etc.). The distribution on which many meaningful data sets are judged is the “Normal Distribution, or “Normal Curve” (often called the Bell Curve). Generally, sophisticated statistical analyses involve the mean (averages) and standard deviations of groups.

Inferential Statistics

The second key function of statistics is to help make decisions. To draw an inference (or infer) decisions are made on statistical reasoning (essentially “yes” and “no” answers) regarding available information. In other words, are two or more things different from each other? Or, whether they are simply different-looking?

Inferences and Frequencies

When frequencies of occurrence are compared, one is observing two or more groups and deciding if frequencies are about the same for each. Logic dictates that in daily life there are generally not identical frequencies among groups, but at what point are the frequencies considered similar and when do they become significant? Are the frequencies (or ratio) of male to female U.S. Senators approximately 50-50 (like the population); or, is it more like 80-20? Regardless, does the difference occur by chance or is it greater than by chance (using inferential statistics such as Chi Square)?

Inferences and Relationships

Another “yes and “no” question concerns whether a given relationship is greater than one might expect based on chance. When observing variables that are measurable but not “controlled,” there may be some level of co-relation or correlation. Outdoor temperature can be measured for high, lows, and averages each day in our hometown, but it cannot be controlled. The question is, is there a relationship with the daily temperature each day and the number of pints of ice cream consumed each day in our hometown? The latter is not controlled either. The point is, one could determine indeed if, as the outside temperature rises, the number pints of ingested ice cream also rises; but, to what degree? The correlation will not be perfect, due to the rise and fall of sales in winter and summer that may or may not be attributed to the temperature. Inferential statistics, determining a correlation coefficient ranging from -1.00 to 0 to +1.00, can provide an answer.

Inferences and Hypotheses

Sometimes the problem of interest is whether two or more conditions are like one another; or, whether one is different. This could include having one control group and one treatment group for comparison. The treatment group is manipulated or treated in some way to determine if the target variable has any impact. For instance, one group of 20 children is taught to read code-breaking books and one group reads comic books. The treated group receives code-breaking training, and the control group of children reads comic books of choice. At the end of 30 days, each group is given a coded message to break, each child is given the code and the elapsed time is measured for each child in each group. It would be expected that the average time needed to solve the code would have some degree of variability, but when the times are compared for each group (using an independent t-test possibly) the code-training team is found faster. And, the speed difference is greater than what would be explained by chance, which validates to some degree that code training works.

Inferences and Probabilities

Each of the previous examples dealt with whether there were differences between groups. But were the differences statistically significant? When reporting significance, researchers note with an asterisk (13.67*, for example) the computed value of the finding is significantly different from chance. Researchers link the computed value to a preselected probability level:

13.67* p < .05 – less than 05% probability of occurring by chance

13.67* p < .01 – less than 01% probability of occurring by chance

13.67* p < 001 – less than 999 times out of 1000, or 99%, probability of occurring by chance

13.67* p < .10 – less than 10% probability of occurring by chance

Inferential Statistics are more powerful when the sub-sets or samples of the population of interest are adequately represented. To have statistical power or robustness, each group should contain at least 100 subjects per group, although some will argue a number as low as 25 subjects per group is sufficient. The bottom line is, there should be a sample size large enough to observe a normal distribution.

The Qualitative Approach

Whereas quantitative research deals with numbers, measurement, prediction, controlling, etc., qualitative research explores, digs, and seeks to understand what things mean. Quantitative and Qualitative Research are independent of each other, mostly.

· Qualitative research targets “meaning” over fact, it is a systematic empirical inquiry into meaning (finding depth or a richer picture of a given phenomenon)

· Qualitative research focuses on understanding rather than knowing (understanding leads to greater insight and illumination)

· Qualitative research looks at differences in kind rather than differences in degree (what happens when things change, what cultures and traditions how do they change, for example?)

Qualitative Methods

· Observations - Paying careful attention to and documenting via observation (journal-writing of sorts)

· Interviews - Determining what individuals think and believe (there are several ways in which this may be done)

· Focus Groups - These play a key role in areas of marketing and research, when ideas and insights can be obtained from small group collective group of individuals

· Material Analysis - this includes study of how humans make and use things, whereby the collective total of things for a given group provides a material record or culture – telling more about those who build something and prize it

· Archive and Historical Records Analysis - searching these types of can be rich sources of information for insight and meaning

· Interpretive Analysis - this involves “careful” reading of texts, customs, patterns of behavior, habits, celebrations, rituals, etc., to dig under the surface for connections to areas that may initially seem unrelated

· Participant Observation - Researchers join into the lives and activities of those people they are studying, often with intent of improving the lives of those they are studying (Think Bissinger’s 1990 book, “Friday Night Lights” which involves a few of the aforementioned methods in conjunction with this one).

· More specific methods include: Ethnography, Grounded Theory, Case Studies, Narrative Analysis, Oral History, Critical Theory, Action Research

Association Studies

When a research project is or has been conducted mainly to assess the degree and direction of association (whether correlation or prediction) between 2 or more variables, it is considered an Association Study (as far as Dr. Conkle is concerned).

Difference Studies

In research studies where the (described) procedures are seeking an answer to significant differences between 2 or more groups, it is considered a difference study. It is challenging to find cause-and-affect relationships unless at least 2 groups of subjects (people, lab rats, etc.) are involved – with 1 exposed to a potential “cause” and the other is not (often called a control group). Comparisons of groups, and their important means (averages) on some variable of interest, are the primary reason these are called difference studies.

Meta-Analysis Studies

Due to technology advances in recent years, a new type of Quantitative Research as arisen in which multiple studies, that have the same focus, are combined to derive a single result that provides clearer conclusions and has considerably more persuasive power than leaving the earlier studies as independent works that may seem contradictory on the surface, or that do not seem to have common findings. This technique is called Meta-Analysis, a glorified version of a literature review with complex statistical analysis. Original data are not collected but aggregate data from numerous studies are analyzed using special statistical formulae. Effect Size is the resulting number when between-group and within-group differences are compared among all the studies of similar focus. The reporting of Effect Size (via statistical analyses) is a key to differentiating informative or narrative literature reviews that have been published from a Meta-Analysis!

Understanding Titles and Abstracts

Research Report Titles should provide sufficient information for readers to determine if the work is potentially useful for their purpose, or if they should look for more useful articles. There are five types of titles that can help readers understand the type of article it may be: “Situation,” “Process,” “Equation,” “Theoretical,” and “Indirect.”

Situation = Describes a certain situation or ongoing state of affairs, particular activities, or specific populations (or each of the latter

two); Situation Titles describe research where observation is prevalent concerning the variables of interest

Process = Something that is either happening in which the researcher is directly involved, or something the researcher intends to make

happen; they are observing something or implementing a process – they are manipulating a variable or exerting some form of influence or control to determine if change occurs. This is often done in educational or societal research, and can be found equally in qualitative or quantitative research.

Equation = These are most common in quantitative study reports. Such titles convey equations to describe key variables of the study

and their relationships to each other.

Theoretical = This type of title is less common than others, because it is used only when “theory” is a key element of the report. Such

titles are used when viewing a phenomenon relative to a larger or more abstract theoretical area.

Indirect = This type of tile is very uncommon, because it is designed to raise reader interest without really giving much detail

concerning the article’s content.

Abstracts are the second thing a reader typically will see, and it should definitely help readers decide whether to read the complete report or if reading the complete article could be beneficial, or if it will be a total waste of their time.

The Abstract (if good) will be between 100 and 250 words, depending on the journal specifications for authors. A good Abstract will:

· Provide an overview of the article

· Addresses how and why the study was conducted

· Summarizes key findings

· Describes the participants (subjects) and setting of the study

· Clearly states the study’s purpose

· Specifies the type of analysis used

· Summarizes key conclusions

An Abstract may be written as sentences that make-up a paragraph. Another way Abstracts are constructed is in paragraph form with parenthetical sub-headings inserted in the sentences. Still another way to format Abstracts is, to see it in outline form with sub-headings followed by one or more sentences summarizing key information regarding each sub-heading.

Study Purposes and Rationales

As indicated earlier, traditionally the Introduction has not been labeled in Research Reports, because it was presumed that readers understood it was the initial section of the article, but recently some journals are showing the Introduction heading in articles. The Introduction (in many articles) will not have sub-headings, although it does contain parts. Long Introductions, however, may contain sub-headings such as: Context, Background, Rationale, Purpose, Research Question(s) (or Hypothesis or Hypotheses), and Argument.

Researchers conduct studies because it matters to them. It is even better when researchers conduct their research because it adds to the body of knowledge and it matters to many others what the results may be. Regardless, there is always a Purpose and Rationale for a research study.

There are four types of Research Purposes: Exploration, Extension, Expansion, and Correction.

Exploration = With this type of research, the authors are working in an area where things are often poorly understood,

and they are exploring .

Extension = With this type of research, the authors explicitly state that they are building on ( extending ) previous work

(whether the earlier research was theirs or others’)

Expansion = With this type of research, the authors are attempting to extend ( expand ) the body of knowledge into

new or complex areas

Correction = With this type of research, the authors attempt to set the record straight on an issue with their current article.

There are five types of Research Rationale (these are implied in the context of the Introduction): Crisis, Importance, Gap-filling,

Depth, and Commitment.

Crisis = A Crisis Rationale argues (seeks to convince readers) that the research is important because it looks at an area of

education or society as a whole.

Importance = An Importance Rationale argues (seeks to convince readers) that the research problem is an important matter and

that the field or profession should not ignore.

Gap-filling = A Gap-filling Rationale argues (seeks to convince readers) that the research topic is already accepted as important to the

field, and that this study goes a step further and fills holes in the body of knowledge

Depth = These rationales have often been related to qualitative studies. Researchers must make a case for why their study is

important and that the topic of focus should be seen in a deeper manner. This type of research and rationale must meet the

“so what?” test – why does the study matter in the least?

Commitment = These rationales also are associated with qualitative studies, most closely with participatory action research, and

provides reasons why this type research approach is the best for a given problem.

Research Questions

For a research study to exist, there MUST BE at least one question asked that requires an answer or solution. Once research questions are established, it then vital to justify it/them. Arguments are how research questions are justified, and they can take many forms. Regardless of form, the question and argument must be grounded in the existing literature, which means there will be citations of the most relevant sources and all of those will be referenced appropriately at the end of the article in the References section.

Research Questions can have problems, and there are multiple things that can make a question bad, below are four examples:

· It is difficult to find or identify

· It is not stated clearly

· It cannot be tested properly

· It is a statement of opinion rather than a question

Arguments justify research questions, because without arguments to support them the question(s) are/is simply a claim. One must ask the following questions (and hopefully see answers in the Introduction of the article) when reading critically:

· Why is/are the question(s) important?

· Is/are the question(s) justified and can it/they be examined?

· What sort of context sets-up where the authors are coming from?

· Do(es) the research question(s) make any sense?

The general principles to which all arguments must attend are:

· It is clear

· It is logical

· It makes a case for the research question(s)

Regardless of the argument content, there are three forms they can take: Set-up, Support, or Set-up and Support.

Set-up = this type of argument builds a case before the research question is asked. Authors do not want to actually state the question

before they have explained why it is important, where it comes from conceptually, and how it is logical to ask it. This is the

most common situation.

Support = If authors believe the research question(s) should make sense to readers, automatically, they will still provide a literature

review to develop a case for the question(s).

Set-up and Support = This type of argument is used when an author feels they should first put the research question in some context,

and then support the question.

To decide what type of argument is being made in an article, a simple process can be used:

· Find the research question(s)

· If the article begins with a research question, then it is often using a support argument

· If the introductory part of the article ends with a research question, it is often using a set-up argument

· If the research question(s) come(s) somewhere in the middle, then a set-up and support approach is often being used

Understanding Methods and Procedures

Following the Introduction section is the Methods (aka, Methodology or Procedures) sections of the work. In this section the authors discuss the “who,” “what,” “where,” “when,” and “how” of the study; or, a description of Subjects, Instruments or Instrumentation, and Procedures (S-I-P) followed for the study.

Subjects

When considering the subjects involved (not “used”) in a research project, there are characteristics (or demographic variables) of interest. Those (the “Who”) should be described in the “S” part of the Methods section. Because most educational research involves the actions or behaviors of human beings, there are observations or manipulations of those actions and behaviors. Ideally, every “relevant” human being would be a study participant in every research study; BUT, that is not practical or logistically possible. To obtain subjects for each study conducted, sampling strategies are implemented. Sampling, is the act obtaining a subset of subjects from an overall population of interest for a given research problem (or research question). There are many types of sampling available for researchers, among those commonly utilized are: Random Sampling, Stratified Sampling, Purposive Sampling, Secondary Sampling, and Convenience Sampling.

Random Sampling - This is a very commonly used strategy. It occurs when the researchers desire some, but not all, of the people

or items that are available. The number of subjects is selected for the study, then they are randomly selected from the available population pool. A computerized random number generator can be used, or a Table of Random Numbers utilized, to ensure the sample is truly random, and representative (in theory) of the population it represents. This strategy permits good “Inference” and allows “Generalization” back to the overall population. Random selection also permits dividing the population into two or more groupings (one or more treatment groups and a control group) for statistical comparison(s).

Stratified Sampling - A Stratified Random Sample can be used to select a specific number or percentage of subjects (items or people)

of a certain type, attribute, quality, or characteristic. This ensures that the sample of interest shares as many of the important characteristics of the target population (e.g., age, gender, ethnicity, social status, economic status, etc.) as possible.

Purposive Sampling - This type of sample is most often found in Qualitative Research. The sample includes persons having unique

backgrounds, experiences, or characteristics that make them the target of individual or small group study.

Secondary Sampling - Secondary samples are found in studies whereby researchers utilize databases of existing information that have

been generated for earlier purposes. In the 20th and 21st century (historically), and in an information-rich age of technology, researchers are capable of viewing data, and looking at/for patterns that have developed over time - or for testing hypotheses. This one of the few sampling techniques that rarely involves living subjects directly.

Convenience Sampling - This is exactly what it sounds like, taking samples of subjects from the overall population of interest that is

conveniently available for study. This a popular sampling technique not only due to the “convenience” of getting subjects, it is also inexpensive, time-efficient, and makes good use of local resources that are readily available.

Instrumentation

Typically, research studies have key variables that are identified, from the beginning, to measure. Instruments of some sort are used to collect and measure those variables. This is the “What” part of research Methods. To answer research questions, data must generally be collected using some form of instrument(s). Innumerable research instruments exist to gather data, following are some major types of instrumentation: Checklists, Psychological Tests, Cognitive Ability Tests, Physical Abilities Tests, Attitude/Opinion Surveys, etc.

For some studies, it is not possible or desirable to use common research instruments. In such cases, other data collection strategies and procedures are used. Following are some examples:

· Basic Observation – the researchers observe and document what occurs

· Audio- or Video-taping subjects, alone or in groups

· Measuring medical or physiological samples (e.g., blood, urine, muscle biopsies, blood gases, saliva, hair, etc.

· Analyzing academic projects or test data

· Gathering data from the public domain sources, including the Internet

· Gaining access to private or specialized data sources and archived collections

Procedures

After Research Questions have been asked, there must be some thought given to “How” they will be answered. To do this, attention turns to what type of research design and analysis are most appropriate. There are innumerable designs and analyses available for researchers to help answer posed research questions.

“Design” is the actual plan used to answer research questions. Some of the major issues of practicality that can arise when research designs are implemented include:

1) Identifying and properly measuring the correct variables is vital for answering research questions (e.g., a 40 yd. sprint is not a good measure of aerobic or cardio-respiratory fitness; catching a baseball in a glove is not a good measure for catching all types of basketball passes, though it is indeed a measure for “catching in general”)

2) Relevant variables must be carefully and specifically identified in quantitative studies. Careful instrumentation and data collection are crucial.

3) Controlling as many variables as possible, that may interfere with findings in quantitative studies, is important. Some variables are impossible to control or remove, but it is still a key issue to resolve as much as is possible.

4) With qualitative studies, it is vital to ensure that participants are giving an accurate, reliable and true picture of their actions, behaviors, and responses. Getting data from multiple sources and ensuring there is agreement among the sources is critical. This process is known in Qualitative Research as Triangulation .

5) In qualitative research, as much of a complete picture as possible is the goal. To ensure this, researchers often collect data until nothing new appears and everything is repeating. This process is known as Saturation .

To build on the basic research design of a study (e.g., frequencies, means, standard deviations, etc.), researchers there are technique-specific designs that are commonly used.

Correlational Design - Two or more variables are compared to determine their patterns of association or relationship with each other.

Such a design is often used as preliminary step of experimental designs, or can be an end unto itself. An example of this is,

smoking and lung cancer instances were compared to determine a correlation, once smoking was found associated with lung cancer more precise research was conducted to determine whether smoking caused lung cancer (cause and effect).

Experimental Design - Hypotheses are tested in this type of design, by isolating and controlling for relevant variables. If isolation

and control are precise enough, researchers can make a claim that one or more variables might cause the changes in one or

more other variables. Some researchers vow that this is the only kind of “True Science,” or “Real Research” due to

determining “causation.” It is the most advantageous of all quantitative research designs, and is desirable whenever it is

possible to use it. For example, a control group lifts weights using any system they want, a second group uses System A and a third group uses System B. It might be found that Systems A and B produce more muscular strength gains than doing a random weight workout plan, but System B is significantly more beneficial than System A.

Quasi-Experimental Design – In educational research, this is a very common design. Pre-existing and ongoing processes are treated

as if they were actual research treatments or manipulations, and the changes or impacts are measured. For example, studies have long shown that students in small class sizes achieve higher and earn better grades than students in large classes.

Meta-Analytical (Critical Review, Systematic Review) Design - This is a study of previous studies, or an analysis of previous

analyses from multiple related and relevant studies on a given problem. This design collects results from similar studies, to determine what key results the given studies have in common.

Computer Simulation – As technology has improved, an emerging design is to have computers “crunch the numbers” on massive

amounts of data and create computer simulations of various processes of interest (flying planes, driving cars, sport situations, etc.).

Grounded Theory - This is a qualitative design, but it is the one that most closely relates to a quantitative design. This design uses

careful observation strategies, in which “grounded theorists” seek to gather data and observations to form a baseline, and then build coded structures that permit building new theories. This is the direct opposite of traditional theory, whereby theories are first proposed and then tested.

Analysis – When considering Analysis, this is what is involved with testing data that has been generated by a chosen research design.

There are innumerable analytical strategies available to researchers. Another Handout may/will be posted by Dr. Conkle regarding Statistics and Analyses (depending on what course you are in); but, following is a sample of common Analysis techniques.

Correlational - These analytical designs produce correlation coefficients. The most common types include a Pearson Product-

Moment Correlation in which two variables are compared to determine an association or relationship. Multiple Correlations (in which more than two variables are compared, and Spearman’s Rho Correlations (used for ranking data), and Point Bi-serial Correlation (used for Categorical or Demographic type data).

Experimental / Quasi-Experimental - These analytical designs test hypotheses. Some of the more common techniques are the

Independent t-test (where two groups are compared for significant differences); the Dependent t-test (where measures from one group at two separate times or conditions are compared); and, ANOVA [ANalysis Of VAriance] where more than groups or conditions are compared. Concerning ANOVAs, there are many types of this and which one is used depends on the research or test conditions – so readers should pay close attention to what the researchers are attempting to test when any sort of ANOVA is used.

Modeling - A simple model design uses Multiple Regression as a tool. Multiple Regression seeks to improve the predictability of a

target variable (such as a weather forecast, or probable winner of an athletic contest, among more serious research problems) by considering its relationship with other relevant variables. This is an extension of correlational analysis. As the modeling techniques become more complex on the modeling ladder, tools such as Logistic Regression, Cohort Analysis, and Structural Equation Modeling can be used. There are many other types of Modeling Analyses that can be used, but thos specified here are most common.

Logistical Concerns in Research

Typically, researchers seek findings that are as “Generalizable” to the overall population as is possible. Generalization is an important aspect and reason for conducting research studies. Any part of a study that deals with time or timing of the process, that answers the “When” question. Any aspect of the study that deals with location(s) in the process, that answers the “Where” question of design in the Methods section of a research report or article.

When? – The most important role that time plays is when it is built into the study’s design. There are three main uses of time that are found in research:

· Repeated Measures, where the sample is measured on more than two occasions on some target variable (pre-test and post-test maybe, after some sort of intervention or practice, or an acquisition test and a retention test at some point much later, Graduate “Comps” for example compared to tests during each course).

· Another type of Repeated Measures design would be to identify a sample and divide the subjects into two groups or more (Treatment Groups[s] and a Control Group). Each group (theoretically identical if random assignment to groups was done) is measured initially on some target variable, after the research treatment, is re-measured on the variable(s) of interest. If the treatment of interest is effective then one group should have significantly different performance data – in the case of two groups, the treatment group should significantly outperform the control group.

· Longitudinal designs involve taking measures on targeted variables across long periods of time. Measures are taken on a key set of variables at the study’s beginning, then observed and systematically re-measured at regular intervals over the course of one or more years. Studies with twins that have been conducted are possibly the best example of this.

· Sometimes Cohort Studies, when one era is targeted for special study, are planned. The Cohort Effect is a confounding variable in some Longitudinal Research. This sometimes leads to study of phenomena that arise. One example includes each Freshman Class that enters a school (each class is a Cohort), and then each class of students are followed and compared for the 4, 5, or 6 years they are in school. Another example is, comparing Baby Boomers, Generation, X, Generation Y, Millennials, etc. on specific target variables over a portion, or all, of the lifespan (e.g., Medical Records, among countless other things).

Where? – Physical location (e.g., state, nation, region, etc.) may be important in a study, for it can be intentionally minimized/limited

or it may be intended as a more broad and comprehensive aspect of the design. However, to make the study’s findings as Generalizable as possible, the setting of the study should be minimized as much as possible. In some cases, the experience of one group of subjects may be very different than another group of subjects and potential geographical or cultural differences may be of interest. The latter is especially true when considering Qualitative Research. Location factors may play a major role when ethnographies are of interest, special locations for the study could be key when planning the study.

Understanding Research Results

Every research report will have this! It is where the authors answer their research questions. Because most research of interest (in physical education and sport-related areas – or “Education”) is Quantitative, four types of research questions can be asked and answered, depending on the nature of the planned study.

· Are characteristics of the various samples, groups, and/or sub-groups representative of the target population? And, what tools can be used to organize and describe resulting data so systematic decisions can be made regarding those characteristics?

· What are the various relationships among key variables that are measured in the study? What tools can be used to determine those associations and whether they are significant?

· What tools can be used to test hypotheses that are raised in the study, to determine if statistically significant differences exist?

· What tools can be used to organize various relationships, among key variables, into models that explain and/or predict target behaviors or objectives; and, to determine if such models are significantly effective?

Tools for Organizing and Describing

Research Questions may not outright state a study’s characteristics, but they are implied (and are detectable by practiced readers of research). The sample and the data are dealt with and clarified characteristics addressed as one reads reports of research.

Sample (Subjects’) Characteristics – Is there a good match between the sample and the population of interest?

Is a group of Jr. high football players at one school representative of all football players - all football players in one state, in one county, in one city, etc.? In some cases, sampling is not an issue because it is possible to study the entire population of interest; but, most of the time a sample must be taken and generalized back to the population – if the sample was obtained correctly. When researchers are seeking to get a sample, two issues must be addressed:

is the sample’s characteristics similar to the population characteristics; and, when there are groups and/or sub-groups, are all of those groups and sub-groups similar enough to each other in terms of their basic demographics?

When comparing sample and population, it is helpful when the population demographics (e.g., characteristics of a population expressed statistically, such as age, sex, education level, income level, marital status, occupation, religion, birth rate, death rate, average size of a family, average age at marriage, etc.) are known. For some studies group to group comparisons are made and population characteristics are less significant. In cases such as that, demographic and categorical variables are determined for each group as part of the study, and measures are compared by the key demographic categories. If the demographics appear similar from each sample group, then there is a research assumption that they mimic the general population in many respects.

Data Characteristics

Know that data is a plural term, and datum is a singular term. Data would be the best term in almost all cases concerning research results, because there will usually be multiple data sets or numbers involved. Each individual subject or item contributes a single measure (datum). In research, the focus is on collective measures (or data). In quantitative research, most of the time, the focus is on group behaviors or performances. That is why groups are treated as distributions.

In terms of research literacy, and statistical literacy specifically, collected measures from a group form a (normal) distribution. A distribution is a set of measurements of a single variable for a given sample or population. Every distribution has two indices that communicate two vital things: the typical score for the distribution, and the range of variability for all scores. The typical score is a Measure of Central Tendency, and the latter deals with dispersion.

Central Tendency – helps identify those scores that are clustered in some way, and there are three measures of such clustering:

· Mean – the average score in a collection of scores. This is the most useful of the three, statistically. This is the most precise MCT (measure of central tendency), therefore it is the basis for most prediction models and hypothesis testing statistics.

· Median – the exact middle score of all scores, if an odd number of scores, or the average of the two middle scores if there is an even number of scores. Medians are sometimes useful when there are dramatic outliers (the highest and lowest scores are extremely far apart).

· Mode – the most frequently occurring score of all scores.

Dispersion (or Measures of Variability – MOV) – Because it is extremely rare that all scores on a measure will be the same,

there is a need to determine how scores are dispersed from a central cluster or MCT (such as the Mean). There are two forms of MOV or dispersion.

· Range – When a Range is narrow, it is known that the highest and lowest score are close to the typical score and that all scores are very similar (little variation). When a Range is wide, it is known that the highest and lowest scores are far apart from the typical score and there could be a great difference among all scores between the two extremes (a lot of variation). All that information is good, but not great.

· Standard Deviation – As opposed to knowing a high score and a low score, the Standard Deviation takes into account the value of each and every score. From a statistical perspective, scores fall within a specific standard deviation on the normal distribution, surrounding the mean score. Scores can be one standard deviation above or below the group mean, two standard deviations above or below the group mean, etc. Anything outside the first Standard Deviation above or below (68% of the scores, 34% below and 34% above the mean) can be considered different from the mean.

MCTs and MOVs give readers a good picture of the distributions that occur in a research study.

Presenting Results in Research

Results sections of research reports should paint a simple and clear picture of the research findings. Sometimes a study is rather simple and the Results and Discussion sections are combined in the article or report. But, sometimes the study is complex and there are separate sections. Some of the more common quantitative results will now be covered.

Descriptive Results - Most studies will report descriptive findings. These provide a clear and unambiguous picture of the sample’s

distributions on key variables. They can include:

Percentage data of key demographics, such as age, gender, height weight, etc.

Means and Standard Deviations on all independent and dependent variables

Any information that is relevant to determining if the sample is representative of the overall population of interest – this also helps determine whether all variables are normally distributed (along the Normal Curve)

Frequency-based Results - The following things are addressed with these results:

Are data clearly organized, preferably in Tables and Figures that are reader-friendly?

Are data presented unambiguously (leaving no doubts regarding their meaning), so that any discussion of them is easy

to read?

If a Chi-square analysis was performed, was the value for the Chi-square test clearly labeled and easy to find?

If a Chi-square analysis was performed, did the researchers report whether the results were significant, and at what level of probability (e.g., p < .05)?

Correlational Results - The following things are addressed with these results:

Are all variables identified and labeled?

Is it reported whether a direct (positive, they move in the same direction to an extent), or inverse (a negative, or a contrary, relationship between two variables such that they move in opposite directions) relationship exists? If there is no + or – sign, it is presumed that there is a positive correlation.

If there are multiple correlations reported, are they presented in a Table for easy reading and interpretation?

Are the level of probability (p) and degrees of freedom (df) reported and do they seem correct?

t-test Results - The following things are addressed with these results:

Are all variables clearly identified?

For an independent t-test, are the mean and standard deviations reported?

For a dependent t-test, are the mean and standard deviations reported?

For either type of t-test, are the level of probability (p) and degrees of freedom (df) reported and do they seem correct?

ANOVA - The following things are addressed with these results:

Are all variables clearly identified?

Are data presented unambiguously, so that discussion of them is easy to follow?

Is the total number of subjects (N) shown for each group, and the overall N of participants in the study?

Are the means and standard deviations reported for each group?

Are the F ratio with the appropriate within groups and between groups df clearly reported, along with whether it is significant at the selected p level?

Regression Analysis Results - The following things are addressed with these results:

Are all variables identified and labeled?

Is it reported whether a direct (positive, they move in the same direction to an extent), or inverse (a negative, or a contrary, relationship between two variables such that they move in opposite directions) relationship exists? If there is no + or – sign, it is presumed that there is a positive correlation.

Are the correlations, that were used to form the regression equation, presented in a Table for easy reading, interpretation, and identification?

Is the standardized slope, or Beta, reported, and is the t-test of significance of the slope reported?

Is the R2 score reported, along with its F-test score?

Multivariate Analyses (MANOVA) – once considered extremely complex, modern computer applications make it simple. The

concept involves replacing the dependent or independent variable (or both) with a set of related variables. Multiple independent variables are a set of related predictor variables that are used. Multiple dependent variables are a matrix of related dependent variables that replace a single dependent variable. Each MANOVA has its own set of rules and terms, but they are not too different from what has been read so far.

Understanding Discussion and Conclusions in Research

In this section readers should find a summary of what happened in the study, where major findings are reported. Explanations are also given where the author(s) explain(s) how the findings matched their initial expectations or hypotheses; and, current findings are compared/contrasted to key literature cited in the Introduction section. Occasionally, researchers must also explain why some of the results turned out the way they did.

Researchers often reflect, in this section, on the potential impact or implications of their findings- relative to theory and/or practical application. Limitations found or experienced in the current study should also be discussed in this section, and how those limitations influenced the study and how they may affect the ways the results are interpreted.

Finally, recommendations are often made (and should be) regarding how the current study can be built upon in the future, or replicated to see if similar results are found. Expansion of the body of knowledge is how “truth” is found, or what is “known.”

Some considerations when consuming research:

Read the article multiple times! Read once for a general understanding. Read once and compare to the outline below, making mental notes. Read a 3rd time and make written notes. Read a 4th time to ensure that your assignment is written at the level of quality that graduate students should demonstrate.

Be sure to search for articles published in Academic, Scholarly, or Scientific Journals.

Be sure to search for peer-reviewed or refereed journals.

Abstracts that stand alone, Commentaries, Editorials, Literature Reviews (that are not comprehensive Meta-Analyses), and Opinion works are not suitable for these assignments. Consensus and Position Papers are not appropriate as well (that was the 1st assignment for the course).

Be wary of works (having headings similar to the following (for assignments) unless directions state/indicate these are acceptable. Ask Dr. Conkle first if it’s acceptable for his courses if you are uncertain.

Editorial or Commentary or Opinion

Position Paper/Statement or Consensus Paper/Statement

Literature Review or Review Article (Ask about these or read very carefully to determine if it is a Meta-Analysis)

Not all appropriate works will have a “Header” indicating it is what you are seeking, but some will show wording at the top similar to the following:

Original Research

Scientific Article

Scholarly Article

Meta-Analysis Article

Comprehensive Review Article

The Research Report (Article) Critique Outline

A report of research (ideally) will contain a massive amount of information (but found primarily in a Thesis or Dissertation). This permits other researchers to replicate a study using the exact same methodology, or to tweak/improve the methodology, or to study other populations (if not a sample form the same universe as the previous study/studies). With few exceptions, all the following information would be in a Thesis or Dissertation, and in some studies published in Journals. However, due to editorial policies of most journals, some of the info must be omitted to fit within the 10-25 page limitations of many scientific or scholarly journals. Whether analyzing and critiquing a published research report for an assignment or evaluating such sources for a Lit Review, the more of these questions that are answered in the source, the higher quality that source is. If most of the info is missing in a source, it is likely a very poor source. Use the following outline to analyze articles for their quality (or lack thereof).

Being a conscientious consumer of research requires taking seriously your role or status as a graduate or professional school student. You are not only learning about what it takes to conduct research, you are familiarizing yourself with how to critically read and comprehend how to read others’ research results. You should be becoming accountable for developing and using research knowledge and skills. As a graduate student you should demonstrate that you are able to distinguish not only fact from opinion, but also to distinguish what you know based on information from research outcomes compared to what you do NOT know. It is also a matter of being aware concerning what you can and cannot reasonably know after research findings have been read.

Use the following questions for analysis and critical purposes. ELABORATE on and DISCUSS the elements, steer away from yes” and “no” responses, and minimal sentences that provide no explanation. For example, do not simply state that variables exist in a title; discuss and explain WHAT the key variable are. Demonstrate that you are “Consuming the Research.”

Any “Research Critique Assignment” Should be Written in PAST TENSE (e.g., PED 511 and PED 514)

(This is IMRaD Format)

TITLE

Titles are intended to summarize the main idea(s) of a(n) article/manuscript/paper, simply, and if possible, in an engaging manner for readers. The elements of a proper title are shown below:

1] Are major variables mentioned in the title (A variable in research refers to a person, place, thing, or phenomenon that is being

measured in some way. The difference between a dependent and independent variable is that the meaning of each is implied

by what the words tell us about the variable being used.)?

2] Are subjects or sample (generally these are people/participants; but, can be animals, or something inanimate) described/

mentioned in the title – this is who or what the data focus regards?

3] Is any geographic location mentioned in the title?

4] Is any curricular area (school subject), or specific physical activity or sport, mentioned in the title?

5] Do “waste-words” begin the title *(A, An, or The as the initial or FIRST word IS CONSIDERED a waste-word!)?

Words such as Methods, Results, Study, Investigation, Experiment, or Experimental, in the initial few words , are also generally considered waste-words. A, An, or The as words later in a title are often quite acceptable.

6] Is the most important word or phrase at or near the beginning of the title, how is this known or obvious?

7] Is the title 12 words or less? APA (in a former edition) specified < 12 words maximum (e.g., 6th edition), prior editions of APA

specified a recommendation of < 15 words maximum . As of the APA 7th Edition, there is no longer a prescribed limit on

the number of words for title lengths, but simple and concise titles are preferred.

Dr. Conkle still prefers the 15-word or less guideline.

ABSTRACT

100-250 words typically, if it is present it should clearly be between the title and beginning of the article (sometimes shown as a summary of what is to come in the article, other times this feature is omitted). Note that an Abstract may appear in several ways - and by names such as Resume`, Summary, or may have no designation at all but is there - but there will be some indication of the background that led to a given study and the following elements:

1] Does the abstract summarize the research effort?

2] Does the abstract mention the problem (e.g., objective, purpose, aim, hypothesis, research question, problem statement)?

3] Does the abstract mention methodology?

4] Does the abstract mention the results?

5] Does the abstract mention any conclusions?

6] Is the abstract brief, in past tense, and in complete sentences?

7] Is an abstract completely missing?

INTRODUCTION

1] Does the article's context cite previous literature, especially the most recent studies, and early studies showing a basis for that line

of study? An abundance of studies more than 5-10 years old may be considered “dated.” One rule-of-thumb is:

sources that have been published within the past 5 years (35% – 45%), with at least 75% being from the past 15 years.

2] Are citations mostly from research reports, or is there an abundance of opinion and theory papers cited –

Review the References at the end of an article.

3] Is justification given for conducting the current study?

4] Is this a previously unstudied problem, or is it a replication study

(i.e., similar/tweaked methods, or exact same methods as a previous study)?

[(Is there a clear statement of an objective, purpose, aim, hypothesis, research question, problem statement –

they communicate the same type of info)] for #5 and #6

5] Are two or more variables obvious in the problem statement?

6] Are two or more variables in the hypothesis / hypotheses, or research question(s)?

7] Is this a descriptive study, association study, difference study; mixed-methods study; or, a meta-analysis;

is it Quantitative or Qualitative?

8] Are the problem statement and hypothesis or research questions stated so data may be tested and collected objectively?

METHODOLOGY

Typically these are the ordered in this way, but some authors may place them out of this order, or may combine parts of one element in another:

Subjects (or Sample or Participants)

1] How many subjects were involved in the study (N = )?

A] Gender B] Age or Experience or Grade Level C] Ethnicity and/or Race D] SES

E] Degrees Earned G] Geographic Location(s) F] Any other Demographic or Behavioral characteristics ?

2] What is the population of subjects, and / or sample of participants or subjects, for the study?

3] How were the sample, subjects, selected for the study form the population or universe of interest?

4] How were subjects assigned to groups?

5] How were treatments assigned to groups?

6] Is there evidence that the study was approved by an IRB, Ethics Committee, or Human Subjects Review Committee, etc.

to ensure ethical and safe treatment of subjects? Sometimes this is stated elsewhere, including possibly after the

References for the work.

Instrumentation

1] What instruments/techniques/tools were chosen for data collection?

2] Are the instruments commercial, self-constructed, used by permission of another entity?

3] Do the study's measurements seem reliable and valid?

Procedures (can include numerous pieces of information)

1] Step-by-Step procedures are described?

2] Time and materials required are mentioned?

3] Procedures and conditions prior to the study are discussed?

4] If not covered in the “Subjects” section, was there discussion of: How were the population and sample, subjects, were

selected for the study? How subjects were assigned to groups? And, how treatments were assigned to groups?

5] Were there incentives provided to the study’s participants?

6] What were the monetary costs associated with conducting the study?

7] Were biases or perceptions of the researcher(s) reported?

8] In the case of a Qualitative Study, what was/were the role(s) of the researcher(s)?

Data Treatment

1] Which variable(s) is/are constant?

2] What are the variables of treatment and no-treatment?

3] Who was responsible for administering the treatments?

Data Collection

1] Where were the data collected?

2] When were the data collected?

3] Who collected the data?

Data Analysis

1] How are data summarized and organized:

a] Association = are correlation/regression coefficients shown?

b] Difference = are means and standard deviations or variances shown?

c] Is it a Descriptive study

d] Is it a Mixed Methods Study that involves Experimental and Descriptive elements

e] Meta-Analysis = is it a (statistical analysis, evaluation, and integration) study of relevant previously published studies?

* a meta-analysis is not a simple literature review of previous works

* it is structured, analytical, critical and leads to specific conclusions concerning a problem

* it is a definitive methodology that quantifies results and permits statistical techniques for analysis

* it explicitly details how the literature search was conducted for the study

* it states (criteria regarding) how potential sources / studies were excluded / included

* it explains the coding of study characteristics

* it explains the analytical procedures followed

* “Effect Size” is reported, the common metric of very different studies that is expressed in terms of

standard deviation units!

2] How were data analyzed, and were the statistical tests appropriate?

3] Who performed the data analysis?

4] Is validity information clearly stated (sometimes this is early in the Results section, but should be here )?

5] Are reliability estimates clearly stated (sometimes this is early in the Results section, but should be here )?

6] Is / Are one or more reliability coefficient(s) reported?

7] Is the type of reliability given:

a] Internal Consistency?

b] Stability?

c] Equivalence?

d] Inter-rater Reliability

RESULTS

1] Do the results or findings clearly answer the problem, question, or hypothesis?

2] Are statements made concerning rejection or acceptance / retaining of the hypothesis?

3] Are comprehensible tables illustrated?

4] Are comprehensible figures illustrated?

5] Are results summarized well in-text?

DISCUSSION

1] Are interpretations of the findings insightful?

2] Are results of the current study interpreted in terms of the previous literature?

3] Are suitable explanations given for any contradictory results?

4] Do interpretations support the stated justification of the study?

5] Do conclusions state clear answers to the research problem(s) mentioned in the introduction?

6] Are conclusions generalized to the appropriate population(s)?

7] Are research design imperfections explained?

8] Are appropriate recommendations made:

a] Is modification of the research design suggested?

b] Should design components be changed?

c] Are wholesale suggestions made for study replication (a “no-no”)?

d] Are restricted suggestions made for study replication (a “yes-yes”)?