week 4 cled770 db

The Dissertation Journey: A Practical and Comprehensive Guide to Planning, Writing,

and Defending your Dissertation

Selecting and Describing the Methodology

Contributors: By: Carol M. Roberts

Book Title: The Dissertation Journey: A Practical and Comprehensive Guide to Planning, Writing, and

Defending your Dissertation

Chapter Title: "Selecting and Describing the Methodology"

Pub. Date: 2010

Access Date: July 10, 2021

Publishing Company: Corwin Press

City: Thousand Oaks

Print ISBN: 9781412977982

Online ISBN: 9781452219219

DOI: http://dx.doi.org/10.4135/9781452219219.n12

Print pages: 141-165

© 2010 Corwin Press All Rights Reserved.

This PDF has been generated from SAGE Knowledge. Please note that the pagination of the online

version will vary from the pagination of the print book.

Selecting and Describing the Methodology

Selecting and describing the methodology

Selecting the Methodology

Beginning the climb on the dissertation mountain involves choosing a dissertation topic, conducting a review of the literature, and selecting and describing a research methodology. These are not linear processes; they undulate back and forth and often go on simultaneously. Reviewing the literature grounds you in understanding what is known and not known about your study's topic and helps provide the basis for selecting an appropriate methodology. Whatever methodology you choose, you need to understand the techniques and processes of that particular method. Very few students remember well the content from their research methodology or statistics courses.

Selection Considerations

Students frequently ask, “How do I go about selecting a methodology for my study?” The answer isn't simple; it is possible to identify several different methodological approaches for a single topic. Methodology selection rests primarily on the (1) problem to be investigated, (2) purpose of the study, (3) theory base, and (4) nature of the data. Selection may also depend on your research skills and those of your committee members. I recommend that one of your committee members possess the expertise in the methodology you select. How comfortable are you with statistics? Do you have the required literary writing skills necessary for qualitative research? It is essential that you are able to express yourself clearly and explicitly. Don't worry if you do not feel comfortable with your level of knowledge about research methodology or with the skills required to conduct an original research study. In my experience, most students do not come to the dissertation process confident and eager to apply research skills. “Learning by doing” is the name of the game. With guidance from your committee, your learning evolves over time as you proceed through each stage of the dissertation. Don't try to make your study fit a predetermined research methodology.

The research approach you select for your study will be quantitative, qualitative, or a combination of the two. In this section, I present a comparison of these two paradigms but focus on the qualitative approach because of the increased interest and use of this methodology in education and the social sciences. This book, however, is not a methodology text, and I refer you to the Further Reading section at the end of this chapter for detailed information in the various methodological approaches.

The Methodology of Research

All research methodology can be classified under two broad generic categories: quantitative or qualitative. Each has a variety of submethodologies, or designs, with their own protocol for collecting and analyzing data. A hybrid approach is obtained when quantitative and qualitative approaches are used together. Blending these two approaches generally allows greater depth of understanding and insight than what is possible using just one approach. Plus, blending helps overcome the biases inherent in each method.

Quantitative versus Qualitative Research

What is the difference between qualitative and quantitative research? Staindack and Staindack (1988) explained it this way: “Qualitative research differs from quantitative research in its theoretical/philosophical rationale” (p. 4). In philosophical terms, the quantitative approach is called logical positivism. Inquiry begins with a specific plan—a set of detailed questions or hypotheses. Researchers seek facts and causes of human behavior and want to know a lot about a few variables so differences can be identified. They collect data that are primarily numerical and result from surveys, tests, experiments, and so on. Most quantitative approaches manipulate variables and control the research setting. Quantitative designs include descriptive

SAGE © 2010 by Carol M

SAGE Books

Page 2 of 20 SAGE Books - Selecting and Describing the Methodology

research, experimental research, quasi-experimental research, ex post facto/causal comparative research, and correlational research.

Not everything that can be counted counts, and not everything that counts can be counted.

—AlbertEinstein

The qualitative approach is based on the philosophical orientation called phenomenology, which focuses on people's experience from their perspective. Inquiry begins with broad, general questions about the area under investigation. Researchers seek a holistic picture—a comprehensive and complete understanding of the phenomena they are studying. They go into the field to collect data. They may make observations; conduct in-depth, open-ended interviews; or look at written documents. Rather than numbers, the data are words that describe people's knowledge, opinions, perceptions, and feelings as well as detailed descriptions of people's actions, behaviors, activities, and interpersonal interactions. Qualitative research may also focus on organizational processes. In other words, qualitative researchers look at the essential character or nature of something, not the quantity (how much, how many). This approach is sometimes called naturalistic inquiry because the research is conducted in real-world settings; no attempt is made to manipulate the environment. Researchers are interested in the meanings people attach to the activities and events in their world and are open to whatever emerges. Qualitative research is really an umbrella term that refers to several research genres that share certain characteristics. They go by different names, such as case study research, historical research, ethnography, grounded theory, narrative analysis, action research, and hermeneutics.

Why Do Qualitative Research?

Strauss and Corbin (1990) offer five reasons for doing qualitative research:

• 1. The conviction of the researcher based on research experience

• 2. The nature of the research problem

• 3. To uncover and understand what lies behind any phenomenon about which little is yet known

• 4. To gain novel and fresh slants on things about which quite a bit is already known

• 5. To give intricate details of phenomena that are difficult to convey with quantitative methods (p. 19)

The most salient differences between qualitative and quantitative approaches are listed in Figure 12.1.

SAGE © 2010 by Carol M

SAGE Books

Page 3 of 20 SAGE Books - Selecting and Describing the Methodology

Figure 12.1 Research Methodologies: A Comparison

Both research orientations play an important role in extending knowledge. Whichever you select for your study, be sure to read widely in that methodological area so you are knowledgeable about the analysis and data collection procedures necessary to conduct your study. Remember, in the end, you must justify your choice of methodology and clarify why it was the best way to conduct your study. Because there are no inferential statistics to be performed in qualitative research, some students mistakenly believe it to be easier to conduct than a quantitative study. This is not true! Analyzing huge amounts of qualitative data into meaningful themes and patterns is an awesome task requiring considerable time and effort. According to Patton (2002), “On average, a one-hour interview will yield 10 to 15 single-spaced pages of text; 10 two-hour interviews will yield roughly 200 to 300 pages of transcripts” (p. 440).

Mixed Methods

Although qualitative and quantitative approaches are grounded in different paradigms, it is possible to combine them into one study. The mixed-methods approach is expanding as a viable methodology in the social and human sciences, evidenced by a variety of books and journals reporting and promoting mixed- methods research. Creswell (2002) reported that “entire books now exist about procedures for conducting mixed methods studies—similar books were not available a decade ago (Greene & Caracelli, 1997; Newman & Benz, 1998; Reichardt & Rallis, 1994; Tashakkori & Teddlie, 1998)” (p. 208).

Here are some ways to incorporate both qualitative and quantitative approaches into a single study. Numerical data may be included in a qualitative study, and narrative data (such as open-ended questionnaire responses) may be included in a quantitative study. For example, you may start gathering data with a survey to get a broad perspective and then select cases to study in depth by conducting open-ended interviews. You may also start with a case study design and find variables to ask on a survey. Strauss and Corbin (1990) offered other examples of how qualitative and quantitative approaches can be combined: “One might use qualitative data to illustrate or clarify quantitatively derived findings; or, one could quantify demographic findings. Or, use some form of quantitative data to partially validate one's qualitative analysis” (p. 19).

Qualitative and quantitative approaches in a single study complement each other by providing results with greater breadth and depth. Combining what with a possible why adds power and richness to your explanation of the data. With quantitative methods, you can summarize large amounts of data and reach generalizations based on statistical projections. Qualitative research tells a story from the viewpoint of the participants that provides rich descriptive detail. Figure 12.2 is an example of a table from a dissertation by Clark (2002) that combines qualitative and quantitative data.

SAGE © 2010 by Carol M

SAGE Books

Page 4 of 20 SAGE Books - Selecting and Describing the Methodology

Figure 12.2 Example: Summary of Responses from Interviews Describing Collaboration Activities

Gay and Airasian (2003) offered a practical resource for understanding how to mix qualitative and quantitative methods. They offered three models of mixed-methods research:

• 1. The QUAL-Quan model where qualitative data are collected first and are more heavily weighted than quantitative

• 2. The QUAN-Qual model where quantitative data are collected first and are more heavily weighted than qualitative

• 3. The QUAN-QUAL model where qualitative and quantitative are equally weighted and are collected concurrently (pp. 184–185)

Following are some resources that explain in detail various research designs within the qualitative, quantitative, and mixed-methods paradigms. Numerous books are available on research methodology, many highly theoretical—designed for professional researchers. I selected these books for their readability, clarity in explaining research concepts, and usefulness in writing dissertations.

Further Reading

Qualitative Research

Corbin, J., & Strauss, A. (2008). Basics of qualitative research ( 3rd ed. ). Thousand Oaks, CA: Sage. Denzin, N. K., & Lincoln, Y. S. (Eds.). (2005). The Sage handbook of qualitative research ( 3rd ed. ). Thousand Oaks, CA: Sage. Lightfoot, S. L. (1985). The good high school: Portraits of character and culture. New York: Basic Books. (An exemplary model for case study research) Merriam, S. B. (2001). Qualitative research and case study applications in education. San Francisco: Jossey- Bass. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage. Patton, M. Q. (2002). Qualitative research & evaluation methods ( 3rd ed. ). Thousand Oaks, CA: Sage. Rossman, G. B., & Rallis, S. F. (2003). Learning in the field: An introduction to qualitative research ( 2nd ed. ). Thousand Oaks, CA: Sage.

SAGE © 2010 by Carol M

SAGE Books

Page 5 of 20 SAGE Books - Selecting and Describing the Methodology

Journals

• Forum: Qualitative Social Research • International Journal of Qualitative Methods • The Qualitative Report • Qualitative Research Journal

Resources on the Web

Association for Qualitative Research

• http://www.aqr.org.au/

Forum: Qualitative Social Research

• http://qualitative-research.net/fqs

Qual Page: Resources for Qualitative Research

• http://www.qualitativeresearch.uga.edu/QualPage

Internet E-mail Discussion Groups (listservs)

QUAL-L Qualitative Research Mailing List

• http://www.scu.edu.au/schools/gcm/ar/arr/qual.html

QUALRS-L Qualitative Research for the Human Sciences

• http://www.lsoft.com/scripts/wl.exe?sl1=qualrs-l&h=listserv.uga.edu

NOTE: These lists of resources are not exhaustive and may change over time.

Quantitative Research

Babbie, E. (2001). Survey research methods ( 9th ed. ). Belmont, CA: Wadsworth. Cox, J., & Cox, K. (2007). Your opinion, please! How to build the best questionnaires in the field of education ( 2nd ed. ). Thousand Oaks, CA: Corwin. Fowler, F. J. (2002). Survey research methods ( 3rd ed. ). Thousand Oaks, CA: Sage. Gravetter, F. J., & Wallnau, L. B. (2008). Statistics for the behavioral sciences ( 8th ed. ). Belmont, CA: Wadsworth. Marsh, C. (2009). Exploring data: An introduction to data analysis for social scientists ( 2nd ed. ). New York: Polity. Tufte, E. R. (2001). The visual display of quantitative information ( 2nd ed. ). Cheshire, CT: Graphics Press.

SAGE © 2010 by Carol M

SAGE Books

Page 6 of 20 SAGE Books - Selecting and Describing the Methodology

Journals

Journal of Applied Quantitative Methods

• http://jaqm.ro

Journal of Statistics Education

• http://www.amstat.org/PUBLICATIONS/JSE

Resources on the Web

Rice Virtual Lab in Statistics

• http://onlinestatbook.com/rvls.html

Statistics.com

• http://www.statistics.com

Electronic Statistics Textbook

• http://www.statsoft.com/textbook/stathome.html

Mixed-Methods Research

Creswell, J. W. (2009). Research design: Qualitative, quantitative, and mixed methods approaches ( 3rd ed. ). Thousand Oaks, CA: Sage. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage. Gay, L. R., Mills, G., & Peter, A. (2008). Educational research: Competencies for analysis and applications ( 9th ed. ). Upper Saddle River, NJ: Merrill Prentice Hall. Neuman, W. L. (2005). Social research methods: Quantitative and qualitative approaches ( 6th ed. ). Boston: Allyn & Bacon. Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in the social and behavior sciences. Thousand Oaks, CA: Sage.

Journals

Journal of Mixed Methods Research

• http://mmr.sagepub.com

Describing the Methodology

The methodology chapter of a dissertation describes the design and the specific procedures used in conducting your study. It is vital that this section is clear, comprehensive, and sufficiently detailed so that other researchers can adequately judge the results you obtain and can validly replicate the study. In a quantitative study, the methodology chapter usually contains the following sections: introduction, research design, population and sample, sampling procedures, instrumentation, data collection procedures, data

SAGE © 2010 by Carol M

SAGE Books

Page 7 of 20 SAGE Books - Selecting and Describing the Methodology

analysis, and limitations. Qualitative studies typically use different terminology in describing the methodology section. For example, a qualitative study's sections often include the following: rationale and assumptions for the qualitative design, type of design, researcher's role, site selection, data sources, data collection techniques, managing and recording data, data analysis procedures, methods for verification or trustworthiness, and limitations. Following is a description of these sections.

Introduction

You may introduce the methodology chapter several ways, depending on the style and preference of your advisor and committee. Generally, there is an opening paragraph stating the chapter's organization. This is sometimes followed with a restatement of your study's purpose and research questions. A brief description of the problem might also be included.

Research Design

In this section, state the type of research and design used in the study as well as the rationale for your selection. The research design you select is based on the purpose and nature of your study. Some alternative designs are historical, descriptive, developmental, case study, correlational, ex post facto, true experimental, and quasi-experimental. Isaac and Michael (1995) provided an excellent resource to help you understand and describe the research design appropriate for your study. Following is an example from a dissertation of one way to describe a descriptive study's methodology:

Example

Descriptive research methodology was used to identify the current socialization processes for beginning elementary principals in the central coast region of California. This method was selected because it is a means to describe systematically, factually, and accurately the characteristics of an existing phenomenon. (Isaac & Michael, 1981)

In this study the phenomenon includes the assistance and support experiences of beginning principals in learning needed information, determining the expectations for a given role, and understanding and adjusting to the operating norms of the school and district. … (Boullion, 1996)

Note that the researcher did not just give a definition of descriptive research from a noted authority, she also related the definition directly to her study and why it was appropriate for her study.

Population and Sample

The population and sample (or data sources) section includes a description of the individuals who participated in your study and the procedures used to select them. Ideally, an entire population would be used to gather information. However, this is usually not feasible as most groups of interest are either too large or are scattered geographically. When you don't have an opportunity to study a total group, select a sample as representative as possible of the total group in which you are interested. Gay and Airasian (1996) provided a clear definition of the terms sampling and population to help in distinguishing between the two:

Sampling is the process of selecting a number of individuals for a study in such a way that the individuals represent the larger group from which they were selected. The individuals selected comprise a sample and the larger group is referred to as a population. … The population is the group of interest to the researcher, the group to which she or he would like the results of the study to be generalizable. (pp. 111–112)

Remember

• 1.

SAGE © 2010 by Carol M

SAGE Books

Page 8 of 20 SAGE Books - Selecting and Describing the Methodology

The bigger your sample, the more it represents the total population and the less sampling error is present.

• 2. Before deciding how many to select for your sample, you must know the size of your population so that you can reliably draw the appropriate sample size.

• 3. Your sample size represents the number of individuals to be contacted for their participation in your study. It does not represent the number of individuals who must respond.

Sampling Procedures

Your study's credibility relies on the quality of procedures you used to select the sample. These procedures should be described in detail since they determine the generalizability of your findings. Your description should include the following:

• 1. The specific type of sampling used, such as probability sampling (random, systematic, stratified, cluster) or nonprobability sampling (purposive, expert)

• 2. The number of individuals included and where they are located

• 3. Why you selected this particular number and the unit of analysis

• 4. The criteria you used for inclusion in the sample

• 5. A step-by-step account of exactly how you went about selecting your sample

Following is an example from a dissertation of one way to describe sampling and rationale for criteria selection:

Example

The population for this study consisted of forty-six (46) elementary schools in California implementing one of three national school reform projects. … Purposive sampling was used to identify schools within the population that met specific criteria. The criteria for selection included

• 1. Schools in their first or second year of implementation of their selected national school reform project,

• 2. Schools where the current principal was also the principal at the time of initiating the national school reform project, and

• 3. Schools willing to participate in the study.

Rationale for Selection of Criteria

The rationale for selecting the first criterion was twofold. The ability of individuals to accurately recall information regarding the period of time prior to implementation of their reform project would be difficult after more than two years. … The second criterion is related to the role of the principal in initiating any change process. … Because this research study focused on reform projects that change the structure and culture of a school, having the same principal who was also the principal at the time of the initiation phase was necessary to this study. The third criterion suggests that each principal's willingness to participate is critical to this study. Since the design of this research involved detailed questionnaires, participants needed to be willing to take the time necessary to respond. (Chaffee, 1995)

SAGE © 2010 by Carol M

SAGE Books

Page 9 of 20 SAGE Books - Selecting and Describing the Methodology

Instrumentation

This section includes a description of all instruments used to collect data—questionnaires, interview schedules, observation forms, and so on. Each instrument should be described in detail in the methodology section. Provide the following information, as relevant:

• 1. Appropriateness of the instrument for your population and setting

• 2. The validity and reliability of the instruments (Validity is the degree to which your instrument truly measures what it purports to measure. In other words, can you trust that findings from your instrument are true? Reliability is the degree to which your instrument consistently measures something from one time to another. If you measured the same thing again, would you find the same results?)

• 3. How the instrument is administered and scored

• 4. Interrater reliability—a check on the consistency between raters, or between a rater and an expert (This information is necessary when measurement involves subjective interpretation, such as open- ended questions.)

• 5. Type of response categories—rating scales, check lists, ranking, and so on (If you develop a new instrument, you should also recount how it was developed and include a description of the field tests that were conducted and the subsequent revisions. Place in the appendix a copy of all instruments used, unless they are copyrighted.)

Remember

Copyrighted instruments are not reproduced in a dissertation. If you wish to use copyrighted instruments, permission should be obtained in writing from the holder of the copyright.

Developing Your Own Instrument

If you are unable to locate a satisfactory instrument that adequately measures your study's variables or concepts, you may either modify an existing validated instrument or create your own instrument. It is appropriate to change the wording or eliminate questions when modifying an instrument for a different population. However, keep in mind that the changes you make may affect the reliability and validity of the instrument. If you modify an instrument, it is your responsibility to justify the changes made and to provide information about the reliability and validity of the revised instrument.

Helpful Hint

A good idea: When developing items for your instrument, it is critical that you align the items with your research questions to ensure that all research variables are adequately covered in your instrument. A good technique is to create a matrix in which you display your research questions on the left side and the questionnaire items on the right. The following shows an alignment matrix.

SAGE © 2010 by Carol M

SAGE Books

Page 10 of 20 SAGE Books - Selecting and Describing the Methodology

SAGE © 2010 by Carol M

SAGE Books

Page 11 of 20 SAGE Books - Selecting and Describing the Methodology

When describing your instrument(s), it is important to explain your rationale for selection. Following is an example of such a paragraph that appeared in a dissertation:

Example

In an exhaustive review of the literature, the research supporting the use of the SACQ far outweighed the criticism of the assessment tool. The SACQ has been used as an assessment tool in more than one hundred dissertations and theses. Even with its limitations, the majority of the research supported the use of the SACQ in understanding student adjustment to college.

(Schultz, 2008, p. 110)

Following this rationale statement, the student then described his instrument's norms, reliability, and validity.

Field Testing

Any time you create your own instrument or modify an existing one, it must be field tested. You can select from five to 10 people to test the instrument and to make judgments about its validity. The people selected should not be involved in the study but should be like those in the study. Look for the following in pretesting an instrument:

• Understandable instructions • Clear wording • Adequate answers • Sufficient detail • Regional differences • Difficult sections • Irrelevant questions • Length • Convenience

Following is another example of how a dissertation student obtained feedback from field testing an online survey instrument. In his dissertation, he wrote:

The web-based questionnaire was administered to eighteen student employees who, after completing the questionnaire online, were asked to provide feedback on access to and navigation within the survey, technical difficulties experienced, clarity of instructions, typographical or grammatical errors, and general observations. The field test results and test participant feedback supported the validity and clarity of the survey and data collection method (see appendix A)

(Schultz, 2008, p. 116).

Schultz (2008) asked respondents to respond to these questions about their field test experience:

• 1. Overall, how easy was it for you to access the survey and navigate from page to page?

• 2. Please describe any technical problems that you encountered while attempting to access or navigate from page to page.

• 3. Were the directions clear and easy to understand? If not, how can they be made easier for first-time students?

• 4. Were there any typographical errors that you discovered?

• 5. Please share any other comments or suggestions you may have that would help make this survey

SAGE © 2010 by Carol M

SAGE Books

Page 12 of 20 SAGE Books - Selecting and Describing the Methodology

more successful. (pp. 204–207)

Following the field test, it is usually necessary to revise your instrument to reflect the various recommendations from the field test respondents. Be sure to state in this section what revisions were made to your instrument. The examples that follow should help to clarify these directions:

Example 1

A pilot study was conducted in one of the districts selected for the sample. Within that district, two elementary schools that were not included in the sample were chosen for the pilot. The principals were selected because they had been implementing shared decision a minimum of one year and were willing to cooperate in the study. … The principals were interviewed using the Interview Guide. The researcher selected one teacher and one parent from the leadership team for individual interviews. All six sessions were tape recorded with permission of the participants. … Appropriate changes were made in the instrument to clarify the questions and to provide an easier format for recording answers for purposes of limiting the data to that which was most pertinent to the study. Question six was expanded to include more prompts, and clarified to include no more than three of the groups most involved in the decision making process. Question ten was expanded to. … (Walkington, 1991)

Example 2

The modified interview schedule was field tested using five beginning principals from outside the central coast region in California who were first-year principals during the 1992–1993 school year. The field test utilized telephone interviews to simulate the actual data collection process. In addition to responding to the interview schedule, these principals gave feedback on the interview schedule by answering the following questions:

• 1. Is the interview schedule too long? (If yes, what would you suggest be dropped?)

• 2. Are the directions and wording clear and unambiguous? (If not, please note directions or words that are unclear.)

• 3. Is the format conducive to ease of response?

• 4. Do some of the questions need to be rephrased or dropped?

• 5. Are there additional questions that I should ask? (If yes, which questions would you suggest?)

As a result of the field test, no substantive changes were made to the interview schedule. Only minor modifications in wording were needed to make the questions flow more smoothly during the interviews. (Boullion, 1996)

Response Rate

The following questions are often asked by doctoral students conducting a questionnaire study:

• 1. What is an acceptable response rate for questionnaires?

• 2. How can I increase my response rate?

A major disadvantage of questionnaire studies is a low response rate, typically much lower than for personal interviews. Response rates for personal interviews are about 95%, whereas mail survey return rates are usually between 20% and 40%. The rule of thumb regarding an appropriate response rate is as follows:

SAGE © 2010 by Carol M

SAGE Books

Page 13 of 20 SAGE Books - Selecting and Describing the Methodology

Below 50% there is no defense

Below 60% is questionable but could be OK

You should try for 70% or above

There are several strategies you can employ to improve your response rate. Some examples are as follows:

• 1. Send a letter of endorsement by someone with whom the respondent can identify.

• 2. Use a professional-looking form.

• 3. Prepare an introductory letter that sells the respondent on the legitimacy and value of your study, its benefit to him or her, and guarantees of confidentiality and a copy of the results.

• 4. Offer inducements to respond. Appeal to the respondent's goodwill and altruism by stating you need his or her help. Other methods include such things as offering lottery tickets, a raffle prize, a $1.00 bill, a tea bag, and so on enclosed with the questionnaire form.

Probably the most effective method to increase your response rate is to use follow-ups such as a reminder postcard sent five days after the questionnaire. To those who don't respond, approximately two weeks later send a second follow-up that includes another reminder letter and a replacement questionnaire with a return envelope. After approximately two months, you can send by certified mail another letter and replacement questionnaire.

Data Collection Procedures

This section describes in detail all of the steps taken to conduct your study and the order in which they occurred. It is important that your writing is clear and precise so that other researchers can replicate your study. Your description should state how and when the data were collected. The following example from a dissertation describes one way to report data collection procedures:

Example

Data collection began in September of 1995 and was completed by October. On September 9, 1995, surveys (Appendix C) were mailed to the eighteen (18) principals of the schools in the sample population. Each of the 18 principals received a telephone call from the researcher on September 9, 1995, to inform them that the survey was in the mail, and that they would be asked to complete and return it within one week. A cover letter was included describing the purpose of the research (Appendix C). Respondents were assured that neither their personal identity nor the identity of their school would be released in the dissertation. … Principals were asked to complete and return the survey to the researcher within one week. Principals who had not responded within two weeks received a follow-up telephone call from the researcher. … By October 26, 1995, there were a total of sixteen completed surveys, an 89 percent response rate. (Chaffee, 1996)

Remember

Since sampling procedures were described in the “Sample and Population” section and your measures explained in the “Instrumentation” section, you do not need to repeat this information in this section.

Helpful Hint

A good idea: To help you efficiently deal with organizing data collection, create a Source of Data Chart. This chart keeps track of each data source in your study (e.g., who was interviewed, who received questionnaires,

SAGE © 2010 by Carol M

SAGE Books

Page 14 of 20 SAGE Books - Selecting and Describing the Methodology

and what documents were analyzed). It also organizes the data sources by your research questions. See Appendix B for an example of a Source of Data Chart.

Best Time to Collect Data

When to collect data is a critical issue because it can greatly affect your response rate. It is important for you to consider the availability of your population. For example, in education there are several windows of opportunity when people are available. September, Christmas, and June are not the best times given the typical school calendar of events. Your best opportunity to collect data is usually October through November and January through April. These dates can vary if the school system is on a year-round schedule because people are “off track” throughout the school year.

Data collection always takes longer than you realize. A rule of thumb is to set a reasonable timeline, then double it! It takes time to schedule interviews, field test, travel, and follow-up on non-respondents. Refer to Appendix C for an example of a participant letter and questionnaire.

Data Analysis

This section includes an explanation of how you analyzed the data as well as your rationale for selecting a particular analysis method. If your study is quantitative, report the descriptive or inferential statistical tests and procedures you used, how they were treated, and the level of statistical significance that guided your analysis. Since statistical tests may vary by research question, you should explain your tests and procedures for each question. An example follows.

Research questions four through nine focused on the differences in students’ attitudes in looped and conventional classrooms. Composite means and standard deviations were computed for each of the attributes: self-concept, motivation, instructional mastery, and sense of control. The data were analyzed using t-test computations to determine if a significant difference existed between students in looped and conventional classrooms on each of the attitudes assessed. (Johnston, 2000)

If your study is qualitative, provide a description of matrices used to display the data and identify the coding processes used to convert the raw data into themes or categories for analysis. Your description should include specific details about how you managed the large amount of data associated with qualitative analysis. Include information about use of software, sticky notes, index cards, or other processes used. This helps readers understand how you reduced or transformed the data.

Every researcher approaches the coding process differently. There is no one right way to code textual data. One excellent guide to help you understand the coding process is provided by Tesch (1990). He described, in eight steps, a systematic process to analyze textual data:

• 1. Get a sense of the whole. Read all the transcriptions carefully. Perhaps jot down some ideas as they come to mind.

• 2. Pick one document (e.g., one interview)—the most interesting one, the shortest, the one on the top of the pile. Go through it asking yourself, “What is this about?” Do not think about the “substance” of the information but its underlying meaning. Write thoughts in the margin.

• 3. When you have completed this task for several informants, make a list of all topics. Cluster together similar topics. Form these topics into columns that might be arrayed as major topics, unique topics, and leftovers.

• 4. Now take this list and go back to your data. Abbreviate the topics as codes and write the codes next to the appropriate segments of the text. Try this preliminary organizing scheme to see if new categories and codes emerge.

SAGE © 2010 by Carol M

SAGE Books

Page 15 of 20 SAGE Books - Selecting and Describing the Methodology

• 5. Find the most descriptive wording for your topics and turn them into categories. Look for ways of reducing your total list of categories by grouping topics that relate to each other. Perhaps draw lines between your categories to show interrelationships.

• 6. Make a final decision on the abbreviation for each category and alphabetize these codes.

• 7. Assemble the data material belonging to each category in one place and perform a preliminary analysis.

• 8. If necessary, recode your existing data. (pp. 142–145)

The following example illustrates how one dissertation student explained her coding process. She describes a five-step process for analyzing interview transcripts. This process involved going from a holistic perspective to individual parts and back to a holistic look at the data.

Example

Step 1: Initial Reading of Transcripts

After all thirty-one tape-recorded interviews were transcribed, the researcher reviewed all the data twice before developing a preliminary list of categories, themes, and patterns. Several prominent themes emerged from the initial reading. Each theme was given an initial coding.

Step 2: Organization and Coding of Responses

Next, the responses were sorted and grouped by research question. The researcher read through all the responses for each research question, highlighting pertinent information, and developed a master coding list of response categories (see Appendix F). Within each research question, response categories were counted by frequency.

Step 3: Review of Total Transcripts and Final Coding

Using the master coding list developed in Step 2, the researcher coded the full transcript of each participant, noting when second or third references were made in a response category. The coding list was then finalized.

Step 4: Completion of Data Analysis and Report of Findings

The analysis of each response to research questions and analysis of each interview transcript were conducted. This resulted in themes, patterns, and categories for the research questions.

Step 5: Review of Total Transcript to Ascertain Validity of Findings

The researcher reviewed all the transcripts a final time to ascertain that the findings and the main themes and patterns were consistent with the data. A comparison of the literature was made to determine which findings were supported or not supported by the literature. (Boullion, 1996)

A variety of qualitative software products are available for analyzing qualitative data; however, they do take time to learn how to use them well. A book titled Computer Programs for Qualitative Data Analysisby Weitzman and Miles (1995) offers guidance in learning the various software programs.

Validating the Findings

In this section on data analysis, it is important to include how you addressed the issue of validity. Qualitative researchers often use the term trustworthiness to refer to the concept of validity. It's the credibility factor that

SAGE © 2010 by Carol M

SAGE Books

Page 16 of 20 SAGE Books - Selecting and Describing the Methodology

helps the reader trust your data analysis. For example, in qualitative studies, techniques such as triangulation, member checks, and interrater reliability are used to validate findings. Following is an example of how to report the process used to establish interrater reliability:

Example

Interrater reliability is established through a process in which two or more people independently analyze the same qualitative data and then compare the findings. This process of multiple analysis reduces the potential bias of a single researcher collecting and analyzing the data (Patton, 1990). An expert practitioner in the socialization of elementary principals as well as in content analysis of qualitative data analyzed six transcripts from the interview data. The researcher and the expert analyzed the same six transcripts. Both the researcher and the expert used the research questions as a conceptual framework in analyzing the data. The researcher found thirteen common themes, patterns, and categories while the expert found twelve. These findings were the same 92 percent of the time. (Boullion, 1996)

Here is an example of how one researcher reported triangulated data. She validated interview information with archival data.

Example

In addition to the responses from the interviews, the case data from archival records such as board policies, memos, newsletters, and other such documents were reviewed. These records were reviewed to validate or extend the statements made by the superintendents or the stakeholders in each of the districts. (Butt, 1993)

This example explains the process of validating case study findings by obtaining feedback from respondents:

Example

To further validate the findings, drafts of each case report were submitted to the coordinator at each school site. They each reviewed the presentation of data for their site. There was overwhelming concurrence with the researcher's findings. Based on the comments from the coordinators, minor corrections were made regarding the titles of personnel delivering services, the use of funds, and the responsibilities of staff. There was also clarification of a factor that impeded services at one site and an addition to the factors impeding the delivery of services at another site. (Kinley, 1996)

Remember

When explaining how you handled your data, be sure to relate how it was reported—standardized scores, raw data, percentages, mean, median, and so on. Also state how it was displayed—matrices, tables, graphs, charts, figures, or narrative text.

Limitations

Limitations are particular features of your study that you know may negatively affect the results or your ability to generalize. Limitations are usually areas over which you have no control. Some typical limitations are sample size, methodology constraints, length of the study, and response rate.

All studies have some limitations, and it is important that you state them openly and honestly so that people reading your dissertation can determine for themselves the degree to which the limitations seriously affect the study. Following is an example that describes the limitations of sample size and methodology:

SAGE © 2010 by Carol M

SAGE Books

Page 17 of 20 SAGE Books - Selecting and Describing the Methodology

Example 1

The major limitation of the study is the relatively small sample size. Six districts and twelve schools were selected for the study. The source of information used to obtain this sample may not be complete. This may affect the generalizability of the study to other districts.

Another limitation is the use of the observation method. There is a possibility of observer bias anytime data are obtained from observations. (Walkington, 1991)

Example 2

• 1. This descriptive study was limited to participants’ self-reported perceptions of their experiences adjusting to college.

• 2. Individual results are based on a volunteer sample and do not necessarily define the population to which the individuals belong.

• 3. Since there were only six participants from the __ campus, the findings may not be generalized to that campus.

• 4. The SACQ provides a snapshot measurement of participants’ self-reported perceptions. Their emotional state may be at a unique point given the survey was administered during the last three weeks of participants’ first college semester. (Schultz, 2008, p. 119)

The following section offers a checklist of the elements to include in your methodology chapter.

Checklist of Elements to Include in the Methodology Chapter

After you have written your first draft of the methodology chapter, check off the following elements. Mark your draft where each of these elements is located.

Research Design

• _____ Type of research • _____ Rationale for selection • _____ Appropriateness to your study

Population and Sample or Participants

• _____ Description of respondents • _____ How many • _____ Where they are located • _____ Rationale for selection • _____ Size of population or sample • _____ Criteria for inclusion • _____ Specific type of sampling used • _____ Step-by-step sampling procedures

Instrumentation

• _____ Detailed description of all instruments

SAGE © 2010 by Carol M

SAGE Books

Page 18 of 20 SAGE Books - Selecting and Describing the Methodology

• _____ Type of response categories • _____ Appropriateness of instruments to your study • _____ Information on validity • _____ Information on reliability • _____ How instruments are administered and scored • _____ Interrater reliability procedures • _____ Criteria for judging competence • _____ How agreements will be assessed • _____ Percentage of data checked for agreement • _____ Statistics used to calculate agreement • _____ Field test or pilot test • _____ Describes test respondents • _____ How many • _____ Where located • _____ Feedback questions • _____ Reactions of respondents • _____ Procedures followed • _____ Revisions made to instruments • _____ Response rate • _____ Procedures used to increase response rate (e.g., follow-up methods)

Data Collection Procedures

• _____ How data were collected • _____ When data were collected • _____ Where data were collected • _____ Procedures stated in order of occurrence

Data Analysis

• _____ Includes how data were reported and displayed • _____ Provides information on validity and reliability of data (trustworthiness if a qualitative study) • _____ Explains methods used to analyze data • _____ States rationale for use of analysis techniques

Limitations

• _____ Limitations stated

Summary

Selecting a methodology requires understanding the two major research paradigms: qualitative and quantitative approaches. Which one you select depends primarily on the problem investigated, the purpose of your study, and the nature of the data. Qualitative studies generate words that describe people's actions, behaviors, and interactions, whereas quantitative studies generate numbers derived from questionnaires, tests, and experiments. Often both approaches are combined in a single study, resulting in greater breadth and depth.

By describing your methodology clearly and precisely, you will make it possible for other researchers to adequately judge the worth of your findings and replicate your study. You must include detailed descriptions about your research design, population and sample, sampling procedures, instrumentation, data collection procedures, data analysis, and limitations.

SAGE © 2010 by Carol M

SAGE Books

Page 19 of 20 SAGE Books - Selecting and Describing the Methodology

Now that you completed your introductory and methodology chapters, it is time to meet with your committee to discuss and critically analyze your proposed study. The next chapter provides some guidelines for holding the proposal meeting.

• mixed methods • qualitative research • response rate • dissertation • qualitative approaches • interview schedule • transcripts

http://dx.doi.org/10.4135/9781452219219.n12

SAGE © 2010 by Carol M

SAGE Books

Page 20 of 20 SAGE Books - Selecting and Describing the Methodology