Short Sections

Validity procedures help assure the quality of the research, especially taking into account potential bias in the way the data/research findings are generated, interpreted, and reported. Incorporating validity procedures as part of the research design and implementation helps generate quality data. They bring awareness to the interactions among the participants/collaborators, and increase the credibility of the findings/learning. Validity procedures are a required component of the Leadership Project Proposal. It may be helpful to think of validity procedures also as “quality procedures.”

From the Leadership Project Guide (April, 2011; December, 2013, p. 8)

Validity procedures take into account potential bias in the way the data/research findings are generated, interpreted, and reported. Since any observation is seen through a particular lens, good research design usually means conducting the inquiry in a way that incorporates multiple perspectives, so that no one vantage point is relied on to describe the reality of what is being investigated. Action research practitioners have developed validity procedures to guard against the bias of the inquirers, especially unconscious bias and self-deception.

There are three main areas where applying validity procedures improve the quality of the inquiry: data gathering process, interactions between participants, reporting the findings.

Examples of procedures used while gathering data include multiple cycles of action and reflection, multiple data sources (triangulation), and diversity of participants to ensure multiple perspectives. These are generally considered defining elements for action research projects. Multiple data sources might include interviews, observations, journal notes, and surveys (Kuhne & Quigley, 1997).

Examples of procedures that support interactions between participants include managing distress, authentic collaboration, balancing action with reflection, balance between chaos and order. Skills that support participants in engaging in the inquiry include being present and open, bracketing and reframing, and guarding against defensive routines and group think by challenging consensus collusion through the role of “devil’s advocate.” These validity procedures and skills raise the awareness of the inquirers during the inquiry to free them from “the distortion of uncritical subjectivity” (Heron & Reason, 2001, p. 150) to create what Heron and Reason (2001) call “critical subjectivity” (p. 149). In using the term “critical subjectivity” they posit that pure “objectivity” is not possible and the need to be openly aware and critical about the inevitably subjective and biased observations and interpretations of researchers—be they outside experts in a traditional research design or participants in a Collaborative Inquiry. Since no researcher(s) are free from subjectivity and bias, the best that can be expected is to be aware of these, try to correct for them when possible, and understand how they influence the research outcomes. Validity procedures help bring uncritical biases to open awareness, and thus facilitate a “critical subjectivity” among the researchers (Reason & Heron, 1999).

At the end of the project there are validity procedures to guard against the biases of the authors who report the findings is to ask participants to review and critique a draft of the project report. Assess the validity of the project in your paper: to what extent were validity procedures implemented? How well did the project incorporate the “choice-points” proposed by Bradbury and Reason (2006) in the conclusion to the Handbook for Action Research? End of excerpt from the Leadership Project Guide.

In summary, to create research findings that are as “valid” or “true” as possible, the inquirers must guard against biased observation in data gathering and take into account potential biases in interpretation. Action research practitioners have developed a variety of validity procedures to guard against the bias of the inquirers, especially unconscious bias and self-deception.

Don’s List of the Validity Procedures complied from the Module VI Readings

The 1st three elements are so basic to action research design that they are not usually labeled as separate procedures:

1. Promoting multiple perspectives from a diversity of co-participants,

2. Emphasizing shared leadership and democratic participatory input,

3. Engaging in iterative cycles of action and reflection (the four phases of inquiry multiple times) that allows for refining the questions via

(a) “convergence”—several times probing the same issue each time in more detail or

(b) ”divergence”—different issues each new cycle (Bray et al.109).

Additional procedures in the action research literature help minimize biases and improve quality:

[The following items without citations are taken from two similar sources: Reason & Heron, 1999 which does not have handy page numbers, and from pp. 149-150 The Handbook of Action Research (2006) chapter 12 (copyright 2001) by Heron & Reason, “The Practice of Co-operative Inquiry: Research ‘with’ rather than ‘on’ People.”]

4. Balancing action with reflection.

5. Developing “critical attention” and “affectionate curiosity”

6. Encouraging “authentic collaboration” by the commitment to the egalitarian process of action research with shared leadership. (Bray et al., p. 110; Heron & Reason, p. 150).

7. Creating safe conditions for collaboration and learning – such as non-violent communication protocols; being present and open; attending to emotionally charged or distressful subject or process issues as they arise. Periodic check-ins, time-outs, and de-briefings (Bray et al., p. 111)

8. Distress management/facilitation to avoid buried emotions and unblock group process, mediate clashes of ideas (Bray et al. p, 112)

9. Welcoming “chaos” and balancing chaos and order

10. Encouraging the role of “Devil’s Advocate” to challenge “group think.”(Bray et al. p. 108).

11. Bracketing and reframing to look beyond initial assumptions.

12. Gathering data from multiple methods including f-2-f discussions, journaling, interviews, surveys, consulting publications, using art and stories in presentational knowing, verification from “outside experts”, field notes, recordings etc. Multiple methods of data gathering. AKA “triangulation” (Kuhne & Quigley pp. 31-33)

13. Multiple ways of knowing: drawing on personal experiences and knowing through the Extended Epistemology which includes experiential, presentational, propositional, and practical multiple ways of knowing. Especially pay attention to incorporating presentational knowing expressed variously through movement, sound, drawing, story, drama. (Heron & Reason, 2006, p. 149; Reason & Heron, 1999.)

14. Incorporating perspectives of informed outsiders, to assure that your own observations and generalizations as convener and final report writer are grounded in and representative of the reality of what happened. (Bray et al., p. 108-109)

15. Validation through observed change; change at the level of the participants involved and of the group/organization (Bray p. 110 “catalytic authenticity” that produces a change in the participants and/or their context (Bray et al. p. 110 citing Reason, 1992).

16. This “validation through observed change” is similar to what Bradbury and Reason suggest as evidence of significant quality research by looking at the consequences of the research. They posit questions that can be raised during the implementation of the project while there is the opportunity to move it toward greater validity. Bradbury, H. & Reason, P. (2006). Conclusion: Broadening the bandwidth of validity: Issues and choice-points for improving the quality of action research. In P. Reason & H. Bradbury (Eds.), Handbook of action research. Thousand Oaks, CA: Sage Publications.

Tracking significance by asking:

a. To what degree are those involved empowered and energized by the action research.

To what extent is there a growing critical consciousness so “they may ideally say ‘that was our research and it helped us to see ourselves and our context anew and to act in all sorts of new ways’.” (p. 344)

b. ‘Do people whose reputations and livelihoods are affected act differently as a result of the inquiry?’ (p. 344)

c. Is the inquiry useful? The work/results valuable and worthwhile? Does it lead people to “say that ‘work is inspiring; that work helps make me live a better life’?” (p. 345)

d. Does the action research increase enduring consequences and development of sustainability in the absence of the initiating researcher?

e. What, if anything, has changed or transformed? At the individual (first-person research), action research team/group (second person research practice) or wider (third-person) context?

These last two important validity procedures below are not in the Module VI or XI assigned reading materials, and are useful to consider when writing the project paper:

17. How do the politics and power dynamics of the research context and setting affect the research and what precautions are taken to reduce the negative effects of these dynamics? (Conducting the research within an organization that may have different priorities than the convener; note that all the Projects are conducted as an academic requirement within an academic calendar with both enabling and limiting consequences. (Recommended reading posted in XI Moodle: Kasl, E. & Yorks, L. “ ‘Whose Inquiry Is This Anyway?’: Money, Power, Reports and Collaborative Inquiry” in Adult Education Quarterly XX(X) 1-24, 21010.)

18. Adding validity by having research participants read over the draft of your final report to verify that it represents what they experienced. [suggested in Donald Stone’s “Knowing How We Know About the New Religions” in Jacob Needleman and George Baker, Understanding the New Religions, pp. 141-152, Seabury Press, NY, 1978.