answer four discussions topics

10

High-Quality Organizational Learning

Why Do Some After-Action Reviews Work and Others Don't?

A brief report that appeared in the Israeli daily Maariv on 2 July, 1997, raises a question about the quality of learning from after-action reviews:

On 31 June 1996, a foot soldier carrying a wireless set was electrocuted when the set's antennae made contact with an overhead high tension electric cable. This was the sixth accident of this kind in the Israel Defense Forces. A committee appointed to investigate the accident discovered that specific instructions issued based on lessons learned from the previous accidents had not been followed. (p. 4)

The report clearly documents a failure in organizational learning. Five accidents were followed by after-action reviews, yet the same accident occurred again, and with a lethal outcome, on a sixth occasion. The Israel Defense Force is not the only organization that fails, occasionally, to learn from its experience. Why or when do such failures occur, and what can be done to avoid them?

An obvious cause of failures of learning is a low-quality learning process. As we pointed out in  Chapter 2 , the mere existence of organizational learning mechanisms (OLMs) is insufficient to ensure productive learning. In  Chapter 3 , we presented five cultural norms that increase the likelihood of productive learning. In this chapter, we offer a more inclusive list of factors that influence the probability that lessons learned from after-action reviews will be remembered, thereby lowering the likelihood that past errors will be repeated.

The factors were identified in a study that compared two episodes of low-quality and two episodes of high-quality after-action reviews in an elite unit of the Israel Defense Forces. The unit operates a variety of OLMs, the most important of which are after-action reviews. Every combat or training mission is followed by an after-action review, and preparation for operational missions includes a review of relevant lessons learned from similar missions. After-action reviews in the unit vary in quality. Some are considered exemplary by the officers and soldiers who participated in them; others are regarded as a waste of time. The principal criterion on which these judgments are based is the effectiveness of assimilation, namely, the long-term implementation of lessons learned. The episode at the beginning of the chapter is an example of failure according to this criterion.

Using assimilation as a criterion for determining the success of after-action reviews makes sense because long-term implementation of lessons learned prevents repetition of accidents and errors. There are two qualifiers for this criterion. First, the lessons learned have to be valid, that is, effective solutions to the causes of the problem. In addition, they have to be reviewed periodically to test their continued relevance to the organization's mode of operation, as well as to changing environmental conditions.

Aserious limitation of the assimilation criterion is that it requires the passage of time in order to know if a given learning process was successful. To correct this limitation, we decided to study the ways in which the processes that produced effective or ineffective assimilation differed systematically. This way we hoped to complement the outcome criterion of assimilation with criteria that pertain to the after-action reviewprocess.

Owing to its unorthodox and dangerous combat methods, the elite unit under study experiences training accidents. The study concerned four reviews of such accidents. The first step was to ask some 30 officers to identify after-action reviews that represented high- and low-quality reviews and, based on their evaluations, select two high- and two low-quality reviews on which there was consensus regarding their quality. The reviews spanned a period of several years and were sequenced as a low-quality review, two high-quality reviews, and a low-quality review. We interviewed every available officer who participated in each review and analyzed all the documentation that was available on them.

The accidents required more complex after-action reviews than the postflight reviews discussed in  Chapter 8  because of the complexity of the exercises, the number of persons involved, the severity of the outcome, the uncertainties regarding their causes, and the number of required changes. To convey a sense of the two types of reviews, we present an outline of one example of each.

 LOW-QUALITY AFTER-ACTION REVIEW

The review concerned the death of a soldier during a dangerous combat exercise. The review proceeded in four phases: information gathering, drawing of lessons learned, implementation, and assimilation.

Information Gathering

As soon as he was informed that a soldier was missing, the exercise commander convened a debriefing on the ground to assess the situation and begin a search. The unit's commander conducted a more systematic review after the soldier was found dead. Although the information-gathering phase is intended only to ascertain the facts of the accident, the commander drew some key lessons learned during this phase. The corps to which the unit belonged appointed an external investigative committee that four hours later began interviewing officers and soldiers who participated in the exercise. Another external investigation was launched by the military police, as required by the Israel Defense Forces regulations. The phase of information gathering continued for 2 days.

Drawing of Lessons Learned

We could not obtain reliable details on this phase either from the interviewees or from the documents (which we take to indicate its shallowness). The usual format requires the commander of the exercise, the officers in charge, and other participants to provide information on the objectives, planning, and execution of the exercise from their different perspectives and answer the questions of the unit's commander and other officers who conduct the review. At the conclusion of this process, the unit's commander decides on lessons learned regarding training procedures, combat doctrine, and logistics.

Implementation

Three committees were appointed by the unit's commander to translate the lessons leaned into changes in training procedures, combat doctrine, and logistics. The changes were summarized in a document that was presented in a conference attended by the unit's high- and middle-rank officers. This phase continued for 2 months. The overall duration of the review was markedly short for an accident that involved the death of a soldier.

Assimilation

To ensure assimilation, the lessons learned were refreshed in several “safety symposia” in the period following the after-action review. The after-action review of a similar accident some 2 years later (the high-quality process below) revealed that some key relevant lessons learned were never implemented.

 HIGH-QUALITY AFTER-ACTION REVIEW

This episode occurred under the successor of the commander who succeeded the officer who commanded the unit during the low-quality review. The two accidents were similar—a soldier who disappeared during an exercise of the same combat technique was later found dead. The two reviews consisted of the same four phases.

Information Gathering

The deputy commander arrived on the scene as soon as the soldier was declared missing. He debriefed the officers and soldiers to assess the situation and direct the search. Several additional debriefings were held later, both on the grounds and in the commander's office. No conclusions were drawn at this stage. The external investigations by the Military Police and the committee appointed by the corps commander began their work shortly after the accident. This phase continued for 3 days.

Drawing of Lessons Learned

An initial after-action review was held in the unit 4 days after the accident. Its report did not identify the causes of the accident but proposed three hypotheses, one of which was considered the most plausible. In addition, it offered 22 recommendations. Two of these were the same recommendations of the external committee in Episode 1, which had not been assimilated. The commander accepted most of the recommendations (this phase continued for 5 days).

Implementation

Eight days after the accident, the commander appointed five committees to study in depth the conclusions and lessons learned of the initial review relating to different subjects (e.g., training and standard operating procedures). Each committee was headed by the most senior officer knowledgeable on its subject. The committees were instructed to conduct extensive investigations and not to limit themselves to narrow issues relevant to the accident. The five committees were coordinated by a steering committee headed by a senior officer. The committees prioritized their agendas and dealt with subjects relevant to routine operations of the unit first. Proposed lessons learned were tested by the training and operational subunits and the results were fed back to the committees, which changed them if necessary. Detailed instructions for implementation were issued to the relevant subunits and units of the corps. The set of final lessons learned and the review process were documented in detail. This process continued for 3 months.

Assimilation

During the next 2 years, the unit held several symposia to refresh the lessons learned, and their implementation was closely inspected. An investigation 3 years after the accident showed that all the lessons learned were operative.

Why did the two cases produce such different assimilations? An examination of their outlines ( Table 10.1 ) reveals that although they followed the same four phases, the, low- and high-quality reviews differed in their breadth and duration. Finer grained analysis produced seven more specific attributes that could account for their different outcomes:

· The commander's leadership style

· The extent to which the review was systematic

· The review's (low vs. high) place in the unit's agenda

· The productiveness of the internal inquiry

· The effectiveness of the external committee

· The plausibility of the lessons learned

· The effectiveness of the implementation and assimilation processes

Table 10.1 After-Action Review Process in the High- and Low-Quality Cases

These attributes can be used both to guide effective reviews and to evaluate their quality. We now discuss each of them in detail.

Leadership Style

Chapter 6  proposed that leadership is a key influence on the probability that organizational learning will take place and that it will be productive. The comparison between the low- and high-quality reviews showed that this factor is especially important when the conditions for learning are difficult. After-action reviews of fatal accidents take place under trying circumstances. The unit is pressured by higher echelons, the media, and the families of the dead and injured soldiers. There are strong feelings of sorrow, guilt, and fear among those who were directly involved and unit members whose friends have died or are likely to be punished. Everyone worries that results of the after-action review may affect him personally. Two behaviors distinguished between commanders of the high- and low-quality reviews. The first, support, was specific to their circumstances. The other, receptiveness, is relevant to any process of organizational learning.

Support. The commanders of the unit in the high-quality reviews supported their subordinates by being sensitive to their emotional needs and shouldering responsibility for the outcomes of the review. Both were explained by a commander:

My principal concern was to preserve the unit. Fatal accidents can cause units to fall apart, and to preserve them you must take care of their people. The officer in charge of the exercise was a reservist who expected to return home after a short period in service. Instead he faced the prospect of going to jail, possibly for a long period. Such experiences destroy people. At the outset of the review, I told the participants that they should relax because I was responsible for whatever happened in the unit, for better or worse. I tried to impress on them that I trusted them and that they were not alone.

In contrast, the commander during a low-quality review was visibly stressed and left an impression that his only concern was to save his own skin:

In four meetings, he and other officers screamed at four different junior officers…. What disturbed me most was that our commander was willing to take responsibility for success but not for failures. It is impossible to draw [valid] lessons learned when you know that if, god forbid, something happens, you will be left on your own.

Receptivity. The purpose of after-action reviews is to introduce required changes in current practices and procedures. This purpose is not likely to be achieved if the commander is not receptive to the suggestions and opinions of others. The commander may reject or not implement them outright, or signal through his reluctance to listen that making suggestions, to begin with, is a waste of time. Overruling the objections of others, the commander of a low-quality review decided on lessons learned that were subsequently not followed in practice and rescinded by his successor. The same commander rejected suggestions involving new techniques or novel ideas because “We tried it before. It will not work.” As a result, “Instead of going forward with new methods and technologies, the unit went backward. A week after he left, his successor began to adopt the suggestions that were rejected.”

Systematic Design of the Review Process

Other researchers of after-action reviews offered the following guidelines for conducting after-action reviews:

· Invite the right people.

· Appoint a facilitator.

· Revisit the objectives.

· Go through the project (i.e., action) step by step.

· Ask what went well? Find out why these aspects went well and express the learning as advice for the future.

· Ask what could have gone better.

· Ensure that participants leave the meeting with their feelings acknowledged.

· Record the meeting.

These guidelines are appropriate for relatively simple, one-shot reviews, such as the postflight reviews described in Chapter 8 .

The criterion of systematic design is intended for complex reviews such as those discussed in the present chapter. It specifies the assignment of committees to investigate a broad range of issues, manning them with persons that have relevant professional and organizational authority, providing them with sufficient resources, appointing a steering committee to coordinate their work, and monitoring their progress periodically through meetings that critique and integrate their work. High-quality reviews were designed according to this prescription:

Work was divided among several committees. One analyzed the training program. Other committees analyzed operational plans (to find if the training program was compatible with them), the unit's combat doctrine, and so on and so forth. The committees were headed by senior officers and composed of senior and middle-rank officers with the best expertise on their subject matter in the unit. Each committee had a detailed plan with specific goals, subgoals, and a timetable. All the committees met every 2 days to present their findings, discuss them, and decide on how to proceed.

Compare this with the testimony of a participant in a low-quality review who said that “I do not remember a real process. The most serious debriefing was conducted on the grounds about half an hour after the accident, and that was it.”

This quote points to another important principle of systematic design: Separate information gathering from making conclusions and the drawing of lessons leaned. The separation is important for conducting a thorough information search, basing lessons learned on all potentially available information, and preventing commanders from committing themselves to premature conclusions that they will have difficulty reversing. The quote shows that the commander of the low-quality review violated this principle. Lessons leaned were drawn at an early stage of the after-action review, the stage which should have been devoted to gathering information. In contrast, the commander of the high-quality review took deliberate steps to separate the two phases. At the beginning of a preliminary debriefing, he instructed participants that the objective was to gather as much information as possible soon after the accident and that he would make no conclusions based on this debriefing. An added benefit of his clarification was that “it helped people to talk freely without worrying about the consequences of what they were saying.”

The Place of the Review in the Unit's Agenda

In  Chapter 5 , we claimed that when the learning effort is closer or more relevant to the core mission, there is a higher likelihood that it will receive resources and attention, catch on, and succeed. Safeguarding soldiers’ lives is high on any fighting unit's agenda. Nevertheless, the high- and low-quality reviews differed in the extent to which participants remembered them as central or marginal in the unit's ongoing operation. The difference can be attributed to the attention that the different commanders paid to the review process. One high-quality review was actually headed by the commander himself. He was deeply involved in all its activities and saw to it that the reviewand its outcomes were presented to the unit's senior officers, who were therefore familiar with its details. Another indication of a review's place in the unit's agenda was the time that was devoted to it relative to other tasks. A participant in a high-quality review remembered the following:

We thought that the review was of high quality and so felt obliged, professionally and morally, to participate. All of us who participated in the review except for one officer were simultaneously engaged in other missions—the unit was particularly busy at the time with operational missions as well as maintaining its training operations. I was involved in the planning of an operation that took a long time. Although I did not downplay the importance of the review, I finally had to spend more time on that [other] operation.

The officer's efforts to participate in the review in spite of competing operational duties is a clear indication of its high place in the unit's agenda. Significantly, he and other officers felt morally obliged to participate because they perceived the review as a serious investigation into the death of a comrade.

Another indication of the review's place in the unit's agenda is participants’ ability to remember its details. Participants in high-quality reviews remembered them in detail; those who participated in low-quality reviews had a hard time doing that:

In my opinion, the process was a complete failure. I arrived to the unit 2 or 3 weeks after the accident, and they were already busy with the implementation of lessons learned. The accident could have happened a year rather than 2 months before I arrived. The review did not receive any special attention.

The place of the review in the unit's agenda sends a message about the importance of its lessons learned, which clearly affects the probability that these will be remembered and implemented.

The Productiveness of the Internal Inquiry

This criterion pertains to the learning process itself: To what extent do the processes of gathering and analyzing information by the unit's members enable them to draw valid conclusions and lessons learned? According to the multi-facet model, a positive answer to this question depends on two factors: (1) the extent to which the culture in the after-action reviews conforms to the learning norms and (2) the extent to which participants feel psychologically safe during the review process. We did not find evidence for accountability in either low-or high-quality reviews, and participants in both types of reviews exhibited integrity. The latter was most clearly observed in the reports of the external committees in both types of reviews, which included candid admissions of errors by officers who testified to the committees. For example, the officer who supervised one of the exercises admitted that he had not reviewed the safety regulations before the exercise (as required) because “We do this exercise routinely.” Another supervising officer admitted that he neglected to check the area in which the soldier was found dead. Similar to the pilots and navigators in Chapter 8 , members of this unit attributed the norm of integrity that was ingrained in it to the long process of socialization of its members:

Right from basic training in the unit, we are told to report after any exercise on what we did [correctly] or failed to do with an emphasis on the latter…. Reporting truthfully is a norm that is part of the culture of the unit. Compared to other organizations with which I am familiar, people here are honest, and you can obtain from them something that approximates the [objective] truth.

Our analysis did show consistent differences between the high-and low-quality reviews in terms of inquiry, transparency, and psychological safety:

Inquiry. The following quote from a participant in a low-quality review demonstrates lacking inquiry:

Although my roommate OKed the exercise, no one came to talk with him…. The debriefing on the ground did not find anything unusual. The commander decided on some minor changes and that was it. I do not remember any follow-up discussions. During the internal review, we were hardly asked two and a half questions on the accident, and most attention was paid to the injury of a soldier during the search.

Lacking inquiry is shown in the above quote through the narrow and unfocused information search, the fact that no attempt was made to go beyond the obvious (“anything unusual”), and the making of quick decisions without serious analysis. Not surprisingly, the status quo was left essentially unchanged, in spite of the fact that a soldier had been killed. Compare this with the manifestation of inquiry in a participant's description of how interim conclusions served to broaden and deepen the analysis in high-quality reviews:

During an after-action review, you try to map all the factors that may have determined the outcome. You begin from the present and go backward looking at the training course, the operational process, beliefs, behaviors, and then—based on common sense, and debriefs, many debriefings—you focus and come to a conclusion. Now this is your understanding of the accident, but different people may come to different understandings from different perspectives. That is why it is important to take into account the information and conclusions of other committees and the external investigation. At the end, you come to a reasoned conclusion based on your analysis and that of others in relation to training, safety, combat doctrine, and so on.

Transparency. Manifestations of transparency were usually associated with those of integrity, inasmuch as both entailed disclosure of error. Here is how one participant in a high-quality review described how the two values were manifested in the learning process:

The unit is generally characterized by openness [i.e., transparency] and truthfulness [i.e., integrity]. I do not think that anyone tried to hide something or to force something on others. Everyone said his mind. Other people could be angry or disagree with him, but he was heard. I attribute this to the spirit of the unit, which is influenced by the commander as well as the way we debrief right from basic training, particularly the way in which errors are dealt with in them.

The following quote underscores the subtle distinction between integrity and transparency. Integrity is admitting error to yourself or others when it is discovered (as opposed to becoming defensive), whereas transparency is allowing others to observe your actions so that they provide you with valid feedback on them:

There was openness at least to hear what people had to say. There were different committees, and people came to say what they knew and thought. In the final analysis, the intention is not to cover up or falsify what has happened. People truly wish that accidents will not be repeated. Listening to people with different perspectives allows you to come to different conclusions.

Psychological safety. Psychological safety is important for productive learning. In its absence, participants become occupied with defending themselves instead of trying to understand what went wrong and how to improve for the future. Commanders’ sensitivity and support played a decisive role in the presence or absence of psychological safety in the reviews that we studied. One officer described how the commander's leadership style and lack of sensitivity lowered his psychological safety this way:

The atmosphere during the review was uncomfortable. It had an effect on me and must have affected the others. An officer was late, and the commander let him know that in his customary threatening style. A soldier was seriously injured when he fired a flare during the search. As the subject came up in the review, I remember saying that I did not order to fire the flare. I must have been anxious because I would not have mentioned it otherwise. One officer ran to call a medic instead of helping the injured soldier. My impression during the incident was that he was shocked, but in the review he said that he was acting deliberately. In my opinion, he was trying to make his actions look better.

The Effectiveness of the External Committee

The rationale for appointing external committees is clear: Enlist relevant expertise from outside the unit and correct potential biases in the work of internal reviews with a review by persons who can study it objectively. To be effective, external committees should possess expertise and objectivity and be perceived to posses them. The credibility of the external committee in a low-quality review was damaged irreparably because it included an officer who was on leave from the unit but scheduled to return as its commander. The credibility of another was destroyed because the accepted opinion in the unit was that its members lacked the required expertise to understand the exercise. Neither committee managed to affect the lessons learned that were decided on by the respective commanders. In contrast, participants in one high-quality review were impressed, not just with the objectivity and expertise of the external committee but by the time and effort that they put into their work. Ninety percent of the recommendations of this committee were accepted by the unit's commander.

The Plausibility of Lessons Learned

There are two methods to ensure that lessons learned will be implemented: Apply external force and generate internal commitment. Applying external force relies on monitoring the implementation processes and punishing those who are caught disobeying them. Generating internal commitment depends on convincing the unit's members that the review has unearthed the causes of the accident and that the lessons learned are plausible, that they provide effective and applicable remedies to these causes. The plausibility of lessons learned was often judged by how they were drawn. Thus, a participant in a low-quality review reported that its lessons learned were perceived as “panicky” because the commander decided on them at an early stage without waiting for the conclusion of the full review. In contrast, a participant in a high-quality review thought that “the review produced the correct lessons learned because it considered all the aspects of the accident and elicited a wide range of opinions from different persons.”

Judgments of plausibility were sometimes determined by the compatibility of the lesson learned with the unit's culture. A key lesson learned in the low-quality accident was hotly contested as impractical. The underlying issue was a tension between safety and risk taking that was ignored in the review. The informal norm in the unit was that model fighters take risks to accomplish their missions. Ignoring this norm, the change in procedure decided on in the review limited soldiers’ freedom of movement to increase their safety. The change was contested as “impractical” and was never implemented in spite of the risks involved. A high-quality review recognized the conflict between formal procedures that stress safety and the unit's culture and introduced new safety regulations in conjunction with a campaign that educated soldiers on the risks involved and that promoted the idea that model fighters avoid unnecessary risks.

The Effectiveness of the Implementation and Assimilation Processes

Implementation is often mentioned as the bottom-line criterion for high-quality organizational learning. Because of our interest in the prevention of repeated errors in the long term, we extend this criterion to include assimilation. The effectiveness of implementation and assimilation is partly dependent, of course, on satisfying the previous six criteria of the review process. As we'll show, these form a causal chain that extends from the commander's leadership style to successful assimilation. In addition, the low- and high-quality reviews differed in terms of specific actions that were taken in the latter and neglected in the former to ensure that lessons learned were implemented and assimilated. These included (a) detailed documentation of the lessons learned; (b) dissemination of the lessons learned to all relevant units with explanations of their underlying rationale, changes in mode of operation that they entailed, and risks that were involved in disobeying them; (c) close supervision and monitoring of implementation of the lessons learned; and (d) regular “refresher” conferences, in which the above are repeatedly illustrated with “war stories” on the accidents (such as the story of the original accident) that the lessons learned were intended to prevent.

As we have just mentioned, these six criteria form a causal chain with the commander's leadership style to determine the review's success or failure. The commander affected all the factors that differentiated between the low- and high-quality reviews, except for the external committee. He determined the design of the review process, and his leadership style influenced the productiveness of the learning process. Proper design had two outcomes: First, systematic data collection and analysis increased the probability that the causes of the accident would be identified correctly and that the correct lessons learned would be drawn. Second, involving as many unit members in the review as possible by assigning them to various subcommittees and testing the plausibility of preliminary lessons learned improved understanding of the rationale of new procedures and regulations and commitment to their implementation. The commander also determined the place of the review in the unit's agenda, which, in turn, determined the amount of time that they could devote to the process and sent a message to the unit at large regarding the importance of the review and the implementation of its lessons learned.

The receptivity of the commander and the latitude that he left to the committees to pursue their investigations were key factors in promoting the value of inquiry. The commander's support shielded members from fear and promoted integrity, transparency, and issue orientation.

The seven factors identified in this chapter are not limited to determining the quality of after-action reviews in military units. The extent to which the review is designed and carried out systematically, the leadership style, receptiveness and support of higher echelons, the productiveness of the learning process, the expertise and objectivity of external resources, the plausibility of the lessons learned, and the effectiveness of implementation will affect the success of assimilating lessons learned in every type of organization. For mangers who wish to maximize assimilation, the chapter has a general lesson beyond ensuring that the seven criteria are achieved: To succeed, they must see that the lessons learned win both the hearts and minds of the organization members. It is not sufficient that lessons learned be valid in some objective sense or in the opinions of experts. Unless the organization members judge them to be valid and important, they will not make the necessary effort to change their current patterns of behavior. That is why it is important to involve as many members as possible in the process, to let those who are directly involved participate in a meaningful way, and to persuade the remaining members of the organization of the importance of lessons learned through extensive communication and the symbolic value of serious support and resources.

_________________

AUTHOR’S NOTE: This chapter was written in cooperation with Neta HorinNaot and is based on Norin-Haot.