Dossier assignment
C© Risk Management and Insurance Review, 2011, Vol. 14, No. 2, 299-309 DOI: 10.1111/j.1540-6296.2011.01200.x
EDUCATIONAL INSIGHTS
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING IN RISK MANAGEMENT EDUCATION John Garvey Patrick Buckley
ABSTRACT This article draws a link between the risk management failures in the financial services industry and the educational philosophy and teaching constraints at business schools. An innovative application of prediction market technology within business education is proposed as a method that can be used to encourage students to think about risk in an open and flexible way. This article explains how prediction markets also provide students with the necessary experience to critically evaluate and stress-test quantitative risk modeling techniques later in their academic and professional careers.
INTRODUCTION The financial and economic crisis that we continue to endure presents a serious challenge to the teaching and learning strategies employed in universities. Business graduates are expected to have a deep knowledge of the theory that forms the bedrock of the financial system as well as the mathematical competence necessary to apply asset pricing and risk management methodologies. However, the techniques and models used to control and manage risk are often taught in an environment that does not provide sufficient space and time for rigorous debate and critical analysis.
Students are often presented with subject knowledge in a way that the content has al- ready been carefully selected and sequenced by their lecturer. The education literature already notes that this method of providing teaching materials prevents an active learn- ing dynamic (Kinchin, Chadha, and Kokotailo, 2008). In the early stages of university business programs, the often large class sizes limit the opportunity for students to engage in realistic decision-making scenarios. The project described in this article is founded on providing students with an early testing ground for the application of risk management theory. The creation of a closed market populated by other class members is a departure from the traditional approach where students learn about the use of statistical mea- sures of risk such as standard deviation and correlation and become familiar with their
John Garvey is a Lecturer in Risk Management and Insurance, Kemmy Business School, Uni- versity of Limerick; e-mail: john.garvey@ul.ie. Patrick Buckley is at the Kemmy Business School, University of Limerick; e-mail: Patrick.buckley@ul.ie.
299
300 RISK MANAGEMENT AND INSURANCE REVIEW
practical relevance to industry standards such as beta or value-at-risk through lectures and formulaic practice. The application by students of statistical methods in a real-time insurance market demonstrates the relevance of human behavior and expectations in driving market dynamics.
Beyond the confines of the university campus we can observe increasing pressure on the insurance system to underwrite risks previously considered uninsurable. The insurance system is absorbing potential claims associated with catastrophic risks posed by natural hazards such as earthquakes and windstorms and in some cases man-made hazards associated with technologies as nuclear, biological, and chemical engineering. This trend is occurring at a time when the industry is beset by narrower profits as large volumes of capital compete for a limited range of risks. There is now a large category of insured risks that are being priced and underwritten using techniques that do not apply the age-old mathematical comforts of the law of large numbers and the central limit theorem.
This article describes an innovative teaching mechanism that has been applied to a large group of undergraduate students at the Kemmy Business School, University of Limerick. We document how the teaching and learning environment has been dramati- cally changed through the introduction of a prediction market where students estimate and transfer insurance risks. The market structure encourages students to think about risk outside the confines of the lecture theatre. The competitive nature of the mar- ket and the sparse historical information that is made available require students to explore the strengths and limitations of traditional risk management techniques. Impor- tantly, the students’ participation in this dynamic and complex environment coincides with their introduction to formal ways of thinking about risk management. Because of this, the market activity provides a reference point during lectures so that students engage in dialogue and listen in an open and flexible way. The dynamic nature of the market and its direct and timely link with the course content encourages students to learn at a “deep” level. It provides them with skills that they can bring to bear in the learning process outside of the specifics of this module.
In this article, we document the prediction market structure as it is used in an under- graduate risk management module taken by 430 undergraduate students. The module is an introduction to a specialty stream in risk management and insurance. Graduates in this specialty go on to work in roles as varied as risk analysis, insurance and rein- surance underwriting, and fund management. These roles primarily require an ability to accurately identify and assess risks using historical data in a variety of quantitative risk models. In practice, risk decision making is also influenced by the existing risk profile of the organization, the requirements of regulators, as well as pressures relating to performance. The many technical skills required in risk decision making must often be applied with subjective elements of judgment. The prediction market allows students to observe the reflexive nature of their decisions in a dynamic environment.
The article is structured as follows. The Introduction section introduces the motivation for the current study. “Risk, Insurability and Education” provides a context for the use of prediction markets in risk management education by focusing on the challenges faced by the insurance industry and the changing nature of insurability. “The Insurance Loss Market” discusses the importance of class interaction and critical thinking in the context of education and risk management. This section also describes the design of the Insurance Loss Market. “Results on Risk Decision Making and Learning” describes the
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING 301
results of the research and examines the effectiveness of prediction markets in engaging students and augmenting learning outcomes. “Conclusions” discusses the future of risk management education and the development of innovative techniques that inform risk decision making.
RISK, INSURABILITY, AND EDUCATION Within the education environment and business schools in particular, the constraints of time and demands from employers for practical and technical knowledge leaves little space for the exploration of how decisions are made in the absence of known ex ante probability distributions. Third-level education in risk management focuses on how practitioners undertake decisions when faced with ex ante probability distributions that are known. Graduates who specialize in risk management and finance learn a great deal about the quantitative and technical aspects of risk decision making. Popular, quanti- tative models, such as value-at-risk, are a generally incorporated into taught modules at both undergraduate and postgraduate levels. In this teaching environment, Frank Knight’s important distinction between risk and uncertainty is rarely linked directly to industry practice and is likely to be relegated to an historical artifact (Knight, 1921).
The assumption that we can accurately estimate ex ante probability distributions is the foundation for many of the risk models used by the insurance and banking industries and interpreted by regulators. For academics, both as researchers and as teachers there is a recognition that effective business education should provide students with the op- portunity to actively apply and evaluate decision making in an environment that closely approximates real-world decisions. In this article, we show that this can be achieved by providing student’s with this opportunity early in an undergraduate business program before their perspective on risk is influenced by traditional thinking and contemporary risk models. As we can observe from the ongoing financial crisis, of the set of risks that are priced and managed within the financial system an increasing proportion extend beyond the limiting parameters required for models such as value-at-risk. If they are to become effective risk management professionals, it is important that graduates become aware of the Knightian uncertainty of the real world, rather than imposing a strict mathe- matical framework on their decisions. The management of uncertainty can be achieved much more effectively through conservatism and avoidance and simple diversification methods where possible.
There is a growing awareness that traditional teaching methods in risk management and finance are somewhat narrow. This awareness has grown most acutely over the past 2 years as we have seen the near collapse of the banking system and the failure of a number of institutions. However, the failure in risk management is the most recent and devastating in a lineage that can be traced back through Enron, LTCM, and Barings Bank. These risk management failures have prompted a variety of responses from corporations and regulators. Within education, business graduates now have a greater awareness of the limitations of quantitative risk models and there has been a general trend toward including new subject areas such as governance and ethics for those engaged in finance and risk management. Although this trend is laudable in some respects, a criticism of this approach might be that graduates compartmentalize the different subject areas, and are unlikely to later draw on issues relating to governance and ethics when they are engaging in risk management.
302 RISK MANAGEMENT AND INSURANCE REVIEW
Within business education a number of techniques have been developed that allow stu- dents the opportunity to apply their knowledge of relevant theory in a realistic setting. In risk management education the breadth of case study and market applications is proof of the need to sharpen traditional teaching techniques so that university students fully appreciate the challenges of risk management. Projects using computer simula- tion have been described by Hoyt, Powell, and Sommer (2007), Born and Martin (2006), and Joaquin (2007). Hoyt et al. introduce commercially available software produced by Riskmetrics to examine value-at-risk. Similarly, Born and Martin simply adopted the software provided by AIR Worldwide to allow students to apply the software used in catastrophe modeling. Joaquin describes the application of spreadsheet-based sim- ulation in loss modeling. While these approaches are effective in allowing students to practice and refine their skills, they are essentially static in nature and as with many risk models there are significant model assumptions made a priori. The project described in this article is also very different from the insurance market simulation used by Russell (2000). Rather than simulate an insurance market, we use actual, real-time insurance data and prediction market software. The activity of market participants (in this case, the students) creates the pricing dynamic by evaluating likely insurance losses.
Other approaches in creating an insurance market type environment generally take the form of a case-study-type project that requires students to recommend specific business decisions. The application of classroom games is described by Barth et al. (2004) and Eckles and Halek (2007). The effect of risk framing on choices under uncertainty is explored in the games structured by Barth et al., while the impact of asymmetric information is a specific objective in the classroom games structured by Eckles and Halek. The dynamic environment created by an interactive prediction market provides a forum to undertake decisions and compete against peers that is distinct from these earlier projects. By using a prediction market and obtaining data on an underlying “asset,” in this case state-wide insurance industry losses, we are not imposing decision parameters on students. Instead, students evaluate and reevaluate their decision-making criteria and learn to appreciate the emotional and psychological inputs into risk decision making in a realistic setting.
THE INSURANCE LOSS MARKET We describe here a prediction market structure as it is applied in an undergraduate business program at the University of Limerick. Prediction markets are also known as collective intelligence networks, and the software required for their operation is available from a number of commercial providers. Prediction market platforms allow multiple users to make forecasts about the probability of future events as diverse as movie box office sales and election results. By forecasting a specific outcome, individual market participants marginally influence the expected probability of that outcome. With large numbers of market participants accurate and reliable estimates of event probabilities are likely to emerge. The dynamic nature of the prediction market allows these probabilities to fluctuate in real time as participants act and react to the arrival of new information.
The prediction market described here used software provided by QMarkets, one of a number of commercial providers. The increasing popularity of prediction markets and the greater breadth of applications have encouraged the creation of open source software that allows users to download and create their own prediction markets. Thus, this type of project could be easily replicated in other educational settings.
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING 303
We describe here the application of a prediction market that is designed specifically for an undergraduate module, called Principles of Risk Management. This module intro- duces students to the qualitative and quantitative skills required in risk assessment, risk control, and risk financing. The module is delivered in a traditional format through a series of lectures and tutorials that were offered over a 12- 1 week semester. The learning outcomes identified are reinforced through student participation in a custom-designed prediction market called the Insurance Loss Market (ILM). This market allows the 430 undergraduate students registered for the module to forecast weekly losses in the in- surance industry. Specifically, students are required to predict weekly insured property losses estimates for California, New York, and Florida.1 The details of the forecasting and trading process are detailed in the next section. The market dynamic allows students to activate their skills in mathematical competency and qualitative risk assessment in real time. During each 5-day period, each student was required to undertake at least one trade in each of the three states. The ILM was open for trading 24 hours a day and it was run over a 10-week period. At the market close on each Friday, their forecasts were evaluated against the gross property loss estimate as notified by data provider, Xactware. The simplicity of the ILM interface and data provided by Xactware concealed a sophisticated process that allowed for the provision of highly accurate data at the end of each week.2
Market Operation At the beginning of every week, Monday 9 a.m., each student is provided with 5,000 units in notional “risk” capital that they must allocate to loss bands in each of the three U.S. states. Figure 1 provides a screenshot of the ILM interface. Historical data on insurance losses for the three states are made available to the students at the beginning of the semester, and the first 2 weeks of the semester are used to allow students to famil- iarize themselves with the operation of the market. During this period students learn quickly about the variability in weekly insurance losses. Given the element of “luck” in making an accurate prediction students were required to use a number of aspects of risk management so that their capital allocation strategy performed consistently from week to week. As discussed in the following section, the students who performed consistently
1 The data providers, Xactware, included 5 years of loss data for each of the three states. These were made available to students at the beginning of the semester and they were encouraged to consult this data bank when undertaking decisions. Although there was a degree of “luck” attached to forecasting losses, the exercise demonstrated to students how to apply historical data could be useful, but had to be used with care. In addition, the element of accuracy required of the students was reduced by requiring them to forecast loss bands rather than point estimates.
2 The process flow used by Xactware to generate the data can be described as a “full-cycle claims workflow.” Each week, Xactware typically receives a first notice of loss from an insurer that includes the type of loss, the physical address of the loss location, along with varying amounts of supporting information dealing with coverage types/amounts, and a description of the circumstances surrounding the loss. This information is then forwarded to either a claims adjuster, repair contractor, independent adjuster or someone else who is responsible for completing an estimate of repairs. That recipient connects to the Xactware network, using a local installation of their estimating application (Xactimate), and proceeds to complete a unit cost repair estimate of the damages. Once completed, the recipient uploads the final estimate to the network (XactAnalysis) where Xactware mine the various data elements contained in that detailed repair estimate.
304 RISK MANAGEMENT AND INSURANCE REVIEW
FIGURE 1 ILM Screenshot
Note: The New York market is shown. Trading activity by market participants implies that there is a 10.6 percent probability that losses in New York will be >$9m and < = $10m for the week ending October 9, 2009.
well are those who recognize that the “luck” element can be reduced through allocating capital across a number of loss bands in each state.
As trading activity commences the market dynamic will produce an expected distribu- tion of likely outcomes as participants evaluate historical information, such as recent weather patterns, insurance hazards and loss statistics as well as forward-looking infor- mation such as hurricane development, weather forecasts, and potential hazards such as wildfires posed by prolonged period of data. There is wide availability of new informa- tion on weather-related hazards such as fires, windstorms, and hail as well as other rele- vant information. Market participants must evaluate the importance of the available his- torical information as well as the relevance of new information when making a decision.
As participants select a specific loss band, its value increases and simultaneously the value of all other loss bands will decrease proportionately. In order to increase trad- ing and improve liquidity, most prediction markets use an automated market maker. When a buyer or a seller posts an order, the automated market maker automatically fills the order and adjusts the price of the asset using a mathematical formula. In this case, it is not necessary to match buyers and sellers. By allowing transactions to occur immediately it reduces the complexity of the market interface, which has the effect of lowering knowledge barriers and promoting participation (Christiansen, 2007). Detailed descriptions of the operation of automated market markers are given by Hanson (2007) who describes the market scoring rule and Pennock (2004) who describes the dynamic parimutuel market maker.
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING 305
In a similar manner to assets traded on a liquid market, the value of units in a specific loss band may make it prohibitive, thus forcing students to make alternative selections or wait for the unit value of a loss band to fall. Many aspects of market activity are similar to that carried out in the insurance markets each day as insurance and reinsurance underwriters allocate, trade, and transfer insurance risks.
Importantly, participants in the ILM are predicting events in “real time.” This overcomes many of the weaknesses of alternative risk-decision methodologies used in education and industry, such as simulations using an historical event or historical asset behavior. The type and level of activity in the market is at the discretion of each participant and the decisions they make in this regard are seen as key part of the learning process. All decisions are taken on an individual basis; however, consultation with classmates is encouraged. In order to retain participation throughout the semester, ILM participants must undertake one trade in each state each week. There is no upper limit on the number of trades they can undertake and they can continue to trade as often as they like (“buying” or “selling” risks) throughout the week until the ILM closes on Friday at 17:00. There are no transactions costs imposed on student portfolios. Later that day or early the following week the actual loss estimates for that trading period are received from Xactware. The closing position of each participant is reconciled against the actual loss data and is used to estimate the value of each student’s portfolio, as shown in Equation (1).
PortfolioA= Cash Balance + (UnitsCA × 100) + (UnitsFL × 100) + (UnitsNY × 100). (1)
The portfolio value for Participant A is calculated as the number of units they hold in the correct loss band for each U.S. state multiplied by 100 (100 percent) plus the cash they did not allocate. The metric for evaluating activity and decision making in the ILM places primary importance on the forecasting accuracy.
RESULTS ON RISK DECISION MAKING AND LEARNING The primary objective of this research is to create a challenging learning environment for risk management students. This environment should encourage a more critical perspec- tive on risk decision making and the popular quantitative techniques that are applied in practice. One of the interesting aspects of using the prediction market was the immediate change in mindset that it produced among the students taking the module in Principles of Risk Management. As mentioned, the ILM was live for a 10-week period during the fall semester 2009. This was preceded by 1 week in which students were encouraged to access the ILM for a trial period of 1 week. The simplicity of the questions and the nature of the underlying risks being evaluated facilitated immediate participation by a large proportion of the class. During the initial weeks of the semester very few instructions were provided to participants.
The minimal level of guidance provided during this initial phase was deliberate and it had the desired effect of creating discomfort among participants as they attempted to evaluate the possible range of gross property losses in New York, Florida, and Cal- ifornia during that week. This “hands-off” approach allowed participants to evaluate the decisions they were making in an unbounded atmosphere, with little consideration for the norms recommended by risk management theory and practice. This approach
306 RISK MANAGEMENT AND INSURANCE REVIEW
gave rise to informal queries from students during the trial week, such as: “What is the right approach?,” “When should I decide on the appropriate loss band?” as well as other comments that included, “Isn’t this just gambling” and “It is hard to get enough data to make a decision.” Decision making (trading) in the market is motivated both by fluctuating values in a specific loss bands as it increased or decreased in popularity and also through relevant external risk information provided by sources such as the National Hurricane Center.
Assessment for the module was designed to promote a high level of participation in the ILM structure.3 The level of activity in the ILM is also revealed in Figure 3, which summarizes the average number of trades undertaken in California, Florida, and New York. We can observe that in the initial week, there were 13.12 trades undertaken by students in the California market, 12.74 in the Florida market, and 10.11 in the New York market. In the 10-week period, the average number of trades undertaken showed a marginal decrease. In the final week of the market the average number of trades for California was 7.61, and for Florida and New York trades undertaken averaged 6.29 and 9.22, respectively. It is worth noting that, throughout the entire 10 weeks, participation in the market exceeded the minimum participation limits that were set as part of the module requirement.
Following the first week of live trading in the ILM, participants were provided with historical data that gave gross property losses for each of the three states for the 5-year period 2004 to 2008.4 The provision of this information coincided with the beginning of a series of lectures on risk assessment and risk measurement. These lectures intro- duced students to fundamental concepts such as randomness and variability around an expected value as well as the useful characteristics of normality.
Students were encouraged to examine the historical loss data and explore how it could be used in their ILM decisions. An experienced risk management professional would immediately recognize that the historical data would provide only very crude predictive information. For those participating in the ILM, the recognition that historical data must be used carefully was learned though the interactive experience of evaluating and undertaking and reversing decisions.
As the weeks progressed and students became more familiar with the dynamic of the ILM we reduced the width of the loss bands.5 From the fifth week of live trading on the ILM loss bands were held constant. This allowed us to evaluate progress in participants’ ability to undertake decisions and control their risk exposure. A comparison
3 Twenty-five percent of the total marks in Principles of Risk Management were assigned to ILM part of the module. Marks were assigned on a weekly basis with a total of 8 marks available for participation (minimum of 1 transaction in each insurance region), 9 marks for performance relative to peers (Maxiumum of 9 marks (top 20 percent finish, relative to peers) and declining by 1 mark for 10 percent bands), and a maximum of 8 marks available for a one-page report on students’ decision-making behavior in the ILM.
4 Data provided by Xactware for quarterly (3-month) periods. 5 Changes to loss bands were initiated in California in Week 4 where bands were reduced from
$5m (e.g., losses will be > = $10 million and < $15 million) to $1m (e.g., losses will be > = $10 million and < $11 million). Narrower bands were applied to all states by Week 5 and remained narrow for the remaining 5 weeks of live trading.
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING 307
FIGURE 2 Weekly Trading by Region on the Insurance Loss Market
FIGURE 3 Weekly Data on the Number of Positions Held by Market Participants
Note: The number of participants categorized with low-level diversification fell, while participants holding three or more positions increased across the 10-week period.
of the distinct trends in trading behavior between Figures 2 and 3 demonstrates a strong learning dynamic among the student population. Figure 3 shows the number of positions (loss bands) held by market participants each week. We can see quite clearly that there is a strong trend among participants to decrease exposure to a specific loss band. This trend coincides with drop in the number of trades undertaken in each week, observed in Figure 2. This shows that market participants are recognizing the uncertainty of the environment, and although they may use historical data as a guide, they are managing their exposure by selecting a wider range of loss bands. In this context, the fall in the number of trades undertaken by participants appears to be a recognition that the difficulty in profiting by actively trading insurance exposures based on sparse information that is available to all participants.
308 RISK MANAGEMENT AND INSURANCE REVIEW
FIGURE 4 Average Number of Positions Held per Week
Note: Participants are ranked and grouped by performance.
Given the sparse historical data available, the ILM environment is one of Knightian uncertainty and it forces participants to evaluate and manage risk without recourse to robust statistical measures. In the early weeks of ILM activity, participants relied heavily on the most recent weeks’ loss experience. Activity centered on one or two loss bands while those loss bands that appeared distant from recent experience remained untraded. Participants were undertaking highly risky behavior where a minor weather event could easily counter their market position. The increasing use of diversification as a mechanism for managing risk is one of the key outcomes from the market. Furthermore, when market participants are grouped according to performance, we can see that those who performed strongest over the 10-week period demonstrated the greatest engagement in overall diversification.
Weekly performance was based on the value of each participant’s portfolio when the markets were resolved at 17:00 GMT each Friday as summarized in Equation (1). Figure 4 illustrates the trading behaviors of participants ranked by their overall performance. Those who performed strongest, the top 20th percentile, engaged in a markedly higher level of diversification. This provides robust evidence of the validity of the ILM as a teaching methodology in risk assessment and risk management.
CONCLUSIONS This article describes the creation of a market in insurance losses and its application in risk management education. The unique application of real-time insurance losses and prediction market technology allowed students to explore the practical considerations in managing and trading insurance exposures. Incorporating this teaching instrument into university education has clearly had a positive impact in engaging students in the subject area and teaching them about the dynamics underlying the insurance system. More broadly, the use of prediction market technology in risk management education is shown here to improve critical thinking and provide an important starting point for introducing students to more sophisticated risk modeling and risk management tech- niques. The availability of historical insurance loss data through commercial providers
USING TECHNOLOGY TO ENCOURAGE CRITICAL THINKING AND OPTIMAL DECISION MAKING 309
such as Xactware as well as the wide number of prediction market software means that the project described here can be applied in other universities. Furthermore, this approach to augmenting the teaching of risk management can by operated as a joint venture among universities, thus allowing a larger number of participants to forecast, trade, and discuss insurance risks in an educational setting.
REFERENCES Barth, M., J. Hatem, and B. Yang, 2004, A Pedagogical Note on Risk Framing, Risk
Management and Insurance Review, 7(2): 151-165. Born, P., and W. Martin, 2006. Catastrophe Modeling in the Classroom, Risk Management
and Insurance Review, 9(2): 219-229. Christiansen, J. D., 2007, Prediction Markets: Practical Experiments in Small Markets
and Behaviours Observed, Journal of Prediction Markets, 1(1): 17-41. Eckles, D., and M. Halek, 2007. The Problem of Asymmetric Information: A Simulation
of How Insurance Markets Can Be Inefficient, Risk Management and Insurance Review, 10(1): 93-105.
Hanson, R., 2007, Logarithmic Market Scoring Rules for Modular Combinatorial Infor- mation Aggregation, Journal of Prediction Markets, 1(1): 3-15.
Hoyt, R., L. Powell, and D. Sommer, 2007, Computing Value at Risk: A Simulation Assignment to Illustrate the Value of Enterprise Risk Management, Risk Management and Insurance Review, 10(2): 299-307.
Joaquin, D., 2007, Loss Modeling Using Spreadsheet-Based Simulation, Risk Management and Insurance Review, 10(2): 283-297.
Kinchin, I. M., D. Chadha, and P. Kokotailo, 2008, Using PowerPoint as a Lens to Focus on Linearity in Teaching, Journal of Further and Higher Education, 32(4): 333-346.
Knight, F. H., 1921, Risk, Uncertainty, & Profit (New York: Harper & Row). Pennock, D. M., 2004, A Dynamic Pari-Mutuel Market for Hedging, Wagering, and In-
formation Aggregation, in: Proceedings of the 5th ACM Conference on Electronic Commerce (New York: ACM), pp. 170-179. doi:10.1145/988772.988799.
Russell, D., 2000, Two Classroom Simulations in Financial Risk Management and Insur- ance, Risk Management and Insurance Review, 3(1): 115-124.
Copyright of Risk Management & Insurance Review is the property of Wiley-Blackwell and its content may not
be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written
permission. However, users may print, download, or email articles for individual use.