summary
This article was downloaded by: [144.37.1.110] On: 24 November 2014, At: 15:52 Publisher: Institute for Operations Research and the Management Sciences (INFORMS) INFORMS is located in Maryland, USA
Interfaces
Publication details, including instructions for authors and subscription information: http://pubsonline.informs.org
Kroger Uses Simulation-Optimization to Improve Pharmacy Inventory Management Xinhui Zhang, Doug Meiser, Yan Liu, Brett Bonner, Lebin Lin
To cite this article: Xinhui Zhang, Doug Meiser, Yan Liu, Brett Bonner, Lebin Lin (2014) Kroger Uses Simulation-Optimization to Improve Pharmacy Inventory Management. Interfaces 44(1):70-84. http://dx.doi.org/10.1287/inte.2013.0724
Full terms and conditions of use: http://pubsonline.informs.org/page/terms-and-conditions
This article may be used only for the purposes of research, teaching, and/or private study. Commercial use or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher approval, unless otherwise noted. For more information, contact [email protected].
The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or support of claims made of that product, publication, or service.
Copyright © 2014, INFORMS
Please scroll down for article—it is on subsequent pages
INFORMS is the largest professional society in the world for professionals in the fields of operations research, management science, and analytics. For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org
Vol. 44, No. 1, January–February 2014, pp. 70–84 ISSN 0092-2102 (print) � ISSN 1526-551X (online) http://dx.doi.org/10.1287/inte.2013.0724
© 2014 INFORMS
THE FRANZ EDELMAN AWARD Achievement in Operations Research
Kroger Uses Simulation-Optimization to Improve Pharmacy Inventory Management
Xinhui Zhang Kroger Operations Research Group, Cincinnati, Ohio 45242; and Department of Biomedical, Industrial and Human
Factors Engineering, Wright State University, Dayton, Ohio 45435, [email protected]
Doug Meiser Kroger Operations Research Group, Cincinnati, Ohio 45242, [email protected]
Yan Liu Kroger Operations Research Group, Cincinnati, Ohio 45242; and Department of Biomedical, Industrial and Human
Factors Engineering, Wright State University, Dayton, Ohio 45435, [email protected]
Brett Bonner Kroger Operations Research Group, Cincinnati, Ohio 45242, [email protected]
Lebin Lin Kroger Operations Research Group, Cincinnati, Ohio 45242; and Department of Biomedical, Industrial and Human
Factors Engineering, Wright State University, Dayton, Ohio 45435, [email protected]
The Kroger Co. is the largest grocery retailer in the United States. It operates 2,422 supermarkets and 1,950 in-store pharmacies. Improving customer service is at the heart of Kroger’s business strategy. Toward this end, Kroger’s operations research team, in collaboration with faculty from Wright State University, developed an innovative simulation-optimization system for pharmacy inventory management. In pharmacy applications, traditional standard statistical distributions fall short of providing accurate pharmacy demand distributions. To overcome business resistance to complex formulas, this simulation-optimization approach uses empirical distributions to model demand, provides end users with a visual intuitive experience, and delivers optimal or near-optimal results in milliseconds through local search heuristics. The system was implemented in October 2011 in all Kroger pharmacies in the United States, and has reduced out-of-stocks by 1.6 million per year, ensuring greater patient access to medications. It has resulted in an increase in revenue of $80 million per year, a reduction in inventory of more than $120 million, and a reduction in labor cost equivalent to $10 million per year.
Keywords: simulation-optimization; inventory management; local search.
The Kroger Co. was founded by Barney Krogerin 1883. In the past 130 years, it has grown to become the largest grocery retailer in the United States and the fifth-largest retailer in the world. Kroger employs more than 339,000 associates who serve cus- tomers in 2,422 supermarkets and multidepartment stores spanning 31 states under two dozen local banner names, including Kroger, King Soopers, City Market, Dillons, Owen’s, Jay C, Food 4 Less, Pay Less,
Baker’s, Gerbes, Scott’s Food and Pharmacy, Fred Meyer, Fry’s, QFC, Ralphs, and Smith’s. The company also operates 790 convenience stores, 344 fine jewelry stores, 1,141 supermarket fuel centers, and 37 food- processing plants in the United States. Recognized by Forbes in 2010 as the most generous company in America (Forbes 2010), Kroger supports hunger relief, breast cancer awareness, military personnel and their families, and more than 30,000 schools and grassroots
70
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 71
organizations. The Kroger Co. was ranked 23rd on the Fortune 500 list with fiscal year 2012 sales of $97 billion.
Kroger operates 1,950 in-store pharmacies nation- wide as part of its convenient one-stop shopping strategy. In the fiscal year 2012, its pharmacies filled more than 160 million prescriptions with a total retail value of approximately $8 billion. Kroger distributes its drugs to these stores from two sources; it pro- vides the majority of the drugs through three of its own warehouses and the remainder through expe- dited orders using third-party warehouses. Depend- ing on the host store’s demand and its distance from the warehouse, a pharmacy store receives its drug orders with other nongrocery products on one of the fixed schedules of two to three deliveries per week (e.g., Monday, Wednesday, and Friday or Tuesday and Friday).
To match purchase orders against delivery sched- ules, Kroger pharmacy employs a periodic review reorder point (s) and order-up-to-level (S) policy to manage its inventory. Typically, one or two days before a scheduled delivery, called the review period, the pharmacy checks its inventory positions before plac- ing orders. If a drug’s inventory level is at or below the reorder point s, the pharmacy will create an order to raise the inventory level to an order-up-to level S. If the inventory position is above the reorder point s, the pharmacy will take no action until at least the next review period. It rounds up each order quantity to be a multiple of a prespecified bottle size, such as 500 units per bottle (i.e., a 50 mg capsule) for amoxicillin, a commonly prescribed antibiotic. When a pharmacy receives a prescription from a patient, a pharmacist dispenses the drug if it is in standard dosage or com- pound ingredient to form specific vials of medication, and advises the patient on the dosage, interactions with other medications, and side effects to ensure that the drug is used safely and effectively.
To provide customers with the correct medicines in a timely manner, pharmacists are constantly chal- lenged by the large selection of drugs and the highly irregular, intermittent, sporadic demands, which are specific to each store and rarely match standard sta- tistical distributions. At any given time, a doctor may write a prescription for any one of about 24,000 drugs; however, most pharmacies carry only a small fraction
of that number, typically between 2,000 and 3,000. Because of geographical location, prevailing disease, and population composition, each of the 1,950 phar- macy stores serves a unique customer demographic and each drug in a store has a demand distribution distinct from that of the same drug at any other store. Managing the large number of drugs at these stores created an enormous inventory problem for Kroger pharmacy. To overstock and unproductively tie up pharmacy assets in inventory—subject to expiration and obsolescence as a result of the introduction of new drugs—is not good practice. However, exces- sively low inventory leads to out-of-stock (OOS) pre- scription drugs, which decreases customer loyalty and deprives patients of access to medications they need.
Kroger’s pharmacies, despite being a profitable and rapidly growing segment of the company’s business, had for years addressed the conundrum of inventory management by relying on heuristic rules and man- agement instinct to manually set inventory policies, such as reorder points and order-up-to levels, of each drug at each store. This manual procedure demanded substantial time from pharmacists who did not have sufficient knowledge, skill, or time to determine proper inventory levels, and often led to unneces- sary overstock and OOS situations. Improving cus- tomer service is at the heart of Kroger’s customer-first business strategy. Toward this end, in March 2010, the pharmacy division asked its operations research (OR) group to investigate scientific inventory man- agement methods to improve its customer service, decrease inventory investment, and decrease its phar- macy inventory management time. After analyzing the existing rules-based inventory management sys- tem, the Kroger OR group, in collaboration with OR faculty from Wright State University, was confident that a scientific approach would yield tremendous benefits.
Challenges Although the potential benefits of a scientific approach were clear, traditional methods do not always apply because of specific pharmacy demand characteristics. The Kroger OR team faced several challenges in its quest to apply traditional analytic methods to solve the pharmacy inventory problem.
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 72 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
The first challenge was the business resistance to complicated inventory formulas. Many of the tradi- tional inventory models are based on complex math- ematical and statistical formulas. The complexity of the development of these formulas, and the formu- las themselves, can be daunting and often result in confusion and resistance from managers who want to deploy solutions that their teams can easily learn, accept, and use (Silver et al. 1998, Tiwari and Gavirneni 2007).
The second challenge centered on the adequacy of standard distributions used in the traditional inventory models to accurately describe pharmacy demand distributions. In the formulation of analyti- cally tractable models and the derivation of results, assumptions must be made and approximations used; for example, normal or Poisson distributions are often used to model lead-time demand. These assumptions, however, do not always hold at Kroger. The demand of drugs in a pharmacy exhibits multiple streams for specific sizes; for example, some prescriptions are written for 30 days, others are written for 90 days. Because pharmacy prescriptions are sporadic, this results in multiple peaks in the demand distribu- tion. As a result, many of the demand distributions are multimodal and do not fit the unimodal statis- tical distributions seen in traditional inventory mod- els. Figure 1 illustrates historical demand (top panel) and empirical distributions (bottom panel) over the replenishment lead time plus an order period of such a drug. The order period, typically two or three days, is the time between placing an order and the time at which the next order can be placed for a given drug at a given pharmacy.
In this simple example, each instance of demand falls mostly into a few discrete values—the basic dose for this drug is one pill per day; thus, a 30-day supply is 30 pills, a 60-day supply is 60 pills, and a 90-day supply is 90 pills. The demand (top panel) reflects the occurrence of multiple independent streams of 30-, 60-, and 90-day demand from customers; however, an occurrence of a 90-unit demand could be either one 90-day supply for one customer, or a combina- tion of 30- and 60-day supplies for two customers, or three independent 30-day supplies for three cus- tomers. In the pharmacy business, the demand for
the vast majority of drugs is highly variable, intermit- tent, and irregular. If the volume of 90- and 30-day demands exceeds the volume of the 60-day demands, as is the case for this drug, two peaks in the demand distribution result, as the bottom panel of Figure 1 illustrates. Thus, an accurate model requires a multi- modal distribution.
Although not all empirical demand distributions are multimodal, we have observed that many pre- scription demands cannot be modeled using standard statistical distributions. In addition, the pharmacy inventory problem has a complicated cost structure that must account for inventory holding, ordering, and partial or complete OOS costs. For example, being able to partially fill a prescription, such as providing a week of supply, is better than being completely out of stock.
These complex demand distributions and cost structures make it very difficult to apply traditional inventory theory; as an alternative, we have devel- oped a simulation-optimization solution approach to solve the pharmacy inventory problem (Law and Kelton 2000). Simulation-optimization approaches can resolve concerns over black-box complex formulas, capture empirical distributions to model the various shapes of demand distributions, and provide results that are visually appealing, easy to understand, and match pharmacist intuition. However, simulation- optimization for inventory systems, such as (s1S) sys- tems, presents computational challenges when one considers the need to find solutions for more than 2,000 drugs at each of the 1,950 stores. Because of the huge financial value (i.e., hundreds of millions of dollars of pharmacy inventory investment is at stake), our objective was to develop an efficient algo- rithm to find near-optimal solutions to the millions of instances that must be solved weekly—a goal that the pharmacy division determined to make it responsive to seasonal changes.
Kroger’s Simulation-Optimization Approach for Inventory Management To resolve this computational complexity challenge, we adopt a sample path approach to transform a stochastic inventory optimization problem into a deterministic optimization problem, which we solve
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 73
50 100
0
50
100
150
Days
D e m
a n d in
u n its
0 150
0 50 100 150
H is
to ri ca
l f re
q u e n cy
0 .0
0 0
0 .0
0 5
0 .0
1 0
0 .0
1 5
0 .0
2 0
0 .0
2 5
F itt
e d d
is tr
ib u tio
n
Quality demanded during order replenishment time
0 2
0 4
0 6
0 8
0
Figure 1: Historical data show multiple streams of independent 30-, 60-, and 90-day demand in the top panel. These streams imply multiple speaks in the underlying demand distribution of the lead time plus an order period, shown in the bottom panel, suggesting that a unimodal distribution is not appropriate for modeling the lead-time demand.
effectively with a local search algorithm. In design- ing the simulation-optimization approach, we believe, as Albert Einstein did, that everything should be as simple as possible, yet not simpler. To ensure that Kroger’s business users understand our solution, the Kroger OR team decided to use a simple and elegant approach and started with a simulation model in a spreadsheet—the platform widely used by both prac- titioners and academics.
Spreadsheet Models for Inventory Management The spreadsheet simulation model is a retrospective simulation, which is designed to mimic the phar- macy periodic inventory system. It takes as input the historical demand patterns of, for example, the previ- ous year, simulates the ordering process of each drug for a given (s1S) policy, evaluates the performance of the policy, and uses an Excel solver to find the opti- mum solution from among the alternatives.
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 74 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
Figure 2: The spreadsheet model shows the performance of various inventory policies and employs Excel solver to find the optimization solution or inventory policy.
Figure 2 illustrates the spreadsheet simulation model, which consists of four sections: drug informa- tion and inventory policy or decision variable, process simulation, graphic section, and results.
Drug information and decision variable section (D2:F3): Cell B2 holds the drug to be optimized, which is identified by its national drug code, cell B3 holds the number of prescriptions of the drug, cell B5 the package size of orders, cell D3 the value of the reorder point (s), and cell F3 the value of the order-up-to level (S). Cells D3 and F3 are the decision variables and quantities of s and S, respectively.
Process simulation section (A6:L32): Column A is the query key used to aggregate the demand of pre- scriptions for the drug on any particular day (listed in Column B). Column C shows the aggregate demand. Column D represents the days of a week, which deter- mine whether this is a review period, as column E shows (1 represents a review period, and 0 other- wise). In the example, the review periods are Monday, Wednesday, and Friday. Column F identifies the cur- rent inventory on hand, and column G the inventory positions. Column H represents the decision to order (coded as 1) or not. If the inventory position is less
than or equal to s, and the day is a review period, then an order, rounded to a multiple of the package size as defined in cell B5, is issued to reach S. This order quantity is shown in column I. The dates and quantities of order arrivals, shown in column J, are calculated based on the drug’s lead time. Columns K and L show the ending inventory and the number of the drug’s out-of-stocks on each day, respectively.
Graphic section (N7:R32)2 This section shows the drug’s average inventory at the end of each day and its aggregate demand each day. The plot of the-end- of-the-day inventory gives a clear picture of the per- formance of the inventory policy defined by (s1S). The plots also provide key insights for the develop- ment of the local search algorithm.
Result section (J2:N3): The total number of orders, the average inventory, and the total number of OOS prescriptions are calculated in cells J3, K3, and L3, respectively. These measures are translated into an objective value (see cell N3) based on management- specified parameters for inventory holding cost, fixed cost of placing an order, and OOS costs.
The spreadsheet model can be easily connected to Kroger’s enterprise information system to retrieve a
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 75
0 1
0 0
2 0
0 3
0 0
Q u a n tit
y d is
p e n se
d
0 50 100 150
0 1
0 0
2 0
0 3
0 0
4 0
0 5
0 0
Days
E n d in
g in
ve n to
ry
Figure 3: Historic demand and results of optimal inventory performance (final inventory) give pharmacists a visual tool to evaluate specific inventory policies.
pharmacy’s transactions. By changing the national drug code number (cell B2), the demand for that drug is automatically retrieved through the concatenation of drug, year, month, and day as the query key to generate the fixed demand. By fixing this demand, the spreadsheet simulation model mimics the inven- tory process for the given (s1S) policy, and translates the policy’s performance into a single value defined as the total ordering, holding, and stockout costs. In doing so, we have translated a stochastic inven- tory problem into a deterministic optimization prob- lem in which various techniques can be used to find the optimal solution. Based on the simulation model described previously, an Excel Solver can be easily used by setting cell N3 as the objective, and cells D3 and F3 as the decision variables, to find the optimal or near-optimal inventory policy of the resulting deter- ministic optimization problem.
The spreadsheet simulation-optimization model is easy to construct—the first author designed it in two weeks—and the Excel solver is able to find near- optimal inventory policies. The model is easy to un- derstand, does not require any specific knowledge of statistical distributions, reflects the operational pro- cess, and matches management’s intuition. It also allows pharmacists to experiment with different inventory polices derived from, for example, heuristic rules, compare their results, and observe the perfor- mance of near-optimal inventory policies. For exam- ple, Figure 3 illustrates the Excel output (transformed for presentation purposes) of the demand (top panel) of a drug and the final inventory (bottom panel) of the optimal inventory policy.
By doing so, the spreadsheet model provides a visual and interactive tool for optimizing the inventory or- dering problem. It helped us address management’s
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 76 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
concerns and overcome its resistance, and enhanced management and pharmacist confidence in our solu- tion. The consistently enthusiastic reception it received at the pharmacies paved the way for the later nation- wide implementation of our solutions. As Bell et al. (1999) point out for visual interactive simulation- optimization approaches, “decision makers have an even more positive view of the model than do model builders” (p. 163), and the spreadsheet model has become a strong marketing tool for OR solutions. Here, we quote a corporate pharmacy financial man- ager, Dennis Bird, on his experience with the simu- lation-optimization model (from our video presen- tation, available online as supplemental material at http://dx.doi.org/10.1287/inte.2013.0724):
To change the behavior of how our associates order products in almost 2,000 pharmacies, a solution needs to be simple, easy to explain why, and work better than the old way 000 0 We started to get excited with the opportunity the day the operations research team showed us the simulation model that we could visu- ally see the performance of different inventory policies. We could play with the model, which gave us hands- on experience within seconds. By creating the simu- lation first, the operations research team gained my confidence and that of the broader pharmacy team 000 0 The system eliminates the dichotomy that has puzzled the pharmacy team for years between out-of-stocks and too much inventory.
However, the spreadsheet model is cumbersome and computationally inefficient; thus, it is unsuitable for national implementation. In light of this limita- tion, we implemented the simulation model in the R statistics language (Revolution Analytics 2013), a well-known open-source package used for statisti- cal computing and analysis, and we developed cus- tomized algorithms to efficiently solve the inventory simulation-optimization problem.
Local Search Algorithm to Optimize the Inventory System Based on the underlying demand distributions, com- putational studies to tackle the inventory simulation- optimization problem can be classified into two cat- egories: those based on continuous distributions and those based on discrete distributions. For the former, see Kleijnen and Wan (2007) for studies on response surface, perturbation analysis, and the scatter search- based OptQuest method; for the latter, see Zheng and Federgruen (1991) and Fu and Healy (1997) for
studies on gradient-based and retrospective search algorithms.
Our simulation-optimization approach follows a sample-path problem, as Fu and Healy (1997) propose. In essence, from empirical distributions, sampling is used to generate a demand of T periods and sim- ulation is used to provide the functional evaluation of the long-run average cost over various inventory policies. By fixing the demand, this approach trans- forms a stochastic inventory problem to a determinis- tic optimization problem in which various techniques can be used to find the optimal solution. Neverthe- less, the resulting optimization problem of minimiz- ing the long-run average costs under an (s1S) policy is not convex; therefore, a complete enumeration of order size Q, defined as S − s, is necessary to find the global optimum (Fu and Healy 1997). In our imple- mentation, we set T at 440 days to achieve a balance of efficiency and solution quality.
Because of the discrete nature of demand in a phar- macy, we devised a local search-based heuristic that starts from an initial inventory solution or policy, and moves or adjusts the (s1S) values around the neigh- borhood of the current solution to search for near- optimal inventory policies. The heuristic alternates mainly between two phases: the first, denoted by Procedure 1, attempts to find the reorder point under specific order sizes Q to achieve a balance of inven- tory and shortage costs; the second, denoted by Proce- dure 2, aims to balance the ordering and inventory costs by varying the value of order size Q.
These procedures consist of several moves. Specif- ically, we accomplish Procedure 1 through Move 1 in which we increase or decrease (s1S5, each by an equal amount; we accomplish Procedure 2 through either (1) Move 2 in which we increase s but keep S unchanged to decrease the order size Q, or (2) Move 3 in which we increase S and leave s unchanged to increase the order size Q0 For details of the algorithm, see the appendix.
Although the procedures are similar to those used in the literature for inventory optimization, the most salient feature of our algorithm is its selection of neighbor solutions based on the simulation results, thus enabling the algorithm to achieve rapid conver- gence. To illustrate, we present an example of how these moves work for the inventory problem outlined in Figure 2. In this problem, the demand is generally
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 77
Figure 4: The simulation results from an 4s1S5 policy of (180, 220) suggest either an increase of s by 20 units to increase negative inventory from −20 to 0 or a decrease of s by 10 units to reduce the lowest positive inventory from 10 to 0 in search of a local optimum under Q = 40.
30, 60, or 90 units, with occasional demand for 10, 15, or 45 units. We assume that the package size is 10 units per bottle and an order is a multiple of this package size.
Starting Solution 1 41801 2205: For simplicity, we start the algorithm with a reorder point equal to the maximum demand of an order period; in this case, this value is 180. We calculate the economic order quantity (EOQ) to be 40, which gives a starting solu- tion of (180, 220) with an objective value of 25.17. Figure 4 shows how the ending inventory (top panel) and demand (bottom panel) evolve over time for 4s1S5 = 41801 2205.
Let us start with Procedure 1 and Move 1 (i.e., we increase or decrease (s1S5 by an equal amount). As we can see from Figure 4, the maximum negative ending inventory is −20 and the minimum positive inventory is 10. If we are to increase s, increasing it by 20 is intu- itive to bring the highest negative inventory to 0. If we are to decrease s, decreasing it by 10 units is intuitive to bring the minimum positive inventory to 0. Increas- ing the reorder point, for example, from 180 to 200, leads to a better objective; therefore, we adopt it. By keeping Q at 40, we can move to the next inventory policy, given by 42001 2405 with an objective value of 12.97; Figure 5 shows the ending inventory for this.
Solution 2: Inventory policy 42001 2405 exhibits no excess inventory or OOS situations and has no
negative inventory. It is the local optimum under Q = 40, because no Move 1 values will result in better solutions. This concludes the search in Procedure 1.
When Procedure 1 reaches a local optimum, a change of order size Q is necessary; we perform this in Procedure 2 in which we can either increase or decrease Q. Silver et al. (1998) state that when s and S (or Q5 are determined simultaneously, Q is always larger than the EOQ; therefore, we initially increase the size of Q.
Notice that in the ending inventory plot for Solu- tion 2, 42001 2405, the minimum positive inventory level is 30. If we are to increase the size of Q, two options are available. The first is to apply Move 2, which decreases s by 30, while keeping S at 240; this would move from 42001 2405 to 41701 2405. The second is to apply Move 3, which increases S by 30 units, while keeping s at 200; this would move from 42001 2405 to 42001 2705. By increasing the order size Q, the policy is likely to increase the inventory holding cost. Here, we select the first option to offset this inventory increase, and finish at 41701 2405. Nev- ertheless, if we apply the second option, which we do not show for brevity, it will result in the same local optimal solution under the same order size (Q = 70). Similar moves, which we also omit, also exist for the case in which we decrease Q.
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 78 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
Figure 5: In this simulation of (200, 240) with the objective value of 12.97 under Q = 40, the minimum inventory is 30, which suggests that a potential change in order size of 30 units will be required to find a better solution.
Figure 6: A simulation of (170, 240) suggests a parallel reduction in both s and S by (i.e., Move 1), because all ending inventory levels are above 0.
Solution 3: Inventory policy 41701 2405 shows an objective value of 12.31. Figure 6 depicts the ending inventory. Notice that the minimum ending inventory is 5, which is above 0, suggesting a parallel reduc- tion in both s and S (i.e., Move 1), which leads to the solution 41651 2355.
Solution 4: Inventory policy 41651 2355 shows an objective value of 11.96. Figure 7 depicts the ending
inventory. We confirmed this solution to be the best solution in the discrete solution space by an exhaus- tive enumeration of all possible (s1S) combinations (see Table 1).
In Table 1, the numbers in the top row (column headings) represent the order size Q, and the numbers in the first column are the reorder points (s). Notice that in this example, the algorithm takes only four
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 79
Figure 7: A simulation of (165, 235) is locally optimal with respect to Q, and we confirmed it to be the global optimum.
steps to converge, producing solutions for 41801 2205, 42001 2405, 41701 2405, and 41651 2355 to arrive at the optimum for the problem.
Computational results: Our empirical experience shows that our algorithm on average takes only a few evaluations of specific (s1S) policies and tens of mil- liseconds to find near-optimal solutions to the inven- tory problem. We implemented it on an Intel-based personal computer with 8 GB of RAM and an i7-2600
s\Q 0 10 20 30 40 50 60 70 80 90 100
150 103053 90.12 90.62 63.56 63.87 51.17 24.19 24.51 24.33 18.04 18.52 155 103088 90.47 90.97 63.91 50.62 24.32 24.54 24.96 25.28 18.39 18.87 160 104023 90.82 43.72 50.66 50.37 24.67 24.83 25.71 12.03 18.74 19.22
165 90098 43.57 44.07 51.01 41.32 25.02 25.24 11.91 12.38 19.09 19.57
170 91033 43.92 44.42 51.36 24.47 25.37 25.39 12.31 12.73 19.44 13.12 175 91068 44.27 31.17 24.51 24.82 25.72 25.94 12.66 13.08 12.99 13.47 180 44043 31.02 31.52 24.86 25.17 26.07 12.69 13.01 13.43 13.34 13.82 185 44078 31.37 31.87 25.21 25.52 12.82 13.04 13.36 13.78 13.69 14.17 190 45013 31.72 25.42 25.56 25.87 13.17 13.39 13.71 14.13 14.04 14.52 195 31088 25.27 25.77 25.91 26.22 13.52 13.74 14.06 14.48 14.39 14.87 200 32023 25.62 26.12 23.26 12.97 13.87 14.03 14.41 14.83 14.74 15.22 205 32058 25.97 26.47 13.01 13.32 14.22 14.44 14.76 15.18 15.09 15.57 210 26015 26.32 26.82 13.36 13.67 14.57 14.73 15.11 15.53 15.44 15.92 215 26048 26.67 27.17 13.71 14.02 14.92 15.14 15.46 15.88 15.79 16.22 220 26083 27.02 13.92 14.06 14.37 15.27 15.43 15.81 16.23 16.14 16.62
Table 1: The table shows the objective function values of the example’s various inventory policies, and illustrates the problem’s solution space; the heuristic takes only four iterations (solutions in bold) to find the optimum solution.
processor running at 3.40 GHz. The total computa- tion time to sequentially solve the problem for each of 1,950 stores with an average of 2,000 drugs per store is approximately six hours, or equivalently an average of 10 milliseconds per drug per store. To test optimal- ity, we compared the results of our methodology with that of the grid search for the 1,621 drugs in a test store. The algorithm was able to achieve the global optima discovered by the grid search for 93 percent
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 80 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
of the drugs. The simulation-optimization procedure produces visually appealing output and is versatile enough to incorporate various forms of stockout costs and inventory policies.
Business Results and Benefits The Kroger pharmacy division and its OR team started to roll out the inventory simulation-optimi- zation methodology to the enterprise in early Octo- ber 2011. They implemented the system in six divi- sions every two weeks, which gave the warehouses time to regain inventory positions and prevented sud- den OOS situations throughout the supply chain. By the end of November 2011, Kroger had completed the implementation across all 18 divisions: on a weekly basis, the simulation-optimization inventory manage- ment system reads the demand transaction history for up to 220 days, reviews delivery schedules for each store, forecasts demand by sampling from empir- ical data the distribution for each drug based on its demand history, finds the optimal reorder point and order-up-to levels, and sends the inventory settings to the central pharmacy system.
The results have been overwhelmingly positive. The system has significantly improved customer ser- vice, thus ensuring that patients have greater access
Figure 8: Implementing our model provided significant benefits: A reduction in stockouts of approximately 1.6 million per year and an increase in revenues of almost $80 million per year.
to medications when they need them. It has also yielded both tangible and nontangible benefits for the pharmacy division. Figure 8 illustrates the percent- age of OOS drugs and the number of prescriptions sold between July 2011 and April 2013. The horizontal axis represents the end of each fiscal period (Kroger divides its fiscal year into 13 periods, each consisting of exactly 28 days, with the first period typically start- ing on the first day of February). The dashed plot and right vertical axis correspond to the number of pre- scriptions sold (in millions) per month, and the solid plot and left vertical axis correspond to the percentage of OOS drugs. The vertical dashed line between fis- cal periods 10 and 11, corresponding to October 2011, indicates the time of model implementation.
Before Kroger implemented our model (the region to the left of the vertical dashed line in Figure 8), its stockouts were steady at 3.4 percent. At the end of May 2012, seven months after implementation, this rate had decreased to 2.4 percent.
The number of OOS drugs increased slightly begin- ning in June 2012 (i.e., period 9 in 2012). Kroger initially attributed this to seasonal adjustments and business growth; the pharmacy division had seen a sig- nificant volume increase because of several insurance contracts. However, the company recently discovered
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 81
that because of a system transfer, the simulation- optimization results were not being uploaded into the central server for periods 9–13, exactly the interval in which the number of OOS drugs increased. Once the system uploads were resumed, the OOS drugs dropped quickly back to 2.3 percent.
Our conservative estimate of a one percent OOS drug reduction translates into a decrease in OOS pre- scriptions of 1.6 million per year, despite a volume increase.
Financial and Nonfinancial Benefits for the Pharmacy Division The tangible financial benefits include the following:
1. Additional revenues as a result of a decrease in OOS prescription drugs and an increase in collateral sales: Although a majority of customers whose pre- scription drugs are OOS will have their prescription partially filled, some will fill their prescription else- where. As a consequence, Kroger could lose all future pharmacy purchases by these customers and some of their collateral nonpharmacy purchases. Account- ing for the customers who go to another pharmacy, the prescription and collateral purchases from these 1.6 million OOS prescription drugs per year translate into an increase in annual revenue of $80 million for Kroger pharmacies.
2. Labor savings: Customers who choose to have their prescription partially filled will have to come back for the remainder of the prescription, which requires additional labor that would be otherwise unnecessary. Multiplying the annual labor reduction by a blended cost rate for pharmacy employees results in estimated labor savings of about $10 million per year; note that the annual compensation for the entire 15-person OR group is less than $2 million.
3. Inventory reduction: Accompanying the service level increase and reduction of OOS drugs, we have also seen a dramatic reduction in inventory. Although fluctuations exist, we have seen a steady reduction in inventory from 45 to 38 days of supply; this translates into a one-time reduction of more than $120 million in inventory investment.
From a qualitative customer service perspective, the reduction of OOS drugs decreased the number of times that customers were inconvenienced by an OOS drug each year by 1.6 million, impacting approx- imately 1 of every 100 customer interactions. In the
case of antibiotics, pain-relief medicines, and blood pressure-control medicines, the benefits to customers are difficult to quantify, but they clearly improve the immediate well-being of these customers. A positive experience is the key to retaining customer loyalty and driving future growth.
Extending the Model for Meat and Seafood Since we implemented our model and its associated methodology at Kroger, we have embellished both and transformed them into a more general inventory optimization framework for the company’s perish- able product lines of business. These products include meat and seafood, which have large demand varia- tions and can suffer from significant spoilage if their inventory is not properly managed. Future enhance- ments to our system include (1) a multiple linear regression-based forecasting model for each product based on underlying predictor variables, such as sales promotions, day of the week, week of the month, and holiday status, and (2) a simulation-optimization approach to optimize inventory settings for each product on each day of the upcoming week with the goal of minimizing spoilage and OOS drugs. This approach can accommodate changes in sales based on freshness and marked-down products. Initial results in pilot stores show that the enhanced system signifi- cantly outperforms Kroger’s current processes, and a conservative estimate shows that Kroger can reduce product spoilage by 50–60 percent. This could reduce costs by tens of millions of dollars annually for the meat and seafood business division. Fresh, available meat is key to future sales of these products and their associated revenues.
Scientific Inventory Management as a Core Competency at Kroger The success of its inventory system has been critical to Kroger, which aims to provide the right products at the right time for its customers. Inventory manage- ment is an integral part of ensuring that the company continues to be successful. The easy-to-understand, visually appealing inventory system has won the sup- port of Kroger’s executives, and has accelerated the use of scientific inventory management as a core com- petency throughout the company. Extending the sys- tem to its other divisions has become one of Kroger’s strategic goals.
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 82 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
Growth of Kroger’s Operations Research Team The successful implementation of the inventory simulation-optimization system has shown that OR can bring many benefits to Kroger. It has significantly improved the reputation of OR and contributed to its growth throughout the company. Before the imple- mentation of this system in March 2011, the Kroger OR team consisted of only four full-time employ- ees and two contractors. As a result of the success- ful implementation, the OR team has nearly tripled in size; as of April 2013, the team had 12 full- time employees and five contractors. The OR team has also undertaken an increasing variety of projects across Kroger’s business activities, including its man- ufacturing, supply chain and logistics, store opera- tions, merchandising, and research and development departments.
Transportability and Operations Research Advances We designed our system to be transparent to users, agile in its application, and responsive to user feedback. The computational approach is indepen- dent of particular demand distributions and is easy for business people to accept. Table 2 highlights the differences between our simulation-optimization approach and the traditional approaches to inventory management.
The simulation-optimization system leverages com- putational capabilities and algorithms, such as local search, to effectively find optimal or near-optimal inventory policies under various cost structures. Given the advancements in computational power, we believe that the simulation-optimization system we
Traditional approaches Simulation-optimization approach
Based on analytic formulas Based on simulation Complex and difficult to understand Intuitive and easy to understand Resistance by business users Acceptance by business users Reliance on specific distributions Use of empirical distributions Induces errors with demand models Accurately models demand patterns Difficult to change Agile and adaptive to change Black box Transparent
Table 2: The table illustrates the advantages the simulation-optimization approach we implemented at Kroger over traditional inventory manage- ment approaches.
discuss in this paper offers an effective and intuitive approach to solve inventory optimization problems, increases our ability to address inventory problems in a comprehensive, analytic manner, and is readily transportable to other industries.
Appendix. Local Search Algorithm In the presentation of the algorithm, let f be the objective function value and superscript ′ the neighboring solution. Let Q+ and Q− be an upper and a lower bound for Q, respectively, and ãQ be the direction of change for Q (i.e., ãQ = +1 represents an increase of Q and ãQ = −1 represents a decrease of Q). Let ãs be the change of direction for s (i.e., ãs = +1 represents an increase of s and ãs = −1 represents a decrease of s). Let B be the set of order quantities and let P be the set of inventory policies and their corresponding objective function values encountered in the search process, and consists of the three-tuples (s1S1f ). We can summarize our iterative algorithm for finding near-optimal (s1S) poli- cies as follows.
Step 1. Initialization. Start the algorithm with s = s′, the reorder point, as the maximum demand of an order period; set Q = EOQ, set S = S′ = s + Q; let f be the current objec- tive function value and set its value to +�; set ãQ = +1 to increase Q and ãs = 1 to increase the reorder point; set B = Q.
Step 2. Simulation. Perform simulation over 440 days under (s′1S′), denote the simulation objective value as f ′, and add (s′1S′1f ′) into set P . Denote I4t5 as the ending inventory of each period t, and let I+ = mint2I4t5>08�I4t5�9 be the minimum positive inventory and I− = mint2I4t5<08�I4t5�9 be the absolute value of the maximum negative ending inventory.
Step 3. Increase or decrease reorder point. If f ′ < f , set s = s′, S = S′, f = f ′, continue the search as follows. If ãs = +1, set s
′ = s+I− and S′ = s+Q to try to raise the small- est negative inventory to zero; else, ãs = −1. Set s
′ = s − I+
and S′ = s + Q to try to lower the lowest positive inventory to zero. Go to Step 2. Otherwise, f ′ > f ; therefore, we have found a local optimal under Q; go to Step 4.
Step 4. Increase or decrease Q? Let Qã be the minimum of I+ and I− associated with (s1S), if ãQ = +1, then assign Q ← Q + Qã; otherwise, Q ← Q − Qã. If no Q exists inside (Q−1Q+5, set ãQ = −ãQ, reversing the search direction for order size Q. If Q already exists in set B, go to Step 6; oth- erwise, add Q to set B and go to Step 5.
Step 5. Perform move to a new (s′1S′5. If ãQ = +1, then move to (s′ = s − Qã1S = S5 and set ãs = 1. If ãQ = −1, then move to (s′ = s1S′ = s + Q) and set ãs = −1. Go to Step 2.
Step 6. Termination. Output the best solution in set P . In Step 1, we start with an order size equal to EOQ. Silver
et al. (1998) state that in the simultaneous determination of s and S, (or Q), Q is always larger than EOQ. As such, we set ãQ = +1 to increase EOQ. The reorder point can be set
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management Interfaces 44(1), pp. 70–84, © 2014 INFORMS 83
in different ways, for example, the maximum demand over the lead time or with any analytical approximation solu- tions. We use the maximum demand over lead time (i.e., replenishment lead time plus an order period).
In Steps 2 and 3, we perform simulation with given val- ues of s and S. Based on the ending inventory resulting from the simulation, we adjust both the reorder and order- up-to quantities by the same amount to keep the order size unchanged. Specifically, if the current solution improves, then we continue the search until we obtain a local opti- mum under a fixed order size. In making these adjust- ments, if we increase the reorder point, then the algorithm increases the reorder by I− to bring the maximum (i.e., min- imum absolute value of) negative inventory to zero (reduc- ing stockout cost); if we decrease the reorder point, then the algorithm decreases the reorder point by I+ to bring the lowest positive inventory to zero (reducing inventory cost).
In Steps 4 and 5, we perform an adaptive search with respect to the order size Q. Rather than enumerate all possible breakpoints, which could be time consuming, we increase or decrease the order size based on the ending inventory level derived from the simulation. We can change the order size by either changing the reorder point s or changing the order-up-to level S; in either case, Steps 2 and 3 will lead to the same local optimum.
Finally, we stop if we have not found an improvement or if Q is outside the predefined bounds. We then output the final best solution encountered in the search process in Step 6. Tighter bounds on s, S, and Q could impact the com- putational efficiency of the algorithm in that they affect the number of breakpoints to be evaluated by Q. Following the approach proposed by Zheng and Federgruen (1991) and by Gallego (1998), our computational experiments show that for most of the drugs that Kroger stocks, the algorithm can converge quickly within a few iterations.
Supplemental Material Supplemental material to this paper is available at http://dx .doi.org/10.1287/inte.2013.0724.
References
Bell PC, Anderson CK, Staples DS, Elder M (1999) Decision-makers’ perceptions of the value and impact of visual interactive mod- eling. Omega 27(2):155–165.
Forbes (2010) American companies that give back the most. Accessed October 1, 2013, http://www.forbes.com/pictures/ mkl45ejjg/1-kroger-company/.
Fu MC, Healy KJ (1997) Techniques for optimization via simulation: An experimental study on an (s1S) inventory system. IIE Trans. 29(3):191–199.
Gallego G (1998) New bounds and heuristics for (Q1r) policies. Management Sci. 44(2):219–233.
Kleijnen JP, Wan J (2007) Optimization of simulated systems: OptQuest and alternatives. Simulation Model. Practice Theory 15(3):354–362.
Law AM, Kelton DW (2000) Simulation Modeling and Analysis, 3rd ed. (McGraw-Hill Education, New York).
Revolution Analytics (2013) Unleash the power of open source R, the world’s most sophisticated data analysis software. Accessed October 1, 2013, http://www.revolutionanalytics .com/what-is-open-source-r/.
Silver EA, Pyke DF, Peterson R (1998) Inventory Management and Production Planning and Scheduling (John Wiley & Sons, New York).
Tiwari V, Gavirneni S (2007) Recoupling inventory control research and practice: Guidelines for achieving synergy. Interfaces 37(2):176–186.
Zheng Y-S, Federgruen A (1991) Finding optimal (s1S) policies is about as simple as evaluating a single policy. Oper. Res. 39(4):654–665.
Xinhui Zhang is an associate professor of industrial engi- neering at Wright State University. He received his PhD in operations research and industrial engineering from the University of Texas at Austin. Dr. Zhang’s research interests are simulation, optimization, and the solution of practical problems in the area of manufacturing, logistics and trans- portation, media management, and service operations. He has led several high impact research projects such as airline crew recovery, advertising allocation, production planning, staff scheduling, vehicle routing, inventory simulation opti- mization, marketing research, and enrollment management and optimization. He is an active member of INFORMS.
Doug Meiser joined The Kroger Co. in 2004 as a forecast- ing system administrator, as he started the MBA program at Northern Kentucky University. In 2007, Meiser transitioned to research and development to lead the development and growth of the operations research team. Since that time, the team has worked and implemented projects in facility lay- outs at the stores, distribution centers, and manufacturing plants, along with many efforts in research and develop- ment, inventory optimization, staff scheduling, and strate- gic supply chain analysis. Meiser completed his MBA in 2008, and before joining The Kroger Co. completed his BS in mathematics and physics from NKU.
Yan Liu is an associate professor in the Department of Biomedical, Industrial, and Human Factors Engineering at Wright State University. She earned her MS and PhD degrees in industrial engineering at Purdue University in 2001 and 2006, respectively, and her BS degree in mechani- cal engineering at the University of Science and Technology, Beijing, China in 1998. Dr. Liu’s current research interests include data mining, information visualization, modeling and simulation, design of interactive systems, computer- supported cooperative work, and usability evaluation. She is serving on the editorial boards of International Jour- nal of Human-Computer Interaction and Human Factors and Ergonomics in Manufacturing and Service Engineering.
Brett Bonner is a senior director for research and devel- opment at The Kroger Co. Previously, Brett worked for Federal Express and Kellogg Company where he held the positions of managing director and vice president, FedEx
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.
Zhang et al.: Kroger Uses Simulation-Optimization for Pharmacy Inventory Management 84 Interfaces 44(1), pp. 70–84, © 2014 INFORMS
Internet Technologies Corporation. He has created numer- ous operational and technical enhancements. Significantly, he has 19 patents with additional patents pending. His team received the Planet Retail “Innovator of the Year” award in 2011. Brett’s organizations specialize in serial inno- vation using a wide spectrum of technologies, inventions, and algorithms. Direct measurements of bottom line con- tributions of his innovations now exceed $9 billion. Brett graduated with a BS in mechanical engineering from the University of Memphis. He was the valedictorian of the Herff College of Engineering. He also attended gradu- ate programs at Western Michigan University, Rennesslaer
Polytechnic Institute, and Concord University School of Law. He is a registered professional engineer.
Lebin Lin is a PhD candidate majoring in industrial engi- neering at Wright State University. He received both his BS degree in software engineering and BA degree in English at Dalian Jiaotong University, China in 2007. He then obtained his MS degree in computer science in the College of Engi- neering and Computer Science at Wright State University in 2010. His PhD study focuses on the algorithm design for the pharmacy inventory simulation and optimization sys- tem for Kroger Co. He was awarded the 2011 University Assistantship from Wright State University.
D ow
nl oa
de d
fr om
i nf
or m
s. or
g by
[ 14
4. 37
.1 .1
10 ]
on 2
4 N
ov em
be r
20 14
, a t
15 :5
2 . F
or p
er so
na l
us e
on ly
, a ll
r ig
ht s
re se
rv ed
.