423 Week 5: Formal source selection

Defense AT&L: March–April 2015 2

From the Under S ecretar y of Defense for Acquisition , Technolog y, and Logistic s

Getting “Best Value” for the Warfighter and the Taxpayer Frank Kendall

W e use the phrase “best value” fairly often, usually to describe the type of source-selection process or evaluation criteria we will use in a competitive acquisition. Under the Better Buying Power initiatives, we have emphasized using a more monetized and less subjective definition of best value. As a way to spur innovation, we also have emphasized communi- cating the “value function” to the offerors so they can bid more intelligently.

Some reluctance and understandable concern arose about the unintended consequences of trying to define best value in monetary terms. In fact, this decision can’t be avoided. I would like to explain why it is unavoidable, provide some examples of using this approach, and discuss how we can avoid those unintended consequences some of us worry about. I’ll also touch on the proper use of Lowest Price, Technically Acceptable (LPTA)—which is a form of monetized best value, but with a very restrictive definition and range of applicability.

A “traditional” best-value source-selection process combines disparate metrics in to one overall evaluation. In a recent example that I reviewed, four separate and unrelated metrics were proposed for the source selection: risk (high, medium or low), cost ($), performance (a composite scaled metric) and degree of small business utilization (with its own scale). Think how this would have played out in the source-selection decision making. Setting aside the small business metric, assume that there was a slightly more expensive and higher-risk but much higher-performing offeror and a slightly less expensive and lower-risk but significantly lower-performing offeror. The Source Selection Authority would have to decide whether the increased price and risk of the higher offeror was worth the difference in performance. That acquisition official, not our customer (the warfighter), would have needed to make the “best value” determination as a subjective judgment by weighing cost against the other two

3 Defense AT&L: March–April 2015

From the Under S ecretar y of Defense for Acquisition, Technolog y, and Logistic s

integratable into their offerings. In addition, communicat- ing this information to industry allows uncompetitive firms to avoid wasting company funds (allowable Bid and Proposal costs in overhead that the government reimburses) on pro- posals that have no chance of success. We have to define best value if we want industry to offer it to us.

There is a side benefit to monetizing best value criteria in that the objective source-selection criterion are harder to contest successfully. I don’t believe we should design our source- selection criteria or acquisition strategies around minimizing the likelihood of a protest, whether it is a successful or an unsuccessful protest. But I don’t mind having that feature as a byproduct of our approach. Avoiding successful protests is about setting down the rules for source selection, follow- ing them religiously, documenting the decisions we make so we can explain them if challenged, and maintaining the pro- cess integrity. All our source selections, of any type, should be conducted in this manner. At the end of the day, however, no one should be able to argue with the government about the monetary value we place on a specific feature or level of performance before we conduct a source selection (as long as we have a reasonable rationale for our choices and aren’t being arbitrary). This judgment also is easier to defend if it is transparent and communicated to offerors well before we start the source-selection process.

About 15 years ago, while in industry, I tried for months to get the Air Force to provide some allowance, some competitive

metrics. In effect, that individual in the acquisition chain would make the precise cost versus performance and risk judgment we intend when we recommend monetizing the value of per- formance and including it in the evaluated price.

The likely bias for an acquisition official making the source selection is to take the lowest-price offer; it’s much easier to defend than the subjective judgment that the higher-cost of- feror was worth the difference in price. Is this the best way for us to do “best value” source selections? To the extent we can do so, we are better off defining “best value” by a single parameter we can readily compare. The easiest way to express that parameter is in dollars—using value-based adjusted price for evaluation purposes (e.g., bid price with predefined dollar- ized reductions for performance above threshold).

I believe there are some very good reasons to take the ap- proach of monetizing performance metrics. First of all, it forces our customers—the operators who set requirements—to con- sider how much they are willing to pay for higher performance. Our normal practice in the requirements process is to define two levels of performance—threshold and objective. Unless we provide industry an incentive to do otherwise, we can ex- pect it to bid the threshold levels of performance and no more. The simple reason is that we usually don’t give industry any competitive incentive to offer higher performance. The lower threshold levels of performance almost always are the lowest- cost levels of performance.

Getting the requirements community to consider what it would be willing to pay for different levels of performance also has an important side benefit: It forces that user community to recognize that its requirements are not free and to engage the acquisition community on prioritizing those requirements. We must work as a team to be effective. Involving our customers in decisions about best value before releasing the final Request for Proposals (RFPs) builds our mutual understanding of the real-life trade-offs needed in almost any product or service acquisition. Monetizing best value to industry also provides benefits that accrue to the government. By not providing in- dustry with a business reason to offer higher performance, we create a disincentive for innovation. We want industry to be in a position to make informed judgments about what level of performance to offer. The easiest way to accomplish this is to tell industry exactly, in dollars and cents, what higher levels of performance are worth to us. Industry then can compare its costs of meeting higher performance levels to our willingness to pay and decide what performance to offer.

We also should provide this information as early as possible, so industry has time to react to the information, including, when possible, time to develop new technologies that are

We usually don’t give industry any competitive incentive to

offer higher performance. The lower threshold levels of performance almost always are the lowest-cost levels of

performance.

Defense AT&L: March–April 2015 4

credit, for my company’s AIM 9X air-to-air missile’s above- threshold performance. We had a novel design with excep- tional off bore-sight capability, well above the threshold re- quirement. I didn’t succeed and we lost the competition, but the Air Force also lost the opportunity to acquire an innovative design with superior performance. I find it hard to believe that performance had no value whatsoever to the Air Force. In any event, we received no credit in the source selection for offering what we were certain was a better product.

We have been using the technique of monetizing performance differences in source selections under Better Buying Power 2.0 and will continue this emphasis under BBP 3.0, but the practice didn’t start with BBP.

One early use was in the second KC-46 Tanker competition. There was a successful protest by the losing offeror in the first competitive best-value source selection conducted in 2008. In the second competition in 2009, we moved to much more objective source-selection criteria, using evaluated price as the primary metric. In addition to folding fuel costs and operational efficiency into the evaluated price, we allowed for consider- ation of a long list of “desired but not required” features, but only if the evaluated prices were within 1 percent for the two offerors before we considered these features. Essentially, we bound the value of all these objective features as being worth no more to us than 1 percent of evaluated price. Notice that this had nothing to do with the cost of those features.

Value or worth to the buyer has nothing to do with cost; it is only about what we would be willing to pay for something. The tanker situation is analogous to buying a car and deciding what

options to include. All those options, the “fully loaded” version of the tanker if you will, were only worth a 1 percent price dif- ferential to us. Having this information allowed industry to be a smarter offeror and propose a product more in line with our “value function.”

More recently I had an experience with the acquisition strat- egy for a tactical radio program where the program manager intended to use a LPTA approach. He was asking for threshold performance and didn’t plan to provide any credit to higher performance in the evaluation criteria.

I asked him hypothetically if he would want to buy a radio with twice the range and twice the message completion rate for 1 percent more. The answer, of course, was yes. We changed the evaluation criteria. Sometimes LPTA makes sense but it doesn’t make sense if we are willing, as we usually are, to pay a little more for a much better product. LPTA may be an easier way to do a solicitation and a source selection, but that shouldn’t be our metric. The warfighter and the taxpayer deserve better from us. LPTA is appropriate when we have well-defined standards of performance and we do not place any value on, and are therefore unwilling to pay for, higher performance.

LPTA is used in many acquisitions for services. As discussed above, it may be appropriate—if there is no value to the gov- ernment in performance beyond well-defined thresholds.

The arguments against monetizing best value include a con- cern recently expressed by an Army program executive officer: Industry is likely to game the system to try to win. He was right, of course. We want “best value.” Industry wants to win. Nevertheless, I don’t find this to be a strong argument against monetizing best value. I do find it to be a strong argument for getting it right and making sure we align our source-selection criteria with what we want (what we value). If we have properly defined what is important to us and what we are willing to pay for that “best value,” industry will position itself to meet our best-value proposition.

There are various possible ways to meet our best-value propo- sition—and from industry’s point of view, that’s not gaming us; that‘s doing what it takes to win. Our concern should be with getting the “best value” criteria right. We need to mon- etize best value in a way that doesn’t permit an unintended consequence imposed on us by a crafty proposal team. I have worked on a reasonable number of proposals from the industry side and I know the concern has some validity. When we set source-selection criteria, we need to do our own red-teaming process to ensure we don’t produce unintended and nega- tive consequences. Basically, this is just a matter of running through the range of possible approaches to bidding to see if

Sometimes LPTA makes sense but it doesn’t make sense if we are willing, as

we usually are, to pay a little more for a much better

product.

5 Defense AT&L: March–April 2015

we have neglected an excursion that has an unintended and negative effect. You can count on industry to do the same.

I have also heard the concern that industry may inflate its pric- ing to come just under what we are willing to pay, even if the cost is substantially lower. In a competitive acquisition, we should be able to count on the fact there will be other bidders to prevent this behavior. Offerors have to beat the competition, regardless of the government’s willingness to pay. Incidentally knowing our published budget figures also provides industry with a strong indication of what we could pay for the product. In any case, we must use either competition or, in a sole-source environment, discussions about actual costs to ensure we get a reasonable price for the warfighter and the taxpayer. Monetiz- ing best value doesn’t change those processes.

In development contracts, we often are concerned about risk, and it’s fair to ask whether it is possible to monetize risk considerations. We can set subjective risk scales for evalu- ation purposes and do so routinely, using High, Medium, and Low—or a more finally grained alternative. Translating these comparisons into relative monetary value takes some thought, but it can be done. One has to be careful because risk valuations can be very nonlinear. For example, “low-risk” and “medium-risk” offerors might have fairly small differ- ences in “value,” but a high-risk offeror could (and probably should) have prohibitively high cost adjustments to over- come. We would expect both low- and medium-risk offers to be obtainable but with cost and schedule impact differences. A high-risk offer has a finite probability of being outside the realm of the possible.

A better way to handle risk factors is to create thresholds or “gates” as opposed to comparative assessments. If an offer has acceptable risk, it is considered responsive and evaluated for cost and performance. If an offer has high risk, it is elimi- nated from the competition. This is one of the many areas in which we have to use professional judgment and a real un- derstanding of the actual risks involved in order to make a good decision.

It is argued that this approach is more difficult and time con- suming. A former senior official once told me that “conve- nience” was the biggest determiner of an acquisition strategy. I certainly hope that is not so. We do have finite capacity, but we owe our customers our best efforts in every acquisition. I am not persuaded that monetizing best value is prohibitively difficult. It is a new approach for many in the requirements community, and they won’t be comfortable with it until they have more experience.

My first attempt to use this approach was on the Combat Rescue Helicopter program. It took several attempts to get

the user community to stop bringing me cost estimates for various levels of performance. Ultimately, the users concluded that the cost premium the Air Force was willing to pay for objective performance was only about 10 percent. This infor- mation caused one company to drop out of the competition. I’m not troubled by that result. It would have been a waste of time for that company to prepare an offer. It does take a little more effort up front to define best value in monetary terms. However, the source-selection process is made simpler, and, more importantly, we can get better results for our customers. That is the metric that should matter most to us.

As we build our teamwork with both the warfighters who set requirements and with industry which tries to win business by meeting those requirements, I believe there will be more acceptance and support for monetizing best value. It is in ev- eryone’s interest and well worth the effort.

(Editor’s Note: For further review of industry and govern- ment assessments of LPTA, see the Acquisition Discussion articles beginning on pp. 16-17.)

Farewell to James S. McMichael

Dr. James S. McMichael—vice president and a former three-term acting president of the Defense Acquisition University in 1991-1992, in 2010 and in 2012-2014—retired in January. He had been DAU’s vice president since 2005.

Earlier, Dr. McMichael was director of acquisition education, training and career development in the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. In that position, he was the principal advocate for workforce man- agement and formulated policies and programs to ensure workforce quality and professionalism.

McMichael also has served as the technical director for the Navy Personnel Research and Development Center in San Diego, Calif.; special advisor for manpower, personnel and training in the Office of the Chief of Naval Operations; and chairman of the psychology department at Long Island Uni- versity in New York, where he taught for eight years.

McMichael is a graduate of Princeton University and re- ceived his advanced degrees at the University of Delaware. He was a fellow at Princeton’s Woodrow Wilson School of Public and International Affairs from 1982 to 1983.