Discussion 5.1
LDots01
ChoosingaTrulyExternalEvaluator.pdf
372
Marilyn Ray, Finger Lakes Law & Social Policy Center, 96 Besemer Road, Ithaca, NY 14850; phone: (607) 539-7778; e-mail: [email protected].
American Journal of Evaluation, Vol. 27 No. 3, September 2006 372-377 DOI: 10.1177/1098214006289787 © 2006 American Evaluation Association
Choosing a Truly External Evaluator
Marilyn Ray Finger Lakes Law and Social Policy Center
Setting the Scene
A proposal has been published by a consortium of foundations for an “external” evaluator to evaluate a replication at two new sites of a program they have been funding for many years. A proposal is received from Dr. Porto-Novo, who has been the external evaluator of the ini- tial program for about 10 years. She has developed much of her reputation and that of her group, Riga, Inc., as well as the majority of Riga’s budget, from this work. High-quality pro- posals are also received from several other fully qualified evaluators.
A team consisting of two staff members from the initial program, two foundation staff members, and an external evaluation expert has reviewed the proposals. The team has given several of the proposals high rankings, but it is split about the award. Staff members from the initial program want to award the contract to Riga, but the staff representatives of the foun- dation argue against this. They explain that they feel that Riga has such a close relationship with the agency that it can no longer be considered external. Serving as the external evalua- tion expert on the team, I am asked to help break the impasse. I can see advantages and dis- advantages to both arguments.
Gathering Information to Inform My Decision
I begin my task by thinking about several key questions and decide to gather information to address them to help me reach a decision. I discuss each of these questions and what I dis- cover about their answers below.
What Is the Definition of an External Evaluator?
First, I refresh my understanding of what makes an external evaluator different from an internal evaluator. The dictionary tells me that external means “not intrinsic; . . . of, relat- ing to, or connected with the outside” (Webster’s Ninth New Collegiate Dictionary, 1984). This definition is helpful in that it specifies that an external evaluator cannot be intrinsic, but how does “relating to” the outside apply? I decide to search for the federal definition of a con- sultant to see if I can find something more readily applicable and find the following: “A con- sultant is a non-employee who provides advice and expertise in a specific program area” (Substance Abuse and Mental Health Services Administration, 2006). In applying this definition
Ray / Choosing a Truly External Evaluator 373
to program evaluation, I understand that an external evaluator is not on the staff of the organization whose program is being evaluated but rather is a paid consultant who works under contract for a specified period of time. Riga technically meets this definition, and the evaluators who submitted the other high-quality proposals definitely meet it.
To further clarify this issue, I turn to a biophysicist friend, who tells me that in the physi- cal sciences, a replication study uses the same treatment, conditions, and research protocols as were used in the original study, but on new subjects, and is always conducted by a scien- tist other than the one who carried out the first study. In our scenario here, we know that the program, or “treatment,” is being replicated, and that should Riga win the contract, it will con- duct the evaluation. What we do not know is whether the evaluation protocols used for eval- uating the initial program will be replicated as well as the initial program model. I suspect that there is no such expectation, though not replicating the evaluation protocols could lead to very different results. But that challenging and interesting issue is beyond the scope of this discussion.
At the 2002 annual conference of the American Evaluation Association (AEA), held in Washington, D.C., I attended a session sponsored by the Independent Consultant Topical Interest Group on ethics in practice, “The Enron Scandal: Independent Consultants as Evaluators” (chaired by Christy Lynch, with panelists Mike Hendricks, John Seeley, and Melanie Hwalek). I remember a question being asked of the panelists about how long an inde- pendent consultant could contract with a single agency to evaluate one of its programs and still be considered external and independent. The responses from the experienced evaluators on the panel ranged from 2 to 5 years. All panelists agreed that after some period of a few years, an evaluator can become fiscally dependent on the income from a single source and/or so integrated into the staff that he or she is no longer really independent.
In addition, in being independent of the agency whose program is under review, external evaluators are thought to bring two special assets to an evaluation: a vision or understanding, sometimes called objectivity, of issues and concerns that is not constrained by the day-to-day operations of a program and freedom from the pressures and constraints that can derive from evaluating the work of colleagues and one’s boss.
Dr. Porto-Novo and Riga have been the external evaluators for the initial program for about 10 years, and according to the scenario, derive most of their income as well as their reputa- tion from this work. It is hard to fit these facts into either the definitions of an external eval- uator or the responses from the panelists in the AEA session. These findings all seem to weigh heavily against Riga being considered an external evaluator and getting the contract.
This brings to mind another reason why the length of time Riga has served the initial program might serve as a flag: They may have developed the blinders that frequently come with becoming ingrained in a program over a period of years. On this point, I recall several years ago, when I was the managing evaluator for a statewide evaluation of the New York State Child Support Guidelines under contract to the New York State Department of Social Services. One aspect of the guidelines calls for a court to examine the ability of a noncusto- dial parent to cover the medical costs of the child in question. During one of the stakeholder meetings, a man whose knowledge of data and statistical procedures I had come to value very highly during the course of the evaluation spoke up about our not needing to collect data on the medical coverage because “we have that in our Medicaid records.” This man had been working at the department so long that he had forgotten that the majority of judgments of divorce are not for recipients of public benefits or Medicaid. Although such custodial parents may be in danger of falling into poverty and becoming eligible for public benefits if medical costs are not appropriately distributed in the judgment of divorce, searching Medicaid records would not be sufficient to address the question of how this section of the child support law
374 American Journal of Evaluation / September 2006
was being applied. This is the kind of trap that Riga will have to avoid if it becomes the eval- uator on the replication study. Riga will have to think creatively in ways that it may have lost sight of over the years of work on the initial program. In other words, Riga will need to be able to approach the evaluation of the replication programs and the interpretation of the data from it with new thinking and in fresh ways. The question before the review team is whether Riga’s long association with the program has caused it to have some blinders or biases that could negatively affect the quality of the evaluation and/or the interpretation of data from the evaluation of the replicated programs.
What Are the Underlying Ethical Considerations That Should Guide This Decision?
Next, I turn to the AEA’s (2004) Guiding Principles for Evaluators for underlying ethical considerations that I can use to guide my thinking for this decision. As I review these guide- lines, the one that applies most clearly to the situation posed in the scenario is Guiding Principle C, “Integrity/Honesty”: “Evaluators display honesty and integrity in their own behavior, and attempt to ensure the honesty and integrity of the entire evaluation process.” Section 2 of this principle states, “Before accepting an evaluation assignment, evaluators should disclose any roles or relationships they have that might pose a conflict of interest (or appearance of a conflict) with their role as an evaluator.”
Although a specific conflict of interest is not stated explicitly in the scenario, Riga’s poten- tial for a conflict of interest is readily apparent to the consortium of foundations and may in fact be a primary reason why it published a request for proposals for an external evaluator. Riga’s potential conflict of interest arises from its close association with the initial program and the funders for many years and its current dependence on the reputation and income it has derived from this work. This dependency implies a possible vested interest in finding that the initial program’s outcomes are generalizable to other agencies and contexts.
A tension between two other principles could arise as a result of the conflict of interest and could cause Riga some difficulties as well. They are Guiding Principles A, “Systematic Inquiry,” and D, “Respect for People” (American Evaluation Association, 2004). Section 1 of Guiding Principle A reads, “To ensure the accuracy and credibility of the evaluative informa- tion they produce, evaluators should adhere to the highest technical standards appropriate to the methods they use.” Section 3 of Guiding Principle D states,
Because justified negative or critical conclusions from an evaluation must be explicitly stated, evaluations sometimes produce results that harm client or stakeholder interests. Under this cir- cumstance, evaluators should seek to maximize the benefits and reduce any unnecessary harms that might occur, provided this will not compromise the integrity of the evaluation findings.
The tension between the two principles arises if there is a conflict of interest that puts Riga in the position of choosing between using the highest technical standards for the evaluation and potentially doing harm to one of its long-term clients or to itself. The stakeholders referred to in Guiding Principle D.3 in this case would include both the initial program and the funders. Riga’s evaluation could produce negative findings, thus putting Dr. Porto-Novo in the position of having to decide how much weight to give them. In these circumstances, negative findings could harm the program stakeholders and Riga, but giving insufficient weight to the findings would compromise the technical quality of the evaluation, which in turn would harm the funders. We know that these are possible challenges Riga might face, and they remain potentially negative considerations in making this award to Riga. Although any evaluator might have to weigh negative findings and consequent harm to a stakeholder from systematic inquiry, it is especially problematic in the case of Riga because decisions it makes
Ray / Choosing a Truly External Evaluator 375
about the findings could be attributed to self-interest instead of adherence to the guiding principle of limiting harm.
What Is the Purpose of the Replication?
It seems to me that going back to the reason why the coalition of foundations has decided to replicate the program, evaluate the replications, and hire an external evaluator for this work might help me come closer to making a decision. It appears from the scenario that the fun- ders want to know the extent to which the findings from the initial program are agency and/or context specific or whether they can be generalized to other agencies and contexts. I assume, but do not know for certain, that one of the primary motivations behind the replication pro- ject is scarce foundation resources and the resulting need to support only the most effective programs possible. The replication of the initial program and evaluation of the replications seem to me like a reasonable approach to this task.
Why Do Staff Members From the Initial Program Want to Award the Contract to Riga?
At this point, I begin thinking about the many reasons that initial program staff members might want to award the evaluation of the replication study to Riga. They may have had expe- rience, for example, with how hard it can be to locate an evaluator whose skills, knowledge, approach, and personality are compatible with their programs. They know that Dr. Porto- Nova has these attributes. For another example, they may have previously had a negative experience with an evaluator. Although we do not know this to be true, if it is, it would help explain their desire for Riga to remain the evaluator on the new programs. They appear to trust Dr. Porto-Novo and Riga, and they also seem to value the work she has done for their program in the past. She has in-depth substantive knowledge of the program, and her previous evalua- tion reports have assisted them in retaining funding for the program for about 10 years. Because of her familiarity with the original program, the program staff members may also feel, not unreasonably, that she would be ideal for evaluating the fidelity of its implementa- tion in the replications.
How Can the Decision Regarding the Evaluation Award Best Serve the Purposes of the Coalition of Foundations?
The coalition of foundations that have supported the initial program appears to want to know whether the program works with other agencies and contexts. The funders have always required external evaluators for their funded programs, apparently feeling that someone who is not on the staff of an agency brings added credibility and objectivity that enhances the validity, reliability, and dependability of the evaluation.
Before I make my decision, I decide to make a table portraying the pros and cons of award- ing the contract to Riga or to another evaluator I have discovered during my information gath- ering (Table 1).
The more I think about this scenario and work with the issues it presents, the more con- vinced I am that it is not really an ethical challenge at this stage, when the evaluator is being selected, but rather a professional challenge (Smith, 2002). Dr. Porto-Novo is not faced with, for example, hiding her 10 years as the evaluator of the initial program. That fact is well known. If selected to evaluate the replication programs, however, she might be faced with an ethical challenge or challenges. If, for instance, she finds evidence that might be of potential harm to either of her long-time stakeholders, the agency that ran the initial program, or the
376 American Journal of Evaluation / September 2006
funders and has to decide whether to collect and report this evidence, downplay its impor- tance, or omit it from the final report, she will face an ethical challenge. But at this stage in the selection process, I find nothing that makes this an ethical challenge for the evaluation proposal reviewers, including me.
Nothing in the scenario with which we have been presented suggests that the funders ques- tion the technical quality of the evaluations they have received to date from Riga. The scenario states that the representatives from the funders who sit on the proposal review team want to award the contract to one of the evaluators other than Riga that submitted high-quality propos- als because they feel that Riga has “such a close relationship with the agency that it can no longer be considered external.” Notwithstanding the substantive knowledge and intimate under- standing of the program that Riga would bring to the evaluation of the replication programs, the funders appear to feel that there is an inherent conflict of interest for Riga and that Riga’s long- standing relationship with the program makes it a de facto internal evaluator. This, I think, makes the decision of whom to hire a programmatic one rather than an ethical one.
My Decision
I decide to vote with the representatives from the coalition of foundations to award the con- tract to evaluate the replication program to one of the evaluators other than Riga. My decision is based on my agreement with the coalition of foundation representatives that Riga does not really meet the definition of an external evaluator because the length of its evaluation work with the initial program is so long as to make it appear, if not to be, internal evaluators and because of its apparent potential for facing two different forms of conflicts of interest. The first possible conflict of interest arises because both Riga’s reputation and the bulk of its income rests with finding positive results from the replication study, as well as Riga’s possi- ble need to protect itself from the consequences should there be less than favorable findings from the evaluation of the replication programs. There is a further possibility of a conflict of interest should Riga be placed in the position of needing to address the potential harm to the initial program, one of its long-term clients, caused by negative findings from the evaluation of the replication programs to be in compliance with Guiding Principle D.3.
Table 1 Pros and Cons for Making the Evaluation Award
Riga/Dr. Porto-Novo Another Evaluator
Pros + In-depth, substantive knowledge of the program + High-quality proposal + Known competence and “fit” with agency + Meets relevant sections of the Guiding + High-quality proposal Principles for Evaluators + Biases assumed to be known + Meets the definitions of an external evaluator
+ No known conflicts of interest
Cons – Does not meet the definitions of an external evaluator – Unknown blinders and biases – Potential conflict of interest (i.e., vested interest in – Unknown fit with agency
the success of the replication programs) – Potential for being challenged to choose between
adhering to the highest professional technical standards and the “cause-no-harm” clause
– Potential for “insider” blinders
References
American Evaluation Association. (2004, July). Guiding principles for evaluators. Retrieved December 4, 2005, from http://www.eval.org/Publications/GuidingPrinciples.asp
Smith, N. L. (2002). An analysis of ethical challenges in evaluation. American Journal of Evaluation, 23, 199-206. Substance Abuse and Mental Health Services Administration. (2006). Sample budget and justification (no match
required). Available at http://www.samhsa.gov/grants06/downloads/sample_budget_no_match.doc Webster’s ninth new collegiate dictionary. (1984). Springfield, MA: Merriam-Webster.
Ray / Choosing a Truly External Evaluator 377
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJDFFile false /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams true /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveEPSInfo true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /ACaslon-Ornaments /AGaramond-BoldScaps /AGaramond-Italic /AGaramond-Regular /AGaramond-RomanScaps /AGaramond-Semibold /AGaramond-SemiboldItalic /AGar-Special /AkzidenzGroteskBE-Bold /AkzidenzGroteskBE-BoldIt /AkzidenzGroteskBE-It /AkzidenzGroteskBE-Light /AkzidenzGroteskBE-LightOsF /AkzidenzGroteskBE-Md /AkzidenzGroteskBE-MdIt /AkzidenzGroteskBE-Regular /AkzidenzGroteskBE-Super /AlbertusMT /AlbertusMT-Italic /AlbertusMT-Light /Aldine401BT-BoldA /Aldine401BT-BoldItalicA /Aldine401BT-ItalicA /Aldine401BT-RomanA /Aldine401BTSPL-RomanA /Aldine721BT-Bold /Aldine721BT-BoldItalic /Aldine721BT-Italic /Aldine721BT-Light /Aldine721BT-LightItalic /Aldine721BT-Roman /Aldus-Italic /Aldus-Roman /AlternateGothicNo2BT-Regular /Anna /AntiqueOlive-Bold /AntiqueOlive-Compact /AntiqueOlive-Italic /AntiqueOlive-Roman /Arcadia /Arcadia-A /Arkona-Medium /Arkona-Regular /AssemblyLightSSK /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-Demi /AvantGarde-DemiOblique /BakerSignetBT-Roman /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /BaskervilleBook-Italic /BaskervilleBook-MedItalic /BaskervilleBook-Medium /BaskervilleBook-Regular /BaskervilleBT-Bold /BaskervilleBT-BoldItalic /BaskervilleBT-Italic /BaskervilleBT-Roman /BaskervilleMT /BaskervilleMT-Bold /BaskervilleMT-BoldItalic /BaskervilleMT-Italic /BaskervilleMT-SemiBold /BaskervilleMT-SemiBoldItalic /BaskervilleNo2BT-Bold /BaskervilleNo2BT-BoldItalic /BaskervilleNo2BT-Italic /BaskervilleNo2BT-Roman /Bauhaus-Bold /Bauhaus-Demi /Bauhaus-Heavy /BauhausITCbyBT-Bold /BauhausITCbyBT-Medium /Bauhaus-Light /Bauhaus-Medium /BellCentennial-Address /BellGothic-Black /BellGothic-Bold /Bell-GothicBoldItalicBT /BellGothicBT-Bold /BellGothicBT-Roman /BellGothic-Light /Bembo /Bembo-Bold /Bembo-BoldExpert /Bembo-BoldItalic /Bembo-BoldItalicExpert /Bembo-Expert /Bembo-ExtraBoldItalic /Bembo-Italic /Bembo-ItalicExpert /Bembo-Semibold /Bembo-SemiboldItalic /Berkeley-Black /Berkeley-BlackItalic /Berkeley-Bold /Berkeley-BoldItalic /Berkeley-Book /Berkeley-BookItalic /Berkeley-Italic /Berkeley-Medium /Berling-Bold /Berling-BoldItalic /Berling-Italic /Berling-Roman /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /Bodoni-Poster /Bodoni-PosterCompressed /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /BremenBT-Black /BremenBT-Bold /CaflischScript-Bold /CaflischScript-Regular /Carta /Caslon224ITCbyBT-Bold /Caslon224ITCbyBT-BoldItalic /Caslon224ITCbyBT-Book /Caslon224ITCbyBT-BookItalic /Caslon540BT-Italic /Caslon540BT-Roman /CaslonBT-Bold /CaslonBT-BoldItalic /CaslonTwoTwentyFour-Black /CaslonTwoTwentyFour-BlackIt /CaslonTwoTwentyFour-Bold /CaslonTwoTwentyFour-BoldIt /CaslonTwoTwentyFour-Book /CaslonTwoTwentyFour-BookIt /CaslonTwoTwentyFour-Medium /CaslonTwoTwentyFour-MediumIt /CastleT-Bold /CastleT-Book /Caxton-Bold /Caxton-BoldItalic /Caxton-Book /Caxton-BookItalic /Caxton-Light /Caxton-LightItalic /CelestiaAntiqua-Ornaments /Centennial-BlackItalicOsF /Centennial-BlackOsF /Centennial-BoldItalicOsF /Centennial-BoldOsF /Centennial-ItalicOsF /Centennial-LightItalicOsF /Centennial-LightSC /Centennial-RomanSC /CenturyOldStyle-Bold /CenturyOldStyle-Italic /CenturyOldStyle-Regular /CheltenhamBT-Bold /CheltenhamBT-BoldItalic /CheltenhamBT-Italic /CheltenhamBT-Roman /Christiana-Bold /Christiana-BoldItalic /Christiana-Italic /Christiana-Medium /Christiana-MediumItalic /Christiana-Regular /Christiana-RegularExpert /Christiana-RegularSC /Clarendon /Clarendon-Bold /Clarendon-Light /ClassicalGaramondBT-Bold /ClassicalGaramondBT-BoldItalic /ClassicalGaramondBT-Italic /ClassicalGaramondBT-Roman /CMTI10 /CommonBullets /ConduitITC-Bold /ConduitITC-BoldItalic /ConduitITC-Light /ConduitITC-LightItalic /ConduitITC-Medium /ConduitITC-MediumItalic /CooperBlack /CooperBlack-Italic /CopperplateGothicBT-Bold /CopperplateGothicBT-BoldCond /CopperplateGothicBT-Heavy /CopperplateGothicBT-Roman /CopperplateGothicBT-RomanCond /Copperplate-ThirtyThreeBC /Copperplate-ThirtyTwoBC /Coronet-Regular /Courier /Courier-Bold /Courier-BoldOblique /Courier-Oblique /Critter /CS-Special-font /DextorD /DextorOutD /DidotLH-OrnamentsOne /DidotLH-OrnamentsTwo /DINEngschrift /DINEngschrift-Alternate /DINMittelschrift /DINMittelschrift-Alternate /DINNeuzeitGrotesk-BoldCond /DINNeuzeitGrotesk-Light /Dom-CasItalic /Dom-CasualBT /Ehrhard-Italic /Ehrhard-Regular /EhrhardSemi-Italic /EhrhardtMT /EhrhardtMT-Italic /EhrhardtMT-SemiBold /EhrhardtMT-SemiBoldItalic /EhrharSemi /ElectraLH-Bold /ElectraLH-BoldCursive /ElectraLH-Cursive /ElectraLH-Regular /EnglischeSchT-Bold /EnglischeSchT-Regu /ErasContour /ErasITCbyBT-Bold /ErasITCbyBT-Book /ErasITCbyBT-Demi /ErasITCbyBT-Light /ErasITCbyBT-Medium /ErasITCbyBT-Ultra /EUEX10 /EUFB10 /EUFB5 /EUFB7 /EUFM10 /EUFM5 /EUFM7 /EURB10 /EURB5 /EURB7 /EURM10 /EURM5 /EURM7 /EuropeanPi-Four /EuropeanPi-One /EuropeanPi-Three /EuropeanPi-Two /Eurostile /Eurostile-Bold /Eurostile-BoldExtendedTwo /Eurostile-ExtendedTwo /EUSB10 /EUSB5 /EUSB7 /EUSM10 /EUSM5 /EUSM7 /ExPonto-Regular /Fenice-Bold /Fenice-BoldOblique /FeniceITCbyBT-Bold /FeniceITCbyBT-BoldItalic /FeniceITCbyBT-Regular /FeniceITCbyBT-RegularItalic /Fenice-Light /Fenice-LightOblique /Fenice-Regular /Fenice-RegularOblique /Fenice-Ultra /Fenice-UltraOblique /FlashD-Ligh /Folio-Bold /Folio-BoldCondensed /Folio-ExtraBold /Folio-Light /Folio-Medium /FontanaNDEeOsF /FontanaNDEeOsF-Semibold /FormalScript421BT-Regular /Formata-Bold /Formata-MediumCondensed /FournierMT-Ornaments /FrakturBT-Regular /FranklinGothic-Book /FranklinGothic-BookItal /FranklinGothic-BookOblique /FranklinGothic-Condensed /FranklinGothic-Demi /FranklinGothic-DemiItal /FranklinGothic-DemiOblique /FranklinGothic-Heavy /FranklinGothic-HeavyItal /FranklinGothic-HeavyOblique /FranklinGothic-Medium /FranklinGothic-MediumItal /FranklinGothic-Roman /FrizQuadrataITCbyBT-Bold /FrizQuadrataITCbyBT-Roman /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura /FuturaBlackBT-Regular /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldCondensed /FuturaBT-BoldCondensedItalic /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-ExtraBlack /FuturaBT-ExtraBlackCondensed /FuturaBT-ExtraBlackCondItalic /FuturaBT-ExtraBlackItalic /FuturaBT-Heavy /FuturaBT-HeavyItalic /FuturaBT-Light /FuturaBT-LightCondensed /FuturaBT-LightItalic /FuturaBT-Medium /FuturaBT-MediumCondensed /FuturaBT-MediumItalic /Futura-ExtraBold /Futura-ExtraBoldOblique /Futura-Heavy /Futura-HeavyOblique /Futura-Light /Futura-LightOblique /Futura-Oblique /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Garamond-Antiqua /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-Halbfett /GaramondITCbyBT-Bold /GaramondITCbyBT-BoldCondensed /GaramondITCbyBT-BoldCondItalic /GaramondITCbyBT-BoldItalic /GaramondITCbyBT-BoldNarrow /GaramondITCbyBT-BoldNarrowItal /GaramondITCbyBT-Book /GaramondITCbyBT-BookCondensed /GaramondITCbyBT-BookCondItalic /GaramondITCbyBT-BookItalic /GaramondITCbyBT-Light /GaramondITCbyBT-LightCondensed /GaramondITCbyBT-LightCondItalic /GaramondITCbyBT-LightItalic /GaramondITCbyBT-LightNarrow /GaramondITCbyBT-LightNarrowItal /GaramondITCbyBT-Ultra /GaramondITCbyBT-UltraCondensed /GaramondITCbyBT-UltraCondItalic /GaramondITCbyBT-UltraItalic /Garamond-Kursiv /Garamond-KursivHalbfett /Garamond-LightCondensed /Garamond-LightCondensedItalic /GaramondThree /GaramondThree-Bold /GaramondThree-BoldItalic /GaramondThree-Italic /GaramondThreeSMSspl /GaramondThreespl /GaramondThreeSpl-Bold /GaramondThreeSpl-Italic /GarthGraphic /GarthGraphic-Black /GarthGraphic-Bold /GarthGraphic-BoldCondensed /GarthGraphic-BoldItalic /GarthGraphic-Condensed /GarthGraphic-ExtraBold /GarthGraphic-Italic /Geometric231BT-HeavyC /GeometricSlab712BT-BoldA /GeometricSlab712BT-ExtraBoldA /GeometricSlab712BT-LightA /GeometricSlab712BT-LightItalicA /GeometricSlab712BT-MediumA /GeometricSlab712BT-MediumItalA /Giddyup /Giddyup-Thangs /GillSans /GillSans-Bold /GillSans-BoldCondensed /GillSans-BoldItalic /GillSans-Condensed /GillSans-ExtraBold /GillSans-Italic /GillSans-Light /GillSans-LightItalic /GillSans-UltraBold /GillSans-UltraBoldCondensed /Gill-Special /Giovanni-Bold /Giovanni-BoldItalic /Giovanni-Book /Giovanni-BookItalic /Glypha /Glypha-Bold /Glypha-BoldOblique /Glypha-Oblique /Goudy /Goudy-Bold /Goudy-BoldItalic /Goudy-ExtraBold /Goudy-Italic /GoudyOldStyleBT-Bold /GoudyOldStyleBT-BoldItalic /GoudyOldStyleBT-ExtraBold /GoudyOldStyleBT-Italic /GoudyOldStyleBT-Roman /GoudySans-Bold /GoudySans-BoldItalic /GoudySansITCbyBT-Bold /GoudySansITCbyBT-BoldItalic /GoudySansITCbyBT-Medium /GoudySansITCbyBT-MediumItalic /GoudySans-Medium /GoudySans-MediumItalic /Granjon /Granjon-Bold /Granjon-BoldOsF /Granjon-Italic /Granjon-ItalicOsF /Granjon-SC /GreymantleMVB-Ornaments /Helvetica /Helvetica-Black /Helvetica-BlackOblique /Helvetica-Black-SemiBold /Helvetica-Bold /Helvetica-BoldOblique /Helvetica-Condensed /Helvetica-Condensed-Black /Helvetica-Condensed-BlackObl /Helvetica-Condensed-Bold /Helvetica-Condensed-BoldObl /Helvetica-Condensed-Light /Helvetica-Condensed-LightObl /Helvetica-Condensed-Oblique /Helvetica-Light /Helvetica-LightOblique /Helvetica-Narrow /Helvetica-Narrow-Bold /Helvetica-Narrow-BoldOblique /Helvetica-Narrow-Oblique /HelveticaNeue-BlackCond /HelveticaNeue-BlackCondObl /HelveticaNeue-Bold /HelveticaNeue-BoldCond /HelveticaNeue-BoldCondObl /HelveticaNeue-BoldExt /HelveticaNeue-BoldExtObl /HelveticaNeue-BoldItalic /HelveticaNeue-Condensed /HelveticaNeue-CondensedObl /HelveticaNeue-ExtBlackCond /HelveticaNeue-ExtBlackCondObl /HelveticaNeue-Extended /HelveticaNeue-ExtendedObl /HelveticaNeue-Heavy /HelveticaNeue-HeavyCond /HelveticaNeue-HeavyCondObl /HelveticaNeue-HeavyExt /HelveticaNeue-HeavyExtObl /HelveticaNeue-HeavyItalic /HelveticaNeue-Italic /HelveticaNeue-Light /HelveticaNeue-LightCond /HelveticaNeue-LightCondObl /HelveticaNeue-LightItalic /HelveticaNeueLTStd-Md /HelveticaNeueLTStd-MdIt /HelveticaNeue-Medium /HelveticaNeue-MediumCond /HelveticaNeue-MediumCondObl /HelveticaNeue-MediumExt /HelveticaNeue-MediumExtObl /HelveticaNeue-MediumItalic /HelveticaNeue-Roman /HelveticaNeue-ThinCond /HelveticaNeue-ThinCondObl /HelveticaNeue-UltraLigCond /HelveticaNeue-UltraLigCondObl /Helvetica-Oblique /HelvLight /Humanist521BT-Bold /Humanist521BT-BoldCondensed /Humanist521BT-BoldItalic /Humanist521BT-ExtraBold /Humanist521BT-Italic /Humanist521BT-Light /Humanist521BT-LightItalic /Humanist521BT-Roman /Humanist521BT-RomanCondensed /Humanist521BT-UltraBold /Humanist521BT-XtraBoldCondensed /Humanist777BT-BlackB /Humanist777BT-BlackItalicB /Humanist777BT-BoldB /Humanist777BT-BoldItalicB /Humanist777BT-ItalicB /Humanist777BT-LightB /Humanist777BT-LightItalicB /Humanist777BT-RomanB /ICMEX10 /ICMMI8 /ICMSY8 /ICMTT8 /ILASY8 /ILCMSS8 /ILCMSSB8 /ILCMSSI8 /Imago-Book /Imago-BookItalic /Imago-ExtraBold /Imago-ExtraBoldItalic /Imago-Medium /Imago-MediumItalic /Industria-Inline /Industria-InlineA /Industria-Solid /Industria-SolidA /Insignia /Insignia-A /IPAExtras /IPAHighLow /IPAKiel /IPAKielSeven /IPAsans /JoannaMT /JoannaMT-Bold /JoannaMT-BoldItalic /JoannaMT-Italic /KlangMT /Kuenstler480BT-Black /Kuenstler480BT-Bold /Kuenstler480BT-BoldItalic /Kuenstler480BT-Italic /Kuenstler480BT-Roman /KunstlerschreibschD-Bold /KunstlerschreibschD-Medi /Lapidary333BT-Black /Lapidary333BT-Bold /Lapidary333BT-BoldItalic /Lapidary333BT-Italic /Lapidary333BT-Roman /LASY10 /LASY5 /LASY6 /LASY7 /LASY8 /LASY9 /LASYB10 /LatinMT-Condensed /LCIRCLE10 /LCIRCLEW10 /LCMSS8 /LCMSSB8 /LCMSSI8 /LDecorationPi-One /LDecorationPi-Two /Leawood-Black /Leawood-BlackItalic /Leawood-Bold /Leawood-BoldItalic /Leawood-Book /Leawood-BookItalic /Leawood-Medium /Leawood-MediumItalic /LegacySans-Bold /LegacySans-BoldItalic /LegacySans-Book /LegacySans-BookItalic /LegacySans-Medium /LegacySans-MediumItalic /LegacySans-Ultra /LegacySerif-Bold /LegacySerif-BoldItalic /LegacySerif-Book /LegacySerif-BookItalic /LegacySerif-Medium /LegacySerif-MediumItalic /LegacySerif-Ultra /LetterGothic /LetterGothic-Bold /LetterGothic-BoldSlanted /LetterGothic-Slanted /Life-Bold /Life-Italic /Life-Roman /LINE10 /LINEW10 /Lithos-Black /Lithos-Regular /LOGO10 /LOGO8 /LOGO9 /LOGOBF10 /LOGOSL10 /LOMD-Normal /LubalinGraph-Book /LubalinGraph-BookOblique /LubalinGraph-Demi /LubalinGraph-DemiOblique /LucidaMath-Symbol /LydianBT-Bold /LydianBT-BoldItalic /LydianBT-Italic /LydianBT-Roman /LydianCursiveBT-Regular /Marigold /MathematicalPi-Five /MathematicalPi-Four /MathematicalPi-One /MathematicalPi-Six /MathematicalPi-Three /MathematicalPi-Two /Melior /Melior-Bold /Melior-BoldItalic /Melior-Italic /MercuriusCT-Black /MercuriusCT-BlackItalic /MercuriusCT-Light /MercuriusCT-LightItalic /MercuriusCT-Medium /MercuriusCT-MediumItalic /MercuriusMT-BoldScript /Meridien-Medium /Meridien-MediumItalic /Meridien-Roman /Minion-Black /Minion-Bold /Minion-BoldCondensed /Minion-BoldCondensedItalic /Minion-BoldItalic /Minion-Condensed /Minion-CondensedItalic /MinionExp-Italic /MinionExp-Semibold /MinionExp-SemiboldItalic /Minion-Italic /Minion-Ornaments /Minion-Regular /Minion-Semibold /Minion-SemiboldItalic /MonaLisa-Recut /MSAM10 /MSAM10A /MSAM5 /MSAM6 /MSAM7 /MSAM8 /MSAM9 /MSBM10 /MSBM10A /MSBM5 /MSBM6 /MSBM7 /MSBM8 /MSBM9 /MTEX /MTEXB /MTEXH /MTGU /MTGUB /MTMI /MTMIB /MTMIH /MTMS /MTMSB /MTMUB /MTMUH /MTSY /MTSYB /MTSYH /MTSYN /MusicalSymbols-Normal /Myriad-Bold /Myriad-BoldItalic /Myriad-CnBold /Myriad-CnBoldItalic /Myriad-CnItalic /Myriad-CnSemibold /Myriad-CnSemiboldItalic /Myriad-Condensed /Myriad-Italic /Myriad-Roman /Myriad-Sketch /Myriad-Tilt /NeuzeitS-Book ] /NeverEmbed [ true ] /AntiAliasColorImages false /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox false /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (U.S. Web Coated \050SWOP\051 v2) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown /SyntheticBoldness 1.000000 /Description << /FRA <FEFF004f007000740069006f006e00730020007000650072006d0065007400740061006e007400200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000500044004600200064006f007400e900730020006400270075006e00650020007200e90073006f006c007500740069006f006e002000e9006c0065007600e9006500200070006f0075007200200075006e00650020007100750061006c0069007400e90020006400270069006d007000720065007300730069006f006e00200061006d00e9006c0069006f007200e90065002e00200049006c002000650073007400200070006f0073007300690062006c0065002000640027006f00750076007200690072002000630065007300200064006f00630075006d0065006e007400730020005000440046002000640061006e00730020004100630072006f0062006100740020006500740020005200650061006400650072002c002000760065007200730069006f006e002000200035002e00300020006f007500200075006c007400e9007200690065007500720065002e> /JPN <FEFF3053306e8a2d5b9a306f30019ad889e350cf5ea6753b50cf3092542b308000200050004400460020658766f830924f5c62103059308b3068304d306b4f7f75283057307e30593002537052376642306e753b8cea3092670059279650306b4fdd306430533068304c3067304d307e305930023053306e8a2d5b9a30674f5c62103057305f00200050004400460020658766f8306f0020004100630072006f0062006100740020304a30883073002000520065006100640065007200200035002e003000204ee5964d30678868793a3067304d307e30593002> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e0020005000440046002d0044006f006b0075006d0065006e00740065006e0020006d00690074002000650069006e006500720020006800f60068006500720065006e002000420069006c0064006100750066006c00f600730075006e0067002c00200075006d002000650069006e0065002000760065007200620065007300730065007200740065002000420069006c0064007100750061006c0069007400e400740020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f0062006100740020006f0064006500720020006d00690074002000640065006d002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /PTB <FEFF005500740069006c0069007a006500200065007300740061007300200063006f006e00660069006700750072006100e700f5006500730020007000610072006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006d00200075006d00610020007200650073006f006c007500e700e3006f00200064006500200069006d006100670065006d0020007300750070006500720069006f0072002000700061007200610020006f006200740065007200200075006d00610020007100750061006c0069006400610064006500200064006500200069006d0070007200650073007300e3006f0020006d0065006c0068006f0072002e0020004f007300200064006f00630075006d0065006e0074006f0073002000500044004600200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002c002000520065006100640065007200200035002e0030002000650020007300750070006500720069006f0072002e> /DAN <FEFF004200720075006700200064006900730073006500200069006e0064007300740069006c006c0069006e006700650072002000740069006c0020006100740020006f0070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f8006a006500720065002000620069006c006c00650064006f0070006c00f80073006e0069006e006700200066006f00720020006100740020006600e50020006200650064007200650020007500640073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /NLD <FEFF004700650062007200750069006b002000640065007a006500200069006e007300740065006c006c0069006e00670065006e0020006f006d0020005000440046002d0064006f00630075006d0065006e00740065006e0020007400650020006d0061006b0065006e0020006d00650074002000650065006e00200068006f0067006500720065002000610066006200650065006c00640069006e00670073007200650073006f006c007500740069006500200076006f006f0072002000650065006e0020006200650074006500720065002000610066006400720075006b006b00770061006c00690074006500690074002e0020004400650020005000440046002d0064006f00630075006d0065006e00740065006e0020006b0075006e006e0065006e00200077006f007200640065006e002000670065006f00700065006e00640020006d006500740020004100630072006f00620061007400200065006e002000520065006100640065007200200035002e003000200065006e00200068006f006700650072002e> /ESP <FEFF0055007300650020006500730074006100730020006f007000630069006f006e006500730020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006e0020006d00610079006f00720020007200650073006f006c00750063006900f3006e00200064006500200069006d006100670065006e00200070006100720061002000610075006d0065006e0074006100720020006c0061002000630061006c006900640061006400200061006c00200069006d007000720069006d00690072002e0020004c006f007300200064006f00630075006d0065006e0074006f00730020005000440046002000730065002000700075006500640065006e00200061006200720069007200200063006f006e0020004100630072006f00620061007400200079002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /SUO <FEFF004e00e4006900640065006e002000610073006500740075007300740065006e0020006100760075006c006c006100200076006f0069006400610061006e0020006c0075006f006400610020005000440046002d0061007300690061006b00690072006a006f006a0061002c0020006a006f006900640065006e002000740075006c006f0073007400750073006c00610061007400750020006f006e0020006b006f0072006b006500610020006a00610020006b007500760061006e0020007400610072006b006b007500750073002000730075007500720069002e0020005000440046002d0061007300690061006b00690072006a0061007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f006200610074002d0020006a00610020004100630072006f006200610074002000520065006100640065007200200035002e00300020002d006f0068006a0065006c006d0061006c006c0061002000740061006900200075007500640065006d006d0061006c006c0061002000760065007200730069006f006c006c0061002e> /ITA <FEFF00550073006100720065002000710075006500730074006500200069006d0070006f007300740061007a0069006f006e00690020007000650072002000630072006500610072006500200064006f00630075006d0065006e00740069002000500044004600200063006f006e00200075006e00610020007200690073006f006c0075007a0069006f006e00650020006d0061006700670069006f00720065002000700065007200200075006e00610020007100750061006c0069007400e00020006400690020007300740061006d007000610020006d00690067006c0069006f00720065002e0020004900200064006f00630075006d0065006e00740069002000500044004600200070006f00730073006f006e006f0020006500730073006500720065002000610070006500720074006900200063006f006e0020004100630072006f00620061007400200065002000520065006100640065007200200035002e003000200065002000760065007200730069006f006e006900200073007500630063006500730073006900760065002e> /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f00700070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f80079006500720065002000620069006c00640065006f00700070006c00f80073006e0069006e006700200066006f00720020006200650064007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f0067002000730065006e006500720065002e> /SVE <FEFF0041006e007600e4006e00640020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006e00e40072002000640075002000760069006c006c00200073006b0061007000610020005000440046002d0064006f006b0075006d0065006e00740020006d006500640020006800f6006700720065002000620069006c0064007500700070006c00f60073006e0069006e00670020006f006300680020006400e40072006d006500640020006600e50020006200e400740074007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e0020006b0061006e002000f600700070006e006100730020006d006500640020004100630072006f0062006100740020006f00630068002000520065006100640065007200200035002e003000200065006c006c00650072002000730065006e006100720065002e> /ENU <FEFF005500730065002000740068006500730065002000730065007400740069006e0067007300200066006f00720020006300720065006100740069006e00670020005000440046002000660069006c0065007300200066006f00720020007300750062006d0069007300730069006f006e00200074006f002000540068006500200053006800650072006900640061006e002000500072006500730073002e002000540068006500730065002000730065007400740069006e0067007300200063006f006e006600690067007500720065006400200066006f00720020004100630072006f006200610074002000760036002e0030002000300038002f00300036002f00300033002e> >> >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice
