Discussion post - ethical dilemma
Vogue1988
jgme-d-11-00075.1.pdf
A Primer on the Validity of
Assessment Instruments Gail M. Sullivan, MD, MPH
1. What is reliability?1
Reliability refers to whether an assessment instrument gives
the same results each time it is used in the same setting with
the same type of subjects. Reliability essentially means
consistent or dependable results. Reliability is a part of the
assessment of validity.
2. What is validity?1
Validity in research refers to how accurately a study answers
the study question or the strength of the study conclusions.
For outcome measures such as surveys or tests, validity refers
to the accuracy of measurement. Here validity refers to how
well the assessment tool actually measures the underlying
outcome of interest. Validity is not a property of the tool
itself, but rather of the interpretation or specific purpose of
the assessment tool with particular settings and learners.
Assessment instruments must be both reliable and valid for
study results to be credible. Thus, reliability and validity must be
examined and reported, or references cited, for each assessment
instrument used to measure study outcomes. Examples of
assessments include resident feedback survey, course evaluation,
written test, clinical simulation observer ratings, needs
assessment survey, and teacher evaluation. Using an instrument
with high reliability is not sufficient; other measures of validity
are needed to establish the credibility of your study.
3. How is reliability measured?2–4
Reliability can be estimated in several ways; the method will
depend upon the type of assessment instrument. Sometimes
reliability is referred to as internal validity or internal
structure of the assessment tool.
For internal consistency 2 to 3 questions or items are
created that measure the same concept, and the difference
among the answers is calculated. That is, the correlation
among the answers is measured.
Cronbach alpha is a test of internal consistency and
frequently used to calculate the correlation values among
the answers on your assessment tool.5 Cronbach alpha
calculates correlation among all the variables, in every
combination; a high reliability estimate should be as close to
1 as possible.
For test/retest the test should give the same results each
time, assuming there are no interval changes in what you are
measuring, and they are often measured as correlation, with
Pearson r.
Test/retest is a more conservative estimate of reliability
than Cronbach alpha, but it takes at least 2 administrations
of the tool, whereas Cronbach alpha can be calculated after
a single administration. To perform a test/retest, you must
be able to minimize or eliminate any change (ie, learning) in
the condition you are measuring, between the 2
measurement times. Administer the assessment instrument
at 2 separate times for each subject and calculate the
correlation between the 2 different measurements.
Interrater reliability is used to study the effect of
different raters or observers using the same tool and is
generally estimated by percent agreement, kappa (for binary
outcomes), or Kendall tau.
Another method uses analysis of variance (ANOVA) to
generate a generalizability coefficient, to quantify how
much measurement error can be attributed to each potential
factor, such as different test items, subjects, raters, dates of
administration, and so forth. This model looks at the overall
reliability of the results.6
5. How is the validity of an assessment instrument determined?4,7,8
Validity of assessment instruments requires several sources
of evidence to build the case that the instrument measures
what it is supposed to measure.9,10 Determining validity can
be viewed as constructing an evidence-based argument
regarding how well a tool measures what it is supposed to
do. Evidence can be assembled to support, or not support, a
specific use of the assessment tool. Evidence can be found in
content, response process, relationships to other variables,
and consequences.
Content includes a description of the steps used to
develop the instrument. Provide information such as who
created the instrument (national experts would confer
greater validity than local experts, who in turn would have
more validity than nonexperts) and other steps that support
the instrument has the appropriate content.
Response process includes information about whether
the actions or thoughts of the subjects actually match the
test and also information regarding training for the raters/
observers, instructions for the test-takers, instructions for
scoring, and clarity of these materials.
Relationship to other variables includes correlation of
the new assessment instrument results with other
performance outcomes that would likely be the same. If
Gail M. Sullivan, MD, MPH, is Editor-in-Chief, Journal of Graduate Medical Education.
Corresponding author: Gail M. Sullivan, MD, MPH, Editor-in-Chief, Journal of Graduate Medical Education, 515 N State St, Suite 2000, [email protected]
DOI: 10.4300/JGME-D-11-00075.1
E D I T O R I A L
Journal of Graduate Medical Education, June 2011 119
there is a previously accepted ‘‘gold standard’’ of
measurement, correlate the instrument results to the
subject’s performance on the ‘‘gold standard.’’ In many
cases, no ‘‘gold standard’’ exists and comparison is made to
other assessments that appear reasonable (eg, in-training
examinations, objective structured clinical examinations,
rotation ‘‘grades,’’ similar surveys).
Consequences means that if there are pass/fail or cut-off
performance scores, those grouped in each category tend to
perform the same in other settings. Also, if lower performers
receive additional training and their scores improve, this
would add to the validity of the instrument.
Different types of instruments need an emphasis on
different sources of validity evidence.7 For example, for
observer ratings of resident performance, interrater agreement
may be key, whereas for a survey measuring resident stress,
relationship to other variables may be more important. For a
multiple choice examination, content and consequences may
be essential sources of validity evidence. For high-stakes
assessments (eg, board examinations), substantial evidence to
support the case for validity will be required.9
There are also other types of validity evidence, which
are not discussed here.
6. How can researchers enhance the validity of their assessment instruments? First, do a literature search and use previously developed
outcome measures. If the instrument must be modified for use
with your subjects or setting, modify and describe how, in a
transparent way. Include sufficient detail to allow readers to
understand the potential limitations of this approach.
If no assessment instruments are available, use content
experts to create your own and pilot the instrument prior to
using it in your study. Test reliability and include as many
sources of validity evidence as are possible in your paper.
Discuss the limitations of this approach openly.
7. What are the expectations of JGME editors regarding assessment instruments used in graduate medical education research? JGME editors expect that discussions of the validity of your
assessment tools will be explicitly mentioned in your
manuscript, in the methods section. If you are using a
previously studied tool in the same setting, with the same
subjects, and for the same purpose, citing the reference(s) is
sufficient. Additional discussion about your adaptation is
needed if you (1) have modified previously studied
instruments; (2) are using the instrument for different
settings, subjects, or purposes; or (3) are using different
interpretation or cut-off points. Discuss whether the
changes are likely to affect the reliability or validity of the
instrument.
Researchers who create novel assessment instruments
need to state the development process, reliability measures,
pilot results, and any other information that may lend
credibility to the use of homegrown instruments.
Transparency enhances credibility.
In general, little information can be gleaned from single-
site studies using untested assessment instruments; these
studies are unlikely to be accepted for publication.
8. What are useful resources for reliability and validity of assessment instruments?
The references for this editorial are a good starting
point.
References
1 American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association; 1999.
2 Downing SM. Reliability: on the reproducibility of assessment data. Med Educ. 2004;38(9):1006–1012.
3 Beckman TJ, Ghosh AK, Cook DA, Erwin PJ, Manderkar JN. How reliable are assessments of clinical teaching?: a review of the published instruments. J Gen Intern Med. 2004;19(9):971–977.
4 Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments. Am J Med. 2006;119(2):166e7–166e16.
5 Bland JM, Altman DG. Statistics notes: Cronbach’s alpha. BMJ. 1997;314:572.
6 Brennan RL. Generalizability Theory. New York, NY: Springer-Verlag; 2001. 7 Downing SM. Validity: on the meaningful interpretation of assessment
data. Med Educ. 2003;37(9):830–837. 8 Downing SM, Haldyna TM. Validity threats: overcoming interference
with proposed interpretations of assessment data. Med Educ. 2004;38(3):327–333.
9 Kane M. Validating high-stakes testing programs. Educ Meas Issues Pract. 2002;1:31–41.
10 Kane M. The assessment of professional competence. Eval Health Prof. 1992;15(2):163–182.
E D I T O R I A L
120 Journal of Graduate Medical Education, June 2011
T he following are corrections to the June 2011 issue.
1. Sullivan GM. A primer on the validity of assessment
instruments. J Grad Med Educ. 2011;3(2):119–120.
On p 119, the sentence should read: Cronbach alpha
calculates correlation among all the variables, in every
combination, and generates one number that the closer it is
to 1, the higher the reliability estimate.
2. Salem JK, Jones RR, Sweet DB, Hasan S, Torregosa-
Arcay H, Clough L. Improving care in a resident practice
for patients with diabetes. J Grad Med Educ.
2011;3(2):196–202.
The Figure legends should read:
Figure 1 Description of Sample Selection for Outcomes
Analysis
Figure 2 Timeline for Implementation of Interventions
3. Saeed F, Majeed MH, Kousar N. Easing international
medical graduates entry into us training. J Grad Med Educ.
2011;3(2):269.
The lead author’s name is Fahad Saeed, MD.
4. Sweeney A, Stephany A, Whicker S, Bookman J,
Turner DA. Senior Pediatric Residents as Teachers for an
Innovative Multidisciplinary Mock Code Curriculum.
J Grad Med Educ. 2011;3(2):188–195.
The Figure 3 label for the seventh column is:
Communicating Effectively.
5. Le-Bucklin KT, Hicks R, Wong A. Impact of a
Teaching Rotation on Residents’ Attitudes Toward
Teaching: A 5-Year Study. J Grad Med Educ.
2011;3(2):253–255.
The Results section of the Abstract should read:
Results: Four categories showed significant
improvement, including feeling prepared to teach
(P , .0001), having confidence in their teaching ability
(P , .0001), being aware of their expectations as a teacher
(P , .0001), and feeling that their anxiety about teaching
was at a healthy level (P 5 .0037). There was an increase
in the level of enthusiasm, but the P value did not reach a
significant range (P 5 .12). The level of enthusiasm started
high and was significantly higher on the pretest than every
other tested category (P , .0001).
Footnote c to Table 2 should read: P value as calculated
using the Mann-Whitney U test.
E R R A T A
446 Journal of Graduate Medical Education, September 2011
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 30%) /CalRGBProfile (None) /CalCMYKProfile (U.S. Sheetfed Coated v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.4 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed false /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams false /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments false /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages true /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 600 /ColorImageDepth 8 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /FlateEncode /AutoFilterColorImages false /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.40 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 600 /GrayImageDepth 8 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /FlateEncode /AutoFilterGrayImages false /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.40 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly true /PDFXNoTrimBoxError false /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox false /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (Euroscale Coated v2) /PDFXOutputConditionIdentifier (FOGRA1) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /False /CreateJDFFile false /SyntheticBoldness 1.000000 /Description << /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e002c00200075006d002000650069006e0065006e00200042006500720069006300680074002000fc00620065007200200064006900650020005000440046002f0058002d0033002d004b006f006d007000610074006900620069006c0069007400e4007400200065007200680061006c00740065006e00200075006e00640020005000440046002d0044006f006b0075006d0065006e007400650020006e00750072002000640061006e006e0020007a0075002000650072007300740065006c006c0065006e002c002000770065006e006e0020007300690065002000fc0062006500720020006400690065007300650020004b006f006d007000610074006900620069006c0069007400e400740020007600650072006600fc00670065006e002e0020005000440046002f00580020006900730074002000650069006e0065002000490053004f002d004e006f0072006d0020007a0075006d002000410075007300740061007500730063006800200076006f006e0020006400690067006900740061006c0065006e00200044007200750063006b0076006f0072006c006100670065006e002e0020005700650069007400650072006500200049006e0066006f0072006d006100740069006f006e0065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e0020005000440046002f0058002d0033002d006b006f006d00700061007400690062006c0065006e0020005000440046002d0044006f006b0075006d0065006e00740065006e002000660069006e00640065006e002000530069006500200069006d0020004100630072006f006200610074002d00480061006e00640062007500630068002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f0062006100740020006f0064006500720020006d00690074002000640065006d002000520065006100640065007200200034002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /FRA <FEFF004f007000740069006f006e00730020007000650072006d0065007400740061006e007400200064002700e900760061006c0075006500720020006c006100200063006f006e0066006f0072006d0069007400e9002000e00020006c00610020006e006f0072006d00650020005000440046002f0058002d003300200065007400200064006500200063006f006e0064006900740069006f006e006e006500720020006c0061002000700072006f00640075006300740069006f006e00200064006500200064006f00630075006d0065006e007400730020005000440046002000e000200063006500740074006500200063006f006e0066006f0072006d0069007400e9002e0020005000440046002f0058002000650073007400200075006e00650020006e006f0072006d0065002000490053004f00200064002700e9006300680061006e0067006500200064006500200063006f006e00740065006e00750020006700720061007000680069007100750065002e00200050006f0075007200200065006e0020007300610076006f0069007200200070006c0075007300200073007500720020006c006100200063007200e9006100740069006f006e00200064006500200064006f00630075006d0065006e00740073002000500044004600200063006f006e0066006f0072006d00650073002000e00020005000440046002f0058002d0033002c00200063006f006e00730075006c00740065007a0020006c00650020004700750069006400650020006400650020006c0027007500740069006c0069007300610074006500750072002000640027004100630072006f006200610074002e00200049006c002000650073007400200070006f0073007300690062006c0065002000640027006f00750076007200690072002000630065007300200064006f00630075006d0065006e007400730020005000440046002000640061006e00730020004100630072006f0062006100740020006500740020005200650061006400650072002c002000760065007200730069006f006e00200034002e00300020006f007500200075006c007400e9007200690065007500720065002e> /JPN <FEFF005000440046002f0058002d00336e9662e0306e30ec30dd30fc30c87528304a30883073658766f84f5c62107528306b4f7f75283057307e30593002005000440046002f00580020306f30b030e930d530a330c330af30b3002030f330c630f330c4590963db306b304a3051308b002000490053004f00206a196e96306730593002005000440046002f0058002d003300206e9662e0306e658766f84f5c6210306b306430443066306f0020004100630072006f006200610074002030e630fc30b630ac30a430c9309253c2716730573066304f30603055304430024f5c62103057305f00200050004400460020306f0020004100630072006f0062006100740020304a30883073002000520065006100640065007200200034002e003000204ee5964d30678868793a3067304d307e30593002> /PTB <FEFF005500740069006c0069007a006500200065007300740061007300200063006f006e00660069006700750072006100e700f5006500730020007000610072006100200063007200690061007200200064006f00630075006d0065006e0074006f007300200050004400460020006500200065006d0069007400690072002000720065006c0061007400f300720069006f007300200073006f00620072006500200063006f006e0066006f0072006d0069006400610064006500200063006f006d0020006f0020005000440046002f0058002d0033002e0020005000440046002f0058002000e900200075006d0020007000610064007200e3006f002000640061002000490053004f00200070006100720061002000740072006f0063006100200064006500200063006f006e0074006500fa0064006f00200067007200e1006600690063006f002e002000500061007200610020006f00620074006500720020006d00610069007300200069006e0066006f0072006d006100e700f50065007300200073006f00620072006500200063006f006d006f00200063007200690061007200200064006f00630075006d0065006e0074006f0073002000500044004600200065006d00200063006f006e0066006f0072006d0069006400610064006500200063006f006d0020006f0020005000440046002f0058002d0033002c00200063006f006e00730075006c007400650020006f0020004700750069006100200064006f002000550073007500e100720069006f00200064006f0020004100630072006f006200610074002e0020004f007300200064006f00630075006d0065006e0074006f0073002000500044004600200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002c002000520065006100640065007200200034002e00300020006500200070006f00730074006500720069006f0072002e> /DAN <FEFF004200720075006700200064006900730073006500200069006e0064007300740069006c006c0069006e006700650072002000740069006c00200061007400200072006100700070006f007200740065007200650020006f006d0020006f0076006500720068006f006c00640065006c007300650020006100660020005000440046002f0058002d00330020006f00670020006b0075006e002000700072006f0064007500630065007200650020005000440046002d0064006f006b0075006d0065006e007400650072002c002000680076006900730020006400650020006f0076006500720068006f006c0064006500720020007300740061006e00640061007200640065006e002e0020005000440046002f005800200065007200200065006e002000490053004f002d007300740061006e0064006100720064002000740069006c00200075006400760065006b0073006c0069006e0067002000610066002000670072006100660069006b0069006e00640068006f006c0064002e00200059006400650072006c006900670065007200650020006f0070006c00790073006e0069006e0067006500720020006f006d0020006f007000720065007400740065006c007300650020006100660020005000440046002d0064006f006b0075006d0065006e007400650072002c00200064006500720020006f0076006500720068006f006c0064006500720020005000440046002f0058002d0033002c002000660069006e00640065007200200064007500200069002000620072007500670065007200760065006a006c00650064006e0069006e00670065006e002000740069006c0020004100630072006f006200610074002e0020005000440046002d0064006f006b0075006d0065006e007400650072006e00650020006b0061006e002000e50062006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200034002e00300020006f00670020006e0079006500720065002e> /NLD <FEFF004700650062007200750069006b002000640065007a006500200069006e007300740065006c006c0069006e00670065006e0020006f006d00200064006500200063006f006d007000610074006900620069006c006900740065006900740020006d006500740020005000440046002f0058002d003300200074006500200063006f006e00740072006f006c006500720065006e00200065006e00200061006c006c00650065006e0020005000440046002d0064006f00630075006d0065006e00740065006e002000740065002000700072006f006400750063006500720065006e002000640069006500200063006f006d007000610074006900620065006c0020007a0069006a006e002e0020005000440046002f0058002000690073002000650065006e002000490053004f002d007300740061006e0064006100610072006400200076006f006f00720020006800650074002000750069007400770069007300730065006c0065006e002000760061006e002000670072006100660069007300630068006500200069006e0068006f00750064002e002000520061006100640070006c0065006500670020006400650020006700650062007200750069006b00650072007300680061006e0064006c0065006900640069006e0067002000760061006e0020004100630072006f00620061007400200076006f006f00720020006d00650065007200200069006e0066006f0072006d00610074006900650020006f00760065007200200068006500740020006d0061006b0065006e002000760061006e0020005000440046002d0064006f00630075006d0065006e00740065006e002000640069006500200063006f006d007000610074006900620065006c0020007a0069006a006e0020006d006500740020005000440046002f0058002d0033002e0020004400650020005000440046002d0064006f00630075006d0065006e00740065006e0020006b0075006e006e0065006e00200077006f007200640065006e002000670065006f00700065006e00640020006d006500740020004100630072006f00620061007400200065006e002000520065006100640065007200200034002e003000200065006e00200068006f006700650072002e> /ESP <FEFF0055007300650020006500730074006100730020006f007000630069006f006e00650073002000700061007200610020007200650061006c0069007a0061007200200075006e00200069006e0066006f0072006d006500200073006f0062007200650020006c006100200063006f006d007000610074006900620069006c006900640061006400200063006f006e0020005000440046002f0058002d003300200079002000670065006e006500720061007200200064006f00630075006d0065006e0074006f007300200050004400460020007300f3006c006f00200073006900200073006f006e00200063006f006d00700061007400690062006c00650073002e0020005000440046002f005800200065007300200075006e002000650073007400e1006e006400610072002000490053004f0020007000610072006100200065006c00200069006e00740065007200630061006d00620069006f00200064006500200063006f006e00740065006e00690064006f00200067007200e1006600690063006f002e002000500061007200610020006f006200740065006e006500720020006d00e1007300200069006e0066006f0072006d00610063006900f3006e00200061006300650072006300610020006400650020006300f3006d006f00200063007200650061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006d00700061007400690062006c0065007300200063006f006e0020005000440046002f0058002d0033002c00200063006f006e00730075006c007400650020006c006100200047007500ed0061002000640065006c0020007500730075006100720069006f0020006400650020004100630072006f006200610074002e0020004c006f007300200064006f00630075006d0065006e0074006f00730020005000440046002000730065002000700075006500640065006e00200061006200720069007200200063006f006e0020004100630072006f00620061007400200079002000520065006100640065007200200034002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /SUO <FEFF004e00e4006900640065006e002000610073006500740075007300740065006e0020006100760075006c006c006100200076006f006900740020006d00e400e4007200690074007400e400e40020005000440046002f0058002d0033002d00790068007400650065006e0073006f0070006900760075007500640065006e0020006a00610020006c0075006f00640061002000730065006e0020006d0075006b006100690073006900610020005000440046002d0061007300690061006b00690072006a006f006a0061002e0020005000440046002f00580020006f006e002000490053004f002d007300740061006e006400610072006400690073006f006900740075002000670072006100610066006900730065006e002000730069007300e4006c006c00f6006e0020006500730069007400790073006d0075006f0074006f002e0020004c0069007300e40074006900650074006f006a00610020005000440046002f0058002d0033002d00790068007400650065006e0073006f00700069007600690065006e0020005000440046002d0061007300690061006b00690072006a006f006a0065006e0020006c0075006f006e006e00690073007400610020006f006e002000410064006f006200650020004100630072006f0062006100740020002d006b00e400790074007400f6006f0070007000610061007300730061002e0020005000440046002d0061007300690061006b00690072006a0061007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f006200610074002d0020006a0061002000520065006100640065007200200034002e00300020002d006f0068006a0065006c006d0061006c006c0061002000740061006900200075007500640065006d006d0061006c006c0061002000760065007200730069006f006c006c0061002e> /ITA <FEFF00550073006100720065002000710075006500730074006500200069006d0070006f007300740061007a0069006f006e00690020007000650072002000760065007200690066006900630061007200650020006c006100200063006f006e0066006f0072006d0069007400e0002000610020005000440046002f0058002d003300200065002000630072006500610072006500200064006f00630075006d0065006e00740069002000500044004600200073006f006c006f00200069006e0020006300610073006f00200064006900200063006f006e0066006f0072006d0069007400e0002e0020005000440046002f0058002d0033002000e800200075006e006f0020007300740061006e0064006100720064002000490053004f00200070006500720020006c006f0020007300630061006d00620069006f00200064006900200063006f006e00740065006e00750074006f0020006700720061006600690063006f002e002000500065007200200075006c0074006500720069006f0072006900200069006e0066006f0072006d0061007a0069006f006e0069002000730075006c006c006100200063007200650061007a0069006f006e006500200064006900200064006f00630075006d0065006e00740069002000500044004600200063006f006e0066006f0072006d0069002000610020005000440046002f0058002d0033002c00200063006f006e00730075006c00740061007200650020006c0061002000470075006900640061002000640065006c006c0027007500740065006e007400650020006400690020004100630072006f006200610074002e0020004900200064006f00630075006d0065006e00740069002000500044004600200070006f00730073006f006e006f0020006500730073006500720065002000610070006500720074006900200063006f006e0020004100630072006f00620061007400200065002000520065006100640065007200200034002e003000200065002000760065007200730069006f006e006900200073007500630063006500730073006900760065002e> /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e500200072006100700070006f007200740065007200650020006f006d0020005000440046002f0058002d0033002d006b006f006d007000610074006900620069006c00690074006500740020006f00670020006c0061006700650020005000440046002d0064006f006b0075006d0065006e00740065007200200062006100720065002000680076006900730020006b006f006d007000610074006900620065006c002e0020005000440046002f005800200065007200200065006e002000490053004f002d007300740061006e006400610072006400200066006f00720020006700720061006600690073006b00200069006e006e0068006f006c006400730075007400760065006b0073006c0069006e0067002e00200048007600690073002000640075002000760069006c0020006800610020006d0065007200200069006e0066006f0072006d00610073006a006f006e0020006f006d002000680076006f007200640061006e0020006400750020006f007000700072006500740074006500720020005000440046002f0058002d0033002d006b006f006d00700061007400690062006c00650020005000440046002d0064006f006b0075006d0065006e007400650072002c0020006b0061006e002000640075002000730065002000690020006200720075006b00650072006800e5006e00640062006f006b0065006e00200066006f00720020004100630072006f006200610074002e0020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50070006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200034002e00300020006f0067002000730065006e006500720065002e> /SVE <FEFF0041006e007600e4006e006400200065006e00640061007300740020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006e00e40072002000640075002000760069006c006c00200072006100700070006f007200740065007200610020006f006d0020005000440046002f0058002d0033002d007300740061006e00640061007200640020006f0063006800200073006b0061007000610020005000440046002d0064006f006b0075006d0065006e007400200073006f006d0020006600f6006c006a00650072002000640065006e006e00610020007300740061006e0064006100720064002e0020005000440046002f0058002000e4007200200065006e002000490053004f002d007300740061006e00640061007200640020006600f6007200200075007400620079007400650020006100760020006700720061006600690073006b007400200069006e006e0065006800e5006c006c002e0020004d0065007200200069006e0066006f0072006d006100740069006f006e0020006f006d002000680075007200200064007500200073006b00610070006100720020005000440046002d0064006f006b0075006d0065006e007400200073006f006d0020006600f6006c006a006500720020005000440046002f0058002d003300200068006900740074006100720020006400750020006900200061006e007600e4006e00640061007200680061006e00640062006f006b0065006e0020006600f600720020004100630072006f006200610074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e0020006b0061006e002000f600700070006e006100730020006d006500640020004100630072006f0062006100740020006f00630068002000520065006100640065007200200034002e003000200065006c006c00650072002000730065006e006100720065002e> /ENU (Settings for the Rampage workflow.) >> >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice
