For Eng.Kelvin Only

Chapter 3: e-Selection

Jerard F. Kehoe David N. Dickter Daniel P. Russell Joshua M. Sacco

The purpose of this chapter is to describe issues and guidelines that will help organizations design, implement, and sustain e-enabled employment selection processes and systems. e-Enabled employment selection is only part of the potential scope of e-enabled HR. Indeed, perhaps the greatest promise of eHR is the complete integration of all HR systems and processes based on common HR data and information and on interdependent tools and processes. For example, in the broadest sense, fully e-enabled HR could provide the data-gathering tools, analysis capabilities, and decision support resources for business managers to recruit, hire, pay, promote, terminate, assign, develop, appraise, and reward employees in ways that fully engage them in managing their own outcomes, maximize the contribution of each employee, and directly support the business strategy.

Organization Issues

The manner in which organizations will implement e-enabled selection programs will depend on whether these programs are being implemented as part of a broad e- enablement of HR processes and systems or are being implemented independent of other HR processes and systems. Both are possible. Indeed, it is even possible that specific e-enabled assessment applications may be implemented independent of each other. For example, one large company e-enabled its employment testing processes at the same time that related processes such as internal staffing processes and employee competency management systems were also being e-enabled. Rather than fully integrate all of these processes, the testing processes for external and internal candidates were integrated into a single e-enabled testing process, but the competency assessment processes were not integrated into the e-enabled employment assessment processes largely because these processes were ―owned‖ by different organizations and a full integration would have required a substantial redesign of either the employment or the competency assessment processes, which the organization was not ready to do.

In any case, the scope of the e-enablement will determine the key strategic issues and policy decisions that should be made along the way. The primary purpose of this section is to describe the key strategic and policy issues for organizations that are associated with e-enabled assessment applications. Of course, many of these strategic and policy issues relating to selection also apply more generally to other HR processes and systems.

The strategic issues relating to e-enabled selection relate to the purposes of the e- enablement and the guiding principles that will govern the way the e-enabled selection processes and systems are designed and used.

Purposes

Generally, the fundamental purposes for e-enabling selection processes are the same as for all other HR e-enablement: to minimize cost and to maximize the utilization of the organization’s human capital. e-Enabled selection processes that do not reduce costs or do not improve utilization of the workforce are not likely to be implemented and, if implemented, are not likely to be sustained. In addition to these two fundamental purposes, a third purpose—sustainability—should also be acknowledged and managed.

Cost Reduction

In the design and planning of the e-enabled processes and systems, the organization should build into the applicable database and management tools the capability to measure costs at the transaction level. To manage costs, the organization should view tracking and management systems as part of the e-enabled process. For example, if an organization is implementing an e-enabled employment system that includes 800- number scheduling, web-based testing, and face-to-face interviewing and job offering, part of the effort to design that system should be to define those elements of the end-to- end process that will have identifiable costs associated with them and develop reporting capabilities to routinely aggregate those costs for monitoring purposes. In the case of employment systems, the major cost factors are shown in Exhibit 3.1.

Exhibit 3.1: Cost Drivers for e-Enabled Selection.

 Costs of sourcing candidates into the employment process  Costs of administering the individual screening events  Loss rates between events in the whole process (Key loss factors include the

percentage of candidates who decline job offers, the percentage of candidates who are screened out by test events, and the percentage of candidates who fail to show up for scheduled testing events)

 Administrative costs associated with elements of the process that may not be e- enabled, such as interviewing, offer negotiation, drug testing, background checking, and the like

Maximum Utilization of Human Capital

As with cost, managing the purpose of maximizing the utilization of human capital requires that measures be designed into the e-enabled selection system. This may not be easy. In general two types of measures are needed—one that measures the ―capital‖ represented by each person who goes through the process and another that measures

the extent to which the process results in decisions that maximize the utilization of that ―capital.‖ Consider the example of an e-enabled employment testing process. The measure of ―capital‖ for each person going through the process includes the set of test scores, interview ratings, résumé quality indices, background check results, drug test results, and any other quantifiable evaluation of the skills, experiences, abilities, or other attributes the candidate brings to the organization. Metrics of the extent to which the ―capital‖ is utilized may include a variety of measures, as described in Exhibit 3.2.

Exhibit 3.2: Examples of Measures of Human Capital Utilization.

 Ratio or difference between average test scores for selected candidates to average test scores for all candidates

 Percentage of candidates who satisfy minimum requirements such as for drug tests or background checks

 Percentage of candidates who satisfy employment requirements as a function of recruiting source

 Retention rates, post-hire  Percentage of candidates who exceed agreed-on standards of ―high capability‖

even if that standard is higher than the minimum employment requirements  New hire performance management results

A frequent question in this matter of utilization measurement is whether outcome measures such as turnover rates and supervisor satisfaction with new hires should be included. On the one hand, such measures can be direct manifestations of the extent to which the e-enabled employment process is delivering employees who satisfy organization requirements. On the other hand, such measures are not under the direct control of the e-enabled employment process. Our experience has taught us that, first, such measures are valuable to collect because of their close link to organization needs, and, second, if such measures are collected then it is all the more important to collect the measures of ―capital‖ described above so that the organization may analyze the link between the desired outcomes that the employment process does not control and the measures of ―capital‖ that the employment process does control.

The experience of one company that e-enabled its employment selection process may be instructive. Prior to e-enablement, employment process metrics such as time-to-fill and cost-per-hire measures were routinely gathered and reported to the client organizations as client-oriented measures of the employment transaction management process. In contrast, measures of applicant quality, that is, ―capital,‖ such as ability test scores, interview scores, and the like were not routinely reported to the clients. Rather, the industrial/organizational psychologists responsible for developing and validating the selection procedures regularly monitored applicants’ skill and ability scores and incorporated them into validation documents as needed. In effect, the time and cost of employment testing was viewed as a process to be managed by attending to metrics, but the outcomes of the testing processes—the measures of applicant capital—were not

viewed as a transaction process to be managed by attending to regular metrics. The introduction of vendor-provided e-enabled testing gradually began to change that mindset, in part to provide test outcome as one measure of the accuracy of the vendor’s scoring and reporting processes.

Sustainability

Unlike cost reduction and human capital utilization, sustainability is not, itself, a valued business end. But because sustainability is necessary to ensure the ongoing business results, it is wise to consider sustainability a purpose of the e-enablement design and planning. Sustainability refers to the organization’s willingness and ability not only to maintain the e-enabled system but to progressively evolve the system to satisfy changing requirements and capitalize on improvements in technology and the science of selection. Our experience has been that at least five major elements are important for the progressive sustainability of e-enabled assessment processes (see Exhibit 3.3).

Exhibit 3.3: Key Elements That Enable Sustainability.

 Clear ownership  Funding strategy  Business contribution  User satisfaction  Professional support

Of these five elements, ownership, funding, and contribution are requirements for the sustainability of virtually any business process. Ownership refers to the business owner of the e-enabled selection process. Typically, ownership is defined by budget and authority, although they do not necessarily reside in the same person or role. Funding strategy refers to the understood mechanism by which the budget is funded that pays for the process and system. Business contribution refers to the level of understanding among all business stakeholders of the value of the e-enabled selection strategy to the business. In addition to these three factors, user satisfaction and professional support are uniquely important to sustain e-enabled selection processes. e-Enabled selection generally has three categories of users: the people—external candidates and internal employees—who are assessed by the e-enabled system; the employees who use the system to make business decisions; and the HR specialists who use the system to execute their business function. While user satisfaction is determined in part by ease of use, accessibility, and the capacity to deliver the needed transaction, user satisfaction with selection processes is also frequently a function of the decision outcome for the user, for example, whether the applicant passed the employment screening or not. One implication of this reality is that organizations that are building user satisfaction measures into the e-enabled system should look for opportunities to request user satisfaction feedback before the users are faced with the ultimate outcome of the decisions resulting from their use of the system. In addition, assessment systems can

be designed to incorporate features known to influence user satisfaction such as clear feedback that is relevant to the user’s need, information about how the system works, and the ability to practice or train on the system.

The importance of professional support derives from the fact that selection processes are the domain of professional expertise in human skills and abilities, psychometrics, job performance, and HR systems design and, in employment-affecting applications, are regulated by anti-discrimination legislation that depends on professional standards for a demonstration of job relevance and business necessity. It is this professional expertise that largely determines whether the assessment system will accomplish the second purpose of human capital maximization and whether legal requirements will be satisfied.

Guiding Principles

The organization’s purpose for e-enabled assessment addresses the question of why the organization chooses to support such an HR practice. Guiding principles or policies, on the other hand, reflect how the organization wants to do business in this domain of HR processes. These guiding principles are largely determined by the organization’s overall culture and HR strategy. Also, these guiding principles are much more specific to the selection application than are the purposes. These principles will inform not only the design of the e-enabled system and processes but will also inform the policies that will be established to govern the organization’s behavior.

Each organization that implements e-enabled assessment systems and processes will need to develop guiding principles regarding several issues relating to the management of human resources based on assessment data. In most organizations, these issues include the following:

1. Risk management relating to employment discrimination; 2. Buy versus build and the roles of internal HR IT and third-party providers; 3. HR expertise versus automaticity in the e-enabled assessment process; 4. Whose functions the system will be designed to support; 5. Ownership; 6. The integration of e-enabled assessment with other HR systems and processes;

and 7. ―Free market‖ versus ―regulated‖ processes.

Risk Management

The e-enablement of personnel selection processes is likely to have certain unique implications for managing the risk associated with employment discrimination laws. Prior to e-enablement, most manually delivered selection processes involve the intervention of functional HR specialists, such as employment office recruiters, interviewers and test administrators, and business unit HR staff who would typically have accountability for minimizing risk by monitoring indicators of risk such as group

differences in selection or promotion rates and by administering assessment procedures according to prescribed rules and practices. To the extent that e-enablement of selection and employment allows employees and managers to provide these services for themselves and reduces the roles of HR specialists, the organization will need to determine how the design of the process will continue to minimize risk. Typically, the choice is between increased reliance on process rules and increased training and broader responsibility among users who may not be functional specialists. Increased reliance on process rules may take the form of specific automatic system triggers that are built into the process either to stop or to warn users when a risky situation is evident. For example, an e-enabled employment process could monitor the ratios of majority and minority candidates who are receiving job offers or who are satisfying skill requirements and trigger a warning to a process owner when the ratios indicate an undesirable discrepancy between groups. Similarly, system rules could prevent job offers from being processed until a vacancy had been posted for a minimum amount of time. In contrast, following a ―tools, not rules‖ guiding principle, an organization could design systems that attempt to maximize the discretion and capability of managers to gather and use assessment information with as few rules as possible. In this case, the system could be designed to provide users with risk levels for various possible decisions. For example, an employment system could be designed to provide skill and group membership information about the available group of candidates to inform the manager as much as possible about the sources of risk for any particular hiring decision. Our experience has been that the most effective method for minimizing risk in large complex organizations with highly diffused accountability across units is to build rules and process boundaries into the system and capitalize on automatic rule checking as much as possible. In smaller organizations with strong cultures of individual accountability, such automatic checks may not be necessary if users are adequately trained in managing the risk factors in employment.

An additional concern for e-employment systems is the organization’s legal accountability for electronically sourced résumés. The question is whether persons who merely submit résumés that are ―seen‖ by an employer must be regarded as applicants in a regulatory sense. If yes, then an employer may have an obligation to make a good faith effort to gather and document race, ethnicity, and gender information for potentially tens of thousands of such applicants. This is not a new obligation, but the potential of e- employment to greatly increase the number of applicants may add a considerable administrative burden and may also substantially increase potential class sizes in possible class action lawsuits relating to employment discrimination. As of this chapter’s publication date, there is no firm answer to the question. However, a federal interagency task force consisting of the Department of Justice, the Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP), the Equal Employment Opportunity Commission (EEOC), and the Office of Personnel Management issued on March 4, 2004, proposed new guidance in the Federal Register (69 Fed. Reg. 10152). Essentially, the proposal was that individuals sourced via Internet-based processes should be considered ―applicants‖ if three criteria are met. First, ―the employer has acted to fill a particular position‖; second, ―the individual has followed the employer’s standard procedures for submitting applications‖; and, third, ―the individual has indicated an

interest in the position.‖ While these proposed criteria provide some clarification, they leave unaddressed the issue of whether the individual must also possess the basic qualifications for the position in order to be considered an applicant. For example, must an employer treat an individual with no technical training or experience as an applicant for a technical position if the three suggested criteria are satisfied? Shortly after this task force issued its proposed guidance, the OFCCP, in the Federal Register on March 29, 2004, proposed conforming regulations that are similar but also added the criterion that the individual possess the ―advertised, basic qualifications‖ for the position in question. After these agencies gather comments about their proposed guidance and regulations, they may choose to make modifications. The outcome will significantly impact the design of e-enabled employment processes to ensure compliance with the new guidance and regulations.

Currently, it is our experience that some organizations (using the EEOC’s previous guidance that an applicant is one who expresses an interest in working for the employer) and individuals whose résumés the organization has received are inviting voluntary submission of race/ethnicity and gender information. In the cases we are aware of, the invitation is made either by postcard or email.

Other organizations are operating based on more restrictive applicant definitions that in some cases include the requirements that the job seeker possess the basic qualifications for the job and that the organization is acting to fill vacancies for which the job seeker has expressed an interest. This latter approach typically defines ―basically qualified‖ as surviving some form of résumé-sorting process designed to identify those candidates who have the training, education, work experience, degrees, licenses, certificates, or other documented achievements required by the job. In sum, in early 2004 the jury is out, as it were, on this question.

Buy Versus Build

One of the most difficult strategic decisions an organization may have to make is whether to buy or build and, if buy, how to integrate the roles of the third-party provider(s) with the existing HR IT group. This may be a difficult decision if not primarily for the cost implications, for the role implications. In some ways, the decision to buy is becoming increasingly attractive as vendors are developing more specialized and flexible systems for e-enabled assessment applications as well as for broader e-enabled HR management. For common HR processes, internal HR IT groups are simply less and less able to compete with vendors who develop standardized, yet flexible, tools. As a result, the dynamics of this decision are becoming less about cost and more about roles. At the far end of the spectrum is the increasingly popular outsourcing approach in which the organization not only adopts a vendor’s system but also moves functional specialists from the business to the vendor to provide the support for the business’s use of that system. A hybrid approach is to adopt a vendor’s systems, retain functional specialists in the business, and integrate the ongoing management of the vendor into the role of the specialist. At the other end of the spectrum, the business acquires the

vendor’s system, tailors it to its specific requirements, and absorbs management responsibility for the system.

In the case of e-enabled selection systems, given the growing range of specialized and flexible products now on the market, it is increasingly likely that organizations will choose to buy and to rely on an ongoing relationship with the vendor to support the organization’s use of the system.

Expertise Versus Automaticity

Similar to the buy versus build issue, organizations are also likely to be faced with the fundamental question of how much HR expertise to build into the e-enabled system. In other words, does e-enablement provide an opportunity to reduce the number of people who provide HR expertise by, in effect, building that expertise into the system? This is likely to be a very relevant question for the e-enablement of assessment processes. Employment processes, in particular, tend to be staffed by HR functional specialists, at least in medium to large organizations. To e-enable an entire employment process or even just the selection assessment component raises the prospect of replacing HR specialists with systems. Further, this issue is not only about the replacement of HR specialists who support the delivery of employment transactions such as test administrators, but it is also about the possible replacement of HR generalists who guide employees and managers in deciding how and when to use an organization’s employment process.

For example, if an organization sought to maximize employee self-service by the design of an e-enabled employment system, it is likely the organization would design into the system the information necessary for any employee to use the system to apply for a job, be assessed, post a job vacancy, review available candidates, choose and hire candidates, and initiate the on-boarding process—all with minimal HR expert support. The fundamental decision is what type and level of HR expertise is best exported into e-enabled processes.

From the standpoint of long-term success, the best answer is probably not to move as much expertise into the system as is technically possible. In professional disciplines such as HR, many specialists typically have both expertise and accountability and the decisions, advice, and counsel that represent accountability are necessarily rooted in the specialized expertise. Effective accountability requires expertise. Separating expertise from accountability by building expertise into the system is likely to weaken the ability of remaining HR staff to be effectively accountable. A principle that organizations might consider is that expertise that is not associated with significant accountability is a good candidate for systemization. (For example, in the case of employment selection, HR staff who manually score employment tests frequently do not have significant accountability for the overall effectiveness of the employment process, aside from the correctness of the manually produced test scores.) Expertise that is associated with significant accountability may be a poor candidate for systemization.

Functions That Should Be e-Enabled

The design of any e-enabled assessment system requires that the users of the system be identified. For assessment systems, there are three categories of users, as mentioned above: (a) people, such as job candidates, who will be assessed by the system; (b) HR specialists, such as employment process managers, who will be accountable for the functions performed by the system; and (c) managers, such as hiring supervisors, who will use the system to make personnel decisions.

The interface of each group of users with the system must be specified in advance to define the appropriate system requirements. Generally, this is not simply a description of the interface of these users with the current system. Almost certainly, the e- enablement will change the roles of each user with respect to the system. So this process requires the redesign of user relationships and roles with the e-enabled system. The trend of e-enablement has generally increased the involvement and interaction of the people being assessed and, especially, of the managers who use the system to make decisions. This is the fundamental question of self-service. Whose role is migrated to the now self-serving employee and manager?

Organizations whose HR strategy is to increase the self-service capabilities of non- specialist employees will design systems to facilitate or enable a wider range of functions to be performed by those employees. Organizations whose HR strategy is to retain a significant role for HR specialists in the delivery of assessment processes will design such systems primarily to support lower cost, more efficient HR functionality.

Ownership

This issue of user roles is closely related to the issue of system ownership. To the extent that e-enabled selection systems are designed to facilitate HR-managed processes, the HR organization should be viewed as the owner of the system. In that case, the vendor supporting the system would be accountable to the HR organization for service level agreements negotiated with that HR organization. On the other hand, if organizations are using e-enablement to advance a strategy of employee self-service that reduces the role of HR staff, then alternative ownership and accountabilities should be considered.

In effect, self-service is a business management strategy that moves more of the cost of HR processes directly onto the operational budgets of business units. This happens because the responsibility for executing HR processes becomes a larger part of the work responsibilities of non-HR employees and managers. In this case, an increased role for operational business units in defining and owning service level agreements with both the vendor and the remaining HR organization would be consistent with the business unit’s increased financial responsibility. Of course, even in this case, it remains true that HR expertise is the primary source for the design and operational oversight of the e-enabled process.

This more collective accountability might be operationalized in a variety of ways, including the creation of a joint council of HR and business representatives to periodically review service level metrics and to establish priorities for changes to the system.

Integration with Other Systems

e-Enabled employment selection may function independently of other HR systems and processes or may be integrated to some extent with those other systems. Even if the initial intent is for the selection system to be independently e-enabled, the organization would be wise to consider possible benefits of future integration and to provide for such possibilities in the design of the assessment system. Perhaps the narrowest application of e-enabled selection systems that we have seen has been solely for employment test administration only in specific locations.

But more generalized assessment systems could be designed to gather and use not only employment test data and competency data but also performance data, training and development data, compensation data, job history and progression data, background data, and health data. This level of integration would require organizations to carefully specify requirements for data confidentiality and privacy, data access, and data use. Of course, these are not new requirements.

However, e-enabled systems are likely to require a reconsideration of these data requirements because e-enablement frequently is part of other strategic changes that may give employees more access and ownership of their own data and may invite HR strategists to analyze new combinations of data such as performance and health data. These considerations of systems integration are rooted in the design of the core employee database that would jointly support multiple applications. Certainly, large scale, commercially available HR management systems such as PeopleSoft and SAP represent e-enabled solutions to integrated HR systems and processes.

“Free Market” Versus Regulation

An important but sometimes subtle issue that organizations should address in the design of e-enabled selection systems is the degree of regulation that will be imposed on the system processes and user capabilities. (This point is separate from the related point about legal risk management which, itself, may impose regulations on the systems and users.) Because selection processes govern employee movement, organizational stability may suggest certain regulations.

At one end of the spectrum, systems may be designed to be open, free market systems in which all employees have unfettered access and all managers have unlimited ability to make the personnel decisions most important to their local organizations. Consider, for example, an e-enabled assessment system that serves the purpose of advertising internal vacancies, gathering employee skill data, and enabling managers to evaluate, select, and offer jobs to the most qualified employees. An open,

free market system would place few if any limitations on employees’ access to the posted vacancies or their ability to provide skill information to be considered for the vacancies. Similarly, managers’ access to candidates and their ability to select and offer candidates jobs would not be limited. In this approach, the embedded pressures of supply (of skilled employees) and demand (available positions) would ―govern‖ the operation of the system. The disadvantages of such an open system, however, are that it enables employees to move without regard to the investment of their current organization in their skills, it creates the uncontrolled possibility of understaffed organizations, and it tends to place relatively more of the burden of initial recruiting and staffing on organizations that happen to be dominated by ―feeder pool‖ jobs.

Similar concerns can be seen in another application of assessment systems that is designed to develop, assess, and certify job skills which, when achieved, trigger automatic compensation growth. An open system that does not regulate employees’ access to such development, assessment, and certification resources may result in an organization providing higher levels of compensation to more skilled employees than it can use at any one time.

Our experience has been that organizations typically choose to govern employees’ access to and managers’ use of systems that support personnel movement and compensation. This type of regulation typically takes the form of HR policies relating to time-in-title requirements, non-releasability options, candidate prioritization rules, approval requirements for movement or promotion, ―ownership‖ of employees by ―home‖ organizations, and the like. Generally, the purpose of these regulations is to control employee movement and progression to ensure (a) a fair return on organizations’ investment in the development of employees; (b) a stable number of skilled workers; (c) equal opportunity and access for employees; and (d) a general homeostasis in the organization’s workforce.

Of course, many factors, including an organization’s culture, business strategy, industry, work climate, management-labor relationship, and HR strategy will determine the degree of regulation. But it is also likely to be true that the environment in which e- enablement becomes an attractive business strategy is an environment in which many of those factors will be changing. It is likely that an organization will be dissatisfied with e-enablement regulations that merely reconstruct existing policies and requirements. Organizations are more likely to identify appropriate policies and governance if they view the implementation of e-enabled HR systems as part of a larger change management process that is designed to capitalize fully on the capabilities of such systems to reduce costs and maximize the organization’s use of human capital.

Technology Considerations

Perhaps the most important theme in this section is that the selection system’s technology is merely a tool to support the company’s hiring goals; technology should not be seen as an end unto itself. It is vitally important to recognize that technology simply (one hopes) makes the hiring support function easier and more efficient and effective. It

is misguided to add technology to a selection process simply because competitors are doing it, because one’s previous employer used it, or because it seems ―cool.‖ Technology should be employed because it complements the way the company does business and because it will allow HR to do its job better.

For example, one large, decentralized company contracted to develop a web-based hiring system for entry-level positions. The company was very interested in leveraging all the advantages of a centralized, high-tech solution. However, the initial roll out of the system revealed several issues. Most applicants were walk-ins (rather than recruited via the Internet) and the company’s locations did not have connectivity. In addition, the managers administering the system did not have the computer skills required to operate a sophisticated applicant tracking and assessment system. Initial users were frustrated by low applicant flow as well as the difficulties in establishing connectivity at the locations. Realizing that the success of the project was in jeopardy, the project team decided to quickly introduce a paper-based alternate process to address the needs of the end users.

In hindsight, the company could have saved a great deal of money and user frustration by properly assessing the needs of the field and not developing a web-based driven process.

Choosing the Technology

One key decision point is which technology platform to utilize. Computerized testing started as applications on local PCs and has evolved into complex Internet-based applications. Today, companies have many options available based on their unique implementation requirements.

Local, un-networked PC-based applications are still in use by many organizations with diverse locations. These systems provide a consistent administration and scoring environment. However, maintaining a centralized database of test results is nearly impossible, as are rolling out updates to test scoring, cut scores, and test content. Vendors may be less inclined to offer this option due to the difficulty in monitoring test usage (that is, volume-based test licensing) and ensuring proper administration. These types of applications can be built either by a vendor or by internal IT resources.

The technology may be implemented as a new module within an existing company system (HRIS) or as an intranet application. For example, certain types of HRIS systems can be programmed to administer and/or score selection assessments. Alternatively, a typical web-based assessment tool can be implemented on a company’s intranet. This approach allows for centralized databases of test results and easier roll out of scoring, cut score, and test content changes. These types of systems are usually ―home-grown,‖ as most test vendors are reluctant to support their products when used in these types of implementations. Therefore, these strategies are very dependent on the company’s ability to allocate IT resources to the project as well as their ability to work closely together to ensure the usability of the final product.

The last implementation method is an Internet application administered via an application service provider (ASP) model by the testing vendor. This implementation model is becoming the most common method for implementing e-assessments via computer. It allows the company to completely outsource the implementation, maintenance, and support of the assessment(s). ASP implementation is often the least taxing on internal IT resources and easier to quickly implement in the short term for companies with reliable Internet access from most or all of its locations.

Table 3.1 provides some of the key decision points that influence which implementation method to choose and also indicates where each method falls in terms of each. As can be seen, no single approach is the clear winner. Rather, an organization must decide which consideration(s) are most critical; based on our experience, cost, ease of updating, and demands on internal IT resources are often at the top of the list.

Developing and Implementing the Technology Solution

In developing and implementing the technology solution, the following topics are critical: (a) implementation requirements; (b) cost considerations; (c) end-to-end versus modular solutions; (d) access channels; (e) professional standards; (f) user interface; and (g) data management.

Table 3.1: e-Selection Decision Points.

Technology Alternatives

Considerations PC-Based Intranet (Internal to Company)

Application Service Provider (by Vendor)

Cost Low Moderate High

Centralized Databasing

Low High High

Ease of Updating Difficult Easy* Easy

Support Difficult Moderate Easy

Implementation Timeline

Moderate Moderate* Fast

Integration with Other HR Systems

Difficult Moderate–Easy Moderate–Easy

Demands on Internal IT Resources

Moderate–High High Low

*Entries fluctuate depending on the ability to quickly and easily task IT resources within the organization.

Implementation Requirements

Implementation of any new selection system is complex and the integration of technology makes that implementation more complex. In fact, the technology platform and its features, functions, and reliability can often determine the difference between a successful and an unsuccessful implementation. To the extent that all major stakeholders effectively participated in gathering and refining requirements, implementation can go rather smoothly. Their involvement is clearly an important step in implementing any new selection process.

The main difference between a successful and unsuccessful implementation process is that corporate and local IT resources must be engaged in the process early on. Often the company’s own internal project managers are not aware of all the IT stakeholders who can make or break an implementation. IT resources within most companies are quite scarce. Therefore, it is important to get their buy-in as early as possible and rely on them to assist in identifying all the relevant IT stakeholders. Due to the fact that IT touches almost every part of a company today, the IT organization is often segmented. Including one IT group is often not enough to ensure a smooth implementation of a technology system. For example, the involvement of IT security, HRIS managers, network administrators, local desktop administrators, IT procurement, as well as other groups may be necessary.

Cost Considerations

The PC-based and internal administration models seem to be the least expensive in many instances due to their reliance on internal resources. However, the cost to the organization in terms of work hours could be great, as these resources are essentially creating a system from scratch. Once the system is complete, the ongoing support and maintenance of the system may be relatively inexpensive, depending on needed upgrades, system reliability, and similar features. ASP models are often more expensive, in terms of real cash outlay, due to the level of services provided by the vendor. However, the company must consider the cost and service levels provided by internal IT resources to determine the appropriate solution.

Undoubtedly, testing volumes aside, customization is the main factor that influences the cost of developing and implementing a technology tool to support selection. Numerous considerations can lead to the need for customizations, for example:

 Look and feel—to mirror other corporate applications (or websites), incorporate ―branding,‖ logos, or color schemes;

 Types of applicant data collected—name format, identifying information (name, address, EEO information, Social Security number);

 Organization setup—to match the company’s divisional, regional, or HR administrative support structure and various access levels for local versus corporate administrators;

 Hiring process—if the technology is supporting multiple pro-cess steps, the alignment of the technology with established processes and procedures;

 Customized versus existing assessment content—although the majority of these costs will typically be from non-technology sources, implementing new assessment content requires a great deal of care and attention to detail;

 Reporting—often overlooked when defining requirements is whether existing reports are interpretable, useful, and otherwise appropriate for the proposed selection process;

 Data transfers and systems integration—with other applicant tracking systems and HRIS, including data uploads of scheduled or registered candidates, current employee data, company location data, as well as download of applicant data;

 Platform integration—ensuring that the technology tool will run successfully on the company’s existing IT platforms and hardware; and

 Volume—based on a per-head model or unlimited use license for a set time period.

Other factors that may influence cost are functionality and features, up-time requirements, maintenance and other support requirements, and hosting agreements.

End-to-End Versus Modular Solutions

Companies will consider purchasing an end-to-end solution versus a modular solution, depending on their current processes and existing systems. Considering the relative ease of integrating technology solutions, standards in data transfer protocols, and increased security in data management, companies can create a very workable solution from a number of products and services offered by various vendors. Choosing the best pieces from a number of different sources, the company can create a highly customized solution. However, managing multiple vendors and ensuring proper coordination among them can be very challenging.

In the early years of computerized selection and assessment, modular solutions were most popular. Vendors concentrated on ensuring that their specific assessments and technology implementations were sufficient for clients’ needs. Given that most products today are at least sufficient for most implementations, vendors are now broadening their offerings to give more end-to-end solutions to their clients. Many vendors are creating partnerships with other vendors in the supply chain so that they are able to provide a full-service offering to their current (or potential) clients. Alternatively, other vendors are merging or developing capabilities in other areas to accomplish this goal.

Companies with existing relationships with vendors for different parts of the hiring process may wish to continue those partnerships. For example, one large employer utilizes three different vendors for their hiring process. The company has successfully created a cohesive team among the vendors to develop a seamless process whereby candidates never know that they are being passed back and forth to different systems. One vendor takes in applicants via a telephone application system and passes that information to the applicant tracking vendor. The applicant tracking system passes the

applicant to yet another vendor’s system to administer a web-based assessment. Finally, the candidate is returned to the applicant tracking system (along with results in real-time) to complete the process.

Certainly, client companies are interested in streamlining vendor management and attaining economies of scale in purchasing outsourced services. Therefore, companies are more often seeking one vendor who can deliver a total end-to-end solution (and often other services such as performance management, government HR-related reporting, and benefits and compensation management).

Access Channels

Before the rise of e-selection, applicants would report to the potential employer at a specified date to complete the steps involved in the selection process. Now, with e- selection, applicants can enter a selection system through a variety of means. Table 3.2 describes the standing of the most common e-selection access channels with regard to several key considerations in selection system design and administration. There are five major access channels that we will review here:

 Testing Centers Within Existing Business Locations—Applicant goes to organization’s location and takes assessment. Employees administer assessment(s) and organization provides any required technology.

 Third-Party Testing Centers—Applicant goes to third-party location, where third party proctors assessment and provides any necessary testing technology.

 Interactive Voice Response (IVR)—Applicant calls IVR system, listens to assessment questions, and responds via voice or touch tone to recorded questions.

 Anywhere Access via the Web—Applicant logs onto assessment website from a time and location of his or her choosing. No test administrator.

 Semi-Proctored Environments—Applicant goes to organization’s field location (branch or store) and completes assessment in public area of the location. No proctor, but applicant is visible to employees and/or customers.

As can be seen in Table 3.2, there are a number of considerations, and no access channel is clearly more desirable than the others. Rather, an organization must decide how important each of the key considerations is given their situation and design their e- selection process accordingly.

Table 3.2: Access Channels.

Existing Business Locations

Third- Party Testing Centers

Interactive Voice Response (IVR)

Anywhere Access via the Web

Semi- Proctored Environments

Control Over the Applicant’s Behavior

High Moderate–

Low Low Moderate

Table 3.2: Access Channels.

Existing Business Locations

Third- Party Testing Centers

Interactive Voice Response (IVR)

Anywhere Access via the Web

Semi- Proctored Environments

High

Convenience for the Applicant

Low Moderate High High Moderate– High

Provider Technology

High Low High High High

Infrastructure Requirements

Administration Technology Requirements

High High Low Moderate

High

Content Flexibility High High Low Moderate

High

Control Over the Testing Environment

High Moderate None None Moderate

Logistical Requirements for the Organization

High Moderate Low Low High

Cost Moderate High Low Moderate

High

Potential to Turn Customers into Employees

Low– High*

Low Low High High

*Depends on whether customers typically go to business locations (retailers).

For example, organizations administering cognitive ability test items should strongly consider the degree of control over the applicant’s behavior and testing environment when deciding on an appropriate access channel. This is important because it is possible that applicants would receive help from friends if the test were taken via ―anywhere access‖ over the web. Because cognitive ability items have objectively correct responses, this erodes confidence in the organization’s inferences concerning the applicant’s true standing on the construct of interest. Alternatively, items that do not have an objectively correct response, or those that are less transparent, might be better candidates for anywhere access. This specific concern has already spawned efforts to develop new item types that are more amenable to these unproctored environments (Schmidt, Russell, & Rogg, 2003).

The above example points out some of the complexities involved in choosing an access channel and also suggests a key way in which access channels can be aligned with an organization’s overall selection strategy. In particular, an organization can decide to use different access channels for different steps in the selection process based on how important each consideration is, given the testing procedures under consideration as well as other relevant factors (logistical and technology constraints, applicant volumes). Typically, screening items, where minimum, objectively verifiable qualifications are evaluated (for example, holding a valid driver’s license for a job that requires driving) would be more amenable to anywhere access via the web or interactive voice response due to low concerns about candidate behavior, higher applicant volumes that make the initial technology investment more worthwhile, and a need for a relatively simple type of item content. On the other hand, an Internet-based work sample administered by computer toward the latter stages of a selection process might be more amenable to one of the other three access channels listed because it would be administered to fewer applicants, have restrictive technology requirements, be important to control the applicant’s behavior during the simulation, and be able to pair it with follow-up in-person interviews. As can be seen from these examples, while these emerging access channels provide great flexibility to organizations and applicants, a variety of factors must be considered before their true value can be leveraged.

Some examples of these considerations include:

 A large geographically diverse food service organization desires to do some quick prescreening of applicants prior to face-to-face interviews. HR leadership knows that there is no connectivity in the restaurant and that the applicant pool probably does not have ready access to the Internet. Therefore, they opt for an IVR screening system that prequalifies candidates for an interview. Managers can check to ensure the candidate was, in fact, qualified on the prescreening. The corporate office can monitor usage via reports and data feeds from the vendor.

 Other companies use web-based or IVR systems to prescreen applicants before scheduling them for an onsite testing session.

 Retailers typically have Internet connectivity and room for in-store kiosks. These companies often opt for fully web-based processes administered to walk- in candidates utilizing sophisticated kiosk computers with touch screens in a semi-proctored environment. Often the company offers a parallel process over the Internet for applicants who prefer to apply from home.

 Hiring for blue-collar positions has not typically taken place over the Internet, based on an assumption that these candidates do not have access. However, recently we have not seen Internet access to be a barrier for virtually any applicant pool. The challenge for blue-collar processes is that these candidates typically are not ―walk-ins,‖ but are usually invited in for large testing sessions with scores of candidates in a single session. The challenge for those events is more often the organization’s ability to provide proctored computer facilities to that many applicants at one time.

 Finally, companies that place more emphasis on cognitive skills and technical knowledge (telecommunications, for example) may require assessment in a fully proctored setting, but do not wish to take on the labor and facilities costs for in- house testing. More often, these companies are outsourcing proctored testing to a third-party testing center that may actually administer assessments for many different companies at one location.

Professional Standards

The most critical thing to remember is that e-assessments are subject to the same professional standards as other assessments. Just as it is risky to implement a paper assessment without proper validity evidence, it is risky to implement an e-assessment without proper validity evidence. The Internet has simply made access to these types of assessments easier than ever. However, we liken this to the availability of regulated drugs from so-called discount websites without a physician’s prescription. Use at your own risk!

Consequently, in this section we will focus on the specific standards that are especially pertinent to issues that arise from the new challenges associated with e- assessments, with an emphasis on those found in the Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures (SIOP, 2003) and the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education [AERA, APA, NCME], 1999). The professional standards outlined in these publications are the major standards espoused by industrial/organizational psychologists who are likely to be constructing e-assessments. In addition, we refer readers to the American Psychological Association’s Psychological Testing on the Internet task force report (Naglieri et al., 2004).

In our view, the most critical areas requiring special attention are administration and test security. This is because e-assessments largely examine the same underlying constructs as more traditional assessments (see our earlier discussion on equivalence in the testing channels section). Regarding administration, several sentences of the Standards capture the essence of the challenge facing e-assessments. ―When directions to examinees, testing conditions, and scoring procedures follow the same detailed procedures, the test is said to be standardized [emphasis added]. Without such standardization, the accuracy and comparability of score interpretations would be reduced. For tests designed to assess the examinee’s knowledge, skills, and abilities [KSAs], standardization helps to ensure that all examinees have the same opportunities to demonstrate their competencies. Maintaining test security also helps to ensure that no one has an unfair advantage‖ (AERA, APA, NCME, 1999, p. 61).

Standardization that parallels traditional assessments is truly impossible to achieve with some e-assessment access channels. Specifically, anywhere access via the web and semi-proctored environments, by their very nature, do not offer standardized testing

environments; however, the Standards suggest that standardization is of most concern with regard to KSA testing, as opposed to personality or biodata testing. Hence, practitioners should carefully consider whether KSA testing is appropriate via anywhere access to the web or in semi-proctored environments (we do not see this as a concern with IVR due to its limited content flexibility). This is because it is extremely difficult to control not only the test environment, but more importantly, whether the applicant is receiving inappropriate aid from another person, using prohibited assistance devices (for example, a calculator), or even is, in fact, the person taking the test. Clearly, any one of these concerns could seriously call into question the integrity of the results, and this last issue is squarely addressed by the Principles: ―The identity of all candidates should be confirmed prior to administration‖ (p. 86).

On the other hand, technology can aid the organization’s efforts to standardize a selection process. In very decentralized organizations, individual hiring managers may pick and choose which components of a selection process to utilize. The technology used to administer the process can be configured to require a test score or a score on each interview question before the hire can be processed in the payroll system (or associated HRIS).

Second, test security becomes an issue as well. For instance, the Standards say that ―test users have the responsibility of protecting the security of test materials at all times‖ (p. 64), and the Principles contain similar language: ―Selection procedure items that are widely known or studied in an organization are usually less effective in distinguishing among candidates on relevant constructs‖ (p. 89). Clearly, this becomes virtually impossible to enforce with anywhere testing via the web, and perhaps to a somewhat lesser extent, on-site semi-proctored environments. This is because an applicant can very easily retain copies of all the test materials by taking screenshots, copying items down, or other means. While the organization delivering the test content certainly has an intellectual-property-based incentive to maintain test security, anywhere testing via the web largely requires abandonment of this principle. Clearly, the organization must make an informed decision about whether insecure test content outweighs the convenience of remote web testing (and many organizations have already decided that it does). We think this will give rise to efforts to develop valid, job-related items that are either opaque or clearly objectively verifiable in order to minimize this concern.

Other important, although less critical, areas for additional attention in e-assessments that were not discussed at length here include e-assessments for applicants with disabilities, the potential of excluding large portions of an applicant pool without convenient access to a particular access channel (such as anywhere access over the web), the increased complexity involved in validating the accuracy of scoring procedures, and maintaining security of large e-assessment databases with confidential applicant information. The interested reader should also see the APA’s Internet Task Force report (Naglieri et al., 2004) for an extended discussion of these and other relevant issues.

User Interface

Just like websites and computer applications in general, user interfaces for e- assessments vary widely. This is because there are no commonly accepted standards for the look and feel of e-assessments. Thus, interfaces can vary from simple web pages with text-based questions and radio button style response options to engaging multimedia experiences. More typically, however, the look and feel of e-assessments tend to be in between these two extremes, that is, ―professional‖-looking interfaces that are not designed to be glamorous. This helps keep development costs and bandwidth requirements to a minimum while conveying that the organization takes its assessments seriously.

That said, in our consulting practices, we have seen an increasing interest in developing engaging applicant experiences in organizations’ e-assessments. This is most typically accomplished by developing rich, multimedia assessments that are not only designed to impress applicants, but also to convey a realistic preview of the job at the same time. For instance, such an assessment for a call center job might include realistic voice interactions and a computer interface that closely mimics what a customer service representative might use on the job. Thus, these high-fidelity experiences form two purposes—assessment and applicant attraction or attrition based on the job preview. Although this requires more resources than traditional e- assessments, we believe this trend will continue to increase in the years to come.

Data Management

Traditional assessments require very little technology. Typically, all that is required is a word-processed document that can be administered to applicants. Alternatively, e- assessments have substantial technical requirements that can add a number of complexities to assessment implementation. In this section, we will discuss three key issues: assessment setup, quality assurance/control, and reporting.

Regardless of the interface used (see the user interface section above), e- assessments are almost invariably stored and deployed using relational databases such as Oracle or SQL server. This means that, unlike spreadsheet programs like Microsoft Excel, which store data in a single table, assessment content and examinee data are stored in multiple tables. For instance, in our consulting practices, we have seen relational databases that include up to several dozens of tables for complex work simulations. Clearly, designing relational databases of this size adds significant time to development efforts.

The first key implication of this complexity is that quality assurance becomes critically important, as well as very challenging. This is mainly because, rather than simply reviewing a list of items in a document, someone has to actually test-drive a beta version of the software. This occurs because the staff who created the content is almost never expert in the database systems delivering the test, providing ample room for errors. This not only must be done to ensure the content is 100 percent correct, but also to ensure that the item scoring, scale scoring, and related cutoffs are correct as well. For lengthy assessments, this can require substantial amounts of time. For example,

many dozens of trials are often required to validate scoring systems, and even then, unless all possible scoring combinations are attempted, there will invariably be situations that occur in practice when the assessment is deployed to tens or hundreds of thousands of candidates that have not arisen in testing. One approach to accelerate the testing process is to develop interim scoring ―utilities‖ or ―simulations‖ that expedite the entry of applicant data. Of course, the downside of such utilities is that there can be disconnects between the utilities and the final implemented version of the e-assessment. Clearly, this adds substantial time, expense, and complexity to e-assessment deployments and revisions.

One area in which e-assessments clearly shine is reporting and analysis. Because data are stored in computer systems, it is possible to design real-time reporting and analysis processes to vastly accelerate the decision-making process for individual applicants, as well as overall process flow. Taking this one step further are online business intelligence platforms that allow users to flexibly perform online, real-time analysis of data or to generate custom reports using simple point-and-click tools. For example, applicant data collected via an IVR system may be downloaded for reporting by the company’s HRIS or accessed via a web-based business intelligence reporting system. The accessibility of the data for reporting makes auditing the selection system much easier for HR leadership. Thus, managing the process and ensuring compliance and consistency can be made easier with technology.

The flip side of this is that candidate data might not be stored in a format that is readily available for analysis. Rather, a programmer might need to restructure the data in a flat file, or the data analyst might be required to do this on the desktop. Either approach adds to the time required, potential for error, and complexity of desktop-based analyses.

Finally, data standards are beginning to emerge in HR in general, and these standards will likely begin to govern integration of systems in e-assessment and selection. Specifically, extensible mark-up language (XML) was designated by the HR- XML Consortium and the Object Management Group as the standard for passing data between various HR systems (Weiss, 2001). While these standards will likely be transparent to the end user (as well as to most HR buyers), the result will be increased ease in linking modules from various vendors to create the best e-enabled selection process. For example, the HR director may like the functionality of a certain applicant tracking system, prefer the content of a different vendor’s assessment products, and at the end of the day need to put all the data into her company’s HRIS. Having all systems and vendors ―communicate‖ via XML will reduce the time needed to coordinate integration of the various modules into the total solution. From an industry perspective, ease in systems integration will encourage partnerships among vendors with complementary services.

Tips and Guidelines

As Tippins (2002) discussed, the most valid of selection systems can be invalidated by shoddy implementation. The same is true with e-assessment selection processes.

Technology adds a level of complexity to implementation that can make the whole project go very well or very wrong. In our experience, there are several important steps an organization can take that will ease the design and implementation:

 Involve internal IT resources early.

Describe your plans to them and have them assist in the RFP process if you are planning to use a vendor or assist in design if you are building internally.

Ask internal IT to consult on service levels and support issues. They likely have experience with these agreements with other vendors.

Leverage contacts in IT to determine all the various IT reviews and approvals that may be needed to ensure that the project can proceed (for example, security review, network accesses, web services).

 Remember the real purpose of the hiring process and the needs of the key stakeholders. Do not allow technology to drive the organization’s purpose, strategy, or policy in designing the new selection system.

 Gather input from key stakeholders early and ask for feedback from them often during the design process.

Carefully document your current process and gather requirements on what all stakeholders need from the process and what they would like to change.

Once the final ―dream process‖ is finalized, request feedback from key stakeholders again. This is often best done in a face-to-face meeting so that stakeholders can discuss and negotiate needs and desires directly rather than having to work through a messenger. This process also allows others to assist in solving problems that may seem ―mutually exclusive.‖

Begin the technology design process after all major stakeholders have approved these requirements. The project will more likely stay on time and within budget if the major process decisions have already been made.

 Be realistic and fair about project timelines. As for most projects, expect ―slippage‖ in the timeline on your end as well as the vendor.

Create a carefully planned work plan and timeline at the beginning of the project—and expect it to change almost immediately. Look for ways to make up the time at later steps in the project, but don’t count on being able to make up all the time.

Hold the vendor to the project timeline, within reason. Forgiveness for slight delays by the vendor will go a long way when you need to accelerate other steps due to delays on your end.

When delays are caused by the customer organization, do not expect the vendor to make up all of the time by accelerating their work. The vendor may be able to make up some of the time, but pushing for the system to be launched early may result in errors that can cause the whole process to lose credibility.

 Create a true partnership among the key project team members and other stakeholders. As for most projects, it is important to create a culture of partnership that encourages easy and open flow of information.

 Allow appropriate time to quality check the technology tool and train new users—then add some more.

Avoidable errors that could have been detected and resolved earlier will undermine the acceptance of the new process.

If users are not properly trained, they will not use the new process and tools. Their frustrations with the system will become known and erode support for the project.

 Avoid the temptation to make that one last little change or addition.

As we mentioned earlier, customization is expensive. Customization at the last minute before a big implementation is deadly! Establish a moratorium on changes in the last two to four weeks of development—and stick to it!

If last-minute changes are a must, delay the implementation launch until the change is fully tested and confirmed to be functioning as planned.

Asking these questions and following these tips will allow most organizations to avoid the most common pitfalls in technology implementations. Although following all our advice doesn’t guarantee a perfect project, it will help most organizations avoid the expensive mistakes.

To better illustrate these points, we can provide an example of a very smooth and successful implementation:

Company A is a large decentralized organization that implemented a selection system via the Internet and in-store kiosks. The locations all had high-speed Internet connectivity maintained for other business purposes by the corporate IT department. In addition, the users were already using computer systems in their jobs every day. The kiosks were purchased and specifically designed for the hiring application. In fact, the kiosks were dedicated to the hiring process application. The IT team worked collaboratively with the vendors to configure and test the applications, connectivity, systems interfaces, and anticipated candidate loads. The organization and vendors were open and honest about challenges and issues so that all parties could work together to solve any potential barriers to success. The organization’s IT resources tested the applications in a central lab to identify and rectify major issues. Later, the

applications were tested again at the actual hiring locations prior to training. While the vendors were creating customized systems and content, the organization’s HR team built process maps and scripts for how the new system would be integrated into current workflows. The implementation team from the organization conducted one week of training and orientation at the hiring location on the actual hardware prior to launching the new process. The implementation team stayed on site and monitored live candidate activity daily for the first few weeks of implementation. Daily status calls were held between the end users, the organization’s implementation team, and the vendors to uncover and immediately address any issues. The organization also created a help desk for user questions and problems immediately. All the preparation and care that went into this implementation fostered strong field support, acceptance, and high compliance.

Future Technology Developments

One thing is sure about e-selection technology—expect it to keep changing. The first wave of innovations is likely to be seen (and is already emerging) as an integration of the various systems involved in the selection process, such as integration of prescreening, testing, interviewing, offer, and other selection steps into an overall applicant tracking system. Integration can be accomplished through partnerships, mergers, or an expansion of services by current incumbent firms. Integrated services usually allow for more convenient, faster, and cost-effective service delivery.

Companies will continue to desire to make the application process more convenient and more efficient. Despite some of the complexities discussed earlier in this chapter, unproctored Internet-based testing will become more prevalent, and organizations will want to measure more aspects of performance in less time. These demands will almost certainly prompt innovations in novel item types designed to reduce transparency. These new item types should allow for reliable and valid unproctored Internet testing. In addition, assessment developers will strive to create more engaging experiences for applicants to increase interest and measure constructs in novel ways.

Finally, general computer technology changes will certainly affect the future of e- selection. High-technology companies are already looking for ways to incorporate new technologies, such as mobile computing devices (Palm Pilots

® , Internet-enabled cell

phones). However, these advancements may be further off due to limited screen size and low resolution of these devices. As pure assessment delivery becomes stable and established, organizations will increasingly focus on the applicant experience. They will put a greater focus on multimedia applications, which may make assessment ―fun‖ for candidates (perhaps trying to limit faking by getting candidates to forget they are applying for a job). These new multimedia assessments may begin to mask situational judgment tests as ―reality‖ television shows or computer decision-making games (such as The Sims or SimCity). High-tech simulations of plane cockpits or other vehicle controls could serve as assessments of candidates’ ability to understand directions and problem solve, as well as of reaction time and manual dexterity. Someday soon, we will

even likely see true virtual reality work simulations that include ―day in the life‖ scenarios that are part assessment and part realistic job preview.

As technology begins to drive the development of new assessments, I/O psychologists will be challenged to research the equivalence, fairness, and job relevance of the constructs being measured. In addition, researchers will seek to understand and demonstrate the ways that these assessments can solve existing problems in selection (such as faking and face validity) and solve new issues brought about by the assessments (such as construct validity and fairness).

Designing and Managing e-Selection Processes

Implementing an e-selection process entails most of the decisions involved in a traditional paper-and-pencil program. Thus, the current section highlights many of the elements that would be addressed during the design and implementation of any new assessment system, whether technology-focused or not (also see Tippins, 2002, for an excellent treatment on implementing large-scale selection programs in general). This section also includes a special focus on the improvements e-selection offers over paper-and-pencil assessment.

Organizations planning an e-enabled employment selection system must consider a variety of procedural requirements, including processes to design up-front; vendor selection and project steps; assessment steps and protocols for the test event; feedback to candidates and internal clients; methods of processing candidates after testing; and management of the candidate flow and test program itself. We now turn to the design issues, which are listed in Exhibit 3.4 also.

Exhibit 3.4: Key Issues for Designing e-Selection Processes.

 Design the selection process step-by-step from every user’s point of view: candidate, recruiter, test administrator, hiring manager, information technology manager, HR researcher/administrator.

 Make use of instantaneous and automated information flow to improve each user’s e-selection experience.

 Establish policies and consistent procedures for restricting access to the test and results.

 Protect proprietary and sensitive information as you would for a paper-and- pencil testing program.

Designing the e-Enabled Selection Process

Flow Chart Current Process

A useful first step in understanding the scope of the work is to make a flow chart of the current assessment process for pre-employment and/or employee development. An example of a pre-employment assessment process is shown in Figure 3.1. Various assessment steps, decisions, and outcomes are shown, and the graphic lends itself to an understanding of the e-selection inputs (types of assessments) and outputs (pass/fail results and whether the candidate remains in the applicant pool). Figure 3.1 also illustrates how e-selection might be included in the process.

Figure 3.1: Sample Employment Process Flow Chart.

Flow Chart New Process

Next, the organization should draft the desired process flow that will result from the e- selection implementation(s) under consideration. Process improvements, efficiencies, and cost savings should be evident from the diagram. For instance, some steps in the new process may involve less staff time and may be easy to make, such as changing from a team of proctors who usher in candidates, distribute test materials, and read test instructions to a single monitor who oversees a testing room while also attending to another task. The monitor also needs no inventories of test booklets, answer keys, or other materials. Scoring and processing candidates will be easier than paper-and-pencil processes, and precision and efficiency replace cumbersome and sometimes error- prone manual procedures.

Users

In drafting the new flow, it is important to consider how the various stakeholders and clients will use the system. Gilliland and Cherry (2000) provide a very insightful discussion of the ―customers‖ of selection programs and their differing needs and objectives. Larger organizations will benefit by assembling task teams to represent the interests of various client groups impacted by e-selection. Clients include the HR users

of the system, such as recruiters, administrators, research staff, and the company’s technology experts (for example, HR network administrators). To a lesser extent, it may be beneficial to involve managers from the operating business functions, such as hiring managers and staff trainers and senior-level supervisors who will benefit indirectly from the assessment system. Other clients may be represented in absentia. Examinees are important clients not only because they are users of the system, but because they may be customers and/or shareholders and will form opinions about the organization based on reactions to the assessment process. It may be worthwhile to survey examinees about the usability of the e-selection system soon after it is up and running. Finally, outside organizations such as unions and federal compliance agencies may also be considered clients of the assessment process.

Scoring Systems

The process flow will include implicit assumptions about the methods of scoring assessments and using the results. For example, e-selection should allow scoring to be rapid or instantaneous, test results should be readily combinable with other tests results, and score reports should be available that are readily interpretable and printable for administrative use. These assumptions should be checked. Similarly, the choice of various technology-enabled scoring methods (automated PC-based methods and/or self-scanning of test answer sheets; scanning and scoring by a vendor organization; web-enabled scoring and distribution of results) have variable effects on the availability of results and the timing of next steps in the assessment process. Organizations considering e-selection should also take into account the long-term effects of decisions to retain legacy systems (for example, database management systems that must converse with the new system) and modes of administration (using both paper-pencil and e-selection).

Equivalence

Organizations that are transitioning from existing paper-and-pencil selection procedures to e-enabled versions of the same procedures will have to address the measurement equivalency of the previous paper-and-pencil assessments and the new e-enabled assessment procedures. Even organizations whose e-enabled selection system is novel and not designed to be an alternative form of a previous paper-and- pencil system may have a need to know the measurement equivalence of the e-enabled scores and corresponding paper-and-pencil assessments. Paper-and-pencil alternative assessment procedures may have to be identified for fail-safe reasons—the network crashes—or for manual delivery alternatives that would deliberately not use the e- enabled system, such as mass testing.

Measurement equivalence is usually important for two reasons. First, the degree of equivalence will affect the transferability of any previous validation evidence that may have been accumulated. Second, for practical reasons it is very likely to be important to know the scores from the e-enabled assessments that are administratively equivalent to scores from paper-and-pencil versions of the same (alternative) assessment procedures.

Adequate measurement equivalence is necessary to treat two scores as administratively equivalent. Two scores from two different assessment procedures are administratively equivalent if they produce the same outcomes.

This issue of equivalence applies both to ability tests and to self-report personality and biodata inventories. For ability tests, substantial research has shown that the computerization of paper-and-pencil tests does not change their measurement characteristics except for speeded tests (Mead & Drasgow, 1993). For assessments in which socially desirable responding is an issue, such as personality inventories and biodata inventories, two recent meta-analyses have reached similar overall conclusions that responses to computerized measures do not appear to be any more or less prone to socially desirable responding than responses to paper-and-pencil measures (Dwight & Feigelson, 2000; Richman, Kiesler, Weisband, & Drasgow, 1999). Furthermore, the earlier trend for computer administration to yield less socially desirable responding than paper-and-pencil administration appears to be lessening over time. Later research tends to show smaller differences between computer and paper-and-pencil administration. However, these overall results appear to change when instruments are not administered anonymously, as is the case with employment assessments. In that case, there is some evidence that computer administration may increase socially desirable responding compared to paper-and-pencil administration. In any case, users should not assume that computerized personality assessment will counteract the motivational effect of applying for a job, and users should not assume that computer administration will reduce socially desirable responding compared to paper-and-pencil administration. Overall, the equivalency research results are clearer for ability tests than for personality and biodata inventories.

In most e-enabled selection programs, the organization will have a need to equate the computer administration scores with paper-and-pencil scores from the same tests or highly similar tests. While this equating is most meaningful when the two measures are known to have measurement equivalence, equating that is useful for administrative purposes may also take place, even if measurement equivalence is unknown or not high. Score equating is a process of calibrating the scores on a computer-administered assessment to be administratively equivalent to scores on the paper-and-pencil counterpart. This is a scaling process, not a process for inducing measurement equivalence. Two useful calibration methods are equipercentile equating by which computer and paper-and-pencil scores at the same percentile rank in their corresponding distributions of scores are equated and equiprediction equating by which computer and paper-and-pencil scores that predict the same criterion score are equated. Measurement equivalence aside, equiprediction equating is more meaningful because the equating is based on equal outcomes that are of interest to the organization. In most cases, however, the organization will not have the data available to perform equiprediction equating and will revert to some other equating method such as equipercentile equating.

Administrative Decisions

From a process flow chart, such as shown in Figure 3.1, it is important to identify the decision rules that will be required to evaluate the outcome of each step. Organizations must make choices about how to use the assessment scores. For instance, will test results be expressed as quantitative scores or profile interpretations, with recommendations from a vendor about the ―match‖ of that profile with the vacant position? Will there be pass/fail cut scores, or will decisions be made top-down? Or will results be grouped into bands, such as ―definitely hire,‖ ―possibly hire,‖ and ―definitely reject‖? Might two or more test scores be considered additively, such that a lower score on one part compensates with a higher score on another part (―compensatory‖ scoring)? Or would each step in the assessment process be a requirement that must be met, in sequence, upon conditionally qualifying on the previous step (a ―multiple hurdles‖ approach)? (For further discussion on these topics, see Guion, 1998, and Tippins, 2002.) In general, the more complicated the decision rule, the more beneficial the use of e- selection will be in processing candidates.

Another outcome from the assessment process is the information that should be known about the candidate. The data to be tracked are the same as for paper-and- pencil testing, for example, has the candidate been tested previously, and if so, has sufficient time elapsed that a re-test is reasonable? Has the candidate held the job previously or for other reasons should be grandfathered or exempted? As the size and complexity of the testing program increases, the availability of readily accessible testing databases (for example, a testing and tracking system) becomes increasingly important.

Managing the e-Enabled Process

Preparing to Test: Processes to Arrange in Advance

As shown in Figure 3.1, the first testing step may occur before the organization has had any formal contact with the candidate. That is, the company may wish to prescreen candidates, particularly if there are many applicants for a small number of job openings. Options include websites for entering a résumé and answering questions about desired work, web-delivered assessments such as biodata and work attitudes, telephone-based assessments the candidate takes using the touch-tone keypad, and in-store kiosks with electronic applications. The next step often involves automatically notifying an HR manager or recruiter of the results. For instance, some companies have a system whereby if a candidate qualifies on the self-administered test at an in-store computerized kiosk, an HR representative is automatically summoned for a follow-up interview and any further testing.

It may be worthwhile to prepare candidates in some way for the exam. Possibilities include general information about the organization’s testing program, descriptions of tests required for various jobs the company hires for, or practice examinations applicants can take so that they are familiar with the instructions (as is permitted for college-entry SAT and ACT exams in the United States). Possible reasons for permitting practice include reducing test anxiety and equalizing levels of familiarity with test instructions between re-testers and first-time examinees. Such test information may

or may not be necessary and is a matter of choice for the organization. Many assessment systems provide practice test items, tutorials, or other instructions to take just before the test that will be sufficient for candidates to reduce test anxiety and provide any last-minute preparations that might be required.

When scheduling candidates, there should be an opportunity to self-identify the need for disability accommodation on testing. Ideally, administration instructions during testing should also mention special accommodation as a follow-up—whether read by an instructor or appearing onscreen—so that candidates have ample opportunity to identify their special needs. Technology may be able to accommodate the needs of the examinee with longer test times or verbal instructions read by the computer in lieu of written material.

Some organizations will consider administering assessments without a proctor or method of identifying and observing the candidate (for example, remote testing outside of a controlled environment). For developmental assessments that result in private feedback to the participant and do not qualify/disqualify, label the candidate, or otherwise involve high-stakes outcomes, unproctored or remote assessment may be convenient and cost-effective. However, for selection purposes, knowing the identity of the candidate and ensuring standardized testing conditions may be essential— especially for ability tests. Nevertheless, some organizations do opt to do some unproctored assessments (questions answered at an in-store kiosk) or remote assessments (biodata items administered over the telephone or Internet). Ideally, any remote, unproctored assessments should be followed by assessments in a controlled, supervised setting.

Test/Event Administration

Certainly, test administrators will need to become familiar with the new characteristics and capabilities of e-selection tests, as well as new issues that are created or solved by the use of technology. Test administration issues include usability, test access, feedback, and scoring. The Principles (SIOP, 2003) includes guidelines for administering selection procedures that apply equally well to paper-and-pencil and e- selection.

Usability issues are important when testing both inexperienced and experienced computer users. Some companies that have implemented computer-based or Internet testing have encountered examinees who are unfamiliar with computers and do some surprising and legendary things (for example, stepping on the computer mouse, thinking it is a foot pedal) and others may simply experience some anxiety associated with the computer that interferes with their ability to perform well on the tests. It is important to know the applicant pool in order to determine whether some mouse training or other instruction should be part of the testing system, particularly if the candidates are applying for work with little or no computer experience, and therefore may be expected to be less than proficient.

Computerized testing systems must also be built for the experienced computer user, with test security in mind. That is, the test should not permit examinees to take the test dishonestly (use the calculator when taking an arithmetic test or access the Internet), compromise the test items (cut/paste or print the contents of the screen), or interrupt the test (reboot the computer by pressing a combination of keys). All non-essential functions of the keyboard, mouse, or other controllers should be locked out.

It is also important to pay attention to test timing. First, a caveat about test timing is in order. Not all timed tests operate correctly, particularly if the test is administered over the Internet, where bandwidth can affect the operation of the test clock or timer. When creating a test or choosing a vendor product, it is very important to check the timer and be certain that it does not run slowly/_quickly or continue running out when the test is loading, taking time away from the examinee unfairly. On an encouraging note, computerized tests provide a certain administrative freedom at the test site, particularly if the tests are self-administered at a computer station, such that simultaneous start and end times may not be necessary. The traditional mass-testing event with verbal instructions to ―Put your pencils down‖ may be replaced by quietly ushering candidates in to take their exams and visit the proctor when they are finished.

Note that a good vendor product should not pose many technical difficulties that might require restarting the test. Obvious technical problems such as a loss of power should be infrequent and, even so, should be anticipated, such that the test software stores answers and permits continuity even if power is lost temporarily or the test is interrupted for other reasons.

Managing access to the test is a critical role for the test administrator. With paper-and- pencil testing, test booklets are generally numbered and kept under tight control of a test administrator. Similarly, some vendors restrict access to web-delivered tests by essentially numbering each test with a ―key‖ or identifier so that the test can be taken only once. And well-developed computerized tests will lock out non-test functions that would permit candidates to compromise test items by cutting/pasting or printing text, just as they would prevent other types of cheating described above.

Automated test delivery and immediate scoring can speed the test process. Immediate scoring enables the selection system to combine test events into a single score, providing a more complex picture than has often been possible in real time. Previously, a multiple-hurdles, and often multiple-day, process was required due to the need to wait for manual scoring or interpretation of results. The organization must plan how to use this information. As with paper-and-pencil testing, organizations also have a variety of choices of feedback, such as whether to give candidates detailed information versus qualified/disqualified (or even to provide feedback onscreen after a section is completed; this is useful for developmental assessments and less so for employment tests), and whether to make the proctor available to answer face-to-face questions versus emailing or sending results by mail, et cetera. Feedback should be available in a printable format so that each candidate will have proof of his or her result in exchange for the time invested in testing.

After the Test/Event

Knowing immediately whether or not the candidate has qualified for the next assessment step is very useful in processing candidates. Examinees may move immediately to the next pre-employment step. Recruiters need not wait for results; the assessment system may notify them, such as by sending an email with the afternoon’s candidate results with an interview roster. Alternatively, recruiters may search a database of just-arrived score reports.

Computerized testing produces a database of searchable score results and other test characteristics, such as specific tests delivered, history, and dates of testing. All are useful for various administrative functions, such as generating reports of passing rates and other hiring statistics and for integrating with the company’s HR database. For example, the recruiter will know when the candidate tested and whether it is too soon to retest (per the organization’s policy), and will know whether the candidate’s test records qualify him or her for other positions in the organization. Further, depending on the candidate’s level of performance, the on-file results might qualify him for one or more positions other than the one for which he was testing. As was described earlier, the organization must create decision rules for the use of test records, for example, whether to use the highest score or the most recent score (Tippins, 2002), as well as who will have access to the database and who will be able to modify records.

Managing the Applicant Flow and Test Program

Related to the use of a test database, the organization may manage the application process effectively by tracking the number of candidates available to test, the number who have qualified on one or more steps, expected pass rates, and so forth. This enables recruiters to know when to source more candidates or to stop (for example, if previously qualified candidates who were not offered positions due to a temporary hiring freeze could now receive an offer).

Finally, some computerized test delivery software permits electronic management of accounts (such as ordering and managing access to test inventory online, and using the interface to grant others privileges to view test information online). This feature is especially useful for large testing programs.

The preceding section introduced a variety of important issues that may be considered when designing an e-selection system. Key questions are shown in Exhibit 3.5.

Exhibit 3.5: Some Questions to Ask About e-Selection Systems.

Inventory and Administrative Rights: How will tests be inventoried/ordered/_purchased? Is this process simple/automated? Who has administrative and record-sharing rights? Can these rights be delegated?

Access and Security: How will a candidate gain access to tests—open access versus privilege-based? What will be the provisions for test security and standardization of the test environment? How will the test system guard against tampering?

Ability to Apply Test Policies: Will the test system permit special disability accommodation (for example, extended time on the test)? Will information be stored/applied to determine who should be tested (former incumbents, people who took the test recently and were disqualified)?

Scoring: What scoring options will be available (for example, can individual results be combined into an overall test battery score with a choice of scoring weights and algorithms)? Will it be possible to apply norms to scores automatically?

Results: When/how will test results be available—storage, communication with other HR systems, interpretable and printable score reports? What information will be collected for analyses of EEO results, pass rates, and test functioning (race/gender, identifying information, test date, location, item-level data)? What information will be exported, and in what data format?

Vendor Management and Partnership

Although technology-savvy companies may opt to develop some e-selection tools on their own, few companies would consider implementing an entire e-selection system without the aid of a vendor specializing in these services. Just as the implementation of an e-selection process follows the same general guidelines as an assessment process without technology, managing a vendor of technology follows the same steps as managing a vendor of other services. Steps include defining the scope of work; negotiating a contract and project plan—including stipulating who will be accountable for delivering products and services and what the criteria are for acceptable work—and monitoring timelines and milestones as the project progresses.

Scope of Work

After the organization maps the process flow (as shown in Figure 3.1), the scope of work for the e-selection implementation begins to come into focus. Elements of the scope of work will include a description of the product and service deliverables, with reference to the process flow, as well as anticipated deadlines for completion of each deliverable. Frequently the deadlines are divided into phases, such as a small pilot test and analysis of outcomes, followed by a general rollout and a series of updates. With e- selection implementations, it is important to include ongoing maintenance and client staff training in the scope of work. Rarely does the project end with the first release of the software.

Negotiating the Contract

When seeking to contract services to a vendor, many companies issue a request for proposal (RFP), with either a detailed set of guidelines and expectations or a call for general proposals written at each vendor’s discretion. It is best to be as specific as possible about the work expected from the vendor so that neither party is surprised later by the other’s requirements and ability to deliver. All reasonable proposals will detail timelines, consulting expenses, and any special charges the vendor will recover from the client. Both client and vendor must provide opportunities for fielding questions about the work to be performed and the solutions offered. Companies often provide a deadline for vendor queries and require vendors to present their ideas. Due to the complexity of the services involved and the need for numerous interchanges to learn about the company’s current technology and the vendor’s ability to customize a solution, creating and revising the proposal can be an iterative process for both client and vendor.

While the organization and its vendors share responsibility for mapping the new process onto the existing one, some areas of responsibility are more clearly assignable to one party or another. The client organization is responsible for proper use of the assessment system, such as meeting the hardware, software, and network requirements; ensuring standardized test conditions; and using assessment products consistently and fairly. The vendor is responsible for quality assurance on the test delivery, scoring accuracy, and delivery of score data and reports. Such work should not be placed on the client organization’s shoulders.

Service Level Agreements

Performance standards for the vendor are defined in specific service level agreements (SLAs) between e-selection vendors and their clients. These ―agreements‖ are exhibits or attachments within the actual contract or statement of work and include performance objectives or goals. It is very important for vendors and clients to clearly articulate needs and services as well as agree on objective measures to evaluate performance.

Some common SLAs are listed below:

 Availability/uptime is defined as number of outages, duration of each, and total outage time for some predefined period (usually one month). When developing this SLA, it is important for the customer to determine when the system will be used and require acceptable uptime within those hours. It will be inefficient to require the vendor to agree to uptime outside of normal hours of use (and therefore more expen-sive for the customer).

 Scheduled downtime for maintenance is another SLA that will be negotiated between the vendor and customer organization. Both companies will typically have standard ―maintenance windows‖ that fall during off or low usage hours and are easily coordinated.

 System performance metrics are as unique as the functionality of the particular system under consideration. For example, SLAs may be set to govern scoring completion time and transaction response time (the interval from when the user requests a page to when he or she receives the last byte of the required page).

 Technical support should be a part of any standard set of services provided. However, there is a wide range of services that may or may not be included here. For example, support hours should reflect the hours of use; contact with the vendor’s help desk may be open to all users or filtered through a central communication team within the customer’s organization; and vendors may limit the number of support hours within a specified period (typically a year). In addition, customers should work with vendors to define problem routing processes, resolution times, follow-up reports (for example, root cause analysis), and escalation procedures.

Managing Milestones

Although it is obvious that timelines for implementation should be realistic, organizations and their vendors too often share a tendency toward wishful thinking—the organization’s frequent unfamiliarity with the technical issues involved in implementing the program, coupled with the vendor’s desire to win the business, sometimes result in failing to meet ambitious deadlines. As early as possible in the discussions about the scope of work, organizations and consulting companies should review timelines for reasonableness and discuss what will be done if certain deadlines are not met, and how to keep the project on track. Later, these discussions can be built into the contract, including consequences of failure to deliver on schedule.

During the development and implementation of the assessment system, the client will have responsibilities and accountabilities as well as the vendor. For instance, various portions of the assessment may require client input as subject-matter experts (feedback as users of the new testing system or on HR processes). SME cooperation is essential to meeting key milestones.

Ongoing Maintenance

When the software, hardware, and technology infrastructure are complete, the organization and its vendors will be required to collaborate further to see that training is effective and that ongoing maintenance concerns are met (for example, software updates to improve usability, document procedures, or provide additional services). Note that once the assessment system is up and running, working with the vendor should be expected to involve some loss of control for the client organization, such as the inability to modify the assessments without assistance from an outside software expert. The client organization must accept this delegation of work, as must any company that outsources services and expertise.

Finally, when designing and managing an e-selection process, consider the tips and guidelines shown in Exhibit 3.6.

Exhibit 3.6: Tips and Guidelines for Designing and Managing e-Selection Processes.

1. Know your own selection process and have an idea of your desired state before you shop for a vendor. The e-selection end state must be of value to your stakeholders; it will be difficult to defend the implementation against later criticism if the reasons for implementation decisions are not clear or appear to be more convenient for the vendor than for your company.

2. Research the vendor’s products as many ways as possible. Beyond speaking to references and trying out demo versions offered by the vendor, ask to see e- selection systems currently running on other client sites. Ask an existing client for an ―inside tour‖ of a working e-selection system. Try out the sites to get the candidate’s experience, examine the online application, try out the in-store kiosk, or dial into the IVR application system.

3. Use the data to show the value of e-selection. Virtual storage allows the collection of vast amounts of data, not only about the candidates and hiring process, but about individual and group- or organization-level performance results.

4. Know that the implementation is never finalized or immutable. The vendor will have periodic upgrades and improvements, both vendor-initiated and as a result of your comments and suggestions, and ongoing management, training, and technical support is integral to your continued success.

References

American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Dwight, S. A., & Feigelson, M. E. (2000). A quantitative review of the effect of computerized testing on the measurement of social desirability. Educational and Psychological Measurement, 60, 340–360.

Gilliland, S. W., & Cherry, B. (2000). Managing ―customers‖ of selection processes. In J. Kehoe (Ed.), Managing selection in changing organizations: Human resource strategies (pp. 158–196). San Francisco: Jossey-Bass.

Guion, R. M. (1998). Assessment, measurement, and prediction for personnel decision. Mahwah, NJ: Lawrence Erlbaum.

Mead, A. D., & Drasgow, F. (1993). Equivalence of computerized and paper-and- pencil cognitive ability tests: A meta-analysis. Psychological Bulletin, 114(3), 449–458.

Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., & Velasquez, R. (2004). Psychological testing on the internet: New problems, old issues. American Psychologist, 59, 150-162.

Richman, W. L., Kiesler, S., Weisband, S., & Drasgow, F., (1999). A meta-analytic study of social desirability distortion in computer-administered questionnaires, traditional questionnaires, and interviews. Journal of Applied Psychology, 84, 754–775.

Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures (4th ed.). College Park, MD: Author.

Schmidt, D. B., Russell, D. P., & Rogg, K. R. (2003, April). Reducing call center associate turnover through effective and focused selection procedures. Paper presented at the Society for Industrial and Organizational Psychology 18th Annual Conference, Orlando, Florida.

Tippins, N. (2002). Issues in implementing large-scale selection programs. In J. W. Hedge and E. D. Pulakos (Eds.), Implementing organizational interventions: Steps, processes, and best practices. San Francisco: Jossey-Bass.

Weiss, J. R. (2001). Six things you should know about XML. The Industrial- Organizational Psychologist, 39(2), 30–34.