Information Systems

Page 1 of 7

Operation Enterprise IT: A Hybrid Consolidation

Project Initiation: March 2016 Project Completion: June 2017

Prepared for NASCIO, State IT Recognition Awards: “Enterprise IT Management Initiatives” Submitted by: Ed Toner CIO, State of Nebraska

501 S 14th Street Lincoln, NE 68509-5045 Phone: 402-471-2761 Email: [email protected]

On

Page 2 of 7

Executive Summary In recent history, Nebraska has been commonly referenced among the handful of states making up the Midwest Silicone Prairie. Business in the state, has long seen competitive growth in the technology sector, and the industry continues to bring jobs along with a highly talented applicant pool to the state. Government tech at the State of Nebraska makes it their mission to stay ahead of the needs of their tech-savvy citizens by innovating more effective, more efficient, customer focused solutions. In 2016, the state achieved this trifecta when the Office of the CIO (OCIO) led a sixteen-month consolidation effort to centralize the enterprise data center and established a single network to support the enterprise’s infrastructure, and establish standardized desktop support for the agencies. In July 2015, when he was appointed by Governor Pete Ricketts as Nebraska’s CIO, Ed Toner approached the State basing these decisions on his 20-year professional portfolio in the private sector. This portfolio already included consolidating key information technology systems, resources and services through business mergers. Toner applied his experience to form a unique plan for the State’s IT consolidation. His plan included the establishment of best practice standards for State IT. Consolidation became synonymous with Nebraska’s core ITIL practices and programs, such as the single service desk solution, IT service catalog, change management process, enhanced information security and operations center, and of course, IT and project governance. The Nebraska model for consolidation is perhaps most unique for its rapidity to implement a roadmap provided by the CIO and his team. An established, clear timeline and scope of the State’s network and server consolidation remained consistent throughout the project. Also indicative of the Nebraska model was the OCIO’s centralized-hybrid focus on consolidation. Enterprise functions were centralized while line of business applications were left untouched, to be managed by the Agency. Toner took a progressive approach to marketing his model, and advocated his roadmap amongst the Agency leaders across the State. The CIO’s initiative received gubernatorial support, and his team rallied OCIO customers around the idea of creating opportunity through more efficient, effective business processes with a customer focused model of providing more quality IT support. The customer agencies were involved in every step of the project with no additional cost to the state’s taxpayers. In 2016, data center and network consolidation largely revolved around the elimination of redundant servers, and the tools and resources that were needed to maintain them. By December 2017, the OCIO had assisted the agencies in over 206 physical to virtual machine migrations in addition to elimination of 133 physical servers from disparate data closets and individual agencies across the state. This work positioned the State of Nebraska to realize a cost savings of $9.9 million in the first twelve months (March 2017), and an annual savings in the first two years of $15.4 million (December 2017). Exemplar

Page 3 of 7

Prior to Toner’s arrival at the State, Nebraska had tried and failed to initiate consolidation efforts. Under Governor Rickett’s support, and with Toner’s leadership, consolidation was successfully completed without incident. The support from State leadership was one factor that impacted consolidation success. Another successful step to the model was the OCIO’s consistent practice of discovery meetings within the agencies leading up to consolidation to see how the agencies were previously supported. The opportunity for collaboration came as a result of these, and the OCIO learned the importance of communication to the agencies about the project. Key messages remained focused on a means to reduce IT expenditures and increase IT support. The OCIO used consistent communication to socialize consolidation messages to the agencies. These messages were shared through the CIO Blog, IT Consolidation FAQ’s, and updates to the Roadmap. The OCIO even used proven metrics and best practices to circumvent the potential for crippling political opposition. Previously mentioned, Nebraska had established core ITIL practices and programs at the State prior to March 2016, this created a foundation of customer service at the base of consolidation. With these practices in place, OCIO leadership could begin to measure such things as service availability, time to resolve support requests, rate of successful changes, and the total savings of IT expenditures for the enterprise. Any opposition could be met with a measured metric. This would not have been possible without the adoption of a single service desk solution, and yet this was done at no expense to state taxpayers. No cost was added to for the ITSM tool because the State already owned it, prior to consolidation the resource had been underutilized. While the IT Service Manager (ITSM) tool chosen by the OCIO provided the customers increased visibility to IT processes, the solution was put in place under the management of the OCIO’s Project Management Office (PMO). PMO served another role to eliminate data the silos that previously existed between State organizations. Establishing a PMO at the start of consolidation presented the CIO with more project governance. With CIO support, the Project Managers (PM) began working with the agency IT managers to migrate their servers into the enterprise data center. PMs were also responsible to bring network team and agency teams together with the common goal of adding all of the different agencies onto the enterprise network Concept Prior to 2016, the majority of the State agencies managed IT functions as independent departments within individual organizations. This decentralized approach fostered the duplication of data and allowed disparate servers to take up valuable real estate inside of data closets in agencies across the enterprise. The decentralized status gave the agencies limited vision and provided no operational control. Technology came at a higher expense while the model itself failed to optimize resources across the state. By outlining a hybrid centralization model, CIO, Ed Toner, pinpointed the collaborative approach to management between his team and IT management within the different agencies. The OCIO’s vision received gubernatorial support, and in March 2016, the OCIO began efforts to centralize the enterprise infrastructure, optimizing support on a single network for the State’s

Page 4 of 7

agencies, boards and commissions. The Phase One initiative began the long-term project of consolidation. The goal of the larger project was stated to bring greater value and efficiency to the IT services provided by the State of Nebraska. The project has three distinct phases each dependent upon the success of the previous phase and very similar in the approach.

o Phase One – Initiated March 2016 Network consolidation of resources and tools

o Phase Two – Initiated September 2016

Server/Data Center consolidation of resources and tools o Phase Three – Initiated January 2017

Desktop Support consolidation of resources and tools The data that the OCIO collected from phase one, prior to March 2016, were incorporated into plans for subsequent phases. The success or failure of the project depended on the cooperation of all agencies, so the discovery portion of the project was essential to establishing the central role played by the OCIO. Affected agencies included all eighteen of the cabinet agencies. Issues with any one Agency could have negatively impacted the productivity of the consolidation project team and subsequent phases. Relying on the IT staff in each of the affected agencies to perform much of the work added depth to support this project. Consolidated employees began moving to work alongside their counterparts in the becoming central IT agency. Because of this collaboration the project cost the OCIO only 3,000 working hours. Once they began working in the same office, network coordinators and support analysts were given the ability to hold standup meetings each morning, share data, and leverage skillsets to identify preventative measures for potential events during the course of the project. Data collected during each phase assisted the OCIO staff in determining the state’s projected savings, forecasting future performance benefits. Phase Two of consolidation presented a majority of the cost savings to taxpayers. This phase focused on greater efficiency of IT operations, greater customer service and higher levels of reliability for the agencies. The Nebraska IT Commission (NITC) standards provided accessibility standards and security guidelines which the data center and network consolidation team followed. During consolidation the NITC was able to enhance its guidelines which furthered the state’s ability to benefit from the project. The OCIO set in motion an ambitious and rapid plan for consolidation, but the centralized tech agency wanted to have two active data centers to effectively eliminate the State’s Disaster Recovery site. The OCIO built a strategy for consolidating the infrastructure on an all-flash solution with a NetApp MetroCluster array spanning the 50 miles from Lincoln to Omaha Data Centers. This structure enabled instantaneous failover and disaster recovery. Since installing the NetApp all-flash solution, the Nebraska CIO’s office virtualized nearly 90% of its servers and eliminated an additional 100 physical servers. The OCIO also architected the mainframe environment to move seamlessly between the two data

Page 5 of 7

centers by replicating transaction at both data center locations. Critical applications such as those hosted on the State’s Mainframe were required to be accessible at all times for the State to successfully serve its citizens efficiently. The State of Nebraska addressed this by implementing a design that allows its mainframe to move from the primary site in Lincoln to the secondary site in Omaha in case of a failure at either site. The State decidedly made certain that critical applications would have the ability to failover to a secondary location. This attention to technical detail played a significant role to protect State business from data loss. The solution enabled the State to begin to provide real-time replication of all data in a separate recovery site with point-in- time recovery. This has allowed the tech agency to restore earlier versions of customers’ data. Significance An innovative “Hybrid” Centralization model was put in place by the State of Nebraska’s OCIO to allow a centralized team to perform enterprise functions with a common vision. The IT personnel from the agencies, who were brought over to the OCIO with consolidation, joined a team with added depth in support to perform more IT functions and support the migration to the State’s network and servers for each of their former agencies. The enterprise functions achieved during the initial three phases of consolidation include:

o The State’s enterprise data and operations center o A consolidated, single state network o Two completely redundant data centers with failover ability o Procurement reviews and IT project governance o Enterprise Service Desk (Site) Support

Critical components for the completion of this project include executive support from the Governor and oversight from the CIO, a solid ITSM framework and transparency. The Hybrid model avoided unnecessary conflict and indicated agency management would maintain authority over agency-specific activities and functions. This was important to the OCIO’s collaboration efforts and allowed the project to remain in scope. Impact Standardized Tech Support via Standardized IT hardware will result in the ease of hardware management by IT staff, reduce travel throughout the State to support the IT hardware, and create depth of support for future advancement and opportunities for the IT staff. Faster Processes and Automation As the OCIO begins to automate processes such as password reset, morning health reports, OST billing process that included SAN/Backups, ShareFile, Cloud, Fax, Managed Domain, Secure File Transfer Protocol Service, VPN service and County services Support staff are able to focus more time on automating other processes. Automated paper-based process that previously took weeks now happen in real time. With the maturity of these processes, consolidation will lead to more future IT innovation for the State. IT Governance As a result of consolidation, the OCIO gained visibility like never before into

Page 6 of 7

the different agencies, and the IT work being done there. With the help of Project Management office the State CIO has gained an ability to make well-informed decisions for future enterprise IT projects. Reduced IT Cost will continue to compound year over year, as proven within the first two years of the larger project. 2016 consolidation effort achieved an annual cost savings of $4.5 million. In 2017, the OCIO found an additional annual cost savings of $6.5 million. A combined savings to date is figured to be $28,105,096. Cost reductions are passed down to the agencies through rate reductions. All of this has been achieved with a focus on tools and resources. Any cost reductions in staffing were gained through attrition. Measured Results proven by metrics will continue to be critical to demonstrating the progress of the consolidation to the agencies. The OCIO will be able to provide accurate analysis and interpret the gains for business processes. As one example, the OCIO tracks time to resolve IT support requests. Steady improvement illustrated by the graph below used a trend line to show the average days open for an incident or service request in 2016 (6.3 days). In 2017 the time was reduced to resolve within an average of 4.5 days. These charts were used in a CIO Blog from January 2018 to communicate to the agencies and IT staff how the OCIO analyzes data to tell the consolidation story.

Page 7 of 7