Research
vivi_5488
Chapter8AllAboutData.pptxYou will summarize PowerPoint, For the article summary. Need 180-200words. Please use simple words and grammar.
1
All About Data
2
Conceptual BI Data Architecture
If you don’t have high quality, relevant data, you can’t do business intelligence
Source: Howson
3
BI Data Breadth
Early BI initiatives unlocked transactional system data
Central data warehouses or independent data marts provide a central place to explore without impacting transactional systems
Integrating external market data was always a goal that has become mainstream with the advent of big data
Many BI teams are challenged with bringing extended data into their existing architecture
4
Source: Howson
Criteria for BI Success
5
Source: Howson
BI Data Quality
Quality = consistency, completeness and accuracy
New data sources and applications = a decline in data quality
Transactional system quality has improved over time
Users do not expect the same level of quality from new data sources
Users accept that newly available, granular data is “dirty”
6
Source: Howson
The Importance of Data Quality
7
Source: Howson
Band Aid May Be the Only Aid for BI Quality
8
Source: Howson
The Big Problem of Bad Data
“Information quality is the second biggest threat to humankind after global warming” – Larry English
96,000 hospital patients die each year from misleading data quality errors
In 2007, the cost of errors and rework resulting from poor data quality in the U.S. was $1.5 trillion+. In 2011 this loss climbed to $3.0 trillion
In 2012 Gartner confirmed 38% of companies did not know what poor data quality costs them, 33%+ confirmed it cost them $1 million per year
9
Source: Howson
The Source Data Maze
Assess the best source from multiple, redundant sources
Coordinating with “upstream” and “downstream” systems is a project
10
Source: Howson
11
Source Data Maze Example: Current State
12
Source Data Maze Example: Phase 2 Original Scope
Policy level data remains fragmented between Legacy B and Data Mart A
Leaves Legacy B, a “burning” platform, in operation
13
Source Data Maze Example Revised Phase 2 Scope
100% of policy level data resides in Data Mart A
Removes Legacy C, a “burning” platform
14
Source Data Maze Example, Phase 3 Scope
15
Source Data Maze Example, Proposed End State
Note: This “end state” evolved over time to
include all businesses. The goal changed
from “establish Data Mart A as our
small business data foundation” to
“Use the data warehouse as our
universal data foundation and have
Data Mart A focus on small business
as a true data mart.”
(Your Prof saw the same evolution
elsewhere when companies attempted
to accrue ROI from their data
warehouse investments.)
Legacy BI Decommissioning
It is probable legacy BI applications exist
Your return on investment in new BI applications will be considerably lower if you do not decommission, or “turn off” the old systems
Legacy BI decommissioning is a major opportunity!!
Transitioning to new BI applications and decommissioning legacy applications is a major project management challenge
16
Data Governance
Your Professor managed two data governance initiatives and was impacted by two more. They failed. Why?
Data governance is difficult to quantify
Other business priorities tend to override it
It is difficult to manage and vague goals and milestones are the “kiss of death”
17
Source: Howson
Challenged Data Governance Initiative 1
| Description | Challenges |
| A major reinsurer chartered a Business Intelligence Center of Excellence (BI COE) Part of that BI COE’s charter was to establish data governance procedures as the company implemented a new ERP system | Since the team was focused on integrating onto a single ERP reporting platform and developing a suite of new reports, data governance was postponed. This became a political issue since some leaders never agreed to this After the ERP integration was finished and reporting was implemented a data governance proposal including a metadata (data about data) strategy was published that management again rejected In the interim, two senior subject matter experts developed a data glossary the business urgently demanded as part of the data governance that they never used |
18
Challenged Data Governance Initiative 2
| Description | Challenges |
| As they converted 10 business units onto a single platform to provide analytical data for sale to clients, a prominent IT services company’s business leaders fielded questions from leadership and clients about their strategy for ensuring data quality They expanded the scope of their conversion project to include data governance | Senior leadership assigned a project manager who drafted a data governance strategy that urged management to invest adequate time and resources to support data governance This data governance initiative never received adequate management support because it was not considered with the original budget. While data governance was pertinent, budget control was more pertinent to them so they removed data governance from scope |
19
Challenged Data Governance Initiative 3
| Description | Challenges |
| A Business Intelligence Center of Excellence (BI COE) at a property and casualty (P&C) insurer established a data governance program with great fanfare This BI COE had achieved success and sufficient business support for this data governance initiative | Considering the BI COE’s previous success, stakeholders concluded this data governance effort would be successful. Their recruitment of “star” employees reinforced this expectation Surprisingly, this positive energy evaporated as the team became frustrated with a lack of direction and slow progress The data governance manager presented a vague set of goals. Six months later, she presented her goals again. Once again they were vague, with no clear milestones or schedule. Frustration increased The data governance manager eventually transitioned to another role and her “stars” either left the company or were relocated. The project never delivered value |
20
Challenged Data Governance Initiative 4
| Description | Challenges |
| A healthcare insurer with a large Business Intelligence Center of Excellence (BI COE) introduced new management The BI COE’s new leader confirmed data governance was a critical success factor for the enterprise He appointed a respected leader to deliver data governance. He was not dedicated to the project. In addition, several team members were also assigned part time | The business sponsor soon became frustrated with the lack of forward progress and decided to “pilot” data governance as part of a visible, ongoing project. Project management on this unrelated project aggressively pushed back, citing schedule and budget issues Meanwhile, the data governance manager conducted multiple meetings with no agenda or recorded action items. Attendees soon concluded this data governance initiative was not an effective use of time The business sponsor “secretly” engaged other project managers to assume management of data governance Over time the data governance leader transitioned to another job, his team members went back to their former full time jobs and data governance was not implemented |
21
A Data Governance Challenge: The Common Coding Package
Situation
A prominent reinsurer leveraged a large ERP implementation to drive the definition of a set of common business codes in a “common coding package”
Multiple meetings were held in North America and Europe
Aggressive debates ensued; management realized the U.S. and non U.S. representatives’ approaches were not aligned
What do you think happened?
22
A Data Governance Challenge: The Common Coding Package (Cont’d)
Result
The resulting “common coding package” was full of exceptions and special rules
Management realized they would never devise common business definitions
Hope for a unified enterprise data governance initiative was adversely impacted
23
A Case of Local Data Governance
Situation
A P&C insurer’s data warehouse fed “downstream BI applications” with bad data
These “downstream” teams realized the bad data came from the data warehouse
They soon learned the data warehouse team implemented their “edits” or “scrubbing” routines covering a subset of data they deemed important to their users. These edits did not cover the data required by the downstream BI applications
A “fire drill” ensued where several teams came together to resolve this issue and a long term solution was planned
What do you think happened?
24
A Case of Local Data Governance (Cont’d)
Result
As a formal data governance initiative floundered, local data governance triggered by this crisis proved quite effective
25
To Achieve Data Governance Success…
Make it a priority
Assign an executive sponsor with legitimate power
Dedicate an aggressive project manager
Fund it
Staff it (and use that staff)
26
To Achieve Data Governance Success (Cont’d)
Publish clear, measurable milestones and a schedule, then deliver
Act as a service provider to peer groups
Don’t “pilot” with other initiatives without their active support and/or insufficient budget
Local initiatives forged by crisis can be “patched together” for a holistic data governance program
27
Master Data Management (MDM)
Goal: Achieve consistent reference data
Custom transaction systems tend to devise their own codes
Data entry errors are prevalent (Lowly vs. Lohle)
MDM is “the stepchild nobody wants” because it is exacting and invisible
Without standard master data, ease of use, “clean” roll ups and low cost maintenance is elusive
28
Source: Howson
A Master Data Management Example
The team defined a “cross reference table,”
generated and maintained its own translation
key (989)
Random numbers were assigned in the ETL
(Note: Allocate a sufficient number of bytes
to cover potential length. Imagine if you have
one thousand records and you only allocate
three bytes. You’re done after “999!”)
Potential for the maintenance of multiple
aggregations (rollups)
29
Source: Howson
Separate Codes and Hierarchies For MDM
30
Source: Howson
Bidirectional MDM
Bidirectional flow ensures the “point of sale” or “point of customer contact”
can remain responsive while integrity is also maintained
31
Source: Howson
The Quality Mantra: Fix Problems at the Source
Consider the large P&C insurer’s data quality example. Our driving mantra was “fix the problems
at the source”
This seems logical, but its not easy. “Upstream and downstream” systems have different priorities
and don’t often “march together.” That’s why this quote rings true:
32
Source: Howson
Source: Howson
The Data Quality Conundrum
In a perfect world, do option one
In the real world, if you do option one you’ll get nothing done! Most do the second option
Concerns
When is the data good enough (it will never be 100%)?
Until the BI solutions surface the pain of poor data quality, that pain may never be addressed
33
Source: Howson
Proceed with Bad Data and Rigorous Expectation Management
34
Source: Howson
Use this Diagram to Assess Your Data Quality
35
Source: Howson
Proceeding with Bad Data
You’ll never be perfect and if you don’t proceed, you’ll never deliver
Be careful about who you give access to
Start with knowledge workers
Expand as data quality issues are understood and rectified
Data quality varies with purpose
Financial, regulatory and medical reports require high quality
Big data presents granular, “dirty” data with less consistency
36
Source: Howson
IMPORTANT: Plan for Adequate Fix Builds
A fix build is time allocated after each testing phase, or “pass,” that enables developers to fix issues that were captured in the previous testing phase
While adhering to aggressive schedules, most project teams underestimate both the number and time they require for fix builds (unfortunately, many have not even heard of them)
As delays occur delivery schedules become compressed and fix builds are often sacrificed
Do not underestimate the need for ample quality assurance (QA) phases with multiple fix builds. Plan for more than you need!!
37
Reference List
Howson, C. (2014). Successful business intelligence: Unlock the value
of BI and big data. New York. McGraw Hill Education.
ISBN: 9780071809184
38
3 Feeds – Policy, Event and TransactionAdaptikUmbrella Policies3 Feeds – Policy, Event and TransactionData Mart ALegacy AAdaptik Daily Reports -Policy/Rule(Umbrella)Legacy BLegacy C ReportsAnalyticDatabaseLegacy CExpress MODELLegacy A calls E model all Com and for E eligible WC, AUTO policiesCognos3 Canned Reports/Query StudioAll Auto, Com, WC PoliciesAccess DatabasesOther Sources
Data Mart A
Legacy A
Adaptik Daily Reports -Policy/Rule (Umbrella)
Legacy B
Adaptik
Legacy C Reports
Analytic Database
Legacy C
Cognos
All Auto, Com, WC Policies
Express MODEL
3 Canned Reports /Query Studio
Legacy A calls E model all Com and for E eligible WC, AUTO policies
Access Databases
Umbrella Policies
Other Sources
3 Feeds – Policy, Event and Transaction
3 Feeds – Policy, Event and Transaction
S.C./Utah – Com PricingAdaptikODSData Mart ALegacy ALegacy BLegacy C ReportsAnalyticDatabaseLegacy CExpress MODELLegacy A calls E model all Com and for E eligible WC, AUTO policiesCognos3 Canned Reports/Query Studio – Legacy Report RationaliazationAll Auto, Com, WC PoliciesAdds Umbrella and S.C./Utah Com to Data Mart AAccess DatabasesOther SourcesUmbrella and Com For S.C./UtahUmbrella/Com for S.C., Utah
Adaptik
Access Databases
ODS
Data Mart A
Legacy A
Other Sources
Legacy B
Legacy C Reports
Analytic Database
Legacy C
Express MODEL
Legacy A calls E model all Com and for E eligible WC, AUTO policies
Cognos
Umbrella and Com For S.C./Utah
3 Canned Reports /Query Studio – Legacy Report Rationaliazation
All Auto, Com, WC Policies
S.C./Utah – Com Pricing
Adds Umbrella and S.C./Utah Com to Data Mart A
Umbrella/Com for S.C., Utah
3 Feeds – Policy, Event and TransactionAdaptik~ 40 Data Elements – Replaces Legacy C and Legacy C ReportsData Mart ALegacy AAdaptik Daily Reports – Rules (Umbrella, Com (SC, UT)Legacy BAnalyticDatabase Express MODELLegacy A calls E model all Com and for E eligible WC, AUTO policiesCognos3 Canned Reports For Express – Query Studio – Report Rationalization For Legacy A ReportingAll Auto, Com, WC PoliciesUmbrella/CMP for S.C., UtahODS2 Feeds – Policy and Event (Legacy C reporting fields only)S.C./Utah – Com Pricing
3 Feeds – Policy, Event and Transaction
Adaptik
~ 40 Data Elements – Replaces Legacy C and Legacy C Reports
2 Feeds – Policy and Event (Legacy C reporting fields only)
Data Mart A
Legacy A
Adaptik Daily Reports – Rules (Umbrella, Com (SC, UT)
Legacy B
Analytic Database
Express MODEL
Legacy A calls E model all Com and for E eligible WC, AUTO policies
Cognos
3 Canned Reports For Express – Query Studio – Report Rationalization For Legacy A Reporting
All Auto, Com, WC Policies
Umbrella/CMP for S.C., Utah
ODS
S.C./Utah – Com Pricing
Adaptik~ 40 Data Elements – Replaces Legacy C ReportsODSData Mart ALegacy ALegacy BAnalytic Express MODELLegacy A calls TE model all Con and for TE eligible WC, AUTO policiesCognos3 Canned Reports For Express – Query Studio – Report Rationalization For Legacy C Reporting – Adaptik Report RationalizationAll Auto, Com, WC PoliciesUmbrella/Com for Transitioned States
Adaptik
~ 40 Data Elements – Replaces Legacy C Reports
ODS
Data Mart A
Legacy A
Legacy B
Analytic
Express MODEL
Legacy A calls TE model all Con and for TE eligible WC, AUTO policies
Cognos
3 Canned Reports For Express – Query Studio – Report Rationalization For Legacy C Reporting – Adaptik Report Rationalization
All Auto, Com, WC Policies
Umbrella/Com for Transitioned States
AdaptikODSData Mart APredictive model and Business RulesCognosCanned reprots/ Self ServiceAll Small Business
Adaptik
ODS
Data Mart A
Predictive model and Business Rules
Cognos
Canned reprots/ Self Service
All Small Business
