PPT

FinalDeliverables.pdf

Home >Computer Science homework help >PPT

The Endgame, or Putting it All Together

1Module 6: The Endgame

Module 6: The Endgame 1

The Endgame, or Putting it All Together

During this lesson the following topics are covered: • Brief review of YoyoDyne case study • Using a core set of materials to deliver presentations for two

different audiences • Comparing the main focus areas for sponsors and analyst

audiences • Using a framework to organize the main pieces of your final

presentations • Tips for sharing your code and technical documentation

Creating the Final Deliverables

Module 6: The Endgame 2

This lesson covers presents and discusses a fictional case study.

Module 6: The Endgame 2

YoyoDyne Churn Prediction Case Study

Situation Synopsis

• Retail Bank, Yoyodyne Bank wants to improve the Net Present Value (NPV) and retention rate of the customers .

• They want to establish an effective marketing campaign targeting customers to reduce the churn rate by at least five percent.

• The bank wants to determine whether those customers are worth retaining. In addition, the bank also wants to analyze reasons for customer attrition and what they can do to keep them.

• The bank wants to build a data warehouse to support marketing and other related customer care groups.

Mini Case Study: Churn Prediction for Yoyodyne Bank

Module 6: The Endgame

Shown is a synopsis of the YoyoDyne Churn Prediction Case Study, which was introduced in Module 2. We will use this scenario throughout this lesson to show examples of how you would write a final narrative summary for the case study example.

3Module 6: The Endgame

Use Analytic Plan to Guide Final Presentation

Components of Analytic Plan

Retail Banking: Yoyodyne Bank

Discovery Business Problem Framed

How do we identify churn/no churn for a customer?

Initial Hypotheses Transaction volume and type are key predictors of churn rates

Data & Scope 5 months of customer account history

Model Planning - Analytic Technique

Logistic regression to identify most influential factors predicting churn

Result & Key Findings

Once customers stop using their accounts for gas and groceries, they will soon erode their accounts and churn. If customers use their debit card fewer than 5 times per month, they will leave the bank within 60 days.

Business Impact

If we can target customers who are high-risk for churn, we can reduce customer attrition by 25%. This would save $3 million in lost customer revenue and avoid $1.5 million in new customer acquisition costs each year.

4Module 6: The Endgame

Mini Case Study: Churn Prediction for

Retail Banking

This slide represents an outline of an analytic plan for the mini case study. In addition to guiding your model planning and methodology, the analytic plan can serve as a guide for the main points of the final presentation. Within the analytic plan are components that can be used as inputs for writing about the scope, underlying assumptions, modeling techniques, initial hypotheses, and key findings.

4Module 6: The Endgame

Key Aspects of Final Presentation Material

• Reflect on the project:  Consider the context of the problems you set out to solve.

 Identify observations about the model outputs, scoring, results.

 Identify Key Messages, and any unexpected insights.

• Tailor outputs to the audience

5Module 6: The Endgame

Project Sponsor Presentation Analyst Presentation

Focus What How

Objectives

• Show that you met the project goals

• Give your sponsor talking points to evangelize the work • Emphasize ROI and business value • Mention if the models can be deployed

within sponsor’s SLA

• Show how you met the project goals

• Share your methods so analysts can learn from it for future projects • Discuss methods, techniques, and

technologies used. • Provide specific model accuracy and

speed (example: how well will it meet SLAs).

As shown, the Sponsor presentation focuses on the “what”…..What did you do, what is the ROI and business value. The Analyst deck focuses on the “how”….how did you do it, how did you opt to solve the problem, etc.

Ideally, you should consider starting the development of the final presentation during the project, rather than at the very end. This approach will ensure that you always have a version of the presentation with working hypotheses to show stakeholders, in case you need to show a work in process (WIP) version on short notice. In fact, many analysts write the executive summary at the outset of a project, then continually refine it over time so that at the end of the project, pieces of the final presentation are already done. This approach also avoids forgetting key points or insights you may discover during the project, and reduces the amount of work you will need to do on the presentation at very end of the project.

Module 6: The Endgame 5

Develop Core Material you can use to Deliver Presentations to 2 Main Audiences

Presentation Component

Project Sponsor Presentation Analyst Presentation

Project Goals • List top 3 agreed upon goals • List top 3 agreed upon goals

Main Findings • Emphasize key message • Emphasize key message

Approach • High Level Methodology • High Level Methodology • Relevant details on modeling techniques and technology

Model Description • Overview of the modeling technique • Overview of the modeling technique

3 key points supported with data

• Support key points with simple charts and graphics (example: bar charts)

• Show details to support the key points • Analyst-oriented charts and graphs (ROC curves, histograms) • Visuals of key variables and significance of each

Model Details • Omit this section, or discuss only at a

very high level

• Show the code or main logic of the model, Include the model type, variables, technology used to execute it and score data.

• Identify key variables and impact of each • Describe expected model performance and any caveats • Detailed description of the modeling technique • Discuss variables, scope, predictive power

Recommendations

• Focus on business impact of doing this, including risks and ROI

• Give the sponsor salient points to help him or her evangelize the work within the organization

• Supplement recommendations with any implications for the modeling, or for deploying in a production environment.

6Module 6: The Endgame

= Same components for both presentations = Different components for Sponsor vs. Analyst presentation

Shown above are the main components of the final presentations for the project sponsor and an analyst audience. Notice that you can create a core set of material in these 7 areas, which can be used for the two presentation audiences. Three areas (Project Goals, Main Findings and Model Description), can be used as is for both presentations. Others areas need additional elaboration, such as the Approach. Other areas, such as the Key Points, require different levels of detail for the analysts and Data Scientists than for the project sponsor.

6Module 6: The Endgame

Project Goals

Module 6: The Endgame 7

Presentation Component

Project Sponsor Presentation Analyst Presentation

Project Goals • List top 3 agreed upon goals • List top 3 agreed upon goals

Example 1 of Goals slide Example 2 of Goals slide, with Situation overview

The Goals portion of the final presentation is generally the same, or very similar, for sponsors and for analysts. In each case, you will need to reiterate the goals of the project in order to lay the groundwork for the solution and recommendations that are shared later in the presentation. In addition, the Goals slide serves to ensure there is a shared understanding between the project team and the sponsors and confirm they are aligned in moving forward in the project. Generally, the goals are agreed on early in the project and it is good practice to write them down and share them to ensure the goals and objectives are clearly understood by both the project team and the sponsors.

The slide shows two examples of slides on Project Goals. Example 1 shows three goals, describing the need to create a predictive model to anticipate customer churn and the expectation that the resulting model is at least as accurate as the current methods that YoyoDyne bank uses, and will be able to run in their production environment within the bank’s SLA.

Example 2 shows a variation on the Project Goals slide. This shows a summary of the situation prior to listing the goals. Keep in mind that when delivering final presentations these deliverables get shared within organizations and the original context can be lost, especially if the original sponsor leaves or changes roles. For this reason, it is good practice to briefly recap the situation prior to showing the project goals. Adding a situation overview to the Goals slide does make it appear busier, so you will need to determine whether to split this into a separate slide or keep it together, depending on your audience and your style of delivering the final presentation.

Module 6: The Endgame 7

Main Findings (Executive Summary)

Presentation Component Project Sponsor Presentation Analyst Presentation

Main Findings (executive summary)

• Emphasize key message • Emphasize key message

Module 6: The Endgame 9

• Enable reader to grasp full synopsis in 1 slide

• Frame outcomes in terms of business value

• Generally same, or very similar, for both types of audiences

Writing a solid Executive Summary to portray the main findings of a project is crucial. In many cases, it may be the only portion of the presentation hurried managers will read. For this reason, it is imperative to make the language clear, concise and complete. Someone consuming the executive summary should be able to grasp the full story of the project and the key insights in a single slide. In addition, this is an opportunity to provide key talking points for the executive sponsor to use to evangelize the project work with others in the customer’s organization.

Be sure to frame the outcomes of the project in terms of business value, which is especially important if presentation is for the sponsor.

Module 6: The Endgame 9

Anatomy of an Executive Summary

Module 6: The Endgame 10

Key

Message

Major

Points

Business Impact

SLA

Shown above is an Executive Summary, with the main components called out in captions. Keep in mind this is not the only format for conveying the Executive Summary – this will vary based on the author’s style, although many of the key components are common themes in Executive Summaries.

We recommend making your key message very clear and conspicuous at the forefront of the slide. This can be set apart with color as shown above, or you can use some other technique to draw attention to it. Your key message may become the talking point that executives or the project sponsor takes away from the project and uses to support your recommendation for a pilot project, so this needs to be succinct and compelling. To make this message as strong as possible, measure the value of the work and quantify the cost savings, revenue, time savings, or other benefits to make the business impact concrete.

Support the key message with 3 major points. You can have more than three major points, although going beyond three ideas makes it difficult for people to recall the main points, so you need to ensure that you keep the ideas clear and limit it to the few most impactful ideas you want the audience to take away from your work. If you list 10 key points, your message will become diluted and the audience may remember only 1 or 2. In addition, since is an analytical project, be sure to make one of your key points related to if, and how well, your work will meet the sponsor’s SLA. Finally, although not required, it may be a good idea to support your main points with a visual, which will support the major points written at right. Visual imagery will serve to make a visceral connection with the reader, and help him or her retain the main message.

Module 6: The Endgame 10

Approach

Module 6: The Endgame 11

Presentation Component

Project Sponsor Presentation Analyst Presentation

Approach • High Level Methodology • High Level Methodology • Relevant details on modeling techniques

and technology

Example Approach slide, for Sponsors Example Approach slide, for Analysts

Note: Green boxes highlight differences between slides

In the Approach portion of the presentation, you will need to explain the methodology you pursued on the project. This can include interviewing domain experts, the groups you collaborated with in the organization, and a few statements about the solution you developed. The objective of this slide is to ensure the audience is clear on the course of action you pursued, and understands it well enough to explain it to others in the organization. Also be sure to include any additional comments related to your working assumptions as you did the work, this can be critical in defending why you pursued a specific course of action.

When explaining the solution you developed, keep it at a high level for the project sponsors. If presenting to analysts or data scientists, add in additional detail about the type of model used, the technology and the actual performance of the model during your tests. Finally, as part of the description on your approach, you may also want to mention constraints from systems, tools, or existing processes, and any implications for how these things may need to change due to this project.

Module 6: The Endgame 11

Model Description

Module 6: The Endgame 12

Presentation Component

Project Sponsor Presentation Analyst Presentation

Model Description • Overview of the modeling

technique • Overview of the modeling technique

Although the Model Description slide can be used for both audiences, the interests and objectives differ for each. For the sponsor, you will need to articulate the general methodology without getting into too much detail. You will need to convey the basic methodology followed in your work so the sponsor can communicate this to others within the organization. To do this, focus on explaining the general methodology you used in way that will enable your sponsor to convey it to others, and provide talking points. Mention the scope of the data used to illustrate thoroughness and provide confidence that you used an approach that was an accurate portrayal of their problem and as free from bias as possible. One of the key traits of a good Data Scientist is the ability to be skeptical of one’s own work. This is an opportunity to view the work and the deliverable with a critical eye and consider how it will be received by the audience. Try to make sure it is an unbiased view of the project and the results.

Assuming the model will meet the agreed upon SLAs, mention that the model will meet the SLAs based on performance of the model within the testing or staging environment.

Analysts will want to understand the details of the model, including the decisions you made in constructing the model, and the scope of the data extracts for testing and training. Be prepared to explain your thinking on this, as well as the speed of running the model within the test environment.

Module 6: The Endgame 12

• Identify key points based on your insights and observations resulting from the data and model scoring results

• Illustrate your key points with charts and visualizations

• Use simpler charts for Sponsors

Key Points Supported With Data

Module 6: The Endgame 13

Presentation Component

Project Sponsor Presentation

Analyst Presentation

Key Points Supported With Data

• Support key points with simple charts and graphics (such as bar charts)

• Show details to support the key points • Analyst-oriented charts and graphs (ROC curves, histograms) • Visuals of key variables and significance of each

Shown is an example slide showing some supporting detail regarding the rate of bank customers who would churn in various months. When developing your key points, consider the insights that will drive the biggest business impact and are defensible with data. For project sponsors, use simple charts such as bar charts, which illustrate data clearly and will allow the audience to understand the value of the insights. This is also where you will foreshadow some of your recommendations, and begin tying together your ideas to demonstrate what led to your recommendations and why. Creating clear, compelling slides to show your key points will make the recommendations more credible and more likely to be acted upon by the customer.

For analyst presentations, you will need more granular graphics. In this case, you may want to show a dot density chart or a histogram of a data distribution to support decisions you made in your modeling techniques. We will further discuss basic concepts of data visualizations in the next lesson.

Module 6: The Endgame 13

Model Details

Presentation Component

Project Sponsor Presentation

Analyst Presentation

Model Details • Omit this section,

or discuss only at a very high level

• Show the code or main logic of the model, Include the model type, variables, technology used to execute it and score data.

• Describe expected model performance and any caveats • Detailed description of the modeling technique, variables, scope,

predictive power

Module 6: The Endgame

Model details are needed to share additional information with people who have a more technical understanding than the sponsors, those who will need to implement the code, or colleagues on the analytical team.

In this section, discuss the variables you used in the model and explain how or why you selected the ones you did. In addition, you should share the actual code (or at least an excerpt) you developed to explain what was created and the basic mechanics of how it will operate. This will also serve to foster discussion related to any additional constraints or implications related to the main logic of the code. In addition, you can use this section to illustrate details of the key variables and the predictive power of the model, using analyst- oriented charts and graphs, such as histograms, dot density charts & ROC curves.

Discuss the speed with which the model can run in the test environment, the expected performance in a live, production environment, and the technology needed. This kind of discussion will address how well the model can meet the organization’s SLA.

Finally, include any additional caveats of the model and model performance, such as systems or data the model will need to interact with, performance issues, or how to feed the outputs of the model into existing business processes. Describe the relationships of the main variables on your objectives (such as the effects of key variables on predicting churn, or the relationship of key variables to other variables). You may even consider making suggestions to improve the model, highlight any risks to introducing bias into the modeling technique, or describe certain segments of customers who may skew the overall predictive power of the methodology.

Module 6: The Endgame 14

Recommendations

Presentation Component

Project Sponsor Presentation Analyst Presentation

Recommendations

• Focus on business impact of the project, including risks and ROI

• Give the sponsor salient points to help him or her evangelize the work within the organization

• Supplement recommendations with any implications for the modeling, or for deploying in a production environment.

Module 6: The Endgame

Create a set of recommendations that will include how to deploy the model from a business perspective within the organization and any other suggestions on the rollout of this logic, depending on your knowledge of the domain area from the discovery phase.

Attempt to measure the impact of the improvements and state how to leverage this within the recommendations. For instance, you might mention that every customer retained represents a time savings of 6 hrs for one of the bank’s account managers or $50k in savings of new account acquisitions (due to marketing costs, sales, and system-related costs).

Focus on the actions you recommend to operationalize the work and the benefits the customer will receive as a result of implementing these recommendations.

Module 6: The Endgame 15

Quick Summary of Final Presentation Components

Module 6: The Endgame 16

Shown is a summary of the main components of the final presentation.

Module 6: The Endgame 16

Top 10 Tips, Tricks, & Pitfalls to Avoid for the Final Presentation

1. Be visual. Generally, the more visual the better. Up to a point. 2. Be MECE (Mutually Exclusive and Collectively Exhaustive). 3. Tie your ideas together….don’t force people to tie your ideas together, guide people

and help them draw logical connections.

4. Don’t forget that not everyone has gone through the Discovery phase like you have. 5. Context is key. Orient people to the project itself, as well as the graphics you use,

the terminology and jargon (spell out acronyms).

6. Don’t assume people see the obvious benefits. 7. Measure and quantify the benefits. Be specific. “$8.5M in annual cost savings” is

much stronger than “Great Value”.

8. Be patient. You may have to tell your story more than once…consider these sessions opportunities to refine your message and share good work that was done.

9. Let the intended audience guide you in shaping the right message and level of detail.

10. Avoid long bulleted lists ☺

Module 6: The Endgame 17

Here is a list of tips, tricks, and common pitfalls to avoid when creating presentations. One of the biggest mistakes authors make when creating presentations is forgetting to take time to set the context in presentations and in presenting the findings. When doing this, keep in mind that you always need to orient the viewer to the work you have created. This is true when writing your goals and situational overview for the project as a whole, and also for creating charts to give the right amount of context to orient the viewer to your main message. Context is key in conveying information in a way the audience will easily understand.

Module 6: The Endgame 17

Overview of Code & Technical Documentation

• Consider the interests of your technical audience:  How will the project affect them?

 In what ways will it change their day-to-day roles, or existing processes?

 Be aware of the implications of your work on their roles as you create these technical deliverables.

• 2 Technical deliverables:  Code

 Technical specifications and documentation.

Module 6: The Endgame

Now that you have learned a framework for authoring the final presentations, you also need to consider how to deliver the actual code you developed and the technical documentation that you need to support it. In doing this, consider how the project will affect the end users, and the technical people who will need to implement the code you developed.

Think through the implications of how your work will affect the recipients of the code, the kinds of questions they will have, and their interests. For instance, indicating that your model will need to perform real-time monitoring may require extensive changes to IT runtime environment, so you may need to consider a compromise of batch processes or nightly processes. In addition, you may need to get the technical team talking with the project sponsor to ensure the implementation and SLA will meet the business needs during the technical deployment.

Plan to address questions from IT related to how computationally expensive is will be to run the model in the production environment. You can provide a sense of this based on how well the model ran in the test scenarios and if there is additional fine-tuning that can be performed to optimize performance in the production environment.

Module 6: The Endgame 18

Considerations for Technical Specifications & Documentation

• Inputs & Pre-processing:  Discuss the expected pre-processing steps before data goes to the model code.

 Document expected input, data format, source tables, and units.

 Describe the processing script are you using.

 Explain how the outputs are created.

• Exception handling :  Explain how to deal with exceptions to the model.

 Provide guidance for making decisions on the exceptions.

• Post-processing:  After you create the output, discuss any post-processing before going to the next

step.

 Interpreting a threshold as opposed to a simple yes/no.

Module 6: The Endgame

Approach the documentation as if it’s for an API (application programming interface)

When considering the final deliverables, think as if you are delivering an API. To do this, you will need to consider the inputs, outputs, and other system constraints to enable a technical person to implement it, even if they have not had a connection to the analytics project up to this point.

Think about the documentation you create as a way to introduce the data your model needs, the logic it is using, and how other related systems will need to interact with it in a production environment for it to operate well.

Consider writing specifications including specific details about the inputs the code will need, the data format and structures (such as do you need structured data, and does the expected data need to be numeric or string formats). Describe any transformations that need to be done on the input data before the code can use it, and if you created any scripting to perform these tasks. These kinds of details are important when other engineers need to modify your code, or point to a different dataset or table if and when their environment changes.

Regarding exception handling, consider how you handle data that is outside the expected data ranges of the model parameters, and how you will handle missing values, null values, zeros, NAs, or data that is in an unexpected format or type. Consider how you will treat these exceptions, or if there are implications on downstream processes that will need to be accounted for.

Regarding the model outputs, you will need to explain to what extent you post-process the output. For example, if you return a probability of churn, identify the scoring threshold to decide which customer accounts to flag as "in danger“ or at risk of churn. In addition, you will need to make provisions for adjusting this threshold and training the algorithm, either in an automated learning fashion or with human intervention.

Module 6: The Endgame 19

Providing Your Code

• Test for accuracy in the production environment

• Ensure the code will run quickly and meet SLAs

• Include comment lines in the code

• Hold a briefing with the engineers who will implement the code

Module 6: The Endgame

Although you will need to create technical documentation, many times engineers receive the code and try to use it without reading through all of the documentation. For this reason, it is important to add extensive comments in the code itself. This will serve to guide the people implementing the code on how to use it, what pieces of the logic are supposed to do, and guide them in stepping through the code and becoming familiar with it. If you can do a thorough job adding comments in the code, you will make it easier for someone else to maintain the code and help tune it in the runtime environment. In addition, this will help the engineers make edits to the code when their environment changes or they need to modify processes that may be providing inputs to the code or receiving its outputs.

Module 6: The Endgame 20

The Endgame, or Putting it All Together

During this lesson the following topics were covered:

• Brief review of YoyoDyne case study

• Using a core set of materials to deliver presentations for two different audiences

• Comparing the main focus areas for sponsors and analyst audiences

• Using a framework to organize the main pieces of your final presentations

• Tips for sharing your code and technical documentation

Summary

Module 6: The Endgame 21

This lesson covered a fictional case study.

Module 6: The Endgame 21