RESEARCH SYNTHESIS

gnsv.srinivas

Analytics_The_Agile_Way_----_Part_TWO_Agile_Methods_and_Analytics.pdf

Home >Business & Finance homework help >Operations Management homework help >RESEARCH SYNTHESIS

p02 77 31 May 2017 9:32 AM

P A r t T W O

Agile Methods and Analytics

P art One covered the requisite background on data and analytics.

Now it’s time to introduce Agile methods, and specifically Scrum.

In Part Two, you will learn how the Agile world differs mark-

edly from its phase-gate counterpart. To paraphrase Allen Coin, Agile

methods are “like finishing a ship while at sea. This way you have to

focus on the most important things.”*

This part contains the following chapters:

■ Chapter 4: A Better Way to Work: The Benefits and Core Values of Agile Development

■ Chapter 5: Introducing Scrum: Looking at One of Today’s Most Popular Agile Methods

■ Chapter 6: A Framework for Agile Analytics: A Simple Model for Gathering Insights

* See http://bit.ly/2o52mOD.

Simon, Phil. Analytics : The Agile Way, John Wiley & Sons, Incorporated, 2017. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/harrisburg-ebooks/detail.action?docID=4901712. Created from harrisburg-ebooks on 2020-12-01 07:12:19.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2o52mOD

p02 78 31 May 2017 9:32 AM

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

c04 79 31 May 2017 9:24 AM

C h A p t e r 4 A Better Way to Work The Benefits and Core Values of Agile Development

The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and starting on the first one.

—Mark Twain

T he dirty little secret in the software industry is that information

technology (IT) projects fail more often than not. Sure, not every

failure qualifies as a public spectacle like the Healthcare.gov debacle,

but the track record of Waterfall projects has never been anywhere

close to good.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

80 ▸ A n A l y t i c s

c04 80 31 May 2017 9:24 AM c04 81 31 May 2017 9:24 AM

The CAse AgAinsT TrAdiTionAl AnAlyTiCs ProjeCTs

In 2014, I spoke at Netflix headquarters about my then-current book,

The Visual Organization. From the moment I entered the building, I

detected a decidedly different vibe compared to that at most organiza-

tions on my book tour.

Perhaps it began with the boldly displayed Tech Emmy in the com-

pany’s lobby.* It continued with the data visualizations on the wall

and the employees’ profound knowledge of data, analytics, and tech-

nology. (I was nowhere near the smartest person in the room.) To be

sure, it was an unforgettable experience.

Since that time, I have reflected quite a bit on the two hours I

spent at the streaming-video giant. Netflix made quite an impression

on me. No wonder that it’s one of the most successful companies of the

Internet Age. Sure, it built a better mousetrap and benefited from how

Blockbuster management grossly misjudged consumer trends. What’s

more, Netflix has made its fair share of blunders.

* For my obligatory selfie, see http://tinyurl.com/zos87j7.

R e m e m b e R Q w i k s t e R ? In 2011, Netflix announced that it would be effectively splitting its business into two separate services and related websites. Many subscribers would now need to manage two rental queues: one for streaming movies and TV shows, and one for physical DVDs in red envelopes. (For a variety of reasons, don’t expect the latter part of Netflix’s business to go the way of the dodo anytime soon.) Nearly one million customers promptly canceled their accounts. More took to social media to vent before Reed Hastings abruptly walked back Qwikster.

Ultimately, it was a silly idea. Nonetheless, it proves that Netflix is willing both to take big risks and to listen to data when those risks don’t pan out.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://tinyurl.com/zos87j7

A B e t t e r W A y t o W o r k ◂ 81

c04 80 31 May 2017 9:24 AM c04 81 31 May 2017 9:24 AM

Understandable but Pernicious

The project mentality is prevalent and certainly understandable. It’s

also pernicious: It encumbers organizations from really embedding

analytics into their cultures—something that has never been more

essential.

For starters, projects, by definition, need beginning and end dates.

Make no mistake: This is a problem in the context of analytics. All too

often in my decade working on IT projects, employees would pooh-

pooh new responsibilities and ways of working because “things will go

back to normal” when the project concludes. This is especially true if

the project doesn’t go as planned. If you consider that concern trivial,

think again. Companies’ overall record implementing new technolo-

gies is dismal.

Beyond that, a project mind-set contravenes the notion of con-

tinuous delivery, something that successful organizations have right-

fully adopted for years. Intelligent professionals today recognize

that analytics is never “finished.” Rather, it needs to be consistently

refined, audited, expanded, and even retired when it no longer

makes sense.

tip Agile methods are far better suited for analytics precisely because they

recognize that the work is never done.

A different Mind-set at netflix

Many organizations continue to approach analytics as a traditional,

discrete IT “project” of a finite duration. Netflix, however, certainly

does not. My visit to its headquarters in Los Gatos, California, and con-

versations with its employees corroborated my prior beliefs.

In this regard, among others, Netflix goes against the grain—and

it’s hard to argue with its results. I’d also put Facebook, Amazon, and

Google/Alphabet in the same boat. These companies routinely use

analytics to make key business decisions. Put differently, they don’t

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

82 ▸ A n A l y t i c s

c04 82 31 May 2017 9:24 AM c04 83 31 May 2017 9:24 AM

treat analytics as one-time IT project. Rather, they consider analytics to

be a business project that never really ends. They consistently augment

their corpuses of knowledge by embracing new data sources. They

have built infrastructures that support instant analysis of complex data

sources and types, not to mention mind-boggling volumes.

Proving The sUPerioriTy of Agile MeThods

Over the years, many academics, industry experts, and CIOs have

questioned the sanity of blindly continuing to follow failure-laden

processes such as the Waterfall method. Expensive consultants armed

with proprietary, complex, and fancy-sounding methodologies have

often just made things worse. To this end, Agile methods started gain-

ing popularity in the mid-1990s and early 2000s.

Yet, not everyone was on board. Skeptics abounded. Many tradi-

tionalists prematurely dismissed Agile methods because they seemed

just too, well, weird. Old-school IT folks knew only one way to work.

They often had trouble grasping the idea that frequently shipping

small batches is even possible, let alone that it almost always beats

boiling the ocean. Beyond that, they could think solely in terms of

detailed and complicated business requirements, not more pragmatic

user stories. (We’ll cover these in Chapter 5.)

Eager to see how Agile methods compared to their predecessors,

Dr. David Rico began studying them in earnest. In 2009, he and his

coauthors penned The Business Value of Agile Software Methods. The book

details the economics of different software-development models. As

Dr. Rico et al. described in the text:

We found 79 studies with quantitative data on the benefits of Agile methods. They cited an average of 26% improvements in cost, 71% improvements in schedule, and 122% improvements in productivity performance. Quality improvements averaged 75% and customer satisfaction improvements averaged 70%. Over 29 of these studies had the data necessary to estimate an average return on investment of 2,633%.

At least 26 of these studies involving over 726 programmers yielded an average productivity of over 21

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A B e t t e r W A y t o W o r k ◂ 83

c04 82 31 May 2017 9:24 AM c04 83 31 May 2017 9:24 AM

lines of code per hour. This is roughly 10 to 20 times the productivity rate associated with traditional methods. At least 21 of these studies involving over 323 programmers yielded an average defect density of about 2 defects per thousand lines of code.

All of these factors combine to make Agile methods more productive to use than traditional methods. Agile methods result in lower maintenance and total life cycle costs due to greater productivity and efficient defect removal. On average, Agile methods are about 25 times more efficient than traditional methods.

Dr. Rico and his colleagues conclusively proved the benefits of

Agile methods on software-development projects. He also wondered

about teams that formerly worked on phase-gate projects. How did

they feel in comparison to their experiences on Agile teams?

Figure 4.1 shows the unequivocal answer to that question through

results from internal team surveys. The solid line reflects the scores for

Agile teams, while the dashed line reflects teams’ scores on Waterfall

projects.

figure 4.1 Agile Methods: Before and After Source: the Next Wave of technologies.

Transparency 5

4.5 4

3.5 3

2.5 2

1.5 1

0 0.5

Communications—Internal

Communications—External

Quality

Planning

Team Performance

Teamwork

Development Process

Empowerment

Issue Resolution

Change Management

Work/Life Balance

Sustainable Pace

Leadership

Satisfaction

Personal Involvement

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

84 ▸ A n A l y t i c s

c04 84 31 May 2017 9:24 AM c04 85 31 May 2017 9:24 AM

The results could not be clearer. Agile teams report substantial

increases along a number of key areas: communications, teamwork,

empowerment, team performance, planning, quality, and so on. In

fact, Agile methods beat phase-gate methods across the board.

But how exactly does this happen? Is the absence of unrealis-

tic phase-gate deadlines enough? Do Agile teams follow foolproof

five-point checklists? Not exactly. The answer lies in how Agile teams

work.

The CAse for gUidelines over rUles

Ah, rules. Lovely rules. Where would most companies be without them?

It’s a subversive question in many organizations, a sign that you

are some type of freethinking anarchist. Still, it’s a valid question to

ask. After all, rules often don’t accomplish what they set out to do.

(If you don’t believe me, Google the law of unintended consequences, a

subject that we’ll revisit in Chapter 11.) By trying to articulate exactly

what is and what is not permitted, organizations allow for massive

loopholes, not to mention a general feeling of “management doesn’t

trust us” by the rank and file.

It’s not hard to find instances in which workers may not have

violated a specific policy in the employee manual, but their behavior

clearly violated its spirit. In a nutshell, this is the problem with rules.

So says Dov Seidman in his excellent book How: Why How We Do

Anything Means Everything. Seidman compellingly argues that con-

veying general principles governing our behavior is almost always

more effective than promulgating myriad rules that forbid specific

actions.

That may be fine in theory but a little hokey in practice. Have any

organizations made this jump?

As it turns out, yes. A few high-profile companies have realized

the inefficiency of trying to manage employee behavior by rules and

corporate fiats. For instance, Brazil’s Semco S.A. operates in a num-

ber of industries, including real estate, banking, and web services. As

an organization, its diverse lines of businesses hardly makes it unique.

Most of its mainstream press stems from the radical management

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A B e t t e r W A y t o W o r k ◂ 85

c04 84 31 May 2017 9:24 AM c04 85 31 May 2017 9:24 AM

style that Semco S.A. executives have adopted: The company forgoes

traditional management altogether. Instead, employees rely on their

own discretion to make almost all decisions.

You may not have heard of Semco S.A., but you probably know of

another company that embraces a very different type of management

philosophy.

Patty McCord used to serve as chief talent officer at Netflix before

hanging out her own shingle. Writing for Harvard Business Review, she

detailed the ex-employer’s über-progressive view on policies (that she

helped create):

We also departed from a formal travel and expense policy and decided to simply require adult-like behavior there, too. The company’s expense policy is five words long: “Act in Netflix’s best interests.” In talking that through with employees, we said we expected them to spend company money frugally, as if it were their own. Eliminating a formal policy and forgoing expense account police shifted responsibility to frontline managers, where it belongs. It also reduced costs.1

scarcity and Trade-offs on Agile Projects

In his 1932 paper titled “An Essay on the Nature and Significance of

Economic Science,” Lionel Robbins defined economics as the study

of scarcity.2 It’s a remarkably simple definition, yet it makes perfect

sense. Governments can’t do everything; they have to make choices

based on their leaders’ priorities. A dollar spent on national defense

means one less dollar available for infrastructure improvement or

social programs.

The same tenet applies to organizations. Even a colossus such as

Amazon cannot pursue each crazy and expensive idea that it consid-

ers. It has to make choices about which new products and services

to develop. This also applies to divisions, departments, groups, and

individuals.

Written in 2001, The Manifesto for Agile Software Develop-

ment formally codifies the trade-offs inherent in any project. It

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

86 ▸ A n A l y t i c s

c04 86 31 May 2017 9:24 AM c04 87 31 May 2017 9:24 AM

explicitly states that Agile teams should adhere to the following

four principles:

1. Individuals and interactions over processes and tools.

2. Working software over comprehensive documentation.

3. Customer collaboration over contract negotiation.

4. Responding to change over following a plan.*

It’s easy to read the second of these four tenets and become a little

concerned. After all, this is a book on analytics, not software develop-

ment. You need not worry, though. Software development and analyt-

ics are more similar than dissimilar. What’s more, Agile is, at its core,

flexible. As such, we can easily apply the same principles to analytics.

Further, the four tenets do not imply that, for instance, documen-

tation isn’t important. It most certainly is, but the Agile Manifesto rec-

ognizes that there are only so many hours in a day. As such, all else

being equal, producing working software—or, in this case, analytics—

takes precedence over documenting what you’re doing for the future.

Ditto for responding to change over following a plan.

The specific Tenets of Agile Analytics

Making these high-level guidelines and principles explicit increases

the chances that Agile methods are ultimately successful. Still, it’s not

practical to work for months under the vague direction of a four-point

doctrine. No, more specific direction is needed in the form of specific

precepts. Here, I am borrowing liberally from Ken Collier’s excellent

book Agile Analytics.

■ The highest priority is to satisfy the user community through

the early and continuous delivery of analytics.

■ Agile methods value speed and end-user buy-in over perfection.

It’s expected and entirely natural for flaws to exist, especially in

early stages.

■ Agile teams welcome changing requirements, even late in

development. This is possible because Agile processes embrace

change.

* See http://agilemanifesto.org.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://agilemanifesto.org

A B e t t e r W A y t o W o r k ◂ 87

c04 86 31 May 2017 9:24 AM c04 87 31 May 2017 9:24 AM

■ Agile teams deliver analytics frequently. They are often able to

integrate new data sources and types every few weeks.

■ Everyone on a project must share ownership and work together

on a daily basis—not just Scrum teams. This includes stakehold-

ers, end users, developers, analysts, and data scientists.

■ Without sufficient trust and resources, Agile teams and projects

will fail.

■ E-mail generally sucks. It is not a true collaboration tool. Often

the best way to brainstorm, convey information, and resolve

conflicts is via in-person conversations.

■ Agile methods are designed to maximize team productivity, not

any one individual’s productivity. Local optimization usually

causes global degradation.

■ Relevant, timely, and meaningful analytics is the primary mea-

sure of progress.

■ The best work often emerges from self-organizing teams.

■ There is a fundamental trade-off among a project’s scope, sched-

ule, and cost/resources.* For instance, if you reduce the time

that a team has to complete a project, then you have to either

decrease its scope or provide it with greater resources.

■ Agile teams must work at a sustainable pace. Team members

need time to think, contemplate, and experiment. When deal-

ing with so much data, the “right” or “best” way may not be

readily apparent.

■ At regular intervals, Agile teams should reflect on how to

become more effective. Single project postmortems offer little

value and opportunity to improve.

tip If it seems as though keeping a full-time team focused on Agile prin-

ciples and methods is a full-time job, you’re absolutely right. Scrum

teams (discussed in the next chapter) hold a specific seat for such a

person: the Scrum Master.

* A similar maxim is “Fast, cheap, and good: pick any two of the three.”

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

88 ▸ A n A l y t i c s

c04 88 31 May 2017 9:24 AM

Chapter review and disCussion Questions

■ Consider Waterfall and Agile methods. When it comes to launching and implementing technology projects, which are generally better and why?

■ Are there any circumstances in which the Waterfall method makes sense? If so, what are they? Why do they make sense? If not, why not?

■ Why are guidelines generally better than rules?

■ What are the four main guidelines from the Agile Manifesto?

■ What are some of the specific tenets of Agile methods? How do they differ from older methods?

Now that we have established the overarching tenets of Agile projects,

it’s time to get more specific. The next chapter dives into the ins and

outs of one of today’s most popular ways of deploying software and

analytics: Scrum.

noTes

1. Patty McCord, “How Netflix Reinvented HR,” Harvard Business Review, January/Feb- ruary 2014, http://bit.ly/1xrhZjR, retrieved March 11, 2017.

2. Lionel Robbins, “An Essay on the Nature and Significance of Economic Science” (London: Macmillan, 1932).

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/1xrhZjR

C h a P t e r 5 Introducing Scrum Looking at One of Today’s Most Popular Agile Methods

When you’re finished changing, you’re finished.

—Benjamin Franklin

A s mentioned in the Introduction, for decades, organizations

largely followed Waterfall or phase-gate methods on IT projects.

Gantt charts representing one- or two-year timeframes ruled the

day. (See Table I.1 in the Introduction.) Failure was common, expensive,

and often devastating.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

90 ▸ A n A l y t i c s

Fortunately, this has started to change. Agile methods are no longer

weird. Startups have embraced them for years, but mature organiza-

tions are finally starting to see the light. They are opting for smaller

batches and Agile methods to further their software-development

efforts. In a parallel way, these same organizations’ analytics efforts

can benefit from adopting a similar mind-set. In a nutshell, this is the

premise of this book.

This chapter briefly defines Agile methods, roles, techniques, and

terminology with a particular emphasis on Scrum. It is not supposed to

be comprehensive. Dozens or even hundreds of books exist on every

mainstream Agile method.

tip Agile analytics is equal parts what you do and how you do it.

A Very BrIef HIStory

The origins of Agile methods date to 1984 when Hirotaka Takeuchi and

Ikujiro Nonaka authored a paper in the Harvard Business Review entitled

“The New New Product Development Game.” Takeuchi and Nonaka

surveyed several Japanese product-development firms such as Canon,

Honda, and NEC Corporation. Their objective: to see why these firms

were so successful for so long developing global consumer products.

Whereas U.S. organizations approached projects in very rigid, linear,

and phased fashions, their Japanese counterparts did the opposite—that

is, they employed cross-functional teams and shorter, more frequent,

iterations. The results could not have been more different.

Ken Schwaber and Jeff Sutherland formally codified Scrum in

1995, although the first informal Scrum team was formed in 1993.

Today, the Scrum Alliance defines the term as:

a simple yet incredibly powerful set of principles and practices that help teams deliver products in short cycles, enabling fast feedback, continual improvement, and rapid adaptation to change.*

* See https://www.scrumalliance.org.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

https://www.scrumalliance.org

i n t r o d u c i n g s c r u m ◂ 91

It is “a simple, team-based framework to develop complex systems

and products.” Note that Scrum is only one type of Agile development

method. Other ones include Extreme Programming, Feature-Driven

Development, and the Dynamic Systems Development Method. While

there are true differences among these approaches, they are more

similar than dissimilar.

With the background out of the way, it’s now time to introduce

the language of Scrum.

Scrum teAmS

To paraphrase John Goodman’s brilliantly bombastic Walter Sobchak

in The Big Lebowski, the beauty of a Scrum team lies in its simplicity.

Scrum teams consist of only three roles: product owner, Scrum Mas-

ter, and team member. That’s it. Figure 5.1 represents this visually.

figure 5.1 Simple Visual of Scrum team Makeup Source: Phil Simon.

Product Owner

Team MemberScrum Master

As we’ll see in a bit, these roles inhere specific responsibilities;

Scrum is not anarchy. Also note that there are no hierarchies—and

this is very much by design. Let’s say that you are on a Scrum team

and your formal job title is business analyst. Also on your team is Lydia,

your friend’s boss and the director of marketing. Lydia may outrank

you on the org chart, but for the purposes of the Scrum team, you are

equals. Scrum is very egalitarian that way.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

92 ▸ A n A l y t i c s

Product owner

The product owner is responsible for maximizing return on invest-

ment. She is authorized solely to ask the team to do work and change

the priority of product backlog items. She ensures that the team mem-

bers understand customer needs. She keeps the product vision and

prioritizes user stories. Put differently, she is not just another coder.

That’s not to say that hers is an easy job. On the contrary, it is

often very demanding. She’ll often have to deal with issues that run

the gamut from legal to compliance to marketing to budgets to man-

agement and the like. What’s more, the product owner must remain

objective while managing any inherent tensions that develop within

the team. In the words of Mindi Schnase, project manager for Renown

Health and a Certified Scrum Master:

The product owner should have the type of personality who can “provide the team with high-level requirements and objectives for the product, but will allow the team to determine how to accomplish these goals.”*

Finally, the product owner generally writes acceptance criteria.

(We’ll get to that shortly.)

Scrum master

The Scrum Master is an expert and advisor, coach, and facilitator. This

role is essential for organizations using Scrum for the first time. He

may tell the product owner “no.” Ideally, he has the maturity, experi-

ence, knowledge, and humility to function effectively in this capac-

ity. Put differently, this role might not be a great fit for a 21-year-old

recent college graduate who lacks much proper business experience.

The Scrum Master occasionally bulldozes impediments. To quote

Schnase again:

If someone can’t get into a network folder they need or the testing environment is down, Scrum Masters can help

* For more, see http://bit.ly/2mVy5CD.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2mVy5CD

i n t r o d u c i n g s c r u m ◂ 93

escalate and resolve these issues. Scrum Masters need to be able to communicate with any stakeholder, in addition to being assertive and enthusiastic.*

If you think that this means being the occasional bad guy, you’re

right. What’s more, Scrum Masters feel a keen sense of ownership

over projects. As Mike Cohn, founder of Mountain Goat Software, so

eloquently put it:

An orchestra conductor once explained that he has no real power over how the individual musicians play. Yet he feels a tremendous responsibility toward helping them be the best musicians they can be.1

Scrum Masters may minimize their involvement as the team

coheres, becomes more productive, and increases its understanding

of Scrum.

team member

On a Scrum team, you’ll find far more players than coaches. In other

words, team members constitute the majority of Scrum teams. Team

members operate with high degrees of authority. They solely deter-

mine estimates (i.e., how much time and effort each task takes) and

they collaborate freely, and ideally, without regard to title/role.

At bottom, teams and team members face a common objective

irrespective of the project: to continue to deliver user stories. That’s

it. This means maximizing the team’s productivity—not necessarily that

of any one individual. For a team to be effective, a high degree of col-

laboration needs to exist within a team and among team members.

Individuals who are overly possessive about completing “their” user

stories often don’t want to help others—and the team’s productivity

suffers for it. This is typically where the Scrum Master steps in.

This brings us to the question of whether teams should be cross-

functional. While there’s an open debate in the Scrum community

about the matter, I’m of the opinion that the answer is yes. In software

* Ibid.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

94 ▸ A n A l y t i c s

development, stacking a team exclusively with programmers often

yields disappointing results. It’s best to include business analysts and

quality-assurance testers as well. Permit me the following analogy: In

basketball, 7-foot centers can certainly rebound and block shots, but

can they effectively bring the ball up the court? Can they protect the

ball from opponents’ guards with quick feet and quicker hands?

uSer StorIeS

For decades, gathering business requirements has been the bane of

many IT projects. Fortunately, Scrum explicitly recognizes the futility

of this exercise. In its stead, Scrum employs user stories that capture

the descriptions of desired features from the point of view of employ-

ees, customers, clients, or other end users. In so doing, user stories

help create simplified and understandable descriptions of business

requirements.

All user stories should contain each of the following three

elements:

1. Role: Type(s) of users

2. Goal: What they want

3. Benefit: Why

An example is in order.

On November 30, 2016, Netflix announced that its subscribers

could download movies to their devices for future viewing. The next

day, I asked two of my students to describe this new feature in the

form of a proper user story. Both of them correctly said something

along these lines:

As a Netflix subscriber, I would like the ability to download shows and movies so I can watch them in areas with no or poor network connectivity.

It’s important here to describe what proper user stories intention-

ally omit: how. That is, user stories should specify neither the technolo-

gies nor the data required to solve the business problem. Scrum team

members do this. (Don’t worry; we’ll return to this subject later in this

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 95

chapter). Ideally, team members maintain a great deal of autonomy in

tackling their user stories.

Clear, concise user stories are essential for all Agile projects—

including analytics. Lamentably, many user stories just aren’t up to

snuff.

tip Don’t confuse user stories with actual tasks. Team members decom-

pose selected user stories into tasks for each sprint. Don’t expect to

identify all of the tasks ahead of time.

epics: too Broad

As evinced by their name, epics are user stories that Scrum teams can-

not complete during a single sprint, let alone easily. Epics often com-

bine many smaller user stories into one. You may be wondering why

this is a problem. If a user story qualifies as an epic, then it falls prey to

many of the problems associated with Waterfall projects. Here are two

examples of epics:

1. As a supply-chain manager, I need to see all inventory data

throughout the company, receive alerts when inventory levels

for key products fall below certain thresholds, understand

my delivery network via geoanalytics, and view information

regarding potential substitutes if weather delays or union

issues prevent normal raw-material production.

2. As the head of training and development, I need to view

employee survey results in real time to immediately send

struggling managers e-mails equipped with links to resources

and online training programs. I also need to see which manag-

ers have taken which courses and how they performed.

In the first epic, there is no doubt that each of these tasks is

important to supply-chain managers. Ditto for the second scenario.

However, including each of these tasks in a single user story is a

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

96 ▸ A n A l y t i c s

recipe for disaster. It’s far better to separate each into several user

stories that Scrum teams can complete over the course of one or

more sprints.

too Narrow/Detailed

At the other end of the spectrum are user stories that are far too granu-

lar. They include:

1. As the head of sales, I need to view each prospect’s name, lead

origin, lead date, address, and zip code in an Excel spreadsheet

sorted by lead origin so I can run mail merge in Microsoft

Word.

2. As the head of analytics, I need the ability to run multivari-

ate regression equations and post the results on my company

website, allowing nontechnical users to import the results in

Tableau for future visualization and analysis.

Just right

The following user stories pass the Goldilocks Test—they are neither

too broad nor too narrow:

1. As the head of talent management, I need to view attrition

rates of high-potential employees to determine whether they

are leaving at an accelerated rate.

2. As a bookstore owner, I need to see real-time inventory levels

and trends to determine which books to order.

the Spike: A Special user Story

Regardless of their required effort or anticipated complexity, all user

stories inhere some degree of risk. Remember that Agile methods

explicitly reject the conceit of phase-gate ones: that a project manager

or sponsor knows precisely how long each item will take.

But risk is not binary; there are degrees. It’s foolish to take core

decisions about development frameworks, programming languages,

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 97

and statistical techniques lightly. Yes, they may not be permanent, but

they aren’t easily reversed.

In the case of big, risky decisions, spikes are very useful. These are

specific types of user stories that seek “to gain the knowledge neces-

sary to reduce the risk of a technical approach, better understand a

requirement, or increase the reliability of a [user] story estimate.”* As

Dean Leffingwell wrote in Agile Software Requirements:

Since spikes do not directly deliver user value, they should be used sparingly and with caution.

The output of a spike is demonstrable, both to the team and to any other stakeholders. This brings visibility to research and architectural efforts and also helps build collective ownership and shared responsibility for the key decisions being taken.

And, like any other user story, [the product owner accepts spikes] when the acceptance criteria for the spike have been fulfilled.†

For instance, let’s say that your team is trying to decide between

competing statistical software packages. On the table are R, SPSS,

and SAS Enterprise Miner. This is a big decision and the wrong one

will set the firm and the team back months. To avert that very sce-

nario, the product owner invests 40 hours of the team’s time in this

user story.

BAcklogS

Product and sprint backlogs underpin Scrum. A product backlog lists

all desired product features in the form of user stories. (These also

go by the names of backlog items or just plain stories.) The product

owner is formally responsible for the product backlog. By defini-

tion, it constantly evolves. Items at the top of the backlog tend to

be smaller and better defined than those at the bottom. The latter

qualify as nice to have.

* For more on this, see www.scaledagileframework.com/spikes. † We’ll get to acceptance criteria shortly.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://www.scaledagileframework.com/spikes

98 ▸ A n A l y t i c s

The sprint backlog serves as the team’s to-do list for the sprint.

As Figure 5.2 shows, the sprint backlog is a subset of the product

backlog.

Product Backlog

Sprint Backlog

figure 5.2 relationship between Product and Sprint Backlogs Source: Phil Simon.

Unlike the product backlog, the sprint backlog is finite. The team

generates this backlog during sprint planning and it remains the team’s

focus throughout the duration of the sprint.* Team members may

change the tasks required to complete these user stories. However, the

user stories themselves should remain constant.

SPrINtS AND meetINgS

In Scrum, a sprint represents the period in which a team completes a

predetermined number of user stories. Typically, a sprint lasts one or

two weeks. Figure 5.3 displays the general structure of a one-week

sprint.

At a high level, Scrum is designed to maximize speed, user accep-

tance, and, ultimately, the odds of successful outcomes. If this is true,

you may be asking, then why do its teams waste so much time in

meetings? Nothing could be further from the truth. Each meeting in

Figure 5.3 is intended to serve a specific and valuable purpose.

* Extreme Programming is another type of Agile development that allows for the sprint

backlog to change mid-sprint.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 99

Sprint Planning

Sprint planning usually takes about two hours. At its core, the meeting

seeks to achieve two things: a sprint goal and a sprint backlog. That’s it.

The product owner and team members shouldn’t develop and

agree on verbose and complicated goals. Rather, goals are ideally short,

one- or two-sentence descriptions of what the team plans to achieve

during the sprint.

The sprint backlog reflects the specific user stories that the team

commits to completing during the sprint. Team members should begin

thinking about the specific tasks they need to complete to deliver the

figure 5.3 Schedule for a One-Week Sprint Source: Phil Simon.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

100 ▸ A n A l y t i c s

agreed-on user stories. Note that the product owner doesn’t assign

tasks akin to what a traditional project manager would. Scrum is sup-

posed to be collaborative. Think conversations, not fiats.

Daily Stand-up

Also known as daily scrums, teams hold these mandatory 15-minute

daily meetings in the morning. These meetings should be brief and

focused. If you wonder why it’s called a stand-up, remember that meet-

ings tend to outlive their utility when people are comfortable (read:

sitting down).

Each team member should quickly answer the following three

questions:

1. What did you do yesterday?

2. What will you do today?

3. Are there any impediments in your way?*

Goals here include transparency and problem identification; the

stand-up is neither a traditional status meeting nor a technical dis-

cussion. Should team members start pontificating or losing focus, the

Scrum Master should intervene.

Story time

Also known as backlog grooming, this optional meeting typically takes

place when the sprint is a little more than halfway complete. The goal

is also straightforward: to maintain small and well-understood user

stories at the top of the backlog at all times. Without such user stories,

the team will likely lose momentum when the next sprint begins. Note

that the focus of this meeting should be upcoming user stories, not the

ones in the current sprint.

Demo

For Scrum team members, this is usually the most enjoyable part of

the sprint. Here the team shows off its work to stakeholders—that is,

* For a more detailed description, see http://bit.ly/2nkjNvm.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2nkjNvm

i n t r o d u c i n g s c r u m ◂ 101

people who do not work on the Scrum team. Goals include maximizing

transparency and gaining stakeholder trust. By showing each iteration

of the product development, stakeholders are less likely to be surprised

at the final release compared to phase-gate projects.

Sprint retrospective

Scrum beats phase-gate methods across the board, but it certainly isn’t

perfect—no method, project, or team is. Scrum is designed to practice

what it preaches: to not only launch the “product” more efficiently,

but also to improve the process and efficiency of the team.

In the second vein, the sprint retrospective is crucial. It attempts to

answer the following three high-level questions:

1. What did we learn during the sprint?

2. What went wrong during the sprint?

3. How can we improve next time?

Note that this meeting is not supposed to serve as a gripe session.

Scrum Masters should ensure that team members express all issues in

a professional way.

releASeS

Typically, each product or model release consists of several sprints.

That is, after two or three weeks of completing user stories, the team

has developed sufficient features to justify a new version. It’s time to

release the new iteration into the wild so others can enjoy the fruits of

the team’s labor.

Aside from regular or scheduled releases, other factors may come

into play when determining releases:

■ Date: E-commerce sites and apps pay particular attention to Cyber Monday, Mother’s Day, and Valentine’s Day.

■ Functionality: If a competitor launches a killer feature, a product owner may prioritize new user stories to staunch the

bleeding.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

102 ▸ A n A l y t i c s

■ Exigent circumstances: Users discover a key problem or bug that warrants immediate attention. This often pushes back pre-

viously scheduled user stories.

Each of these departures is acceptable. Unlike the Waterfall

method, Scrum is designed to handle unforeseen changes and crises.

eStImAtIoN tecHNIqueS

Generally speaking, Waterfall projects suffer from the difficulty of esti-

mating how long individual tasks—let alone entire phases—will take.

This dooms many projects from the start. For instance, will it take

three or four months to collect, clean, deduplicate, and load enterprise

data into a new business-intelligence tool or data warehouse? Even a

modest 20 percent delay in one step of a phase-gate project affects all

other phases, immediately rendering the project significantly behind

schedule.

Agile methods such as Scrum avoid this all-too-real scenario in

several ways. First, through sprints, Scrum teams work to complete

tasks in small batches, not large ones. In this case, the team first com-

pletes user stories around collecting data—and only a certain type of

data at that. Perhaps it first tackles customer data and then proceeds

to product or employee data. By breaking work into more manageable

parts, Scrum teams tend to be far more successful than their Waterfall

counterparts.

But Scrum teams excel for another reason: the superior nature of

their estimates. More specifically, Scrum teams rely on relative esti-

mates—not absolute ones. A simple, nontechnical example illustrates

the point.

on lawns and relative estimates

I recently moved into a new home in Arizona not too far from Arizona

State University’s Tempe campus. I had lived outside of Las Vegas,

Nevada, for the prior five years. My new backyard is about half as large

as my old one, and with real grass on half of it. (I prefer it to its artifi-

cial equivalent.) Figure 5.4 shows a simple diagram.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 103

Nevada Arizona

figure 5.4 Simple representation of Grass at author’s Former and Current home Source: Phil Simon.

As I prepared to mow my new lawn in March for the first time,

I didn’t know how exactly long it would take. I did know, however,

that it would take less time than it took for me to mow the lawn at my

Nevada home. In other words, I used a relative estimate and I was right.

This doesn’t make me exceptional; it makes me a human being.

We are generally terrible at making absolute estimates, but adept at

making relative ones. We can effectively differentiate between and

among items with one caveat: the items need to be sufficiently dis-

parate. If I were standing on the ground and looking up at two build-

ings, I wouldn’t necessarily be able to tell which one was taller; it all

depends on the height of the buildings and my proximity to them. In

Figure 5.5, the two buildings are dissimilar enough that, even in an era

of alternative facts, any sober adult can tell that the one on the left is

much larger.

figure 5.5 two Very Different Buildings Source: Phil Simon.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

104 ▸ A n A l y t i c s

But what about the buildings shown in Figure 5.6? Would you be

able to tell which one is taller if you were standing near the entrance

of one of them?

figure 5.6 two Very Similar Buildings Source: Phil Simon.

No, you would be guessing. The buildings aren’t sufficiently

different.

fibonacci Numbers

It turns out that this concept dates all the way back to an Italian math-

ematician named Leonardo of Pisa (aka Fibonacci). In his 1202 text

Liber Abaci, he introduced the eponymous Fibonacci sequence:

0 + 1 = 1 1 + 1 = 2 1 + 2 = 3 2 + 3 = 5 3 + 5 = 8 5 + 8 = 13 8 + 13 = 21

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 105

As you can see, Fibonacci numbers get big quickly; the numbers in

the sequence are meant to denote differences in a way that humans

can easily discern. For instance, 3 is 50 percent greater than 2. The

number 13 is 62.5 percent greater than 8. Figure 5.7 shows the first

few numbers of the sequence in a graphical form.

21 13

3 2 1 1

figure 5.7 Fibonacci Sequence Source: By —Own work, CC BY-SA 4.0.*

Note that in Scrum, Fibonacci numbers also go by the moniker

story points.

t-Shirt Sizes

Some Scrum teams prefer to use T-shirt sizes in lieu of Fibonacci num-

bers for relative estimates. In this case, small, medium, large, and extra

large replace 8, 13, 21, and so on. As we’ll see in Chapter 7, some of

my students did this very thing when working with the University

Tutoring Center on their semester-long capstone project.

* https://commons.wikimedia.org/w/index.php?curid=38708516.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

https://commons.wikimedia.org/w/index.php?curid=38708516

106 ▸ A n A l y t i c s

When teams Disagree

For several reasons, Scrum teams tend to be inherently more collab-

orative than their counterparts on Waterfall projects. For one, the for-

mer are more egalitarian. With only three roles, formal job titles don’t

matter—or at least they shouldn’t. As a result, there’s less chance for

politics, especially as teams congeal. Second, Scrum teams generally

consist of five to nine members. Finally, team members are supposed

to work with a great deal of autonomy. User stories describe what

employees want; they do not mandate how to deliver specific features.

That’s not to say, though, that team members don’t disagree from

time to time. This is fairly common with newer teams. When this hap-

pens, the teams typically play either Planning Poker or the Team Esti-

mation Game. Both are simple yet effective ways to resolve conflicts.

Planning Poker

Let’s say that Dinesh, Gilfoyle, and Nelson disagree on the number

of points to assign to a user story.* Table 5.1 represents their honest

estimates.

table 5.1 estimates for User Story Points

Team Member Estimate

Dinesh 8

Gilfoyle 5

Nelson 21

Source: Phil Simon.

Other team members include Erlich (product owner), Jared

(Scrum Master), and Richard, Russ, and Monica (all team members).

To start, each estimator receives separate cards with the estimates

of Dinesh, Gilfoyle, and Nelson. For the user story, Erlich reads the

description of the user story. He also moderates a discussion and

answers questions.

After discussion, Richard, Russ, and Monica each select cards rep-

resenting their guesses. The three then concurrently flip them over.

* This is a not-so-subtle nod to the hysterical HBO series Silicon Valley.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 107

Now all participants can see each other’s estimates. Table 5.2 shows

their selections.

table 5.2 Planning Poker, round 1

Team Member Estimate

Richard 5

Monica 8

Russ 21

Source: Phil Simon.

Because their estimates are the highest and lowest, respectively,

Richard and Russ explain their points of view. Erlich takes notes. After

discussion, the team repeats the process. As is often the case, the team

quickly reaches consensus, shown in Table 5.3.

table 5.3 Planning Poker, round 2

Team Member Estimate

Richard 5

Monica 5

Russ 21

Source: Phil Simon.

Team Estimation Game

Another way to quickly reach agreement involves placing user stories

in order of perceived difficulty or amount of work. This jibes nicely

with our innate ability to accurately make relative estimates, if not

absolute ones.

Erlich selects 20 stories from the product backlog. He writes them

down on index cards or Post-it notes. Finally, he distributes them

equally to the team members.

Dinesh begins the game by picking out a user story and placing it

on the whiteboard, arranged by perceived story size. Russ goes next

with his first user story. He thinks that his requires more points than

Dinesh’s, but Gilfoyle disagrees. A skilled coder, he can complete

it much quicker than Russ can. Russ moves his user story closer to

the left.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

108 ▸ A n A l y t i c s

This process repeats until something similar to Figure 5.8 eventu-

ally begins to take shape.

< Smaller Bigger > Too Big

figure 5.8 team estimation Game, round 1 Source: Figure from Phil Simon.

When everyone agrees with how the user stories rank, it’s time to

tidy up. The team completes the game by assigning Fibonacci num-

bers to each column. In the end, the whiteboard looks something like

Figure 5.9.

3 5 8 21

figure 5.9 team estimation Game, round 2 Source: Figure from Phil Simon.

It’s not uncommon for a Scrum team to breeze through 20 or more

user stories in an hour. As an added benefit, the game often creates a

positive dynamic among group members.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 109

otHer Scrum ArtIfActS, toolS, AND coNcePtS

Scrum teams not only benefit from employing superior estimation

techniques. Its practitioners enjoy a distinct advantage over their

phase-gate counterparts in the form of velocities, burn-down charts,

Kanban boards, and more. Let’s delve into each of them.

Velocities

A sprint’s velocity simply represents the total number of story points

from completed user stories. For instance, let’s say that a team com-

pletes four user stories during its first sprint. Table 5.4 shows the team’s

velocity for sprint number 1.

table 5.4 Sample Points for User Story and Sprint Velocity

User Story Points

1 5

2 8

3 5

4 3

Sprint Velocity 23

Source: Phil Simon.

This begs a chicken-and-egg question: How many points’ worth of

user stories should a team attempt to complete during a sprint?

It’s wise to start conservatively. There’s no sense in committing to

obscene estimates. After completing the first sprint, the team may still

not know how much it can realistically accomplish next time. If this

happens, it helps to use prior velocities as a guide, a technique called

yesterday’s weather. Finally, as teams cohere and members learn each

others’ strengths and weaknesses, velocities should increase over time.

Burn-Down charts

A burn-down chart graphically reflects each sprint’s velocities. Figure

5.10 shows a sample burn-down chart.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

110 ▸ A n A l y t i c s

300

225

150

0 Sprint 1

S to

ry P

o in

Sprint 2 Sprint 3 Sprint 4 Sprint 5

figure 5.10 Generic Burn-Down Chart Source: Figure from Phil Simon.

Figure 5.10 is simple by design. It does not show any increases

in total story points associated with the project or launch. In reality,

though, this happens frequently as a product owner adds more user

stories to the product backlog. Remember that Scrum does not require

teams to gather all user stories prior to commencing.

Also note in Figure 5.10 that the slope of the line increases over

time—technically, the slope becomes more negative. This happens

because, as mentioned earlier, team velocities tend to increase over time.

Definition of Done and Acceptance criteria

Your team has completed a task, but how do you really know that it’s

really done? More important, does everyone agree that that task is, in

fact, finished?

For this very reason, a shared definition of done is essential. In

the software-development world, it usually means that something is

ready to ship. This is an end, and done probably includes both code and

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

i n t r o d u c i n g s c r u m ◂ 111

design reviews as well as different types of testing. Let’s say that a team

member completes a user story by adding some new code. That addi-

tion, however, breaks other key features of a product. Is that new user

story really done after all?

Think of acceptance criteria as cousins of done. Moreover, they are

the conditions that a software product must satisfy to be accepted by

a user, customer, or a receiving system. Note that you can’t get a little

bit pregnant. There is no such thing as partial acceptance: a criterion is

either met or it is not.

kanban Boards

On long Waterfall projects, project managers typically relied on Micro-

soft Project and very involved Gantt charts. Together, they offered

the illusion of predictability and control. I can’t say that either tool is

inherently unhelpful or confusing, but neither allows team members

to easily understand what’s going on.

By way of contrast, a simple Kanban board can effectively convey the

status of user stories and tasks on a sprint.* Figure 5.11 shows a sample.

To Do DoneIn Progress Tested

figure 5.11 Sample Kanban Board for analytics Project Source: Phil Simon.

* For more on Kanban boards’ many benefits and the rationale behind using them, see

http://bit.ly/2mNVCn6.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2mNVCn6

112 ▸ A n A l y t i c s

Many Scrum teams use physical Kanban boards made up of dif-

ferent colors of index cards. This type of decidedly low-tech method

is especially common when team members are colocated. It may

seem odd, but individuals often enjoy the process of physically mov-

ing a task or user story from the “in process” column to the “done”

column. For distributed or virtual teams, many project-manage-

ment applications such as Trello* allow the same movement among

columns.

ChaPTEr rEViEw and diSCUSSion QUESTionS

■ What are the three roles on Scrum teams? What does each do?

■ What are user stories? Why are they important? What are their three parts?

■ Which is more accurate: relative or absolute estimates? Why?

■ What is a burn-down chart? What should its general direction be?

■ Should sprint velocities increase or decrease over time? Why?

■ Why are Kanban boards helpful?

Chapters 4 and 5 introduced Agile methods and Scrum. Before con-

cluding Part Two, it’s time to broach one last important concept: a

framework for tying this all together.

* See http://bit.ly/2nbAN8Q.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2nbAN8Q

113

c06 113 31 May 2017 9:26 AM

C h A p t e r 6 A Framework for Agile Analytics A Simple Model for Gathe ring Insights

If I can’t picture it, I can’t understand it. —Albert Einstein

A t least in the software-development world, Agile methods are old

hat today. Companies such as Amazon, Google, Facebook, Apple,

Twitter, Microsoft, and countless others in the technology sector

have long recognized the superiority of Scrum compared to the Water-

fall method. Based on the success of these companies and the need to

adapt quickly to a remarkably dynamic business environment, Agile

methods have penetrated other industries.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

114 ▸ A n A l y t i c s

c06 114 31 May 2017 9:26 AM c06 115 31 May 2017 9:26 AM

Founded in 1892 in Schenectady, New York, General Electric

(GE) is one of the oldest and most storied enterprises in the world.

In a way, though, that doesn’t matter. As former executives at Black-

Berry, Kodak, and Blockbuster can attest, previous success does not

guarantee future success. To adapt to the realities of the twenty-

first century, GE’s management recognized the need to get with the

times—and, increasingly, this means adopting Agile practices, such

as Scrum.

Consider Brad Surak, now GE Digital’s Chief Operating Officer

(COO). Surak began his career as a software engineer. As such, he

was intimately familiar with Agile. He piloted Scrum with the leadership team responsible for developing industrial Internet applications and then, more recently, began applying it to the new unit’s management processes, such as operating reviews.1

Although the notion of Agile analytics is relatively new, it is quickly

gaining steam. As Part Three shows, organizations are using data and

analytics to solve a wide variety of business problems. Before we arrive

at proper case studies, we’ve got some work to do.

This brief chapter provides a simple and general framework for

gleaning insights in an iterative or Agile fashion. The framework dis-

played in Figure 6.1 seeks to avoid the costly mistakes of Waterfall

analytics projects.

Even at a high level, the intent here should be obvious: You

should not attempt to analyze every conceivable data source in one

large batch. You’ll be much better served by completing a series of

smaller batches. Ditto for spending months attempting to build the

perfect model.

tip Don’t try to boil the ocean.

Let’s cover each of these steps in a fair amount of detail.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 115

c06 114 31 May 2017 9:26 AM c06 115 31 May 2017 9:26 AM

PerForm Business Discovery

Analytics doesn’t exist in a vacuum. Sure, at some intellectual level

you may wonder why your customers are churning or you can’t accu-

rately predict inventory levels at your company. Still, at this point,

hopefully you are attempting to solve a real business problem, not

conduct an interesting but largely academic exercise.

To that end, you should start with key questions such as the

following:

■ What are we trying to achieve?

■ What behavior(s) are we trying to understand, influence, and/

or predict?

Figure 6.1 A Simple Six-Step Framework for Agile Analytics Source: Model adapted from Alt-Simmons’ book Agile by Design. Figure created by Phil Simon.

Perform Business Discovery

Evaluate and

Improve

Score and

Deploy

Model Data

Prepare Data

Perform Data

Discovery

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

116 ▸ A n A l y t i c s

c06 116 31 May 2017 9:26 AM c06 117 31 May 2017 9:26 AM

■ What type of data would we need to address these issues?*

■ Is the project even viable?

■ Does our organization possess the time, budget, and resources

to undertake the project?

■ Is our organization committed to the project? Or will the

project fade into the background as more important priorities

crop up?

■ What happens if we don’t answer these questions? What if the

project takes longer than expected?

At this point, members of your team may very well disagree about

the answers to some of these questions. For instance, not everyone

may concur about whether the project is even viable. Disagreement is

healthy as long as it is respectful.

To assess the viability of any analytics endeavor, it’s wise to hold

discovery workshops. These brainstorming sessions can flush out

ideas. Perhaps Saul is a skeptic because he saw a similar organizational

project fail five years ago. He doesn’t realize that new leadership, tech-

nologies, data sources, and business realities have changed the game.

Maybe Penny is a Pollyanna because this is her first project and she

just assumes that everyone will follow her lead.

Resist the urge to skip this step. Next, try to start with a testable

hypothesis or working theory of why a problem is occurring. For

instance:

■ Initial hypothesis: Customers are leaving because our prod- ucts are too expensive.

■ Null hypothesis: Customers are not leaving because our prod- ucts are too expensive.

Don’t worry about completely answering this question from the

get-go. Remember that this is a cycle. You’ll have plenty of time to

introduce additional hypotheses, variables, and data sources. Odds are

that a single simple hypothesis won’t explain the entirety of the busi-

ness problem that you’re addressing in this stage anyway.

* We’ll get to whether that data exists in the next phase.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 117

c06 116 31 May 2017 9:26 AM c06 117 31 May 2017 9:26 AM

PerForm DAtA Discovery

If you’re of a certain age, as I am, you remember a much different

data landscape 20 years ago. Across the board, individuals and compa-

nies accessed far less data when making decisions. In a way, this made

decision making easier. For instance, employees didn’t have to worry

about collecting and analyzing data from social networks because

they didn’t exist. The World Wide Web was just getting started. You

couldn’t answer as many questions as comprehensively as you can

today—at least in theory.

Today, we have the opposite problem. The arrival of Big Data

means that discovery has never been more important. Critical data-

related questions include:

■ Where does the desired data “live”?

■ Is it even available?

■ Is it legal to use? Is it free to use?

■ Are we able to retrieve the data in a clean and usable format?

Or do we need to scrape it using one of the tools mentioned

earlier? (See “Getting the Data” in Chapter 2.)

■ Is use of the data restricted? (For instance, Twitter limits access

to its firehouse. The company intentionally throttles users who

attempt to access too much data, especially first-time users.)

■ Can you pay to circumvent those restrictions? How much?

■ How long will it take to access/acquire the data?

■ How old is our data? Has it aged well?

■ If the data exists inside of the enterprise, which organizations

and departments own the data? Are they willing to share it with

you? (Don’t assume that the answer is yes.)

■ Is the data complete, accurate, and deduplicated?

At this point, it’s wise to remember Douglas Hofstadter’s wonderfully

recursive law: “It always takes longer than you expect, even when you

take into account Hofstadter’s Law.”* Avoid committing to overly aggres-

sive timelines for analytics. Remember that perfect is the enemy of good.

* For more, see http://bit.ly/2m2p7pF.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2m2p7pF

118 ▸ A n A l y t i c s

c06 118 31 May 2017 9:26 AM c06 119 31 May 2017 9:26 AM

Also, know going in that it’s unlikely that you’ll solve your prob-

lem via a single data source, no matter how robust or promising it is.

Start with what you know, but expect surprises. If a finding doesn’t

surprise you at some point, then you’re probably not looking hard

enough. Recognize that you’re never going to get all the desired data.

Finally, it’s wise at this point to digest the data that you have unearthed

for a little while. Remember spikes (which are discussed in Chapter 5).

Yes, you can always restart your efforts, but you won’t recoup the time.

Agile methods such as Scrum don’t include time machines.

PrePAre the DAtA

Odds are that your data will contain at least a few errors, inconsisten-

cies, and omissions, especially at first. Data quality isn’t sexy, but it’s a

really big deal. In fact, data preparation may take a great deal of time

and prevent you from getting started in earnest. You’ll probably need

to parse, scrub, collect, and manipulate some data. Consider the fol-

lowing example.

In one of my Enterprise Analytics classes, a group of my students

agreed to help a local retail business analyze its data against industry

benchmarks. (I call the small business A1A here.) My students thought

that they would be receiving pristine data in a format to which they

had become accustomed. In other words, they thought that A1A’s

data would be transactional (i.e., long, not wide). Table 6.1 shows the

expected format.

As Table 6.1 shows, each transaction exists as a proper record

in a sales table. (This is the way that contemporary systems store

transactional data.) Lamentably, my students learned that A1A kept

its data in the antiquated format demonstrated in Figure 6.2.

table 6.1 expected Client Data

Customer_ID PurchDate PurchAmt ProductCode

1234 1/1/08 12.99 ABC

1234 1/19/08 14.99 DEF

1234 1/21/08 72.99 XYZ

Source: Phil Simon.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 119

c06 118 31 May 2017 9:26 AM c06 119 31 May 2017 9:26 AM

Table 6.2 contains the same data as Table 6.1, but this isn’t a

potato-po-tah-toe situation. Each table represents the data in a very

different way. Note that storing data in this manner was much more

common in the 1980s. (For more on this, see “How Much? Kryder’s

Law” in Chapter 1.)

table 6.2 Actual Client Data

Customer_ID Purch Date1

Purch Amt1

Product Code1

Purch Date2

Purch Amt2

Product Code2

1234 1/1/08 12.99 ABC 1/19/08 14.99 DEF

1235 1/1/12 72.99 XYZ 1/19/08 14.99 DEF

1236 1/1/08 12.99 ABC 1/19/08 72.99 XYZ

Source: Phil Simon.

By way of background, A1A hired temps to manually enter its sales

data in Microsoft Excel. Not unexpectedly, the temps lacked a back-

ground in system design and data management. As such, they kept add-

ing new columns (technically, fields) to the spreadsheet. This may seem

like an inconsequential difference, but from a data perspective, it most

certainly was not. If a customer booked 200 sales over the years with

A1A, then the spreadsheet would contain more than 600 different fields

with no end in sight. While the current version of Excel supports more

than 16,000 different fields,* A1A’s data was, quite frankly, unwieldy.

My students had to wade through hundreds of columns to trans-

form the data into a far more usable and current format. Transpos-

ing data is time consuming. What’s more, this was not the only issue

related to the data’s structure. As a result of their discoveries, the stu-

dents spent the majority of the semester rebuilding A1A’s database

from scratch. They couldn’t get to what they considered the good stuff

(read: the analytics) until the very end of the project.

When preparing data for analytics, ask yourself the following key

questions:

■ Who or what generates the data? (Remember from Chapter 1

the burgeoning Internet of Things. A machine may generate the

data, but that doesn’t mean that the data is completely accurate.)

* See http://bit.ly/2nu46RU.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2nu46RU

120 ▸ A n A l y t i c s

c06 120 31 May 2017 9:26 AM c06 121 31 May 2017 9:26 AM

■ If people are responsible for generating the data, are they trained

in how to enter it properly? Was there turnover in the organiza-

tion that could introduce inconsistencies and errors?

■ Is the data coming directly from the system of record or from

another source, such as a data mart or data warehouse?

■ How is the data currently generated and has that ever changed?

■ How much data is generated?

■ What if the data is flawed or incomplete? What are the

downsides?

■ Is certain data absolutely required to proceed? What types of

proxies can we use if we are left with no choice?

■ How complex is the data?

■ How frequently is the data updated?

tip Often, heat maps, simple SQL statements, pivot tables, histograms, and

basic descriptive statistics can provide valuable insights into the state

of your data. A day of data preparation may save you six weeks’ time

down the road.

moDel the DAtA*

Many professionals are afraid of building models, and some of my

students are a little apprehensive as well. I understand the hesita-

tion. After all, it sounds a little daunting. What happens if you get

it wrong?

Here’s the rub: As George E. P. Box once said, “Essentially, all

models are wrong, but some are useful.”

tip The question isn’t whether a model is completely accurate; no model

is. The real question hinges on whether a model is useful.

* Note that I use the terms forecasting, modeling, and predictive analytics interchangeably.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 121

c06 120 31 May 2017 9:26 AM c06 121 31 May 2017 9:26 AM

At a high level, the goal of any model is to understand, describe,

and/or predict an event. (See “Types of Analytics” in Chapter 3.) This

holds true whether you are trying to predict customer churn, the most

relevant search results, or, as we’ll see shortly, basketball outcomes.

Taking a step back, you want to know the following:

■ Which variables are important

■ The absolute and relative importance of these variables

■ Which variables ultimately don’t matter

In a business context, most models lead—or at least should lead—to

specific actions designed to improve business outcomes. To this end, the

model may focus on customers, prospects, employees, users, or partners.

A d v i c e o n B u i l d i n g M o d e l s I’m fond of metaphors. For instance, IT projects are like landing planes. It’s best to slowly ease into a landing, not try to stop on a dime.

Along the same lines, I often analogize data and analytics endeavors to building houses. Let’s say that you hire an architect to build a duplex. As your house starts to become a reality, you decide that you don’t need a fourth bedroom after all. Remove the closet and voilà! It’s an office.

Most structural changes, however, aren’t feasible past a certain point. There’s no way to inexpensively turn that duplex into a ranch without starting from scratch. Keep the house analogy in mind when creating models.

Beyond that, ensure that your modeling software handles additional volumes and types of data. What’s more, your program of choice should accommodate additional complexity should you choose to add it. Different tools and programming languages are better suited for different data types and sizes than others.

the Power of a simple model

Many books tackle building models, and I won’t attempt to summarize

them here. (Remember, this book emphasizes breadth over depth.)

For now, heed the following advice: It’s best to start simply, especially

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

122 ▸ A n A l y t i c s

c06 122 31 May 2017 9:26 AM c06 123 31 May 2017 9:26 AM

at first. Fight the urge to overcomplicate initial models. As the follow-

ing anecdote illustrates, models need not be terribly sophisticated to

bear fruit.

R e l i v i n g t h e 1 9 9 7 n c A A t o u R n A M e n t For decades, millions of Americans have filled out brackets for the National Collegiate Athletic Association (NCAA) Men’s College Basketball Tournament. In 1997, the World Wide Web was exploding and ESPN.com made “bracketology” easier than ever. For the first time, you could submit your brackets online and see how your picks compared against those of everyone else in the world. Oh, the technology!

At the time, I was finishing up my graduate degree at Cornell University. I didn’t know much about college basketball, but I knew a few things about data and building models. With graduation looming and my future full-time job secured, I wasn’t lacking for free time. (Ithaca, New York, is chilly in March, and I don’t ski.) I asked a bunch of my friends if they were interested in participating in a different type of NCAA pool.

Ten of them agreed, and on Wednesday, March 12, 1997, we drafted players in a serpentine order. (The 64-team tournament started on Thursday.) Under snake drafts, the person with the first pick selects and then waits until the end of the second round for his next slot—the 20th overall. He would then pick the 21st player and then not again until the 40th selection. The person with the second pick would have to wait until the 19th to go again and so forth.

Tournament Rules

Rather than attempt to predict which teams advanced in each round, I asked my friends if they wanted to kick in $15 each in a player-based tournament.

Players’ scores would represent the total of their points, rebounds, and assists. Players would accumulate points as long as their teams remained alive. If their teams lost, then they could no longer add to their teams’ totals because they no longer suited up. Whichever team had the highest combination of points at the end of the tournament would win the $150 prize. All points, rebounds, and assists counted as one point each. A 20-point scorer would count for as many points as a 10-point scorer who grabbed 10 rebounds per game.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 123

c06 122 31 May 2017 9:26 AM c06 123 31 May 2017 9:26 AM

Draft Strategy

In their quests to win the prize, my friends employed very qualitative approaches. That is, they picked the players whom they recognized, even if they played for teams expected to lose in the first round.

I did not.

I didn’t care if a player was fundamentally better than any other. I only cared about expected values. For instance, a star like Tim Duncan of Wake Forest (a number-three seed) was certainly valuable, but I only expected him to play three games. In the third round, Wake Forest would most likely play a number-two seed. (Although exceptions abound, higher-seeded teams have tended to beat lower-seeded ones throughout the tournament’s history.)

Getting the Data

I downloaded each player’s season statistics from ESPN.com and built a simple model in Microsoft Excel. I then started ranking players. For instance, let’s say that Duncan averaged 20 points, eight rebounds, and five assists per game over three tournament games. His expected value would be 99: [(20 + 8 + 5) × 3]. Duncan ultimately played only two tournament games that year as Stanford (a number-six seed) knocked off Wake Forest 72–66. Duncan only accounted for 92 points.* Bawdy numbers, but hardly worthy of a top-three pick. (My model projected that he would account for 116.1 total points over three games, 26.1 percent lower than his final score.)

If this sounds complicated, I assure you that it wasn’t. It took me a little more than an hour to download the data, build my model, and project player expected values.

The Draft: My Model in Action

With my first-round pick I grabbed Bobby Jackson, a guard from the University of Minnesota. No surprise there. Jackson averaged 19.4 points, 7.4 rebounds, and three assists that year. Most important, the Golden Gophers earned a number-one seed in the Midwest Regional. I

(Continued )

* In two games, Duncan actually averaged 22 points, 22 rebounds, and two assists.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

124 ▸ A n A l y t i c s

c06 124 31 May 2017 9:26 AM c06 125 31 May 2017 9:26 AM

expected Jackson to play at least five games for me, and in the process, accrue a boatload of points.

In the fifth round, I selected Scott Padgett of Kentucky. Padgett was nowhere near as skilled as Jackson, never mind Duncan, a college stud and future NBA immortal. Still, Padgett’s stats were, put kindly, understated. In the 1996–1997 season, over 32 games, he averaged 9.6 points, 5.1 rebounds, and 1.8 assists (16.5 total). This was respectable, but hardly worthy of a first- or even third-round pick. Best of all, Padgett flew well under my friends’ radar.* He should not have even been available when I snagged him in the fifth round.

I had my eye on Padgett, though, because that year Kentucky was ranked first in the West Regional. Number-one seeds are the most likely to advance to the Final Four. In so doing, they play five games minimum and more if they advance to the championship game. If Padgett only suited up for five games, my model predicted that he would net me 82.5 total points. (Remember that, all else being equal, more games equal more points.)

When I grabbed Padgett with my fifth-round pick, my friends chuckled. After all, Padgett was decent but hardly an elite player. No bother. I didn’t care about his objective basketball skills; I only cared about his projected points under my model. (For my part, I was silently laughing at their silly picks; they grabbed players with high name recognition but low expected values.)

Why I Will Buy Scott Padgett Dinner if I Ever Meet Him

It turns out that I had the last laugh with my friends. My remarkably simple model netted me the $150 prize.

In 1997, Kentucky lost to Arizona in the NCAA Championship. Padgett played in seven total games. In the process, he netted me 129 points, nearly 56 percent more than what my model had predicted. (I didn’t even need his final-game contribution in the Arizona game; I had cinched my victory and all the other players on my friends’ teams had been eliminated.)

* Not that I know nearly as much as he does about analytics, but it turns out that Billy Beane was employing a similar philosophy at the same time as general manager of the Oakland A’s.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 125

c06 124 31 May 2017 9:26 AM c06 125 31 May 2017 9:26 AM

Padgett was my most valuable player not because of his total points, but because of what he cost me. (Remember, he was my fifth-round pick.) Figure 6.2 shows the performance of Duncan, Jackson, and Padgett relative to my model.

Figure 6.2 player extra Value Source: Data from Basketball-reference.com.

Scott PadgettTim DuncanBobby Jackson –25.00%

0.00%

25.00%

50.00%

75.00%

There are two morals of my little yarn. First, models need not be

complicated to be effective. Why not start with Occam’s razor? Sec-

ond, to succeed you still need to get a little lucky. As Branch Rickey

once wrote, “Luck is the residue of hard work and design.”

Forecasting and the human Factor

Up until now, this book has emphasized the decidedly nonhuman

components of data and analytics. To be sure, awareness of data types

and structures, the different types of analytics, and the framework dis-

cussed in this chapter are critical. Put differently, absent this knowl-

edge it’s nearly impossible to attain any sustainable level of success with

analytics. (Of course, there’s always dumb luck.)

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

126 ▸ A n A l y t i c s

c06 126 31 May 2017 9:26 AM c06 127 31 May 2017 9:26 AM

Before proceeding, it’s critical to accentuate a decidedly human

point. A note from the Introduction bears repeating: Data and analyt-

ics generally do not make decisions by themselves. We human beings

do, and we can act in a deliberate way that will maximize our odds

of success. (Part Three shows how much leadership and openness to

analytics drive successful outcomes.)

Philip Tetlock is an Annenberg University Professor at the Uni-

versity of Pennsylvania. For 20 years, he studied the accuracy of

thousands of forecasts from hundreds of experts in dozens of fields.

His 2005 book Expert Political Judgment examined why these alleged

“experts” so frequently made wildly inaccurate predictions in just

about every field.*

In 2011, Tetlock began The Good Judgment Project along with

Barbara Mellers and Don Moore. The multiyear endeavor aimed to

forecast world events via the wisdom of the crowd. Think of them as

forecasting tournaments with mystifying results: Predictions from nonex-

perts were “reportedly 30 percent better than intelligence officers with

access to actual classified information.”2

understanding superforecasters

Intrigued by this discovery, Tetlock and Dan Gardner wrote a 2015

follow-up book, Superforecasting: The Art and Science of Prediction. Tet-

lock wanted to know why a very small percentage of people rou-

tinely performed exceptionally well, even in areas in which they

lacked any previous knowledge. He called this group of people

superforecasters.

Table 6.3 shows some of the differences between superforecasters

and their regular brethren.

Brass tacks: Let’s say that you need to solve a problem of relative

sophistication. You give the same data to groups of superforecasters

and vanilla experts. As a result of their mind-sets, the former is far

more likely to produce a superior solution than the latter. To para-

phrase Isaiah Berlin’s essay, foxes are better than hedgehogs.

* This was the basis for one of my favorite talks in my public-speaking days, “I’m an

Expert. Don’t Trust Me.” Watch it at http://bit.ly/2nywOV6.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/2nywOV6

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 127

c06 126 31 May 2017 9:26 AM c06 127 31 May 2017 9:26 AM

score AnD DePloy

My NCAA Tournament model described earlier was simple on two

levels: First, it didn’t require anywhere near the data or statistical tech-

niques required to predict a complex business, economic, or medical

outcome. Second, I was testing a discrete event, not an ongoing pro-

cess. That is, my model had no use beyond March Madness in 1997,

although I could have refined it over time.

The vast majority of business models couldn’t be more different

than my little—albeit effective—Excel spreadsheet. Because these

forecasts attempt to describe and predict ongoing events of far greater

complexity, they need to evolve over time. Customer churn, employee

attrition, and credit-card delinquency rates don’t end when a team

wins a trophy and cuts down the net.

The score-and-deploy phase begins the process of assessing the

viability of the model. Questions may well include:

■ Is your model working well? How well and how do you really

know?

■ Are you measuring what you sought to measure?

■ Even if you’re looking at the right (independent) variables, are

their weights appropriate?

table 6.3 regular Forecasters versus Superforecasters

Regular Forecasters Superforecasters

Myopic and provincial. They tend to start with an inside view and rarely look outside. These folks generally can’t get away from their own predispositions and attachments.

Ignorant in a good way. They tend to start with an outside view and slowly adopt an inside view. That is, they look heavily at external data sources, news articles, and crowd indicators, especially as starting points.

Lazy. They doubt that there is interesting data lying around.

Stubborn. They believe strongly that there is interesting data lying around, even if they can’t quickly find it.

Tend to rely on informed hunches and make the data conform to those hunches.

Engage in active, open-minded thinking. They go wherever the data takes them, even if it contradicts their preexisting beliefs.

Tend to believe in fate. Tend to reject fate and understand that someone has to win. Examples here include lotteries, markets, poker tournaments, and so on.

Source: Principles from Superforecasting: The Art and Science of Prediction by Philip Tetlock and Dan Gardner. Table from Phil Simon.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

128 ▸ A n A l y t i c s

c06 128 31 May 2017 9:26 AM c06 129 31 May 2017 9:26 AM

■ How confident are you in your predictions?

■ Knowing that you’ll never reach complete accuracy, what is an

acceptable level of uncertainty?

It’s important to note that you might not have access to a pristine

and comprehensive dataset to run through your model. At some point,

odds are that you will have to decide between including a smaller but

cleaner dataset and a larger but impure one. Finally, keep an eye out for

tactical and operational issues. If people aren’t adhering to your model’s

recommendations for whatever reason, it will ultimately suffer.

evAluAte AnD imProve

You’ve developed the first or fifth iteration of your model and want to

see if it’s describing or predicting what you expected. It’s now time to

audit your model. At a high level, model updates take one of the fol-

lowing three forms:

1. Simple data refresh: In this case, you replace a model’s exist- ing dataset with a different one. The new dataset may include

newer records, older ones, or a combination of both.

2. Model update: This is a complete or partial rebuild. (This may entail new variables and associated weights.)

3. Combination: This method fuses the first two. That is, you sig- nificantly alter the model and run a different dataset through it.

Again, it’s hard to promulgate absolute rules here because cer-

tain events are much harder to explain—let alone predict—than others.

A model that explains 20 percent of the variance of a complex issue

might exceed expectations, while one that explains 65 percent may be

woefully inadequate.

Questions here typically include:

■ What data sources are you missing? Which ones are worth

including?

■ Which data sources may dry up? What will you do if that hap-

pens? (It’s a mistake to assume that a source will be freely avail-

able forever just because this is the case today.)

■ Which data sources should you retire?

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

A F r A m e w o r k F o r A g i l e A n A l y t i c s ◂ 129

c06 128 31 May 2017 9:26 AM c06 129 31 May 2017 9:26 AM

■ Which weights need adjusting? By how much?

■ Will your model improve or diminish if you make fundamental

changes?

■ What are the time implications?

■ What happens if you make a big mistake? (A major risk to a

company’s core product or service is very different than one to

some “moonshot.”)

I don’t know the answers to questions such as these for your

organization’s specific problems. Regardless of what you’re trying to

achieve, though, it’s imperative to regularly review your models for

efficacy. Put differently, disavow yourself of the “set-it-and-forget-

it” mind-set. As described in the Introduction (see “Analytics and the

Need for Speed”), the world changes faster than ever today. At a bare

minimum, complacent companies risk missing big opportunities. In

the extreme, they may become obsolete. Before leaving his post as

CEO at Cisco Systems, John Chambers gave a keynote speech in which

he predicted that 40 percent of today’s companies will “not exist in a

meaningful way in 10 years.”3

tip After completing the cycle, it’s time to repeat it. Ideally, you have

developed better questions than you asked the first time and even a

few answers.

CHAPTER REvIEW AnD DISCuSSIon QuESTIonS

■ What are the six steps in the framework for Agile analytics?

■ Why is it essential to complete every step in the framework?

■ Why is business discovery so essential?

■ Do models need to be complicated to be effective? Why or why not?

■ Are experts particularly adept at making accurate predictions? Why or why not?

■ What are the personality characteristics that make for better forecasting?

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

130 ▸ A n A l y t i c s

c06 130 31 May 2017 9:26 AM

Part Two has covered the essentials of analytics and one specific Agile

method: Scrum. It also provided a general framework for performing

analytics. It’s now time to move from theory to practice.

Part Three details a number of organizations’ efforts to make sense

of data and deploy analytics. Yes, it’s case-study time. As we’ll soon

see, with analytics, moving from theory to practice is often easier said

than done.

notes

1. Darrell Rigby, Jeff Sutherland, and Hirotaka Takeuchi, “Embracing Agile,” Harvard Business Review, May 2016, http://bit.ly/23JbM5k.

2. Alix Spiegel, “So You Think You’re Smarter Than a CIA Agent,” Parallels, NPR, April 2, 2014, http://n.pr/231BDSa.

3. Julie Bort, “Retiring Cisco CEO Delivers Dire Prediction: 40% of Companies Will Be Dead in 10 Years,” Business Insider, June 8, 2015, https://read.bi/1HmHyIU.

C op

yr ig

ht ©

2 01

7. J

oh n

W ile

y &

S on

s, In

co rp

or at

ed . A

ll rig

ht s

re se

rv ed

http://bit.ly/23JbM5k

http://n.pr/231BDSa

https://read.bi/1HmHyIU