The Executive Dashboard and the Trinity Mindset

JL12345

Web_Analytics_An_Hour_a_Day_----_Chapter_11_Month_6_Three_Secrets_Behind_Making_Web_Analytics_Actionabl....pdf

Home >Computer Science homework help >The Executive Dashboard and the Trinity Mindset

263

■ M

O N

T H

6 : T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

A N

A L

Y T

IC S

A C

T IO

N A

B L

Month 6: Three Secrets Behind Making Web Analytics Actionable

There was a wonderful catch phrase in the early

’90s: D2I, or data to information. It stressed that

data itself was not very valuable unless it was

converted into information. It represented the

very wise thought that we should not simply stop

at the implementation of analytics tools (any ana-

lytics tools) but strive to put extra effort into con-

verting that data into information that is useful

for decision making.11

30652c11.qxd 5/1/07 12:48 PM Page 263

Kaushik, Avinash. Web Analytics : An Hour a Day, John Wiley & Sons, Incorporated, 2007. ProQuest Ebook Central, http://ebookcentral.proquest.com/lib/apus/detail.action?docID=302273. Created from apus on 2020-09-07 12:30:22.

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

264

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Since that time, the challenge of converting data has become exponentially more complex because we have so much more data and business online or offline has become a lot more complex as well.

Although we have gotten better over time at trying to capture an increasing amount of data and processing it into information, it has been observed that there is a much greater challenge to deal with: inducing decision makers to take action on the information (insights) being provided from the data.

Week 1: Leveraging Benchmarks and Goals in Driving Action

Moving the mountain of actionability is a complex challenge that encompasses people, mindsets, roles, responsibilities, existing processes, social environments, vested interests, internal politics, and much more. All those variables, which you are already aware of, should give you a glimpse into how hard it is to motivate organizations to take action. On a scale of 1 to 10 (with 10 being the maximum challenge), collecting data is a 3.5, converting data into insights is a 7, and getting your boss and your boss’s boss and your boss’s boss’s boss to take action is an 18.5. That about sums it up.

You will spend the first week underscoring how you can leverage benchmarks and goals to drive action in your organization (this can apply to your web analytics program or any other analytics program).

Monday and Tuesday: Understand the Importance of Benchmarks and Setting Goals

According to Wikipedia, “Benchmarking (also best practice benchmarking or process benchmarking) is a process used in management and particularly strategic manage- ment, in which organizations evaluate various aspects of their processes in relation to best practice, usually within their own sector. This then allows organizations to develop plans on how to adopt such best practice, usually with the aim of increasing some aspect of performance.”

The power of benchmarking is that it allows you to look at your performance from a different perspective, not your own. You can consider the performance of your metrics within this other context that helps you understand how you are doing and enables you to take actions.

But perhaps the most amazing thing about benchmarking is that suddenly it is not you (or your boss or your tool) making a judgment on the data. You are able to present data in the context of a neutral party, which often is conducive to move deci- sion makers from questioning you or poking holes in your analysis to looking at the data and making decisions (because it is not about the messenger anymore, it is about the message).

One of the primary keys to driving action is context (for example, via use of rel- evant benchmarks). When you present a table or a graph or a bulleted insight in an

30652c11.qxd 5/1/07 12:48 PM Page 264

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

265

■ W

E E

K 1

: L E

V E

R A

G IN

G B

E N

C H

M A

R K

S A

N D

G O

A L

S IN

D R

IV IN

G A

C T

IO N

email without putting the information in context, it is less likely to induce action. The following is an example of website performance in December 2006:

• Number of visits: 173,539

• Number of unique visitors: 89,559

• Average time on site: 130 seconds

• Page views per visitor: 3.1

These are standard metrics that do tell us something but they don’t induce action because the context is missing. What do we compare these numbers to? Are they good or bad? What action can I take?

Consider the golden rule of actionability: Never present metrics without con- text—some context, any context.

You can simply segment the data at a top-line level to provide that context. Here is how it would look:

• Number of visits:

• All traffic: 173,539

• PPC campaigns: 77,140 (44 percent! Really high.)

• Number of unique visitors:

• All traffic: 89,559

• PPC campaigns: 39,695 (roughly the same repeat visitors)

• Average time on site:

• All traffic: 130 seconds

• PPC campaigns: 180 seconds (nice!)

• Page views per visitor:

• All traffic: 3.1

• PPC campaigns: 4.1 (really nice!) You can see how at the end of this second effort, you would have informed deci-

sion makers a bit better with just a little more context. Another method is to apply trends to the numbers that would give you more

context. Here is how that would look:

• Number of visits:

• All traffic: Nov: 33,360 Dec: 173,539

• PPC campaigns: Nov: 14,323 Dec: 77,140

• Number of unique visitors:

• All traffic: Nov: 19,000 Dec: 89,559

• PPC campaigns: Nov: 8,493 Dec: 39,695

30652c11.qxd 5/1/07 12:48 PM Page 265

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

266

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

• Average time on site:

• All traffic: Nov: 131 seconds Dec: 130 seconds

• PPC campaigns: Nov: 178 seconds Dec: 180 seconds

• Page views per visitor:

• All traffic: Nov: 3.0 Dec: 3.1

• PPC campaigns: Nov: 3.9 Dec: 4.1

Notice that with each step, we are getting more context in which we can think. At the end of this little table while you have a lot more data you also have a lot more insights that you can pick out right away and maybe one that you could take action on.

Hopefully, you’ll see that providing even a little bit of context helps move us fur- ther along the journey of driving action. Two of the most powerful ways to provide context that drive action are benchmarks and goals.

Benchmarking is usually done by using external benchmarks (indicating how others are doing for the same metric), but it does not have to be. You can also use internal benchmarks. What is important is that you use the power of benchmarking to induce action in your companies.

Wednesday: Leverage External Benchmarking

If you can find benchmarks for metrics that you are using in your company, consider that a gift. The reason external benchmarks are so powerful is that you are comparing yourself to others in the industry and usually a mostly neutral authority is presenting the benchmarks (this facet goes over great with senior management). Let’s go over some examples of how you can leverage external benchmarks.

An example of a great external benchmark you can use is the American Cus- tomer Satisfaction Index (ACSI). The ACSI is sponsored by the University of Michigan and measures most of the US economy. It presents its scores and ratings online at www.theacsi.org for almost all American industries.

Let’s say that you, dellcompetitor.net, are a new company in the great PC busi- ness and you have been measuring customer satisfaction on your website for some time now and you are really happy with your scores (Figure 11.1).

Should you be happy? Well, you can actually go to www.theacsi.org and get scores for not just Dell but also other major computer makers and judge your own per- formance against their trends (Figure 11.2).

By comparing your performance against that of a competitor, it quickly becomes apparent that your performance is okay over the last year but there is a wide gap between your performance and that of Dell. If before you were not able to get your company to focus on customers and take action, you can rest assured that your boss will take action now. There is something almost amazing about comparing yourself to your competitors and looking bad and wanting to go out and do something about it. Leverage this wonderful psychological gift.

30652c11.qxd 5/1/07 12:48 PM Page 266

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

267

■ W

E E

K 1

: L E

V E

R A

G IN

G B

E N

C H

M A

R K

S A

N D

G O

A L

S IN

D R

IV IN

G A

C T

IO N

Figure 11.1 Customer satisfaction trend: dellcompetitor.net

Figure 11.2 ACSI customer satisfaction trends: personal computers

It is advisable for you to put enough thought into ensuring that your metric can be compared directly against the benchmark (that is, that you are comparing apples to apples). But in general, I am a fan of taking whatever you can get from the outside and getting as close as you can to apple-to-apple measurements. If your methodology is

Ja n-

Fe b-

M ar

-0 5

Ap r-

M ay

-0 5

Ju n-

Ju l-0

5 Au

g- 05

p- 05

ct -0

30652c11.qxd 5/1/07 12:48 PM Page 267

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

268

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

slightly different, you can observe the trends of the numbers over time and benchmark against the trend and not absolute changes. It is certainly not a pure approach, as many in our industry would rightly say, but it can still be incredibly powerful, especially if trended over time, in moving your organization in the right direction to take actions.

Sometimes, though, pure benchmarks such as the ACSI might not be available— for example, in our web analytics (clickstream) space. Each existing web analytics tool measures things differently, and there are so many different types of businesses. We are always struggling with understanding our web analytics metrics performance because usually we are measuring just in our silo. Not having the benefit of an external bench- mark means that it is a bit harder to understand performance and drive action.

One excellent option for you is to compare your web analytics metrics against the Fireclick Index (http://snipurl.com/fireclick). Fireclick provides web analytics solutions (among other things) and collects data from all its customers. It has created a publicly available index providing benchmarks for key web analytics metrics (Figures 11.3 and 11.4).

Figure 11.3 Fireclick Index: business metrics

Figure 11.4 Fireclick Index: site metrics

With access to these weekly, monthly, and yearly trends, you can compare your web analytics performance against the benchmark. Your web analytics tool might not be Fireclick and you might measure unique visitors differently. But over time if you hold those two inconsistencies as constant, the trend of deltas between your perform- ance and the Fireclick index can be extremely helpful and actionable.

30652c11.qxd 5/1/07 12:48 PM Page 268

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

269

■ W

E E

K 1

: L E

V E

R A

G IN

G B

E N

C H

M A

R K

S A

N D

G O

A L

S IN

D R

IV IN

G A

C T

IO N

The Fireclick Index is also available by vertical industries (Figure 11.5), which will get you even closer in terms of your ability to do a Washington-apples-to-Fuji- apples comparison.

Figure 11.5 Fireclick Index: industry-specific metrics

To summarize, external benchmarks are perhaps the most powerful out there when it comes to moving from insights to action. They can be a powerful neutral authority in convincing management that the measurement of your website’s perform- ance is not based on your own personal opinion but on an external, trusted benchmark.

Thursday: Leverage Internal Benchmarking

If your business is truly unique or you are not able to find any kind of external bench- marks that you can trust, look inward and create benchmarks internally within your company, using your own data. It is surprising that we don’t do this more often. Again the goal here is to give more context to the performance of your key metrics in a way that helps you understand the performance better and drives action. Let’s look at some methods of benchmarking internally.

Table 11.1 shows how you can easily create an internal benchmark by showing performance for eight days instead of seven.

� Table 11.1 Internal benchmarking: measuring eight days vs. seven

Date Day of Week Total Visitors Unique Visitors Orders Units Units/Order

8/25/04 Wednesday 18,988 17,017 165 199 1.21

8/24/04 Tuesday 19,091 17,148 160 249 1.56

8/23/04 Monday 19,157 17,104 151 183 1.21

8/22/04 Sunday 9,181 8,403 74 113 1.53

8/21/04 Saturday 9,416 8,561 71 83 1.17

8/20/04 Friday 15,605 14,129 128 157 1.23

8/19/04 Thursday 18,246 16,403 140 178 1.27

8/18/04 Wednesday 19,095 17,181 151 170 1.13

Total 1,040 1,332

30652c11.qxd 5/1/07 12:48 PM Page 269

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

270

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Most often we show seven-day trends, but they don’t tell the full story. By com- paring eight-day trends (Wednesday this week to Wednesday last week, for instance) we are able to have a quick benchmark that tells us how to think about our perform- ance this week. For example it is not just important o know that on Wednesday you achieved higher results than Tuesday, but comparing this Wednesday to the same time period last week provides additional context around seasonality. This simple compari- son starts the process of asking the much required “what happened” and “why it hap- pened” discussions.

You can transpose this and imagine its application for 13-month trends so you can compare the same month year over year, or apply the data to the same quarter last year (as is extremely common on Wall Street).

Every web analytics tool out there should make this simple tweak to allow their data to become a smidgen more actionable.

To build on this idea, you can benchmark not only by using time as an element but also by using contributing segments, as shown in Figure 11.6.

Figure 11.6 Internal benchmarking: contributing segments approach

Figure 11.6 shows the annual trend for revenue. Notice that each bar is seg- mented (and it does not matter whether the segments are types of products, campaign categories, buckets of customers, or something else). The trend by itself would be quite meaningless. But by using segmented trends, you can create a sense of internal bench- marking and understand over time which segments are performing better or worse (be they products or customers).

$4,000,000

$3,000,000

$2,000,000

$1,000,000

Au g-

Se p-

O ct

-0 5

N ov

-0 5

D ec

-0 5

Ja n-

Fe b-

M ar

-0 6

Ap r-

M ay

-0 6

Ju n-

Ju l-0

30652c11.qxd 5/1/07 12:48 PM Page 270

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

271

■ W

E E

K 1

: L E

V E

R A

G IN

G B

E N

C H

M A

R K

S A

N D

G O

A L

S IN

D R

IV IN

G A

C T

IO N

If you are lucky enough to work in a company that has many websites, it is even easier to create internal benchmarks. Just use data from your different websites. You can’t independently judge the performance of each of your websites, because the data from each is a silo without enough context, but if you pool the data, you can get some nice magic going (Table 11.2).

� Table 11.2 Internal benchmarking: intra-company approach

a) Search engine traffic as a percentage of total site traffic

Search Traffic Jan 02 Feb 02 Mar 02 Apr 02 May 02 Jun 02 Avg

brand q.com 6.2% 4.8% 4.3% 6.8% 9.0% 5.8%

brand b.com 20.7% 12.1% 13.7% 13.4% 13.8% 14.5% 15.1%

brand a.com 1.9% 0.9% 0.9% 0.8% 1.7% 1.7% 1.3%

b) Google as a percentage of search engine traffic

Google Jan 02 Feb 02 Mar 02 Apr 02 May 02 Jun 02 Avg

brand q.com 19.0% 25.4% 36.4% 35.5% 23.6% 28.9%

brand b.com 55.6% 57.8% 61.7% 61.3% 66.4% 66.6% 59.3%

brand a.com 11.5% 7.0% 8.4% 8.2% 9.1% 9.7% 9.0%

Suddenly questions become so much easier to answer. For example: Is brand b.com performing well, or is brand a.com? In a data silo, it would be hard for you to judge performance, even if you have trends over time. But now very quickly you can see that brand a.com is in trouble when it comes to search engines. As you compare performance against other internal sites (the second table in Table 11.2), it becomes apparent that the problem for brand a.com is Google (perhaps the Google spider is not indexing the site).

Another good idea for creating internal benchmarks is to use 3-month or 12-month averages on your graphs to create a benchmark against which you can index your performance.

The goal with internal benchmarks is to provide context of any kind from the data we already have. This context can provide insights into the current time period’s performance that would allow you to drive action.

Friday: Encourage and Create Goals

Ask anyone whether setting goals is important and you will hear a chorus of “yes, of course!” There is probably not a single reader of this book who would disagree with the importance of setting goals either. Yet if asked for an honest raising of hands from

30652c11.qxd 5/1/07 12:48 PM Page 271

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

272

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

those whose key business metrics have goals, most hands in the room would stay down. We all know that goals are important, yet so few of us actually create them.

If you want to induce action in your organization, you have to have goals. Pure and simple. Goals are critical to decision making for the following reasons:

• They are a great way to focus the organization. They clearly delineate what is important and everyone knows exactly where they are going.

• They promote teamwork and a culture of accountability.

• Goals that have been approved by senior management are great at ensuring that you can find the right funding and staffing to succeed (or at least have a fair shot at asking for resources). With no goals, you don’t even know where to start.

• Goals are a tie-breaker for critical forks in the road and even speed up decision making. (“Well, that path does not line up with our goal so we can’t go down that one.”)

• They provide the perfect way to measure performance (both of the business and of employees at annual review time). Figure 11.7 shows an example of a trend that we will commonly see reported. It

can be conversion rate or it can be one of the many other rates in our portfolio.

Figure 11.7 Important business metric trend

It looks like you are making great progress on this metric. In general, the trend is up over the last 13 months. But that is a long time, and during that time your busi- ness grew and your staff grew. So is this good enough?

Overlaying the goal for this metric over the actual trend transforms its ability to communicate to you and drive action (Figure 11.8). It looks like the goal for the metric

2.50%

2.00%

1.50%

1.00%

.50%

0.00%

Fe b-

M ar

-0 5

Ap r-

M ay

-0 5

Ju n-

Ju ly

-0 5

Au g-

Se p-

O ct

-0 5

N ov

-0 5

D ec

-0 5

Ja n-

Fe b-

30652c11.qxd 5/1/07 12:48 PM Page 272

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

273

■ W

E E

K 1

: L E

V E

R A

G IN

G B

E N

C H

M A

R K

S A

N D

G O

A L

S IN

D R

IV IN

G A

C T

IO N

was to gradually improve from 1.2 percent to 1.9 percent over 13 months (rather rea- sonable). For the most part, the actual numbers fell far behind the goals for that time period.

Figure 11.8 Important business metric trend with clearly articulated goal

If you had this graph for the six months (May–Oct) when the trend consistently lagged behind the goals, would you have taken action faster? Perhaps.

Maybe the most important reason to encourage setting goals is that the process will make you think. What do I mean by that?

Let’s say that your job is to improve the trend in Figure 11.7 and let’s call the metric conversion rate.

The first thing to understand is that conversion rate = orders / unique visitors. So to improve conversion rate, you will have to improve the number of orders you take on the site or you will have to get more unique visitors.

To increase unique visitors (hopefully qualified), you might ask these questions:

• Where do our visitors come from?

• Which ones are most qualified? What is our site and segmented bounce rate?

• What is our acquisition strategy for traffic? PPC, SEO, affiliates, direct market- ing? What else?

• What is the effectiveness of our acquisition strategy for the last couple of years? What else is new that we can try?

• What are our competitors doing that we can learn from?

• What kinds of skills do we have in our team to support improvements in traffic to our website?

2.50%

2.00%

1.50%

1.00%

.50%

0.00%

Fe b-

M ar

-0 5

Ap r-

M ay

-0 5

Ju n-

Ju ly

-0 5

Au g-

Se p-

O ct

-0 5

N ov

-0 5

D ec

-0 5

Ja n-

Fe b-

30652c11.qxd 5/1/07 12:48 PM Page 273

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

274

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

• Can we leverage our ecosystem websites or this new user-generated content (UGC) or blog websites? To increase the orders on the website, you might ask these questions:

• What is the abandonment rate on our website?

• What are the typical page views per visitor or time on site?

• What is the most influential content on the site? How do we know what con- vinced people to buy?

• Do we have an effective merchandising strategy? What do people like buying?

• Do we have the budget to offer discounts on the site?

• Do we know why people come to the site? How come less than 2 percent end up buying?

• Why do people bail in the ordering process? Do we have the voice of the customer?

• How is our behavior targeting strategy working? We don’t have one, you say? Do we need to? What’s the repeat visits rate on our website?

• Can we do experimentation and testing? What are the top pages that we can optimize to improve orders?

• What are sales trends for the same products in stores or on our phone channel?

• What truly innovative things can we do to influence the purchase decision?

Take a pause and imagine this. Simply to improve a seemingly “dumb” metric such as conversion rate, you have to ask all these questions, find all these answers, explore data that you have and lots that you might not have, and put your senior exec- utives on the spot to make some tough calls.

How much harder and time-consuming will it be if you have to do this for your top five key business metrics?

Go into goal setting knowing that it is going to be an incredible amount of work. But in the end you will have all the benefits outlined earlier and you will have a clear shot at a competitive advantage.

One final tip on setting goals specifically for the Web: There is a tendency to set five-year goals for the Web. That is usually suboptimal because we can’t even look for- ward two years—things change so fast. So here is a best practice:

• Have real, concrete six-month goals.

• Have goals, with some stretch built in, 12 months out that you can commit to.

• Set goals for 24 months out if you have to, but be explicit that they are, for the most part, conjecture.

30652c11.qxd 5/1/07 12:48 PM Page 274

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

275

■ W

E E

K 2

: C R

E A

T IN

G H

IG H

IM P

A C

T E

X E

C U

T IV

E D

A S

H B

O A

R D

This will ensure that you have goals that are built on a solid foundation and remain relevant as you try to achieve them. It will also mean that they will accomplish their purpose of being a performance indicator and a catalyst for driving action.

Week 2: Creating High Impact Executive Dashboards

We live in a world that is overflowing with data (our world of web analytics espe- cially). The fundamental pace of business is accelerating bit by bit each day, our com- petition is coming from new areas, and there is a desire to have more accountability from our executives and businesses to deliver results. In such an environment, dash- boards empower a rapid understanding of business performance by tracking the critical business data in an easy-to-understand manner. Effective dashboards can be a powerful communication medium and greatly accretive to driving actions.

Yet studies indicate that more than 75 percent of marketers are dissatisfied with their ability to measure the effectiveness of either short-term or long-term performance and have a hard time making data-driven decisions. The core problem behind this dis- satisfaction is not that marketers don’t have enough data (that is often a minor issue). The core problem is the inability to identify what the most critical metrics are and to communicate performance for those metrics in a way that induces action.

Dashboards can vary by industries, by business functions, by altitude (organiza- tion or decision maker level) and by the sophistication of available skills and tools. This section covers the basic, and not so basic, rules that you can follow while creating your own dashboards. These rules will enable you to create effective dashboards that are geared toward driving actions.

Monday: Provide Context—Benchmark, Segment, and Trend

You will spend the first day of this week focused on applying the lessons from the prior week’s hard work (hence I have grouped a big bunch of work for you here all in one day!). All three of today’s recommendations help in driving action and are critical first things to cover when creating effective dashboards.

Use Benchmarks

Never report a metric all by itself. Rule number one of great dashboards is that there is no metric on a dashboard that exists without context. There are many ways you can show context. You can use benchmarks (internal or external), goals, or even prior per- formance to give some kind of context. But without context, it is impossible for a met- ric to provide any value on the dashboard, even if it is the most important metric for your business.

30652c11.qxd 5/1/07 12:48 PM Page 275

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

276

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Always Segment

The goal in a dashboard is to communicate not just the performance of one metric but also to improve actionability. Segmentation is a key tactic that makes it easy to under- stand what might be causing a great performance or a bad one.

Figure 11.9 shows a dashboard element that embodies the first two basic rules (have goals and apply segmentation). The metric, Recommendation Index, has a clear goal that is highlighted prominently. It also shows the performance of four key cus- tomer segments for a rapid understanding of where opportunities for focus and improvement lie.

Figure 11.9 Four dashboard metrics segmented, trended, and measured against the goal

Trending Rocks

Absolute a-given-point-in-time numbers segmented against goals are good. If you seg- ment them, they are even better. But if you trend them over time, it can really move your dashboard toward actionability (just as an example, imagine all the seasonality that comes to the fore). In Figure 11.9, in each of the months it is clear that the fourth segment is doing worse when it comes to performance against goals. But it is the trend that highlights that the problem is getting worse over time and perhaps, now, is in des- perate need of attention.

In Figure 11.10 you see a simple trend of visitors to the website. It is insightful to see a long-term trend and summarize performance of the site. Figure 11.10 also merges the preceding rule on segmentation. By segmenting this trend for your four big sources of traffic, you can observe that your strategy of investing in email marketing is having its intended effect of increasing traffic (for example, in August).

30652c11.qxd 5/1/07 12:48 PM Page 276

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

277

■ W

E E

K 2

: C R

E A

T IN

G H

IG H

IM P

A C

T E

X E

C U

T IV

E D

A S

H B

O A

R D

Figure 11.10 Dashboard metrics: segmented trends over time illustrating performance

Trends are a lot more effective than absolute numbers. They provide a big pic- ture and help highlight consistent misses or problems.

Tuesday: Isolate Your Critical Few Metrics

There is such a thing as too many metrics (Figure 11.11).

Figure 11.11 Example of a nonactionable dashboard

Mar Apr May Jun Jul Aug Sep Oct Nov Dec

334K 294K 317K 381K 270K

563K 499K 609K

372K 522K

Direct Corporate Email Search

30652c11.qxd 5/1/07 12:48 PM Page 277

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

278

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

It should be obvious by looking at this illustration, Figure 11.11, that this is not an effective dashboard. It tracks too many metrics, it attempts to segment some and color-code performance, but in the end it is nearly impossible to distill a cogent under- standing of what happened and what action should be taken now.

I advise you to spend lots of time trying to understand exactly what critical few metrics drive the business. As someone inarticulately put it, “What do we care about if the crap hits the fan?” What are your bottom-line business critical few? Your answer to that question will make or break your dashboard’s ability to empower decision making.

As a general rule of thumb, your dashboard should contain fewer than 10 metrics. Remember that you will need a goal for each of these metrics and you are going to seg- ment most, if not all, of them (and represent the goals and segments on your dashboard). By the time you are finished with that, you might not have space left for anything else.

Wednesday: Don’t Stop at Metrics—Include Insights

No dashboard should exist without including a cogent set of insights (in words) that summarize performance and recommend action. Most often dashboards are a collec- tion of numbers and dials and graphs, but they leave it to the awareness and intelli- gence of the reader to infer what all that data might indicate. Perhaps more sadly, what such insight-free dashboards are missing is the benefit of all the analysis that went into creating them. Even if they are segmented and trended, you have only summary-level data for critical metrics in the dashboard. Having a section for insights allows the intel- ligence from the analyst to bubble up to the highest level.

Figure 11.12 shows a dashboard from the White House website. Notice that the graphs are quite small, yet sort of legible. Two of the graphs clearly indicate that the data was segmented. Most important, the section in the middle includes a brief written summary of the trends.

You can go one further and ensure that you have a section, way up on top, in the dashboard that shares your insights so that the people looking at the dashboard (the ones with no context of your business) can understand it better. Here are sections you should consider adding to your dashboard:

• Performance summary: What was up, what was down, where is the big gap? This can be just one line.

• Insights summary: What were some of the causes of the hits and misses? What were the underlying shifts in the business? What is the root cause? This can be one or two lines.

• Recommended actions: What should we do next? How do we reverse the decline? What is the new opportunity on the horizon? What’s a threat? Based on all the data, what are the top three things to fix? This should be a top-five list.

30652c11.qxd 5/1/07 12:48 PM Page 278

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

279

■ W

E E

K 2

: C R

E A

T IN

G H

IG H

IM P

A C

T E

X E

C U

T IV

E D

A S

H B

O A

R D

Figure 11.12 Example dashboard with trends and insights

Do not create dashboards without insights and recommended actions. These are the only way to push accountability down the chain, to not just send numbers and reports and only send up analysis. It is the only way to ensure that senior leaders will be able to take action sooner and with fewer questions.

Thursday: Limit Your Dashboard to a Single Page

It might not be the most obvious basic rule, but if your dashboard does not fit on one page, you have a report, not a dashboard. Additional layers of this rule are as follows:

• Page Size = A4

• Print margin = minimum of 0.75 inch (all sides)

• Font size = minimum of 10 (for metrics), minimum of 12 (for goals/benchmarks)

That should not leave a lot of room for doubt. This rule is important because it encourages rigorous thought to be applied in

selecting the metric. It should act as a natural barrier to cramming in too much infor- mation, make data presentation easier, make the dashboard more understandable (hence more likely to promote action), and make it portable (don’t underestimate the power of being able to carry a piece of paper around with 100 percent of your business performance on it).

30652c11.qxd 5/1/07 12:48 PM Page 279

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

280

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

It might seem like an easy-enough rule to follow, but the fact that this is a task for a whole day should reflect how hard it is to pull this off. Pull out any dashboard that you have handy for your company and try to apply this rule. You’ll see instantly how hard this is. But it is also absolutely critical if you are to communicate effectively and drive action.

Friday: Know That Appearance Matters

There is no politically correct way of saying this except that appearance matters. Dash- boards must be pretty for them to be used as effective communication vehicles.

Here is the first definition of pretty from Merriam-Webster’s Collegiate Dictio- nary: artful, clever.

Creating dashboards that are artful and clever enables the information presented to be understood more clearly, which will lead to faster decisions.

A standard dashboard section might look like Figure 11.13. It is fairly clean, it communicates sort of okay, and presents metrics and numbers in an easy-to-understand 2 × 2 matrix. Overall, not bad.

Figure 11.14 shows similar data but in a much different format. You can quickly see how Figure 11.13 might communicate much more effectively than Figure 11.12. There are enhancements, including having the bubble size for each metric communicate importance. It is extremely clear what needs most attention; the numbers are there almost as an afterthought. With Figure 11.13, you have to think a lot less and explain a lot less.

The visual appeal of the dashboard matters a great deal more than we realize. Your call to action is to produce pretty stuff!

Figure 11.13 Standard dashboard element

High

Low

Low Impact

Status Quo Required

Maintain or Improve

Look & Feel (75, 1.4) Product Information (76, 1.9)

Content (78, 0.7) Site Performance (80, 0.7)

Monitor Top Priority

Functionality (73, 1.9) Navigation (69, 1.4)

Search (72, 1.5)

30652c11.qxd 5/1/07 12:48 PM Page 280

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

281

■ W

E E

K 3

: U S

IN G

B E

S T

P R

A C

T IC

E S

F O

R C

R E

A T

IN G

E F

F E

C T

IV E

D A

S H

B O

A R

D P

R O

G R

A M

Figure 11.14 “Pretty” dashboard element

Week 3: Using Best Practices for Creating Effective Dashboard Programs

Creating a dashboard that is effective is an important foundational element of inducing action based on your data. The prior section covered specific rules that should be applied in the process of creating a dashboard. This section covers best practices that should be followed if you are to create a successful actionable program around dash- boards. These best practices are one level above the tactical rules and they are critical in ensuring that your dashboards are actually used and live for a long time rather than being rituals you simply go through every week or month.

Monday: Create Trinity Metrics That Have a Clear Line of Sight

In several places in this book, I have covered the importance of having a Trinity mind- set. Applying that mindset is especially critical when it comes to creating actionable dashboards. It does not have to be complicated, but you should deliberately determine whether you have all three elements of the Trinity represented in your dashboard to ensure that you are showing the complete picture.

It is equally important to have a clear line of sight to business goals. Having this line of sight can be the difference between your email about the dashboard being opened and it going directly into the Delete folder after month one.

Apply the Trinity Approach

In web analytics, we typically err toward including lots and lots of clickstream data in our dashboards. They are typically light on outcome metrics (revenue, problem resolu- tion, and so forth) and really light on customer experience metrics.

Senior management often can’t connect to the clickstream metrics because the met- rics often don’t mean much and for your site probably don’t change that often either. So

High

Low

Status Quo Required Maintain or Improve

Monitor Top Priority

Look and feel 54, 0.4

Content 44, 0.4

Sc or

Ease of use 43, 1.0

Accessibility 39, 1.1 Search

37, 1.1

Problem resolution 36, 0.8

Timeliness 31, 2.4

30652c11.qxd 5/1/07 12:48 PM Page 281

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

282

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

the average time on the site probably does not change much at all no matter what you do, and the same thing can be said about page views or others “key” clickstream metrics.

Management can connect to outcomes (especially those that affect the bottom line). Many dashboards are starting to include these. But the ones that management will connect with the most are metrics related to customers, such as improvements in customer satisfaction or task completion rates or percent of people recommending the website. The wonderful thing is that all of these metrics measure strategic elements that will yield results over the long term.

When you create your dashboard, ensure that you are covering all the elements of the Trinity so that your dashboards balance the short-term and the long-term met- rics and are able to connect with senior management.

Table 11.3 shows metrics that could bubble up to form a Trinity-inspired dash- board. These metrics also show the value of your website to the company ecosystem very effectively.

� Table 11.3 Trinity-inspired dashboard elements

“Trinity” Element Metric Rationale for Inclusion

Behavior Visits Shows overall site traffic (online sessions)

Unique Visitors Shows unique visitors to the website

Return Visitors Helps keep track of visitor retention/engagement

New Visitors Tracks the key metric of attracting new prospects to the

website

Measures how long it takes to complete purchase process

online

Search Traffic (External) Traffic from search engines

Internal Search Usage Shows internal search engine usage

Experience Customer Satisfaction Overall customer satisfaction with the site

Computes the likelihood that customers recommend our

website

Overall Task Completion % of website visitors that were able to successfully com-

plete their tasks

Outcomes Revenue Most important metric showing finance results

Visits to Purchase Related to convince to buy and site efficiency

AOS Can identify changes in either customer or available prod-

uct mix

Conversion The critical measurement of how well the website is per-

forming

Direct Traffic Measurement of the site efficiency not related to any mar-

keting programs

Campaign Traffic Measurement of efficiency of marketing programs

Checkout Abandonment Number of visitors who bailed during the checkout process

Likelihood to Rec-

ommend Website

Avg. Time on Site,

Purchasers

30652c11.qxd 5/1/07 12:48 PM Page 282

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

283

■ W

E E

K 3

: U S

IN G

B E

S T

P R

A C

T IC

E S

F O

R C

R E

A T

IN G

E F

F E

C T

IV E

D A

S H

B O

A R

D P

R O

G R

A M

Maintain a Clear Line of Sight

Often after a quick scan of our dashboard, a reader will not clearly understand a met- ric’s effect on the company’s bottom line. The perfect antidote to this is to apply a test that will check how a change in a metric will improve the company’s bottom line. What’s the line of sight?

It does not matter whether you are running an e-commerce website or a support website. All the metrics on a dashboard should have a clear line of sight to the strategic business objectives. This will ensure that the dashboard will be embedded into the way of life of the company.

The ultimate goal for your company, if public, is to deliver shareholder value. Shareholder value is measured by the stock price, your company revenue and earnings, and growth. Your CEO probably has key business objectives for the year that will deliver against all of these shareholder metrics. Here is an exercise that you can do: take each of the critical few metrics from your dashboard (remember, you just have 10) and map them to the CEO’s objectives for the year.

If you are able to map page views per visitor all the way up to a CEO objective, you have a clear line of sight. If you can’t, it is not a good metric.

Going through the line of sight exercise will help you understand whether you are working on things that are important to the company and whether you’ll be able to get the kind of exposure you need to take action if you need to. It also means that if you are able to improve these metrics, you will be a superstar because it will be easy to see how you have added value by improving the CEO’s business objectives.

Tuesday: Create Relevant Dashboards

Having made it so far in this chapter, and this week, I am sure you feel that dashboards are these tablets of stone that have to be carefully chiseled with important metrics and then worshiped. Quite the contrary. Most dashboards become irrelevant quickly, espe- cially on the Web, because business does move at the speed of light (whether you are a Fortune 100 company or a small or medium business). Actionable dashboards have structures and processes built in to ensure that they remain timely and relevant, as you’ll see here.

Short Lag Time

Dashboard metrics should have a short time lag. You should be able to collect them, process them, and report in the required time frame. You can imagine how reporting for January performance in the first week of March might be rather suboptimal when it comes to taking action. This will certainly mean that you will have to make tough choices in selecting your metrics and even abandon some that might not meet the short time lag rule. You can take those metrics, if they are still critical, and add them to a

30652c11.qxd 5/1/07 12:48 PM Page 283

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

284

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

separate addendum that gets presented every quarter or every two months. This should keep your dashboard clean, focused, and actionable.

The short lag time rule also applies in a slightly different perspective as well. Met- rics you choose for your dashboard should be able to provide a performance signal in the measured time period. For example, if a metric changes only every three months, it is perhaps suboptimal for a weekly dashboard. You can, akin to the computational lag time requirement, move it to a different addendum that gets presented less frequently.

Churn

Contrary to popular belief, dashboards are not carved in stone and hence are not per- manent affairs. Dashboards, like humans, constantly evolve. That is exactly what your mindset for dashboards should be and it can be incredibly hard because organizations like stability, and senior management often likes predictability when it comes to meas- uring success. You can keep a few (25 percent) of your metrics stable for a long period of time, but you should plan on the dashboard having some level of churn all the time (metrics should be eliminated, almost deliberately, as soon as it is discovered that they are no longer relevant). If you have a dashboard that measures monthly performance, then over the course of a year you should have churned out at least 15 percent of the metrics because as the Web changes, at least a couple of your key metrics will change with it. Generally, you should not be churning more than 25 percent of your metrics in a year; more than that indicates that you didn’t do an effective job identifying your key metrics.

Everything evolves. Businesses change, people come and go, high-level priorities evolve, we become smarter (or we become dumber but acquire people who are smarter than us!), our competitors think of new and clever things, and so forth. Why should our dashboards and metrics on dashboards stay the same over the span of a year?

Planning for the evolution and churn is mandatory. Ensuring that evolution is the only way to ensure that your dashboards don’t become stale and end up as pieces of paper that don’t add any value and consume way too much of your time.

Wednesday: One Metric, One Owner

This is the primary way that you will ensure that action will be taken. Ensure that every metric on the dashboard has a clearly identifiable person who is to be, in an ideal world, held responsible for all facets of delivering on the metric.

In your respective dashboards, and during presentations to senior management and stakeholders, ensure that the name of the owner is prominently displayed and that the team knows who the go-to person is (Table 11.4). Having clear ownership will ensure that your dashboard will continue to have the care and feeding it needs for its sub-element metrics. But it will also ensure that when you are in meetings and your CMO wants the answers, it is clear who will provide leadership when it comes to resolving each metric and taking action.

30652c11.qxd 5/1/07 12:48 PM Page 284

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

� Ta

bl e

11 .4

Tr in

it y-

in sp

ir ed

d as

h b

oa rd

e le

m en

ts —

w it

h o

w n

er s

“T ri

n it

y” El

em en

t M

et ri

c O

w n

er R

at io

n al

e fo

r In

cl us

io n

B eh

av io

r V

is it

s St

ev en

Sh ow

s ov

er al

l s it

e tr

af fi

c (o

n lin

e se

ss io

n s)

U n

iq ue

V is

it or

s St

ev en

Sh ow

s un

iq ue

v is

it or

s to

t h

e w

eb si

R et

ur n

V is

it or

s H

el ps

k ee

p tr

ac k

of v

is it

or r

et en

ti on

/ e

n ga

ge m

en t

N ew

V is

it or

s Tr

ac ks

t h

e ke

y m

et ri

c of

a tt

ra ct

in g

n ew

p ro

sp ec

ts t

o th

e w

eb si

A vg

.T im

e on

S it

e, Pu

rc h

as er

s M

ic h

el l

M ea

su re

s h

ow lo

n g

it t

ak es

t o

co m

pl et

e pu

rc h

as e

pr oc

es s

on lin

Se ar

ch T

ra ff

ic (

Ex te

rn al

) Ke

vi n

Tr af

fi c

fr om

s ea

rc h

e n

gi n

In te

rn al

S ea

rc h

U sa

ge Ke

vi n

Sh ow

s in

te rn

al s

ea rc

h e

n gi

n e

us ag

Ex pe

ri en

ce Cu

st om

er S

at is

fa ct

io n

O w

en O

ve ra

ll cu

st om

er s

at is

fa ct

io n

w it

h t

h e

si te

Li ke

lih oo

d to

R ec

om m

en d

W eb

si te

O w

en Co

m pu

te s

th e

lik el

ih oo

d th

at c

us to

m er

s re

co m

m en

d ou

r w

eb si

O ve

ra ll

Ta sk

C om

pl et

io n

O w

O ut

co m

es R

ev en

ue V

ic to

r M

os t

im po

rt an

t m

et ri

c sh

ow in

g fi

n an

ce r

es ul

V is

it s

to P

ur ch

as e

Jo h

n R

el at

ed t

o co

nv in

ce t

o bu

y an

d si

te e

ff ic

ie n

A O

S D

av e

Ca n

id en

ti fy

c h

an ge

s in

e it

h er

c us

to m

er o

r av

ai la

bl e

pr od

uc t

m ix

Co nv

er si

on D

av e

Th e

cr it

ic al

m ea

su re

m en

t of

h ow

w el

l t h

e w

eb si

te is

p er

fo rm

in g

D ir

ec t T

ra ff

ic M

ea su

re m

en t

of t

h e

si te

e ff

ic ie

n cy

n ot

r el

at ed

t o

an y

m ar

ke ti

n g

pr og

ra m

Ca m

pa ig

n T

ra ff

ic M

ea su

re m

en t

of e

ff ic

ie n

cy o

f m ar

ke ti

n g

pr og

ra m

Ch ec

ko ut

A ba

n do

n m

en t

M ar

ie N

um be

r of

v is

it or

s w

h o

ba ile

d du

ri n

g th

e ch

ec ko

ut p

ro ce

30652c11.qxd 5/1/07 12:48 PM Page 285

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

286

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Of course, ownership will not be successful without a competent support struc- ture. It is the responsibility of senior management, the CMO in this case, to ensure that the resources are available to the metric owners so that they can deliver what they are being held responsible for.

Thursday: Walk the Talk

This recommendation is geared more toward your most senior management than toward you (well, you too). It is important to walk the talk when it comes to dash- boards. They can’t be these one-time events that happen every week (and the related emails go directly to the trash box) or happen every month as simply a big gabfest.

Make them a way of life, carry them around, bring them to meetings, stick them to the walls of your cubicles. They should be facilitators of active and everyday decision mak- ing (even monthly dashboards). Senior management should show, through walking the talk, that they are using the dashboard to prioritize the work that happens in the team (rather than having a dashboard and yet relying on their gut to drive work). They should use dashboards to measure the personal success of the metric owners. Some facet of their own compensation (even if 2 percent) should be tied to meeting the dashboard metrics.

Walking the talk will promote a culture that gels around a common set of met- rics and objectives. And that at the end of the day is priceless.

Friday: Measure the Effectiveness of Your Dashboards

In a book that has stressed customer satisfaction and the voice of the customer, you should have anticipated this recommendation: survey the key stakeholders and con- sumers of your dashboard.

Ask your stakeholders to rate various facets of the dashboard’s effectiveness and its ability to drive action. Ask them how you are doing and how else can you improve, and just for good measure make the survey anonymous (it is amazing what people won’t tell you even in the most open and honest company environments, because they are just trying to be nice).

The voice of the customer will be critical to keeping the dashboard meaningful and relevant over a longer period of time and ensuring that it is adding value.

If you collect feedback, it is imperative that you present your findings and also that you incorporate the VOC. This is the only way to reinforce the importance of the VOC to you and also to ensure that you will receive good feedback when you ask for it again. If you ask and you receive, ensure that you give back. It’s simple.

Week 4: Applying Six Sigma or Process Excellence to Web Analytics

A common challenge that stymies lots of companies, especially advanced ones, is that although they have clickstream data and advanced web analytics tools, have embraced the Trinity mindset, are into testing and experimentation as well as surveys and usability,

30652c11.qxd 5/1/07 12:48 PM Page 286

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

287

■ W

E E

K 4

: A P

P L

Y IN

G S

IX S

IG M

A O

R P

R O

C E

S S

E X

C E

L L

E N

C E

T O

W E

B A

N A

L Y

T IC

they find that for some reason web analytics is not quite as institutionalized as they would like it to be. Somehow even after such an investment in tools and skills, they are still not quite able to induce action. It is all still ad hoc. It is as if a lens or filter needs to be applied that would help bring a sense of focus, consistency, and predictability from which the company can benefit.

Methodologies and approaches inspired from Six Sigma or process excellence can help solve this problem.

Process excellence (PE) is a set of activities designed to create excellent processes. Process excellence includes Six Sigma but it also includes other initiatives as well.

Six Sigma is a business improvement methodology that comes from Motorola and was made truly famous at the General Electric Company (GE). The overall spirit of Six Sigma is to look at the business as a series of processes. The goal is to improve processes by eliminating any “defect” in each of the processes. Quoting from Wikipedia, “The core of the Six Sigma methodology is a data-driven, systematic approach to problem solving, with a focus on customer impact.”

The common theme between both ideas is to think of business as a collection of processes that should be well defined, measured, optimized, and executed with an eye toward reducing the number of defects in the process to deliver for the end customer.

I believe, along with many others, that what is missing from web analytics thus far is the Six Sigma/PE-inspired mindset that forces us to think of activities we perform in terms of processes that need to be defined, documented, measured, and optimized. By learning from Six sigma/PE, we can cross the final hurdle holding us back in deliver- ing sustainable actionability from web analytics for our businesses. To move that final step will require a fundamental mindset shift for companies and their resources and how work is done today. We will move from cowboy land, just a little bit, to living in well-organized suburbs, if not cities. I am sure that prospect is not appealing to some, but it is very much required.

Six Sigma and PE are extremely complex topics and hence I will not go into any great detail for either. This week you will focus on two topics that are inspired by Six Sigma/PE and that will be greatly beneficial for web analytics practitioners. The first topic is the fantastic beauty of thinking about everything we do as part of a process, and the second topic is how we can improve our processes byusing the Six Sigma methodology DMAIC (define, measure, analyze, improve, control).

Monday: Everything’s a Process

We don’t tend to think of web analytics overall, or the work we do in our day-to-day lives, as a process. But the reality is that every day the web ecosystem in your company is one complicated process that is taking inputs, processing them, and creating outputs. It is perhaps more accurate to say that it is a collection of processes solving for the whole.

Let’s consider an example. Figure 11.15 shows a simple example of a web opera- tions process.

30652c11.qxd 5/1/07 12:48 PM Page 287

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

288

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Figure 11.15 Typical web operations process

Each step of the process could be handled by one team or different teams in the company. But each step depends on the prior one for success. Each step in this high level process consists of sublayers of processes that help make it successful. For exam- ple, Figure 11.16 shows that the Scope and Schedule step can also be viewed as a process in and of itself.

Figure 11.16 Web operations process—step 2 drill-down

You can quickly see how each of the steps in the high-level process will have sublay- ers. In some cases, each of the sublayers will have their own sub-sublayers that are worth working through and documenting (until a point of diminishing returns, of course).

The exercise of understanding what it takes to execute each step in our day to day delivery of key tasks and taking time to document it with a process excellence mindset is critical to creating a true system that does the following:

• Functions efficiently

Plan Scope & Schedule Deliver to IT Build, Test, & Deploy

Analyze

Create & Submit Work Request

Clarify Work Request

Prioritize Work Schedule & Commit

1 2 3 4

Plan Scope & Schedule Deliver to IT Build, Test, & Deploy

Analyze

30652c11.qxd 5/1/07 12:48 PM Page 288

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

289

■ W

E E

K 4

: A P

P L

Y IN

G S

IX S

IG M

A O

R P

R O

C E

S S

E X

C E

L L

E N

C E

T O

W E

B A

N A

L Y

T IC

• Has the ability to grow and scale

• Provides a shared understanding of what is supposed to happen and when

• Can be measured and optimized

That last one ties back to Six Sigma. For any well-defined process, the goal with Six Sigma is to put critical measures in place for each process, measures that would empower the understanding of “defects” in the process. This would result in decreased variations in the process. Consider Figure 11.17.

Figure 11.17 Improving the sigma by reducing variations

Let’s say that this figure shows a plot of all the times when a daily delivery of a package from your UPS driver occurs (though it could represent your daily website release or report automatically published—or imagine your own situation). The deliv- ery is supposed to happen every day at 9 A.M.

The first graph illustrates a delivery schedule that is widely dispersed with points (defects) that occur far from the mean. The variations from the mean reduce the pre- dictability of the process, increase its complexity, and result in suboptimal customer impact.

Defining the process, understanding the critical customer measures, and opti- mizing the process steps will result in a graph that looks like the one on the right in Figure 11.17. There are significantly fewer variations from the mean. The net results from reducing variations in a process are as follows:

• Greater predictability in the process

• Lower costs resulting from less rework and waste by reducing inefficiencies, complexity, and errors

• Products and services that perform better and last longer

• Delighted customers (and company employees)

By clearly defining the various processes that make up web analytics, we can understand the process much better, measure it better, and optimize it better over

Too early Too late Too early Too late

Defects Defects

Delivery time Delivery time

Reduce variation

Spread of variation too wide compared to

specifications

Spread of variation narrow compared to specifications

30652c11.qxd 5/1/07 12:48 PM Page 289

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

290

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

time—everything from website page development, to experimentation and testing, to creation and delivery of dashboards, to daily scheduled reports, to checking for missing tags, and so forth.

Figure 11.18 illustrates a potential process that clearly identifies the big steps to follow for effective merchandising on a typical website.

Figure 11.18 Sample web merchandizing and launch process

The process identifies what happens in each step and who is responsible for ownership of that step, and it documents from start to finish the expectations of each step. It is very simple, yet amazingly powerful in providing a solid understanding of what happens and assigning roles and responsibilities. It should be obvious that each step can be measured and optimized for its unique success metrics.

One last example of web processes, Figure 11.19, documents the process, in detail, of moving from requirements gathering to release for an improved customer experience on the websites.

Identify problem and set scope

Conduct the deep analysis and recommend

fixes

Present findings Exhaustive

analysis, PPT, actionable

recommendations & findings

summary, XLS

Analyze recommendations

and prioritize

Make design/ merchandizing

changes

Implement changes

Measure success

Marketers & Analysts Analysts

AnalystsMarketers Designers Developers

30652c11.qxd 5/1/07 12:48 PM Page 290

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

291

■ W

E E

K 4

: A P

P L

Y IN

G S

IX S

IG M

A O

R P

R O

C E

S S

E X

C E

L L

E N

C E

T O

W E

B A

N A

L Y

T IC

Figure 11.19 Sample customer experience improvement process

Each of the steps in this diagram have associated ownerships, roles, responsibili- ties, expectations of start-to-finish efforts, success metrics, and all the specific deliver- ables that need to be completed to move on to the next one. You can easily imagine not just how this would deliver a better end customer experience but how any com- pany can itself ensure that the process can be repeated and performed with the least amount of variation.

Process Creation: Not a Scary, Complex Proposition

There is a common misperception that defining a process takes a lot of time and effort and that all

processes need to look like the one shown here.

Continues

Development

Requirements

Competitive Research

User Research

Site visits (personas) Market research

Call logs, etc.

Critical Tasks

Ease of Use Goals

Business Goals

Planning & Design

User Flows

Conceptual Design Information architecture

Interaction design (wireframes)

Content development

Detailed Design Visual design

Finalize content Design documentation

User Test

Revise

Review UI Implementation

Test

User Test

Revise

Release

Benchmark

Track & Prioritize Design

recommendations Features

Issues, etc.

30652c11.qxd 5/1/07 12:48 PM Page 291

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

292

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

Process Creation: Not a Scary, Complex Proposition (Continued)

You don’t have to worry that you have to use a particular software program or create a certain look

and feel or format to document a process. It does not have to be painful work. What is important is

that you spend time identifying each process in your teams, document by using the formats that you

are most comfortable with, identify ownership of each step in the process, and create success metrics

to measure its performance over time so that you can optimize it and reduce defects (variations).

The following is a process that uses a completely different presentation layer but accomplishes the

same purpose.

Website JavaScript Tagging Process

Provide specific business requirements as to what is expected from the analytical tool. Owner: Mar-

keter. Timeline: None.

Meet with website development lead to understand site structure, operations, cookie strategy, and

so forth. Owner: Analyst. Timeline: 3 days.

Partner with the vendor to create a customized JavaScript tag for the site to collect the data that is

required. The JavaScript tag and instructions are then sent to the website development lead for

implementation. Owner: Analyst. Timeline: 3 days.

Ensure that the tag is implemented exactly as provided and appears on all pages on the site.

Owner: Dev lead. Timeline: 1 day.

Validate that the provided code has been implemented correctly and exactly as provided on the QA

website. Owner: Analyst. Timeline: 1 day.

Validate that some data is flowing through to the application (clicks or page views from the QA

site). Owner: Analyst. Timeline: 2 days.

Provide feedback to the development lead for any fixes (only if required). Owner: Analyst. Timeline:

1 day.

Development lead makes the corrections (only if required). Owner: Dev lead. Timeline: 1 day.

Website migrates from QA to production. Owner: Dev lead. Timeline: 1–10 days.

Three days after launch, analyst validates that data is flowing okay and configures security access

for all marketers. Owner: Analyst. Timeline: 3 days.

Conduct training one week after launch and set up basic reports for marketers. Owner: Analyst.

Timeline: 1–29 days.

It is not important how you document the processes that all combine to deliver to your company’s

customers. It is important that you go through the exercise to identify the critical processes and

document them with an eye toward improving predictability, reducing complexity and inefficien-

cies, and setting clear goals, roles, and responsibilities to create delight for your company’s cus-

tomers and employees.

30652c11.qxd 5/1/07 12:48 PM Page 292

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

293

■ W

E E

K 4

: A P

P L

Y IN

G S

IX S

IG M

A O

R P

R O

C E

S S

E X

C E

L L

E N

C E

T O

W E

B A

N A

L Y

T IC

Tuesday through Thursday: Apply the DMAIC Process

In the prior section, you learned about the importance of processes and reviewed sam- ples of processes that apply directly to our world of web analytics. Identifying your critical process is necessary for you to move into our second Six Sigma/PE-inspired arena: DMAIC (define, measure, analyze, improve, control), shown in Figure 11.20.

Figure 11.20 DMAIC continuous improvement process

DMAIC is a rigorous process for continuous improvement. DMAIC is relevant and important because it helps us go one step beyond simply identifying a process. It is a closed-loop process that helps eliminate unproductive steps, helps focus on new measurements, and in its essence applies technology for improvement.

It is helpful to think of the DMAIC as a structured problem-solving approach. By applying DMAIC, we can be assured that the processes we have put in place are per- forming at peak levels and delivering for our customers. For this reason, I believe that DMAIC is particularly relevant and applicable to web analytics. So much of what we do is structured around data and focuses on solving for our customers (internal and exter- nal). Applying the rigorous DMAIC approach not only will improve our web analytics supporting processes but can inspire the kind of discipline that is so often missing. It

Define

Measure Analyze

Improve

Control

30652c11.qxd 5/1/07 12:48 PM Page 293

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

294

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

also provides the structure and guidance that we so sorely need (in our approaches, in our reports, in our dashboard, in driving our decision makers to action by providing analysis and insights directly driven by company customers).

DMAIC is a closed-loop process; let’s go through each step one at a time.

Define

The goals of the Define phase are to clearly identify and validate the opportunity to improve the process and to articulate this definition in a clear and measurable way. The basic steps of this phase are as follows:

• Define who the customers are and what their requirements are for products and services from the process.

• Use tools such as stakeholder analysis, voice of customer techniques (surveys, and so forth) to identify the critical customers and their Critical to Quality fac- tors (CTQs) that have the greatest effect on quality. It is important to distill down the list of critical few metrics.

• Create a project charter that will contain the problem definition, goal, business case, and high-level project place for the rest of the phases.

At the end of this phase, you have a clear understanding of how to deliver to your customers.

Measure

The goals of this phase are to understand the critical measures of success and to estab- lish a baseline performance by undertaking an extensive (as optimal) data collection and benchmarking effort.

• Develop a data collection plan for each step of the process.

• Collect data from as many sources as possible to determine the metrics that could measure the CTQs.

• Measure the current rate of failures (defects) in the process, with a defect being any output that does not meet customer requirements.

• Identify the inputs into the process that contribute to failures.

• Validate the data collection process to ensure that the data is being collected correctly, and if required, update any data collection process that might be incorrect.

This exercise produces a solid understanding of how the process is performing today, which establishes a baseline measure. Usually at the end of this stage, we have initial identifications of the metrics that we will dive deeper into for the next phase.

30652c11.qxd 5/1/07 12:48 PM Page 294

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

295

■ W

E E

K 4

: A P

P L

Y IN

G S

IX S

IG M

A O

R P

R O

C E

S S

E X

C E

L L

E N

C E

T O

W E

B A

N A

L Y

T IC

Analyze

You should love this phase. Here we dive deep and conduct a root-cause analysis that will help us understand issues that are causing defects in the process. The steps are as follows:

• Develop a hypothesis about causes of defects in the process.

• Analyze the data; perform root-cause analysis on reasons for failures to identify opportunities to improve. The goal is to determine true sources of defects that are leading to customer dissatisfaction.

• Create a specific list of sources of defects in the process.

• Validate the outcomes of the analysis with the stakeholders.

At the end of this phase, you should have a list of improvements to the process, some of which will be quick hits and others that will take much longer and require support of senior management teams.

Improve

All the tough work you have done thus far pays off now. You get to prioritize the spe- cific ideas for defect reduction identified in the prior phase and you get to implement them. Yay!

• Firm up the solutions and assess and address any risk to the process from imple- menting improvements.

• Identify any new measures that will be required to gauge success in the process from implementing improvements (if things change dramatically).

• Adjust the process to implement the recommended changes and kick off the measurement of the customer CTQs identified in the earlier phases.

• If improvements are not made for any reason, quantify the effects on the busi- ness by using data from the Measure and Analyze phases and present this to the key stakeholders.

This is a fun stage because you get to see the fruits of your labor. It is amazing to see process efforts reduced from ten days to three days, or bugs on the live site go down to zero, or reports being published that actually get used and induce action.

Control

This phase helps ensure the long-term successful adoption of the new process improve- ments. The goal is not to stop until it is clear that improvements have become a way of life.

• Measure postimplementation success of the process improvements and changes.

30652c11.qxd 5/1/07 12:48 PM Page 295

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .

296

C H

A P

T E

R 1

1 :

M O

N T

H 6

: T

H R

E E

S E

C R

E T

S B

E H

IN D

M A

K IN

G W

E B

N A

L Y

T IC

S A

C T

IO N

A B

L E

■

• Establish ongoing monitoring of the process to ensure that it stays “in control.”

• Collect VOC from the key stakeholders and company customers to quantify the impact of the changes from their perspectives.

It is optimal at the end of the Control phase to do a postmortem of the biggest lessons learned and present them to the wider community. This enables you to share the progress made and to identify other areas in the company that could benefit from the knowledge gained from your DMAIC process.

Friday: Reflect on What You’ve Learned

To repeat just slightly, by using our first inspiration from Six Sigma/PE, you define and identify processes in order to reduce failures and meet customer requirements. Then you can apply our second inspiration to establish what the customer CTQs are and work toward improving existing processes to improve delivery on those CTQs.

Both of these processes—creation and improvement—do not have to be time- consuming or complex. As long as you diligently apply the principles shared in this chapter, you can complete some projects in a few days and realize immediate benefits (for example, how long does it take from the time data is captured to deliver a report to your customers?). Other projects can take much longer (for example, why is it that absolutely no one has made a single decision by using your half-million-dollar invest- ment in web analytics tools?). However, all projects will have commensurate rewards if you persist in documenting the process, identifying your customer CTQs and points of failure (the root causes of defects), and addressing them by making changes (even big ones such as changing your organizational model if that’s the root cause!).

My recommendation is to try this on a small scale, as in the examples in the “Everything’s a Process” section, and you’ll get a feel for how powerful Six Sigma/PE can be in revolutionizing your life.

30652c11.qxd 5/1/07 12:48 PM Page 296

C o p yr

ig h t ©

2 0 0 7 . Jo

h n W

ile y

& S

o n s,

I n co

rp o ra

te d . A

ll ri g h ts

r e se

rv e d .