Advertising Experiments at Yelp

Advertising Experiments at Yelp

(mini-case in data analytics)

Yelp is a restaurant review platform with an impressive number of online reviews written by ordinary restaurant goers. Yelp has compelling data demonstrating these reviews have an economically important influence of the restaurant choices people make, which is exciting for the executive team at Yelp and has caught a lot of attention within the industry.

However, doubts have been raised about the efficacy of their main source of revenue – selling ads to restaurants. To better understand the issue, they have run a randomized controlled test with a control group and two treatment groups: One treatment to test the impact of their current ads on restaurant sales, and the other treatment to test the impact of an alternative ad design that they are considering switching to.

You are a recent hire at Yelp - three months on the job, having moved from Logan to San Francisco after graduating with your degree in Marketing. You work in the Marketing Analytics area, and the Jeremy Stoppleman, the CEO of Yelp, just dropped by your desk and asked if you would analyze the test results and make some recommendations to him based on what you find. In particular, he wants to understand whether their ads really do work, and whether they should stick with the current ad design or switch to the new alternative design.

Background on Advertisements

On Yelp, each restaurant has a profile page with operational information including its hours, phone number, and location. Yelp users who have visited the restaurant are encouraged to leave reviews for other users. Users can also discover and search for restaurants on the Yelp platform using filters, and make reservations and even order food through the restaurants profile page.

The majority of Yelp’s revenue comes from selling ads to restaurants. The sales team cold-calls restaurants and tries to convince them to advertise on the platform. Advertisements (labelled as “Sponsored” in the search results) are placed in the premium position above the organic search results that users conduct. The advertising package sold to restaurants costs $300 / month, and advertisers are required to sign a minimum one-year contract.

Yelp uses a search algorithm similar to Google’s, that determines when and which ads are shown given a user’s search for restaurants on the platform. Restaurants have little say in what search terms will trigger their ads, however, they are guaranteed that their ads will be shown a minimum of 1,000 times per month to Yelp users.

Yelp’s current search algorithm shows ads for restaurants triggered by the type of cuisine within a one-half-mile radius of the user. For example, if a user is standing in Boston’s Harvard Square and searches for Italian food, the algorithm will choose two Italian restaurants within a half mile of Harvard Square to show “Sponsored” ads.

The engineering team has run a variety of tests looking at how users responds to different types of ads in different searches, and designed a alternative search algorithm (which was used in the test). In the alternate algorithm, rather than choosing two restaurants by cuisine type (i.e. Italian) the alternative algorithm shows advertisements when a user searches for a specific restaurant and then selects two other restaurants with similar ratings and hours.

While Yelp management is reasonably satisfied with how the current algorithm shows ads, they are open to the possibility that the alternative algorithm might be significantly better (or worse) in providing the benefits to the advertisers.

The Experiment

For the experiment, Yelp randomly selected 30,000 restaurants that were active on their platform but not currently advertising, yielding a sample that is representative of the population of restaurants in the USA.

For the one-month duration of the test, 10,000 restaurants were randomly selected to receive free ads using the current algorithm and design approach; another 10,000 restaurants were randomly selected to receive free ads using the “new” alternative algorithm / design; the remaining 10,000 restaurants received no advertisements - their listings were shown in the organic search results only. None of the restaurants were informed of the experiment to keep the data and the test secret.

The Data

Jeremy just emailed to you a spreadsheet containing the experiments results so you can analyze the data.

There are two different types of restaurants: Chain (Chili’s, Applebee’s, Olive Garden, etc), and Independents (El Toro Viejo, Elements, Angie’s, etc). You’ll need to look at those separately.

You have the following data for each group (Chain, Independent, Control)

# of ads shown using the current algorithm

# of ads shown using the new algorithm

# of ads shown just using the organic search (the control group)

For each restaurant, you have three pieces of information:

Page Views

Calls to the Restaurant

Reservations Made

Your task is to analyze the data, and write a one-page P&G memo summarizing your findings and recommendations to CEO Jeremy Stoppleman.

As you do your analysis, don’t just stop at figuring out if one ad performed better than the other. Look deeper. What other things did you find? Are there any things that look odd and perhaps need to be reported to Mr. Stoppleman? Look at it not just from the perspective of Yelp, also look at it from the perspective of the restaurant owner as well. You are an analyst. You need to analyze everything, not just what you’re asked to do. Become an expert.