Contact InformationJoleen Schultz
For press inquiries:
Intuition Can’t Beat Experimentationby Professors Ayelet Gneezy and Uri Gneezy
In the summer of 2009, we received a call from Joe, a winery owner in Temecula. Joe asked for our help pricing his wines – clearly one of the most important decisions he needs to make. We were happy to go visit Joe, taste some wines and possibly help him in the process.
When we asked him how he’d chosen prices in the past, we encountered the usual suspects: looking at how other wineries price similar wines, intuition, inertia, etc. Joe expected the business professors to come over, look around, do some quick calculations – and come up with the magic number that would make him rich. You can imagine how disappointed he was when, having spent some time with him (and his wine), we told him we had no idea what the “right” price was and that the magic number didn’t exist. He almost took away the wine he’d already poured for us.
In an attempt to save our drinks, we did offer him help. The help was a method – no magic, equations, or superior knowledge – just a simple experimental design. This article provides a brief introduction to a new and exciting field in behavioral economics: experimentation in firms. The new part of this method is the use of economic techniques in the process of designing and running experiments and then analyzing the results. The main takeaway is that in many market instances, experimenting (rather than guessing) is the most accurate, simple and often the cheapest way to know how to approach a challenge, be it the pricing, presentation or promotion of a product.
Why Do We Need Experiments in Firms?
Business experiments are research investigations that present companies with an opportunity to get fast and accurate data regarding important decisions. Manipulating various factors in the environment could be used to better understand the causal relationship between a change in strategy and a response in consumers’, employees' or other stakeholders’ behavior. Typical examples of business experiments are the use of a promotion on a test population, with the goal of determining the optimal conditions (discount depth, discount type, framing, etc.) for maximizing a promotion’s success. This test is different from commonly used focus groups, because participants make real life decisions without even knowing that they are part of a study. When designed properly, business experiments can provide invaluable insights and reveal surprising results, which the company can then implement on a larger scale.
Numerous challenges are associated with predicting market reaction to changes. Research has shown that imagining consumers’ thought processes and reactions is difficult. Product, service and market information is typically more readily available to the manager, but this availability does not necessarily make the prediction easier or more accurate. Even the most educated guesses remain imprecise at best. With this difficulty in mind, an important tool for predicting how the market will receive your efforts is experimentation. A well-controlled experiment provides an unbiased snapshot of how consumers may react to changes.
To understand the spirit of things, imagine going to your doctor, who prescribes a new treatment for you. When you ask her what she bases this treatment on, she gives you the same answer Joe gave us: “By looking at what others have done, intuition, inertia.” This answer would be suspicious at best and not very reassuring. You would prefer your doctor base her decision on empirical research!
Examples: Pricing Wines, Pricing Photos
Let’s go back to Joe, the winery owner. Pricing wines is a particularly tricky task since quality is objective. Intuition asserts that price and quality will be positively correlated. Often, when product attributes are objective and measurable (e.g., the weight of a laptop when lighter is better), finding evidence counter to this basic intuition is difficult. Consequently, substantive literature provides evidence that when the quality of a product is hard to evaluate, increasing its price (without changing actual quality) increases its attractiveness to consumers. Is this also the case with wines?
Visitors to Joe’s winery, as with others in this region, can taste different wines and subsequently choose to buy from the available selection. Consumers typically come to this region for wine trips, going from one winery to another, sampling and buying wine. The wine we experimented with was a 2005 Cabernet Sauvignon. The price Joe had previously chosen for it was $10, and it sold well.
For our experiment, we manipulated the price of the Cabernet to be $10, $20 or $40 on different days during two weeks. Each experiment day, Joe greeted the visitors and told them about the tasting. Then visitors went to the counter, where they met the person who administered the tasting and gave them a single printed page containing the names and prices of the nine included wines, ranging from $8 to $60, of which visitors could try six of their choice. As in most wineries, the list was constructed from light to heavy, starting with white wines, moving to red wines and concluding with dessert wines. Visitors typically choose wines going down the list, and the Cabernet Sauvignon was always number 7. Tastings take between 15 and 30 minutes, after which visitors decide whether to buy any of the wines.
The results shocked Joe. Visitors were almost 50 percent more likely to buy the Cabernet when he priced it at $20 than when he priced it under $10. Using an almost costless experiment and adopting prices accordingly, Joe increased the winery’s total profits by 11 percent.
Interestingly, following this experiment, Joe happily adopted the results and changed the price of this wine to $20. However, he did not adopt the experimental method and didn’t plan to in the future.
On a larger-scale experiment, we (together with Leif Nelson and Amber Brown) tested a new pricing strategy in one of Disney’s amusement parks. We suggested Disney test a new model of corporate social responsibility that allows customers to take greater ownership in their donations. In the field experiment with over 100,000 participants, we manipulated two factors in the sale of souvenir photos. First, customers saw either a traditional fixed price or could choose their price (including $0). Second, half of those customers saw a variation in which half of the revenue went to charity.
We found that at a standard fixed price, the charitable component only slightly increased demand. However, when participants could choose their price, the same charitable component created a substantially more profitable treatment. Switching from corporate social responsibility to what we termed shared social responsibility works in part because customized contributions allow customers to directly express social welfare concerns through the purchase of material goods.
From a business perspective, Disney’s profit from this change will amount to over $600,000 a year, just in this one location in the park. More generally, this change also increased the benefit to the charity and presumably to the customers who felt they were doing something good.
So why don’t businesses experiment more? A number of barriers make implementing experimentation in firms difficult. For example, the students who helped us write this article were recruited in order to help with a field experiment on incentives in a large company. This was back at the end of the summer, nearly a year ago. We naively believed that by now we’d have the data that they could analyze. In reality, the study is still buried somewhere in the big organization, waiting for management approval.
At existing companies with standard procedures, instilling a culture of experimentation is hard, especially if past success has occurred independent of experimentation. Some of the typical barriers are managers that are intimidated by the uncertainty involved in a change and the unknown. Going the traditional way without introducing new methods is familiar and as long as it works, it seems safer (“if it ain’t broke, don’t fix it”). Another barrier is that managers feel they’ve been hired to provide solutions and make tough decisions to enhance the firm’s performance. In other words, they feel they are expected to have ready answers for the challenges the firm faces. Opting for experimentation may appear to imply they don’t and could compromise their level of expertise – as if they have failed to do their job.
One could overcome these barriers in two distinct ways: top-down and bottom-up. Under top-down, the company's managing team would need to overcome the typical “short-term earnings first” mindset and encourage (and even reward) experimentation that can improve the firm’s performance. This approach would require some level of training that would provide the relevant employees with the skill set to design and run experiments, analyze the data and draw conclusions. Under a bottom-up approach, lower-level managers could conduct smaller-scale field studies and then present the results to the management, providing them with the costs and benefits associated with expanding the experimentation and potentially defining new practices (prices, promotions, etc.).
Changing a tried – if not so true – mindset is no small feat. We suggest starting with future leaders by providing them with the concepts and basic skill sets associated with experimentation via the curriculum of MBA students. Education, combined with hands-on experience, would help decrease suspicion and skepticism and provide tomorrow’s leaders with some tools that would allow them to make a difference in the way companies make decisions.
Uri Gneezy is the Arthur Brody Endowed Chair in Behavioral Economics and professor of economics & strategy at the Rady School. Before joining the Rady School, Gneezy was a faculty member at the University of Chicago, Technion and Haifa. He received his Ph.D. from the CentER for Economic Research in Tilburg.
Ayelet Gneezy is an assistant professor of marketing at the Rady School. Prior to graduate school, she worked as a consultant in marketing services to companies in various industry areas including consumer goods, banking services and the nonprofit sector. Gneezy received her Ph.D. in marketing from the University of Chicago Booth School of Business.
Research assistance was provided by Eric Dorflinger, Joshua Rutenberg, Ning Ge, Poornima Suresh, Sam Chi and Toshi Nakamura.