The Influence of Software Process Maturity and Customer Error Reporting on Software Release and Pricing
Terrence August and Marius Niculescu
Management Science, December, 2013, 59(12), 2702-2726
With modern software assurance strategies that leverage users’ systems and their interconnectedness, a software firm must carefully select product release dates that account for several important trade-offs. On one hand, by releasing earlier, the firm’s product is available for a longer time in the market before it becomes outmoded; the firm reduces its cost of detecting and fixing software bugs due to error reporting; and the firm may even enjoy competitive benefits associated with being first to market (Cohen et al. 1996). On the other hand, earlier-released software bears a lower initial quality which has several effects: users incur costs associated with security attacks and poor application performance in the interim; users, in turn, impose goodwill costs on the firm; and the speed at which consumers adopt the software, i.e., the rate of diffusion, is reduced (Keizer 2007).
These aforementioned trade-offs will be strongly influenced by the distinct characteristics of the firm, its product, and the corresponding market. One particularly relevant characteristic of a software firm is its software process maturity which is “...the extent to which a specific process is explicitly defined, managed, measured, controlled, and effective” (Paulk et al. 1993). The federally funded Software Engineering Institute (SEI) established a set of Capability Maturity Models (CMMs) to help organizations improve processes with emphasis on those related to software development. Some of these basic models were superceded by SEI’s Capability Maturity Model Integration (CMMI), but the essence is much the same. CMMI defines five levels of maturity in software-producing organizations that range from Level 1, which is characterized by ad hoc processes that can complete goals, to Level 5, where processes are already rigorously defined and managed in a quantitative way, and the focus now lies on optimizing them (SEI 2006). One useful aspect of the CMMI is that firms at the same level tend to be similar on other dimensions as well. For example, Table illustrates the relationship between a firm’s software process maturity and the quality of its software, measured in terms of defects per thousand lines of code. This positive relationship between process maturity and quality as well as other software development characteristics has also been established in the academic literature (see, e.g., Harter et al. 2000, Harter and Slaughter 2003). Hence, by understanding how bug density interacts with the trade-offs that affect software release which are described above, we can provide broad implications on how firms at a given level of process maturity should manage software adoption and even identify the value that underlies achieving a higher level of maturity.
In this paper, using an analytical model, we formally examine how a software firm adjusts its release timing and pricing in a setting where it can harness customer error reporting. In this context, we specifically study the influence of the firm’s pricing power, its software process maturity, and the characteristics of its product class, as well as its relative quality within that class on its timing and pricing decisions.
We characterize relevant bounds on a firm’s software release time and relate these bounds to the firm’s process maturity and its product’s level of functionality. We then explore how consumer feedback affects adoption dynamics, and, in turn, the firm’s associated revenues and costs. For sufficiently high software development process maturity, the bounds collapse suggesting that the firm should release its product as soon as it can generate adoption. Said differently, firms that utilize mature development processes (e.g., those that have obtained a higher CMM Level) should release their products to the market immediately when they have a release candidate available. Although there are always some negative effects on adoption stemming from reduced quality due to bugs early in the product lifecycle, at a high level of maturity, a firm’s initial software quality still tends to be high and an early release boosts adoption through network effects which can further increase software quality stemming from customer error reporting. Thus, early release strategies can lead to better software products in the marketplace when firms are mature in their development processes.
Next, we examine how firms adapt their software release timing to changes in the customer error reporting rate. The net effect of the interaction between the optimal release time and the level of error reporting is critically determined by goodwill costs, the cost of processing user-generated error reports, error reporting rates of the firm and userbase, bug-fixing rates, functionality, and the various parameters describing the adoption curve. When the ratio between goodwill costs and the processing costs associated with user-generated error reports is relatively low and the strength of the error reporting rate from users compared to the firm is also low, then the optimal release time increases in response to an increase in the error reporting rate. When goodwill costs are low, the firm has incentives to release its product early on. With an increase in the fraction of users contributing to error reporting, the firm can release even earlier to further harness these contributions particularly in light of the small goodwill costs. On the other hand, if the cost of processing user reports is large in comparison to goodwill costs and the relative strength of total user error reporting rate relative to the firm’s is small, then the firm also has incentives to incur further detection costs itself and release a more polished product on the market later. In this case, the latter effect of this trade-off is stronger, and the firm may optimally prefer to shift its release time outward.
In contrast, when goodwill costs are high, despite obvious drawbacks of releasing its software early, a software firm will overall prefer to release earlier as the fraction of users participating in error reporting increases provided that the cost of processing these user reports is not too substantial. In this case, it is critical that the firm is able to remedy defects quickly in order to effectively offset the high goodwill costs while benefitting from the increased user error report contributions as they increase software quality.
We also study how firms adapt their software release timing to changes in the amount of product functionality. When the impact of this change on user contributions is positive and contributions are large, an increase in functionality boosts the total detection rate stemming from users. In this case, there is again a distinct trade-off. First, there are incentives to release the software earlier in order to benefit from stronger user contributions. On the other hand, the firm also wants to limit both goodwill costs stemming from early release as well as potential additional costs associated with processing of user-reported defects. We find that when firms have higher development maturity, and correspondingly less bugs in their products, they can release earlier in order to benefit from more user contributions as long as processing costs are not too high. High maturity and lower processing costs together limit the downside costs, making it optimal to decrease release time as a consequence to increased functionality. On the other hand, when a firm has lower software maturity, it may find it preferable to delay release to limit exposure to the additional processing and goodwill costs stemming from the higher bug density found in its product.
Connecting our findings with what we observe in industry, we discuss a recent example. The Apple iOS 5 mobile operating system was released relatively quickly after its previous iteration and initially contained flaws and pre-configured settings that severely diminished handset battery performance (Albanesius 2011). Although Apple does not sell its mobile operating system separately to customers, the goodwill impact of poor quality can be high given its complementary handset and application (app) market businesses. Apple uses mature software development processes and takes a structured approach to quality assurance, using many channels to receive user feedback. Moreover, a great number of Apple iOS 5’s incremental new features were aimed at enhancing user experience and facilitating the development of apps on its platform. Both of these classes of enhancements are likely to have generated a significant amount of attention and usage from their targeted communities, leading to potentially substantial feedback in case of software flaws. Our findings discussed above suggest that Apple had incentives to release earlier, which was confirmed ex-post.
Broadening our study, we explore how beta testing interacts with the firm’s release time decision. Because the focus of our work centers on release timing and pricing in the context of user error reporting contributions, we abstract from initial software development decisions taking a software product as given (i.e., a release candidate is available at time zero). In the typical software release cycle, beta testing is a precursor to the release candidate stage, and therefore issues such as when to cease development (quality choice) and begin alpha/beta testing become more relevant. In that light, in this work, we do not intend to provide a comprehensive understanding of how to manage beta testing. Instead, we hope to provide some basic insights into how the existence and degree of beta testing might impact a firm’s software release timing, which is a fundamental concern in our study.
We find that when a firm with high process maturity produces a high quality product and there are not many defects in the code, it should delay software release as the beta test size increases. When there are few defects in the code, the firm will not incur significant additional processing costs due to user reported bugs. However, it incurs testing costs at each moment while searching for these defects. As the size of the beta test increases, the firm faces a trade-off: release earlier to reduce its own testing cost while harnessing a larger beta testing population for a shorter period of time, or release later to utilize the larger beta testing population for a longer period of time, which improves quality and reduces goodwill costs but also leads to higher testing costs. For a high maturity firm, the trade-off described above tilts toward the latter, and it should prefer a delayed release strategy. For example, Blizzard Entertainment, a producer of very popular PC games, has long been known for the high quality of their products. Recently, they were in the beta testing phase of the third edition of one of their most successful role-playing games. During that phase, they announced that both the beta test would be extended and the size of the beta testing group would also be increased, two moves that should be performed in lockstep for a high maturity producer (Hachman 2011).
Above, we discussed a software firm’s optimal release timing and profitability when it utilizes release time as its primary lever to manage adoption and error reporting. In our study, we also explore a setting where a software firm has some pricing power, optimally selecting a single price to charge throughout the selling horizon of the product. There are many instances where software firms prefer to set price at an optimal level and keep it to a large extent at that level for the entire selling horizon. For example, in the past several years, AutoCAD suite and DivX video software bundle have been priced at $3995 and $19.99 respectively for a perpetual license, whereas WinEdt’s non-commercial single user license has been priced at a steady $40 over the last decade. Our aim is to better understand what role each lever plays to manage adoption and harness the benefits of error reporting.
If error contributions from users are large and goodwill costs are low, the preceding implications for higher maturity firms expand to firms of all maturity, i.e., they tend to release immediately. When a firm releases its software at the earliest time possible, it uses a two-pronged strategy. First, immediate release jump-starts adoption and permits error reporting feedback earlier in the software’s life cycle. Second, the firm finds it profitable to price the software (and, implicitly, shape adoption) in a manner that can help prevent overexposure to lower initial quality. Generally, when a firm can set a single, optimal price, we find that comparative statics of the proportion of user participation in error reporting and software functionality on the firm’s price align well with our comparative statics on release time (i.e., the optimal price moves in the same direction as the optimal release time, for similar reasons).
For example, when software is produced by a highly mature firm, it has fewer bugs at the release candidate stage. Given that the firm is performing extensive testing across the entire codebase, if it can efficiently harness user contributions to help detect the bugs that remain, it has incentives to reduce its own internal testing. This strategy can be achieved by releasing early and pricing in a balanced way such that the installed base leads to a significant reduction in testing costs without severely impacting revenue. The higher the user error reporting rate, the lower the optimal price is because a larger resulting installed base is an effective substitute to internal testing; detection costs are outsourced to the user base without compromising quality. This effect is strongest when goodwill costs are high because, in this case, the potential benefits of faster quality improvement by harnessing users are the largest. Second, the cost associated with processing user error reports should not be too high such that leveraging users remains economical.
Although our work is normative in nature, we briefly present some suggestive evidence that our finding here can be supported. Over time, Microsoft developed and improved Windows Error Reporting (WER) to harness consumer bug detections toward improving quality (Glerum et al. 2009). Microsoft also released successive generations of products such as Office Professional (i.e., 2000, XP, 2003, 2007, and 2010). The suggested retail prices of these versions of Office Professional, as announced by Microsoft, were $599, $579, $499, $499, and $499, respectively, and in this last case for the 2010 version, an alternative option to purchase the product key card instead was made available for $349 (Microsoft 2001, Arar 2003, Microsoft 2007, Meredith 2010). Given that Microsoft has high software maturity, part of its decreasing price trend over versions can be explained by the substantial increase in error reporting and its strategic outsourcing of its testing costs.
Glerum, K., K. Kinshumann, S. Greenberg, G. Aul, V. Orgovan, G. Nichols, D. Grant, G. Loihle, and G. Hunt (2009). Debugging in the (very) large: ten years of implementation and experience. In Proc. ACM SIGOPS 22nd symposium on Operating systems principles, SOSP ’09, Big Sky, Montana, pp. 103–116.
Paulk, M. C., B. Curtis, M. B. Chrissis, and C. V. Weber (1993). Capability maturity model for software, version 1.1. Technical Report. Carnegie Mellon Univ., Software Engineering Institute. CMU/SEI-93-TR-024.
- Workshop on Information Systems and Economics 2011 (WISE), Shanghai, China, December 8, 2011
- INFORMS Annual Meeting 2011, Charlotte, NC, November 15, 2011