A Methodology to Support Software Release Decisions

Software is everywhere and has become a major worldwide industry. We find software embedded, for example, in watches, coffee makers, cars, televisions, airplanes, telephones, reservation systems, and medical equipment. Software not only pervades a multitude of products, but also is an important corporate asset, and demand is increasing. Yet software projects are characterized by schedule and budget overruns and the delivery of unreliable and difficult to maintain software products.

Despite an exponential increase in the demand for and dependence on software, many software manufacturers exhibit unpredictable behavior. It is sometimes difficult to determine, for example, a software product’s release date, its features, the associated development costs, or the resulting product quality.

Without knowing the release date, a software manufacturer experiences difficulty planning product promotions, customer training, and maintenance support. Resource utilization across projects may become inefficient and difficult to manage when projects fail to meet schedules. Customers have difficulties planning for the introduction of new software into their organizations when a scheduled release date is missed.

The exponential growth of software suggests that end-users will be exposed to more defects if the software industry is not able to exponentially improve its defect potentials and removal efficiencies. Combined with increased competition and smaller market windows, software manufacturers likely will be exposed to higher levels of uncertainties when releasing software. The decision to release a software product will become an even more complex and important decision.

A Software Release Methodology Supporting Strategic Value

Existing methodologies, models, and standards reveal limited guidance for structuring software release decisions in a methodological way to support strategic value. A decision has strategic value when it has the potential for large prospective financial losses to a software manufacturer or its customers or end-users. Software release decisions often have strategic value due to high costs for reversing the decision. Prospective loss outcomes also may arise long after the decision has been made, for example, in cases where liability leads to lawsuits.

A software release decision can be seen from different perspectives:

·     Maximizing behavior. A software release decision is a trade-off between early release to capture the benefits of an earlier market introduction and the deferral of product release to enhance functionality or improve quality. If a software product is released too early, the software manufacturer incurs post-release costs of later fixing failures. If a software product is released too late, the additional development cost and the opportunity cost of missing a market window could be substantial. These two alternatives need to be compared to determine which alternative maximizes economic value.

·        Optimizing behavior. A release decision deals with the difficulty of verifying the correct implementation of functional and non-functional requirements. How much testing is needed? Software manufacturers must find the optimal level of information because information has its price in cost and time. In practice, cost and time constraints will normally be present in retrieving complete and reliable information, so this search for information should be taken into account as an economic activity. This leaves the software manufacturer with the problem of finding the optimal level of information where marginal value equals marginal costs and marginal yield is zero. This optimal level is difficult, if not impossible, to find.

·        Satisfying behavior. Decision-making in the real world is often unstructured and normally involves various stakeholders who may, for example, have reasons to release a system or software product due to political or business pressures even though they know it still contains defects. A study of spacecraft accidents, for example, reveals that, although inadequate system and software engineering occurred, management and organizational factors played a significant role, including the diffusion of responsibility and authority, limited communication channels, and poor information flows.

·       Decision Implementation. A decision is only considered successful if there is congruence between the expected outcome and actual outcome, which sets requirements for decision implementation. In practice, there are many obstacles to the successful implementation of almost any decision, including:

            
1.    The reduced importance of a decision once it is made and implemented
2.    The control of the outcome of a decision by stakeholders not involved in its making 
3.    The development of new situations and problems to command the attention of the decision- makers  once the choice has been implemented

Research Findings

This research reviewed these different perspectives from both a theoretical and an empirical point of view by studying practical examples. The results helped to frame a proposed methodology, called release decision methodology, to address a software release decision from different perspectives. (See Appendix [Sidebar].) The methodology, consisting of a coherent set of practices, combines insights from economics, software management, and social psychology disciplines.

Studies in three different organizations validated the methodology. One participating organization is a leading global financial services company that provides financial services and products to retail and business markets. Services include insurance, pensions, occupational health and safety, asset management, investments, leasing, real estate, venture capital, and mortgage finance.

The research examined a project in this organization’s IT department, which develops custom systems for internal and external use. The initial estimate for the schedule was 10 months and for the pre-release cash outflows, Euro 15M. Budgets were reserved for the technical infrastructure and post-release cash outflows for maintenance and exploitation. During the first months, the project encountered several setbacks: technical problems surfaced and the development budget turned out to be too optimistic. Progress control was lacking, mainly through the absence of clearly-defined milestones or quality gates. These problems increased, and in November 2001, the project was re-defined. Both Senior Management and Marketing exerted pressure on the Product Development Team to release the product as soon as possible. However, the team was faced with an unstable product under test and had to use a veto several times to postpone a scheduled release date. When the product was released, uncertainty was high because many known problems were not resolved (although not considered critical), and the organization judged that continued testing would reveal more defects, including potentially critical ones, that could severely hamper the correct functioning and stability of the product.

After the product was released, a special task force assumed responsibility for temporarily performing corrective maintenance activities. This team needed more than a year to resolve the known and newly-detected defects. Despite the original requirement to develop a maintainable product, the organization decided in 2004 to start a pre-study toward a totally new product to replace this product because corrective maintenance and functional enhancements proved difficult and costly. In other words, the early release of the product saved the organization additional testing cost, but the post-release maintenance cost turned out to be significantly higher than expected. A retrospective review of this project using the software release methodology enabled this organization to assess the project from a release decision point of view.

Figure 1 illustrates how the organization scored on the identified practices in the methodology. Lack of a product development strategy (release definition) and lack of information as input to the decision-making process (release information) led to a poorly structured release decision-process without consensus among the stakeholders involved (release decision). Sufficient financial resources "saved" the organization on the short-term by patching the released software.

Figure 1: Radar Presentation of Case Study Results.

Based on these validation results in a practical setting, the software release methodology displayed a descriptive and a judgmental character, and it can therefore support understanding, analyzing, assessing, and improving the capability of software manufacturers in this problematic area, at least in the studied environments. Currently, ongoing research in software manufacturer environments is under way to study the effects of applying the methodology at the start of projects to proactively aim for release decision success.

Appendix: Release Decision Methodology

Figure 2: Release Decision Methodology.

In the release decision methodology framework, four areas in the software release decision-making process are distinguished, each addressing the process from different perspectives. A process area is defined as a cluster of related practices that, when performed collectively, achieve a set of goals considered important for establishing process capability in that area. Each process area consists of four relevant practices, describing "what" is to be accomplished but not "how." Through this approach, the descriptions of practices still offer the possibility for interpretation and customization to the external market environment and to internal strategic and functional characteristics of a software manufacturer organization.

Identified process areas are (see Figure 2):

1.     Release Definition. Decision-making is mainly viewed from a quantitative perspective, assuming that information is near perfect: complete and reliable. It emphasizes the maximizing behavior approach with emphasis on mathematics, economics, and statistics. In software release decisions, decision-making from a quantitative perspective is concerned with the definition and control of a product development strategy setting the managerial objectives with their priorities and ensuring they are attainable. The availability of a product development strategy enables the comparison and evaluation of different release alternatives, answering the question: which alternative maximizes economic value?

2.     Release Information. This process area is concerned with the search for alternatives during product development, for example, the identification and collection of information that is needed to compare and evaluate different release alternatives. This search is derived from the formulated product development strategy. Decision-making is also viewed from a quantitative perspective, but with the recognition that information is imperfect in the sense that not everything can be expressed in numbers and that information has its price in time and money. For this process mathematics, economics, and statistics still play an important role, but the maximizing behavior approach is extended with an optimizing behavior approach: what is the optimal volume of information? Insufficient information increases uncertainty and hampers the decision-making process, whereas too much information is a waste of scarce resources. There is an optimum above which the cost for searching for more information exceeds the benefits.

3.     Release Decision. Decision-making is viewed from a psychological, sociological and socio-psychological perspective, addressing factors that influence individual and group behavior. It recognizes the imperfections of information, and stakeholders involved in the choice will possibly have different preferences with respect to the decision outcome; an open decision-making process. The challenge is to use a judgmental strategy to reach a decision outcome that meets the formulated objectives and is agreeable to all stakeholders involved. The concept of optimizing behavior is extended with a satisfying behavior approach: which outcome satisfies the needs of all stakeholders involved?

4.     Release Implementation. Decision-making is viewed from an implementation perspective once a decision has been made and is implemented, assuming a successful decision requires follow-up and control. For software release decisions, it is necessary to identify the factors that ensure congruence between the expected and the actual outcome. To increase organizational learning, the decision-making process and its outcome should be evaluated.