Defect Management Process

Without a doubt, the best approach to defects is to eliminate them altogether.  While that would be ideal, it is virtually impossible given current technology.  In the meantime, developers need strategies to find defects quickly and minimize their impact.  Identifying and implementing the best defect prevention techniques (which is a large part of identifying the best software development processes) should be a high priority activity in any defect management program. 

 

Defect Prevention : Implementation of techniques, methodology and standard processes to reduce     the risk of defects.

Deliverable Baseline :  Establishment of milestones where deliverables will be considered complete and ready for further development work.  When a deliverable is baselined, any further changes are controlled.  Errors in a deliverable are not considered defects until after the deliverable is baselined.

Defect Discovery :  Identification and reporting of defects for development team acknowledgment.  A defect is only termed discovered when it has been documented and acknowledged as a valid defect by the development team member(s) responsible for the component(s) in error.

Defect Resolution : Work by the development team to prioritize, schedule and fix a defect, and document the resolution.  This also includes notification back to the tester to ensure that the resolution is verified.

Process Improvement : Identification and analysis of the process in which a defect originated to identify ways to improve the process to prevent future occurrences of similar defects.  Also the validation process that should have identified the defect earlier is analyzed to determine ways to strengthen that process.

Management Reporting : Analysis and reporting of defect information to assist management with risk management, process improvement and project management.



Defect Prevention
Defect prevention should begin with an assessment of the critical risks associated with the system.  Getting the critical risks defined allows people to know the types of defects that are most likely to occur and the ones that can have the greatest system impact.  Strategies can then be developed to prevent them.  The major steps for defect prevention are as follows

Defect Prevention Process

Identify Critical Risks :  Identify the critical risks facing the project or system.  These are the types of defects that could jeopardize the successful construction, delivery and/or operation of the system.

Estimate Expected Impact :  For each critical risk, make an assessment of the financial impact if the risk becomes a problem.

Minimize Expected Impact : Once the most important risks are identified try to eliminate each risk.  For risks that cannot be eliminated,  reduce the probability that the risk will become a problem and the financial impact should that happen.

Deliverable Baseline
A deliverable (e.g. work product) is baselined when it reaches a predefined milestone in its development.  This milestone involves transferring the product from one stage of development to the next.  As a work product moves from one milestone to the next, defects in the deliverable have a much larger impact on the rest of the system, and making changes becomes much more expensive.  A deliverable is subject to configuration management (e.g., change control) once it is "baselined." 

For purposes of this model, a defect is defined as an instance of one or more baselined product components not satisfying their given set of requirements.  Thus errors caught before a deliverable is baselined would not be considered defects.  For example, if a programmer had responsibility for both the programming and the unit testing of a module, the program would not become baselined until after the program was unit tested.  Therefore a bug discovered during unit testing would not be considered a defect.  If, on the other hand, an organization decided to separate the coding and unit testing, it might decide to baseline the program after it was coded, but before it was unit tested.  In this case, a bug discovered during unit testing would be considered a defect (See Figure below).

Even for organizations that do not recognize this concept, a deliverable is, for practical purposes, baselined when the deliverable passes to the next stage of development.  For example, developers should consider a program specification baselined when a programmer uses it as the basis to code a program.  A program should be considered baselined when developers pass it on for integration testing.  Finally, developers should consider a requirements specification baselined if it is being used as the basis for a technical design.

The concept of baselining is important because it requires an organization to decide both the level of formality that is appropriate, and the point in the process when the formality takes effect.  In general, a deliverable should be baselined when changes to the deliverable, or defects in the deliverable, can have an impact on deliverables other people are working on.

Deliverable baselining involves the following activities:

Identify Key Deliverables:  Select those deliverables that will be baselined and the point within the development process where the deliverable will be baselined.

Define Standards for Each Deliverable:  Set the requirements for each deliverable and the criteria that must be met before the deliverable can be baselined.

Defect Discovery
If technology can not guarantee that defects will not be created, then defects should be found quickly before the cost-to-fix becomes expensive.  For purposes of this model, a defect has been discovered when the defect has been formally brought to the attention of the developers, and the developers acknowledge that the defect is valid.  A defect has not necessarily been discovered when the user simply finds a problem with the software.  The user must also report the defect and the developers must acknowledge that the defect is valid.  Levinson and Turner provide an example where users reported problems for years before the developers of the software admitted there was a defect [LEV93].  Since it is important to minimize the time between defect origination and defect discovery, strategies need to be implemented that uncover the defect, and facilitate the reporting and acknowledge of the defect.

To make it easier to recognize defects, organizations should predefine defects by category.  This is a one-time event, or an event that could be performed annually.  It would involve the knowledgeable, respected individuals from all major areas of the IS organization.  The group should be run by a facilitator.  The objective is to identify the errors/problems that occur most frequently in the IS organization and then get agreement that they are, in fact, defects.  A name should be attached to each category of defect.  The objective of this activity is to minimize conflicts over the validity of defects.  For example, developers may not want to acknowledge that a missing requirement is a defect, but if it has been previously defined as a defect category then that conflict can be avoided.

Defect Discovery Process
The steps involved in defect discovery are as follows:

 

Click on the following links to read more about these steps:

 

Find Defect :  Discover defects before they become major problems.

Report Defect :  Report defects to developers so that they can be resolved.

Acknowledge Defect :  Obtain development acknowledgement that the defect is valid and should be addressed.

 

Defect Resolution

Once the developers have acknowledged a valid defect, the resolution process begins.  The steps involved in defect resolution are as follows:


Defect Resolution Process

Click on the following links to read more about these steps:
Prioritize Risk  :  Developers determine the importance of fixing a particular defect.
Schedule Fix and Fix Defect :  Developers schedule when to fix a defect.  Then developers should fix defects in order of importance.
Report Resolution :  Developers notify all relevant parties how and when the defect was repaired.

Process Improvement
This is perhaps the activity that is most ignored by organizations today, but offers one of the greatest areas of payback.  NASA emphasizes the point that any defect represents a weakness in the process.  Seemingly unimportant defects are, from a process perspective, no different than critical defects.  It is only the developer's good luck that prevents a defect from causing a major failure.  Even minor defects therefore represent an opportunity to learn how to improve the process and prevent potentially major failures.  While the defect itself may not be a big deal, the fact that there was a defect is a big deal.
Based on the study's findings, participants should go back to the process that originated the defect to understand what caused the defect.  Then they should go back to the validation process that should have caught the defect earlier in the process.  Not only can valuable insight be gained as to how to strengthen the review process, this step serves to make everyone involved in these activities take them more seriously.  This human factors dimension alone, according to some of the people the research team interviewed, can have a very large impact on the effectiveness of the review process.
NASA, takes an additional step of asking the question:  If this defect could have gotten this far into the process before it was captured, what other defects may be present that have not been discovered.  Thus not only is the process strengthened to prevent defects, it is strengthened to find defects that have been created, but not yet discovered.  This aggressiveness should be mandatory on life critical systems.  The well-known "Thearc-25" case of a radiation machine that kept issuing "Malfunction 25" messages is one example where problems might have been caught earlier if this step had been pursued [LEV93].

Management Reporting

 It is important that the defect information, which is a natural by-product of the defect management process, be analyzed and communicated to both project management and senior management.  This could take the form of defect rates, defect trends, types of defects, failure costs, etc.  From a tactical perspective, defect arrival rate (rate at which new defects are being discovered) is a very useful metric that provides insight into a project's likelihood of making its target date objectives.  Defect removal efficiency is also considered to be one of the most useful metrics, however it can not be calculated until the system is installed.  Defect removal efficiency is the ratio of defects found prior to product operation divided by the total number of defects found in the application.

 

Information collected during the defect management process has a number of purposes:

 

·         To report on the status of individual defects.

 

·         To provide tactical information and metrics to help project management make more informed decisions -- e.g., redesign of error prone modules, the need for more testing, etc.

 

·         To provide strategic information and metrics to senior management -- defect trends, problem systems, etc.

 

·         To provide insight into areas where the process could be improved to either prevent defects or minimize their impact.

 

·         To provide insight into the likelihood that target dates and cost estimates will be achieved.

 

 

 

Management reporting is a necessary and critically important aspect of the defect management process, but it is also important to avoid overkill and ensure that the reports that are produced have a purpose and advance the defect management process.

 

The basis for management reporting should be the information collected on individual defects by the project teams.  Thus the information collected during the defect management process and the classification of individual defects needs to be considered by each organization.