Improving software quality with a reliability improvement warranty.

Folse, Raymond O. ; Cope, Rachelle F. ; Cope, Robert F., III 等

ABSTRACT

A Reliability Improvement Warranty (RIW) contract establishes a fixed price for a given level of performance based upon the anticipated number of failures and the cost of each repair action. The anticipated number of failures over the warranty period is determined by assuming that the reliability of the warranted item will improve from an initial level to some specified level. The philosophy behind RIW is that once the fixed price warranty contract is established the profit realized is dependent upon the equipment's reliability.

In our current software industry, the need for a commitment to software quality is evident. Although the subject of software warranty has been addressed in literature, the subject of RIW for software has not been given much attention. In this research, we explore RIW as a vehicle for improving the reliability of software. We focus on the development of a model to determine the cost of a software-related RIW contract and employ the Pearl-Reed economic growth model to emulate the effect of equipment modifications on the reliability of an item under contract.

INTRODUCTION

The software industry has been in existence for about five decades. During this time, it has survived its backwards progression of maturity, and has taken some strides forward in the development of quality software. However, the question of software quality still remains an evasive issue. In an internal e-mail on January 15, 2002 to employees, Bill Gates coined the term "trustworthy computing" to describe his ambitious goal of software improvement. "As an industry leader, we can and must do better," he wrote.

At this point in time, no one would disagree with Bill Gates. Gates' e-mail was sent the very day the company recovered from a five-day stretch of system glitches causing Microsoft's Windows update feature to fail intermittently. Users were unable to download software or security-related updates. A month earlier, Microsoft had to fix a potentially serious security hole in Windows XP, which it touts as its most secure operating system yet. Also in 2001, a spate of Internet worms infected Windows computers at thousands of companies (Hulme, 2002).

Many information technology (IT) professionals consider the commitment long overdue. Poor software quality and security remain major problems for many businesses as they struggle with a continual flow of upgrades and fixes for software applications. It's an issue that keeps IT departments busy and one that can put their business data at risk.

Microsoft is by no means alone in dealing with reliability shortcomings. Carnegie Mellon University's CERT Coordination Center, a security watchdog group, says the number of software vulnerabilities reported last year more than doubled to nearly 2,500 (Hulme, 2002).

So what's wrong? A fundamental problem with software quality is that programmers make mistakes. Research has shown that even experienced programmers inject about one defect into every 10 lines of code, according to the Software Engineering Institute. If 99% of those are caught, that's still 1,000 bugs in a 1 million-line software application.

In our research, the concept of a Reliability Improvement Warranty (RIW) contract is offered as a vehicle for improving the reliability of software. In linking the theme of using a RIW as a means of improving software reliability, we develop a model to determine the RIW's cost.

Although warranty cost has been a well-explored research topic, the subject of the cost of a RIW has not been given as much attention. Mamer (1987) researched the discounted and per unit cost of three common varieties of warranty without discussing RIW. Murthy and Blischke (1992) concentrated more on the effect of RIW on market behavior and the resulting social welfare implications. To extend the research, our work employs the Pearl-Reed economic growth model to emulate the effect of equipment modifications on the reliability of an item under contract.

SOFTWARE QUALITY

Quality is obviously a subjective term. The definition of quality software will depend on who the "customer" is and their overall situation. Each type of "customer" will have their own slant on quality. For example, the accounting department might define quality in terms of profits while an end-user might define quality as user-friendly and bug-free. In general, one might define quality software as software that is reasonably bug-free, delivered on time and within budget, meets requirements and/or expectations, and is maintainable.

The software life cycle for a system begins when an application is first conceived and ends when it is no longer in use. It includes aspects such as initial concept, requirements analysis, design, test planning, coding, integration, testing, and delivery. Figure 1 that follows depicts the software life cycle. Once delivered, the system requires maintenance, updates, and re-testing. Eventually the system will no longer be relevant and be phased out.

[FIGURE 1 OMITTED]

According to Cho (1980), there are five common problems in the software life cycle that inhibit developers from achieving the goal of quality software. They include poor system requirements, unrealistic schedules, inadequate testing, scope creep of the requirements, and miscommunication. These five common problems can be marginally resolved if:

* Systems analysts develop clear, complete, detailed, cohesive, attainable, testable requirements that are agreed to by all stakeholders.

* Management provides realistic schedules that allow adequate time for planning, design, testing, bug repair, re-testing, changes, and documentation.

* An adequate test plan is executed that starts testing early with adequate time for testing and bug repair.

* Initial requirements should change as little as possible. If changes are necessary, they should be adequately reflected in related schedule changes.

* Walkthroughs and inspections when appropriate should be required to minimize miscommunications. Prototypes should be used throughout the project to clarify the customers' expectations relative to the system.

Unfortunately most software developers do not have the discipline to invoke the solutions to these common problems. In a recent survey by Information Week Magazine, 97% of 800 business-technology managers reported problems with software flaws in the past year, with 9 out of 10 reporting higher costs, lost revenues, or both (Hayes, 2002). Thus, a new approach is required to motivate developers to improve the quality of software.

SOFTWARE RELIABILITY

The reliability of a software system is a dynamic characteristic that is a function of the number of failures. Reliability is related to the probability of an error occurring. A program may contain known errors but may still seem reliable to its users. These users may apply the system in such a way that the system is always reliable. For example, a certain version of Microsoft Word may have an error in its Insert Object Math Equation Editor tool. But if a user never selects this tool, then the system will seem completely reliable (Sommerville, 1995).

To represent the failure rate of software systems, the Poisson distribution is generally used. This distribution leads to a simple negative exponential distribution to define the probability as a function of time. The reciprocal of the failure rate is the mean time between failures (MTBF). Thus, if is the constant failure rate, then the MTBF is 1/ . (Note: Throughout this paper, MTBF will be used rather than the failure rate since it is more convenient in expressing the reliability of a system.)

SOFTWARE WARRANTIES

If there is some doubt that there are persistent problems with the quality of software, look at the warranty on software systems. The warranted system is not guaranteed to meet specific quality standards. A software warranty is a manufacturer's guarantee to replace software that proves to be defective with a working copy of the same product and version. Software warranties do not cover interoperability problems that may arise from one software program interacting with another.

A software warranty is assurance that the supplier of the system will back the quality of the item in terms of correcting any legitimate problems with the item at no cost for a particular period of time or use. Warranties ensure that suppliers accept liability for the level of performance and quality they are offering, making software quality control a necessary element in order to offer software warranties (Brennan, 1994).

In order to establish a quality control program for software, the use of statistical quality control is a reality that cannot be escaped. The use of statistical quality control has come to be a powerful and widely used tool in the manufacturing industry. In manufacturing, a product's warranty is determined through the methods of statistical quality control (Schulmeyer, 1990). The manufacturer passes the cost of the warranty on to the customer in the form of a higher priced good. The manufacturer relies on the quality control principles in order to guarantee that the number of defective items produced does not cause the manufacturer's costs to exceed the additional cost carried by the customer.

The proposition for software warranties is that statistical quality control is just as applicable to software development. However, this is considered to be a myth by many software professionals, particularly those having no background in statistical quality control. It is not surprising, then, that a total commitment to product quality is lacking in the software industry. It is a widely accepted belief in the industry that current technology limits developers from offering a meaningful warranty. In fact, most off-the-shelf software packages offer no warranty at all except disclaimers. However, Cho (1987) has been an advocate of solving the many problems of the software industry in order to make software warranties an attainable dream to software users and developers. His methodology stresses the use of statistical quality control throughout every stage of the software life cycle.

In Cho's work, the software product population defective rate, obtained by statistical sampling, measures the goodness (or defectiveness) of the software. It can also provide a vehicle for which the software warranties can be delivered. According to Cho (1987), a software warranty can be written in terms of the following requirements:

Software Input Domain. Conventionally, software input domain
 comprises four components: types of input,
 characteristics of each type of input,
 rules for using the input, and constraints
 on using the input.

Software Product Unit The product unit definition identifies
Definition. the desired output of the software.

Software Product Unit Based on the product unit definition, a
Defectiveness Definition. product unit will be considered defective
 if the desired output is not usable.

Sampling Plans. The user should specify the most
 appropriate statistical methods
 consistent with the product unit
 definitions developed as part of the
 modeling activity.

Sampling. The most appropriate statistical sampling
 methods consistent with the module product
 unit definitions during the modeling
 activity should be identified.

Acceptance Sampling. A statistical sampling plan that most
 economically meets the specified
 producer's risk and user's risk levels can
 be selected.

The Defective Rate Less The user should establish the product unit
than [alpha] population defective rate for each module.

RELIABILITY IMPROVEMENT WARRANTY

It is clear that one of the problems with the quality of software development is that there are no incentives for software developers to develop trustworthy software. The Department of Defense had the same problem of reliability with various subsystems in airplanes and weapons. They have been able to offset this problem with the application of the RIW concept, which was first developed in 1966 between Lear Seigler, Inc. and the Navy.

It is important to note initially that a RIW is not a warranty in the classic sense with respect to materials and workmanship. A typical item placed under RIW would be an airplane engine. It calls for the contractor to replace or repair, at his option, any warranted item within a specified time unit (in operating hours, calendar time or both), except on cases of obvious misuse. The contract establishes a fixed price for a given level of performance. This price is based upon the anticipated number of failures and the cost of each repair action. The anticipated number of failures over the warranty period is determined by assuming that the reliability of the warranted item will improve from the initial level to some specified level as a result of the contractor's planned reliability improvement program. The incentive comes in the form of an increased fee paid to the contractor if it can be demonstrated that the reliability of the item has been increased (Blischke and Murthy, 1996). In addition, a study by the Logistic Management Institute demonstrated investment in an RIW contract produced a net savings in total cost to the Defense Department, which helped to offset the effects of inflation and dwindling defense budgets (Logistic Management Institute, 1974).

The contractor, who is responsible for all failures, initiates an on-going reliability improvement program to improve the warranted item's initial reliability, [[theta].sub.i], to a specified reliability, [theta]*. The contract requires the manufacturer to replace or repair, at his option, any warranted item over the life of the contract.

Reliability information generated and recorded over time can be used to observe trends in the reliability of the product. The term "growth" can be used because it is assumed that the reliability of the product will increase over time as design changes and repairs are implemented. In other words, reliability growth is a projection of the reliability of a system, component, or product at some future development time. This projection is based upon information currently available from predictions or prior experience on identical or similar systems. Monitoring reliability, the MTBFs, and the failure rate of a system, equipment, or product, can establish an increasing trend in reliability, the MTBFs, or a decrease in the failure rate.

Such reliability growth occurs from corrective and/or preventive actions based on experience gained from early failures and corrective actions to the system, design, production, and operation processes. These actions represent an obvious reason for improved reliability. The philosophy behind RIW is that once the fixed price warranty contract is established, the profit realized by the contractor is dependent upon the equipment's reliability. Thus, contractors are motivated to focus their attention on the reliability of the items under contract through the use of "no cost" (to the buyer) engineering change proposals (USAF, 1974).

The advantages of the RIW for the buyer are reduced maintenance cost and increased reliability. The seller has the incentive to develop an on-going reliability (quality control) program because of the profit potential.

COST OF THE RIW CONTRACT

The contractor faces risk and uncertainty when he engages into a RIW contract. The risk and uncertainty increases as the length of the warranty increases. One risk function utilized by contractors to compensate for this is

R([T.sub.w]) = (1 + r) [T.sup.w/12] (1)

where r is the annual rate of risk (Balaban and Retterer, 1973).

In equation 2, the total cost of employing a RIW contract for a given number of items for a warranty period is estimated by

[C.sub.RWI] = ([C.sub.MOD] + [C.sub.DMC])* R([T.sub.w])(1 + X/100) (2)

where

[C.sub.RWI] = total cost of the RIW contract,

[C.sub.MOD] = total expected modification cost of the contractor,

[C.sub.DMC] = direct maintenance support cost to the contractor,

[T.sub.w] = the length of the warranty period chosen for the RIW contract,

R([T.sub.w]) = risk factor, and

X = percent profit of the contractor.

Next, methods of determining the expected modification cost of the contractor, [C.sub.MOD], and direct maintenance support cost to the contractor, [C.sub.DMC], are explored. Before these costs can be defined, models for distributing modification factors for determining the expected times of the modifications during the warranty period, and for computing the expected MTBF over the warranty period must be developed.

THE EFFECT OF SYSTEM MODIFICATIONS ON THE MTBF

Modifications of a software system under a RIW are made through the use of software engineering change proposals. Each change proposal is designed to improve the current MTBF by a factor of M. Namely, if [[theta].sub.new] is the current MTBF, then the new MTBF [[theta].sub.new] = M* [[theta].sub.old]. Within the lifetime of a RIW contract, a finite number of modifications may be implemented to cause the initial MTBF, [[theta].sub.i], to approach a specified MTBF, [theta]*, which was agreed upon during contract negotiations. Thus, a technique must be found to determine the improvement factor, M, defined by each modification.

The value of each M is bound such that M [greater than or equal to] but M < M', where M' is an upper bound for all such modifications pertaining to the particular item. Furthermore, the value of each M is dependent upon [theta]*, the specified MTBF the contractor must try to reach with each modification, and upon [theta], the current MTBF of the population of items procured.

One such function that is consistent with these assumptions is

M([theta]) = M' + (1 - M') [(M' 0 M*/M' - 1)[theta]*/[theta]] (3)

where

M* = the improvement factor expected if [theta] = [theta]*.

Equation 3 is a form of the Pearl-Reed curve often used in economic growth models. Balaban and Retterer (1973) used a similar form of this equation in their study.

A numerical example is now considered to demonstrate the nature of the function M. Let the initial MTBF be 50 hours, the specified MTBF be [theta]* = 111 hours, M' = 4 and M* = 1.3. For these values the modification function is

M([theta]) = 4 + (1 - 4)[[(4 - 1.3)/(4 - 1)].sup.(111/ [theta])] = 4 - 3[(0.90).sup.(111/2)].

Table 1 that follows indicates the values of M and the new MTBF, [[theta].sub.new], after each modification is employed. Two modifications are required for [[theta].sub.new] to become greater than or equal to the specified MTBF of 111 hours.

TIME OF A MODIFICATION

The time at which a modification can be introduced is a random variable [T.sub.m]. It is reasonable to assume that modifications will not occur before some minimum time [T.sub.[alpha]] has occurred after procurement of the items or after a previous modification has been employed. One distribution that can be used to define [T.sub.m] is the negative exponential distribution defined in equation 4,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (4)

for [T.sub.m] [T.sub.[alpha]] and where the constant, d, represents the rate of modification. The cumulative > distribution function F is defined in equation 5,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (5)

for [T.sub.[alpha]<] [T.sub.m] [less than or equal to] T. Further, assume that a certain procurement results in k-modifications with values [M.sub.1], [M.sub.2], [M.sub.3], ..., [M.sub.k], and each modification occurs at some time [T.sub.m]. The time between modifications is restricted so that each modification occurs only after some time [T.sub.[alpha]] has occurred. Thus if k-modifications are involved, then the times of each are ordered as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Given that a modification occurs on some time interval ([T.sub.[alpha]] , T$), then the expected time of each such modification can be obtained by employing basic principles of probability theory. It can be shown that the expected time of each modification can be found by employing equation 6,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (6)

AVERAGE MTBF OVER [T.sub.W]

The average MTBF of the software system over the warranty period represents another important measure. If we assume that k-modifications occur respectively, at times [T.sub.m1], [T.sub.m2], [T.sub.m3], ... [T.sub.mn], then the MTBF varies from [[theta].sub.0] over [0, [T.sub.m1]], to [[theta].sub.1] over [[T.sub.m1], [T.sub.m2]], to [[theta].sub.2] over [[T.sub.m2], [T.sub.m3]], ..., to 2k over [[T.sub.mk], [T.sub.w]]. The values of the modification times [T.sub.m1], [T.sub.m2], [T.sub.m3], ... [T.sub.mk] can be estimated by employing equation 6 which defines the expected modification time.

Thus an estimate for the average MTBF is defined in equation 7,

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

COST OF MODIFICATION

The cost of modification is a difficult cost to predict. Studies have shown that in general the greater the reliability improvement, the higher the cost. Certainly there are instances where high reliability improvement has resulted from low cost modification while low reliability improvement has resulted from high cost modification. In general the cost of modification is an increasing function of M and this assumption will be used.

A study by Mercurio and Skaggs (1973) utilized multiple regression analysis to obtain the cost of reliability improvement in terms of the resultant MTBF and the quantity of parts in the item modified. Balaban and Retterer (1973) in their study utilized a cost function in terms of the item modified. The cost function adopted in this study is essentially the same as Balaban and Retterer's except the cost is defined in terms of the amount of modification to increase the MTBF of the item. The function is denoted and defined in equation 8,

C(M) = 1.06([e.sup.[(M-1)/10M]] -1)P (8)

where P = the purchase price of the item and 1 [less than or equal to] M < M'.

Presented below in Table 2 are the costs associated with the modifications defined previously in Table 1, assuming P = $10,000. The general shape of function C is illustrated in Figure 2.

[FIGURE 2 OMITTED]

If it is found that k-modifications are profitable, then the total cost of these modifications can be found using equation 9,

[C.sub.MOD] = U [k.summation over (i=1)] C([M.sub.i]). (9)

Additionally, the contractor will typically incur modification costs at time [T.sub.mi]. If we assume the estimated modification costs are included by the contractor in the price of the RIW contract and the contractor pays the contract price at time 0, then these costs are discounted. Equation 10 that follows determines the total cost of all the modifications by discounting and amortizing each C(Mi) with I = yearly interest rate. The modification times, [T.sub.mi], are again estimated by equation 6.

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] (10)

DIRECT MAINTENANCE SUPPORT COST CDMC

The contractor incurs a total direct maintenance support cost, [C.sub.DMC], because the terms of the RIW contract make him responsible for failures. The value of the total direct maintenance support cost is determined by multiplying the expected number of failures by the cost per failure. This cost is determined by applying equation 11,

[C.sub.DMC] = ([U.sub.o][H.sub.o][T.sub.w][C.sub.F])/[??]. (11)

where

[U.sub.o] = the number of operational units,

[H.sub.o] = the average number of hours a unit operates in a month,

[T.sub.W] = the length of the warranty in months,

[C.sub.F] = the contractor cost per failed unit, and

[??] = the average MTBF over the contract defined by equation 7.

EXAMPLE--HYPOTHETICAL PURCHASE

Table 3 below defines the parameters to be used for a hypothetical software purchase of an enterprise's information system.

The value of the rate of modification was derived assuming the probability that a software contractor would inaugurate a software modification over a 3 to 24 month period was 0.90. By employing equation 5 and the given information, a value for the rate of modification, d, can be found.

The key values M* and M' which are required in the modification improvement function are defined as M' = 4 and M* = 1 + 0.20([theta]* - [[theta].sub.i])/ [[theta].sub.i] where [theta]* is the specified MTBF and [[theta].sub.i] is the initial MTBF.

Table 4 lists the data elements for the hypothetical procurement. The results of applying the model for the cost of a RIW contract are summarized in Table 5. Notice that the unit cost of the RIW contract is $1,870.70 and the MTBF increases 61.7% from 312.4 hours to 505.0 hours over the warranty period for the given example.

CONCLUSION

The concepts of using a RIW contract to increase the quality of software along with a model to determine the cost of a RIW contract have been presented. In our current software industry, the need for reliability improvement can be easily justified. With the pressure on software corporations to hurriedly put software on the market, it is no wonder that the vast majority of consumers fall victim to some type of error. It is vital to the leaders in the software industry to acquire and maintain trust and loyalty among customers. The main focus of this research lies in the ability to assign a tangible cost to a RIW for software. An integral part of this research is the use of the Pearl-Reed economic growth function, which determines the effects of modifications on the reliability of warranted items.

In this study, it has been shown that a plausible interpretation can be obtained by employing the methodology presented. Thus, a basis for an extension has been formed. Future studies can explore the use of other economic growth models relative to software modifications involved in the use of the concept of the RIW contract in the area of software warranties.

REFERENCES

Balaban, H.S. & B.L. Retterer. (1973, June). The use of warranties for defense avionics procurement. Annapolis, Maryland: The ARINC Research Corporation. ARINC Research Corporation Report 0637-02-1-243.

Blischke, W.R. & D.N.P. Murthy. (1996). Product warranty handbook. New York: Marcel Dekker, Inc.

Brennan, J.R. (1994). Warranties: Planning, analysis and implementation. New York: McGraw-Hill, Inc.

Cho, C.K. (1987). Statistical methods applied to software quality control. Published in G.G. Schulmeyer & J.I. McManus, (Eds.), Handbook of software quality assurance, . New York: Van Nostrand Reinhold Company.

Cho, C.K. (1980). An introduction to software quality control. New York: John Wiley & Sons, Inc.

Hayes, M. (2002, May 20). Software quality counts. Information Week Magazine, 38-54.

Hulme, G.V. (2002, January 21). Software's challenge: It's time for developers to think and act differently. Information Week Magazine, 20-23.

Logistic Management Institute (1974, March). Criteria for evaluating weapon system reliability, availability and costs. Washington, D.C.: Logistics Management Institute. LMI Task No. 73-111.

Mamer, J.W. (1987, July). Discounted and per unit costs of product warranty. Management Science, 33(7), 916-930.

Mercurio, S.P. & C.W. Skaggs (1973). Reliability acquisition cost study. Griffin Air Force Base, New York: Rome Air Development Center. General Electric Company Report RADC-TR-73-334.

Murthy, D.N.P. & W.R. Blischke (1992). Product warranty management--III: A review of mathematical models. European Journal of Operational Research, 63(1), 1-34.

Schulmeyer, G.G. (1990). Zero defect software. New York: McGraw-Hill, Inc.

Sommerville, I. (1995). Software engineering. Reading, MA: Addison-Wesley Company.

U.S. Department of the Air Force [USAF] (1974). Interim guidelines: Reliability improvement warranty. Washington, D.C.: U.S. Department of the Air Force.

Raymond O. Folse, Nicholls State University

Rachelle F. Cope, Southeastern Louisiana University

Robert F. Cope III, Southeastern Louisiana University

Table 1: Modification Functional Values

 [[theta].sub.new] =
Modification [theta] M([theta]) [M.sup.*] [theta]

1 50.0000 1.6257 81.2839
2 81.2839 1.4020 113.9818

Table 2: Cost of Modification

Modification M C(M)

1 1.6257 $415.93
2 1.4020 $208.34

Table 3: Parameters for the Example

Parameter Symbol Value

Expected Lifetime of Software [T.sub.L] 120 months
Minimum period before Modification [T.sub.win] 3 months
Length of RIW Contract [T.sub.W] 75 months
Discount Interest Rate I 10%
Risk Factor R 4%
Contractor Profit Factor X 10%
Rate of modification d 0.1096

Table 4: Data for the Example

Variable Symbol Value

Unit Price P $15,461.80
Operating Hours per Month [H.sub.o] 52
Number of Operational Units [U.sub.o] 27
Initial MTBF Specified [[theta].sub.i] 312.4
MTBF [[theta].sup.*] 472.3
Contractor Cost per Failed Unit [C.sub.F] $500.00

Table 5: Results for the Example

Variable Symbol Value

Average MTBF over Contract [??] 416.5
Final MTBF FMTBF 505.0
Number of Modifications k 5
Contractor Direct Maintenance [C.sub.DMC] $36,904.21
 Support Cost
Contractor Modification Cost [C.sub.MOD] $9,678.18
Cost of the RIW Contract [C.sub.RIW] $65,474.58
Unit Cost of RIW Contract $1,870.70