Software quality assurance on the HP Printed Circuit Design System project - technical

Software Quality Assurance on the HP Printed Circuit Design System Project

EFFORTS TO ENSURE SOFTWARE QUALITY must be planned for and kept visible throughout the entire project life cycle. Such efforts were particularly important during the development of Hewlett-Packard's Printed Circuit Design System (HP PCDS). Focusing on quality alone has the inherent danger of never releasing the product because it is not perfect. Developing methods that result in a product that meets high quality standards without sacrificing aggressive goals for a product release date is a monumental challenge for HP as well as the rest of the industry. Two key ingredients in addressing this challenge are quality assurance and engineering productivity.

QA Plan

Often a quality assurance plan is viewed as a necessary nuisance at best and busy work at worst. For a QA Plan to contribute to bettering the quality of the software, engineers and management should view it as a contract among themselves detailing their quality efforts.

The QA plan includes the release criteria for the project, which are the basis for the rest of the document. Although release criteria are usually custom tailored for each project, there are some items that are fairly common, such as a clear downward trend of the defect rate, execution of all of the individual test plans, no known critical defects remaining, etc. The rest of the plan describes how it is anticipated that the release criteria will be met.

An optimal QA plan requires extensive effort from everyone. The management team provides the framework for specifying what areas need coverage and ensuring that the coverage is adequate. The details about how to perform the testing are left to the engineering team. It is vital that everyone knows the plan, believes it, and is committed to fulfilling it. To that end, the QA plan is not a static document, but a dynamic one. It is impossible to foresee all events that might cause certain sections of the document to become infeasible, inadequate, or even unnecessary. Thus the plan becomes much more valuable to a project team if it is kept current, reflecting the real intentions of the team. As the testing progresses, results are entered. Performance measurement results are especially important for comparing later revisions of the product. The results give the management team invaluable information for deciding when the formal QA phase should end. A side benefit of keeping the QA plan current is that it automatically becomes a final document for the product archives.

Defect Tracking

Software defects some in many forms. Many are very innocuous, such as the misspelling of a word or unclear error messages. Others are of such a critical nature that the application software aborts unexpectedly and data is lost.

Throughout the life cycle of any software project a large number of defects are encountered and fixed. Attempting to keep track of thousands of defects in a large software project using a manual method is untenable and especially abhorrent, given the fact that HP is in the computer business. Having an automated defect tracking system is viewed as a necessity for helping the management team balance resources and provide the engineers with data on locating and fixing defects. The two major defects tracking systems used in HP are STARS and DTS. STARS (Software Tracking And Reporting System) runs on HP 3000 Computers under the MPE operating system and is used by marketing and field sales organizations. DTS (Defect Tracking System) runs on HP 9000 Computers under the HP-UX operating system.

HP PCDS Development

Since HP's Printed Circuit Design System was developed in the HP-UX environment, it was felt that the engineers would be more productive using DTS. With other systems for tracking defects, it is not uncommon for the R&D lab to be unaware of defects because the procedure for reporting them is too cumbersome. However, with DTS the ease of entering defect reports facilitated the logging of the vast majority of defects encountered. Furthermore, because DTS was installed on every engineer's workstation, engineers found it very convenient to use DTS as a lab notebook. Notes about what caused a defect and ideas on how to fix it were stored directly with the defect data packet. Reminders to enter copyright notices, say, were entered as "defects." Ideas for new features were entered as enhancement requests.

Throughout the HP PCDS project, the management team was kept up to date with accurate statistics that were used to measure the quality of the software. For example, one measurement that was closely monitored was the number of defects reported versus time. A clear downward trend of this curve is evidence that QA efforts are improving the quality of the software.

Although there were many advantages for the lab in using DTS, the needs of the field, sales, and marketing organizations could not be ignored. Before manufacturing release, the outstanding defects from DTS were copied into the STARS data base. Defects from the field and sales organizations are entered in STARS and then copied over to DTS for the lab engineers. There is now software that automatically provides a link between DTS and STARS which keeps the two data bases synchronized. Although supporting two defect tracking systems simultaneously is certainly not easy, the advantage make the effort worthwhile.

Environmental Issues

It is unreasonable to expect anyone to produce quality results in an atmosphere that is not conducive to the task being performed. In the ideal case, the physical environment for a software project should be completely unobtrusive, allowing the engineers to focus their efforts on software development, rather than on manipulating that environment. With each engineer equipped with a workstation, the environment immediately becomes somewhat less than ideal because the numerous workstations create a need for some level of systems administration.

The use of a local area network (LAN) and the addition of a full-time systems administrator can turn this potential liability into an asset. For example, the systems administrator provides expertise in installing and updating systems, troubleshooting hardware and operating systems failures, and maintaining the LAN hardware and software. The LAN provides many services that improve the productivity of engineers. The use of electronic mail over the LAN promotes communication with little effort on the part of the engineers. The systems administrator facilitates this activity by establishing and maintaining electronic mailboxes for all the LAN users. Fig. 1 illustrates a method that minimizes the overhead generated by maintenance. Using this method, changes can be handled easily by the systems administrator, since only the files on the central system need to be modified. The addition and deletion of user mailboxes is very straightforward for the rest of the lab.

The LAN also plays an important part in the engineer's development environment. Typically, a team of engineers works on the same software module. This requires that each engineer have access to all of the module's files and that there be an easy way to prevent inadvertent simultaneous updates to the same file by more than one person. The use of SCCS (source code control system), a standard toolset provided by HP-UX, with the LAN provides a foundation on which to build a stable working environment. SCCS requires that files be checked out for the purpose of making modifications, and uses a semaphore to allow only one user to check out a file at a time. While a file is being edited by one user, other users can still get a ready-only version of the file. SCCS also keeps version information so that, if need be, an earlier revision of a file can be restored.

In a distributed environment, there may be a tendency to have each engineer responsible for a defined set of files, with the SCCS files resident on the engineer's local workstation. This approach unfortunately creates extra maintenance for everyone when files shift responsibility or if there is joint ownership of files. Keeping the SCCS files on one workstation designated as a team master system avoids this confusion, yet the LAN still allows the necessary flexibility. Although the SCCS files reside on a remote machine over the LAN, an engineer still has access to them by remote file access (RFA). Makefiles can be set up to perform all commands necessary to build the software module, referencing the location of the SCCS files relative to the local machine. By doing this, each engineer has a local copy of the module which is built from the latest versions of the SCCS files. While an engineer has a file checked out for editing, the other members of the team will have copies of the file before the checkout. Only after the engineer has made appropriate tests is the revised file checked back in and made available to all. In this way, the entire team has access to the latest software, but is protected from the experimentation of individuals.

Additional benefits are derived from the physical environment chosen. Since HP PCDS was developed on the product's target hardware, testing was more convenient for the engineers. The LAN was extensively used when defects were encountered. The engineer could view data files on a remote system or copy them to a local workstation using the LAN. Then debugging activities could take place at the engineer's desk. In other cases, a special version of the code that could be used with the debugger was copied to the test system and the debugging activities took place on the test system. The LAN reduced the overall need for physical media such as magnetic tape for the transferring of files.

System Integration

At some point, all modules must be integrated and the product as a whole must be built on a single workstation. The process must be highly reliable, because at the end of the project cycle, time cannot be wasted trying to figure out how to build the system, or chasing down the reason that some file won't compile. To have continuing confidence in the process, it was decided to rebuild the HP PCDS product every night. A workstation serving as a product master system was dedicated to this task of system integration. This system provided a central repository for files shared by the team masters. Every night each team would build files required by other teams. These would be copied to the master system before a prescribed time. Teams would then copy the files required from other teams from the master system and proceed to build their respective modules. This distributed method of building the product ensured that interteam dependencies were checked continually. This method was incremental in nature because only the files that had changed were recompiled.

On a periodic basis, it was necessary to verify that the whole product could be built entirely from scratch on a single workstation. This had the advantage of uncovering any implicit assumptions that might not be valid. This was done by copying all the SCCS files from the team masters to the master system. The product was then built, and an installation tape was produced for verification and testing. By establishing this build-and-system-integration process early in the HP PCDS project, valuable time was not consumed at the end of the project, the most critical phase, attempting to get the process under control.

Formal QA Activities

Before entering the QA phase, some QA activities have already been accomplished. Design and code reviews certainly contribute to the quality of the software, yet must be done during the design and implementation phases of the project. During the QA phase of the project, the QA plan becomes the guiding force. To complete the project and ensure that the product has high quality, the plan must be executed. The confidence in the product's quality is in direct proportion to the amount of testing that has been performed on the product. However, testing hours alone do not tell the whole story. There must be a variety of testing, with emphasis placed on testing the right things.

Engineers are very good at deriving "white box" tests. That is, since they have developed the software, they know how it functions, and are best able to develop tests that stress boundary conditions, limits, and other structural tests.

A group of engineers from a different section of the lab formed a small test team. This test team had the charter to test the HP PCDS product against its manuals. Nothing is more frustrating to a customer than having the behavior of the software be completely different from what is described in the documentation. The test team members were able to bring their engineering skills to bear on the problem without being biased by intimate knowledge of the internal workings of the software.

Another opportunity for testing came in the form of support engineer and field engineer training classes. Getting prerelease software stable enough to support training classes during the early phase of the project helped establish installation and system integration processes and illuminate deficiencies and weaknesses in the product. The training classroom had the serendipitious advantage of providing ideal resources for hardware configuration testing during idle times.

Another whole category of testing is more important to customers. This is fitness for use, that is, can the customer use the product in its intended application? Software engineers rarely have strong backgrounds in the use environments for the applications they are developing. The addition of an application resource (AR) team composed of typical end users fills this gap. In the case the HP PCDS, several experienced printed circuit board designers were assembled as the AR team. Their job was to use HP PCDS to design actual printed circuit boards. It was extremely advantageous that the AR team was in the same physical location as the software lab. During the process of designing a board, the AR team would give instant feedback to the lab engineers concerning problems, suggestions, etc.

The AR team was particularly qualified to evaluate the results produced by HP PCDS. With an experienced eye, they could quickly spot a problem that a software engineer would never notice. It is typical for prospective buyers of a printed circuit board design system to request that a printed circuit board board representative of their requirements be designed on the system before purchase. This benchmarking activity was very professionally accomplished by the AR team.

The AR team also coordinated and gave support to the alpha test sites within Hewlett-Packard. Alpha sites were chosen such that HP PCDS would have a wide exposure to different printed circuit board technologies. There was a strong partner relationship between the lab and the alpha sites. The alpha sites benefited from their participation by receiving the software much earlier than the rest of HP and by being able to provide input regarding desired product refinements. Naturally, the design team benefited from the extra end-user testing. The AR team and the alpha sites had the experience to evaluate the quality of HP PCDS properly from the customer's perspective. They were the "voice of reality."

Acknowledgments

I would like to express my sincere appreciation to the following people. Elaine Regelson provided me with guidance and enthusiastic support, which encouraged me to do a better job than I thought I was capable of. Rick Roeling joined the project at a time when I was desperate for some help. i am still amazed at how quickly he came up to speed and how he was able to take over the responsibility of administering DTS. Many of the benefits derived from using DTS were a direct result of Rick's efforts. Pati Porter served as our systems administrator and developed the methods for keeping the electronic mail system and the LAN running smoothly. Also, I am indebted to Jon Gustafson, Debbie Lienhart, Mark Mayotte, Elaine Regelson, and Denese Van Dyne for their efforts in helping me prepare this article.

COPYRIGHT 1988 Hewlett Packard Company
COPYRIGHT 2004 Gale Group