Tools, teamwork defuse politics of performance - goals of software performance engineering - Data Center and Systems Support: Software Performance Management - buyers guide

TOOLS, TEAMWORK DEFUSE POLITICS OF PERFORMANCE

With the considerable size of today's computing workloads, budgetary constraints and hardware limitations, software performance engineering (SPE) has become increasingly more important over the last several years.

Much of this visibility comes from the popularity of performance management and capacity planning, where the careful management of large system complexes can save a great deal of money. SPE ensures that application development not only satisfies functional requirements, but also performance requirements.

There is a problem that hinders the use of SPE for many shops, however. It is a political barrier between the application development group and other groups that have a vested interest in performance. This wall keeps internal department from communicating information that can effectively increase the performance of software systems, and therefore decrease overall MIS operating cost.

Lack of communication and cooperation is the greatest danger. This allows issues to slip away without being resolved. MIS and the corporation can pay dearly for system inefficiencies, and sometimes do not even know it.

A commitment from management to improve communications is important. Establishing a common goal of software development--the success of the corporation--is also critical to achieving staff support. Finally, the use of performance analysis tools can identify system problems while eliminating finger pointing.

"Using a report that the computer produced takes the personality out of the analysis; then it's not like someone is pointing a finger or criticizing a developer," said John Boettcher, performance analyst for the Aid Association for Lutherans (AAL), Appleton, Wis.

Performance problems that result from lack of attention may be tolerable in many cases. What a corporation must safeguard against, however, is outright failures that result in project cancellations or customer discontent. Organizations that have experienced large failures are reluctant to speak out about such incidents. The following profiles of such situations may sound familiar, however:

Patient: Manufacturer

Location: United States

Doctor: Corporate Board of Directors

Symptoms: After a $25 million investment and three years of design work, a new inventory control system was ready for coding. Because of the large size of the project, and its strategic importance, a third party was brought in to estimate the system's performance characteristics. A review of the physical design indicated response times far greater than service-level requirements, and a substantial cost to rework the complex design.

Diagnosis: The cooperation between participating departments seemed to be present, but the many suggestions made were ignored due to project milestone deadlines.

Treatment and Prognosis: Because of the cost to rework the project and a growing application backlog, the existing work was sold to another company. The strategic value of the system was lost.

Patient: Insurance carrier

Location: Canada

Doctor: Corporate management

Symptoms: A new policy inquiry and update system would service over 3,000 users. The system was designed for a new database system that was to be installed. After nearly five years in development, the early test phase revealed that performance was so bad that a $25 million investment in new hardware would be necessary to implement the system.

Diagnosis: Many years of political strife had resulted in almost nonexistent communication between departments. Performance was a tertiary consideration over functionality and deadlines during development, since neither department would cooperate.

Treatment and Prognosis: The cost of reworking the project was excessively high, so it was scraped. The insurance company replaced management that oversaw the project, and let go the extra employees who were no longer needed.

Patient: Bank

Location: United Kingdom

Doctor: Senior IS management

Symptoms: A wide variety of software projects had been implemented with less than acceptable performance. Tuning existing applications became difficult because there was a lack of supporting evidence reflecting the performance characteristics of suspect transactions. A number of internal conflicts were responsible.

Diagnosis: Technical support claimed that the application development group was unreasonable and demanding. Application development claimed that technical support was not serving their needs. It was not clear which group, if either, was ultimately responsible for the problem.

Treatment and Prognosis: The bank brought in consultants to discover and resolve the problems between the two groups. The consultants facilitated and mediated meetings. Eventually, the IS group members stopped coming to the meetings. For lack of internal cooperation, the bank cancelled the project. Communication problems remain the status quo.

These situations are not uncommon. Experts in the software performance arena can give numerous examples of consulting engagements that were contracted due to the inability of internal IS to address performance.

The benefits of high-performance software are clear: use satisfaction and productivity, controlled IS operating costs, lower software maintenance costs, upper management satisfaction and harmony among IS and users. The pitfalls of poorly performing software, of course, are the opposite of these benefits.

There are other consequences as well. Dr. Connie U. Smith, in her book, Performance Engineering for Software Systems [Addison Wesley, 1990], said, "The most serious consequence of performance failures is the possibility of a business failure. The inability to operate on a peak business day, to respond to customer inquiries, to generate bills or process payments can have serious financial consequence. Similarly, excessive operational costs lower profit margins. These performance failures not only cost money to fix; they detract from development resources that could be used on other beneficial projects."

Beyond the financial costs of new systems, user satisfaction is another critical issue. Said Smith, "Performance refers to the response time or throughput as seen by users. Many users subconsciously base their perception of [computer] service more on system responsiveness than on functionality. Negative perceptions, based on poor responsiveness of new systems, seldom change after correction of performance problems."

WHY POLITICS ARISE

Political problems are not unsolvable, but an understanding of politics is essential. Ken Kapps, an instructor with the IBM Management Institute, Chicago, teaches IBM's Effectiveness Skills course. He maintains, "To accomplish what you need to do, you must be able to work in the formal organization 'informally.' There are people whose titles don't suggest they have a lot of power, when in reality they do. You must understand formal power versus informal power."

Communication is key. There is a tendency to use technology to communicate with each other. "This is like giving each member of your family a pad of Post-it notes to communicate with each other," said Kapps.

He cited an example of a West Coast company working on a software project. Working management was on the West Coast, and senior management was on the East Coast. Both groups communicated about the project's progress solely by electronic mail (E-mail). One afternoon, the project team sent an E-mail to senior management that was rather upsetting. Senior management's immediate reaction was to send an E-mail back. However, they had recently completed training on communication and politics. They did decide to send an E-mail, but the message stated that senior management would fly out to the West Coast the next morning. Kapps said the trip resolved the problem and proved invaluable to the overall relationship.

For Ted Rock, systems programming manager of Pansophic Systems, Inc., Lisle, Ill., delicate communication was key when their application development group hinted that response time was not all it could be. After investigating, Rock found that the developers themselves had created a PROC (a set of procedures) that performed a number of allocations for each developer at logon. As the development staff grew and the PROC proliferated, the response-time problem got worse. Rather than throwing the problem back at the developers, Rock's team did some testing and presented a solution to development in the form of a suggestion on how to tune the PROC. A simple solution, and no ruffled feathers.

Although personalities are a driving force in political situations, it is a lack of common goals that fuels the problem for the long term. The goal of application developers is to deliver a software system that meets the functional requirements of the users, and to deliver the system on time. Response time, throughput and availability are the goals of counterpart groups such as database administration, technical support and performance management.

Where these goals are so important to a group, and the suggestion of change does not reveal an obvious benefit, there is no motivation for a commonality of goals. There is also no motivation for internal policing of issues such as performance. This is not surprising to Greg Boone, a consultant to Ernst & Young's Information Technology and Strategy Center in Boston. He said, "Developers are not rewarded on the efficiency of a system after it is implemented; they are rewarded on promises of delivery."

Beyond the lack of common goals, there is no intersection between groups that most often experience political barriers. In many cases, these groups report to different managers.

It is important to note that high performance should not necessarily be the number one issue. High performance characteristics are beneficial, but "fast, bad customer satisfaction" is not the answer either, according to James Newkirk, former staff vice president of TWA's Information Resource Management group, and now group vice president and technology group partner for the MTW Group, Independence, Mo. "The goal of IS is to provide the best information system to serve the corporation," said Newkirk. "The user needs to be able to concentrate on the customer."

He suggested there should be a balance between high performance and rich functionality. If richer functionality will benefit the user and the corporation, and if enough computing resources are available, then the balance should lean toward functionality for the user. "The axiom that hardware is cheaper than people is absolutely correct," added Newkirk.

At the same time, when the balance is entirely weighted toward functionality, the potential exists for performance problems. When performance problems strike, finger pointing begins. Criticism usually elicits defense, and the circle of politics is complete.

Many firms use consultants to assist with the performance issues of both new and existing software. The use of a consultant can alleviate the need to train staff. Consultants can also bring new ideas and insights to a project.

Developers who are going to have their work critiqued are not always excited, however. Bill Inmon, a senior principal with American Management Systems, Denver, and noted author and lecturer on software performance, said people will call him in advance to ask that certain issues not be brought up during a consulting engagement. From the consultant's standpoint, Inmon said, "The more sensitive the issues are, the more germane it is to bring it out into the open. It's much better for the long-term health of the organization."

Inmon also finds that many consulting engagements come from shops where politics are a hindrance to the performance issue. "If the groups involved know at the outset of a project that performance is important, and that performance experts are going to be brought in, there is a greater likelihood that no one will be upset."

Most agree that development is not against performance by any means. "The bottom line is that [development] doesn't want bad performance either," said Rich Fiori, senior analyst in the Data and Development Support group at Arizona Public Service (APS), Phoenix. At APS, Fiori attacks potential issues head on by sending memos, but stressed that, "Things have to be communicated in positive terms with positive language, and you've got to support the people you're making suggestions to by having evidence, and offering help."

Whereas some shops are sensitive to communications that might disrupt the peace, APS management is sensitive to performance and generally allows free communication about potential problem areas. Fiori also asserted that management backing is essential." have to have that or you won't go anywhere," he said.

Supporting the corporation is probably the most convincing and understandable motivation for addressing software performance. For example, Allied Signal's Garrett Engine Division, Phoenix, has adopted a company-wide continuous improvement and total quality program. Arnold Acedo, computer services manager, said the program has had a positive impact on how different groups work together. Individuals now have a common goal.

Acedo uses a team approach to address many IS problems that arise. Depending on the type of problem, individuals from different IS groups are included who can lend expertise to the situation. One such team was formed recently to deal with a tightly squeezed batch window that threatened online production hours.

Rather than identifying the largest problem and pointing a finger at who was responsible, the team was formed under the assumption that improvements could likely be made in many areas. Among the team of roughly 18 people were several from the development staff. "We are both after customer satisfaction," said Acedo.

As an additional benefit, team members from different groups get the opportunity to work more closely together and establish a better working relationship. The combined expertise offers additional diversification of views on how to solve problems.

Bob Lind, a programmer analyst for Garrett Engine Division, is on the batch window team. He said, "I felt it was an honor to be included." Lind said the experience has also given him a "greater appreciation for the overall organization."

From Lind's perspective, going into the project open-minded and staying away from an accusatory attitude are the qualities team members must have. The continuous improvement culture fosters this approach and eliminates the finger pointing.

What has made the team successful over time is the synergism created by its success. "The team found some simple solutions to things in the first couple meetings and immediate results were there. This helped team members know that they really can make a difference. It helped solidify the group and didn't seem like it was a waste of time," said Lind.

Bob Gray, senior manager of Performance Management for MCI Telecommunications, Arlington, Va., also feels it is the success of the corporation that will establish the common goals between performance and development groups. "We constantly express our recommendations in business terms, and the ultimate profitability of the corporation," said Gray. Selling software performance in this manner reveals a mutual self-interest, and lets developers see a personal benefit from working with the performance group.

Although "selling" software performance may be key, Gray cautioned, "Be careful not to sell more than you can deliver. If you do, you run the risk of compromising your credibility, and whoever it was you didn't come through for won't be interested in working with you again. You have to sell just the right amount."

CONQUERING TECHNIQUES

Many techniques are being used to quell political hindrances to the development process: communication, teamwork approaches, JAD (joint application design), modifying methodologies, automation, culture changes, performance analysis consultants and others.

During a major effort to change the development process, many of these techniques were used while Newkirk was staff VP at TWA. To affect this type of change, Newkirk said, "The absolute key ingredient is that you have to establish a direction, establish goals and objectives, and management has to be committed." Management promoted the notion that IS was building systems for the corporation. "We had some strong political issues and barriers," said Newkirk. But he believes 75% of those problems have been overcome by the effort.

Newkirk's group made some basic changes: They initiated weekly staff meetings; placed a greater emphasis on project management, including performance; promoted teamwork more heavily; brought individuals into projects when it was their time; and made refinements to existing guidelines.

Speaking of the political communication barriers, Newkirk said, "JAD is how we really started breaking it down. JAD got all the groups together to better understand the issues." Newkirk prefers using "joint application development" versus design, because he said users should be involved throughout the process.

TWA also implemented a strategic information plan, information engineering methodology and the Information Engineering Facility (IEF) integrated Case tool from Texas Instruments, Houston. "This was a statement from management that we were willing to spend some money to support this."

Most shops would make this type of radical change one step at a time. In Newkirk's case, "We decided to take the pill. It was a tremendous culture shock, but we got over it." This all-at-once technique can help ensure the attempt to change does not get dropped after only a minor effort, and assures it does not drag on for an unreasonable amount of time. For those employees who cannot accept the change, and cannot understand the need to support the corporation, it is perhaps time for them to move on.

Communication is the basic starting point to address political impediments. Communication, however, is not just a matter of being friends, it is a matter of understanding. Leilani Allen, VP of Information Resource Management at Aon Corporation, Chicago, said, "If the development group understands up front who is going to be involved, and why, the performance issues have a much greater likelihood of being successfully resolved. Also, if those responsible for logical design (developers and data administrators) and those who create physical designs (database administrators) work together in the context of a life-cycle methodology, that is also a big help to communications."

At Aon, transactions that require high performance are identified in the design phase as part of a process that gathers "volumetrics" (data such as number of transactions and number of users). This allows management to allocate a resource with performance expertise to work with the project team.

Allen also suggested that if a development methodology is being used, as in their case, particular performance or service-level requirements should be documented by the user up front. "A rigorous methodology clarifies the performance issues, the approach to be taken, and provides for accountability," she said.

COMMUNICATION AUTOMATION

Although communication between people should not be stifled or stopped, there are software tools that help remove the political overtones from critiquing production applications or application designs. The output of a software analysis tool is now a communication tool. Even where problems are detected by individuals, supporting evidence is always a helpful ally.

Software tools can also provide insight to the system's ability to handle the application. Performance problems do not always come from poorly designed applications. On the systems side, there are hundreds of configuration and tuning possibilities. Glynn Giacone, Crystal product manager for BGS Systems Inc., Waltham, Mass., said, "One of the reasons [developers] are reluctant to take on the responsibility of performance is that many of the factors are out of their hands."

Convincing developers to address performance is a matter of "getting development to buy into the fact that they share in the responsibility," he said.

Giacone described the optimal scenario as one where there is a basic agreement between development and the performance advocates to address the performance situation. Performance experts should perform the analysis with input from development, and the performance of the system in question should be quantified.

"This scenario points out whether it is an application-oriented problem, or a configuration problem," Giacone said. "The approach provides protection for everyone, and development can see that it is not just a means of critiquing their work."

Having software tools is one thing, getting people to use them is another. AAL's Boettcher has been helping developers with performance analysis on an invitational basis. "A few years ago, nobody would have asked. Now people ask us to monitor a CICS transaction, purchased software or an overnight job; or ask if I will model a new system design," he said. AAL uses BGS' Crystal to model systems and give feedback to developers on potential problem areas. During testing, AAL uses The Monitor for CICS, from Landmark Systems, Inc., Vienna, Va., and the Strobe application tuning product, from Programart, Cambridge, Mass., to further identify problem areas.

The tools provide an effective way of revealing and communicating performance inefficiencies for production applications, said Boettcher. The tools have also improved the understanding between departments, he said.

Fiori of APS said their use of a number of tools is based on each tool's strengths in different life-cycle areas. To work with and evaluate system designs, APS uses Bachman/Analyst, from Bachman Information Systems, Burlington, Mass.; Pose, from Computer Systems Advisers, Woodcliff Lake, N.J.; and DesignEnhancer-Design Review for DB2, from Applied Computer Research, Phoenix.

"They make an effective communication tool because it's basically an outside expert that's making the comments," said Fiori. As important as identifying possible problem areas, tools that evaluate designs can also supplement existing arguments for design changes.

Software analysis tools are also used to support MCI's SPE efforts. But MCI's Gray believes that data from tools is not what sells the notion of SPE, it is how the information is presented.

MCI uses Best/1 and Crystal, from BGS Systems Inc., and Paws and GPSM, from Scientific and Engineering Software, Inc., Austin, Texas, for modeling software systems. Harvard Graphics from Software Publishing Corp., Mountain View, Calif., is then used to represent the statistics for presentation purposes. "Your success is directly related to the quality of the presentation," said Gray. [Note: Paws has been superseded by SES/Workbench.]

Software performance engineering has much to offer. Although functionality is the primary goal, performance should be considered an integral part of functionality.

The issues in support of SPE are many. Customer demands continue to increase; with this comes a higher degree of difficulty to increase and sustain user satisfaction. Technology is becoming more complex; new software platforms under which systems are being built are not well understood and are subject to many performance variables. Management is more cautious about large hardware expenditures; this dilemma is aggravated even more as a result of the current economy.

The most complex issue of all is people. It is also people who are the most valuable asset to the IS organization. More often than not, technology gets most of the attention. The reality is, people who drive technology should get most of the attention. MTW's Newkirk summarized, "We have got to stop being technical bigots. We need to align ourselves with user departments and be business analysts."

COPYRIGHT 1991 Wiesner Publications, Inc.
COPYRIGHT 2004 Gale Group