Logistics Test And Evaluation: An Overview - weapons system testing

Subtle flaws in design can have multimillion-dollar effects-and sometimes potentially catastrophic ones. A vigorous logistics test and evaluation process is essential to ferret out these potential problems, and make sure that when fielded a system is supportable, maintainable, safe, survivable, and transportable. An Office of the Secretary of Defense (OSD) study suggests that this is a job we need to do better; here is a guide on how to go about it, illustrated with specific cases that show its value.

Logistics is a primary element in a thorough test and evaluation of new or modified weapons systems. It is critical to the warfighter that we provide an early assessment of the overall support characteristics of the system we are evaluating.

In our evaluations of the system under test (S UT), we need to remain focused on the goal of providing a system that maximizes its operational availability ([A.sub.o]) within the life-cycle cost (LCC) of the program. The system must minimize the amount of time that maintainers expend getting it ready to perform its mission, recovering the system after the mission, and returning it to a mission-ready status again. That is, we must build systems that maximize their [A.sub.o]s or we build a ground target for the enemy.

A key principle of logistics test and evaluation (LT&E) is to ensure that the system under development is able to achieve the readiness objectives for both peacetime and wartime scenarios. This is accomplished through a unified and iterative approach to the management and technical activities that allow support considerations to influence system requirements and the design process. The support requirements and the design must be optimally related to each other in order to maximize the system [A.sub.o] throughout its life cycle. During the LT&E process, we systematically evaluate each of the logistic support elements and their interrelationship with other elements. Part of this process is to identify system deficiencies, provide enhancement options, and then evaluate corrections made to the system. This continuous process will help to ensure the system maximizes its [A.sub.o] when it is fielded.

So what drove us to reexamine the method we used to evaluate the SUT? One impetus was the OSD study that suggested the Services were not systematically evaluating the SUT as well as could be done. While we may have thought we were doing a good job of putting systems out in the field that were reliable and maintainable and able to achieve their [A.sub.o], the feedback from the field was that we could do much better.

Reliability and maintainability are what really determine [A.sub.o]. Recall the definition of [A.sub.o]:

[A.sub.o] = MTBM/MTBM + MDT

where MTBM equals the mean time between maintenance, and MDT equals the maintenance delay time. The formula is based on the assumption that the system is designed with reliability. Reliability is the probability that an item will perform a required function under specified conditions for a specified period of time, or at a given point in time. Maintainability is the probability that an item will be retained in, or restored to, a specified condition within a given period of time. Reliability and maintainability, along with performance, should act as equal partners in the requirements generation process.

LIFE-CYCLE COSTS

In a typical acquisition program, the system program office (SPO) has committed more than 80 percent of the LCC before the engineering and manufacturing development test program has begun. The program hasn't written the check, but they have made key decisions on the design of the system and the support characteristics that essentially commit those funds. Based on design and its reliability and maintainability predictions, the SPO will determine of the number of spares of each particular type that will be purchased, what support equipment will be used, and whether new equipment will be procured. These predictions are also used to determine the types of skills needed and the varying skill levels required, and other manpower considerations. During LT&E, we are chartered to validate the contractor's estimates and provide that data to the program office. Under- or over-estimating the reliability and maintainability of the component pieces of the system will cause already limited dollars to be allocated unwisely. Accomplishing LT&E early and continuously throughout the development, test, and evaluation (DT&E) program allows SPO personnel to determine what needs to be adjusted and by how much, to provide the required system support throughout the program's life cycle.

Most of the LCC of a system is spent during the operations and support (O&S) phase. Typically 10 percent of the program's LCC is spent during the research, development, test, and evaluation (RDT&E) phase of the program. While we may view production as the most costly portion of the program per unit of time, it really only amounts to some 30 percent of the LCC. Based on these figures, it becomes readily apparent that the largest cost driver in the life of a system is the O&S phase. With a system such as the B-52 bomber, which is doubling (or more) its life expectancy, it is easy to see that O&S costs will form 75-90 percent of its LCC. Essentially, it costs almost twice as much to maintain and support a system once it has been fielded as it does to acquire and produce that same system. For a major system such as the B-2, which costs $44 billion in RDT&E and production, the Air Force can expect to spend about $66 billion more in the O&S portion of that aircraft's life cycle.

COORDINATING THE LT&E PROCESS

The LT&E of today is a significantly different process. Previously we used maintainability demonstrations, support equipment compatibility demonstrations, and technical order verifications, but all were conducted independently. The current LT&E discipline makes a synergistic application of all the logistics support elements. The goal of LT&E is to test, evaluate, analyze the system, and then report those findings to the customer. Established guidance exists that addresses the need to accomplish LT&E. (In fact, when Air Force Instruction [AFI] 99-101 was being drafted, we were asked to develop the wording for LT&E.)

Now that we have guidance, what do we use for test and evaluation criteria? What will define satisfactory systems performance? The place to start is the operational requirements document (ORD). What are the requirements the user has given us for the system? How have those requirements been translated into the system specification? For example, what are the mean time between failure (MTBF) and mean time to repair (MTTR) requirements?

Human factor requirements come from both the ORD and the system specification. For example, will we be able to complete all maintenance tasks while in chem gear, or even cold weather gear? Is equipment being designed for ease of accessibility and maintenance? Are line-replaceable units (LRUs) being designed and installed in such a way that one person can accomplish the task if required?

Supportability analysis predictions are a critical part of how we evaluate the system. If the LRU has been designed to work for 350 hours before it fails, then that is the criteria used to evaluate its success or failure. If it fails continually at 100 hours, the customer needs to know, in order to reevaluate system design. If spare LRUs were purchased based on a 350-hour MTBF, there would now be an insufficient quantity of spares.

We need to ensure that, although we test each part of the system, we are able to evaluate the overall system supportability, maintainability, safety, survivability, and transportability. You can almost directly translate these requirements into the logistic support elements (LSEs).

When comparing the test data with the LSE predictions, be aware that a new design or new use of an LRU may suffer infant mortality or have initially low reliability. Test the system long enough to distinguish between development pains and fundamental problems (i.e., the unit not working as it should and the data telling you it's not getting any better). Report these problems to the customer and also provide them with recommended changes to improve the shop replaceable unit (SRU)/LRU, or system. Technical Order 00-35D-54 provides not only a means to report system, LRU, and SRU deficiencies, but a means to recommend enhancements to the system. This is a reason why we use people who have current operational experience to perform the LT&E. They know what is happening in the field and know how to make the system better.

FINDING PROBLEMS EARLY

The AFI99-101 stressed that the DT&E communities look at the maintainability and suitability (supportability) of the system early in the development program. Although at this point we may be assessing prototypes or preproduction equipment, we can still make an early determination of how the system will work when updated, modified, or changed, based on our DT&E findings. Consider the training that was provided to the maintenance team and make determinations by their ability to maintain the system: Was the training adequate? Are the technical orders they are working with adequate to accomplish the tasks? (They may not be validated and verified.) Is the support equipment they are using the equipment that will be fielded? If not, have they been able to evaluate the proposed support equipment designs? What is the number of personnel needed to support and test the system in the test environment; will this number be adequate for the system when it is fielded? Do we have the correct Air Force Speciality Codes (AFSCs ) identified with the correct skill levels? Finally, are the maintenance procedures correct? Are we asking an engine technician to maintain an instrument system, just because it is an engine instrument?

In a typical program it is many years from the development of the ORD to the point at which a system is available for testing in the engineering and manufacturing development phase. During that time, events can change the user requirements: The threat may change, or the program funding may be reduced or increased. The ORD format will identify all of the LSEs and will provide the author the opportunity to input requirements into the ORD for testing those elements to ensure that a supportable system is fielded. The insertion of human factors and LSA predictions is another way to evaluate the data that we gather on the system. These items, the 10 LSEs, the user's requirements, human factors, and the supportability analysis predictions, when tested and evaluated together, provide a comprehensive analysis of whether the system, when fielded, will be able to attain the required [A.sub.o].

Other documents provide additional information. The program office will develop the logistics support plan, which describes how the customer and the acquisition personnel envision life-cycle support for the system. LT&E is the primary method used to test the plan and determine if it is effective. The contractor will also have a plan describing how the system will be supported from production to disposal. This plan should be developed using a relational data base, which flags how changes in one element will affect others. (For example, if the numbers and skills of the personnel in the field change, the data base will identify other support elements that will be affected.)

The test and evaluation master plan (TEMP), written by the test planning working group, is one of the primary management documents for this phase of the program. The TEMP contains five sections:

* a system description;

* an integrated program schedule with a focus on the test schedule;

* DT&E objectives;

* the objectives of the operational testers; and

* the test program resource requirements.

Examples of program resources are: number of personnel required, location of testing, duration of testing, number of test articles, number of spare assets, funding by fiscal year, and training. Program resources are what we, the test community, need to complete the test and evaluation of the system.

LT&E SUCCESSES

Let's look for a moment at some of the results of accomplishing LT&E. During testing of a new F-15 crypto key loading device, the contractor had recommended a new extension cable. Maintenance personnel determined that it wasn't necessary to use the cable to load the device. This saved $58 per cable and about $1,044.00 per F-15 fighter wing per year. It doesn't seem like much, until you look at a small cost like this in the context of shrinking budgets. How many fighter wings of F-15s do we have, and how long will the F-15 remain in the inventory? Little costs mount up, and eventually impinge upon money that we we need to spend on other new equipment or upgrades.

The results of successful LT&E can be potentially much more dramatic: In the original global positioning satellite modifications on the F-15, the seal for the controlled reception pattern antenna had metal washers that were not captive in the seal and were prone to slip off the seal during installation and removal. As a result of input from the maintainers, the finished product eliminated the need for the seal and the loose hardware. Another modification of the global positioning satellite recommended by maintenance personnel was the relocation of the proposed antenna electronics unit, which chafed against the right rudder pedal cable. The possible consequences of this initial design flaw--loss of the aircraft and possibly of the crew--demonstrate the value of this kind of assessment.

An example of what can happen if the maintainers are not involved early in the DT&E process is demonstrated by the modified SUU-73/A pylon. When the pylon was installed as designed, certain engine access panels could not be opened. So every time access was required, the fuel tank pylon or the weapons pylon would have to be removed--an 8-manhour job. Needing excess time to repair aircraft is just the same as not having enough aircraft to accomplish the mission.

Here's a good example of how being involved early in the acquisition process has an impact on the design. An electro-environmental specialist looking at the mockup at the plant in Long Beach, CA, noticed that the C-17 cargo compartment liquid oxygen filler connection had protruding screws, which would prevent the filler cart hose from being attached properly. When brought to the attention of the manufacturer, the flaw was described as merely an error in the mockup. Through persistence, the specialist found and studied the design drawing, and determined that it was assembled correctly. The connector would not have been able to lock, and maintenance would have been unable to service the system. The drawing was changed and the assembly line was checked to ensure that the change was incorporated.

The following example shows how a small item can run into big money because reliability figures weren't accurately estimated. A wingtip lamp for the C-17, both front and rear positions, is the same lamp that is installed in the KC-10 (DC-10) to illuminate the wing's leading edge during flight. On the KC-1O it is mounted in the side of the fuselage skin in a very benign environment, and needs to be replaced every 676 hours. That same lamp on the wingtip of the C-17 is in a different operating environment (higher vibration and temperatures) and the lamp only lasts for 25 hours in the forward position and 50 hours in the aft position.

The lamp did not come close to its predicted value, and if insufficient spares of this mission-critical part were purchased, all of the C-17s could soon be grounded for want of a lamp. The impact of the lamp's invalid reliability figure of 676 hours MTBF on the life-cycle cost of the C-17 is an estimated additional $4.5 million. With the actual MTBF numbers in hand, the system program office can at least adjust the procurement requirements for spares.

Here is another example of the effectiveness of early LT&E in identifying design shortfalls. The original estimate for the C-17 water coalescer bag was that it would be changed approximately once a year (about every 1,000 to 1,200 flying hours), which was the normal change rate for the current fleet of airlifters (C-130, C-141, C-5). We determined during the test program that because of the design of the auxiliary power unit's intake and exhaust, and the proximity to the air conditioning system intake, the bags (sometimes called the water separator socks) were becoming clogged every 25 flying hours. It didn't take long for supply bags to run out. Through some redesign and changes in the maintenance procedures, the bags now last about 250 hours (still a dramatic LCC impact). If it doesn't get fixed, the cost will be an estimated $29 million.

Another instance of the value of early design review occurred during the B-2 program, when egress technicians at the manufacturing facility reviewing the mockup design noticed the upper escape hatch was secured with hi-torq fasteners.

The upper hatch has to be removed once a year for inspection of the ejection seat. The maintenance demonstrations showed that with the hi-torqs installed, it would take two people 214 manhours to remove the hatch. This amounted to a 5-day effort, working around the clock. The egress technicians wanted to change the hi-torqs for screws and nutplates, but the engineers were reluctant, asserting that the hatch was a stress panel. The egress technicians pointed out that the F-4 aircraft was basically a flying stress panel, and used screws and nutplates predominately throughout the aircraft to secure panels. It was agreed to evaluate the change to screws and nutplates. After a series of successful tests at the Holloman Air Force Base, NM, sled track, it was agreed to use screws and nutplates. The maintenance task time was reduced to 18 manhours (2 people)--in other words, one normal work shift. This seemingly simple change meant the aircraft increased its [A.sub.o] dramatically. What this means is the inspection c an be done overnight and the aircraft will be ready to fly and fight again the next day.

CONCLUSION

Logistics test and evaluation is a major portion of the DT&E process. The goal is to field systems that are supportable from the very beginning. That doesn't mean that we won't continue to modify weapons systems. It means that the modifications we install will be the result of new technology, not something that we will constantly have to modify just to meet the original reliability and maintainability goals. We will make sure the user gets what they ask for and that it meets the ORD requirements for A0. If it is supposed to have a 95 percent fully mission capable rate, is that what is occurring in the field? If it is not at 95 percent, why not? Does the radar work for 500 hours before it fails, as it was required to do? If not, why not? Is the user spending all of its O&S money on modifying the system to make it meet those original requirements, or are they spending it on the things necessary to maintain their proficiency to be prepared in times of crisis?

Ensuring that the warfighter receives maintainable and supportable systems is the goal of LT&E. While it will generally have a larger LCC impact if discovered during the DT&E process. LT&E should continue throughout the operational T&E phase and in fact should be a part of any modifications that will effect the supportability of the system.

Tony Parker has been associated with the Air Force in the aircraft maintenance and logistics career field since 1961. After retirement from active duty, he returned to government service as a logistics manager. He is currently employed at the Air Force Flight Test Center's 412th Logistics Group. He is a staff officer in the Logistics Test and Evaluation Office.

DISCLAIMER

The opinions, interpretations, conclusions, and recommendations herein are those of the author and are not necessarily endorsed by the U.S. Air Force.

COPYRIGHT 1999 Defense Acquisition University Press
COPYRIGHT 2004 Gale Group