Candid voices: sure it is effective, but is it suitable?

Introduction

Sure, your weapon or avionics system may be operationally effective, but is it operationally suitable? Is it reliable, maintainable, and available when a maintainer or operator needs it to be? These are questions the operational suitability analysts (OSA) of the 28th Test Squadron's newest division at Eglin AFB, Florida, ask members of test teams on a daily basis. The mission of the 28th is to evaluate the effectiveness and suitability of weapon and avionics systems that are being procured or improved to support current and future Air Force air combat missions.

In recent years, the amount of suitability analysis performed for a test had been based on the project manager's and team member's experience (or lack thereof) in suitability. Emphasis on system performance, costs, and schedules resulted in an unstructured approach to suitability analysis. This meant that effectiveness portions of an operational test and evaluation (OT&E) were done very well, while suitability was sometimes lacking. In an effort to establish a formal mechanism to ensure a standardized approach to suitability analysis, the 28th stood up the Operational Suitability Division in February 2004. This article describes just a few of the issues and concerns being addressed by logisticians in the Operational Suitability Division and how it supports the Air Force OT&E mission.

The mission of this new division is to ensure reliable, available, maintainable, and cost-effective systems are designed to meet the user's peacetime and wartime readiness requirements with the necessary support infrastructure. Operational suitability analysts make certain that suitability is included in the system performance specifications so that the system is designed to be supportable. They ensure all necessary support resources (technical data, spares, facilities, support equipment, training, manpower, and so forth) have been acquired, proven, and provided to the users.

Operational suitability analysts operate the same way as their effectiveness-driven operations analyst counterparts. As projects are approved and project managers request team members, operational suitability analysts are assigned to projects based on the career-field experience needed and test priority determined by Headquarters Air Combat Command (ACC). Once assigned, each operational suitability analyst works to achieve two immediate divisional goals. First, they ensure that suitability is addressed as early as possible in the life of the project by seeking opportunities to provide suitability inputs during the writing of the project or test plan. Second, they make sure that suitability analysis efforts produce the desired outcomes or products to support the warfighter. That is, they find the problems before the warfighter does.

Operational suitability analysts perform as maintenance and logistics subject-matter experts while evaluating suitability issues. They do this by developing and reviewing test plans and final reports; formulating specific suitability test objectives, methods of evaluation, and performance and evaluation criteria; retrieving and analyzing maintenance data; developing questionnaires for maintenance technicians; evaluating technical data, tools, and support equipment; validating equipment diagnostics; and assisting project managers in reviewing, submitting, and tracking deficiency reports.

Operational suitability analysts also serve as the project manager's maintenance liaison during test execution by ensuring operationally realistic scenarios are addressed and developed. They work hand in hand with maintenance evaluation teams, making sure all maintenance actions and findings are documented and reported. Additionally, operational suitability analysts identify direct and indirect maintenance and logistics impacts of the system under test by staying aware of changes in maintenance concepts, inspection requirement intervals, availability of spares, and changes in manning or training requirements. Their aim is to anticipate all impacts on the new system so the warfighter does not have to develop costly workarounds after it reaches the field.

What Is Suitability?

Operational suitability is the degree to which a system can be placed satisfactorily in field use with consideration given to availability, compatibility, transportability, interoperability, reliability, wartime usage rates, maintainability, safety, human factors, manpower supportability, logistics supportability, documentation, and training requirements. The most operationally effective system can be deemed ineffective if it cannot be supported and, therefore, is unsuitable for operational employment. Considerations to reliability, maintainability, and availability make up the bulk of suitability testing, which are intricately related and are discussed briefly next.

Reliability is the duration or probability of failure-free performance. The challenge in reliability testing is to reduce the amount of system maintenance and servicing downtime, thus increasing the availability of the system. A common term used to express reliability is mean time between failures (MTBF). MTBF is expressed as the total operating time (for example, flying time, driving time, or system-on time) divided by the total number of failures. The definition of what is considered a failure must be included in the test plan to ensure it includes all operational influences, not just system design problems. Usually, long test periods are needed to measure system reliability accurately. With time and money always a constraint, sometimes a larger number of items are tested for a short time instead of a few items for a long time; for example, testing six items for 200 hours each versus testing three items for 400 hours each. Although the test time for each scenario is the same (1,200 total hours), the suitability analysts need to determine if there are any decreased reliability issues or wear out failures between 200 and 400 operating hours. There are risks involved with this approach, but they can be minimized by using other test data to demonstrate the risk is acceptable and that significant wear-out failure modes have not occurred in longer duration testing and are unlikely to occur during operational use.

Maintainability is defined as the ability of an item to be retained in or restored to a specified condition when maintenance is performed by personnel having specified skill levels, using prescribed procedures and resources, at each prescribed level of maintenance and repair. There are three dimensions to determining system maintainability. First, there is the average corrective maintenance time required to restore the system to a mission-capable condition. This is how long a system will be under repair after mission critical failures. The second dimension encompasses the corrective maintenance time for any failure. Times to correct any maintenance actions may be longer or shorter than the time to correct mission critical failures. When determining the average corrective maintenance time, the total number of hours of active repair time divided by total number of incidents requiring corrective maintenance, it is important to define the meanings of corrective (unscheduled) and preventive (scheduled) maintenance and define start and stop times for each measure. Finally, the third dimension to consider is the manpower required to perform the required maintenance actions. If it takes 3 man-hours for an average repair, there is a considerable difference between one person's working 3 hours or three people, each working 1 hour. Improved fault isolation through more accurate built-in test capabilities and automatic test equipment also can increase maintainability. Improved maintainability can reduce the number of spares and maintenance actions while simultaneously reducing the need for specialized test equipment and personnel.

Availability addresses the degree to which an item is in an operable state at the start of a mission or when demanded at some undetermined time in the future. Operational availability is considered a function of reliability and maintainability. The type of system must be considered and can range from the entire aircraft to smaller individual systems that make up the entire weapon system. Some systems may be required to operate continuously 24 hours a day. Others spend time in a ready status and perform their mission at defined intervals. Operational availability is measured by dividing the total system uptime by the total uptime plus total downtime. In aircraft maintenance terms, this is the mission-capable rate. A few failures randomly distributed during short test periods may skew the calculated availability and misrepresent what actually may be observed in an operational environment. An immature system may experience numerous failures (infant mortality break rate) and may take longer to return to an operational status because a learning curve has not been established. In these cases, the limitations on the availability measure must be recognized. Other actions like administrative logistics delay times (ALDT) also must be addressed. These include time taken for maintenance and supply technicians to cross reference part and stock numbers, time to order the part, and time for the part to be delivered. ALDT should be representative of the actual time occurring in operational units. Improved availability not only will increase the number of available assets on a daily basis but also will allow units to perform a particular mission with fewer assets.

Early OSA Success

Shortly after standing up 1 year ago, the Operational Suitability Division provided OSA support to the ACC-directed operational utility evaluation (OUE) of the fighter aircraft command-and-control enhancement (FACE) pod on the F-16 and A-10 aircraft. US Central Command Air Forces validated an urgent and compelling requirement to establish a robust, beyond-line-of-sight command-and-control communications capability for fighter aircraft operations throughout Afghanistan, without additional communications infrastructure. This new capability would allow the combined air operations center to contact an aircraft via a satellite telephone call and pass real-time weather, target, and intelligence information to the pilot. To get this new capability into the hands of the warfighter as quickly as possible, organizations from both Air Force Materiel Command and ACC conducted a combined developmental and operational quick reaction test.

Test events consisted of both ground and flight events, with the ground events including technical order validation, ground checks, and loading verification procedures. Early OSA involvement identified reliability, maintainability, and availability issues associated with the FACE pod. Although current unit support equipment was suitable to upload and download the FACE pod, not all toolboxes contained the deep well 7/16 socket required to tighten FACE pod snubbers. During loading verification, the short umbilical cord on the prototype FACE pods made it difficult to install on an LAU-105 missile launch rail. Options for carrying AIM-9 missiles and a FACE pod on a dual rail adapter/launcher on the A-10 were not determined adequately; this was forwarded to the A-10 System Program Office for resolution.

Ground maintenance tests were able to determine that external built-in test and system status lights on the FACE pods were incompatible with night-vision goggles because the type of light-emitting diode used represented a source of hostile lighting for other aircraft during nighttime operations. In the end, the FACE pod proved not only effective but also suitable for use during this quick reaction OUE. Based on testing performed by the operational suitability analysts and other logistics team members representing all the Air Force logistics disciplines, the 53d Wing Commander was able to recommend fielding this system for immediate use in the area of responsibility within 1 year of receiving the tasking to design and test a new capability. The lead operational suitability analyst on this project, Master Sergeant Steve Clay, became so knowledgeable on FACE pod operation and loading procedures, he was selected as the Ace subject-matter expert. He subsequently was tasked to supervise the load training of 926th Fighter Wing (Air Force Reserve Command) maintenance personnel in preparing for their upcoming deployment in support of Operation Iraqi Freedom, the first operational use of the FACE pod.

Conclusion

While only a year old, this new group of operational suitability analysts already has made an impact on the 53d's tests and, most important, the warfighter. However, there is still a lot of room for improvement. The 28th conducts approximately 50 tests annually. With only five operational suitability analysts assigned, it is not feasible to have an operational suitability analyst assigned to every project. We will continue to add more suitability analysts and increase our capabilities. As systems become more expensive to operate and test, we are examining modeling and simulation tools. These new capabilities would allow us to utilize data gathered from limited test resources and extrapolate the information to simulate additional test articles with high confidence levels, thereby modeling actual anticipated results in the operational environment. Our goal is to find the problems before the warfighter does.

Captain John W. Garrison, USAF

Master Sergeant Stephen W. Clay, USAF

Technical Sergeant Jeffrey J. Kile, USAF

Captain Garrison is the Operational Suitability Division Deputy Chief, Master Sergeant Clay is the Weapon Suitability Branch Chief and Technical Sergeant Kile is the Integrated Avionics Suitability Branch Chief All are career aircraft maintainers assigned to the Operational Suitability Division of the 28th Test Squadron at Eglin AFB, Florida.

COPYRIGHT 2005 U.S. Air Force, Logistics Management Agency
COPYRIGHT 2005 Gale Group