Can FAA Salvage Its IT Disaster?

Delta Air Lines Flight 705 from Atlanta to Salt Lake City was at cruising altitude on Feb. 27 when a female passenger began to complain of severe chest pains. Fortunately, the tool was now in place that could help an air traffic controller quickly engineer an emergency landing route into Memphis.

INTERACTIVE TOOLS Spiral-development tools that speed your project planning.

This small feat represents one of the first victories the Federal Aviation Administration can claim after billions of dollars have been wasted trying to get control of air traffic over North America.

The Boeing 767-400, carrying 283 passengers, needed to make an unplanned landing in Tennessee. In an FAA facility at Memphis International Airport, air traffic controller Kenny Ellis got busy figuring out the safest and most direct route through the busy midday skies. There were more than a dozen planes just in his mid-altitude sector of the sky at the time.

Using a sophisticated new computer system called the User Request Evaluation Tool, or URET, Ellis could nearly instantly approve a flight path and coordinate with the tower controllers responsible for the plane's final approach. "URET lets you have more confidence in your decisions," says Ellis. "Just by a quick glance at the alert boxes on the left of the screen, I could see that URET didn't predict him to be in conflict with any other aircraft."

He then checked a graphical depiction of the new route, projected 20 minutes into the future and overlaid a map of his airspace and surrounding sectors. "From that display, I could see the point where his route was estimated to enter Memphis Approach Control's airspace." With the path and entry point clear, he was able to okay the best possible emergency route.

Better en route traffic management could save hundreds of millions of dollars per year in lost time and fuel costs for airlines operating in the United States, the ultimate end-users of this new system. The FAA hopes URET will eliminate as much as $100 million of the estimated $1 billion in routing-related waste that accrues, flight by flight, across the civil aviation system each year. Early indications are that that goal may be exceeded.

Memphis was a key test-site before URET's national rollout began in late 2001. One early beneficiary: Federal Express, the package delivery firm, which uses Memphis as its main hub for receiving, sorting and resending packages destined for all parts of the United States. With the new URET system, all FedEx planes get optimal routes. The freight shipper runs 130 planes a night out of Memphis. Ninety have standard, "most-efficient'' routes. The other 40 now get fast routes designed by URET, where only a handful got assistance before.

"The staffing is minimal on the midnight shift, but the box-haulers want to go as far down the road as they can on a direct path," says Memphis-based air traffic controller Phillip Hardee. The fewer en route course changes, the quicker and cheaper the flight. "In the past you could accommodate them according to your workload. Now, you can help them all."

Such early user satisfaction among both controllers and carriers could make URET a significant success for the Federal Aviation Administration, the agency in charge of the nation's air traffic infrastructure. The controllers get faster, easier-to-use information, the airlines save money, and passengers may get a smoother ride by avoiding storms and arrive a little earlier. In the case of Delta Flight 705 and other nearby planes, the safety level may even go up a bit as a passenger in pain is rushed to an airstrip near a waiting hospital.

But URET's on-time, close-to-budget delivery may make it something more: a demarcation point in the turnaround of what has been one of the largest and least successful information technology projects in history.

Challenge: Establish Credibility

Modernization of the U.S. air traffic control system has been a disaster story of epic proportions. The General Accounting Office estimates the cost of the push to modernize its information systems that the FAA started in 1981 will top $45 billion by 2005. Until recently, there was little to show for it. The most visible failure: the Advanced Automation System, which aimed to replace at one stroke several key computer and display systems, including the core mainframe architecture and the display screens that show controllers the planes in a given sector. That project cost more than $2.6 billion before the plug was pulled in 1994. By comparison, the Hoover Dam cost just $2.1 billion in today's dollars.

The GAO estimates $1.5 billion of the $2.6 billion was completely wasted, meaning none of the work could be salvaged. Modernization limped along for the next few years, always haunted by the FAA's record of ineptitude, which in turn made the frequently cash-strapped airlines reluctant to invest in equipment for their planes to work with the new systems.

Now, though, there is evidence that things are getting better at the FAA. "We are seeing a lot of positive signs," says GAO analyst Belva Martin, citing a change in management focus under FAA administrator Jane Garvey and the incremental development strategy behind programs such as URET. "They are moving in the right direction."

Controller Ellis, a 17-year veteran, also sees a change. "Overall, the FAA is doing better at delivering what we need."

Garvey, appointed by President Clinton in 1997, has pushed for changes in the way systems are developed and deployed. She emphasizes buy-in from users, including controllers, who have more say over the look and feel of new equipment, and cost-conscious airlines, now losing $7 billion a year, which have more input into where and when systems are deployed.

This is a critical moment for Garvey's modernization plan. After the Sept. 11 attacks on the U.S., she said in a speech to the National Air Traffic Control Association, "Our modernization plans will move ahead full throttle. Traffic will rebound. Demand will come back. Aviation is simply too important; too integral to our economy, to the fabric of our society, to our quality of life. It would be the height of irresponsibility for us to think or plan otherwise."

The Department of Transportation Inspector General's Office did suggest in a December report that the FAA pause to re-examine URET's parent program, Free Flight, and its broader plans for technology changes. The idea was raised "to assess security risks with the Free Flight concept, which technologies should receive priority, and how the agency's diverse efforts can be more effectively linked together."

However, FAA officials say they have already gone through that process and concluded they can't afford to let their plans slide. Flight delays were plaguing air passengers and freight before Sept. 11, and even with the terrorist-induced slowdown in traffic the rate of delays rose slightly in 2001, according to FAA statistics.

"Systems of this magnitude require long lead times to implement," says Amr El Sawy, general manager of the Mitre Center for Advanced Aviation Development, a federally funded research lab that works with the FAA. "What would you say when the traffic comes back? This requires consistent commitment and a consistent push on deployment."

Indeed, with the number of passengers dropping as much as 30% in the last quarter of 2001 and 350 aircraft getting taken out of the skies and parked, timing may be working in the FAA's favor. It has a chance to get its new systems in place before capacity becomes an issue at America's airports again.

Nobody inside the agency or out pretends the turnaround is complete—or that the job will ever truly be done, given the size of the system and the continuing advance of technology. Sitting in his tenth-floor office in the FAA's box-like headquarters just across the street from the Smithsonian museum complex, assistant administrator for research and acquisitions Steven Zaidman is guardedly positive. "We've got some successes, we've got some losers," says the rumpled and world-weary veteran of some of the FAA's darkest hours.

The GAO continues to flag a variety of FAA initiatives as among the government's most high-risk technology projects. A key example is the Standard Terminal Automation Replacement System (STARS) project to bring new technology to the controllers who manage airport arrivals and departures. STARS has grown from the $940 million project envisioned when Raytheon won the contract in 1996 to $1.4 billion, and is running about 4 years behind, with completion now expected in 2008. Even that schedule may prove ambitious if deployment of a related digital radar system continues to lag, according to the GAO.

An inspector general's office report also singled out URET for its cost, saying the FAA now estimates the first release will overshoot its original $172 million budget by 14 to 21%, meaning it will end up somewhere between $194.8 million and $207.6 million. However, the agency says that extra cost has to be measured against URET's benefits and speed of deployment.

"The knock against the FAA in the past has been that we couldn't deliver what we promised," says John Thornton, acting director of the Free Flight office responsible for URET.

The Creed of Spiral Development

Free Flight was created by Garvey to bring new technologies like URET to the field more quickly. Free Flight's secret sauce is a technique called "spiral development"—known within the agency as "build a little, test a little, deploy a little"—an approach quite different from the FAA's traditional practice of delivering (or failing to deliver) enormous projects in one big package on a specified date.

Lockheed Martin is a longtime FAA contractor and the lead vendor on the $629 million URET program. "There are some differences in working with Free Flight that helped make this program successful," says Diane DeSua, director of the URET program at Lockheed Martin. "Their relationship with their users, the agreements with the unions from the beginning, working with the controllers, it all helped keep the requirements stable."

Also important was the early involvement of the airlines, which through the industry group RTCA Inc. (organized in 1935 as the Radio Technical Commission for Aeronautics) helped decide where URET should be deployed first.

Spiral development is now becoming religion within the FAA proper, and is credited in part with reviving the STARS program. Costs grew and speed had suffered as controllers objected to changes in the hardware and software they touched, and the agency ultimately agreed to make modifications to its "off- the-shelf" approach that weren't in the original plan.

Once that was decided, however, the agency worked to develop a process that would help controllers buy into plans and results. The project was restructured so the first phase would include only the hardware and software used by controllers, grafted onto the existing backend computer that tracks planes in the airport vicinity.

This Early Display Capability (EDC) was rolled out to relatively low-volume airports in El Paso, Texas, and Syracuse, N.Y., in December 1999 and January 2001, respectively. By giving controllers practical experience with the new workstations, the STARS team gathered feedback for later full-service releases.

Garvey also created a chief information officer position, now filled by former AT&T International CIO Dan Mehan. Mehan doesn't directly control air traffic systems development but has a role in setting strategy. In addition to such traditional technology tasks as upgrading the e-mail system, the CIO sets standards and promotes development and process-improvement methodologies. He also has the point on information security, more pressing after 9/11.

But the bulk of the responsibility for acquiring air traffic control hardware and software, contracting for development services, supervising the contracts, and testing new systems lies with Zaidman, the assistant administrator for research and acquisitions. Other FAA divisions set requirements for new systems and make sure safety rules are enforced.

Even with the focus on buy-in and lowering risks on big projects, there are plenty of critics who say the progress the FAA has made is inadequate. "The problem is we don't really have a grand plan," says John-Paul Clarke, a professor of aeronautics and astronautics at MIT. He sees the successes the FAA points to as baby steps too incremental to solve the system's real problems, such as delays and controller productivity in the face of sometimes-overwhelming traffic volume.

With demand expected to resume its historical growth in lockstep with the nation's Gross National Product, Clarke says the FAA needs to do more to meet that demand. "What the FAA is working on is all incremental, small Band-Aids that will give you small improvements—1% or 2% here and there. We need something bold, something that will handle 50% more capacity."

The Operational Evolutionary Plan (OEP), under which the FAA tries to show how all its programs fit together, isn't a "grand plan," Clarke says. "That's let's see what we're doing and put it all together in one document."

OEP Director Charlie Keegan says the document is more than that—it's where the FAA lays out its commitments to the industry about what it will deliver and when.

Keegan, previously the founding director of the Free Flight program, says working with the industry to determine what needs to be done and setting deadlines is a big cultural change for the FAA. As for complaints that it's not enough, he declares, "We don't see any magic available." The best solution for the system's problems is to make determined, steady progress, not only on technology but also on basics like paving new runways, he says.

The Birth of STARS

George L. Donohue, a systems engineering professor at George Mason University who once ran research and acquisitions for the FAA, says much of what's wrong with the agency stems from an inability to stand up to its unions, who wind up holding veto power over what gets deployed.

His bitterness stems largely from STARS, the Raytheon system selected on his watch. STARS was originally developed for Norway, and has since been deployed elsewhere in Europe and the world. Donohue thought it could be deployed in the U.S., with a few tweaks. But the National Air Traffic Controllers Association (NATCA) complained it hadn't been consulted adequately and warned the system would be unsafe if deployed in its original form.

The controllers wanted STARS to have an ABC keyboard like the one they had grown used to, rather than the QWERTY keyboard Raytheon proposed. Although not necessarily the most logical or ergonomic, QWERTY is the standard typewriter and computer keyboard, named after a key sequence on the left side. Controllers wanted STARS to mimic consoles of the existing Automated Radar Terminal System (ARTS) system. That not only meant giving STARS an ABC keyboard but also a bunch of knobs and dials resembling the analog controls on an ARTS console.

Controllers said they feared that in a crisis they might lose critical time reaching for a knob that wasn't there or hesitating over an unfamiliar keyboard. The knobs provide fast access to functions such as changing the range of radar data displayed on the screen.

Bill Blackmer, the safety and technology director for NATCA, says there is nothing so crazy about controllers demanding hardware tailored to their high-pressure job. Besides, TRACON controllers don't sit at a keyboard and type. Instead, they occasionally hit a few keys required to accomplish a specific function.

"These programs were being built by people used to building PC computers," Blackmer says. "The engineers thought we'd sit in front of the computer and go through all the windows and menus. They didn't realize that time is the enemy of the air traffic controller."

Today, the FAA's strategy is that working with its unions makes more sense than working against them. In any case, the FAA heads charged with leading the development and deployment of new systems don't have the power to change the rules of the game. Given the hand they have been dealt, they are playing the game more skillfully than before, and they are starting to stack up some wins.

Challenge: Fixing System of Systems

The challenge is immense.

Current air navigation systems are organized by assigning planes to preset flight paths, defined by a series of radio beacon codes; flight progress is monitored by radar, with controllers speaking instructions over radio. The FAA would like to move to a system that makes more use of satellite-assisted navigation, giving pilots moving displays of the other planes in their vicinity, and decreasing reliance on voices by adding a digital data link between controllers and pilots.

First, the agency has to break free of the limitations of systems designed in the 1950s and '60s. For example, one of the backbone systems, the Host Computer System, continues to run on software written in a dead mainframe computer language, Jovial, and even those scarce Jovial programmers still working have a hard time navigating all the patches that have been applied to the software over the years.

The system has been upgraded by adding new mainframe hardware, and a new front end was added through a project called the Display System Replacement (DSR) that gave the controllers Unix-based workstations.

Host is one of many systems that was supposed to be replaced by AAS, but the agency is only now ramping up a replacement project. Under the FAA's new strategy of proceeding a step at a time, replacing an old mainframe system that still basically works just hasn't been a priority.

The air traffic control system is really several interlocking systems. Airport tower controllers are in charge of landing, take-off, runway assignments, and preventing collisions on the ground.

Flight arrivals and departures are managed by Terminal Radar Approach Control facilities (TRACONs), which track and direct planes from just outside the range at which they are visible from the tower (about 5 miles) to a range of 40 or 50 miles. Although a TRACON is typically located at a major airport, TRACON controllers also have to worry about traffic bound for smaller airports in their region, helicopter flights over nearby cities, and so on.

For years, TRACONs have used the computer system called ARTS that was originally built by IBM and is now in the hands of Lockheed Martin, which acquired IBM's old Federal Systems business. The replacement for ARTS, Raytheon's STARS, wound up running about four years behind schedule.

Both ARTS and STARS track arriving and departing flights using radar data, display the position of each plane on a controller's screen, and help with the process of making sure planes don't get too close together. There are many versions of ARTS in use, including some with updated color displays that rival those offered by STARS, but STARS is supposed to provide one common replacement system that makes more use of open standards like Unix and off-the-shelf hardware. As a departing flight moves away from the airport, TRACON controllers take it to the edge of the area they control and hand it off to the next control center in its flight path. Usually, for a cross-country flight, that's an En Route facility—one of the air traffic control centers that specialize in managing the long-haul portion of a flight.

Here, the key computer systems are Host, the legacy back end, and DSR, the more modernized controller's workstation. It's also where URET is being deployed, as a sort of sidecar to DSR for the second member of a controller team. While one controller watches the real-time radar monitor, his partner uses URET's 20-minute projections of flight paths to detect and avert conflicts that would put one plane into the path of another.

Challenge: Managing Risk

The Free Flight office has its home in Washington's stately Southern Railroad building, far from FAA headquarters. "Free Flight was set up to act, look, and feel different than what people see at FAA," says acting director Thornton, a one-time controller who was fired in the 1982 PATCO strike. This is where spiral development was nurtured, a program office with its own staff, reporting to Garvey.

As it finishes its $838 million introductory mission of developing a group of projects called Free Flight Phase 1 (FFP1), Free Flight internally has the look of a winner. One indication of its newfound status: original director Keegan is now overseeing the FAA's Operational Evolution Plan, the master blueprint of modernization. "Charlie's promotion is an example of how the lessons learned here have moved there," says Thornton.

Another measure of progress is that Free Flight's leaders seem to know when to quit. In September, a program called pFast, which figures out the best runway assignments for inbound planes, was mothballed despite successful deployment and testing at the Dallas/Forth Worth airport—and $40 million spent.

This is part of the spiral method of assessing risks early on, before committing to a big rollout. Customizing pFast to work within the topographies of other cities would have been prohibitively expensive. "Making it work at six or seven early sites would be $100 million extra, and the ROI isn't there," says Thornton. "It's not a success, but it is an important signal that we are doing business differently."

By contrast, Free Flight's biggest win to date is URET, which helps reroute planes in flight, including Delta 705. Controllers love it, and airlines stand to save a lot of money with it. Models show there is an average possible savings of one to two minutes per flight in the U.S. Saving one minute per flight en route would add up to $1 billion in savings per year to the airlines in terms of fuel, salaries, and depreciation, says Dave Knorr, Free Flight Performance Metrics Manager.

"We are going after a portion of that," he says. "We saw with the prototypes in Memphis and Indy that we could capture more than one-tenth of it." If so, URET could generate savings equal to its $296.5 million development and initial rollout budget in just three years.

A large, complex program, with 625,000 lines of code that had to be integrated with the FAA's core Host Computer System, URET is a good example of the spiral development method in use. Originally developed by Mitre, a federally funded research lab, the tool was refined in close collaboration with the controllers and started delivering benefits while still a prototype. A "hardened" version of the tool produced by Lockheed Martin will be deployed to seven control facilities this year and the remaining 13 en route centers by 2005.

"Mitre got the requirements stable and let us understand the human factors. That provides a lot of value," says Lockheed Martin URET program manager Diane DeSua. "We reuse the trajectory modeling algorithms and some concepts, but our system is totally re-architected." Because Mitre's research on URET dates back so far, the software was a mix of C and PL/1 code that would have been hard to maintain long-term. Lockheed Martin did a complete rewrite in Ada.

Controllers like Ellis and Hardee in Memphis were originally promised a decision-support tool, but they soon found that URET made obsolete the paper "flight strips" on which they had always kept track of a plane's flight data. Somewhat narrower than an index card, these strips kept controllers apprised of air speed, altitude, and direction.

Now data are posted on screen. Constant updates and easy visibility allow quicker reactions to changing conditions, so they can track more routes further into the future, optimize flight paths for more planes at once, and get more information on routing changes to the Host computer. "It was advertised as a tactical support tool, but we saw that with a few modifications it could be more than that," says Hardee. "It was our suggestion that we be able to send flight plan amendments directly to the Host computer."

Knorr says controllers can process up to 60% more amendments, or changes to flight plans en route, and that changes that once didn't get entered into the system if controllers were busy are now available for everyone to see and use.

"We were aware of the spiral approach all the way along," says Ellis. Controllers used to getting complete systems dumped in their laps have had input on usability. "They are real open to suggestions we have, even now," says Ellis. True, says Jesse Wijnjtes, URET project manager at Free Flight. "We are just now moving into feedback from our initial seven deployments, so we can continue to improve as it rolls out to the remaining centers next year."

Another difference between URET and previous projects is that Lockheed Martin maintains responsibility for software maintenance and support, instead of turning it over to the FAA as has been typical in the past. That makes maintenance more expensive for the FAA, but the extra cost is meant to be offset by better service and responsiveness. Time will tell if this strategy bears out.

Challenge: Stepping up Pace

But URET is not immune from criticism. Former acquisitions chief Donohue agrees it's a genuine functional improvement, but says it was a long time in the making, with the original prototype created in the 1980s. Algorithms that would automatically resolve conflicts between aircraft flight paths were dropped so URET could be promoted as a "decision support tool" that controllers would find less threatening, he says.

Mitre officials say the next version of URET will give controllers conflict-resolution capability in a way they like. URET PARR (Problem Assessment Resolution and Ranking) will eventually be handed off to Lockheed Martin and refined into a component of Free Flight Phase 2.

The PARR version will further boost productivity by simultaneously analyzing multiple alternatives for assigning new flight paths and avoiding collisions with other planes. For example, instead of experimenting with trial flight plans that assign a plane to a different altitude until they hit on one that works, URET PARR would let controllers see at a glance which altitudes will work (coded green) and which won't (coded red). Controllers like this version better because it lets them make the final call.

Clearly, air traffic control modernization is not a pure software development problem. It's a systems development problem that involves hardware, software, procedures, human factors, unions, and politics.

In some ways, the challenge is different only in degree from implementing a new supply chain that depends as much on trucks and trust between partners as it does on computer software. But issues of life and death loom larger here than in corporate projects, making it much more difficult to tell users to make do with whatever functionality their software provides.

"You've got to understand people aren't telling you the product will be late or the phone call will be dropped. They're telling you that it may be less safe," says FAA CIO Mehan.

"When our systems lock up, aircraft do not stop moving," says NATCA's Blackmer. He rejects the idea that controllers are opposed to all change, saying many of them are frustrated that the modernization process isn't moving forward faster. "The AAS really is the FAA's Vietnam. Every time they think of doing something big, they say, 'Oh, no, we failed at that.' It's so ingrained within the FAA now that it paralyzes us with inaction."

And yet, the incremental approach is delivering progress where before there was virtually none. "Clearly, ATC is not going to be changed, en masse, overnight," says Agam N. Sinha, the vice president at Mitre CAASD. The FAA designed its Operational Evolutionary Plan to keep pace with the 30% growth in traffic anticipated for 2010, he notes. "Some people say that's not enough because the delays in the air traffic system are not acceptable today.''

However, a revised traffic forecast released in March suggests that the one-billion-passenger milestone the FAA expected to see in 2010 won't arrive until 2013. The FAA admits to considerable uncertainty about how quickly air traffic will recover from the impact of 9/11, but it projects that passenger statistics will continue to be depressed by about 12% through Sept. 2002. After that, the agency predicts a return to a growth rate of 4% per year.

Meanwhile, perhaps some real progress is possible. The FAA has done a better job in recent years of setting performance goals and trying to manage to them. For example, it continues to squeeze a higher rate of availability out of key systems and facilities, having boosted overall availability to 99.56% in 2001, up from just over 99% in 1992. But the fact remains that the agency is still missing its targets in several key areas, such as cutting delays and reducing controller operational errors.

"There are people who have suggested what we really need is a revolution," Sinha adds. "You can define lots of good systems that can, maybe, do the job better. But the question is, how do you get from here to there? If you're not able to define that, you're not able to solve the problem."

Federal Aviation Administration Base Case

Headquarters: 800 Independence Avenue, S.W., Washington, D.C. 20591

Phone: (202) 366-4000

Business: U.S. Federal agency overseeing civil aviation, including air traffic control and airports

Top Technology Executives: Steve Zaidman, Associate Administrator for Research and Acquisitions; Dan Mehan, Chief Information Officer

Financials: $14.2 billion (FY 2002 budget)

Challenge: Modernize air traffic control despite lengthy inability to upgrade systems effectively

Baseline Goals: Deploy new terminal area control system without exceeding revised $1.4 billion budget Reduce airplane delays from 253 per 100,000 departures, to no more than 171 Cut the number of fatalities from 3.7 in every 10 million departures, to no more than 1 Reduce the errors controllers make to no more than 5 in every 1 million maneuvers they manage, from 7.3

Copyright © 2002 Ziff Davis Media Inc. All Rights Reserved. Originally appearing in Baseline.