DAU's systems engineering department revamping Sys-301 course: systems engineering competencies at core of recent changes - Systems Acquisition Process

Systems engineering is an interdisciplinary engineering management process that evolves and verifies an integrated, life cycle balanced set of system solutions that satisfies customer needs. It clearly is at the heart of the systems acquisition process, and the DoD relies heavily on systems engineers to provide technical support to program managers. In fact, the Systems Planning, Research, Development and Engineering (SPRDE) career field has more members than any of the other 12 Department of Defense (DoD) acquisition career fields. One way the DoD ensures that its systems engineers possess the needed competencies to perform their jobs is through a certification process that includes specific training requirements.

The Original SYS-301 Course

In 1991 Congress passed the Defense Acquisition Workforce Improvement Act (DAWIA). In response, DoD created the Acquisition Workforce Certification Program. This program established education, training, and experience criteria for each of the 13 acquisition career fields. The SPRDE career field has as one of its criteria for level III certification completion of a two-week course called Systems 301 (SYS-301), Advanced Systems Planning, Research, Development and Engineering (ASPRDEC).

As the first step in developing this course, the Defense Acquisition University (DAU) conducted a number of workshops with field activities to get their input on what material should be covered. We received over 400 distinct suggestions, ranging from broad areas such as configuration management to specific skills such as being able to do word processing. These inputs were narrowed down into about 30 topic areas by combining related items. Lesson plans, including learning objectives, were then developed for each topic area. A DoD-level functional board oversaw this process and by 1994 the first class was held.

Need for Change

In subsequent years, course materials were updated to ensure currency and minor changes were made to most lessons to incorporate suggested improvements, but the learning objectives and course structure remained relatively stable. By the end of 1999, however, it was becoming obvious that the skills and competencies needed by senior technical managers were changing. Issues such as the need for systems interoperability and increasing use of software to perform system functions were becoming vitally important. On top of this, a major change in how the DoD would do systems acquisition was soon to be expressed in the new 5000 series of acquisition regulations.

Recognizing that it was time to make a substantial revision to the SYS-301 course, DAU's Systems Engineering Department began a process of interviewing 20 program managers and technical managers from the three Services, other agencies, and industry. Managers were asked what they felt were the skills that senior technical managers needed to be most effective, These skills were then consolidated into 15 skill areas.

Once these skill areas were compared to the SYS-301 course content, we discovered that several subjects covered in the course--such as International Acquisition and Environment, Safety and Health (ESH)--had not been mentioned in the interviews. Since the goal of this research was to provide direction for a curriculum revision, we decided to modify the original 15 areas to ensure that all topics already covered in the course were included.

Figure 1 displays the final list of 22 areas. We termed these "Systems Engineer Competencies." By no means is this a comprehensive list of all competencies required by systems engineers. Domain-specific technical knowledge, organizational knowledge, management skills, and many behavioral competencies are not included. These competencies were beyond the scope of SYS-301, and it was assumed they would be acquired through other means.

The interview process did not prioritize the competencies and used too small a sample to treat it as representing the thoughts of the entire acquisition community. The next step, therefore, was to develop a questionnaire that could be administered to a broad cross-section of senior program managers, engineers, and technical managers.

The first part of the questionnaire asked a number of demographic questions. The second part, displayed in Figure 2, asked respondents to rate each of the 22 competencies as having high, medium, or low importance. In order to distinguish those of highest importance, respondents were asked to rate not more than eight competencies as high.

The questionnaire also asked for an assessment of the degree to which DoD's technical workforce possessed each competency. We did this to allow an analysis showing where there were gaps between: 1) how important a competency is, and 2) the current level of competence. Space was provided for respondents to add any competencies not on the list that they thought were important.

The survey was administered to 137 students while they attended the SYS-301 class, to 96 students in the Advanced Program Management Course (APMC), and to 90 senior technical managers throughout DoD. Respondents were able to fill out the questionnaire online with their input going directly to the Center for Research, Defense Acquisition University They were then able to analyze and sort responses by Service, years of experience, rank, and several other categories using the "Group-Systems Survey Tool" by Group-Systems.com

Study Results

Figure 2 shows the overall rankings for the entire population of 323 responses, with competencies listed in descending order of importance. The corresponding assessment of the degree to which people in the SPRDE career field possess these competencies is listed in the right column of Figure 2.

The first four competencies were rated "high" importance by the majority of the respondents. The next nine had at least twice as many "high" ratings as "low" ratings. Only the last three had more "low" than "high" ratings. Of interest is the frequency of "low" ratings for the observed degree of competence. Only five of the 22 competencies received more "high" than "low" ratings. Half had more than twice as many "low" ratings as "high" ratings.

For the most part, the three sample groups had similar rankings for both "importance" and "degree observed." There were, however, some notable exceptions.

* The Advanced Program Management Course students ranked "operational cost reduction" and "financial management" much higher in importance and "test integration" much lower in importance.

* Senior technical managers ranked "ethics" much higher in importance.

* The ASPPDEC students ranked "modeling and simulation" much higher and "adaptability" much lower in importance.

* The ASPRDEC students ranked "assimilation and communication of technical information" and "adaptability" much lower for degree observed.

* Senior managers ranked "user focus" much lower for degree observed.

Since the three groups had different perspectives due to the nature of their jobs, some differences were to be expected.

Analysis

Our next step was to determine what this research suggested as to how we should adjust the SYS-301 course material. Rather than merely focus on the most important competencies, we felt it important to take into account how well the SPRDE population was already doing in each area. The thought here was that areas where we are already doing well may not need extra emphasis, even if they are important. Conversely areas where we are doing poorly may need extra emphasis, even if they are not the most important.

To help us in this assessment, we did a "gap analysis." We assigned numerical values to high (8), medium (4), and low (1) ratings and then averaged the responses to get numerical values for each competency. The results (Figure 3, next page) show that significant gaps exist in about half of the competencies.

After evaluating this information we drew the following conclusions about how much emphasis each area should receive in SYS-301. This does not imply that any of these areas are unimportant. It just recognizes that time constraints limit what material can be covered in one class. To the extent students need additional training in areas we don't strongly emphasize, other resources are available to provide that training.

STRONGLY EMPHASIZE

Total Systems View (H)

Risk Management (H)

Teaming (H)

User Focus (M)

Assimilation and Communication of Technical Information (M)

Software Management (L)

Management of Changing Technology (M)

EMPHASIZE

Operational Cost Reduction (L)

Adaptability (L)

Evolutionary Acquisition/Open Systems (M)

Logistics Integration (L)

Test Integration (L)

EMPHASIZE SOMEWHAT

Modeling and Simulation (M)

Post Production Support (L)

Design for Manufacturing (M)

Financial Management (L)

Contract Technical Management (M)

Configuration Mmanagement (L)

LITTLE EMPHASIS

Earned Value (L)

Ethics (M)

Environment, Safety and Health (M)

International Acquisition (M)

The next step was to determine how much emphasis each area was already receiving in SYS-301. All faculty members then teaching the class were asked to estimate how many hours were spent in each area. Areas with more than 4 hours were ranked "high," those with between 2 and 4 hours were ranked "medium," and areas with less than 2 hours were ranked "low". These ratings are shown in parentheses in the areas of emphasis covered above.

Students in three ASPRDEC classes were also asked their assessments of what was actually being taught. They agreed closely with the instructors except for the area "management of changing technology" which they felt should have much more emphasis. A comparison between what was needed and what was being taught shows agreement in most areas, but it also highlighted several areas where changes were indicated.

More Emphasis

Software Management

Operational Cost Reduction

Management of Changing Technology

Less Emphasis

International

Environment, Safety and Health

Ethics

DoD Workforce Report

During this same period, the Office of the Under Secretary of Defense for Acquisition, Technology and Logistics was conducting a study of competencies required by the DoD acquisition work-force. This study resulted in a report titled "Future Acquisition and Technology Workforce Report." This report listed 435 functional competencies. Of these, 274 were relevant to the SPRDE career field. These were grouped into 33 "environmental trends."

A comparison with our study results indicated very good agreement. For example, two of the environmental trends were "increased emphasis on interoperability" and "increased emphasis on software development." A number of environmental areas addressed the need to reduce operational costs through a variety of means.

Implementation

With this information in hand, we began to incorporate changes into the course material. A new two-hour lesson titled "Architecture and Interoperability" addressed systems architectures and current policy on interoperability requirements and certification. This knowledge is then used in a case study where students develop a system architecture and look at interface requirements. Six hours of instruction in software acquisition were added. Topics covered include policy, development strategies, the software life cycle, best practices, and software risk management.

In order to make space for this new material, we decided to eliminate the Contracting Issues lesson and shorten the ESH and International Acquisition lessons. While the survey results didn't make a strong case to eliminate the contracting lesson, we felt that most of the students had already received more contracting training than we were providing. We also felt that those students needing more training in this area would be better served by taking a separate contracts course. The ESH and International Acquisition lessons were shortened in response to our survey results.

Some additional material on operational cost reduction was added to existing lessons, but this area requires further work. While the Ethics lesson, which is based on the Challenger incident, remains, we are using it to also address issues of effective decision making in addition to purely ethics issues.

The Way Ahead

We found the survey on systems engineering competencies very useful in helping us develop a road map to revise SYS-301. While many of the results seem intuitively obvious, it was important to have inputs from a broad cross-section of the acquisition community before we proceeded. We did find some obvious gaps between what was being taught and what our customers felt they needed. We also had what we felt was a revelation upon seeing the significant gaps between the perceived importance of a set of competencies and the perception of how skilled our SPRDE workforce is in these areas. Clearly, there is a great need for continued training.

While we changed SYS-301 significantly, we will continue to make additional changes that will address those study issues that have not yet been implemented. Our changes will also include new material required to keep the course current as the acquisition environment continues to evolve.

SYS-301 is not the only systems engineering course to have undergone change. In June of 2001 DAU introduced a revised Systems 201 (SYS-201) course--Intermediate Systems Planning, Research Development & Engineering--that converted what was formerly a two-week, in-residence course into a hybrid course with a distance learning portion followed by one week in class. One of the options we are considering for SYS-301 is to convert it to a hybrid course in the future.

Editor's Note: For questions/comments, contact Falk at martin.falk@dau.mil.

FIGURE 1

Systems Engineer Competencies

1. Total Systems View

Ability to think beyond engineering and consider all functions and
stakeholders in the Systems Engineering process. Ability to understand
the entire acquisition process.

2. Teaming

Ability to build, work in, motivate, and lead high-performing
multidisciplinary teams.

3. User Focus

Ability to understand the user's perspective and requirements. Ability
to conduct requirements analysis and to structure Research and
Development work effort to match user needs.

4. Contract Technical Management

Ability to understand contractors' processes and perspectives, to work
with contractors and provide informed assessments of their progress, and
to understand the source selection process.

5. Configuration Management

Ability to manage and communicate changes to systems in all phases of
the life cycle.

6. Post Production Support

Ability to identify improvements to systems for the purpose of
Operations and Support (O&S) cost reduction, safety, replacing obsolete
parts, reliability, tech insertion, etc. Ability to use Systems
Engineering process to implement these changes.

7. Financial Management

Ability to understand the Planning, Programming and Budgeting System
(PPBS) system, sources and uses of funds, and how budget issues impact
the program.

8. Operational Cost Reduction

Ability to assess design impact on Total Operational Cost and identify
means to reduce Total Ownership Cost (TOC). Ability to understand
designing for change using techniques such as open systems
architectures. Ability to understand Cost As an Independent Variable
(CAIV).

9. Risk Management

Ability to plan, assess, handle, and monitor risk.

10. Management of Changing Technology

Awareness of current state-of-the-art, and mechanisms to introduce
technology. Ability to accurately assess technology maturity of a
system. Ability to understand Information Technology and how to
effectively acquire and use it Ability to understand spectrum
management, system security, and Joint Technical Architecture.

11. Earned Value

Ability to understand Earned Value (EV) principles, evaluate EV data,
and make recommendations.

12. Software Management

Knowledge of software development principles and techniques. Ability to
integrate software considerations into the systems engineering process,
Ability to assess development progress and identify risks and pitfalls.

13. Design Impacts on Manufacturing

Understanding of producibility issues and how to manage the design for
producilility.

14. Modeling and Simulation

Ability to understand uses of Modeling and Simulation (M&S). Ability to
use M&S throughout the life cycle and to assess the contractor's use of
M&S. Ability to work in an integrated data environment.

15. Ethics

Ability to understand ethical considerations and adhere to ethical
principles.

16. Environment, Safety and Health (ESH)

Ability to understand Environment, Safety and Health (ESH) requirements
and how to design systems to effectively meet those requirements.

17. International Acquisition (IA)

Ability to understand International Acquisition (IA) policy and
techniques. Ability to utilize offshore technology in system design
where it provides a benefit Ability to understand interoperability
requirements.

18. Test Integration

Ability to assist in test planning and design. Ability to respond to
issues arising during test.

19. Logistics Integration

Ability to understand designing for supportability. Ability to develop
or evaluate design changes in response to supportability issues.

20. Evolutionary Acquisition/Open Systems Architecture

Ability to develop Evolutionary Acquisition (EA) design strategies and
ensure system design supports the EA approach. Ability to understand
open systems architectures.

21. Assimilation and Communication of Technical Information

Ability to evaluate technical issues, assess program performance, make
recommendations, and effectively present these issues to diverse
audiences.

22. Adaptability

Ability to respond quickly and effectively to changing conditions or
events that impact the program's systems engineering process.

FIGURE 2

Questionnaire

    Competency                        Importance      Observed

 1. Total Systems View                   High         Moderate
 2. User Focus                           High         Moderate
 3. Risk Management                      High         Moderate
 4. Teaming                              High         Moderate
 5. Assimilation & Communication of  Moderate-High    Moderate
    Technical Information
 6. Software Management              Moderate-High  Moderate-Low
 7. Management of Changing           Moderate-High  Moderate-Low
    Technology
 8. Test Integration                 Moderate-High    Moderate
 9. Operational Cost Reduction       Moderate-High  Moderate-Low
10. Adaptability                     Moderate-High  Moderate-Low
11. Logistics Integration            Moderate-High    Moderate
12. Configuration Management         Moderate-High    Moderate
13. Contract Technical Management    Moderate-High    Moderate
14. Evolutionary Acquisition/Open      Moderate     Moderate-Low
    Systems Architecture
15. Financial Management               Moderate     Moderate-Low
16. Design Impacts on Manufacturing    Moderate     Moderate-Low
17. Ethics                             Moderate     Moderate-High
18. Modeling & Simulation              Moderate     Moderate-Low
19. Post Production Support            Moderate          Low
20. Earned Value                     Moderate-Low   Moderate-Low
21. Environment Safety & Health      Moderate-Low     Moderate
22. International Acquisition        Moderate-Low        Low

Falk is the functional manager for Systems Engineering at DAU's Fort Belvoir, Va., campus. He has over 30 years of acquisition experience, including assignments as the Belvoir Research, Development and Engineering (RD&E) Center's Operations Director and as Chief Scientist at U.S. Army, Europe. Falk is a graduate of APMC 98-03. Also contributing to this article was DAU Professor of Acquisition Management Bill McGovern. A graduate of PMC 95-1, McGovern also serves as Director of DAU's Management Deliberation Center.

COPYRIGHT 2002 Defense Acquisition University Press
COPYRIGHT 2003 Gale Group