New Frontiers

Discovery

February 2010 0

National Aeronautics and Space Administration

Dennon Clardy, Program ManagerDiscovery & New Frontiers

Project Management Challenge2010

Improving Life-Cycle Cost Management of Spacecraft Missions

Used with Permission

New Frontiers

Discovery

Overview of Programs

1

• Uncoupled program addressing high-priority science objectives in solar system exploration• Frequent opportunities for science community to propose full investigations• Fixed-price cost cap (cradle to grave), excluding the launch vehicle• Full and open competition• Principal Investigator-led project

• Established in 1992• $425M cap per mission (FY09)• Mission opportunities every 1.5 to 2 yr• Open science competition for all solar

system objects, except for the Earth and the Sun

• Established in 2003• $625M cap per mission (FY09)• Mission opportunities every 3 to 4 yr• Addresses high-priority investigations

identified by the National Academy of Sciences

February 2010

New Frontiers

Discovery

February 2010 2

Juno

Mars Pathfinder

CONTOUR

ASPERA-3 on Mars Express

D&NF Science Mission Targets

– The Moon (Lunar Prospector, M3, GRAIL)

– Mars (Mars Pathfinder, ASPERA-3)

– Inner Planets (MESSENGER, Strofio)

– Outer Planets (New Horizons, Juno)

– Comets (CONTOUR, Stardust, Deep Impact, EPOXI, NExT)

– Asteroids (NEAR, Dawn)

– Interplanetary Space (Genesis)

– Extra-Solar System (Kepler)

Dawn

Genesis

Stardust

MESSENGER

NEARLunar Prospector

M3 on Chadrayaan-1

New Horizons

Kepler

GRAIL

New Frontiers

Discovery

Program Organization

3

Phase B, C, D, ESMD confirmation review at end of Phase B.

SMD AA makes confirmation decision.

PM becomes heavily engaged following selection.

Programmatic Authority, Responsibility, Reporting, & Accountability

Scientific Reporting & Accountability

NASA Headquarters

Program Office

Projects

Associate Administratorfor the

Science Mission Directorate

PrincipalInvestigators

MissionProgram Executive

Project Manager(Implementing Organization)

Agency PMC

NASA Center PMC orAppointed Board

D&NF Program Executives

Mission Manager

D&NFProgram Manager

MissionProgramScientist

D&NF Program

Scientist s

D&NFProgram Director

MSFC CMC

February 2010

New Frontiers

Discovery

Study Impetus: Cost and Schedule

February 2010 4

Mission Cost Growth (percent of proposed cost)

Mission Schedule Growth (launch date slip)

Legend:Historic Mission Data Included in Study

0 00

5

10

15

20

25

30

35

40

45

50

Perc

ent C

ost G

row

th

-20.0

-10.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

Perc

ent C

ost G

row

th

Legend:Historic Mission Data Included in Study

New Frontiers

Discovery

Look for and address the “boulders” that are under the control of the program and project

Study Impetus & Research Plan

“Assess the cost escapes that have occurred on recent D&NF missions. Determine how the cost escapes are making it through our processes and determine what reasonable things we can do to either prevent them or manage them better.”

Assess LCC growth at decision gates throughout the mission development process, identifying causes affecting cost increases over the missions’ life cycle

Identify factors that contribute to the occurrence of unplanned costs and significant mission cost cap overruns

Based on findings, provide specific recommendations and implementation plans to improve current processes and provide a greater level of insight to make better-informed decisions throughout the mission life-cycle

February 2010 5

New Frontiers

Discovery

Study Implementation

INADEQUATE PLANNING FOR OPERATIONS / PHASE E Project

OPTIMISTIC HARDWARE / SOFTWARE INHERITANCE and TECHNOLOGY READINESS ASSUMPTIONS Project

INEXPERIENCED PROJECT TEAM FOR PLANETARY MISSIONS Project

INSUFFICIENT MANAGEMENT OF AND INSIGHT INTO CONTRACTOR TASKS/PERFORMANCE Project

LACK OF OR INADEQUATE INTEGRATED PROJECT SCHEDULES Project

UNIQUE/SPECIAL TASKS/WORK OUTSIDE OF AREAS OF DEMONSTRATED EXPERTISE Project

ADDITION OF NEW NASA REQUIREMENTS AFTER SELECTION Project

INSTABILITY IN NASA PROGRAM BUDGETS Project

NO LESSONS LEARNED FEEDBACK/FEEDFOWARD PROCESS Program

INADEQUATE RISK MANAGEMENT PROCESS AT TRANSITION TO IMPLEMENTATION Program

INADEQUATE CONSIDERATION OF INDEPENDENT REVIEW TEAM FINDINGS AND RECOMMENDATIONS Program

INABILITY TO PROVIDE CREDIBLE COST ESTIMATES EARLY IN DEVELOPMENT Program

February 2010 6

List of Major Findings

AssociatedRecommendations

PI’s

PM’s

PE’sTMCO

Mission Mgrs

Institutions

Mission 4Gate

Analysis

Mission 1Gate

Analysis

Mission 2Gate

Analysis

Mission 3Gate

Analysis

Mission 5Gate

Analysis

Milestone Timelines LCC & Development Histories Findings Matrices

Contributors

New Frontiers

Discovery

Major Findings

CONSIDERATION OF REVIEW TEAM FINDINGS

INEFFECTIVE MANAGEMENT STRUCTURE

INTEGRATED PROJECT SCHEDULES

INSUFFICIENT PROJECT INSIGHT

PROJECT TEAM INEXPERIENCE

INADEQUATE MISSION REPLANS

INADEQUATE PLANNING FOR OPERATIONS / PHASE E

HERITAGE and TECHNOLOGY ASSUMPTIONS

FAULT PROTECTION AND AUTONOMY

February 2010 7

We did not find a “magic bullet” in any of our findings...There is no single action that ensures cost and schedule control...

New Frontiers

Discovery

Consideration of Review Team Findings

NASA commissions senior-level expert review panels, yet does not always address (carefully consider and mitigate or refute) the panel’s conclusions and recommendations

– In most instances, cost drivers had been identified as a weakness, finding, or deficiency during reviews (source selection, milestone, or internal reviews)

– There was little evidence indicating a consistent approach to considering, analyzing, or responding to the findings and recommendations

Review teams are there to help the project: project management needs to make the best use of their knowledge and expertise

– Schedule sufficient time for review and analysis of issues (findings, issues, concerns, or actions) from reviews

– Foster a project atmosphere that moves beyond “not invented here” to ensure that issues are addressed

– Ensure follow-through with risks, liens or threats, or budget adjustments

– Document the disposition of all findings in a manner that facilitates resuscitating issues as the situation changes

February 2010 8

New Frontiers

Discovery

Ineffective Management Structure

Ineffective management structure and unclear roles and responsibilities resulted in cost and schedule impacts to missions

– Primarily occurred within projects involving multiple organizations • Inconsistent project reporting and decision-making• Unclear lines of technical authority• Unconnected systems engineering across multiple organizations• Unclear responsibility for system integration

– The management structure issues directly compounded the effects of other embedded project issues (e.g., heritage and technology problems, cost control, prime contractor inexperience, etc)

An effective Project and technical management structure should be validated and continually reassessed

– Project formulation needs to ensure a clearly documented project organizational structure, with a clear delineation of responsibilities and decision making authority

– Site visits should verify that the key parties clearly understand the organizational structure and the decision process

– Reviews should include a briefing and assessment of the effectiveness of the Project organization structure

February 2010 9

New Frontiers

Discovery

Integrated Project Schedules

The lack of a comprehensive, integrated Project schedule results in uncoordinated activities, inefficiencies in resource management, and increased costs

– Missing critical milestones and major events resulting in underestimated resources, and insufficient data for tracking performance

– Missing logical relationships (interdependencies), or unidentified or incomplete critical paths, resulting in underestimated resources, schedule delays, and poor decision making

– Multiple separate, uncoordinated schedules resulting in incomplete data for tracking performance, missing logical relationships, and unidentified critical paths

Project management should ensure valid, comprehensive, Integrated Master Schedule is in place on each project that forms the basis of and facilitates EVM reporting

– Document the proposed approach to developing and maintaining an integrated schedule, including a description of roles and processes for integrating multiple schedules, if any, into a single master schedule

– Use the Integrated Baseline Review (IBR) to confirm the content of the schedule, the processes for maintenance, and its ability to support weekly, monthly, and annual reviews and planning

– Perform a detailed review of the actual integrated schedule and associated processes as part of project milestone reviews (PDR, CDR, SIR) and periodically throughout the life cycle

February 2010 10

New Frontiers

Discovery

Insufficient Project Insight

Insufficient project management and technical insight into contractor performance results in poor communications, schedule delays, and technical problems that manifest as cost over-runs

– Specifications, processes, and procedures that did not meet institutional standards due to organization “cultural” differences or insufficient flow down of requirements

– Rework, retest, and waivers to hardware and software– Hardware mishaps– Additional personnel to perform the appropriate level of insight is added to correct issues

Project management should ensure that the structure and rationale for the approach to the insight and oversight is documented and understood at all levels

– Document the oversight approach (including rationale and staffing levels) for appropriate oversight based upon implementing center, prime contractor, and subcontractor/vendor experience

– Include a clear plan for appropriate mission assurance oversight at the prime and subcontractors/vendors

– Review and independently assess the effectiveness of their planned oversight approach at major milestones

February 2010 11

New Frontiers

Discovery

Project Team Inexperience

February 2010 12

Teams with major players with limited experience in planetary mission development are a major contributor to program management issues resulting in cost over-runs

– Complex or poorly-defined management structure, roles and responsibilities, and communications

– Inadequate development schedules and implementation of performance measuring techniques

– Inadequate performance oversight and configuration management: institute to prime and prime to sub

– Inaccurate cost estimates, inadequate cost control and management of reserves

Given there may be strategic benefits to selecting “new” team members, the Project and Program should assume a greater cost risk and operate with increased insight/oversight for less experienced team members

– Assess the relevant experience for major partners and identify associated risks– Develop mitigation plans, including mentoring and active sharing of lessons learned from

more experienced partners– Identify and carry a cost threat, adjusted to reflect continuing team performance– Review and independently assess the risks and associated mitigations at major milestones

New Frontiers

Discovery

Inadequate Mission Replans

February 2010 13

The impact of significant changes to mission scope, schedule, or funding profiles were not sufficiently understood, resulting in unexpected cost increases and schedule delays

– Program- and Project-driven changes, regardless of lifecycle timeframe, often result in underestimates of the effects (i.e., duration and complexity) on the operations phase

• Replans during development tend to concentrate on Phase C/D, so impacts to operations receive limited analysis and review

• Operations is often not well-defined early enough in the project development cycle to understand the full impact of any changes

Program and Project management need to work together to ensure sufficient time to perform a detailed assessment, develop a detailed revised baseline, and complete an independent evaluation

– Carefully consider ancillary effects such as heritage assumptions (see Heritage and Technology Assumptions)

– Specifically address any operational considerations and impacts (see Inadequate Planning for Operations/Phase E )

– Document the cost and schedule impact, along with a complete description of all associated drivers

New Frontiers

Discovery

Inadequate Planning for Operations/Phase E

February 2010 14

Phase E cost increases result from poorly scoped mission operations. Even moderate yearly underestimates can present significant LCC impacts for missions with long Phase E durations

– Early focus on hardware development schedules and cost (big ticket items)– Underestimating the complexity of the operation, ignoring special requirements for a

particular mission type, or inadequate planning for sustaining engineering

Project Management should emphasize from the beginning the need for an full and accurate life cycle cost estimate

– Emphasize the importance of a detailed operations concept that identifies plans for equipment maintenance/replacement and parts obsolescence, longevity plan for retaining/replacing key personnel

– Develop a Phase E staffing estimate by primary operations functional area that can clearly be traced to and supported by the operations concept

– Review and independently assess the operations concept and staffing estimates, including revisions to the operations concept driven by project replans

– Consider the risk of significant Phase D activities deferred into Phase E and the potential impact to operations staffing

– Manage deferred Phase D work as development effort, including appropriate metrics and schedules

New Frontiers

Discovery

Heritage and Technology Assumptions

February 2010 15

Optimistic hardware/software inheritance and technology readiness assumptions cause significant cost and schedule growth in Phases C/D

– Heritage is used to justify decreased cost and risk, but can increase cost and risk – Heritage is more than just a specific piece of hardware or software, it must be considered to

include all associated aspects (available documentation, knowledgeable personnel, operational paradigm, spacecraft design, etc.)

– New or reapplied technology may be required to enable a mission

Project Management should realistically assess both the benefits and risks associated with heritage and new technology, and ensure appropriate mitigations are identified and implemented

– Clearly demonstrate the relationship between heritage/technology rankings and cost/reserve estimates

– Create a living reference for the relationship between heritage/technology rankings and cost/reserve estimates, updated throughout formulation and implementation

– Capture high risk heritage/technology assumptions project risks and tracked for mitigation actions

– Cost factors/threats should be updated after each review, particularly heritage reviews– Review and independently assess the risks and associated mitigations at major milestones– Select membership of SRB or internal review teamsto ensure members possess needed

areas of expertise

New Frontiers

Discovery

Fault Protection and Autonomy

Missions underestimate the time and effort required to design, implement, and test fault protection and autonomy (FPA) capabilities

– Developing an integrated hardware, software, and operations approach to FPA starting early in the development life cycle

– Defining appropriate autonomy requirements and the proper level of fault protection– Estimating resource requirements, including hardware (test beds)

Institutions, programs, and projects need to develop a better understanding of the cost drivers associated with FPA to improve our processes and estimating capabilities

Project management needs to ensure FPA is addressed as a separate and equal engineering discipline

– Emphasize the importance of a comprehensive FPA approach starting with the initial system concept

– Develop an FPA development resource estimate that can clearly be traced to and supported by the operations concept

– Define FPA to the same level of maturity as the hardware and software for the mission concept throughout the mission life cycle

– Derive FPA from the operations concept, and implement a systems approach encompassing hardware, software, and operations

February 2010 16

New Frontiers

Discovery

Embedding and Impact Realization Timing

February 2010 17

Insight

Inexperience

Fault Prot.

Heritage/Tech.

Mgt. Structure

Phase E

Replan

Integrated Schedule

Impact PeriodEmbedding Period

Most FindingsEmbedded pre CR

Most ImpactsOccur post CDR

Impact PeriodEmbedding Period

Ability to descope content(cost) falls sharply post CR

Cost impacts dominate Phase D

Mitigation Actions must Focus on pre CR period in Project life cycle

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

SRR PDR/CRR/CR CDRLaunchMRR

FRRLRR

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

SRR PDR/CRR/CR CDRLaunchMRR

FRRLRR

New Frontiers

Discovery

Conclusion

February 2010 18

We did not find a “magic bullet” in any of our findings...There are no surprises here...

except that we continue to have problems in these areas

The Discovery and New Frontiers Program Office Life Cycle Cost Study was performed under the direction of Paul Gilbert (MSFC), led by Bryan Barley (MSFC), and supported by Kenny Mitchell

(MSFC-retired) and Marilyn Newhouse (CSC)

So what do we do with the results?

– First, avoid the temptation to acknowledge and then put on a shelf along with most of the previous cost growth studies

– Second, while there were no surprises, that does not mean there are no actions the entire community can take to significantly improve performance against approved budgets

– D/NF Program Office is implementing a fairly disciplined approach to • Continually identify actions we can take to avoid the traps identified in the study• Avoid the “checklist” mentality and ensure that project management activities and tools directly

contribute to planning, tracking, and controlling cost and schedule

– We are also available to engage in more detailed discussions concerning the study approach, findings, and potential actions with groups inside and outside of the D/NF community

New Frontiers

Discovery

February 2010 19

Supplemental Data and Backup

Improving the Life-Cycle Cost Management of Planetary Missions

New Frontiers

Discovery

Acronyms

February 2010 20

AO Announcement of OpportunityAR Acceptance ReviewARR ATLO Readiness ReviewATLO Assembly, Test, and Launch OperationsCDR Critical design ReviewCR Confirmation ReviewCRR Confirmation Readiness Review CSR Concept Study ReportD&NF PO Discovery and New Frontiers Program OfficeDOE Department of Energy DPI Deputy Principle InvestigatorEM Engineering ModelEVM Earned Value ManagementFPP Flight Practices and ProceduresFRR Flight Readiness ReviewFTE Full-Time EquivalentFY Fiscal YearIAT Independent Assessment TeamICE Independent Cost EstimateIRT Independent Review TeamLCC Life-Cycle CostsLRR Launch Readiness ReviewLV Launch VehicleMM Mission ManagerMOA Memorandum of AgreementMRR Mission Readiness ReviewPBR Project Budget ReportPCA Program Commitment AgreementPDR Preliminary Design Review

PDS Planetary Data SystemPI Principle InvestigatorPLRA Program Level Requirements Appendix

(to the Program Plan)PM Project ManagerPO Program OfficePSD Planetary Science DivisionRTG Radioisotope Thermoelectric GeneratorRM Resource ManagerRY Real YearSMD Science Mission DirectorateSRB Standing Review BoardSRR Systems Requirements ReviewTMCO Technical, Management, Cost, and OtherWBS Work Breakdown Structure

New Frontiers

DiscoveryBackgroundGeneric Mission Development Process

February 2010 21

Decision Gates are mission milestones at which NASA makes decisions that commit funding to a project’s Life Cycle Cost (LCC): Announcement of Opportunity (AO) Selection of Proposals – start of Phase A Selection of a project for development – start of Phase B Preliminary Design Review (PDR) Confirmation Review (CR) Program Level Requirements Appendix (PLRA) Phase C/D/E design and mission-readiness reviews

AO Development AO Implementation

Step 1 Step 2

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

AO Release

SRR PDR/CRR/CR CDRLaunch

6 – 24 mo 4 mo 6- 8 mo 6 – 12 mo 6 – 12 mo 24 – 36 mo 12 – 60+ mo

Competitive

Science/Mission Selection led by SMD Program Implementation led by Discovery PO

MRRFRR

LRR

AO Development AO Implementation

Step 1 Step 2

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

AO Release

SRR PDR/CRR/CR CDRLaunch

4 mo 6- 8 mo 6 – 12 mo 6 – 12 mo 24 – 36 mo 12 – 60+ mo

Competitive

Science/Mission Selection led by SMD Program Implementation led by Discovery PO

MRRFRR

LRR

New Frontiers

DiscoveryStudy Implementation Issues Affecting Data Collection

Collection, analysis, and synthesis of data much more intensive than anticipated

– Assumptions were made when necessary to correlate across missions, but data still valid for identifying driving issues

Lack of official/formal program documentation

– Little traceability to program-level decisions or direction (e.g., official letters, documents)

– Inconsistent records of year-to-year or life-cycle phased cost commitments and obligations to projects

Understanding of common program operating principles

– Institutions (i.e., projects) not clear on definition of cost cap

February 2010 22

Difficult to obtain and capture important/key aspects in the life of a mission due to programmatic practices

New Frontiers

Discovery

August 2007 23

High Level Summary of the Findings Timing of the Embedding and Impacts

Ability to descope content (cost) falls sharply post CR

Cost impacts dominate Phase D

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

SRR PDR/CRR/CR CDRLaunchMRR

FRRLRR

Phase A(Concept Studies)

Phase B(Preliminary Design)

Phase C(Detailed Design)

Phase D(Development)

Phase E(Operations & Data

Analysis)

Formulation Implementation

SRR PDR/CRR/CR CDRLaunchMRR

FRRLRR

Finding 2: Insight

Finding 8: Inexperience

Finding 6: Fault Protection

Finding 1: Heritage/Technology

Finding 7: Management Structure

Finding 3: Phase E

Finding 4: Replan

Finding 5: Integrated Schedule

Impact IntervalEmbedding Interval

Most findings embedded pre-CR Most impacts occur post-CDR