-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
1/24
Using Parametric Software Estimates
During
Program Support Reviews
Chris MillerOffice of the Deputy Director,
Software Engineering and System Assurance
SYSTEMS & SOFTWARE ENGINEERINGOffice of the Deputy Under Secretary of Defense
for Acquisition and TechnologyUS Department of Defense
October 2008
Version 2.0
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
2/24
Systems & Software Engineering Chris Mil ler, Octob er 2008 Slide 2
OUSD (AT&L) OrganizationMay 2006
USD, AcquisitionTechnology & Logistics
DUSD, Acquisition &
Technology
Dir, Joint Advanced
Concepts
Dir, Systems and
Software Engineering
Dir, Portfolio
Systems Acquisition
Industrial
Programs
Defense Procurement
and Acquisition Policy
Small Business
Programs
Defense Contract
Management Agency
Defense Acquisition
University
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
3/24
Systems & Software Engineering Chris Mil ler, Octob er 2008 Slide 3
Director, Systems &
Software Engineering
Systems and Software Engineering
Director, Systems &
Software Engineering
Deputy Director
Enterprise Development
Deputy Director
Developmental Test
& Evaluation
Deputy Director
Software Engineering &
System Assurance
Deputy Director
Assessments & Support
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
4/24
Systems & Software Engineering Chris Mil ler, Octob er 2008 Slide 4
Elements of a DoD Strategy for Software
Support Acquisition Success
Ensure effective and efficient software solutions across the acquisition
spectrum of systems, SoS and capability portfolios
Improve the State-of-the-Practice of Software Engineering
Advocate and lead software initiatives to improve the state-of-the-practices through transition of tools, techniques, etc.
Leadership, Outreach and Advocacy
Implement at Department and National levels, a strategic plan for
meeting Defense software requirements
Foster Software Resources to meet DoD needs
Enable the US and global capability to meet Department software
needs, in an assured and responsive manner
Promote World-Class L eadership for Defense Software Eng ineer ing
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
5/24
Systems & Software Engineering Chris Mil ler, Octob er 2008
Notional View ofSoftware Measurement
Software Engineering and Systems Assurance (SSA) initiatives Program feasibility analysis using estimation models
Software Resources and Data Report: Feasibility Study
Revision of MIL-HDBK-881A to improve software guidance
Integration of software metrics with EVM to assess consistency
of estimates
Concepts - Requirements - Arch/Design Development -Maintenance
Determine methods of
obtaining cost estimating
dataUse estimation tools,
techniques, processes,& practices
Generate SW
appropriate
WBS
Earned Value
Management (EVM)
for SW
Link SW quality
indicators to EVM
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
6/24
Systems & Software Engineering Chris Mil ler, Octob er 2008 6
Driving Technical Rigor Back Into ProgramsProgram Support Reviews
Program Support Reviews provide insight into aprograms technical execution focusing on:
SE as envisioned in programs technical planning
T&E as captured in verification and validation strategy
Risk management - integrated, effective and resourced
Milestone exit criteria as captured in Acquisition Decision Memo
Acquisition strategy as captured in Acquisition Strategy Report
Independent, cross-functional view aimed at providingrisk-reduction recommendations
The PSR reduces r isk in the technical and prog rammatic
execut ion of a prog ram
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
7/24Systems & Software Engineering Chris Mil ler, Octob er 2008 7
Program Support Review Activity(as of August 2008)
PSRs/NARs completed: 57
AOTRs completed: 13
Nunn-McCurdy Certification: 13
Participation on Service-led IRTs: 3
Technical Assessments: 13
Decision Support Reviews
DAE Review
5%
OTRR
9%
Other
25%
Pre-MS C
19%
Pre-MS A
3%Pre-MS B
26%
Nunn-
McCurdy
13%
Service-Managed Acquisitions
Marine
Corps
10%Army
27%Navy
21%
Air Force
33% Agencies9%
Programs by Domain Area
Unmanned 4%
Land 13%
C2-ISR 13%
Ships 8%Munitions 3%
Rotary Wing
21%
Comms 3%
Space 4%
Business 2%Missiles 9%
Fixed Wing
18%
Other 2%
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
8/24Systems & Software Engineering Chris Mil ler, Octob er 2008 8
DoD Software Performance:What Were Seeing*
Software issues are significant contributors to poorprogram execution Schedule realism (compressed, overlapping)
Software requirements not well defined, traceable, testable
Immature architectures, COTS integration, interoperability,obsolescence (electronics/hardware refresh)
Software development processes not institutionalized, planningdocuments missing or incomplete, reuse strategies inconsistent
Software test/evaluation lacking rigor and breadth
Lessons learned not incorporated into successive builds Software risks/metrics not well defined, managed
*Based on over 60 program reviews over the past 3 years
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
9/24Systems & Software Engineering Chris Mil ler, Octob er 2008 9
Software Engineering and SystemAssurance (SSA) Role
SSA produced software estimates to support severalPSR teams in 2007 Program A: SSA was asked to assess software schedule
feasibility prior to MS B
Program B: Significant software issues
Opportunity for SSA to support program decision makingby providing software estimates Estimation activities aimed at gauging overall program feasibility
and quantifying magnitude of top program risks Focus on support for engineering vs. budgeting decisions
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
10/24Systems & Software Engineering Chris Mil ler, Octob er 2008 10
Program A
PSR of aircraft program with ambitious schedule Three years (36 months) from Milestone B to LRIP
Modifications needed to meet U.S. requirements
Developed three software estimates Software size (SLOC) estimates provided by program office
Re-estimated both new and reused code Based on reuse and code growth
Estimates for optimistic, typical, and pessimistic were identified
as Min, Mid, and Max Used three parametric models COCOMO II, SLIM, and SEER-SEM
Most input parameters set at nominal
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
11/24Systems & Software Engineering Chris Mil ler, Octob er 2008 11
Program A
Estimation & Feasibility Analysis Given an adjusted code size (ASIZE) of 918K to 1590K SLOCs
A range of 5375 to 9571 person months should be expected over a 62to 74 calendar month schedule
All three models forecasted 65 to 68 months (assuming a 50%confidence-level)
Result: Analysis revealed existing acquisition strategy was not feasible
Service added more schedule to acquisition strategy
DUSD(A&T) estimates change in acquisition strategy saved $5 billion
COCOMO II Min Mid MaxASize (KSLOC) 918 1,204 1590
Effort (PM) 5375 7147 9571
Duration (CM) 62 68 74
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
12/24Systems & Software Engineering Chris Mil ler, Octob er 2008 12
Program B
Major subcontractor with significant software content did not use any
parametric estimating tool
Firm Fixed Price subcontract
At one point there was a substantial conflict on estimates between
major subcontractor and prime, related to requirements
One portion of software grew from ~250KSLOC to 863KSLOC (3.5x),reflecting shall not degrade current capability requirement
Increase in software engineering staff by 57 percent (40 people)
over 9 month period Based on schedule, contractor should be drawing down software staff
Significant variation in code estimates during review
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
13/24Systems & Software Engineering Chris Mil ler, Octob er 2008 13
CSCI Supplier Original NM Review 2 wks. later New Reuse Complete % Test coverage to-date
DR's/Problem
reports
1 A 40,000 374,739 250,844 201,144 49,700 75% 68% 47
2 B 27,000 50,363 50,363 40,363 10,000 90% 85% 0
3 A 20,000 38,962 38,962 38,962 0 71% 60% 5
4
C 325,000 415,249 401,732 131,921 269,811 98% 98%
1
5 5
6 0
7 D 65,000 283,255 25,467 18,367 7,100 82% 60% 0
8 C 33,000 373,587 100,129 68,129 32,000 99% 99% 0
9 E 15,000 7,879 7,879 2,000 5,879 100% 100% 1
10 F 114,000 218,609 218,609 16,409 202,200 99% 95% 0
11 G 26,000 25,622 34,544 8,922 25,622 100% 100% 2
12 H 6,000 16,580 16,580 15,580 1,000 100% 100% 2
13 H 2,000 34,806 34,806 17,206 17,600 100% 100% 0
14 I 33,000 42,355 42,355 31,655 10,700 99% 97% 1
15 G 15,000 126,238 119,626 6,433 113,193 100% 100% 0
16 K 23,000 26,404 26,404 2,604 23,800 91% 91% 1
17 L 7,000 12,500 12,500 1,000 11,500 100% 100% 1
18 C 1,000 29,121 29,121 25,621 3,500 100% 100% 7
19 M NA 100 1,000 1,000 Unavail 100% 100% 0
20 C NA 2,600 2,600 400 2,200 81% 43% 0
752,000 2,078,969 1,413,521
Program B
Changes in SLOC estimates
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
14/24Systems & Software Engineering Chris Mil ler, Octob er 2008 14
Program B Schedule Estimates
0
200
400
600
800
1000
1200
0 5 10 15 20 25 30
Months
StaffMon
ths
50 KSLOC (Nominal)
150 KSLOC (Nominal)
100 KSLOC (Nominal)
150 KSLOC (Optimal Schedule)
100 KSLOC (Optimal Schedule)
50 KSLOC (Optimal Schedule)
Time from Detailed Requirements Complete Through End of Developmental Testing
75 KSLOC remaining
based on last estimate
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
15/24Systems & Software Engineering Chris Mil ler, Octob er 2008 15
Program B
Estimation and Feasibility Analysis Contractor appears to be driven to meet schedules versus costs
Significant increase in effort for minimal schedule savings
Uncertainty in scope of remaining software development Need to develop firmer size estimates
Recommendations PMO reach a decision on unstable requirements to prevent further code
growth in or out Program office bring in parametric estimating consultant to review
contractors estimates for most volatile software components
Result Acquisition Decision Memorandum requires Service to conduct
software review
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
16/24Systems & Software Engineering Chris Mil ler, Octob er 2008 16
Integrating Management Indicators
Estimation& Planning
Measurement
& Analysis
Risk
Management
SDP:
WBS, IMP, IMS,
Metric Definitions
Data
& Insight
Prioritized
Risks
Risk
Mitigation
Status
Program
Context
Insight
Needed
SSA S ft M t &
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
17/24Systems & Software Engineering Chris Mil ler, Octob er 2008
SSA Software Measurement &Analysis Initiatives
Software EVM Study/Pilot Focus: Development of cost controls for software component of
program Execution: Partnering with DCMA, ARA, PA&E, Contractors
Cost Estimation / SRDR Feasibility Study Focus: Analysis on software cost estimation practices, availabilityof historical data, and increasing confidence in estimates
Execution: Partnering with PA&E/DCARC and Hill AFB STSC toevaluate existing data sources
Work Breakdown Structure Handbook Update Focus: Recommendations for update of MIL HDBK 881 toimprove handling of software
Execution: Partnering with Services, DCMA, ARA, DAU, NDU,PA&E/DCARC and using NDIA expert panel
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
18/24Systems & Software Engineering Chris Mil ler, Octob er 2008
Software EVM Study/Pilot
Development of a methodology combining EVM andsoftware metrics analysis to predict cost and scheduleoverruns
Piloting on a 5-year ACAT 1D SW developmentprogram, as part of a cross-cutting study
Pilot indicator shows ETC (estimate-to-complete)forecasts for:
Existing program management plans
EVM measures
Software metrics (i.e, growth profile of size, effort, and defects)
Equivalent ETC forecasts provides an increasedconfidence in project plans
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
19/24Systems & Software Engineering Chris Mil ler, Octob er 2008
Integrated EVM Concept
CPMO Spiral - Sample Summary
Performance Measurement Baseline
$71
$478
$706
$792
$941
$1,051
$1,145
$1,318
$1,420
$47
$228$291
$338
$565715
$1,032
$1,251
$1,370
$1,471
$1,559 $1,579
$0
$200
$400
$600
$800
$1,000
$1,200
$1,400
$1,600
$1,800
Oct07 Nov07 Dec07 Jan08 Feb08 Mar08 Apr08 May08 Jun08 Jul08 Aug08 Sep08 Oct08 Nov08 Dec08
TIME (Months)
DOLLARS($K
BAC BCWS (PV) BCWP (EV) ACWP (AC) ETC
BAC = $1420
Data Date
EV M etrics
Calculated Schedule Range:
09/08/08 - 10/12/08
Calculated Estimate at Complete
Range:
$1,797K - $2,515K
Cum ulative CPI = 0.79SW Metrics
Calculated Schedule Range:
05/14/08 - 06/18/08
Calculated Estimate at Complete
Range:
SW MetricETC
Program
Plan ETC
EV ETC
Acquisition Oversight confidence increases as ETCs overlap
i.e., as multiple measures indicate to same conclusion
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
20/24
Systems & Software Engineering Chris Mil ler, Octob er 2008
SRDR Feasibility Study
SSA funded a STSC feasibility study on normalizingDCARC data
Purpose of the study was to determine if the DCARC data setcould be used to develop a parametric software estimation model
Dr. Randall Jensen at Hill AFB is the creator of Sage and SEER-
SEM
Preliminary study results Data was submitted using a wide variety of definitions that made
it difficult to use directly, but was successful in normalizing thedata and validating the results
The data collected in the SRDR forms allows a first-levelparametric model to be constructed (lack of environmentinformation prohibits the creation of a detailed model)
Minimum development time projection model
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
21/24
Systems & Software Engineering Chris Mil ler, Octob er 2008
Minimum Software Development TimeProjection Model
Source: Long, L. G. et al, Aerospace Corp Report,2004
Impossible
Zone
Analysis of CSCI data: Apparent Maximum Size of 200 KSLOC (per CSCI)
Apparent Minimum Schedule (Impossible Zone)
Environment & Complexity are primary drivers of variation
Apparent Maximum Schedule of 50 months
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
22/24
Systems & Software Engineering Chris Mil ler, Octob er 2008
Updating MIL-HDBK-881A forSoftware Overview
Revision objectives: Revise to make handbook acceptable of software
engineering practice
Correct errors
Notable changes:
Replace 'material item' with ' acquisition program' Add words to make 'artifacts' equal to 'products'
Include words that make 'product-oriented' and 'DoDAFarchitecture' views interchangeable WRT to creating a WBS
Results Compiled a Comment matrix
Drafted a new Appendix B (for software)
During final review and clean up, the working groupwas made aware of 90s effort (draft MIL-HDBK-171)
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
23/24
Systems & Software Engineering Chris Mil ler, Octob er 2008
MIL-HDBK-881A Next Steps
The Software Cost Control working group goal is toprovide an community-coordinated set ofrecommendations
Reached out to industry members of NDIA to review the
suggested changes Ensure recommended changes are improvements from industry
perspective
Obtain additional examples to include in Appendix B
NDIA feedback is due on September 19th
Submission of suggested changes to PA&E ARA willtake place after collection of NDIA feedback andconsolidate into a single baseline
-
7/28/2019 6_USC COCOMO Forum - Miller - PSR Feas Analysis 1021008
24/24
Questions/Discussion
Contact Information:
Chris MillerSSE/SSA SupportSoftware Engineering & Systems AssuranceODUSD(A&T) Systems & Software [email protected]
mailto:[email protected]:[email protected]