research profile systems engineering...

seari.mit.edu © 2008 Massachusetts Institute of Technology 1

RESEARCH PROFILE

Systems Engineering EconomicsOctober 21, 2008

Dr. Ricardo Valerdi

Massachusetts Institute of Technology

[email protected]

mit.edu/~rvalerdi/www


Outline

• Description of systems engineering economics– Reuse

– Human Systems Integration

– Optimism

– Adoption

– Integrating systems and software

– Heuristics for cost estimation


Research Description

SYSTEMS ENGINEERING ECONOMICS

This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering

value to stakeholders. Examples include:

• Measurement of productivity and quantifying the ROI of systems engineering

• Advanced methods for reuse, cost modeling, and risk modeling

• Application of real options in systems and enterprises • Leading indicators for systems engineering

effectiveness


Why measure systems engineering?Cost Overrun as a Function of SE Effort

NASA Data

Honour, E.C., “Understanding the Value of Systems Engineering,” Proceedings of the INCOSE International Symposium, Toulouse, France, 2004.


Feasibility Plans/Rqts. Design Develop

and TestPhases and Milestones

Relative

Size

Range

Operational

Concept

Life Cycle

Objectives

Life Cycle

Architecture

Initial

Operating

Capability

x

0.5x

0.25x

4x

2x

Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.

Cone of Uncertainty


COSYSMO

SizeDrivers

EffortMultipliers

Effort, $

Calibration

# Requirements# Interfaces# Scenarios# Algorithms

- Application factors-8 factors

- Team factors-6 factors

Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008.


Reuse Terminology

New:

Items that are completely new

Managed:

Items that are incorporated and require no added SE effort other than technical management

Adopted: Items that are incorporated unmodified but require verification and validation

Modified: Items that are incorporated but require tailoring or interface changes, and verification

and validation

Deleted: Items that are removed from a legacy system, which require design analysis, tailoring

or interface changes, and verification and validation

Notes:• New items are generally unprecedented• Those items that are inherited but require architecture or implementation changes should be

counted as New


Modified vs. New Threshold

Reuse Continuum

Modified

Adopted

New 1.0

0

Deleted

Managed

Reu

se w

eig

ht0.65

0.51

0.43

0.15

Wang, G., Valerdi, R., Fortune, J., “Reuse in Systems Engineering,” aimed for IEEE Systems, Man and Cybernetics – Part: C. (under review)


Human Systems Integration

• The overall objective of the research is to contribute to the goals of making acquisitions more cost effective and to better support the warfighter by effectively meeting HSI requirements.

• The proposed project will develop approaches to validate two hypotheses: (1) The percentage of HSI effort can be estimated as a function of the total

systems engineering effort and used for predictive purposes on future programs; and

(2) Current systems engineering leading indicators can be extended for HSI and used to improve the predictability of HSI programmatic and technical performance on a program.

• Specific Aims: The study will focus on tools and methods to incorporate human systems integration into new and existing systems engineering economic models and indicators.


Design task(tools/methods)Design accomplished through:

• Requirements analysis

• Quality function

deployment

• Feasibility analysis• Operational requirements & maintenance concept

• Functional analysis

• Design trade-off studies

• Simulation & modeling

• Requirements allocation

• Reliability & maintainability

analyses

• Human system integration

• Supportability analysis

• Test and evaluation

• Risk analysis

• Other supporting analyses

FunctionalGrouphardware

FunctionalGroupsoftware

FunctionalGrouphuman

EquipmentComputerSoftwareunits

HumanActivities/duties

HumanTasks/Subtasks

HardwareStructure

SoftwareStructure

MPRequirements

ComponentIntegration &prototypes

SoftwareComponentintegration

PersonnelDevelopment &Training

EquipmentTesting

SoftwareTesting

PersonnelTesting

Requirements Analysis

Functional analysis(systems level)

Evaluation(system integrationAnd testing)

Equipment& Accessories

SoftwareConfiguration∫ ∫

Day-to-day designIntegration activities

Modified from Blanchard & Fabrycky, Systems Engineering and Analysis, 2006, pp. 106

Integration of Hardware, Software, & Human Life Cycles

Design Requirements(criteria)*

Design for:• Performance

• Cost-system effectiveness

• Reliability

• Maintainability

• Political, Social, & Tech

Feasibility

• Human Factors

• Safety

• Environment

• Occupational Health

• Manpower

• Personnel

• Training

• Survivability

• Habitability

• Vulnerability

• Supportability

• Producibility

• Reconfigurability

• Affordability

• Disposability

• Flexibility (growth)

* applicable to all levels in the system structure and tailored to specific program needs


Systems Engineering Defined

• Acquisition and Supply

– Supply Process– Acquisition Process

• Technical Management

– Planning Process– Assessment Process– Control Process

• System Design

– Requirements Definition Process– Solution Definition Process

• Product Realization

– Implementation Process– Transition to Use Process

• Technical Evaluation

– Systems Analysis Process

– Requirements Validation Process

– System Verification Process

– End Products Validation Process

EIA/ANSI 632, Processes for Engineering a System, 1999.


Cost Driver Clusters

UNDERSTANDING FACTORS

– Requirements understanding

– Architecture understanding

– Stakeholder team cohesion

– Personnel experience/continuity

COMPLEXITY FACTORS

– Level of service requirements

– Technology Risk

– # of Recursive Levels in the Design

– Documentation Match to Life Cycle Needs

OPERATIONS FACTORS

– # and Diversity of Installations/Platforms

– Migration complexity

PEOPLE FACTORS

– Personnel/team capability

– Process capability

ENVIRONMENT FACTORS

– Multisite coordination

– Tool support

Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008.


Optimism in Systems Engineers


Koehler, D. J., Harvey, N. (1997). Confidence judgments by actors and observers. Journal of Behavioral Decision Making. 10, 221-242. Russo, J. E. and Schoemaker, P. J. H. (1992). Managing

overconfidence. Sloan Management Review, 33(2), 7-17.

Why are other disciplines better at estimating?


Quantifying Optimism Bias

Confidence (fi)

Acc

urac

y (d

i)

0 0.5 1

1

0.5 fi > di optimistic

fi = di calibrated

fi < di pessimistic

( )∑=

−

=N

i

ii dfN

scoreBrier1

21_

fi = respondent’s probability that their judgment is correctdi = outcome of the respondent’s accuracyN = total number of judgments

Where fi is a subjective probability di is an objective (empirical) probability


Cox, W. M., Alm R., “You Are What You Spend,” NY Times, Feb 10, 2008.


What Makes an SE Tool Adoptable?

(supply side)

• Well documented

• Trialabilty

• Low barrier of entry

• Transparency

• Demonstrates value

• Variety of incentives

• Tailorable

• Information freshness

• Relative advantage

• Compatibility

• On-going peer support

• Credibility

• Agility

• Flexibility

• Failure modes

• Enabled by IT


Ranking of Adoption Attributes (n=35)

Adoption Attributes

2.63

2.49

2.4

2.29

2.29

2.23

2.23

2.11

1.97

1.89

1.77

1.57

0 0.5 1 1.5 2 2.5 3

Well_Documented

Credibility

Demonstrates_value

Low_Barrier_of_Entry

Information_Freshness

Transparency

Compatibility

Tailorable

On-going_Peer_Support

Variety_of_Incentives

Relative_Advantage

Trialability

Att

rib

ute

Score

Must-be

One-dimensional

Attractive

Valerdi, R. “Cultural Barriers to the Adoption of Systems Engineering Research,”2nd Asia-Pacific Conference on Systems Engineering, Yokohama, Japan, September 2008.


Dimensions of Organizational Culture(demand side)

Social Science

• Power distance – the extent to which a society accepts the unequal distribution of power in the organization

• Uncertainty avoidance – the extent to which people are comfortable or uncomfortable with uncertainty and little structure

• Individualism – the extent to which individuals are supposed to be self-reliant and look after themselves, versus being more integrated into a group

• Masculinity or Femininity – hardness vs. softness; toughness vs. tenderness

• Long term or short term orientation –the culture’s members having a stance on delayed, or immediate, gratification

Management

• Innovation and risk taking – willing to experiment, take risks, encourage innovation

• Attention to detail – paying attention to being precise vs. saying its “good enough for chopped salad”

• Outcome orientation – oriented to results vs. oriented to process

• People orientation – degree of value and respect for people. Are people considered unique talents, or is an engineer an engineer an engineer?

• Individual vs. Team orientation – are individuals most highly noted, or are collective efforts

• Aggressiveness – taking action, dealing with conflict

• Stability – openness to change

Hofstede, G., Culture and organizations: Software of the mind. London: McGraw-Hill, 1991.

O’Reilly, C., Chatman, J., & Caldwell, D., People and organizational culture: A profile comparison approach to assessing person-organization fit. Academy of Management Journal, 34, 487-516, 1991.


Culture of Technology

TechnologyPractice

Cultural AspectGoals, values, and ethical codes, beliefin progress, awarenessand creativity

Organizational AspectEconomic and industrial activity,professional activity, users andconsumers, trade unions

Technical AspectKnowledge, skill, and technique,tools, machines, chemicals, resources, products and wastes

General meaningof “technology”

Restricted meaningof “technology”

Pacey, A., The Culture of Technology, MIT Press, 1983.


COCOMO II COSYSMO

Software Systems Engineering

Wang, G., Valerdi, R., Roedler, G., Ankrum, A., Gaffney, J., “Harmonizing Systems and Software Estimation,” working paper.


COCOMO II COSYSMO

Rele

van

t ta

sks

n

ot

co

vere

d

Software Systems Engineering


Contract Engineering WBS Based On Standards

1.0 – System/Project1.1 – Integrated Project Management (IPM)

1.2 – Systems Engineering

1.3 – Prime Mission Product (PMP)1.3.1 – Subsystem / Configuration Item (CI) 1…n (Specify Names)

1.3.2 – PMP Application Software

1.3.3 – PMP System Software

1.3.4 – PMP Integration, Assembly, Test & Checkout (IATC)

1.3.5 – Operations/Production Support

1.4 – Platform Integration

1.5 – System Test & Evaluation (ST&E)

1.6 – Training

1.7 – Data Management

1.8 – Peculiar Support Equipment

1.9 – Common Support Equipment

1.10 – Operational / Site Activation

1.11 – Industrial Facilities

Product-oriented construct, by tailoring MIL-HDBK 881A and

ANSI/EIA 632

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management

Six Functions:

1. Systems Engineering

2. Software Engineering

3. Electrical Engineering

4. Mechanical Engineering

5. Support Engineering

6. Project Engineering Management


Rechtin’s

Systems

Architecting

Heuristics

COSYSMO

Model

COSYSMO

Tool

Model

Development

Heuristics

Model

Calibration

Heuristics

Model

Usage

Heuristics

Cost

Estimation

Heuristics

inspired implemented in

experiencelead to

confirmed

Valerdi, R. “Zen in the Art of Cost Estimation,” 2nd Asia-Pacific Conference on Systems Engineering, Yokohama, Japan, September 2008.

Experiential Closed Loop


Elements of Successful Systems Architects

1. Know the engineering fundamentals on which each architecture is based

• Common sense, derived from specific engineering fundamentals and the experience of other architects, is needed to reduce the search to practical dimensions

2. Experience and judgment are necessary• Hands-on system problem solving is mandatory

3. Acquire the insights gained from experience in the design laboratories on the job

Rechtin, E. 1991. Systems Architecting: Creating & Building Complex Systems, Upper Saddle River: Prentice Hall.


Development-related Heuristic

Heuristic #5: Some system characteristics are

more likely to be cost penalties than cost savings.

Migration Complexity

Nominal High Very High Extra High

Legacy

contractor

Self; legacy system is well documented. Original team largely available

Self; original development team not available; most documentation available

Different contractor; limited documentation

Original contractor out of business; no documentation available

Effect of

legacy

system on

new system

Everything is new; legacy system is completely replaced or non-existent

Migration is restricted to integration only

Migration is related to integration and development

Migration is related to integration, development, architecture and design


Calibration-related Heuristic

Heuristic #6: All calibrations are local.

Before local calibration

After local calibration


Implications for Systems Engineering

� Modeling of socio-technical phenomena• Architecture understanding, process

capability, stakeholder team cohesion, requirements understanding

� Quantification of emergent properties• Impacts of “ilities” on system costs

• Drivers of complexity

• Reuse

• Diseconomies of scale


Agenda

9:00 Welcome and Introductions Dr. Donna Rhodes, SEAri Director

9:30 SEAri – Overview of the SEAri Research Program Dr. Donna Rhodes, Research Director

10:00 Research Profile: Socio-Technical Decision Making and Designing for Value Robustness

Dr. Adam Ross, Research Scientist

10:45 Break

11:00 Research Report: Designing Systems for Survivability Matt Richards, Doctoral Research Assistant

11:30 Research Report: Real Options in Enterprise Architecture Tsoline Mikaelian, Doctoral Research Assistant

noon LUNCH

1:00 Research Profile – Systems Engineering in the Enterprise Dr. Donna Rhodes, Principal Research Scientist

1:30 Research Report: Leveraging Organizational Culture, Standard Process, and Team Norms to Enable Collaborative Systems Thinking

Caroline Lamb, Doctoral Research Assistant

2:00 Stretch Break

2:10 Research Profile: Systems Engineering Economics Dr. Ricardo Valerdi, Research Associate

2:50 Research Poster Session with Refreshments SEAri Research Assistants

4:15 Participant Feedback and Recommendations for Research SEAri Leadership

4:55 Closing Remarks Dr. Donna Rhodes

5:00 Adjourn

research profile systems engineering...

Documents