Stemming Architectural Decayin Software Systems

Nenad MedvidovićComputer Science DepartmentViterbi School of Engineering

University of Southern CaliforniaLos Angeles, CA, USA

neno@usc.eduhttp://csse.usc.edu/~neno/

Part I

SoftwareArchitecturesRealandImagined

ABitofTerminology• Asoftwaresystem’sarchitectureisthesetofprincipaldesigndecisions aboutthesystem– Softwarearchitectureistheblueprintforasoftwaresystem’sconstructionandevolution

• Designdecisionsencompasseveryfacetofthesystemunderdevelopment– Structure– Behavior– Interaction– Non-functionalproperties– Deployment– …

TemporalAspectofArchitecture• Designdecisionsare madeandunmadeoverasystem’slifetime– Attimet asystemhasonlyonearchitecture

• Prescriptivearchitecture(PA) capturesdesigndecisionsmadepriortosystemconstruction– as-designed

• Descriptivearchitecture(DA) describeshowthesystemhasbeenbuilt– as-implemented

iRODS – DescriptiveArchitecture

HowManySystemsStartoffiRODS – PrescriptiveArchitecture …andEndup

WhatHappened?• Softwaredecay

– Drift– introductionofdesigndecisionsintoasystemthatarenotencompassedorimpliedbyitsarchitecturaldesign

– Erosion– introductionofdesigndecisionsintoasystemthatviolateitsarchitecturaldesign

• Decayingsystemsbeginto“smell”«Moreonthislater…

DecayinRealSystems?

Linux– PrescriptiveArchitecture

Linux– DescriptiveArchitecture

Top-LevelArchitecture– AnotherView

AnotherExampleHadoopDistributedFileSystem– PrescriptiveArchitecture

HDFS– DescriptiveArchitecture

Hadoop – HDFS+MapReduce

Hadoop – CompleteArchitecture

Hadoop – CompleteArchitecture,AnotherView

OneMoreExampleBashPrescriptiveArchitecture

BashDescriptiveArchitecture

Part II

Decay

CanWe“Smell”Decay?• Yes,bothinthedesignandcode• Softwaresmell

• Commonlymadedesignorimplementationdecision• Negativelyimpactsyoursystem’slifecycleproperties• Itisnotabug– itdoesn’tbreakyoursystem

• Ourgoalistodiscoverarchitecturaldesignsmellsautomatically

• Inspiredby• Refactoring:ImprovingtheDesignofExistingCode

byMartinFowler

ACatalogueofArchitecturalSmells• BrickConcernOverload• BrickUseOverload• BrickDependencyCycle• UnusedInterface• AmbiguousInterface• DuplicateComponentFunctionality• ScatteredFunctionality• ComponentEnvy• ConnectorEnvy• ConnectorChain• ExtraneousAdjacentConnector• …

ExamplesofSmellsfromRealSystems

ThisdocumentcontainsnotechnicaldatasubjecttotheEARortheITAR.

LinuxArchitecture

Linux– MemoryManagerSubsystem

BashArchitecture

Bash– JobControlComponent

Bash– CommandsComponent

Hadoop – CompleteArchitecture

Hadoop – DependencyCycle

Hadoop – ComponentUseOverload

Hadoop – BrickConcernOverload

ValueAggregator

Hadoop –* EnvyInterDataNode Protocol

Part III

Recovery(andthenrefactoring)

FableofTwoSystems

PrescriptiveArchitecture DescriptiveArchitecture

NewsFlash!

PrescriptiveArchitecture DescriptiveArchitecture

iRODS – DescriptiveArchitecture

HowManySystemsStartoffiRODS – PrescriptiveArchitecture …andEndup

SmellsImpactRealSystemsSmellyfilesaremoreissueprone

Smellyfilestendtobemorechangeprone

HowDoWeKnow?• Architecturerecovery

– Theprocessofdetermininga system’sarchitecturefromitsimplementation-levelartifacts

– Sourcecode,executablefiles,Java.classfiles,…• Difficultinpractice

– Sizeofcodebases– Irrelevantdetails– Misleadingdetails– Missinginformation– Hardtoobjectivelyassessexistingtechniques

• Still,automatedsolutionsareavailable

WhatAreTheseSolutionsYouSpeakof?• ACDC– AlgorithmforComprehension-DrivenClustering

– Structuralpattern-basedclustering

• ARC– ArchitectureRecoveryUsingConcerns– Concern-basedhierarchicalclusteringbasedonsimilaritymeasure

• Bunch-NAHC& Bunch-SAHC– Hill-climbingalgorithmformaximizingModularizationQuality

• LIMBO– scaLable InforMation BOttleneck– Probabilistichierarchicalclustering

• WCA-UE& WCA-UENM– Weigted CombinedAlgorithm– Dependency-basedhierarchicalclustering

• ZBR– Zone-BasedRecovery– Hierarchicalclusteringbasedontextualinformation

• PKG– ImplementationPackageStructure

DoTheyReallyWork?

BashfromACDC

BashfromBunch BashfromZBR

Bash“GroundTruth”

AMoreIn-DepthStudy• Eightarchitecturesofsixopen-sourcesystems• Previouslyobtainedground-truthsforeach

ArchStudio 4 IDE Java 280K 54comp.Bash 1.14.4 OSShell C 70K 25Hadoop 0.19.0 DataPrc Java 200K 68Linux-C 2.0.27 OS C 750K 7Linux-D 2.0.27 OS C 750K 120Mozilla-C 1.3 Browser C/C++ 4M 10Mozilla-D 1.3 Browser C/C++ 4M 233OODT 0.2 DataMgt Java 180K 217

ProximitytoGroundTruth

ClusterComparison

Part IV

UnderstandingArchitectureasa“BigData”problem

Software ArchitectureARCADEArchitectureRecovery,Change,andDecayEvaluator

SourceCode

IssueRepository

RecoveryTechniques IssueExtractor

IssuesArchitecturesArchitecturalSmellDetector

Architectural-SmellInstances

ChangeMetricsCalculator

DecayMetricsCalculator

ChangeMetrics

DecayMetrics

RelationAnalyzer

CorrelationData

EmpiricalStudyofChangeandDecay

1. Inwhatwaysdoarchitectureschange?

2. Whenandhowdoarchitecturesdecay?

3. Whatistherelationshipbetweenarchitecturalsmellsandimplementationissues?

42

SeveralSubjectSystems

43

System ApplicationDomain Versions Time MSLOCActiveMQ MessageBroker 20 8/04-12/05 3.4Cassandra DistributedDBMS 127 9/09-9/13 22.0Chukwa DataMonitoring 7 5/09-2/14 2.2Hadoop DataProcessing 63 4/06-8/13 30.0Ivy DependencyManager 20 12/07-2/14 0.4JackRabbit ContentRepository 97 8/04-2/14 34.0Jena SemanticWebFramework 7 6/12-9/13 2.7JSPWiki WikiEngine 54 10/07-3/14 1.2Log4j Logging 41 01/01-06/14 2.4Lucene SearchEngine 21 12/10-1/14 5.1Mina NetworkFramework 40 11/06-11/12 2.3PDFBox PDFLibrary 17 2/08-3/14 2.7Struts WebApps 36 3/00-2/14 6.7Xerces XMLLibrary 22 3/03-11/09 2.3

AFewBackgroundBits• VersioningScheme

– major.minor.patch release• Changemetrics

– MojoFM– a2a– c2c

• Decaymetrics– #structuraldependencies– Changeproneness– Couplingandcohesion– Smelldensityandcoverage

44

1.5.3

1.6.0

1.6.1

2.0.0

RecoveryTechniquesUsed

• PKG – package structure recovery

• ACDC* – algorithm forcomprehension-drivenclustering

• ARC** – architecture recovery usingconcerns

*V.Tzerpos etal.,ACDC:analgorithmforcomprehension-drivenclustering,InWorkingConferenceonReverseEngineering(WCRE),2000

**J.Garciaetal.,Enhancingarchitecturalrecoveryusingconcerns,InInternationalConferenceonAutomatedSoftwareEngineering(ASE),2011

HowArchitecturesChange

ValueunitispercentageLowernumbersmeanmorechange

0

10

20

30

40

50

60

70

80

90

100

Ivy Lucene JSPWiki Ivy Lucene JSPWiki Ivy Lucene JSPWiki Ivy Lucene JSPWiki

Major MinMajor Minor Patch

Averagea2avaluesbetweenversions

ACDC ARC PKG

ArchitectureSimilarity

“Reversed”architecturechanges

Onaverage,architecturechangesrangefrom15-25%

Changesdifferbetweendifferentviews

Major<MinMajor <Minor<Patch

System vs.ComponentLevel• Architecture changes occur within components even when the

system’s overall architectural structure remains relatively stable

Architecturalsimilaritybetweenminorversionsof“Ivy”

ARCview: architecturechangesmorethan80%withincomponents

• Dramaticarchitecturechangecanoccuracrossminorversions

0

10

20

30

40

50

60

70

80

90

100

Minimuma2avaluesbetweenminorversions

ACDC ARC PKG

RQ3 – WhenSignificantChangeOccurs

Architecturechanges> 50%

ArchitectureSimilarity

AtWhatPointDoesChangeBecomeDecay?

ApacheChukwa0.3.0 ApacheChukwa0.4.0

ArchitecturalDecay

50

0

20

40

60

80

100

120

colospfdc

Versionsv1 v123

Cassandra’sarchitecturesrecoveredusingARC

SmellsImpactRealSystemsSmellyfilesaremoreissueprone

Smellyfilestendtobemorechangeprone

Smellyfilesarecontinuouslyinvolvedinmanyissues

On-GoingWork• Identify,understand,andcataloguesmells• Identifypatternsindicatingdecay• Collectanddocumentground-truths• Improvearchitecturalrecovery

– Bettercodeanalysis– Dynamicsystemaspects

• Studycorrelation/causalityofarchitecturalchange&decaywith– implementationissues – codedecay– refactoring – self-adaptivebehavior

Acknowledgments• Supporters

– NSF– BoschRTC– NASA/JPL

• Projectparticipantsandcollaborators- Pooyan Behnamghader,USC- Yuanfang Cai,DrexelU.- EricDashofy,AerospaceCorp.- ChrisDouglas,Microsoft- EderFigueroa,USC- AlessandroGarcia,PUC-Rio- JoshuaGarcia,UCIrvine- Muzzammil Imam,USC- IgorIvkovic,U.ofWaterloo

- IvoKrka,Google- Duc Le,USC- DanielLink,USC- Isela Macia,PUC-Rio- ChrisMattmann,NASA/JPL- DanielPopescu,Google- ArmanShahbazian,USC- Yixue Zhao,USC

– Infosys– NorthropGrumman– Huawei