a prototype of a full-scale scada system installation ...1046339/fulltext01.pdf · a prototype of a...
TRANSCRIPT
INOM EXAMENSARBETE ELEKTROTEKNIK,AVANCERAD NIVÅ, 30 HP
, STOCKHOLM SVERIGE 2016
A prototype of a full-scale SCADA system installation using an operator training simulator module as power grid
IVAN HIDAJAT
KTHSKOLAN FÖR ELEKTRO- OCH SYSTEMTEKNIK
KTHROYALINSTITUTEOFTECHNOLOGYMasterofScienceinElectricalEngineering
Master’sDegreeThesis
Aprototypeofafull-scaleSCADAsysteminstallationusinganoperatortrainingsimulatormoduleaspowergrid
KTHSupervisor CandidateProf.MathiasEkstedt IvanHidajatABBSupervisorMagnusOlofssonGöranEkström
Stockholm,SwedenAugust24
SammanfattningModulenOperatorTrainingSimulator(OTS)syftartillatthjälpaelkraftsystemsoperatörerattförbättrasinakunskaperavelkraftsystemetsbeteende. Mångakraftföretaghar integreratOTS i sina SCADA-system för utbildningsändamål. Det samma gäller ABB som har enOTSintegreradisittNetworkManagerSCADA/EMS-system.Dock,vidanvändandetavOTSharNetworkManagerinteegenskapersåsomdetskullehafomdetopereradeettriktigtelkraftnät.DettaprojektsyftartillattlösadettaproblemochettalternativttillvägagångssättförOTSintegrationtillSCADAföreslagits.Arbetet inleddes med en undersökning av ABB:s SCADA / OTS med en analys av dessavvikelsertillettfaktisktSCADAstyrsystem.EftereninledandefassominvolveradetestningochstuderaSCADA/OTSarbetsflöde,fortsattearbetetmedattskapaalternativalösningarför att undanröja avvikelsernamellan de två systemen. Kärnan i projektet innefattade attkonstruera,implementeraochtestanyaalternativförintegrationavOTSochSCADAsystem.Denframtagnaprototypensfunktionalitet,nätverksarkitektur,kommunikationsprotokollochdataflödeundersöktesoch jämfördesmeddeegenskaperpåett SCADA-system för verkligdrift. Resultaten av utvärderingen visade att prototypen var en bra representation av ettverkligtSCADA-system.
i
Abstract
TheOperatorTrainingSimulator (OTS)aims tohelppowersystemoperators to improve theirproficiency of the power systembehavior in the current increasing complexity of the controlsystem.ManypowerutilitycompanieshaveintegratedtheOTSintotheirSCADAsystem(Walve&Edstrom,1998)(Demjen,Kadar,Meszaros,&Szendy,1994)fortrainingpurposes.ThereisnoexceptionforABB’s;theyintegratedtheOTSintotheirNetworkManagerSCADA/EMSsystem,anditisthemaintoolforthisproject.
However,theABB’sSCADA/OTSsuffersfromincompleterealSCADAproperties,e.g.,PCUandRTU, due to its sandbox nature. Consequently, it does not realistically represent the fullpropertiesandfunctionalitiesofanactualSCADAsystem.Toaddressthis issue,analternativeapproachforOTSintegrationtotheSCADAhasbeenproposed.
Thework startedwithanexplorationof the currentABB’s SCADA/OTSwithananalysisof itsdeviations to an actual SCADA control system.After a preliminary stage that involved testingandstudyingSCADA/OTS’sworkflow,theworkcontinuedwithcreatingalternativesolutionstoeliminatethedeviationsbetweenthetwosystems.Then,thecoreworkoftheprojectinvolvedimplementationandtestingthenewalternativeprototypesincethereisapossibilitythatKTHand/orABBwillusethesimulatorforafutureproject.
To finalize the work, the prototype’s functionality, network architecture, communicationprotocolsanddataflowwereexaminedandcomparedwiththeactualSCADAtoevaluatethequality of the design. The results of the evaluation showed that the prototype was a goodrepresentationofarealSCADAsystem.
ii
Acknowledgement
This thesis project has been carried out at ABB Software Enterprise department in Västerås,Sweden,forsixmonthsperiodfromMarch2016toAugust2016.Beforetheprojectstarted, Ispent one year studying at EindhovenUniversity of Technology and half a year at KTHRoyalInstitute of Technology as a Smart Electrical Network and System (SENSE) KIC InnoEnergystudent.
Firstofall,IwouldliketothankmyABB’sdirectsupervisorMagnusOlofssonandGöranEkströmfor giving me the possibility of working on an exciting project. Also, to Professor MathiasEkstedt,my supervisor from KTH for his guidance and feedback. I would like to thank ABB’ssoftwareengineers,especially,MikaelAndersson,MikaelEklundandMikaelStenviforbeingsosupportiveandhelpfulandtoallthepeoplefromtheoffice,whointegratedmeverywellandmademefeelwelcomedintheoffice.Specialthanksgotomyproject’scolleagueMohammadNaivasalZaheer,withouthim,wewouldnotreachthisfar.
AchievingthisMaster’sDegreewaspossibleduetothescholarshipprovidedbyKICInnoEnergy.HenceIwouldliketothankandgiveabighugtomyprogramcoordinatorHansEdinfromKTH,whohaveguidedandhelpedussincethebeginningoftheprogramandmadetheprogramtobeunforgettablememories.
Finally, Iwould like to thankmy familyand friends,whoalways supportedme from farawayduring my studies and time doing this thesis work, who always opened for questions anddiscussions. It is impossibletomentionall theenrichingand inspiringpeople Iencountered inthislastphaseofmymasterstudies,buttheyhaveastronginfluencethatmadethisthesisworkpossible.
Thankyou.
iii
Table of Contents
Chapter1:Introduction.................................................................................................................1
1.1 OTSIntegration................................................................................................................1
1.2 ProblemStatement..........................................................................................................3
1.3 ObjectiveoftheThesis.....................................................................................................4
1.4 AssumptionsandScope...................................................................................................4
1.5 Methodology....................................................................................................................5
1.5.1 Projectworkflow.......................................................................................................5
1.5.2 ReportStructure.......................................................................................................7
Chapter2:ExperimentToolsandComponents.............................................................................8
2.1 VMwarevCloud................................................................................................................8
2.2 ABB’sNetworkManager..................................................................................................8
2.2.1 Servers......................................................................................................................9
2.2.2 Avantidatabase......................................................................................................10
2.2.3 WS500.....................................................................................................................11
2.2.4 NetworkModel.......................................................................................................12
2.2.5 OTS..........................................................................................................................12
2.3 Protocols,Applications,andModules............................................................................14
2.3.1 RemoteServerProtocol(RSP).................................................................................14
2.3.2 IEC60870-5-104......................................................................................................14
2.3.3 RP570......................................................................................................................14
2.3.4 RemoteCommunicationServer(RCS).....................................................................14
2.3.5 PCU400....................................................................................................................15
2.3.6 ModuleM8.............................................................................................................15
2.3.7 ModuleM3.............................................................................................................15
2.3.8 ModuleM9.............................................................................................................15
Chapter3:DesignAlternativeImplementation...........................................................................16
3.1 OriginalSCADA/OTSDeviations.....................................................................................16
3.2 AlternativeDesignRequirements..................................................................................17
3.3 TheChosenDesignAlternative......................................................................................19
3.4 Dataflow........................................................................................................................21
iv
3.5 Modifications.................................................................................................................22
3.5.1 vCloudNetworkConfiguration...............................................................................22
3.5.2 M8andM3Configuration.......................................................................................23
3.5.3 AdditionalSourceCode...........................................................................................23
Chapter4:ValidatingandAnalyzingtheNOTS............................................................................24
4.1 DesignValidation............................................................................................................24
4.1.1 CommunicationNetworkConfigurationCheck......................................................24
4.1.2 DataFlowTest.........................................................................................................28
4.1.3 ProtocolCheck........................................................................................................31
4.1.4 FunctionalityTest....................................................................................................32
4.2 ScalabilityTest................................................................................................................33
4.3 Re-AssessingtheDesignRequirements.........................................................................34
4.4 NOTSDeviations.............................................................................................................36
4.5 Investigatingadditionalfunctionalities..........................................................................38
Chapter5:Conclusion,Improvements,andFutureWorks..........................................................39
5.1 Conclusion......................................................................................................................39
5.2 PossibleImprovements..................................................................................................40
5.3 FutureWorks..................................................................................................................40
References...................................................................................................................................41
AppendixA:Alternativedesignsolutions....................................................................................44
AppendixB:Dataflowtest..........................................................................................................45
v
Abbreviation
AGC AutomaticGenerationControl
API ApplicationProgramInterface
AQL AvantiQueryLanguage
BMS BusinessManagementSystem
CCM ControlCenterModel
DNP DistributedNetworkProtocol
EMS EnergyManagementSystem
EPRI ElectricPowerResearchInstitute
EXR ExecuteResponse
GOOSE GenericObjectOrientedSubstationEvent
HMI Human-MachineInterface
IEC InternationalElectrotechnicalCommission
IED IntelligentElectronicDevice
IOA InputOutputAddress
IP InternetProtocol
IVST IndicationValueSpecificwithTime
LAN LocalAreaNetwork
MMS ManufacturingMessageSpecification
MU MergingUnit
NIC NetworkInterfaceCard
NOTS NewOperatorTrainingSimulator
NXR NegativeExecuteResponse
OCST ObjectCommandSpecificwithTime
OPF OptimalPowerFlow
OTS OperatorTrainingSimulator
vi
PCU ProcessingControlUnit
PD ProductDevelopment
PSM PowerSystemModel
PSO ProfessionalServicesOrganization
QM QueueManager
QoS QualityofService
RCS RemoteCommunicationServer
RM RecordManipulation
RP570 RTUProtocol570
RSP RemoteServerProtocol
RTU RemoteTerminalUnit
SCADA SupervisoryControlandDataAcquisition
SE StateEstimation
SMV SampledMeasuredValues
SQL StructuredQueryLanguage
TCP TransmissionControlProtocol
XML ExtensibleMarkupLanguage
vii
ListofFigures
FIGURE1.1TYPICALSCADAARCHITECTUREBEFORE(LEFT)ANDAFTER(RIGHT)ANOTSINTEGRATION...............................................3FIGURE1.2PROJECTWORKFLOWDURINGTHEFIVEMONTHSWORKINGDURATION..........................................................................5FIGURE2.1ABB'SNETWORKMANAGEREXTENSIVEAPPLICATIONSLIST[13].................................................................................8FIGURE2.2NETWORKMANAGER'SSERVERSETUP....................................................................................................................9FIGURE2.3AVANTIINTERPROCESSCOMMUNICATIONFLOW.....................................................................................................10FIGURE2.4NETWORKMANAGER'SPOWERGRIDNETWORKMODEL...........................................................................................12FIGURE2.5OTSCOMMUNICATIONFLOWANDITSINTERNALPROCESSES.....................................................................................13FIGURE3.1THENOTSDATAFLOW.....................................................................................................................................20FIGURE3.2COMMUNICATIONNETWORKCONFIGURATIONFORTHENOTS..................................................................................22FIGURE4.2TEST1PRINTSCREEN,LEFTPCUTORTUVS.RIGHTADSERVERTORTU....................................................................26FIGURE4.3TEST2PRINTSCREEN,LEFTRTUTOPCUVS.RIGHTRTUTOOTS.............................................................................26FIGURE4.4TEST3PRINTSCREEN,LEFTOTSTORTUVS.RIGHTDESERVERTORTU.....................................................................27FIGURE4.5TEST4PRINTSCREEN,LEFTOTSTOSCADAVS.RIGHTRTUTOSCADA.....................................................................27FIGURE4.6SCADA'SPROCESSQUEUELOG...........................................................................................................................28FIGURE4.7MODULERCSLOG............................................................................................................................................28FIGURE4.8PCU'SRSPBUSLOG.........................................................................................................................................29FIGURE4.9RTU'SRSPBUSLOG.........................................................................................................................................29FIGURE4.10PRINTOUTOFRECEIVEDTELEGRAMFROMTHERTUONTHEOTSMACHINE................................................................29FIGURE4.11OTS'PROCESSQUEUELOG...............................................................................................................................29FIGURE4.12RTU'SRSPBUSLOG.......................................................................................................................................30FIGURE4.13PCU'SRSPBUSLOG.......................................................................................................................................30FIGURE4.14MODULERCSLOG..........................................................................................................................................30FIGURE4.15ACTIONONE,TRIPPINGACIRCUITBREAKER..........................................................................................................32FIGURE4.16ACTIONTWO,RAISINGAGENERATORSETPOINTVALUE..........................................................................................32
pg.1
Chapter 1: Introduction
1.1 OTS Integration
Inrecentyear,electricalpowersystemhasbecomemorecomplicatedduetothenetworkexpansionandinterconnectionbetweencountries.Consequently,powersystemoperatorshavetohandleanincreasingcomplexitycontrolsystem.Also,theyneedtorecognizetheimpactfromneighboringgrids.ThesewerethemainreasonbehindthedevelopmentoftheOTS.WithOTS,anoperatorabletosimulatecontrolactionsandmonitortheresponseandthebehavioroftheirpowersystem.
Since early 1990 there have been a lot of research and development on the OTS (Chu,Dobrowolski,Barr,McGeehan,Scheurer,&Nodehi,1991).Ithasbecomeanattractivetopicinthe electrical power system field due to its promising future outcomes to enable a morereliable,efficientandeconomicaloperation.SomeoftheearlyprominentprojectswereARISTOandEPRI(Walve&Edstrom,1998)(Wu,Moslehi,&Bose,2005).
TheARISTOprojectstartedattheRoyalInstituteofTechnology,Sweden,inthelate1980s.TheprojectsupportedbyVattenfallandSvenskaKraftnatandcompletedin1993(Walve&Edstrom,1998). ARISTO was later integrated into Svenska Kraftnat SCADA/EMS system, which hasimprovedofthesimulatorduetothepossibilitiesofloadingastartsolutionfromthereal-timeSCADAsystemandusingreplicasoftheSCADA’sHMIconsoles.ARISTOaccuracyandinteractivegraphicuser interfacemake it commonlyused inSwedishUniversityProject suchasKTHandChalmers.
AnotherOTS thatwas built and used in the industry is the Electric Power Research Institute(EPRI)simulator.TheEPRIsimulatorwasdevelopedinPaloAlto,theUnitedStatesasinquiriesfrom the electrical power industry. It has become a great tool for power system operationstudies and has been utilized and reported in various IEEE and PES papers (Koda, Bose, &Anderson,1998)(Prais&Bose,1987)(Zhang&Bose,1989)(Bose&Chen,1987).Similarly,theprogramprovidesafast,accuratepowersystemsimulation, includingfasttopologyprocessor,networksolutionwithdynamicenergysourcemodels, five-secondupdatetime, islanding,andblackstartcapability.ItisalsocustomizedtohaveanexactduplicateofthecontrolcenterHMIandcontrols.
The previous examples above illustrate the transition of an OTS system from a standaloneapplication into subpart of a proprietary SCADA system. Similarly, ABB also has the OTSintegrated into theirNetworkManager SCADA system,and it is theexperiment toolused forthisproject.
pg.2
pg.3
1.2 Problem Statement
Toprovideaneffectiveandaccuratetrainingplatformfortheoperators,theOTSshallsatisfyaset of requirements. An essential requirement is, “The simulator have to be realisticallyrepresentingthefullpropertiesandfunctionalitiesofpowersystem’sSCADA.”(DyLiacco,etal.,1983).However,thecurrentOTSintegrationapproachdoesnothavefullpropertiesofaSCADA.
AtypicalSCADAarchitecturepre-andpost-OTSintegration(Sindelar&Novak,2012)(Hua,Zhou,&Yu,2004)areillustratedinFigureFel!Formatmallenärintedefinierad..1.InanactualSCADA,operatorcommandsaresenttothepowersystemthroughthePCUandRTU.Then,thepowersystemreturnsnewmeasurementstotheSCADAHMIthroughthesamepath.ThehistoriandatabasestoresallinformationthatpassesthroughtheSCADAbuswhichincludescommands,indications,andmeasurements.Thecontrolapplicationcalculatesadvancedcontrol
functionssuchasSE,OPF,andAGC,anddisplaystheresultsontheHMI.
FigureFel!Formatmallenärintedefinierad..1TypicalSCADAarchitecturebefore(left)andafter(right)anOTSintegration
Inpost-OTSintegration,SCADAredirectsoperatorcommandstotheOTS.TheOTSissimulatinga power systembehavior (DyLiacco, et al., 1983) (Walve& Edstrom, 1998) then returns newmeasurementstotheSCADA.ThefigureaboveshowsthatthePCUandtheRTUareremovedaftertheOTSintegration.ItmeansanyincidentsoroccurrencesthathappenedwithinthePCU
pg.4
and the RTU are being overlooked, and it might affect the simulation accuracy. Moreimportantly,itviolatestherequirementofthesimulatorinrepresentingthefullpropertiesandfunctionalitiesof thepowersystem’sSCADA. Inthefollowingdiscussions,SCADAwithanOTSintegrationwillbecalledtheSCADA/OTS.
1.3 Objective of the Thesis
ThemainobjectivetheprojectistoimplementanotherdesignalternativeforSCADA/OTSthateliminatesthedeviationsofthecurrentdesigntoanactualSCADAtoproducemoreaccurateandrealisticsimulationresults.
Todothis,itisnecessarytostudythecurrentABB’sSCADA/OTSsystemthoroughly.Althoughthemaindeviationhasbeenmentionedintheproblemstatementsection,thesmalleronesarealsobeinginvestigatedandhandledduringthework.Thedeviationsareexaminedfromfourdimensions,functionality,networkarchitecture,communicationprotocolsanddataflow.Everydimensionisinvestigatedbeforecomingwithnewdesignrequirements.Finally,basedontherequirements,anewSCADA/OTSdesignsolutionisimplemented.
1.4 Assumptions and Scope
Assumptionsandprojectscopeneedtobedeterminedtoproceedwiththeproject.Theywerecreatedtonarrowdownthecomponentsthatneedtobestudiedandinvestigated.Moreover,timeandresourceslimitationwerealsothemainconsiderationtocompletetheprojectwithinthetimeconstraint.Thoseassumptionsandprojectscopeare:
• TheOTScansimulatepowersystembehaviorrealistically.
• LayersaboveSCADA/EMSLAN,e.g.,businesslayer,isoutofscope.TheOTSwasmainlydesignedforpowersystemoperator,notforentrepreneursormarketplayers.
• CommunicationprotocolsbelowaRTUlayerarenotcoveredbecauseitisnotdevelopedbyABB’ssoftwareenterprisedepartment.
pg.5
1.5 Methodology
1.5.1 Project workflow
TheprojecthasbeencarriedoutforfivemonthsinABB’sofficeinVästerås(Sweden).Theworkhas been closely followed and supported by two departments within ABB, the ProductDevelopment(PD)andProfessionalServiceOrganization(PSO).Meetingwasdoneweeklywiththe thesis supervisor from University and the representative of each department for workprogressreviewandupcomingweekplans.
Figure1.2Projectworkflowduringthefivemonthsworkingduration
The technical approach used to perform the thesiswork can be illustrated chronologically inFigure1.2.Thefirststage(PhaseI)wasmainlysettingupandlearningdifferentprocessesandcomponents in the systems. It was last for five weeks and could be defined to sub-steps asfollows:
• Created, set up and configured the necessary servers and directories in the virtualenvironmentfortheexperimentandtestingplatform.Alsoinstalledtheupdatedversionof NetworkManager to the system, incorporate OTS and checked if the functionalityworkedasitshould.
• Studiedandtestedthenetworkmanagerthoroughly.Theseincludegettingfamiliarwiththe server connection,data flow,human interfaces,Avanti database, debugging tools,andsourcecodes.
• Madesimplemodificationstothesourcecodesforexpectedoutcomes,suchasaddingandactivatingqueuesandprocesses,manipulatingthedatabaseandcreatinglogfiles.
pg.6
Thesepreliminarystepsallowedathoroughunderstandingofhowtooperateanddevelopthenetworkmanager,SCADA,andOTS.
Nextwasthedevelopmentandimplementationphase(PhaseII).Thesecondphaselastedforafulltenweeks,anditconsistedof:
• Brainstormed design alternatives and alteration ideas for the OTS. The considerationbasedondataflowrealism,codingdifficulty,resourceavailabilityandABB’sregulation.
Onceatentativedesign ideawasdeterminedandapprovedbythesupervisors, itwasbrokendownintosmallerparts.Everypartinvolvingmultipleiterations,whichconsistedof:
• CommunicatedtheideatoABB’ssoftwareengineersanddiscussedtheonthefeasibilityandpossiblewaytomakethemodification.
• Wrotecode(inFORTRANandClanguage)toapplythechanges.
• Testedanddebuggedthesystemextensivelyafterthe integrationofnewsourcecode,investigated forunpredictedbehaviorsandcasesaffected thatmightoccurdue to themodification.
Thedevelopmentphasehasbeendesignedwithagoodlevelofmodularity.Majormodificationswere done in separated source code files thatwere built on the top of the originalNetworkManager Source code set, soonlyminor changesneed tobemadewhenever a system resetwasrequired.
The accomplishment of design was followed by a revision and discussion meeting with allsupervisorsforpossibleimprovements.Theseimprovementswouldgotothesameiterationasaboveuntiltheproductwasfinalized.
Thefinalphase(PhaseIII)fortheworkconsistedofmultipletesting,evaluation,documentationandfinalreportwriting.Amoredetailedelaborationasfollows:
• Validation check for all protocols and data traffic that was implemented during thedevelopmentphase,ensuringtheyoperatedaccordingly.
• Changesweremadetoimplementthescalabilitytest.
• Scalability check by boosting the traffic tomonitorwhen the system starts losing anddroppingpackages,alsowhatarethebottlenecks.
• Experimental evaluation of the overall product, documentation of the process andwrappingupthefinalreport.
pg.7
1.5.2 Report Structure
The report is subdivided into three parts, apart from the introductory chapter. The first part(chapter2)isanexplanationoftheexperimenttools.Thesecondpart(chapter3)isabouttheworkonimplementingandconstructingthedesignalternative.Lastly,thethirdpart(chapter4and5)iscoveringtesting,validation,evaluationandconclusion.
Chapter 2 starts with an explanation of the VMware vCloud and ABB’s Network ManagerfollowedbytheprocedureofactivatingtheOTS.Additionally,applicationsandmodulesthatareusedfortheprojectaredescribedinthischapter.
Chapter3coverstheworkonimplementingthechosendesign.ItbeginswithlistingtheoriginalSCADA/OTSdeviations toanactual SCADAandcreating thedesign requirements toeliminatethose deviations. Then followed by a detailed explanation on the network architecture,communicationprotocolsanddataflowofthenewdesign.Sourcecodestosupportthedesignisalsodescribedattheendofthechapter.
Chapter4containsthecorepartofthethesiswork.ItstartswithtestingtothenewOTSdesign,then analysis of the new design based on the predefined requirements followed with asummaryofthewholetestingandanalysisprocess.
Chapter5isthefinalpartthework,withaconclusionandpossiblefuturework.
pg.8
Chapter 2: Experiment Tools and
Components
2.1 VMware vCloud
ABBhaveintegratedtheVMwarevCloudtosimplifyit’sIToperationsinbuildingandoperatingasoftware-defineddatacenterarchitectureinsidetheirprivatecloud.ThevCloudprovidesvirtualizedinfrastructureservicessuchasnetwork,security,andavailabilitytoautomatetheconfigurationandcontrolofapplicationsbasedonthedefinedpolicies(VMware,2014).
ThethesisworkwasimplementedinsideABB’svirtualcloudsystemandutilizedthevCloudfunctionalities,whichmeansallmachinesinthisprojectarevirtual.Itenablesquickoperationsinconfiguringthenetwork,switchingnetworkinterfaces,addingandremovingservers.Moreimportantly,itreducestheuseofphysicalresourcessignificantly.
2.2 ABB’s Network Manager
TheNetworkManagerisABB’sone-stopcontrolcentersolutionforpowersystemoperation.Itoffers not only a primary SCADA system but many other integrated functionally, whichillustratedinFigure2.1.
Figure2.1ABB'sNetworkManagerextensiveapplicationslist(ABB,2015)
pg.9
Theseapplicationsarebuiltonanopenplatform inanopenarchitecture thatenablesahighlevelofflexibilityforsystemconfiguration(ABB,2015).ThepurposeofhavingEMS,GMS,andDMScalculationistoenhanceoperatorawarenesstohelptomakethedecision.
This project was implemented using NetworkManager SCADA/EMS application with an OTSintegration.EMSandDMSprovideaverysimilarfunctionalitysuchasstateestimation,reactivepower reserve, voltage stability analysis, network sensitivity, etc.However, theyoperate at adifferentvoltagelevel;EMSisfortransmissionsystem(highvoltage),andDMSisfordistributionsystem(medium-lowvoltage).
ThefollowingsubsectiondescribestheNetworkManager’scomponentsthatarerelatedtothework.
2.2.1 Servers TheNetworkManagerrequiresfourserverstooperate,anactivedirectoryserver,developmentenvironmentserverandtwoapplicationserversasillustratedin
Figure2.2.Allfourserverneedtobeconnectedtothesamenetwork.Eachservercanberepresentedasaphysicalcomputer,however,sincetheprojectisrunninginavirtualcloudenvironment,theserverswillbevirtualmachines.
Figure2.2NetworkManager'sserversetup
• ActiveDirectoryServer:WindowsServer2012R2StandardActiveDirectoryoperatesasthecentralauthoritytoensurenetworksecurity.Usershall
pg.10
beregisteredintheActiveDirectoryandgonethroughauthenticationprotectionmechanisms(usernameandpassword)beforeabletoaccesstheotherservers.
• DataEngineeringServer:WindowsServer2012R2StandardDataEngineeringServerisaplatformtorunWS500(HMI)softwareandPCU400application.AbriefdescriptionofWS500andPCU400willcomelaterinthischapter.
• ApplicationServerIandII(standby/backup):LinuxRedHatEnterpriseApplicationserversareaplatformfortheNetworkManagerandotherapplicationsoftware.LinuxRedHatEnterprisewasthechosenOSduetothequalitycomputingenvironmentitpremierenterpriseoffers.ItcansupportlargeSymmetricMultiprocessingsystemsandproventohaveahighdegreeofreliability,whichisanessentialrequirementfortheapplicationserver
TheapplicationserverrequiresahighlevelofreliabilitybecauseitstoresSCADA’sdata(historian),performstheEMScalculationsandrunstheOTSsimulation.Toimprovethereliability,itusesaredundantconfiguration.ApplicationServerII,theredundantstandbyserver,isalwaysreadytotakeoverwhentheprimaryserverfails.
2.2.2 Avanti database
TheAvantidatabaseistheintegralcomponentofNetworkManagerArchitecturethatservesasthe real-time repository for all PSM and CCM data. It utilized Oracle database technologies(ABB,1990)toperformthecentraldatarepositorytask.ThedatabasecanbeseenascabinetstofillalltheoperationinformationoftheNetworkManager,anditcanbeaccessiblefromeveryprogramsorserverthatauthenticatedtousethedatabase.
The inter-processdata communicationprovidedbyAvanti is done in the following fashionasillustrated in Figure 2.3. A sender process sendsmessages to the receiver process through aqueue.Aprocesscanhavemultiplequeuesattachedandusuallyeachqueueservesaspecifictask(e.g.,RTUCommand,updateHMI,etc.).
Figure2.3Avantiinterprocesscommunicationflow
pg.11
The database is designed to support redundant server configurations, specifically for theapplication servers. All specified updates on the primary on-line application servers aretransferredtothesecondarystandbyapplicationserver.
AvantiQueryLanguage(AQL)TheAQLisalanguagethatisusedtotalkandmanipulatetheAvantidatabase.AQLcommandscanbereadeitherfromaterminalortextfile.TheAQLismodeledafterthestandarddatabaselanguageSQL.Thus,itiseasytofollowforpeoplewhoarealreadyfamiliarwithSQL.
TheseareutilitiesontheAvantidatabasethatareusedduringtheproject:
• QueueManager(QM):QMisautilitytocheckstate,kill,activate,andattachqueuesorprocesses.
• RecordManipulation(RM):RMisautilitytopopulatetheAvantidatabase,duringtheprojectitwasmostlyusedtosendanartificialmessagetoaqueueorprocess.
• AQL:AQLisautilitytoreadandmodifythecontentoftheAvantidatabase.Thisutilityeasesustonavigatethroughthedatabase.
2.2.3 WS500
The Operator Workstation WS500 is a Human Machine Interface (HMI) for the NetworkManager.Itisauser-friendlyinterfacethatshowsdataandprocessstatustotheoperatorsandenablesoperatorstodoanin-depthcontroltothedataacquisitionandpowersystem.
pg.12
2.2.4 Network Model
The Network Manager provides a power grid network model for the OTS to simulate. ThisnetworkmodelispresentedinFigure2.4.
Figure2.4NetworkManager'spowergridnetworkmodel
The network model above is accessible through WS500. The network consists of fourteensubstations and sixteen RTUs because some substations have two RTUs, which are Amherst(AMHE) and Troy. Proceeding, three substations will frequently bementioned, and they areAmherst,Troy,andBowman(BOWM).
2.2.5 OTS
OTSisasubfunctionoftheNetworkManagerthatenablesanenvironmentforthetrainingofpower system operation. It provides a realistic representation of the static and dynamicresponsesofthepowersystemmodeldisplayedinFigure2.4.Itcomposesofeventprocessor,power flow, network equipment dynamics, load dynamics, unit dynamics, system frequencydynamics,andexternalAGCmodulesthatareperiodicallyexecutedevery5seconds.
Theactivationprocessandcommunication flowshiftafterOTSactivation iselaborated in thefollowing;
pg.13
OTSActivation
OncetheserversandtheNetworkManagerwereinstalled,nextstepwastoactivatetheOTSbyrunning build_OTS.exe, which can be found on the Application Server. After the OTS isactivated,allcommandsareredirectedtotheOTS,insteadofthePCU.
Finally,aninitialstatesolutionshallbeaddedtotheSCADA/OTS.AstatedumpcanbeobtainedfromtheSCADAproduction server.The simulatoruses the state solutionas itsbaseor initialstate. The operator can always load the file to reset the simulator back to the base state.
OTS’sInternalProcesses
TheprocessesinsidetheOTScanbesimplifiedtothreesubfunctions,p1,pots,andp2.Thepurposeofeachprocesswillbeelaboratedfurtherinthissubsection.
Figure2.5OTScommunicationflowanditsinternalprocesses
The process p1 ismodeled to simulate processes that happened between SCADA’s PCU andRTU.Anoperator’scommandsoriginatedfromWS500aretranslatedbyp1fortheOTSsystemtorecognize.Next,p1forwardsthetranslatedcommandtopots.Commandwillbeexecutedasinarealpowersystem,thennewindications,measurements,andbetransmittedtoprocessp2.Processp2willcomparethenewvalueswiththelastreceivedvalues.AsinanactualRTU,thep2 will be driven by the new value that deviates above the given threshold, and it will beforwarded to update the Avanti database through Data Update queue. Finally, the updatedvalueswillbedisplayedontheWS500inthenext(5-second)refreshcycle.
pg.14
2.3 Protocols, Applications, and Modules
2.3.1 Remote Server Protocol (RSP)
TheRSPwasdesignedmostlytosupportRTU-andmasterstation-protocolindependence.WhendatafromtheRTUsreachesthePCU,itisconvertedtoRSPformatandforwardedtothemasterstation. The RSP enables themaster station to process RTU’s data even if they use differenttypes of RTU’s protocols due to RSP protocol converting and data point cross-referencecapability.Inthisproject,theRSPisusedforcommunicationbetweenthePCUandtheNetworkManagerprocesses.
2.3.2 IEC 60870-5-104
In 2000, IEC 60870-5-104 (IEC 104) was released by The International ElectrotechnicalCommission (IEC) as one of the standard communication protocols for sending telecontrolmessages between the central control station and substation (IPCOMM). Unlike any otherprevious standards, IEC 104 utilizes standard TCP/IP protocol for its data transmission. ThebiggestadvantageofusingTCPlinkisthatitenablescommunicationviaastandardnetworkandsimplifiedmanagementofconnecteddevices.
2.3.3 RP570
TheRP570 isaprotocol that isusedbetween theSCADA frontend (PCU)and thecontrolledsubstation(RTU).RP570isanabbreviationofRTUProtocolfromIEC57version0(ABB,1997).TheRP570protocolwasastandardprotocolpopularinearly2000.Recently,itwasreplacedbyIEC 68070-5-101/104. Similar to IEC 68070-5-101/104,ABB created amodule to simulate thecontrolstation(master),H13,andtosimulatethecontrolledstation(slave),H20.
2.3.4 Remote Communication Server (RCS)
The Remote Communication Server is a subsystem of the Application Server that handlescommunicationbetweentheApplicationServertoRTU,throughPCU400.Inanotherword,theRCS acts as a gateway for the Application Server to PCU400. Incoming and outgoingcommunication traffic of theApplication Server andNetworkManager goes through theRCSandfollowsRSP.
pg.15
2.3.5 PCU400
The PCU400 act as a SCADA front end for the Network Manager. PCU400 main task is tocommunicatewithsubstations(RTU),morethanoneRTUcanbeconnectedtoasinglePCU400.Additionally,theseRTUcanusedifferentprotocolstotalkwithPCU400.
PCU400alsocommunicateswithoneormoreMasterStation (SCADASystem).Thedata fromtheRTUscanbespreadtomultipleMasterStationbyPCU400.
2.3.6 Module M8
The module M8 acts as a converter between IEC 60870-5-104 and the ABB internal RTU,independentprotocolRSPandisimplementedusingtheXLDtoolkitforthePCU400system.
BeforethePCU400cancommunicatewiththeRTUs,configuration,andengineeringdatahastobedefined.
When the PCU400 starts, itwill start all protocol drivers defined. The protocol driverwill, inturn, load its configuration files and request engineering data. The engineering data will beloadedeitherfromtheapplicationserverorthedatabaseloaderinthePCU400,dependingonthesetup.
2.3.7 Module M3
ThemoduleM3actsasanIEC870-5-104controlledstationorRTUslave.DatareceivedfromthecontrollingstationispassedthroughtheRSPbusandissenttothecontrolledstation.Thereisaninternaldatabasethatstorespointconfigurationandvalues.
2.3.8 Module M9
ThemoduleM9isanRTUmastersimulator.TheM9isaRSPsimulationmodulethathelpstosimulatetheRSPdatawithoutconfiguringtherealmasterprotocolsandconnectingRTUs.TheM9istypicallyusedasareplacementfortheactualprotocolforthecommunicationlinesinthePCUconfiguration,whichveryusefulfortestingpurpose.
Despiteabovefunctionalities,moduleM9wasutilizedforadifferentpurposeinthisproject.ThemoduleM9whichmountedontheRTUmachinehasawebinterfacethatisconnectedtoport.Hence,itprovideda“door”forexternalapplicationtosendtelegramstotheportandentertheinternalRSPbus.ThisarrangementwasduetothelimitationofmoduleM3whichcanonlytalktoM8andinternalRSPbus.
pg.16
Chapter 3: Design Alternative
Implementation
3.1 Original SCADA/OTS Deviations
Beforecreatingthedesignrequirement,thedeviationsbetweentheoriginalSCADA/OTSandtheactualSCADAwereinvestigated.Althoughtheyhavebeenbrieflymentionedintheproblemstatement,however,moredetailedandspecificdifferenceswillbedescribedinthissubsection.Thesedeviationswerethebaseforconstructingthealternativedesignrequirements.FollowingisthelistoftheSCADA/OTSdifferencestoarealSCADA;
Functionality
• TheOTSmodulesmentionin2.2.5andtheWS500refreshevery5seconds,however,inanactualSCADA,theyoperatecontinuously(Wu,Moslehi,&Bose,2005).
• TheOTSmodulesforsimulatingthepowersystemresponsesrunperiodicallyevery5seconds,unlikeintheactualpowersystemwhichoperatescontinuously.
• TheOTSsimulatesonlytheslowdynamicsresponsesofthepowersystem,becausefastdynamicssuchasharmonics,synchronousresonance,andinter-machineoscillationsarenotobservedandcontrolledbyon-lineoperators.
NetworkArchitecture
• ThePCU,theRTU,andtheircommunicationnetworkdonotexist.
• TheSCADA/EMSandtheOTSarelocatedinthesameareanetwork.
CommunicationProtocols
• TheOTSprocessesdonotsimulateanactualPCUtoRTUprotocol.However,therearefunctionalsimilaritiesbetweenthem,forexample,p1translatesthecommandforpotsandp2forwardsonlywhenthereisasignificantvaluechangewhichalsohappenedintherealRTUwhereittranslatesthetelegrambeforesendsittothepowersystemandreactswhenthereisavaluechange(event-driven).However,theOTSprocessesuseaproprietaryprotocolthatspecifiesonitssourcecode,anditisnotastandardPCUto
pg.17
RTUprotocol.
Dataflow
• ThereisnoactualtrafficgoingtothePCUandtheRTUbecausetheydonotexist.
• AstandardcommunicationprotocolforPCUtoRTU,e.g.,IEC104orRP,isnotused.
3.2 Alternative Design Requirements
A variety of requirements has been taken into account when deciding the possible designalternatives.TheserequirementswerestronglyinfluencedbytheoriginalSCADA/OTSdeviationsmentionedintheprevioussubsection.
Allrequirementswillbere-assessedinsection4.3.Therequirementsarespecifiedasfollows;
Functionality
• ThefunctionsofferedbytheoriginalSCADA/OTSmustnotberemovedordisabledduetotheassumptionmentionin1.4.
• TheWS500shalloperateinreal-time.
• Thefunctionsshallgiveaconsistentresult.
NetworkArchitecture
• IncorporatePCU’sandRTU’scommunicationnetworks.
• ThecommunicationnetworkinfrastructureshouldmatchanactualSCADAsystem,e.g.PCU,RTU,andOTSlocatedintheseparatenetwork.
• TheAdditionalnetworksshouldhaveenoughcapacitytoaccommodatecommunicationtrafficfromSCADAtotheOTS,viceversa.
• Everycommunicationnetworkshouldonlyhavetheintendeddatatraffic,forexample,PCUtoRTUnetworkshouldnotcontaindatatrafficfromtheSCADAtoPCUortheRTUtoOTSnetwork.
Protocols
pg.18
• Thedesignshouldusetoday’sstandardcommunicationprotocolforPCUtoRTUcommunicationchannel.Accordingto(Wu,Moslehi,&Bose,2005)(Tomsovic,Bakken,Venkatasubramanian,&Bose,2005)(Marihart,2001)andsupportedbyABB’sengineers,IEC60870-5-104orIEC104istheprotocol.
• TheIEC104telegramframeformatmustfollowthestandardconfigurationfoundintheinternationalIEC60870-5-104documentation(InternationalElectrotechnicalCommission,2016).
• KeeptheproprietaryNetworkManagerSCADAprotocolsforSCADA’sinter-processesandinter-modulescommunication
Dataflow
• ThedatashouldflowinthesamepathasanactualSCADAsystemthroughtheintendedcommunicationnetwork.
• Thedesignshouldsendthedatareliably.Noduplicatepackets,consistentpayloads,andreachestherecipientcorrectly.
• Thenewdataflowshouldnotaffectthetransmissiontimetotheextentofhavinganimpactontheperformanceofthesimulationresult.
pg.19
3.3 The Chosen Design Alternative
Multiple design alternatives have been considered and well thought before choosing thepreferable one. It is presumably themost alignedwith the requirements set in 3.2. Anotherdesignalternativethatconsideredcanbeseenin
pg.20
AppendixA.
Figure 3.1 illustrates the OTS design alternative’s communication flow and networkconfiguration. This design will be calledNOTS. All the terms found in Figure 3.1 have beendiscussedandcanbereferredinchapter2.
TheNOTSwas using twoNetworkManager Instances and a standardwindowsmachine. TheuppercloudrepresentedoneNetworkManagerSCADA/EMSInstance,whichdoesnothaveOTSinstalled.ThismachinewillbecalledtheCleanSystem.ThelowercloudwasanotherNetworkManagerSCADA/EMSinstancewithOTSinstalled.TheRTUwasmountedonanormalwindowsmachine.
Figure3.1TheNOTSdataflow
pg.21
3.4 Data flow
The data flow is described in chronological order starting from operator’s input in the CleanSystem’sWS500untilWS500receivedanddisplayedthenewmeasurements.
1. The operator gives command on Clean System’sWS500 (trip or close circuit breaker,raiseorlowertapchangerorsetgeneratorvalue)
2. ThecommandisforwardedtoCleanSystem’sSCADA.
3. TheSCADAconverts thecommand intoRSPtelegramandsends it toPCUthroughtheRCSmodule.
4. ModuleM8inthePCUreceivesthetelegram,convertsittoIEC104telegramanddeliverittotheRTU.
5. Module M3 in the RTU convert the telegram back to RSP format and forward it tomodule M9. Additionally, there is an additional program that listens to the RSP buscommunication traffic. It will grab the command telegram and send it to the OTSthroughaTCPconnection.
6. ModuleM9returnsanEXRorNXRacknowledgmenttotheCleanSystem’sSCADAusingthepathmentionedinstep3to5.
7. On the other end, theOTS receives the telegram sends from the additional program,thenitrecalculatesnewloadflowandsendstheoutputtoitslocalWS500andmoduleM9intheRTUthroughaTCPconnection.
8. ModuleM9returnsthetelegramwithnewmeasurementstotheCleanSystem’sSCADAthroughthePCU.
9. Finally,theupdatedmeasurementsarestoredintheCleanSystem’sSCADAAvantidatabaseandbedisplayedontheWS500.
ThedataflowaboveisfullyalignedwiththeactualSCADA’sdataflow(Yang&Barria,2011)andABB’sreferencesystem.However,ithastogothroughacoupleofteststovalidatethatthedatafollowstheintendedflow.Thesetestswillbeelaboratedlaterinsection4.1.
pg.22
3.5 Modifications
The following sectionexplains the configurations and changesmade to implement theNOTS.
3.5.1 vCloud Network Configuration
ThedesignsolutionincorporatestwoNetworkManagerinstancesandawindowmachinethatemulatesanRTU.ThesemachinesareconnectedinawaytoenablecommunicationflowexplainedinFigure3.1andsection3.4.Fortunately,havingavirtualsetupinthevCloud
environmentsimplifytheeffortincreatingthedesirednetworkarchitecture.
Figure3.2CommunicationnetworkconfigurationfortheNOTS
Thenetworkconfigurationwasarrangedas inFigure3.2tosupporttheNOTScommunicationflow. It consisted of five private networks, SCADA, RTU, OTS, SCADA-RTU, and RTU-OTSnetworks.The intermediatenetworkswerecreatedbecausevClouddoesnotsupportadirectmachine tomachineconnectionwhentheyare locatedonadifferentnetwork.Themachinesareconnected toanetwork throughtheNIC, forexample, theRTUandOTSmachinecontainthreeNICsthatareconnectedtothreenetworks.
Ideally, the OTS can only communicatewith the SCADA through the RTU. However, a directconnection was established for the OTS to the SCADA through the SCADA-OTS network toaddressthedatatrafficforthenon-simulatedRTUstosimplifythetestingprocess.Additionally,theroutingtableofeachmachineneedstobeadjustedaccordingly.
pg.23
3.5.2 M8 and M3 Configuration
In module M8, the IP address of the RTU slave and the communication port have to bespecified. Similarly, the RTU master’s IP address and the communication port have to beconfigured inmoduleM3. Additionally, the RTU requires XML files tomap each point in thepowersystemproperly.ThereisamasterexcelfiletogeneratetheseXMLfiles.ThenecessaryinformationtoloadthemasterfilecanbefoundinAvantidatabase.
3.5.3 Additional Source Code
rtuSendLog(Powerscript)TheavailableABB’smodulesdonothavethecapabilitytosendouttelegramsfromtheRTUtoanexternalmachine,asaworkaround,rtuSendLogwascreatedtologmessagesreceivedintheRTU’sRSPbusandforwardallthecommandmessagestoOTSoverTCP/IPlink.
rtulisten(Clanguage)rtulistenisanextensionprogramforrtuSendLog.ItreceivestelegramssentbyrtuSendLogandforwardsittoprocessp1.
p2redir(Clanguage)AnenormousamountoftelegramscontainingupdatedindicationsandmeasurementsarereleasedbytheOTSafteritfinishedrecalculatingaloadflow.However,withsuchheavytraffic,itisimpossibletotrackandanalyzethecommunicationnetworkfromtheOTSbacktoSCADA.Asaresult,p2redirwascreatedtoselectwhichRTU(s)trafficgoesthroughtheRTU,whilethenon-selectedRTUstrafficissentdirectlytoSCADAthroughaTCPconnection.
sendtoM9(Clanguage)ThisfunctionsendsthesimulatedRTUsfromp2redirtomoduleM9intheRTUmachine.ModuleM9isequippedwithawebinterface.Hence,sendtoM9willsimplysendthepacketstotheportnumberofthewebinterfaceandIPaddressoftheRTUmachinewithTCPchannel.ThemessageisconstructedsuchasmoduleM9understandsandabletoreturnthepacketstoSCADAthroughM3andM8.
serversync(Clanguage)serversyncisanextensionforp2redir.Itreceivestheupdatedindicationsandmeasurandsofthenon-simulatedRTUssentbyp2redirandforwardsthemtoSCADA’sprocess.Additionally,ItperformsanimportanttasktokeepbothOTS’s,andCleanSystem’sdatabasesynchronized.
pg.24
Chapter 4: Validating and Analyzing the
NOTS
4.1 Design Validation
Avalidationcheckwascarriedouttoensurethedesignhavebeenimplementedcorrectly.Thenetworkconfiguration,dataflowandprotocolsofthedesignprototypeareexaminedtocheckiftheyhavemetthepredefinedrequirements.
4.1.1 Communication Network Configuration Check
ThecommunicationnetworkinFigure4.1wasanextensionfromtheconfigurationinFigure3.2.TheIPaddressesweresetuptoenableTCP/IPcommunicationbetweenmachines.TheinternalIPaddressesoneachnetworkwereidentical(xx.xx.109.0/24)becausethesameNetworkManagercopywasused,butitwasnotanissuesincetheaddresseswasNAT-edwhentheywentoutoftheirlocalnetwork.Additionally,changinginternalIPaddresseswascausingauthenticationproblem.
Figure4.1CommunicationnetworkconfigurationfortheNOTSwithIPaddresses
pg.25
Thenetworkconfigurationwastestedtoverifythatalltheconnectionswereinplaceandoperatingproperly.Moreimportantly,wastoconfirmthatRTUcouldonlyreceivetrafficfromthePCUandtheOTS.
pg.26
Test1:FromtheSCADALANonlythePCUabletocommunicatetotheRTUthroughSCADA-RTUnetwork.ThetestwastopingtheRTUfromthePCUandtheADServer.
FigureFel!Formatmallenärintedefinierad..2Test1printscreen,leftPCUtoRTUvs.rightADservertoRTU
Test1results:AsexpectedthePCUwasabletotalktotheRTU,butADserveroftheCleanSystemfailed.
Test2:TheRTUshallcommunicatewiththePCUthroughtheSCADA-RTUnetwork,alsotheOTSthroughtheRTU-OTSnetwork.ThetestwastopingthePCUandtheOTSfromtheRTU.
Figure4.3Test2printscreen,leftRTUtoPCUvs.rightRTUtoOTS
Test2results:Bothpingswentthrough,meaningPCU-RTUandRTU-OTSconnectionswereoperatingproperly.
pg.27
Test3:OntheOTSLANonlytheOTScantalkbacktotheRTUthroughtheRTU-OTSnetwork.ThetestwastopingtheRTUfromtheOTSandDEserverfromtheOTSnetwork.
Figure4.4Test3printscreen,leftOTStoRTUvs.rightDEservertoRTU
Test3result:TheOTS’spingwentthrough,itconfirmedthatRTUandOTScouldcommunicatewitheachother.TheOTSDEserver’spingdidnotgothroughasitsupposedtobe.
Test4:TheSCADA-OTSbackchannelshouldenableOTStosendnon-simulatedRTU’spacketsdirectlytotheCleanSystem’sSCADAwithoutgoingthroughRTUandPCU.TheRTU,ontheotherhand,canonlycommunicatewithSCADAthroughthePCU.Hence,thistestincludespingingtheCleanSystem’sSCADAfromtheOTSandtheRTU.
Figure4.5Test4printscreen,leftOTStoSCADAvs.rightRTUtoSCADA
Test4result:Asexpected,theOTS’spingwentthroughandtheRTU’spingdidnot.
pg.28
Thetestshaveproventhatcommunicationnetworkswereoperatingproperly.Additionally,onlythePCUandtheOTScouldpingtheRTU.ItimpliesthattheSCADA-RTUandtheRTU-OTSnetworkcontainonlytheintendedcommunicationtraffic.Consequently,thesystemwaspreparedforfurthertests.
4.1.2 Data Flow Test
ThedataflowtestwasdonetoconfirmthattheNOTSfollowstheintendeddataflowdescribedinsection3.4.Todothis,thetrafficpathswereloggedtocapturethetelegramsafteratrippingcircuitbreakeractioncommandwasreleased.Thecontentofthetelegramwaspartiallyblurredtokeepitconfidential.
1. Operator’sactionfromtheCleanSystem’sWS500wasforwardedtotheSCADAthroughaprocessqueue.
Figure4.6SCADA'sprocessqueuelog
2. TheSCADAsentthecommandtoPCUthroughmoduleRCSwithRSP.
Figure4.7ModuleRCSlog
pg.29
3. ThePCUreceivedthetelegram;thenmoduleM8sentittoRTUinIEC104form.
Figure4.8PCU'sRSPbuslog
4. ThemoduleM3ofRTUreceivedtheIECtelegram,convertedandforwardedthetelegramtoM9throughtheRTU’sRSPbus.
Figure4.9RTU'sRSPbuslog
5. Theadditionalsourcecode,namelyrtuSendLog,grabbedandforwardedthecommandtelegramfromtheRTU’sRSPbuglogfiletotheOTS.
Figure4.10PrintoutofreceivedtelegramfromtheRTUontheOTSmachine
6. ThetelegramwasforwardedtotheOTS’sprocessp1throughaprocessqueue,andtheOTSstartedtorecalculateanewloadflow.
Figure4.11OTS'processqueuelog
pg.30
7. ThenewindicationsandmeasurementsofthenewstateoftheOTSweresentbacktothemoduleM3intheRTUthroughmoduleM9.
Figure4.12RTU'sRSPbuslog
8. ThemoduleM3forwardedthetelegramtothePCUinIEC104format.
Figure4.13PCU'sRSPbuslog
9. ThePCUreceivedandsentthetelegramtotheSCADA’sprocessesthroughmoduleRCS.
Figure4.14ModuleRCSlog
Theresultdemonstratedthatthedesiredcommunicationroutementionedabovehadbeencorrectlyfollowed.Additionally,itshowedthetransmissiontimeofsendinganopenbreakercommand.Ittook1.1secondsfortheSCADAtosendthecommandtotheOTSandreceivedbacktheupdatedindication.
Whenanopenbreakeractionoccurred,theSCADAformsacommandtelegramthenforwardsittotheOTSthroughthePCUandtheRTU.Then,processp1takesthetelegramandtranslatesitforprocesspots.Consequently,thepotsupdatesthedatabaseandp2immediatelydetectsachangeofcircuitbreakerstatusandreturnthenewindicationtotheSCADA.Hence,thereisanimmediatereplyforpowersystem’sstaticresponses.However,theeffectsofthesimulatedpowersystemmeasurementscanonlybeseenaftertheOTS’s5-secondsperiodiccycle.Every5secondstheOTSprocessesnewpowerflow,networkequipmentdynamics,loaddynamics,unitdynamics,andsystemfrequencydynamics.
pg.31
4.1.3 Protocol Check
TherearethreedifferentapplicationlayerprotocolsimplementedontheNOTSanddescribedasfollows;
• ABB’sproprietaryNetworkManagerinter-processcommunicationprotocolthatoperatesonthetopoftheAvanticommunicationplatform.Theprotocolisdefinedinsidetheprocesses'sourcecode.WheredoesNOTSusethisprotocol:communicationbetweenprocessesinsidetheSCADAandOTS,e.g.,betweenp1andpots.ReferringtoFigure4.6andFigure4.11,theyhaveproventheapplicationofthisprotocolforcommunicationbetweeninternalNetworkManagerprocesses.
• ABB’sproprietaryRSPforcommunicationbetweenmodulesonthetopofTCP/IPprotocolthroughtheRSPbus(ABB,1997).WheredoesNOTSusethisprotocol:communicationbetweenmoduleRCSandmoduleM8,andmoduleM3andmoduleM9.Figure4.7,Figure4.9,Figure4.12,andFigure4.14aretheintendedinter-modulecommunicationtouseRSP.TheyprintedRSPafterthetimestampasaconfirmationRSPisimplemented.
• IEC104protocolforcommunicationbetweenPCUandRTUwithTCP/IPprotocolonitslinkandnetworklayerprotocol.WheredoesNOTSusethisprotocol:communicationbetweenPCUandRTUFigure4.8andFigure4.13illustratedtheIEC104telegramformat(ABB,2014)(ABB,2015)(Statkraft,2015).ItindicatesthatIEC104hasbeensuccessfullyimplementedforcommunicationbetweenPCUandRTU.Additionally,(ABB,2014)(ABB,2015)explainsthatM8andM3arecommunicatingontheTCP/IPlink,whichsupportedbythenetworkconfigurationinFigure4.1.
pg.32
4.1.4 Functionality Test
TwosimplesimulationsweredonetocheckthefunctionalityoftheNOTS.TheactionwascarriedoutintheCleanSystem’sWS500.Thefirstactionwastrippingacircuitbreakertomonitorifthepowerinthatparticularlineturnstozero.Thesecondactionwassetgeneratorvalueto400MWandmonitorraise.
Figure4.15Actionone,trippingacircuitbreaker
Figure4.16Actiontwo,raisingageneratorsetpointvalue
Thesimulationresultwasonpoint,aftertrippingthecircuitbreaker,activepowerbecamezero.Similarly,thegeneratoroutputraisedprogressivelyafterthesetpointvaluewaschanged.
pg.33
4.2 Scalability Test
AScalabilitytestwasperformedtoevaluatethecapacityofthecommunicationnetworks.Theconstant growth of power system size has shown the importance of knowing themaximumlimit/capacityoftheNOTS.ThetestwasdonesimplybyenablingmoreRTU’straffictothePCU-RTU communication link becausepreviously only threeRTUswereused. They areRTUs fromAmherst,Bowman,andTroystation.
ThetestwashaltedafterconfigurationofthreeadditionalRTUswithRP570inadditiontothefirst threeRTUswith IEC104.Thetestwasstoppednotbecausethesystemcouldnothandletheextratraffic,butduetothefollowingreason:
• IEC 104 protocols are only supported by the RTUs in Amherst, Bowman, and Troystations.
• OtherRTUscanonlyimplementolderRP570protocol.
• Implementation RP570 is not aligned with the predefined communication protocolrequirements.
Sinceitisnotpossibletoassessthecapacityofthenewdesign,followingarethehypotheticalbottlenecks:
• PCUtoRTUThePCUandRTUcanbeconfiguredtosupportupto255tasks,whicheachtaskrepresentsanRTU.However,apowersystemSCADAwillneverfit255RTUs.Additionally,thePCUprocessestookabout2-3%ofthemachineCPUresourceswithdualcoresand4GBoframundernormaltraffic.Hence,asinglesimulatorcansupportupto25-30PCUtoRTUconnections.
• OTStoM9ABBhasnevertestedmoduleM9tohandleheavydatatrafficsinceitwasinitiallycreatednottoreceiveahugenumberofpayload.
• RTUtoOTSrtuSendLogisanadditionalsourcecodetoworkaroundthelimitationofmoduleM3.ThemoduleM3couldonlysendtelegramsbacktoitsmaster,themoduleM8.Asaresult,rtuSendLogwascreatedtoreadtheRTUlogfilesandforwardthecommandtelegramtotheOTS.
pg.34
4.3 Re-Assessing the Design Requirements
Inthissection,theNOTSisreassessedwiththeinitialdesignrequirementsspecifiedinsection3.2.Theconclusionishighlydependentuponthetestdoneintheprevioussubchapterandtheknowledgeduringtheproductdevelopment.
Theassessmentneedstoanswerthesefollowingkeyquestions:
• Doesthedesignfulfillalltherequirements?
• Ifthereisanunmetrequirement,whatcausesit?
Functionality
• TheNOTSdoesnotremoveordisableanyoriginalfunctionalitiesofferedbytheSCADA/OTS.Itisbecausetheadditionalprogramsmentionedin3.5.3werecreatednottointerferewiththefunctionalitiesoftheinternalprocesses.
• TheNOTShasarealtimeWS500sinceitdeployedtheSCADA/EMSNetworkManagerwithoutOTS.
• Referringtotheresultfrom4.1.4andfewotherteststodifferentcircuitbreakersandgenerators,theoutcomeofthesimulatorproventobeconsistent.
NetworkArchitecture
• TheNOTShassuccessfullyintegratedthePCU,theRTU,andtheircommunicationnetworksintoitsdesignsolution.
• TheNOTSincorporatedanadditionalnetworkthatconnectsOTSdirectlytotheSCADAasillustratedinFigure3.2whichdoesnotexistinanactualSCADA.Initially,theintentionofhavingthenetworkwastosimplifythedevelopmentprocessbylimitingthetrafficpassingthroughthePCUtoRTUnetworkandberemovedonceitpassedthetestingphase.However,duetothelimitationfoundduringthescalabilitytest,ithastostay.SincetheSCADArequiresallRTUsinformationandupdatefromtheOTS,thebackchannelnetworkisnecessarytohandlethenon-IEC104RTUsinformationtraffic.
• ItwasnotpossibletocheckifthenetworksabletoaccommodatetrafficforalltheRTUsbecauseonlythreeRTUssupportIEC104inthecurrentdatabase.However,therearethreepotentialbottlenecksfortheNOTS.First,thePCUmachinecanonlyhandlearound30RTUsbecauseeachPCUtoRTUconnectiontook3%CPUresources.Second,themoduleM9limitedcapacitybecauseitwasnotdesignedtohandleahugeamountoftraffic.Finally,thertuSendLogprogramthathastoreadfromtheRTUlogfile.Itwould
pg.35
presumablyhavereliabilityissueifthelogfilestartedtoupdatefasterthantheprogramprocessingtimetoopenandreadthefile.
• Thetestsinsection4.1.1haveconfirmedthattheconnectionshavebeenproperlyconfiguredonlycontaintheintendeddatatraffic.
Protocols
• TheprotocolcheckshowsthatIEC104protocolwassuccessfullyimplementedforPCUtoRTUcommunication.IEC104protocolisdesirablebecauseitadoptedtheTCP/IPprotocolasIP-basedprotocolsabletoconstructanetworkbyusingpublicwires,itmeansthephysicalequipmentthatusedtobuildthenetworkbeforeIPbecomeslessofconcerninthecontrolcenter.Moreover,InternetQoSperformanceandencryptionprotocolshaveimprovedsignificantlyinthelastfewyears.Asaresult,thereisnoreasonforSCADAnottouseIP-basedprotocols.
• Thedocuments(ABB,2014)(ABB,2015)(Statkraft,2015)haveensuredthattheNOTSusesstandardIEC104telegramformat.
• TheNetworkManagerproprietaryprotocolsforinter-processandinter-modulecommunicationhavenotbeenchangedasdiscussedinsection4.1.3.
Dataflow
• Thetestresultinsection4.1.2showedthattheNOTShavesuccessfullydesignedtofollowthedesiredcommunicationroute.ItisidenticalwiththerouteusedbyanactualSCADAaccordingtoABB’sSCADAexpertsand(Yang&Barria,2011).
• Othernon-idealflowsaretheRSPcommunicationtoandfrommoduleM9ontheRTUandRTUcommunicationtoOTS.InanactualSCADA,whenaphysicalRTUreceivesatelegram,itsendselectricpulsestocontroltherelaysandpowersystemdevices.However,itisnotapossiblesetupfortheNOTSbecausetherearenoactualphysicalRTUsandpowersystemdevices.Additionally,theRTUtopowersystemprotocolsandtrafficdeviationsarenotwithinthescopeofthisprojectasmentionedinsection1.4.
• Thetestsin3.4provedthatthedatawastransmittedonlytotheintendedrecipientreliably,whichlaterstrengthenwiththefunctionalitysimulations.
• Accordingtothetestsdonein4.1.2andAppendixB,theNOTStookanaverageof1.055secondstosendacommandandreceivedastaticresponseofthepowersystemwithavarianceoflessthan10%.Additionally,ittooklessthan300msforRTUtoreturnthenewindicationtoSCADA.Ontheotherhand,inarealSCADAsystem,itwouldtakemuch
pg.36
longertimeforRTUtoreachthecontrolcenter.Accordingto(PowerGripCorporationofIndia,2012),thetimerangingfrom2to15seconds,whichishighlydependentonthecommunicationchanneltechnology(fiberoptic,digitalmicrowave,etc.).Duetothefasterrate,theNOTScouldprovideamoreaccurateobservationofthenetworkthananactualSCADA.
Inshort,thedesignfulfillsmostoftheinitialdesignrequirements.Amajorunmetrequirementwas a result of the limited support of IEC 104 protocol in Network Manager database.Consequently,anadditionalnetwork isnecessarytosupporttheNetworkManager limitation.Due to this limitation, it was not possible to test the PCU-RTU network capacity. There arehypotheticallimitationsforthePCU-RTUnetworkcapacity.However,furthertestingisrequiredtoconfirmiftheycouldresultinanunmetrequirement.
4.4 NOTS Deviations
ThereassessmentresultsindicatethattheNOTShavefewdifferenceswiththeactualSCADA.Thosedifferencesandtheirimpactonthesimulationresultwillbediscussedinthefollowing;
Functionality
• TheNOTSsimulatesthepowersystem’sdynamicresponsesperiodicallyevery5seconds,whileinanactualSCADA,theresponseshappenedimmediately.Asaresult,NOTScanonlysimulatestheslowpowersystembehaviorsuchasautomaticgenerationcontrol,loadfrequencycontrol,frequencyvariations,powerswing,etc.
• PartofRTUfunctionalityistakenoverbyOTS’sp2process.Intheactualsystem,eventorvaluechangedetectionisdonebytheRTU,anditformedanIEC104telegramandsentittothePCU.However,intheNOTS,thep2doestheeventdetectionandforwardthenewvaluetotheRTUtobetranslatedandforwardedtothePCU.
NetworkArchitecture
• TheadditionalbackchannelfromtheOTStoSCADAtohandlethenon-IECcommunicationtraffic.
pg.37
Protocols
• ThecommunicationtrafficbetweentheRTUandtheOTS,alsotheOTSandtheSCADAthatusestheadditionalbackchanneldonotfollowanyapplicationlayerprotocol.However,theyuseanidealTCP/IPfortheirnetworkandlinklayers.
Dataflow
• ThetrafficintheSCADA-OTSnetworkdoesnotexistinanactualSCADA
• ThetimingofthedataflowfromtheOTStotheRTUisnotideal.Itisaconsequencefromthefunctionalitydeviation.TheNOTSsimulatesdynamicresponsesofapowersystemevery5seconds.Hence,theRTUhavetowaituntilthenext5-secondscycleiscompletedbeforeitreceivesnewmeasurements.However,anactualpowersystem’smeasurementsshouldcomerightafteracommandissent.
• TheRTUdoesnotreceiveanidealtrafficcontentfromtheOTS.Inreality,thepowersystemsendseverymeasurementtotheRTU,then,theRTUdetectedaneventandcreatedatelegramtosendthePCU.However,intheNOTS,eventdetectionfunctionalityistakenoverbyOTS’sprocessp2.Hence,OTSsendsonlytheevent/measurementchangetoRTUinsteadofsendingeverymeasurement.
ThesedifferencesaretranslatedtolimitationsfortheNOTSasdescribedbelow;
However,onthetopofthat,theNOTSpossessesanidealdataacquisitionfunctionality,e.g.,p2detectswhenthereisasignificantchangeinthemeasurement;thenitsendstheupdatedmeasurementtothePCUthroughtheRTU,whichisanidealrepresentationofIEC104eventdrivenprotocolmechanism.ThenitgoestoSCADAandisdisplayedontheHMI.Additionally,theidealismwasconfirmedupbyABB’sengineer.
TheOTSrunsthemodulesperiodicallyevery5seconds;thismeanstheNOTSisonlylimitedtosimulatetheslowdynamicresponseofthepowersystem,whichistheoriginalintentionoftheOTS.However,duetomanydemandsforhavingafasterOTS,ABBiscurrentlyworkingtolowerthecycletimeto2secondsforabiggerdynamicresponsescope.Hence,thislimitationcanbeminimizedinthefuture.
TheadditionalcommunicationchannelwascreatedtosupportstationsapartfromAmherst,Bowman,andTroybecausetheydonotsupportIEC104communicationprotocols.Itmeanstheoperator’sactionscanbedoneforAmherst,Bowman,andTroypowersystemdevices.However,allthevaluesandmeasurementsforotherstationsinthepowergridmodelthatillustratedinFigure2.4arestillvalidbecauseoftheupdatedvaluesforallnon-IEC104RTUsisalsoforwardedtotheSCADAusingtheadditionalbackchannel.
pg.38
ThedataflowdifferencefromOTStoRTUdoesnothaveanyimpactonthefunctionalityofthesimulator.AlthoughthetrafficfromOTStoRTUisnotideal,butmoreimportantlythetrafficfromRTUtoPCUisidealandusestherequiredprotocol.Additionally,thecommunicationprotocolbetweenRTUandOTSisnotwithinthescopeofthisproject.
4.5 Investigating additional functionalities
TheintegrationofPCU-RTUnetworkcommunicationhasopenednewideastoaddnewfunctionstothesimulator,forexample,communicationprotocoltestingorcyber-attacksimulation.PCU-RTU’sprotocolisnotlimitedtoIEC104,ifthereisanewprotocolinthefuture,itcouldalsobeimplementedusingthesamedesign.Networkcommunicationcyber-attackhasgainedmuchattentionlately,especiallyafterUkraine’spowergridwashacked(Bernat,2016).WithNOTS,cyber-attackcanbesimulatedbypenetratingthePCU-RTUcommunicationandinvestigatetheresponseofthesimulator.Ongoingprojectsareinvestigatingtheimpactofcyber-attackstothepowersystemutilizingasimilarsimulationsetup(Fovino,Carcano,Masera,&Trombetta,2009)(Queiroz,Mahmood,Hu,Tari,&Yu,2009).
pg.39
Chapter 5: Conclusion, Improvements, and
Future Works
5.1 Conclusion
TheOTSisahugeeffortinhelpingpowersystem’soperatortogetfamiliarwithtoday’scomplexpowersystemnetworktoperformeffectivelyandefficiently.Italsopotentiallyhasanenormousimpactonpowerutilitycompaniesandthecommunitiesinhavingamorereliableelectricitysupply.Thus,theOTSsystemneedsacontinualimprovement.
Atthebeginningofthework,theideawastorefactortheOTSmoduletohaveanidealnetworkcommunicationanddatatrafficprofileastotheactualSCADAsystem.Designrequirementsweresetintheearlystagetokeeptrackandevaluatethedesignduringthedevelopmentphase.
Thefinalizedprototypehassuccessfullymetmostofthedesignrequirements,mostimportantlyintegratingthePCU’sandRTU’scommunicationnetworkwithIEC104protocolandallowinganidealdatatraffic.However,thereisanadditionalbackchannelhandlingnon-IEC104traffic,whichshouldnotexistintheidealcase.
Inpractice,theNOTScouldenablePCUtoRCUcommunicationforallRTUsusingtheavailableRP570protocolsandremovetheadditionalbackchannel.However,therequirementwastoimplementIEC104becauseitisthecurrentstandardprotocolsformasterstationtocontrolledstationcommunication.Fortunately,thisissuecanbesolvedoncetheNetworkManagerenablesIEC104communicationprotocolsforallRTUsinthefuture.
Tosummarize,theprototypehasthreeRTUs,Amherst,BowmanandTroy,thatcompliedwithallpredefinedrequirementsandtheprojectobjectives.Whereas,theotherRTUsarehinderedbythelimitationoftheNetworkManager’sdatabase.Asaresult,theoperator’sactionsimulationcanbeperformedonthepowersystemdevicesinthosestations.
pg.40
5.2 Possible Improvements
• SystemautomationAutomatizationoftheNOTSstillhasroomforimprovement.Currently,RSPbusloggingandrtuSendLogprogramarerunmanually,inthefuture.bator.exefilecanbegeneratedtoautomatethesefunctions.Additionally,PCU-RTUconfigurationalsoneedstobeautomated,inparticularontheXMLfilesgeneration.
• PersistentTCPconnectionrtuSendLogandrtulistencanbeenhancedbyusingpersistentTCPconnectionwithmultithreadingtoavoidopeningandclosingsocketforeverytelegramexchange.
5.3 Future Works
• IEC104implementationforallRTUsImplementationofIEC104forallRTUstohaveacompleteversionofthesimulator.ItcanbedonebyaddingIOAforthemeasurementsandindicationpointsontheAvantidatabase.
• IntegratingtheRTU-OTScommunicationatM3moduleAtthemomenttheM3doesnotsupportcommunicationdirectlytotheOTS.Asaworkaround,rtuSendLogandrtulistenprogramswerecreatedtoaddthefunctionalitytothesystem.ImplementingcommunicationdirectlyfromthemoduleM3totheOTSwillmakethedataflowmorefluidandmakethesystemmorereliable.
• IntegratingstandardprotocolsundertheRTUlayerAlthoughtheABB’sSoftwareEntreprisedepartmentdoesnotdevelopprotocolsunderneaththeRTUlayer,itispossibletocollaboratewithotherdepartmentstoimplementprotocolsbetweentheRTUandtheOTS.TheseprotocolsarepresentedbyIEC61850,whichusesGOOSE,MMSandSMVapplicationlayerprotocolsonthetopofTCP/IPoveranEthernet(Adamiak,Baigent,&Mackiewicz).
• Cyber-attacksimulationTheimplementationofthePCUtoRTUcommunicationlinkhasletdataflowoutsideoftheNetworkManager’sbox.TheexposedPCUtoRTUlinkhasenabledtheNOTSforcyber-attacksimulation.Theattackercouldpenetrateandattackthelink,whilepowersystem’soperatorcouldstudyandmonitorwhathappenedifsuchattackhappened.
pg.41
References ABB.(1990).AvantiReferenceManualGeneralIntroduction.Vasteras:ABB.
ABB.(2015).IEC60870-5-104RTUMasterProtocolFunctionDescription.ABB.
ABB.(2014).IEC870-5-101&104SlaveFunctionDescription.ABB.
ABB.(2015).NetworkManagerTraining-EMS,GMS,DMSApplications.ABB.
ABB.(2015).NetworkManager:SCADA/EMSandSCADA/GMS.(ABB)Retrievedfromhttp://new.abb.com/docs/librariesprovider139/default-document-library/network-manager-scada-ems-gms-overview_abb.pdf?sfvrsn=8
ABB.(1997).RemoteServerProtocolProtocolSpecification.ABB.
ABB.(1997).RP570ProtocolDescription.ABB.
Adamiak,M.,Baigent,D.,&Mackiewicz,R.(n.d.).IEC61850CommunicationNetworksandSystemsinSubstation.Retrievedfromhttp://www.gegridsolutions.com/multilin/journals/issues/spring09/iec61850.pdf
Bernat,J.A.(2016,March3).Wired.Retrievedfromhttps://www.wired.com/2016/03/inside-cunning-unprecedented-hack-ukraines-power-grid/
Bose,A.,&Chen,Y.(1987).ChoosingtheAppropriateBoundaryforAdaptiveReduction.PICA.Montreal.
Chu,R.F.,Dobrowolski,E.J.,Barr,E.J.,McGeehan,J.,Scheurer,D.,&Nodehi,K.(1991).TheUsesofanOperatorTrainingSimulatorforSystemRestoration.IEEE.
Demjen,C.,Kadar,P.,Meszaros,P.,&Szendy,D.(1994).AnIntegratedDispatcherTrainingSimulatorforUseinaRegionalElectricityDistributionCentre.ComputingandControlEngineering,66-70.
DyLiacco,T.E.,Enns,M.K.,Schoeffler,J.D.,Quada,J.J.,Rosa,D.L.,Jurkoshek,C.W.,etal.(1983).ConsiderationsinDevelopingUtilizingOperatorTrainingSimulator.IEEETransactionsonPowerApparatusandSystems,PAS-102,3672-3679.
Fovino,I.N.,Carcano,A.,Masera,M.,&Trombetta,A.(2009).AnExperimentalInvestigationofMalwareAttacksonSCADAsystems.InternationalJournalofCriticalInfrastructureProtection,2,139-145.
Hua,B.,Zhou,J.,&Yu,J.(2004).IntegrationofExistSCADA/EMSwithDispatcherTrainingSimulatorSystem.IEEE.
pg.42
InternationalElectrotechnicalCommission.(2016).IEC60870-5-104.
IPCOMM.(n.d.).IpcommGmbH.Retrievedfromhttp://www.ipcomm.de/protocol/IEC104/en/sheet.html
Koda,V.,Bose,A.,&Anderson,M.M.(1998).PowerPlantModelsforOperatorTrainingSimulators.IEEE/PESSummerPowerMeeting.Portland.
Marihart,D.J.(2001).CommunicationstechnologyguidelinesforEMS/SCADAsystems.IEEETrans.PowerDel.,16(2),181-188.
PowerGripCorporationofIndia.(2012).UnifiedRealTimeDynamicStateMeasurement.Gurgaon:PowerGridCorporationofIndia.
Prais,M.,&Bose,A.(1987).ATopologyProcessorthatTracksNetworkModificationsOverTime.IEEE/PESSummerPowerMeeting.SanFrancisco.
Queiroz,C.,Mahmood,A.,Hu,J.,Tari,Z.,&Yu,X.(2009).BuildingaSCADASecurityTestbed.InternationalConferenceonNetworkandSystemSecurity.
Sindelar,R.,&Novak,P.(2012).SimulationIntegrationFramework.IEEE.
Statkraft.(2015).ProtocolImplementationDocumentforIEC60870-5-104Ed.2.StatkraftEnergiAS.
Tomsovic,K.,Bakken,D.E.,Venkatasubramanian,V.,&Bose,A.(2005).DesigningtheNextGenerationofReal-TimeControl,Communication,andComputationforLargePowerSystems.ProceedingsoftheIEEE,93(5),965-979.
VMware.(2014).WMWarevCloudSuite.PaloAlto:VMware.
Walve,K.,&Edstrom,A.(1998).TheTrainingSimulatorARISTO-DesignandExperiences.Sweden:IEEE.
Wu,F.F.,Moslehi,K.,&Bose,A.(2005,November11).PowerSystemControlCenters:Past,Present,andFuture.ProceedingsoftheIEEE,pp.1890-1908.
Yang,Q.,&Barria,J.A.(2011).CommunicationInfrastructuresforDistributedControlofPowerDistributionNetworks.IEEETransactionsonIndustrialInformatics,7,316-327.
Zhang,G.,&Bose,A.(1989).ScenarioBuildingforOperatorTrainingSimulatorUsingaTransientStabilityProgram.IEEE/PESWinterMeeting.NewYork.
pg.43
pg.44
Appendix A: Alternative design solutions Adesignalternativethatwasexaminedduringtheproject,however,duetothecompleximplementationandnotidealdataflow,itdidnotgetselected.
pg.45
Appendix B: Data f low test Thisappendixcontainstwoothertrialsdataflowtestdescribeinsection4.1.2.
Trial2:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
pg.46
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
pg.47
Trial3:
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
pg.48
______________________________________________________________________________
______________________________________________________________________________
TRITA TRITA-EE 2016:169
www.kth.se