Download - Troubleshooting Mcrnc
-
WCDMA RAN, Rel. RU50 andRU50 EP1, OperatingDocumentation, Issue 05
TroubleshootingMulticontroller RNC DN0976768Issue 02BApproval Date 2015-02-23
NokiaNetworks
-
The information in this document applies solely to the hardware/software product (Product) specifiedherein,andonlyasspecifiedherein.
ThisdocumentisintendedforusebyNokiaSolutionsandNetworks'customers(You)only,anditmaynotbeusedexceptforthepurposesdefinedintheagreementbetweenYouandNokiaSolutionsandNetworks(Agreement)underwhich thisdocument isdistributed.Nopartof thisdocumentmaybeused,copied,reproduced,modified or transmitted in any form ormeanswithout the priorwritten permission ofNokiaSolutionsandNetworks. If youhavenotentered intoanAgreementapplicable to theProduct,or if thatAgreementhasexpiredorhasbeenterminated,Youmaynotusethisdocument inanymannerandYouareobligedtoreturnittoNokiaSolutionsandNetworksanddestroyordeleteanycopiesthereof.
The document has been prepared to be used by professional and properly trained personnel, andYouassumefullresponsibilitywhenusingit.NokiaSolutionsandNetworkswelcomeYourcommentsaspartoftheprocessofcontinuousdevelopmentandimprovementofthedocumentation.
This document and its contents are provided as a convenience to You. Any information or statementsconcerningthesuitability,capacity, fitnessforpurposeorperformanceof theProductaregivensolelyonanasisandasavailablebasisinthisdocument,andNokiaSolutionsandNetworksreservestherighttochangeanysuchinformationandstatementswithoutnotice.NokiaSolutionsandNetworkshasmadeallreasonableeffortstoensurethatthecontentofthisdocumentisadequateandfreeofmaterialerrorsandomissions,andNokiaSolutionsandNetworkswill correcterrors thatYou identify in thisdocument.But,NokiaSolutionsandNetworks'totalliabilityforanyerrorsinthedocumentisstrictlylimitedtothecorrectionofsucherror(s).NokiaSolutionsandNetworksdoesnotwarrantthattheuseofthesoftwareintheProductwillbeuninterruptedorerror-free.
NOWARRANTYOFANYKIND,EITHEREXPRESSORIMPLIED, INCLUDINGBUTNOTLIMITEDTOANY WARRANTY OF AVAILABILITY, ACCURACY, RELIABILITY, TITLE, NON-INFRINGEMENT,MERCHANTABILITYORFITNESSFORAPARTICULARPURPOSE, ISMADE INRELATIONTOTHECONTENT OF THIS DOCUMENT. IN NO EVENT WILL NOKIA SOLUTIONS AND NETWORKS BELIABLE FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT,INCIDENTALOR CONSEQUENTIAL OR ANY LOSSES, SUCH AS BUT NOT LIMITED TO LOSSOFPROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA THAT MAYARISEFROMTHEUSEOFTHISDOCUMENTORTHEINFORMATIONINIT,EVENINTHECASEOFERRORSINOROMISSIONSFROMTHISDOCUMENTORITSCONTENT.
ThisdocumentisNokiaSolutionsandNetworksproprietaryandconfidentialinformation,whichmaynotbedistributed or disclosed to any third parties without the prior written consent of Nokia Solutions andNetworks.
Nokia isa registered trademarkofNokiaCorporation.Otherproductnamesmentioned in thisdocumentmaybetrademarksoftheirrespectiveowners,andtheyarementionedforidentificationpurposesonly.
Copyright2015NokiaSolutionsandNetworks.Allrightsreserved.
f Important Notice on Product Safety Thisproductmaypresentsafetyrisksduetolaser,electricity,heat,andothersourcesofdanger.
Only trained and qualified personnel may install, operate, maintain or otherwise handle thisproductandonlyafterhavingcarefullyreadthesafetyinformationapplicabletothisproduct.
The safety information is provided in the Safety Information section in the Legal, Safety andEnvironmentalInformationpartofthisdocumentordocumentationset.
Nokia Solutions and Networks is continually striving to reduce the adverse environmental effects of itsproductsandservices.Wewouldliketoencourageyouasourcustomersanduserstojoinusinworkingtowardsacleaner,saferenvironment.Pleaserecycleproductpackagingandfollowtherecommendationsforpoweruseandproperdisposalofourproductsandtheircomponents.
IfyoushouldhavequestionsregardingourEnvironmentalPolicyoranyoftheenvironmentalservicesweoffer,pleasecontactusatNokiaSolutionsandNetworksforanyadditionalinformation.
TroubleshootingMulticontrollerRNC
2 DN0976768 Issue:02B
-
Table of ContentsThisdocumenthas71pages
Summaryofchanges..................................................................... 7
1 OverviewofMulticontrollerRNCTroubleshooting......................... 81.1 TroubleshootingMulticontrollerRNCrecommendations................81.2 Informationsourcesinfaultsituations............................................91.3 Problemtypes.............................................................................. 101.4 Generictroubleshootingprocedure.............................................. 11 2 ReportingproblemtoNSN........................................................... 14 3 Symptomsdatacollection............................................................ 193.1 Standardsymptomsreport...........................................................193.2 Collectingstandardsymptomreports...........................................233.3 Listingthestandardsymptomreports.......................................... 283.4 Copyingstandardsymptomsreportstoremotemachine.............293.5 Deletingthestandardsymptomreports....................................... 30 4 Genericsoftwaretroubleshootinginstructions............................. 324.1 Displayingallblackboxfiles......................................................... 324.2 Viewingactivealarmsandalarmshistory.................................... 334.3 Collectingcoredumpfileinformation........................................... 34 5 Configurationmanagementtroubleshooting................................ 365.1 Savingconfigurationsnapshotfails..............................................365.1.1 Description................................................................................... 365.1.2 Symptoms.................................................................................... 365.1.3 Recoveryprocedures................................................................... 36 6 Softwaremanagementtroubleshooting....................................... 376.1 Installationofanewsoftwaredeliveryfails.................................. 376.1.1 Description................................................................................... 376.1.2 Symptoms.................................................................................... 376.1.3 Recoveryprocedures................................................................... 376.2 Activationofanewsoftwaredeliveryfails....................................376.2.1 Description................................................................................... 376.2.2 Symptoms.................................................................................... 386.2.3 Recoveryprocedures................................................................... 386.3 Troubleshootingsoftwareconfigurationmanagement................. 386.3.1 Executingpre-downloadscriptfails..............................................386.3.2 ParsingtargetBD.xmlfails............................................................396.3.3 Executingpre-activationscriptfails..............................................396.3.4 UpgradingeSWfails.................................................................... 406.3.5 Alarm2518-NOVALIDFALLBACKCOPYFORDEFAULT
PACKAGEistriggered................................................................. 41
TroubleshootingMulticontrollerRNC
Issue:02B DN0976768 3
-
7 Networkingtroubleshooting..........................................................437.1 Packetlossincertaintrafficflowsunexpectedlyhigh.................. 437.1.1 Description................................................................................... 437.1.2 Symptoms.................................................................................... 437.1.3 Recoveryprocedures................................................................... 437.2 MultihopBFDsessionsarenotestablished................................ 467.2.1 Description................................................................................... 467.2.2 Symptoms.................................................................................... 467.2.3 Recoveryprocedures................................................................... 467.3 OSPFisnotworkingproperly...................................................... 487.3.1 Description................................................................................... 487.3.2 Symptoms.................................................................................... 487.3.3 Recoveryprocedures................................................................... 497.4 IPsignalinglinkactivationfails.....................................................527.5 Thestateofallsubsystemsintheremotenetworkelementis
unavailable(UA)...........................................................................53 8 Hardwaretroubleshooting............................................................ 548.1 Notrafficdetected........................................................................ 548.1.1 Description................................................................................... 548.1.2 Symptoms.................................................................................... 548.1.3 Recoveryprocedure.....................................................................548.2 SFPportsaddedforUSPUsandCSPUsdonotshowcorrectly
theiroperationalstatenoralarmsareraised................................558.3 TroubleshootingwithmcJANEtool.............................................. 56 9 Performancemanagementtroubleshooting................................. 699.1 ThresholdmonitoringalarmisnotsenttoNetAct........................ 699.2 TransportandHWmeasurementmanagementinmcRNC..........709.2.1 McRNCmeasurementmanagementconcepts............................ 709.2.2 McRNCmeasurementmanagementcommands......................... 71
TroubleshootingMulticontrollerRNC
4 DN0976768 Issue:02B
-
List of FiguresFigure1 ForcingchangeoftheSPFpoststate................................................ 64Figure2 Exampleusageofthecpcommand................................................... 65Figure3 Checkingtheportmonitoring..............................................................66Figure4 Removingportmonitoring.................................................................. 67
TroubleshootingMulticontrollerRNC
Issue:02B DN0976768 5
-
List of TablesTable1 NSNCaseType..................................................................................15Table2 NSNProblemPriority......................................................................... 16Table3 Supportedplugins...............................................................................19Table4 Parametersrelatedtosave symptom-reportcommand.............. 24Table5 Mappingbetweenpip-markandQueueID......................................44Table6 PriorityvaluesfordifferentDSCPcodes............................................ 45
TroubleshootingMulticontrollerRNC
6 DN0976768 Issue:02B
-
Summary of changesChangesbetweendocumentissuesarecumulative.Therefore,thelatestdocumentissuecontainsallchangesmadetopreviousissues.
Changes between issues 02A(2014-05-23, RU40) and 02B(2015-02-23,RU40)Newsubchapterhasbeenadded.Troubleshooting software configurationmanagement
Changes between issues 02(2013-09-15, RU40) and 02A(2014-05-23,RU40)OSPF is not working properlyhasbeenupdated.ThefeaturemanagementnamehasbeenchangedtoOSPFForRedundancy.
Changes between issues 01D (2012-11-23, RU30) and 02(2013-09-15,RU40)Activation of a new software delivery fails
Commandshavebeenupdated.
Installation of a new software delivery fails
Commandshavebeenupdated.
Displaying all blackbox files
InformationonpermissionsrequiredforexecutingthecommandinthissectionisaddedtoBeforeyoustartsection.
Collecting core dump file information
Commandshavebeenupdated.
TroubleshootingMulticontrollerRNC Summaryofchanges
Issue:02B DN0976768 7
-
1 Overview of Multicontroller RNCTroubleshooting
1.1 Troubleshooting Multicontroller RNCrecommendationsIfyouhaveacontractwithNokiafortheoperationandmaintenanceofthenetwork(orsomeotheragreement),theactionsyouneedtotakeinafaultsituationmaybedifferentfromtheonessuggestedinthetroubleshootinginstructions.Ifthegeneralprinciplesareinconflictwiththeoperationandmaintenancecontractoranyothercontract,carryoutactionsasagreedinthecontract.
TheoperationandmaintenancepersonnelthatcarryouttroubleshootingshouldbefamiliarwiththehardwareandsoftwareofNokianetworkelements.
Electrostatic precautionsWhenhandlingplug-inunits,itisimportanttouseElectrostaticPrecautions(ESP).Thismeansthatyoumustbeearthedtoequipmentracksusinganapprovedwriststrapandconnectinglead.ApprovedESPequipmentmakesaresistiveconnectiontoensurethesafetyofthepersonnelandtopreventasuddenstaticdischargeduringconnectiontotheearthingpoint.
Security proceduresYouarerecommendedtoestablishsecurityproceduresatyoursitetoensureappropriatestaffandterminalaccesstothepersonnel.
Disaster recovery planEstablishadisasterrecoveryplantohelpthepersonneltodealwithemergencysituations.Rememberthatemergencysituationscanbebestavoidedbydetectingabnormalconditionsearly.Adisasterrecoveryplanshouldcovervariousdisasterscenariosanddisasterrecoveryproceduresforpersonnel.Theoperationandmaintenancepersonnelshouldalsobeabletocontactthepersonswhoarecapableofdealingwiththeprobleminquestion.Therefore,eachsiteshouldhaveanescalationplanavailablewithappropriatecontactinformation.
Escalation planAnescalationplanofferscontactlistsofinternalandexternalsupportpersonnelandservicesavailabletotackleproblems.Itshouldcontaininformationonwhotocontactandinwhatkindofsituations,forexample,airconditioning,powerback-upsystemandNokiaEmergency/HelpDesknumbers.
Preventive maintenancePerformpreventivemaintenanceroutinesonaregularbasis.Forexample,carryoutregularalarmandunitstatesurveillance.
OverviewofMulticontrollerRNCTroubleshooting TroubleshootingMulticontrollerRNC
8 DN0976768 Issue:02B
-
SafecopyingTakingregularsafecopiesofthesoftwareanddatabasesofthenetworkelementensuresthatyouhaveafunctionalcopyofthesoftwarewhichyoucanuseifthereareproblemswiththesoftwareorhardwareofthenetworkelement.Howoftenyouneedtotakesafecopiesdependsonthesizeandtypeofthenetworkelement.
Performance monitoringThepurposeofperformancemonitoringistomeasuretheoverallqualityofthesystem.Performancemonitoringcanhelpyoutodetectverylowrateorintermittentproblemsandpossibledegradationofsomepartofthesystem.
Theperformancemonitoringparametersfornetworkelementsofdifferentkindsandsizesvary.
DocumentationEstablishaprocedureforkeepingthedocumentationup-to-dateandmakesurethattheoperationandmaintenancepersonnelhaveaccesstoallrelevantexternalandinternaldocuments.
Network element diaryItisrecommendedthatyoumaintainanetworkelementdiary.Thediaryshouldbenetworkelement-specific,butyoucanstoreitintheOperationandMaintenanceCentreifthenetworkelementisnotusuallymanned.
Startfillinginthenetworkelementdiaryalreadywhenthenetworkelementisbeingsetupandinstalled.
Youarerecommendedtorecordthefollowingeventsinthenetworkelementdiary:
Hardwarechanges Softwareandhardwareupdates(forexample,changenotesandcorrection
deliveries) Essentialmodificationstotheconfigurationorroutinginthenetworkelement Safecopying Operationalfailures Anyotherrelevantinformation
Anetworkelementdiarycanprovideusefulinformationonthesystem'sperformanceinthepastandhintsonwhatmightcausethecurrentproblems.
1.2 Information sources in fault situationsUseatleastthefollowinginformationsourceswhencarryingouttroubleshooting.
AlarmsAlarmsaretheprimarysourceofinformationinmostsituationswheretroubleshootingisneeded.Alarmsareprintedoutonthealarmprinterand/orotherdevicesthatyouhavespecifiedforthenetworkelement.
TroubleshootingMulticontrollerRNC OverviewofMulticontrollerRNCTroubleshooting
Issue:02B DN0976768 9
-
Diagnosis reportsDiagnosisreportscontaindataontheplug-inunitsthatthesystemsuspectstobefaulty.Diagnosisreportsareusuallyprintedoutonthereportprinterand/orotherdevicesthatyouhavespecifiedforthenetworkelement.
Error messagesGeneralerrormessagesofthesystemtellwhythesystemcannotcarryoutatask.Theycanappearinthesupplementaryinformationfieldsofalarmsanddiagnosticprintouts,intheprintoutsofthestartingphasesmonitoredthroughaserviceterminal,andinSCLIandElementManagercommandoutputs.
Statistical reportsDifferentstatisticalreportscontainusefuldata,forexample,ontrafficonspeechandsignallingcircuits,useofservicesandloadandavailabilitymeasurements.Monitorandassessstatisticaldataregularlyasthisdatacanindicateforthcomingproblemsbeforetheyaffectthetraffic.Formoreinformation,seeperformancemanagementdocumentation.
Logs and other relevant statistical informationDifferentlogs(forexample,computerandoperatingsystemlogsandSCLIsessionreports)containusefuldatathatcanbeattachedtothefaultreportwhenyouneedNokia'helptosolveaproblem.Takethelogsfromtheunitsendingthealarmandtheunitthatistheobjectofthealarm,andifthealarmisI/Orelated,fromtheOMU.
Unit statesCheckalsotheunitstates.
1.3 Problem typesHerearesomeproblemtypesyoumayencounter.
Reproducible problemsYoucanreproducethesymptomsusingasetofactions.Reproducibleproblemscanbesolvedbynarrowingdownthepossiblecausesoftheproblemtoasinglecauseortoanumberofcausesandapplyingcorrectiveactions.Thisrequiresknowledgeofhowthesystemworksandteststoeliminatewrongconclusions.
Intermittent problemsYoucannotreproducethesymptomsconsistentlyusinganysetofactions.However,anintermittentproblemcanreproduceitselfrandomly.Insuchacase,somekindoftracingormonitoringofthesystemmayleadyoutotheoriginofthetrouble.
Ifanintermittentproblemoccursveryseldomandithasnoseriousconsequences,itmaybebesttojustignoretheproblem.Youcanalsoperformgeneralmaintenanceandseeiftheproblemdisappears.Iftheproblemoccursoccasionally,trytoconcludewhichfactorsseemtoaffectorcontributetotheappearanceoftheproblem.
OverviewofMulticontrollerRNCTroubleshooting TroubleshootingMulticontrollerRNC
10 DN0976768 Issue:02B
-
Several related or isolated troubles active at the same timeStudywhetherthesymptomsrelatetoeachotherornotandtrytoisolatetheproblemsifpossible.
1.4 Generic troubleshooting procedureDescriptionDependingonwhetheryouhaveacontractwithNokiaontheoperationandmaintenanceofthenetwork(orsomeotheragreement),theactionsthatyouneedtotakeinafaultsituationmaybedifferentfromtheonespresentedhere.
WhenyoususpectthataNokianetworkelementisnotperformingasitshould,carryoutatleastthefollowingchecks.
SymptomsANokianetworkelementisnotperformingasitshould.
Recovery proceduresTroubleshooting process before calling Nokia Solutions andNetworks help desk
Steps
1 Evaluate how serious the consequences of the trouble areIftheproblemhasveryseriousconsequences,youmayhavetocallforexperthelporapplyanemergencyplanimmediately.
2 Analyse the situation where the problem or failure first appearedConsiderthefollowingbeforeyoucarryoutanycorrectiveactions.
Whatistheproblem? Whereistheproblem? Whendidtheproblemoccur? Whatwerethecircumstancesthatledtotheproblem? Whatistheimpactoftheproblem(forexample,towhatextentdoesthefault
affecttheendcustomers)? Whoisresponsiblefortakingcareoftheproblem?
3 Try to eliminate the possibility of human error
Forexample,recentchangesinthesoftwareorhardwareconfigurationofthesystem(forexample,circuitsaddedorequippingchanged)arepossiblesourcesofproblems.Thechangesmayhavebeencarriedoutincorrectly.Checktheconfigurationparameters,physicalconnections,strappings,plug-inunitsandsoon,verycarefully.
TroubleshootingMulticontrollerRNC OverviewofMulticontrollerRNCTroubleshooting
Issue:02B DN0976768 11
-
show functional-unit unit-info Humanerrorisaverycommoncauseofproblemstherefore,checkand
double-checkeverypossibleproblemsource.Typehistory.
Afailurecanalsooccurspontaneously(forexample,theremoteendsystemmayhaveproblemsoraservicebreakdown,oraplug-inunitmayfailduetoageing).Checkthealarms,clearcodes,unitandlinkstatesandlogsasdescribedbelow.
4 Make an accurate description of the symptomsYoumaynotbeabletosolvetheproblemyourself.Adetaileddescriptionofthesituationwherethesymptomsoccurredcanhelpanexpertsolvetheproblem.Gatheralsodataonthefailureevent.Asymptomdescriptionshouldcontainallthebasicfacts,suchas:
Date,nameofthepersonwhodetectedthetrouble,phonenumberande-mailaddress
Detailsofthesystem;forexample,whatequipmentandsoftwareisinuse Descriptionofthesymptoms(alarms,errormessages,clearcodes,faultystates
oftheunitsandlinksandsoon) Anyotherrelevantinformation,forexample,logandmessagemonitoringfiles.All
datamaybevaluableeveniftheyseemirrelevantatthetime.Storethisinformationpreferablyinelectronicformat.
5 Check and analyse the alarm situation
Checkthealarmsthatarecurrentlyon.Youarerecommendedtoalsostudythealarmhistory.Displaythealarmhistorysothatitshowsallalarmeventsfromthetimeperiodwhichstartsonehourbeforetheoccurrenceoftheproblemsituation,andendsonehouraftertheproblemsituationwasover.Youshoulddisplaythealarmhistory.# show alarm activeThesystemmaysetanalarmandcancelitimmediately.Youcanfindthesealarmsinthealarmhistory.Alarmsbehavinginthiswaymayindicatethatsomepartofthesystemisabouttobreakdownoritsfunctionalityhasbeenreduced.
Checkalsothediagnosticreports.
6 Option Description
If thefaultcanbelocatedbasedonthealarmsituation
Then Carry out the appropriate maintenance actions
Ifyouhaveendedupwithmorethanoneprobableandpossiblecauseforthetrouble,changeonlyonethingatatimeotherwiseyoucannotbesureofwhichchangecorrectedthefailureorproblem.
Rememberthatrandomactionscanmakeproblemsworse.Generally,youshouldnottakeanyradicalcorrectiveactionsifyouarenotsurewhattheproblemisandwhattheconsequencesofthecorrectiveactionsare.Losingtrafficbecauseofincorrectactionsisnotwhatyouwant.
OverviewofMulticontrollerRNCTroubleshooting TroubleshootingMulticontrollerRNC
12 DN0976768 Issue:02B
-
7 Option Description
If youcannotlocatethefaultbasedsolelyonthealarmsituation
Then Try to narrow down the possible problem source
Analyseandcategorisesymptomsandlistpossiblecausesforthesymptoms.Sometimestherecanbeseveralrelatedorisolatedtroublesactiveatthesametime.Studywhetherthesymptomsrelatetoeachotherornot.Prioritisesymptomsandcollectfurtherfactsifneeded.
Basedontestsandyourknowledgeofthesystem,eliminatesymptomsthatarenotrelevanttothetroubleyouaretryingtosolve.Thiswayyoucanfocusonsymptomsandcausesthataremorelikelytoproduceasolutiontotheproblem.Examinewhatworksandwhatdoesnot.
However,eventhoughyoumaynotbeabletoanalysewhatthecauseofthetroubleis,youcancarryoutgeneralmaintenancetoeliminatesometrivialcausesoftroubles,suchasloosecablesandbadconnections.
8 Use measurements to trace any abnormal trendsFormoreinformation,seeperformancemanagementdocumentation.
9 Check the states of units, links and circuitsCheckunitstates.show functional-unit unit-info
10 Fill in a problem report if neededDescribetheproblemindetailintheproblemreport.Includeallrelevantinformationthatyouhaveavailablefromtheproblemsituationanddescribealsothecorrectivemeasuresthatyouhavecarriedoutaftertheproblemoccured.
TroubleshootingMulticontrollerRNC OverviewofMulticontrollerRNCTroubleshooting
Issue:02B DN0976768 13
-
2 Reporting problem to NSNProblemreportsareusedtocommunicateproblemsandfailurestoservicepersonnel.Reportonlyonefaultinoneproblemreport.Includetheattachmentsinthecompressedformat.
Tomaketheinvestigationofaproblemfaster,includethefollowinginformationintheproblemreport:
atitlethatgivesabriefdescriptionoftheproblem aclearandexactdescriptionoftheproblemitself problembackgroundinformation:
softwareandhardwarereleases situationinthebeginning,forexample,thefirstsymptomsoftheproblem situationaftertheproblemoccured operationsyoumadewhichpossiblycausedthefailure,forexample:hardware,
software,parameter,featureorconfigurationchanges,includingopertionsintransportnetworkandthirdpartyequipment
describeiftheproblemcanbereproducedandwhatactionsarerequiredtoreproducethisproblem
troubleshootingandrecoveryactionsthatyoumade
symptomdatareportsMostoftheMulticontrollerRNCconfigurationdataandlogsessentialforinvestigationoftheproblemcanbecollectedwiththestandardsymptompluginreportgroup-RNCandsubreport-MessageMonitor.ThisdataisalwayscollectedifProblemReportissubmittedtoNSN.TocollecttheMulticontrollerRNCstandardsymptomdata,executethefollowingSCLIcommands:
symptomdataandmessagemonitoring:save symptom-report name include group-RNCinclude subreport-MessageMonitor
basicsymptomdata:save symptom-report name include group-RNC
Formoreinstructionsonthestandardsymptomsdatacollectionframework,seeStandard symptoms data collection.Thedatamustbecollectedassoonaspossibleafteranabnormalsituationhastakenplaceandbeforeanyrecoveryactionisperformed,suchasMulticontrollerRNCrestartsorreplacinghardware.Thisisimportantbecausetheinformationstoredabouttheproblem(forexample,blackboxofacertainunit)maygetoverwrittenintheprocessoftimeorbelostbecauseofrecoveryactions.Thestandardsymptomreportgroup-RNCisexpectedtobecompletedinaround15minutesdependingonMulticontrollerRNCconfiguration.Recoveryactionscanbestarted,ifneeded,assoonassymptomreportgenerationiscompleted.Acopyofthesymptomreportcanbetransferredtolocalmachinelaterataconvenienttime.Whenyousendoutaproblemreport,makesurethatallthepossibleattachmentsareincludedintheproblemreport,toavoidunnecessaryinformationrequests.ForOMSsymptomdatacollection,seeTroubleshooting OMSdocument.
Ifpossible,includeadditionalproblemspecificsymptomdata.Forexample,problemspecificmessagemonitoringlogs,networkanalyzerinterfacetraces.
ReportingproblemtoNSN TroubleshootingMulticontrollerRNC
14 DN0976768 Issue:02B
-
NetActKPI/counterreportincasetheproblemconcernscertainKPI/counter Inamultivendorenvironment,includedetailedinformationontheotherthirdparty
products. NSNCaseType;Emergency,TroubleResolutionorTechnicalQuery.Fordetails,see
#c105225791/table_bjv_g12_rp NSNProblemPriority:Major,MediumorMinor.Fordetails,see
#c105225791/table_ndz_pd2_rp.
Table 1 NSNCaseType
NSN Case Type
NSN Definition
Examples
Emergency ThiscasetypeisforTotalandPartial outages.
Availablepriority-Critical.
ThiscasetypeisforcasesunderEmergencySupportservice.
Total Outage
Totallossofvoiceanddatatrafficcapability.Anunscheduledeventmustbelongerthan15seconds.
Partial Outage
Lossofgreaterthan10%oftheprovisionedcapacityfororiginationand/orterminationofvoiceand/ordatatraffic.Totallossofoneormorecriticalservices.Anunscheduledeventmustbelongerthan15seconds.
Total Outage:
AllIublinksaredown. AllWCELsareinincorrectstate. RNCspontaneousrestart(unplanned). RNCsystemrestart(planned),alllinks
aredown.
Partial Outage:
Lossof10%ormoreofvoiceordatatrafficcapacityforatleast15seconds.
Spontaneousrestartsofactivecomputerunit(s)whenthereisnoredundantunitavailabletotakeovertheservicesprovidedbytherestartedone.
Oneormorenetworkinterfacesaredownforover15seconds.
TotallossofsubscriberrelatedRNCfunctionality(ISHO,HSDPA,HSUPA,Internetbrowsing,andsoon)
RNCoperationandmaintenancefromNetActistotallylost.
Emergencycallsnotpossible(forexample,112or991).
TroubleResolution
Thiscasetypeisforproblemswhichcausedbyasuspectedoranidentifieddefect.
Availableproblempriorities-Major,Medium,andMinor.
SWdefectisidentifiedorsuspected.Acorrectionwillberequiredifthedefectisconfirmed.
HWdesigndefectissuspected.Acorrection(HWretrofitorSWchange)willberequiredifthedefectisconfirmed.
Anerrororomissionintheproducttechnicalliteratureissuspected.Adocumentationcorrectionwillberequiredifthedefectisconfirmed.
Technical Query Thiscasetypeisfortechnicalquestionsregardingdailynetworkoperationsandmaintenanceissues.
Technicalquestionsonproceduresorfeaturesthatarecoveredinthedocumentationshippedwiththeproduct.
TroubleshootingMulticontrollerRNC ReportingproblemtoNSN
Issue:02B DN0976768 15
-
Table 1 NSNCaseType(Cont.)
NSN Case Type
NSN Definition
Examples
AvailableproblemprioritiesMajorandMedium.
InformationrequestsonNSNproductthatwillbeusedtohelpinterfacethisproductwithathirdpartyproduct.
Table 2 NSNProblemPriority
NSN ProblemPriority
NSN Definition
Examples
Critical ProblemsunderEmergencySupportservice.
N/A
Major OnlyTotalorPartialoutageswhicharenotavoidablewithaworkaroundsolution.
N/A
Medium Lossoflessthan10%oftheprovisionedcapacityfororiginationand/orterminationofvoiceand/ordatatraffic.
TotalorPartialoutagesavoidablewithaworkaroundsolution.
Partiallossofoneormorecriticalservices.
totalormorethan10%lossofvoiceordatatrafficcapacityforatleast15seconds,avoidablewithworkaround
singlerestartofcomputerunits configurationchanges(RNW,
HWandSW)arenotworking activationofafeaturefails singleperformance
measurementisnotworkingcompletely
partiallossofsubscriberrelatedRNCfunctionality(ISHO,HSDPA,HSUPA,Internetbrowsing,andsoon)
partiallossofalarmmanagementofobjects(BTS,functionalunits)
problemswithback-up majorerrorsindocumentation,
forexample,analarmorparameterdescriptionismissingfromdocumentation
vitaldocumentsaremissingfromthedocumentationlibrary
Minor Minorfaultnotaffectingoperationorservicequality
failuresnotseriouslyaffectingtraffic
errorsincommandlinesyntaxoroutput
minorerrorsindocumentation
ReportingproblemtoNSN TroubleshootingMulticontrollerRNC
16 DN0976768 Issue:02B
-
MostoftheMulticontrollerRNCconfigurationdataandlogsessentialforprobleminvestigationcanbecollectedwiththestandardsymptompluginreportgroup-RNCandsubreport-MessageMonitor.ThisdataisalwayscollectedifProblemReportissubmittedtoNokiaSolutionsandNetworks.
TocollecttheMulticontrollerRNCstandardsymptomdata,executethefollowingSCLIcommands:
symptomdataandmessagemonitoring:save symptom-report name include group-RNCinclude subreport-MessageMonitor
basicsymptomdata:save symptom-report name include group-RNC
Formoreinstructionsonthestandardsymptomsdatacollectionframework,seeStandard symptoms data collection.
ThedatamustbecollectedassoonaspossibleafteranabnormalsituationhastakenplaceandbeforeanyrecoveryactionisperformedsuchasMulticontrollerRNCrestartsorreplacinghardware.Thisisimportantbecausetheinformationstoredabouttheproblemmaygetoverwrittenintheprocessoftimeorlostduetorecoveryactions.
Tosaveorcollectthebasicsymptomreportgroup-RNC,enterthefollowingcommand:
save symptom-report name include group-RNC includesubreport-MessageMonitor
_nokadmin@CFPU-0 [RNC-37] > save symptom-report name walle20141016 include group-RNC include subreport-MessageMonitorCFPU-0@RNC-37 [2014-10-16 13:48:57 +0200]Mode : Normal modeMax. execution time limit : 30 minutesMax. size of each part : 100 MBMax. size of full report : 350 MBReportName FileName Chunk Subreport Store---------- --------- ------ -------- -----walle20141016 /stdsymp/walle20141016_0.tar Part 1 ipmgmt CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 MessageMonitor CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncipconfig CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncrnw CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncsignaling CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rnchw CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncinfo CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncmon CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncuplane CFPU-0walle20141016 /stdsymp/walle20141016_0.tar Part 1 rnchas CFPU-0
TroubleshootingMulticontrollerRNC ReportingproblemtoNSN
Issue:02B DN0976768 17
-
walle20141016 /stdsymp/walle20141016_0.tar Part 1 rncalarm CFPU-0
Successfully collected symptom report data for 11 out of 11 subreports
Thestandardsymptomreportgroup-RNCisexpectedtobecompletedinaround15minutesdependingonMulticontrollerRNCconfiguration.
g Whenusingthe"to"and"from"optionswiththestandardsymptomreportgroup-RNC,thesyslog,syslog.1filesarenotfilteredaccordingtothetimerangeprovided.Thetimerangeisnotapplicabletosyslogandsyslog.1files.
Recoveryactionscanbestarted,ifneeded,assoonassymptomreportgenerationiscompleted.Copyofthesymptomreportcanbetransferredtolocalmachinelaterataconvenienttime.
Whenyousendoutaproblemreport,makesurethatallthepossibleattachmentsareincludedintheproblemreport,toavoidunnecessaryinformationrequests.
ReportingproblemtoNSN TroubleshootingMulticontrollerRNC
18 DN0976768 Issue:02B
-
3 Symptoms data collection
3.1 Standard symptoms reportStandardsymptomreportisaframeworkusedtocollectsymptomdatafromMulticontrollerRNCtosupporttroubleshootingofproblems.Theframeworkcollectsstandardsymptomsdatabyrunningindividualormultipleplugins.Someofthepluginsareputintogroups,thusitispossibletocollectthesymptomdataonthegroupleveltoo.Theframeworkallowseasyenhancementwithadditionalplugins,iftheavailablestandardpluginsarenotsufficient.Thecollectedsymptomsdataisstoredin/mnt/backup/stdsympdirectory.
Thefollowingtableprovidesdetailsaboutstandardpluginsandtheirfunctionality:
Table 3 Supportedplugins
Plugin Name
Description
group-RNC ItcollectsMulticontrollerRNCconfigurationdataandlogsessentialforinvestigationoftheproblem.
w NOTICE:ThisdatamustalwaysbecollectedifProblemReportissubmittedtoNokia.Thisisthereportgroupcontainingthefollowingsub-reports:subreport-rnchw subreport-rncinfo subreport-rncipconfig subreport-rncsignaling subreport-rncrnw subreport-rnchas subreport-rncalarmsubreport-rncuplanesubreport-rncmon
group-Backupandrestore
Itcollectsthefollowingbackupandrestore-relatedlogs:
subreport-clusterinfo subreport-syslogsubreport-corefiles subreport-processinfosubreport-clusterstatus
group-Networking Itcollectsthefollowinggeneralstatisticsofthenetwork:
subreport-clusterinfo subreport-osinfosubreport-processinfo subreport-clusterstatussubreport-syslog subreport-corefilessubreport-alarmsinfo subreport-debugsubreport-routing subreport-ipmgmt
group-SignalingProtocol
Itcollectsthefollowingreportsrelatedtothesignalingprotocolsandthesignalingnetworkmanager:
subreport-signalingSS7 subreport-signalingSCCPsubreport-signalingNetManager subreport-signalingPacketCaptureSctp subreport-signalingPacketCaptureSccpUser subreport-syslog subreport-corefiles subreport-alarmsinfo subreport-debug
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 19
-
Table 3 Supportedplugins(Cont.)
Plugin Name
Description
group-IpalPlatform Thisisthereportgroupcontainingthefollowingsub-reports:subreport-IpalSignalling subreport-IpalCallManagement subreport-IpalCommonsubreport-IpalTransportsubreport-IpalBasicServices
subreport-alarmsinfo Itcollectsalarmdataforaselectedperiodoftime.
subreport-clusterinfo Itcollectsclusterandnodeinformation.
subreport-clusterstatus
Itcollectsinformationaboutthestatusofcluster,nodes,recoverygroupsorunits.
subreport-corefiles Itcollectscoredataavailablewithinthe/crashdirectoryoftheactivenode.
subreport-databaseinfo
ItcollectsthedatabaseinformationsuchasDatabaseDeployed,DiskSpace,PostgresConfigurationfiles,Processlists,andEnterpriseDatabasedetails.
subreport-debug Itcollectsdebugdatacollectedfrommasterdebuglog.
subreport-fastpath Itcollectsinformationfromnodeswhichhave6windipstack.Forexample,ngctl.dat,fpdebug,fpstat.
subreport-ipmgmt ItcollectstheIPnetworkingconfigurationdata.
subreport-licinfo Itcollectslicensemanagementrelatedinformation.
subreport-licdebug Itcollectslicensemanagementrelatedinformation.
subreport-ipsec ItcollectsinformationaboutIKEtemplate,IPsecrules,VPNs,andconfigurationfiles.
subreport-MessageMonitor
Itcollectstwomessagelogsforthefollowingtraffic:
callscenarios operationandmaintenace(O&M)scenarios
Messagemonitoringparametersarestoredinthemsgmon.ini fileunder/opt/nsn/SS_SysReport/stdsymp/plugins/path.
Bydefault,thebufferssizeissetto8MB(0x8FFFFF),anddurationofthemonitoringis30seconds.Themessagemonitoringisperformedintworuns,thefirstisforcallscenarios,andthesecondisforO&Mscenarios.
Tomodifythesettingsthatareusedtocapturemessagelogswithsubreport-MessageMonitor,proceedasfollows:
Symptomsdatacollection TroubleshootingMulticontrollerRNC
20 DN0976768 Issue:02B
-
Table 3 Supportedplugins(Cont.)
Plugin Name
Description
1. Switchto rootuserandbashshell(typeexitlaterontoreturntofsclish).set user username rootProvidethepasswordforroot(defaultisroot).
2. Copythemsgmon.ini fileto/mnt/backup/shareandmodifyit.cp /opt/nsn/SS_SysReport/ stdsymp/plugins/msgmon.ini /mnt/backup/ share/msgmon.ini
Torestoredefaultsettings,deletethemsgmon.ini filefrom/mnt/backup/sharedirectory.
subreport-osinfo Itcollectsinformationaboutoperatingsystemversion,clusteruptime,diskusage,sharedmemory,buildlabel,imagevariants,currentsnapshotname,andcompleteRPMinformation.
subreport-processinfo Itcollectsprocess-relatedinformation.
subreport-routing Itcollectsthecurrentroutinginformation,IPaddress,androutinginstances.
subreport-signalingNetManager
Itcollectsthering-basedbuffertracefilescreatedfornet-managerinthefollowinglocation:
/trace
subreport-signalingPacketCaptureSccpUser
Itcollectsthetcpdumppcapfilescreatedforcapturingtheflowofpacketsbetweensccpandsccp-user.
subreport-signalingPacketCaptureSctp
ItcollectsthetcpdumppcapfilescreatedforcapturingtheflowofpacketsattheSCTPlevel.
subreport-signalingSCCP
ItcollectsthetracefilesforSCCPsubsystem,ldapconfiguration,andoutputofSCLIshowcommandatthestartingandstoppingtimeofsignalingdiagnostics.
subreport-signalingSS7
ItcollectsthetracefilesforSS7subsystem,ldapconfiguration,andoutputofSCLIshowcommandatthestartingandstoppingtimeofsignalingdiagnostics.
subreport-syslog Itcollectsrelevantdatafromsyslog.
subreport-techservice Itcollectstheoutputofsomesoftwarecommandslikefsswcli,unameandsoon,andalsosomescriptsrequiredbytechservicepeople.
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 21
-
Table 3 Supportedplugins(Cont.)
Plugin Name
Description
subreport-tracemgmt Itcollectstracing-relatedinformationusefulfordebuggingpurposessuchas:
buildversion tracing-relatedrpmversions configurationfiles LDAPfragmentsundertracing featuresrelatedtotracingLDAPfragment contentsoffptl_adminsharedlibrary listoftracefiles listofpluginlibraries processstatesofNodeTraceManager(NTM)and
ClusterTraceManager(CTM) listofbuffers/adminbuffers diskusageoftracing-relatedfilesystems consistencyoftracingconfiguration
subreport-tracesnapshot
Itcollectsthesnapshotfordefault_platformbufferintheCLA-0node.
subreport-IpalBasicServices
Itcollectsdataondistributedcomputingservices,suchasfunctionalunitstates,nameservicesinformation,andfeaturemanagementdata.
subreport-IpalCallManagement
Itcollectsinformationaboutcallmanagementanduserplanemanagement,suchascalldetails,connectioninformation,anduserplaneserviceallocation.
subreport-IpalCommon Itcollectsgeneralinformationfortroubleshooting.Thisincludessoftwarebuildversion,hardwareconfigurationinformation,recoveryunits/recoverygroups/nodeconfiguration,andbasiccounters.
subreport-IpalSignalling
Itcollectsinformationaboutsignalingconnections,includingallNBAP/SCTPlinkinformationandallSIGTRANlinkinformation.
subreport-IpalTransport
ItcollectsinformationabouttraffictransportthatishandledbyEIPU,suchastransportforwardingtable,GTPtunnelIDinformation,IPBRconfiguration.
subreport-rnchw ItcollectsinformationonMulticontrollerRNCfunctionalunits.
subreport-rncinfo Itcollectsinformationonthesoftwarebuilds,diskspaceusage,availablesnapshots,recoverygroupsstates,PRFILEparameters,andsoon.
Thisincludesalsorelevantinformationfromthesyslog.
Symptomsdatacollection TroubleshootingMulticontrollerRNC
22 DN0976768 Issue:02B
-
Table 3 Supportedplugins(Cont.)
Plugin Name
Description
subreport-rncipconfig ItcollectsinformationontheIP-relatedconfiguration,thatis,IPinterfaces,IPaddresses,routing,userplaneresources(ipbrandiproobjects),andsoon.
subreport-rncsignaling
ItcollectsinformationonSIGTRAN-relatedconfiguration,thatis,statusoflocalandremotesubsystems,localapplicationserver(s),SCTPassociations,SCTPprofiles,M3UAlimits,SSPfiltertimers,andsoon.
subreport-rncrnw ItcollectsinformationontheRNWstatus,thatisBTSOMconnectionsstatus,NBAPconnectionstatus,WBTSandWCELobjectsstatus,IPNBobjectstatus,IUCSandIUCSIPobjectsstatusesIUPSandIUPSIPobjectstatuses.
subreport-rnchas Itcollectsinformationonthenode,recoverygroups,andrecoveryunits.
ThisincludesalsoHASrelatedinformationfromthesyslog.
subreport-rncalarm Itcollectsinformationaboutactivealarms,alarmshistory,andsoon.
subreport-rncuplane Itcollectsinformationontheuserplaneconfigurationandstatus.
subreport-rncmon ItcollectsinformationoncontrolplaneUEspecificmonitoringforabnormalcallsaspartofHPLloggingfunctionality.
3.2 Collecting standard symptom reportsPurpose
Followthisproceduretosave(collect)thestandardsymptomreports.
Steps
1 Save or collect the standard symptom reports.Tosave(collect)thestandardsymptomreport,enterthefollowingcommand:save symptom-report name Thefullsyntaxofthecommandis:save symptom-report name {[exclude [include ] [quick-mode ] [timeout ] [report-max-size ] [single-file-max-size ]} from date-time to date-time
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 23
-
w NOTICE:Itisrecommendedtousenormal-modeforthesymptomsdatacollectionforgroup-RNC.Itisbecause,forgroup-RNC,thedatacollectioninnormal-modecompletesinthetimeallocatedforquick-mode.
Simultaneousmultiplesessionsofsymptomreportsdatacollectionwiththesamereportnameisnotallowed.Ifareportfilealreadyexists,collectingstandardsymptomreportcannotbeinitiatedwiththesamereportname.Whenusingthe"from date-time"and"to date-time"optionswiththestandardsymptomreportgroup-RNC,thesyslog andsyslog.1filesarenotfilteredaccordingtothetimerangeprovided.Thetimerangeisnotapplicabletosyslogandsyslog.1files.Symptomreportsthatareolderthan10daysareremovedfromthefilesystemautomatically.Theparametersinthecurlybracketsarenotallowedinthecommandsyntaxafterthe "from date-time"and"to date-time"parameters.Whenretrievingsymptomreportsfromthepreviousyear,thevalueofthefromdate-time andto date-time parametersdonotgobackfurtherthan333daysfromthecurrentdate.Thefollowingmessagesaredisplayedaccordingtothetriggeringscenariosencounteredduringcommandexecution:
Usecase1:Thedatespecifiedinthefrom date-time parameterisfromthepreviousyear.Messagedisplayed:From/To dates cannot be older than 333 days from current dateDate has to be given in specified format, refer usage
Usecase2:Thedatespecifiedinthefrom date-time parameterisfromthepreviousyearwhilethedatespecifiedintheto date-time parameterisinthecurrentyear.Messagedisplayed:From/To dates cannot be older than 333 days from current dateDate has to be given in specified format, refer usage
Iftheuseronlyspecifiedthefrom date-time parameter,oriftheuseronlyspecifiedtheto date-time parameter,thentheonethatwasnotspecifiediscalculatedbyadefaultoffsetof10days.Forthedescriptionoftheparametersofthesave symptom-report command,seeTable4:Parametersrelatedtosavesymptom-reportcommand.
Table 4 Parametersrelatedtosave symptom-reportcommand
Parameter
Description
name Thismandatoryparametercreatesasingleormultiple(itdependsonsubsequentoptions)tarfile(s)withtheprovidedreport-name.Thereport(s)arestoredunderthedirectory/mnt/backup/stdsympoftheMulticontrollerRNC.Thereportfile(s)generatedcontainsubreport(s)thatarecompressedarchivefile(s)(with.tar.gzextension)containingoneormorefile(s).
Symptomsdatacollection TroubleshootingMulticontrollerRNC
24 DN0976768 Issue:02B
-
Table 4 Parametersrelatedtosave symptom-reportcommand(Cont.)
Parameter
Description
Inthereportfile(s),thereisalsoonetextfilethatsummarizesexecutedcommands,theirexecutionstatusandtime.Itsformatis:__summary.txt
Specialcharactersforreport-namearenotallowed.Numbersarenotallowedasfirstcharacter.Thereportnamelengthmustnotexceed25characters.
[exclude ] Ifthisoptionalparameterisused,thestandardsymptomreportoftheselectedpluginandgrouparenotcollected.
[include ] Ifthisoptionalparameterisused,thestandardsymptomreportcanbecollectedfromaspecifiedpluginorgroup.
[quick-mode ] Thisoptionalparameterisusedinemergenciestogatherimportantandrelevantinformationratherthanhavingallinformationavailable.Whenquickmodeisenabled,thesymptomsubreportsautomaticallyincludedatathatiscriticalandquicklyretrieved.If"yes"isselected,eachsubreportincludesonlydatathatisquicktocollect.If"no"isselected,eachsubreportincludesalldatathatisneededinasupportcase.Thedefaultvalueforthisoptionis"no".
[single-file-max-size]
Thisoptionalparametercreatesthefinalreportinchunksorpiecesoftarfile,eachwithamaximumsizespecifiedastheargument.Eachchunkisatarfilewhichcontainsreportsthatareindependentlyanalyzable.Reportproducedbysinglesubreportisnotsplitevenifthereportsizeexceedsthespecifiedmaximumsize.Thisimpliesthatsizeofasinglefilemaysometimesbebiggerthanthespecifiedmaximumsinglefilesize.Thedefaultvalueforthisoptionis10MB.Asizeof0meansnolimit.
[report-max-size ]
Thisoptionalparameteracceptsthemaximumallowedsizelimit(inMB)forthedatagenerated.Whenthesizeofthedatageneratedreachesthespecifiedlimit,theframeworkstopsthedatacollectionafterthe
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 25
-
Table 4 Parametersrelatedtosave symptom-reportcommand(Cont.)
Parameter
Description
currentlyexecutingsubreportcompletesitsexecution.Sincethedatagenerationisnotabruptlystoppeduponreachingthesizelimit,theresultingsizeofthesymptomdatacollectedmaystillexceedthespecifiedlimit.Thedefaultvalueforreport-max-sizeis30MB.Asizeof0meansnolimit.
[timeout ] Thisoptionalparameterspecifiesthetimeoutvalueforsymptomdatacollection.Whentimeoutoccurs,thepluginthatiscurrentlybeingprocessedfailstobecollected,andallthequeuedplugins(ifany)areskipped.Timeoutoptionisstrictlyfollowedexceptwhenpluginsgointokernelmode.Inkernelmode,delaysoccurbecausethesignalsarenotprocesseduntilitlogsoutofthekernelmode.Permittedrangeis0-60.Avalueof0meansthereisnotimeout.Timeoutoptionallowsyoutostopthesymptomdatacollectionafterthespecifiedtimeoutvalueexpires.Thedefaulttimeoutis30minutes.
from date-time
Thisoptionalparametersavessymptomreportuptoaparticulardate.AccepteddateformatsareYYYY.MM.DD-HH:MM:SSorDD.MM.YYYY-HH:MM:SS.Whentryingtocollectsymptomreportsfromthepreviousyear,makesuretheparameterdoesnotgobackfurtherthan333daysfromthecurrentdate.
to date-time Thisoptionalparameterspecifiesthedatetillwhichthestandardsymptomreportmustbecollected.TheaccepteddateformatsareYYYY.MM.DD-HH:MM:SSorDD.MM.YYYY-HH:MM:SS.Whentryingtocollectsymptomreportsfromthepreviousyear,makesuretheparameterdoesnotgobackfurtherthan333daysfromthecurrentdate.
Examples
a) Tocollectthebasicsymptomreportwithaspecificname(RNC311),andwithasubreportgroup(group-RNC),executethefollowingcommand:save symptom-report name RNC311 include group-RNC
Symptomsdatacollection TroubleshootingMulticontrollerRNC
26 DN0976768 Issue:02B
-
b) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwithamessagemonitoring(subreport-MessageMonitor),executethefollowingcommand:save symptom-report name RNC311 include group-RNC include subreport-MessageMonitor
c) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andfromthegivendate(2013.08.05-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00
d) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),fromthegivendate(2013.08.05-09:00:00)toaparticulardate(2013.08.06-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC from date-time 2013.08.05-09:00:00 to date-time 2013.08.06-09:00:00
e) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),fromthetimetheMulticontrollerRNCwascommissionedtoaparticulardate(2013.08.06-09:00:00),executethefollowingcommand:save symptom-report name RNC311 include group-RNC to date-time 2013.08.06-09:00:00
f) Tocollectallthestandardsymptomreportswithaspecificname(RNC311),executethefollowingcommand:save symptom-report name RNC311
g) Tocollectallthestandardsymptomreportswithaspecificname(RNC311ipconifg),andwiththespecificsubreport(rncipconifg),executethefollowingcommand:save symptom-report name RNC311ipconfig include subreport- rncipconfig
h) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththelimitedsizeofasinglereportfile(5MB),executethefollowingcommand:save symptom-report name RNC311 include group-RNC single-file-max- size 5
i) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththemaximumsizeofthereport(10MB),executethefollowingcommand:save symptom-report name RNC311 include group-RNC report-max-size 10
j) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwithtimeoutfordatacollection(10minutes),executethefollowingcommand:save symptom-report name RNC311 include group-RNC timeout 10
k) Tocollectthestandardsymptomreportwithaspecificname(RNC311),withasubreportgroup(group-RNC),andwiththesinglesubreportexcludedfromthegroup(subreport-rnchw),executethefollowingcommand:save symptom-report name RNC311 include group-RNC exclude subreport-rnchw
Expected outcome:Thefilesarestoredasindirectory/mnt/backup/stdsymp,withthenameusedinthesave symptom-reportcommand.
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 27
-
3.3 Listing the standard symptom reportsPurpose
Followthisproceduretolistthestandardsymptomreportoffiles,pluginsorgroupsavailableontheMulticontrollerRNC.
Before you start
Defaultsystemusernameandpasswordare"_nokadmin"/"system".
Steps
1 List the symptom reports.Tolistallthestandardsymptomreports,enterthefollowingcommand:show symptom-report all
Sample OutputCFPU-0@RNC-311 [2013-08-06 07:52:43 +0200]ReportName FileName Chunk/Total Store---------- -------- -------- -----AllRNC-311 /stdsymp/AllRNC-311_0.tar part 1/2 CFPU-0AllRNC-311 /stdsymp/AllRNC-311_1.tar part 2/2 CFPU-0MessageMonitor /stdsymp/MessageMonitor_0.tar part 1/1 CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 CFPU-0rncmon /stdsymp/rncmon_0.tar part 1/1 CFPU-0RoutingFails /stdsymp/RoutingFails_0.tar part 1/1 CFPU-0history /stdsymp/history_0.tar part 1/1 CFPU-0
w NOTICE:TheChunk/Totalfielddisplaysthenumberofchunkspresentfortheparticularreport.Thisassiststheuserinlearningwhetherthefilesofthereporthavebeenlistedproperly.
2 List the detailed contents of a particular symptom report.show symptom-report name
ExampleTolisttheparticularstandardsymptomreport(RNC-311),enterthefollowingcommand:show symptom-report name RNC-311
Symptomsdatacollection TroubleshootingMulticontrollerRNC
28 DN0976768 Issue:02B
-
Sample OutputCFPU-0@RNC-311 [2013-08-06 07:57:14 +0200]ReportName FileName Chunk/Total Subreport Store---------- --------- --------- ----- -----RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rnchw CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncinfo CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncipconfig CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncsignaling CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncrnw CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rnchas CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncalarm CFPU-0RNC-311 /stdsymp/RNC-311_0.tar part 1/1 rncuplane CFPU-0
3 Display the list of available plugins.ExampleTodisplaythelistofavailableplugins,enterthefollowingcommand:show symptom-report plugin-list
4 List the group of plugins based on the group names.ExampleTolistthegroupofpluginsbasedonthegroupnames(group-RNC),enterthefollowingcommand:show symptom-report group group-RNC
3.4 Copying standard symptoms reports to remotemachinePurpose
FollowthisproceduretocopystandardsymptomsreportsfromMulticontrollerRNClocalstoragetoremotemachine.
Before you start
Defaultsystemusernameandpasswordare"_nokadmin"/"system".
Expected outcome
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 29
-
Thestandardsymptomsreportsaresuccessfullycopiedtoexternalstorage.
Steps
1 Switch to bash shell (type exit later on to return to fsclish).shell bash full
Confirmthecommandbypressing"y".
2 Use SCP to copy standard symptoms report to remote machine.Notethatthereportfilesarestoredin /mnt/backup/stdsympdirectory.scp /mnt/backup/stdsymp/.tar @:/Notethatyoucanusewildcardsymbol"*"inthetocopyallthereportfilesfrom/mnt/backup/stdsympdirectory.IfthesameremotemachineisusedtostorethereportsfromdifferentMulticontrollerRNCs,itisrecommendedtoincludespecificMulticontrollerRNCnameandidentifierinthefoldernametodifferentiatethereports.
w NOTICE:ThereportfilescanalsobecopieddirectlyfromtheMulticontrollerRNCtoexternalmachineusingSFTPorSCPprotocol.Thereisadefaultaccount_nokadminwhichprovidestheaccesstoMulticontrollerRNCfilesystem.Thedefaultpasswordfor_nokadminuseris"system.Thereportsarestoredunder/stdsympdirectory.
3.5 Deleting the standard symptom reportsPurpose
Followthisproceduretodeletethestandardsymptomreports.
Before you start
Defaultsystemusernameandpasswordare"_nokadmin"/"system".
Steps
1 Delete the particular standard symptom report.Todeleteaparticularstandardsymptomreport,enterthefollowingcommand:delete symptom-report name
ExampleTodeleteaparticularstandardsymptomreportwithnameRNC311,enterthefollowingcommand:delete symptom-report name RNC311
Sample Output:CFPU-0@RNC-311 [2013-08-06 08:29:10 +0200]ReportName FileName Store
Symptomsdatacollection TroubleshootingMulticontrollerRNC
30 DN0976768 Issue:02B
-
---------- -------- -----RNC311 /stdsymp/RNC311_0.tar CFPU-0
2 Delete all the standard symptoms reports.Todeleteallthestandardsymptomreports,executethefollowingcommand:delete symptom-report all
Sample Output:CFPU-0@RNC-311 [2013-08-06 08:32:00 +0200]ReportName FileName Store---------- -------- -----AllRNC311 /stdsymp/AllRNC311_0.tar CFPU-0AllRNC311 /stdsymp/AllRNC311_1.tar CFPU-0MessageMonitor /stdsymp/MessageMonitor_0.tar CFPU- 0RoutingFails /stdsymp/RoutingFails_0.tar CFPU- 0history /stdsymp/history_0.tar CFPU- 0rncmon /stdsymp/rncmon_0.tar CFPU- 0
TroubleshootingMulticontrollerRNC Symptomsdatacollection
Issue:02B DN0976768 31
-
4 Generic software troubleshootinginstructions
4.1 Displaying all blackbox filesPurpose
Thissectionprovidesinstructionsonhowtodisplayalltheblackboxfilesinthesystem.Theseblackboxfilesarenormallylocatedin/srv/Log/crashfolder.
Before you start
Makesureyouhavetheauthoritytothesecondarygroup_nokfsuicrashlogandthepermissionfsASView.Toavoidaddingtheuseraccounttoalonglistofgroups,youarerecommendedtoaddthepermissionfsASViewtogroup_nokfsuicrashlogusingthecommandadd user-management group-to-permission gid_nokfsuicrashlog permid fsASViewandthenassignthegroup_nokfsuicrashlogtothetargetuseraccount.
1 Show all blackbox file groups for all crashed processes.Enterthefollowingcommand:show troubleshooting blackbox listExpected outcomeAlltheblackboxnamesarelisted.Eachnameindicatesthereisaprocesscrash.Anexampleoftheoutputisasfollows.Apr 10 16:34 CLA-1-11945-534657b0-snmpmdserver-ABRTApr 21 09:27 CLA-0-11435-53547401-snmpmdserver-ABRT
Steps
1 Show all blackbox file groups for all crashed processes.Enterthefollowingcommand:show troubleshooting blackbox listExpected outcomeAlltheblackboxnamesarelisted.Eachnameindicatesthereisaprocesscrash.Anexampleoftheoutputisasfollows.Jun 23 14:34 CFPU-0-21089-4e02db77-sokeri-ABRTJun 23 15:30 CFPU-0-21441-4e02eb8e-ilalarm-ABRTJun 24 14:24 CFPU-0-7052-4e042aaa-slapd-ABRTJul 18 13:15 CFPU-0-7905-4e23c149-lastproc-ABRTJul 18 13:15 CFPU-0-7871-4e23c188-starter-ABRT
Genericsoftwaretroubleshootinginstructions TroubleshootingMulticontrollerRNC
32 DN0976768 Issue:02B
-
4.2 Viewing active alarms and alarms historyThealarmsystemindicatespotentialfaultsinthesystemaswellasfaultsthatrequirecorrectiveactions.Afteranalarmisraised,thefaultcausingthealarmmustbesolved.Thesolutioncanbeanautomaticrecoveryoramanualcorrectiveaction.Alarmsaretypicallyusedinsituationswhereitispossibletogiveinstructionsforcorrectiveactionsinthealarmdescription,suchasreplacingahardwareunit.
Thealarmscanalsoberaisedthroughthealarmsystemtoindicatethatthesystemisnotworkingnormally,forexample,whentheharddiskisfullandthesystemcannotwritetoit.Suchalarmsareclearedautomaticallywhenthesystemreturnstoitsnormalstate.Youcannotclearthesealarmsmanually.
Multicontroller RNC alarm handling MulticontrollerRNCOMSprovidesFaultManagementapplicationthatenablesuser
toperformoperationssuchas:
managingMulticontrollerRNC,MulticontrollerRNCOMS,andFlexiBTSalarms viewingMulticontrollerRNC,MulticontrollerRNCOMS,andFlexiBTSalarms
history
Forinformationonhowtocheckandmanagethealarms,seeManaging Faults withOMS.
g TochecktheactivealarmsandalarmshistorydirectlyfromMulticontrollerRNCcommandlineinterface,enterthefollowingcommands:# zahoUsage: zaho [ -h | --help | -a | --alarms | -p | --alarmhistory ][ -x | --alarmx | -s | --summary ] Obligatory parameters: -h | --help : this help -a | -- alarms : active alarms -x | --alarmx : shows alarms with certain alarm number -s | --summary : summary of the alarmsExample: zaho -a : prints all active alarms zaho -ax 12345 : prints all 12345 alarms from active alarms zaho -ps : prints summary about alarm history
ForinformationontheMulticontrollerRNCspecificalarms,checkthefollowingreferencedocumentation:Multicontroller RNC Notices (0-999)Multicontroller RNC Disturbances (1000-1999)Multicontroller RNC Failure Printouts (2000-3999)Multicontroller RNC, IPA-RNC and I-HSPA Adapter Base Station Alarms (7000-7900)
Flexi BTS alarm handling FlexiBTSalarmsaremanagedusingOMSFaultManagementapplication.Asan
alternative,theycanbealsocheckedbyusingBTSSiteManager.Formoreinformation,seeChecking BTS alarmsofTroubleshooting Flexi Multiradio BTSWCDMA.
ForinformationontheFlexiBTSspecificalarms,see Flexi Multiradio BTS WCDMAFaults.
TroubleshootingMulticontrollerRNC Genericsoftwaretroubleshootinginstructions
Issue:02B DN0976768 33
-
Multicontroller RNC OMS alarm handling MulticontrollerRNCOMSalarmsaremanagedusingOMSFaultManagement
application. ForinformationonthespecificMulticontrollerRNCOMSalarms,see:OMS Alarms.
4.3 Collecting core dump file informationPurpose
Whenanapplicationcrashes,itwritesthecontentsofitsexecutionenvironment(stacks,variablesandsoon)intoafile.Thisfileisknownasacoredump.
Acoredumpisusefulforproblemsolving,sinceitprovidesthefollowingdetails:
Theapplication'staskduringfailure Theapplication'sexecutionenvironment
Inadditiontothecoredump,thesystemalsogathersinformationsuchasrecentsyslogentriesandinformationaboutthenode.Thisinformationisalsosavedtofilesandlabeledthesamewayasthecorefiles.
Before you start
Defaultsystemusernameandpasswordare"_nokadmin"/"system".
Steps
1 Open an SSH connection to Multicontroller RNC IP address.
2 List all the blackbox files in the system.show troubleshooting blackbox listTheseblackboxfilesarenormallylocatedin/srv/Log/crashfolder.
3 Switch to root user and the bash shell.set user username root
Providethepasswordforrootuser(defaultisroot).
4 Collect the log and the core files.Tocheckthefilesthatcanbeaccessedfromthenode,wherethelogrecoverygroupisrunning,enterthefollowingcommand:ls /srv/Log/crashThefilenameswillbeinthefollowingformat:----..gzWhere:
nodeisthenameofnodewherethecrashoccurred.
Genericsoftwaretroubleshootinginstructions TroubleshootingMulticontrollerRNC
34 DN0976768 Issue:02B
-
pidistheprocessidofthecrashedprocess. timestampisthetimeofcrashinsecondssince1970-01-0100:00:00UTCin
hexadecimalnotation. nameisthenameofthecrashedprocess. signalisthesignalthatcausedtheprocesstodumpthecore.
typecanbeoneofthefollowing:
core-thecoredumpfile syslog-asnapshotofsyslogentriesfromthetimeofcrash debug -asnapshotofdebugentriesfromthetimeofcrash blackbox.tar -informationaboutthenode
5 Copy the files from the Multicontroller RNC to external machine.scp /srv/Log/crash/ @://
6 Send the files to your Nokia Solutions and Networks representative.
TroubleshootingMulticontrollerRNC Genericsoftwaretroubleshootinginstructions
Issue:02B DN0976768 35
-
5 Configuration management troubleshooting
5.1 Saving configuration snapshot fails5.1.1 Description
Ifthereisnofreespaceonthedisk,savingtheconfigurationsnapshotisnotpossible.
5.1.2 SymptomsThesave snapshotcommandfailswithanerrormessage.
5.1.3 Recovery proceduresIfsavingconfigurationsnapshotfails,followeitherofthesesteps:
1 Check the existing configuration snapshots and delete the old configurationsnapshots that are not needed.Todisplaytheexistingconfigurationsnapshots,enterthefollowingcommand:show snapshot listTodeleteallconfigurationsnapshotsthatarenolongerneeded,enterthefollowingcommand:delete snapshot config-name
2 Check the existing software volumes and delete the old software deliveriesthat are not needed.Todisplaytheexistingsoftwarevolumes,enterthefollowingcommand:show sw-manage listTodeletetheoldsoftwaredeliveriesthatarenolongerneeded,enterthefollowingcommand:delete sw-manage delivery
Configurationmanagementtroubleshooting TroubleshootingMulticontrollerRNC
36 DN0976768 Issue:02B
-
6 Software management troubleshooting
6.1 Installation of a new software delivery fails6.1.1 Description
Theinstallationofanewsoftwaredeliveryfailsifthesoftwareimageiscorruptedoralreadyinstalled.
6.1.2 SymptomsItisnotpossibletoinstallanewsoftwaredelivery.
6.1.3 Recovery proceduresDefaultsystemusernameandpasswordare"_nokadmin"/"system".
1 Check if the software image is corrupted.Ifthesoftwareimageiscorrupted,copyitagainfromtheprovidedtransfermedium.
w NOTICE:Theintegrityofthesoftwaredeliveryimagecanbeverifiedbyusingthemd5sumtool.
2 Check if the image to be installed is already existing.Iftheimagetobeinstalledalreadyexisting,thenthenewsoftwareimagecannotbeinstalledonthesystem.
3 Check the log files related to software management.
a) Switchtobashshell(typeexitlaterontoreturntofsclish).Confirmthecommandbypressing"y".
b) Logfilesarestoredtothefollowinglocations:/var/log/ilsw_upgrade.log/var/log/swm.log
6.2 Activation of a new software delivery fails6.2.1 Description
InthemcRNChardware,whenactivatinganewsoftwaredelivery,thereisafallbackmechanismtoactivatethepreviousdeliveryifthenewdeliverydoesnotbootup.Thesystemswitchestothepreviousdeliveryafterfivefailedbootattempts.
TroubleshootingMulticontrollerRNC Softwaremanagementtroubleshooting
Issue:02B DN0976768 37
-
6.2.2 SymptomsAfteractivationofanewsoftwaredelivery,thesystemtakesalongtime(aboutfivetimeslongerthannormally)tobootup.Whenthesystemisoperationalagain,thepreviousdeliveryisactivatedinsteadofthenewsoftwaredelivery.
6.2.3 Recovery proceduresDefaultsystemusernameandpasswordare"_nokadmin"/"system".
1 Contact Nokia Solutions and Networks representative and attach the followinglog files to carry out the recovery actions.
a) Switchtobashshell(typeexitlaterontoreturntofsclish).Confirmthecommandbypressing"y".
b) Logfilesarestoredtothefollowinglocations:/var/log/ilsw_upgrade.log/var/log/swm.log/var/log/fsconfigure.log
6.3 Troubleshooting software configurationmanagement
6.3.1 Executing pre-download script failsSymptomsTheexecutionofDownloadtaskonNetActinterfacefails.Thefollowingerrorlogs(asanexample)arefoundinthe ilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):
2011-09-14 17:56:58 ILSWMAN: Execute pre_downloading_02_fail.sh error! Status: 12011-09-14 17:56:58 SOKERI: Download pre-checking error! Status: 255, Additional Status: 25
Reason analysisBeforedownloadingthesoftwaredelivery,thesystemexecutesthepre_downloading_xx.shscripttodopre-downloadinspectionifsuchscriptexists.AnerrorintheexecutionofthescriptcausesthefailureoftheDownloadtask.
Recovery proceduresSteps
1 Make a copy of syslog file and ilsw_upgrade.log file (both under /var/logdirectory) respectively.
Softwaremanagementtroubleshooting TroubleshootingMulticontrollerRNC
38 DN0976768 Issue:02B
-
2 Send the copied files to Nokia Customer Service Center.
6.3.2 Parsing targetBD.xml failsSymptomsTheexecutionofDownloadtaskonNetActfails.Thefollowingerrorlogs(asanexample)arefoundinthe ilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):
2011-09-14 16:35:58 SOKERI: Parse target BD SWPackages/default/TargetBD.xml start.2011-09-14 16:35:59 SOKERI: Parse target BD error! Status: 17466, Additional Status: 18
Reason analysisWhentheoperatorstartsaDownloadtaskonNetAct,mcRNCdownloads targetBD.xmlfilebeforedownloadingtheactualsoftwaredelivery.ThereasonforthefailedparsingofthetargetBD.xmlfilecanbe:
TheformatofthetargetBD.xmlfileisincorrect. Keyinformationismissinginthefile.
g Inthisexample,thetargetBDfileisnamedastargetBD.xml.Thisnamecouldbedifferentintherealenvironment.
Recovery procedures
1 Make a copy of the needed filesThefilesthatneedtobecopiedare:
targetBD.xmllocatedin/var/opt/nsn/SS_ILOMU/SS_ILSWMAN/build/ folder sysloglocatedin/var/logfolder ilsw_upgrade.loglocatedin/var/logfolder
2 Send the copied files to Nokia Customer Service Center.
6.3.3 Executing pre-activation script failsSymptomsTheexecutionofDownload and ActivateorActivatetaskonNetActfails.Thefollowingerrorlogs(asanexample)arefoundintheilsw_upgrade.logfile(thatlocatesin/var/logfolderonmcRNC):
2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh start2011-09-16 09:35:56 ILSWMAN: Execute pre_activation_02.sh error! Status: 2552011-09-16 09:35:57 SOKERI: Pre-Activate error! Status: 255, Additional Status: 25
TroubleshootingMulticontrollerRNC Softwaremanagementtroubleshooting
Issue:02B DN0976768 39
-
Reason analysisBeforeactivatingthesoftwaredelivery,thesystemexecutesthepre_activation_xx.shscripttodopre-activatinginspectionifsuchscriptexists.AnerrorintheexecutionofthescriptcausesthefailureoftheDownload and ActivateorActivate task.Thereasonforthefailedexecutionofthescriptvariesdependingonwhatarespecifiedinthescript.
Recovery procedures
1 Make a copy of syslog file and ilsw_upgrade.log file (both under /var/logdirectory) respectively.
2 Send the copied files to Nokia Customer Service Center.
6.3.4 Upgrading eSW failsSymptomsTheeSWupgradefailsduringthemcRNCupgrade.ThestatusINSTALLATION_FAILEDisdisplayedontheACTIVE/MAINBANKorthePENDINGBANKfieldwhenyouexecutetheSCLIcommandshow sw-managed embedded-sw status.
Reason analysisAsuccessfuleSWupgradehascertainrequirementsonthediskspace.Whenthefreediskspaceinthe/tmpfolderisbelow7MB,ortheusageofthe/dev/mtdblock1filesystemexceeds83%,theeSWupgradefails.
Recovery procedures
1 Enter the bash shell as a root user.Toenterthebashshellasarootuser,executethefollowingcommand:shell bash full
2 Check the available space of each LMP.TochecktheavailablespaceofLMP,executethefollowingcommandforeachLMP:ssh LMP-1-X-1 "df h"
Example output for LMP-1-1-1# ssh LMP-1-1-1 "df -h" Filesystem Size Used Avail Use% Mounted on /dev/mtdblock1 117M 103M 14M 88% / tmpfs 506M 2.6M 504M 1% /dev/shm /dev/mtdblock0 10M 2.6M 7.5M 26% /boot none 32M 31M 1M 97% /tmp none 64M 0 64M 0% /mnt/var none 64M 22M 43M 34% /opt/fastpath/tmpfs none 64M 22M 43M 34% /opt/fastpath2/tmpfs /dev/mtdblock5 117M 96M 21M 83% /mnt/backupfs
Softwaremanagementtroubleshooting TroubleshootingMulticontrollerRNC
40 DN0976768 Issue:02B
-
3 Delete some unnecessary files to free more space.Deletesomeunnecessaryfilesinthe/tmpdirectoryorroot/filesystemofeachLMPuntil:
Thefreediskspaceinthe/tmpdirectoryisabove7MB. Theusageofthefilesystemmountedonroot/(/dev/mtdblock1intheoutput
ofstep2)isbelow83%.
g Donotdeleteanyfilesthatareimportantforthesystemfunctionality.Forexample,itissafetodeletethefilesinthe/tmp/sysrepdirectory.
6.3.5 Alarm 2518 - NO VALID FALLBACK COPY FOR DEFAULTPACKAGE is triggeredSymptomsAlarm 2518isseenbothfromtheNetActAlarmMonitorgraphicaluserinterface(GUI),andthemcRNCcommandlineinterfacewhenexecutingshow alarm activeSCLIcommand.Anexampleofthecommandoutputisasfollows:
Alarm ID : 550Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGEManaged object : fshaRecoveryUnitName=OMUTestServer- 0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRootSeverity : 3 (major)Cleared : noClearing : automaticAcknowledged : noAck. user ID : N/AAck. time : N/AAlarm time : 2012-01-04 11:17:47:633 CSTEvent type : x2 (processing error)Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServer-0,fsipHostN ame=CLA- 0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRootIAppl Addl. Info : 4Appl. Addl. Info :
Reason analysisThefaultiscausedbyoutdatedfallback(FB)build.ItisrequiredthattheFBbuildmustbesavedatafrequencyofeverysevendays.
Recovery proceduresSteps
1 Enter SCLI command interface on mcRNCTogetintoSCLIcommandinterface,enterthefollowingcommand:fsclish
TroubleshootingMulticontrollerRNC Softwaremanagementtroubleshooting
Issue:02B DN0976768 41
-
2 Create an FB buildEnterthecommandasfollows:save sw-manage app-sw-mgmt fb-buildExpected outcomeThefollowingoutputisdisplayed:FB build has been saved successfully.Unexpected outcomeThefollowingoutputisdisplayedwhentheactivebuildisnotabackup(BU)build.Current build is not BU.FormoreinformationoncreatinganFBbuild,seeCreatingfallbackbuildfromactivebackupbuild.
3 Check if Alarm 2518 has been cancelledAfterthenewFBbuildiscreated,thealarmiscancelledautomatically.Youcanseetheresultbyeitherofthefollowingways:
Usetheshow alarm activeSCLIcommandortheshow alarm activefilter-by specific-problem 2518command.Alarm2518stillexistsintheoutput,buttheClearedfieldisnowyesandSeverityhaschangedto6(cleared).Forexample:Alarm ID : 550Specific problem : 2518 - NO VALID FALLBACK COPY FOR DEFAULT PACKAGEManaged object : fshaRecoveryUnitName=OMUTestServer- 0,fsipHostName=CLA-0,fsFragmentId=Nodes,fsFra gmentId=HA,fsClusterId=ClusterRootSeverity : 6 (cleared)Cleared : yesClearing : automaticAcknowledged : noAck. user ID : N/AAck. time : N/AAlarm time : 2011-12-19 16:21:00:749 CSTEvent type : x2 (processing error)Application : fshaProcessInstanceName=IL_Sokeri,fshaRecoveryUnitName=OMUTestServer-0,fsipHostN ame=CLA- 0,fsFragmentId=Nodes,fsFragmentId=HA,fsClusterId=ClusterRootIAppl Addl. Info : 4Appl. Addl. Info :
ChecktheactivealarmsintheAlarmMonitoroftheNetActGUI.Alarm2518hasdisappeared.
Softwaremanagementtroubleshooting TroubleshootingMulticontrollerRNC
42 DN0976768 Issue:02B
-
7 Networking troubleshooting
7.1 Packet loss in certain traffic flows unexpectedlyhigh
7.1.1 DescriptionThenetworkelementkeepsthepacketdropfromdifferenttrafficflowsundercontrolduetotheQualityofService(QoS)systemfeatureimplementedontheingress(incoming)andegress(outgoing)sides.TheQoSoningresssidemanagescaseswhenNPUisoverloadedandQoSonegresssidemanagescaseswhenoutgoinginterfacecapacityisoverloaded.Inaddition,theinternaltrafficismanaged.
Thepacketlossincertaintrafficflowmaybeunexpectedlyhighbecause:
Theegressport rate-limitthresholdislessthantheconvergedtrafficflowsvolumeofferedandACLrulestosettheproperpipmarkforanyoutgoingstreamsmightnotbecreated
IncomingpacketsmighthavewrongDSCPmark.
7.1.2 SymptomsThepacketlossincertaintrafficflowsunexpectedlyhigh.
7.1.3 Recovery procedures
1 Analyze the traffic flows coming out of the interface.Toverifywhetherthepacketlosshappensintheegressinterface,collecttheegressinterfaceQOSstatisticsbyusingtheperformancemanagementmeasurementjobs,whilethetrafficisongoing.Whileanalyzingthereportfile,lookforanydiscardedpacketcountforthespecificegressinterface.Iftherearenosignificantdiscardedpackets,thenskipstep1andstep2whichhandlesegresssideandproceedtostep3whichhandlestheingressside.
2 Check the rate-limit value for the egress interface.Tochecktherate-limitvaluefortheegressinterface,enterthefollowingcommand:show networking ether owner iface\
Checktherate-limitvalueintheoutput.Ifthe rate-limitvalueisequaltothetrafficpassed,thelowestprioritytrafficmaybedropped.Topreventpacketloss,therate-limitthresholdmaybeincreased.Ifthisactionisnotdesired,thenproceedtostep2.Tosetthenewrate-limit threshold,enterthefollowingcommand:set networking ether iface rate-limit
TroubleshootingMulticontrollerRNC Networkingtroubleshooting
Issue:02B DN0976768 43
-
g Iftheegressinterfaceisalinkaggregationgroupinterface,thenmakesurethattheratelimitvaluedoesnotexceedthesummationofalltheportsthatwasunderthelinkaggregationgroupinterface.
3 Check if the traffic flow is appropriately handled by egress interface QoS.EachqueuefromaqueuesetattachedtotheinterfaceimplementparticularPHB.ThequeueisselectedforeachpacketaccordingtothePacketInternalPriority.PIPismarkedbyclassificationfunctionoftheACL.Tocheckifthepip-markfortheflowunderconsiderationisconfiguredproperly,enterthefollowingcommand:show networking aclruleThefollowingtableprovidesthemappingbetweenthequeue IDand pip-mark.
Table 5 Mappingbetweenpip-markandQueueID
pip-mark
queue ID
0x00 0
0x01 1
0x02 2
0x03 3
0x04 4
0x05 5
0x06 5
0x07 5
Tomaptheselectedtraffictotherightqueueaslistedinthetable,seethesection,Configuring ACL settings.Thepriorityofthequeueisdeterminedbytheweightassignedtothequeueintheattachedqueueset.Increasingthelengthofthequeuecanalsoreducethepacketlossduringshorttrafficbursts.However,systemresourcesandtrafficlatencyhastobetakenintoaccount.FormoreinformationrefertoConfiguring queue sets section.Also,theothertrafficflow(s)maygethigherprioritythantheflowunderconsideration,thusoccupyingthesystemresources.Tocheckwhethertheattachedqueuesethasappropriateweightandlength,enterthefollowingcommand:show networking qset name Tomodifytheweightandlengthofthequeue,enterthefollowingcommand:set networking qset test1 queue weight \length queue weight \ length
Networkingtroubleshooting TroubleshootingMulticontrollerRNC
44 DN0976768 Issue:02B
-
4 Verify that the incoming packet has expected DSCP value.
a) ChecktheDSCPvaluefromtheMulticontrollerRNC.
StarttcpdumponMulticontrollerRNCsidetomonitortrafficonthedesiredinterface.
Switchtorootuserandthebashshell.(Typeexitlaterontoreturntofsclish.)set user username rootProvidethepasswordforrootuser.
Enabletcpdumpforselectedinterface.#tcpdump -i -w/mnt/backup/tcp_dump.pcap
OpenthecollectedtracewithprotocolanalyzerandverifytheDSCPvalueofthedesiredtrafficflow.
Or
a) ChecktheDSCPvaluefromtheexternalworkstation.
Runprotocolanalyzerandcapturethetrafficofthedesiredinterface. VerifytheDSCPvalueofthecollectedtrafficflow.
ThefollowingtableliststheexpectedDSCPcodesforexternaltraffic.TheDSCPcodesnotmentionedinthetableareclassifiedasbesteffortclassandarehandledwithlowestpriority.
Table 6 PriorityvaluesfordifferentDSCPcodes
PHB Class
DSCP Code
Priority
nc 0x30 Highpriority
ef 0x2E
af41 0x22 Mediumpriority
af42 0x24
af43 0x26
af31 0x1A Lowpriority
af32 0x1C
af33 0x1E
af21 0x12
af22 0x14
af23 0x16
TroubleshootingMulticontrollerRNC Networkingtroubleshooting
Issue:02B DN0976768 45
-
Table 6 PriorityvaluesfordifferentDSCPcodes(Cont.)
PHB Class
DSCP Code
Priority
af11 0x0A
af12 0x0C
af13 0x0E
be 0x00
7.2 Multihop BFD sessions are not established7.2.1 Description
ThemultihopBFDsessionsmaynotbeestablishedifthereareproblemsinBFDservice,configuration,ornetworkconnectivity.
7.2.2 SymptomsThemultihopBFDsessionsarenotestablishedorsessionsdonotreachUPstate.
7.2.3 Recovery procedures
1 Check the BFD service.TochecktheBFDservice,followthesesteps:
a) VerifythestateoftheFSBFDServerrecoveryunitinthenodewhereBFDisconfigured.Enterthefollowingcommand:show has state managed-object //FSBFDserverThefollowingoutputisdisplayed:root@CFPU-0 [RNC-89] > show has state managed-object /CFPU- 0/FSBFDserverOBJECT ADMINISTRATIVE OPERATIONAL USAGE ROLE PROCEDURAL DYNAMIC/CFPU-0/FSBFDServer UNLOCKED ENABLED ACTIVE ACTIVE - -Iftherecoveryunitislocked,unlockitusingthefollowingcommand:set has unlock managed-object //FSBFDServer
2 Check BFD alarms.VerifythatthereareactivealarmsforBFD(specificproblem70348).Enterthefollowingcommand: show alarm active filter-by specific-problem70348
IftheactiveBFDalarmsarefound,verifyBFDconfiguration(seestep4)andnetworkconnectivity(seestep5).
Networkingtroubleshooting TroubleshootingMulticontrollerRNC
46 DN0976768 Issue:02B
-
3 Check the BFD runtime information.VerifythattheBFDsessionsarefoundintheruntimeenvironmentandthatthesessionstateiscorrect.Enterthefollowingcommand:show networking instance bfd runtime node /The Initstatemeansthattheremotesystemiscommunicatingandthelocalsystembringsthesessionup,buttheremotesystemdoesnotrealizeit.Ifthesessionsarenotfound,verifytheBFDservicestateprovidedbyHAS.Ifthesessionstateisdown,verifythebasicIPconnectivity.
g Diagnosticfieldshowsadiagnosticcodespecifyingthereasonforchangeinsessionstate.
4 Check BFD configuration.TochecktheBFDconfiguration,followthesesteps:
a) VerifythatBFDisenabledonthecorrectnodewithcorrectparameters.Enterthefollowingcommands:show networking monitoring bfd config allIftheBFDisnotenableonthecorrectnodewithcorrectparameter,seethesection,Configuring IP Connection for Multicontroller RNCformoreinformation.
b) VerifythatlocalIPaddressforBFDsessionisfoundfromthenodeandisassignedtothecorrectVRFinstance.Enterthefollowingcommand:show networking address
g BFDsessionscanbeconfiguredbeforeIPaddressisaddedtotheinterface.However,theBFDsessioncontrolpacketaresentonlyaftertheIPaddressisadded.IfthelocalIPaddressfortheBFDsessionisnotfoundandassignedtothecorrectVRFinstance,seethesection,ConfiguringtheBFDmultihopsessionsectionformoreinformation.
c) VerifythattherearenoACLrulesthatblockBFDpackets.Enterthefollowingcommand:show networking aclruleIfrequiredaddruleforallowingBFDpacketsinboththedirections,seethesection,Configuring IP Connection for Multicontroller RNCformoreinformation.
g MultihopBFDcontrolpacketsaretransmittedinUDPwithdestinationport4784.
5 Check the IP connectivity between routers.TochecktheIPconnectivitybetweenrouters,followthesesteps:
a) VerifybasicIPconnectivitybetweennetworkdevicesusingpingcommand.PingfromthecorrectVRFinnodeyouhaveconfiguredBFDsession.Ifthereisnoresponsetopingverifytheinterfacestate.ToexecutethepingcommandfromtheSCLI,startthebashshell(infullmode)andthenusesshtoconnecttothecorrectnode.Enterthefollowingcommands:shell bash full ssh ping -V or ping exit
TroubleshootingMulticontrollerRNC Networkingtroubleshooting
Issue:02B DN0976768 47
-
IfyouwanttomakesurethatthenodewhereBFDisconfigured,acceptsICMP(protocol1)packetsininputandoutputchain,enterthefollowingcommand:show networking aclrule owner /IfthenodedoesnotacceptICMP(protocol1)packets,thenaddacceptrulesbyenteringthefollowingcommands:add networking aclrule / index 20000chain\ input protocol 1 accept add networking aclrule/ index 20001 chain\ output protocol 1 accept
g Theneighboringroutermaybeconfiguredtodroppingrequestsandmaycausepingnottorespond.b) VerifythatthestatefortheinterfacewhereBFDsessionisconfiguredhasadmin
stateupandhasrunningflagset.Enterthefollowingcommand:show networking interface runtimeIfrunningflagisnotpresent,verifythatcableisproperlyconnectedtotheinterface.Iftheadminstateisnotup,seeConfiguring IP Connection forMulticontroller RNCformoreinformation.
c) VerifythattheBFDcontrolpacketsaresentandreceivedontheinterfacewhereBFDsessionisconfigured.ToexecutethiscommandfromSCLI,startthebashshell(infullmode)andthenusesshtoconnecttothecorrectnode.Enterthefollowingcommands:shell bash full ssh tcpdump -i port 4784 exitThiscommanddoesnotrequire rootaccess.IfonlyoutgoingBFDpacketsareseen,verifytheconfigurationattheremoteendpoint.
w NOTICE: Nokiaadvisesthatonlyadvancedusers,deeplyfamiliarwiththebehaviorofthenetworkelement,mustusecommandsofthefull bashingeneralandthetcpdumpcommandinparticular.Eveniftherootaccountordedicatednonrootaccountaccessisused,thecommandsfromthefull bashshellcanbeeasilyexploited,Forexample,tochangesecurity-relevantconfiguration.Alsothetcpdumpcommandinparticularwhenaccidentallyoron-purposemisusedcancausesignificantperformancepenalties(adropof0-100%)innetworking(throughputandlatencies),diskI/Oorprocessingcapacity.Intheworstcasethismight,forexample,causeautomaticrecoveryactions.Alsosomepartsofthetcpdumpcommandmightnotworkasexpectedduetothepeculiaritiesofthenetworkelementarchitecture.
7.3 OSPF is not working properly7.3.1 Description
IfthereareproblemswithOSPFconfigurations,interfacestate,ornetworkconnectivitytheOSPFneighboradjacenciesarenotformedandthereforeOSPFrouteadvertisementdoesnotwork.
7.3.2 SymptomsOSPFadjacenciesarenotformedandtheroutingtabledoesnotincludeOSPFroutes.ThereisnonetworkconnectivitytonetworksadvertisedbyOSPF.TheOSPFneighborsarenotseenorOSPF not configurederrormessageisdisplayedwhenmonitoringtheOSPFneighboradjacencystatefromtheSCLIcommand.
Networkingtroubleshooting TroubleshootingMulticontrollerRNC
48 DN0976768 Issue:02B
-
7.3.3 Recovery procedures
1 Check the routing service.CheckthestateofFSRoutingDaemonServer recoveryunitinthenodewhereOSPFisconfigured.Enterthefollowingcommand:show has state managed-object //FSRoutingDaemonServerThefollowingoutputisdisplayed:OBJECT ADMINISTRATIVE OPERATIONAL USAGE ROLE PROCEDURAL DYNAMIC//FSRoutingDaemonServer UNLOCKED ENABLED ACTIVE ACTIVE - -IftheFSRoutingDaemonServerrecoveryunitislocked,theOSPFneighborquerytothecorrespondingnodewillresultinanOSPF not configured error.IfFSRoutingDaemonServerrecoveryunitislocked,unlockitusingthefollowingcommand: set has unlock managed-object//FSRoutingDaemonServer
2 Check the OSPF runtime information.TochecktheOSPFruntimeinformation,followthesesteps:
a) Verifythattheneighboradjacencystatehasreached2-wayorfullstate.Enterthefollowingcommand:show networking instance node ospf neighbors
g Thefulladjacencyisformedonlywithdesignatedrouterandbackupdesignatedrouter.Thetimetoreach2-wayorfullstatedependsontheconfiguredOSPFparameters.IfthecommandfailswiththeOSPF not configurederror,verifyroutingservicestate(seestep1)andOSPFconfiguration(seestep3).IfOSPFneighboradjacencyarenotformed,checktheconfiguration(seestep3).
b) VerifythatthebasicIPconnectivitybetweenroutersisworking(seestep4).c) VerifythattheOSPFhellopacketsaretransmittedandreceived.Enterthe
followingcommand:show routing instance node \ospf packetsIftheOSPFhellopacketsarenottransmittedandreceived,checkthatOSPFisconfiguredproperly(seestep3).
d) VerifythatOSPFroutesarefoundinroutingtable.Enterthefollowingcommand:show routing instance node \route ospfOSPFinternalroutesaremarkedwithO infrontoftherouteandexternalroutesaremarkedwithOE.Ifsomeoftheroutesaremissingchecktheconfiguration(seestep3).
3 Check OSPF configuration.TochecktheOSPFconfiguration,followthesesteps:
a) VerifythattheOSPFlicensehasbeeninstalledandthefeatureadminstateisON.Enterthefollowingcommand:show license feature-mgmt name OSPFForRedundancy
TroubleshootingMulticontrollerRNC Networkingtroubleshooting
Issue:02B DN0976768 49
-
g OSPFdoesnotstartifthelicenseisnotinstalledorifthelicenseisinstalledbutisinactiveorhasexpired.Alsoifthefeatureadminstateisofforconfig.Ifthefeatureadminstateisoff or config,enterthefollowingcommandtosetittoon:set license feature-mgmt name OSPFForRedundancy feature-admin-state on
IftheOSPFlicenseisnotinstalled,thenenterthefollowingcommandtoinstallthelicense:add license file Where,file specifiesthelocationofOSPFlicense.
b) VerifythatOSPFisenabledonthecorrectinterfaceandOSPFarea.Enterthefollowingcommands:show routing instance node \ospf configIftheOSPFisnotenabledonthecorrectinterfaceandarea,seeConfiguring IPConnection for Multicontroller RNCformoreinformation.
c) VerifythatanIPv4addressissetfortheinterfacewhereOSPFv2isconfigured.Enterthefollowingcommand:show networking addressExampleIftheOSPFv2isconfiguredfortheeth_sfp1interface,enterthefollowingcommand:show networking instance default address owner /IPNIU1-0\iface eth_sfp1Thefollowingoutputisdisplayed:address instance defaultinterfaces : eth_sfp1 type : dedicated address : 200.0.0.1/24 owner : /IPNIU1-0IftheIPv4addressismissing,addittothecorrespondinginterface,seeConfiguring IP Connection for Multicontroller RNCformoreinformation.
d) Verifythattherouter-idisset.Enterthefollowingcommand:show routing instance node \router-idThesystemautomaticallyassignsrouterIDbasedontheIPv4addressesfound.Notethat router-idmus