peer policy policing with netflow nanog 25 june 9, 2002

Peer Policy Policing withNETFLOW

NANOG 25June 9, 2002

Matthew MeyerTraffic Engineering

NANOG 25June 9, 2002

» 200 + On Net Cities» 27 On Net Countries» Nearly 100,000 route miles» 17 Metro Networks

The Global Crossing Network

Peer Policy Policing With Netflow

» Discovering and engaging the wayward packet flows that stumble onto your network

» Giving default free networking a fighting chance» Get off my lawn» Bottom line: Just detecting a peer defaulting

traffic us

Peer Policy Policing with NetflowDefining the problem

» Telecom & Internet-space companies going into Ch11

» Punctuated mass customer moves due to Ch7 backbone liquidations

» Peering less flexible» Some will resort to uncouth methods to mitigate

the congestion and sidestep potential costs

Peer Policy Policing with NetflowDefining the problem

» Fewer players, larger peerings» Peering inherits more flux and less

flexibility to deal with it» Some more liberal peering channels may

dry up or become heavily utilized

Peer Policy Policing with Netflow

» Time to think like a bean counter» Is peering being abused?» Effect: Lower capex due to longer upgrade

cycles» End goal: Knowing that we run a tight ship and

being alerted when uninvited traffic enters the network

Addressing the Problem


» Not rocket science» 1:100 Netflow sampling» Sampling points: All traffic arriving on our

border routers» Currently set to do peer-as type flow export

Measurement


» One centrally located collector» Collector handling approximately 20 selected

routers» Collector iBGP peers with border routers» Records route table changes every 5 minutes» Dual Pentium III, 1G memory, multiple Ultra-160

SCSI drives, directly connected to backbone

Measurement


DEFAULTING PEER REPORT: Rec'd Peer Bytes percentage of totalrouter interface destined for peer Bytes for interfacebr2.HUB1.gblx.net_so-2/1/3.0 0.011M 0.006 <-Peer Abr2.HUB1.gblx.net_so-2/1/0.0 0.026M 0.008 <-Peer Bbr2.HUB1.gblx.net_so-3/1/0.0 0.087M 0.008 <-Peer Cbr2.HUB1.gblx.net_so-2/1/2.0 0.145M 0.011 <-Peer Dbr2.HUB1.gblx.net_at-2/2/0.0 0.167M 0.024 <-Peer Ebr2.HUB1.gblx.net_so-1/2/3.0 0.339M 0.017 <-Peer Fbr2.HUB1.gblx.net_so-3/1/2.0 2.464M 0.246 <-Peer Gbr2.HUB1.gblx.net_so-0/0/0.0 3319.615M 56.722 <-uplinkbr2.HUB1.gblx.net_so-1/0/0.0 3381.523M 61.515 <-uplink

Measurement


EXAMPLE OF FLOWDATA/Ixia/SeeFlow/bin/rseeas2as -S '20020603 00:00' br2.w00t1.gblx.net

Facets:TimeInterval : 06/04/2002 16:50:49.217018 - 06/04/2002 19:31:52.879363 UTCRouterIpv4Addr : 10.10.10.10InputIfIndex : 67InputIfIpv4Addr : 10.0.0.1InputIfName : so-1/2/3.0RouterName : br2.w00t1.gblx.netSrc AS Dst AS Packets Pkts/sec Bytes Bits/sec------- ------- ------------- ------------- ------------- -------------1111 2222 654.061K 67.683 321.386M 266.058K1111 3333 177.794K 18.398 130.125M 107.723K99 44444 139.861K 14.473 91.889M 76.070K1111 3549 257.006K 26.595 78.603M 65.071K1111 5555 72.634K 7.516 65.807M 54.478K[~300 more lines clipped]

Measurement


» Extracted with Ixia tools» 24 hour cumulative byte count per interface +

dest-as key pair» Created a peer-as list» Ignored incorrectly reported Netflow data

according to routing policy

Manipulating the Data

0.000 20.000 40.000 60.000 80.000 100.000PeerPeerPeerPeerPeerPeerPeer

Peer?UplinkUplink

Traffic Summary

Inbound/Internal Inbound/External


» Our design is hierarchical» Peers tend to be on dedicated peering routers» Our peering in consistent and rich» Collecting closer to the core would not catch this

behavior universally

Where to Look


» BGP import policy gets in the way of trusting source AS

» Trace levels of false peer to peer traffic associated with most peering interfaces

» In initial beta, no peers have been found blatantly defaulting to us

Analysis


» For the moment peer defaulting does not seem to be a problem

» We can move forward and easily complete a detection system

» Feeling more confident about possible tighter peering ahead

So Far So Good


» Change flow export style from peer-as to origin-as

» Putting the discovery ‘on cron’» Long term: »Distribute collection»Build some visualization» Integrate with RRDtool

What’s Next


» Good exercise in ‘Netflow 101’» Sampling capability excellent» Data quality excellent» Restored confidence in Netflow reliability

Retrospect

GLOBAL REACH.SEAMLESS NETWORK.

THANK YOU

peer policy policing with netflow nanog 25 june 9, 2002

Documents