Improved BGP convergence via Ghost Flushing

Yehuda Afek

Anat Bremler-Barr Shemer Schwarzd

המרכז הבינתחומי הרצליה

Problem: BGP Convergence

[Labovitz,Ahuja,Bose,Jahanian] BGP may take up to 15 minutes to converge.

Here: Reduce the worst case from minutes to seconds, in a practical way

Problem: BGP Convergence

[Labovitz,Ahuja,Bose,Jahanian] Events Time (sec’s, minRouteAdver=30)

E-Down 30•n n 10,000, up to 15 minutes

E-UpE-Up 30 30•d d d d 30, d=diameter

E-Longer 2•30•l l == path length

E-ShorterE-Shorter 30 30•d d

Here: E-down = l time units (unit = link delay)

E-Longer = 30•d

Agenda

BGP overview The BGP convergence problem Ghost buster rule Ghost flushing rule Simulation results

BGP protocol

Distance (Path) vector protocol

Receive AS-path from the neighbors

Chooses the best one (shortest)

Eliminates Routing loops using the AS-path

Two kinds of messages: Announcements and Withdrawal

Problem: Ghost information

One Ghost (old information) makes many, and in the network it continues recursively

dst

2

1 3

4

dst: 0 dst: 0

dst: 0

0

dst: 0

dst: 0t=0

withdraw

Problem: Ghost information

dst

2

1 3

4

dst: 2 0 dst: 1 0

dst: 1 0

0

dst: 1 0

dst:}{ t=1

annc:2 0

annc:1 0 annc:1 0

annc:1 0

One Ghost (old information) makes many, and in the network it continues recursively

Problem: Ghost information

dst

2

1 3

4

dst:}{ dst: 1 2 0

dst: 1 2 0

0

dst: 3 1 0

dst:}{ t=2

withdraw

One Ghost (old information) makes many, and in the network it continues recursively

Problem: Ghost information

dst

2

1 3

4

dst:}{ dst: 2 1 0

dst: 2 1 0

0

dst: 3 1 0

dst:}{ t=3t=4t=5t=6

One Ghost (old information) makes many, and in the network it continues recursively

minRouteAdver: Wait 30 sec’s before sending the next announcement (BGP)

t=28t=27t=24t=22t=20t=17t=16t=14t=11t=10t=8t=7t=31

annc: 3 1 0 annc: 2 1 0

annc: 2 1 0

E_Down convergence

Time Message

BGP30nnE

In the clique (size 4) example the scenario ends after 62 sec

(= 30(n-2) )

Without MinRouteAdver

Avalanche of Messages O(n!)

Explore all possible paths of length 1, 2

…

dst

2

1 3

4

dst: 0 dst: 0

dst: 0

0

dst: 0

dst:}{

t=0

1 : 1 03 : 3 0 4 : 4 0

1 : 1 02 : 2 0 3 : 3 0 1 : 1 02 : 2 0 4 : 4 0

2 : 2 03 : 3 0 4 : 4 0 withdraw

al

Without MinRouteAdver

dst

2

1 3

4

dst: 2 0 dst: 1 0

dst: 1 0

0

dst: 1 0

dst:}{

t=0.1

1 : 1 03 : 3 0 4 : 4 0

1 : 1 03 : 3 0 4 : 4 0 1 : 1 03 : 3 0 4 : 4 0

2 : 2 03 : 3 0 4 : 4 0

annc: 2 0

annc: 1 0

annc:1 0

annc: 1 0

Avalanche of Messages O(n!)

Explore all possible paths of length 1, 2

…

Without MinRouteAdver

dst

2

1 3

4

dst: 20 dst: 2 0

dst: 2 0

0

dst: 3 0

dst:}{

t=0.2

1 : 1 2 03 : 3 0 4 : 4 0

1 : 1 2 02 : 2 0 3 : 3 0

1 : 1 2 02 : 2 0 4 : 4 0

2 : 2 03 : 3 04 : 4 0

annc: 2 0

annc:3 0

annc:2 0

annc:2 0

Avalanche of Messages O(n!)

Explore all possible paths of length 1, 2

…

Without MinRouteAdver

dst

2

1 3

4

dst: 3 0 dst: 4 0

dst: 3 0

0

dst: 3 0

dst:}{

t=0.3

1 : 1 2 03 : 3 0 4 : 4 0

1 : 1 2 02 : 2 1 0

3 : 3 0

1 : 1 2 02 : 2 1 0

4 : 4 0

2 : 2 3 03 : 3 04 : 4 0

annc:3 0

annc:3 0

annc:4 0

Avalanche of Messages O(n!)

Explore all possible paths of length 2, 3

…

annc:3 0

Without MinRouteAdver

dst

2

1 3

4

dst: 4 0 dst: 4 0

dst: 1 2 0

0

dst: 4 0

dst:}{

t=0.4

1 : 1 2 03 : 3 1 0

4 : 4 0

1 : 1 2 02 : 2 1 0 3 : 3 1 0

1 : 1 2 02 : 2 1 0

4 : 4 0

2 : 2 3 03 : 3 1 0

4 : 4 0

annc:4 0

annc: 1 2 0

annc:4 0

Avalanche of Messages O(n!)

Explore all possible paths of length 2, 3

…

annc:4 0

E_Down convergence

Time Message

BGP with MinRouteAdver

30nnE

BGP without MinRouteAdver

hnh=one link delay

n!E

Related Work

Introducing the problem [Labovitz,Ahuja,Bose,Jahanian], [Labovitz,Wattenhofer,Venkatachary,Ahuja] real life evidence theoretical analysis

Experimental analysis [Griffin,Premore] Solution

Works in Counting to Infinity: Adding states [Garcia-Luna-Aceves] – EIGRP like… Route Poisoning with Hold-down [Cisco:Rutgers]–

IGRP like... Routes consistency [Pei,Zhao,Wang,Massey,Mankin,Wu,Zhang]

Ghost flushing rule

If ASpath to dst is longer and cannot send annoucement (due to minRouteAdver rule ) then send withdrawal

Motivation: Flush the ghost information ASAP

Ghost Flushing example

dst

2

1 3

4

dst: 0 dst: 0

dst: 0

0

dst: 0

dst: 0t=0

withdraw

Ghost Flushing example

dst

2

1 3

4

dst: 2 0 dst: 1 0

dst: 1 0

0

dst: 1 0

dst:}{ t=1

annc:2 0

annc:1 0 annc:1 0

annc:1 0

Ghost Flushing example

dst

2

1 3

4

dst:}{ dst: 1 2 0

dst: 1 2 0

0

dst: 3 1 0

dst:}{ t=2

withdraw

Longer ASpath &minRouteAdver timer Send “flushing” withdrawal

withdraw

withdraw

withdraw

Ghost Flushing example

dst

2

1 3

4

dst:}{ dst:}{

dst:}{

0

dst:}{

dst:}{ t=3

withdraw

withdraw

withdraw

Analysis: Time convergence of ghost flushing rule, E_down

In each time unit (=h, maximum link delay), ghost information is erased to a distance greater by one

After k time units, ghost information ASpath with length < k has disappeared.

Longest Ghost ASpath = n (in theory). Hence (worst case) time convergence: nh

E_Down convergence

Time Message

BGP with MinRouteAdver

30nnE

BGP without MinRouteAdver

hnh=one link delay

n!E

Ghost flushing hn2Ehn/30

Ghost Buster Rule

The convergence time is better than expected !!!!

Explanation: The minRouteAdver blocks the

propagation of ghost information, while the flushing withdrawal “eats” the ghost information.

Bad (wrong) news propagate slowly

Analysis: Ghost buster rule

Add to the ghost flushing rule:

Router sends announcement, only after delta time

MinRouteAdver similar to delta:

Common implementation: MinRouteAdver per peer And, timer almost always on (lots of BGP announcements !)

Analysis: Time convergence of ghost buster rule

The ghost information disappears at time t: d+t/(delta+h) = t/h Every delta+h time the length of the maximum ghost

ASpath is increased by one. Every h time, the length of the minimum ghost ASpath

is increased by one. After the failure the length of the maximum ghost

ASpath is d (diameter). Hence: t = kdh/(k-1) d, where k = (delta+h)/h is the rate of the algorithm

E_Down convergence

TimeMessage

BGP with MinRouteAdver

30nnE

BGP without MinRouteAdver

hnh=one link delay

n!E

Ghost flushing hn2Ehn/30

Ghost flushingWith Ghost buster

kdh/k-1 dhd=diameterk=(delta+h)/h

2Ehkd/30(k-1)

The effect on E_longer

dst

2

1 3

4

0 5

6

7

BGP: Convergence time dominated:

1. Time until ghost information vanishes

2. Time until backup path propagates in

Ghost flushing: helps the first factor

The effect on E_longer

Original BGP may err:

MinRouteAdver peer stores wrong ASPath

BGP may err and send the packet in the wrong direction

Ghost flushing: send withdrawal to a peer. Perhaps by a chance there may be an alternative path there.

Simulation: BGP code

Shortest path metric

Delay on link between 0.2 to 2 sec

MinRouteAdver randomly in 0 to 30 sec

Simulation: Clique E-down

0

100

200

300

400

500

600

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18Number of Nodes

Tim

e

( se

c

)

Original BGP

Modified BGP

Simulation: ISP topology

1

3

7

5

8

4

9

dst

0

20

40

60

80

100

10 20 30 40 50 60 70 80 90 100 110 120

Convergence Time

Pe

rce

nta

ge

0102030405060708090

100

10 20 30 40 50 60 70 80 90 100 110 120

Convergence Time

Pe

rce

nta

ge

Example: Core Internet (ASes)

5.14981106298111111954111121882.5261733920242063610341161550101727112436103143351898175222296310451

Ghost Flushing

BGPIn-degree

Out-degree

E_longer: Convergence Time

dst

2

1 3

4

0 5

6

7

0

100

200

300

400

500

600

10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44

Number Of Nodes

Tim

e

Original BGPModified BGP

Conclusion

Reduced convergence time from minutes to sec’s.

Does not hurt in other cases

Ghost flushing - no change at BGP messages

Ghost buster solution – a new counting to infinity solution

BGP very sensitive to minor modifications.