Interconnection in Regional MarketsHow it’s broken and why it’s such a problem.-Tom Paseka

Disclaimer• I’m going to talk about Tier 1 Networks• I define a Tier 1 as a network whose routes are not transited by any other

network, except to their customers• I don’t care about paid settlement.

• I’m looking at primary locations where multiple Tier-1’s interconnect• I’m not looking at a number of locations, for example: Sydney, where three

Tier 1’s are present, but with next-to-no network.• All of this data is based on the BGP view of AS13335. Yours may be

different.

Primary Markets

Historical Interconnection

Historical Interconnection• These locations evolved in part due to the IXPs that operate there.• Eg:

Frankfurt - DE-CIXAmsterdam - AMS-IXLondon - LINXAshburn - EquinixPalo Alto - PAIX (now Equinix)etc.

• These locations continue to grow as major interconnection points with a huge density of networks.

Understanding Current Interconnection• Looking at our Transit Providers tables in San Jose• Their Peer routes only , no customers. • We receive 5 x Full Tables but looking at 3 transits:• NTT Com AS2914• TATA AS6453• Cogent AS174

• Looking at combination of their peer BGP community tag, as well as location based tags, we can see how many routes are locally exchanged.

San Jose/Palo Alto/SF Bay Area• AS2914 NTT Communications:• Looking at community 2914:420, for all their peer (not customer) routes:• 372,244 Routes

• Now, combination [2914:420 2914:1008], for SF Bay Area location + peer routes:• 360,033 Routes

• That’s roughly 97% local!

San Jose/Palo Alto/SF Bay Area• AS6453 TATA Communications:• Looking at community 6453:86, for all their peer (not customer) routes:• 401,220 Routes

• Now, combination [6453:86 6453:1300], for SF Bay Area location + peer routes:• 383,941 Routes

• That’s roughly 96% local!

San Jose/Palo Alto/SF Bay Area• AS174 Cogent:• Looking at community 174:21[01]00, for all their peer (not customer) routes:• 346,788 Routes

• Now, combination [ 174:21[01]00 174:22013 ], for USA location + peer routes:• 337,181 Routes

• That’s roughly 97% in country!

(Cogent doesn’t have, or doesn’t expose city based communities).

Understanding Current Interconnection• Almost all interconnection between peers on the Tier 1 networks happens locally.

• This is EXCELLENT news for interconnection

• The story is completed over most of the traditional primary markets for interconnection.

• USA location considered because of interconnection

• Europe not the same, Many Tier 1’s have little or no presence

• Eg: AT&T AS7018, XO AS2828, Verizon AS701, Sprint AS1239, et al.

• Numbers hover around 75% of routes (Because of above).

• Some funky traffic engineering pollutes the numbers, but for the most part things are good.

Secondary Markets / Regional InterconnectionA bleak outlook

Regional Growth

Regional Growth• Getting closer to eyeballs matters!• It’s how CDN’s beat the speed of light with performance / delivery. • But, so many users are far away from these “primary markets”.

Tokyo, Japan• Looking at two Tier 1’s• NTT AS2914 and TATA AS6453• Same principal as previous tests

• NTT: Just 45,842 routes or 12% of peered routes are learnt in Tokyo• TATA: Just 29,557 routes or 7.5% of peered routes as learnt in Tokyo• This is despite a number of Tier 1 networks being present there. • But, Tokyo might be a bad example because of “other” reasons.

Tokyo, Japan

Asia31%

North America

68%

Europe2% South America

<1%

REgional breakdown of Route Location - NTT Tokyo

Tokyo40%

Osaka<1%

Hong Kong38%

Sin-gapor

e22%

Asia City breakdown of Route Location - NTT Tokyo

Dublin, Ireland• Cogent AS174• 452 routes learned in Ireland• 0 Routes from Peers.

• GTT AS3257• 2718 routes learned in Ireland• 2162 routes learned from Peers• Majority of these from a (major)

eyeball network, not another Tier 1• (though this is still good for delivery!

)

Dublin, Ireland• Almost no inter-connection between Tier 1 networks• Even though multiple have presence in the same buildings• Saving grace is INEX (@comepeerwithme), but this is a talk about transit

interconnection.

One star. Would not interconnect again.

Singapore, Singapore• Looking at two Tier 1’s• NTT AS2914 and TATA AS6453• Same principal as previous tests

• NTT: 87,616 routes or an improved 23% of peered routes are learnt in Singapore

• TATA: 85,544 routes or 22% of peered routes as learnt in Singapore• Definitely an improvement, but shows that connectivity here might be

suboptimal.

Singapore, Singapore• Local routes, looking at two largest local ISPs: Singnet and Starhub• As seen from NTT and Tata, not more than 80% of local routes are seen within

Singapore• Across all of Asia, it improves slightly to 82%• For at least 18% of traffic going over these two providers, traffic will trombone

through the USA to come back to Singapore

Singapore, Singapore

Asia

Not Asia

NTT SINGNET/STARHUB ROUTES

Asia

Not Asia

TATA SINGNET/STARHUB ROUTES

This sounds broken!

Issues arise• Enterprise internet connectivity can be inefficient• Applications suffer• Economy suffers• Network is less resilient

Issues arise• What makes a market secondary? • San Francisco Bay Area: population ~7.5million• Phoenix Metropolitan Area: population ~4.5million• Does that mean cities/regions with less than 5 million people

suffer?• Los Angeles <> Phoenix = ~370 miles• Los Angeles <> San Jose = ~330 miles

Issues arise• Dublin: population ~1.8Million• Need more people to move in before better internet

connectivity comes around?

Issues arise• What can drive improvement?• Is this lack of facilities? • Regulatory? • Do we care?

Take Backs

Take Backs

In a regional, or secondary market, don’t be surprised to have traffic

trombone!

Take Backs

User Experience Suffers…

Take Backs

Peering can help!

Take Backs

But, things are getting better….

Take Backs

More still needs to happen!

Questions?