criteo infraestructure: hadoop datacenter
TRANSCRIPT
CR ITEO INFRASTRUCTUREHA DOO P D A T A C E N T E RH U G R E N N E S - S E P T E M B E R , 3 0 T H 2 0 1 5
Matthieu BLUMBERGSenior Engineering ManagerInfrastructure Operations
3
Global OverviewOf Machines and Men
Why running our own infrastructure ?
• 2005 : no cloud• Dynamic content• Economies of scale• Availability• Latency
2011 2012 2013 2014
Criteo servers
€ per year / server
40 Gbps15.000 servers
+1M req/s+6MW
6
Hadoop NG ?We can haz moar servers !
Current DC is full
New design Scale outAutomationNon blocking
No vendor lock-in
PDU PDU
1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120
1101 1102 1103 1104 1105 1106 1107 1108 1109 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220
1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318
CRAC CRAC
1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
CRAC CRAC
1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518
1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618
1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716
CRAC CRAC
1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816
CC #4
CC #1 CC #2
CC #3
901Fabric4
902Fabric3
903Fabric2
904Fabric1
905Access5
906Access4
907Access3
908Access2
909Access1
910Edge2
911Edge1
912Infra2
913Infra1
Supe
rSpine
Spine
Edge
Infra
Door
Doo
r
Doo
r
Doo
r
Doo
r
Doo
rDoo
rDoo
r
Doo
r
Doo
r
Door
+9 POD2880 Nodes (Expansion)
+2 POD640 Nodes (Expansion)
Network
Power : 1 MWPower : 1 MW
5 POD1600 Nodes
2 POD : 640 Nodes
• Traditional redundant tree
• Suitable for vertical scaling
• SPOF on each level
• Cost-ineffective
• Efficient if using vendor features
n7k n7k
n5672 n5672
FEXFEX FEX
n5672 n5672
FEXFEX FEX
Servers
Not suitable for Hadoop !
Clos, Charles (Mar 1953). "A study of non-blocking switching networks". Bell System Technical Journal 32 (2): 406–424.
• Scales “infinitely” with commodity hardware
• Cable plan is easy and scalable
• Close to traditional network topologies
• Each station is at equal amount of hops
• Bandwidth is fully utilizedSuper Spine
Super Spine
ToR ToR
1 POD = 16 Server Racks
80 MTP (16 POD x 5 Uplink)
…
Super Spine
Super Spine
80 MTP (4 Rack Fabric x 20)
Spine1
Spine2
Spine3
Spine4
Spine5
4x 4x 4x 4x
16x
20x
Criteria ILIAD EQUINIX TELECITY
Infrastructure quality 4 5 5Site operability 4 4 3Room urbanization 5 4 2Site security 4 5 4site scalability 3 5 324/7 Services 4 4 4Pricing 4 5 3SLA 4 3 5Contract terms 4 5 3TOTAL (max 170) 140 156 126Without Pricing Criteria 77,4% 84,5% 71,6%Including Pricing Criteria 82,4% 91,8% 74,1%
Arista Juniper Cisco Dell HuaweiFeatures 5 5 5Capacity 5 5 3Documentation 4 5 1Support 4 5 4Pricing 4 5 3
22 25 16
Features 5 5 5 5Capacity 4 4 4 2Density 4 2 5 1Power 3 3 5 1Documentation 4 5 4 1Support 4 0 4 4Pricing 4 3 5 2
32 17 36 9
7 785 7 124 8 932 2 202 2 220 1 512 2 296
Total
ToR CostSuper Spines Cost
Total
Super Spines
ToR
1 an de projet
Hadoop1200 noeuds& 39 PBNe suffisent plus !
Juin 2014
• Design réseau validé• Vendeurs définis
Octobre 2014
• Premières livraisons• Construction de la salle
Février 2015
• Mise en baie• Câblage• Recette
Mai 2015
Serveurs commandésMars 2015
3 appels d’offres :• Datacenter• Réseau• Serveurs
Juillet 2014L’hébergeur est retenuDécembre 2014
Go live !
Juin 2015
Thank you [email protected]
