Michael Zelenkov Yandex.Search

Implementing IPv6 in Yandex.Search

IPv6 support inside Yandex.Search

inet6num: 2a02:6b8::/32 netname: RU-YANDEX-20080815 descr: YANDEX LLC country: RU org: ORG-YA1-RIPE admin-c: YNDX1-RIPE tech-c: YNDX1-RIPE status: ALLOCATED-BY-RIR mnt-by: RIPE-NCC-HM-MNT mnt-lower: YANDEX-MNT mnt-routes: YANDEX-MNT source: RIPE # Filtered organisation: ORG-YA1-RIPE org-name: YANDEX LLC org-type: LIR

First Yandex IPv6 net

Running IPv6 on internal technology network - fastbone

2009

Yandex uses two physically separated networks: backbone and fastbone.

dual  stack  

backbone  

fastbone  

802.1q

IPv4+IPv6

IPv4

dual  stack  

backbone  

fastbone  

802.1q

IPv6

IPv4

IPv4 IPv6

32 bit 128 bit

IPv4 IPv6

127.0.0.1 ::1

§2.2 RFC 4291

3. An alternative form that is sometimes more convenient when dealing with a mixed environment of IPv4 and IPv6 nodes is x:x:x:x:x:x:d.d.d.d, where the 'x's are the hexadecimal values of the six high-order 16-bit pieces of the address, and the 'd's are the decimal values of the four low-order 8-bit pieces of the address (standard IPv4 representation).

…………… ws5-023.yandex.ru IN A 93.158.151.38 ws5-027.yandex.ru IN A 93.158.151.39 ws5-031.yandex.ru IN A 93.158.151.40 …………… ws5-023.search.yandex.net IN AAAA 2a02:6b8:0:287::93.158.151.38 ws5-027.search.yandex.net IN AAAA 2a02:6b8:0:287::93.158.151.39 ws5-031.search.yandex.net IN AAAA 2a02:6b8:0:287::93.158.151.40 ……………

DNS IPv4-mapping

Welcome to FreeBSD! # ping6 2a02:6b8:0:14a1::178.154.167.10 PING 2a02:6b8:0:14a1::178.154.167.10(2a02:6b8:0:14a1::b29a:a70a) 56 data bytes 64 bytes from 2a02:6b8:0:14a1::b29a:a70a: icmp_seq=1 ttl=64 time=0.029 ms ^C --- 2a02:6b8:0:14a1::178.154.167.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.029/0.029/0.029/0.000 ms

/etc/rc.conf.d/ip6addrctl ipv6_enable="YES” /etc/rc.conf.d/auto_linklocal ipv6_enable="YES" sysctl net.inet6.ip6.accept_rtadv=0

IPv6 support on FreeBSD

IPv4 backbone network/mask IPv6 backbone network/mask vlan IPv4 fastbone network/mask vlan IPv6 fastbone network/mask vlan myt4 178.154.167.0/25 2a02:6b8:0:14a1::/64 667 178.154.151.0/25 767 2a02:6b8:0:a05::/64 764 # . /etc/ya.subr # ya_network_info 178.154.167.1 ya_net_num=99 ya_DC=myt4 ya_net=178.154.167.0/25 ya_VLAN=667 ya_ip=178.154.167.1 ya_defrouter=178.154.167.126 ya_porter=178.154.167.1 ya_porter_backup=178.154.164.1 ya_porters="178.154.167.1 178.154.164.1 178.154.165.1 178.154.172.1 178.154.173.1 178.154.174.1 178.154.175.1 178.154.149.1" ya_broadcast=178.154.167.127 ya_netmask=255.255.255.128 ya_net6=2a02:6b8:0:14a1::/64 ya_ip6=2a02:6b8:0:14a1::178.154.167.1 ya_defrouter6=fe80::1 ya_prefix6=64 ya_FB_net=178.154.151.0/25 ya_FB_VLAN=767 ya_FB_ip=178.154.151.1 ya_FB_defrouter=178.154.151.126 ya_FB_netmask=255.255.255.128 ya_FB_ipv6_net=2a02:6b8:0:a05::/64 ya_FB_ipv6_VLAN=764 ya_FB_ipv6_ip=2a02:6b8:0:a05::178.154.167.1 ya_FB_ipv6_defrouter=2a02:6b8:0:a05::1 ya_FB_ipv6_netmask=64 #

/etc/ya.subr

# cat /etc/rc.conf.local ipaddr="178.154.167.1" hostname="porter080.yandex.ru" adapter_options="-lro -tso" # # ifconfig | grep inet inet 178.154.167.1 netmask 0xffffff80 broadcast 178.154.167.127 inet6 fe80::22cf:30ff:fe00:8e8%em0 prefixlen 64 scopeid 0x4 inet6 2a02:6b8:0:14a1::b29a:a701 prefixlen 64 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc inet 127.0.0.1 netmask 0xff000000 inet 178.154.151.1 netmask 0xffffff80 broadcast 178.154.151.127 inet6 fe80::22cf:30ff:fe00:8e8%vlan767 prefixlen 64 scopeid 0xd inet6 2a02:6b8:0:a05::b29a:a701 prefixlen 64 inet6 fe80::22cf:30ff:fe00:8e8%vlan764 prefixlen 64 scopeid 0xe

#

/etc/rc.d/ya-netif & /etc/rc.d/ya-fastbone

# netstat -rn -f inet Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 178.154.167.126 UGS 0 52124803 em0 5.45.194.0/24 178.154.167.126 UGS 0 0 em0 5.45.202.0/24 178.154.167.126 UGS 0 0 em0 ............. Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 => default fe80::1%em0 UGS em0 ::1 link#12 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 .............. 2a02:6b8:b000::/64 fe80::1%em0 UGS em0 2a02:6b8:f000::/64 2a02:6b8:0:a05::1 UGS vlan764 fe80::/10 ::1 UGRS lo0 ..............

/etc/rc.d/ya-routing

dual  stack  

backbone  

fastbone  

802.1q

IPv6

IPv4+IPv6

# host ws5-000.yandex.ru ws5-000.yandex.ru has address 93.158.148.33 ws5-000.yandex.ru mail is handled by 10 outbound-relay.yandex.net. # host ws5-000.search.yandex.net ws5-000.search.yandex.net has IPv6 address 2a02:6b8:0:287::5d9e:9421 # host ws5-000.fb.yandex.ru ws5-000.fb.yandex.ru has IPv6 address 2a02:6b8:0:a02::5d9e:9421 # host fb-ws5-000.search.yandex.net fb-ws5-000.search.yandex.net has IPv6 address 2a02:6b8:0:a02::5d9e:9421 #

yandex.ru & search.yandex.net

named.conf

# hostname –s porter080 # # # cat /etc/resolv.conf # Generated on 05/07/13 at 14:49:10 by resolvconf for porter080 (inet 178.154.167.1 netmask 255.255.255.128) domain yandex.ru nameserver 127.0.0.1 nameserver 141.8.146.1 options timeout:1 attempts:1 # # host porter080 porter080.yandex.ru has address 178.154.167.1 porter080.yandex.ru mail is handled by 10 outbound-relay.yandex.net. # host porter080.search Host porter080.search not found: 3(NXDOMAIN) # host porter080.search.yandex.net porter080.search.yandex.net has IPv6 address 2a02:6b8:0:14a1::b29a:a701 #

short names & resolv.conf

troubles with ipmi

23

http://www.intel.com/support/network/sb/CS-028553.htm

24

http://lists.ipxe.org/pipermail/ipxe-devel/2012-August/001781.html

25

Some graphs

26

Some graphs

27

Some graphs

28

Some graphs

To the IPv6-future!

Michael Zelenkov

mzelenkov@yandex-team.ru @mzelenkov