Download - Kauli SSPにおけるVyOSの導入事例
Case studies of VyOSin Kauli SSP
Flandre Scarlet favorite Platform Engineer
Kazuhito Ohkawa
at
Kauli, Inc.
Agenda
- Self‐Introduction- About Kauli SSP- Case studies in Kauli SSP of VyOS- Tuning tips - About microburst traffic(digress)
Self‐Introduction
- おおかわ かずひと Kazuhito Ohkawa(twitter@SatchanP)
- Aug 2012 Joined Kauli, Inc. Platform Engineer
- My Lover THE IDOLM@STER : Yayoi, Mami Touhou Project : Flandre, Sakuya
- Private Rallyist This is a my co-driver and three-dimensional parking of impreza.
About Kauli SSP
SSPとは
SSPとは、「Supply Side Platform」(サプライサイドプラットフォーム)の略で、オンライン広告において、広告枠を提供しているメディア(Webサービス、アプリデベロッパー)など媒体社の広告枠販売や広告収益最大化などを支援するツールのこと。 主に、広告のインプレッションが発生するごとに最適な広告を自動的に選択し、収益性の向上を図るという仕組みが提供されるが、アドネットワーク、アドエクスチェンジの一元的管理、リアルタイム入札(RTB)への対応など、具体的な提供機能はサービスによって異なる。
DSP、SSP - SMMLab(ソーシャルメディアマーケティングラボ)
smmlab.jp/?p=30268
About SSP
A supply-side platform or sell-side platform (SSP) is a technology platform with the single mission of enabling publishers to manage their advertising impression inventory and maximize revenue from digital media. As such, they offer an efficient, automated and secure way to tap into the different sources of advertising income that are available, and provide insight into the various revenue streams and audiences. Many of the larger web publishers of the world use a supply-side platform to automate and optimize the selling of their online media space.[1]
A supply-side platform on the publisher side interfaces to an ad exchange, which in turn interfaces to a demand-side platform (DSP) on the advertiser side.
This system allows advertisers to put online advertising before a selected target audience.[2] Often, real-time bidding (RTB) is used to complete DSP transactions.[3]。
http://en.wikipedia.org/wiki/Supply-side_platform
About RTB
Audience
Media
AD
Select the DSP in conditions.Request in parallel.
Request for SSP
Browse
Bid winner is DSP B
Kauli connected DSPs
Displayed DSP B's AD
Many connections for Ad delivery.Up to 400 million Ad per day.
All traffic via the VyOS.
Agony of SSP Platform Engineer
Very very very many many many traffics...
As well internal and external...
Various traffics, cookie sync, banner,
flash and movies, JS tags...etc...
About 80 % traffic is short packet...
Claim for delay of Ad...
SSP isn't profitable! Many media rewards!
SSP Handmade Servers
Infrastructure engineers of SSP.I can not recommend!
Case studies in Kauli SSP of VyOS
Mainly running on a physical server
Gen-1
Intel Core i7 870
RAM 16G
Intel 82574L x2
M/B ASUS
HDD
Gen-2
Intel Xeon E3-1280 v3
RAM 32G
Intel I350/I210
M/B Supermicro
SSD
Using at the Default Gateway for all servers
L3 Core
LVSDR
Real Servernginx
VyOSDefault GW
IP Masquarede
Internet
DMZ
LAN
SSP Server
RTB Requests
Peak traffic graphs of Default Gateway
Logic of LVS-DR
LVSVIP : 8.8.8.8
SRC : 8.8.4.4DST : 8.8.8.8
Real ServerIP : 10.1.1.2
Client AIP : 8.8.4.4
SRC : 8.8.4.4DST : 8.8.8.8
MAC : 0000.0000.0000
lo : 8.8.8.8MAC : 0000.0000.0000
SRC : 8.8.8.8DST : 8.8.4.4
VyOSDefault GWIP : 10.1.1.1
Source address is LVS VIPSolved by MAC Address
Make possible by loopback
SRC : 8.8.8.8DST : 8.8.4.4
LAN
Internet
FP Filter off
Router is unnecessary, If server have global IPs
LVSVIP : 8.8.8.8
SRC : 8.8.4.4DST : 8.8.8.8
Real ServerIP : 8.8.8.9
Client AIP : 8.8.4.4
SRC : 8.8.4.4DST : 8.8.8.9
MAC : 0000.0000.0000
lo : 8.8.8.8MAC : 0000.0000.0000
SRC : 8.8.8.8DST : 8.8.4.4DMZ
Internet
Scaling VyOS router by OSPF/ECMP after replacement
L3 Core
LVSDR
Real Server L3 SwitchDefault GW
Internet
OSPF ECMP
VyOS VyOS VyOS
Other VlanReal Server
LVSDR
Checking new data center application by Cloud Bridge
Vyatta Vyatta
Internet
LVS-DRSSP Server
DB
KVS
Index
Cloud Bridge
SSP Server
New Data Center Old Data Center
IndexKVSDB
Internet
Sakura cloud between VPN
VyOS VyOS
Internet
Data Center Sakura Cloud
Internet
API Server
IPSec
Crawler Crawler
Tuning Tips
NUMA I/ONAPI
circular bufferCPU Affinityconntrack
Use a uni-processor server (NUMA I/O)
PCI Express controller is integrated into the CPU in the sandy bridge.High access costs between processors.or using memory mirroring...
NIC
CPU1 CPU2 RAMRAM
PCI Express
QPI
It is printed on motherbord
Reconsider the polling of buffer (NAPI)
Buffer overflows even Intel's I350.(Amazing!)It is set the maximum value at 4096.Confirmed with ifconfig and ethtool -S.
ifconfig:RX packets:1215382409979 errors:0 dropped:9836789 overruns:9836789 frame:0
ethtool -S:rx_no_buffer_count: 220474
Change the NAPI kernel parameters
- net.core.netdev_budget
Increase the processing queue.
- net.core.dev_weightShorten the polling sensation.
However CPU usage rises.
circular buffer
igb is not set to the maximum value.And too large buffer will cause a delay.Consider the balance to CPU by NAPI and circular buffers.
# ethtool -g eth0Ring parameters for eth0:Pre-set maximums:RX: 4096RX Mini: 0RX Jumbo: 0TX: 4096Current hardware settings:RX: 256RX Mini: 0RX Jumbo: 0TX: 256
# ethtool -G eth0 rx 4096 tx 4096
CPU Affinity
Case of multi-queue, specific cpu core only high load.Adjust these manually.
$ cat /proc/interrupts | egrep 'eth|CPU' CPU0 CPU1 CPU2 CPU3 50: 1406514518 0 0 0 PCI-MSI-edge eth0-rx-0 51: 84923776 383727140 0 0 PCI-MSI-edge eth0-tx-0 52: 2951 0 0 0 PCI-MSI-edge eth0 53: 2 31961537 1787069187 0 PCI-MSI-edge eth1-rx-0 54: 1 6218033 0 510452860 PCI-MSI-edge eth1-tx-0 55: 115 0 0 0 PCI-MSI-edge eth1
$ sudo cat /proc/irq/5[0-1,3-4]/smp_affinity0001000200040008
conntrack tuning
Here is the essential part in the IP Masquarede.Maybe 10G-40G class of IP Masquarede also possible.Established time is very short.The high cost of connection open and close processing.
Setting value depends on the memory.
conntrack parameter
- hash-sizeconntrack table hashes.Processed faster conntracks scan by hashed.Hash algorithm is chaining scheme.
- table-sizeRaw conntrack tables.
- expect-table-sizeUse FTP, SIP, H.323...http://conntrack-tools.netfilter.org/conntrack.html
Raw conntrack table samples
tcp 6 128 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43860 dport=80 packets=6 bytes=698 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43860 packets=4 bytes=419 [ASSURED] mark=0 secmark=0 use=2
Setting conntrack tables and hash size
- table-size CONNTRACK_MAX = RAMSIZE (bytes) / 16384 / (x / 32) x = 32bit or 64bit
- hash-size tablesize / 8
- expect-table-size In preference
True upper limit of conntrack
Focus on the status of the conntrack table.[ASSURED] is not dropping from conntrack tables.
Comparison with the [ASSURED] total value and the maximum value.
Sample:tcp 6 23 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43708 dport=80 packets=6 bytes=663 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43708 packets=4 bytes=542 [ASSURED] mark=0 secmark=0 use=2
Shorten the timeout of conntrack table
conntrack table is supposed to be used recursively.But our traffic has very many hosts.Unable to keep conntrack table.
Short set a time-out so it not overflow conntrack table.timeout { icmp 3 other 600 tcp { close 10 close-wait 1 established 10 fin-wait 10 last-ack 30 syn-recv 60 syn-sent 5 time-wait 3 } udp { other 30 stream 10 } }
Microburst traffic(digress)
About microburst traffic
Microburst is not visible, but our network have it.Can be confirmed by various phenomena.One example is a packet discard of switchs.
Read the signs of microburst
Expand the graph in a narrow range.Spikes confirm.
Read the signs of microburst
This is a poll of 1 minute sensation.Ave 85 Packets discard/sec = 85Packets * 60 = 5160
5160 packets lost in a moment.
I have prepared a movie today.
Thank you for your attention!