kauli sspにおけるvyosの導入事例

39
Case studies of VyOS in Kauli SSP Flandre Scarlet favorite Platform Engineer Kazuhito Ohkawa at Kauli, Inc.

Upload: kazuhito-ohkawa

Post on 20-Aug-2015

5.900 views

Category:

Technology


2 download

TRANSCRIPT

Page 1: Kauli SSPにおけるVyOSの導入事例

Case studies of VyOSin Kauli SSP

Flandre Scarlet favorite Platform Engineer

Kazuhito Ohkawa

at

Kauli, Inc.

Page 2: Kauli SSPにおけるVyOSの導入事例

Agenda

- Self‐Introduction- About Kauli SSP- Case studies in Kauli SSP of VyOS- Tuning tips - About microburst traffic(digress)

Page 3: Kauli SSPにおけるVyOSの導入事例

Self‐Introduction

- おおかわ かずひと Kazuhito Ohkawa(twitter@SatchanP)

- Aug 2012 Joined Kauli, Inc. Platform Engineer

- My Lover THE IDOLM@STER : Yayoi, Mami Touhou Project : Flandre, Sakuya

- Private Rallyist This is a my co-driver and three-dimensional parking of impreza.

Page 4: Kauli SSPにおけるVyOSの導入事例

About Kauli SSP

Page 5: Kauli SSPにおけるVyOSの導入事例

SSPとは

SSPとは、「Supply Side Platform」(サプライサイドプラットフォーム)の略で、オンライン広告において、広告枠を提供しているメディア(Webサービス、アプリデベロッパー)など媒体社の広告枠販売や広告収益最大化などを支援するツールのこと。 主に、広告のインプレッションが発生するごとに最適な広告を自動的に選択し、収益性の向上を図るという仕組みが提供されるが、アドネットワーク、アドエクスチェンジの一元的管理、リアルタイム入札(RTB)への対応など、具体的な提供機能はサービスによって異なる。

DSP、SSP - SMMLab(ソーシャルメディアマーケティングラボ)

smmlab.jp/?p=30268

Page 6: Kauli SSPにおけるVyOSの導入事例

About SSP

A supply-side platform or sell-side platform (SSP) is a technology platform with the single mission of enabling publishers to manage their advertising impression inventory and maximize revenue from digital media. As such, they offer an efficient, automated and secure way to tap into the different sources of advertising income that are available, and provide insight into the various revenue streams and audiences. Many of the larger web publishers of the world use a supply-side platform to automate and optimize the selling of their online media space.[1]

A supply-side platform on the publisher side interfaces to an ad exchange, which in turn interfaces to a demand-side platform (DSP) on the advertiser side.

This system allows advertisers to put online advertising before a selected target audience.[2] Often, real-time bidding (RTB) is used to complete DSP transactions.[3]。

http://en.wikipedia.org/wiki/Supply-side_platform

Page 7: Kauli SSPにおけるVyOSの導入事例

About RTB

Audience

Media

AD

Select the DSP in conditions.Request in parallel.

Request for SSP

Browse

Bid winner is DSP B

Kauli connected DSPs

Displayed DSP B's AD

Page 8: Kauli SSPにおけるVyOSの導入事例

Many connections for Ad delivery.Up to 400 million Ad per day.

All traffic via the VyOS.

Page 9: Kauli SSPにおけるVyOSの導入事例

Agony of SSP Platform Engineer

Very very very many many many traffics...

As well internal and external...

Various traffics, cookie sync, banner,

flash and movies, JS tags...etc...

About 80 % traffic is short packet...

Claim for delay of Ad...

SSP isn't profitable! Many media rewards!

Page 10: Kauli SSPにおけるVyOSの導入事例

SSP Handmade Servers

Page 11: Kauli SSPにおけるVyOSの導入事例

Infrastructure engineers of SSP.I can not recommend!

Page 12: Kauli SSPにおけるVyOSの導入事例

Case studies in Kauli SSP of VyOS

Page 13: Kauli SSPにおけるVyOSの導入事例

Mainly running on a physical server

Gen-1

Intel Core i7 870

RAM 16G

Intel 82574L x2

M/B ASUS

HDD

Gen-2

Intel Xeon E3-1280 v3

RAM 32G

Intel I350/I210

M/B Supermicro

SSD

Page 14: Kauli SSPにおけるVyOSの導入事例

Using at the Default Gateway for all servers

L3 Core

LVSDR

Real Servernginx

VyOSDefault GW

IP Masquarede

Internet

DMZ

LAN

SSP Server

RTB Requests

Page 15: Kauli SSPにおけるVyOSの導入事例

Peak traffic graphs of Default Gateway

Page 16: Kauli SSPにおけるVyOSの導入事例

Logic of LVS-DR

LVSVIP : 8.8.8.8

SRC : 8.8.4.4DST : 8.8.8.8

Real ServerIP : 10.1.1.2

Client AIP : 8.8.4.4

SRC : 8.8.4.4DST : 8.8.8.8

MAC : 0000.0000.0000

lo : 8.8.8.8MAC : 0000.0000.0000

SRC : 8.8.8.8DST : 8.8.4.4

VyOSDefault GWIP : 10.1.1.1

Source address is LVS VIPSolved by MAC Address

Make possible by loopback

SRC : 8.8.8.8DST : 8.8.4.4

LAN

Internet

FP Filter off

Page 17: Kauli SSPにおけるVyOSの導入事例

Router is unnecessary, If server have global IPs

LVSVIP : 8.8.8.8

SRC : 8.8.4.4DST : 8.8.8.8

Real ServerIP : 8.8.8.9

Client AIP : 8.8.4.4

SRC : 8.8.4.4DST : 8.8.8.9

MAC : 0000.0000.0000

lo : 8.8.8.8MAC : 0000.0000.0000

SRC : 8.8.8.8DST : 8.8.4.4DMZ

Internet

Page 18: Kauli SSPにおけるVyOSの導入事例

Scaling VyOS router by OSPF/ECMP after replacement

L3 Core

LVSDR

Real Server L3 SwitchDefault GW

Internet

OSPF ECMP

VyOS VyOS VyOS

Other VlanReal Server

LVSDR

Page 19: Kauli SSPにおけるVyOSの導入事例

Checking new data center application by Cloud Bridge

Vyatta Vyatta

Internet

LVS-DRSSP Server

DB

KVS

Index

Cloud Bridge

SSP Server

New Data Center Old Data Center

IndexKVSDB

Internet

Page 20: Kauli SSPにおけるVyOSの導入事例

Sakura cloud between VPN

VyOS VyOS

Internet

Data Center Sakura Cloud

Internet

API Server

IPSec

Crawler Crawler

Page 21: Kauli SSPにおけるVyOSの導入事例

Tuning Tips

Page 22: Kauli SSPにおけるVyOSの導入事例

NUMA I/ONAPI

circular bufferCPU Affinityconntrack

Page 23: Kauli SSPにおけるVyOSの導入事例

Use a uni-processor server (NUMA I/O)

PCI Express controller is integrated into the CPU in the sandy bridge.High access costs between processors.or using memory mirroring...

NIC

CPU1 CPU2 RAMRAM

PCI Express

QPI

Page 24: Kauli SSPにおけるVyOSの導入事例

It is printed on motherbord

Page 25: Kauli SSPにおけるVyOSの導入事例

Reconsider the polling of buffer (NAPI)

Buffer overflows even Intel's I350.(Amazing!)It is set the maximum value at 4096.Confirmed with ifconfig and ethtool -S.

ifconfig:RX packets:1215382409979 errors:0 dropped:9836789 overruns:9836789 frame:0

ethtool -S:rx_no_buffer_count: 220474

Page 26: Kauli SSPにおけるVyOSの導入事例

Change the NAPI kernel parameters

- net.core.netdev_budget

Increase the processing queue.

- net.core.dev_weightShorten the polling sensation.

However CPU usage rises.

Page 27: Kauli SSPにおけるVyOSの導入事例

circular buffer

igb is not set to the maximum value.And too large buffer will cause a delay.Consider the balance to CPU by NAPI and circular buffers.

# ethtool -g eth0Ring parameters for eth0:Pre-set maximums:RX: 4096RX Mini: 0RX Jumbo: 0TX: 4096Current hardware settings:RX: 256RX Mini: 0RX Jumbo: 0TX: 256

# ethtool -G eth0 rx 4096 tx 4096

Page 28: Kauli SSPにおけるVyOSの導入事例

CPU Affinity

Case of multi-queue, specific cpu core only high load.Adjust these manually.

$ cat /proc/interrupts | egrep 'eth|CPU' CPU0 CPU1 CPU2 CPU3 50: 1406514518 0 0 0 PCI-MSI-edge eth0-rx-0 51: 84923776 383727140 0 0 PCI-MSI-edge eth0-tx-0 52: 2951 0 0 0 PCI-MSI-edge eth0 53: 2 31961537 1787069187 0 PCI-MSI-edge eth1-rx-0 54: 1 6218033 0 510452860 PCI-MSI-edge eth1-tx-0 55: 115 0 0 0 PCI-MSI-edge eth1

$ sudo cat /proc/irq/5[0-1,3-4]/smp_affinity0001000200040008

Page 29: Kauli SSPにおけるVyOSの導入事例

conntrack tuning

Here is the essential part in the IP Masquarede.Maybe 10G-40G class of IP Masquarede also possible.Established time is very short.The high cost of connection open and close processing.

Setting value depends on the memory.

Page 30: Kauli SSPにおけるVyOSの導入事例

conntrack parameter

- hash-sizeconntrack table hashes.Processed faster conntracks scan by hashed.Hash algorithm is chaining scheme.

- table-sizeRaw conntrack tables.

- expect-table-sizeUse FTP, SIP, H.323...http://conntrack-tools.netfilter.org/conntrack.html

Page 31: Kauli SSPにおけるVyOSの導入事例

Raw conntrack table samples

tcp 6 128 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43860 dport=80 packets=6 bytes=698 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43860 packets=4 bytes=419 [ASSURED] mark=0 secmark=0 use=2

Page 32: Kauli SSPにおけるVyOSの導入事例

Setting conntrack tables and hash size

- table-size CONNTRACK_MAX = RAMSIZE (bytes) / 16384 / (x / 32) x = 32bit or 64bit

- hash-size tablesize / 8

- expect-table-size In preference

Page 33: Kauli SSPにおけるVyOSの導入事例

True upper limit of conntrack

Focus on the status of the conntrack table.[ASSURED] is not dropping from conntrack tables.

Comparison with the [ASSURED] total value and the maximum value.

Sample:tcp 6 23 TIME_WAIT src=10.x.x.xx dst=1xx.xx.xx.xx sport=43708 dport=80 packets=6 bytes=663 src=1xx.xx.xx.xx dst=1x.x.x.xx sport=80 dport=43708 packets=4 bytes=542 [ASSURED] mark=0 secmark=0 use=2

Page 34: Kauli SSPにおけるVyOSの導入事例

Shorten the timeout of conntrack table

conntrack table is supposed to be used recursively.But our traffic has very many hosts.Unable to keep conntrack table.

Short set a time-out so it not overflow conntrack table.timeout { icmp 3 other 600 tcp { close 10 close-wait 1 established 10 fin-wait 10 last-ack 30 syn-recv 60 syn-sent 5 time-wait 3 } udp { other 30 stream 10 } }

Page 35: Kauli SSPにおけるVyOSの導入事例

Microburst traffic(digress)

Page 36: Kauli SSPにおけるVyOSの導入事例

About microburst traffic

Microburst is not visible, but our network have it.Can be confirmed by various phenomena.One example is a packet discard of switchs.

Page 37: Kauli SSPにおけるVyOSの導入事例

Read the signs of microburst

Expand the graph in a narrow range.Spikes confirm.

Page 38: Kauli SSPにおけるVyOSの導入事例

Read the signs of microburst

This is a poll of 1 minute sensation.Ave 85 Packets discard/sec = 85Packets * 60 = 5160

5160 packets lost in a moment.

I have prepared a movie today.

Page 39: Kauli SSPにおけるVyOSの導入事例

Thank you for your attention!