[kgc 2012] online game server architecture case study performance and security

Performance and Security CTO Online Game Server Architecture Case Study

Performance Security Server Architecture Case Study Publishing Platform

On game, what is my role as system engineer?

TCP vs. UDP ? ODBC vs. ADO ? ? HDD vs. SSD ? DBC ? ? ? ? IP vs. DNS ? Questionnaire

Scale-out vs. Scale-up ? DB ? ? Memcached ? ? ? ? RAID 10 RAID5 ? SSD RAID ? Questionnaire

Books 2004

Lobby Gateway PvP World NAT Database DBC Validation Login GameDungeon Terms Chat Management Relay

Sample of Server Configuration N C Rookie Level CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW

Room Server: CP Server: Sample of Server Configuration CP Server: Function of Servers Patch Server: NAT Server: UDP Hole Punch Auth Server:

Sample of Server Configuration CP Server GameDB CP Server Function of Servers Rookie Level CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW

Sample of Server Configuration How can a client connect among CP Servers? CP Server L4 CP Server

Rookie Level Sample of Server Configuration Single point of failure CPSvr CPSvr CPSvr RoomSvr GameDB AuthSvr Client Client UserDB PatchSvr NATSvr DBGW UDP Game Data DBGW ? ? ? ? ? ? ?

Books 2011~2012

High Availability Virtualization Scale-out vs. Scale-up Cloud Distribute system Open Source CompilerScalability Terms Synchronous vs. Non-synchronous system KVS

Books 2001~2010

Cryptography Stream Cipher Kerckhoff Protocol Block CipherSSL CAPTCHA Key: Private, Public, Session, Symmetric Algorithm Security Hash Terms RSA

Online Game Server Platform Network Program DB Chat LobbyLogin Business Logic Cryptography The three key functions for game server

Online Game Server Platform Game + Dungeon + NPC + PvP World + Channel + Lobby Authentication + Login Chat + Messenger + Mail + Friends Management + Duplicate login + OP Tools Logical Architecture Module

Online game server Architecture

Build a new game server architecture Client and Server Lobby Client Web Launcher Game Client

Build a new game server architecture Authentication Servers Lobby1 Client Web What is the IP address or Hosts name of Authentication Server? Auth1 Auth1 UserDB

Build a new game server architecture Patch Servers Lobby1 Client Web What is the IP address or Hosts name of Authentication Server? Auth1 Auth1 UserDB Patch1

Game Server Farm Build a new game server architecture Lobby & Game Servers Lobby1 Client Web What is the IP address or Hosts name of Lobby Server? Lobby2 Lobby3 Lobby1 Game1 Game2 Game3 Game1

Game Server Farm Build a new game server architecture Web Server Farm Lobby1 Client What is the IP address or Hosts name of Lobby Server? Lobby2 Lobby3 Lobby1 Game1 Game2 Game3 Game1 Web Server Farm Web1 Web2 Web3 L4 Virtualization

Build a new game server architecture Authentication system Client ID & Password for authentication? Web Server Farm Web1 Web2 Web3 L4 Virtualization Authentication Cookie

Build a new game server architecture Authentication system Client ID & Password for authentication? Web Server Farm Web1 Web2 Web3 L4 Virtualization Authentication Cookie Authentication Cookie = ? Auth1 GameUserDB

Authentication Cookie 7aec312v1285345f31073df1e745caf6aeb0d3 cc0c9690b100b6c3011898a5b0d3fa22a98f52 cb6xf4c53e0344a1fe20c75670b01c310daab9 5a31b75f426a48722c7fbc16f8ec05e7988d36 8b94735b017048d77cffeb060cb20a602f8be3 e8f3ed6f8f43fffddc7bb7cdc6606df9df7a33b6 o42 javascript: document.cookie

Game Server Farm Build a new game server architecture Game Database Server Lobby1 Client WebLobby2 Lobby3 Game1 Game2 Game3 Auth1 UserDB Patch1 GameDB

K Online Architecture C/S MMORPG

Online game system Client Internet Game Server Game DB Client display & Business logic process Client side internet bandwidth Business logic process Database performance Bottleneck points

Client display & Business logic process Video RAM & Bottleneck points

Client Display Bottleneck points

( ) ( ) Game Server & Database Performance , . , . DB . Bottleneck points resolve

One game server Game DB Game Server Bottleneck points resolve

Space Partition Method Game DB Game Server Game Server Game Server Game Server Bottleneck points resolve

Space Copy Method Game DB Game Server Game Server Game Server Game Server Bottleneck points resolve

Instance Dungeon Method Game DB Game Server ID#1 ID#2 ID#3 Bottleneck points resolve

Parallel World Method Game DB#1 Game Server Game DB#2 Game Server Game DB#3 Game Server Game DB#4 Game Server Bottleneck points resolve

Use in combination with Parallel World and Space Copy Method Game DB#1 Game Server Game Server Game Server Game Server Game DB#3 Game Server Game Server Game Server Game Server Game DB#2 Game Server Game Server Game Server Game Server Game DB#4 Game Server Game Server Game Server Game Server Bottleneck points resolve

All in One Game DB#1 Game Server Game Server Game Server Game Server 1 2 3 Game DB#3 Game Server Game Server Game Server Game Server 1 2 3 Game DB#2 Game Server Game Server Game Server Game Server 1 2 3 Game DB#4 Game Server Game Server Game Server Game Server 1 2 3 Bottleneck points resolve

Character Database Bottleneck points resolve Game DB#1 Server Server Server Server 1 2 3 Game DB#2 Server Server Server Server 1 2 3 Game DB#3 Server Server Server Server 1 2 3 Game DB#4 Server Server Server Server 1 2 3 Billing DB Character DB

Web: 10% < 90% Game: 90% > 10% SSD: Solid State Drive Write : Read KVS: Key-Value Store How to resolve Database Table Compress Compare between Web and Game DB

Scale Out vs. Scale Up

Scale horizontally (scale out) Database scalability: http://en.wikipedia.org/wiki/Scalability One technique supported by most of the major database management system (DBMS) products is the partitioning of large tables, based on ranges of values in a key field. In this manner, the database can be scaled out across a cluster of separate database servers

Scale vertically (scale up) With the advent of 64-bit microprocessors, multi-core CPUs, and large SMP multiprocessors, DBMS vendors have been at the forefront of supporting multi-threaded implementations that substantially scale up transaction processing capacity Database scalability: http://en.wikipedia.org/wiki/Scalability

Storage Area Network Scale up HDD vs. SSD Database: Hardware spec

RAID Configuration Scale up RPM(Revolutions Per Minute) Hardware BUS Interface

RPM Scale up High speed disk HDD Spindle [rpm] Average rotational latency [ms] 4,200 7.14 5,400 5.56 7,200 4.17 10,000 3.00 15,000 2.00 Game Database

Scale up RAID Configuration http://en.wikipedia.org/wiki/RAID

Level Minimum # of drives Read Benefit Write Benefit Image RAID 0 2 nX nX RAID 1 2 nX 1X RAID 4 3 (n-1)X (n-1)X RAID 5 3 (n-1)X (n-1)X

Nested (hybrid) RAID RAID 0+1: striped sets in a mirrored set (minimum four drives; even number of drives) provides fault tolerance and improved performance but increases complexity. The key difference from RAID 1+0 is that RAID 0+1 creates a second striped set to mirror a primary striped set. The array continues to operate with one or more drives failed in the same mirror set, but if drives fail on both sides of the mirror the data on the RAID system is lost. RAID 1+0: (a.k.a. RAID 10) mirrored sets in a striped set (minimum four drives; even number of drives) provides fault tolerance and improved performance but increases complexity. The key difference from RAID 0+1 is that RAID 1+0 creates a striped set from a series of mirrored drives. The array can sustain multiple drive losses so long as no mirror loses all its drives. From Wikipedia

RAID 10 versus RAID 5 A common opinion (and one which serves to illustrate the dynamics of proper RAID deployment) is that RAID 10 is inherently better for relational databases than RAID 5, because RAID 5 requires the recalculation and redistribution of parity data on a per-write basis. While this may have been a hurdle in past RAID 5 implementations, the task of parity recalculation and redistribution within modern storage area network (SAN) appliances is performed as a back-end process transparent to the host, not as an in-line process which competes with existing I/O. (i.e. the RAID controller handles this as a housekeeping task to be performed during a particular spindle's idle timeslices, so as not to disrupt any pending I/O from the host.) The "write penalty" inherent to RAID 5 has been effectively masked since the late 1990s by a combination of improved controller design, larger amounts of cache, and faster drives. The effect of a write penalty when using RAID 5 is mostly a concern when the workload cannot be de-staged efficiently from the SAN controller's write cache. From Wikipedia

RAID 10 versus RAID 5 The choice between RAID 10 and RAID 5 for the purpose of housing a relational database depends upon a number of factors (spindle availability, cost, business risk, etc.) but, from a performance standpoint, it depends mostly on the type of I/O expected for a particular database application. For databases that are expected to be exclusively or strongly read-biased, RAID 10 is often chosen because it offers a slight speed improvement over RAID 5 on sustained reads and sustained randomized writes. If a database is expected to be strongly write-biased, RAID 5 becomes the more attractive option, since RAID 5 does not suffer from the same write handicap inherent in RAID 10; all spindles in a RAID 5 can be utilized to write simultaneously, whereas only half the members of a RAID 10 can be used. However, for reasons similar to what has eliminated the "read penalty" in RAID 5, the 'write penalty' of RAID 10 has been largely masked by improvements in controller cache efficiency and drive throughput. From Wikipedia

Scale up http://en.wikipedia.org/wiki/SATA Name Raw bandwidth(Mbit/s) Transfer speed(MB/s) eSATA 3,000 300 SATA revision 3.0 6,000 600 SATA revision 2.0 3,000 300 SATA revision 1.0 1,500 150 USB 3.0 5,000 400 USB 2.0 480 60 Fibre Channel over optic fibre 10,520 1,000 Fibre Channel over copper cable 4,000 400 Thunderbolt 10,000 1,250 Bus Interface

Performance Charts HDD vs. SSD HDD: Write Access Times: score (in ms) [SRC] http://www.tomshardware.com/charts/hard-drives-and-ssds,3.html

Performance Charts HDD vs. SSD HDD: Read Access Times: score (in ms)

Performance Charts HDD vs. SSD SSD: Write Access Times: score (in ms) x80

Performance Charts HDD vs. SSD SSD: Read Access Times: score (in ms)

Performance Charts HDD vs. SSD SSD: Sequential Write: score (in MB/s)

Performance Charts HDD vs. SSD SSD: Sequential Read: score (in MB/s)

Angelbird Crest 6 Master Samsung MZ-7PC256N Samsung MCCOE64G5MPP IOMeter Web Server Benchmark 6537.10 6876.00 1241.85 IOMeter 4K Random Reads Benchmark 37298.00 58946.00 6166.33 IOMeter Database Benchmark 9883.00 8910.00 319.63 CrystalDiskMark Random Read, 4KB(QD =32) [MB/s] 212.00 319.00 25.22 CrystalDiskMark Random Write, 4KB(QD =32) [MB/s] 19.00 154.00 0.76 CrystalDiskMark Sequential Read, [MB/s] 184.00 403.00 100.54 CrystalDiskMark Sequential Write, [MB/s] 262.00 537.00 78.96 CrystalDiskMark Random Read, 4KB(QD =1) [MB/s] 33.00 26.00 24.27 CrystalDiskMark Random Write, 4KB(QD =1) [MB/s] 512.00 121.00 0.74 Performance Charts HDD vs. SSD SSD: Compare with Angelbird and Samsung x5 x6 x30 x8 x25 x1.8 x3.3 x1.3 x691 x5 x9 x27 x12 x202 x4 x6 x1 x163

Performance Charts HDD vs. SSD ORCA Disk Run Percent HDD SSD

RAID configuration for SSD Papers

Flash SSD RAID Papers Intels iOmeter

Flash SSD RAID Papers

Scale out a-1, b-2, , z-26 Distribute Game Database System by Partitioning RDBMS . . . Game DB(1) Game DB(26) Login Server Lobby ServerUser . . . . . . . . .

Scale out Hash(user id) = n Distribute Game Database System by Partitioning Hash(user_id) = n RDBMS . . . Game DB(1) Game DB(26) Login Server Lobby ServerUser . . . . . . . . .

Game Server Farm Build a new game server architecture View of high availability Lobby1 Client Web Lobby2 Lobby3 Game1 Game2 Game3 Patch1Auth Database Server Farm GameDB UserDB M M S S Replication

Scale Out KVS(Key-Value Store) Database: Middleware User RDBMS Game DB Login Server Lobby Server . . . . . . . . . . . . KVS Server

Cache Write-through vs. Writer-back Write-back(Write-behind) Write is done synchronously both to the cache and to the backing store. Write-through Writing is done only to the cache. A modified cache block is written back to the store, just before it is replaced.

Write-through vs. Write-back Write-back cache is more complex to implement, since it needs to track which of its locations have been written over, and mark them as dirty for later writing to the backing store. The data in these locations are written back to the backing store only when they are evicted from the cache, an effect referred to as a lazy write. For this reason, a read miss in a write-back cache (which requires a block to be replaced by another) will often require two memory accesses to service: one to write the replaced data from the cache back to the store, and then one to retrieve the needed datum. http://en.wikipedia.org/wiki/Cache_(computing)#Writing_policies

Write-through vs. Write-back

http://ihchoi.tistory.com/11 Compress of Database Table * 1000 Page 48.5% , Row 77.8%

http://ihchoi.tistory.com/11 Compress of Database Table CPU 118% , Page 35.6% , Row 32.8%

http://ihchoi.tistory.com/11 Compress of Database Table CPU 235% , Page 249.8% , Row 150.8% * 100

Moors law Multi-core processor Memory SSD Virtualization Scale-out vs. Scale-up

5 6,000 , 3 8 (8) 360 (16) K Online Game System

K Online Game System Billing System K Online World2 W3 W4 W5 WEB 1 5 1 World1 2 6 2 3 7 4 8 3 6 0 Field Instance Game DB CU 6,000

K Online Game System SV Backend DBMS CU 6,000 Backup dbsv DBMS Backup logsv worldsv authsv Frontend Internet loginsv gmsv1 msgsv gmsv2 gmsv3 proxy proxy proxy proxy proxy cli cli cli

Case Study Game Client Web Board

W Publishing Platform WAG: Authentication WPG: Payment WPIN: Personal Identification Number WOTP: One Time Password Game Security WPCBG: PCBang WIRG: Item Register Game Support

W Publishing Platform WAG CashDB WPG WPCBG PC Bang DB Game Server XML-RPC JSON AVRO Channel Game Client AuthDB JDBC JDBC x

XML-RPC is a remote procedure call (RPC) protocol which uses XML to encode its calls and HTTP as a transport mechanism JSON-RPC is a remote procedure call protocol encoded in JSON. It is a very simple protocol (and very similar to XML-RPC), defining only a handful of data types and commands. http://en.wikipedia.org/wiki/XML-RPC http://en.wikipedia.org/wiki/JSON-RPC Remote procedure call (RPC) protocol Avro is a remote procedure call and serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. http://en.wikipedia.org/wiki/Thrift_(protocol)

W Personal Identification Number

W One Time Password

How to hack Diablo 3, Ive OTP?

2channel Authentication system Case by Dream Security

2channel Authentication system Compare 2channel + 2factor 1channel + 2 factor 2channel + 2factor ARS

SHA-3 http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html

SHA-3 The NIST hash function competition is an open competition held by the US National Institute of Standards and Technology for a new SHA-3 function to replace the older SHA-1 and SHA-2, which was formally announced in the Federal Register on November 2, 2007. NIST has selected five SHA-3 candidate algorithms to advance to the third (and final) round: BLAKE Grstl (Knudsen et al.) JH Keccak (Keccak team, Daemen et al.) Skein (Schneier et al.) http://csrc.nist.gov/groups/ST/hash/sha-3/Round3/index.html

SHA-3 NIST noted some factors that figured into its selection as it announced the finalists: Performance: "A couple of algorithms were wounded or eliminated by very large [hardware gate] area requirement it seemed that the area they required precluded their use in too much of the potential application space." Security: "We preferred to be conservative about security, and in some cases did not select algorithms with exceptional performance, largely because something about them made us 'nervous,' even though we knew of no clear attack against the full algorithm." Analysis: "NIST eliminated several algorithms because of the extent of their second-round tweaks or because of a relative lack of reported cryptanalysis either tended to create the suspicion that the design might not yet be fully tested and mature." Diversity: The finalists included hashes based on different constructions, including the HAIFA and sponge hash constructions, and hashes with different sources of nonlinearity, including S-boxes and the interaction between addition and XOR.

SHA-3 WINNER NIST announced Keccak as the winner of the SHA-3 Cryptographic Hash Algorithm Competition and the new SHA-3 hash algorithm in a press release issued on October 2, 2012. Keccak was designed by a team of cryptographers from Belgium and Italy, they are: Guido Bertoni (Italy) of STMicroelectronics, Joan Daemen (Belgium) of STMicroelectronics, Michal Peeters (Belgium) of NXP Semiconductors, and Gilles Van Assche (Belgium) of STMicroelectronics.

8F Seung Kwang Bldg., 143-8 Samsung-Dong, Gangnam-Gu, Seoul, 135-877, Korea / Seungmin Shin TEL: +82-2-2050-4307 E-MAIL: joo @ windysoft.net ! CONTACT ME

[kgc 2012] online game server architecture case study performance and security

Entertainment & Humor