relational and document database with mysql
TRANSCRIPT
Relational and Document Databases with MySQL MyNA(日本MySQLユーザ会)会 2016年5月 Twitter: @RDBMS
本内容は個人の見解であり、 所属組織を代表するものではありません。
MySQLドキュメントストア
3
ドキュメントデータ
• 形式・様式 (ex: XML, JSON) • ツリー構造 ( [], {} ) • スキーマレス
+-----------------------------------------------------------------------------+ | body | +-----------------------------------------------------------------------------+ | {"id": 1, "name": "自転車", "price": 10000, "Conditions": ["NEW", 2015]} | | {“id”: 2, “name”: “テレビ", "price": 30000, "Conditions": ["USED", 2013]} | | {"id": 3, "name": "冷蔵庫", "price": 10438, "Conditions": ["NEW", 2015]} | | {"id": 4, "name": "冷蔵庫", "price": 50000, "Conditions": ["NEW", 2015]} | | {"id": 5, "name": "自転車", "price": 25000, "Conditions": ["NEW", 2015]} | +-----------------------------------------------------------------------------+
Document Store
• 高速な検索 • 更新の最適化 • オペレーション
4
CREATE TABLE T_JSON_DOC (body json);
MYSQLのドキュメントストア機能拡張
5
• JSONサポート JSONデータ型(utf8mb4)・JSONファンクション Generated ColumnとIndexによる検索の高速化
MySQL 5.7.9(GA)
• X Protocol MySQLサーバをドキュメントストアとして拡張する為に、Xプラグイン(mysqlx)により実装
• X DevAPI SQL処理とドキュメントに対してのCRUD処理 Connector/Node.js, Connector/J, Connector/Netに実装
• mysqlsh コマンドラインクライアント (Javascript, Python, SQL)
MySQL 5.7.12
JSONデータ ネイティブJSONデータ型 (バイナリ形式) Insert時のJSON構文バリデーション機能 組み込みJSON関数 (保存、検索、更新、操作) ドキュメントにインデックス設定可能 SQLとの統合を容易にする新しいインライン構文 utf8mb4の文字セットとutf8mb4_binの照合
外部サイト/SNS
モバイル デバイス
コマース /ポータル
その他
(data JSON);
REST/JSON
SELECT NAME,CountryCode from world.City where CountryCode ='JPN' limit 1; +-------+-------------+ | NAME | CountryCode | +-------+-------------+ | Tokyo | JPN | +-------+-------------+ SELECT JSON_OBJECT('CITY',NAME,'Country',CountryCode) from world.City where CountryCode ='JPN' limit 1; +------------------------------------------------+ | JSON_OBJECT('CITY',NAME,'Country',CountryCode) | +------------------------------------------------+ | {"CITY": "Tokyo", "Country": "JPN"} | +------------------------------------------------+
select feature from NEW57.features where json_extract(feature,'$.properties.STREET') = 'MARKET' limit 1¥G ************** 1. row ************** feature: {"type": "Feature", "geometry": {"type": "Polygon", "coordinates": [[[-122.39836263491878, 37.79189388899312, 0], [-122.39845248797837, 37.79233030084018, 0], [-122.39768507706792, 37.7924280850133, 0], [-122.39836263491878, 37.79189388899312, 0]]]}, "properties": {"TO_ST": "388", "BLKLOT": "0265003", "STREET": "MARKET", "FROM_ST": "388", "LOT_NUM": "003", "ST_TYPE": "ST", "ODD_EVEN": "E", "BLOCK_NUM": "0265", "MAPBLKLOT": "0265003"}}
MySQL 5.7.12 ～ コネクター, ドライバー, プロトコル
MySQL
Plugins
X Protocol Plugin Memcached Plugin Core
MySQL Connectors and Drivers
X Protocol Std Protocol
Memcached driver
X Protocol 33060
Std Protocol 3306
SQL API CRUD and SQL APIs
Memcache Protocol
X and Std Protocols
MySQL Shell
X Protocol
•非同期APIサポート – 並列処理とバッチ処理をサポート
–パイプライン方式 – 複数リクエストを送信, ラウンドトリップを削減
– CRUD ==大量に小さなPKを処理, 独立した複数のクエリーを処理
• ミドルウエアとの親和性
–ルーティング、シャーディング、読み取り書き込みスプリッティング (XSESSION)
• オープンスタンダードの利用: TLS (Transport Layer Security), SASL(Simple Authentication and Security Layer), Protobuf (Protocol Buffers)等
8
Protobuf:
https://developers.google.com/protocol-buffers/
+-------------+----------------+--------------------+ | PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_DESCRIPTION | +-------------+----------------+--------------------+ | mysqlx | 1.0 | X Plugin for MySQL | +-------------+----------------+--------------------+
The X Protocol focuses on: • extensibility • performance • security
INSTALL PLUGIN mysqlx SONAME 'mysqlx.so';
X Protocol Query Time
Client Server Network
Stage Time
network path latency 1ms
exectime 0. 1ms
クラッシックリクエスト/レスポンス Total: 4x path + 2x exectime = 4.2ms
パイプライン処理 Total: 2x path + 2x exectime = 2.2ms
9
Pipelining messages is a core feature of the Mysqlx Protocol. It sends messages to the server without waiting for a response to save latency. (no mandatory handshake) 参考) https://dev.mysql.com/doc/internals/en/x-protocol-messages-message-structure.html
X Protocol
10
エラーを無視して続行: Mysqlx.Expect::Open([-no_error]) 最初のエラーで失敗 : Mysqlx.Expect::Open([+no_error])
With expectations pipelined, the server will handle errors in a consistent, reliable way. In case error reporting isn't a major topic one can combine multi-row INSERT with pipelining and reduce the per-row network overhead. This is important in case the network is saturated.
詳細： https://dev.mysql.com/doc/internals/en/x-protocol.html
Expectation
Mysqlx.Expect::Open([-no_error])
参考情報
Spec: http://dev.mysql.com/doc/internals/en/x-protocol.html
Message Def: https://github.com/mysql/mysql-server/tree/5.7/rapid/plugin/x/protocol Protobuf: https://developers.google.com/protocol-buffers/
$ protoc -I --python_out=... .../mysql.proto
11
Pipeline https://dev.mysql.com/doc/internals/en/x-protocol-implementation-pipelining.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/crud_pipe.test Expectations https://dev.mysql.com/doc/internals/en/x-protocol-expect-expectations.html https://github.com/mysql/mysql-server/blob/5.7/mysql-test/suite/xplugin/t/expect_noerror.test
独自クライアントを作成する場合
Pipeline及びExpectationsに関して
X DevAPI
• X Pluginを有効にする事で、X Protocol経由で通信可能 • ドキュメントとテーブルのコレクションに対してのCRUD処理 • NoSQLライクな構文でドキュメントに対しCRUD処理可能 • Fluent API
prod = sess.getSchema("prod") res = prod.users. find("$.name = 'Milk'"). fields(["name", "properties"])
X Plugin (MySQL) ⇔ X Protocol ⇔ X DevAPI (Driver)
12
MySQL Connector/node.js MySQL Connector/J MySQL Connector/Net MySQL Shell
MySQL Connectors include X Dev API • Use SQL, CRUD APIs
スキーマレスドキュメントおよびリレーショナルテーブルに対応 - Classic APIsに加えて、これらの全てが追加されます
13
Operation Document Relational
Create Collection.add() Table.insert()
Read Collection.find() Table.select()
Update Collection.modify() Table.update()
Delete Collection.remove() Table.delete()
参照) http://dev.mysql.com/doc/x-devapi-userguide/en/crud-operations-overview.html
14
[root@misc01 nodejs]# cat sample_node_X_API.js const mysqlx = require('mysqlx'); mysqlx.getSession({ host: 'localhost', port: 33060, dbUser: 'demo_user', dbPassword: 'password' }).then(function (session) { return session.createSchema("test_schema").then(function (schema) { return schema.createCollection("myCollection"); }).then(function (collection) { return Promise.all([ collection.add( {baz: { foo: "bar"}},{foo: { bar: "baz"}}).execute(), collection.find("$.baz.foo == 'bar'").execute(function (row) {console.log("Row: %j", row); }).then(function (res) {console.log("Collection find done!");}), collection.remove("($.foo.bar) == 'baz'").execute().then(function () { console.log("Document deleted");}), collection.drop() ]); }).then(function () { return session.dropSchema("test_schema"); }).then(function () { return session.close(); }); }).catch(function (err) { console.log(err.stack); process.exit(); }); [root@misc01 nodejs]# node sample_node_X_API.js Row: {"_id":"630f0d3b-f6fd-1d99-6d80-a8e90352","baz":{"foo":"bar"}} Collection find done! Document deleted Connector：mysql-connector-nodejs-1.0.2.tar.gz
2016-05-06T15:15:05.364983+09:00 37 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-06T15:15:05.369200+09:00 37 Query CREATE DATABASE `test_schema` 2016-05-06T15:15:05.382450+09:00 37 Query CREATE TABLE `test_schema`.`myCollection` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-06T15:15:05.464364+09:00 37 Query INSERT INTO `test_schema`.`myCollection` (doc) VALUES ('{¥"baz¥":{¥"foo¥":¥"bar¥"},¥"_id¥":¥"c5d6964d-af2a-0b87-36aa-4f5bc18b¥"}'),('{¥"foo¥":{¥"bar¥":¥"baz¥"},¥"_id¥":¥"3f106b27-e14a-31e0-5297-51da7f1c¥"}') 2016-05-06T15:15:05.481912+09:00 37 Query SELECT doc FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.baz.foo') = 'bar') 2016-05-06T15:15:05.528923+09:00 37 Query DELETE FROM `test_schema`.`myCollection` WHERE (JSON_EXTRACT(doc,'$.foo.bar') = 'baz') 2016-05-06T15:15:05.551710+09:00 37 Query DROP TABLE `test_schema`.`myCollection` 2016-05-06T15:15:05.565824+09:00 37 Query DROP DATABASE `test_schema` 2016-05-06T15:15:05.643891+09:00 37 Quit
General Log
MySQL Shell • 開発および管理用のシェルの統合 • 一般的なスクリプト・インターフェースを介して 利用可能な MySQLコマンドラインクライアント。 • PythonやJavaScriptなどのスクリプト言語で さまざまな製品と対話するための完全な開発用API
16
[root@misc01 admin]# mysqlsh --help | egrep -i "Start in" --sql Start in SQL mode using a node session. --sqlc Start in SQL mode using a classic session. --js Start in JavaScript mode. --py Start in Python mode. [root@misc01 admin]#
バッチ処理に利用可能 shell> mysqlsh --file code.js shell> mysqlsh < code.js shell> echo "show databases" | mysqlsh –sql 2016年5月現在:Development Release: 1.0.3
17
[root@misc01 MID2016]# mysqlsh --uri demo_user@localhost/NEW57 -ppassword Creating an X Session to demo_user@localhost:33060/NEW57 Default schema `NEW57` accessible through db. Welcome to MySQL Shell 1.0.3 Development Preview ……. Currently in JavaScript mode. Use ¥sql to switch to SQL mode and execute queries. mysql-js> db.createCollection("x_posts"); <Collection:x_posts> mysql-js> db.x_posts.add({"title":"Hello World", "text":"This is the first post via mysqlx"}); Query OK, 1 item affected (0.01 sec) mysql-js> db.x_posts.find("title = 'Hello World'").sort(["title"]); [ { "_id": "baee9a744308e61168170800279cea3c", "text": "This is the first post via mysqlx", "title": "Hello World" } ] 1 document in set (0.00 sec) MySQL Shell Version 1.0.3 Development Preview
session.getSchema(‘NEW57')
2016-05-26T23:24:45.694581+09:00 11 Query /* xplugin authentication */ SELECT `authentication_string`, <SNIP> 2016-05-26T23:24:45.695786+09:00 11 Query show databases 2016-05-26T23:24:45.697994+09:00 11 Query select schema(), @@lower_case_table_names 2016-05-26T23:24:45.698455+09:00 11 Query select connection_id() 2016-05-26T23:24:45.698899+09:00 11 Query use `NEW57` 2016-05-26T23:24:45.699441+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.701333+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:24:45.703443+09:00 11 Query SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2 2016-05-26T23:24:45.706091+09:00 11 Query SHOW FULL TABLES FROM `NEW57` 2016-05-26T23:25:15.862883+09:00 11 Query CREATE TABLE `NEW57`.`x_posts` (doc JSON,_id VARCHAR(32) GENERATED ALWAYS AS (JSON_UNQUOTE(JSON_EXTRACT(doc, '$._id'))) STORED NOT NULL UNIQUE) CHARSET utf8mb4 ENGINE=InnoDB 2016-05-26T23:25:38.026317+09:00 11 Query INSERT INTO `NEW57`.`x_posts` (doc) VALUES ('{¥"_id¥":¥"de3e8cb74d23e6112c1d0800279cea3c¥",¥"text¥":¥"This is the first post via mysqlx¥",¥"title¥":¥"Hello World¥"}') 2016-05-26T23:25:57.011923+09:00 11 Query SELECT doc FROM `NEW57`.`x_posts` WHERE (JSON_EXTRACT(doc,'$.title') = 'Hello World') ORDER BY JSON_EXTRACT(doc,'$.title')
root@localhost [NEW57]> SELECT table_name, COUNT(table_name) c FROM information_schema.columns WHERE ((column_name = 'doc' and data_type = 'json') OR (column_name = '_id' and generation_expression = 'json_unquote(json_extract(`doc`,''$._id''))')) AND table_schema = 'NEW57' GROUP BY table_name HAVING c = 2; +------------------------+---+ | table_name | c | +------------------------+---+ | Innovation_Day | 2 | | Innovation_Day_Confirm | 2 | | X_JSON | 2 | | x_posts | 2 | | X_PYTHON | 2 | +------------------------+---+
テーブル構造(doc json列＋generated columnのPK(_id) から、通常のテーブルとドキュメントストアを判別
mysql-js> db.getCollections() { "Innovation_Day": <Collection:Innovation_Day>, "Innovation_Day_Confirm": <Collection:Innovation_Day_Confirm>, "X_JSON": <Collection:X_JSON>, "X_PYTHON": <Collection:X_PYTHON>, "x_posts": <Collection:x_posts> }
mysql-js> db.getTables() { "T_GIS": <Table:T_GIS>, "T_JSON_DOC": <Table:T_JSON_DOC>, "T_JSON_DOC_TXT": <Table:T_JSON_DOC_TXT>, "T_TDE": <Table:T_TDE>, "citylots": <Table:citylots>, "employees": <Table:employees>, "employees_json": <Table:employees_json>, "employees_txt": <Table:employees_txt>,
MySQL5.7~ ドキュメント関連機能の連携デモ JSONデータ型, MySQL Shell, X Protocol, 生成列, FTS
Twitter API: (JSONデータ) https://api.twitter.com/1.1/xxxx
localhost, port:33060 Schema: NEW57
myDb = mySession.getSchema('NEW57') 省略… timeline = json.loads(res.text) for tweet in timeline: myDb.X_PYTHON.add(tweet).execute()
Shell> mysqlsh --py < demo_python_twitter.py
+-------+--------------+------+-----+---------+------------------+ | Field | Type | Null | Key | Default | Extra | +-------+--------------+------+-----+---------+------------------+ | doc | json | YES | | NULL | | | _id | varchar(32) | NO | PRI | NULL | STORED GENERATED | | name | varchar(64) | YES | MUL | NULL | STORED GENERATED | | text | varchar(512) | YES | MUL | NULL | STORED GENERATED | +-------+--------------+------+-----+---------+------------------+
(1)Twitter APIに接続しJSONフォーマットのTweetを取得
(2) 取得したJSONデータを.addでテーブルに追加
(3) Generated ColumnやFTSを利用してデータ参照処理
OAuth1Session
mysqlx
20
有難うございました
21
MySQL5.7からの新機能を活用してみて下さい!! 問題・改善要求： https://bugs.mysql.com/