杜伟业 ( richard to) richardto@leccotech 技术总监 灵高科研有限公司
DESCRIPTION
DM10:Sybase SQL Expert 12.5 新一代智能 SQL 语句优化技术 The New Generation of Intelligent SQL Tuning Technology. 杜伟业 ( Richard To) [email protected] 技术总监 灵高科研有限公司. 什么是反应时间. 1/10秒是用户认为系统能够立即反应的极限. 1 秒是用户觉得没有被中断的极限. 10秒是用户能将注意力继续集中在与计算机的对话上的时间极限. - PowerPoint PPT PresentationTRANSCRIPT
www.leccotech.com Where the experts go for performance
DM10:Sybase SQL Expert 12.5DM10:Sybase SQL Expert 12.5
新一代智能新一代智能 SQLSQL 语句优化技术语句优化技术The New Generation of Intelligent SQL Tuning TechnologyThe New Generation of Intelligent SQL Tuning Technology
杜伟业 ( Richard To) [email protected]技术总监 灵高科研有限公司
www.leccotech.com Where the experts go for performance
什么是反应时间1/10 秒是用户认为系统能够立即反应的极限1 秒是用户觉得没有被中断的极限10 秒是用户能将注意力继续集中在与计算机的对话上的时间极限
引用自 Jakob Nielsen 的 “ Usability Engineering” 第五章(Morgan Kaufmann, San Francisco 出版 )
www.leccotech.com Where the experts go for performance
我们可以关注的地方可调整的部分 :硬件操作系统数据库网络应用程序
许多人在试图提高数据库应用程序性能时只将注意力放在了前三项:硬件、操作系统、数据库…
www.leccotech.com Where the experts go for performance
数据库系统各部分对性能的影响
应用程序数据库操作系统
硬件网络
www.leccotech.com Where the experts go for performance
40%
60%
应用程序其它
最 少 60% 的 数 据 库 性 能 问 题 是 由 应 用 程 序 引 起 的
性能的挑战
www.leccotech.com Where the experts go for performance
我们可以优化什么 ?•数据库设计问题•编程问题•索引问题•SQL 问题
www.leccotech.com Where the experts go for performance
什么是 SQL ? ( 结构化查询语言 )
访问和更新数据 对应用程序的性能产生最重大的影响 SQL 语句消耗 70% 到 90% 的数据库系统资源 SQL 语句可以有多种写法 易学,但难以精通 需要考虑在数据和系统环境变化后仍能达到最佳化
www.leccotech.com Where the experts go for performance
服务器收到 SQL语句SQL语句解析
数据库的优化器决定执行路径
变量赋值执行
数据库是如何处理 SQL 语句的 ?
www.leccotech.com Where the experts go for performance
SQL
数据库的优化器是如何工作的 ?内部重写并产生多种执行计划
执行计划1执行计划2执行计划3成本估计
执行计划 1 成本 =1000执行计划 2 成本 =3000执行计划 3 成本 = 500
执行
它是否尝试了所有的的途径来重写你的 SQL语句 ?
成本的估计有多精确 ?
www.leccotech.com Where the experts go for performance
优化器说明什么是执行计划SQL 语法如何影响优化器的决定
www.leccotech.com Where the experts go for performance
什么是 SQL 优化 ?
www.leccotech.com Where the experts go for performance
解决这个问题的传统途径
www.leccotech.com Where the experts go for performance
执行计划SET SHOWPLAN ONGO
SELECT * FROM EMPLOYEE WHERE EMP_ID > 73712GO
SET SHOWPLAN OFFGO
QUERY PLAN FOR STATEMENT 1 (at line 1).
STEP 1 The type of query is SELECT.
FROM TABLE EMPLOYEE Nested iteration. Table Scan. Forward scan. Positioning at start of table. Using I/O Size 2 Kbytes for data pages. With LRU Buffer Replacement Strategy for data pages.
Total estimated I/O cost for statement 1 (at line 1): 20600.
iSQL
www.leccotech.com Where the experts go for performance
IO 和时间统计信息SET STATISTICS IO ONGOSET STATISTICS TIME ONGO
SELECT * FROM EMPLOYEE WHERE EMP_ID > 73712GO
SET STATISTICS IO OFFGOSET STATISTICS TIME OFFGO
iSQL Server Message: Number 3631, Severity 10Line 1:Total actual I/O cost for this command: 0. Total writes for this command: 0 Server Message: Number 3631, Severity 10Line 1:Total actual I/O cost for this command: 0. Total writes for this command: 0 Execution Time 0. SQL Server cpu time: 0 ms. SQL Server elapsed time: 3614 ms.Parse and Compile Time 0. SQL Server cpu time: 0 ms.Table: EMPLOYEE scan count 1, logical reads: (regular=1030 apf=0 total=1030), physical reads: (regular=0 apf=0 total=0), apf IOs used=0 Server Message: Number 3631, Severity 10Line 1:Total actual I/O cost for this command: 2060. Total writes for this command: 0 Execution Time 16. SQL Server cpu time: 1600 ms. SQL Server elapsed time: 7730 ms.(18371 rows affected)Parse and Compile Time 0. SQL Server cpu time: 0 ms. Execution Time 0. SQL Server cpu time: 0 ms. SQL Server elapsed time: 0 ms.Parse and Compile Time 0. SQL Server cpu time: 0 ms.
www.leccotech.com Where the experts go for performance
Traceon
DBCC TRACEON (3604, 302, 310)GO
SELECT * FROM EMPLOYEE WHERE EMP_ID > 73712GO
iSQL Beginning selection of qualifying indexes for table 'EMPLOYEE',varno = 0, objectid 1593824790. The table (Allpages) has 18373 rows, 1030 pages, Data Page Cluster Ratio 0.998890
Table scan cost is 18373 rows, 1030 pages, using no data prefetch (size 2K I/O), in data cache 'default data cache' (cacheid 0) with LRU replacement
Selecting best index for the SEARCH CLAUSE:EMPLOYEE.EMP_ID > 73712.000000
Estimated selectivity for EMP_ID, selectivity = 0.999946.
Estimating selectivity of index 'EMPLOYEE_1961957491', indid 1 scan selectivity 0.999946, filter selectivity 0.999946 18372 rows, 1032 pages, index height 2, Data Row Cluster Ratio 1.000000, Index Page Cluster Ratio 0.000000, Data Page Cluster Ratio 0.998890
…..
www.leccotech.com Where the experts go for performance
使用 EXIST 还是 IN ?
SELECT * FROM A WHERE A.CITY IN (SELECT B.CITY FROM B)
SELECT * FROM A WHERE EXISTS (SELECT ‘X’ FROM B WHERE B.CITY = A.CITY)
EXIST or IN
www.leccotech.com Where the experts go for performance
表连接与子查询通情况下表连接 (Table join) 要好过子查询 (sub-query)
源语句 SELECT * FROM A WHERE A.CITY IN (SELECT B.CITY FROM B)重写后
SELECT A.* FROM A, B
WHERE A.CITY = B.CITY /* if B.CITY is unique in B */
www.leccotech.com Where the experts go for performance
如果你确实知道问题所在,可以强制使用一个特殊的表操纵路径( driving path ) 。源语句SELECT * FROM A, B WHERE A.STATE = B.STATE
重写后的语句/* 禁止使用 B 的索引来改变表操纵路径 (B drive A) */SELECT * FROM A, B WHERE A.STATE = ISNULL(B.STATE,B.STATE)
Joins — Change Table Driving Path
www.leccotech.com Where the experts go for performance
要使用索引就不能在该索引字段上进行操作源语句 /* A.key1 上的索引被使用 */SELECT * FROM A WHERE A.Key1 = @ver1 AND A.Key2 = @ver2
重写后 /* 禁止使用字段 A.key1 上的索引,使用字段 A.Key2 上的索引 */SELECT * FROM A WHERE ISNULL(A.Key1, A.key1) = @ver1 AND A.Key2 = @ver2
Use Indexes
www.leccotech.com Where the experts go for performance
学习使用强制选项 (Forces Options)
SET FORCEPLAN ONINDEX hintPARALLEL hintSET SORT_MERGE ONSET JTC ONSET TABLE COUNT
Forces
www.leccotech.com Where the experts go for performance
使用 FORCEPLAN 强制选项强制使用特殊的表操纵路径 (driving path) 。
原来的方法 /* 禁止使用 B 的索引来改变表操纵路径 (B drive A)*/SELECT * FROM A, B WHERE A.STATE = ISNULL(B.STATE,B.STATE)
用 FORCEPLAN重写后SET FORCEPLAN ONGOSELECT * FROM B, A WHERE A.STATE = B.STATEGO
Joins — Change Table Driving Path
www.leccotech.com Where the experts go for performance
强制索引 (Force Index)
SELECT * FROM EMPLOYEE WHERE EMP_ID > 73712
Force Index
备注:仅在事先知道使用使用索引 x 将会较快的情况下采用
(INDEX EMP_ID_INX) QUERY PLAN FOR STATEMENT 1 (at line 1).
STEP 1 The type of query is SELECT.
FROM TABLE EMPLOYEE Nested iteration. Table Scan. Forward scan. Positioning at start of table. Using I/O Size 2 Kbytes for data pages. With LRU Buffer Replacement Strategy for data pages.
Total estimated I/O cost for statement 1 (at line 1): 20600.
QUERY PLAN FOR STATEMENT 1 (at line 1).
STEP 1 The type of query is SELECT.
FROM TABLE EMPLOYEE Nested iteration. Index : EMP_ID_INX Forward scan. Positioning by key. Keys are: EMP_ID ASC Using I/O Size 2 Kbytes for index leaf pages. With LRU Buffer Replacement Strategy for index leaf pages. Using I/O Size 2 Kbytes for data pages. With LRU Buffer Replacement Strategy for data pages.
Total estimated I/O cost for statement 1 (at line 1): 23382.
www.leccotech.com Where the experts go for performance
变量使用上的普遍问题SELECT * FROM employee WHERE (emp_id > @range_lower OR @range_lower IS NULL) AND (emp_id < @range_upper OR @range_upper IS NULL)
使用配合输入范围边界的索引范围搜索 (index range search)
SELECT * FROM employee WHERE emp_id > ISNULL(@range_lower , min of emp_id) AND emp_id < ISNULL(@range_upper, max of emp_id)
www.leccotech.com Where the experts go for performance
SELECT * FROM A WHERE EXISTS (SELECT ‘x’
FROM B WHERE A.key = B.key)
SELECT * FROM A a1 WHERE EXISTS IN (SELECT ‘x’ FROM A a2, B
WHERE a1.key = B.key AND a1.Unique_Key = a2.Unique_Key)
复杂的 SQL 转化
语义上等同于 :
www.leccotech.com Where the experts go for performance
这条 SQL 语句有多少个不同的等价写法 ?
SELECT * FROM DEPARTMENT WHERE DPT_ID IN (SELECT EMP_DEPT FROM EMPLOYEE WHERE EMP_ID > 50000)
更多的信息:DEPARTMENT EMPLOYEE记录数 = 410 记录数 = 18370索引 - DPT_ID 索引 - EMP_ID
www.leccotech.com Where the experts go for performance
最新的 SQL 优化技术
基于人工智能 (A.I.) 的 SQL 转换技术
使用人工智能 (A.I.) 来重写 SQL 语句
www.leccotech.com Where the experts go for performance
ESR 人工智能引擎优化的 SQL 1
优化的 SQL 2优化的 SQL 3
优化的 SQL 4优化的 SQL 5
优化的 SQL 6
源代码
Sybase SQL Expert 12.5 使用的技术
不良的 SQL
最好的 SQL
外部 SQL重写(External SQL Rewriter --ESR)
www.leccotech.com Where the experts go for performance
人工智能 ( A.I.) SQL 转换技术
优化后的 优化后的 SQLSQL
SQL11 SQL12 SQL13 SQLn1 SQLn2 SQLn3 SQLnn…SQL1n…
SQL1 SQL2 SQL3 SQLn...
SQL111 SQL112 SQL113 SQL11n... SQLnn1 SQLnn2 SQLnn3 SQLnnn...
Feedback Searching Engine反馈式搜索引擎
www.leccotech.com Where the experts go for performance
什么是递归的 SQL 转换 (Recursive SQL
Transformation)?
SELECT * FROM AWHERE A.C1 IN (SELECT B.C1 FROM B WHERE EXISTS (SELECT ‘x’ FROM C WHERE B.C2=C.C2 ))
IN to EXISTS
EXISTS to IN
SELECT * FROM A WHERE EXISTS (SELECT ‘x’ FROM B WHERE EXISTS (SELECT ‘x’ FROM C WHERE B.C2=C.C2) WHERE A.C1=B.C1)
SELECT * FROM A WHERE A.C1 IN (SELECT B.C1 FROM B WHERE B.C2 IN (SELECT C.C2 FROM C))
IN to EXISTS
EXISTS to IN
SELECT * FROM A WHERE A.C1 IN (SELECT B.C1 FROM B WHERE EXISTS (SELECT ‘x’ FROM C WHERE B.C2=C.C2 ))
SELECT * FROM A WHERE EXISTS (SELECT ‘x’ FROM B WHERE B.C2 IN (SELECT C.C2 FROM C) AND A.C1=B.C1)
SELECT * FROM A WHERE A.C1 IN (SELECT B.C1 FROM B WHERE B.C2 IN (SELECT C.C2 FROM C))
SELECT * FROM A WHERE EXISTS (SELECT ‘x’ FROM B WHERE B.C2 IN (SELECT C.C2 FROM C) AND A.C1=B.C1)
SELECT * FROM A WHERE A.C1 IN (SELECT B.C1 FROM B WHERE EXISTS (SELECT ‘x’ FROM C WHERE B.C2=C.C2))
SELECT * FROM A WHERE EXISTS (SELECT ‘x’ FROM B WHERE EXISTS (SELECT ‘x’ FROM C WHERE B.C2=C.C2) AND A.C1=B.C1)
使用 2 个转换规则
www.leccotech.com Where the experts go for performance
SQL 优化技术 ESR
1 秒
2.4 秒
1.3 秒
0.3 秒
0.8 秒
SQL1SQL1SQL2SQL2SQL3SQL3SQL4SQL4SQL5SQL5SQL6SQL6SQL7SQL7SQL8SQL8SQL9SQL9SQL10SQL10SQL11SQL11SQL12SQL12SQL..SQL..SQLNSQLN
人工智能重写 执行计划 性能 结果
相同的结果相同的结果SQL10SQL10
或或SQL12SQL12
源代码SQL10SQL10
SQL12SQL12
www.leccotech.com Where the experts go for performance
Number of Records (Where X Between :A and :B)
Tim
e
SQL1
SQL2
Which SQL is the Best?
www.leccotech.com Where the experts go for performance
边做边学式训练在线执行计划帮助
SQL比较器 SQL优化器
www.leccotech.com Where the experts go for performance
演示 2
边做边学式训练SQL 比较器 & 执行计划帮助
www.leccotech.com Where the experts go for performance
Sybase 12.5 最新的技术
Sybase Abstract Plan
www.leccotech.com Where the experts go for performance
什么是 Abstract Plan?
捕获查询的 SQL语句并存放该查询的 abstract plan在系统表 Sysqueryplans 中
提交给数据库的 SQL语句会跟已存于数据库内的 SQL语句进行比较,如果匹配,存于数据库的 abstract plan 将会用来执行该查询。
www.leccotech.com Where the experts go for performance
Abstract Plan
优化器优化器 执行基于保存的
abstract plan 来产生执行计划
基于 数据库统计信息 来产生执行计划SQL
Sybase
www.leccotech.com Where the experts go for performance
使用 Abstract Plan- /* 创建 abstract plan 组 demo1 */- sp_add_qpgroup demo1- Go- /* 保存 abstract plan 到组 demo1 */- set plan dump demo1 on- Go- /* 执行 SQL */- Select * from A, B
- /* 从组 demo1 加载计划 (Plan) */- set plan load demo1 on- Go
- Execution Plan /* 下一次你重新执行同一条 SQL 语句 */- QUERY PLAN FOR STATEMENT 1 (at line 1).- Optimized using an Abstract Plan (ID : 1989579095).
www.leccotech.com Where the experts go for performance
保存 SQL 的计划 (Plan)- /* 在组 demo1 下创建 abstract plan */- /* 以使用字段 A.key2 上的索引 */
- CREATE PLAN- “SELECT * FROM A WHERE A.Key1 = 1 AND
A.Key2 = 2”- “( i_scan Index_key2 A ) - ( prop A - ( parallel 1 ) - ( prefetch 2 ) - ( lru ) - )” into demo1
www.leccotech.com Where the experts go for performance
使用 Abstract Plan 的好处 即使没有源代码的情况下也可以对 SQL语句进行优化 执行计划将不会受以下情况影响:
o 数据库升级引起的数据库内部优化器的改变o 新的 ASE 特性o 调整选项的改变,如:并行深度 (parallel degree) ,表分区 (table
partitioning) 和索引 (indexing)
www.leccotech.com Where the experts go for performance
示范
Abstract Plan使用 Abstract Plan 优化 SQL
www.leccotech.com Where the experts go for performance
使用 Abstract Plan 优化 SQL 语句SQL Expert
www.leccotech.com Where the experts go for performance
SQL 扫瞄器……………………………………………………………………………………………………………………………..
…… SQL1 ………… SQL2 …….……….. SQL3 …..…… SQL4 ………………………….... SQL5 ...……………...
……………… SQL6 …………….. SQL7 ………………………………. SQL8.………………………………………
….. SQL9 ……………………………………………… SQL10 ………………………………………………………..
……………………………. SQL11 …………………………………….. SQL12 …………… SQL13 ………………..
…………… SQL14 ………………………… SQL15 …………………………………………………………………..
………………………. SQL16 ……………………………… SQL17 ……………… SQL18 ……….…………………
………………………………………………………………………………………………………………………………
……………………………………………………………………………………………………………………………...
……………………. SQL1 ………………………………….………. SQL2 ………………………………………...…..
…………………………………….…… SQL3 …………….. SQL4 ……………………. SQL5 ...……………….……
…………….….. SQL6 …………………………… SQL7 ………………….…..…...…………………………………..
………………….…………….…. SQL8 ……………………….… SQL9…………… SQL10 ………………….……..
…..… SQL11 ………… SQL12 ………………………… SQL13 ………………………………..… SQL14 …...……..
.………………. SQL15 …………………… SQL16 …………………………………………..…………………………
…………. ………………………………………………………………………………………………………………….
程序 A
程序 B
SQL1 SQL2 SQL3 SQL4 SQL5
SQL6 SQL7 SQL8
SQL9 SQL10
SQL11 SQL12 SQL13
SQL14 SQL15
SQL16 SQL17 SQL18
SQL1 SQL2SQL3 SQL4 SQL5
SQL6 SQL7SQL8 SQL9 SQL10
SQL11 SQL12 SQL13 SQL14
SQL15 SQL16
www.leccotech.com Where the experts go for performance
定位有问题的 SQL 语句数据库对象或监视结果
程序源代码
反馈式搜索引擎 优化的 SQL 1优化的 SQL 2
优化的 SQL 3优化的 SQL 4
优化的 SQL 5优化的 SQL 6
有问题的 SQL 语句
SQL 语法优化器
SQL 扫瞄器
www.leccotech.com Where the experts go for performance
演示 3
SQL 扫瞄器定位有问题的 SQL 语句
www.leccotech.com Where the experts go for performance
高生产力高质量
高质量 VS 高生产力
www.leccotech.com Where the experts go for performance
谢 谢杜伟业 ( Richard To) [email protected]技术总监 灵高科研有限公司