surrogate-assisted tuning for computer … tuning for computer experimentswith qualitative and...
TRANSCRIPT
Surrogate-Assisted Tuning for Computer Experiments with Qualitative and Quantitative Parameters
Ray-Bing Chen
Department of Statistics,
National Cheng Kung University
11
ATAT2018 in NCKU
• Tainan City
• 國立成功大學 (National Cheng Kung University)
44
� ������ � ���� � � �������������� ��� ������������
� ������������������ �
����� ���!��� � �� ����� ��� ���� ������������������ �
�"������#�$��!��� � �� ���� �� �� ����������
%�"��� �������� �
&��$�����!��� � �� ���� �� �� ����������
�������� �������"��
����������������� ���'��� �� �� �����(
Collaborators
Outlines
55
Tuning as an optimization problem
Surrogate Assisted Tuning (SAT)
Surrogate construction
Infill criteria
Numerical results
Conclusion
Optimization Problem
66
Motivation Problem
Find quantitative and qualitative factors to minimize run time of Algebraic Multigrid (AMG) solver
Search set: possible combinations 15*7*2*3=63077
Algebraic Multigrid Solver FactorsData type Factor Range Levels
Quantitative(continuous)
θ 0.02~0.11 [15]ω 0.6~1.8 [7]
Qualitative(category)
Smoother GS, SGS 2Cycle V, F, W 3
Motivation Problem
88
Minimize AMG execution time
Problem Domain
Auto-Tuning as an Optimization Problem
99
Target function
Minimal total time
Unknown response value
Expensive evaluation
Q&Q Input factors
x : Quantitative
z : Qualitative
Derivative Free Optimization
• Here the responses are generated via a “black-box function”.
• There is no close form of the objective function. • Derivative-free optimization approach:
• Grid Search Method• Direct Search and Pattern Search
–Kolda et al. (2003)–No gradient (derivative-free)–Iterative optimization algorithm–At each iteration, search the next point from a
pattern
)1,9.0(with |1)23(||12)23(|),(min
0
3/73/7
xxyyyxxyxf
Surrogate Assistant Approach
• Useful statistical tools:– Experimental designs– Model building
• Two well-known methods:– Response Surface Methodology (RSM)– Design and Analysis of Computer Experiment
(DACE)
• Response Surface Methodology– Box and Wilson (1951)– Noise response: y = f (x) + – Experimental Design + Regression model– Approximate f by a lower-order polynomial (Taylor
expansion)– Central Composite Design
• Design and Analysis of Computer Experiments (DACE)– Sacks et al. (1989) and Jones et al. (1998)– Model by a Gaussian Process:
– Kriging method or radial bases
– Space-filling designs
)),(exp())(),(( and )()()(
2121 xxdxzxzCorrxzxfxy T
))(),(()( ),(ˆ)()(ˆ iii iiT xzxzCorrxrxrcxfxy
Surrogate Assisted Tuning (SAT)
1616
Surrogate Assisted Tuning
1717
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
1818
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
1919
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2020
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2121
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2222
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2323
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2424
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2525
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Surrogate Assisted Tuning
2626
Surrogate construction
Statistical models:
Gaussian Process (Kriging) (Sacks et al., 1989)
Overcomplete basis surrogate method(Wang, Chen, 2008, Chen, Wang, Wu, 2011)
Infill criterion
Expected Improvement (EI)(Jones et al., 1998)
Works well for stationary data with quantitative factors only
How about Q&Q?
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Q & Q Input Factors
2727
Quantitative and Qualitative (Q&Q) factors
Statistical model
Q&Q Gaussian ProcessReferences: Qian, Wu, & Wu. (2008); Zhou, Qian, & Zhou (2011)
Infill criterion
propose here
Focus
2828
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
SAT- Initial Design
2929
Qualitative sub-domains
Whole domain
Whole Domain
3030
Individual LHD
Latin Hypercube Design (LHD)
Individual LHD
LHD on each qualitative sub-domain
Chen, Hsieh, Hung, Wang (2013) 3131
Bad design LHD Space-filling LHD
Q & Q Design
Peter Z. G. Qian and C. F. Jeff Wu. Sliced space-filling designs. Biometrika, 96(4):945–956, 2009.
Jian-Feng Yang, C. Devon Lin, Peter Z.G. Qian, and Dennis K.J. Lin. Construction of sliced orthogonal Latin hypercube designs. Stat. Sin., 23(3):1117–1130, 2013
3232
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
SAT- Surrogate Construction
3333
“Divide-and-Conquer” approach
more smaller search space: 6 independent dim
strong relations for QN factors; no relations for QL factors
“All-in-One” approach
one large search space
relations for all variables
Ideas
3434
AMG qualitative factors: (smoother, cycle)
Qualitative sub-domains are
Divide-and-Conquer
3535
Smoother
SGS
GS
Cycle V F W
“Divide-and-Conquer” approachIndependent Gaussian Process
Sub-domain GP (SDGP)
“All-in-One” approachQualitative and Quantitative Gaussian Process
Whole domain QQGP (WDQQ)
Ideas
3636
SDGP: Independent Sub-Domain Surrogates
An surrogate in each sub-domain is constructed
Gaussian Process (GP) is applied
Statistical surrogate3737
SDGP construction - Step 1
3838
Assume observed data follows Gaussian Process
SDGP construction - Step 2
3939
Estimate R, β, σ2 from observations
WDQQ: Global Whole-Domain Q&Q Surrogate
Main idea:
To estimate of the correlations between QL factors
To share information between QL factors
To reduce the size of initial experiment design
Qian, Wu, & Wu. (2008); Zhou, Qian, & Zhou (2011)
4040
Summary on Surrogate construction
4141
SDGP WDQQ
Approach Divide-and-Conquer All-in-One
Domain Local domain Global domain
Estimation of R Cheap Expensive
Ill-condition of R Grows slower Grows faster
Size of initial design (may be reduced)
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
SAT - Infill Criterion
4242
Infill Criteria
Classical problem (quantitative only)
Classical Expected Improvement (Classical EI)
Q & Q Problem
Minimum-shared EI (MSEI)
Minimal Prediction (MP)
4343
Classical Expected Improvement (EI)
GP-based Prediction
Improvement function
Expected Improvement
4444
Classical EI
Expected Improvement
4545Figures taken/modified from Forrester: Engineering Design via surrogate Modelling: A Practical Guide (2008)
EI Criterion
EI criterion
Take x with the largest EI(x) as the next point
Theorem
EI(x) > 0 for all unexplored points
EI(x) = 0 for all explored points
4646
EI Formula
Balance between
Prediction-based local exploitation
Error-based global exploration
4747
Minimum-shared Expected Improvement
Motivation
Extend the EI to across all sub-domains
Add new design via maximizing EI among all sub-domains
Improvement function
WDEI
4848
Whole-Domain Expected Improvement
WDEI criterion
Take w with largest as the next point
4949
Sub-Domain Expected Improvement (SDEI)
Motivation
Minimize the target function is our main goal
Approach
Step 1: Choose the sub-domain with minimum prediction
Step 2: Add an new design in that domain with EI criterion
5050
Minimal Prediction
Summary on Infill Criteria
5151
SDEI WDEI
Idea Filter sub-domain by minimal prediction Extend classical EI
Search Domain Local Global
Exploitation/Exploration Exploit. > Explore. Balanced
Features Cheaper; may stick to local minimum
More expensive; globally converge
SAT Framework
5252
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
SDGPWDQQSDGPWDQQ
SDEIWDEISDEIWDEI
Numerical Results
5353
Testing Examples
AMG solver (real data)
Oscillation-2D (gabor)
5454
Experiment Settings
Stopping criterion
M: maximum iteration number
Repetition for N times
Different initial designs
Robustness w.r.t. initial designs
5555
Metrics & Quantile Curve
5656
Metrics & Quantile curve
5757
Comparison
Robustness
Efficiency
Two surrogates
SDGP / WDQQ
Two infill criteria
SDEI / WDEI
5858
P1: AMG Solver
Find quantitative and qualitative factors to minimize run time of Algebraic Multigrid solver
Search set: possible combinations 15*7*2*3=630
5959
Algebraic Multigrid Solver FactorsData type Factor Range Levels
Quantitative(continuous)
θ 0.02~0.11 [15]ω 0.6~1.8 [7]
Qualitative(category)
Smoother GS, SGS 2Cycle V, F, W 3
P1: AMG True Surface (Six 15x7 Subdomain)(amg_ani_cg)
6060
V F W
SGS
GS
MinMin
P1: AMG Results (5%, 50%, 95% quantile, median) (amg_ani_cg)
6161
SDGP
WDQQ
WDEISDEI
Divide-and-Conquer
All-in-One
P1: AMG results
6262
P2: Oscillation (3 QL, 2 QN)
6363
P2: Oscillation (3 QL, 2 QN)
6464
SDGP
WDQQ
WDEISDEI
Divide-and-Conquer
All-in-One
P3: Oscillation (3 QL, 2 QN)
6565
Conclusion
6666
Guidelines
Surrogates
SDGP for simple surface
WDQQ for complicated surface
SDGP, if modeling resource is limited
Infill criteria
WDEI is suggested in general
SDEI is better if surrogates differs significantly
6767
Surrogate selection
6868
SDGP WDQQ
Surface Simple Complicated
Cost Cheap Expensive
Pros and Cons for surrogates
Infill Criterion selection
6969
SDEI WDEI
Simple Fast Slow
Oscillating Stick to local min Convergent globally
Convergence w.r.t. surface property
For large size of the qualitative factors:
Simplify the correlation structures for QQ-GP.
Treed Gaussian Process (tGP, Gramacy and Lee, 2008)
Other GP model structures:
Branching and Nested Factors
Performance Tuning of Next‐Generation Sequencing Assembly
Other models: Radial basis function (RBF)
Initial DesignInitial Design
Surrogate Construction
Surrogate Construction
Optimal?Optimal?Identify OptimaIdentify Optima
Function EvaluationFunction
Evaluation
New Design
New Design
Infill CriteriaInfill Criteria
Y
Thank you.
7171
Questions/comments/collaborations are welcome!