kernel based swarm optimization for renewable energy application

Kernel-based Swarm Optimization for Renewable Energy Application

Seminar, FCI, Cairo University (17-July-2016)

Sarah Osama Talaat

للماجستير التسجيل محاضرةPre-master seminar

Acknowledgment Some slides are adapted from several

presentation and sites from the internet and the others including the idea is my own materials


2

Overview Kernel-based Method.

Kernel Trick (Kernel Substitutions). Kernel Function Support Vector Machine (SVM): An Example.

Meaning of Swarm. Meaning of Swarm Intelligence (SI) Characteristics of SI System. SI Advantages. Examples of SI Algorithms.


3

Overview

Problem Definitions.

Research Objectives.

Research Motivation (Applied Research).

Literature Review.

Proposed Model.

4


5

Kernel-based Method

The kernel-based approaches take their name from kernel function. It Allows us to build

interesting extensions of many well-known algorithms by making use of the kernel trick,

also known as kernel substitution.

The general idea is that, expand the dimensionality by a kernel function.


6

Kernel Functions Using a kernel function it transforms data from input space into a high-dimensional feature space in which

it searches for a separating hyperplane. So, that nonlinear data can also be separated using hyperplane in

high dimensional space.

Some commonly used kernel functions are polynomial kernels, Gaussian RBF kernels and Mercer kernels.

The RBF kernel is usually used for its flexibility in fitting data, other popular kernels such as the

polynomial or sigmoid.

Kernel function can be used in any machine learning (ML) methods like neural network (NN) and support

vector machine (SVM).


7

What is SVM

Support Vector Machine (SVM) is kernel-based supervised learning algorithm, which is the

combination of Machine learning theory, optimization algorithms from operation research and

kernel techniques from Mathematical analysis.

SVM’s that deal with classification problems are called Support Vector Classification (SVC)

and SVM’s that deal with modeling and prediction are called Support Vector Regression

(SVR).


8

Class AClass B

Linear SVM

Separating hyperplane

X0

SVM: An Example


Support Vectors (SV)

9

Class AClass B

Linear SVM

X0

How would you classify these data?

SVM: An Example Cont’d…


10

Class AClass B

X0

Any of these would be fine ? which is best?



11

Class AClass B

Linear SVM

X0

Margin

Margin is defined as the width that the boundary could be increased by before hitting a data point, that has no interior data points.



12

Class AClass B

Linear SVM

X0

The linear classifier with the maximum margin is the best, is an optimal hyperplane.



13

Class AClass B

Linear SVMX

Y



14

Class AClass B X

Here, what are we going to do?

NON-Linear SVM



15

Class AClass B X

Here, what are we going to do?

NON-Linear SVM



16

Class AClass B

NON-Linear SVM

XZ

Y



17

A swarm is a large number of

homogenous, simple agents interacting

locally among themselves, and their

environment.

Meaning of Swarm


18

Swarm Intelligence (SI) can be defined as a

relatively new branch of Artificial Intelligence

that is used to model the collective behavior of

social swarms in nature, such as ant colonies,

honey bees, and bird flocks.

Meaning of SI


19

Properties of SI System

Swarm intelligence system characterized by: It is composed of many individuals. The individuals are either all identical or belong to a few

typologies. The interactions among the individuals are based on simple

behavioral. The overall behavior of the system results from the

interactions of individuals with each other and with their environment.


20

SI AdvantagesThe main advantages of the swarm intelligence approach compared with a classical approach are the following:

Flexibility: SI systems are highly scalable; The control mechanisms used

in SI systems are not too dependent on swarm size.

Robustness: even when one ore more individuals fails, the group can

still perform its tasks.

Self organization: the group needs relatively little supervision or top

down control. Seminar, FCI, Cairo University (17-July-2016)

21

Examples of SI Algorithms


Chicken swarm optimization.

particle swarm optimization (PSO) algorithm.

Whale optimization algorithm.

Elephant optimization algorithm.

Bacterial foraging optimization algorithm, etc.

22

Problem Definitions


Machine learning methods (i.e. SVM) can't classify nonlinear data. The kernel- based approach make the

nonlinear data classify using linear classifier by making use Kernel-trick that use a kernel function.

Kernel function it use to mapped original input space to some higher-dimensional feature space in which

it searches for a separating hyperplane. So, that nonlinear data can also be separated using hyperplane in

high dimensional space.

Now we can say that the kernel function is a heart of kernel-based approaches. With kernel trick, it is

possible to efficiently accomplish a variety of machine learning tasks, (e.g. classification, regression and

clustering) that will impact on the training result, generalization performance and classification,

regression and clustering accuracy.

23

Problem Definitions


When we talk about kernel functions optimization we will look to it from to two aspect:1. Type of kernel functions: It belongs to polynomial family or exponential family.2. Meta parameters of kernel functions:

Each family has a set of Meta parameters the selection of suitable Meta parameters is one of major challenges and usually handled by try and error. For example, machine learning library like SVM lib selects Meta parameters by trying a range of values given by user.

Our aim is to get the optimal Meta parameters for kernel functions by using swarm optimization algorithms.

24


Research Objectives

Optimize Meta parameters of kernel functions using swarm

optimization algorithms.

Construct a prediction model for renewable energy.

Test the proposed model.

Research Motivation25

Important of research

(Applied research)


26

Literature Review


Optimization Algorithms

Conventional Algorithms

Nature-inspired metaheuristics

i.e. Swarm

27

Literature Review

Kernel function optimization using

conventional algorithms



28

Reference Optimization Algorithm

Machine Learning Method

Contribution

Ali, Shawkat, and Kate A. Smith, "Automatic parameter selection for polynomial kernel", Information Reuse and Integration, IRI 2003, IEEE International Conference on, IEEE, 2003.

Classical statistical theory

SVM Select a suitable parameter of polynomial kernel,

Meng, Xianbing, Yu Liu, Xiaozhi Gao, and Hengzhen Zhang, "A new bio-inspired algorithm: chicken swarm optimization." Advances in swarm intelligence. Springer International Publishing, 86-94, 2014.

chaos optimization algorithm (COA).

SVM Optimize the SVM key parameters.

Literature Review

29

Literature ReviewThis algorithms types have a set of demerits include:

The convergence to an optimal solution depends on the chosen optimal solution. Most algorithms may get stuck at local optimum. An algorithm efficient in solving one optimization problem may not be efficient in solving

a different optimization problem. They can find only a single optimized solution in a single simulation run. Algorithms are not efficient in handling problems having discrete variables. They become too slow if the number of variables are large. Algorithms can't be efficiently used on parallel machine.


30

Literature Review

Kernel function optimization using

nature-inspired metaheuristic algorithms


31

Literature Review

Because of the demerits in the conventional algorithms appeared this kind of algorithms which is designed for:

Search using structure without getting stuck. Don’t keep trying the same solutions. Combine one or more properties of good solutions when generating new solutions.

And is characterized by: It tends to move relatively quickly towards very good solutions, so it provides a

very efficient way of dealing with large complicated problems. Useful in cases where traditional methods get stuck at local minimas.


32



Contribution

Mucherino, Antonio, and Onur Seref, "Monkey search: a novel metaheuristic search for global optimization." Data Mining, Systems Analysis and Optimization in Biomedicine. Vol. 953. No. 1. AIP Publishing, 2007

Simulated

annealing

SVM • Optimized the parameters of kernel functions on 6 different kernel functions (Linear, Polynomial, Gaussian (Radial Basis Function(RBF) ), Sigmoid, Cauchy and Log).

• The result show that, the best improvements obtained when use (RBF, Polynomial and Linear), while other kernel achieved bad result even using without any optimization.

Literature Review


33



Contribution

Gandomi, Amir Hossein, Xin-She Yang, and Amir Hossein Alavi, "Cuckoo search algorithm: a metaheuristic approach to solve structural optimization problems", Engineering with computers 29.1, 17-35, 2013.

Simulated

annealing

SVMOptimized the parameters

of RBF kernel functions

Literature Review


34



Contribution

Xing, Bo, and Wen-Jing Gao, "Cat Swarm Optimization Algorithm."Innovative Computational Intelligence: A Rough Guide to 134 Clever Algorithms. Springer International Publishing, 93-104, 2014

Particle swarm

optimization

(PSO)

SVR • Proposed a HPSO–SVR model to improve the regression accuracy based on SVR parameters optimization.

• The results showed that the HPSO–SVR model outperforms the previous models.

• Specifically, the new HPSO–SVR model can successfully identify all the optimal values of the SVR parameters with the lowest prediction error values.

Literature Review


35



Contribution

Xin-She Yang, “Recent

Advances in Swarm

Intelligence and Evolutionary

Computation”, Studies in

Computational Intelligence,

Volume 585, Springer

International Publishing

Switzerland, 2015.

cuckoo search

and compare with:

grid search

(GS), genetic

algorithm (GA)

and particle

swarm

optimization

(PSO)

SVR • Optimize the parameters of SVR .

• The performance of cuckoo search to optimize the parameters of SVR outperforms that of traditional methods that are GS, GA, and PSO.

• Use three popular artificial intelligent algorithms, BP, Elman, and SVR.

• The SVR functions are the best among these two models.

Literature Review



36

Proposed Model

38

Any Questions!?


Thanks and Acknowledgement39


kernel based swarm optimization for renewable energy application

Education