icvs2013 - opencv for ios

8/10/2019 ICVS2013 - Opencv for IOS

1/38

OpenCV for iOS

ICVS 2013

Alexander Shishkov

[email protected]


2/38

Agenda

1.

OpenCV for iOS2.

Hello, World! application

3.

OpenCV camera example


3/38

Why bother?

Its fun

Its trendy

It can make you rich

Its impressive way to

show some technology


4/38

OpenCV for iOS

Started with GSoC 2011 project

Firstly released 2012-07-04 with OpenCV 2.4.2

Thanks to C++ !Objective C interoperabilitynative OpenCV for iOS just works (excludingsome parts of highguimodule):

working with camera reading/writing video files

namedWindow, imshow,


5/38


6/38

Changelog

2.4.6

fixed bug with incorrect video saving with CvVideoCamera

added rotateVideoflag to the CvVideoCameraclass

added functions for conversion between

UIImageand cv::Mat


7/38

iOS specific part

Camera stack

#import

CvAbstractCamera, CvPhotoCamera, CvVideoCamera

Conversion functions #import

UIImageToMat, MatToUIImage


8/38


9/38

iOS Anatomy

Apps

Media (CoreImage, CoreGraphics, OpenGL ES,

AVFoundation)

Cocoa Touch (UIKit etc.)

Core Services (Databases, iCloud, GDC, )

Core OS (Unix, POSIX, )


10/38

OpenCV+iOS Anatomy

Apps

Media (CoreImage, CoreGraphics, OpenGL ES,

AVFoundation)

Cocoa Touch (UIKit etc.)

Core Services (Databases, iCloud, GDC, )

Core OS (Unix, POSIX, )

OpenCV for iOS is just another framework


11/38

Framework layout


12/38

OpenCV Framework

Prebuilt framework

from opencv.org

From source code

cd ~/

git clone https://github.com/Itseez/opencv.git

git checkout b 2.4 origin/2.4

python opencv/platforms/ios/build_framework.py ios


13/38

Agenda

1.

OpenCV for iOS2.


3.



14/38


15/38

iOS Simulator iOS Device

Cost Free* $99 per year

Prerequisites OSX 10.7 + Xcode

4.3.x or later

+ iOS-5 capable

deviceTarget CPU x86 (SSE, no

Neon)ARM (Neon, no

SSE)

Speed fast slower

RAM lots of limited

OpenGL + +

Photo Library, StillPhoto processing

+ +

Camera - +

Accelerometer - +

Debugging,Logging

+ +

Performanceevaluation

+/- +


16/38


17/38

Hello world

#import

int main(int argc, char *argv[])

{

//printf("\n\nHello, world!\n");

@autoreleasepool {

//NSLog(@"\n\nHello, world!);

NSLog([[NSString stringWithUTF8String:"\n\nHello"]

stringByAppendingString:@", world!"]);

return 0;

}

Its a superset of C

Since OSX 10.7 & iOS 5 memory is automatically managed

#import == #include with automatic guards.

[recipient messageWithArg1:A andArg2:B!

];


18/38

Cocoa(Touch) key concept: MVC

View is created visually as aStoryboardelement

For each View component (image view,

button, slider) we add corresponding

IBOutlet-markedproperty in theController class

For each action we add IBAction-marked

method in the Controller

Model-View-Controller(For simple demos Model

is a part of Controller class.) View

Controller

Model

update update

user action notify


19/38

Class declaration

#import

@interface ViewController : UIViewController {

UIImage* image;

}

@property (weak, nonatomic) IBOutlet UIImageView *imageView;

@property (weak, nonatomic) IBOutlet UIBarButtonItem *loadButton;

-(IBAction)effectButtonPressed:(id)sender;

@end

@interface @end delimits class declaration

there must be one and only one base class (UIViewController in our sample)

data members are declared inside {}.

@property declares a property, IBOutletqualifier denotes outlets.

No need to declare overloaded methods, only the newly added

Actions have IBActionreturn type


20/38

Class implementation#import ViewController.h

@implementation ViewController

@synthesize imageView;

- (void)viewDidLoad {

[super viewDidLoad]; // call the superclass method

NSString* filename = [[NSBundle mainBundle]

pathForResource:@"helloworld" ofType:@"png"];

UIImage *image = [UIImage imageWithContentsOfFile:filename];

if( image != nil ) imageView.image = image;

}

@end

@implementation @end delimits class implementation

@synthesize automatically generates correct code for reading and writing properties

viewDidLoadis the place to put additional initialization. Xcode generates stub for it

Some methods, including viewDidLoad, require to call the original superclass method

To show the image, just assign it to the image property of UIImageViewer


21/38

Lets now add OpenCV!


22/38

Hello world1.

Download framework from OpenCV website

2.

Add framework to your project

3.

Change project language to Objective C++.

4.

Add #import to the *.pch file.


23/38

UIImage !cv::Mat

UIImage* MatToUIImage(const cv::Mat& m) {

}

void UIImageToMat(const UIImage* image, cv::Mat& m) {

}

These are not super-fast functions,

minimize such conversions

#import


24/38

Hello world

Add OpenCV code

if( image != nil ) {

cv::Mat m, gray;

UIImageToMat(image, m);

cv::cvtColor(m, gray, CV_RGBA2GRAY);

cv::GaussianBlur(gray, gray, cv::Size(5, 5), 1.2, 1.2);

cv::Canny(gray, gray, 0, 50);

m = cv::Scalar::all(255);

m.setTo(cv::Scalar(0, 128, 255, 255), gray);

imageView.contentMode = UIViewContentModeScaleAspectFit;

imageView.image = MatToUIImage(m);

}


25/38

Live demo #1

25


26/38

Agenda

1.

OpenCV for iOS2.


3.



27/38

Adding Camera input

Camera is not supported by Simulator "

Setting up camera, retrieving frames,

displaying them, handling rotations etc.

takes a lot of code. Thats why OpenCV

encapsulates this logic.

Its built using delegate pattern. The

delegate will be ourViewController

.


28/38

Camera interfaceUpdate ViewControllerinterface part

#import

#import //or ios.h

using namespace cv;

@interface ViewController : UIViewController

{

UIImage* image;

CvVideoCamera* videoCamera;

}

@property (nonatomic, retain) CvVideoCamera* videoCamera;

Protocols and delegates are not exclusive things in standard Cocoacomponents; well-designed components may introduce custom protocols

and use delegates. See cap_ios.h for details.

ARC works with pointers to your own classes as well (e.g. with

CvVideoCamera*)


29/38

Camera interfaceSetup camera in ViewController viewDidLoadmethod

- (void)viewDidLoad

{

[super viewDidLoad];

self.videoCamera = [[CvVideoCamera alloc] initWithParentView:imageView];

self.videoCamera.delegate = self;self.videoCamera.defaultAVCaptureDevicePosition =

AVCaptureDevicePositionFront;

self.videoCamera.defaultAVCaptureSessionPreset =

AVCaptureSessionPreset352x288;

self.videoCamera.defaultAVCaptureVideoOrientation =AVCaptureVideoOrientationPortrait;

self.videoCamera.defaultFPS = 30;

}

CvVideoCameraclass is super-easy to use. Leave the defaultresolution, fps etc. or customize them as needed

Using lowest possible resolution and reasonable frame rate can save a

lot of power and make apps more responsive


30/38

Camera interface

Modify button callback

- (IBAction)captureButtonPressed:(id)sender {[self.videoCamera start];

}

- (void)processImage:(Mat&)image;

{

// Do some OpenCV stuff with the image

Mat image_copy;

cvtColor(image, image_copy, CV_BGRA2BGR);

// Invert image

bitwise_not(image_copy, image_copy);

cvtColor(image_copy, image, CV_BGR2BGRA);

}


31/38

iOS performance tips

Do expensive initializations once

(e.g. in NSViewControllersviewDidLoad

method)

Use smaller resolution and FPS

Plan your work with memory, avoid redundantmemory allocations and copies (reuse buffers)

Profile your apps using Xcode profiler But avoid premature optimization


32/38

iOS performance tips (2)

Use Accelerate.framework

Use OpenGL and CoreGraphics for rendering,

CoreImage for image processing

Use Grand Dispatch Central (implements data-

level and task-level parallelism)

Use NEON SIMD intrinsics & OpenGL ES shaders


33/38

Live demo #2

33


34/38

Multi-threading

Backends:

Intel Threading Building Blocks (Intel TBB)

Microsoft Concurrency Runtime

OpenMP

Grand Central Dispatch (GCD)

C=

void parallel_for_(const Range& range, const ParallelLoopBody& body);

class LoopBody : public cv::ParallelLoopBody

{

public:

void LoopBody::operator() (const cv::Range& range) const

{}

}


35/38

Future plans

Improve documentation and tutorials

Add iOS samples

Publish few applications in App Store Support iOS 7


36/38

Resources

http://developer.apple.com(including excellent introductory guides to

Objective-C, Cocoa and iOS programming)

http://docs.opencv.org/master/doc/tutorials/ios

(iOS tutorials for the latest OpenCV version) http://code.opencv.org/svn/gsoc2012/ios/trunk

(our GSoC 2012 iOS project repository)

http://answers.opencv.org

(Q&A forum for OpenCV)


37/38

Resources

Coming soon

OpenCV for iOS minibook

Chapters: Getting Started with iOS

Displaying Image from Resources

Linking OpenCV to iOS Project Detecting Faces with Cascade Classifier Printing Postcard

Working with Images in Gallery

Applying Retro Effect Taking Photos From Camera


38/38

Thank you!

Any questions? [email protected]

[email protected]

icvs2013 - opencv for ios

Documents