第38回 名古屋cv・prml勉強会 「kinect v2本の紹介とpclの概要」
TRANSCRIPT
[PR] Kinect v2本の紹介とPCLの概要
Tsukasa Sugiura
@UnaNancyOwen
Self-Introduction
Tsukasa SugiuraMicrosoft MVP for Kinect for Windows
@UnaNancyOwen
http://UnaNancyOwen.com
Agenda
Kinect v2の概要
Kinect v2本の紹介
Kinect v2本のプレゼント
Kinect v2とPCL
Kinect for Windows v1 Sensor
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
RGB CAMERA
Kinect for Windows v2 Sensor
MULTI-ARRAY MIC
3D DEPTH SENSOR
( IR Camera + IR Emitters )
RGB CAMERA
Kinect for Windows v2 Sensor
Image by iFixit
IR EMITTERS
IR CAMERA
Specifications
Kinect for Windows v1 Kinect for Windows v2
Color 640×480 @ 30fps 1920×1080 @ 30fps
Depth 320×240 @ 30fps 512×424 @ 30fps
Sensor Structured Light(PrimeSense Light Coding)
Time of Flight(ToF)
Range of View 0.8~4.0 m 0.5~8.0 m
Range of Detection 0.8~4.0 m 0.5~4.5 m
Angle of ViewHorizontal / Vertical
57 / 43 degree 70 / 60 degree
Microphone Array ◯ ◯
Specifications
Kinect for Windows v1 Kinect for Windows v2
BodyIndex 6 people 6 people
Body 2 people 6 people
Joint 20 joint/people 25 joint/people
Hand State Open / Closed Open / Closed / Lasso
Gesture ☓ ◯
Face ◯
Speech / Beamforming ◯ ◯
System / Software Requirements
OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU * DirectX 11 supported
USB * USB 3.0 (Intel or Renesas Host Controller)
Compiler * Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
Basic Features
Color
1920×1080@30fps / 15fps (Lighting Condition)
RGBA, YUV, BGRA, Bayer, YUY2
Basic Features
Depth
512×424@30fps
500~8000[mm]
ToF (Time of Flight)
Basic Features
Infrared / LongExposureInfrared
512×424@30fps
16bit (higher 8 bits)
Basic Features
BodyIndex
512×424@30fps
500~4500[mm]
6 people
Body Area : 0~5, Other Area : 255 (5 < Index)
255
0 1
Basic Features
Body
500~4500[mm]
6 people
25 joint / people (Add Tip, Thumb, Neck)
Orientation (Quaternion)
Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)
Basic Features
Audio
Beamforming (+/-50 degree)
Speaker Estimation
Speech Recognition
Application Features
Gesture
Gesture Recognition using Machine Learning
Discrete (detected true/false), Continuous (progress 0.0f~1.0f)
Learning Classifier Tool “Visual Gesture Builder”
Video by http://youtu.be/-XYoblrnDpg
Application Features
Face
Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)
Activity, Appearance, Expression Activity … Eye Closed, Mouth Open / Moved, Looking Away
Appearance … Wearing Glasses
Expression … Happy
Application Features
HDFace
For Creating 3D Face Model
Points (1347), Triangles (2340), Hair Color, Skin Color
Fitting Face Model
Application Features
Other
Kinect Fusion (3D Shape Reconstruction)
Controls (Assist in implementation of NUI)
Kinect v2 Book
Kinect for Windows SDK プログラミングKinect for Windows v2センサー対応版
中村薫, 杉浦司, 高田智広, 上田智章
秀和システム
480ページ
3,400円+税
Publisher : http://www.shuwasystem.co.jp/products/7980html/4395.html
Sample : https://github.com/K4W2-Book/K4W2-Book
Reference : https://github.com/K4W2-Book/Docs
Contents
Basic Streams Color, Depth, Infrared, BodyIndex, Body, Audio
Application Futures Speech, Fusion, Gesture, Control, Face, HDFace
Contents
NUI Tools Kinect Studio, Visual Gesture Builder, Configuration Verifier
3rdParty Library, Tools, Math., Sensing Unity, Point Cloud Library, Vector
Review Copy
Point Cloud Library
About PCL(Point Cloud Library)
3次元点群のためのオープンソースのライブラリ
様々な点群処理が含まれている
http://pointclouds.org/
Point Cloud Library
PCL Features
Point Cloud Library
PCL All-in-one Installer
ビルド済みのPCLと依存ライブラリを一括でインストールする
ただし、公式サイトでは旧い開発環境向けのパッケージしか配布していない
(PCL 1.6.0 All-in-one Installer MSVC 2008/2010)
Download : PCL 1.7.2 All-in-one Installer MSVC 2012/2013/2015
Self-Build
PCLと依存ライブラリをビルドする
Reference : Building PCL with Visual Studio
PCL All-in-one Installer
Point Cloud Library
PCL (1.7.2)
Dependencies Library
Boost (1.57.0) … C++準標準ライブラリ
Eigen (3.2.4) … 行列ライブラリ
FLANN (1.8.4) … 最近傍探索ライブラリ
Qhull (2012.1) … 計算幾何ライブラリ
VTK (6.2.0) … 可視化ライブラリ
(OpenNI / OpenNI2) … RGB-Dセンサーライブラリ
PCL Grabber
OpenNI/OpenNI2
オープンソースのRGB-Dセンサーライブラリ
ただし、OpenNIはVisual Studio 2010までしか利用できない
PCL 1.7.2からOpenNI2がサポートされた
openni_grabber / openni2_grabber
Kinect for Windows SDK v1/v2
Microsoft公式のKinectセンサーのソフトウェア開発キット
最新のVisual Studioで利用できる
kinect_grabber / kinect2_grabber
PCL Project
Create Project
Visaul C++のプロジェクトを作成、プロパティシートを読み込む
PCL.props
CMakeでVisual C++のプロジェクトを自動生成する
CMakeLists.txt
PCL Project
CMake
CMakeLists.txt
cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
project(solution)
find_package(PCL 1.7 REQUIRED)
include_directories(${PCL_INCLUDE_DIRS})
include_directories($ENV{KINECTSDK20_DIR}/inc)
link_directories(${PCL_LIBRARY_DIRS})
link_directories($ENV{KINECTSDK20_DIR}/Lib/$(PlatformTarget))
add_definitions(${PCL_DEFINITIONS})
set(HEADER kinect2_grabber.h)
set(SOURCE main.cpp)
add_executable(project ${HEADER} ${SOURCE})
target_link_libraries(project ${PCL_LIBRARIES})
target_link_libraries(project Kinect20.lib)
Basic Usage Example
Source Code
main.cpp
#include "kinect2_grabber.h"#include <pcl/visualization/pcl_visualizer.h>
typedef pcl::PointXYZRGB PointType;
int main( int argc, char* argv[] ){
// PCL Visualizerboost::shared_ptr<pcl::visualization::PCLVisualizer> viewer(
new pcl::visualization::PCLVisualizer( "Point Cloud Viewer" ) );
// Point Cloudpcl::PointCloud<PointType>::ConstPtr cloud;
// Retrieved Point Cloud Functionboost::mutex mutex;boost::function<void( const pcl::PointCloud<PointType>::ConstPtr& )> function =
[&cloud, &mutex]( const pcl::PointCloud<PointType>::ConstPtr& ptr ){boost::mutex::scoped_lock lock( mutex );cloud = ptr;
};
Basic Usage Example
Source Code// Kinect2Grabberpcl::Grabber* grabber = new pcl::Kinect2Grabber();
// Register Callback Functionboost::signals2::connection connection = grabber->registerCallback( function );
// Start Grabbergrabber->start();
while( !viewer->wasStopped() ){// Update Viewerviewer->spinOnce();
boost::mutex::scoped_try_lock lock( mutex );if( cloud && lock.owns_lock() ){
if( cloud->size() != 0 ){/* Processing Point Cloud */
// Update Point Cloudif( !viewer->updatePointCloud( cloud, "cloud" ) ){
viewer->addPointCloud( cloud, "cloud" );viewer->resetCameraViewpoint( "cloud" );
}}
}}
Basic Usage Example
Source Code// Stop Grabbergrabber->stop();
return 0;}
Kinect2Grabber Supported Point Types
pcl::PointXYZRGB … 3次元位置(XYZ)と色(RGB)
pcl::PointXYZ … 3次元位置(XYZ)
Down Sampling
VoxcelGrid#include <pcl/filters/voxel_grid.h>
// Create Objectpcl::VoxelGrid<PointType> grid;
// Settinggrid.setLeafSize( 0.01f, 0.01f, 0.01f );
// Inputgrid.setInputCloud( *input );
// Processingpcl::PointCloud<PointType>::Ptr output( new pcl::PointCloud<PointType> );grid.filter( *output );
Point CloudをVoxcel Gridフィルターでダウンサンプルする
Voxcel Gridの間隔を0.01[m]に設定
http://pointclouds.org/documentation/tutorials/voxel_grid.php
Down Sampling
Result
Segmentation
SACSegmentation#include <pcl/segmentation/sac_segmentation.h>
// Create Objectpcl::SACSegmentation<PointType> seg;
// Setting seg.setOptimizeCoefficients( true ); seg.setModelType( pcl::SACMODEL_PLANE );seg.setMethodType( pcl::SAC_RANSAC );seg.setMaxIterations( 1000 ); seg.setDistanceThreshold( 0.01 );
// Inputseg.setInputCloud( *input );
// Processingpcl::PointIndices::Ptr inliers( new pcl::PointIndices );pcl::ModelCoefficients::Ptr coefficients( new pcl::ModelCoefficients );seg.segment( *inliers, *coefficients );
Point Cloudから平面(SACMODEL_PLANE)を検出する
最大反復回数を1000回、距離閾値を0.01[m]に設定
http://pointclouds.org/documentation/tutorials/planar_segmentation.php
Segmentation
Draw Segmentation Result// Drawing Detection Areapcl::PointCloud<PointType>::Ptr output( new pcl::PointCloud<PointType> );output = input;for( size_t i = 0; i < inliers->indices.size(); i++ ){
int index = inliers->indices[i];output->points[index].r = 255;output->points[index].g = 0;output->points[index].b = 0;
}
Segmentation
Result
Extracting
ExtractIndices#include <pcl/filters/extract_indices.h>
// Create Objectpcl::ExtractIndices<PointType> extract;
// Settingextract.setIndices( inliers );
// Inputextract.setInputCloud( *input );
// Processingpcl::PointCloud<PointType>::Ptr output_positive( new pcl::PointCloud<PointType> );extract.setNegative( false );extract.filter( *output_positive );
pcl::PointCloud<PointType>::Ptr output_negative( new pcl::PointCloud<PointType> );Extract.setNegative( true );extract.filter( *output_negative );
Point Cloudから検出した平面を抽出する
Positiveは検出領域、Negativeは非検出領域
http://pointclouds.org/documentation/tutorials/extract_indices.php
Extracting
Result
Extracting
Result
Tips
Convert ConstPtr tp Ptr// Convert pcl::PointCloud<T>::ConstPtr pcl::PointCloud<T>::Ptrpcl::PointCloud<PointType>::ConstPtr ConstPtr( new pcl::PointCloud<PointType> );pcl::PointCloud<PointType>::Ptr Ptr;Ptr.reset( new pcl::PointCloud<PointType>( *ConstPtr ) );
Save/Load Point Cloud#include <pcl/io/pcd_io.h>
// Save Point Cloudpcl::PointCloud<PointType>::Ptr cloud( new pcl::PointCloud<PointType> );pcl::io::savePCDFileBinary( "pointcloud.pcd", *cloud );
// Load Point Cloudpcl::PointCloud<PointType>::Ptr cloud( new pcl::PointCloud<PointType> );pcl::io::loadPCDFile( "pointcloud.pcd", *cloud );
Reference
PCL Official
Documentation - Tutorials
Documentation - API Reference
Users Forum
Japanese
DERiVE - PCLを触ってみよう！
第13回名古屋CV・PRML勉強会 - PCL (Point Cloud Library)
PCL Development
GitHub - PointCloudLibrary/pcl (issues/pull requests)
PCL Next All-in-one Installer
Point Cloud Library
PCL (1.8.0)
Dependencies Library
Boost (1.59.0) … C++準標準ライブラリ
Eigen (3.2.6) … 行列ライブラリ
FLANN (1.8.4) … 最近傍探索ライブラリ
Qhull (2012.1) … 計算幾何ライブラリ
VTK (6.3.0) … 可視化ライブラリ
OpenNI2 … RGB-Dセンサーライブラリ
RealSense … Intel RealSense SDK