the not so short introduction to kinect

The Not So ShortIntroduction to Kinect一份不太简短的 Kinect 介绍

Hang xiesonicisdreaming@gmail.comhttp://Xhbang.com

About Me

Chongqing University bachelor （单身汉）A boy couldn't stop dreamingBike travelling

About Him

•Dr.•Sensor related•Poet （魔幻现实主义）

About the Lab

•50% Idle•25% Computer•20% Programmer•5% Student/Paper

About You?

Programming dummy?Linux || Windows?Understand 3D?Work with the Kinect?

The Contentswhat u will learn today

体感操作， Kinect ， OPENNI 介绍基本程序设计Working with Depth ImageWorking with Point Clouds （ * ）Working with Skeleton Data & TrackingKinect Hacks

昨夜西风凋碧树 , 独上高楼 , 望尽天涯路

Buy it!

Inside Kinect

TeardownPrincipleHardwarePerformace

Hardware & teardown

http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown

Kinect Hardware

•RGB camera•IR depth sensor•IR projector•Audio device

Take a closer look

Noise at edges(shoulders)Depth shadowsMisalignment between the color and depth imagesFrame rate(30 vs 60)

Behind Kinect

TOF

•Time of Flight•以色列公司 3DV （ 09 年微软收购）•光飞行时间，飞秒 (1e−15) 级快门•砷化镓，使其时钟频率提高到上百•高成本，难以适应消费电子品•Natal 并不是基于 ToF 的原理

Light Coding （光编码）

•连续的照明（而非脉冲）•普通的 CMOS 感光芯片（低成本）•结构光技术（三维纵深的“体编码” ）•激光散斑（ laser speckle ）

http://www.javaforge.com/wiki/103639 http://www.wipo.int/pctdb/en/wo.jsp?WO=2007043036 http://www.freepatentsonline.com/7433024.pdf

Exemplar

•机器学习系统•TB 计的数据训练集

1.寻找移动部位（背景分离，分割遮罩）2.辨认身体部位可能性3.模型匹配：生成骨架系统

Audio

•Something we forget•Supported by OpenNI, not by SensorKinect

More About Audio （题外话）•“Sound Wave”•微软研究院和华盛顿大学•利用多普勒效应，使用笔记本内置的扬声器和麦克风开发出类似 Kinect 的姿势感知系统•低成本•识别精度能达到 90% 以上•用手势玩俄罗斯方块游戏（视频）

Concepts

http://en.wikipedia.org/wiki/Natural_user_interface

Next generation interface

Wii Remote

ASUS Xtion

Kinect

Project Glass

Development

http://www.openni.org/

http://code.google.com/p/simple-openni/

http://openkinect.org/wiki/Main_Page

https://github.com/OpenKinect/libfreenect

Architecture

Quick view(*)

OpenNI

•Short for “Open Natural Interface”•Multi-langue•Cross-platform•Standard & Infrastructure

向后兼容

使用 openNI 开发的程序在以后的版本中都可以继续使用 , 因为开发人员可以毫无顾虑的更新 openNI到最新的版本。

OpenNI Framework(1)

抽象分层视图

•Top: 代表软件，在 OpenNI 顶部，实现了自然交互的应用程序。•Middle: 代表 OpenNI ，提供通信接口，同时连接传感器和中间件组件，后者分析来自传感器的数据。•Bottom: 硬件设备，捕捉现场的视频和音频内容。

OpenNI Framework(2)

通用的框架工具集

•一个 USB 抽象访问层•一些基本数据类型的实现（包括列表，哈希，等等）•系统日志和转储•内存和性能分析•活动（授权要注册的回调函数到一个特定的事件）•任务调度•不是标准 OpenNI 的一部分（ * ）

Callback func

•函数指针•注册•事件响应

OpenNIの构成

Generator Metadata Capability

Generator

注：绿色已经有，红色计划中，后略

生产节点

•单位集•生产性角色•使用较低水平的生产节点•“有意义的三位数据”

Sensor Related

•Device•Depth Generator•Image Generator•IR Generator•Audio Generator

Middleware Related

•Gesture Generator•Scene Analyzer•Hands Generator•User Generator•//provided by NITE

Record & Play

•Recoder•Player•Codec

Matadata

Capability

NITEの构成

SessionDetector

Middleware

•Full body analysis•Hand point analysis•Gesture detection•Scene Analyzer

Session

Detector

衣带渐宽终不悔，为伊消得人憔悴。

Hello world!

•Preparation•The program

Software Preparation

•OpenNI+NITE•Avin’s SensorKinect•Processing(*)•SimpleOpenNI(*)

https://github.com/avin2/SensorKinect

Here’s the code,too young too simple

import SimpleOpenNI.*;SimpleOpenNI kinect;

void setup(){ size(640*2, 480); kinect = new SimpleOpenNI(this);

kinect.enableDepth(); kinect.enableRGB();}

void draw(){ kinect.update();

image(kinect.depthImage(), 0, 0); image(kinect.rgbImage(), 640, 0);}

Understand the code

import SimpleOpenNI.*;SimpleOpenNI kinect;

•Import the library•Declare the SimpleOpenNI object

void setup(){ size(640*2, 480); kinect = new SimpleOpenNI(this);

kinect.enableDepth(); kinect.enableRGB();}

•Declare the size of app•Instantiate the object•Call two method

void draw(){ kinect.update();

image(kinect.depthImage(), 0, 0); image(kinect.rgbImage(), 640, 0);}

•Get fresh data from kinect•Place the image

Working with depth map

•R G B•纯黑代表无穷远，纯白代表无穷近•Demo:ex03_basic_depth_plus_rgb

More about depth map

•Projective•Realworld•Z值相同，代表深度，单位 mm•ConvertProjectiveToRealWorld()

The Kinect coordinate system

Demo

chPC_ex12_obj_in_point_cloud

范围以及精确性

•最近距离： about 0.5(?)•最远距离：  10m•GetDeviceMaxDepth()•官方建议： 1.2 – 3.6米•近远精确性比较

Working with Point Clouds(*)

Welcome to the Third Dimension3d reconstructionE.g:KinectFusion PCL

http://pointclouds.org/

DEMO

chPC_ex05_rotating_point_cloud

Working with Skeleton Data

•User Detection•Joints & Skeleton•Distance in 3D•Scene Map, Com•Tracking

User Detection

•Calibration or not•Stages in Process

Joints & Skeleton

•Joints•Distance between two joints in 3D

WindowsSDK: 20 joints(all)

OpenNI: 24 joints

DEMO

chSK_ex02_skel_anatomy

Scene Map

•Background remove•User pixels•Center of Mass

Demo

•chSK_scene_map•chSK_scene_image•chSK_center_of_mass

How

•kinect.enableScene();•sceneMap = kinect.sceneMap();•if(sceneMap[i] == n)

Tracking

Demo:PointViewer

众里寻他千百度，蓦然回首，那人却在，灯火阑珊处。

Kinect Hacks

Apps out of boxCreate your own cool stuffsLiving coding with VS & Cpp

Media Art

Demo: chDE_face_changing

How

•Collision detection•kinect.getJointPositionSkeleton()•kinect.convertRealWorldToProjective()•Picture scala

Computer Interface

Demo:Minority Report(*)Demo:2D TrackPadDemo:Hotpoint 3D

FAAST

•Key event mapping•Game playing

http://projects.ict.usc.edu/mxr/faast/

Sex Game Controller

•DEMO: 宅男妄想地带•Microsoft: NO!!

Sorry, Bill

Control emacs(*)

•Map(Joint position + gesture, key event)

Kinect Visual Camera(*)

http://www.kinectdevs.com/forums/kinect-virtual-camera-kinect-webcam-f17

The following demo need a better computer for higher GPU

support

kinect-ultra(*)

一秒钟凹凸曼合体！完成你童年时候的梦想Video

http://code.google.com/p/kinect-ultra/

kinect-kamehameha(*)

MikuMikuDance(*)

初音妹子Video

http://www.kinect-hacks.com/openni

What do you want to make?

reference•http://viml.nchc.org.tw•Visualization and Interactive Media Laboratory of NCHC•国家科學視算與互動媒體實驗室

•http://www.javaforge.com/wiki/103532•Kinect for PC Wiki•中文

Thanks!You are the controller.Q & AHang xiesonicisdreaming@gmail.comhttp://www.xhbang.com

This is basic template

Do not delete this page