[Kinect] use with Matlab

I have recently purchased a Kinect camera (although I do not even own an X-Box). It’s main purpose is to get the multi-touch working on it. Well, multi-touch: multi-sky-touch that is. For this i would be programming in C, but for now it is easier to develop in Matlab, so let’s focus on Matlab.

Kinect Hardware connections

The Kinect is provided with a normal USB connector and split to a normal power adapter. It gets quite hot after a while.

Kinect Drivers
There are three options to get the driver working on your Windows machine (and one for your OS X beast):

  • CLNUI untested
  • Open-Kinect
  • OpenNI/NITE

Open-Kinect

If we are going to use Open-Kinect with Matlab, we will be using an external DLL written for Matlab after the Kinect is working. For this we will use the Image Acquisition Toolbox (IMAQ). I had a lot of trouble with this. Though, this seemed like the good option since the Image Acquisition-toolbox is made especially for these kinds of streaming video data, for which you would otherwise need buggy loops. So after some problems with IMAQ i switched to the other driver. Also, you need Visual Studio for this approach.

Open-NI and NITE

This worked great for me, although this is not entirely open-source, but i was not going to touch the source-code much anyway. So, lets get this working: Do this:

  1. Download Kinect Drivers (LINK) and press the DOWNLOAD button. Search for the file dpinst-x86.exe and run it.
  2. Download the unstable binary which worked for me (LINK) otherwise try the unstable binary. Scroll down and press LATEST UNSTABLE and download.
  3. Download the unstable NITE binary (LINK), same fashion as (2), the install key is 0KOIk2JeIBYClPWVnMoRKn5cdY4=
  4. Download the unstable binary from (LINK) and install.
  5. Plug in the Kinect. If you go to the device manager it should say Xbox NUI Audio unknown, this is obvious, and under PrimeSensor you’ve got your Cam and Motor
  6. For this step check http://www.studentguru.gr/blogs/vangos/archive/2011/01/20/how-to-successfully-install-kinect-windows-openni-nite.aspx
  7. Run a test to make sure it works from C:Program FilesOpenNISamplesBinRelease and C:Program FilesPrime SenseNITESamplesBinRelease. If they do not work, do not proceed. Switch binaries; stable or unstable this time.
  8. Get http://sourceforge.net/projects/kinect-mex/ , press DOWNLOAD and unpack: then get to Matlab and run one of the test .m files. You can only execute the files that have _sample in them, the rest is compiled stuff you should not touch. Also, don’t touch the .XML files. Now, hacking begins!
  9. If you have any problems, drop a line in the comments.

Specifications

  • Streaming RGB ánd depth image: 20 FPS
  • Streaming depth image only: 30~35 FPS
  • Streaming without viewing: 35~50 FPS

Here a sample video in which i 3D reconstruct my desk-area.


[youtube=http://www.youtube.com/watch?v=i2i5p3U3WTg&w=560&h=345]

update The Kinect also records audio ofcourse. There are, as far as i know, no drivers yet to get audio from the Kinect. This holds at least true for Matlab, and i’m not sure if they’ve managed to import audio to windows anyway.

update2 This Kinect method does not work for Matlab 2008. Try upgrading your version.

Tim Zaman

MSc Biorobotics. Specialization in computer vision and deep learning. Works at NVIDIA.

You may also like...

83 Responses

  1. rose says:

    I want to convert kinect depth video to 3D point cloud….how could I proceed with the code….Please help me…

  2. zizou says:

    help me please

  3. Peugas says:

    hi,

    What do you mean by:
    “Streaming without viewing: 35~50 FPS” ??

    Regards

  4. Mohsin says:

    hello, great work Tim, well i am working on 3D modeling by using kinect on matlab. is there any kinect toolbox ??? where should i install it?? how this toolbox interface with kinect?

  5. nuiCapture can export Kinect point cloud data easily to Matlab.

  6. Reza says:

    Hi Tim,

    I had a openni for kinect on my computer. Last week I uninstalled it and then I installed microsoft SDK. This week I decided uninstall microssft SDK and again install openni. But the problem is: openni can not be installed successfully. How can I solve this probelm?

    Sincerely

    Reza

  7. GB says:

    Hey Tim great job here!
    THe links in point 2,3 and 4 seem out of date as they redirect to the OpenNI main page. Can you detail which code should be downloaded in those points?
    cheers
    GB

    • Tim says:

      Hello hello! Well this post is in technical terms considered ancient, so i guess i can’t (nor does openni) support this anymore 🙁

  8. hong says:

    Am I right that we have to rename the .mexw32 (or .mexw64) to .mex and remove all .m files in /Mex? I am new with Matlab and mex stuff. When I did so, and execute the sample_nIRImage.m, I received an error of “…: could not find library or dependents: …mxNiCreateContext.mex”. Anyone has a clue what I might have done wrong? OpenNI and NITE work fine (their samples run).

  9. hong says:

    hi, i am not sure if this is appropriate question. has anyone managed to compile the kinect-matlab for octave? i m having problem. i can’t find a kinect wrap in octave for openni, there seems to be one for OpenKinect. will appreciate any direct or help in this. many thanks.

  10. Mohammad says:

    Thanks for the code. I followed the instruction step by step. Everything is fine and as it supposed to be. I see the device in “Device Manager” exactly as you describe in step 6. However, in step 7 there is no such a file or directory! Furthermore, when i run the sample code in MATLAB i get the following message and do not receive any image from the device:

    Device: PrimeSense/SensorV2/5.0.4.4: The device is not connected!
    IR: PrimeSense/SensorV2/5.0.4.4: Can’t create any node of the requested type!

    What should i do?

    Regards

    • Mohammad says:

      Hi, I found the problem, I installed the Redist version of OpenNI Binary, middle ware, and hardware. That’s why i couldn’t see the sample directory.

      Now, I installed the Developer edition. I checked with the samples, i can not find the device. So, got stocked at step 7! I will try the stable version.

  11. ehsan says:

    hi
    i have 64-bit win and matlab , i can’t run your code , gave this error
    Invalid MEX-file ‘C:UsersehsanDesktopkinect-mex1.3-windows[v1.3] for OpenNI Unstable
    Build for Windows v1.0.0.25MexmxNiCreateContext.mexw64’: The specified module could not be
    found.

    please help me!!!!

    • Alex says:

      I am having the exact same problem. The example executable work but I get the error: …for OpenNI Unstable Build for Windows
      v1.0.0.25MexmxNiCreateContext.mexw64′: The specified module could not be found.

      I’m not really sure how to fix this, because the 64bit mex files and the 64bit drivers are all provided.

      • Tim says:

        I’m not running 64bit, so not sure, i have heard of more people with problems on 64 bit, sorry but since i have a 32bit machine for now, i cant run any tests. Maybe someone else?

  12. tiqbal says:

    Hi Tim,

    Thanks a lot for the tutorial. I got the RGB and the depth image using OpenNI in MATLAB. Now I am clue less, how to calibrate this two camera? Means, how to get the depth of a RGB pixel? Cause, what I am seeing is that the RGB and the depth image is not completely aligned. I searched a lot, but I did not find any solution for windows and matlab. Please help, your help will be highly appreciated.

    Thanks.

  13. ames says:

    Hey Tim,
    the links in 2 3 and 4 have been changed . could you please watch the links and lead me which files should i download?
    thnx

  14. Silva says:

    Hey Tim,

    thanks for the Matlab code, it helped me a lot !

    I recorded some “.oni”-files with the OpenNI recorder. Is it possible to read those .oni-files with your Matlab code by modifying the Config .xml-files (adding the filename somewhere in the Production Nodes)?

    Thank you for your help!

  15. Shikhar says:

    Hey TIM,
    I am using a Windows 7 64 bit machine. I have installed OpenNI, NITE, PrimeSense drivers all for 64 Bit and the samples work fine.
    However when i try to run the mex files on my Matlab R2010a, this is the error i get

    ??? Invalid MEX-file ‘F:Kinect worksopen nite
    installerskinect-mex1.11-windowskinect-mex1.11-windows[v1.11] for OpenNI Unstable Build for
    Windows v1.0.0.25mxNiCreateContext.mexw64’: The specified module could not be found.
    .
    Error in ==> sample_niImage at 75
    context = mxNiCreateContext(‘SamplesConfig.xml’);

    Can u help me out… Need to get it working urgently. The readme says that for a 64bit machine you have to compile OpenNI and drivers from their source file… what does i mean??? is that the problem??? plz reply soon.

    • Tim says:

      Hi, sorry, as i do not have a 64bit machine to test with i can’t really be much help. Make sure to download all the files you need that are indicated in 64 bit. If it says you have to compile it yourself, you have to compile it y our self. that means: get a copy of Visual Express (microsoft) and compile it..

  16. Subrat Rath says:

    Hello..please could anyone tell me if kinect works with MATLAB 2009b ?
    Actually i am trying to integrate ” MSC MD Adams ” , ” MATLAB ” and ” Kinect Sensor “. I learned that Adams 2010 is compatible with Matlab 2009b. So, i am trying to find out if kinect works with this version. Please suggest. Thank you 🙂

  17. WA says:

    Hi
    I have a question, perhaps you can help.
    I am not using Kinect but a ToF camera and have a depth image. I also have an RGB image from another camera. I have a way to calibrate and find intrinsics and extrinsics and hence find the transformation matrices between the two cameras which means I am able to relate the ToF data to RGB data. How can I visualize it? How can I create a 3D color image in MATLAB?
    Regards

  18. KBM says:

    Hi!
    I project to use the kinect with matlab to measure the shape of different object (like a cup). (I need an ‘accurate’ XYZ points cloud.)

    Which driver should I use: the OpenNI or the Official one? Is there any difference?
    You have developed a useful tools (mx files) to use with the OpenNi and Matlab, is there any other (mx files) to use with the official Kinect SDK ?
    Thank you for your help ^_^

  19. Amir says:

    I have a question i tried to run ur code,, and i find it running but i get no image from the camera there is always nothing,, and i find a warning message ( Open failed: OpenNI library can’t find any module! Check whether Config/SamplesConfig.xml is available.))

    Can u help me ? thanks alot

    • Tim says:

      First try if you kinect actually works with the sample code inside of your folder provided, before checking my code.. Does that work? Otherwise you might just be sitting too far away of the camera (my code assumes a distance of about 70cm)

      • Amir says:

        yes it works but i recieve nothing when i run the code,,,,, i find no error in ,atlab just this :

        PrimeSense/SensorV2/5.0.1.31: The device is not connected!
        and
        PrimeSense/SensorV2/5.0.1.31: Can’t create any node of the requested type!

  20. dave holden says:

    Hi Tim,

    Very cool demos. I got the kinect-matlab code working in Matlab but I don’t understand how you got such a hi-res scan of your face (on your youtube video). I get 0’s in the depth image when I get close enough to get a high-res image of an object like my face. Did you figure how to reset the minimum distance from the xml or something?

    Thanks,
    Dave

  21. Anon says:

    Hey Tim, thank you so much for the tutorial. I have a quick question, you said the kinect streams around 20-40 fps, why is it that I am getting less than 10fps when I am running the sample.m files(from the kinect mex) – I have everything disabled I just have a loop that refreshes and updates a figure. Would getting IMAQ be helpful?(I want to remain in the matlab working environment) or should I get the parallel processing toolbox (I am running on a eight-threaded computer) Thank you in advance!

  22. vincenzo says:

    Hi Tim,
    I’m working with Kinect in Matlab, using your code and I would know the error of
    Kinect while it calculates the distance of an object.
    I know that the range of the distance of kinect is about 0.5 m – 7 m,
    so I’m doing an experiment.
    I put an object away from Kinect example 3 m, in the range, than I
    run the program sample_niImage and it gives me the depth map. The distance
    calculated kinect is very close to 3 m, so I search for values in the
    depth map close to 3 m.
    To do this i use the command find in matlab.
    In the other case I put my object out of the range , at 0.3 m or 8 m
    away from kinect, and the program gives me a value of distance in the
    depth map. Because the depth map is 460*640 and the calculated value
    contains an error, how do I know what is the distance value of my
    object?

    • Tim says:

      How about holding a paper in front of the kinect, 1 meter away (you measure this). Then you take a snapshot, and measure the difference…?

      • vincenzo says:

        I’m sorry but I’m not very practical to matlab and I did not realize that there is a button with which to place the mouse in a point on the ‘depth of image you get the depth value of that point.

  23. khaled emam says:

    is it possible to get a live stream as a webcam from kinect or not yet .. as i want to use kinect with simulink for the moment

    • Tim says:

      obviously: yes, ofcourse you can get a livestream. you can have a simultanious livestream of the depth image, raw IR image, XYZ point cloud, ÁND real webcam feed at the same time in matlab, and thus in simulink.

  24. matt says:

    Tim,

    thanks so much for the guide, after a bit too much time fiddling in ubuntu this was a breeze with your guide on the windows partition!

    I may be wrong but I think your step 3 link should point to: http://www.openni.org/downloadfiles/openni-compliant-middleware-binaries/33-latest-unstable , as it is you’ve got a repeat link and no “primesense NITE” link.

    looking forward to playing with this, cheers

    • Tim says:

      Now that’s good to hear. Just another few weeks until the official driver release i think; then ill update this page as well.

  25. khaled emam says:

    How can i use the kinect matlab software posted on the mathworks website to get a depth information of any pixel using the raw depth image ?

  26. tariq says:

    how to get the point cloud data in the format. txt which consists of the value XYZ using Matlab

    • Tim Zaman says:

      The data is stored in your “workspace”. Rightclick on the matrix (XYZ data) you want to store.. then select export or something; then you can also export to .txt! But for Matlab-interchange, the .mat format is best.

  27. hermann says:

    It works, but …additionally I wanna do something more (with matlab programming of course) with the raw IR/Depth data.

    Although I use another device (named XtionPro, from PrimeSense and its partner) which seems similiar to kinect except that is has no RGB camera, you mex file and sample_niIRImage still works.

    The next step…I wanna implement PointViewer sample of NITE with matlab, which can track the hand movement. The C++ code is available, however I have no experience on making mex file from the provided C++ library.

    So…Maybe you are interested in providing more mex files as matlab interface to OpenNI/NITE ? I think it will attract more matlab users.

    • Tim Zaman says:

      Actually i recommend waiting for the official microsoft kinect drivers, i expect them to be released this month. ok? 🙂 also, this website will be entirely renovated next week, so im a bit busy! Plus, learning c++ is very useful and i recommend it to you!

  28. khaled hossam emam says:

    everything works perfect , the matlab sample files give no error but there is no output appearing ..only black screens ??

  29. DMZ says:

    Hi again Tim,

    The function mxNICreateContext can read SamplesConfig.xml (for RGB+depth image) and SamplesIR_DepthConfig.xml (IR+depth). But it can only read one XML file per Matlab session. Do you have the same problem? How can I resolve, if it’s possible?

    • DMZ says:

      This is the error throwed when I try to read the second xml file:

      Open failed: Bad Parameter sent to the device!
      Check whether SamplesIR_DepthConfig.xml is available.

  30. Xiao says:

    Excellent tutorial. Every thing works for me exceptc the matlab won’t execute the sample programes. I got this ‘Invalid MEX-file, … mxNiCreateContext.mexw64 is not a valid Win32 application’. I’m runing the matlab on a 64 bit windows 7. not sure if this might have something to do with the mex file? need your help.

    • Tim Zaman says:

      (1) do the sample (openni/primesense) executables work?? (2) does matlab say invalid win32 application?; that would be weird since it doesnt want win32 mode. (3) have you tried win32 compatible mode?.. also make sure openni and primesense are both for 64b

    • Xiao says:

      Ok, I simply switched to 32bit matlab and the samples started. However I don’t think the program is running correctly. The led light just flashes green, infrared projector is dark too. Don’t know why.

      • Tim Zaman says:

        Just hold on another month and Microsoft will release official Kinect drivers; and then, all academic hell will break loose! (in a positive way)

  31. Reza says:

    Hi Tim,

    How can i record video by matlab

  32. DMZ says:

    Very good tutorial. Kinect works perfectly, and I can execute OpenNi and NITE samples. But I have a problem with Matlab .m files. When I execute it throws this:
    “??? Attempt to execute SCRIPT mxNiCreateContext as a function.
    Error in ==> sample_niIRImage at 35
    context = mxNiCreateContext(‘SamplesIR_DepthConfig.xml’);”

    Do you know where the problem is?

    • Tim Zaman says:

      try restart and reinstall of the matlab mex codes. very weird case, and make sure the XML file that is in the same folder is correct. Otherwise i have not yet seen this error.

      • DMZ says:

        I have in Matlab workspace two XML files: SamplesConfig.xml and SamplesIRConfig.xml.. and I don’t know what name should be correct.
        I’m using Matlab 7.0.1, maybe this is the problem. Now I’m upgrading.

      • DMZ says:

        Sorry, Kinect Mex 1.11 includes 4 XML files.

      • Tim Zaman says:

        1.1 is for the openni stable build, and 1.11 is for the openni unstable, make sure that’s correct. it has nothing to do with the XML files, thats for sure. there was some trouble with 2008 i recon, try to upgrade; if it doesnt work then i’ll dive into it.

      • DMZ says:

        Thank you for your interest Tim. I have installed version 2010b and it works perfect!! 🙂

  33. Anonymous says:

    Hi Tim,
    How can i understand matlab is Ok with this program?

    • Tim Zaman says:

      Matlab works great with the kinect, it is so fast it can stream 30fps real-time. Could you maybe elaborate your question?

    • Anonymous says:

      Hello,

      7- Run a test to make sure it works.

      please give me an example

      • Tim Zaman says:

        You have installed OpenNI and Prime Sense. They are installed in Program Files. If you indeed installed them, they install sample .exe files. Try one of the .exe files in ..Program FilesOpenNISamplesBinRelease, for instance the file “SimpleViewer.net.exe”.

      • Reza says:

        After executing SimpleViewer.net.exe , I have a small black window after 2 seconds a white blank window is apperad. Is it correct?

      • Tim Zaman says:

        no this is not correct. first the black small screen appears, then the info from the kinect. you can see that the kinect works as well when the lens on the left emits some red light (infrared laser)

  34. Reza says:

    Hi,

    my computer is 64 bit i have problem with dpinst-x86

    • Tim Zaman says:

      dpinst-x86 is for 32 bit! you have to take the 64bit version =)

      • Reza says:

        Thank you
        I try C:Program FilesOpenNISamplesBinRelease and C:Program FilesPrime SenseNITESamplesBinRelease for 64 bit version either
        stable version and unstable version but it dose not working ?

      • Tim Zaman says:

        Try dpinst-amd64.exe (if you have a 64-bit processor). Maybe you could try forcing to run it in 32bit mode or something, isn’t that possible? Otherwise i wouldn’t know, and if you’re really stuck you could try the Matlab IMAQ adapter, which I personally dislike.

  35. JK says:

    Hey this might be an easy question for you, but I am not very experienced with Matlab and need to use it to process the data coming in from the Kinect to do object detection. I was wondering how you directly input the data to perform an operation on it. I am building a robot that will autonomously navigate a rocky area and matlab seems to be the most capable program. After performing the object detection, i’m going to convert the operation into C and use that in the program I am building. Is there any way you could please help me out??

  1. June 7, 2013

    […] have been wrappers for interfacing with the Kinect device before, but the latest update to MATLAB makes it native to the Image […]