[3D] Realtime Face Reconstruction with Matlab and Kinect

This is what my program does:

  1. Gets 3D + live image
  2. Converts 3D coordinates to real-world coordinates
  3. Makes a trigonial mesh
  4. Fills in the entire trigonial mesh with the correct colors (from live image)
  5. Starts from 1
Specs:
  • ~2 seconds per frame (0.5 FPS)
  • Accurate to ~5mm
  • Works from 50cm to at least 700cm
  • 0.3 MP Resolution
  • If not set to realtime (=postprocessing) framerate would be 30FPS
See below this youtube of my face, which consists of around 40.000 datapoints: http://www.youtube.com/watch?v=Rlp9rRKj-ZU And then here follows the code. Looks basic? Yeah it’s fairly basic.
  1. clc
  2. clear all
  3. close all
  4. % Create context and read xml file at first
  5. context = mxNiCreateContext('SamplesConfig.xml');
  6. %% Initialise FIGURE
  7. width = 640;
  8. height = 480;
  9. % depth image
  10. % figure(1), h1 = imagesc(zeros(height,width,'uint16'));
  11. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  12. % rgb image
  13. % figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));
  14. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  15. % rgb+depth image
  16. %figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;
  17. %           set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  18. %           h4 = imagesc(zeros(height,width,'uint16')); hold off;
  19. % converted XYZ plot
  20. %figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));
  21. %           axis([-800 800 -600 600 -10000 -1 ]); view(180,90);
  22. %           xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');
  23. %           title('XYZ ?iThe world coordinate system?j');
  24. figure(5);  %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')
  25.             %axis equal
  26.             %shading interp %colours are interpolated inside triangles
  27. %% acquire images
  28. for k=1:2
  29. option.adjust_view_point = true;
  30. % Acquire rgb and depth image
  31. tic
  32. [rgb, depth] = mxNiImage(context, option);
  33. real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system
  34. [M]=-mn32m3(real_XYZ); %dit duurt erg lang
  35. [Mrgb]=mnrgb2mrgb(rgb); %dit duurt erg lang
  36. itmp=sum(M~=0); %locate existing info
  37. stmp=itmp~=0; %positive if there is info
  38. M=M(:,stmp);
  39. Mrgb=Mrgb(:,stmp);
  40. itmp=M(3,:)>-800;
  41. stmp=itmp==1;
  42. M=M(:,stmp);
  43. Mrgb=Mrgb(:,stmp);
  44. p=M';
  45. C=double(Mrgb')/255;
  46. t=delaunay(p(:,1),p(:,2));
  47. colormap(C);
  48. trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')
  49. axis equal;view(200,66)
  50. shading interp
  51. %real_X = real_XYZ(:,:,1);
  52. %real_Y = real_XYZ(:,:,2);
  53. %real_Z = real_XYZ(:,:,3);
  54. % update FIGURE
  55. % set(h1,'CData',depth);
  56. % set(h2,'CData',rgb);
  57. %set(h3,'CData',rgb);
  58. %set(h4,'CData',depth);
  59. %set(h4,'AlphaData',double(depth/50));
  60. %
  61. %tmp = real_XYZ(:);
  62. %tmp(find(real_XYZ==0)) = NaN;
  63. %tmp = reshape(tmp,[height width 3]);
  64. %set(h5,'XData',-tmp(:,:,1));
  65. %set(h5,'YData',-tmp(:,:,2));
  66. %set(h5,'ZData',-tmp(:,:,3));
  67. drawnow;
  68. disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);
  69. end
  70. % Delete the context object
  71. mxNiDeleteContext(context);
MN322M3.m function
  1. function [M]=mn32m3v2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=[X Y Z];
MNRGB2MRGB.m function
  1. function [M]=mnrgb2mrgbv2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=uint8([X Y Z]);
]]>

You may also like...

28 Responses

  1. tariq says:

    I have tried your program [3D] Realtime Face Reconstruction with Matlab and Kinect but I got a problem.
    Open failed: Memory allocation failed!
    Check whether SamplesConfig.xml is available.
    ??? Error using ==> cgprechecks at 47
    Not enough unique points specified.
    Error in ==> delaunay at 59
    cgprechecks(X, nargin-1, cg_opt);
    Error in ==> rekonstruksi at 58
    t=delaunay(p(:,1),p(:,2));

  2. Tim says:

    Okay that’s not so bad. Line 51 makes sure you’re within 80cm of the camera. So just cut out that part, or sit closer to the Kinect: otherwise it doesnt have any info to process: that’s why you’re getting the error, it doesnt have any data -i guess-. Check that. Also, try playing with the functions first and try to understand what the code does then.
    T

  3. Arthur da Igreja says:

    Congratulations on your project! I´m developing applications with Kinect also, have you tried the ”warp” command to apply the image in the surface provided by the variable ”depth”? It worked fine for me.

  4. love to kinect says:

    dear Tim, thank so much from your code, but it’s not worked for me.
    i run the 2011 version of matlab, and get the following error, can help me? regards.
    —-
    One or more of the following nodes could not be enumerated:
    ??? Error using ==> delaunay
    The input points must be a double array.
    Error in ==> a at 58
    t=delaunay(p(:,1),p(:,2));

    • Tim says:

      Not sure.. i used 2009, maybe even 2008. Just check out what goes wrong step by step by asking “class(p)” for instance, to see what classes everything are. Know how the code works if you use it; otherwise you gain nothing from it =)

  5. love to kinect says:

    dear tim, can help me in more detail?
    if it possible to make a tutorial movie for this code ? or should more accurate text ?
    i couldn’t run this code with matlab 2009b, 2010a, 2010b, 2011a !!
    we get this error with 2009a version,
    ??? Error using ==> cgprechecks at 47
    Not enough unique points specified.
    Error in ==> delaunay at 59
    cgprechecks(X, nargin-1, cg_opt);
    i’m so sad because i couldn’t run it ! please help me, if it possible.
    thanks a million.

  6. Gabriel says:

    Hi Tim,
    I was wondering if you could help me. I am trying to do the same thing you do, but from scratch, that is, I have depth image, rgb image, the intrinsic parameters and that’s it . The first thing I would like to do, is to project to real world the pixels in depth image. I would later take care of the color..
    I do it by using this code,
    %I1(i,j)=depth(i,j)
    [m n]=size(I1);
    k=1;
    c_x =3.3930780975300314e+02;
    c_y = 2.4273913761751615e+02;
    f_x= 5.9421434211923247e+02;
    f_y= 5.9104053696870778e+02;
    for i=1:m
    for j=1:n
    mundo_real(k,:)=[ (i-c_x)*I1(i,j)/f_x (j-c_y)*I1(i,j)/f_y I1(i,j) ];
    % mundo_real(k,:)=[i I1(i,j) j];
    k=k+1;
    end
    end
    However, I am not getting what I want.
    Do you think you can help me?
    Thanks in advance,
    Gabriel

    • Tim says:

      I don’t see your point.. i mean, i do the same thing with this code with just the depth image and rgb image. with a little tweaking you can almost ignore your parameters. Also, the code you point to truly sucks as it has a for-loop in a for-loop and for-loops should be avoided (super slow). As you can see i do the same thing without for-loops.
      The thing you have to do is easy: take your depth image (XYZ pointcloud) make it into a triangular polygonic form (use delaunay function as in my code above) and just apply the rgb image onto it, as i do in the code above.
      T

  7. John says:

    Hi Tim,
    You stated that the accuracy is ~ 5 mm ? The depth image resolution of Kinect is at least 1 cm ( http://openkinect.org/wiki/Imaging_Information ). For example, ff I choose my nose and exam the depth value around that, some pixels have the same depth values. Therefore, I suspect the accuracy of Kinect. Any idea?

    • Tim says:

      Oh come on, you’re talking nonsense. Can’t you see the above video i made from scanning my own face? In this post? You can see a lot of detail there, the quality is around 2mm if you ask me.

  8. Seongsu mun says:

    dear Tim, thank so much your code.
    I don’t know about mn32m3. Is it a function provided by matlab? Or did you make it?
    please answer me.
    Thank you~

    • Tim says:

      Hi. mn32m3 is a code i made myself, it is provided in the post. The code’s name is “mn32m3” which means (MxNx3 to Mx3) so it does a fairly simple but important matrix transformation.

  9. Phuoc Nguyen says:

    Hi Tim,
    How many poses did you use for reconstruction?
    Regards,
    Phuoc Nguyen

    • Tim says:

      for the 3d reconstruction of my face you see in this post i used only one.

      • Phuoc Nguyen says:

        Hi Tim,
        Thank you for reply. If you used one pose there are some parts of face would be invisible. How can you deal with that? I didn’t see you use any interpolation method in your code. Correct me if I am wrong.
        Regards,
        Phuoc Nguyen

        • Phil says:

          Delaunayrtriangles take care of that. and, the back of the face is invisible, right.. its just one frame. use your kinect and see for yourself.

  10. Bryan says:

    Dear Tim,
    I am really enjoying your MATLAB programs.
    However, since I am a real newbie, I could not get through the demo program of yours, “Real Time Face Reconstruction using Matlab”.
    Is therer any other Matlab functions to install? I’ve got the following errors
    during the Matlab execution. I use Win7(32bit). Other Demo programs acquiring IR images, Depth Images, RealworldCoordinates, all work flawlessly.
    Thank you so much.
    ??? Error using ==> delaunay
    The input points must be a double array.
    Error in ==> sample_FaceConstruction at 58
    t=delaunay(p(:,1),p(:,2));
    Bryan

  11. Rui says:

    Hi Tim,
    I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?
    Bests,
    Rui

  12. hano says:

    Dear Tim,
    I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?
    Bests,
    hano

  13. khadhraoui says:

    code matlab

  14. Jesus says:

    I’m sorry. I am running this sample but Matlab show me one error:
    ??? Undefined function or method ‘mxNiImage’ for input arguments of type ‘struct’.
    Error in ==> main at 41
    [rgb, depth] = mxNiImage(context, option);
    How can implement that function or how can I solve that problem.
    Thank you very much !

  15. Ghost says:

    Can you update this project? please Will be very nice.

  16. Adam says:

    Hi Tim,
    I admire your work and wanna try that on my computer.
    However, I have trouble to find the “download folder” you mentioned in the comments.
    I was wondering if you can help me out.