Tim Zaman

Science Adventures

[3D] Realtime Face Reconstruction with Matlab and Kinect

by · Published · Updated

This code written in Matlab demonstrates how to reconstruct a face from information captured by the Kinect. What it does, is that it makes a pointgrid, and fills that grid in with the appropriate color.

I managed to map the texture on the pointgrid aquired from the Kinect 3D image.

This is what my program does:

  1. Gets 3D + live image
  2. Converts 3D coordinates to real-world coordinates
  3. Makes a trigonial mesh
  4. Fills in the entire trigonial mesh with the correct colors (from live image)
  5. Starts from 1

Specs:

  • ~2 seconds per frame (0.5 FPS)
  • Accurate to ~5mm
  • Works from 50cm to at least 700cm
  • 0.3 MP Resolution
  • If not set to realtime (=postprocessing) framerate would be 30FPS

See below this youtube of my face, which consists of around 40.000 datapoints:

And then here follows the code. Looks basic? Yeah it’s fairly basic.

  1.  
  2. clc
  3. clear all
  4. close all
  5.  
  6. % Create context and read xml file at first
  7. context = mxNiCreateContext('SamplesConfig.xml');
  8.  
  9. %% Initialise FIGURE
  10. width = 640;
  11. height = 480;
  12. % depth image
  13. % figure(1), h1 = imagesc(zeros(height,width,'uint16'));
  14. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  15. % rgb image
  16. % figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));
  17. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  18. % rgb+depth image
  19. %figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;
  20. %           set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  21. %           h4 = imagesc(zeros(height,width,'uint16')); hold off;
  22. % converted XYZ plot
  23. %figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));
  24. %           axis([-800 800 -600 600 -10000 -1 ]); view(180,90);
  25. %           xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');
  26. %           title('XYZ ?iThe world coordinate system?j');
  27. figure(5);  %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')
  28.             %axis equal
  29.             %shading interp %colours are interpolated inside triangles
  30.  
  31.  
  32.  
  33.  
  34. %% acquire images
  35. for k=1:2
  36. option.adjust_view_point = true;
  37. % Acquire rgb and depth image
  38. tic
  39. [rgb, depth] = mxNiImage(context, option);
  40.  
  41. real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system
  42.  
  43. [M]=-mn32m3(real_XYZ); %dit duurt erg lang
  44. [Mrgb]=mnrgb2mrgb(rgb); %dit duurt erg lang
  45.  
  46. itmp=sum(M~=0); %locate existing info
  47. stmp=itmp~=0; %positive if there is info
  48. M=M(:,stmp);
  49. Mrgb=Mrgb(:,stmp);
  50.  
  51. itmp=M(3,:)>-800;
  52. stmp=itmp==1;
  53. M=M(:,stmp);
  54. Mrgb=Mrgb(:,stmp);
  55.  
  56.  
  57. p=M';
  58. C=double(Mrgb')/255;
  59. t=delaunay(p(:,1),p(:,2));
  60. colormap(C);
  61.  
  62. trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')
  63. axis equal;view(200,66)
  64. shading interp
  65.  
  66.  
  67. %real_X = real_XYZ(:,:,1);
  68. %real_Y = real_XYZ(:,:,2);
  69. %real_Z = real_XYZ(:,:,3);
  70.  
  71.  
  72. % update FIGURE
  73. % set(h1,'CData',depth);
  74. % set(h2,'CData',rgb);
  75. %set(h3,'CData',rgb);
  76. %set(h4,'CData',depth);
  77. %set(h4,'AlphaData',double(depth/50));
  78. %
  79. %tmp = real_XYZ(:);
  80. %tmp(find(real_XYZ==0)) = NaN;
  81. %tmp = reshape(tmp,[height width 3]);
  82.  
  83. %set(h5,'XData',-tmp(:,:,1));
  84. %set(h5,'YData',-tmp(:,:,2));
  85. %set(h5,'ZData',-tmp(:,:,3));
  86.  
  87. drawnow;
  88. disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);
  89. end
  90. % Delete the context object
  91. mxNiDeleteContext(context);

MN322M3.m function

  1. function [M]=mn32m3v2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=[X Y Z];

MNRGB2MRGB.m function

  1. function [M]=mnrgb2mrgbv2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=uint8([X Y Z]);

Tags:

Tim Zaman

MSc Biorobotics. Specialization in computer vision and deep learning. Works at NVIDIA.

You may also like...

28 Responses

  1. Adam says:
    April 30, 2014 at 09:46

    Hi Tim,
    I admire your work and wanna try that on my computer.
    However, I have trouble to find the “download folder” you mentioned in the comments.
    I was wondering if you can help me out.

  2. Ghost says:
    May 9, 2012 at 17:12

    Can you update this project? please Will be very nice.

  3. Jesus says:
    May 5, 2012 at 02:09

    I’m sorry. I am running this sample but Matlab show me one error:

    ??? Undefined function or method ‘mxNiImage’ for input arguments of type ‘struct’.

    Error in ==> main at 41
    [rgb, depth] = mxNiImage(context, option);

    How can implement that function or how can I solve that problem.
    Thank you very much !

  4. khadhraoui says:
    December 9, 2011 at 22:09

    code matlab

  5. hano says:
    November 5, 2011 at 14:03

    Dear Tim,

    I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?

    Bests,
    hano

  6. Rui says:
    October 24, 2011 at 15:37

    Hi Tim,

    I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?

    Bests,
    Rui

  7. Bryan says:
    October 5, 2011 at 13:48

    Dear Tim,

    I am really enjoying your MATLAB programs.
    However, since I am a real newbie, I could not get through the demo program of yours, “Real Time Face Reconstruction using Matlab”.
    Is therer any other Matlab functions to install? I’ve got the following errors
    during the Matlab execution. I use Win7(32bit). Other Demo programs acquiring IR images, Depth Images, RealworldCoordinates, all work flawlessly.
    Thank you so much.
    ??? Error using ==> delaunay
    The input points must be a double array.

    Error in ==> sample_FaceConstruction at 58
    t=delaunay(p(:,1),p(:,2));

    Bryan

  8. Phuoc Nguyen says:
    July 9, 2011 at 15:19

    Hi Tim,
    How many poses did you use for reconstruction?
    Regards,
    Phuoc Nguyen

    • Tim says:
      July 9, 2011 at 20:04

      for the 3d reconstruction of my face you see in this post i used only one.

      • Phuoc Nguyen says:
        July 10, 2011 at 09:42

        Hi Tim,
        Thank you for reply. If you used one pose there are some parts of face would be invisible. How can you deal with that? I didn’t see you use any interpolation method in your code. Correct me if I am wrong.
        Regards,
        Phuoc Nguyen

      • Phil says:
        July 10, 2011 at 09:45

        Delaunayrtriangles take care of that. and, the back of the face is invisible, right.. its just one frame. use your kinect and see for yourself.

  9. Seongsu mun says:
    July 5, 2011 at 07:00

    dear Tim, thank so much your code.

    I don’t know about mn32m3. Is it a function provided by matlab? Or did you make it?

    please answer me.

    Thank you~

    • Tim says:
      July 5, 2011 at 12:06

      Hi. mn32m3 is a code i made myself, it is provided in the post. The code’s name is “mn32m3” which means (MxNx3 to Mx3) so it does a fairly simple but important matrix transformation.

  10. John says:
    June 21, 2011 at 04:52

    Hi Tim,

    You stated that the accuracy is ~ 5 mm ? The depth image resolution of Kinect is at least 1 cm ( http://openkinect.org/wiki/Imaging_Information ). For example, ff I choose my nose and exam the depth value around that, some pixels have the same depth values. Therefore, I suspect the accuracy of Kinect. Any idea?

    • Tim says:
      June 21, 2011 at 12:43

      Oh come on, you’re talking nonsense. Can’t you see the above video i made from scanning my own face? In this post? You can see a lot of detail there, the quality is around 2mm if you ask me.

  11. Gabriel says:
    June 15, 2011 at 20:58

    Hi Tim,
    I was wondering if you could help me. I am trying to do the same thing you do, but from scratch, that is, I have depth image, rgb image, the intrinsic parameters and that’s it . The first thing I would like to do, is to project to real world the pixels in depth image. I would later take care of the color..
    I do it by using this code,
    %I1(i,j)=depth(i,j)
    [m n]=size(I1);
    k=1;
    c_x =3.3930780975300314e+02;
    c_y = 2.4273913761751615e+02;
    f_x= 5.9421434211923247e+02;
    f_y= 5.9104053696870778e+02;

    for i=1:m
    for j=1:n
    mundo_real(k,:)=[ (i-c_x)*I1(i,j)/f_x (j-c_y)*I1(i,j)/f_y I1(i,j) ];
    % mundo_real(k,:)=[i I1(i,j) j];
    k=k+1;
    end
    end
    However, I am not getting what I want.
    Do you think you can help me?
    Thanks in advance,
    Gabriel

    • Tim says:
      June 15, 2011 at 21:17

      I don’t see your point.. i mean, i do the same thing with this code with just the depth image and rgb image. with a little tweaking you can almost ignore your parameters. Also, the code you point to truly sucks as it has a for-loop in a for-loop and for-loops should be avoided (super slow). As you can see i do the same thing without for-loops.

      The thing you have to do is easy: take your depth image (XYZ pointcloud) make it into a triangular polygonic form (use delaunay function as in my code above) and just apply the rgb image onto it, as i do in the code above.

      T

  12. love to kinect says:
    May 27, 2011 at 09:24

    dear tim, can help me in more detail?

    if it possible to make a tutorial movie for this code ? or should more accurate text ?
    i couldn’t run this code with matlab 2009b, 2010a, 2010b, 2011a !!

    we get this error with 2009a version,

    ??? Error using ==> cgprechecks at 47
    Not enough unique points specified.

    Error in ==> delaunay at 59
    cgprechecks(X, nargin-1, cg_opt);

    i’m so sad because i couldn’t run it ! please help me, if it possible.

    thanks a million.

  13. love to kinect says:
    May 26, 2011 at 22:05

    dear Tim, thank so much from your code, but it’s not worked for me.
    i run the 2011 version of matlab, and get the following error, can help me? regards.

    —-

    One or more of the following nodes could not be enumerated:

    ??? Error using ==> delaunay
    The input points must be a double array.

    Error in ==> a at 58
    t=delaunay(p(:,1),p(:,2));

    • Tim says:
      May 26, 2011 at 22:16

      Not sure.. i used 2009, maybe even 2008. Just check out what goes wrong step by step by asking “class(p)” for instance, to see what classes everything are. Know how the code works if you use it; otherwise you gain nothing from it =)

  14. Arthur da Igreja says:
    May 5, 2011 at 18:52

    Congratulations on your project! I´m developing applications with Kinect also, have you tried the ”warp” command to apply the image in the surface provided by the variable ”depth”? It worked fine for me.

  15. Tim says:
    April 10, 2011 at 11:02

    Okay that’s not so bad. Line 51 makes sure you’re within 80cm of the camera. So just cut out that part, or sit closer to the Kinect: otherwise it doesnt have any info to process: that’s why you’re getting the error, it doesnt have any data -i guess-. Check that. Also, try playing with the functions first and try to understand what the code does then.
    T

  16. tariq says:
    April 10, 2011 at 10:57

    I have tried your program [3D] Realtime Face Reconstruction with Matlab and Kinect but I got a problem.

    Open failed: Memory allocation failed!
    Check whether SamplesConfig.xml is available.
    ??? Error using ==> cgprechecks at 47
    Not enough unique points specified.

    Error in ==> delaunay at 59
    cgprechecks(X, nargin-1, cg_opt);

    Error in ==> rekonstruksi at 58
    t=delaunay(p(:,1),p(:,2));