[3D] Reconstruct object from multiple Kinect captures in Matlab

This code written in Matlab demonstrates how to reconstruct 3D information from an object, using multiple orientated images captured with the Kinect. You need at least 1 image for the reconstruction, and then you rotate the object, and let the kinect take a picture: so it makes a full 3D scan using the Kinect, that combines multiple shots.

This is what my program does:

  1. Takes an image
  2. You rotate the object
  3. Takes another image
  4. …repeat x times
  5. Then 3D-rotates the images to form one big point cloud
  6. Makes a trigonial mesh

This code does not use smart algorithms like SLAM or something.

Figure out how it works yourself, its fairly obvious:

  1. % sample
  2. % MEX sample file to acquire and rgb and depth image used via Kinect and
  3. %     convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ
  4.  
  5. % MEX Function Reference:
  6. % -------------------------------------------------------------------------
  7. % [Name]
  8. % mxNiConvertProjectiveToRealWorld
  9. %
  10. % [Function]
  11. % Convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ
  12. % [u[pix] v[pix] d[mm]] -> [X[mm] Y[mm] Z[mm]]
  13. %
  14. % [Usage]
  15. % real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth);
  16. %
  17. % [Arguments]
  18. % context
  19. % The structure which includes object for acquiring images.
  20. % It is not used in Matlab directly, but used in mex file.
  21. %
  22. % depth
  23. % Depth image
  24. %
  25. % [Return]
  26. % real_XYZ
  27. % converted XYZ value on world coordinate system
  28. %
  29. % [Sample]
  30. % sample_niConvertCoordinate.m
  31. % -------------------------------------------------------------------------
  32. %
  33. % -------------------------------------------------------------------------
  34. % [Name]
  35. % mxNiConvertRealWorldToProjective
  36. %
  37. % [Function]
  38. % Convert XYZ on world coordinate system to pixel (u,v) and depth value[mm]
  39. % [X[mm], Y[mm], Z[mm]] -> [u[pix], v[pix], d[mm]]
  40. %
  41. % real_XYZ
  42. % XYZ value on world coordinate system
  43. %
  44. % [Usage]
  45. % projective_XYZ = mxNiConvertRealWorldToProjective(context, real_XYZ);
  46. %
  47. % [Arguments]
  48. % context
  49. % The structure which includes object for acquiring images.
  50. % It is not used in Matlab directly, but used in mex file.
  51. %
  52. % real_XYZ
  53. % XYZ value on world coordinate system
  54. %
  55. % [Return]
  56. % projective_XYZ
  57. % pixel (u,v) and depth value
  58. %
  59. % [Sample]
  60. % sample_niConvertCoordinate.m
  61. % -------------------------------------------------------------------------
  62.  
  63. clc
  64. clear all
  65. close all
  66.  
  67. % Create context and read xml file at first
  68. context = mxNiCreateContext('SamplesConfig.xml');
  69.  
  70. %% Initialise FIGURE
  71. width = 640;
  72. height = 480;
  73. % depth image
  74. % figure(1), h1 = imagesc(zeros(height,width,'uint16'));
  75. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  76. % rgb image
  77. % figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));
  78. %            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  79. % rgb+depth image
  80. %figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;
  81. %           set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);
  82. %           h4 = imagesc(zeros(height,width,'uint16')); hold off;
  83. % converted XYZ plot
  84. %figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));
  85. %           axis([-800 800 -600 600 -10000 -1 ]); view(180,90);
  86. %           xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');
  87. %           title('XYZ ?iThe world coordinate system?j');
  88. figure(5);  %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')
  89.             %axis equal
  90.             %shading interp %colours are interpolated inside triangles
  91.  
  92. disp('go for first image?')
  93. pause
  94.  
  95.  
  96. %% acquire images
  97. for k=1:2
  98. option.adjust_view_point = true;
  99. % Acquire rgb and depth image
  100. tic
  101. [rgb, depth] = mxNiImage(context, option);
  102.  
  103. real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system
  104.  
  105. [M]=-mn32m3v2(real_XYZ); %dit duurt erg lang
  106. [Mrgb]=mnrgb2mrgbv2(rgb); %dit duurt erg lang
  107.  
  108. itmp=sum(M~=0,2); %locate existing info
  109. stmp=itmp==3; %positive if there is info
  110. M=M(stmp,:);
  111. Mrgb=Mrgb(stmp,:);
  112.  
  113. itmp=M(:,3)>-950;
  114. stmp=itmp==1;
  115. M=M(stmp,:);
  116. Mrgb=Mrgb(stmp,:);
  117.  
  118. itmp=M(:,2)<50;
  119. stmp=itmp==1;
  120. M=M(stmp,:);
  121. Mrgb=Mrgb(stmp,:);
  122.  
  123. %M(:,1)=M(:,1)+40;
  124. %M(:,2)=M(:,2)-60;
  125. M(:,3)=M(:,3)+830;
  126.  
  127. if k==1
  128.     Mrgb1=Mrgb;
  129.     M1=M;
  130.     disp('1st im OK, press a key for next image')
  131.     pause
  132. elseif k==2
  133.  
  134.     q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis
  135.     Mrgb2=Mrgb;
  136.     M2=M*q;
  137.  
  138.  
  139.     clear M Mrgb itmp stmp real_XYZ
  140. end
  141.  
  142.  
  143. disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);
  144. end
  145.  
  146. Mrgb=[Mrgb1;Mrgb2];
  147. M=[M1;M2];
  148.  
  149. C=double(Mrgb)/255;
  150. t=delaunay(double(M(:,1)),double(M(:,2)));
  151. colormap(C);
  152. toc
  153. trisurf(t,M(:,1),M(:,2),M(:,3),1:size(M,1),'edgecolor','none')
  154. axis equal;view(-150,66)
  155. shading interp
  156. drawnow;
  157.  
  158. figure
  159. plot3(M(:,1),M(:,2),M(:,3),'.')
  160.  
  161.  
  162.  
  163. % Delete the context object
  164. mxNiDeleteContext(context);

MN322M3.m function

  1. function [M]=mn32m3v2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=[X Y Z];

MNRGB2MRGB.m function

  1. function [M]=mnrgb2mrgbv2(MN)
  2. X=MN(:,:,1)';
  3. Y=MN(:,:,2)';
  4. Z=MN(:,:,3)';
  5. X=X(:);
  6. Y=Y(:);
  7. Z=Z(:);
  8. M=uint8([X Y Z]);

Tim Zaman

MSc Biorobotics. Specialization in computer vision and deep learning. Works at NVIDIA.

You may also like...

17 Responses

  1. suvarna says:

    hi tim,..
    what are the software needed to run the above code?

  2. Aditya says:

    Why am i getting this error and what must i do to rectify it?

    ??? Undefined function or method ‘mxNiCreateContext’
    for input arguments of type ‘char’.

    Error in ==> Ad at 68
    context = mxNiCreateContext(‘SamplesConfig.xml’);

  3. Jesus says:

    Dear Mr. Tim
    Your website is really good and I like read it.
    At present, Im traying to run the project “[3D] Realtime Face Reconstruction with Matlab and Kinect” http://www.timzaman.nl/?p=495&lang=es
    But I have one error in ‘mxNiImage’ function (Undefined function or method)
    I have Matlab R2011a. Can you help me please?

  4. Fadel says:

    Hi Tim,
    i have already made 2 RGB photos with Kinect, i would like to make a point cloud for one bild of those in Matlab, is this possible? you habe a function for this?
    Thnaks
    fadol

  5. John says:

    Hi Zim,

    I think that the mex-functions that you are using does not calibrate well (align the depth image and RGB image). How did you solve this problem so that you can reconstruct the 3D object?

    Thanks.

    • Tim says:

      Yeah you can’t change the MEX files.. But i did not do anything with it. It might be your Kinect, but i can’t imagine that being the case.. i have no idea. If it’s not correct: try changing the XYZ information yourself by adding and subtracting; maybe even scaling.

      T

  6. John says:

    Why did you do the “minus” here?

    [M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

    • Tim says:

      No Idea. I just wrote this very quickly, but i guess its just to invert the coordinates. No biggy.

      • John says:

        Thanks for reply.
        Sorry for having so many questions. But please explain it to me.
        1. Why do you have to ignore point which has y-coordinate <50 here?
        itmp=M(:,2)<50;
        stmp=itmp==1;
        M=M(stmp,:);
        Mrgb=Mrgb(stmp,:);

        2. by the way, I try to search for information about the world coordinate system but I could not find it. Is the world coordinate system origin the Kinect position?

        3. Why do you add 830 mm to z-coordinate of all points?
        M(:,3)=M(:,3)+830;

      • John says:

        Hi Tim,
        if you do

        [M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

        and try to reconstruct a book cover, the character is inverse.

        I will recommend

        function [M]=mn32m3v2(MN)
        X= – MN(:,:,1)’;
        Y=MN(:,:,2)’;
        Z=MN(:,:,3)’;
        X=X(:);
        Y=Y(:);
        Z=Z(:);

        M=[X Y Z];

        so that it will become normal.

        anyway, thanx for the tutorial

  7. John says:

    Hi Tim. I got the rotationmad3D.m file from Matlab exchange. However, one problem is

    ??? Error using ==> delaunay
    Error computing the Delaunay triangulation. Not enough
    unique points specified.

    Error in ==> reconstruct3D at 151
    t=delaunay(double(M(:,1)),double(M(:,2)));

    This also happen with the 3D face reconstruction.
    Can you help me to solve it?
    Thanks

    • Tim says:

      This means no points have been specified: that means your kinect doesnt capture any points: make sure your Kinect works with matlab before you run this code.

      • John says:

        You are so smart. I am trying to understand your code. Your program works for me now.

      • satish says:

        i need codings for the 3d construction from the two 2d images that is one front side face image and one side view face image pls reply me this is my final year project my email is satish.bayya3001@gmail.com

  8. vincenzo says:

    Hello Tim,
    I have a problem.
    1st im OK, press a key for next image
    itr=1 : FPS=0.323031

    ??? Undefined function or method ‘rotationmat3D’ for
    input arguments of type ‘double’.

    Error in ==> sample3D_Reconstructor at 134
    q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis

    I’m using Matlab (R2010a), I mentioned the code sample_3DReconstructor.m, mn32m3v2.m, mnrgb2mrgbv2.m ,the two functions, then I copied the three files in the same folder I put your first code, Kinet-mex …
    Thanks for your help.
    This code can be used by a robot to rebuild ambente where is it?

    • Tim says:

      you forgot the function rotationmad3D i think.. not sure, at least as you can see its missing. Also, you can rebuild the environment simply by using the SLAM algorithm. look into it, it will take a while to get it running :).