# [3D] Reconstruct object from multiple Kinect captures in Matlab

This code written in Matlab demonstrates how to reconstruct 3D information from an object, using multiple orientated images captured with the Kinect. You need at least 1 image for the reconstruction, and then you rotate the object, and let the kinect take a picture: so it makes a full 3D scan using the Kinect, that combines multiple shots.

This is what my program does:

1. Takes an image
2. You rotate the object
3. Takes another image
4. …repeat x times
5. Then 3D-rotates the images to form one big point cloud
6. Makes a trigonial mesh

This code does not use smart algorithms like SLAM or something.

Figure out how it works yourself, its fairly obvious:

1. `% sample`
2. `% MEX sample file to acquire and rgb and depth image used via Kinect and`
3. `%     convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ`
4. ` `
5. `% MEX Function Reference:`
6. `% -------------------------------------------------------------------------`
7. `% [Name]`
8. `% mxNiConvertProjectiveToRealWorld`
9. `%`
10. `% [Function]`
11. `% Convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ`
12. `% [u[pix] v[pix] d[mm]] -> [X[mm] Y[mm] Z[mm]]`
13. `%`
14. `% [Usage]`
15. `% real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth);`
16. `%`
17. `% [Arguments]`
18. `% context`
19. `% The structure which includes object for acquiring images.`
20. `% It is not used in Matlab directly, but used in mex file.`
21. `%`
22. `% depth`
23. `% Depth image`
24. `%`
25. `% [Return]`
26. `% real_XYZ`
27. `% converted XYZ value on world coordinate system`
28. `%`
29. `% [Sample]`
30. `% sample_niConvertCoordinate.m`
31. `% -------------------------------------------------------------------------`
32. `%`
33. `% -------------------------------------------------------------------------`
34. `% [Name]`
35. `% mxNiConvertRealWorldToProjective`
36. `%`
37. `% [Function]`
38. `% Convert XYZ on world coordinate system to pixel (u,v) and depth value[mm]`
39. `% [X[mm], Y[mm], Z[mm]] -> [u[pix], v[pix], d[mm]]`
40. `%`
41. `% real_XYZ`
42. `% XYZ value on world coordinate system`
43. `%`
44. `% [Usage]`
45. `% projective_XYZ = mxNiConvertRealWorldToProjective(context, real_XYZ);`
46. `%`
47. `% [Arguments]`
48. `% context`
49. `% The structure which includes object for acquiring images.`
50. `% It is not used in Matlab directly, but used in mex file.`
51. `%`
52. `% real_XYZ`
53. `% XYZ value on world coordinate system`
54. `%`
55. `% [Return]`
56. `% projective_XYZ`
57. `% pixel (u,v) and depth value`
58. `%`
59. `% [Sample]`
60. `% sample_niConvertCoordinate.m`
61. `% -------------------------------------------------------------------------`
62. ` `
63. `clc`
64. `clear all`
65. `close all`
66. ` `
67. `% Create context and read xml file at first`
68. `context = mxNiCreateContext('SamplesConfig.xml');`
69. ` `
70. `%% Initialise FIGURE`
71. `width = 640;`
72. `height = 480;`
73. `% depth image`
74. `% figure(1), h1 = imagesc(zeros(height,width,'uint16'));`
75. `%            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`
76. `% rgb image`
77. `% figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));`
78. `%            set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`
79. `% rgb+depth image`
80. `%figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;`
81. `%           set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`
82. `%           h4 = imagesc(zeros(height,width,'uint16')); hold off;`
83. `% converted XYZ plot`
84. `%figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));`
85. `%           axis([-800 800 -600 600 -10000 -1 ]); view(180,90);`
86. `%           xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');`
87. `%           title('XYZ ?iThe world coordinate system?j');`
88. `figure(5);  %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')`
89. `            %axis equal`
90. `            %shading interp %colours are interpolated inside triangles`
91. ` `
92. `disp('go for first image?')`
93. `pause`
94. ` `
95. ` `
96. `%% acquire images`
97. `for k=1:2`
98. `option.adjust_view_point = true;`
99. `% Acquire rgb and depth image`
100. `tic`
101. `[rgb, depth] = mxNiImage(context, option);`
102. ` `
103. `real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system`
104. ` `
105. `[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang`
106. `[Mrgb]=mnrgb2mrgbv2(rgb); %dit duurt erg lang`
107. ` `
108. `itmp=sum(M~=0,2); %locate existing info`
109. `stmp=itmp==3; %positive if there is info`
110. `M=M(stmp,:);`
111. `Mrgb=Mrgb(stmp,:);`
112. ` `
113. `itmp=M(:,3)>-950;`
114. `stmp=itmp==1;`
115. `M=M(stmp,:);`
116. `Mrgb=Mrgb(stmp,:);`
117. ` `
118. `itmp=M(:,2)<50;`
119. `stmp=itmp==1;`
120. `M=M(stmp,:);`
121. `Mrgb=Mrgb(stmp,:);`
122. ` `
123. `%M(:,1)=M(:,1)+40;`
124. `%M(:,2)=M(:,2)-60;`
125. `M(:,3)=M(:,3)+830;`
126. ` `
127. `if k==1`
128. `    Mrgb1=Mrgb;`
129. `    M1=M;`
130. `    disp('1st im OK, press a key for next image')`
131. `    pause`
132. `elseif k==2`
133. ` `
134. `    q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis`
135. `    Mrgb2=Mrgb;`
136. `    M2=M*q;`
137. ` `
138. ` `
139. `    clear M Mrgb itmp stmp real_XYZ`
140. `end`
141. ` `
142. ` `
143. `disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);`
144. `end`
145. ` `
146. `Mrgb=[Mrgb1;Mrgb2];`
147. `M=[M1;M2];`
148. ` `
149. `C=double(Mrgb)/255;`
150. `t=delaunay(double(M(:,1)),double(M(:,2)));`
151. `colormap(C);`
152. `toc`
153. `trisurf(t,M(:,1),M(:,2),M(:,3),1:size(M,1),'edgecolor','none')`
154. `axis equal;view(-150,66)`
155. `shading interp`
156. `drawnow;`
157. ` `
158. `figure`
159. `plot3(M(:,1),M(:,2),M(:,3),'.')`
160. ` `
161. ` `
162. ` `
163. `% Delete the context object`
164. `mxNiDeleteContext(context);`

MN322M3.m function

1. `function [M]=mn32m3v2(MN)`
2. `X=MN(:,:,1)';`
3. `Y=MN(:,:,2)';`
4. `Z=MN(:,:,3)';`
5. `X=X(:);`
6. `Y=Y(:);`
7. `Z=Z(:);`
8. `M=[X Y Z];`

MNRGB2MRGB.m function

1. `function [M]=mnrgb2mrgbv2(MN)`
2. `X=MN(:,:,1)';`
3. `Y=MN(:,:,2)';`
4. `Z=MN(:,:,3)';`
5. `X=X(:);`
6. `Y=Y(:);`
7. `Z=Z(:);`
8. `M=uint8([X Y Z]);` Tim Zaman

MSc Biorobotics. Specialization in computer vision and deep learning. Works at NVIDIA.

### 17 Responses

1. suvarna says:

hi tim,..
what are the software needed to run the above code?

2. Aditya says:

Why am i getting this error and what must i do to rectify it?

??? Undefined function or method ‘mxNiCreateContext’
for input arguments of type ‘char’.

Error in ==> Ad at 68
context = mxNiCreateContext(‘SamplesConfig.xml’);

3. Jesus says:

Dear Mr. Tim
Your website is really good and I like read it.
At present, Im traying to run the project “[3D] Realtime Face Reconstruction with Matlab and Kinect” http://www.timzaman.nl/?p=495&lang=es
But I have one error in ‘mxNiImage’ function (Undefined function or method)
I have Matlab R2011a. Can you help me please?

4. Fadel says:

Hi Tim,
i have already made 2 RGB photos with Kinect, i would like to make a point cloud for one bild of those in Matlab, is this possible? you habe a function for this?
Thnaks

• Tim says:

You mean 2x2D pictures into a 3D picture? check this out http://www.timzaman.nl/?p=179&lang=en . Otherwise, the kinect already makes a pointcloud from your image, so what you are saying doesn’t actually make sense at all.

5. John says:

Hi Zim,

I think that the mex-functions that you are using does not calibrate well (align the depth image and RGB image). How did you solve this problem so that you can reconstruct the 3D object?

Thanks.

• Tim says:

Yeah you can’t change the MEX files.. But i did not do anything with it. It might be your Kinect, but i can’t imagine that being the case.. i have no idea. If it’s not correct: try changing the XYZ information yourself by adding and subtracting; maybe even scaling.

T

6. John says:

Why did you do the “minus” here?

[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

• Tim says:

No Idea. I just wrote this very quickly, but i guess its just to invert the coordinates. No biggy.

• John says:

Sorry for having so many questions. But please explain it to me.
1. Why do you have to ignore point which has y-coordinate <50 here?
itmp=M(:,2)<50;
stmp=itmp==1;
M=M(stmp,:);
Mrgb=Mrgb(stmp,:);

2. by the way, I try to search for information about the world coordinate system but I could not find it. Is the world coordinate system origin the Kinect position?

3. Why do you add 830 mm to z-coordinate of all points?
M(:,3)=M(:,3)+830;

• John says:

Hi Tim,
if you do

[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

and try to reconstruct a book cover, the character is inverse.

I will recommend

function [M]=mn32m3v2(MN)
X= – MN(:,:,1)’;
Y=MN(:,:,2)’;
Z=MN(:,:,3)’;
X=X(:);
Y=Y(:);
Z=Z(:);

M=[X Y Z];

so that it will become normal.

anyway, thanx for the tutorial

7. John says:

Hi Tim. I got the rotationmad3D.m file from Matlab exchange. However, one problem is

??? Error using ==> delaunay
Error computing the Delaunay triangulation. Not enough
unique points specified.

Error in ==> reconstruct3D at 151
t=delaunay(double(M(:,1)),double(M(:,2)));

This also happen with the 3D face reconstruction.
Can you help me to solve it?
Thanks

• Tim says:

This means no points have been specified: that means your kinect doesnt capture any points: make sure your Kinect works with matlab before you run this code.

• John says:

You are so smart. I am trying to understand your code. Your program works for me now.

• satish says:

i need codings for the 3d construction from the two 2d images that is one front side face image and one side view face image pls reply me this is my final year project my email is satish.bayya3001@gmail.com

8. vincenzo says:

Hello Tim,
I have a problem.
1st im OK, press a key for next image
itr=1 : FPS=0.323031

??? Undefined function or method ‘rotationmat3D’ for
input arguments of type ‘double’.

Error in ==> sample3D_Reconstructor at 134
q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis

I’m using Matlab (R2010a), I mentioned the code sample_3DReconstructor.m, mn32m3v2.m, mnrgb2mrgbv2.m ,the two functions, then I copied the three files in the same folder I put your first code, Kinet-mex …
Thanks for your help.
This code can be used by a robot to rebuild ambente where is it?

• Tim says:

you forgot the function rotationmad3D i think.. not sure, at least as you can see its missing. Also, you can rebuild the environment simply by using the SLAM algorithm. look into it, it will take a while to get it running :).