# [3D] Reconstruct object from multiple Kinect captures in Matlab

This code written in Matlab demonstrates how to reconstruct 3D information from an object, using multiple orientated images captured with the Kinect. You need at least 1 image for the reconstruction, and then you rotate the object, and let the kinect take a picture: so it makes a full 3D scan using the Kinect, that combines multiple shots.

**This is what my program does:**

- Takes an image
- You rotate the object
- Takes another image
- …repeat x times
- Then 3D-rotates the images to form one big point cloud
- Makes a trigonial mesh

This code does *not *use smart algorithms like SLAM or something.

Figure out how it works yourself, its fairly obvious:

`% sample`

`% MEX sample file to acquire and rgb and depth image used via Kinect and`

`% convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ`

`% MEX Function Reference:`

`% -------------------------------------------------------------------------`

`% [Name]`

`% mxNiConvertProjectiveToRealWorld`

`%`

`% [Function]`

`% Convert pixel (u,v) of depth image and depth[mm] to world coordinate XYZ`

`% [u[pix] v[pix] d[mm]] -> [X[mm] Y[mm] Z[mm]]`

`%`

`% [Usage]`

`% real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth);`

`%`

`% [Arguments]`

`% context`

`% The structure which includes object for acquiring images.`

`% It is not used in Matlab directly, but used in mex file.`

`%`

`% depth`

`% Depth image`

`%`

`% [Return]`

`% real_XYZ`

`% converted XYZ value on world coordinate system`

`%`

`% [Sample]`

`% sample_niConvertCoordinate.m`

`% -------------------------------------------------------------------------`

`%`

`% -------------------------------------------------------------------------`

`% [Name]`

`% mxNiConvertRealWorldToProjective`

`%`

`% [Function]`

`% Convert XYZ on world coordinate system to pixel (u,v) and depth value[mm]`

`% [X[mm], Y[mm], Z[mm]] -> [u[pix], v[pix], d[mm]]`

`%`

`% real_XYZ`

`% XYZ value on world coordinate system`

`%`

`% [Usage]`

`% projective_XYZ = mxNiConvertRealWorldToProjective(context, real_XYZ);`

`%`

`% [Arguments]`

`% context`

`% The structure which includes object for acquiring images.`

`% It is not used in Matlab directly, but used in mex file.`

`%`

`% real_XYZ`

`% XYZ value on world coordinate system`

`%`

`% [Return]`

`% projective_XYZ`

`% pixel (u,v) and depth value`

`%`

`% [Sample]`

`% sample_niConvertCoordinate.m`

`% -------------------------------------------------------------------------`

`clc`

clear all

close all

`% Create context and read xml file at first`

context = mxNiCreateContext('SamplesConfig.xml');

`%% Initialise FIGURE`

`width = 640;`

`height = 480;`

`% depth image`

`% figure(1), h1 = imagesc(zeros(height,width,'uint16'));`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% rgb image`

`% figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% rgb+depth image`

`%figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% h4 = imagesc(zeros(height,width,'uint16')); hold off;`

`% converted XYZ plot`

`%figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));`

`% axis([-800 800 -600 600 -10000 -1 ]); view(180,90);`

`% xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');`

`% title('XYZ ?iThe world coordinate system?j');`

figure(5); %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')

`%axis equal`

`%shading interp %colours are interpolated inside triangles`

disp('go for first image?')

`pause`

`%% acquire images`

for k=1:2

`option.adjust_view_point = true;`

`% Acquire rgb and depth image`

`tic`

[rgb, depth] = mxNiImage(context, option);

real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system

[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

[Mrgb]=mnrgb2mrgbv2(rgb); %dit duurt erg lang

itmp=sum(M~=0,2); %locate existing info

stmp=itmp==3; %positive if there is info

M=M(stmp,:);

Mrgb=Mrgb(stmp,:);

itmp=M(:,3)>-950;

`stmp=itmp==1;`

M=M(stmp,:);

Mrgb=Mrgb(stmp,:);

itmp=M(:,2)<50;

`stmp=itmp==1;`

M=M(stmp,:);

Mrgb=Mrgb(stmp,:);

`%M(:,1)=M(:,1)+40;`

`%M(:,2)=M(:,2)-60;`

M(:,3)=M(:,3)+830;

if k==1

Mrgb1=Mrgb;

M1=M;

disp('1st im OK, press a key for next image')

`pause`

elseif k==2

q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis

Mrgb2=Mrgb;

M2=M*q;

`clear M Mrgb itmp stmp real_XYZ`

`end`

disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);

`end`

Mrgb=[Mrgb1;Mrgb2];

M=[M1;M2];

C=double(Mrgb)/255;

t=delaunay(double(M(:,1)),double(M(:,2)));

colormap(C);

`toc`

trisurf(t,M(:,1),M(:,2),M(:,3),1:size(M,1),'edgecolor','none')

axis equal;view(-150,66)

`shading interp`

`drawnow;`

`figure`

plot3(M(:,1),M(:,2),M(:,3),'.')

`% Delete the context object`

mxNiDeleteContext(context);

MN322M3.m function

function [M]=mn32m3v2(MN)

X=MN(:,:,1)';

Y=MN(:,:,2)';

Z=MN(:,:,3)';

X=X(:);

Y=Y(:);

Z=Z(:);

M=[X Y Z];

MNRGB2MRGB.m function

function [M]=mnrgb2mrgbv2(MN)

X=MN(:,:,1)';

Y=MN(:,:,2)';

Z=MN(:,:,3)';

X=X(:);

Y=Y(:);

Z=Z(:);

M=uint8([X Y Z]);

hi tim,..

what are the software needed to run the above code?

Why am i getting this error and what must i do to rectify it?

??? Undefined function or method ‘mxNiCreateContext’

for input arguments of type ‘char’.

Error in ==> Ad at 68

context = mxNiCreateContext(‘SamplesConfig.xml’);

Dear Mr. Tim

Your website is really good and I like read it.

At present, Im traying to run the project “[3D] Realtime Face Reconstruction with Matlab and Kinect” http://www.timzaman.nl/?p=495&lang=es

But I have one error in ‘mxNiImage’ function (Undefined function or method)

I have Matlab R2011a. Can you help me please?

Hi Tim,

i have already made 2 RGB photos with Kinect, i would like to make a point cloud for one bild of those in Matlab, is this possible? you habe a function for this?

Thnaks

fadol

You mean 2x2D pictures into a 3D picture? check this out http://www.timzaman.nl/?p=179&lang=en . Otherwise, the kinect already makes a pointcloud from your image, so what you are saying doesn’t actually make sense at all.

Hi Zim,

I think that the mex-functions that you are using does not calibrate well (align the depth image and RGB image). How did you solve this problem so that you can reconstruct the 3D object?

Thanks.

Yeah you can’t change the MEX files.. But i did not do anything with it. It might be your Kinect, but i can’t imagine that being the case.. i have no idea. If it’s not correct: try changing the XYZ information yourself by adding and subtracting; maybe even scaling.

T

Why did you do the “minus” here?

[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

No Idea. I just wrote this very quickly, but i guess its just to invert the coordinates. No biggy.

Thanks for reply.

Sorry for having so many questions. But please explain it to me.

1. Why do you have to ignore point which has y-coordinate <50 here?

itmp=M(:,2)<50;

stmp=itmp==1;

M=M(stmp,:);

Mrgb=Mrgb(stmp,:);

2. by the way, I try to search for information about the world coordinate system but I could not find it. Is the world coordinate system origin the Kinect position?

3. Why do you add 830 mm to z-coordinate of all points?

M(:,3)=M(:,3)+830;

Hi Tim,

if you do

[M]=-mn32m3v2(real_XYZ); %dit duurt erg lang

and try to reconstruct a book cover, the character is inverse.

I will recommend

function [M]=mn32m3v2(MN)

X= – MN(:,:,1)’;Y=MN(:,:,2)’;

Z=MN(:,:,3)’;

X=X(:);

Y=Y(:);

Z=Z(:);

M=[X Y Z];

so that it will become normal.

anyway, thanx for the tutorial

Hi Tim. I got the rotationmad3D.m file from Matlab exchange. However, one problem is

??? Error using ==> delaunay

Error computing the Delaunay triangulation. Not enough

unique points specified.

Error in ==> reconstruct3D at 151

t=delaunay(double(M(:,1)),double(M(:,2)));

This also happen with the 3D face reconstruction.

Can you help me to solve it?

Thanks

This means no points have been specified: that means your kinect doesnt capture any points: make sure your Kinect works with matlab before you run this code.

You are so smart. I am trying to understand your code. Your program works for me now.

i need codings for the 3d construction from the two 2d images that is one front side face image and one side view face image pls reply me this is my final year project my email is satish.bayya3001@gmail.com

Hello Tim,

I have a problem.

1st im OK, press a key for next image

itr=1 : FPS=0.323031

??? Undefined function or method ‘rotationmat3D’ for

input arguments of type ‘double’.

Error in ==> sample3D_Reconstructor at 134

q=rotationmat3D(pi/2,[0 1 0]); %[0 0 1]=z axis

I’m using Matlab (R2010a), I mentioned the code sample_3DReconstructor.m, mn32m3v2.m, mnrgb2mrgbv2.m ,the two functions, then I copied the three files in the same folder I put your first code, Kinet-mex …

Thanks for your help.

This code can be used by a robot to rebuild ambente where is it?

you forgot the function rotationmad3D i think.. not sure, at least as you can see its missing. Also, you can rebuild the environment simply by using the SLAM algorithm. look into it, it will take a while to get it running :).