# [3D] Realtime Face Reconstruction with Matlab and Kinect

This code written in Matlab demonstrates how to reconstruct a face from information captured by the Kinect. What it does, is that it makes a pointgrid, and fills that grid in with the appropriate color.

I managed to map the texture on the pointgrid aquired from the Kinect 3D image.

**This is what my program does:**

- Gets 3D + live image
- Converts 3D coordinates to real-world coordinates
- Makes a trigonial mesh
- Fills in the entire trigonial mesh with the correct colors (from live image)
- Starts from 1

**Specs:**

- ~2 seconds per frame (0.5 FPS)
- Accurate to ~5mm
- Works from 50cm to at least 700cm
- 0.3 MP Resolution
- If not set to realtime (=postprocessing) framerate would be 30FPS

See below this youtube of my face, which consists of around 40.000 datapoints:

And then here follows the code. Looks basic? Yeah it’s fairly basic.

`clc`

clear all

close all

`% Create context and read xml file at first`

context = mxNiCreateContext('SamplesConfig.xml');

`%% Initialise FIGURE`

`width = 640;`

`height = 480;`

`% depth image`

`% figure(1), h1 = imagesc(zeros(height,width,'uint16'));`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% rgb image`

`% figure(2), h2 = imagesc(zeros(height,width,3,'uint8'));`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% rgb+depth image`

`%figure(3), h3 = imagesc(zeros(height,width,3,'uint8')); hold on;`

`% set(gcf,'MenuBar','none'); set(gca,'Position',[0 0 1 1]);`

`% h4 = imagesc(zeros(height,width,'uint16')); hold off;`

`% converted XYZ plot`

`%figure(4); h5 = mesh(zeros(height,width,'double'),zeros(height,width,'double'),zeros(height,width,'double'));`

`% axis([-800 800 -600 600 -10000 -1 ]); view(180,90);`

`% xlabel('X [mm]'); ylabel('Y [mm]'); zlabel('Z [mm]');`

`% title('XYZ ?iThe world coordinate system?j');`

figure(5); %h6=trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')

`%axis equal`

`%shading interp %colours are interpolated inside triangles`

`%% acquire images`

for k=1:2

`option.adjust_view_point = true;`

`% Acquire rgb and depth image`

`tic`

[rgb, depth] = mxNiImage(context, option);

real_XYZ = mxNiConvertProjectiveToRealWorld(context, depth); % return XYZ[mm, mm, mm] on the world coordinate system

[M]=-mn32m3(real_XYZ); %dit duurt erg lang

[Mrgb]=mnrgb2mrgb(rgb); %dit duurt erg lang

itmp=sum(M~=0); %locate existing info

stmp=itmp~=0; %positive if there is info

M=M(:,stmp);

Mrgb=Mrgb(:,stmp);

itmp=M(3,:)>-800;

`stmp=itmp==1;`

M=M(:,stmp);

Mrgb=Mrgb(:,stmp);

p=M';

C=double(Mrgb')/255;

t=delaunay(p(:,1),p(:,2));

colormap(C);

trisurf(t,p(:,1),p(:,2),p(:,3),1:size(p,1),'edgecolor','none')

axis equal;view(200,66)

`shading interp`

`%real_X = real_XYZ(:,:,1);`

`%real_Y = real_XYZ(:,:,2);`

`%real_Z = real_XYZ(:,:,3);`

`% update FIGURE`

`% set(h1,'CData',depth);`

`% set(h2,'CData',rgb);`

`%set(h3,'CData',rgb);`

`%set(h4,'CData',depth);`

`%set(h4,'AlphaData',double(depth/50));`

`%`

`%tmp = real_XYZ(:);`

`%tmp(find(real_XYZ==0)) = NaN;`

`%tmp = reshape(tmp,[height width 3]);`

`%set(h5,'XData',-tmp(:,:,1));`

`%set(h5,'YData',-tmp(:,:,2));`

`%set(h5,'ZData',-tmp(:,:,3));`

`drawnow;`

disp(['itr=' sprintf('%d',k) , ' : FPS=' sprintf('%f',1/toc)]);

`end`

`% Delete the context object`

mxNiDeleteContext(context);

MN322M3.m function

function [M]=mn32m3v2(MN)

X=MN(:,:,1)';

Y=MN(:,:,2)';

Z=MN(:,:,3)';

X=X(:);

Y=Y(:);

Z=Z(:);

M=[X Y Z];

MNRGB2MRGB.m function

function [M]=mnrgb2mrgbv2(MN)

X=MN(:,:,1)';

Y=MN(:,:,2)';

Z=MN(:,:,3)';

X=X(:);

Y=Y(:);

Z=Z(:);

M=uint8([X Y Z]);

Hi Tim,

I admire your work and wanna try that on my computer.

However, I have trouble to find the “download folder” you mentioned in the comments.

I was wondering if you can help me out.

Can you update this project? please Will be very nice.

Too busy with other projects! Havent touched the kinect in ages.

I’m sorry. I am running this sample but Matlab show me one error:

??? Undefined function or method ‘mxNiImage’ for input arguments of type ‘struct’.

Error in ==> main at 41

[rgb, depth] = mxNiImage(context, option);

How can implement that function or how can I solve that problem.

Thank you very much !

It’s supposed to be in one of the packages you have downloaded with the functions.

code matlab

Thanks for taking the time to comment. lol

Dear Tim,

I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?

Bests,

hano

Hi Tim,

I am trying the thing you were doing. However, Ican’t find the SamplesConfig.xml and function – mxNiConvertProjectiveToRealWorld. Could you share the code please?

Bests,

Rui

Should be in the downloadfolder that contain the “MX” functions

Dear Tim,

I am really enjoying your MATLAB programs.

However, since I am a real newbie, I could not get through the demo program of yours, “Real Time Face Reconstruction using Matlab”.

Is therer any other Matlab functions to install? I’ve got the following errors

during the Matlab execution. I use Win7(32bit). Other Demo programs acquiring IR images, Depth Images, RealworldCoordinates, all work flawlessly.

Thank you so much.

??? Error using ==> delaunay

The input points must be a double array.

Error in ==> sample_FaceConstruction at 58

t=delaunay(p(:,1),p(:,2));

Bryan

Well, i guess if you are a newbie you should start with newbie projects 🙂 there are plenty of those! 😀

Hi Tim,

How many poses did you use for reconstruction?

Regards,

Phuoc Nguyen

for the 3d reconstruction of my face you see in this post i used only one.

Hi Tim,

Thank you for reply. If you used one pose there are some parts of face would be invisible. How can you deal with that? I didn’t see you use any interpolation method in your code. Correct me if I am wrong.

Regards,

Phuoc Nguyen

Delaunayrtriangles take care of that. and, the back of the face is invisible, right.. its just one frame. use your kinect and see for yourself.

dear Tim, thank so much your code.

I don’t know about mn32m3. Is it a function provided by matlab? Or did you make it?

please answer me.

Thank you~

Hi. mn32m3 is a code i made myself, it is provided in the post. The code’s name is “mn32m3” which means (MxNx3 to Mx3) so it does a fairly simple but important matrix transformation.

Hi Tim,

You stated that the accuracy is ~ 5 mm ? The depth image resolution of Kinect is at least 1 cm ( http://openkinect.org/wiki/Imaging_Information ). For example, ff I choose my nose and exam the depth value around that, some pixels have the same depth values. Therefore, I suspect the accuracy of Kinect. Any idea?

Oh come on, you’re talking nonsense. Can’t you see the above video i made from scanning my own face? In this post? You can see a lot of detail there, the quality is around 2mm if you ask me.

Hi Tim,

I was wondering if you could help me. I am trying to do the same thing you do, but from scratch, that is, I have depth image, rgb image, the intrinsic parameters and that’s it . The first thing I would like to do, is to project to real world the pixels in depth image. I would later take care of the color..

I do it by using this code,

%I1(i,j)=depth(i,j)

[m n]=size(I1);

k=1;

c_x =3.3930780975300314e+02;

c_y = 2.4273913761751615e+02;

f_x= 5.9421434211923247e+02;

f_y= 5.9104053696870778e+02;

for i=1:m

for j=1:n

mundo_real(k,:)=[ (i-c_x)*I1(i,j)/f_x (j-c_y)*I1(i,j)/f_y I1(i,j) ];

% mundo_real(k,:)=[i I1(i,j) j];

k=k+1;

end

end

However, I am not getting what I want.

Do you think you can help me?

Thanks in advance,

Gabriel

I don’t see your point.. i mean, i do the same thing with this code with just the depth image and rgb image. with a little tweaking you can almost ignore your parameters. Also, the code you point to truly sucks as it has a for-loop in a for-loop and for-loops should be avoided (super slow). As you can see i do the same thing without for-loops.

The thing you have to do is easy: take your depth image (XYZ pointcloud) make it into a triangular polygonic form (use delaunay function as in my code above) and just apply the rgb image onto it, as i do in the code above.

T

dear tim, can help me in more detail?

if it possible to make a tutorial movie for this code ? or should more accurate text ?

i couldn’t run this code with matlab 2009b, 2010a, 2010b, 2011a !!

we get this error with 2009a version,

??? Error using ==> cgprechecks at 47

Not enough unique points specified.

Error in ==> delaunay at 59

cgprechecks(X, nargin-1, cg_opt);

i’m so sad because i couldn’t run it ! please help me, if it possible.

thanks a million.

dear Tim, thank so much from your code, but it’s not worked for me.

i run the 2011 version of matlab, and get the following error, can help me? regards.

—-

One or more of the following nodes could not be enumerated:

??? Error using ==> delaunay

The input points must be a double array.

Error in ==> a at 58

t=delaunay(p(:,1),p(:,2));

Not sure.. i used 2009, maybe even 2008. Just check out what goes wrong step by step by asking “class(p)” for instance, to see what classes everything are. Know how the code works if you use it; otherwise you gain nothing from it =)

Congratulations on your project! I´m developing applications with Kinect also, have you tried the ”warp” command to apply the image in the surface provided by the variable ”depth”? It worked fine for me.

Okay that’s not so bad. Line 51 makes sure you’re within 80cm of the camera. So just cut out that part, or sit closer to the Kinect: otherwise it doesnt have any info to process: that’s why you’re getting the error, it doesnt have any data -i guess-. Check that. Also, try playing with the functions first and try to understand what the code does then.

T

I have tried your program [3D] Realtime Face Reconstruction with Matlab and Kinect but I got a problem.

Open failed: Memory allocation failed!

Check whether SamplesConfig.xml is available.

??? Error using ==> cgprechecks at 47

Not enough unique points specified.

Error in ==> delaunay at 59

cgprechecks(X, nargin-1, cg_opt);

Error in ==> rekonstruksi at 58

t=delaunay(p(:,1),p(:,2));