[Imaging] Lytro In Matlab

This post tells you how to use images and pictures shot by the Lytro camera with Matlab. Lytro images are called “light field images” or dynamic images are whatever, but whatever they tell you; it’s a normal image made with a “micro-lens-array” in front of the digital sensor. Sounds easy? It is easy.

What can you do with a light-field picture?

Essentially three awesome things no other method does, because you can to it retroactively (a posteriori).
1) You can move the observer, to make a tele image or a wide angle image
2) You can move the observer around (within the distance of the lens size, pinhole style)
3) You can be selective in focussing/blurring (this feature is well known)

First things first. How many megapixels does the Lytro camera have? Simple. It’s a square sensor featuring 3280×3280 pixels. This equates to 11 megapixels. The awesome thing is that it outputs 12-bit information, as opposed to normal camera’s outputting 8bit. The difference is huge, and they need it because they do all kinds of smart things with the information using this micro lens array.

First off, the microlens-array is put on the digital sensor, and it’s not exactly aligned. You can see this here:


Lenticular array misalignment

The amount of misalignment is actually hidden in a metadata file. Look for ‘mla’. Example:

		"mla" : {
			"tiling" : "hexUniformRowMajor",
			"lensPitch" : 1.3898614883422850808e-05,
			"rotation" : -0.00028155732434242963791,
			"defectArray" : [],
			"scaleFactor" : {
				"x" : 1,
				"y" : 1.0004389286041259766
			},
			"sensorOffset" : {
				"x" : -3.5040323734283452459e-06,
				"y" : -1.6298500299453736302e-06,
				"z" : 2.5000000000000001198e-05
			}
		}

You need this kind of information because knowing the exact positions of the micro-lens-array centers is key to our algorithms.

Workflow

Okay we want to import our images into Matlab. Your image should be the larger one with the .lfp extension. The larger one is a TIFF file, while there also exists a JPEG compressed one.
This .LFP file contains the TIFF file and some metadata. This Metadata we can see as four different parts: the image, metadata reference, private metadata, and a table.

In order to convert or split the .LFP file to TIF in order to use it in Matlab, we can use a handy tool called LFP splitter. You need to compile this (if you don’t know how it works, don’t bother, find another project). You can find lfpsplitter here: https://github.com/nrpatel/lfptools , its in C++, just compile it and you’re good.

./lfpsplitter IMG_0001.lfp

This is a command line interface program that converts the .lfp into these files. The image file it produces is actually a .raw file. Now we have to convert this raw file into a Tiff!

raw2tiff -w 3280 -l 3280 -d short IMG_0001_imageRef0.raw IMG_0001.tif

Excellent! We got our .TIF file. It everything went to plan, this TIFF file is still a BGGR bayer array mosaic with 12 bits per pixel. We have to demosaic this! Luckily, Matlab does this natively.

Now what follows is some images, a video, and my Matlab code that uses the .TIF file so that you can play around with your files. If you want some sample files you can contact me or browse around on the web for sample .lfp files.

Selected Images


Crossection of the Lytro

Selected Video

Matlab Code

 
%(c) Tim Zaman 2012, Oct 9th
clc; close all; clear all
 
%Parameters
%wy=8.60; %average width between cells (vertical)
%wx=10.0; %average width between cells (horizontal)
 
%From JSON
rotation= -0.00028155732434242963791;
offset_x= -3.5040323734283452459e-0;
offset_y= -1.6298500299453736302e-06;
offset_z=  2.5000000000000001198e-05;
 
%Calculated
wx=9.94036;
wy=8.60585;
 
imraw=imread('0027.tif');
 
imdem = demosaic(imraw,'bggr');
imdemr=imrotate(imdem,rad2deg(-rotation));
 
imdembig=imresize(imdemr,4,'lanczos3');
 
imdembig=imdembig(34:end,37:end,:);
 
 
imsamp1=imdembig(20:wy*4*2:end,20:wx*4:end,:);
 
imsamp2=imdembig(53:wy*4*2:end,40:wx*4:end,:);
 
 
i=0;
imsamptot=0;
for y=-8:8
    for x=-8:8
        imsamptot=imsamptot+double(imdembig(y+20:wy*4*2:end,x+20:wx*4:end,:));
        i=i+1;
    end
end
 
imagesc(uint16(imsamptot/i))
 
 
 
break;
 
for i=0:0.5:20
    xxx=sin(i)*7;
    yyy=cos(i)*7;
imsamp1=imdembig(yyy+20:wy*4*2:end,xxx+20:wx*4:end,:);
imagesc(imsamp1);
drawnow
pause(0.01)
end
 
while 1==1
[gX,gY] = ginput(1);
gX=((gX/size(imsamp1,2))-0.5)*20;
gY=((gY/size(imsamp1,1))-0.5)*20;
imsamp1=imdembig(gY+20:wy*4*2:end,gX+20:wx*4:end,:);
imagesc(imsamp1);
axis equal
drawnow
end
 
 
 
%imsamp2=imdembig(54:wy*4*2:end,40:wx*4:end,:);
%figure
%imshow(imsamp2);
 
 
 
 
 
 
break;
 
wx=10*4*2;
wy= 8.6*4*2;
 
 
for i=0:0.2:20
 
 
s=sin(i)*wx/2;
c=cos(i)*wy/2;
 
rangey=round(abs(53:wy:size(imdembig,2)-50)+s); %vertical
rangex=round(abs(56:wx:size(imdembig,1)-50)+c); %horizontal
 
imagesc(imdembig(rangey,rangex,:))
drawnow
pause(0.01)
end

Tim Zaman

MSc Biorobotics. Specialization in computer vision and deep learning. Works at NVIDIA.

You may also like...

10 Responses

  1. 李晨 says:

    I want to know the complete code,thank you very much。

  2. Ekta says:

    Hi Tim, thanks for this code and explanation. It is very useful. I came across some sample .lfp files on the web. But, I am looking for some more sample files (preferably using lytro’s version 2 software). Can you please help me out by emailing me some more .lfp files?

  3. Jake says:

    Hi Tim, I am wondering how did you get the information of horizontal/vertical lens dimension w.r.t. pixel dimension (they are set as 9.94 and 8.60 respectively in your m file for 0027.tif); is it related to the pixel scale factor?

    Thanks!
    Jake

    • Ekta says:

      Hi Jake

      Even I am stuck at understanding how Tim has obtained wx and wy values in this code. Did you figure out how? I yes, please do let me know!

      Thanks
      Ekta

  4. Martijn says:

    “The awesome thing is that it outputs 12-bit information, as opposed to normal camera’s outputting 8bit.”

    This is not true. Normal camera’s output 12-bit pixels, some even more. Unless you put them in JPEG mode of course, duh, but we’re talking RAW here.

    • Tim says:

      Depends on your definition of camera. DSLR’s, yes; computer-vision camera’s, often those are 8-bit.

  5. Moonkey says:

    Hi, thanks for your effort! I got this error when I use the code in this page:
    Array dimensions must match for binary array op.

    Error in (line 38)
    imsamptot=imsamptot+double(imdembig(y+20:wy*4*2:end,x+20:wx*4:end,:));

    I have changed filename, rotation and offset_x,y,z, and nothing else. I do not know matlab very much. So can you help me with this? Thanks!

  6. Yuchen Zhang says:

    Hello, do you have the matlab code for refocusing? Now I am studying how to refocus at a different depth from its orginal picture(that’s the .tif file),but after reading Ren Ng’s paper , I still can’t do it. Please help me !