Focus all parts of Lytro image

One of my coworkers, Max Braun, recently took some pictures with the new Lytro camera. For those unfamiliar with the Lytro technology, the data captured for each photo is more than what is captured by a traditional camera. The outcome of this extra data is the ability to refocus images after capture. Try clicking different points on the image below. Notice how you can bring the individual areas (cards, poker chips, Charles) into focus separately.

After seeing this, I naturally wanted to know if I could focus all parts of the images at once. If you don't care how it works, jump to the result. I asked Max for the data file generated by the camera and started poking around. Turns out Lytro creates two types of files. One contains the original raw data captured by the Lytro camera. The other is a subset of this data encoded as jpeg for various focal depths. The latter data is much smaller and it's used to create the web version you see above. I learned all this and more by reading a blog post by Nirav Patel (no relation). Nirav was also kind enough to create a simple tool called lfpsplitter to split apart the jpeg and meta- data encoded in the files. I ran lfpsplitter on the Lytro data which results in a few files:

$ ./lfpsplitter IMG_0004-stk.lfp
Saved IMG_0004-stk_table.txt
Saved IMG_0004-stk_depth.txt
Saved IMG_0004-stk_00.jpg
Saved IMG_0004-stk_01.jpg
Saved IMG_0004-stk_02.jpg
Saved IMG_0004-stk_03.jpg
Saved IMG_0004-stk_04.jpg
Saved IMG_0004-stk_05.jpg
Saved IMG_0004-stk_06.jpg
Saved IMG_0004-stk_07.jpg
Saved IMG_0004-stk_08.jpg
Saved IMG_0004-stk_09.jpg
Saved IMG_0004-stk_10.jpg
Saved IMG_0004-stk_11.jpg

The _table.txt file includes a JSON data structure which gives you a lot of information but most importantly it gives you a map between the jpg and its focal depth. The _depth.txt file is a flattened 20 x 20 array of depth values. This 20 x 20 array maps to the 1080 x 1080 jpeg images. So each value in the array represents 54 x 54 pixels in jpeg images. The values in the array do not exactly match the values found in the _table.txt file, however it seems that using the corresponding area from the image with the closest focal depth is sufficient. So I whipped up a quick Python script to do this. Please forgive my use of PyGame just for loading/manipulating images. :)

import pygame, glob, json, commands
from pprint import pprint

whichFile = "IMG_0004-stk"
jpegFilenames = glob.glob(whichFile + "*.jpg")
shasumToJpeg = {}
for filename in jpegFilenames:
    shasum = commands.getstatusoutput("shasum " + filename)[1]
    shasum = "sha1-" + shasum.split(" ")[0]
    shasumToJpeg[shasum] = pygame.image.load(filename)
print "Loading filenames:", jpegFilenames
pprint(shasumToJpeg)

print "Loading depth table"
depths = [float(x.strip()) for x in open(whichFile + "_depth.txt")]
print len(depths), "depth numbers found"

print "Loading jpeg -> depth table"
metadata = json.load(open(whichFile + "_table.txt"))
imageArray = metadata["picture"]["accelerationArray"][0]["vendorContent"]["imageArray"]

finalJpeg = pygame.Surface((1080, 1080))
for row in xrange(20):
    for col in xrange(20):
        depth = depths[row*20 + col]

        # find closest jpeg
        closestImg = None
        closestDepth = 100000000
        for i, img in enumerate(imageArray):
            delta = abs(img["lambda"] - depth)
            if (delta < closestDepth):
                closestDepth = delta
                closestImg = img["imageRef"]
        closestImg = shasumToJpeg[closestImg]

        left = col * 54
        top = row * 54

        # take the 54 x 54 block from closestImg
        rect = pygame.Rect(left, top, 54, 54)
        subSurf = closestImg.subsurface(rect)
        finalJpeg.blit(subSurf, rect)

pygame.image.save(finalJpeg, "focused.jpg")

Result

Focused all over!