Bringing Photoshop image-stacking to mobile with FFmpeg, Hugin, ImageMagick & Hyper.sh

Intro #

At the end of last year I started paying for a Creative Cloud subscription so that I could use the Adaptive Wide-Angle Filter to correct distorted images taken with my Moment Superfish Lens and play with image stacking to take more interesting night-time shots.

While the Adaptive Wide-Angle in Photoshop is really well implemented, (Example before and after) I found that with the SKRWT 'suite' of apps I could get much the same result from using just my phone.

However, I've been unable to find an app that does image stacking on Android. This is a stacked-edit from Photoshop using photos taken on my phone over Christmas. Note the stars and roads in the distance.

I'd love to be able to create images like this without needing to sit down and wait for Photoshop as well as various uploads and downloads to Dropbox in between.

I'm opposed to using Photoshop in my 'workflow' for the following reasons:

• I need to use my laptop, often this means uploading many photos and loading them into a stack takes ages.
• It is not free
• Pouring over Photoshop for hours on your laptop is hardly in the spirit of "the best camera is the one you have with you"

So with SKRWT covering my Adaptive Wide-angle needs I set out to create a tool for image stacking I could use from my phone.

Fundamentals #

At the most basic level I need to be able to turn videos into stacked stills with the option of tuning various parameters (I can turn sets of images into videos on my phone if need be - Google Photos to create an Animation, then GIF to MP4).

You can do this with the following commands:

ffmpeg -i video.mp4 -r $FPS -f image2 frame_%07d.png
convert frame_* -evaluate-sequence mean stacked.jpg

Sometimes I also need to align the stills picked from the video:

ffmpeg -i video.mp4 -r $FPS -f image2 frame_%07d.png

if [ "$ALIGN" = true ] ; then
	align_image_stack -m -a aligned_ frame_*
	convert aligned_* -evaluate-sequence mean stacked.jpg
else
	convert frame_* -evaluate-sequence mean stacked.jpg
fi

It'd also be nice to use more of the evaluate-sequence modes:

MODE=median
MODE=min
MODE=max
convert frame_* -evaluate-sequence $MODE stacked.jpg

I tested this out on my laptop and it worked well enough. Aligning the images was slow but it takes forever in Photoshop too.

This is the easy bit though. Next I needed to work out how to run this from my phone.

I called my thing stackr #

From the commands above I knew I needed FFmpeg, Hugin and ImageMagick. I also knew that I couldn't run this on Heroku because of timeouts. Similarly, it's awkward to run on Lambda as sometimes a job might take over 5 minutes to complete.

What I needed was to be able to run an arbitrary container with some params. Hyper.sh to the rescue!

I'd recently enjoyed using Hyper.sh to automate my photo website and thought it'd work well here too - mainly because containers can run for as long as required.

I also needed to run a frontend that could handle the video uploads and present the options form. I decided to just run this as a sinatra app on Heroku - using what I knew best in the hope of just getting something working.

This is what it looked like by the time I had an MVP. First upload a video with frames to use in the stack. Choosing the mode, whether to auto align images and FPS at the same time.

Next, something like this happens:

Using a bucket to store the video, the Heroku app spawns a container to generate the stacked image. When it's done, it sends me a notification with Pushover.

With this I can quite easily experiment with different stack modes from my phone

• and with minimal battery impact too.

Some Examples #

I've been inside all weekend so I've got to draw from my past videos for this bit. I only keep the stacked image - not the all the source images.

Standard blurring out the motion of the waves example:

Stacking is also good for removing people from busy shots:

These are both pretty poor quality videos taken on a Galaxy Note II. Here's one of a plane coming in to land at Heathrow this afternoon. No tripod, 4k, pixel 2.

Source Video

Or one from a video I took on the way to work:

Source Video

This one's based of a short video waiting for the lights in putney, just holding it in my hand.

Source Video

You can also create some more unusual ones:

Source Video

Sometimes it's not always best to attempt to align the shots!

Source Video

How can I get it? #

If you're interested to try this out let me know and I can let you in. I'm keeping the site private as each upload as the potential to incur a cost.

I expect to post the code on GitHub in the next day or so.