5D.SlowMo R&D
The SlowMo plug-in for Jaleo
generates an output sequence of a different length from its input sequence by using motion
analysis and morphing techniques. The version of SlowMo in the current Release 3.0 of 5D
Monsters provides for a fixed integer ratio of output frames to input frames, limiting the
results to slowing down the sequence by a constant factor. We would like to produce a
version of SlowMo which uses a curve to lookup the desired output (fractional) output frame
for each input frame. This would allow varispeed effects, including speeding up the output
sequence as well as slowing it down. Unfortunately the Jaleo API does not allow random
access of frames from the input clip which is very necessary.
The vast majority of "time warping"
systems generate an output frame simply by mixing between the input frames that
"bound" it. For example, if we are slowing down an input sequence by a factor of
4, we would want output frames corresponding to input frames 1.0, 1.25, 1.50, 1.75, 2.00
and so on. The simple method is to generate the 3rd output frame simply as a 50% mix of
input frames 1 and 2.
SlowMo, on the other hand, would proceed in two
phases.
Phase One:
- It would take input frames 1 and 2 and estimate a
motion vector at each pixel in frame 1 that would move that pixel to its corresponding
position in frame 2.
- When SlowMo was first developed, a number of
methods of motion estimation were (hurriedly!) examined, including the original Horn and
Schunk optical flow algorithm, phase correlation and block matching followed by motion
vector interpolation. While phase correlation looked promising, block matching and
interpolation was chosen because it worked surprisingly well for many cases of interest,
and in order to get a useful product to market as quickly as possible. It also had speed
advantages and could be readily multi-processed.
- The block matching is carried out on blocks whose
centers are arranged in a fixed grid on the input image. The matching algorithm works on a
resolution pyramid of images, proceeding from low resolution, where it can capture large
motions, then refining these estimates using successively higher resolution versions of
the input images.
- SlowMo's motion estimation method works well for
single fairly large objects moving against plain backgrounds (generally speaking). This is
often found in footage of things shot on blue or green screen - an important class of
sequences.
- On the other hand, we are aware that it does not
work well and was never expected to when there are multiple moving objects, those objects
are small, or when there is a complex background that needs to be protected against
"swimming" effects and other distortions.
Phase Two:
- Having estimated the motion in the image in terms
of a motion vector at each pixel and given the fractional position in time of the output
image between the enclosing input images, the earlier input image is warped forward along
the motion vector paths while the later input image is warped backward along those
vectors. The resulting warped images are then mixed together. In other words, the bounding
input frames are morphed to find the output frame.
We are actively investigating a more
"robust" motion estimation technique, based on "advanced" optical flow
algorithms. These methods will hopefully accurately assign motion vectors to the parts of
the images that move, and should cope much better with multiple moving objects, while not
incorrectly distorting the background. Unfortunately, these algorithms will probably run
significantly more slowly than the current method and will be less amenable to
multiprocessing.
SlowMo was developed with the aim of producing a
speed changing method that could be implemented without years of research and would run
reasonably quickly on current hardware.
Index