To enhance the possibilities to apply external velocity to the fluid solver, PixelFlow contains an optical flow module.

Based on the idea of simple frame-differencing (image1 – image2), the optical flow of subsequent images (Frames-Animation, Movie, WebcamCapture, etc…), can be computed from 3 quantities:

  1. dt – difference of frames (frame_A – frame_B)
  2. dx – sum of horizontal grandients (sobelX_A + sobelX_B)
  3. dy – sum of vertical gradients (sobelY_A + sobelY_B)

The normalized vector(dx, dy) is basically already the flow-vector. dt defines the length and direction (positive or negative).

There are several ways to generate the gradients. I got decent results with using the Sobel-Operator. Since everything can be done on the GPU using shaders the overhead is minimal.

Optical Flow Algorithm
update step
{
  preprocessing (blur, noise-reduction, grayscale/color, etc...) - optional
  gradients (sobel, horizontal+vertical)
  optical flow (GLSL-Shader)
  smooth (gaussian blur + temporal smooth) - optional
}
Noise-ReductioN
  1. Preprocessing
    Most obviously, the input images can (and should) be preprocessed. It depends a lot on the source-footage if its worth to apply more expensive filters (median, bilateral filter, …) and/or simply a gaussian blur.
  2. Thresholding
    A threshold, to omit flow-vectors, of a certain length.
  3. PostProcessing
    Gauss-bluring of the resulting flow and finally some temporal smoothing. (mixing the current and the previous flow)
GLSL-Shader – Optical Flow
/**
 * 
 * PixelFlow | Copyright (C) 2016 Thomas Diewald - http://thomasdiewald.com
 * 
 * A Processing/Java library for high performance GPU-Computing (GLSL).
 * MIT License: https://opensource.org/licenses/MIT
 * 
 */

#version 150

precision mediump float;
precision mediump int;

#define SUM_RGB(v) ((v).r + (v).g + (v).b)

out vec2 glFragColor;

uniform sampler2D    tex_curr_frame ; // current  image
uniform sampler2D    tex_prev_frame ; // previous image
uniform sampler2D    tex_curr_sobelH; // current  gradient horizontal
uniform sampler2D    tex_curr_sobelV; // current  gradient vertical
uniform sampler2D    tex_prev_sobelH; // previous gradient horizontal
uniform sampler2D    tex_prev_sobelV; // previous gradient vertical

uniform vec2  wh;                  // size rendertarget
uniform float scale;               // scale flow
uniform float threshold = 1.0;     // flow intensity threshold

void main(){
  
  vec2 posn = gl_FragCoord.xy / wh;
  
  // dt, dx, dy
  vec3 dt_ = texture(tex_curr_frame , posn).rgb - 
             texture(tex_prev_frame , posn).rgb;
  vec3 dx_ = texture(tex_curr_sobelH, posn).rgb + 
             texture(tex_prev_sobelH, posn).rgb;
  vec3 dy_ = texture(tex_curr_sobelV, posn).rgb + 
             texture(tex_prev_sobelV, posn).rgb;
  
  // sum rgb-channels
  float dt = SUM_RGB(dt_), dx = SUM_RGB(dx_),  dy = SUM_RGB(dy_);
  // gradient length
  float dd = sqrt(dx*dx + dy*dy + 1.0);
  // optical flow
  vec2 flow = scale * dt * vec2(dx, dy) / dd; 
  
  // threshold
  float len_old = sqrt(flow.x*flow.x + flow.y*flow.y + 0.00001);
  float len_new = max(len_old - threshold, 0.0);
  flow = (len_new * flow)/len_old;
  
  glFragColor = flow;
}