Difference between revisions of "Reversing digital filters"

From apertus wiki
Jump to: navigation, search
Line 65: Line 65:
   0.145344  0.318310  0.145344
   0.145344  0.318310  0.145344
   0.025079  0.145344  0.025079
   0.025079  0.145344  0.025079
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f1.png


http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f1.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f1.jpg
Line 73: Line 71:


Tip: for pixel peeping, open the above image in a new tab in Firefox, then open the validation image in another tab. They will align perfectly, so you can now switch between them, back and forth.
Tip: for pixel peeping, open the above image in a new tab in Firefox, then open the validation image in another tab. They will align perfectly, so you can now switch between them, back and forth.
Standard deviations of residuals:
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f1.png
Note: when filter size = 0, the filter becomes simply a scaling factor, so the largest value shows the mismatch between the two images (original vs filtered) after scaling them to look equally bright.


3x3 averaging blur:
3x3 averaging blur:
Line 79: Line 83:
   0.11111  0.11111  0.11111
   0.11111  0.11111  0.11111
   0.11111  0.11111  0.11111
   0.11111  0.11111  0.11111
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f2.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f2.png


Sharpen:
Sharpen:
Line 85: Line 93:
   -0.66667  4.33333  -0.66667
   -0.66667  4.33333  -0.66667
   -0.16667  -0.66667  -0.16667
   -0.16667  -0.66667  -0.16667
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f3.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f3.png


Blur followed by sharpen:
Blur followed by sharpen:
Line 93: Line 105:
   -0.0409430  -0.1381697  0.3357311  -0.1381697  -0.0409430
   -0.0409430  -0.1381697  0.3357311  -0.1381697  -0.0409430
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f4.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f4.png


Laplacian of Gaussian (edge detector):
Laplacian of Gaussian (edge detector):
  f5 = @(x) imfilter(x, fspecial('log'));
  f5 = @(x) imfilter(x, fspecial('log'));
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f5.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f5.png


Laplacian of Gaussian plus the original image:
Laplacian of Gaussian plus the original image:
  f6 = @(x) imfilter(x, fspecial('log')) + x;
  f6 = @(x) imfilter(x, fspecial('log')) + x;
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f6.jpg
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f6.png


===Different filters on odd/even columns===
===Different filters on odd/even columns===

Revision as of 16:59, 4 February 2016

[WIP]

1 Motivation

We are trying to record raw video with existing HDMI recorders (that don't know anything about recording raw).

Unfortunately, it seems that some of these recorders apply some processing on the image, like sharpening or blurring. Therefore, it may be a good idea to attempt to undo some of these filters applied to the image without our permission :P

Note: uncompressed versions of all of the images from this page can be found at http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/

Example:

Source image (R, G, B):

red-input-raw-encoded.jpg green-input-raw-encoded.jpg blue-input-raw-encoded.jpg

Recorded image (Atomos Shogun, ProRes 422):

red-hdmi-from-tif.jpg green-hdmi-from-tif.jpg blue-hdmi-from-tif.jpg

The compression looks pretty strong; according to simulation (compressing the source image with prores in ffmpeg), I would expect the recorded image to look like this:

red-422-prores-ffmpeg.jpg green-422-prores-ffmpeg.jpg blue-422-prores-ffmpeg.jpg

Color versions (source, image from shogun, prores ffmpeg profile 2):

rgb-input-raw-encoded.jpg rgb-hdmi-from-tif.jpg rgb-422-prores-ffmpeg.jpg

My guess: the HDMI recorder appears to sharpen the image before compressing, causing the ProRes codec to struggle.

So... good luck recovering the raw image from this!

2 Intro

General idea: feed some test images to the HDMI, compare with the output image from the recorder, and attempt to undo the transformations in order to recover the original image.

We will start by experimenting with simple linear filters on grayscale images, as they are easiest to work with.

3 Linear filters on grayscale images

Input:

  • source image
  • altered image (with an unknown filter)

To find a digital linear filter that would undo the alteration on our test image, we may solve a linear system: each pixel can be expressed as a linear combination of its neighbouring pixels. For a MxN image and a PxP filter, we will have P*P unknowns and M*N equations.

To simplify things, we'll consider filters with odd diameters, so filter size would be PxP = (2*n+1) x (2*n+1).

If we assume our filter is horizontally and vertically symmetrical, the number of unknowns decreases to (n+1) * (n+1).

If we assume our filter is also diagonally symmetrical, the number of unknowns becomes n * (n+1) / 2.

Let's try some examples.

We will use a training data set (a sample image used to compute the filter), and a validation data set (a different image, to check how well the filter does when the input data doesn't match). This is a simple strategy to avoid overfitting [1][2][3]. Maybe not the best one [4], but for a quick experiment, it should do the trick.

Training and validation images:

training.jpg validation.jpg

Blur:

f1 = @(x) imfilter(x, fspecial('disk', 1));
  0.025079   0.145344   0.025079
  0.145344   0.318310   0.145344
  0.025079   0.145344   0.025079

images-f1.jpg

Left: altered image (in this case, blurred). Middle: recovered image (by undoing the alteration). Right: largest filter identified.

Tip: for pixel peeping, open the above image in a new tab in Firefox, then open the validation image in another tab. They will align perfectly, so you can now switch between them, back and forth.

Standard deviations of residuals:

resid-f1.png

Note: when filter size = 0, the filter becomes simply a scaling factor, so the largest value shows the mismatch between the two images (original vs filtered) after scaling them to look equally bright.

3x3 averaging blur:

f2 = @(x) imfilter(x, fspecial('average', 3));
  0.11111   0.11111   0.11111
  0.11111   0.11111   0.11111
  0.11111   0.11111   0.11111

images-f2.jpg

resid-f2.png

Sharpen:

f3 = @(x) imfilter(x, fspecial('unsharp'));
 -0.16667  -0.66667  -0.16667
 -0.66667   4.33333  -0.66667
 -0.16667  -0.66667  -0.16667

images-f3.jpg

resid-f3.png

Blur followed by sharpen:

f4 = @(x) imfilter(imfilter(x, fspecial('disk', 1)), fspecial('unsharp'));
 -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
 -0.0409430  -0.1381697   0.3357311  -0.1381697  -0.0409430
 -0.1052555   0.3357311   0.9750399   0.3357311  -0.1052555
 -0.0409430  -0.1381697   0.3357311  -0.1381697  -0.0409430
 -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798

images-f4.jpg

resid-f4.png

Laplacian of Gaussian (edge detector):

f5 = @(x) imfilter(x, fspecial('log'));

images-f5.jpg

resid-f5.png

Laplacian of Gaussian plus the original image:

f6 = @(x) imfilter(x, fspecial('log')) + x;

images-f6.jpg

resid-f6.png

4 Different filters on odd/even columns

Average odd/even columns (1 with 2, 3 with 4, similar to a YUV422 subsampling)

g1 = @(x) imresize((x(:,1:2:end) + x(:,2:2:end))/2, size(x), 'nearest');
 0 0.5 0.5 on columns 1:2:N
 0.5 0.5 0 on columns 2:2:N

Blur on odd columns, sharpen on even columns

function y = g2aux(x,f1,f2)
   y = x;
   y(:,1:2:end) = imfilter(x, f1(x))(:,1:2:end);
   y(:,2:2:end) = imfilter(x, f2(x))(:,2:2:end);
end
g2 = @(x) g2aux(x,f1,f2)

Green on odd columns, red on even columns, attempt to recover green (similar to the debayering problem)

function y = g3aux(g,r)
   y = g;
   y(:,2:2:end) = r(:,2:2:end);
end
g3 = @(g) g2aux(g,r)

5 Nonlinear filters

Median filter:

h1 = @(x) medfilt2(x, [3 3])

6 Added noise

Gaussian:

n1 = @(x) x + randn(size(x)) * 50;

Row noise:

n2 = @(x) x + ones(size(x,1),1) * randn(1,size(x,2)) * 10;