Difference between revisions of "Reversing digital filters"

From apertus wiki
Jump to: navigation, search
 
(41 intermediate revisions by the same user not shown)
Line 1: Line 1:
[WIP]
[WIP]


===Motivation===
=Motivation=


We are trying to record raw video with existing HDMI recorders (that don't know anything about recording raw).
We are trying to record raw video with existing HDMI recorders (that don't know anything about recording raw).
Line 7: Line 7:
Unfortunately, it seems that some of these recorders apply some processing on the image, like sharpening or blurring. Therefore, it may be a good idea to attempt to undo some of these filters applied to the image without our permission :P
Unfortunately, it seems that some of these recorders apply some processing on the image, like sharpening or blurring. Therefore, it may be a good idea to attempt to undo some of these filters applied to the image without our permission :P


Note: uncompressed versions of all of the images from this page can be found at http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/
''Note: uncompressed versions of all of the images from this page can be found at http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/''


Example:
Example:
Line 31: Line 31:
So... good luck recovering the raw image from this!
So... good luck recovering the raw image from this!


===Intro===
=Intro=


General idea: feed some test images to the HDMI, compare with the output image from the recorder, and attempt to undo the transformations in order to recover the original image.
General idea: feed some test images to the HDMI, compare with the output image from the recorder, and attempt to undo the transformations in order to recover the original image.
Line 37: Line 37:
We will start by experimenting with simple linear filters on grayscale images, as they are easiest to work with.
We will start by experimenting with simple linear filters on grayscale images, as they are easiest to work with.


=Grayscale images=
===Linear filters on grayscale images===
===Linear filters on grayscale images===


Line 54: Line 55:
Let's try some examples.
Let's try some examples.


We will use a training data set (a sample image used to compute the filter), and a validation data set (a different image, to check how well the filter does when the input data doesn't match). This is a simple strategy to avoid overfitting [http://en.wikipedia.org/wiki/Overfitting][http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set][http://www.kdnuggets.com/2015/01/clever-methods-overfitting-avoid.html]. Maybe not the best one [http://ai.stanford.edu/~ang/papers/cv-final.pdf], but for a quick experiment, it should do the trick.
We will use a training data set (a sample image used to compute the filter), and a validation data set (a different image, to check how well the filter does in other situations, not just on one particular test image). This is a simple strategy to avoid overfitting [http://en.wikipedia.org/wiki/Overfitting][http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set][http://www.kdnuggets.com/2015/01/clever-methods-overfitting-avoid.html]. Maybe not the best one [http://ai.stanford.edu/~ang/papers/cv-final.pdf], but for a quick experiment, it should do the trick.


Blur:
Training and validation images:
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/training.jpg http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/validation.jpg
 
=====Blur=====
  f1 = @(x) imfilter(x, fspecial('disk', 1));
  f1 = @(x) imfilter(x, fspecial('disk', 1));
   0.025079  0.145344  0.025079
   0.025079  0.145344  0.025079
Line 62: Line 67:
   0.025079  0.145344  0.025079
   0.025079  0.145344  0.025079


3x3 averaging blur:
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f1.jpg
 
Left: altered image (in this case, blurred). Middle: recovered image (by undoing the alteration). Right: largest filter identified.
 
Tip: for pixel peeping, open the above image in a new tab in Firefox, then open the validation image in another tab. They will align perfectly, so you can now switch between them, back and forth.
 
Standard deviations of residuals:
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f1.png
 
Note: when filter size = 0, the filter becomes simply a scaling factor, so the largest value shows the mismatch between the two images (original vs filtered) after scaling them to look equally bright.
 
=====3x3 averaging blur=====
  f2 = @(x) imfilter(x, fspecial('average', 3));
  f2 = @(x) imfilter(x, fspecial('average', 3));
   0.11111  0.11111  0.11111
   0.11111  0.11111  0.11111
Line 68: Line 85:
   0.11111  0.11111  0.11111
   0.11111  0.11111  0.11111


Sharpen:
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f2.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f2.png
 
=====Sharpen=====
  f3 = @(x) imfilter(x, fspecial('unsharp'));
  f3 = @(x) imfilter(x, fspecial('unsharp'));
   -0.16667  -0.66667  -0.16667
   -0.16667  -0.66667  -0.16667
Line 74: Line 95:
   -0.16667  -0.66667  -0.16667
   -0.16667  -0.66667  -0.16667


Blur followed by sharpen:
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f3.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f3.png
 
This one was reversed completely. Not bad!
 
So, can we really undo sharpening artifacts completely? Usually, those sharpening artifacts may result in values outside the usual black...white range (0-255 in this experiment), and in practice, these values get clipped. Let's add clipping to our filter:
 
f3c = @(x) max(min(imfilter(x, fspecial('unsharp')),255),0);
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f3c.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f3c.png
 
Not so good this time...
 
=====Blur followed by sharpen=====
  f4 = @(x) imfilter(imfilter(x, fspecial('disk', 1)), fspecial('unsharp'));
  f4 = @(x) imfilter(imfilter(x, fspecial('disk', 1)), fspecial('unsharp'));
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
Line 82: Line 119:
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
   -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798


Laplacian of Gaussian (edge detector):
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f4.jpg
  f5 = @(x) imfilter(x, fspecial('log'));
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f4.png
 
=====Laplacian=====
f5 = @(x) imfilter(x, fspecial('laplacian')) + 128;
  0.16667  0.66667  0.16667
  0.66667  -3.33333  0.66667
  0.16667  0.66667  0.16667
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f5.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f5.png
 
This one is a little harder :)
 
=====Laplacian plus the original image=====
  f6 = @(x) imfilter(x, fspecial('laplacian')) + x;
  0.16667  0.66667  0.16667
  0.66667  -2.33333  0.66667
  0.16667  0.66667  0.16667
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-f6.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-f6.png
 
So, yeah, this algorithm is not exactly magic.
 
===Nonlinear filters===


Laplacian of Gaussian plus the original image:
=====Median 3x3=====
  f6 = @(x) imfilter(x, fspecial('log')) + x;
h1 = @(x) medfilt2(x, [3 3]);
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-h1.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-h1.png
 
Median filter seems pretty hard to undo. Compare it to the 3x3 averaging filter (f2) from above.
 
=====Median 1x3=====
h2 = @(x) medfilt2(x, [1 3]);
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-h2.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-h2.png
 
===Added noise===
 
=====Gaussian=====
n1 = @(x) x + randn(size(x)) * 20;
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-n1.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-n1.png
 
Not exactly the best denoising filter, but it seems to do something :)
 
=====Column noise=====
  n2 = @(x) x + ones(size(x,1),1) * randn(1,size(x,2)) * 10;
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-n2.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-n2.png
 
This one seems a little better, but still doesn't beat --fixpn from raw2dng :)


===Different filters on odd/even columns===
===Different filters on odd/even columns===
Average odd/even columns (1 with 2, 3 with 4, similar to a YUV422 subsampling)  
 
=====Average odd/even columns=====
(1 and 2, 3 and 4, similar to a YUV422 subsampling)  
  g1 = @(x) imresize((x(:,1:2:end) + x(:,2:2:end))/2, size(x), 'nearest');
  g1 = @(x) imresize((x(:,1:2:end) + x(:,2:2:end))/2, size(x), 'nearest');
   0 0.5 0.5 on columns 1:2:N
   0 0.5 0.5 on columns 1:2:N
   0.5 0.5 0 on columns 2:2:N
   0.5 0.5 0 on columns 2:2:N


Blur on odd columns, sharpen on even columns
Trying to recover it with a simple linear filter:
  function y = g2aux(x,f1,f2)
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g1.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g1.png
 
Do we have better luck with two linear filters?
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g1h.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g1h.png
 
Um... nope. Figure out why.
 
=====Blur on odd columns, sharpen on even columns=====
  function y = g2aux(x,fa,fb)
     y = x;
     y = x;
     y(:,1:2:end) = imfilter(x, f1(x))(:,1:2:end);
     y(:,1:2:end) = fa(x)(:,1:2:end);
     y(:,2:2:end) = imfilter(x, f2(x))(:,2:2:end);
     y(:,2:2:end) = fb(x)(:,2:2:end);
  end
  end
  g2 = @(x) g2aux(x,f1,f2)
  g2 = @(x) g2aux(x,f1,f3);


Green on odd columns, red on even columns, attempt to recover green (similar to the debayering problem)
Assuming the image can be recovered with a simple linear filter gives this:
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g2.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g2.png
 
Any luck with two linear filters?
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g2h.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g2h.png
 
Looks like it worked this time!
 
=====Green on odd columns, red on even columns, attempt to recover green (similar to the debayering problem)=====
  function y = g3aux(g,r)
  function y = g3aux(g,r)
     y = g;
     y = g;
     y(:,2:2:end) = r(:,2:2:end);
     y(:,2:2:end) = r(:,2:2:end);
  end
  end
  g3 = @(g) g2aux(g,r)
  g3 = @(g) g3aux(g,r);
 
Recovery with one filter:
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g3.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g3.png
 
Recovery with two filters:
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/images-g3h.jpg
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/gray/resid-g3h.png
 
=RGB images=
 
WIP, just a sneak preview for now.
 
===HDMI recovery experiment===
 
HDMI image (crop from 1080p, made up by dumping raw channels to sRGB, with gamma 2: [R, 0.9*(G1+G2)/2, B]^2):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/hdmi.jpg
 
 
 
3.8K render from the corresponding raw12 image that was sent to HDMI (AMaZE, RawTherapee, then adjusted with gamma 2):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/ideal4k.jpg
 
 
 
HDMI frame, white-balanced and brightness-adjusted (simple scaling on R,G,B):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/hdmi1x-wbs.jpg
 
 
 
HDMI frame, resized 200% (convert -resize 200%):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/hdmi2x.jpg
 
 
 
3.8K image recovered by filtering the HDMI image (first frame):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/recovered4k.jpg
 
While it's far away from the ideal rendering from raw image, the result looks a little better than simply scaling to 200%, right?
 
With a recorder that doesn't compress like crazy, and by sending alternate green channels (G1 on even frames and G2 on odd frames), this method might have a chance to recover some more detail.
 
 
 
Recovered image downsized to 1080p (convert -resize 50%):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/recovered2xto1x.jpg
 
 
 
Ideal 1080p (convert -resize 50%):
 
http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/hdmireschart/ideal1x.jpg


===Nonlinear filters===


Median filter:
h1 = @(x) medfilt2(x, [3 3])


===Added noise===
Will it work on real-world images, or was it all overfitting? We'll have to try and see.


Gaussian:
=YCbCr images=
n1 = @(x) x + randn(size(x)) * 50;


Row noise:
(todo)
n2 = @(x) x + ones(size(x,1),1) * randn(1,size(x,2)) * 10;

Latest revision as of 21:52, 9 February 2016

[WIP]

1 Motivation

We are trying to record raw video with existing HDMI recorders (that don't know anything about recording raw).

Unfortunately, it seems that some of these recorders apply some processing on the image, like sharpening or blurring. Therefore, it may be a good idea to attempt to undo some of these filters applied to the image without our permission :P

Note: uncompressed versions of all of the images from this page can be found at http://files.apertus.org/AXIOM-Beta/snapshots/reversing-digital-filters/

Example:

Source image (R, G, B):

red-input-raw-encoded.jpg green-input-raw-encoded.jpg blue-input-raw-encoded.jpg

Recorded image (Atomos Shogun, ProRes 422):

red-hdmi-from-tif.jpg green-hdmi-from-tif.jpg blue-hdmi-from-tif.jpg

The compression looks pretty strong; according to simulation (compressing the source image with prores in ffmpeg), I would expect the recorded image to look like this:

red-422-prores-ffmpeg.jpg green-422-prores-ffmpeg.jpg blue-422-prores-ffmpeg.jpg

Color versions (source, image from shogun, prores ffmpeg profile 2):

rgb-input-raw-encoded.jpg rgb-hdmi-from-tif.jpg rgb-422-prores-ffmpeg.jpg

My guess: the HDMI recorder appears to sharpen the image before compressing, causing the ProRes codec to struggle.

So... good luck recovering the raw image from this!

2 Intro

General idea: feed some test images to the HDMI, compare with the output image from the recorder, and attempt to undo the transformations in order to recover the original image.

We will start by experimenting with simple linear filters on grayscale images, as they are easiest to work with.

3 Grayscale images

3.1 Linear filters on grayscale images

Input:

  • source image
  • altered image (with an unknown filter)

To find a digital linear filter that would undo the alteration on our test image, we may solve a linear system: each pixel can be expressed as a linear combination of its neighbouring pixels. For a MxN image and a PxP filter, we will have P*P unknowns and M*N equations.

To simplify things, we'll consider filters with odd diameters, so filter size would be PxP = (2*n+1) x (2*n+1).

If we assume our filter is horizontally and vertically symmetrical, the number of unknowns decreases to (n+1) * (n+1).

If we assume our filter is also diagonally symmetrical, the number of unknowns becomes n * (n+1) / 2.

Let's try some examples.

We will use a training data set (a sample image used to compute the filter), and a validation data set (a different image, to check how well the filter does in other situations, not just on one particular test image). This is a simple strategy to avoid overfitting [1][2][3]. Maybe not the best one [4], but for a quick experiment, it should do the trick.

Training and validation images:

training.jpg validation.jpg

3.1.1 Blur
f1 = @(x) imfilter(x, fspecial('disk', 1));
  0.025079   0.145344   0.025079
  0.145344   0.318310   0.145344
  0.025079   0.145344   0.025079

images-f1.jpg

Left: altered image (in this case, blurred). Middle: recovered image (by undoing the alteration). Right: largest filter identified.

Tip: for pixel peeping, open the above image in a new tab in Firefox, then open the validation image in another tab. They will align perfectly, so you can now switch between them, back and forth.

Standard deviations of residuals:

resid-f1.png

Note: when filter size = 0, the filter becomes simply a scaling factor, so the largest value shows the mismatch between the two images (original vs filtered) after scaling them to look equally bright.

3.1.2 3x3 averaging blur
f2 = @(x) imfilter(x, fspecial('average', 3));
  0.11111   0.11111   0.11111
  0.11111   0.11111   0.11111
  0.11111   0.11111   0.11111

images-f2.jpg

resid-f2.png

3.1.3 Sharpen
f3 = @(x) imfilter(x, fspecial('unsharp'));
 -0.16667  -0.66667  -0.16667
 -0.66667   4.33333  -0.66667
 -0.16667  -0.66667  -0.16667

images-f3.jpg

resid-f3.png

This one was reversed completely. Not bad!

So, can we really undo sharpening artifacts completely? Usually, those sharpening artifacts may result in values outside the usual black...white range (0-255 in this experiment), and in practice, these values get clipped. Let's add clipping to our filter:

f3c = @(x) max(min(imfilter(x, fspecial('unsharp')),255),0);

images-f3c.jpg

resid-f3c.png

Not so good this time...

3.1.4 Blur followed by sharpen
f4 = @(x) imfilter(imfilter(x, fspecial('disk', 1)), fspecial('unsharp'));
 -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798
 -0.0409430  -0.1381697   0.3357311  -0.1381697  -0.0409430
 -0.1052555   0.3357311   0.9750399   0.3357311  -0.1052555
 -0.0409430  -0.1381697   0.3357311  -0.1381697  -0.0409430
 -0.0041798  -0.0409430  -0.1052555  -0.0409430  -0.0041798

images-f4.jpg

resid-f4.png

3.1.5 Laplacian
f5 = @(x) imfilter(x, fspecial('laplacian')) + 128;
  0.16667   0.66667   0.16667
  0.66667  -3.33333   0.66667
  0.16667   0.66667   0.16667

images-f5.jpg

resid-f5.png

This one is a little harder :)

3.1.6 Laplacian plus the original image
f6 = @(x) imfilter(x, fspecial('laplacian')) + x;
  0.16667   0.66667   0.16667
  0.66667  -2.33333   0.66667
  0.16667   0.66667   0.16667

images-f6.jpg

resid-f6.png

So, yeah, this algorithm is not exactly magic.

3.2 Nonlinear filters

3.2.1 Median 3x3
h1 = @(x) medfilt2(x, [3 3]);

images-h1.jpg

resid-h1.png

Median filter seems pretty hard to undo. Compare it to the 3x3 averaging filter (f2) from above.

3.2.2 Median 1x3
h2 = @(x) medfilt2(x, [1 3]);

images-h2.jpg

resid-h2.png

3.3 Added noise

3.3.1 Gaussian
n1 = @(x) x + randn(size(x)) * 20;

images-n1.jpg

resid-n1.png

Not exactly the best denoising filter, but it seems to do something :)

3.3.2 Column noise
n2 = @(x) x + ones(size(x,1),1) * randn(1,size(x,2)) * 10;

images-n2.jpg

resid-n2.png

This one seems a little better, but still doesn't beat --fixpn from raw2dng :)

3.4 Different filters on odd/even columns

3.4.1 Average odd/even columns

(1 and 2, 3 and 4, similar to a YUV422 subsampling)

g1 = @(x) imresize((x(:,1:2:end) + x(:,2:2:end))/2, size(x), 'nearest');
 0 0.5 0.5 on columns 1:2:N
 0.5 0.5 0 on columns 2:2:N

Trying to recover it with a simple linear filter:

images-g1.jpg

resid-g1.png

Do we have better luck with two linear filters?

images-g1h.jpg

resid-g1h.png

Um... nope. Figure out why.

3.4.2 Blur on odd columns, sharpen on even columns
function y = g2aux(x,fa,fb)
   y = x;
   y(:,1:2:end) = fa(x)(:,1:2:end);
   y(:,2:2:end) = fb(x)(:,2:2:end);
end
g2 = @(x) g2aux(x,f1,f3);

Assuming the image can be recovered with a simple linear filter gives this: images-g2.jpg

resid-g2.png

Any luck with two linear filters?

images-g2h.jpg

resid-g2h.png

Looks like it worked this time!

3.4.3 Green on odd columns, red on even columns, attempt to recover green (similar to the debayering problem)
function y = g3aux(g,r)
   y = g;
   y(:,2:2:end) = r(:,2:2:end);
end
g3 = @(g) g3aux(g,r);

Recovery with one filter:

images-g3.jpg

resid-g3.png

Recovery with two filters:

images-g3h.jpg

resid-g3h.png

4 RGB images

WIP, just a sneak preview for now.

4.1 HDMI recovery experiment

HDMI image (crop from 1080p, made up by dumping raw channels to sRGB, with gamma 2: [R, 0.9*(G1+G2)/2, B]^2):

hdmi.jpg


3.8K render from the corresponding raw12 image that was sent to HDMI (AMaZE, RawTherapee, then adjusted with gamma 2):

ideal4k.jpg


HDMI frame, white-balanced and brightness-adjusted (simple scaling on R,G,B):

hdmi1x-wbs.jpg


HDMI frame, resized 200% (convert -resize 200%):

hdmi2x.jpg


3.8K image recovered by filtering the HDMI image (first frame):

recovered4k.jpg

While it's far away from the ideal rendering from raw image, the result looks a little better than simply scaling to 200%, right?

With a recorder that doesn't compress like crazy, and by sending alternate green channels (G1 on even frames and G2 on odd frames), this method might have a chance to recover some more detail.


Recovered image downsized to 1080p (convert -resize 50%):

recovered2xto1x.jpg


Ideal 1080p (convert -resize 50%):

ideal1x.jpg


Will it work on real-world images, or was it all overfitting? We'll have to try and see.

5 YCbCr images

(todo)