Difference between revisions of "PLR"

From apertus wiki
Jump to: navigation, search
 
(26 intermediate revisions by one other user not shown)
Line 1: Line 1:
Some attempts to reverse engineer the PLR high dynamic mode from CMV12000.
Some attempts to reverse engineer the PLR high dynamic range mode from CMV12000.


==Registers==
==Registers==
Line 13: Line 13:
I'll use an IT8 chart, exposed at 30 ms (normal exposure) and 100 ms (overexposed). A little dark in the lab today, but shouldn't be a big problem.
I'll use an IT8 chart, exposed at 30 ms (normal exposure) and 100 ms (overexposed). A little dark in the lab today, but shouldn't be a big problem.


To analyze the images, I'll use octave 4.0, compiled [http://marcelojoeng.blogspot.ro/2012/11/compile-octave-using-1632-bits-colour.html with 16-bit image support]. The scripts should run in Matlab as well, with minimal changes.
To analyze the images, I'll use octave 4.0, compiled [https://marcelojoeng.blogspot.com/2012/11/compile-octave-using-1632-bits-colour.html with 16-bit image support]. The scripts should run in Matlab as well, with minimal changes.


=====Linear exposures=====
=====Linear exposures=====


http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/30ms-lin.jpg http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-lin.jpg
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/30ms-lin.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-lin.jpg


Let's check if the first image is really exposed to the right, in octave.
Let's check if the first image is really exposed to the right, in octave.
Line 28: Line 28:
  figure, hold on
  figure, hold on
  colors = 'rgcb';                                  % meaning: red, gren1, green2 (plotted as cyan), blue (raw data from Bayer channels)
  colors = 'rgcb';                                  % meaning: red, gren1, green2 (plotted as cyan), blue (raw data from Bayer channels)
  [g30,c30]  = sample_it8('30ms-lin.DNG', 0);      % read median RGGB swatch values from IT8 chart (first arg is grayscale, second is full color)
  [g30,c30]  = sample_it8('30ms-lin.DNG', 0);      % read median RGGB swatch values from IT8 chart
  [g100,c100] = sample_it8('100ms-lin.DNG', 0);
  [g100,c100] = sample_it8('100ms-lin.DNG', 0);     % first output arg is grayscale, second is full color, each column is a Bayer channel
  for i = 1:4
  for i = 1:4
   plot(c30(:,i), c100(:,i), ['.' colors(i)]);    % plot the 100ms image vs the 30ms one (extremely rough guess for response curve)
   plot(c30(:,i), c100(:,i), ['.' colors(i)]);    % plot the 100ms image vs the 30ms one (extremely rough guess for response curve)
  end
  end


* http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-vs-30ms-lin.png
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-vs-30ms-lin.png
 
* this approximates (but it's not equal to!) the response curve without PLR
* this approximates (but it's not equal to!) the response curve without PLR
* this plot assumes the sensor response in the 30ms image (which was not overexposed) is linear (but it's probably not)
* this plot assumes the sensor response in the 30ms image (which was not overexposed) is linear (but it's probably not)
* the sensor doesn't clip very harshly to white (could be the PLR circuits kicking in with a very low exposure time? to be checked)
* the sensor doesn't clip very harshly to white (could be the PLR circuits kicking in with a very low exposure time? to be checked)


=====PLR exposures=====
=====2-segment PLR exposures=====


Let's start with a 2-segment PLR exposure, 100ms/10ms, vtfl2=32. That means, Number_slopes = 2, Exp_time = 8072, Exp_kp1 = 805, Vtfl=96.
Let's start with a 2-segment PLR exposure, 100ms/10ms, vtfl2=32. That means, Number_slopes = 2, Exp_time = 8072, Exp_kp1 = 805, Vtfl=96.


http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-10ms-32.jpg
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-10ms-32.jpg


Although the image looks overexposed at first sight, the raw levels seem to be alright:
Although the image may look a little overexposed at first sight (because of the magenta cast), the raw levels seem to be alright:
  c = read_raw('100ms-10ms-32.DNG');
  c = read_raw('100ms-10ms-32.DNG');
  prctile(c(:),99)
  prctile(c(:),99)
Line 58: Line 59:
  end
  end


* http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-10-32-plr-vs-30ms-lin.png
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-10-32-plr-vs-30ms-lin.png
 
* the knee point isn't as sharp as in the datasheet
* the knee point isn't as sharp as in the datasheet
* its location is around 400 (todo: find the relationship between this and vtfl)
* its location is around 400 (todo: find the relationship between this and vtfl)
* the second exposure appears a little noisy
* the second exposure segment appears a little noisy


Let's leave Vtfl constant and change exposure time for the second segment (1,5,10,20,50 ms).
======Changing Exp_kp1======


http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-32-tiny.jpg http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-5ms-32-tiny.jpg http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-10ms-32-tiny.jpg http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-20ms-32-tiny.jpg http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-50ms-32-tiny.jpg
Let's leave Vtfl constant and change exposure time for the second segment (0,1,5,10,20,35,50 ms).
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-0ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-5ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-10ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-20ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-35ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-50ms-32-tiny.jpg


Commands for plotting the "response curve" approximations are left as an exercise to the reader. Result:
Commands for plotting the "response curve" approximations are left as an exercise to the reader. Result:


http://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-x-32-plr-vs-30ms-lin.png
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-x-32-plr-vs-30ms-lin.png
 
======Changing Vtfl======
 
Now let's leave exposure time constant (100ms/1ms) and change Vtfl (off,0,8,16,24,32,40,48,56,63):
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-00-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-0-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-8-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-16-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-24-tiny.jpg
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-32-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-40-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-48-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-56-tiny.jpg https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100ms-1ms-63-tiny.jpg
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-1-x-plr-vs-30ms-lin.png
 
======Changing Exp_time======
 
Now let's leave Exp_kp1 and Vtfl constant (0ms/32) and change base exposure time (20,50,100,150 ms):
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/x-0-32-plr-vs-30ms-lin.png
 
* there is a noticeable light leak in the second segment, but it doesn't seem to change with base exposure time.
* most of the noise in these curves seems static (correctable with calibration frames)
 
======Changing Exp_kp2 and Vtfl3======
 
As expected, these settings do not seem to have any noticeable effect when num_slopes = 2.
 
====Mathematical models====
 
=====First models of the response curve for 2-segment PLR exposures=====
 
Let's go back to the scenario with constant Vtfl and variable exposure.
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-x-32-plr-vs-30ms-lin.png
 
 
Notice the knee point appears to move to the right with higher Exp_kp1 values. Why would this happen?
 
If we assume the knee point threshold (Vtfl2) is only monitored while we still have time to fully expose the second segment (Exp_kp1), we can imagine a model of the PLR exposure that looks like this:
function y = plr_model_0(x, exp_time, vtfl_thr, exp_kp1)
    % regular exposure time
    y = x .* exp_time;
   
    % time needed to reach the Vtfl threshold (lower input levels => longer times)
    t_reach_vtfl = vtfl_thr ./ x;
   
    % which pixels reach vtfl?
    % assume this threshold is only monitored while we still have time
    % to fully perform the exp_kp1 (that is, before exp_time - exp_kp1)
    reached_vtfl = t_reach_vtfl < exp_time - exp_kp1;
   
    % how the exposure behaves in the highlights
    y_highlight = vtfl_thr + x .* exp_kp1;
   
    % copy highlight values only for those pixels that reached Vtfl
    y(reached_vtfl) = y_highlight(reached_vtfl);
end
 
Let's see how it compares to our real data.
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-x-32-plr-vs-30ms-lin-model0.png
 
Getting closer.
 
Notice the slopes of the second segment seem to be higher in the real data, compared to our model, as if the real exposure time would be a little higher.
 
Since the slope at exp_kp1=0 doesn't depend on total exposure time, let's assume there is some sort of exposure leak - that is, the actual exposure time is exp_kp1 + some constant (called exp_leak).
 
function y = plr_model_1(x, exp_time, vtfl_thr, exp_kp1, exp_leak)
 
    [ ... ] 
 
    % how the exposure behaves in the highlights
    y_highlight = vtfl_thr + x .* (exp_kp1 + exp_leak);
 
    [ ... ] 
 
end
 
https://files.apertus.org/AXIOM-Beta/snapshots/PLRHDR/wiki/100-x-32-plr-vs-30ms-lin-model1.png
 
Tuning parameters (trial and error): vtfl_thr = 850, exp_leak = 2.5ms.
 
Black level 128 (after subtracting black reference columns, dark frame and dark current nonuniformity). Reference frames obtained from fitting 256 dark frames taken at exposures from 1 to 64 ms, in 1 ms increments, 4 images at each exposure, with raw2dng --swap-lines --calc-dcnuframe *x1*.raw12.
 
This model appears to explain the actual response curves pretty well, but there's still room for improvement.
 
-----
<blockquote>
'''At this point, I would try to find some real response curves and reduce the noise of the test images by averaging multiple frames, then repeat the experiment. This will take a while.''
</blockquote>
-----
 
After averaging 50 frames at each setting, the noise in the curve remained the same, so it must be systematic error (correctable with some sort of calibration frame).
 
Finding response curves from bracketed images is difficult - we can't just scale exposure times hoping we'll get the same curve. We will need a dimmable light source with no ambient light (or some dedicated calibration hardware).
 


[to be continued]
[to be continued]

Latest revision as of 16:01, 26 June 2018

Some attempts to reverse engineer the PLR high dynamic range mode from CMV12000.

1 Registers

  • 79: Number_slopes: 1,2,3.
  • 75-78: Exp_kp1, Exp_kp2: exposure times for highlights (same formula as Exp_time)
  • 106: Vtfl2, Vtfl3: knee point locations (range: 0-63; units: unknown)

Let's check the effect.

1.1 Register effects

I'll use an IT8 chart, exposed at 30 ms (normal exposure) and 100 ms (overexposed). A little dark in the lab today, but shouldn't be a big problem.

To analyze the images, I'll use octave 4.0, compiled with 16-bit image support. The scripts should run in Matlab as well, with minimal changes.

1.1.1 Linear exposures

30ms-lin.jpg 100ms-lin.jpg

Let's check if the first image is really exposed to the right, in octave.

a = read_raw('30ms-lin.DNG');
prctile(a(:),99) - 128        % note: black level is forced to 128 in raw2dng
ans =  2269                   % clipping starts at about 2400-2500 above black

Let's check if the clipping point is, indeed, where I say:

figure, hold on
colors = 'rgcb';                                  % meaning: red, gren1, green2 (plotted as cyan), blue (raw data from Bayer channels)
[g30,c30]   = sample_it8('30ms-lin.DNG', 0);      % read median RGGB swatch values from IT8 chart
[g100,c100] = sample_it8('100ms-lin.DNG', 0);     % first output arg is grayscale, second is full color, each column is a Bayer channel
for i = 1:4
  plot(c30(:,i), c100(:,i), ['.' colors(i)]);     % plot the 100ms image vs the 30ms one (extremely rough guess for response curve)
end

100ms-vs-30ms-lin.png

  • this approximates (but it's not equal to!) the response curve without PLR
  • this plot assumes the sensor response in the 30ms image (which was not overexposed) is linear (but it's probably not)
  • the sensor doesn't clip very harshly to white (could be the PLR circuits kicking in with a very low exposure time? to be checked)
1.1.2 2-segment PLR exposures

Let's start with a 2-segment PLR exposure, 100ms/10ms, vtfl2=32. That means, Number_slopes = 2, Exp_time = 8072, Exp_kp1 = 805, Vtfl=96.

100ms-10ms-32.jpg

Although the image may look a little overexposed at first sight (because of the magenta cast), the raw levels seem to be alright:

c = read_raw('100ms-10ms-32.DNG');
prctile(c(:),99)
ans =  1957

Let's check our "response curve" approximation:

[g100p,c100p] = sample_it8('100ms-10ms-32.DNG', 0);
figure, hold on
for i = 1:4
  plot(c30(:,i), c100p(:,i), ['.' colors(i)]);
end

100-10-32-plr-vs-30ms-lin.png

  • the knee point isn't as sharp as in the datasheet
  • its location is around 400 (todo: find the relationship between this and vtfl)
  • the second exposure segment appears a little noisy
1.1.2.1 Changing Exp_kp1

Let's leave Vtfl constant and change exposure time for the second segment (0,1,5,10,20,35,50 ms).

100ms-0ms-32-tiny.jpg 100ms-1ms-32-tiny.jpg 100ms-5ms-32-tiny.jpg 100ms-10ms-32-tiny.jpg 100ms-20ms-32-tiny.jpg 100ms-35ms-32-tiny.jpg 100ms-50ms-32-tiny.jpg

Commands for plotting the "response curve" approximations are left as an exercise to the reader. Result:

100-x-32-plr-vs-30ms-lin.png

1.1.2.2 Changing Vtfl

Now let's leave exposure time constant (100ms/1ms) and change Vtfl (off,0,8,16,24,32,40,48,56,63):

100ms-1ms-00-tiny.jpg 100ms-1ms-0-tiny.jpg 100ms-1ms-8-tiny.jpg 100ms-1ms-16-tiny.jpg 100ms-1ms-24-tiny.jpg

100ms-1ms-32-tiny.jpg 100ms-1ms-40-tiny.jpg 100ms-1ms-48-tiny.jpg 100ms-1ms-56-tiny.jpg 100ms-1ms-63-tiny.jpg

100-1-x-plr-vs-30ms-lin.png

1.1.2.3 Changing Exp_time

Now let's leave Exp_kp1 and Vtfl constant (0ms/32) and change base exposure time (20,50,100,150 ms):

x-0-32-plr-vs-30ms-lin.png

  • there is a noticeable light leak in the second segment, but it doesn't seem to change with base exposure time.
  • most of the noise in these curves seems static (correctable with calibration frames)
1.1.2.4 Changing Exp_kp2 and Vtfl3

As expected, these settings do not seem to have any noticeable effect when num_slopes = 2.

1.2 Mathematical models

1.2.1 First models of the response curve for 2-segment PLR exposures

Let's go back to the scenario with constant Vtfl and variable exposure.

100-x-32-plr-vs-30ms-lin.png


Notice the knee point appears to move to the right with higher Exp_kp1 values. Why would this happen?

If we assume the knee point threshold (Vtfl2) is only monitored while we still have time to fully expose the second segment (Exp_kp1), we can imagine a model of the PLR exposure that looks like this:

function y = plr_model_0(x, exp_time, vtfl_thr, exp_kp1)
   % regular exposure time
   y = x .* exp_time;
   
   % time needed to reach the Vtfl threshold (lower input levels => longer times)
   t_reach_vtfl = vtfl_thr ./ x;
   
   % which pixels reach vtfl?
   % assume this threshold is only monitored while we still have time
   % to fully perform the exp_kp1 (that is, before exp_time - exp_kp1)
   reached_vtfl = t_reach_vtfl < exp_time - exp_kp1;
   
   % how the exposure behaves in the highlights
   y_highlight = vtfl_thr + x .* exp_kp1;
   
   % copy highlight values only for those pixels that reached Vtfl
   y(reached_vtfl) = y_highlight(reached_vtfl);
end

Let's see how it compares to our real data.

100-x-32-plr-vs-30ms-lin-model0.png

Getting closer.

Notice the slopes of the second segment seem to be higher in the real data, compared to our model, as if the real exposure time would be a little higher.

Since the slope at exp_kp1=0 doesn't depend on total exposure time, let's assume there is some sort of exposure leak - that is, the actual exposure time is exp_kp1 + some constant (called exp_leak).

function y = plr_model_1(x, exp_time, vtfl_thr, exp_kp1, exp_leak)
  
   [ ... ]  
  
   % how the exposure behaves in the highlights
   y_highlight = vtfl_thr + x .* (exp_kp1 + exp_leak);
  
   [ ... ]  
  
end

100-x-32-plr-vs-30ms-lin-model1.png

Tuning parameters (trial and error): vtfl_thr = 850, exp_leak = 2.5ms.

Black level 128 (after subtracting black reference columns, dark frame and dark current nonuniformity). Reference frames obtained from fitting 256 dark frames taken at exposures from 1 to 64 ms, in 1 ms increments, 4 images at each exposure, with raw2dng --swap-lines --calc-dcnuframe *x1*.raw12.

This model appears to explain the actual response curves pretty well, but there's still room for improvement.


'At this point, I would try to find some real response curves and reduce the noise of the test images by averaging multiple frames, then repeat the experiment. This will take a while.


After averaging 50 frames at each setting, the noise in the curve remained the same, so it must be systematic error (correctable with some sort of calibration frame).

Finding response curves from bracketed images is difficult - we can't just scale exposure times hoping we'll get the same curve. We will need a dimmable light source with no ambient light (or some dedicated calibration hardware).


[to be continued]