General info about pattern noise: https://web.archive.org/web/20210202034649/http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/ and http://isl.stanford.edu/~abbas/ee392b/lect07.pdf
The CMV12000 sensor suffers from dynamic row noise.
That means, a scalar offset gets added to each row. The offset is not correlated between different frames, so we can't remove it using a calibration frame (dark frame or whatever).
One can observe this noise by looking at the difference between two images taken at identical settings. There are two main components that appear obvious in such a difference frame: random noise (per pixel, increases on brighter pixels) and row noise (per line).
There are two ways to deal with this noise, after performing Black Calibration:
The application note AN01 from CMOSIS says:
"The noise is also present in the black reference columns (8 left and 8 right), so when enabled (reg 89 = 1), these can be used for row noise correction by for example making a relative row profile of these black columns and subtract this from the image."
However, simply subtracting each row average of the black columns from our image is not going to work. Here's why:
Kalman filter theory: http://robocup.mi.fu-berlin.de/buch/kalman.pdf
From page 3, if we know how noisy our estimations are, the optimal weights are inversely proportional with the noise variances:
x_optimal = (x1 * var(x2) + x2 * var(x1)) / (var(x1) + var(x2))
Here, let's say R = x1 is row noise (stdev = 1.6 at gain=x1) and x2 is black column noise.
R = x1 B = mean(black_col') = R + x2 => x2 = B - R x2 can be estimated as mean(black_col') - mean(active_area') stdev(x2) = 1.3.
We want to find k that minimizes var(R - k*B).
var(R - k*B) = var(x1 * (1-k) - x2 * k), => k = var(x1)) / (var(x1) + var(x2).
In particular, for gain = x1, k = 1.6^2 / (1.6^2 + 1.3^2) = 0.6.
So, we don't have to simply subtract the black columns. Rather, we'll subtract the static offset (median value) first, and then, we'll subtract the remaining variations multiplied by 0.6 at gain=x1.
Things get a little more complex because the static offset is different on odd and even rows, and it also appears to change from the left side to right side of the frame. More details on the Raw preprocessing page.
A closer look at the frequency spectrum of the black columns, compared to the spectrum of the row noise from a dark frame, revealed a strong fixed-frequency component present only in the black columns. Attempting to fix row noise with the above procedure would introduce some of this fixed frequency component in the main image as well.
In the example image from below, this component has a frequency of 1/41.27 pixels-1, with an amplitude of 1.14 DN. The value is different in other test images, and appears to be consistent in the images taken during the same experiment. It doesn't change with exposure time. Cause is unknown.
TODO: detailed analysis, FFT graphs...
We'll attempt to filter out this perturbation from the black columns before using them for reducing row noise.
Good news: after removing this perturbation, the optimal black columns multiplier increases to about 0.8 :)
Maybe some nearby rows could offer some hints about the row offset on the line being analyzed?
Looks like yes. Trying to write the row noise as a linear combination of row averages from black columns, using linear regression, not just the ones from the same row, but also from nearby rows, and discarding very small coefficients, gives the following filter:
(y % 2) ? black_col[y-1] * 0.24 + black_col[y] * 0.63 : black_col[y] * 0.38 + black_col[y+1] * 0.43
What's interesting: on even rows, row noise (in the active area) depends on the black column average from the same row, but also on the black column average from the previous row. But on odd rows, it depends on the black column average from the same row and the next row. This probably means they are processed in pairs, and there is some common perturbation that affects both of them.
This gives a minor improvement (0.6 -> 0.5).
The two green channels are expected to be nearly identical, except for high-frequency details like sharp edges. That means, if we take the median difference on each row, we expect to get zero. In practice, we get some nonzero values. We could try to see if there is any correlation between those green channel differences and our row noise.
Let's extract only the green channels from an image: the result will be W x H/2. Let's define green_delta(lag) = row_median(green - circshift(green,-lag)). That means, from each green row, subtract some nearby green row, and take the median difference. We'll compute this at lags -2, -1, 1, 2, mapped to array indices 0, 1, 2, 3.
Linear regression gives an ugly FIR filter that looks like this:
(y % 2) ? black_col[y-2] * 0.17 + black_col[y-1] * 0.14 + black_col[y+0] * 0.17 + black_col[y+1] * 0.16 + black_col[y+2] * 0.14 + green_delta[y] * 0.22 + green_delta[y] * 0.31 + green_delta[y] * 0.38 : black_col[y-2] * 0.12 + black_col[y-1] * 0.13 + black_col[y+0] * 0.14 + black_col[y+1] * 0.14 + black_col[y+2] * 0.12 + black_col[y+3] * 0.12 + green_delta[y] * 0.33 + green_delta[y] * 0.32
As weird as it looks, it seems to work!
Improvement (over the "Kalman" scaling of black columns): 0.6 -> 0.3.
The methods discussed here are implemented in raw2dng.
Troubleshooting or checking the effectiveness of each step:
Tip: the algorithm for filtering row noise is also available in MLVFS, so you can use it on MLV videos (recorded with Magic Lantern) as well.
Checking a single 10ms dark frame, corrected using 256 exposures between 1.2ms and 77ms, various settings. Showing only noise measurements.
# raw data without any processing
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-darkframe --check-darkframe Average : 190.35 Pixel noise : 7.14 Row noise : 5.29 (74.1%) Col noise : 5.41 (75.7%)
# disable black columns, simple darkframe
raw2dng *x1*.raw12 --calc-darkframe --no-blackcol --swap-lines raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe Average : 140.33 Pixel noise : 3.08 Row noise : 1.33 (43.0%) Col noise : 0.09 (3.0%)
# disable black columns, enable dark current frame
raw2dng *x1*.raw12 --calc-dcnuframe --no-blackcol --swap-lines raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe Average : 142.46 Pixel noise : 3.06 Row noise : 1.32 (43.1%) Col noise : 0.08 (2.5%)
# dark current frame, use only static offsets from black columns
raw2dng *x1*.raw12 --calc-dcnuframe --swap-lines raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-rn --check-darkframe Average : 127.46 Pixel noise : 3.06 Row noise : 1.39 (45.4%) Col noise : 0.10 (3.2%)
# dark current frame, enable black columns, don't remove the fixed frequency component from black columns
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-ff --check-darkframe Average : 127.46 Pixel noise : 2.90 Row noise : 0.82 (28.2%) Col noise : 0.10 (3.3%)
# dark current frame, enable black columns, remove the fixed frequency component (default setting)
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --check-darkframe Average : 127.46 Pixel noise : 2.85 Row noise : 0.61 (21.4%) Col noise : 0.10 (3.4%)
# also enable --fixrn (aggressive correction)
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --fixrn --check-darkframe Average : 127.42 Pixel noise : 2.79 Row noise : 0.12 (4.1%) Col noise : 0.10 (3.5%)
So, row noise was reduced from 1.33 (after dark frame correction) to 0.61 after using the black reference columns.
If it's still noticeable, --fixrn should bring it to very low levels, unless you have strong horizontal lines in the image, which might trick the algorithm.
Some experimental row noise filters:
% 2-tap FIR, separate for odd/even rows
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --rnfilter=1 --check-darkframe Average : 127.44 Pixel noise : 2.84 Row noise : 0.51 (18.1%) Col noise : 0.10 (3.4%)
% that ugly FIR that also looks at green channel differences
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --rnfilter=2 --check-darkframe Average : 127.47 Pixel noise : 2.81 Row noise : 0.31 (11.1%) Col noise : 0.10 (3.4%)
Showing half-res image crops pushed by 4 stops (ufraw-batch --wb=auto --exposure=4 --shrink=2).
Algorithm internals for --fixrn:
Corrected with dark frame and dark current only:
Also corrected with black columns:
Also corrected with --fixrn:
Corrected with --rnfilter=2:
Larger images (half-res):
All files used for this test, including scripts, calibration frames and uncompressed images, can be found here: http://files.apertus.org/AXIOM-Beta/snapshots/pattern-noise/
In particular, you might be interested in: