Pattern Noise

1 Intro

General info about pattern noise: https://web.archive.org/web/20210202034649/http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/ and http://isl.stanford.edu/~abbas/ee392b/lect07.pdf

The CMV12000 sensor suffers from dynamic row noise.

That means, a scalar offset gets added to each row. The offset is not correlated between different frames, so we can't remove it using a calibration frame (dark frame or whatever).

One can observe this noise by looking at the difference between two images taken at identical settings. There are two main components that appear obvious in such a difference frame: random noise (per pixel, increases on brighter pixels) and row noise (per line).

2 Correction methods

There are two ways to deal with this noise, after performing Black Calibration:

• use info from black reference columns to reduce dynamic row noise without guessing anything (fast, can be implemented in real-time, see Raw_preprocessing)
• use denoising techniques to reduce the remaining row noise (the guesswork part, slow)

2.1 Reducing row noise using black reference columns

http://github.com/apertus-open-source-cinema/misc-tools-utilities/commit/48de47b2a544dc32bbd5a8fd7701bb44a31ea850#diff-624053a553f49c0036b4d31282e58b2fR301

The application note AN01 from CMOSIS says:

"The noise is also present in the black reference columns (8 left and 8 right), so when enabled (reg 89[15] = 1), these can be used for row noise correction by for example making a relative row profile of these black columns and subtract this from the image."

However, simply subtracting each row average of the black columns from our image is not going to work. Here's why:

Kalman filter theory: http://robocup.mi.fu-berlin.de/buch/kalman.pdf

From page 3, if we know how noisy our estimations are, the optimal weights are inversely proportional with the noise variances:

x_optimal = (x1 * var(x2) + x2 * var(x1)) / (var(x1) + var(x2))

Here, let's say R = x1 is row noise (stdev = 1.6 at gain=x1) and x2 is black column noise.

 R = x1
 B = mean(black_col') = R + x2 =>  x2 = B - R
 x2 can be estimated as mean(black_col') - mean(active_area')
 stdev(x2) = 1.3.

We want to find k that minimizes var(R - k*B).

 var(R - k*B) = var(x1 * (1-k) - x2 * k),
 => k = var(x1)) / (var(x1) + var(x2).

In particular, for gain = x1, k = 1.6^2 / (1.6^2 + 1.3^2) = 0.6.

So, we don't have to simply subtract the black columns. Rather, we'll subtract the static offset (median value) first, and then, we'll subtract the remaining variations multiplied by 0.6 at gain=x1.

Things get a little more complex because the static offset is different on odd and even rows, and it also appears to change from the left side to right side of the frame. More details on the Raw preprocessing page.

2.2 Fixed frequency perturbation in black columns

A closer look at the frequency spectrum of the black columns, compared to the spectrum of the row noise from a dark frame, revealed a strong fixed-frequency component present only in the black columns. Attempting to fix row noise with the above procedure would introduce some of this fixed frequency component in the main image as well.

In the example image from below, this component has a frequency of 1/41.27 pixels^-1, with an amplitude of 1.14 DN. The value is different in other test images, and appears to be consistent in the images taken during the same experiment. It doesn't change with exposure time. Cause is unknown.

TODO: detailed analysis, FFT graphs...

We'll attempt to filter out this perturbation from the black columns before using them for reducing row noise.

Good news: after removing this perturbation, the optimal black columns multiplier increases to about 0.8 :)

2.3 Using nearby rows to reduce the row noise even more

Maybe some nearby rows could offer some hints about the row offset on the line being analyzed?

Looks like yes. Trying to write the row noise as a linear combination of row averages from black columns, using linear regression, not just the ones from the same row, but also from nearby rows, and discarding very small coefficients, gives the following filter:

 (y % 2)
    ?
      black_col[y-1] *  0.24 +
      black_col[y]   *  0.63
    :
      black_col[y]   *  0.38 +
      black_col[y+1] *  0.43

What's interesting: on even rows, row noise (in the active area) depends on the black column average from the same row, but also on the black column average from the previous row. But on odd rows, it depends on the black column average from the same row and the next row. This probably means they are processed in pairs, and there is some common perturbation that affects both of them.

This gives a minor improvement (0.6 -> 0.5).

2.4 Exploiting differences in green channels

The two green channels are expected to be nearly identical, except for high-frequency details like sharp edges. That means, if we take the median difference on each row, we expect to get zero. In practice, we get some nonzero values. We could try to see if there is any correlation between those green channel differences and our row noise.

Let's extract only the green channels from an image: the result will be W x H/2. Let's define green_delta(lag) = row_median(green - circshift(green,-lag)). That means, from each green row, subtract some nearby green row, and take the median difference. We'll compute this at lags -2, -1, 1, 2, mapped to array indices 0, 1, 2, 3.

Linear regression gives an ugly FIR filter that looks like this:

(y % 2)
    ?
      black_col[y-2]    * 0.17 +
      black_col[y-1]    * 0.14 +
      black_col[y+0]    * 0.17 +
      black_col[y+1]    * 0.16 +
      black_col[y+2]    * 0.14 +
      green_delta[0][y] * 0.22 +
      green_delta[1][y] * 0.31 +
      green_delta[2][y] * 0.38
    :
      black_col[y-2]    * 0.12 +
      black_col[y-1]    * 0.13 +
      black_col[y+0]    * 0.14 +
      black_col[y+1]    * 0.14 +
      black_col[y+2]    * 0.12 +
      black_col[y+3]    * 0.12 +
      green_delta[0][y] * 0.33 +
      green_delta[3][y] * 0.32

As weird as it looks, it seems to work!

Improvement (over the "Kalman" scaling of black columns): 0.6 -> 0.3.

2.5 Reducing the remaining row noise by image filtering

Basic algorithm:

1. Filter the image with an edge-aware vertical blur (bilateral filter on pixels from the same column)
2. Subtract the blurred image; the residuals will reveal the row noise (see example)
3. Mask out highlights and strong edges
4. Take the median value from each row of the residuals image
5. Subtract these values from each row of the original image

Source: patternnoise.c

3 Usage

The methods discussed here are implemented in raw2dng.

• black reference columns are used by default, as long as you use a dark frame (since this method is fast and has no side effects)
• to reduce the remaining row noise, use raw2dng --fixrn
• if the image also suffers from column noise, use raw2dng --fixpn

Troubleshooting or checking the effectiveness of each step:

• disable row noise reduction from black columns, but use the static offsets: --no-blackcol-rn
• disable fixed frequency correction for black columns: --no-blackcol-ff
• disable black reference columns completely: --no-blackcol (you need to compute new darkframes if you use this)

Tip: the algorithm for filtering row noise is also available in MLVFS, so you can use it on MLV videos (recorded with Magic Lantern) as well.

4 Tests on a dark frame

Checking a single 10ms dark frame, corrected using 256 exposures between 1.2ms and 77ms, various settings. Showing only noise measurements.

# raw data without any processing

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-darkframe --check-darkframe
   Average     : 190.35
   Pixel noise : 7.14
   Row noise   : 5.29 (74.1%)
   Col noise   : 5.41 (75.7%)

# disable black columns, simple darkframe

raw2dng *x1*.raw12 --calc-darkframe --no-blackcol --swap-lines
 raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe
   Average     : 140.33
   Pixel noise : 3.08
   Row noise   : 1.33 (43.0%)
   Col noise   : 0.09 (3.0%)

# disable black columns, enable dark current frame

raw2dng *x1*.raw12 --calc-dcnuframe --no-blackcol --swap-lines
 raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe
   Average     : 142.46
   Pixel noise : 3.06
   Row noise   : 1.32 (43.1%)
   Col noise   : 0.08 (2.5%)

# dark current frame, use only static offsets from black columns

raw2dng *x1*.raw12 --calc-dcnuframe --swap-lines
 raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-rn --check-darkframe
   Average     : 127.46
   Pixel noise : 3.06
   Row noise   : 1.39 (45.4%)
   Col noise   : 0.10 (3.2%)

# dark current frame, enable black columns, don't remove the fixed frequency component from black columns

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-ff --check-darkframe
   Average     : 127.46
   Pixel noise : 2.90
   Row noise   : 0.82 (28.2%)
   Col noise   : 0.10 (3.3%)

# dark current frame, enable black columns, remove the fixed frequency component (default setting)

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --check-darkframe
   Average     : 127.46
   Pixel noise : 2.85
   Row noise   : 0.61 (21.4%)
   Col noise   : 0.10 (3.4%)

# also enable --fixrn (aggressive correction)

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --fixrn --check-darkframe
   Average     : 127.42
   Pixel noise : 2.79
   Row noise   : 0.12 (4.1%)
   Col noise   : 0.10 (3.5%)

So, row noise was reduced from 1.33 (after dark frame correction) to 0.61 after using the black reference columns.

If it's still noticeable, --fixrn should bring it to very low levels, unless you have strong horizontal lines in the image, which might trick the algorithm.

Some experimental row noise filters:

% 2-tap FIR, separate for odd/even rows

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --rnfilter=1 --check-darkframe
   Average     : 127.44
   Pixel noise : 2.84
   Row noise   : 0.51 (18.1%)
   Col noise   : 0.10 (3.4%)

% that ugly FIR that also looks at green channel differences

raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --rnfilter=2 --check-darkframe
   Average     : 127.47
   Pixel noise : 2.81
   Row noise   : 0.31 (11.1%)
   Col noise   : 0.10 (3.4%)

5 Example

Showing half-res image crops pushed by 4 stops (ufraw-batch --wb=auto --exposure=4 --shrink=2).

• Left: raw sensor data (adjusted black level manually).
• Right: after dark frame and dark current subtraction, but without correction from black columns.
• Note: in the raw data, even and odd rows have different black offsets; that's why we have wrong colors.

---

• Left: after dark frame, dark current and static offsets from black reference columns
• Right: after dark frame, dark current and black reference columns correction, without removing the fixed frequency component

---

• Left: after dark frame, dark current and black reference columns correction.
• Right: after row noise reduction (--fixrn).

---

• Left: --rnfilter=1
• Right: --rnfilter=2

---

Algorithm internals for --fixrn:

• Left: filtered image (vertical blur using a bilateral filter)
• Right: noise image, revealing row noise. Black regions are from edges that were masked.

---

Downsized images:

Corrected with dark frame and dark current only:

Also corrected with black columns:

Also corrected with --fixrn:

Corrected with --rnfilter=2:

Larger images (half-res):

• 10ms+4-no-blackcol.jpg
• 10ms+4.jpg
• 10ms+4-fixrn.jpg
• 10ms+4-rnfilter2.jpg

---

All files used for this test, including scripts, calibration frames and uncompressed images, can be found here: http://files.apertus.org/AXIOM-Beta/snapshots/pattern-noise/

In particular, you might be interested in:

• Raw12 image: 10ms.raw12
• Calibration frames: darkframe-x1.pgm and dcnuframe-x1.pgm
• Final image (--rnfilter=2): 10ms.DNG
• Testing script: fpntest.sh
• Script output: fpntest.log