Pattern Noise

1 Intro

General info about pattern noise: http://theory.uchicago.edu/~ejm/pix/20d/tests/noise/#patternnoise

The CMV12000 sensor suffers from dynamic row noise.

That means, a scalar offset gets added to each row. The offset is not correlated between different frames, so we can't remove it using a calibration frame (dark frame or whatever).

One can observe this noise by looking at the difference between two images taken at identical settings. There are two main components that appear obvious in such a difference frame: random noise (per pixel, increases on brighter pixels) and row noise (per line).

2 Correction methods

There are two ways to deal with this noise, after performing Black Calibration:

• use info from black reference columns to reduce dynamic row noise without guessing anything (fast, can be implemented in real-time, see Raw_preprocessing)
• use denoising techniques to reduce the remaining row noise (the guesswork part, slow)

2.1 Reducing row noise using black reference columns

http://github.com/apertus-open-source-cinema/misc-tools-utilities/commit/48de47b2a544dc32bbd5a8fd7701bb44a31ea850#diff-624053a553f49c0036b4d31282e58b2fR301

The application note AN01 from CMOSIS says:

"The noise is also present in the black reference columns (8 left and 8 right), so when enabled (reg 89[15] = 1), these can be used for row noise correction by for example making a relative row profile of these black columns and subtract this from the image."

However, simply subtracting each row average of the black columns from our image is not going to work. Here's why:

Kalman filter theory: http://robocup.mi.fu-berlin.de/buch/kalman.pdf

From page 3, if we know how noisy our estimations are, the optimal weights are inversely proportional with the noise variances:

x_optimal = (x1 * var(x2) + x2 * var(x1)) / (var(x1) + var(x2))

Here, let's say R = x1 is row noise (stdev = 1.6 at gain=x1) and x2 is black column noise.

R = x1
B = mean(black_col') = R + x2 =>  x2 = B - R
x2 can be estimated as mean(black_col') - mean(active_area')
stdev(x2) = 1.3.

We want to find k that minimizes var(R - k*B).

var(R - k*B) = var(x1 * (1-k) - x2 * k),
=> k = var(x1)) / (var(x1) + var(x2).

In particular, for gain = x1, k = 1.6^2 / (1.6^2 + 1.3^2) = 0.6.

So, we don't have to simply subtract the black columns. Rather, we'll subtract the static offset (median value) first, and then, we'll subtract the remaining variations multiplied by 0.6 at gain=x1.

Things get a little more complex because the static offset is different on odd and even rows, and it also appears to change from the left side to right side of the frame. More details on the Raw preprocessing page.

2.2 Fixed frequency perturbation in black columns

A closer look at the frequency spectrum of the black columns, compared to the spectrum of the row noise from a dark frame, revealed a strong fixed-frequency component present only in the black columns. Attempting to fix row noise with the above procedure would introduce some of this fixed frequency component in the main image as well.

In the example image from below, this component has a frequency of 1/41.27 pixels^-1, with an amplitude of 1.14 DN. The value is different in other test images, and appears to be consistent in the images taken during the same experiment. It doesn't change with exposure time. Cause is unknown.

TODO: detailed analysis, FFT graphs...

We'll attempt to filter out this perturbation from the black columns before using them for reducing row noise.

2.3 Reducing the remaining row noise by image filtering

Basic algorithm:

1. Filter the image with an edge-aware vertical blur (bilateral filter on pixels from the same column)
2. Subtract the blurred image; the residuals will reveal the row noise (see example)
3. Mask out highlights and strong edges
4. Take the median value from each row of the residuals image
5. Subtract these values from each row of the original image

Source: patternnoise.c

3 Usage

The methods discussed here are implemented in raw2dng.

• black reference columns are used by default, as long as you use a dark frame (since this method is fast and has no side effects)
• to reduce the remaining row noise, use raw2dng --fixrn
• if the image also suffers from column noise, use raw2dng --fixpn

Troubleshooting or checking the effectiveness of each step:

• disable row noise reduction from black columns, but use the static offsets: --no-blackcol-rn
• disable fixed frequency correction for black columns: --no-blackcol-ff
• disable black reference columns completely: --no-blackcol (you need to compute new darkframes if you use this)

Tip: the algorithm for filtering row noise is also available in MLVFS, so you can use it on MLV videos (recorded with Magic Lantern) as well.

4 Tests on a dark frame

Checking a single 10ms dark frame, corrected using 256 exposures between 1.2ms and 77ms, various settings. Showing only noise measurements.

# raw data without any processing
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-darkframe --check-darkframe
  Average     : 190.35
  Pixel noise : 7.14
  Row noise   : 5.29 (74.1%)
  Col noise   : 5.41 (75.7%)

# disable black columns, simple darkframe
raw2dng *x1*.raw12 --calc-darkframe --no-blackcol --swap-lines
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe
  Average     : 140.33
  Pixel noise : 3.08
  Row noise   : 1.33 (43.0%)
  Col noise   : 0.09 (3.0%)

# disable black columns, enable dark current frame
raw2dng *x1*.raw12 --calc-dcnuframe --no-blackcol --swap-lines
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol --check-darkframe
  Average     : 142.46
  Pixel noise : 3.06
  Row noise   : 1.32 (43.1%)
  Col noise   : 0.08 (2.5%)

# dark current frame, use only static offsets from black columns
raw2dng *x1*.raw12 --calc-dcnuframe --swap-lines
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-rn --check-darkframe
  Average     : 127.46
  Pixel noise : 3.06
  Row noise   : 1.39 (45.4%)
  Col noise   : 0.10 (3.2%)

# dark current frame, enable black columns, don't remove the fixed frequency component from black columns
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --no-blackcol-ff --check-darkframe
  Average     : 127.46
  Pixel noise : 2.90
  Row noise   : 0.82 (28.2%)
  Col noise   : 0.10 (3.3%)

# dark current frame, enable black columns, remove the fixed frequency component (default setting)
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --check-darkframe
  Average     : 127.46
  Pixel noise : 2.85
  Row noise   : 0.61 (21.4%)
  Col noise   : 0.10 (3.4%)

# also enable --fixrn (aggressive correction)
raw2dng blackframes-gainx1-offset2047-10ms-01.raw12 --swap-lines --fixrn --check-darkframe
  Average     : 127.42
  Pixel noise : 2.79
  Row noise   : 0.12 (4.1%)
  Col noise   : 0.10 (3.5%)

So, row noise was reduced from 1.33 (after dark frame correction) to 0.61 after using the black reference columns.

If it's still noticeable, --fixrn should bring it to very low levels, unless you have strong horizontal lines in the image, which might trick the algorithm.

5 Example

Showing half-res image crops pushed by 4 stops (ufraw-batch --wb=auto --exposure=4 --shrink=2).

• Left: raw sensor data (adjusted black level manually).
• Right: after dark frame and dark current subtraction, but without correction from black columns.
• Note: in the raw data, even and odd rows have different black offsets; that's why we have wrong colors.

---

• Left: after dark frame, dark current and static offsets from black reference columns
• Right: after dark frame, dark current and black reference columns correction, without removing the fixed frequency component

---

• Left: after dark frame, dark current and black reference columns correction.
• Right: after row noise reduction (--fixrn).

---

Algorithm internals:

• Left: filtered image (vertical blur using a bilateral filter)
• Right: noise image, revealing row noise. Black regions are from edges that were masked.

---

Downsized images:

Corrected with dark frame and dark current only:

Also corrected with black columns:

Also corrected with --fixrn:

Larger images (half-res):

• 10ms+4-no-blackcol.jpg
• 10ms+4.jpg
• 10ms+4-fixrn.jpg

---

All files used for this test, including scripts, calibration frames and uncompressed images, can be found here: http://files.apertus.org/AXIOM-Beta/snapshots/pattern-noise/

In particular, you might be interested in:

• Raw12 image: 10ms.raw12
• Calibration frames: darkframe-x1.pgm and dcnuframe-x1.pgm
• Final image: 10ms.DNG
• Testing script: fpntest.sh
• Script output: fpntest.log