AXIOM Alpha Software

1 Image Pipeline

Image Acquisition Pipeline:

HDMI Image Processing Pipeline:

2 Register Memory Space

CMV12000 Register Blocks

CMV12000 to HDMI registers

Image sensor related registers: 0x60xxxxxx

cmv.func contains the functions to deal with this address space

HDMI related registers: 0x80xxxxxx

hdmi.func contains the functions to deal with this address space

0xU0VWWWWW
U = image sensor or HDMI
V = register block
WWWWW =  addresses inside a register block

Register Addresses HDMI:

0x800xxxxx = scan generator
0x801xxxxx = address generator
0x802xxxxx = component matrix 4x4
0x803xxxxx = 4 channel 12bit LUT
0x804xxxxx = HDMI PLL dynamic reconfig

Register Addresses Image Sensor:

0x600xxxxx = CMV SPI registers
0x601xxxxx = address generator/capture control
0x602xxxxx = LVDS input delay registers
0x603xxxxx = column/row noise LUTs

Examples:

0x60000000 = sensor, register block 0, address 0 (CMV register 0)

3 Reading and Writing Sensor Register

Execute the script that contains the sensor interface functions before we can use them:

. ./cmv.func

Set Sensor register 82 to the decimal value "3122"

cmv_reg  82   3122

Read Register 82:

 cmv_reg  82

will output:

0x00000C31

4 Enable/Disable LEDs

Disable LEDs on the back:

fil_reg 15 0x01FF01FF

disable LEDs on the front:

fil_reg 14 0xFFFF0000

Enable all LEDs again:

Front:

fil_reg 14 0x0

Back:

fil_reg 15 0x0

5 Row/Coloumn Noise Correction

To run this calibration an evenly lit image is essential. Be sure to use a lens with little vignetting effects and set the aperture to a range that suffers from least vignetting effects. Diffusion gel and turning focus to infinite can help smooth out light differences.

It is advised to disable automatic image acquisition before running the RCN profiler:

./cmv_rcn3 -zsacr -N 5

Pixels going out of range will show up as blue dots. To prevent this enable clipping by setting register 11 bits: 21:20 to 1:

fil_reg 11 0xFC31F000

Save FPN profile into a file (all 4 channels):

pmem -B 0x60000000 -m R0x60300000+0x8000/W > /tmp/rcn.save

Load FPN profile from a file (all 4 channels):

pmem -B 0x60000000 -m W0x60300000+0x8000/W </tmp/rcn.save

Clear FPN profile:

./cmv_rcn3 -z

If only a particular channel should be saved/loaded the addresses and lengths are different, eg:

pmem -B 0x60000000 -m R0x60304000+0x2000/W > /tmp/rcn_lut1.save

With the real time histogram you can evaluate how evenly lit your image is and can also judge the calibration results. The spikes in the histogram should for optimal results be in the 33% gray range and all line up in one place. With a good RCN profile they should get higher and less wide in the slope.

6 statically linked busybox

http://vserver.13thfloor.at/Stuff/AXIOM/FAKE/

builtin fake devmem

all you need to get it to work is the following:

dd if=/dev/zero of=/tmp/mem bs=1k seek=4M count=1

this will create a sparse 4GB file /tmp/mem, which will be used by the fake devmem values written can be read back, non existing values return 0

/bin/sh and /sbin/devmem both link to busybox on the axiom alpha filesystem so both can be tested with this executeable

7 Capturing an Image

Preparations:

ssh root@*alpha-IP* "./cmv_train3 -a"

Capture the image:

ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-

So we dont need to type the password every time we snap an image:

Install:

sudo apt-get install sshpass

Use:

sshpass -p '1234' ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-

8 cmv_perf3

displays image stats and performance data, the longer it runs the more accurate the stats will be.

./cmv_perf3 -h
This is ./cmv_perf3 V1.0
options are:
-h        print this help message
-f        measure frame rates
-m        measure memory bandwidth
-D <val>  delay between updates
-T <val>  run for a given time

Will get more accurate with ntpd running

ntpd -p 86.59.80.170 -p 86.59.118.153

Typical output:

 ./cmv_perf3
mapped 0x60000000+0x00400000 to 0x60000000.
mapped 0x80000000+0x00400000 to 0x80000000.
hdmi: 29.97 FPS  cseq: 29.97 FPS

9 cmv_snap3

capture one still image and display it over HDMI immediately. If the -z parameter is not used this tool will provide the image data via stdout.

./cmv_snap3 -h
This is ./cmv_snap3 V1.9
options are:
-h        print this help message
-8        output 8 bit per pixel
-2        output 12 bit per pixel
-b        enable black columns
-r        dump sensor registers
-t        enable cmv test pattern
-z        produce no data output
-e <exp>  exposure times
-v <exp>  exposure voltages
-s <num>  shift values by <num>
-S <val>  writer byte strobe
-R <fil>  load sensor registers

./cmv_snap3 -r # without an -e parameter outputs only sensor registers
./cmv_snap3 -e 20ms > image.raw16 # capture an image and write the data into a file

Can also be used for direct capture to remote host:

ssh root@*cameraip* "./cmv_snap3 -e 20ms" > image.raw16 # capture an image and write the data into a file

10 cmv_hist3

Outputs RAW histogram data of the last captured image. Values are tab separated in 4 columns (Order: GRBG[image flip on], RGGB[image flip off]) with 4096 values (when operating in 12 bit mode) each.

Download: http://vserver.13thfloor.at/Stuff/AXIOM/cmv_hdmi2/cmv_hist3

./cmv_hist2
This is ./cmv_hist2 V1.0
options are:
-h        print this help message
-s        acquire snapshot
-B <val>  register mapping base
-S <val>  register mapping size
-M <val>  buffer memory base
-Z <val>  buffer memory size

Real Time Histogram Plot

10.1 Examples

./cmv_hist2 -s

Acquire new snapshot and output histogram

./cmv_hist2

Acquire histogram from last captured snapshot in memory

ssh root@192.168.2.88 "./cmv_hist3 -s" > capt.hist && gnuplot hist.gplot

11 cmv_train3

Executes the LVDS training routines to align LVDS reading timing. Execute this script when the Axiom Alpha prototype is booted up before you capture any images:

./cmv_train3 -a

 ./cmv_train3 -h
This is ./cmv_train3 V1.3
options are:
-h        print this help message
-a        test all bit pattern
-B <val>  memory mapping base
-S <val>  memory mapping size
-A <val>  memory mapping address
-P <val>  training pattern

12 Linearization

This happens as very first step in the image pipeline, before the Row/Column Noise compensation.

./linear_conf.sh *factor* *offset*

Examples:

./linear_conf.sh 1.2 -0.1 # multiple by factor 1.2 and shift offset 10% into the darker area

13 Post Processing images

13.1 Create RGGB separated color images from raw file:

convert -size 4096x3072 -depth 16 -crop +0+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch0.raw
convert -size 4096x3072 -depth 16 -crop -1+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch1.raw
convert -size 4096x3072 -depth 16 -crop +0-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch2.raw
convert -size 4096x3072 -depth 16 -crop -1-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch3.raw

13.2 Simple debayer with ImageMagick:

For flipped images:

convert \( -size 4096x3072 -depth 16 gray:colors_500ms.raw16 \) \
\( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \
-sample 2048x1536 \( -clone 2,3 -average \) -delete 2,3 -swap 0,1 +swap -combine colors_500ms.png

For unflipped images:

convert \( -size 4096x3072 -depth 16 gray:IT8_incand.raw16 \) \
\( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \
-sample 2048x1536 \( -clone 0,1 -average \) -delete 0,1 +swap -combine IT8_incand.png

13.3 Plot Histogram of Raw Image

Create histogram values file with imagemagick:

convert -size 4096x3072 -depth 16 gray:image.raw16 -format "%c" histogram:info: > histogram.hist

Reformat the file:

gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }' 

Or all in one command:

convert -size 4096x3072 -depth 16 gray:image.raw16 -format %c histogram:info:- | gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }' > histogram.info

Then draw it with this gnuplot script:

set notitle
set term svg size 1024, 512
set xlabel "Values"
set ylabel "Number of Pixels"
set multiplot
set obj 1 rectangle behind from screen 0,0 to screen 1,1
set obj 1 fillstyle solid 1.0 fillcolor rgbcolor "white"
set xrange[0:4096]
set yrange[0:8000]
plot 	'histogram.info' using 1:2	with lines lc rgb "#FF0000"	title "name your curve", \
       'histogram2.info' using 1:2	with lines lc rgb "#FF0000"	title "name your curve"
unset multiplot

execute the script with:

gnuplot thescript > output.svg

14 chroot environment on Axiom Alpha

Full raspian image is available on the alpha prototype in a changeroot environment to keep the core system free from too much clutter.

Download:

http://vserver.13thfloor.at/Stuff/AXIOM/raspian.tar.xz

Preferably download from the alpha prototype directly and upack into /opt/raspian, moving files to the prototype using Filezilla has produced some problems.

Map required folders:

mount -o bind /proc /opt/raspian/proc
mount -o bind /tmp /opt/raspian/tmp

Changeroot:

chroot /opt/raspian/ /bin/bash

15 Live HDMI video output

Init:

cd root
./cmv_train3 -a
./setup.sh

Start Video Acquisition:

. ./cmv.func
fil_reg 15 0x01000100

Stop Video Acquisition:

fil_reg 15 0x0

Half HDMI Frequency (1080p60 -> 1080p30 or 1080p50 -> 1080p25)

pll_reg 22 0x2106

Full HDMI Frequency (1080p60)

pll_reg 22 0x2083

16 Update Firmware

u-boot access:

mount /dev/mmcblk0p1 /mnt/

17 Real Time FPN correction

FPN correction supports a 9bit signed value for each column and row.

9 bit signed integer means:

0xFF = 255 (highest possible positive value)
0x1FF = -1
0x100 = -256 (lowest possible negative value

To obtain the correct value (for -7 in this case) to set you can run:

echo $(( -7 & 0x1FF ))

which outputs (in decimal)

505

To set a value use syntax:

devmem address 32 value

or with actual values:

devmem 0x60304000 32 0xFF

Adresses are:

0x60300000 = even columns, 2048 total
0x60304000 = odd columns, 2048 total
0x60308000 = even rows, 2048 total (only first 1576 are used)
0x6030C000 = odd rows, 2048 total (only first 1576 are used)

each row/column address increases by 4 from current to next row/column.

Row/Column 0 is in the top left corner.

17.1 Balance (FPN profiling tool)

Download: http://vserver.13thfloor.at/Stuff/AXIOM/balance

balance can extract a DSNU profile from a darkframes and apply the created profile to correct an image.

Usage:

Extract FPN profile:

./balance -b -o darkframe.raw16 > profile.offsets

Apply FPN profile to image:

./balance -b -O profile.offsets image.raw16 > image_balanced.raw16

Only use the -b parameter if the image uses darkcolumns.

The profile.offsets file format is:

row-numer offset (double) then column number (double)

(3072+4096)x8 = 57344 Bytes = 57KB

One double is 8 bytes (64 bits)

17.2 In-camera FPN profiling

Point the camera at a totally evenly lit non-clipping same color surface like a white wall (out of focus helps to even out minor details) and run:

./cmv_rcn3 -azc

To help visualize how evenly lit a surface is this LUT can help:

./lut_conf.sh -M 0x100000 -N 4096

18 Matrix Color Conversion

Each color channel is going through 4 stages that each look like this: (I+A)*E+O

I = input (25bit unsigned)
A = adjustment (30bit signed)
E = matrix coefficient (18bit signed)
O = offset (48bit signed)

The four stages are connected together that the offset of stage N is the output of stage N-1. The offset of stage 1 is set in registers (32-35). The output of stage 4 is the final result.

For each channel that results in the following fomular:

Ri = SUMj ((Ij + Aij) * Eij) + Oi

Which in other words mean the result of matrix (Eij) multiplied with an input vector (Ij + Aij) plus an offset vector (Oi).

Currently all registers (beside input/output) are shortened to 16bit signed (this could be changed easily) values so the matrix is currently limited to 1/256 of the possible accuracy (due to a 8bit shift of the output).

All input /output register are tested for over/underflow and clipped to 12 bit (this can also be disabled).

Setting/Getting Registers:

. ./hdmi.func # load functions

mat_reg 0 # Read value of register 0 = A0
mat_reg 0 1 # Sett value of register 0 (A0) to 1

4x4 matrix coefficients (A0, A1, A2, A3, B0, B1, B2, B3, B4, C0, C1, C2, C3, D0, D1, D2, D3) are mapped to addresses 0 - 15. In registers 16 - 31 there are adjustment values and from 32 - 35 there are 4 offset values.

Input format per default is (R, G1, G2, B).

All values of the 4x4 matrix can be set conveniently with this script:

Syntax:

./mat4_conf.sh  A0 A1 A2 A3  B0 B1 B2 B3 B4  C0 C1 C2 C3  D0 D1 D2 D3    O1 O2 O3 O4

Examples:

./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1    0 0 0 0   # unity matrix but not optimal as both green channels are processed separately
./mat4_conf.sh  1 0 0 0  0 0.5 0.5 0  0 0.5 0.5 0  0 0 0 1    0 0 0 0   # the two green channels inside each 2x2 pixel block are averaged and output on both green pixels
./mat4_conf.sh  0 0 0 1  0 1 0 0  0 0 1 0  1 0 0 0    0 0 0 0   # red and blue are swapped
./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1    0.5 0 0 0    # red 50% brigther
./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1.5    0 0 0 0    # blue multiplied with factor 1.5
./mat4_conf.sh  .25 .25 .25 .25  .25 .25 .25 .25  .25 .25 .25 .25  .25 .25 .25 .25    0 0 0 0    # black/white
./mat4_conf.sh  -1 0 0 0  0 -0.5 -0.5 0  0 -0.5 -0.5 0  0 0 0 -1    1 1 1 1    # negative

An RGB 3x3 matrix can easily be extended to a 4x4 matrix by using the coefficient for green and offsets twice or better to half both of the green coefficients and use the same value for both greens twice.

19 LUTs

The result from the matrix conversation goes straight into 4x12 bit Look Up Tables (LUT). The output is 16bit wide.

These 4096x16bit values can be set at the register address space: 0x8030xxxx [0x80300000 (ch0 - R), 0x80304000 (ch1 - G1), 0x80308000 (ch2 - G2), 0x8030C000 (ch3 - B)]

For creating classic gamma curves for the LUTs there is a tool called: lut_conf3

Examples:

./lut_conf3 -N 4096 -M 65535 -G 0.6 -B 0x80300000 # create LUT of gamma curve for gamma value 0.6 and set it for red channel (ch0)

-N = number of entries
-M = maximum value of the curve
-G = gamma value
-B = channel address
-F = multiplication factor
-O = offset

Another tool that simplifies setting gamma curves is:

./gamma_conf.sh 0.6

Save LUT into a file (all 4 channels):

pmem -B 0x80000000 -m R0x80300000+0x10000/W > /tmp/lut.save

Load LUT from a file (all 4 channels):

pmem -B 0x80000000 -m W0x80300000+0x10000/W </tmp/lut.save