AXIOM Alpha Software

1 Image Pipeline

Image Acquisition Pipeline:

HDMI Image Processing Pipeline:

2 Register Memory Space

CMV12000 Register Blocks

CMV12000 to HDMI registers

Image sensor related registers: 0x60xxxxxx

cmv.func contains the functions to deal with this address space

HDMI related registers: 0x80xxxxxx

hdmi.func contains the functions to deal with this address space

0xU0VWWWWW
U = image sensor or HDMI
V = register block
WWWWW =  addresses inside a register block

Register Addresses HDMI:

0x800xxxxx = scan generator
0x801xxxxx = address generator
0x802xxxxx = component matrix 4x4
0x803xxxxx = 4 channel 12bit LUT
0x804xxxxx = HDMI PLL dynamic reconfig

Register Addresses Image Sensor:

0x600xxxxx = CMV SPI registers
0x601xxxxx = address generator/capture control
0x602xxxxx = LVDS input delay registers
0x603xxxxx = column/row noise LUTs

Examples:

0x60000000 = sensor, register block 0, address 0 (CMV register 0)

3 Connecting to the camera

Connect the camera trough a micro usb cable.

Zedboard provides a serial to usb bridge

On linux : dmesg | cat tty

Find the correct tty /dev

Use putty and connect at 115200 bauds, standard settings.

./cmv_train3 -a

this trains the lvds lanes and needs to be done every time the camera reboots please run it and see if it has proper output sometimes I saw that due to some loose contact or some other problem all the training values were 0 if this is the case wiggling a bit with the sensor connector or just waiting did the trick every time

./setup.sh

prepares the HDMI output it also takes other parameters like “1080p60” or “1080p50” or “SWIT” from what we saw each monitor requires this to be a bit different check the details by reading the contents of the setup.sh script or just try each line and see if the monitor does something (should stay black but register signal input)

test the HDMI output by loading a single image: ./mimg -w newsmoke_n.rgb16

this does not work if you initiate live image acquisition already with ./fil_reg…. as it will overwrite the framebufer 25 times a second already so all you might end up seeing is a short flash of the above newsmoke image

./fil_reg 11 …

enables live video acquisition it is a function in cmv.func so you need to run . ./cmv.func beforehand

./chroot_init

starts the raspian chroot environment and webserver

you can go to the cameras IP over ethernet then and see the webgui (dont forget to configure ifconfig IP firsthand) Alpha will register DHCP automatically if used with a router

4 Reading and Writing Sensor Register

Execute the script that contains the sensor interface functions before we can use them:

. ./cmv.func

Set Sensor register 82 to the decimal value "3122"

cmv_reg  82   3122

Read Register 82:

 cmv_reg  82

will output:

0x00000C31

5 Enable/Disable LEDs

Disable LEDs on the back:

fil_reg 15 0x01FF01FF

disable LEDs on the front:

fil_reg 14 0xFFFF0000

Enable all LEDs again:

Front:

fil_reg 14 0x0

Back:

fil_reg 15 0x0

6 Row/Coloumn Noise Correction

To run this calibration an evenly lit image is essential. Be sure to use a lens with little vignetting effects and set the aperture to a range that suffers from least vignetting effects. Diffusion gel and turning focus to infinite can help smooth out light differences.

It is advised to disable automatic image acquisition before running the RCN profiler:

./cmv_rcn3 -zsacr -N 5

Pixels going out of range will show up as blue dots. To prevent this enable clipping by setting register 11 bits: 21:20 to 1:

fil_reg 11 0xFC31F000 # clipping on

for reference (dont apply this setting in the FPN process:

fil_reg 11 0xFC01F000 # clipping off

Save FPN profile into a file (all 4 channels):

pmem -B 0x60000000 -m R0x60300000+0x8000/W > /tmp/rcn.save

Load FPN profile from a file (all 4 channels):

pmem -B 0x60000000 -m W0x60300000+0x8000/W </tmp/rcn.save

Clear FPN profile:

./cmv_rcn3 -z

If only a particular channel should be saved/loaded the addresses and lengths are different, eg:

pmem -B 0x60000000 -m R0x60304000+0x2000/W > /tmp/rcn_lut1.save

With the real time histogram you can evaluate how evenly lit your image is and can also judge the calibration results. The spikes in the histogram should for optimal results be in the 33% gray range and all line up in one place. With a good RCN profile they should get higher and less wide in the slope.

7 statically linked busybox

http://vserver.13thfloor.at/Stuff/AXIOM/FAKE/

builtin fake devmem

all you need to get it to work is the following:

dd if=/dev/zero of=/tmp/mem bs=1k seek=4M count=1

this will create a sparse 4GB file /tmp/mem, which will be used by the fake devmem values written can be read back, non existing values return 0

/bin/sh and /sbin/devmem both link to busybox on the axiom alpha filesystem so both can be tested with this executeable

8 Capturing an Image

Preparations:

ssh root@*alpha-IP* "./cmv_train3 -a"

Capture the image:

ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-

So we dont need to type the password every time we snap an image:

Install:

sudo apt-get install sshpass

Use:

sshpass -p '1234' ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-

9 Create Custom Image

Using imagemagick:

convert originalimage.png -depth 16 -size 1920x1080 rgb:output.raw16

10 Display Custom Image

./mimg -h
This is ./mimg V1.4
options are:
-h        print this help message
-w        use word sized data
-D <val>  image color depth
-W <val>  image width
-H <val>  image height
-T <val>  load test pattern
-B <val>  memory mapping base
-S <val>  memory mapping size
-A <val>  memory mapping address

You can load custom images into the HDMI image pipeline. Be sure to disable automatic image acquisition beforehand or you will only be able to see the custom image flashing for a single frame in the video stream:

./mimg -w newsmoke_n.rgb16

11 cmv_perf3

displays image stats and performance data, the longer it runs the more accurate the stats will be.

./cmv_perf3 -h
This is ./cmv_perf3 V1.0
options are:
-h        print this help message
-f        measure frame rates
-m        measure memory bandwidth
-D <val>  delay between updates
-T <val>  run for a given time

Will get more accurate with ntpd running

ntpd -p 86.59.80.170 -p 86.59.118.153

Typical output:

 ./cmv_perf3
mapped 0x60000000+0x00400000 to 0x60000000.
mapped 0x80000000+0x00400000 to 0x80000000.
hdmi: 29.97 FPS  cseq: 29.97 FPS

12 cmv_snap3

capture one still image and display it over HDMI immediately. If the -z parameter is not used this tool will provide the image data via stdout.

./cmv_snap3 -h
This is ./cmv_snap3 V1.9
options are:
-h        print this help message
-8        output 8 bit per pixel
-2        output 12 bit per pixel
-b        enable black columns
-r        dump sensor registers
-t        enable cmv test pattern
-z        produce no data output
-e <exp>  exposure times
-v <exp>  exposure voltages
-s <num>  shift values by <num>
-S <val>  writer byte strobe
-R <fil>  load sensor registers

./cmv_snap3 -r # without an -e parameter outputs only sensor registers
./cmv_snap3 -e 20ms > image.raw16 # capture an image and write the data into a file

Can also be used for direct capture to remote host:

ssh root@*cameraip* "./cmv_snap3 -e 20ms" > image.raw16 # capture an image and write the data into a file

To save all current sensor registers into a file

./cmv_snap3 -r > settings.reg 

To load all current sensor registers from a file

./cmv_snap3 -R settings.reg

13 cmv_hist3

Outputs RAW histogram data of the last captured image. Values are tab separated in 4 columns (Order: GRBG[image flip on], RGGB[image flip off]) with 4096 values (when operating in 12 bit mode) each.

Download: http://vserver.13thfloor.at/Stuff/AXIOM/cmv_hdmi2/cmv_hist3

./cmv_hist2
This is ./cmv_hist2 V1.0
options are:
-h        print this help message
-s        acquire snapshot
-B <val>  register mapping base
-S <val>  register mapping size
-M <val>  buffer memory base
-Z <val>  buffer memory size

Real Time Histogram Plot

13.1 Examples

./cmv_hist2 -s

Acquire new snapshot and output histogram

./cmv_hist2

Acquire histogram from last captured snapshot in memory

ssh root@192.168.2.88 "./cmv_hist3 -s" > capt.hist && gnuplot hist.gplot

14 cmv_train3

Executes the LVDS training routines to align LVDS reading timing. Execute this script when the Axiom Alpha prototype is booted up before you capture any images:

./cmv_train3 -a

 ./cmv_train3 -h
This is ./cmv_train3 V1.3
options are:
-h        print this help message
-a        test all bit pattern
-B <val>  memory mapping base
-S <val>  memory mapping size
-A <val>  memory mapping address
-P <val>  training pattern

15 Linearization

This happens as very first step in the image pipeline, before the Row/Column Noise compensation.

./linear_conf.sh *factor* *offset*

Examples:

./linear_conf.sh 1.2 -0.1 # multiple by factor 1.2 and shift offset 10% into the darker area

16 Post Processing images

16.1 Create RGGB separated color images from raw file:

convert -size 4096x3072 -depth 16 -crop +0+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch0.raw
convert -size 4096x3072 -depth 16 -crop -1+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch1.raw
convert -size 4096x3072 -depth 16 -crop +0-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch2.raw
convert -size 4096x3072 -depth 16 -crop -1-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch3.raw

16.2 Simple debayer with ImageMagick:

For flipped images:

convert \( -size 4096x3072 -depth 16 gray:colors_500ms.raw16 \) \
\( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \
-sample 2048x1536 \( -clone 2,3 -average \) -delete 2,3 -swap 0,1 +swap -combine colors_500ms.png

For unflipped images:

convert \( -size 4096x3072 -depth 16 gray:IT8_incand.raw16 \) \
\( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \
-sample 2048x1536 \( -clone 0,1 -average \) -delete 0,1 +swap -combine IT8_incand.png

16.3 Plot Histogram of Raw Image

Create histogram values file with imagemagick:

convert -size 4096x3072 -depth 16 gray:image.raw16 -format "%c" histogram:info: > histogram.hist

Reformat the file:

gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }' 

Or all in one command:

convert -size 4096x3072 -depth 16 gray:image.raw16 -format %c histogram:info:- | gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }' > histogram.info

Then draw it with this gnuplot script:

set notitle
set term svg size 1024, 512
set xlabel "Values"
set ylabel "Number of Pixels"
set multiplot
set obj 1 rectangle behind from screen 0,0 to screen 1,1
set obj 1 fillstyle solid 1.0 fillcolor rgbcolor "white"
set xrange[0:4096]
set yrange[0:8000]
plot 	'histogram.info' using 1:2	with lines lc rgb "#FF0000"	title "name your curve", \
       'histogram2.info' using 1:2	with lines lc rgb "#FF0000"	title "name your curve"
unset multiplot

execute the script with:

gnuplot thescript > output.svg

17 chroot environment on Axiom Alpha

Full raspian image is available on the alpha prototype in a changeroot environment to keep the core system free from too much clutter.

Download:

http://vserver.13thfloor.at/Stuff/AXIOM/raspian.tar.xz

Preferably download from the alpha prototype directly and upack into /opt/raspian, moving files to the prototype using Filezilla has produced some problems.

Map required folders:

mount -o bind /proc /opt/raspian/proc
mount -o bind /tmp /opt/raspian/tmp

Changeroot:

chroot /opt/raspian/ /bin/bash

18 Live HDMI video output

Init:

cd root
./cmv_train3 -a
./setup.sh

Start Video Acquisition:

. ./cmv.func
fil_reg 15 0x01000100

Stop Video Acquisition:

fil_reg 15 0x0

Half HDMI Frequency (1080p60 -> 1080p30 or 1080p50 -> 1080p25)

pll_reg 22 0x2106

Full HDMI Frequency (1080p60)

pll_reg 22 0x2083

18.1 HDMI Output Parameters

. ./i2c.func

i2c_map 0x39 0x55 5 6 0x0 #RGB output
i2c_map 0x39 0x55 5 6 0x1 #YCbCr 422 output
i2c_map 0x39 0x55 5 6 0x2 #YCbCr 444 output

19 Update Firmware

u-boot access:

mount /dev/mmcblk0p1 /mnt/

20 Real Time FPN correction

FPN correction supports a 9bit signed value for each column and row.

9 bit signed integer means:

0xFF = 255 (highest possible positive value)
0x1FF = -1
0x100 = -256 (lowest possible negative value

To obtain the correct value (for -7 in this case) to set you can run:

echo $(( -7 & 0x1FF ))

which outputs (in decimal)

505

To set a value use syntax:

devmem address 32 value

or with actual values:

devmem 0x60304000 32 0xFF

Adresses are:

0x60300000 = even columns, 2048 total
0x60304000 = odd columns, 2048 total
0x60308000 = even rows, 2048 total (only first 1576 are used)
0x6030C000 = odd rows, 2048 total (only first 1576 are used)

each row/column address increases by 4 from current to next row/column.

Row/Column 0 is in the top left corner.

20.1 Balance (FPN profiling tool)

Download: http://vserver.13thfloor.at/Stuff/AXIOM/balance

balance can extract a DSNU profile from a darkframes and apply the created profile to correct an image.

Usage:

Extract FPN profile:

./balance -b -o darkframe.raw16 > profile.offsets

Apply FPN profile to image:

./balance -b -O profile.offsets image.raw16 > image_balanced.raw16

Only use the -b parameter if the image uses darkcolumns.

The profile.offsets file format is:

row-numer offset (double) then column number (double)

(3072+4096)x8 = 57344 Bytes = 57KB

One double is 8 bytes (64 bits)

20.2 In-camera FPN profiling

Point the camera at a totally evenly lit non-clipping same color surface like a white wall (out of focus helps to even out minor details) and run:

./cmv_rcn3 -azc

To help visualize how evenly lit a surface is this LUT can help:

./lut_conf.sh -M 0x100000 -N 4096

21 Matrix Color Conversion

Each color channel is going through 4 stages that each look like this: (I+A)*E+O

I = input (25bit unsigned)
A = adjustment (30bit signed)
E = matrix coefficient (18bit signed)
O = offset (48bit signed)

The four stages are connected together that the offset of stage N is the output of stage N-1. The offset of stage 1 is set in registers (32-35). The output of stage 4 is the final result.

For each channel that results in the following fomular:

Ri = SUMj ((Ij + Aij) * Eij) + Oi

Which in other words mean the result of matrix (Eij) multiplied with an input vector (Ij + Aij) plus an offset vector (Oi).

Currently all registers (beside input/output) are shortened to 16bit signed (this could be changed easily) values so the matrix is currently limited to 1/256 of the possible accuracy (due to a 8bit shift of the output).

All input /output register are tested for over/underflow and clipped to 12 bit (this can also be disabled).

Setting/Getting Registers:

. ./hdmi.func # load functions

mat_reg 0 # Read value of register 0 = A0
mat_reg 0 1 # Sett value of register 0 (A0) to 1

4x4 matrix coefficients (A0, A1, A2, A3, B0, B1, B2, B3, B4, C0, C1, C2, C3, D0, D1, D2, D3) are mapped to addresses 0 - 15. In registers 16 - 31 there are adjustment values and from 32 - 35 there are 4 offset values.

Input format per default is (R, G1, G2, B).

All values of the 4x4 matrix can be set conveniently with this script:

Syntax:

./mat4_conf.sh  A0 A1 A2 A3  B0 B1 B2 B3 B4  C0 C1 C2 C3  D0 D1 D2 D3    O1 O2 O3 O4

Examples:

./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1    0 0 0 0   # unity matrix but not optimal as both green channels are processed separately
./mat4_conf.sh  1 0 0 0  0 0.5 0.5 0  0 0.5 0.5 0  0 0 0 1    0 0 0 0   # the two green channels inside each 2x2 pixel block are averaged and output on both green pixels
./mat4_conf.sh  0 0 0 1  0 1 0 0  0 0 1 0  1 0 0 0    0 0 0 0   # red and blue are swapped
./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1    0.5 0 0 0    # red 50% brigther
./mat4_conf.sh  1 0 0 0  0 1 0 0  0 0 1 0  0 0 0 1.5    0 0 0 0    # blue multiplied with factor 1.5
./mat4_conf.sh  .25 .25 .25 .25  .25 .25 .25 .25  .25 .25 .25 .25  .25 .25 .25 .25    0 0 0 0    # black/white
./mat4_conf.sh  -1 0 0 0  0 -0.5 -0.5 0  0 -0.5 -0.5 0  0 0 0 -1    1 1 1 1    # negative

An RGB 3x3 matrix can easily be extended to a 4x4 matrix by using the coefficient for green and offsets twice or better to half both of the green coefficients and use the same value for both greens twice.

22 LUTs

The result from the matrix conversation goes straight into 4x12 bit Look Up Tables (LUT). The output is 16bit wide.

These 4096x16bit values can be set at the register address space: 0x8030xxxx [0x80300000 (ch0 - R), 0x80304000 (ch1 - G1), 0x80308000 (ch2 - G2), 0x8030C000 (ch3 - B)]

For creating classic gamma curves for the LUTs there is a tool called: lut_conf3

./lut_conf3 -h
This is ./lut_conf3 V1.5
options are:
-h        print this help message
-d        dump current values
-r        read index/value pairs
-w        write index/value pairs
-z        wrap on limits
-s <val>  scale input value
-m <val>  minimum output value
-N <val>  number of lut entries
-M <val>  maximum output value
-C <val>  sigmoid/sine center
-F <val>  output value factor
-O <val>  output value offset
-B <val>  memory mapping base
-S <val>  memory mapping size
-A <val>  memory mapping address
-G <val>  gamma value

Examples:

./lut_conf3 -N 4096 -M 65535 -G 0.6 -B 0x80300000 # create LUT of gamma curve for gamma value 0.6 and set it for red channel (ch0)

-N = number of entries
-M = maximum value of the curve
-G = gamma value
-B = channel address
-F = multiplication factor
-O = offset

Another tool that simplifies setting gamma curves is:

./gamma_conf.sh 0.6

Save LUT into a file (all 4 channels):

pmem -B 0x80000000 -m R0x80300000+0x10000/W > /tmp/lut.save

Load LUT from a file (all 4 channels):

pmem -B 0x80000000 -m W0x80300000+0x10000/W </tmp/lut.save

Write a LUT pair table into a file:

./lut_conf3 -d > gamma-table.save

Read a LUT pair table file back in:

./lut_conf3 -r < gamma-table.save