AXIOM Alpha Software
1 Image Pipeline
Image Acquisition Pipeline:
HDMI Image Processing Pipeline:
2 Register Memory Space
Image sensor related registers: 0x60xxxxxx
cmv.func contains the functions to deal with this address space
HDMI related registers: 0x80xxxxxx
hdmi.func contains the functions to deal with this address space
0xU0VWWWWW U = image sensor or HDMI V = register block WWWWW = addresses inside a register block
Register Addresses HDMI:
0x800xxxxx = scan generator 0x801xxxxx = address generator 0x802xxxxx = component matrix 4x4 0x803xxxxx = 4 channel 12bit LUT 0x804xxxxx = HDMI PLL dynamic reconfig
Register Addresses Image Sensor:
0x600xxxxx = CMV SPI registers 0x601xxxxx = address generator/capture control 0x602xxxxx = LVDS input delay registers 0x603xxxxx = column/row noise LUTs
Examples:
0x60000000 = sensor, register block 0, address 0 (CMV register 0)
3 Reading and Writing Sensor Register
Execute the script that contains the sensor interface functions before we can use them:
. ./cmv.func
Set Sensor register 82 to the decimal value "3122"
cmv_reg 82 3122
Read Register 82:
cmv_reg 82
will output:
0x00000C31
4 Enable/Disable LEDs
Disable LEDs on the back:
fil_reg 15 0x01FF01FF
disable LEDs on the front:
fil_reg 14 0xFFFF0000
Enable all LEDs again:
Front:
fil_reg 14 0x0
Back:
fil_reg 15 0x0
5 Row/Coloumn Noise Correction
To run this calibration an evenly lit image is essential. Be sure to use a lens with little vignetting effects and set the aperture to a range that suffers from least vignetting effects. Diffusion gel and turning focus to infinite can help smooth out light differences.
It is advised to disable automatic image acquisition before running the RCN profiler:
./cmv_rcn3 -zsacr -N 5
Pixels going out of range will show up as blue dots. To prevent this enable clipping by setting register 11 bits: 21:20 to 1:
fil_reg 11 0xFC31F000 # clipping on
for reference (dont apply this setting in the FPN process:
fil_reg 11 0xFC01F000 # clipping off
Save FPN profile into a file (all 4 channels):
pmem -B 0x60000000 -m R0x60300000+0x8000/W > /tmp/rcn.save
Load FPN profile from a file (all 4 channels):
pmem -B 0x60000000 -m W0x60300000+0x8000/W </tmp/rcn.save
Clear FPN profile:
./cmv_rcn3 -z
If only a particular channel should be saved/loaded the addresses and lengths are different, eg:
pmem -B 0x60000000 -m R0x60304000+0x2000/W > /tmp/rcn_lut1.save
With the real time histogram you can evaluate how evenly lit your image is and can also judge the calibration results. The spikes in the histogram should for optimal results be in the 33% gray range and all line up in one place. With a good RCN profile they should get higher and less wide in the slope.
6 statically linked busybox
http://vserver.13thfloor.at/Stuff/AXIOM/FAKE/
builtin fake devmem
all you need to get it to work is the following:
dd if=/dev/zero of=/tmp/mem bs=1k seek=4M count=1
this will create a sparse 4GB file /tmp/mem, which will be used by the fake devmem values written can be read back, non existing values return 0
/bin/sh and /sbin/devmem both link to busybox on the axiom alpha filesystem so both can be tested with this executeable
7 Capturing an Image
Preparations:
ssh root@*alpha-IP* "./cmv_train3 -a"
Capture the image:
ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-
So we dont need to type the password every time we snap an image:
Install:
sudo apt-get install sshpass
Use:
sshpass -p '1234' ssh root@*alpha-IP* "./cmv_snap3 -e 10ms" | tee snap.raw16 | display -size 4096x3072 -depth 16 gray:-
8 Create Custom Image
Using imagemagick:
convert originalimage.png -depth 16 -size 1920x1080 rgb:output.raw16
9 Display Custom Image
./mimg -h This is ./mimg V1.4 options are: -h print this help message -w use word sized data -D <val> image color depth -W <val> image width -H <val> image height -T <val> load test pattern -B <val> memory mapping base -S <val> memory mapping size -A <val> memory mapping address
You can load custom images into the HDMI image pipeline. Be sure to disable automatic image acquisition beforehand or you will only be able to see the custom image flashing for a single frame in the video stream:
./mimg -w newsmoke_n.rgb16
10 cmv_perf3
displays image stats and performance data, the longer it runs the more accurate the stats will be.
./cmv_perf3 -h This is ./cmv_perf3 V1.0 options are: -h print this help message -f measure frame rates -m measure memory bandwidth -D <val> delay between updates -T <val> run for a given time
Will get more accurate with ntpd running
ntpd -p 86.59.80.170 -p 86.59.118.153
Typical output:
./cmv_perf3 mapped 0x60000000+0x00400000 to 0x60000000. mapped 0x80000000+0x00400000 to 0x80000000. hdmi: 29.97 FPS cseq: 29.97 FPS
11 cmv_snap3
capture one still image and display it over HDMI immediately. If the -z parameter is not used this tool will provide the image data via stdout.
./cmv_snap3 -h This is ./cmv_snap3 V1.9 options are: -h print this help message -8 output 8 bit per pixel -2 output 12 bit per pixel -b enable black columns -r dump sensor registers -t enable cmv test pattern -z produce no data output -e <exp> exposure times -v <exp> exposure voltages -s <num> shift values by <num> -S <val> writer byte strobe -R <fil> load sensor registers
./cmv_snap3 -r # without an -e parameter outputs only sensor registers ./cmv_snap3 -e 20ms > image.raw16 # capture an image and write the data into a file
Can also be used for direct capture to remote host:
ssh root@*cameraip* "./cmv_snap3 -e 20ms" > image.raw16 # capture an image and write the data into a file
To save all current sensor registers into a file
./cmv_snap3 -r > settings.reg
To load all current sensor registers from a file
./cmv_snap3 -R settings.reg
12 cmv_hist3
Outputs RAW histogram data of the last captured image. Values are tab separated in 4 columns (Order: GRBG[image flip on], RGGB[image flip off]) with 4096 values (when operating in 12 bit mode) each.
Download: http://vserver.13thfloor.at/Stuff/AXIOM/cmv_hdmi2/cmv_hist3
./cmv_hist2 This is ./cmv_hist2 V1.0 options are: -h print this help message -s acquire snapshot -B <val> register mapping base -S <val> register mapping size -M <val> buffer memory base -Z <val> buffer memory size
12.1 Examples
./cmv_hist2 -s
Acquire new snapshot and output histogram
./cmv_hist2
Acquire histogram from last captured snapshot in memory
ssh root@192.168.2.88 "./cmv_hist3 -s" > capt.hist && gnuplot hist.gplot
13 cmv_train3
Executes the LVDS training routines to align LVDS reading timing. Execute this script when the Axiom Alpha prototype is booted up before you capture any images:
./cmv_train3 -a
./cmv_train3 -h This is ./cmv_train3 V1.3 options are: -h print this help message -a test all bit pattern -B <val> memory mapping base -S <val> memory mapping size -A <val> memory mapping address -P <val> training pattern
14 Linearization
This happens as very first step in the image pipeline, before the Row/Column Noise compensation.
./linear_conf.sh *factor* *offset*
Examples:
./linear_conf.sh 1.2 -0.1 # multiple by factor 1.2 and shift offset 10% into the darker area
15 Post Processing images
15.1 Create RGGB separated color images from raw file:
convert -size 4096x3072 -depth 16 -crop +0+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch0.raw convert -size 4096x3072 -depth 16 -crop -1+0 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch1.raw convert -size 4096x3072 -depth 16 -crop +0-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch2.raw convert -size 4096x3072 -depth 16 -crop -1-1 -sample 2048x1536 gray:colors_500ms.raw16 gray:colors_500ms_ch3.raw
15.2 Simple debayer with ImageMagick:
For flipped images:
convert \( -size 4096x3072 -depth 16 gray:colors_500ms.raw16 \) \ \( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \ -sample 2048x1536 \( -clone 2,3 -average \) -delete 2,3 -swap 0,1 +swap -combine colors_500ms.png
For unflipped images:
convert \( -size 4096x3072 -depth 16 gray:IT8_incand.raw16 \) \ \( -clone 0 -crop -1-1 \) \( -clone 0 -crop -1+0 \) \( -clone 0 -crop +0-1 \) \ -sample 2048x1536 \( -clone 0,1 -average \) -delete 0,1 +swap -combine IT8_incand.png
15.3 Plot Histogram of Raw Image
Create histogram values file with imagemagick:
convert -size 4096x3072 -depth 16 gray:image.raw16 -format "%c" histogram:info: > histogram.hist
Reformat the file:
gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }'
Or all in one command:
convert -size 4096x3072 -depth 16 gray:image.raw16 -format %c histogram:info:- | gawk -F, '(NF>3) { printf "%d\t%d\n", $2/16, $1 }' > histogram.info
Then draw it with this gnuplot script:
set notitle
set term svg size 1024, 512
set xlabel "Values"
set ylabel "Number of Pixels"
set multiplot
set obj 1 rectangle behind from screen 0,0 to screen 1,1
set obj 1 fillstyle solid 1.0 fillcolor rgbcolor "white"
set xrange[0:4096]
set yrange[0:8000]
plot 'histogram.info' using 1:2 with lines lc rgb "#FF0000" title "name your curve", \
'histogram2.info' using 1:2 with lines lc rgb "#FF0000" title "name your curve"
unset multiplot
execute the script with:
gnuplot thescript > output.svg
16 chroot environment on Axiom Alpha
Full raspian image is available on the alpha prototype in a changeroot environment to keep the core system free from too much clutter.
Download:
http://vserver.13thfloor.at/Stuff/AXIOM/raspian.tar.xz
Preferably download from the alpha prototype directly and upack into /opt/raspian, moving files to the prototype using Filezilla has produced some problems.
Map required folders:
mount -o bind /proc /opt/raspian/proc mount -o bind /tmp /opt/raspian/tmp
Changeroot:
chroot /opt/raspian/ /bin/bash
17 Live HDMI video output
Init:
cd root ./cmv_train3 -a ./setup.sh
Start Video Acquisition:
. ./cmv.func fil_reg 15 0x01000100
Stop Video Acquisition:
fil_reg 15 0x0
Half HDMI Frequency (1080p60 -> 1080p30 or 1080p50 -> 1080p25)
pll_reg 22 0x2106
Full HDMI Frequency (1080p60)
pll_reg 22 0x2083
17.1 HDMI Output Parameters
. ./i2c.func
i2c_map 0x39 0x55 5 6 0x0 #RGB output i2c_map 0x39 0x55 5 6 0x1 #YCbCr 422 output i2c_map 0x39 0x55 5 6 0x2 #YCbCr 444 output
18 Update Firmware
u-boot access:
mount /dev/mmcblk0p1 /mnt/
19 Real Time FPN correction
FPN correction supports a 9bit signed value for each column and row.
9 bit signed integer means:
0xFF = 255 (highest possible positive value) 0x1FF = -1 0x100 = -256 (lowest possible negative value
To obtain the correct value (for -7 in this case) to set you can run:
echo $(( -7 & 0x1FF ))
which outputs (in decimal)
505
To set a value use syntax:
devmem address 32 value
or with actual values:
devmem 0x60304000 32 0xFF
Adresses are:
0x60300000 = even columns, 2048 total 0x60304000 = odd columns, 2048 total 0x60308000 = even rows, 2048 total (only first 1576 are used) 0x6030C000 = odd rows, 2048 total (only first 1576 are used)
each row/column address increases by 4 from current to next row/column.
Row/Column 0 is in the top left corner.
19.1 Balance (FPN profiling tool)
Download: http://vserver.13thfloor.at/Stuff/AXIOM/balance
balance can extract a DSNU profile from a darkframes and apply the created profile to correct an image.
Usage:
Extract FPN profile:
./balance -b -o darkframe.raw16 > profile.offsets
Apply FPN profile to image:
./balance -b -O profile.offsets image.raw16 > image_balanced.raw16
Only use the -b parameter if the image uses darkcolumns.
The profile.offsets file format is:
row-numer offset (double) then column number (double)
(3072+4096)x8 = 57344 Bytes = 57KB
One double is 8 bytes (64 bits)
19.2 In-camera FPN profiling
Point the camera at a totally evenly lit non-clipping same color surface like a white wall (out of focus helps to even out minor details) and run:
./cmv_rcn3 -azc
To help visualize how evenly lit a surface is this LUT can help:
./lut_conf.sh -M 0x100000 -N 4096
20 Matrix Color Conversion
Each color channel is going through 4 stages that each look like this: (I+A)*E+O
I = input (25bit unsigned) A = adjustment (30bit signed) E = matrix coefficient (18bit signed) O = offset (48bit signed)
The four stages are connected together that the offset of stage N is the output of stage N-1. The offset of stage 1 is set in registers (32-35). The output of stage 4 is the final result.
For each channel that results in the following fomular:
Ri = SUMj ((Ij + Aij) * Eij) + Oi
Which in other words mean the result of matrix (Eij) multiplied with an input vector (Ij + Aij) plus an offset vector (Oi).
Currently all registers (beside input/output) are shortened to 16bit signed (this could be changed easily) values so the matrix is currently limited to 1/256 of the possible accuracy (due to a 8bit shift of the output).
All input /output register are tested for over/underflow and clipped to 12 bit (this can also be disabled).
Setting/Getting Registers:
. ./hdmi.func # load functions
mat_reg 0 # Read value of register 0 = A0 mat_reg 0 1 # Sett value of register 0 (A0) to 1
4x4 matrix coefficients (A0, A1, A2, A3, B0, B1, B2, B3, B4, C0, C1, C2, C3, D0, D1, D2, D3) are mapped to addresses 0 - 15. In registers 16 - 31 there are adjustment values and from 32 - 35 there are 4 offset values.
Input format per default is (R, G1, G2, B).
All values of the 4x4 matrix can be set conveniently with this script:
Syntax:
./mat4_conf.sh A0 A1 A2 A3 B0 B1 B2 B3 B4 C0 C1 C2 C3 D0 D1 D2 D3 O1 O2 O3 O4
Examples:
./mat4_conf.sh 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 # unity matrix but not optimal as both green channels are processed separately ./mat4_conf.sh 1 0 0 0 0 0.5 0.5 0 0 0.5 0.5 0 0 0 0 1 0 0 0 0 # the two green channels inside each 2x2 pixel block are averaged and output on both green pixels ./mat4_conf.sh 0 0 0 1 0 1 0 0 0 0 1 0 1 0 0 0 0 0 0 0 # red and blue are swapped ./mat4_conf.sh 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0.5 0 0 0 # red 50% brigther ./mat4_conf.sh 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1.5 0 0 0 0 # blue multiplied with factor 1.5 ./mat4_conf.sh .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 .25 0 0 0 0 # black/white ./mat4_conf.sh -1 0 0 0 0 -0.5 -0.5 0 0 -0.5 -0.5 0 0 0 0 -1 1 1 1 1 # negative
An RGB 3x3 matrix can easily be extended to a 4x4 matrix by using the coefficient for green and offsets twice or better to half both of the green coefficients and use the same value for both greens twice.
21 LUTs
The result from the matrix conversation goes straight into 4x12 bit Look Up Tables (LUT). The output is 16bit wide.
These 4096x16bit values can be set at the register address space: 0x8030xxxx [0x80300000 (ch0 - R), 0x80304000 (ch1 - G1), 0x80308000 (ch2 - G2), 0x8030C000 (ch3 - B)]
For creating classic gamma curves for the LUTs there is a tool called: lut_conf3
./lut_conf3 -h This is ./lut_conf3 V1.5 options are: -h print this help message -d dump current values -r read index/value pairs -w write index/value pairs -z wrap on limits -s <val> scale input value -m <val> minimum output value -N <val> number of lut entries -M <val> maximum output value -C <val> sigmoid/sine center -F <val> output value factor -O <val> output value offset -B <val> memory mapping base -S <val> memory mapping size -A <val> memory mapping address -G <val> gamma value
Examples:
./lut_conf3 -N 4096 -M 65535 -G 0.6 -B 0x80300000 # create LUT of gamma curve for gamma value 0.6 and set it for red channel (ch0)
-N = number of entries -M = maximum value of the curve -G = gamma value -B = channel address -F = multiplication factor -O = offset
Another tool that simplifies setting gamma curves is:
./gamma_conf.sh 0.6
Save LUT into a file (all 4 channels):
pmem -B 0x80000000 -m R0x80300000+0x10000/W > /tmp/lut.save
Load LUT from a file (all 4 channels):
pmem -B 0x80000000 -m W0x80300000+0x10000/W </tmp/lut.save
Write a LUT pair table into a file:
./lut_conf3 -d > gamma-table.save
Read a LUT pair table file back in:
./lut_conf3 -r < gamma-table.save