AXIOM Beta/Raw Recording
1 Overview
Simplified: The AXIOM Recorder is a small computer connected to the AXIOM Beta via HDMI, we use the HDMI video stream to transport our own custom raw image data from the camera. This stream needs to be repacked or post processed on the AXIOM Recorder to acquire the actual raw image sequence (RAW12 sequence). First software prototypes did this after recording, newer versions do it on-the-fly while recording.
Technical: On the AXIOM Beta we utilize a Full HD (1920 x 1080) HDMI output stream with twice the frame-rate to store 4 pixels (12 bit each) from the image sensor (sensels) in 2 RGB pixels (24 bit each) in the HDMI stream. After recording this data through a HDMI capture device like the Magewell HDMI USB3 Gen 2 on a PC, Laptop or Single Board Computer (SBC) the data needs to be post processed to obtain the original raw data from the image sensor again. The two HDMI images from a single raw image are called A-Frame and B-Frame.
2 Recording Principle
3 Pixel Packing V3 Structure
Traditional HDMI signal:
AXIOM Beta raw mode output:
Essentially from a 2x2 sensel block on the image sensor R, G1 go into the A frame and G2, B into the B frame of a HDMI stream.
Images from the HDMI stream look like this then:
|
|
4 Post Processing Principle
Version 1 Prototype:
Notes:
- Content of single *.rgb file and *.frame sequence should - if everything worked correctly - be 100% bit identical. So *.rgb file could be deleted after *.frame sequence extraction.
5 Recorder Hardware
Basically any PC with USB3 and a fast solid state memory should work fine. So far we have tested 3 devices. The goals are in general to have an as small as possible form factor, as low as possible power consumption (as we will be operating on batteries often) but enough speed/performance to reliably capture footage (also for longer durations) and process / playback the image sequences smoothly. Obviously any recent laptop/notebook or desktop PC will work well but here we want to focus on the much smaller form factors.
5.1 Intel NUC
recommended for AXIOM Recorder use: Yes
Tested an Intel NUC NUC7i5BNK, that's a 2017 model for around 400€
CPU: i5-7260U (Passmark 3868, TDP: 15 W)
Dimensions: approximately 4 × 4 inches (101.6 × 101.6 mm)
M.2 slot with PCIe x4 lanes
5.2 Rockpi 4 Model C
recommended for AXIOM Recorder use: No
4GB RAM (soldered)
CPU: Rockchip RK3399
Dimensions: 85 × 54mm (the smallest of the 3)
M.2 PCIe slot for SSD with four lanes PCIe 2.0
Unfortunately we could not get the M.2 SSD to perform fast enough for our purposes on the Rockpi 4 - maybe the PCIe 2.0 is the bottleneck.
Retail price: 60$
5.3 Lattepanda Alpha 864s
recommended for AXIOM Recorder use: Yes
CPU: Intel 8th m3--8100y (~4000 passmark claimed by manufacturer, TDP 15W)
8GB RAM (soldered)
Dimensions: 115 × 78 × 14 mm (2nd most compact of the 3 devices)
active cooling option is very silent, entirely passive cooling available from manufacturer
has onboard 64GB flash memory for operating system
Retail price: ~400€
5.4 Magewell USB HDMI Gen 2
The exact model we use is called “Magewell USB Capture HDMI Gen 2” Part number: 32060. It retails for around 350-400€ and can be bought used for around 250€, we discovered offers as low as 100$ on ebay.
Note: there are clones (eg. on Amazon) that look almost exactly like the Magewell device and promise similar features:
We were curious and bought one. Unfortunately the electronics inside are totally different and we could not even get the device to display a HDMI signal from the AXIOM Beta so we can not recommend those.
6 Requirements and Introduction
In addition to the AXIOM Beta with 1 or 2 HDMI plugin modules you need a USB Capture HDMI Gen 2 device from Magewell: https://www.magewell.com/products/usb-capture-hdmi-gen-2 connected to the HDMI output of the AXIOM Beta.
A PC/laptop/SBC connected to the AXIOM Beta is also required (see Recorder Hardware above), all instructions here assume this PC/laptop/SBC is running Linux - other operating systems are not supported by this guide - they might work - we have not tested them.
Commands to be run on the Linux PC are indicated like:
$ command
Commands to be run on the AXIOM Beta are indicated like:
[root@beta ~]# command
6.1 AXIOM Beta firmware flashing
For a full guide about firmware flashing/backup see: Firmware Flashing
6.2 Determine Magewell Video Device ID
Install v4l2-ctl:
$ sudo apt install v4l2-ctl
Run the tool:
$ v4l2-ctl --list-devices
This should output a list, in this case telling you that the magewell device is /dev/video2 and /dev/video3 (the internal webcam is /dev/video0 and /dev/video1):
USB Capture HDMI: USB Capture H (usb-0000:00:14.0-2):
/dev/video2
/dev/video3
Integrated Camera: Integrated C (usb-0000:00:1a.0-1.6):
/dev/video0
/dev/video1
6.3 Download, Install & Run Magewell Setup Tool
This raw recording method relies on a USB HDMI capture device like the one produced by Magewell that can capture 1920x1080 at 60 FPS without any compression or image manipulation. .
download setup tool from magewell website: https://www.magewell.com/downloads/usb-capture#/tools/linux-ubuntu
run:
$ sudo usbcaptureutility
under video set:
- input color format: RGB
- input quantization: full-range
- capture color format: RGB
- capture quantization: full-range
- capture saturation: full-range
- Specific Effect: Flip
under advanced set:
- color format: only leave RGB24 on the left, move all other options to the right
- frame rate: only leave 60.00 on the left, move all other options to the right
- resolution, only leave 1920x1080 on the left, move all other options to the right
click “save to device”, disconnect and reconnect device
Settings will be stored in the device even when you disconnect it so these steps only need to be completed once.
6.4 Check Transmission Data Integrity (optional)
To test if the magewell device is receiving data from the AXIOM Beta in a reliable way (good cables are essential for example) the following method can be used:
assuming the magewell input is /dev/videoX:
using Firmware 1.0:
init raw mode on AXIOM Beta
[root@beta ~]# ./kick_manual_raw.sh
stop the acquisition with:
[root@beta ~]# fil_reg 15 0
then load a gradient overlay:
[root@beta ~]# ./mimg -T4
disable overlay:
[root@beta ~]# ./mimg -O -P0
Reset gamma and color matrix settings:
[root@beta ~]# ./mat4_conf.py 1 # ./gamma_conf.sh 1
on magewell connected pc run (assuming the magewell input is /dev/videoX):
$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null
this should output many lines containing the hash:
ddbfb96e4b8d5191bb274885ccc0264d
then let it run for 15 minutes with (assuming the magewell input is /dev/videoX):
$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null | grep -v ddbfb96e4b8d5191bb274885ccc0264d
no hashes should be reported, if hashes are reported this could mean a bad cable/bad connection
while the hash test is running on the camera depending on the HDMI plugin module being connected to the top plugin module slot set:
[root@beta ~]# scn_reg 31 0x5000
and then:
[root@beta ~]# scn_reg 31 0x4000
no different hashes should be reported still or the bottom plugin module slot:
[root@beta ~]# scn_reg 30 0x5000
and then:
[root@beta ~]# scn_reg 30 0x4000
no different hashes should be reported still
now stop the hash comparing line.
7 Usage Instructions
7.1 Initialize Raw Mode on AXIOM Beta
Firmware 1.0:
[root@beta ~]# ./kick_manual_raw.sh
Firmware 2.0:
[root@beta ~]# axiom_start.sh raw
7.2 Clear Overlay
Firmware 1.0:
[root@beta ~]# ./mimg -a -o -P 0
7.3 Check HDMI Stream
assuming the magewell input is /dev/videoX:
$ ffplay /dev/videoX
7.4 Frame-rate Verification
on AXIOM Beta run the performance tool: Firmware 1.0:
[root@beta ~]# ./cmv_perf3
Firmware 2.0:
[root@beta ~]# axiom_perf
For 30 FPS this should output something similar to:
mapped 0x60000000+0x00400000 to 0x60000000. mapped 0x80000000+0x00400000 to 0x80000000. hdmi: 60.27 FPS cseq: 29.97 FPS
If the reported FPS are lower you need to reduce the exposure time (run performance tool again afterwards to check if the changes had the desired effect):
Firmware 1.0:
[root@beta ~]# ./cmv_snap3 -e 11ms -z
Firmware 2.0:
[root@beta ~]# axiom_snap -e 11ms -z
7.5 Raw HDMI Related Commands
The first parameter of scn_reg defines which plugin module/shield slot is affected:
31 - top (North) HDMI plugin module 30 - bottom (South) HDMI plugin module 29 - HDMI Shield
set bottom HDMI plugin module to: live view:
Firmware 1.0:
[root@beta ~]# scn_reg 30 0x0000
Firmware 2.0:
[root@beta ~]# axiom_scn_reg 30 0x0000
set bottom HDMI plugin module to: A channel / A frames:
Firmware 1.0:
[root@beta ~]# scn_reg 30 0x5000
Firmware 2.0:
[root@beta ~]# axiom_scn_reg 30 0x5000
set bottom HDMI plugin module to: B channel / B frames:
Firmware 1.0:
[root@beta ~]# scn_reg 30 0x6000
Firmware 2.0:
[root@beta ~]# axiom_scn_reg 30 0x6000
set bottom HDMI plugin module to: A/B frames alternating:
Firmware 1.0:
[root@beta ~]# scn_reg 30 0x7000
Firmware 2.0:
[root@beta ~]# axiom_scn_reg 30 0x7000
activates the four corner markers (all 4 corner pixels are identical):
Firmware 1.0:
[root@beta ~]# ./raw_mark.sh
Firmware 2.0:
[root@beta ~]# axiom_raw_mark.sh
- red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and see if no frames were dropped during transfer/capture
- green channel: wrsel buffer information
- blue channel: 0x55 / 85 (A-frame) and 0xAA / 170 (B-frame)
Note: Activating the corner markers is essential for post processing the captured files later so do not forget to do this.
set corner marker frame counter max to 256:
Note: This is essential for post processing the captured files later so do not forget to do this.
Firmware 1.0:
[root@beta ~]# scn_reg 2 0x100
Firmware 2.0:
[root@beta ~]# axiom_scn_reg 2 0x100
7.6 Record Raw Video with ffmpeg
$ ffmpeg -i /dev/videoX -map 0 -pix_fmt rgb24 recording.rgb
7.7 Extract Image Sequence
To extract the individual frames from a single *.rgb file after recording do:
$ split recording.rgb recording_frame_ --additional-suffix=.frame -d -b 6220800 --verbose --suffix-length=5
6220800 here corresponds to the file size in bytes of one frame (1920 x 1080 x 24bit / 8)