AXIOM Beta/Raw Recording

1 Overview

Simplified: The AXIOM Recorder is a small computer connected to the AXIOM Beta via HDMI, we use the HDMI video stream to transport our own custom raw image data from the camera. This stream needs to be repacked or post processed on the AXIOM Recorder to acquire the actual raw image sequence (RAW12 sequence). First software prototypes did this after recording, newer versions do it on-the-fly while recording.

Technical: On the AXIOM Beta we utilize a Full HD (1920 x 1080) HDMI output stream with twice the frame-rate to store 4 pixels (12 bit each) from the image sensor (sensels) in 2 RGB pixels (24 bit each) in the HDMI stream. After recording this data through a HDMI capture device like the Magewell HDMI USB3 Gen 2 on a PC, Laptop or Single Board Computer (SBC) the data needs to be post processed to obtain the original raw data from the image sensor again. The two HDMI images from a single raw image are called A-Frame and B-Frame.

2 Recording Principle

3 Pixel Packing V3 Structure

Traditional HDMI signal:

AXIOM Beta raw mode output:

Essentially from a 2x2 sensel block on the image sensor R, G1 go into the A frame and G2, B into the B frame of a HDMI stream.

Images from the HDMI stream look like this then:

4 Post Processing Principle

Version 1 Prototype:

Notes:

• Content of single *.rgb file and *.frame sequence should - if everything worked correctly - be 100% bit identical. So *.rgb file could be deleted after *.frame sequence extraction.

5 Recorder Hardware

Basically any PC with USB3 and a fast solid state memory should work fine. So far we have tested 3 devices. The goals are in general to have an as small as possible form factor, as low as possible power consumption (as we will be operating on batteries often) but enough speed/performance to reliably capture footage (also for longer durations) and process / playback the image sequences smoothly. Obviously any recent laptop/notebook or desktop PC will work well but here we want to focus on the much smaller form factors.

5.1 Intel NUC

recommended for AXIOM Recorder use: Yes
Tested an Intel NUC NUC7i5BNK, that's a 2017 model for around 400€
CPU: i5-7260U (Passmark 3868, TDP: 15 W)
Dimensions: approximately 4 × 4 inches (101.6 × 101.6 mm)
M.2 slot with PCIe x4 lanes

5.2 Rockpi 4 Model C

recommended for AXIOM Recorder use: No
4GB RAM (soldered)
CPU: Rockchip RK3399
Dimensions: 85 × 54mm (the smallest of the 3)
M.2 PCIe slot for SSD with four lanes PCIe 2.0
Unfortunately we could not get the M.2 SSD to perform fast enough for our purposes on the Rockpi 4 - maybe the PCIe 2.0 is the bottleneck.
Retail price: 60$

5.3 Lattepanda Alpha 864s

recommended for AXIOM Recorder use: Yes
CPU: Intel 8th m3--8100y (~4000 passmark claimed by manufacturer, TDP 15W)
8GB RAM (soldered)
Dimensions: 115 × 78 × 14 mm (2nd most compact of the 3 devices)
active cooling option is very silent, entirely passive cooling available from manufacturer
has onboard 64GB flash memory for operating system
Retail price: ~400€

5.4 Magewell USB HDMI Gen 2

The exact model we use is called “Magewell USB Capture HDMI Gen 2” Part number: 32060. It retails for around 350-400€ and can be bought used for around 250€, we discovered offers as low as 100$ on ebay.

Note: there are clones (eg. on Amazon) that look almost exactly like the Magewell device and promise similar features:

We were curious and bought one. Unfortunately the electronics inside are totally different and we could not even get the device to display a HDMI signal from the AXIOM Beta so we can not recommend those.

6 Requirements and Introduction

In addition to the AXIOM Beta with 1 or 2 HDMI plugin modules you need a USB Capture HDMI Gen 2 device from Magewell: https://www.magewell.com/products/usb-capture-hdmi-gen-2 connected to the HDMI output of the AXIOM Beta.

A PC/laptop/SBC connected to the AXIOM Beta is also required (see Recorder Hardware above), all instructions here assume this PC/laptop/SBC is running Linux - other operating systems are not supported by this guide - they might work - we have not tested them.

Commands to be run on the Linux PC are indicated like:

$ command

Commands to be run on the AXIOM Beta are indicated like:

[root@beta ~]# command

6.1 AXIOM Beta firmware flashing

For a full guide about firmware flashing/backup see: Firmware Flashing

6.2 Determine Magewell Video Device ID

Install v4l2-ctl:

$ sudo apt install v4l2-ctl

Run the tool:

$ v4l2-ctl --list-devices

This should output a list, in this case telling you that the magewell device is /dev/video2 and /dev/video3 (the internal webcam is /dev/video0 and /dev/video1):

USB Capture HDMI: USB Capture H (usb-0000:00:14.0-2):
    /dev/video2
    /dev/video3

Integrated Camera: Integrated C (usb-0000:00:1a.0-1.6):
    /dev/video0
    /dev/video1

6.3 Download, Install & Run Magewell Setup Tool

This raw recording method relies on a USB HDMI capture device like the one produced by Magewell that can capture 1920x1080 at 60 FPS without any compression or image manipulation. .

download setup tool from magewell website: https://www.magewell.com/downloads/usb-capture#/tools/linux-ubuntu

run:

$ sudo usbcaptureutility

under video set:

• input color format: RGB
• input quantization: full-range

• capture color format: RGB
• capture quantization: full-range
• capture saturation: full-range
• Specific Effect: Flip

under advanced set:

• color format: only leave RGB24 on the left, move all other options to the right
• frame rate: only leave 60.00 on the left, move all other options to the right
• resolution, only leave 1920x1080 on the left, move all other options to the right

click “save to device”, disconnect and reconnect device

Settings will be stored in the device even when you disconnect it so these steps only need to be completed once.

6.4 Check Transmission Data Integrity (optional)

To test if the magewell device is receiving data from the AXIOM Beta in a reliable way (good cables are essential for example) the following method can be used:

assuming the magewell input is /dev/videoX:

using Firmware 1.0:

init raw mode on AXIOM Beta

[root@beta ~]# ./kick_manual_raw.sh

stop the acquisition with:

[root@beta ~]# fil_reg 15 0

then load a gradient overlay:

[root@beta ~]# ./mimg -T4

disable overlay:

[root@beta ~]# ./mimg -O -P0

Reset gamma and color matrix settings:

[root@beta ~]# ./mat4_conf.py 1
# ./gamma_conf.sh 1

on magewell connected pc run (assuming the magewell input is /dev/videoX):

$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null

this should output many lines containing the hash:

ddbfb96e4b8d5191bb274885ccc0264d

then let it run for 15 minutes with (assuming the magewell input is /dev/videoX):

$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null | grep -v ddbfb96e4b8d5191bb274885ccc0264d

no hashes should be reported, if hashes are reported this could mean a bad cable/bad connection

while the hash test is running on the camera depending on the HDMI plugin module being connected to the top plugin module slot set:

[root@beta ~]# scn_reg 31 0x5000

and then:

[root@beta ~]# scn_reg 31 0x4000

no different hashes should be reported still or the bottom plugin module slot:

[root@beta ~]# scn_reg 30 0x5000

and then:

[root@beta ~]# scn_reg 30 0x4000

no different hashes should be reported still

now stop the hash comparing line.

7 Usage Instructions

7.1 Initialize Raw Mode on AXIOM Beta

Firmware 1.0:

[root@beta ~]# ./kick_manual_raw.sh

Firmware 2.0:

[root@beta ~]# axiom_start.sh raw

7.2 Clear Overlay

Firmware 1.0:

[root@beta ~]# ./mimg -a -o -P 0

7.3 Check HDMI Stream

assuming the magewell input is /dev/videoX:

$ ffplay /dev/videoX

7.4 Frame-rate Verification

on AXIOM Beta run the performance tool: Firmware 1.0:

[root@beta ~]# ./cmv_perf3

Firmware 2.0:

[root@beta ~]# axiom_perf

For 30 FPS this should output something similar to:

mapped 0x60000000+0x00400000 to 0x60000000.
mapped 0x80000000+0x00400000 to 0x80000000.
hdmi: 60.27 FPS     cseq: 29.97 FPS

If the reported FPS are lower you need to reduce the exposure time (run performance tool again afterwards to check if the changes had the desired effect):

Firmware 1.0:

[root@beta ~]# ./cmv_snap3 -e 11ms -z

Firmware 2.0:

[root@beta ~]# axiom_snap -e 11ms -z

7.5 Raw HDMI Related Commands

The first parameter of scn_reg defines which plugin module/shield slot is affected:

31 - top (North) HDMI plugin module 
30 - bottom (South) HDMI plugin module 
29 - HDMI Shield

set bottom HDMI plugin module to: live view:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x0000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x0000

set bottom HDMI plugin module to: A channel / A frames:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x5000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x5000

set bottom HDMI plugin module to: B channel / B frames:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x6000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x6000

Firmware 1.0: [root@beta ~]# scn_reg 30 0x7000

Firmware 2.0: [root@beta ~]# axiom_scn_reg 30 0x7000 set bottom HDMI plugin module to: A/B frames alternating (V3 structure) Firmware 1.0: [root@beta ~]# ./raw_mark.sh

Firmware 2.0: [root@beta ~]# axiom_raw_mark.sh activates the four corner markers (all 4 corner pixels are identical)

red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and see if no frames were dropped during transfer/capture green channel: wrsel buffer information blue channel: 0x55 / 85 (A-frame) and 0xAA / 170 (B-frame)

Note: Activating the corner markers is essential for post processing the captured files later so do not forget to do this. Firmware 1.0: [root@beta ~]# scn_reg 2 0x100

Firmware 2.0: [root@beta ~]# axiom_scn_reg 2 0x100 set corner marker frame counter max to 256

Note: This is essential for post processing the captured files later so do not forget to do this.

7.6 Record Raw Video with ffmpeg

  $ ffmpeg -i /dev/videoX -map 0 -pix_fmt rgb24 recording.rgb