Difference between revisions of "AXIOM Beta/Raw Recording"

Revision as of 13:29, 17 January 2022

1 Overview

Simplified: The AXIOM Recorder is a small computer connected to the AXIOM Beta via HDMI, we use the HDMI video stream to transport our own custom raw image data from the camera. This stream needs to be repacked or post processed on the AXIOM Recorder to acquire the actual raw image sequence (RAW12 sequence). First software prototypes did this after recording, newer versions do it on-the-fly while recording.

Technical: On the AXIOM Beta we utilize a Full HD (1920 x 1080) HDMI output stream with twice the frame-rate to store 4 pixels (12 bit each) from the image sensor (sensels) in 2 RGB pixels (24 bit each) in the HDMI stream. After recording this data through a HDMI capture device like the Magewell HDMI USB3 Gen 2 on a PC, Laptop or Single Board Computer (SBC) the data needs to be post processed to obtain the original raw data from the image sensor again. The two HDMI images from a single raw image are called A-Frame and B-Frame.

2 Recording Principle

3 Pixel Packing V3 Structure

Traditional HDMI signal:

AXIOM Beta raw mode output:

Essentially from a 2x2 sensel block on the image sensor R, G1 go into the A frame and G2, B into the B frame of a HDMI stream.

Images from the HDMI stream look like this then:

4 Post Processing Principle

Version 1 Prototype:

Notes:

Content of single *.rgb file and *.frame sequence should - if everything worked correctly - be 100% bit identical. So *.rgb file could be deleted after *.frame sequence extraction.

5 Recorder Hardware

Basically any PC with USB3 and a fast solid state memory should work fine. So far we have tested 3 devices. The goals are in general to have an as small as possible form factor, as low as possible power consumption (as we will be operating on batteries often) but enough speed/performance to reliably capture footage (also for longer durations) and process / playback the image sequences smoothly. Obviously any recent laptop/notebook or desktop PC will work well but here we want to focus on the much smaller form factors.

5.1 Intel NUC

recommended for AXIOM Recorder use: Yes
Tested an Intel NUC NUC7i5BNK, that's a 2017 model for around 400€
CPU: i5-7260U (Passmark 3868, TDP: 15 W)
Dimensions: approximately 4 × 4 inches (101.6 × 101.6 mm)
M.2 slot with PCIe x4 lanes

5.2 Rockpi 4 Model C

recommended for AXIOM Recorder use: No
4GB RAM (soldered)
CPU: Rockchip RK3399
Dimensions: 85 × 54mm (the smallest of the 3)
M.2 PCIe slot for SSD with four lanes PCIe 2.0
Unfortunately we could not get the M.2 SSD to perform fast enough for our purposes on the Rockpi 4 - maybe the PCIe 2.0 is the bottleneck.
Retail price: 60$

5.3 Lattepanda Alpha 864s

recommended for AXIOM Recorder use: Yes
CPU: Intel 8th m3--8100y (~4000 passmark claimed by manufacturer, TDP 15W)
8GB RAM (soldered)
Dimensions: 115 × 78 × 14 mm (2nd most compact of the 3 devices)
active cooling option is very silent, entirely passive cooling available from manufacturer
has onboard 64GB flash memory for operating system
Retail price: ~400€

5.4 Magewell USB HDMI Gen 2

The exact model we use is called “Magewell USB Capture HDMI Gen 2” Part number: 32060. It retails for around 350-400€ and can be bought used for around 250€, we discovered offers as low as 100$ on ebay.

Note: there are clones (eg. on Amazon) that look almost exactly like the Magewell device and promise similar features:

We were curious and bought one. Unfortunately the electronics inside are totally different and we could not even get the device to display a HDMI signal from the AXIOM Beta so we can not recommend those.

6 Requirements and Introduction

In addition to the AXIOM Beta with 1 or 2 HDMI plugin modules you need a USB Capture HDMI Gen 2 device from Magewell: https://www.magewell.com/products/usb-capture-hdmi-gen-2 connected to the HDMI output of the AXIOM Beta.

A PC/laptop/SBC connected to the AXIOM Beta is also required (see Recorder Hardware above), all instructions here assume this PC/laptop/SBC is running Linux - other operating systems are not supported by this guide - they might work - we have not tested them.

Commands to be run on the Linux PC are indicated like:

$ command

Commands to be run on the AXIOM Beta are indicated like:

[root@beta ~]# command

6.1 AXIOM Beta firmware flashing

For a full guide about firmware flashing/backup see: Firmware Flashing

6.2 Determine Magewell Video Device ID

Install v4l2-ctl:

$ sudo apt install v4l2-ctl

Run the tool:

$ v4l2-ctl --list-devices

This should output a list, in this case telling you that the magewell device is /dev/video2 and /dev/video3 (the internal webcam is /dev/video0 and /dev/video1):

USB Capture HDMI: USB Capture H (usb-0000:00:14.0-2):
    /dev/video2
    /dev/video3

Integrated Camera: Integrated C (usb-0000:00:1a.0-1.6):
    /dev/video0
    /dev/video1

6.3 Download, Install & Run Magewell Setup Tool

This raw recording method relies on a USB HDMI capture device like the one produced by Magewell that can capture 1920x1080 at 60 FPS without any compression or image manipulation. .

download setup tool from magewell website: https://www.magewell.com/downloads/usb-capture#/tools/linux-ubuntu

run:

$ sudo usbcaptureutility

under video set:

input color format: RGB
input quantization: full-range

capture color format: RGB
capture quantization: full-range
capture saturation: full-range
Specific Effect: Flip

under advanced set:

color format: only leave RGB24 on the left, move all other options to the right
frame rate: only leave 60.00 on the left, move all other options to the right
resolution, only leave 1920x1080 on the left, move all other options to the right

click “save to device”, disconnect and reconnect device

Settings will be stored in the device even when you disconnect it so these steps only need to be completed once.

6.4 Check Transmission Data Integrity (optional)

To test if the magewell device is receiving data from the AXIOM Beta in a reliable way (good cables are essential for example) the following method can be used:

assuming the magewell input is /dev/videoX:

using Firmware 1.0:

init raw mode on AXIOM Beta

[root@beta ~]# ./kick_manual_raw.sh

stop the acquisition with:

[root@beta ~]# fil_reg 15 0

then load a gradient overlay:

[root@beta ~]# ./mimg -T4

disable overlay:

[root@beta ~]# ./mimg -O -P0

Reset gamma and color matrix settings:

[root@beta ~]# ./mat4_conf.py 1
# ./gamma_conf.sh 1

on magewell connected pc run (assuming the magewell input is /dev/videoX):

$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null

this should output many lines containing the hash:

ddbfb96e4b8d5191bb274885ccc0264d

then let it run for 15 minutes with (assuming the magewell input is /dev/videoX):

$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null | grep -v ddbfb96e4b8d5191bb274885ccc0264d

no hashes should be reported, if hashes are reported this could mean a bad cable/bad connection

while the hash test is running on the camera depending on the HDMI plugin module being connected to the top plugin module slot set:

[root@beta ~]# scn_reg 31 0x5000

and then:

[root@beta ~]# scn_reg 31 0x4000

no different hashes should be reported still or the bottom plugin module slot:

[root@beta ~]# scn_reg 30 0x5000

and then:

[root@beta ~]# scn_reg 30 0x4000

no different hashes should be reported still

now stop the hash comparing line.

7 Usage Instructions

7.1 Initialize Raw Mode on AXIOM Beta

Firmware 1.0:

[root@beta ~]# ./kick_manual_raw.sh

Firmware 2.0:

[root@beta ~]# axiom_start.sh raw

7.2 Clear Overlay

Firmware 1.0:

[root@beta ~]# ./mimg -a -o -P 0

7.3 Check HDMI Stream

assuming the magewell input is /dev/videoX:

$ ffplay /dev/videoX

7.4 Frame-rate Verification

on AXIOM Beta run the performance tool: Firmware 1.0:

[root@beta ~]# ./cmv_perf3

Firmware 2.0:

[root@beta ~]# axiom_perf

For 30 FPS this should output something similar to:

mapped 0x60000000+0x00400000 to 0x60000000.
mapped 0x80000000+0x00400000 to 0x80000000.
hdmi: 60.27 FPS     cseq: 29.97 FPS

If the reported FPS are lower you need to reduce the exposure time (run performance tool again afterwards to check if the changes had the desired effect):

Firmware 1.0:

[root@beta ~]# ./cmv_snap3 -e 11ms -z

Firmware 2.0:

[root@beta ~]# axiom_snap -e 11ms -z

7.5 Raw HDMI Related Commands

The first parameter of scn_reg defines which plugin module/shield slot is affected:

31 - top (North) HDMI plugin module 
30 - bottom (South) HDMI plugin module 
29 - HDMI Shield

set bottom HDMI plugin module to: live view:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x0000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x0000

set bottom HDMI plugin module to: A channel / A frames:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x5000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x5000

set bottom HDMI plugin module to: B channel / B frames:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x6000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x6000

set bottom HDMI plugin module to: A/B frames alternating:

Firmware 1.0:

[root@beta ~]# scn_reg 30 0x7000

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 30 0x7000

activates the four corner markers (all 4 corner pixels are identical):

Firmware 1.0:

[root@beta ~]# ./raw_mark.sh

Firmware 2.0:

[root@beta ~]# axiom_raw_mark.sh

red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and see if no frames were dropped during transfer/capture
green channel: wrsel buffer information
blue channel: 0x55 / 85 (A-frame) and 0xAA / 170 (B-frame)

Note: Activating the corner markers is essential for post processing the captured files later so do not forget to do this.

set corner marker frame counter max to 256:

Note: This is essential for post processing the captured files later so do not forget to do this.

Firmware 1.0:

[root@beta ~]# scn_reg 2 0x100

Firmware 2.0:

[root@beta ~]# axiom_scn_reg 2 0x100

7.6 Record Raw Video with ffmpeg

$ ffmpeg -i /dev/videoX -map 0 -pix_fmt rgb24 recording.rgb

7.7 Extract Image Sequence

To extract the individual frames from a single *.rgb file after recording do:

$ split recording.rgb recording_frame_ --additional-suffix=.frame -d -b 6220800 --verbose --suffix-length=5

6220800 here corresponds to the file size in bytes of one frame (1920 x 1080 x 24bit / 8)

7.8 Corner Markers

There are 4 corner markers in a *.frame:

red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and see if no frames were dropped during transfer/capture
green channel: wrsel buffer information
blue channel: 0x55 / 85 (A-frame) and 0xAA / 170 (B-frame)

Read corner marker pixel value from a *.frame file:

$ head -c 3 recording_frame_00001.frame | od -t x1

Note the order (of the hexadecimal values):

0000000 43 04 AA 
        R  G  B

AA = indicates this is a B-frame in blue channel
43 = frame counter in red channel

7.9 Recover raw12 from A+B frames

We created a small tool for this called frame-merge, prerequisite, compile & install from source: https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw-via-hdmi/frame-merge

Usage to convert one A+B frame pair to a raw12:

$ frame-merge a-frame.frame b-frame.frame output.raw12

7.10 Convert raw12 to DNG

prerequisite, compile & install from source: https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw2dng

Convert a raw12 to DNG:

$ raw2dng output.raw12 --width=3840 --height=2160 --bayer-order=2

7.11 Identify skipped/double frames in raw recording

There can be several reasons for skipped frames:

bad HDMI/USB cable
recording media too slow
Recorder CPU at full load or doing other stuff on the side (depends on CPU and system of course)

We created a script that can be used to show framecounter (and that way identify missing/double frames) and A/B Frame identification of all files in a folder: https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw-via-hdmi/frame-info

Usage example:

$ python3 frame-info.py -i myfolder/

Example output:

*.frame files found in folder: 698

==========================================================
filename              framecounter      A/B frame
==========================================================
...
test_frame_00077.bgr        142            A-Frame
test_frame_00078.bgr        143            B-Frame
test_frame_00079.bgr        144            A-Frame
test_frame_00080.bgr        145            B-Frame
test_frame_00081.bgr        145            B-Frame     framecounter missmatch
test_frame_00082.bgr        146            A-Frame
test_frame_00083.bgr        147            B-Frame
test_frame_00084.bgr        148            A-Frame
test_frame_00085.bgr        149            B-Frame
...

8 4096 Pixel Width Raw Mode

The HDMI signal can go up to 2048 width so with reconstructing raw data we can achieve 4096 original image sensor width that way.

This allows to get the 8 pixels black coloumns into the recorded frame and enables advanced noise compensation, make sure to enable the black coloumns first though (cmv_snap3 / axiom_snap)

In the Magewell Setup Tool (usbcaptureutility) set 2048x1080 as only resolution on the left column, then save to device and reconnect device.

On the AXIOM Beta run:

Firmware 1.0:

memtool -1 -F 0x0 0x18000000 0x8000000
./kick_manual_raw.sh
./gen_init_hdmi.sh 2048x1080p50 or ./gen_init_hdmi.sh 2048x1080p60
./data_init_hdmi.sh
./rmem_conf.sh 0
./raw_mark.sh

Firmware 2.0:

memtool -1 -F 0x0 0x18000000 0x8000000
axiom_start.sh raw
axiom_gen_init_hdmi.sh 2048x1080p50 or axiom_gen_init_hdmi.sh 2048x1080p60
axiom_data_init_hdmi.sh
axiom_rmem_conf.sh 0
axiom_raw_mark.sh

To view the HDMI signal on the connected magewell run:

$ffplay -video_size 2048x1080 /dev/videoX

9 raw12 Image Viewer

We created a python based GUI (https://github.com/apertus-open-source-cinema/misc-tools-utilities/blob/hdmi-raw-2021/raw-via-hdmi/scripts/viewer.py) that can load and display raw12 preview image rather quickly. The focus here is not accurate color representation or high quality conversion but rather to quickly be able to see the content of an image. For high grade processing & conversion use the raw2dng utility (https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw2dng) instead.

Set up filetype association see Raw12_viewer

10 raw12 Sequence Playback

Installation:

sudo apt install cmake
git clone https://github.com/apertus-open-source-cinema/axiom-recorder
cd axiom-recorder
cargo build --release --all

Running:

cargo run --release  ! RawDirectoryReader --file-pattern '/path/to/your/*.raw12' --bit-depth 12 --height 2160 --width 3840 --first-red-x true --first-red-y false --fps 30 ! GpuBitDepthConverter ! Debayer ! Display

11 Output Latency Tests

Tests were conducted in 24 fps mode.

Test 1: Direct HDMI output from the AXIOM Beta:
Delay is 2 frames (1 from camera internal buffering, 1 from the Monitor)

Test 2: Color HDMI output through Magewell USB3 device on Lattepanda Alpha 864S, displayed with ffplay:
Delay: 5 frames

Test 3: Raw HDMI output through Magewell USB3 device on Lattepanda Alpha 864S, displayed with live debayering in axiom-recorder software
Delay: 7 - 16 frames: