View source for AXIOM Beta/Raw Recording

==Overview==
'''Simplified:'''
The AXIOM Recorder is a small computer connected to the AXIOM Beta via HDMI, we use the HDMI video stream to transport our own custom raw image data from the camera. This stream needs to be repacked or post processed on the AXIOM Recorder to acquire the actual raw image sequence ([[RAW12]] sequence). First software prototypes did this after recording, newer versions do it on-the-fly while recording.

'''Technical:'''
On the AXIOM Beta we utilize a Full HD (1920 x 1080) HDMI output stream with twice the frame-rate to store 4 pixels (12 bit each) from the image sensor (sensels) in 2 RGB pixels (24 bit each) in the HDMI stream. After recording this data through a HDMI capture device like the Magewell HDMI USB3 Gen 2 on a PC, Laptop or Single Board Computer (SBC) the data needs to be post processed to obtain the original raw data from the image sensor again.
The two HDMI images from a single raw image are called A-Frame and B-Frame.

==Recording Principle==
[[File:Axiom_recorder_raw_recording_principle.png]]

==Pixel Packing V3 Structure==

'''Traditional HDMI signal:'''

[[File:Hdmi_output_signal.png]]

'''AXIOM Beta raw mode output:'''

[[File:Axiom_raw_hdmi_output_signal.png]]

Essentially from a 2x2 sensel block on the image sensor R, G1 go into the A frame and G2, B into the B frame of a HDMI stream.

'''Images from the HDMI stream look like this then:'''

{|
|-
| [[File:Axiom_raw_hdmi_a-frame.jpg | 400px]]
| [[File:Axiom_raw_hdmi_b-frame.jpg.jpg | 400px]]
|}


==Post Processing Principle==

'''Version 1 Prototype:'''

[[File:Axiom_beta_raw_recording_principle.png]]

Notes:
* Content of single *.rgb file and *.frame sequence should - if everything worked correctly - be 100% bit identical. So *.rgb file could be deleted after *.frame sequence extraction.

==Recorder Hardware==

Basically any PC with USB3 and a fast solid state memory should work fine. So far we have tested 3 devices. The goals are in general to have an as small as possible form factor, as low as possible power consumption (as we will be operating on batteries often) but enough speed/performance to reliably capture footage (also for longer durations) and process / playback the image sequences smoothly. Obviously any recent laptop/notebook or desktop PC will work well but here we want to focus on the much smaller form factors.

===Raspberry Pi 5===

The Raspberry Pi 5 is a rather powerful but still cheap widely available single board computer. It provides 2 MIPI interfaces and a PCIe interface through flex PCB connectors.

With the Geekworm C790 Raspberry Pi HDMI CSI CSI-2 Adapter we can connect two HDMI Input interfaces to the raspberry pi and with an NVME SSD shield attach fast solid state memory to the system.
We succesfully tested with the Geekworm X1002 PCIe M.2 NVMe SSD PIP PCIe Peripherie Bottom Board for Raspberry Pi 5. 

===Intel NUC===

[[File:Nuc.png | 400px]]

recommended for AXIOM Recorder use: Yes<br />
Tested an Intel NUC NUC7i5BNK, that's a 2017 model for around 400€<br />
CPU: i5-7260U (Passmark 3868, TDP: 15 W)<br />
Dimensions: approximately 4 × 4 inches (101.6 × 101.6 mm)<br />
M.2 slot with PCIe x4 lanes<br />

===Rockpi 4 Model C===

[[File:Rockpi4.png | 400px]]

recommended for AXIOM Recorder use: No<br />
4GB RAM (soldered)<br />
CPU: Rockchip RK3399<br />
Dimensions: 85 × 54mm (the smallest of the 3)<br />
M.2 PCIe slot for SSD with four lanes PCIe 2.0<br />
Unfortunately we could not get the M.2 SSD to perform fast enough for our purposes on the Rockpi 4 - maybe the PCIe 2.0 is the bottleneck.<br />
Retail price: 60$<br />

===Lattepanda Alpha 864s===

[[File:Lattepanda.png | 400px]]

recommended for AXIOM Recorder use: Yes<br />
CPU: Intel 8th m3--8100y (~4000 passmark claimed by manufacturer, TDP 15W)<br />
8GB RAM (soldered)<br />
Dimensions: 115 × 78  × 14 mm (2nd most compact of the 3 devices)<br />
active cooling option is very silent, entirely passive cooling available from manufacturer<br />
has onboard 64GB flash memory for operating system<br />
Retail price: ~400€<br />

===LattePanda 3 Delta===

Similar form factor as Lattepanda Alpha 864s and similar performance but slightly cheaper.<br />
Dimensions: 125 x 78 x 16mm<br />
1x M.2 M Key, PCIe 3.0 2x, Supports NVMe SSD<br />
Retail price: $279.00<br />
Intel® Celeron® N5105<br /> 
recommended for AXIOM Recorder use: needs to be be tested.

===Rock 5 Model B===

Supports PCle 3.0 x4 NVMe SSD<br />
HDMI in support up to 4K@60FPS - does this make the magewell obsolete? -> needs to be tested!<br />
Dimensions: 100 x 72mm<br />
Retail price: starting at ~200€ for 8GB RAM version<br />
recommended for AXIOM Recorder use: needs to be be tested.

===Magewell USB HDMI Gen 2===

[[File:Magewell_hdmi_usb3.png | 400px]]

The exact model we use is called “Magewell USB Capture HDMI Gen 2” Part number: 32060.
It retails for around 350-400€ and can be bought used for around 250€, we discovered offers as low as 100$ on ebay.

'''Note:''' there are clones (eg. on Amazon) that look almost exactly like the Magewell device and promise similar features:

[[File:Magewell_clone.png | 400px]]

We were curious and bought one. Unfortunately the electronics inside are totally different and we could not even get the device to display a HDMI signal from the AXIOM Beta so we can not recommend those.

==Requirements and Introduction==
In addition to the AXIOM Beta with 1 or 2 HDMI plugin modules you need a USB Capture HDMI Gen 2 device from Magewell: https://www.magewell.com/products/usb-capture-hdmi-gen-2 connected to the HDMI output of the AXIOM Beta.

A PC/laptop/SBC connected to the AXIOM Beta is also required (see Recorder Hardware above), all instructions here assume this PC/laptop/SBC is running Linux - other operating systems are not supported by this guide - they might work - we have not tested them.

Commands to be run on the Linux PC are indicated like:
 <pre class="code">$ command</pre>

Commands to be run on the AXIOM Beta are indicated like:
 <pre class="code">[root@beta ~]# command</pre>

===AXIOM Beta firmware flashing===

For a full guide about firmware flashing/backup see: [[AXIOM_Beta/Firmware_Flashing | Firmware Flashing]]

===Determine Magewell Video Device ID===

Install v4l2-ctl:
 <pre class="code">$ sudo apt install v4l2-ctl</pre>

Run the tool:
 <pre class="code">$ v4l2-ctl --list-devices</pre>


This should output a list, in this case telling you that the magewell device is /dev/video2 and /dev/video3 (the internal webcam is /dev/video0 and /dev/video1):
 <pre class="code">USB Capture HDMI: USB Capture H (usb-0000:00:14.0-2):
    /dev/video2
    /dev/video3

Integrated Camera: Integrated C (usb-0000:00:1a.0-1.6):
    /dev/video0
    /dev/video1</pre>

===Download, Install & Run Magewell Setup Tool===
This raw recording method relies on a USB HDMI capture device like the one produced by Magewell that can capture 1920x1080 at 60 FPS without any compression or image manipulation. .

download setup tool from magewell website: https://www.magewell.com/downloads/usb-capture#/tools/linux-ubuntu

run:
 <pre class="code">$ sudo usbcaptureutility</pre>

under '''video''' set:
* input color format: RGB
* input quantization: full-range

* capture color format: RGB
* capture quantization: full-range
* capture saturation: full-range
* Specific Effect: Flip

under '''advanced''' set:
* color format: only leave RGB24 on the left, move all other options to the right
* frame rate: only leave 60.00 on the left, move all other options to the right
* resolution, only leave 1920x1080 on the left, move all other options to the right
[[File:Magewell_setup_tool.png]]

click “save to device”, disconnect and reconnect device

Settings will be stored in the device even when you disconnect it so these steps only need to be completed once.

===Check Transmission Data Integrity (optional)===

To test if the magewell device  is receiving data from the AXIOM Beta in a reliable way (good cables are essential for example) the following method can be used:

assuming the magewell input is ''/dev/videoX'':

using Firmware 1.0:

init raw mode on AXIOM Beta
 <pre class="code">[root@beta ~]# ./kick_manual_raw.sh</pre>

stop the acquisition with: 
 <pre class="code">[root@beta ~]# fil_reg 15 0</pre>

then load a gradient overlay: 
 <pre class="code">[root@beta ~]# ./mimg -T4</pre>

disable overlay: 
 <pre class="code">[root@beta ~]# ./mimg -O -P0</pre>

Reset gamma and color matrix settings:
 <pre class="code">[root@beta ~]# ./mat4_conf.py 1
# ./gamma_conf.sh 1</pre>

on magewell connected pc run (assuming the magewell input is /dev/videoX): 
 <pre class="code">$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null</pre>

this should output many lines containing the hash:
 <pre class="code">ddbfb96e4b8d5191bb274885ccc0264d</pre>

then let it run for 15 minutes with (assuming the magewell input is /dev/videoX):
 <pre class="code">$ ffmpeg -report -i /dev/videoX -map 0 -f framehash -hash md5 - 2>/dev/null | grep -v ddbfb96e4b8d5191bb274885ccc0264d</pre>

no hashes should be reported, if hashes are reported this could mean a bad cable/bad connection

while the hash test is running on the camera depending on the HDMI plugin module being connected to the top plugin module slot set:
 <pre class="code">[root@beta ~]# scn_reg 31 0x5000</pre>
and then:
 <pre class="code">[root@beta ~]# scn_reg 31 0x4000</pre>
no different hashes should be reported still
or the bottom plugin module slot:
 <pre class="code">[root@beta ~]# scn_reg 30 0x5000</pre>
and then:
 <pre class="code">[root@beta ~]# scn_reg 30 0x4000</pre>
no different hashes should be reported still

now stop the hash comparing line.

==Usage Instructions==

===Initialize Raw Mode on AXIOM Beta===

Firmware 1.0:
 <pre class="code">[root@beta ~]# ./kick_manual_raw.sh</pre>

Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_start.sh raw</pre>

===Clear Overlay===

Firmware 1.0:
 <pre class="code">[root@beta ~]# ./mimg -a -o -P 0</pre>

Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_mimg -a -o -P 0</pre>

===Check HDMI Stream===

assuming the magewell input is /dev/videoX:
 <pre class="code">$ ffplay /dev/videoX</pre>

===Frame-rate Verification===

on AXIOM Beta run the performance tool:
Firmware 1.0:
 <pre class="code">[root@beta ~]# ./cmv_perf3</pre>

Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_perf</pre>

For 30 FPS this should output something similar to:
 <pre class="code">mapped 0x60000000+0x00400000 to 0x60000000.
mapped 0x80000000+0x00400000 to 0x80000000.
hdmi: 60.27 FPS     cseq: 29.97 FPS</pre>

If the reported FPS are lower you need to reduce the exposure time (run performance tool again afterwards to check if the changes had the desired effect):

Firmware 1.0:
 <pre class="code">[root@beta ~]# ./cmv_snap3 -e 11ms -z</pre>

Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_snap -e 11ms -z</pre>

===Raw HDMI Related Commands===

The first parameter of scn_reg defines which plugin module/shield slot is affected:
 <pre class="code">31 - top (North) HDMI plugin module 
30 - bottom (South) HDMI plugin module 
29 - HDMI Shield</pre>


'''set bottom HDMI plugin module to: live view:'''

Firmware 1.0:
 <pre class="code">[root@beta ~]# scn_reg 30 0x0000</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_scn_reg 30 0x0000</pre>


'''set bottom HDMI plugin module to: A channel / A frames:'''

Firmware 1.0:
 <pre class="code">[root@beta ~]# scn_reg 30 0x5000</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_scn_reg 30 0x5000</pre>


'''set bottom HDMI plugin module to: B channel / B frames:'''

Firmware 1.0:
 <pre class="code">[root@beta ~]# scn_reg 30 0x6000</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_scn_reg 30 0x6000</pre>


'''set bottom HDMI plugin module to: A/B frames alternating:'''

Firmware 1.0:
 <pre class="code">[root@beta ~]# scn_reg 30 0x7000</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_scn_reg 30 0x7000</pre>


'''activates the four corner markers (all 4 corner pixels are identical):'''

Firmware 1.0:
 <pre class="code">[root@beta ~]# ./raw_mark.sh</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_raw_mark.sh</pre>


* red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and verify that no frames were dropped during transfer/capture
* green channel: wrsel buffer information (write select and read select buffers - 4 bit total) wrsel <= wbuf_sel & rbuf_sel (VHDL) split with upper/lower nibble
* blue channel: 0x55 / 85  (A-frame) and 0xAA / 170 (B-frame)

Note: Activating the corner markers is essential for post processing the captured files later so do not forget to do this.


'''set corner marker frame counter max to 256:'''

Note: This is essential for post processing the captured files later so do not forget to do this.

Firmware 1.0:
 <pre class="code">[root@beta ~]# scn_reg 2 0x100</pre>
Firmware 2.0:
 <pre class="code">[root@beta ~]# axiom_scn_reg 2 0x100</pre>

===Record Raw Video with ffmpeg===

 <pre class="code">$ ffmpeg -i /dev/videoX -map 0 -pix_fmt rgb24 recording.rgb</pre>

===Extract Image Sequence===

To extract the individual frames from a single *.rgb file after recording do:

 <pre class="code">$ split recording.rgb recording_frame_ --additional-suffix=.frame -d -b 6220800 --verbose --suffix-length=5</pre>

6220800 here corresponds to the file size in bytes of one frame (1920 x 1080 x 24bit / 8)

===Corner Markers===

There are 4 corner markers in a *.frame:
*red channel: contains 8 bit frame counter to identify A-frames (even) and B-frames (odd) and see if no frames were dropped during transfer/capture
*green channel: wrsel buffer information
*blue channel: 0x55 / 85  (A-frame) and 0xAA / 170 (B-frame)

Read corner marker pixel value from a *.frame file:
 <pre class="code">$ head -c 3 recording_frame_00001.frame | od -t x1</pre>

Note the order (of the hexadecimal values):
 <pre class="code">0000000 43 04 AA 
        R  G  B

AA = indicates this is a B-frame in blue channel
43 = frame counter in red channel
</pre>

===Recover raw12 from A+B frames===
We created a small tool for this called frame-merge, prerequisite, compile & install from source: 
https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw-via-hdmi/frame-merge

Usage to convert one A+B frame pair to a raw12:

 <pre class="code">$ frame-merge a-frame.frame b-frame.frame output.raw12</pre>

===Convert raw12 to DNG===

prerequisite, compile & install from source: https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw2dng

Convert a raw12 to DNG:
 <pre class="code">$ raw2dng output.raw12 --width=3840 --height=2160 --bayer-order=2</pre>


===Identify skipped/double frames in raw recording===
There can be several reasons for skipped frames:
*bad HDMI/USB cable
*recording media too slow
*Recorder CPU at full load or doing other stuff on the side (depends on CPU and system of course)

We created a script that can be used to show framecounter (and that way identify missing/double frames) and A/B Frame identification of all files in a folder:
https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw-via-hdmi/frame-info

Usage example:
 <pre class="code">$ python3 frame-info.py -i myfolder/</pre>

Example output:
 <pre class="code">*.frame files found in folder: 698

==========================================================
filename              framecounter      A/B frame
==========================================================
...
test_frame_00077.bgr        142            A-Frame
test_frame_00078.bgr        143            B-Frame
test_frame_00079.bgr        144            A-Frame
test_frame_00080.bgr        145            B-Frame
test_frame_00081.bgr        145            B-Frame     framecounter missmatch
test_frame_00082.bgr        146            A-Frame
test_frame_00083.bgr        147            B-Frame
test_frame_00084.bgr        148            A-Frame
test_frame_00085.bgr        149            B-Frame
...</pre>

==4096 Pixel Width Raw Mode==
The HDMI signal can go up to 2048 width so with reconstructing raw data we can achieve 4096 original image sensor width that way.

This allows to get the 8 pixels black coloumns into the recorded frame and enables advanced noise compensation, make sure to enable the black coloumns first though (cmv_snap3 / axiom_snap)

In the Magewell Setup Tool (usbcaptureutility) set 2048x1080 as only resolution on the left column, then save to device and reconnect device.

On the AXIOM Beta run:

Firmware 2.0:
 <pre class="code">sudo su</pre>

 <pre class="code">memtool -1 -F 0x0 0x18000000 0x8000000
axiom_start.sh raw
axiom_gen_init_hdmi.sh 2048x1080p50 or axiom_gen_init_hdmi.sh 2048x1080p60
axiom_data_init_hdmi.sh
axiom_rmem_conf.sh 0
axiom_scn_reg 30 0x7000
axiom_raw_mark.sh
axiom_scn_reg 2 0x100</pre>

To view the HDMI signal on the connected magewell run:
  <pre class="code">$ffplay -video_size 2048x1080 /dev/videoX</pre>

==raw12 Image Viewer==

[[File:Raw12_viewer.png]]

We created a python based GUI (https://github.com/apertus-open-source-cinema/misc-tools-utilities/blob/hdmi-raw-2021/raw-via-hdmi/scripts/viewer.py) that can load and display raw12 preview image rather quickly. The focus here is not accurate color representation or high quality conversion but rather to quickly be able to see the content of an image. For high grade processing & conversion use the raw2dng utility (https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw2dng) instead.


Set up filetype association see [[Raw12_viewer]]

==raw12 Sequence Playback==

Installation:

  <pre class="code">sudo apt install cmake
git clone https://github.com/apertus-open-source-cinema/axiom-recorder
cd axiom-recorder
cargo build --release --all</pre>

Running:

  <pre class="code">cargo run --release  ! RawDirectoryReader --file-pattern '/path/to/your/*.raw12' --bit-depth 12 --height 2160 --width 3840 --first-red-x true --first-red-y false --fps 30 ! GpuBitDepthConverter ! Debayer ! Display</pre>


==Output Latency Tests==
Tests were conducted in 24 fps mode.

'''Test 1:''' Direct HDMI output from the AXIOM Beta:<br />
Delay is 2 frames (1 from camera internal buffering, 1 from the Monitor)

'''Test 2:''' Color HDMI output through Magewell USB3 device on Lattepanda Alpha 864S, displayed with ffplay:<br />
Delay: 5 frames

'''Test 3:''' Raw HDMI output through Magewell USB3 device on Lattepanda Alpha 864S, displayed with live debayering in axiom-recorder software<br />
Delay: 7 -  16 frames: