Difference between revisions of "Calibration files"

From apertus wiki
Jump to: navigation, search
(Created page with "Proposal: 1. Store calibration files in a subdirectory (axiombeta-calibration) 2. That subdirectory can be placed in a way similar to dcraw's badpixels file: dcraw badpixels...")
 
 
(15 intermediate revisions by one other user not shown)
Line 1: Line 1:
Proposal:
=Proposal=


1. Store calibration files in a subdirectory (axiombeta-calibration)
==Location of calibration files==
2. That subdirectory can be placed in a way similar to dcraw's badpixels file:
 
'''Store calibration files in a subdirectory (axiombeta-calibration)''', located in a way similar to dcraw's badpixels file.


dcraw badpixels (from man page):
dcraw badpixels (from man page):


./.badpixels, ../.badpixels, ../../.badpixels, ...
./.badpixels, ../.badpixels, ../../.badpixels, ...


beta calibration files
beta calibration files
Line 14: Line 15:
or maybe also ~/axiombeta-calibration ?
or maybe also ~/axiombeta-calibration ?


3. Inside that subdirectory, we can have a generic profile (one size fits all) and various profiles for different setting combinations (that can cover either just one particular situation, or can also be generic to some degree):
==Types of calibration==
 
We will use two types of calibration: '''sensor profiling''' (e.g. dark calibration, response curves, gain corrections, all that's needed to get a linear radiometric response) and '''color profiling''' (with argyll, for example), so we'll add two subdirectories:
<pre style="white-space: pre-wrap">
axiombeta-calibration/sensor-profiles
axiombeta-calibration/color-profiles
</pre>
In theory, these two things (sensor profiling and color profiling) should be independent (if they are not, we probably did a poor job with calibrating the sensor).
 
==Sensor profiles==


axiombeta-calibration/generic/
====Directory structure====
axiombeta-calibration/gain1_expo1-50/
axiombeta-calibration/gain1_expo100-1000/
axiombeta-calibration/blacksun8192_expo100-1000/


Inside that subdirectory, we can have a '''generic profile''' (one size fits all) and various '''profiles for different setting combinations''' (that can cover either just one particular situation, or can also be generic to some degree):
<pre style="white-space: pre-wrap">
axiombeta-calibration/sensor-profiles/generic/
axiombeta-calibration/sensor-profiles/gain1_expo1-50/
axiombeta-calibration/sensor-profiles/gain1_expo100-1000/
axiombeta-calibration/sensor-profiles/blacksun8192_expo100-1000/
axiombeta-calibration/sensor-profiles/gain1_expo100-1000_plr32-10ms-24-1ms/
</pre>
That means, from all the settings that might affect the calibration, we can build an unique name from the settings that differ from the generic profile. A calibration might cover files at some particular setting value, or some range of values (detected from the supplied input files during calibration).
That means, from all the settings that might affect the calibration, we can build an unique name from the settings that differ from the generic profile. A calibration might cover files at some particular setting value, or some range of values (detected from the supplied input files during calibration).


Line 27: Line 42:
Or, we may want to test the profile against some validation data, store the results, and pick the profile likely to minimize the calibration errors (some solid math skills required for this one).
Or, we may want to test the profile against some validation data, store the results, and pick the profile likely to minimize the calibration errors (some solid math skills required for this one).


4. Inside one profile, we can have the following calibration files (all optional; if missing, no correction is done for that specific parameter):
====Sensor calibration files====
 
'''Inside one profile, we can have the following calibration files''' (all optional; if missing, that calibration step will be skipped):


For black calibration:
For [[Black Calibration]]:
  beforegain.pgm : static offsets whose source is before the gain amplifiers (that means, will be multiplied by gain, then subtracted)
<pre style="white-space: pre-wrap">
  beforegain.pgm : static offsets whose source is before the gain amplifiers
                (that means, will be multiplied by gain, then subtracted)
  aftergain.pgm  : static offsets whose source is after the gain amplifiers
  aftergain.pgm  : static offsets whose source is after the gain amplifiers
  darkcurrent.pgm: dynamic offsets dependent on many variables (exposure, temperature, gain), whose scaling factor can be measured from hot pixels (idea from [http://www.photonics.com/Article.aspx?AID=44298])
  darkcurrent.pgm: dynamic offsets dependent on many variables (exposure, temperature, gain),
                  whose scaling factor can be measured from hot pixels
                  (idea from [http://www.photonics.com/Article.aspx?AID=44298])
</pre>
These 3 files allow computing a dark frame for every gain/expo/temperature combination (at least in theory), and should generalize fairly well to other settings (to be tested).
 
The computed dark frame would be:
beforegain * gain + aftergain + darkcurrent * darkcurrent_scaling
 
where darkcurrent_scaling can be measured from hot pixels.


These 3 files allow computing a dark frame for every gain/expo/temperature combination (at least in theory), and should generalize fairly well to other settings (to be tested).
Therefore, a simple dark frame (without any computation) can be saved in aftergain.pgm. If that's confusing, should we add darkframe.pgm as a synonym?


Other calibration files:
Other calibration files (e.g. [[Response Curves]], [[Pattern Noise]] corrections):
<pre style="white-space: pre-wrap">
  gainframe.pgm  : per-pixel gain variations
  gainframe.pgm  : per-pixel gain variations
  clipframe.pgm  : clipping offsets (might be changed)
  clipframe.pgm  : clipping offsets
  responseXXXX.pgm: per-pixel response curve data point (raw sensor outputs that should give the corrected value XXXX)
  responseXXXX.pgm: per-pixel response curve data point,
                  (raw sensor outputs that should give the corrected value XXXX)
                  (these might replace gain/clip frames completely, need to experiment)
  lut.spi1d      : overall or per-channel response curve (shaper)
  lut.spi1d      : overall or per-channel response curve (shaper)
</pre>
==Color profiles==


5. Color calibration files should be independent of sensor profiling (if they are not, we probably did a poor job with calibrating the sensor), so we'll place them in a different directory:
'''Color profiles are stored under this subdirectory:'''


  axiombeta-calibration/color-profiles/
  axiombeta-calibration/color-profiles/


These depend on other things, like ambient light and maybe lens used, so we could think about something like:
These depend on other things, like ambient light and maybe lens used, so we could think about something like:
 
<pre style="white-space: pre-wrap">
  axiombeta-calibration/color-profiles/D50/
  axiombeta-calibration/color-profiles/D50/
  axiombeta-calibration/color-profiles/Tungsten/
  axiombeta-calibration/color-profiles/Tungsten/
Line 53: Line 86:
  axiombeta-calibration/color-profiles/InTheCave/                  # or the location where the calibration was performed
  axiombeta-calibration/color-profiles/InTheCave/                  # or the location where the calibration was performed
  axiombeta-calibration/color-profiles/D50/EF50mm_f/1.8_II/        # from lens metadata
  axiombeta-calibration/color-profiles/D50/EF50mm_f/1.8_II/        # from lens metadata
 
</pre>
Those variables will be entered manually by the user (or chosen from a pickbox from a GUI), except maybe lens metadata, where it can be obtained from the lens itself (when the lens communication will be functional).
Those variables will be entered manually by the user (or chosen from a pickbox from a GUI), except maybe lens metadata, where it can be obtained from the lens itself (when the lens communication will be functional).


6. Color profile formats:
'''Color profile formats:'''


  - ICC (as saved by argyll)
  - ICC (as saved by argyll)
Line 62: Line 95:
  - ...
  - ...


7. A way to identify which sensor or camera is used (by serial numbers) could be very helpful for those who own more than one camera, or more than one sensor.
=Wish list=
 
'''A way to identify which sensor or camera is used (by serial numbers)''' could be very helpful for those who own more than one camera, or more than one sensor.
 
In this case, the directory structure would be be:
<pre style="white-space: pre-wrap">
~/axiombeta-calibration/cam1234/sensor5678/sensor-profiles
~/axiombeta-calibration/color-profiles
</pre>
That's because sensor profiling is expected to account for the differences between sensors (that means, consistent output), so color profiles should be interchangeable between cameras/sensors (and should only account for lighting/lens differences).
 
Also, with a serial number, we no longer have to use tricks like having to search for the calibration subdirectory up 1/2/3/n levels from the current location.
 
Proposal from danieel:
 
~/.config/axiom/beta/calib/
 
also: "if the data is of low entropy, then just pack the multi-file structure into one .tgz"

Latest revision as of 20:05, 2 March 2018

1 Proposal

1.1 Location of calibration files

Store calibration files in a subdirectory (axiombeta-calibration), located in a way similar to dcraw's badpixels file.

dcraw badpixels (from man page):

./.badpixels, ../.badpixels, ../../.badpixels, ...

beta calibration files

./axiombeta-calibration ../axiombeta-calibration ../../axiombeta-calibration

or maybe also ~/axiombeta-calibration ?

1.2 Types of calibration

We will use two types of calibration: sensor profiling (e.g. dark calibration, response curves, gain corrections, all that's needed to get a linear radiometric response) and color profiling (with argyll, for example), so we'll add two subdirectories:

 axiombeta-calibration/sensor-profiles
 axiombeta-calibration/color-profiles

In theory, these two things (sensor profiling and color profiling) should be independent (if they are not, we probably did a poor job with calibrating the sensor).

1.3 Sensor profiles

1.3.1 Directory structure

Inside that subdirectory, we can have a generic profile (one size fits all) and various profiles for different setting combinations (that can cover either just one particular situation, or can also be generic to some degree):

 axiombeta-calibration/sensor-profiles/generic/
 axiombeta-calibration/sensor-profiles/gain1_expo1-50/
 axiombeta-calibration/sensor-profiles/gain1_expo100-1000/
 axiombeta-calibration/sensor-profiles/blacksun8192_expo100-1000/
 axiombeta-calibration/sensor-profiles/gain1_expo100-1000_plr32-10ms-24-1ms/

That means, from all the settings that might affect the calibration, we can build an unique name from the settings that differ from the generic profile. A calibration might cover files at some particular setting value, or some range of values (detected from the supplied input files during calibration).

When selecting a profile, we may want to choose the one with the narrowest setting range that fits the settings of the file being rendered (that one should probably give the lowest calibration errors). If no match is found, the generic profile will be used.

Or, we may want to test the profile against some validation data, store the results, and pick the profile likely to minimize the calibration errors (some solid math skills required for this one).

1.3.2 Sensor calibration files

Inside one profile, we can have the following calibration files (all optional; if missing, that calibration step will be skipped):

For Black Calibration:

 beforegain.pgm : static offsets whose source is before the gain amplifiers
                 (that means, will be multiplied by gain, then subtracted)
 aftergain.pgm  : static offsets whose source is after the gain amplifiers
 darkcurrent.pgm: dynamic offsets dependent on many variables (exposure, temperature, gain),
                  whose scaling factor can be measured from hot pixels
                  (idea from [http://www.photonics.com/Article.aspx?AID=44298])

These 3 files allow computing a dark frame for every gain/expo/temperature combination (at least in theory), and should generalize fairly well to other settings (to be tested).

The computed dark frame would be:

beforegain * gain + aftergain + darkcurrent * darkcurrent_scaling

where darkcurrent_scaling can be measured from hot pixels.

Therefore, a simple dark frame (without any computation) can be saved in aftergain.pgm. If that's confusing, should we add darkframe.pgm as a synonym?

Other calibration files (e.g. Response Curves, Pattern Noise corrections):

 gainframe.pgm   : per-pixel gain variations
 clipframe.pgm   : clipping offsets
 responseXXXX.pgm: per-pixel response curve data point,
                   (raw sensor outputs that should give the corrected value XXXX)
                   (these might replace gain/clip frames completely, need to experiment)
 lut.spi1d       : overall or per-channel response curve (shaper)

1.4 Color profiles

Color profiles are stored under this subdirectory:

axiombeta-calibration/color-profiles/

These depend on other things, like ambient light and maybe lens used, so we could think about something like:

 axiombeta-calibration/color-profiles/D50/
 axiombeta-calibration/color-profiles/Tungsten/
 axiombeta-calibration/color-profiles/OsramDulux/                 # model of the light bulbs
 axiombeta-calibration/color-profiles/InTheCave/                  # or the location where the calibration was performed
 axiombeta-calibration/color-profiles/D50/EF50mm_f/1.8_II/        # from lens metadata

Those variables will be entered manually by the user (or chosen from a pickbox from a GUI), except maybe lens metadata, where it can be obtained from the lens itself (when the lens communication will be functional).

Color profile formats:

- ICC (as saved by argyll)
- config.ocio files?
- ...

2 Wish list

A way to identify which sensor or camera is used (by serial numbers) could be very helpful for those who own more than one camera, or more than one sensor.

In this case, the directory structure would be be:

 ~/axiombeta-calibration/cam1234/sensor5678/sensor-profiles
 ~/axiombeta-calibration/color-profiles

That's because sensor profiling is expected to account for the differences between sensors (that means, consistent output), so color profiles should be interchangeable between cameras/sensors (and should only account for lighting/lens differences).

Also, with a serial number, we no longer have to use tricks like having to search for the calibration subdirectory up 1/2/3/n levels from the current location.

Proposal from danieel:

~/.config/axiom/beta/calib/

also: "if the data is of low entropy, then just pack the multi-file structure into one .tgz"