Current Server Time: 23:57 (Central Europe)

#apertus IRC Channel Logs

2023/03/20

Timezone: UTC


02:28
aombk2
left the channel
02:29
aombk2
joined the channel
03:13
balrog
left the channel
03:15
balrog
joined the channel
12:36
polyrhythm
anuejn: hokay, I had time to try again this weekend, still not working as expected
12:36
polyrhythm
I give the following command:
12:36
polyrhythm
E:\dev\axiom\axiom-recorder>target\release\cli.exe from-cli RawDirectoryReader --file-pattern E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\*.raw32 --height 3072 --width 4096 ! CinemaDngWriter --dcp-yaml src\nodes_io\default_dcp.yml --path E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\DNG
12:36
polyrhythm
(ignore if the width/height is correct for now)
12:37
polyrhythm
I get
12:37
polyrhythm
using 24 threads
12:37
polyrhythm
using cpu only processing
12:37
polyrhythm
no sample interpretation was specified
12:37
polyrhythm
I tried looking for where to provide a sample interpretation and there seems to be one in base_ifd.yml
12:37
polyrhythm
if I point to that instead of default_dcp.yml, same thing
12:38
polyrhythm
I am on the gpu_node_refactor branch
14:10
vup
polyrhythm: you need to tell the RawDirectoryReader that you are reading a float32 file using the "--fp32" flag
14:12
polyrhythm
vup: woohoo! that did it
14:12
polyrhythm
finally
14:17
polyrhythm
anybody happen to know how to change the debayer/CFA order? right now it seems not right
14:17
polyrhythm
https://i.imgur.com/yaktUE9.png
14:25
vup
polyrhythm: yes, "--bayer=PATTERN", where "PATTERN
14:25
vup
∈ {RGGB, GBRG, GRBG, BGGR}
14:25
polyrhythm
nice
14:30
polyrhythm
for future posterity the final working incantation turned out to be
14:30
polyrhythm
E:\dev\axiom\axiom-recorder>target\release\cli.exe from-cli RawDirectoryReader --fp32 --file-pattern E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\*.raw32 --height 2160 --width 3840 --bayer GBRG ! CinemaDngWriter --dcp-yaml src\nodes_io\base_ifd.yml --path E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\DNG
14:44
polyrhythm
hokay, now I get random errors reading the DNG files in everything other than RawDigger
14:45
polyrhythm
even Adobe DNG Converter can't read them
14:45
polyrhythm
I suspect there's some critical metadata missing from the DNG files
15:17
se6astian
please document the working parameters and final commands in the github readme accordingly
15:18
se6astian
anuejn: can you read copy the DNG header from a known good file maybe?
15:57
vup
se6astian: thats how (part) of the current "header" was developed. Its probably just something "obvious" missing
16:08
se6astian
good good :)
16:23
vup
also in addition to the "cli" way of specifying pipelines we have the yml config files. I think adding a folder with example config files for these operations would be better than adding more and more examples to the README. I would be happy to see PRs for these
16:57
polyrhythm
I'll add a config example for my use case and do a PR
16:59
vup
great
17:00
se6astian
MEETING TIME, who is here?
17:00
polyrhythm
i'm here!
17:01
se6astian
great polyrhythm! you mentioned that even though the files don't load in many tools you were able to load them in rawdigger?
17:01
se6astian
is that the tool you use to profile the colors?
17:02
Bertl
is here
17:02
polyrhythm
I'm able to load them in RawDigger but that's really just an analysis tool, and I think the only reason it can open them is because it's an extremely pared down application that is designed to break apart all the metadata and not do anything to the file itself
17:02
polyrhythm
I have to open it in other software to do any profiling, like LumaRiver
17:02
polyrhythm
just getting Adobe DNG convert to be able to "repair" them would be a good start...
17:03
polyrhythm
right now even the Adobe DNG converter can't open them
17:03
se6astian
I see
17:07
se6astian
well then anuejn and vup are soon or already on it I assume
17:07
se6astian
does rawdigger provide any output what metadata is present and what is expected/missing?
17:07
polyrhythm
it shows you what is there, one sec
17:07
polyrhythm
I will try to take a screenshot or export the data
17:07
se6astian
Bertl any news from your side (hardware setup, software progress)?
17:08
polyrhythm
---- ExifTool ----
17:08
polyrhythm
ExifTool Version Number : 12.50
17:08
polyrhythm
---- File ----
17:08
polyrhythm
File Name : 000002.dng
17:08
polyrhythm
Directory : E:/assets/ref/axiom/221201_XriteSnaps/halo_xrite/DNG
17:08
polyrhythm
File Size : 33 MB
17:08
polyrhythm
File Modification Date/Time : 2023:03:20 10:35:23-04:00
17:08
polyrhythm
File Access Date/Time : 2023:03:20 13:07:57-04:00
17:08
polyrhythm
File Creation Date/Time : 2023:03:20 10:35:21-04:00
17:08
polyrhythm
File Permissions : -rw-rw-rw-
17:08
polyrhythm
File Type : DNG
17:08
polyrhythm
File Type Extension : dng
17:08
polyrhythm
MIME Type : image/x-adobe-dng
17:08
polyrhythm
Exif Byte Order : Little-endian (Intel, II)
17:08
polyrhythm
---- EXIF ----
17:08
polyrhythm
Subfile Type : Full-resolution image
17:08
polyrhythm
DNG Version : 1.4.0.0
17:08
anuejn
is here
17:08
polyrhythm
Make : Apertus
17:08
polyrhythm
Camera Model Name : AXIOM
17:08
polyrhythm
Software : axiom-recorder
17:09
polyrhythm
Unique Camera Model : Apertus AXIOM
17:09
polyrhythm
Samples Per Pixel : 1
17:09
polyrhythm
Compression : Uncompressed
17:09
polyrhythm
Fill Order : Normal
17:09
polyrhythm
Orientation : Horizontal (normal)
17:09
polyrhythm
Resolution Unit : None
17:09
Bertl
se6astian: no progress from my side, was busy with work station update but everything is fine now, so back on track
17:09
polyrhythm
X Resolution : 1
17:09
polyrhythm
Y Resolution : 1
17:09
polyrhythm
Planar Configuration : Chunky
17:09
polyrhythm
Photometric Interpretation : Color Filter Array
17:09
polyrhythm
CFA Repeat Pattern Dim : 2 2
17:09
polyrhythm
CFA Pattern 2 : 1 2 0 1
17:09
polyrhythm
Bits Per Sample : 32
17:09
polyrhythm
Sample Format : Float
17:09
polyrhythm
Image Width : 3840
17:09
polyrhythm
Image Height : 2160
17:09
Bertl
polyrhythm: a pastebin is a great choice ;)
17:09
polyrhythm
Rows Per Strip : 2160
17:09
se6astian
Bertl: right, thanks
17:09
polyrhythm
Frame Rate : 24
17:09
polyrhythm
Strip Offsets : 388
17:09
polyrhythm
Strip Byte Counts : 33177600
17:09
polyrhythm
---- Composite ----
17:09
polyrhythm
CFA Pattern : [Green,Blue][Red,Green]
17:09
polyrhythm
Image Size : 3840x2160
17:09
polyrhythm
Megapixels : 8.3
17:10
anuejn
for me it worked to open the resulting dngs in adobe dng converter
17:11
anuejn
I am a bit puzzeled that id does not work for you
17:11
se6astian
can you exchange DNG files please to check
17:11
polyrhythm
sure, is there a preferred cloud storage thingie
17:12
se6astian
we have a nextcloud on apertus.org if you want I can create an account for you
17:12
polyrhythm
anuejn: you could also unblock by directly converting the raw32 files i need to profile (the colorchart images of LED and kinoflo light or whatever it is)
17:13
anuejn
ahI can also do that
17:14
anuejn
the recorder produced this file for me: https://files.niemo.de/recorder_sample.dng
17:14
anuejn
with the comandline from the email-thread:
17:14
anuejn
cargo run --bin cli -- from-cli RawBlobReader --fp32 --bayer GBRG --width 3840 --height 2160 --file /Users/anuejn/Downloads/Halogen_Xrite_0_8ms.raw32 ! CinemaDngWriter --path target/test
17:16
anuejn
the dngs we currently write are in fact a bit - unusual but that should not be a problem for a well bahaved dng reader
17:16
anuejn
(they do not contain a preview)
17:19
polyrhythm
all my stuff from Japan suddenly arrived so I have to step away for a bit
17:19
polyrhythm
:)
17:19
se6astian
alright :)
17:19
se6astian
anuejn/vup anything else that is new to report?
17:19
se6astian
any progress with rock5b?
17:20
anuejn
nope, sadly not
17:20
Bertl
what's the current problem there?
17:20
Bertl
the data islands?
17:21
anuejn
yup
17:21
anuejn
I started to craft a generator for the packets in python but then discussed with vup that it is probably easier to patch the driver of the hdmi-rx peripheral
17:26
Bertl
does the driver allow for DVI like signalling?
17:28
Bertl
because if so, you might simply drop the guard band and get away without the data islands
17:35
se6astian
anyone else around or anything else to report? otherwise I would conclude the meeting, but feel free to continue HDMI output, DNG metadata, etc. discussion
17:36
se6astian
only updates from me are that more parts arrived for enclosure
17:36
se6astian
and that copper sheets have been selected for creating shims, might have talked about it already last week
17:36
se6astian
etching them like PCBs and laser engraving to label them
17:37
se6astian
MEETING CONCLUDED
17:38
Bertl
thanks for the moderation!
17:40
vup
Bertl: the driver does not as far as we can tell, the SoC supports DVI signalling
17:47
anuejn
Bertl: i do not think so
17:48
anuejn
vup: how do you know?
17:48
anuejn
is it said in the datasheet?
17:53
polyrhythm
hey anuejn is there a difference from using RawBlobReader and RawDirectoryReader?
17:53
polyrhythm
that is the only obvious difference I see in our incantations
17:57
vup
anuejn: yes
18:00
anuejn
polyrhythm: not really (at least in theory)
18:01
anuejn
polyrhythm: can you open my dng with your version of adobes tool?
18:14
polyrhythm
anuejn: does it work for you?
18:14
polyrhythm
this is how it looks when I open it in RawDigger:
18:14
polyrhythm
https://i.imgur.com/g8blsKN.png
18:14
polyrhythm
Adobe DNG Converter can open it too though, so that's interesting
18:14
polyrhythm
i can't see anything with the adobe program, it just a converter, no visual anything
18:17
se6astian
polyrhythm: see PM
18:21
anuejn
polyrhythm: ah yes that is also how it opens in the macos preview. I think this is due to the program not respecting the float32 nature of the image
18:22
anuejn
probably it just interprets it as uint32
18:22
anuejn
so we need to convert it before and either write or find a tool for that
18:23
Bertl
why are we using float32 in the first place?
18:24
anuejn
because these are averaged frames and it seemed reasonable at the time to use float32 to ease implementation and retain precision
18:27
Bertl
okay, makes sense
18:29
polyrhythm
kind of getting in the weeds but I feel like float16 should be more than enough, no?
18:30
anuejn
yup
18:30
anuejn
but that is probably also unsupported?
18:30
polyrhythm
even full-on CG renders typically render at float16 buffers, and that's certainly with more stops of info than whatever this dinky sensor is puttin out...
18:30
polyrhythm
yeah maybe
18:31
anuejn
the point is that processing fp16 on cpus is not too fast
18:31
anuejn
but yeah for storage it might be reasonable
18:31
anuejn
for our applications int16 would also certainly be enough
18:32
anuejn
the trick is more, that the floating point dngs have the same value range as the uint ones but more resolution
18:32
anuejn
so they can be directly substracted
18:36
vup
processing fp16 on cpus is pretty cheap
18:37
vup
if you just convert them to fp32 (which is pretty cheap on x86 cpus nowadays with the f16c extension)
18:38
vup
it could actually be faster than fp32 for some scenarios where you are more mem latency / bandwidth bound than compute bound
18:38
vup
although I read that openblas did not see any performance improvements between fp16 and fp32
18:52
anuejn
oh wow i didnt know of the f16c extension
18:52
anuejn
that is cool
23:45
balrog
left the channel
23:46
balrog
joined the channel
23:51
balrog
left the channel
23:52
balrog
joined the channel