Current Server Time: 21:11 (Central Europe)

#apertus IRC Channel Logs

2023/03/20

Timezone: UTC


01:28
aombk2
left the channel
01:29
aombk2
joined the channel
02:13
balrog
left the channel
02:15
balrog
joined the channel
11:36
polyrhythm
anuejn: hokay, I had time to try again this weekend, still not working as expected
11:36
polyrhythm
I give the following command:
11:36
polyrhythm
E:\dev\axiom\axiom-recorder>target\release\cli.exe from-cli RawDirectoryReader --file-pattern E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\*.raw32 --height 3072 --width 4096 ! CinemaDngWriter --dcp-yaml src\nodes_io\default_dcp.yml --path E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\DNG
11:36
polyrhythm
(ignore if the width/height is correct for now)
11:37
polyrhythm
I get
11:37
polyrhythm
using 24 threads
11:37
polyrhythm
using cpu only processing
11:37
polyrhythm
no sample interpretation was specified
11:37
polyrhythm
I tried looking for where to provide a sample interpretation and there seems to be one in base_ifd.yml
11:37
polyrhythm
if I point to that instead of default_dcp.yml, same thing
11:38
polyrhythm
I am on the gpu_node_refactor branch
13:10
vup
polyrhythm: you need to tell the RawDirectoryReader that you are reading a float32 file using the "--fp32" flag
13:12
polyrhythm
vup: woohoo! that did it
13:12
polyrhythm
finally
13:17
polyrhythm
anybody happen to know how to change the debayer/CFA order? right now it seems not right
13:17
polyrhythm
https://i.imgur.com/yaktUE9.png
13:25
vup
polyrhythm: yes, "--bayer=PATTERN", where "PATTERN
13:25
vup
â {RGGB, GBRG, GRBG, BGGR}
13:25
polyrhythm
nice
13:30
polyrhythm
for future posterity the final working incantation turned out to be
13:30
polyrhythm
E:\dev\axiom\axiom-recorder>target\release\cli.exe from-cli RawDirectoryReader --fp32 --file-pattern E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\*.raw32 --height 2160 --width 3840 --bayer GBRG ! CinemaDngWriter --dcp-yaml src\nodes_io\base_ifd.yml --path E:\assets\ref\axiom\221201_XriteSnaps\LED_Xrite\DNG
13:44
polyrhythm
hokay, now I get random errors reading the DNG files in everything other than RawDigger
13:45
polyrhythm
even Adobe DNG Converter can't read them
13:45
polyrhythm
I suspect there's some critical metadata missing from the DNG files
14:17
se6astian
please document the working parameters and final commands in the github readme accordingly
14:18
se6astian
anuejn: can you read copy the DNG header from a known good file maybe?
14:57
vup
se6astian: thats how (part) of the current "header" was developed. Its probably just something "obvious" missing
15:08
se6astian
good good :)
15:23
vup
also in addition to the "cli" way of specifying pipelines we have the yml config files. I think adding a folder with example config files for these operations would be better than adding more and more examples to the README. I would be happy to see PRs for these
15:57
polyrhythm
I'll add a config example for my use case and do a PR
15:59
vup
great
16:00
se6astian
MEETING TIME, who is here?
16:00
polyrhythm
i'm here!
16:01
se6astian
great polyrhythm! you mentioned that even though the files don't load in many tools you were able to load them in rawdigger?
16:01
se6astian
is that the tool you use to profile the colors?
16:02
Bertl
is here
16:02
polyrhythm
I'm able to load them in RawDigger but that's really just an analysis tool, and I think the only reason it can open them is because it's an extremely pared down application that is designed to break apart all the metadata and not do anything to the file itself
16:02
polyrhythm
I have to open it in other software to do any profiling, like LumaRiver
16:02
polyrhythm
just getting Adobe DNG convert to be able to "repair" them would be a good start...
16:03
polyrhythm
right now even the Adobe DNG converter can't open them
16:03
se6astian
I see
16:07
se6astian
well then anuejn and vup are soon or already on it I assume
16:07
se6astian
does rawdigger provide any output what metadata is present and what is expected/missing?
16:07
polyrhythm
it shows you what is there, one sec
16:07
polyrhythm
I will try to take a screenshot or export the data
16:07
se6astian
Bertl any news from your side (hardware setup, software progress)?
16:08
polyrhythm
---- ExifTool ----
16:08
polyrhythm
ExifTool Version Number : 12.50
16:08
polyrhythm
---- File ----
16:08
polyrhythm
File Name : 000002.dng
16:08
polyrhythm
Directory : E:/assets/ref/axiom/221201_XriteSnaps/halo_xrite/DNG
16:08
polyrhythm
File Size : 33 MB
16:08
polyrhythm
File Modification Date/Time : 2023:03:20 10:35:23-04:00
16:08
polyrhythm
File Access Date/Time : 2023:03:20 13:07:57-04:00
16:08
polyrhythm
File Creation Date/Time : 2023:03:20 10:35:21-04:00
16:08
polyrhythm
File Permissions : -rw-rw-rw-
16:08
polyrhythm
File Type : DNG
16:08
polyrhythm
File Type Extension : dng
16:08
polyrhythm
MIME Type : image/x-adobe-dng
16:08
polyrhythm
Exif Byte Order : Little-endian (Intel, II)
16:08
polyrhythm
---- EXIF ----
16:08
polyrhythm
Subfile Type : Full-resolution image
16:08
polyrhythm
DNG Version : 1.4.0.0
16:08
anuejn
is here
16:08
polyrhythm
Make : Apertus
16:08
polyrhythm
Camera Model Name : AXIOM
16:08
polyrhythm
Software : axiom-recorder
16:09
polyrhythm
Unique Camera Model : Apertus AXIOM
16:09
polyrhythm
Samples Per Pixel : 1
16:09
polyrhythm
Compression : Uncompressed
16:09
polyrhythm
Fill Order : Normal
16:09
polyrhythm
Orientation : Horizontal (normal)
16:09
polyrhythm
Resolution Unit : None
16:09
Bertl
se6astian: no progress from my side, was busy with work station update but everything is fine now, so back on track
16:09
polyrhythm
X Resolution : 1
16:09
polyrhythm
Y Resolution : 1
16:09
polyrhythm
Planar Configuration : Chunky
16:09
polyrhythm
Photometric Interpretation : Color Filter Array
16:09
polyrhythm
CFA Repeat Pattern Dim : 2 2
16:09
polyrhythm
CFA Pattern 2 : 1 2 0 1
16:09
polyrhythm
Bits Per Sample : 32
16:09
polyrhythm
Sample Format : Float
16:09
polyrhythm
Image Width : 3840
16:09
polyrhythm
Image Height : 2160
16:09
Bertl
polyrhythm: a pastebin is a great choice ;)
16:09
polyrhythm
Rows Per Strip : 2160
16:09
se6astian
Bertl: right, thanks
16:09
polyrhythm
Frame Rate : 24
16:09
polyrhythm
Strip Offsets : 388
16:09
polyrhythm
Strip Byte Counts : 33177600
16:09
polyrhythm
---- Composite ----
16:09
polyrhythm
CFA Pattern : [Green,Blue][Red,Green]
16:09
polyrhythm
Image Size : 3840x2160
16:09
polyrhythm
Megapixels : 8.3
16:10
anuejn
for me it worked to open the resulting dngs in adobe dng converter
16:11
anuejn
I am a bit puzzeled that id does not work for you
16:11
se6astian
can you exchange DNG files please to check
16:11
polyrhythm
sure, is there a preferred cloud storage thingie
16:12
se6astian
we have a nextcloud on apertus.org if you want I can create an account for you
16:12
polyrhythm
anuejn: you could also unblock by directly converting the raw32 files i need to profile (the colorchart images of LED and kinoflo light or whatever it is)
16:13
anuejn
ahI can also do that
16:14
anuejn
the recorder produced this file for me: https://files.niemo.de/recorder_sample.dng
16:14
anuejn
with the comandline from the email-thread:
16:14
anuejn
cargo run --bin cli -- from-cli RawBlobReader --fp32 --bayer GBRG --width 3840 --height 2160 --file /Users/anuejn/Downloads/Halogen_Xrite_0_8ms.raw32 ! CinemaDngWriter --path target/test
16:16
anuejn
the dngs we currently write are in fact a bit - unusual but that should not be a problem for a well bahaved dng reader
16:16
anuejn
(they do not contain a preview)
16:19
polyrhythm
all my stuff from Japan suddenly arrived so I have to step away for a bit
16:19
polyrhythm
:)
16:19
se6astian
alright :)
16:19
se6astian
anuejn/vup anything else that is new to report?
16:19
se6astian
any progress with rock5b?
16:20
anuejn
nope, sadly not
16:20
Bertl
what's the current problem there?
16:20
Bertl
the data islands?
16:21
anuejn
yup
16:21
anuejn
I started to craft a generator for the packets in python but then discussed with vup that it is probably easier to patch the driver of the hdmi-rx peripheral
16:26
Bertl
does the driver allow for DVI like signalling?
16:28
Bertl
because if so, you might simply drop the guard band and get away without the data islands
16:35
se6astian
anyone else around or anything else to report? otherwise I would conclude the meeting, but feel free to continue HDMI output, DNG metadata, etc. discussion
16:36
se6astian
only updates from me are that more parts arrived for enclosure
16:36
se6astian
and that copper sheets have been selected for creating shims, might have talked about it already last week
16:36
se6astian
etching them like PCBs and laser engraving to label them
16:37
se6astian
MEETING CONCLUDED
16:38
Bertl
thanks for the moderation!
16:40
vup
Bertl: the driver does not as far as we can tell, the SoC supports DVI signalling
16:47
anuejn
Bertl: i do not think so
16:48
anuejn
vup: how do you know?
16:48
anuejn
is it said in the datasheet?
16:53
polyrhythm
hey anuejn is there a difference from using RawBlobReader and RawDirectoryReader?
16:53
polyrhythm
that is the only obvious difference I see in our incantations
16:57
vup
anuejn: yes
17:00
anuejn
polyrhythm: not really (at least in theory)
17:01
anuejn
polyrhythm: can you open my dng with your version of adobes tool?
17:14
polyrhythm
anuejn: does it work for you?
17:14
polyrhythm
this is how it looks when I open it in RawDigger:
17:14
polyrhythm
https://i.imgur.com/g8blsKN.png
17:14
polyrhythm
Adobe DNG Converter can open it too though, so that's interesting
17:14
polyrhythm
i can't see anything with the adobe program, it just a converter, no visual anything
17:17
se6astian
polyrhythm: see PM
17:21
anuejn
polyrhythm: ah yes that is also how it opens in the macos preview. I think this is due to the program not respecting the float32 nature of the image
17:22
anuejn
probably it just interprets it as uint32
17:22
anuejn
so we need to convert it before and either write or find a tool for that
17:23
Bertl
why are we using float32 in the first place?
17:24
anuejn
because these are averaged frames and it seemed reasonable at the time to use float32 to ease implementation and retain precision
17:27
Bertl
okay, makes sense
17:29
polyrhythm
kind of getting in the weeds but I feel like float16 should be more than enough, no?
17:30
anuejn
yup
17:30
anuejn
but that is probably also unsupported?
17:30
polyrhythm
even full-on CG renders typically render at float16 buffers, and that's certainly with more stops of info than whatever this dinky sensor is puttin out...
17:30
polyrhythm
yeah maybe
17:31
anuejn
the point is that processing fp16 on cpus is not too fast
17:31
anuejn
but yeah for storage it might be reasonable
17:31
anuejn
for our applications int16 would also certainly be enough
17:32
anuejn
the trick is more, that the floating point dngs have the same value range as the uint ones but more resolution
17:32
anuejn
so they can be directly substracted
17:36
vup
processing fp16 on cpus is pretty cheap
17:37
vup
if you just convert them to fp32 (which is pretty cheap on x86 cpus nowadays with the f16c extension)
17:38
vup
it could actually be faster than fp32 for some scenarios where you are more mem latency / bandwidth bound than compute bound
17:38
vup
although I read that openblas did not see any performance improvements between fp16 and fp32
17:52
anuejn
oh wow i didnt know of the f16c extension
17:52
anuejn
that is cool
22:45
balrog
left the channel
22:46
balrog
joined the channel
22:51
balrog
left the channel
22:52
balrog
joined the channel