Current Server Time: 21:31 (Central Europe)

#apertus IRC Channel Logs

2021/11/22

Timezone: UTC


00:58
illwieckz
left the channel
04:01
Bertl_oO
off to bed now ... have a good one everyone!
04:01
Bertl_oO
changed nick to: Bertl_zZ
04:55
felix_
left the channel
04:55
se6astian
left the channel
04:55
felix_
joined the channel
04:55
se6astian
joined the channel
06:53
mithro
left the channel
06:54
mithro
joined the channel
07:14
se6astian
good day
08:28
mithro
left the channel
08:30
mithro
joined the channel
10:38
Bertl_zZ
changed nick to: Bertl
10:38
Bertl
morning folks!
15:53
Bertl
off for now ... bbl
15:53
Bertl
hmm, disregard, meeting coming up ;)
15:59
se6astian
yes please :)
16:00
se6astian
MEETING TIME! who is here?
16:00
Bertl
is here ...
16:01
vup
is here
16:01
se6astian
vup, any news to report?
16:02
vup
a little
16:02
vup
anuejn and I mostly worked on some small fixes and dependency updates for narui (the ui framework for the recorder) and published the 0.1 version: https://crates.io/crates/narui
16:03
anuejn
is here
16:03
vup
next we will probably start to finally merge the integration with the recorder: https://github.com/apertus-open-source-cinema/axiom-recorder/pull/7
16:05
vup
(thats it)
16:05
Bertl
nice
16:06
se6astian
great, anything to add anuejn?
16:06
anuejn
no, I didnt get much done
16:06
se6astian
ok
16:06
se6astian
anyone else who wants to report?
16:07
se6astian
ilia3101 from the magic lantern forum will join us at 19:15
16:07
se6astian
and is interested in helping us with raw processing: https://www.magiclantern.fm/forum/index.php?topic=26299.0
16:07
se6astian
I doubt we will still be meeting then but please return for a warm welcome then in 1h
16:08
se6astian
quick updates from me:
16:08
se6astian
I tested a lattepanda alpha 864s as potential recorder
16:08
se6astian
and it works very well!
16:08
se6astian
also an Intel NUC, also works well
16:08
se6astian
but the lattepanda is slightly smaller
16:08
se6astian
but provides the same features/throughput
16:09
se6astian
even has onboard flash memory which the NUC doesnt
16:09
se6astian
raw12 playback also works well with onboard GPU
16:09
se6astian
so that seems like a very good candidate
16:09
se6astian
rockpi4 eventually went to the drawer
16:10
se6astian
and the pny CS3030 NVME SSD eventually turned out to actually be fast enough for continous recording
16:10
se6astian
I did a few recorder gui additions here and there
16:10
se6astian
or bgr-info script
16:10
se6astian
https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw-via-hdmi/bgr-info
16:10
se6astian
to identify skipped/double frames after capture
16:11
se6astian
as I still suspect the ffmpeg way is not 100% optimal bertl is working on a magewell SDK inspired capture tool
16:11
se6astian
that showed much better performance on his magewell PCIE capture cards IIRC
16:11
se6astian
I need your feedback or ideas regarding the processing step to combine 2 bgr frames into 1 raw12
16:12
Bertl
more reliable indeed, not sure how that maps to USB though, as they use different mechanisms there
16:12
se6astian
currently we use montage (imagemagick) and place the two image side by side, then reinterpret the size
16:12
se6astian
effectively alternating lines from the two files
16:12
se6astian
do you think with a different tool/approach this could be speed up
16:13
vup
I think anuejn and I wanted to support it in the recorder (just did not get to it yet)
16:13
se6astian
maybe our own C tool?
16:13
Bertl
it should be a two liner in C actually :)
16:13
vup
(the recorder would then probably also have the capture integrated)
16:14
Bertl
in the future, there are likely two different 'recording' options for raw data
16:15
Bertl
one is by using alternating frames on the HDMI output, the other is by using two HDMI outputs, one for each frame type (A/B)
16:15
Bertl
in the first case, we need to know which frame (A or B) we are currently on
16:15
Bertl
in the second case we need to synchronize the frames so that we process the correct A and B frame
16:16
se6astian
2 line sounds good :)
16:16
Bertl
for the first case, I was thinking we could utilize an interlace mode if that works
16:16
Bertl
i.e. effectively sending the two frames after each other as a single image
16:17
Bertl
s/after each other/one after the other/
16:17
Bertl
note that this requires some changes in the gateware, so something for future transports
16:17
se6astian
sounds good
16:18
se6astian
with the corner markers we could always identify the matching pairs easily as well
16:18
Bertl
yes, but that wouldn't even be necessary on the interlace version
16:18
se6astian
even better
16:19
se6astian
regarding the grant application with the technical university, I reviewed and updated our text over the weekend
16:19
se6astian
and today got the universities parts of the text
16:19
se6astian
manfred has until wednesday to review/request changes
16:19
se6astian
then we combine texts
16:19
se6astian
and apply
16:20
se6astian
I will play around with binning/skipping a bit in the near future
16:20
se6astian
had a chat with bertl about it already today
16:20
se6astian
also about increasing FPS for smaller resolutions
16:20
se6astian
magewell USB3 device can go up to 120 FPS
16:21
se6astian
1280x720 at 120 fps
16:21
vup
we could even do more if we just combine the data into bigger frames on the beta probably
16:21
se6astian
would be in the same throughput range as 1920x1080@50fps
16:21
se6astian
yes but that requires a more complex gateware IIRC
16:21
se6astian
buffering frames
16:21
vup
well we buffer frames already
16:22
se6astian
you are the experts :)
16:22
vup
well Bertl is the current gatewares expert
16:22
Bertl
it mainly requires that we get rid of the sequencer ;)
16:22
se6astian
I updated the raw documentation in the google doc with FW2.0 commands already partially
16:22
se6astian
but as next step plan to bring it to the wiki
16:22
se6astian
but focusing on FW2.0 there
16:22
Bertl
but that is planned for some time now, so it probably will happen sooner or later
16:23
se6astian
probably leave out the FW1.0 raw hdmi commands altogether
16:23
se6astian
with the current length todo list bertl has probably "later" :)
16:24
Bertl
yeah, no promises there ;)
16:24
se6astian
if someone wants to merge the snap_neon with the actual snap that would also help check one thing of berrtls todo list
16:24
se6astian
https://github.com/apertus-open-source-cinema/axiom-firmware/issues/192
16:24
Bertl
arrr!
16:25
se6astian
Bertl: any other news from your side?
16:25
Bertl
nothing really this week, I'm unfortunately quite busy with unrelated stuff
16:25
se6astian
understood, fingers crossed for next weeks
16:26
se6astian
ah just remembered: we registered for a 4096 block of mac addresses with openmoko yesterday
16:26
se6astian
they are handing out the addresses they bought but since the project was discontinued dont need anymore
16:27
se6astian
github PR for entry was submitted yesterday
16:27
se6astian
lets hope they work through it soon
16:27
Bertl
the idea here is to have universal addresses for each beta :)
16:27
se6astian
yes, currently the image has the mac address hard coded
16:27
se6astian
so all betas have the same
16:28
se6astian
not good
16:28
se6astian
ok, anyone else with news/topics to share/discuss?
16:31
se6astian
then lets conclude the meeting and please join us again here in 45 minutes t meet ilia
16:31
se6astian
MEETING CONCLUDED
16:31
Bertl
thanks
16:32
Bertl
changed nick to: Bertl_oO
16:35
anuejn
sorry, cant be here in 45 minutes but maybe someone can advertise axiom recorder?
16:35
anuejn
and integrating new image processing there
16:35
vup
not sure ill be here, but if I am sure :P
16:36
se6astian
so regarding the c tool for merging bgr files, will fopen() and fwrite() provide the speed up I expect?
16:37
vup
you probably want to benchmark fopen, fread with mmap?
16:40
se6astian
thanks, will check
17:13
Ilia3101
joined the channel
17:14
se6astian
Hi Ilia3101!
17:14
se6astian
welcome to our channel
17:14
Ilia3101
hello!
17:15
vup
hi
17:15
Ilia3101
I'm using the web irc, a bit confused how this workkshello!
17:15
Ilia3101
Can't see what I'm tuyuping
17:15
vup
oh interesting
17:15
se6astian
oh, not good :)
17:16
se6astian
well you can choose to install a dedicated client
17:16
vup
it sounds like the webirc is a bit broken
17:16
Ilia3101
I'll tryn again
17:16
Bertl_oO
changed nick to: Bertl
17:16
Ilia3101
Yeah I need a client
17:16
vup
if you have matrix already, you can also join via that
17:16
Ilia3101
sorry about this
17:16
se6astian
no worries
17:17
se6astian
alternatives: hexchat, pidgin, etc.
17:17
se6astian
chatzilla
17:17
Ilia3101
thank you
17:19
se6astian
shall we wait until you return with a client?
17:20
ilia
joined the channel
17:20
ilia
?
17:20
ilia
Hello
17:20
vup
hi
17:20
ilia
I am on hexchat
17:20
vup
nice
17:20
Ilia3101
left the channel
17:20
vup
does that work better?
17:20
ilia
It is working better.
17:20
Bertl
yay!
17:20
vup
awesome
17:20
ilia
How was the meeting?
17:21
se6astian
good thanks :)
17:21
vup
(note though that you will only receive messages while connected, if you ever miss something we keep logs here: http://irc.apertus.org/)
17:21
se6astian
you can read the log here: http://irc.apertus.org/index.php?day=22&month=11&year=2021#18
17:21
ilia
Ah thank you
17:22
se6astian
its updated in realtime
17:22
se6astian
so regarding the raw processing:
17:22
ilia
Nice
17:22
se6astian
many thanks that you are here :)
17:22
ilia
Yes the raw processing :)
17:22
se6astian
can you tell us a bit about yourself?
17:22
se6astian
or your background
17:24
se6astian
then it might be easier to see what first steps of a collaboration could be
17:24
se6astian
the tools alex created are pretty extensive already so I am not sure myself where to best continue
17:26
ilia
I'm currently a computer science student, but I've been interested in raw image processing for a very long time. I created the MLV App project four years ago, an all-in-one raw video processing program for MLV files. And I'm currently digging deeper in to colour science and camera colour.
17:26
ilia
I definitely need to take a look at what Alex has done.
17:27
vup
oh awesome, now I know where I have seen your name before :o
17:27
se6astian
https://github.com/ilia3101/MLV-App
17:27
se6astian
cool!
17:27
vup
s/:o/:)/
17:28
ilia
So are you planning to develop image processing tools for users of the camera?
17:29
se6astian
well we do not want to reinvent the wheel here but there are some steps we want to do with our camera raw data before we feed it into a raw development software
17:29
se6astian
currently our format is called raw12 https://wiki.apertus.org/index.php/RAW12
17:30
se6astian
its basically just raw 12 bit data from the image sensor
17:30
ilia
I see.
17:30
se6astian
to convert this raw12 file to a DNG (our second format of choice) alex created raw2dng
17:30
se6astian
https://github.com/apertus-open-source-cinema/misc-tools-utilities/tree/master/raw2dng
17:31
ilia
Nice to see
17:31
se6astian
https://wiki.apertus.org/index.php/Raw2dng
17:31
se6astian
it can alter white point, black point etc
17:31
se6astian
but also do flat field correction
17:31
ilia
Nice, all very useful
17:32
ilia
How much do you know about the colour of the sensor?
17:32
se6astian
alex also added the creation of darkframes/dcnuframes, etc directly into raw2dng
17:33
se6astian
and drafted some steps of the calibration routine already
17:33
se6astian
https://wiki.apertus.org/index.php/Factory_Calibration_(firmware_2.0)
17:33
se6astian
again raw2dng can validate the effects the flatfield correction has
17:33
ilia
This is great.
17:34
se6astian
but not all evaluations are integrated yet
17:34
se6astian
alex created a range of octave scripts to analyze some aspects of the results
17:34
ilia
So what type of things still need to be done?
17:34
se6astian
thought this is were our unerstanding of what he created ends
17:35
ilia
So we don't understand everything he created?
17:35
se6astian
the camera can capture stil images internally and write raw12 files (with metadata containing sensor information)
17:35
se6astian
raw2dng requires some of this metadata to do certain things
17:36
se6astian
capturing these still images is rather slow though (think of it like a photo camera application)
17:36
se6astian
now recently we eventually made progress with capturing raw moving images over HDMI
17:36
se6astian
and recovering the raw12 data (again 12 bit) from the sensor with such image sequences
17:37
se6astian
24/25/30 fps
17:37
ilia
That's awesome
17:37
se6astian
in 3840x2160 or even 4096x2160
17:37
se6astian
the problem now is that we cannot transfer metadata over this hdmi stream
17:37
se6astian
also raw2dng originally expected full resolution frames with 4096x3072
17:38
se6astian
so thats where we currently are
17:38
ilia
I see. Is there any possibility to make the camera do this processing?
17:38
ilia
(that raw2dng currently does)
17:38
se6astian
I managed to create darkframes from raw images captured via hdmi but the results were not as good as expected
17:39
ilia
What was the issue with those darkframes?
17:39
se6astian
DCNU frames would also help but for these the missing metadata prevents the creation
17:39
vup
se6astian: hmm so is it actually impossible to transfer the metadata, because of the processing the hdmi capture card does?
17:40
se6astian
the darkframes did not entirely remove the fixed pattern noise, but we just applied it and looked at the results
17:40
vup
or is it "just" a gateware problem and we could just append a row of pixels that contains the metadata?
17:40
se6astian
so it would require a more structured analysis actually
17:40
vup
(my understanding is, that it is the latter)
17:40
ilia
Would DCNU frames remove the noise significantly more than darkframes?
17:40
se6astian
I guess we could repack it into pixels somehow indeed
17:40
se6astian
its 128x16 bit
17:40
vup
yeah I mean thats a trivial amount of data
17:41
se6astian
indeed, probably more tricky to handle them in the hdmi sender gateware though
17:41
se6astian
but also up for discussion if it makes sense to send it with every frame
17:41
se6astian
as the content is pretty much identical for all frames
17:42
se6astian
unless you change exposure time during recording
17:42
se6astian
which can happen of course
17:42
vup
I don't think sending them every frame does a lot of harm
17:42
vup
its such a small amount of data
17:42
se6astian
agreed
17:42
vup
the bigger problem is aquiring the data probably
17:42
se6astian
ilia: Would DCNU frames remove the noise significantly more than darkframes? <- that is what we need to find out :)
17:42
vup
the easiest solution is probably just providing some generic buffer for data that gets appended to every frame
17:42
vup
and make the userspace tools keep that up to date
17:43
se6astian
so in general the goal is to improve the image quality any way we can for footage that comes out of the camera
17:43
Bertl
actually, long term we want to put any metadata into HDMI data islands
17:43
vup
when we have the control daemon that would probably be relatively easy
17:43
vup
Bertl: do we? can we actually receive those properly with common capture cards?
17:43
Bertl
if we do it right, then yes
17:44
se6astian
currently I plan to have the recorder ssh to camera, cerate a register dump and download it to where the captured footage is
17:44
Bertl
data islands are used to provide video information but also to send audio
17:44
se6astian
it wont handle changes during recording
17:44
se6astian
but an easy qucikfix
17:44
se6astian
registers can simply be appended to all raw12 files
17:44
Bertl
capture cards might not support the video aspects but they usually support the audio stream(s)
17:44
vup
Bertl: oh so you want to for example use the audio stuff to just transport arbitrary data?
17:45
Bertl
precisely
17:45
vup
sure that works
17:45
se6astian
ah, very clever!
17:45
vup
But I don't see how that is a lot better than just packing it into the pixel data
17:45
Bertl
it deals with 'lossy' capture solutions
17:45
vup
true
17:46
vup
but with 128x16 bit we could also to ecc or something like that
17:46
Bertl
and it is otherwise unused bandwidth
17:46
vup
of course, its nicer, but I its also quite a lot of work
17:46
se6astian
<ilia> How much do you know about the colour of the sensor? <- so thats the next topic we have barely touched actually (also alex did not really touch the colour topic yet)
17:46
Bertl
and of course, we could also add fake pixels as well
17:46
vup
the other option for aquiring the data would be reading out the sensor registers every frame, is that feasible Bertl
17:46
vup
?
17:46
se6astian
so also an area where help would be much appreciated
17:47
vup
(the sensor on the micro for example is nice in that regard, because it sends the register contents as part of each frame)
17:47
Bertl
vup: the main question there is, whether we need to actually read the data from the sensor
17:47
se6astian
there is a default matrix in the raw2dng code I think
17:47
vup
Bertl: as opposed to having the userspace keep the data up to date?
17:48
vup
yeah thats what I suggested as the first option
17:48
Bertl
or let's rephrase it like this: what sensor data will change without userspace changing the sensor registers?
17:48
vup
yeah probably not a lot :P
17:48
se6astian
the sensor temperature :)
17:48
Bertl
probably only the temperature ;)
17:48
vup
maye some frame counter or so
17:49
vup
(not sure the cmv12k has one)
17:49
Bertl
so in the CMV12k case we can simply get away with buffering the sensor registers
17:49
se6astian
dont think so
17:49
ilia
I am very interested in the colour properties of the sensor. Does the manufacturer provide spectral response data or any colour information?
17:49
se6astian
yes, its in the datasheet
17:49
Bertl
ilia: yes, there is quite some information in the datasheet
17:49
se6astian
let me dig it out
17:50
Bertl
a recent version is available at AMS iirc
17:50
se6astian
https://ams.com/documents/20143/36005/CMV12000_DS000603_4-00.pdf
17:50
ilia
Thank you!
17:51
se6astian
page 17
17:51
vup
ok I unfortunately need to head out now, see you!
17:51
se6astian
thanks vup!
17:51
Bertl
vup: cya
17:52
ilia
So this DCNU data is only 128x16bit in size?
17:52
se6astian
no
17:53
se6astian
128x16bit are the sensor configuration metadata
17:53
ilia
Ah right
17:53
se6astian
exposure time
17:53
se6astian
gain
17:53
se6astian
etc.
17:53
ilia
DCNU is a whole frame then?
17:53
se6astian
yes
17:53
ilia
And you don't have a good way of sending it over HDMI?
17:54
se6astian
DCNU frames are calculated from calibration data
17:54
Bertl
it certainly could be sent over HDMI as well
17:54
se6astian
we can send these images over hdmi
17:54
se6astian
problem is that the metadata is missing
17:54
se6astian
and raw2dng tool expects it
17:55
se6astian
so 2 ideas/approaches:
17:55
se6astian
1. capture calibration data "the traditional way" with single images from inside the camera
17:55
se6astian
and figure out how to crop out the center area so it matches the HDMI stream
17:56
se6astian
full sensor resolution: 4096x3072
17:56
se6astian
vs HDMI stream 3840x2160
17:56
se6astian
or 4096x2160
17:56
se6astian
cropping should be trivial
17:57
se6astian
but we need to figure out if correction data created for stills really works 1:1 for HDMI data
17:57
se6astian
it should but we need to verify
17:57
ilia
So calibration data would have to be combined with the captured footage later?
17:57
ilia
After shooting
17:58
Bertl
yes, it doesn't make much sense to do that in real-time in our case
17:58
Bertl
we had some approaches for that but they were only half-hearted for a quick preview
17:58
se6astian
https://docs.google.com/drawings/d/1OX83QFdwphGZRrPg773UaYnnY0QYonqOttPOoBtNEwI/edit?usp=sharing
18:00
se6astian
alex created a pretty extensive document that gives a good overview: https://docs.google.com/document/d/12gZG4KFiWW4eV_ha-AM2hsw86fjNI4ILNo-CiHn80Z4/edit?usp=sharing
18:02
ilia
Interesting document
18:03
se6astian
sorry if we flood you with information currently :)
18:05
ilia
No worries, it's very informative
18:06
se6astian
good :)
18:07
ilia
I'm still not sure exactly what I could help with though
18:09
se6astian
well, currently we have nobody who has the skills/time to dive into these image processing topics
18:10
se6astian
like: how can we generate DCNU frames for HDMI output
18:10
ilia
Ah well I do have time right now :) And I would very much like help when possible.
18:11
ilia
So would DCNU frame generation have to happen on the camera?
18:11
se6astian
how can we evaluate how good the DCNU frames help reduce the actual fixed pattern noise
18:11
se6astian
no, its done on a connected PC most likely
18:12
se6astian
same for darkframes
18:12
se6astian
gainframes, etc.
18:12
ilia
understood
18:13
ilia
So this requires some sample data to begin
18:13
se6astian
we want to document the process so everyone can conduct the calibration
18:14
se6astian
but also need tools that measure if the calibration had the desired effects, etc.
18:14
ilia
I see
18:14
se6astian
I can help by capturing sample data (dark frames, or color charts, exposure sequences, etc.)
18:14
ilia
That would definitely be very helpful
18:14
se6astian
but I can also setup remote access to a PC and AXIOM Beta if it makes sense
18:15
ilia
Is the data steam over HDMI entirely lossless?
18:15
se6astian
yes
18:15
ilia
That's great
18:15
ilia
So when you mentioned disappointing results with darkframe over HDMI you weren't saying it was worse because of HDMI, but that dark frame is not enough by itself without DCNU
18:17
se6astian
yes, or another possibility is that the created darkframe is not calculated correctly because of the missing metadata
18:17
se6astian
but we can easily verify that by comparing HDMI captured footage/darkframes and internally captured still footage/darkframes
18:18
ilia
Yes
18:19
ilia
I think I might be able to dive in to things once I have some sample data
18:19
se6astian
sounds good!
18:20
se6astian
so next steps I would propose: you look at the raw2dng documentation/reference/factory calibration routine
18:20
se6astian
and create a list for me what sample data I should capture for you
18:21
ilia
I will take a look at that.
18:21
ilia
And think about what I might need
18:21
se6astian
I will then do that and upload
18:21
ilia
A lot of this (DCNU etc) is very new to me so we'll see how it goes.
18:21
ilia
That's great
18:21
se6astian
and we take it from there
18:21
se6astian
one step at a time
18:21
ilia
Yep
18:23
se6astian
what is the best way to reach you?
18:23
se6astian
email/messenger/irc/etc.?
18:24
ilia
I'd say email or Discord. I've also been using Discord a lot lately, haven't fully understood IRC yet.
18:25
ilia
What platforms are best for reaching you?
18:25
ilia
I'll send you my email in Magic Lantern PM
18:26
se6astian
thanks
18:26
se6astian
I will see if I can message you on discord
18:26
se6astian
I a on the magic lantern server but think I need to get authroized somehow
18:26
se6astian
to send direct messages
18:27
se6astian
I am also here on IRC most of the time
18:28
se6astian
which is our main communication platform for realtime interaction
18:28
ilia
Ok that's good to know, I'll start going on here more
18:28
se6astian
great
18:28
se6astian
many thanks!
18:28
se6astian
heading off to dinner now
18:28
ilia
Me too, it was great to chat.
18:28
Bertl
pleasure was ours!
18:29
Bertl
off for now as well ... bbl
18:29
Bertl
changed nick to: Bertl_oO
18:34
ilia
left the channel
18:34
ilia
joined the channel
18:48
ilia
left the channel
19:06
ilia
joined the channel
19:16
ilia
left the channel
19:25
ilia
joined the channel
19:27
ilia
left the channel
22:04
vup
I think there are some decently well working discord to irc bridges
22:04
vup
so in theory we could setup a bridged apertus discord room
22:11
anuejn
sounds like a plan
22:15
illwieckz
joined the channel
22:26
illwieckz
left the channel
22:46
illwieckz
joined the channel