Current Server Time: 16:27 (Central Europe)

#apertus IRC Channel Logs

2022/02/07

Timezone: UTC


23:24
Bertl
off to bed now ... have a good one everyone!
23:24
Bertl
changed nick to: Bertl_zZ
05:17
aombk2
left the channel
05:22
aombk
joined the channel
07:39
Spirit532
left the channel
07:39
Spirit532
joined the channel
10:16
Bertl_zZ
changed nick to: Bertl
10:16
Bertl
morning folks!
14:50
Bertl
off for now ... bbl
14:50
Bertl
changed nick to: Bertl_oO
15:19
balrog
left the channel
16:00
se6astian
MEETING TIME!
16:00
se6astian
who is here?
16:00
se6astian
Bertl notified me he has an urgent appointment currently and will likely miss the meeting
16:00
se6astian
maybe join later on
16:00
vup
is here
16:01
se6astian
vup: any news/progress for us to share?
16:01
vup
sure
16:02
vup
I looked a bit into the flatfields that se6astian captured
16:03
vup
In general, these were captured with a as uniform as possible light source and averaged over 256 frames
16:03
vup
the averaged flatfields were dark current compensated by subtracting the average of a 256 darkframes at ~14ms
16:04
vup
(the flatfields were captured at ~2ms, ~3ms, ~7ms and ~14ms)
16:05
vup
In general these flatfields were captured at 1x gain and under rather low light conditions (the ~14ms exposure only yields a average pixel value of ~50 after darkframe compensation)
16:06
vup
the first thing I looked at, was the standard deviation per pixel over the 256 darkframes for the exposures
16:07
vup
the standard deviation here is on average ~6.2 with a σ (of the standard deviation) of ~0.1
16:07
vup
this is roughly the same as plain darkframes
16:09
vup
now just visually inspecting the flatfields already shows some inhomogenity
16:09
vup
https://files.niemo.de/flatfield.png
16:10
vup
(if all pixels had the exact same response curve we would expect all pixels with the same color filter to have the same value)
16:11
vup
visually the pixel values mostly look similar per column, but seem to change for the columns
16:13
vup
so I tried to find a pattern in the average values along each column again (like I did for the darkframes)
16:13
vup
using only pixels in even columns and rows (to select pixels with the same color filter)
16:15
vup
one can again find some strong patterns when looking at columns in steps of 32
16:15
vup
(looking a column 0, 32, 64, at 1, 33, 65, etc)
16:15
vup
(these numbers refer only to even columns, so column 32 is actually column 64, because we are only looking at even columns)
16:16
vup
plotting these one gets something like this: https://files.niemo.de/column_wise_averages_1.9932959999999997.png
16:16
vup
https://files.niemo.de/column_wise_averages_2.9964.png
16:16
vup
https://files.niemo.de/column_wise_averages_6.996432.png
16:16
vup
https://files.niemo.de/column_wise_averages_6.996432.png
16:16
vup
whoops this one: https://files.niemo.de/column_wise_averages_13.993392.png
16:17
vup
ok, but only looking at a single flatfield without comparing to flatfields with different exposures doesn't really give us a lot to work with
16:17
vup
so the next step is comparing the flatfields with different exposures to each other
16:19
vup
now if the response curve of each pixel would be linear, one would expect that the ratio of pixel values of at two different exposure was equal to the ratio of the exposure times
16:19
vup
this is however not always the case
16:19
vup
is seems that there are "two" groups of pixels, ones that are roughly linear and ones that are not
16:20
vup
the ones that are not seem to only occur in odd rows
16:20
vup
where they make up roughly 50% of the pixels
16:21
vup
it seems that these "nonlinear" pixels are in general a lot less sensitive than the "linear" ones
16:21
vup
which can be clearly seen, when looking at the ratio of the pixel values between two different exposures in dependence of the pixel value of one of the flatfields
16:21
vup
https://files.niemo.de/corr_1.9932959999999997_by_13.993392.pdf
16:21
vup
https://files.niemo.de/corr_2.9964_by_13.993392.pdf
16:22
vup
https://files.niemo.de/corr_6.996432_by_13.993392.pdf
16:22
vup
(here 1.99... by 13.99... means that the ratio is the ration of the pixel values of the ~2ms and the ~14ms exposure and the x axis always uses the pixel values of the first exposure time)
16:23
vup
in these histogram we can clearly see these two groups of pixels
16:23
vup
also note, that which pixel is "nonlinear" seems to be independent of the exposure time (they are always the same pixels)
16:24
vup
indeed the location of these "nonlinear" pixels seems to follow a pretty strong pattern
16:25
vup
https://files.niemo.de/weird_pixels_even_rows_even_cols.png
16:25
vup
https://files.niemo.de/weird_pixels_even_rows_odd_cols.png
16:25
vup
https://files.niemo.de/weird_pixels_odd_rows_even_cols.png
16:25
vup
https://files.niemo.de/weird_pixels_odd_rows_odd_cols.png
16:25
vup
in these images a white pixel signifies a "nonlinear" pixel and a black one a roughly linear one
16:26
vup
(there are four images, because each image only shows pixels with the same color filter)
16:26
vup
I think this is pretty much everything I can say about the flatfields for now
16:26
se6astian
very interesting
16:27
vup
For further investigation there are a couple of things that would be interesting
16:27
se6astian
I will create an image acquisition script soon to gather more data about how the calibration images are captured + more metadata
16:27
vup
- using the full width with dark columns available
16:27
vup
- more different exposure times
16:28
vup
- capturing the different exposure times at different light intensity levels
16:28
vup
- capturing a couple of reference images for each capture with axiom_snap
16:28
vup
- capturing darkframes for the different exposure times aswell
16:29
se6astian
all noted
16:29
vup
the last thing that I am not fully convinced on is, how many frames to capture for each
16:30
vup
assuming a standard deviation of 6.2, the standard error of the mean is 6.2 / sqrt(n)
16:30
vup
so 256 frames gives us a standard error of the mean of 0.3875
16:31
vup
ahh
16:31
vup
I think what we should actually do is, do the averaging already in the recorder
16:31
se6astian
that would be great for upload sizes!
16:31
vup
so then capturing averages of many many frames should be pretty easy
16:31
vup
yes
16:32
vup
and for my available disksize :)
16:32
vup
Ill look into adding that feature
16:32
se6astian
wuld it be a trivial software addition for the recorder?
16:32
se6astian
regarding the light intensity levels, there is the axiom_histogram tool, https://wiki.apertus.org/index.php?title=AXIOM_Beta/Manual#Image_Histogram_Data but its output is not meant to be human readable on the CLI
16:32
se6astian
any idea how to make it easier to use
16:32
se6astian
to see the current exposure levels of the live image
16:33
vup
hmm
16:33
se6astian
a webui addition maybe?
16:34
vup
the way that works is, that it just captures a snapshot and then computes the histogram, right?
16:35
vup
Does that work while streaming raw hdmi currently?
16:35
se6astian
good question, will have to ask herbert
16:35
se6astian
its possible to also capture a axiom_snap while raw streeaming
16:35
vup
I guess then this should be fine aswell
16:35
se6astian
but you get output from several frames as writing the snap is not as fast as HDMI output
16:35
vup
feel free to add a issue to the webui
16:36
se6astian
which would not be a real issue for the histogram graph
16:36
vup
hmm what does "you get output from serveral frames" mean?
16:36
vup
does it not lock the memory buffer until it is done with computing the histogram?
16:36
se6astian
it writes top to bottom IIRC, no locking
16:37
se6astian
so while it writes the image changes
16:37
se6astian
so a block of lines from frame 1
16:37
se6astian
then a block of lines from frame 2....
16:37
vup
hmm that sounds pretty bad, I thought the locking of the memory buffers was there exactly to prevent these kinds of issues?
16:38
se6astian
I dont think cmv_snap does it on its own
16:39
se6astian
a capture script can easily stop hdmi stream, capture raw image, resume hdmi stream
16:39
se6astian
then no issues and only brief stream interuption
16:39
vup
hmm
16:42
se6astian
anyway, just a convenience details for the histogram
16:42
vup
sure
16:42
se6astian
alternatively would it be simpler to add a histogram or something even more simplified in the recorder?
16:43
se6astian
in the debayer module for example
16:43
se6astian
as overlay
16:43
vup
well
16:44
vup
in theory that should not bee that hard
16:44
se6astian
anyway, thats again details
16:44
se6astian
quick updates from me:
16:44
vup
but without hacks that would need narui integration
16:44
se6astian
the wiki has been upgraded and new theme
16:44
se6astian
https://wiki.apertus.org/index.php?title=Main_Page
16:45
vup
(fyi, I just noticed that https://wiki.apertus.org/index.php/CMV12000_Register_Blocks is broken)
16:45
se6astian
also a small working group has been formed to work on the website content/structure improvements
16:45
se6astian
infographics, etc.
16:45
se6astian
next VC for this on Thursday evening (CET)
16:45
se6astian
message me if you want to take part
16:45
se6astian
we are 4 currently
16:46
se6astian
next topic
16:46
se6astian
GSoC org applications are on
16:46
se6astian
we now have two weeks to apply again for 2022
16:47
se6astian
which means you need to decide who wants to mentor tasks this year
16:47
se6astian
and define tasks, etc.
16:47
se6astian
clean up idea page
16:47
se6astian
as the idea page is the central part of the application
16:48
se6astian
also there is an interesting grant currently open
16:48
se6astian
https://forum.openhardware.science/t/apply-here-for-gosh-s-2022-collaborative-development-program-round-1/3380/2
16:49
se6astian
I want to see if we could apply to get the CP enclosure design from prototype to pilot
16:49
se6astian
and involve a few early adopters to get one in the process
16:51
se6astian
thats it from my side
16:51
se6astian
anyone else with news to share?
16:54
se6astian
ok then! many thanks vup for the update and progress!
16:54
se6astian
MEETING CONCLUDED
18:20
balrog
joined the channel
18:37
jn
left the channel
18:42
jn
joined the channel
18:46
davidak[m]
left the channel
18:47
McUles[m]
left the channel
18:58
Bertl_oO
se6astian: vup: the logic for locking a buffer is already there, it just isn't used in the snap tool for now, so it should be fine to lock an arbitrary buffer down first, then simply dump the buffer e.g. with memtool instead of running a snap
18:58
vup
right
18:58
vup
thats sounds like a good idea
18:58
Bertl_oO
you do not want to change any register values anyway and you also do not need to trigger any exposures
18:58
vup
yeah
18:59
Bertl_oO
the raw memory dump can later be converted easily with the neon tool ;)
19:00
Bertl_oO
(which should be done after the capture sequence I guess)
19:01
vup
right
19:06
Bertl_oO
vup: one question, you mentioned the nonlinearity is independent from the exposure time
19:06
Bertl_oO
but the explanation before sounds like there were only two exposure times compared?
19:07
vup
What I meant was, that the pixels that show this behaviour are always the same and do not change with changing exposure time
19:08
Bertl_oO
so you did various comparisons (with different exposure time pairs) and found the same nonlinearity across all of those, yes?
19:09
Bertl_oO
because basically with a dark frame we do a single point calibration, with a dark and flat a two point (linear) calibration and only with different flats (exposure time or light intensity) we can do a multipoint (nonlinear) calibration
19:10
vup
yes exactly
19:10
Bertl_oO
what I was also wondering is, what happened to the light sphere we hacked together? maybe this might be helpful with the flats?
19:10
vup
I was assuming that is what se6astian was using, but maybe thats wrong?
19:11
Bertl_oO
ah, might be and would make sense
20:04
davidak[m]
joined the channel
20:07
McUles[m]
joined the channel
20:18
se6astian
I am not using the integration sphere for the snapshots captured
20:19
se6astian
its stored in the office and as big as my entire desk so I am afraid currently not practical
20:20
se6astian
so for the capture script how do I lock the buffer and dump a buffer with the memory tool?
20:21
se6astian
btw just saw an article about the man of spheres were we also got the parts for the integrating sphere
20:21
se6astian
https://www.derstandard.at/story/2000133180697/runde-sache-besuch-in-wiens-einzigem-kugelgeschaeft
20:35
McUles[m]
left the channel
20:35
davidak[m]
left the channel
20:38
Bertl_oO
interesting, glad to read they somehow survived after all, I remember getting the notifications that they will close down sometime before 2020
20:38
davidak[m]
joined the channel
20:44
se6astian
no they "kept the ball rolling"
20:45
McUles[m]
joined the channel
21:05
Bertl_oO
seems like it ...