Current Server Time: 16:50 (Central Europe)

#apertus IRC Channel Logs

2022/02/07

Timezone: UTC


00:24
Bertl
off to bed now ... have a good one everyone!
00:24
Bertl
changed nick to: Bertl_zZ
06:17
aombk2
left the channel
06:22
aombk
joined the channel
08:39
Spirit532
left the channel
08:39
Spirit532
joined the channel
11:16
Bertl_zZ
changed nick to: Bertl
11:16
Bertl
morning folks!
15:50
Bertl
off for now ... bbl
15:50
Bertl
changed nick to: Bertl_oO
16:19
balrog
left the channel
17:00
se6astian
MEETING TIME!
17:00
se6astian
who is here?
17:00
se6astian
Bertl notified me he has an urgent appointment currently and will likely miss the meeting
17:00
se6astian
maybe join later on
17:00
vup
is here
17:01
se6astian
vup: any news/progress for us to share?
17:01
vup
sure
17:02
vup
I looked a bit into the flatfields that se6astian captured
17:03
vup
In general, these were captured with a as uniform as possible light source and averaged over 256 frames
17:03
vup
the averaged flatfields were dark current compensated by subtracting the average of a 256 darkframes at ~14ms
17:04
vup
(the flatfields were captured at ~2ms, ~3ms, ~7ms and ~14ms)
17:05
vup
In general these flatfields were captured at 1x gain and under rather low light conditions (the ~14ms exposure only yields a average pixel value of ~50 after darkframe compensation)
17:06
vup
the first thing I looked at, was the standard deviation per pixel over the 256 darkframes for the exposures
17:07
vup
the standard deviation here is on average ~6.2 with a σ (of the standard deviation) of ~0.1
17:07
vup
this is roughly the same as plain darkframes
17:09
vup
now just visually inspecting the flatfields already shows some inhomogenity
17:09
vup
https://files.niemo.de/flatfield.png
17:10
vup
(if all pixels had the exact same response curve we would expect all pixels with the same color filter to have the same value)
17:11
vup
visually the pixel values mostly look similar per column, but seem to change for the columns
17:13
vup
so I tried to find a pattern in the average values along each column again (like I did for the darkframes)
17:13
vup
using only pixels in even columns and rows (to select pixels with the same color filter)
17:15
vup
one can again find some strong patterns when looking at columns in steps of 32
17:15
vup
(looking a column 0, 32, 64, at 1, 33, 65, etc)
17:15
vup
(these numbers refer only to even columns, so column 32 is actually column 64, because we are only looking at even columns)
17:16
vup
plotting these one gets something like this: https://files.niemo.de/column_wise_averages_1.9932959999999997.png
17:16
vup
https://files.niemo.de/column_wise_averages_2.9964.png
17:16
vup
https://files.niemo.de/column_wise_averages_6.996432.png
17:16
vup
https://files.niemo.de/column_wise_averages_6.996432.png
17:16
vup
whoops this one: https://files.niemo.de/column_wise_averages_13.993392.png
17:17
vup
ok, but only looking at a single flatfield without comparing to flatfields with different exposures doesn't really give us a lot to work with
17:17
vup
so the next step is comparing the flatfields with different exposures to each other
17:19
vup
now if the response curve of each pixel would be linear, one would expect that the ratio of pixel values of at two different exposure was equal to the ratio of the exposure times
17:19
vup
this is however not always the case
17:19
vup
is seems that there are "two" groups of pixels, ones that are roughly linear and ones that are not
17:20
vup
the ones that are not seem to only occur in odd rows
17:20
vup
where they make up roughly 50% of the pixels
17:21
vup
it seems that these "nonlinear" pixels are in general a lot less sensitive than the "linear" ones
17:21
vup
which can be clearly seen, when looking at the ratio of the pixel values between two different exposures in dependence of the pixel value of one of the flatfields
17:21
vup
https://files.niemo.de/corr_1.9932959999999997_by_13.993392.pdf
17:21
vup
https://files.niemo.de/corr_2.9964_by_13.993392.pdf
17:22
vup
https://files.niemo.de/corr_6.996432_by_13.993392.pdf
17:22
vup
(here 1.99... by 13.99... means that the ratio is the ration of the pixel values of the ~2ms and the ~14ms exposure and the x axis always uses the pixel values of the first exposure time)
17:23
vup
in these histogram we can clearly see these two groups of pixels
17:23
vup
also note, that which pixel is "nonlinear" seems to be independent of the exposure time (they are always the same pixels)
17:24
vup
indeed the location of these "nonlinear" pixels seems to follow a pretty strong pattern
17:25
vup
https://files.niemo.de/weird_pixels_even_rows_even_cols.png
17:25
vup
https://files.niemo.de/weird_pixels_even_rows_odd_cols.png
17:25
vup
https://files.niemo.de/weird_pixels_odd_rows_even_cols.png
17:25
vup
https://files.niemo.de/weird_pixels_odd_rows_odd_cols.png
17:25
vup
in these images a white pixel signifies a "nonlinear" pixel and a black one a roughly linear one
17:26
vup
(there are four images, because each image only shows pixels with the same color filter)
17:26
vup
I think this is pretty much everything I can say about the flatfields for now
17:26
se6astian
very interesting
17:27
vup
For further investigation there are a couple of things that would be interesting
17:27
se6astian
I will create an image acquisition script soon to gather more data about how the calibration images are captured + more metadata
17:27
vup
- using the full width with dark columns available
17:27
vup
- more different exposure times
17:28
vup
- capturing the different exposure times at different light intensity levels
17:28
vup
- capturing a couple of reference images for each capture with axiom_snap
17:28
vup
- capturing darkframes for the different exposure times aswell
17:29
se6astian
all noted
17:29
vup
the last thing that I am not fully convinced on is, how many frames to capture for each
17:30
vup
assuming a standard deviation of 6.2, the standard error of the mean is 6.2 / sqrt(n)
17:30
vup
so 256 frames gives us a standard error of the mean of 0.3875
17:31
vup
ahh
17:31
vup
I think what we should actually do is, do the averaging already in the recorder
17:31
se6astian
that would be great for upload sizes!
17:31
vup
so then capturing averages of many many frames should be pretty easy
17:31
vup
yes
17:32
vup
and for my available disksize :)
17:32
vup
Ill look into adding that feature
17:32
se6astian
wuld it be a trivial software addition for the recorder?
17:32
se6astian
regarding the light intensity levels, there is the axiom_histogram tool, https://wiki.apertus.org/index.php?title=AXIOM_Beta/Manual#Image_Histogram_Data but its output is not meant to be human readable on the CLI
17:32
se6astian
any idea how to make it easier to use
17:32
se6astian
to see the current exposure levels of the live image
17:33
vup
hmm
17:33
se6astian
a webui addition maybe?
17:34
vup
the way that works is, that it just captures a snapshot and then computes the histogram, right?
17:35
vup
Does that work while streaming raw hdmi currently?
17:35
se6astian
good question, will have to ask herbert
17:35
se6astian
its possible to also capture a axiom_snap while raw streeaming
17:35
vup
I guess then this should be fine aswell
17:35
se6astian
but you get output from several frames as writing the snap is not as fast as HDMI output
17:35
vup
feel free to add a issue to the webui
17:36
se6astian
which would not be a real issue for the histogram graph
17:36
vup
hmm what does "you get output from serveral frames" mean?
17:36
vup
does it not lock the memory buffer until it is done with computing the histogram?
17:36
se6astian
it writes top to bottom IIRC, no locking
17:37
se6astian
so while it writes the image changes
17:37
se6astian
so a block of lines from frame 1
17:37
se6astian
then a block of lines from frame 2....
17:37
vup
hmm that sounds pretty bad, I thought the locking of the memory buffers was there exactly to prevent these kinds of issues?
17:38
se6astian
I dont think cmv_snap does it on its own
17:39
se6astian
a capture script can easily stop hdmi stream, capture raw image, resume hdmi stream
17:39
se6astian
then no issues and only brief stream interuption
17:39
vup
hmm
17:42
se6astian
anyway, just a convenience details for the histogram
17:42
vup
sure
17:42
se6astian
alternatively would it be simpler to add a histogram or something even more simplified in the recorder?
17:43
se6astian
in the debayer module for example
17:43
se6astian
as overlay
17:43
vup
well
17:44
vup
in theory that should not bee that hard
17:44
se6astian
anyway, thats again details
17:44
se6astian
quick updates from me:
17:44
vup
but without hacks that would need narui integration
17:44
se6astian
the wiki has been upgraded and new theme
17:44
se6astian
https://wiki.apertus.org/index.php?title=Main_Page
17:45
vup
(fyi, I just noticed that https://wiki.apertus.org/index.php/CMV12000_Register_Blocks is broken)
17:45
se6astian
also a small working group has been formed to work on the website content/structure improvements
17:45
se6astian
infographics, etc.
17:45
se6astian
next VC for this on Thursday evening (CET)
17:45
se6astian
message me if you want to take part
17:45
se6astian
we are 4 currently
17:46
se6astian
next topic
17:46
se6astian
GSoC org applications are on
17:46
se6astian
we now have two weeks to apply again for 2022
17:47
se6astian
which means you need to decide who wants to mentor tasks this year
17:47
se6astian
and define tasks, etc.
17:47
se6astian
clean up idea page
17:47
se6astian
as the idea page is the central part of the application
17:48
se6astian
also there is an interesting grant currently open
17:48
se6astian
https://forum.openhardware.science/t/apply-here-for-gosh-s-2022-collaborative-development-program-round-1/3380/2
17:49
se6astian
I want to see if we could apply to get the CP enclosure design from prototype to pilot
17:49
se6astian
and involve a few early adopters to get one in the process
17:51
se6astian
thats it from my side
17:51
se6astian
anyone else with news to share?
17:54
se6astian
ok then! many thanks vup for the update and progress!
17:54
se6astian
MEETING CONCLUDED
19:20
balrog
joined the channel
19:37
jn
left the channel
19:42
jn
joined the channel
19:46
davidak[m]
left the channel
19:47
McUles[m]
left the channel
19:58
Bertl_oO
se6astian: vup: the logic for locking a buffer is already there, it just isn't used in the snap tool for now, so it should be fine to lock an arbitrary buffer down first, then simply dump the buffer e.g. with memtool instead of running a snap
19:58
vup
right
19:58
vup
thats sounds like a good idea
19:58
Bertl_oO
you do not want to change any register values anyway and you also do not need to trigger any exposures
19:58
vup
yeah
19:59
Bertl_oO
the raw memory dump can later be converted easily with the neon tool ;)
20:00
Bertl_oO
(which should be done after the capture sequence I guess)
20:01
vup
right
20:06
Bertl_oO
vup: one question, you mentioned the nonlinearity is independent from the exposure time
20:06
Bertl_oO
but the explanation before sounds like there were only two exposure times compared?
20:07
vup
What I meant was, that the pixels that show this behaviour are always the same and do not change with changing exposure time
20:08
Bertl_oO
so you did various comparisons (with different exposure time pairs) and found the same nonlinearity across all of those, yes?
20:09
Bertl_oO
because basically with a dark frame we do a single point calibration, with a dark and flat a two point (linear) calibration and only with different flats (exposure time or light intensity) we can do a multipoint (nonlinear) calibration
20:10
vup
yes exactly
20:10
Bertl_oO
what I was also wondering is, what happened to the light sphere we hacked together? maybe this might be helpful with the flats?
20:10
vup
I was assuming that is what se6astian was using, but maybe thats wrong?
20:11
Bertl_oO
ah, might be and would make sense
21:04
davidak[m]
joined the channel
21:07
McUles[m]
joined the channel
21:18
se6astian
I am not using the integration sphere for the snapshots captured
21:19
se6astian
its stored in the office and as big as my entire desk so I am afraid currently not practical
21:20
se6astian
so for the capture script how do I lock the buffer and dump a buffer with the memory tool?
21:21
se6astian
btw just saw an article about the man of spheres were we also got the parts for the integrating sphere
21:21
se6astian
https://www.derstandard.at/story/2000133180697/runde-sache-besuch-in-wiens-einzigem-kugelgeschaeft
21:35
McUles[m]
left the channel
21:35
davidak[m]
left the channel
21:38
Bertl_oO
interesting, glad to read they somehow survived after all, I remember getting the notifications that they will close down sometime before 2020
21:38
davidak[m]
joined the channel
21:44
se6astian
no they "kept the ball rolling"
21:45
McUles[m]
joined the channel
22:05
Bertl_oO
seems like it ...