#apertus IRC Channel Logs

Good Morning!

morning folks!

I am struggling to find a optimal solution to demosaic the borders

off for now ... bbl

values for borders should be copied from next available ones, and then adjusted through luminance or something like that, just so that they are not totally different

so, off for some time, will be back later

Yeah the other stuff is working

Yes definitely

anuejn2: OK, then it might be worth starting a separate page so that we can evaluate what's suitable for where. At the moment that section is kind of an overview, in which case, and if we maintain that feel, a link to a dedicated page referring to the technical details of plugin modules might be best (or at least something to consider) - https://wiki.apertus.org/index.php/AXIOM_Beta/Camera_Structure/Plugin_Modules

hi

How are you?

fine thanks, a bit busy as you might have noticed

Wait, who is "we" ?

hi TofuLynx

we had yesterday (SUN) at 18.00 and today (holiday in germany) at 19.00

thats why i am here at 18.00

Did you see what I wrote after you went afk?

yes

hi

What kind of meeting? I know nothing

Sebastian send a mail on apertus-gsoc about weekly meeting 2-3 days ago I guess

Good that you're already here, but would recommend you to join that group

TofuLynx: it was in the mails

Ok, I have to add myself to the mailing list to be notified

wait, but I am on the group. How do I get notified?

BAndiT1983 : I'm kind of tired because exams going on, but just one exam is left which is on 23, so hopefully after that I can breathe a little!

is end of may general phase of exams?

Some Universities get over with it in early May, but my Uni is kind of late in such stuff. So, it ends on almost end of May

hope that it does not affect gsoc too much

Hopeful not :)

BAndiT1983 : It's almost over, after 23 I'll have almost 2 months off as summer vacation! :D

BAndiT1983: Do I have to prepare something for the meeting?

Good evening supragya!

:)

hi there

Hello,se6astian!

I am good

re

got my mail?

TofuLynx: no, it's just about current states, problems and so on, general overview usually

It's tuff for my network to hold up for IRC packets today..

perfect timing

so a very warm welcome to everyone

this is the first official GSoC, student, mentor, community & team meeting

same time same place, every week

http://irc.apertus.org/

note the timestamps are UTC

the basic idea behin the meetings is that the students and mentors can coordinate

Weekly meeting sounds good :)

:-)

after all we want us and you to also learn from each other

to give everyone a chance to talk/report after another (instead of all at the same time - can get confusing) what worked well so far is that one person leads the meeting

So, who will lead it?

that person then asks every speaker one by one to have a go

for today that will be me, but it doesnt necesarily be me every time

Nice :)

while I collect PMs a questions from my side: Bertl whats the current situation with the IRC bouncers, ZNC?

for students/mentors in particular

great

Bertl : Give me two days I'll set it. Currently all my time goes to final.

supragya: okay, sorry, just saw your supragya_2 :)

I don't trust my Hexchat today...

let me know (pm) if you have any problems with access to the IRC bouncer or need/want an account there

sure :)

ok then Iti_ you were the first to PM me, please feel free to talk now about your progress so far, etc.

So, on April 27 Bertl gave me some tasks to complete. Here are the details of my approach and things that I have completed.

May 6 - Calculating data size and required bandwidth, this gave me some required information.

May 9 - Figuring out what data is used for which visualization (Bertl helped me in this as I wasn’t able to clearly understand some things )

May 13 - Figuring out how does mmap work and what area needs to be mapped.

and also about "How long does it takes to access an entire frame + figure out what access pattern and good and what are not

I still need feedback on those two tasks

This would help me in testing my code on beta (on which I have been provided root access )

After 23, I'll working on the assigned task and will update my status here

great! thanks for the detailed update!

Which I keep updating (Also I am always updating on irc)

https://trello.com/b/pSdCOkMt/untitled-board

That's great!

Iti_: If you need, I have a link for a pdf which explains a lot about low level functions such as mmap

Thanks TofuLynx I did already read on that. But I'd like to see your pdf as well

And, for timestamping my progress, we decided to use a google doc, but it hasnt been used and I am thinking about creating a Trello Board

The unit tests, in my opinion, were the most important thing I learned in that time. They are so valuable and help a lot with bugs

So, after this, I started working on my task: I have to implement a Bilinear Demosaic for OpenCine

However, I found some issues with the OpenCine loading DNG images, which have to be resolved

great, thanks!

anyone else who wants to talk/share?

please do!

My project is to actually figure out a good way to get video streams out of Beta and give a good enough format for that

while many formats showed promise... we wanted the format that we chose to be simple.. that AXIOM Beta can handle easily

MLV seemed like a great choice.. because it is easy to encode and much of it's codebase could be taken up for our use... since it is C

So, we began with an idea in mind as follows:

Keeping AXIOM Beta end as black box (not knowing what it can do), we were to look into what performance was to be expected

An in memory emulation (on i7-4710HQ, DDR3 1333MHz) of MLV through this mechanism showed promise

when passing RAW12 as MLV blocks (in memory)

if we expect x10 or even x20 degradation in speed, using SSD RAID 0

we can still achieve over 300 frames per second RAW12 streams

so it's fast... 15GB/s

nevertheless, the next leg of our discussion will be around HDR using PLR

and how to incorporate that in our MLV based system

but 7000 FPS, 4k raw is 132Gigabyte/s no?

I am more on what to expect..

the result is that 60fps... could be handled quite easily

TofuLynx: PLR (piecewise linear response) is a HDR (high dynamic range) mode of the image sensor

it allows increasing the dynamic range by selectively reducing highlight sensitivity with 2 kneepoints

i'd wait with numbers till the prototype is fully working. memcopy speeds are not telling much :)

also in memory control is very linient...

supragya: what are the considerations with PLR and the container? metadata for non-linear rawdata?

however, we have to look how to put that into MLV

(hi, btw)

we had this discussion and are not quite sure if the CDNG format would support it properly. well CDNG probably will, but we dont know about the tools that should process the CDNG

2. MLV-> CDNG convertor has to be changed accordingly too

In general, I feel we need to know the exact use cases and the representations surrounding this PLR, both me and g3gg0

as a starting point we decided to potentially convert the PLR into linear raw when writing either MLV or CDNG

so the CDNG has linear raw for the start

yep

I need to know how that mapping is to be done

because it isn't completely map-able to linear

PLR poses problems for publisher (MLV-> CDNG)... we would really have to look at encoding once again

or do you need too much post processing for that?

:)

yes, definitely

if "we" cannot map, i doubt some CDNG capable post processing tool will be :)

sure...

ArunM: your turn!

se6astian: just ping us all in (Bertl, you, g3gg0, me, BAndiT1983, alexML etc...) on our meet

time wise

Oops

se6astian: I have a question

After thoroughly understanding the datasheet, I’ve already posted detailed architecture for my project CMV12000 emulation/simulation

If, at the end of GSoC, I had a good documentation of what I did and my experience during GSoC, would you publish it on Apertus website?

TofuLynx: definitely

I have completely programmed two modules (ROW_BUFFER, SPI Interface) and currently in the middle of third (SERIALIZER)

After that I am going to integrate all these modules with SEQUENCER and also write final test bench for the complete system

but i think problems may arise as I start programming SEQUENCER.

Thats all

its not organised, okay i'll upload it tommorow only

make sure to have copyright notice and (FOSS) license on the files

okay

i.e. it's fine if you need some time to clean it up

:-)

anyone else want to share/say/discuss anything?

maybe just as information for the students:

it will happen on the next weekend

se6astian: Where are our other GSoC fellows? just curious!

So, do we have meeting next monday?

Okay, cool!

rahul- seems to be 'virtually' present but not around right now

have a good night everyone! :)

feel free to stick around and discuss/chat of course

good night!

good night g3gg0

Good night everyone!

(have to leave for a hour)

(am I correct ?) :)

almost! "gute nacht" :)

supragya : right xD

supragya if you want we can chat a bit about PLR now

Loved it!

sure

this means you have different images to fuse together

with motion this can lead to problems as the motion continues with each bracketed image

and a lot of data in general

seen such artifacts myself... takes a toll on sharpness at very least

which basically gives you two bracketed images, both reduced in horizontal or vertical resolution

right

correct

that's dual ISO... mlv thing

Bertl: There is available I2S port in Shield section? or any interface protocol can be made in FPGA side?

now PLR is interesting because the fusing of different exposures already happens on the image sensor level

the result is one image 12bit 4096x3072 but you can select 1 or 2 kneepoints for the response curve

and for each kneepoint you can set voltage level and turn-off time IIRC from the datasheet

is kneepoint something like keyframes in timeline (analogy)

and response curve is a bezier?

supragya: yes you can see it that way

but the response curve is in general linear

and the kneepoint defines where two linear sections meet

http://www.nocturncamera.com/image/hrd2.png

and for another interval (T2) the exposure is limited to this voltag

Bertls explanations are visualized here: http://www.nocturncamera.com/image/hrd.png

BAndiT1983: Are you online? Can I add you to the trello board?

you are breaking a straight line to span over higher HDR

but then here are a few issues... are we losing details in middle?

it is always a tradeoff because 12 bits are 12 bits whatever you do :)

roughly speaking, 3 overlapped exposures (with 3 different exposure times) encoded in the same image

is it always 3?

you can have 1 or 2 kneepoints

i seem to get a hold of it.. a little bit

i was wondering of artifacts in such scenario

how they look and when they occur

yes

I mean even if sensors are swapped

other sensors will have completely different features

TofuLynx: yep, am online, was just doing household stuff

was there the question about the pattern already? is it always fixed for the CMV12000 or does it change from sensor to sensor??

as always -> two ways

a few ways would be:

1. to encode PLR info in meta stream and send it over

I also think the best way is to save raw data and metadata (your 1.) and decide in post how to utilize it

it is doable.. but really how many would like to do it, any idea

completely different featureset

but who knows what will become available in the future

because incorporating PLR such as this in system would only bolster use of CMV12k more... and other sensors will have to use.. a different streaming system and a differenct version of recorder maybe

in any case, the concept of saving raw image data and all metadata the sensor provides works with any sensor

for correct benchmarks on PC recoder + AXIOM Beta, I may need as I said... atleast SSD RAID0 to get the capabilities

lets see what we can organize

another topic for PLR but also linear raw data is a viewfinder LUT

BAndiT1983: Can you send me your email via the chat at lab?

bitdepths, etc.

viewfinders normally operate on 8 bit per channel color images

generated from 12 bit raw

problem is that a 12 bit image converted to 8 bit just by using the MSB looks very contrasty

so in the beta we apply a gamma curve

it would be helpful if the raw format could also store this gamma curve somehow

gamma

like a CURVES adjustment?

off for dinner for a bit

per channel or global

a bezier?

hi :)

for preview you may get away with a global curve though

alexML: is that so? the basis of this is the imperfect nature of pixel in relation with others....

or am I missing some reason... why would every pixel need a curve caliberation of it's own

TofuLynx: done

hi supragya, fine, and you?

why are there multiple accounts of yours?

seen the logs

g3gg0: have you seen the logs regarding PLR?

supragya, every sensel has some offset

maybe also on how clean the micro-lenses are

that is taken care of by caliberations (dark frames) right

Ok BAndiT1983, done

with PLR it's not just a single offset, or just an offset + gain

offset... regarding voltage/ readouts ? is it something that is corrected simply by [actualvalue - offset] ?

probably curve

feel free to try; I've got some gigabytes of reference frames to interpret, but don't have a working PoC yet

You will eventually understand it! :)

any slight variation in the knee point of a pixel would cause serious noise artefacts?

g3gg0: I guess not

for this reason does every single pixel need a proper calibration?

maybe some patterns etc, but not deterministic?

I'm pretty sure most of it is deterministic, just hard to calibrate with a global curve

from overexposed to nearly pitch black

hmm sounds the best way is to do X * Y kalman estimators :D

* just lost it completely now :D

yes

so a sensel's value rises non-linear

if a scene continuously gets brighter, the sensel's values will raise at three different rates

the first derivate would have a jump

alexML: already did some quite accurate estimations of the response curve: https://wiki.apertus.org/index.php/PLR

as in?

some pixels/rows/columns/whatever patterns darker/brighter than others

is this because of pixels getting excited around a kneepoint extrapolating the effect?

more like enhancing the effect

but when there are regular patterns of such, people will notice

yes we have an integrating sphere for such tests/calibrations

(its always better to prevent errors than to compensate)

i hope i understood correctly, alex?

so, this per pixel PLR is one big chunk of data that is to be saved per recoding/ per camera?

no LUT

2048 x 1088 x 2^12

9GiB of LUT

as you don't need the full range, it could be compressed in linear areas

its very expensive when processing then

@alexML

(9:16:54 PM) sebastian: per image sensor, per PLR settings most likely

*offloaded

it can pretty likely be calibrated once and stored for later use in postprocessing

the PLR settings can be given along with MLV to encode a CDNG then

i'd place it there for now, yeah

constant

dng has linearization table for compression

so most are 12 or 14 bit

it's the standard way, you cannot omit it just like thatz

gamma curves are also per pixel?

[gamma curves are also per pixel] -> silly question I guess :)

it should be a global thing i suppose

its actually a single value for the entire image, like gamma "0.6"

gamma "0.6" would map 12bits to 8bit how?

dng specs say something about gamma curve

supragya, first is the processing line done, before you convert it down, so linearize before downscaling the values

I would be leaving for now

or whatever daytime it is where you are right now :D

Its 01:00 AM here

Just saying that the pattern issues i have been facing were OpenCine's fault and I have solved it. Now the images appear perfect :D