Current Server Time: 09:37 (Central Europe)

#apertus IRC Channel Logs

2013/12/30

Timezone: UTC


23:17
rexbron_
left the channel
23:18
rexbron
joined the channel
23:19
troy_s
left the channel
23:19
troy_s_
joined the channel
23:26
troy_s
joined the channel
23:27
troy_s_
left the channel
23:32
troy_s
Bertl: Can we get a reliable file extension for the raw data Herb?
23:32
troy_s
Bertl: Something like .AXI
23:33
Bertl
do you have an url to a spec for .axi? and what do you mean by 'reliable' and 'raw data'?
23:36
troy_s
Bertl: Well I'm thinking ahead slightly. I'm about to write a stub for OIIO.
23:36
troy_s
Bertl: And it relies on a file extension.
23:37
Bertl
relying on an extension is always a bad thing, but that's unrelated here I guess
23:39
rexbron
left the channel
23:39
rexbron
joined the channel
23:39
rexbron
left the channel
23:39
rexbron
joined the channel
23:39
troy_s
left the channel
23:39
troy_s_
joined the channel
23:40
troy_s_
Bertl: So raw16 isn't great.
23:40
troy_s_
Make sense?
23:41
Bertl
yeah, well, .raw16 is just raw data, you can name it .data or .wossname as well
23:42
Bertl
i.e. it isn't a file format per se
23:43
Bertl
gabe has written a converter to convert it (lossless) to dng and to regenerate the .raw16 data from those dngs without any loss
23:43
Bertl
not knowing .axi, I can't tell how much work a similar converter would be
23:44
troy_s_
Bertl: It's just a made up extension.
23:45
troy_s_
Bertl: If you are writing the firmware, at least put a magic code or something at the head pehaps?
23:45
troy_s_
Bertl: Because otherwise reader / writers have no way of knowing if the data is useful or in the correct format.
23:45
Bertl
well, that's what I mean, a file format should have some mime magic
23:45
troy_s_
It doesn't need to be complex.
23:45
Bertl
but .raw16 is not intended as a format
23:45
troy_s_
Just a magic header value or width / height or whatever the hell is useful.
23:45
Bertl
i.e. I don't want to create a new format here :)
23:46
troy_s_
Too late.
23:46
troy_s_
You already are dumping data.
23:46
troy_s_
That's a format.
23:46
Bertl
it is raw data
23:46
troy_s_
Every single camera dumps raw data
23:46
troy_s_
they still encapsulate it in a meaninful wrapper
23:46
Bertl
what mime magic/format does .txt have in your book?
23:46
troy_s_
The extension.
23:46
Bertl
(i.e. ascii raw text)
23:46
Bertl
well, .raw16 has the .raw16 extension, no?
23:46
troy_s_
My ponit is that all of OIIO relies on extensions to identify the initial file.
23:47
troy_s_
It's kind of a crap extension, yes?
23:47
troy_s_
:)
23:47
Bertl
hmm, why?
23:47
troy_s_
Because it doesn't tell you what it is from.
23:47
troy_s_
And it isn't exactly a useful descriptor.
23:47
Bertl
and .txt tells you that?
23:47
troy_s_
Yes.
23:47
troy_s_
.TeXT
23:47
Bertl
so where is my AXIOM.txt from?
23:48
troy_s_
like .AXIom
23:48
troy_s_
Oh god.
23:48
troy_s_
If you look up the history of .txt, you can likely find that it includes ASCII
23:48
troy_s_
.raw16 doesn't mean anything
23:48
troy_s_
and in fact the data is well known and understood
23:48
troy_s_
it's the data from a bayer sensor on a prototype axiom camera
23:48
troy_s_
it isn't just random
23:49
troy_s_
but as one of the fellows that might try to write some code for decoding the file, how about an extension and a magic number as a test?
23:49
troy_s_
Something super simple.
23:49
Bertl
yes, but obviously there is no generic format for that which doesn't involve a million tags and complicated structures like dng
23:49
troy_s_
Or a magic word sized code and a version number if you want to get really fancy.
23:49
troy_s_
I'm not telling you that.
23:49
troy_s_
I'm asking for an extension that clearly tells us it is from the axiom camera. :)
23:50
Bertl
we will do that once we know that we 'need' or 'want' a specific axiom format
23:50
troy_s_
Ideally with a simple check to avoid breaking a reader
23:50
troy_s_
So a magic first word header is too much?
23:50
troy_s_
two bytes?
23:50
gcolburn
good news! I patched the bayer2RGB code to work on the .raw16 file
23:50
troy_s_
Or four?
23:50
Bertl
yes, because it breaks the ability to access the data easily
23:50
troy_s_
You aren't hearing me.
23:50
gcolburn
it successfully ran with VNG
23:50
troy_s_
If I try to write a plugin loader and
23:51
troy_s_
I have to rely on raw16 as the extension with no helper check
23:51
troy_s_
the chances of failure go up huge
23:51
Bertl
e.g. now I can do display -size 4096x3072 -depth 16 gray:example.raw16
23:51
troy_s_
In imagemagck?
23:51
Bertl
(without implementing anything)
23:51
Bertl
yes
23:51
troy_s_
Urgggghhhhhhhhhh...adlfkja;lfkjasd;flkasjf;lasjf;lasfkjas;dlfkjasd;lfkj
23:51
troy_s_
Whatever.
23:51
troy_s_
Moving on.
23:51
Bertl
LOL
23:51
troy_s_
Enjoy staring at your black and white bayer pattern
23:52
Bertl
as I said, no problem, just wrap it with 4 bytes if that makes your reader happy
23:52
troy_s_
gcolburn: VNG?
23:52
Bertl
like cat AXI.magic example.raw16 >example.troy
23:54
gcolburn
Bertl: The bayer2rgb code was not calling the 16 bit converter when you do -b 16
23:54
gcolburn
so I believe it was assuming 8
23:54
gcolburn
so I just put the function call in LOL
23:54
troy_s_
gcolburn: VNG?
23:55
gcolburn
and presto, the tiff comes out correct
23:55
Bertl
gcolburn: nice, well, I didn't bother, but good to know
23:55
Bertl
maybe send the changes upstream and check the tool into the axiom repository?
23:57
gcolburn
yeah
23:57
troy_s_
Is it at GitHub?
23:57
gcolburn
what's funny is near as I can tell the -swap option for 16 bit isn't implemented
23:57
gcolburn
I searched the code. it seems to set a flag and not use it
23:58
Bertl
yes, that was what I figured
23:58
gcolburn
I get the same image with or without adding -swap
23:58
gcolburn
but it works, which is all that matters
23:58
Bertl
great!
23:59
troy_s_
gcolburn: Now onto that stub.
00:04
Bertl
troy_s_: isn't there a format in OIIO which already supports raw data in bayer matrix?
00:04
Bertl
(I would assume that there has to be one which could be used)
00:14
Bertl
if not, maybe a generic 'raw' (not in the camera specific raw sense) plugin would be appropriate
00:14
Bertl
i.e. something where you specify the size (width, height), the bitdepth and the bayer pattern order
00:29
troy_s_
Bertl: No
00:29
troy_s_
there isn't.
00:30
troy_s_
Larry has been developing code for too long to make such open ended bits of rubbish.
00:30
troy_s_
It's a very focused library.
00:30
troy_s_
It makes a number of assumptions and design context decisions... only associated alpha, no random different bit depth, only a very limited subset, etc.
00:31
troy_s_
The stub for the camera wouldn't ever be useful upstream either, so it will be a maintained thing. That said, it's vastly more useful than a DNG exporter etc.
00:32
troy_s_
DNG is a great target for the raw data to be converted to for folks who want to dicker with the debayering etc. I'm only going to implement a very focused bayer -> VNG -> RGB (of a given colorspace) implementation if I can get it working.
00:32
troy_s_
The idea that we are testing on DNG is pure lunacy.
00:32
Bertl
so why not do a plugin as I suggested then? should do the trick and doesn't need to support any sizes/bitdepths we do not encounter
00:32
troy_s_
Testing should be much more focused and likely should be a very very very very simple command line thing so that we eliminate all variables.
00:33
troy_s_
I'm doing the Axiom plugin. Or at least I intend to. Trying to get my build environment working as I need.
00:33
troy_s_
I'm only focusing on the raw data format you are dumping from the camera (hence my request to have the extension be .axi or something particular)
00:34
troy_s_
DNG is again, a no argument discussion when it comes to storing the raw data into a single frame if needed. My gut says that we should likely dump the speediest raw format we can to disk though. Rather like ArriRaw.
00:34
troy_s_
And then provide stubs as required to go from ArriRaw to whatever.
00:34
Bertl
what I mean is, why does it need to be an 'axiom' data format, why can't it be a raw, 16 (soon we will switch to 12 :) bit linear (gray) image with bayer pattern?
00:34
gcolburn
troy: you can feel free to pick an extension for the raw files for the OIIO plugin purposes
00:34
troy_s_
It doesn't need to be a data format.
00:35
troy_s_
You misread my request. It was purely identification purposes with a _teeny_ check in there (just in case someone has a malformed file or something)
00:35
Bertl
like a checksum or so?
00:35
troy_s_
Yep. Back in the ages of yore I remember having my readers go apepoop over malformed files.
00:35
Bertl
we can easily append a crc32 at the end, if that helps
00:35
troy_s_
(Which was sadly more common)
00:36
troy_s_
I see your need for a basic "display foo" but I'm merely trying to get a plugin in place that makes our ability to go to say, EXR or DPX or JPG a click away.
00:36
troy_s_
As opposed to the current nightmare of DNG.
00:36
troy_s_
(Which again, I think is a really sub-optimal focus at this point given our design constraints.)
00:37
Bertl
well, my 'to jpg' (or png) is trivial as well
00:37
troy_s_
(What I _really_ don't want to see is DNGs getting posted and then countless idiocy spiralling out of control when someone starts talking about color being off only to discover that their FooBarDNG reader was malforming or doing something odd.)
00:37
troy_s_
Bertl: More trivial with OIIO.
00:38
troy_s_
Bertl: We write one plugin stub and we get TIFF, JPG, DPX, EXR, PNG, TGA, etc. all for free.
00:38
troy_s_
Which isn't a bad thing.
00:38
troy_s_
I'd focus on two points this early on for testing purposes.
00:38
troy_s_
EXR and DPX
00:38
Bertl
I agree, I'm all with you on that
00:38
troy_s_
largely because it accomplishes a crapload with a very simple bit of elegance.
00:39
Bertl
so an crc32 at the end would help you? we can do that easily and older raws can be easily 'upgraded'
00:39
troy_s_
1) You can't screw up an EXR. It is _always_ linearized data. The ONLY conversion we will be coupling in is VNG (best of breed arguably at the moment, and certainly high enough rough quality) and color matrix (hopefully via some better profling)
00:40
troy_s_
I don't know. The readers all start at the beginning. I suppose I could do a quick calc. I'm only thinking that for our purposes some sort of really trivial check would be at least a good step forward for reading files and validating that they are in fact decent.
00:40
troy_s_
2) By focusing on EXR and DPX you eliminate a whole subset of folks that likely aren't quite the target demographic for evaluating the images.
00:41
troy_s_
(as in if someone doesn't know what an EXR or DPX is, they are less likely to download the image and gripe about incorrect image results which is arguably happening already.)
00:41
Bertl
well, I have no clue about OIIO plugins, IMHO the only relevant question is how such a plugin gets selected
00:41
troy_s_
So a simple stub plugin for OIIO gets us well on our way without worrying about all sorts of libraries and cruft.
00:41
troy_s_
Yep
00:41
troy_s_
The first line in OIIO is obviously extension
00:41
troy_s_
Then the rest is up to the plugin to validate.
00:41
troy_s_
Pretty simple and lean.
00:42
Bertl
so you need a specific extension (or set of extensions) and a way to say, yep, that's mine?
00:42
Bertl
i.e. all plugins registering for an extension will get a chance to process the data?
00:46
troy_s_
Bertl: I just think it's probably easier to say "Ok... this plugin wants the Axiom Alpha Prototype file so only load .axi
00:46
troy_s_
Bertl: Just like a jpeg
00:46
troy_s_
Bertl: Which is nothing more than the first test. I'm not saying we commit to anything.
00:46
troy_s_
Bertl: As we can always craft the stub to be whatever we want.
00:47
troy_s_
Bertl: No ridiculous versioning etc.
00:47
troy_s_
But your CRC idea isn't an awful one.
00:47
troy_s_
And our resolution is always fixed right?
00:48
Bertl
at the moment, we generate the 4k images
00:48
troy_s_
(Unless I guess Axiom decides to offer a lower res bayer for offspeeds or whatever... all stuff that isn't really a huge thought at the moment.)
00:48
troy_s_
another quick question Bertl, Gabe is hitting a wall with some of the calibration / profiling issues
00:48
Bertl
i.e. 4096x3072 pixels, color or monochrome (depending on the sensor)
00:48
troy_s_
So I was hoping that you might be able to get that noise reduction in place sooner rather than later?
00:49
troy_s_
I ask this because if we get to a 5 error profile, and evaluate the gamut... what happens if we find out that it is smaller than 709 or sRGB?
00:49
troy_s_
See my concern?
00:49
Bertl
yeah, I'm still fighting the xilinx tools to get the new code finished
00:49
troy_s_
We really should get that profile locked in as if the sensor has a ridiculously small gamut... well you won't be winning any supporters with that sensor.
00:49
troy_s_
And all this work is for naught.
00:49
Bertl
it will fix all the LVDS issues, but the tools are stubborn
00:49
troy_s_
I bet.
00:53
troy_s_
Bertl: Right now I think we are zoning in on getting the image quality up to snuff and all that it entails.
00:53
troy_s_
Bertl: Everything else should be secondary, as there are evaluations that are needed before moving on I'd bet.
00:54
troy_s_
Bertl: Especially in the "Is this sensor suitable" zone.
00:54
troy_s_
That gamut would worry me to move on without further knowledge.
00:54
Bertl
yeah, but I don't see a point in using images where we cannot guarantee that we have the actual sensor daa
00:54
Bertl
*data
00:55
troy_s_
I have a few days before I fly off to Spain that I'll try and get the damn OIIO stub working, and then from then if gcolburn gets better images, we can use a better matrix and evaluate using something like Blender or any other OCIO enabled app.
00:55
troy_s_
(I'll craft up the OCIO configs for viewing on sRGB and perhaps even raw XYZ)
00:55
troy_s_
(For the nerds that want to see that)
00:55
Bertl
with the new LVDS deserializer and training we can guarantee that, and as soon as that works, I'll shift my focus onto denoising
00:55
troy_s_
Bertl: What do you mean?
00:55
troy_s_
Bertl: Is your data corrupted in some way?
00:56
Bertl
as you can see in some images, there is some corruption going on
00:56
Bertl
the Cmosis reference code (for the deserializer) I've been using is insufficient
00:57
Bertl
that's why I redesigned it and already verified that it is working reliably now
00:57
troy_s_
Gotcha.
00:57
troy_s_
I get it.
00:57
troy_s_
But anyways... yes... agree.
00:57
troy_s_
The idea that we get the image 'as best as possible' is probably high on needs at the moment.
00:57
Bertl
unfortunately I had to change a bunch of other things as well, and the xilinx tools are not easy to handle (read trick into doing the right thing :)
00:58
troy_s_
So that we can do secondary evaluations.
00:58
troy_s_
That's stinky
00:58
troy_s_
Are you stuck with them forever?
00:58
Bertl
kind of, they are the prorietary tools which convert the VHDL code to the FPGA binary bitstream
00:59
Bertl
if we switch to Altera, we get a different set of proprietary tools with similar issues :)
00:59
troy_s_
Well I hope you figure it out.
00:59
troy_s_
Yes. Different set of problems I'm sure.
01:00
Bertl
yes, I think already figured out much more than most xilinx employees ever will :)
01:00
Bertl
(at least according to their answers on the forums :)
01:02
gcolburn
:)
01:03
gcolburn
thanks for all your hard work on it Bertl. someday I hope to understand the FPGA programming side in depth :)
01:03
gcolburn
one thing at at time
01:03
Bertl
it is fascinating to some extend, but it also involves a lot of waiting and working around 'broken' tools
01:04
gcolburn
that's got to be really frustrating
01:04
troy_s_
I understand that the stuff is proprietary
01:05
troy_s_
but for the love of all things good and pure would not an open abstraction layer help to reduce code bugs etc?
01:05
troy_s_
(way out of my depth so it is all crack dream)
01:06
Bertl
yeah, probably, for example, the quality of error reporting of the synthesis tools (i.e. the compiler) is really awfull
01:06
Bertl
so, while gcc gives you something like missing semicolon in line 17 column 23
01:07
Bertl
the synthesis tools give you something like: there is an error in some net in foo, or maybe bar, don't know, but here is a list of 1000 nodes to check :)
01:07
troy_s_
Awesome
01:07
troy_s_
Lol
01:07
troy_s_
Grab goat and sacrifice compiling. Awesome.
01:07
Bertl
also the tools happily rename everything as they work
01:08
Bertl
so you label a variable foo, and it ends up being called xilinx_bar_recomp_some__i_7 (no joke)
01:12
gcolburn
that's fun :)
02:52
Bertl
off for a nap .. bbl
04:22
gcolburn
left the channel
07:13
troy_s
joined the channel
07:13
rexbron_
joined the channel
07:13
troy_s_
left the channel
07:13
rexbron
left the channel
12:29
Bertl
morning everyone!
12:34
se6astian
joined the channel
12:35
se6astian
hello!
13:08
Bertl
hey, how's going?
13:24
se6astian
good good
13:25
se6astian
I had an idea yesterday for a hybrid version "between" the "old" and the "new" open modules concept
13:25
Bertl
okay?
13:26
se6astian
an FMC connector between core and the rear part that hosts the bays for eg. 4 moduleslots
13:27
se6astian
so its stacking combined with slide in modules
13:27
se6astian
2x FMC, 4 x PCIe
13:29
se6astian
not sure if its THE best of both worlds but at least it keeps modularity to change the size of the camera high but does not spend a fortune on FMC(or similar) connectors
13:37
se6astian
but I ll make some concept images to illustrate it, as always :)
13:44
Bertl
excellent!
13:46
se6astian
regarding further test chart shots, do I assume correct that its better to wait a little for the finished new firmware than to create new shots with the old one now?
13:48
Bertl
probably
13:48
Bertl
should be finished today, unless the tools trick me again :)
13:49
se6astian
ok
13:49
se6astian
good luck, remember you are the magician with the tricks :)
13:52
Bertl
pulls rabbit out of his hat
13:53
se6astian
the rabbit size I have seen at your place????
13:54
Bertl
sure :)
14:49
Bertl
\o/ tools tamed ... we should have the new image in a few minutes
14:52
se6astian
hurray!
17:14
misuris
joined the channel
17:19
misuris
left the channel
17:26
ratunku
joined the channel
17:27
ratunku
left the channel
17:40
philippej
joined the channel
18:00
philippej
left the channel
18:15
philippej
joined the channel
20:38
Wescotte
left the channel
20:51
jucar
joined the channel
21:29
philippej
left the channel
22:20
jucar
left the channel