23:17 | rexbron_ | left the channel | |
23:18 | rexbron | joined the channel | |
23:19 | troy_s | left the channel | |
23:19 | troy_s_ | joined the channel | |
23:26 | troy_s | joined the channel | |
23:27 | troy_s_ | left the channel | |
23:32 | troy_s | Bertl: Can we get a reliable file extension for the raw data Herb?
| |
23:32 | troy_s | Bertl: Something like .AXI
| |
23:33 | Bertl | do you have an url to a spec for .axi? and what do you mean by 'reliable' and 'raw data'?
| |
23:36 | troy_s | Bertl: Well I'm thinking ahead slightly. I'm about to write a stub for OIIO.
| |
23:36 | troy_s | Bertl: And it relies on a file extension.
| |
23:37 | Bertl | relying on an extension is always a bad thing, but that's unrelated here I guess
| |
23:39 | rexbron | left the channel | |
23:39 | rexbron | joined the channel | |
23:39 | rexbron | left the channel | |
23:39 | rexbron | joined the channel | |
23:39 | troy_s | left the channel | |
23:39 | troy_s_ | joined the channel | |
23:40 | troy_s_ | Bertl: So raw16 isn't great.
| |
23:40 | troy_s_ | Make sense?
| |
23:41 | Bertl | yeah, well, .raw16 is just raw data, you can name it .data or .wossname as well
| |
23:42 | Bertl | i.e. it isn't a file format per se
| |
23:43 | Bertl | gabe has written a converter to convert it (lossless) to dng and to regenerate the .raw16 data from those dngs without any loss
| |
23:43 | Bertl | not knowing .axi, I can't tell how much work a similar converter would be
| |
23:44 | troy_s_ | Bertl: It's just a made up extension.
| |
23:45 | troy_s_ | Bertl: If you are writing the firmware, at least put a magic code or something at the head pehaps?
| |
23:45 | troy_s_ | Bertl: Because otherwise reader / writers have no way of knowing if the data is useful or in the correct format.
| |
23:45 | Bertl | well, that's what I mean, a file format should have some mime magic
| |
23:45 | troy_s_ | It doesn't need to be complex.
| |
23:45 | Bertl | but .raw16 is not intended as a format
| |
23:45 | troy_s_ | Just a magic header value or width / height or whatever the hell is useful.
| |
23:45 | Bertl | i.e. I don't want to create a new format here :)
| |
23:46 | troy_s_ | Too late.
| |
23:46 | troy_s_ | You already are dumping data.
| |
23:46 | troy_s_ | That's a format.
| |
23:46 | Bertl | it is raw data
| |
23:46 | troy_s_ | Every single camera dumps raw data
| |
23:46 | troy_s_ | they still encapsulate it in a meaninful wrapper
| |
23:46 | Bertl | what mime magic/format does .txt have in your book?
| |
23:46 | troy_s_ | The extension.
| |
23:46 | Bertl | (i.e. ascii raw text)
| |
23:46 | Bertl | well, .raw16 has the .raw16 extension, no?
| |
23:46 | troy_s_ | My ponit is that all of OIIO relies on extensions to identify the initial file.
| |
23:47 | troy_s_ | It's kind of a crap extension, yes?
| |
23:47 | troy_s_ | :)
| |
23:47 | Bertl | hmm, why?
| |
23:47 | troy_s_ | Because it doesn't tell you what it is from.
| |
23:47 | troy_s_ | And it isn't exactly a useful descriptor.
| |
23:47 | Bertl | and .txt tells you that?
| |
23:47 | troy_s_ | Yes.
| |
23:47 | troy_s_ | .TeXT
| |
23:47 | Bertl | so where is my AXIOM.txt from?
| |
23:48 | troy_s_ | like .AXIom
| |
23:48 | troy_s_ | Oh god.
| |
23:48 | troy_s_ | If you look up the history of .txt, you can likely find that it includes ASCII
| |
23:48 | troy_s_ | .raw16 doesn't mean anything
| |
23:48 | troy_s_ | and in fact the data is well known and understood
| |
23:48 | troy_s_ | it's the data from a bayer sensor on a prototype axiom camera
| |
23:48 | troy_s_ | it isn't just random
| |
23:49 | troy_s_ | but as one of the fellows that might try to write some code for decoding the file, how about an extension and a magic number as a test?
| |
23:49 | troy_s_ | Something super simple.
| |
23:49 | Bertl | yes, but obviously there is no generic format for that which doesn't involve a million tags and complicated structures like dng
| |
23:49 | troy_s_ | Or a magic word sized code and a version number if you want to get really fancy.
| |
23:49 | troy_s_ | I'm not telling you that.
| |
23:49 | troy_s_ | I'm asking for an extension that clearly tells us it is from the axiom camera. :)
| |
23:50 | Bertl | we will do that once we know that we 'need' or 'want' a specific axiom format
| |
23:50 | troy_s_ | Ideally with a simple check to avoid breaking a reader
| |
23:50 | troy_s_ | So a magic first word header is too much?
| |
23:50 | troy_s_ | two bytes?
| |
23:50 | gcolburn | good news! I patched the bayer2RGB code to work on the .raw16 file
| |
23:50 | troy_s_ | Or four?
| |
23:50 | Bertl | yes, because it breaks the ability to access the data easily
| |
23:50 | troy_s_ | You aren't hearing me.
| |
23:50 | gcolburn | it successfully ran with VNG
| |
23:50 | troy_s_ | If I try to write a plugin loader and
| |
23:51 | troy_s_ | I have to rely on raw16 as the extension with no helper check
| |
23:51 | troy_s_ | the chances of failure go up huge
| |
23:51 | Bertl | e.g. now I can do display -size 4096x3072 -depth 16 gray:example.raw16
| |
23:51 | troy_s_ | In imagemagck?
| |
23:51 | Bertl | (without implementing anything)
| |
23:51 | Bertl | yes
| |
23:51 | troy_s_ | Urgggghhhhhhhhhh...adlfkja;lfkjasd;flkasjf;lasjf;lasfkjas;dlfkjasd;lfkj
| |
23:51 | troy_s_ | Whatever.
| |
23:51 | troy_s_ | Moving on.
| |
23:51 | Bertl | LOL
| |
23:51 | troy_s_ | Enjoy staring at your black and white bayer pattern
| |
23:52 | Bertl | as I said, no problem, just wrap it with 4 bytes if that makes your reader happy
| |
23:52 | troy_s_ | gcolburn: VNG?
| |
23:52 | Bertl | like cat AXI.magic example.raw16 >example.troy
| |
23:54 | gcolburn | Bertl: The bayer2rgb code was not calling the 16 bit converter when you do -b 16
| |
23:54 | gcolburn | so I believe it was assuming 8
| |
23:54 | gcolburn | so I just put the function call in LOL
| |
23:54 | troy_s_ | gcolburn: VNG?
| |
23:55 | gcolburn | and presto, the tiff comes out correct
| |
23:55 | Bertl | gcolburn: nice, well, I didn't bother, but good to know
| |
23:55 | Bertl | maybe send the changes upstream and check the tool into the axiom repository?
| |
23:57 | gcolburn | yeah
| |
23:57 | troy_s_ | Is it at GitHub?
| |
23:57 | gcolburn | what's funny is near as I can tell the -swap option for 16 bit isn't implemented
| |
23:57 | gcolburn | I searched the code. it seems to set a flag and not use it
| |
23:58 | Bertl | yes, that was what I figured
| |
23:58 | gcolburn | I get the same image with or without adding -swap
| |
23:58 | gcolburn | but it works, which is all that matters
| |
23:58 | Bertl | great!
| |
23:59 | troy_s_ | gcolburn: Now onto that stub.
| |
00:04 | Bertl | troy_s_: isn't there a format in OIIO which already supports raw data in bayer matrix?
| |
00:04 | Bertl | (I would assume that there has to be one which could be used)
| |
00:14 | Bertl | if not, maybe a generic 'raw' (not in the camera specific raw sense) plugin would be appropriate
| |
00:14 | Bertl | i.e. something where you specify the size (width, height), the bitdepth and the bayer pattern order
| |
00:29 | troy_s_ | Bertl: No
| |
00:29 | troy_s_ | there isn't.
| |
00:30 | troy_s_ | Larry has been developing code for too long to make such open ended bits of rubbish.
| |
00:30 | troy_s_ | It's a very focused library.
| |
00:30 | troy_s_ | It makes a number of assumptions and design context decisions... only associated alpha, no random different bit depth, only a very limited subset, etc.
| |
00:31 | troy_s_ | The stub for the camera wouldn't ever be useful upstream either, so it will be a maintained thing. That said, it's vastly more useful than a DNG exporter etc.
| |
00:32 | troy_s_ | DNG is a great target for the raw data to be converted to for folks who want to dicker with the debayering etc. I'm only going to implement a very focused bayer -> VNG -> RGB (of a given colorspace) implementation if I can get it working.
| |
00:32 | troy_s_ | The idea that we are testing on DNG is pure lunacy.
| |
00:32 | Bertl | so why not do a plugin as I suggested then? should do the trick and doesn't need to support any sizes/bitdepths we do not encounter
| |
00:32 | troy_s_ | Testing should be much more focused and likely should be a very very very very simple command line thing so that we eliminate all variables.
| |
00:33 | troy_s_ | I'm doing the Axiom plugin. Or at least I intend to. Trying to get my build environment working as I need.
| |
00:33 | troy_s_ | I'm only focusing on the raw data format you are dumping from the camera (hence my request to have the extension be .axi or something particular)
| |
00:34 | troy_s_ | DNG is again, a no argument discussion when it comes to storing the raw data into a single frame if needed. My gut says that we should likely dump the speediest raw format we can to disk though. Rather like ArriRaw.
| |
00:34 | troy_s_ | And then provide stubs as required to go from ArriRaw to whatever.
| |
00:34 | Bertl | what I mean is, why does it need to be an 'axiom' data format, why can't it be a raw, 16 (soon we will switch to 12 :) bit linear (gray) image with bayer pattern?
| |
00:34 | gcolburn | troy: you can feel free to pick an extension for the raw files for the OIIO plugin purposes
| |
00:34 | troy_s_ | It doesn't need to be a data format.
| |
00:35 | troy_s_ | You misread my request. It was purely identification purposes with a _teeny_ check in there (just in case someone has a malformed file or something)
| |
00:35 | Bertl | like a checksum or so?
| |
00:35 | troy_s_ | Yep. Back in the ages of yore I remember having my readers go apepoop over malformed files.
| |
00:35 | Bertl | we can easily append a crc32 at the end, if that helps
| |
00:35 | troy_s_ | (Which was sadly more common)
| |
00:36 | troy_s_ | I see your need for a basic "display foo" but I'm merely trying to get a plugin in place that makes our ability to go to say, EXR or DPX or JPG a click away.
| |
00:36 | troy_s_ | As opposed to the current nightmare of DNG.
| |
00:36 | troy_s_ | (Which again, I think is a really sub-optimal focus at this point given our design constraints.)
| |
00:37 | Bertl | well, my 'to jpg' (or png) is trivial as well
| |
00:37 | troy_s_ | (What I _really_ don't want to see is DNGs getting posted and then countless idiocy spiralling out of control when someone starts talking about color being off only to discover that their FooBarDNG reader was malforming or doing something odd.)
| |
00:37 | troy_s_ | Bertl: More trivial with OIIO.
| |
00:38 | troy_s_ | Bertl: We write one plugin stub and we get TIFF, JPG, DPX, EXR, PNG, TGA, etc. all for free.
| |
00:38 | troy_s_ | Which isn't a bad thing.
| |
00:38 | troy_s_ | I'd focus on two points this early on for testing purposes.
| |
00:38 | troy_s_ | EXR and DPX
| |
00:38 | Bertl | I agree, I'm all with you on that
| |
00:38 | troy_s_ | largely because it accomplishes a crapload with a very simple bit of elegance.
| |
00:39 | Bertl | so an crc32 at the end would help you? we can do that easily and older raws can be easily 'upgraded'
| |
00:39 | troy_s_ | 1) You can't screw up an EXR. It is _always_ linearized data. The ONLY conversion we will be coupling in is VNG (best of breed arguably at the moment, and certainly high enough rough quality) and color matrix (hopefully via some better profling)
| |
00:40 | troy_s_ | I don't know. The readers all start at the beginning. I suppose I could do a quick calc. I'm only thinking that for our purposes some sort of really trivial check would be at least a good step forward for reading files and validating that they are in fact decent.
| |
00:40 | troy_s_ | 2) By focusing on EXR and DPX you eliminate a whole subset of folks that likely aren't quite the target demographic for evaluating the images.
| |
00:41 | troy_s_ | (as in if someone doesn't know what an EXR or DPX is, they are less likely to download the image and gripe about incorrect image results which is arguably happening already.)
| |
00:41 | Bertl | well, I have no clue about OIIO plugins, IMHO the only relevant question is how such a plugin gets selected
| |
00:41 | troy_s_ | So a simple stub plugin for OIIO gets us well on our way without worrying about all sorts of libraries and cruft.
| |
00:41 | troy_s_ | Yep
| |
00:41 | troy_s_ | The first line in OIIO is obviously extension
| |
00:41 | troy_s_ | Then the rest is up to the plugin to validate.
| |
00:41 | troy_s_ | Pretty simple and lean.
| |
00:42 | Bertl | so you need a specific extension (or set of extensions) and a way to say, yep, that's mine?
| |
00:42 | Bertl | i.e. all plugins registering for an extension will get a chance to process the data?
| |
00:46 | troy_s_ | Bertl: I just think it's probably easier to say "Ok... this plugin wants the Axiom Alpha Prototype file so only load .axi
| |
00:46 | troy_s_ | Bertl: Just like a jpeg
| |
00:46 | troy_s_ | Bertl: Which is nothing more than the first test. I'm not saying we commit to anything.
| |
00:46 | troy_s_ | Bertl: As we can always craft the stub to be whatever we want.
| |
00:47 | troy_s_ | Bertl: No ridiculous versioning etc.
| |
00:47 | troy_s_ | But your CRC idea isn't an awful one.
| |
00:47 | troy_s_ | And our resolution is always fixed right?
| |
00:48 | Bertl | at the moment, we generate the 4k images
| |
00:48 | troy_s_ | (Unless I guess Axiom decides to offer a lower res bayer for offspeeds or whatever... all stuff that isn't really a huge thought at the moment.)
| |
00:48 | troy_s_ | another quick question Bertl, Gabe is hitting a wall with some of the calibration / profiling issues
| |
00:48 | Bertl | i.e. 4096x3072 pixels, color or monochrome (depending on the sensor)
| |
00:48 | troy_s_ | So I was hoping that you might be able to get that noise reduction in place sooner rather than later?
| |
00:49 | troy_s_ | I ask this because if we get to a 5 error profile, and evaluate the gamut... what happens if we find out that it is smaller than 709 or sRGB?
| |
00:49 | troy_s_ | See my concern?
| |
00:49 | Bertl | yeah, I'm still fighting the xilinx tools to get the new code finished
| |
00:49 | troy_s_ | We really should get that profile locked in as if the sensor has a ridiculously small gamut... well you won't be winning any supporters with that sensor.
| |
00:49 | troy_s_ | And all this work is for naught.
| |
00:49 | Bertl | it will fix all the LVDS issues, but the tools are stubborn
| |
00:49 | troy_s_ | I bet.
| |
00:53 | troy_s_ | Bertl: Right now I think we are zoning in on getting the image quality up to snuff and all that it entails.
| |
00:53 | troy_s_ | Bertl: Everything else should be secondary, as there are evaluations that are needed before moving on I'd bet.
| |
00:54 | troy_s_ | Bertl: Especially in the "Is this sensor suitable" zone.
| |
00:54 | troy_s_ | That gamut would worry me to move on without further knowledge.
| |
00:54 | Bertl | yeah, but I don't see a point in using images where we cannot guarantee that we have the actual sensor daa
| |
00:54 | Bertl | *data
| |
00:55 | troy_s_ | I have a few days before I fly off to Spain that I'll try and get the damn OIIO stub working, and then from then if gcolburn gets better images, we can use a better matrix and evaluate using something like Blender or any other OCIO enabled app.
| |
00:55 | troy_s_ | (I'll craft up the OCIO configs for viewing on sRGB and perhaps even raw XYZ)
| |
00:55 | troy_s_ | (For the nerds that want to see that)
| |
00:55 | Bertl | with the new LVDS deserializer and training we can guarantee that, and as soon as that works, I'll shift my focus onto denoising
| |
00:55 | troy_s_ | Bertl: What do you mean?
| |
00:55 | troy_s_ | Bertl: Is your data corrupted in some way?
| |
00:56 | Bertl | as you can see in some images, there is some corruption going on
| |
00:56 | Bertl | the Cmosis reference code (for the deserializer) I've been using is insufficient
| |
00:57 | Bertl | that's why I redesigned it and already verified that it is working reliably now
| |
00:57 | troy_s_ | Gotcha.
| |
00:57 | troy_s_ | I get it.
| |
00:57 | troy_s_ | But anyways... yes... agree.
| |
00:57 | troy_s_ | The idea that we get the image 'as best as possible' is probably high on needs at the moment.
| |
00:57 | Bertl | unfortunately I had to change a bunch of other things as well, and the xilinx tools are not easy to handle (read trick into doing the right thing :)
| |
00:58 | troy_s_ | So that we can do secondary evaluations.
| |
00:58 | troy_s_ | That's stinky
| |
00:58 | troy_s_ | Are you stuck with them forever?
| |
00:58 | Bertl | kind of, they are the prorietary tools which convert the VHDL code to the FPGA binary bitstream
| |
00:59 | Bertl | if we switch to Altera, we get a different set of proprietary tools with similar issues :)
| |
00:59 | troy_s_ | Well I hope you figure it out.
| |
00:59 | troy_s_ | Yes. Different set of problems I'm sure.
| |
01:00 | Bertl | yes, I think already figured out much more than most xilinx employees ever will :)
| |
01:00 | Bertl | (at least according to their answers on the forums :)
| |
01:02 | gcolburn | :)
| |
01:03 | gcolburn | thanks for all your hard work on it Bertl. someday I hope to understand the FPGA programming side in depth :)
| |
01:03 | gcolburn | one thing at at time
| |
01:03 | Bertl | it is fascinating to some extend, but it also involves a lot of waiting and working around 'broken' tools
| |
01:04 | gcolburn | that's got to be really frustrating
| |
01:04 | troy_s_ | I understand that the stuff is proprietary
| |
01:05 | troy_s_ | but for the love of all things good and pure would not an open abstraction layer help to reduce code bugs etc?
| |
01:05 | troy_s_ | (way out of my depth so it is all crack dream)
| |
01:06 | Bertl | yeah, probably, for example, the quality of error reporting of the synthesis tools (i.e. the compiler) is really awfull
| |
01:06 | Bertl | so, while gcc gives you something like missing semicolon in line 17 column 23
| |
01:07 | Bertl | the synthesis tools give you something like: there is an error in some net in foo, or maybe bar, don't know, but here is a list of 1000 nodes to check :)
| |
01:07 | troy_s_ | Awesome
| |
01:07 | troy_s_ | Lol
| |
01:07 | troy_s_ | Grab goat and sacrifice compiling. Awesome.
| |
01:07 | Bertl | also the tools happily rename everything as they work
| |
01:08 | Bertl | so you label a variable foo, and it ends up being called xilinx_bar_recomp_some__i_7 (no joke)
| |
01:12 | gcolburn | that's fun :)
| |
02:52 | Bertl | off for a nap .. bbl
| |
04:22 | gcolburn | left the channel | |
07:13 | troy_s | joined the channel | |
07:13 | rexbron_ | joined the channel | |
07:13 | troy_s_ | left the channel | |
07:13 | rexbron | left the channel | |
12:29 | Bertl | morning everyone!
| |
12:34 | se6astian | joined the channel | |
12:35 | se6astian | hello!
| |
13:08 | Bertl | hey, how's going?
| |
13:24 | se6astian | good good
| |
13:25 | se6astian | I had an idea yesterday for a hybrid version "between" the "old" and the "new" open modules concept
| |
13:25 | Bertl | okay?
| |
13:26 | se6astian | an FMC connector between core and the rear part that hosts the bays for eg. 4 moduleslots
| |
13:27 | se6astian | so its stacking combined with slide in modules
| |
13:27 | se6astian | 2x FMC, 4 x PCIe
| |
13:29 | se6astian | not sure if its THE best of both worlds but at least it keeps modularity to change the size of the camera high but does not spend a fortune on FMC(or similar) connectors
| |
13:37 | se6astian | but I ll make some concept images to illustrate it, as always :)
| |
13:44 | Bertl | excellent!
| |
13:46 | se6astian | regarding further test chart shots, do I assume correct that its better to wait a little for the finished new firmware than to create new shots with the old one now?
| |
13:48 | Bertl | probably
| |
13:48 | Bertl | should be finished today, unless the tools trick me again :)
| |
13:49 | se6astian | ok
| |
13:49 | se6astian | good luck, remember you are the magician with the tricks :)
| 13:52 | Bertl | pulls rabbit out of his hat
|
13:53 | se6astian | the rabbit size I have seen at your place????
| |
13:54 | Bertl | sure :)
| |
14:49 | Bertl | \o/ tools tamed ... we should have the new image in a few minutes
| |
14:52 | se6astian | hurray!
| |
17:14 | misuris | joined the channel | |
17:19 | misuris | left the channel | |
17:26 | ratunku | joined the channel | |
17:27 | ratunku | left the channel | |
17:40 | philippej | joined the channel | |
18:00 | philippej | left the channel | |
18:15 | philippej | joined the channel | |
20:38 | Wescotte | left the channel | |
20:51 | jucar | joined the channel | |
21:29 | philippej | left the channel | |
22:20 | jucar | left the channel |