Current Server Time: 19:49 (Central Europe)

#apertus IRC Channel Logs

2014/02/10

Timezone: UTC


01:06
gwelkind
left the channel
02:43
rexbron_
Bertl: Reading the back chat with troy, the standard method for uniquely identifying frames is a reel name and timecode. The reel name should be unique on a per project basis, which usually encodes the camera letter (A, B, C, etc) and a unique and incrimenting number. So a reel name might be A001 or A_CAM_001.
02:44
Bertl
ah, okay, and there is a 'reel' per 'take', yes?
02:45
Bertl
so, two takes shot with the same camera would have different numbers? or just different timecodes?
02:52
rexbron_
Bertl: depends
02:52
rexbron_
Generally, the two minimum required pieces of unique information are the Reel number and the timecode
02:53
rexbron_
as timecode when recorded with seperate audio, for simplicity and two avoid the potential duplicate timecodes, as time of day
02:53
rexbron_
that goes all the way back to film :)
02:54
rexbron_
keycode (TM Kodak) was a batch printed monotonicly increasing number printed between sprockets to uniquely identify footage of film
02:55
rexbron_
for the negative cutter
02:56
Bertl
http://en.wikipedia.org/wiki/Keykode
02:56
rexbron_
for apps like resolve, reel name and timecode are the means of conforming and EDL or other edit back to data that is different than what was used for the offline
02:57
rexbron_
side note: I'm currently conforming the edit for a feature I shot with the BMCC. From 10TB of camera originals, I'm conforming the edit down to 1TB of unique clips, then using resolve to trim the DNG sequences to just what was used, getting a 20 min finished exhibition reel (different than camera reels ;) down to 120gb
03:00
rexbron_
The DNG sequences are coming from TAR archives that take 5-20 min to open depending on if they were done by my DMT or myself when I got home at the end of the day and instead of taring by camera roll, which is a better way to refer to a mag, I just said fuck it and tared the entire day and went to sleep
03:01
rexbron_
older tape based systems tended to prefer record run timecode, but that is not the accepted norm in digital production now
03:02
Bertl
interesting ... why would you open the TAR archives?
03:03
Bertl
or more precisely, I mean, why would you unpack it, not just open it
03:04
rexbron_
Bertl: reading back the archive takes 5-20 to open :)
03:04
rexbron_
it's a 2Tb tar file
03:05
Bertl
I got that, but 'listing' a 2TB tar file takes a few seconds I guess, no?
03:05
rexbron_
We went to tar as a format because file system overhead became an issue when your file count is in the millions
03:05
rexbron_
our backup drives were no longer capible of real time with the camera
03:06
Bertl
what filesystem was used?
03:06
rexbron_
and even on a narritive shoot where you only capture 1/10th of the day, if you don't have enough media (which we didn't) for the entire day, you have to stop
03:06
rexbron_
Bertl: NTFS :P
03:06
Bertl
okay, well, that explains a lot
03:07
rexbron_
Bertl: would exfat perform any better, as that's the only other default option in windows
03:08
Bertl
no, neither ntfs nor (v)fat is suited for large numbers of files or large files in general
03:12
Bertl
but what I meant before is, if you choose to use the tape archive format, why not simply read the metadata and access the file contents directly
03:12
rexbron_
Bertl: as I understand it, the reader needs to seek through the file as there is no index
03:13
rexbron_
at least, that's what 7zip seems to do
03:14
rexbron_
that then gives me a representation of the file to cherry pick shots out of
03:15
rexbron_
I needed to maintain 125MB/s thoughout the entire drive, tar was able to do that
03:16
rexbron_
There is a saying that goes "any port in a storm"
03:16
rexbron_
and you generally don't shoot a TB's worth of test footage in prep
03:18
Bertl
not sure how you use tar, but look for --index-file (this option allows you to have an external index for direct access)
03:20
rexbron_
Bertl: too late :) I've just kept them open once they've been read back
03:22
Bertl
okay :)
03:24
rexbron_
Bertl: Most people have no idea the scale that's involved in shooting raw
03:24
rexbron_
uncompressed raw
03:25
rexbron_
in a narrative setting I usually average 1Tb to 1.5tb of uncompressed 2.5k raw per day
03:26
rexbron_
with 4k, thats more like 3-4Tb per day per camera
03:26
rexbron_
data capture device can be inexpensive but the storeage backend costs are high
03:27
rexbron_
Side note: that's also why RedCode was so crucial to the sucess of Red
03:27
Bertl
what do you consider the 'storage backend'?
03:27
rexbron_
considering that they were doing (lossy) 4k raw in 2008
03:28
rexbron_
the DMT system to dump and review footage on set plus the space to store a copy of the footage x3
03:28
rexbron_
lets say a 3Tb external HD is somehow fast enough for 4k uncompressed on set use
03:29
Bertl
i.e. the recording medium
03:29
rexbron_
Bertl: nope, not camera mags
03:29
Bertl
okay, so you recorded to whatever medium in the camera, like 1TB
03:29
rexbron_
I mean, the hard drive is slower than real time but it's going to keep you for hours after wrap because of a good data manger who reloads frequently to keep the system busy
03:30
Bertl
then you copy it over to your storage server, no?
03:30
rexbron_
Bertl: who's storeage server?
03:30
Bertl
your own, I presume?
03:30
rexbron_
Most producers cry bloody murder when you say you need 3x hard drives
03:30
rexbron_
Bertl: so I need to have a $10k raid array to use the camera?
03:30
Bertl
but they aren't really expensive nowadays
03:31
rexbron_
....
03:31
rexbron_
it's about operating cost
03:31
Bertl
I'd go with like 6 4TB disks in a raud
03:31
Bertl
*raid
03:31
rexbron_
x3 for backup
03:31
Bertl
you can do three backups, or just make 3 raids
03:31
rexbron_
even in raid 0 those 4TB arrays nets you a weeks worth of footage
03:32
rexbron_
24TB/4 :)
03:32
Bertl
but they do not cost very much, the disks
03:32
rexbron_
x3
03:32
rexbron_
but you are now at the point where the cost of running the camera is almost as much as the camera
03:32
Bertl
I don't think that 200 EUR/week or so is critical
03:32
rexbron_
count the low budget guys out
03:32
rexbron_
it's more than 200/week
03:33
rexbron_
shooting BMCC raw costs $150 day
03:33
rexbron_
$50 per copy
03:33
rexbron_
also, I'm not trying to discourage
03:33
rexbron_
just identify issues that will be encountered in the field
03:34
rexbron_
also what kind of raid enclosure are we talking about? the 3x copies are so that at least 2 of them can live offsite over night and permanantly when the film wrapps
03:37
rexbron_
so an inexpensive 8 bay raid enclosure (I haven't ever heard of a 6 drive model) at $350 with 6x 4TB drives @ $210
03:37
Bertl
well, if you go cheap, you can simply put each disk in a plastic case/bag and store it away
03:37
rexbron_
Bertl: not feasable, camera media is expense and so are reshoots, at least 2 copies need to be made on set
03:38
rexbron_
so we are at $1500 for 24TB of media
03:38
rexbron_
that is capible of sustaining realtime performance
03:39
rexbron_
putting the average per day cost at $270 rounding up
03:39
rexbron_
per copy
03:39
rexbron_
per camera
03:39
Bertl
I think you have to decide what you want
03:39
Bertl
either go cheap, have only one backup or simple raid redundancy and no realtime performance
03:40
Bertl
or you value your data/speed and make three (or more) high quality copies with excellent speed
03:40
Bertl
(of course, you can pick your middle ground as well)
03:40
rexbron_
but now we are into high end workflow territory
03:40
rexbron_
Have you decided on an MSRP?
03:40
rexbron_
because that will dictate who considers buying
03:41
rexbron_
and as BMD well knows, there are a lot of loud retards out there who get interested in a product when it's low
03:41
rexbron_
in price
03:42
rexbron_
and scream, bitch and moan that it isn't what they deluded themselves into thinking it was
03:42
rexbron_
to make an analogy
03:42
rexbron_
my brother works in morgages
03:42
rexbron_
you never want to borrow at the limit of what you can afford
03:42
rexbron_
many people bought BMD's affordable raw camera
03:43
rexbron_
without a second though to those "Hidden" costs
03:45
rexbron_
people tend not to see media as consumablle, as HDD are reuseable, but when making a film, those copies are permanent
03:46
Bertl
it might be a good investment to buy a tape drive and store the copies there
03:54
Bertl
LTO-5 does 1.5TB for roughly 20 USD
03:56
Bertl
the one time cost of roughly 1500 USD for the drive will pay off soon
03:58
rexbron_
Bertl: I've been thinking about that as a means to do cold archives at the end of a project but you still need hot online storage on set
03:58
rexbron_
if you are camera media constrained
03:58
rexbron_
which would likely be the case
03:58
Bertl
of course, but that would not be three, only one copy
03:59
Bertl
and you could reuse those disks for every movie
04:15
gwelkind
joined the channel
05:22
gwelkind
left the channel
06:03
gwelkind
joined the channel
06:11
gwelkind
left the channel
06:18
Bertl
off to bed now ... have a good one everyone!
06:28
gwelkind
joined the channel
06:49
gwelkind
left the channel
08:06
SashaC
joined the channel
09:37
se6astian
joined the channel
09:40
se6astian
good morning
10:36
guest
oh, BM4K went down in price by $1000
10:47
se6astian
maybe they will announce the 5K version next month :)
10:50
guest
is there a 5k chip?
10:51
guest
i think the users decided to divert from it and cancel the orders
11:24
se6astian
http://www.cmosis.com/products/standard_products/cmv20000
12:08
guest
that is uncessarily big
13:39
se6astian
like cinema or imax screens are unnecessarily big :)
14:55
SashaC
left the channel
15:06
se6astian
time to leave
15:06
se6astian
see you
15:06
se6astian
left the channel
15:20
Bertl
morning everyone!
15:36
tyrone_
joined the channel
15:45
FergusL
hi Bertl
17:15
jucar
left the channel
17:24
troy_s
Yum. guest remember that 5k sensels isn't even 2k.
17:24
troy_s
guest: A 2k (full HD) sensor is 8k sensels.
17:30
troy_s
Bertl: Good log above.
17:30
troy_s
Bertl: I also believe the more the project emphasizes the design audience, the less crazies will gravitate to it.
17:31
troy_s
(See rexbron_ 's comment on price point and audience attracted)
17:32
troy_s
I also wonder if there is a subsidizing opportunity via the storage. Where supporters can purchase a drive setup with a well-documented markup that funds the project.
17:33
guest
troy_s: why is that? i would say from a 6MP sensor you can get FHD 4:4:4
17:34
jucar
joined the channel
17:43
gwelkind
joined the channel
18:09
se6astian
joined the channel
18:10
se6astian
good evening
18:14
troy_s
guest: 6k = 6k pixels across.
18:14
troy_s
guest: Which is 3k green
18:16
troy_s
Sorry, yes, 6k is bare minimum for 2k.
18:16
guest
what about 4k.. that is 2k too
18:16
guest
even more greens than necessary
18:17
troy_s
"more greens" is sort of relative
18:18
se6astian
time to release the latest article
18:18
guest
if you are that picky, then you should say no bayer sensor is good enough :)
18:19
guest
as color are offsetted there
18:28
se6astian
and its out: https://www.apertus.org/openness-explained-2014
18:28
troy_s
guest: 4k is not 2k
18:29
troy_s
guest: You have to simply do the math on discrete pixel sensels - to do 444 (RGB style) you need 3 sensels per color.
18:29
troy_s
(per pixel rather)
18:29
troy_s
guest: So the bare minimum is 6k x 3k for an HD sensor only (not 3:2)
18:30
troy_s
guest: Otherwise you are subsampling off of already used sensels.
18:30
troy_s
guest: That is not picky... that is the influence of bad math. :)
18:30
troy_s
guest: There is only one way to get to RGB 444 and that is precisely one sensel per pixel.
18:30
guest
at 4k you have 8Mpx that is more than 2k HD
18:30
troy_s
guest: Uh what?
18:31
troy_s
8k is 2.6k
18:31
guest
2k green X2, 2k blue, 2k reds
18:31
troy_s
Along horizontal.
18:31
troy_s
4k is 1.3k
18:31
guest
image is 2D, there is no h/v
18:31
troy_s
(or rather, 4ks = 1.3k)
18:32
troy_s
4ks (4000 sensels) is 1.3k pixels
18:32
guest
isnt
18:32
troy_s
Remember, for each pixel you require exactly THREE sensels
18:32
troy_s
Sense?
18:33
troy_s
Anything short of that most basic math is actually bad math.
18:33
troy_s
(aka subsampling)
18:33
troy_s
So no matter how you try to magic the numbers, 4000 sensels means 4000 sensels total of RGB across the horizontal.
18:33
troy_s
And that is how they 'cheat' the math.
18:33
troy_s
As they are borrowing (subsampling) from alternate rows.
18:33
guest
in bayer you never get rgb on 1 line
18:34
guest
there is no need to
18:34
troy_s
Agree, but the math holds.
18:36
troy_s
(And I also agree that bickering about CMOS versus RGB can be rather pointless.)
18:36
guest
cmos??
18:36
troy_s
(The key takeaway is that 4k isn't 4k in the visual effects / image effects sense.)
18:36
troy_s
CMOS sensor.
18:36
troy_s
Versus say, CCD.
18:38
troy_s
Anyways. I hate the discussions. Frustrating to hell.
18:38
troy_s
(Mostly because many folks simply draw a 1:1 of k thanks to that company.)
18:39
troy_s
(Which sort of skips over the nuances of sampling.)
18:44
troy_s
guest: What have you been up to?
18:55
guest
i am here now
18:57
guest
so.. what is the issue with the 4k sensor, having 4096x2160 photosites - if you take a group of 4, you get a RGB pixel .. do that over the whole surface and you are at 2048x1080 on the end
18:57
guest
there is a little color shift, as the sites are not above each other, but side by side.. can be solved with an OLPF to blur the source
18:57
Bertl
but you can also use the G lightness values to create a 4096x1080 image
18:58
Bertl
or in both directions, a 4096x2160 RGB image
18:59
guest
Bertl: the rgb image with same res as bayer pattern is not 100% okay.. it heavily depends on scene if such processing is unnoticable
18:59
guest
i was rather opposing to troy_s who wants 6k bayer for 2k rgb...
19:00
Bertl
the rgb image with 1/4 the size is not 'okay' either
19:00
Bertl
it also depends on the scene
19:01
troy_s
Ultimately optics were designed to resolve such that a single discrete unit is resolved from the lens.
19:01
troy_s
Variation from that will give you optical issues, and no amount of algo will fix it.
19:02
guest
so what?
19:02
guest
our 4k is still better than REDs 5k after the compression they apply :)
19:02
Bertl
troy_s: no, actually 'modern' optics use low band filters
19:06
troy_s
Bertl: Unfamiliar with that. Most of the optics I have seen have only the glass.
19:06
troy_s
Bertl: Even older SLR lenses.
19:06
troy_s
Not sure what the filter would do? Explain?
19:07
troy_s
Bertl: Need a current image to test with.
19:07
troy_s
Bertl: Where is it?
19:08
troy_s
Bertl: And what are your plans for ApertusRaw? Single blob file or chunks or?
19:08
troy_s
(The more I know about that raw blob the better I can design this thing)
19:08
guest
I already do DNG sequences
19:09
troy_s
guest: Huh?
19:09
troy_s
(Not at all worried about DNG. At all.)
19:09
troy_s
Going to DNG sequences is just another fail point.
19:09
FergusL
left the channel
19:09
troy_s
I'm more interested in focusing on a tool for decoding the raw into more useful formats (IE Debayered)
19:09
FergusL
joined the channel
19:10
troy_s
(and a lab tool for analysis really.)
19:10
troy_s
Bertl: %%
19:13
guest
why is a DNG fail? it is the most compact and versatile file for data capture
19:17
philippej
joined the channel
19:20
philippej
Hi all, interesting discussions going on
19:20
philippej
troy_s: what other, open, options do we have to store raw data?
19:26
troy_s
philippej: Tough question. I could care less how the raw data is stored personally.
19:26
troy_s
philippej: I just would like to know if Bertl intends for it to be a blob or several blobs or whtaever.
19:27
troy_s
(As I would need to know how to load the blobs so that the images are scrubbable)
19:27
Bertl
troy_s: the low pass filters are used to prevent aliasing effect
19:27
Bertl
troy_s: maybe take the x-mas image?
19:27
troy_s
philippej: I suspect though that your question is "What format is best suitable as an _interchange_ for the raw data?" and that answer is likely only DNG currently. I have no interest in it at the moment, and frankly I find it to be rather useless here.
19:28
troy_s
Bertl: Linky?
19:28
Bertl
other than that, raws can be found here: ow about that raw blob the better I can design this thing)
19:28
troy_s
Bertl: And thoughts on the raw blob?
19:28
Bertl
hmm, copy paste error
19:28
Bertl
http://vserver.13thfloor.at/Stuff/AXIOM/RAW/
19:28
troy_s
In your estimation, when you dump the data, will it be a raw file per frame or a binary blob of them?
19:29
troy_s
(several)
19:29
Bertl
now or in the future or in general?
19:29
troy_s
Bertl: So the file is the raw##.xz?
19:29
troy_s
Bertl: When motion happens on it, your best guess\
19:30
troy_s
(as in if /when the camera starts to dump the raw data to disk, how do you see it being stored?)
19:30
Bertl
for now we will do snapshots or short sequences (8-10 frames)
19:30
Bertl
they will usually be just concatenated raws
19:30
Bertl
with a storage solution, it will depend on many other factors
19:31
troy_s
Ok. So many blobs?
19:31
Bertl
like what kind of raid/filesystem/etc we decide upon
19:31
troy_s
So there will be some method to figure out what blobs are part of a given long take?
19:31
troy_s
(It might seem silly to ask this now, but I don't want to be recoding huge chunks on bad guesses on my end.)
19:32
Bertl
whatever storage format we come up with, it will decode to single frames at some point
19:32
troy_s
Where are the daylight raws?
19:32
Bertl
which will be one of the 'known' raw blobs
19:32
troy_s
Hrm.
19:33
troy_s
Bertl: Ok. So raw will be stored as many blobs, each comprising a given number of frames.
19:33
Bertl
it might be just one frame per file lateron
19:34
Bertl
but as I said, it shouldn't matter, you can assume that you always have a single frame to work with
19:34
Bertl
either it is an offset in a file or an offset in a directory :)
19:35
Bertl
the daylight scenes I did are labeled 'scene_daylight'
19:35
troy_s
Yes no daylight raws
19:35
troy_s
scene_daylight_*.png
19:36
Bertl
I can upload those if you need them
19:36
troy_s
Would be great if you would. I'll test against a few raws.
19:36
troy_s
What's the raw sensel resolution?
19:36
Bertl
4096x3072
19:51
guest
troy_s: what are you doing ? (for what is a DNG unusable as a source?)
19:59
Bertl
troy_s: the first daylight raw is up, the other two will take about 15 minutes
20:04
gcolburn
joined the channel
20:09
philippej
left the channel
20:13
troy_s
Bertl: Thanks.
20:14
troy_s
guest: I'm not too concerned about DNG because all that really matters to me is a decent processing application (in this case, decent being something that allows an artist to select between chroma scaling approaches)
20:15
troy_s
guest: And extend out to dumping more useful debayered image formats (Namely DPX) as well as possibly bake in look LUTs.
20:18
Bertl
troy_s: all up now
20:18
guest
what is chroma scaling?
20:19
Bertl
when you change the intensitiy of the colors
20:19
Bertl
*intensity
20:19
guest
you mean saturation then :)
20:19
Bertl
i.e. the chroma component not the luma one
20:20
guest
i found some reference in a book, of chroma scaling being going from 444 to 422
20:20
guest
and reverse
20:21
Bertl
that usually is chroma subsampling, but yes, the terms are not precise
20:21
guest
anyway, using dng as intermediate would at least define the resolution and bayer order... so no questions would needed to be ask :)
20:23
guest
Bertl: have you made any measurement to determine a base ISO / EI ?
20:24
Bertl
sebastian has done some measurements IIRC
20:25
troy_s
guest: Chroma scaling / interpolation.
20:26
troy_s
guest: So no, he doesn't mean saturation.
20:26
troy_s
subsampling / scaling
20:26
troy_s
guest: The issue with DNG is that it doesn't solve anything.
20:26
troy_s
guest: Or rather, it is a useful option, but only to pipelines that feel inclined to worry about interpolations themselves.
20:27
troy_s
guest: So going from Apertus raw to DNG to interpolation is extra layers of crap I'm not at all interested in.
20:27
troy_s
guest: I can have a more streamlined bit of code (which is vastly more useful for testing purposes as we can be 110% certain that DNG hasn't introduced some strange bit of transform or bug or whatever)
20:28
troy_s
guest: By simply providing the first teeny bit of the Apertus Cine Tools to examine / analyze / scale / interpolate the raw images.
20:28
troy_s
guest: So again, DNG solves nothing and worse, it adds layers of complexity for zero gain.
20:28
troy_s
(And given that, even if you do provide DNG, there's only a mitful of tools that allow you to do useful things with the blasted format.)
20:29
guest
it is a container, keeps you data/metadata organized, so you do not need to ask (stupid questions) like what is the resolution :)
20:29
troy_s
Why do I need that again?
20:29
troy_s
I know the resolution now. It won't change.
20:30
troy_s
And if you know anything about DNG, you know that the color transforms are a whole added layer of mess
20:30
guest
it wont change? i bet it will.
20:30
troy_s
Whereas I can use OCIO and change the calibration matrices without any futzing.
20:30
troy_s
Really?
20:30
troy_s
How so?
20:30
troy_s
Change the sensor?
20:30
troy_s
:)
20:30
troy_s
At which point I change the define.
20:30
guest
do a ROI on a sensor
20:30
troy_s
Or if there is a mark 1 and a mark 2
20:30
troy_s
then add a new definition.
20:30
troy_s
DNG is nightmare.
20:30
troy_s
Wank wank.
20:31
guest
why the hell are you recording 4:3? just for +50% of extra storage?
20:31
troy_s
Uh...
20:31
troy_s
Anamorphic?
20:31
troy_s
Or 3:2 frame (aesthetic choice)
20:31
guest
for the 4% of the users
20:31
troy_s
Or overscan
20:31
troy_s
Ever done any VFX work?
20:31
troy_s
You want as much data as possible (tracking for example)
20:32
troy_s
So in terms of folks who are using the camera as the primary intention (cinema crafting), visual effects / post work will make up a sizable chunk
20:32
guest
doing full aperture recording is a choice, you should really not force everybody to have it that way
20:32
troy_s
Betting more like uh... 65-70% Maybe more?
20:32
troy_s
Sorry... disagree entirely. And even if I didn't, you develop for the full frame.
20:33
troy_s
Then worry about restricting.
20:35
troy_s
guest: And besides... until either of us does any lifting such as the work that Bertl has been doing, who the heck are we to suggest anything exactly?
20:41
guest
the slight difference is that he got 4k stills... I have 4k video
20:42
Bertl
there are a bunch of companies who have 4k video already, no?
20:43
guest
with same budget, time and number of workers?
20:44
guest
i think you and i are quite an exception
20:44
Bertl
well, we got 4k video as well, just with a maximum of 8 frames for now (because of the output bandwidth of the zedboard)
20:45
guest
hard to imagine the usefulness of that
20:45
Bertl
it is a prototype to check feasability
20:47
Bertl
but in any case, if it helps you, congrats to your proprietary camera!
20:49
Bertl
https://www.apertus.org/openness-explained-2014
20:51
guest
the last sentence could also say ... legal method to obtain interoperability (which might be more important)
20:53
troy_s
guest: I'm not sure exactly what your point is.
20:54
troy_s
guest: If you shoot video on a cinema camera, you are dealing with stills.
20:54
troy_s
guest: And if you are talking interoperability, DNG isn't exactly the worlds most ideal format. Certainly fine again, if you wish to permit someone to do their own interpolation. Sure.
20:54
troy_s
guest: But not useful at all beyond that.
20:54
troy_s
guest: Most cinema cameras skip all of that and dump to DPX from their conversion software.
20:55
troy_s
(Which coincidentally is precisely what I'm working toward.)
20:55
troy_s
(granted in my limited I'm-an-idiot sort of approach)
20:56
troy_s
(Or scene referred EXRs)
20:59
troy_s
(and again, I'm not diminishing the usefulness of DNG for transport of the format to somewhere that wants to do its own debayering, but even then, given an open raw format, who really needs DNG exactly?)
21:03
gcolburn
left the channel
21:13
tyrone_
@troy_s: wasn't the DNG a concept to store the camera raw data?
21:14
troy_s
tyrone_: It has come up. But I doubt you can beat the speed of simply dumping the data into a blob onto a disk.
21:15
troy_s
tyrone_: The overhead of DNG would make it nigh on impossible to dump to DNG I'd think off of a processor board. And again, I'm not entirely sure that DNG brings much ot the table at all.
21:25
Bertl
tyrone_: atm, there are both formats available, the dng and the raw, and they can be converted into eachother
21:26
guest
dng is just few bytes more on start of each frame, optionally on the end too, it is not that bad, and the benefits are unbeatable - the format is directly usable in various tools - without a need to ask them to implement a future open raw...
21:26
tyrone_
troy_s: i'm not sure how big the overhead is i haven't made a test program. but i think the overhead is pritty small when you can make a dng template where all the raw data can placed in as a replace. i have to check that up first...
21:26
troy_s
tyrone_: Sure. Again, I'd merley focus on the design need.
21:27
guest
Bertl: i am not proprietary, rather commercial. still better than using grants - usually done for projects which cant make it on their own
21:27
troy_s
tyrone_: If DNG brought anything significant to the table, I'd be entirely +1. One could argue that if / when Resolve or such suites adopt it, it would be decent. It also brings added color issue complexity however.
21:28
troy_s
tyrone_: As in it relies on the viewer to A) do interpolation (not entirely certain that what you get is what someone else gets for example) B) color mangling possibly (if the app is doing the conversion correctly via the matrices etc.)
21:28
troy_s
tyrone_: If it were up to me in a post pipeline, I'd likely rely on formats that don't have any implied transforms and leverage a well documented approach to achieve the correctness (EG OCIO)
21:29
Bertl
guest: there is no point in discussing your closed source camera design, it doesn't benefit anyone except maybe you
21:29
Bertl
so let's focus on things which benefit the community and this project
21:30
tyrone_
troy_s: resolve hasn't got it??? even there cameras supports DNG..... have to test that out when i have more time...
21:31
troy_s
tyrone_: It didn't. Haven't looked more recently.
21:31
troy_s
Oh it did in fact land (For the BMCC)
21:32
troy_s
tyrone_: But again, you are still relying on software to do several things that I'd be wary of.
21:32
guest
Bertl: of course. But it is quite out of direction when troy_s is dumping dng, while apertus being cinema-dng member :)
21:32
troy_s
guest: I am not dumping anything.
21:32
Bertl
I'm using the raws for most purposes as well, it is easier to handle for testing
21:32
troy_s
guest: I just could care less about DNG. And until someone can prove otherwise, it's a moot point.
21:33
troy_s
guest: Name one single instance where DNG is useful over a debayered format again and we can enterain your wandering.
21:33
troy_s
guest: And again, exactly as Bertl has suggested, why on EARTH would you add another layer of crap in for evaluating images?
21:34
troy_s
It makes _zero_ sense, and if you were entirely aware (as a camera vendor now?) of all of the complexities around color and transfer curves, you'd also likely be very ready to admit that DNG adds several layers of hidden issues.
21:36
tyrone_
troy_s:what file format would be the best to integrade into the apertus camera? (DPX,TGA,openEXR, tiff,..) :
21:36
guest
the hidden issues so far are the way how certain software handle the format. usually those which do not follow the specs (ignore ceratain tags) are ones to blame. You can set a pretty simple transformation to keep the image straight as needed
21:36
troy_s
tyrone_: Depends on context. I'm partial to having a frame archived into RGB and that would be DPX from my vantage.
21:37
guest
troy_s: and for the why to not use anything else - storage capacity/bandwidth. Anything debayerd is either lower resolution or larger file.
21:37
troy_s
tyrone_: But again, if you want to interpolate yourself using some other application or whatever, then DNG makes sense, although even then, the raw data isn't encrypted so why not just start at the raw image data?
21:38
guest
why not? e.g. to keep exposure metadata with the frame.
21:38
troy_s
guest: I think if you read your first reply above, you can see why DNG is idiocy for this project currently.
21:38
troy_s
guest: Pretty sure there will be metadata in the raw dump too.
21:39
tyrone_
troy_s: the only reason is when you have a custom raw you have to take care that the commercial software supports this format. thats the only reason to use common format
21:39
troy_s
guest: Anyways... write a DNG converter. Go nuts. I don't mind.
21:39
troy_s
guest: It's just not a file format I could give a damn about.
21:40
troy_s
tyrone_: Post production pipes don't use DNG.
21:40
troy_s
tyrone_: And you can rest assured that the industrial formats are very much well followed in the leading apps.
21:40
troy_s
(IE EXR / DPX)
21:40
troy_s
(or rather, _most_ post production pipes don't use DNG. If you are shooting on an BMCC then you will be.)
21:44
tyrone_
troy_s: exr and dpx is fine for me :-) i just don't know how the file formats are structured....
21:45
troy_s
tyrone_: Again, this is a pretty open area. Anyone could write any FooX to FooY if they were inclined.
21:45
troy_s
tyrone_: The context of the application is likely what dictates needs.
21:45
troy_s
tyrone_: I am content with debayered RGB to eliminate one more step in the pipe. If I can get a reasonable debayer I'm happy.
21:46
troy_s
(AND absolutely control color transforms etc.)
21:46
rexbron_
troy_s: Reading the back chat: Resolve supports DNG
21:46
rexbron_
And has since v9
21:46
troy_s
(Also from a testing standpoint, being able to simply spit out a config update via GitHub or whatnot and have everyone on the 'same page' without a recompile, it's better.)
21:46
troy_s
rexbron_: Yes I Googled after I misspoke.
21:47
troy_s
rexbron_: Same issues though for me.
21:47
tyrone_
troy_s i have to work.... thanks for talking.... i will have the irc open...
21:47
troy_s
tyrone_: Ditto.
21:47
troy_s
tyrone_: haven't seen you idle around here.
21:47
troy_s
tyrone_: Where are you from?
21:49
tyrone_
i'm from switzerland.... helping sebastian for the EU Horizon 2020 grant...
21:49
rexbron_
troy_s: I would argue that maintaining a bayer format until the final grade allows you to preserve both storeage space and IO until it's actually needed in it's full quality
21:50
rexbron_
doing a full quality debayer on footage shot with a 25:1 ratio is a waste of time and resources
21:50
rexbron_
if you are going to offlines, better to do a cheap quick and dirty debayer
21:50
rexbron_
and only put the resources towards what actually makes the final cut
21:51
rexbron_
it's akin to print all vs print circles :)
21:51
troy_s
rexbron_: Sure. Then leave it in raw.
21:51
troy_s
(Like most places do. :) )
21:52
troy_s
tyrone_: How cool is that.
21:52
troy_s
rexbron_: And I agree. Which is precisely the goal of my little conversion of the YCbCr lab.
21:52
rexbron_
troy_s: It
21:52
rexbron_
it's still wasteful to have to have a copy of raw blob and a cDNG sequence at the same time
21:52
troy_s
rexbron_: I would need to add conforming somehow at some point.
21:53
troy_s
rexbron_: Raw blob Gzipped isn't a big deal.
21:53
rexbron_
especially if we are getting into large amounts of data
21:53
rexbron_
LOL
21:53
rexbron_
right
21:53
troy_s
rexbron_: CinemaDNG isn't going to help compression unless you add in loss.
21:53
tyrone_
from where are you troy_s?
21:53
rexbron_
Huffman coding cDNG nets 1.5 to 2x compression
21:53
troy_s
rexbron_: Anyways... I'm pro skipping things that bring nothing to the table.
21:53
troy_s
tyrone_: Canada.
21:53
rexbron_
troy_s and I are both Canuks
21:54
troy_s
rexbron_: And a bzip / 7zip / gzip comparison?
21:54
troy_s
(off of the integer based blob)
21:54
rexbron_
would fail hard
21:54
rexbron_
because they rely on dictonary coding
21:54
guest
troy_s: mistake. it can help compression as it can do lossless (jpeg loss + difference to original raw). The reason for unusability of that in recording is that it produces VBR stream
21:54
rexbron_
noise is deadly
21:54
guest
for archive, it could be quite a nice option
21:54
rexbron_
so it nets you liitle whe
21:55
tyrone_
nice places you have there... :-)
21:55
rexbron_
unless axiom integrates NR circuts into the raw
21:55
troy_s
rexbron_: Little isn't a number. I'm all pro huge gains, but knowing the defaults isn't that horrific.
21:55
rexbron_
troy_s: gzip some samples and give us the data :)
21:56
troy_s
rexbron_: Will do.
21:56
rexbron_
I'd wager that Huffman coding the error from JPEG compression would net you a better compression
21:56
troy_s
7 zip is 25.2 to 17.1
21:56
rexbron_
BMD aren't stupid :)
21:56
troy_s
rexbron_: But does that outweigh the color issues? No.
21:56
rexbron_
on a moving scene :)
21:56
troy_s
rexbron_: It's great in a monotheistic world of software sure.
21:56
rexbron_
pan the camera and watch the compression drop
21:56
troy_s
rexbron_: Just not crossing pipeline walls.
21:56
rexbron_
troy_s: in what sense? your going off your religous deep end
21:57
troy_s
rexbron_: Uh... you have matrices baked into that DNG format
21:57
troy_s
rexbron_: And a very canonized method for decoding it. I would NEVER bet money on X viewer decoding it precisely as Y viewer.
21:57
troy_s
rexbron_: Whereas I can assure you that if I hand you an EXR and the lut set you can.
21:58
troy_s
rexbron_: So religious deep end? Nope.
21:58
troy_s
(bz2 by the way is 16.3)
21:58
rexbron_
there are no matrices baked in
21:58
rexbron_
there are tags to specify them but they aren
21:58
rexbron_
baked in
21:58
troy_s
rexbron_: There are matrix tags. And you are to follow them.
21:58
troy_s
We've covered this.
21:58
troy_s
Again, I'm all for as little futzting with the data as possible.
21:59
guest
there are few 3x3 transformations and an optional LUT for nonlinear data if needed
21:59
rexbron_
those matrices are for camera RGB to CIE XYZ, I fail to see why you
21:59
rexbron_
you'd be against those
21:59
rexbron_
and you can ignore them
21:59
rexbron_
they are TAGS
21:59
rexbron_
baked would be applied to the data
21:59
troy_s
Again, go nuts. Add a layer of DNG complexity to your tester.
21:59
troy_s
I don't mind.
21:59
rexbron_
I'd like DNG so I can throw samples through resolve
22:00
troy_s
I just think it is A) stupid for this B) stupid for me C) a wankfest of extra complexity.
22:00
rexbron_
which is free btw
22:00
troy_s
Sure.
22:00
troy_s
And if I give you DPXs you can't?
22:00
troy_s
;)(
22:00
rexbron_
it's sure as fuck ain
22:00
rexbron_
raw
22:00
troy_s
Anyways... moot point. Go find a DNG library and go nuts.
22:00
troy_s
I could care less about the raw if I am going to control the damn interpolation anyways.
22:01
troy_s
And further, of the pipe, exactly one camera works off of DNGs to the best of my knowledge. Every other has a stub that dumps DPX / EXR.
22:02
troy_s
Not exactly a huge deal is it?
22:02
rexbron_
May I take you down over the one camera cDNG comment?
22:02
rexbron_
what about the Ikonoscope?
22:02
guest
oh, I have one more reason to go DNG - but you wont like it... humanly hinted debayer (algo) on specific image regions
22:02
rexbron_
or the KiniRaw?
22:02
rexbron_
or the BMCC or BMPCC
22:02
troy_s
LOL
22:02
rexbron_
so there are 4
22:02
troy_s
Ok. Go nuts.
22:02
guest
and 0.5 for mine :)
22:03
troy_s
I've never heard of the other two so fair.
22:03
troy_s
I've heard of Alexa, F65, F55, etc.
22:03
troy_s
:P
22:03
rexbron_
oh
22:03
troy_s
guest: Your's doesn't count until you push past vapourware.
22:03
rexbron_
and the Aaton Penelope Delta
22:03
troy_s
rexbron_: Which due to the huge success of DNG is dead.
22:03
troy_s
Double :P
22:03
guest
what makes that? torrent for a sequence?
22:03
rexbron_
troy_s: no
22:03
rexbron_
lol
22:04
rexbron_
even True Sense can't deliver perfect sensors in volume
22:04
troy_s
Anyways... I find the chats interesting, but it's like arguing with the same goal in mind.
22:04
rexbron_
and the cost of recalibrating the camera from the prototype sensor to the production run one bankrupted Aaton
22:04
troy_s
And given the goal, I'm not too concerned about DNG to be honest.
22:04
rexbron_
troy_s: thats fine
22:04
troy_s
(and I have a little more faith being able to examine the actual code off of the raw data at this point.)
22:04
rexbron_
so long as you don't put down those who do
22:05
troy_s
Well... I can make a sizable discussion about the merits of DNG with regard to THIS context
22:05
troy_s
and in that case, using that as a basline, it's a foolish choice at the current moment.
22:05
rexbron_
privalged language alert
22:05
troy_s
Absolutely.
22:05
troy_s
Contextual.
22:05
rexbron_
your speaking about yourself
22:05
troy_s
Wait
22:05
troy_s
I'll bite
22:05
rexbron_
you do all your projects in house start to finish
22:06
troy_s
So you are saying that I should write a DNG output format so that I can write a DNG input reader so that I can view the image and perform transforms on the said data?
22:06
rexbron_
they aren't drama, their MVs or other purely aesthetical works
22:06
troy_s
You are telling me that makes MORE sense than going direct from the raw bayer data to evaluation / interpolation? Right?
22:06
rexbron_
Did I say _you_ should write it? :P
22:06
troy_s
Well given that I am in fact writing a lab viewer
22:06
troy_s
Yes
22:06
troy_s
And if _we_ don't put in work, what exactly gets done?
22:06
rexbron_
I get lots of spectator value
22:06
troy_s
And even if SOMEON ELSE wrote it, would you seriously suggest the above?
22:07
troy_s
Because I think that's bogus and you and I know it.
22:07
troy_s
It's simply foolish design thinking and I know you know it.
22:07
troy_s
(Softspot for DNG aside)
22:07
rexbron_
you've lost me
22:07
troy_s
(and again, I'm _not_ anti DNG. I'm anti DNG for testing purposes totally and completely.)
22:07
troy_s
Is writing DNG software smart here? Nope.
22:07
troy_s
Not for evaluating interps.
22:08
rexbron_
troy_s: Yes, I agree with you
22:08
troy_s
rexbron_: Have you used the imagecache in OIIO?
22:08
rexbron_
I
22:08
troy_s
rexbron_: I know you do.
22:08
troy_s
which is why I'm a little confused. :P
22:09
rexbron_
troy_s: Perhaps I missed the context but I though you were refering to what the Axiom should be able to write down to on camera storage
22:09
troy_s
rexbron_: If it can write to DNG quickly, It would become an option.
22:09
troy_s
(and there is a compelling reason (such as onboard compression would be a reasonable one as you cited)
22:10
troy_s
(not sure if that is possible on the boards chosen etc.)
22:36
se6astian
left the channel