#apertus IRC Channel Logs

I have a few queries regarding the RAW data pipeline...

also, what is the separation of tasks between Zync and MachXO2 in USB3 path

X02 and Zync models in use?

got it ... 640HC

the Zynq is the 7020 model used in the MicroZed

although it is kind of pin compatible with the 640 and 2000

it doesn't transport any high speed data

(see USB3 gearwork task)

hi supragya

anyhow, BAndiT1983 : Bertl told me in the morning that the limiting factor of USB3 would be FTDI FT60x

that is 5Gbps

Are we looking at that low of recording fps?

hmm, I was thinking, why not we just piggyback the metadump from the camera with every frame?

and stream it out...

[piggyback] -> leaving the recorder to encode it to whichever format it likes

a Recording computer (PC)

However, as Bertl_oO told me, the USB3 chain is as follows:

Sensor (serial data) -> FPGA (small FIFO buffer) -> AXI Memory Writer (Serial to Memory) -> DDR3 Memory Buffer -> AXI Memory Reader (Memory Block to FIFO) -> (Video Processing Stuff) -> TMDS (HDMI) Encoder -> Serialization to D0-D3 -> Zynq (via Serializer) -> MachXO2 (another FPGA) -> FTDI FT60x -> USB3 output

HDMI encoder would be an obstacle for clear format

what I suggest is that we stop the serialisation D0-D3 for a few moments between frames and add the metadata that camera captures (as Bertl say, a simple register dump) and add it behind the frame before serialisation, all could be done on one channel

if you are sure, that the output would be fast enough, then go on

for meta

still, I need someone like g3gg0, or Bertl on this, as to how it is feasible, where do we induce such piggybacking (in Zync maybe)

Reasoning: As each frame has it's own meta, the conversion to DNG sequence is easy

currently axiom cameras can have the following meta only:

anything else?

i still doubt we can stream out DNG or any other format that needs bigger processing effort

** I struggle to say we can get DNG out **

the recording computer is to take this RAW12 and meta and convert it to DNG (or any other format for that matter) in it's leisure time

how will be raw12 data separated? how will be the metadata recorded along with it?

can we get away with normal SSD or do we need some faster system for recording without losing data? or is the data buffered first, after receiving it over USB?

then this flushed data is taken up (along with appropriate flushed meta) to convert it to suitable format

the stream needs also preparation

which format will meta data have?

[ don't forget about future extensions of meta data] -> 2 points on this

Even if we don't get 5Gbps, we pass the RAW12 with a "Sequence ID"

On an another (maybe GbE), we can stream metadata (that we would have piggybacked)

The PC encoder has to just match the meta with frames

those will be piggybacked behind sensor dump then...

the format is not the worry here, it is the piggybacking technique

then we won't use it

https://lab.apertus.org/T951 -> my mail was to satisfy the point 2 (the final format), I never tried to put forth that being the way we would stream the data out of the camera itself

supragya, no problem here, just trying to get a clear picture of the process

no, everything ok

what do you think about this from your perspective

everything looks clear for now, but the practical side of the thing is totally unknown yet, so can't say much

Yes

for g3gg0!

hi

sure

your call BAndiT1983.

private? I don't have that on plate today

anyways, me too for shower, will be back soon. BAndiT1983: I am fine any way as long as we could communicate :)

%n

re

Important: 16, 50, 53, 67, 71, 84, 88, 76

g3gg0: https://docs.google.com/document/d/1gxJGt74VogWa_CyxGyHXaE6gpYT1UJDx67uYQ9NmMqQ/edit

as far as i understood se6astian, we would have two USB3 chips with up to 3.2 gbits/s each

"Sequence ID" then? this could help?

http://irc.apertus.org/index.php?day=06&month=05&year=2018#105

> the recording computer is to take this RAW12 and meta and convert it to DNG (or any other format for that matter) in it's leisure time

Here is how it works, let me get a few links

that's why "leisure", we don't need realtime as long as we can store the stream somewhere fast enough

hehe you are redesigning what MLV was meant for

to not let camera do much

if you look at [http://www.cmosis.com/products/product_detail/cmv12000]

Bertls says, we store the meta as a register dump

but we need also additional data, like time

and a timestamp

yep

off the camera...

We can have KLV format for that (key, length, value)

keys maybe as (0-frame, 1-lensMeta, 2-sensorMeta)

the difference I feel g3gg0 is this:

we just throw out what we have, as fast we can

it's more like firmware version

piggyback if ofcourse only a solution on single channel

[sID|RAW12 frame][sID|RAW12 frame][sID|RAW12 frame][sID|RAW12 frame][sID|RAW12 frame]

more like (2-CMV12ksensorMeta, 3-someOtherSensorMeta)

to prevent that, consider something like json with generic keys for sensor data

save the effort encoding binary data into JSON but use binary tags and voila

there is also BSON for binary json

we can re-use same memory, as meta data will be the same length for every frame

*slower

I never went for JSON ;)

ah that came from you :)

for performance a fixed one would fit better

the only reason I described regdump to be streamed is performance... as once it has been read... nothing more than streaming has to be done...

memset should be fast, if you pre-allocate

[memset]-> wouldn't we need to call it multiple time

no need for memset

either you just memcpy or set the elements manually

you can also memcopy, my point was, that we should re-use pre-allocated chunk of memory for meta data

while setting manually can reduce size of the regdump, it imposes new restrictions -> for every new sensor, a new struct setter from regdump has to be written (anyways we would have to, but then it would have resided at encoder only)

result: unneded load on zync, Gbe is not utilised fully

so from what i've read, i'd split the writing part into two processes

b) intermediate file -> CDNG

yes

you then have to:

b) extract interesting information from metadata and write them - if they changed since the last metadata

a fitting point to note here:

wasnt it on the "small PC" side ?

we can have two streams and then we have 50fps

MLV is single stream, am i wrong?

(called chunks)

yes

and goal was to write CDNG ?

struct writing is in camera

axiom side is restricted to META/RAW12/...

right?

(that is the problem statement)

and then we get the final stream

can you see this?

sketched correctly?

zync is meant to be within the axiom camera

errh i mean that was obvious :)

all we see to the outer world is the CMOS and the two USB3

light in, USB3 out

we are talking about how it one level above would look like

"meta" is more or less the register dump of the CMOS and some other stuff

otherwise, what g3gg0 says is correct

so okay thats what exists (left side) and what should get implemented (right side)

chunks etc

okay

we all agree?

yes g3gg0

then a code for zync.... to serialise the RAW and meta in stream

maybe not, lets do the next steps

on the right,

so the "magic here" has to buffer the RAW12 stream locally

then it is a two way step:

2. to initiate a process that can access this flushed data to convert it to CDNG or whatever we feel fit, and then flush to disk back

yes, seems legit for now

focus should be on lossless data

never knew...

they store their primary data as proprietary ARRIRAW format, afterwards everything is converted to the needs of the projects/films

also saw somewhere a frame server example for arriraw, maybe in their ads

not the worst solution to me

they do it, to provide preview

A few things have come up now:

2. to finally set the stream format

4. A system to convert to CDNG (Encoder_

nothing to do on zynq side. should have been some other work

that helps

meta on same stream as frames

or different Gbe

how should it look like then? meta/RAW piggyback? seqID|Frame and seqID|meta on differenct channels? or MLV?

the "from-zynq"-format?

and meta?

-> ask and make an assumption

@bertl_oO / @se6astian

if you dont know, ask people who could now and with all the information make an assumption on how it would look like

okay... so this needs further investigation... and that I should ask Bertl_oO or se6astian, choose a fitting format and go on..

are we then doing RAW12 series -> MLV -> CDNG?

O.o 2 conversions, so that we don't have to make efforts on MLV-> CDNG

noooo, MLV is not conversion

chained in series

1 min

if you want to write DNG, which pixel format will you have to use?

the same encoding you get out of the AXIOM?

infact the RAW could be memcpy'ed in the pixel data of DNG

it's just raw12, fitting DNG rather well

don't remember if byte swapping have to be done or not

ok maybe i mixed something then

and modify e.g. MLVFS to accept that RAW12 bitstream for DNGs

as cinema dng is a lot of work to do it right, i'd reuse existing tools

?

if I understand right... a meta is (mode changer) for further frames

so if the meta does not changes between frames, it should not be written

https://www.magiclantern.fm/forum/index.php?topic=13152.0

current system as in ... dumping quickly into cache

serial -> you convert the struct/object to string (serialise to JSON maybe) and compare

o.O

neither in C nor in java

the only string in raw video processing you would ever see is maybe the "author name"

strncmp > memcmp :P

you have binary data and you simply can compare fields in them

yep, he is a fairy

depends on how ">" operator was defined in this context :)

let's not go to other world :)

a file with data in format (to be discussed) that would make it look like a AXIOM stream

other system... reads the cache (concurrently or after the MLV writer is done) and then writes DNG sequence

concurrently is more trouble than required

that can for the first implementation also be done in post

as soon the recording stops, the CDNG conversion could be started

either like the FUSE things work "in-place" e,g, via samba

just image on-set situation, things are filmed, then backuped, if some one would convert live and loase data, then he/se will be a dead man/woman

definitely!

and the other: MLV to CDNG other time (maybe next year)

seems like it

theres a 55 page discussion about cinemadng

> thx for coming chmee, save us from Adobe hell.

there are many comments that will give you a picture of how complex CDNG is

never knew this

> no. cinema dng and "simple" dng differs much - in terms of mandatory tags. i use a cdng-template, i merged myself from looking into another cdng-frames. just to clarify some things:

aye aye, I am afraid now

so should I begin emulation -> stream to MLV flush to disk system?

would you write the RAW12 bitstream unmodified into VIDF frames and then modify the CDNG converter (maybe MLVFS) so it can work with it? (i'd prefer that way)

or:

Kjetil is probably trying to do his magic on the document ;)

the first option is probably a bit more work (modifying two things the same time plus extending MLV's header specification for non-linear stuff?)

the latter is probably the quickest win

when thinking about it, i am not sure

especially when the cache should not contain post-processing (like gamma) the latter one is probably worse

don't know how canon bit encoding is different?

prores also

endianess depends on the MCU/CPU in the camera, some are big, other small

it can be read out by using markers in the beginning, see tiff header

you are losing track, because you are trying to do the details first

my point was the gamma curve in axiom was special, right?

dont remember the term

can't find info about that in wiki

adc slope?

will look in the datasheet, maybe there is something

anyway. latter option would involve that you convert the pixel values in the barebone when writing MLV

so how should I go about? 1 (change MLVFS) or 2 (additional step)

seems doable

can CDNG tools handle this multi slope raw?

(i bet they just implement the minimum they need to for their favorite cameras)

with existing image data

so: i'd start converting the RAW12 to a normal linear raw if the multi slope feature is used

ah, read the specs again, there was a linearization table

I am losing packets on IRC here

sups, which irc client?

HexChat BAndiT1983 !

sups, same here, no problems so far

see you

and i am quite sure even if CDNG could handle that, the tools wont

so for the first prototype: just convert it to a normal linear raw

then you can check your cache-MLV with common tools if the data is fine

e.g. mlv_dump

raw.h

https://bitbucket.org/hudson/magic-lantern/src/0e6493e8ac5e118976b94237b5048c436f379d98/src/raw.h?at=unified&fileviewer=file-view-default

yep

https://bitbucket.org/hudson/magic-lantern/src/0e6493e8ac5e118976b94237b5048c436f379d98/modules/mlv_rec/mlv_dump.c?at=unified&fileviewer=file-view-default

thats how the mlv tools read/write pixels from/to a VIDF stream

if you realize it will get too slow, put that effort into the CDNG writer

lets continue via mail as usual

cu

see you

off to bed now ... have a good one everyone!