We need a mechanism to transport extra data in a GstBuffer. Unlike caps the content is likely to change for each buffer.
Examples:
- Pan and scan (video subregion) position information
- interlacing markers (top field, bottom field)
- theora granulepos, instead of slapping it in the 'offset' field
- MPEG Decoding timestamps, which we currently never send along.
- Feature-Extraction elements can add feature data
- Some image capture sensors can provide histograms with the frames
- Some formats (mpeg-ts from HDV cameras) provide date/gain/shutter-speed/scene-change at a regular interval but not for every frame (around 10 times per second).
- packet priority and timeout could be used by network plugins (such as DCCP) to send more important packets when congestion is detected.
Ideas:
GstStructure in GstBuffer
GstTaglist in GstBuffer
- syncronized downstream events
Store GstStructures in GstIndex
GstStructure is also use in GstTaglist. The structure can be hierarchical. We could require that the producer uses its element name as a root tag. This way the taglist could carry multiple data-sets identified in a unique way. Elements that can use the data need to know the name of the element that created it.
Issues:
- Performance
- copying, freeing buffers need to check/handle the presence of extra metadata
- Transformations
- there might be transformation that render the data useless. e.g. if the data is a histogram, color changes invalidate it. if its a sub-region, size changes invalidate it.
- Mixing
- if we go for (B) we could use the tag merge modes
- Muxing
- using and/or dropping the data, it does not sound sensible to send it further
- Copies (tee)
- make duplicates