VideoHackfest/Conclusions

YUV in pixman

It has been a goal for a while to make video a first-class citizen in the image stack. A concrete API proposal on how to integrate video into pixman was reached and reviewed.

Links:

Actions:

YUV in Cairo

To make it easy for applications to use these different formats, the Cairo API needs to be extended to allow them. A proposal does exist and has been reviewed, details need to be finalized.

Links:

Actions:

locking in Cairo

GStreamer takes a lot of freedom in deciding which threads to schedule elements in. Most hardware backends however require proper serialization of commands. The current gst-plugins-cairo and gst-plugins-gl code provide different, but ugly ways to achieve this. Cairo's internal thread safety guarantees match these requirements very well. But the different Cairo backends don't always keep these guarantees. Interaction with these locking mechanisms from outside applications is not possible yet either.

Links:

Actions:

locking in Mesa

The GLX specification does not allow binding the same GLX context in multiple threads at the same time. This is a requirement for both Cairo's and GStreamer's threading model. An extension was proposed and initial code developed to support this requirement in the same way as Apple's GL does by default. Windows does not support this and would require potentially expensive fallback code.

Links:

Actions:

switch GStreamer to Cairo as default video transport model

The current approach to handling video in GStreamer is very outdated. It does not allow hw-accelerated buffers nor does it provide a unified API to modify video buffers, which leads to fragmentation and duplication. It was agreed that using Cairo to provide an abstraction as a drawing API and abstraction over different backends was a good idea.

Links:

Actions:

gst-plugins-gl

The general consensus was that gst-plugins-gl is a hack. It was necessary in the past to get things to work, but is not a good way forward. However, the functionality or performance provided by the current elements needs to continue working. Developers shared the opinion that gst-plugins-cairo with cairo-gl surfaces is the best way to achieve this.

Links:

Actions:

XRenderPutImage

Currently there is no way to upload video data to the X server for later use. The XV extension is not sufficient for anything more complicated than a simple video player.

Links:

Actions:

JIT in Pixman

ORC was investigated as a potential JIT for pixman. A lot of talk has happened, but no clear conclusions were reached. The idea to share a JIT with Mesa's GLSL stack was also brought up.

Links:

Actions:

Video decoding Acceleration

Hardware video decoding is a feature that everybody is very excited about and wants to integrate into GStreamer as soon as possible. But developers feel that the current APIs available - the focus was on VAAPI and VDPAU - to do hardware acceleration fail to integrate well. No developer managed to create even demo-quality integration with gst-plugins-cairo, even though quite a few tried. In particular, both VDPAU and VAAPI...

... do not integrate with existing frameworks

Both libraries fail to use (or allow easy conversion to) existing objects (GEM handles, GL textures, ...) but only provide access to an encapsulation. This makes it hard for a flexible framework like GStreamer to make use of its features without limiting itself.

... do not have a clear scope

In particular VDPAU advertises itself as a video playback framework using hardware acceleration. Unfortunately, they both limit themselves to only the formats and functionalities supported by the hardware and see fullscreen video playback as the single usecase. This needlessly complicates integration with existing frameworks

... provide a very complicated API

For decoding video, an API like provided by libtheora, libmpeg2 or even ffmpeg is very simple and allows easy integration with both the demuxing and the postprocessing/display side. VAAPI and VDPAU require complex setups and preprocessing of the data to work properly.

... overlap with GL functionality

It was felt that a lot of processing features provided by these libraries are equally well available using shaders in GL or by providing GL extensions. It was unclear why it is necessary to use a separate way to do this.

Links:

Actions:

Timeline

The end goal of all of this is to get the code into users' hands quickly, but allow a smooth and non-disrupting upgrade. Desired dates have been put forth to achieve this: