So far, this is an unordered list of topics that you want tdiscussed at the hackfest. Please add your own.
Introduction
Introduce the involved projects and their internal designs. This is supposed to help hackers unfamiliar with these projects gain a high-level understanding.Challenges
Outline the challenges that the projects have in ensuring smooth behavior. Examples here are the performance pitfalls in video when doing things with 30fps or the pipelining in graphics required to really profit from the speed of modern GPUs.awesome browser demos for everyone
This is the tracker goal: Figure out a way to make http://people.freedesktop.org/~company/stuff/video-demo.html work with minimal CPU usage by default. (FIXME: That page needs ogv videos instead of mp4.)GL vs Xrender
It is not clear if desktops should use GL or X and Xrender for their video handling needs (or Xv?). A long-term plan should be put forward that addresses this question and allows modern more demanding video players (NLEs, browsers) to take advantage of these features. A focus should be taken on the abilities of embedded devices, so they can take advantage of this infrastructure.gst-plugins-cairo
Prototype code for a proposed API to integrate GStreamer with Cairo and make use of hardware acceleration exists (including patches for the WebKit browser to use it). Discussion about the feasability of this API should happen, with a focus on allowing seamless integration of GL and keeping it as easy as possible to update to future changes of the stack (see "GL vs Xrender").Add more image formats to Pixman, Cairo (, Xrender)
Currently, Cairo and Xrender exclusively use premultiplied ARGB. Videos are almost always encoded in YUV formats and unmultiplied RGBA is the format of choice in HTML5 and Gtk, so it would make sense to add support for these formats in a way that allows hardware acceleration.improved SIMD acceleration in Pixman
Pixman supplies fast-paths for the most common operations. People have long desired JITing abilities to speed up uncommon operations.integrate GStreamer threading better with GL/X/Cairo
Gstreamer allows passing buffers over thread boundaries. As long as the buffer is considered modifiable, it will only be accessed from one thread at a time. If the buffer is static, it can be read from multiple threads at once. Modifiability is determined by the buffer's refcount. Operating with buffers created in different threads is a common operation. This provides unique challenges to integration with X and GL, as both require synchronization. Two approaches have been taken:- gst-plugins-gl uses a GL global lock that all elements are expected to take when operating with GL. This effectively synchronizes operations and allows passing buffers through thread boundaries. I did not investigate this way of doing things, but would expect it to cause contention on the GL thread and complicate the API used by GL elements.
- gst-plugins-cairo's X code uses one X Display per thread and passes the drawable ID n buffers. This allows it to recreate the cairo_surface_t for a different Display at any time. However, it requires that surfaces created from buffers are short-lived and destroyed in the same thread they were created. Keeping a surface around or passing it in buffers wreaks havoc.
- Another option would be to make every buffer keep its own X display / GL context and carry it along.
hardware-accelerated video decoding
A lot of tools exist that allow hardware-accelerated decoding of video (XvMC, VAAPI, VDPAU, XvBA). They are not very well integrated into the multimedia frameworks however. This should be improved by outlining requirements and restrictions from both sides and proposing a way forward. Proposal of a generic/extensible way to handle hardware-accelerated decoding in gstreamer.- creating an abstraction for rendering and compositing that provides primitive operations that are reimplemented by multiple backends (VAAPI, VDPAU...). Maybe the abstraction should be able to manage a pool of HW surfaces to constrain the memory footprint in the GPU and provide methods to take and release surfaces.
- defining generic mime types (video/x-va-decoded, video/x-va-composited, ...). Here va means Video Accelerated as a generic concept unlinked to vaapi.
- creating specialized mixer and sink elements for overlaying and rendering.
How to make the most of your hardware with GStreamer
GStreamer is making more and more usage of hardware components to offload the main cpu for various tasks (capture, display, decoding, encoding, post-processing...), but this has mostly been done on embedded devices. Using the feedback of the past experience of writing DSP/GPU/hardware plugins on various devices, we should aim to come up with a sets of rules on how to make the most out of available hardware devices in GStreamer and the pitfalls to avoid when writing hardware-accelerated plugins.