This is a starting point for information related to passing around OpenGL textures inside buffers in GStreamer pipelines.
AdamLofts: I've made some comments inline. Hope this is ok.
Design Constraints
* In general, elements require their own connection to the X server. This requires passing around display in caps, as well as an XID (or XIDs) to the resource in the X server that corresponds to the contents of the buffer.
Are the caps the right place for the display? Can a buffer not just have a reference to its creator (the last element in the chain) which will have a reference to the X Server.
* Neighboring elements (in a chain) may be able to share the same connection to the X server. Not sure if this is a good idea.
* If a shared connection is used by a chain of elements, the buffer_alloc() function of each element should be chained to the next, so that the last element in the chain creates the connection.
* Planar video (e.g., I420) requires multiple textures per buffer.
Unresolved Issues
* Should the XID representing a buffer on the server be an XImage or GLXPixmap? If XImage, there is the added possibility of using X to draw on the images. However, with GLXPixmaps, additional state information such as depth buffer, stencil buffer, etc.
Can a buffer not be represented by just a texture object (or a group of textures) ? You can use frame buffer objects to get the texture to and from main memory.
* Need to make sure that normal buffer operations work, especially _copy(). This requires a subclass of GstBuffer.
* A GLXContext is thread local. Gstreamer makes no guarantees about which thread will call chain() even on only one element. A possible solution is to use thread local storage and share textures between glx contexts.
* There are many types of texture / x pixmap. The shader you use depends on the type of texture, e.g. 24-bit rgb, 8-bit alpha, float. This complicates the interface for allocating buffers.
Ideas/Code
Some development is being done in gst-plugins-bad in sys/glsink.
More ideas in http://alofts.co.uk/git/gstgpu.git/.
Generic code for pushing raw video into an XPixmap would be nice.