Back in June '25, I implemented a new speech synthesis element using the ElevenLabs API.

In this post I will briefly explain some of the design choices I made, and provide one or two usage examples.

POST vs. WSS

ElevenLabs offers two interfaces for speech synthesis:

• Either open a websocket and feed the service small chunks of text (eg words) to receive a continuous audio stream

• Or POST longer segments of text to receive independent audio fragments

The websocket API is well-adapted to conversational use cases, and can offer the lowest latency, but isn't the most well-suited to the use cases I was targeting: my goal was to use it to synthesize audio from text that was first transcribed, then translated from an original input audio stream.

In this situation we have two constraints we need to be mindful of:

• For translation purposes we need to construct large enough text segments prior to translating, in order for the translation service to operate with enough context to do a good job.

• Once audio has been synthesized, we might also need to resample it in order to have it fit within the original duration of the speech.

Given that:

• The latency benefits from using the websocket API are largely negated by the larger text segments we would use as the input

• Resampling the continuous stream we would receive to make sure individual words are time-shifted back to the "correct" position, while possible thanks to the sync_alignment option, would have increased the complexity of the resulting element

I chose to use the POST API for this element. We might still choose to implement a websocket-based version if there is a good story for using GStreamer in a conversational pipeline, but that is not on my radar for now.

Additionally, we already have a speech synthesis element around the AWS Polly API which is also POST-based, so both elements can share a similar design.

Audio resampling

As mentioned previously, the ElevenLabs API does not offer direct control over the duration of the output audio.

For instance, you might be dubbing speech from a fast speaker with a slow voice, potentially causing the output audio to drift out of sync.

To address this, the element can optionally make use of signalsmith_stretch to resample the audio in a pitch-preserving manner.

When the feature is enabled it can be used through the overflow=compress property.

The effect can sometimes be pretty jarring for very short input, so an extra property is also exposed to allow some tolerance for drift: max-overflow. It represents the maximum duration by which the audio output is allow to drift out of sync, and does a good job using up intervals of silence between utterances.

Voice cloning

The ElevenLabs API exposes a pretty powerful feature, Instant Voice Cloning. It can be used to create a custom voice that will sound very much like a reference voice, requiring only a handful of seconds to a few minutes of reference audio data to produce useful results.

Using the multilingual model, that newly-cloned voice can even be used to generate convincing speech in a different language.

A typical pipeline for my target use case can be represented as (pseudo gst-launch):

input_audio_src ! transcriber ! translator ! synthesizer

When using a transcriber element such as speechmaticstranscriber, speaker "diarization" (fancy word for detection) can be used to determine when a given speaker was speaking, thus making it possible to clone voices even in a multi-speaker situation.

The challenge in this situation however is that the synthesizer element doesn't have access to the original audio samples, as it only deals with text as the input.

I thus decided on the following solution:

input_audio_src ! voicecloner ! transcriber ! .. ! synthesizer

The voice cloner element will accumulate audio samples, then upon receiving custom upstream events from the transcriber element with information about speaker timings it will start cloning voices and trim its internal sample queue.

To be compatible, a transcriber simply needs to send the appropriate events upstream. The speechmaticstranscriber element can be used as a reference.

Finally, once a voice clone is ready, the cloner element sends another event downstream with a mapping of speaker id to voice id. The synthesizer element can then intercept the event and start using the newly-created voice clone.

The cloner element can also be used in single-speaker voice by just setting the speaker property to some identifier and watching for messages on the bus:

gst-launch-1.0 -m -e alsasrc ! audioconvert ! audioresample ! queue ! elevenlabsvoicecloner api-key=$SPEECHMATICS_API_KEY speaker="Mathieu" ! fakesink

Putting it all together

At this year's GStreamer conference I gave a talk where I demo'd these new elements.

This is the pipeline I used then:

AWS_ACCESS_KEY_ID="XXX" AWS_SECRET_ACCESS_KEY="XXX" gst-launch-1.0 uridecodebin uri=file:///home/meh/Videos/spanish-convo-trimmed.webm name=ud \
  ud. ! queue max-size-time=15000000000 max-size-bytes=0 max-size-buffers=0 ! clocksync ! autovideosink \
  ud. ! audioconvert ! audioresample ! clocksync ! elevenlabsvoicecloner api-key=XXX ! \
    speechmaticstranscriber url=wss://eu2.rt.speechmatics.com/v2 enable-late-punctuation-hack=false join-punctuation=false api-key="XXX" max-delay=2500 latency=4000 language-code=es diarization=speaker ! \
    queue max-size-time=15000000000 max-size-bytes=0 max-size-buffers=0 ! textaccumulate latency=3000 drain-on-final-transcripts=false extend-duration=true ! \
    awstranslate latency=1000 input-language-code="es-ES" output-language-code="en-EN" ! \
    elevenlabssynthesizer api-key=XXX retry-with-speed=false overflow=compress latency=3000 language-code="en" voice-id="iCKVfVbyCo5AAswzTkkX" model-id="eleven_multilingual_v2" max-overflow=0 ! \
    queue max-size-time=15000000000 max-size-bytes=0 max-size-buffers=0 ! audiomixer name=m ! autoaudiosink audiotestsrc volume=0.03 wave=violet-noise ! clocksync ! m.

Watch my talk for the result, or try it yourself (you will need API keys for speechmatics / AWS / elevenlabs)!

The GStreamer Material Exchange Format (MXF) muxer and demuxer elements so far only supported extracting Vertical Ancillary Data (VANC) as closed captions. Any other VANC data was silently dropped. This was primarily reflected by the sink pad template of mxfmux.

  SINK template: 'vanc_sink_%u'
    Availability: On request
    Capabilities:
      closedcaption/x-cea-708
                 format: cdp
              framerate: [ 0/1, 2147483647/1 ]

mxfmux and mxfdemux have now been extended to support arbitrary VANC data.

SMPTE 436 (pdf) specification defines how the ancillary data is stored in MXF. SMPTE 2038 (pdf) defines the carriage of Ancillary Data Packets in an MPEG-2 Transport Stream acting as a more structured format (ST2038) in comparison to the line-based format (ST436M). mxfdemux converts from ST436M to ST2038 while mxfmux converts from ST2038 to ST436M. So mxfdemux now outputs VANC (ST436M) essence tracks as ST2038 streams and mxfmux consumes ST2038 streams to output VANC (ST436M) essence tracks.

A breaking change was introduced to support this in the muxer, by updating the acceptable caps on the pad. The sink pad template of mxfmux has now changed to meta/x-st-2038 instead of the earlier closedcaption/x-cea-708. Applications can use cctost2038anc for converting closed captions to ST2038.

  SINK template: 'vanc_sink_%u'
    Availability: On request
    Capabilities:
      meta/x-st-2038
              alignment: frame (gchararray)

While the pad templates of mxfdemux haven't changed as shown below, the caps on the source pad are going to be meta/x-st-2038 for VANC data, so applications have to handle different caps now. Closed captions can be extracted via st2038anctocc.

  SRC template: 'track_%u'
    Availability: Sometimes
    Capabilities:
      ANY

The older behaviour is still available via an environment variable GST_VANC_AS_CEA708. In addition, mxfdemux can now read both, 8-bit and 10-bit VANC data from MXF files.

The ST2038 elements available in Rust plugins and described in an earlier post here, have also seen some fixes for correctly handling alignment and framerate.

As part of our ongoing efforts to extend GStreamer's support for ancillary data, I've recently improved the ancillary data handling in the Blackmagic DeckLink plugin. This plugin can be used to capture or output SDI/HDMI/ST2110 streams with Blackmagic DeckLink capture/output cards.

Previously only CEA 608/708 closed captions and AFD/Bar ancillary data was handled in that plugin. Now it can also additionally handle any other kind of ancillary data via GstAncillaryMeta and leave interpretation or handling of the concrete payload to the application or other elements.

This new behaviour was added in this MR, which is part of git main now, and can be enabled via the output-vanc properties on the video source / sink elements.

The same was already supported before by the plugin for AJA capture/output cards.

For example the following pipeline can be used to forward an SDI stream from an one DeckLink card to an AJA card

gst-launch-1.0 decklinkvideosrc output-vanc=true ! queue ! combiner.video \
  decklinkaudiosrc ! queue ! combiner.audio \
  ajasinkcombiner name=combiner ! ajasink handle-ancillary-meta=true

With both the AJA and DeckLink sink elements, special care is needed to not e.g. output closed captions twice. Both sinks can retrieve them from GstVideoClosedCaptionMeta and GstAncillaryMeta, and outputting from both will likely lead to problems at the consumer of the output.

The GStreamer Conference is an annual gathering that brings together developers, contributors, and users of the GStreamer multimedia framework. It serves as a platform for sharing knowledge, discussing the latest advancements, and fostering collaboration within the open-source multimedia community.

This year’s conference was held in London at the impressive Barbican Center, located within the Barbican Estate, a residential complex rebuilt after World War II in the brutalist architectural style.

It was a pleasure to meet in person with the colleagues, from different companies and backgrounds, I usually collaborate with remotely, sharing and discussing their projects and results.

Recently, UbiCast, which generously records and streams the conference, has uploaded all talks of this year conference to their site, exclusively for the GStreamer Conference.

In this blog post, I’ll share the talks delivered by my fellow igalians:

Animate Your Subtimelines in GES #

GstVA and GStreamer-VAAPI updates #

Time Remapping and GES: Implementation Details and Latest Updates #

soothe: a proposal for encoder testing #

GstWebRTC in WebKit, current status & plans #

VVC/H.266 in GStreamer #

Video Reshaping with Skia #

Vulkan Video: pipeline update #

Following the GStreamer Conference, we hosted our Autumn hackfest at Amazon’s offices in the City of London. This time I worked on GStreamer Vulkan.

This year, two other conferences typically held in the US, FOMS and Demuxed, also took place in London. I attended FOMS, where I discovered the vibrant MOQ project.

Finally, I’d like to thank Centricular for organizing the event, especially Tim-Philipp Müller, and even more particularly Igalia for sponsoring it and allowing me to participate in this project that’s close to my heart.

And that’s all, mates. Cheers!

We're pleased to announce that the video recordings for this year's GStreamer conference are now available on the GStreamer Conference 2025 channel on ubicast.tv.

Many thanks to Ubicast for recording the talks again!

While working on other ancillary data related features in GStreamer (more on that some other day), I noticed that we didn't have support for sending or receiving ancillary data via RTP in GStreamer despite it being a quite simple RTP mapping defined in RFC 8331 and it being used as part of ST 2110.

The new RTP rtpsmpte291pay payloader and rtpsmpte291depay depayloader can be found in this MR for gst-plugins-rs, which should be merged in the next days.

The new elements pass the SMPTE ST 291-1 ancillary data as ST 2038 streams through the pipeline. ST 2038 streams can be directly extracted from or stored in MXF or MPEG-TS containers, can be extracted or inserted into SDI streams with the AJA or Blackmagic Decklink sources/sinks, or can be handled generically by the ST 2038 elements from the rsclosedcaption plugin.

For example the following pipeline can be used to convert an SRT subtitle file to CEA-708 closed captions, which are then converted to an ST 2038 stream and sent over RTP:

$ gst-launch-1.0 filesrc location=file.srt ! subparse ! \
    tttocea708 ! closedcaption/x-cea-708,framerate=30/1 ! ccconverter ! \
    cctost2038anc ! rtpsmpte291pay ! \
    udpsink host=123.123.123.123 port=45678

Now you might be wondering how ST 291-1 and ST 2038 are related to each other and what ST 2038 has to do with RTP.

ST 291-1 is the basic standard that defines the packet format for ancillary packets as e.g. transmitted over SDI. ST 2038 on the other hand defines a mechanism for packaging ST 291-1 into MPEG-TS, and in addition to the plain ST 291-1 packets provides some additional information like the line number on which the ST 291-1 packet is to be stored. RFC 8331 defines a similar mapping just for RTP, and apart from one field it provides exactly the same information and conversion between the two formats is relatively simple.

Using ST 2038 as generic ancillary data stream format in GStreamer seemed like the pragmatic choice here. GStreamer already had support for handling ST 2038 streams in various elements, a set of helper elements to handle ST 2038 streams, and e.g. GStreamer's MXF ANC support (ST 436) also uses ST 2038 as stream format.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• Fix showtime video player showing washed-out colours for HDR videos when subtitles are active
• core: performance improvements for elements with many source pads
• aacparse: support streams which do not have frequent LOAS config
• av1parse: Fix duplicated frames issue in frame splitting
• fmp4mux: Fix EAC3 datarate calculation and substream writing
• gtk4painablesink: fixes glitches with padded buffers such as for sub-sampled video formats with odd sizes
• mpegtsmux: PUSI flag and ID3 tag handling fixes
• rtpbaseaudiopay2: Fix marker bit handling for DISCONT and RESYNC buffer flags
• rtpvp9pay: Fix parsing of show-existing-frame flag, fixes compatibility with vavp9lpenc
• splitmuxsink: accept pads named 'sink_%u' on the muxer for fmp4 muxer support
• webrtcsink: Correct lock ordering to prevent deadlock
• gst-plugins-rs meson build gained an `auto_plugin_features` option and no longer requires all gstreamer libraries to be available
• v4l2 device monitor fixes
• x265enc: advertise latency based on encoder parameters instead of hard-coding it to 5 frames
• cerbero package builder: Add Rust support for 32-bit Linux x86
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.8 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly and will be published on the Downloads page.

For one of our recent projects, we worked on adding multitrack audio capabilities to the GStreamer FLV plugin following the Enhanced RTMP (v2) specification. All changes are now merged upstream (see MR 9682).

Enhanced RTMP

As the name suggests, this is an enhancement to the RTMP (and FLV) specifications. The latest version was released earlier this year and is aimed at meeting the technical standards of current and future online media broadcasting requirements, which include:

• Contemporary audio/video codecs (HEVC, AV1, Opus, FLAC, etc.)
• Multitrack capabilities (for concurrent management and processing)
• Connection stability and resilience
• and more

FLV and RTMP in GStreamer

The existing FLV and RTMP2 plugins followed the previous versions of the RTMP/FLV specifications, so they could handle at most one video and one audio track at a time. This is where most of the work was needed, to add the ability to handle multiple tracks.

Multitrack Audio

We considered a couple of options for adding multitrack audio and enhanced FLV capabilities:

• Write completely new element(s), preferably in Rust (or)
• Extend the current FLV muxer and demuxer elements

Writing a fresh set of elements from scratch, perhaps even in Rust, would have potentially made it easier to accommodate newer versions of the specification. But the second option, extending the existing FLV muxer/demuxer elements turned out to be simpler.

Problems to Solve

So, at a high level, we had two problems to solve:

1. Handle multiple tracks

   As mentioned above, the FLV and RTMP plugins were equipped to handle only one audio and one video track. So we needed to add support for handling multiple audio and video tracks.

2. Maintain backwards compatibility

   There should be no breakage in any existing applications that stream using the legacy FLV format. So, the muxer needs a mechanism to decide whether a given audio input needs to be written into the FLV container in the enhanced format or the legacy format.

A two-step solution

We arrived at a two-step solution for the implementation of multiple track handling:

1. Use the audio template pad only for the legacy format and define a new audio_%u template for the enhanced format. That makes it clear which stream needs to be written as a legacy FLV track or an enhanced FLV track. The index of the audio_%u pads is also used as the track ID when writing enhanced FLV.

2. Derive a new element from the existing FLV muxer called eflvmux, which defines the new audio_%u pad templates. The old flvmux will continue to support only the legacy codec/format. That way, the existing applications that use flvmux for legacy FLV streaming will not face any conflicts while requesting the pads.

Minor Caveat

Note that applications that use eflvmux need to specify the correct pad template name (audio or audio_%u) when requesting sink pads to ensure that the input audio data is written to the correct FLV track (legacy or enhanced).

Some formats such as MP3 and AAC are supported in both legacy and enhanced tracks, so we can't just auto-detect the right thing to do.

Interoperability issues

An interesting thing we noticed while testing streaming of multitrack audio with Twitch.tv is that when we tried to stream multiple enhanced FLV tracks or a mix of single legacy track and one or more enhanced FLV tracks, none of the combinations worked.

On the other hand, OBS was able to stream multitrack audio just fine to the same endpoint. Dissecting the RTMP packets sent out by OBS revealed that Twitch can accept at most two tracks, one legacy and one enhanced, and the enhanced FLV track's ID needs to be a non-zero value. To our knowledge, this is not documented anywhere.

It was a simple matter of track ID semantics which could be easily missed without referring to the OBS Studio code. This is also the case with FFmpeg which we recently noticed.

So we have requested a clarification on the track ID semantics from the enhanced RTMP specification maintainers and got a confirmation that 0 remains a valid value for track ID. As mentioned in the specification, it can be used to represent the highest priority track or the default track.

However, when streaming to servers like Twitch you may need to take care to request only pads with index greater than 0 from eflvmux because it may not accept tracks with ID 0.

Sample Pipelines to test

Here are some sample pipelines I used for testing the muxer and demuxer during the implementation.

Scope for other features

The FLV muxer and demuxer have undergone significant structural changes in order to support multiple audio tracks. This should make it easy to update the existing multitrack video capability merge request as well as add support for advanced codecs listed in the specification, some of which (like H265 and AV1) are already in progress.

There is also a work-in-progress merge request to add the eRTMP related support to the rtmp2 plugin.

P.S.: You can also refer to my talk on this topic at the GStreamer Conference that took place in London last month. The recording will be soon published on Ubicast.

At the GStreamer project, we produce SDKs for lots of platforms: Linux, Android, macOS, iOS, and Windows. However, as we port more and more plugins to Rust 🦀, we are finding ourselves backed into a corner.

Rust static libraries are simply too big.

To give you an example, the AWS folks changed their SDK back in March to switch their cryptographic toolkit over to their aws-lc-rs crate ^[1]. However, that causes a 2-10x increase in code size (bug reports here and here), which gets duplicated on every plugin that makes use of their ecosystem!

What are Rust staticlibs made of?

To summarise, each Rust plugin packs a copy of its dependencies, plus a copy of the Rust standard library. This is not a problem on shared libraries and executables by their very nature, but on static libraries it causes several issues:

• Rust leaks unexported symbols from native staticlibs
• On some platform, linking against multiple Rust staticlibs is impossible

First approach: Single-Object Prelinking

I won't bore you with the details as I've written another blog post on the subject; the gist is that you can unpack the library, and then ask the linker to perform "partial linking" or "relocatable linking" (Linux term) or "Single-Object Prelinking" (the Apple term, which I'll use throughout the post) over the object files. Setting which symbols you want to be visible for downstream consumers lets dead-code elimination take place at the plugin level, ensuring your libraries are now back to a reasonable size.

Why is it not enough?

Single-Object Prelinking has two drawbacks:

• Unoptimized code: the linker won't be able to deduplicate functions between melded objects, as they've been hidden by the prelinking process.
• Windows: there are no officially supported tools (read: Visual Studio, LLVM, GCC) to perform this at the compiler level. It is possible to do this with binutils, but the PE-COFF format doesn't allow to change the visibility of unexported functions.

Melt all the object files with the power of dragons' fire!

As said earlier, no tools on Windows support prelinking officially yet, but there's another thing we can do: library deduplication.

Thanks to Rust's comprehensive crate ecosystem, I wrote a new CLI tool which I called dragonfire. Given a complete Rust workspace or list of static libraries, dragonfire:

1. reads all the static libraries in one pass
2. deduplicates the object files inside them based on their size and naming (Rust has its own, unique naming convention for object files -- pretty useful!)
3. copies the duplicate objects into a new static library (usually called gstrsworkspace as its primary use is for the GStreamer ecosystem)
4. removes the duplicates from the rest of the libraries
5. updates the symbol table in each of the libraries with the bundled LLVM tools

Thanks to the ar crate, the unpacking and writing only happens at stage 3, ensuring no wasteful I/O slowdowns takes place. The llvm-tools-preview component in turn takes care of locating and calling up llvm-ar for updating the workspace's symbol tables.

A special mention is deserved to the object files' naming convention. Assume a Rust staticlib named libfoo, its object files will be named as:

• crate_name-hash1.crate_name.hash2-cgu.nnn.rcgu.o
• On Windows only: foo.crate_name-hash1.crate_name.hash2-cgu.nnn.rcgu.o
• On non-Windows platforms: same as above, but replacing foo with libfoo-hash

In all cases, crate_name means a dependency present somewhere in the workspace tree, and nnn is a number that will be bigger than zero whenever -C codegen-units was set to higher than 1.

For dragonfire purposes, dropping the library prefix is enough to be able to deduplicate object files; however, on Windows we can also find import library stubs, which LLVM can generate on its own by the use of the #[raw-dylib] annotation ^[2]. Import stubs can have any extension, e.g. .dll, .exe and .sys (the latter two coming from private Win32 APIs). These stubs cannot be deduplicated as they are generated individually per imported function, so dragonfire must preserve them where they are.

Drawbacks of object file deduplication

Again there are several disadvantages of this approach. On Apple platforms, deduplicating libraries triggers a strange linker error, which I've not seen before:

ld: multiple errors: compact unwind must have at least 1 fixup in '<framework>/GStreamer[arm64][1021](libgstrsworkspace_a-3f2b47962471807d-lse_ldset4_acq.o)'; r_symbolnum=-19 out of range in '<framework>/GStreamer[arm64][1022](libgstrsworkspace_a-compiler_builtins-350c23344d78cfbc.compiler_builtins.5e126dca1f5284a9-cgu.162.rcgu.o)'

This also led me to find that Rust libraries were packing bitcode, which is forbidden by Apple. (This was thankfully already fixed before shipping time, but we've not yet updated our Rust minimum version to take advantage of it.)

Another drawback is that Rust's implementation of LTO causes dead-code elimination at the crate level, as opposed to the workspace level. This makes object file deduplication impossible, as each copy is different.

For the Windows platform, there is an extra drawback which affects specifically object files produced by LLVM: the COMDAT sections are set to IMAGE_COMDAT_SELECT_NODUPLICATES. This means that the linker will outright reject functions with multiple definitions, rather than realise they're all duplicates and discarding all but one of the copies. MSVC in particular performs symbol resolution before dead-code elimination. This means that linking will fail because of unresolved symbols before dead code elimination kicks in; to use deduplicated libraries, one must set the linker flags /OPT:REF /FORCE:UNRESOLVED to ensure the dead code can be successfully eliminated.

Results

With library deduplication, we can make static libraries up to 44x smaller when building under MSVC ^[3] (you can expand the tables below for the full comparison):

• gstaws.lib: from 173M to 71M (~2.5x)
• gstrswebrtc.lib: from 193M to 66M (~2.9x)
• gstwebrtchttp.lib: from 66M to 1,5M (~ 44x)

The results from the melding above can be compared with the file sizes obtained using LTO on Windows ^[4] (remember it doesn't actually fix linking against plugins):

• gstaws.lib: from 71M (LTO) to 67M (melded) (-5.6%)
• gstrswebrtc.lib: from 105M to 66M (-37.1%)
• gstwebrtchttp.lib: from 28M to 1,5M (-94.6%)

Conclusion

This article presents several longstanding pain points in Rust, namely staticlib binary sizes, symbol leaking, and incompatibilities between Rust and MSVC. I demonstrate the tool dragonfire that aims to address and work around, where possible, these issues, along with remaining issues to be addressed.

As explained earlier, dragonfire treated libraries are live on all platforms except Apple's, if you use the development packages from mainline; it's on track hopefully for the 1.28 release of GStreamer. There's already a merge request pending to enable it for Apple platforms, we're only waiting to update the Rust mininum version.

If you want to have a look, dragonfire's source code is available at Freedesktop's GitLab instance. Please note that at the moment I have no plans to submit this to crates.io.

Feel free to contact me with any feedback, and thanks for reading!

1. See its default-https-client feature at lib.rs, you will find it throughout the AWS SDK ecosystem. ↩︎

2. https://doc.rust-lang.org/reference/items/external-blocks.html#dylib-versus-raw-dylib ↩︎

3. In all cases the -C flags are debug=1 + codegen-units=1 + opt-level=s; see this comment for the complete results across all platforms. ↩︎

4. Source: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1895 ↩︎

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• cea608overlay: improve handling of non-system memory
• cuda: Fix runtime kernel compile with CUDA 13.0
• d3d12: Fix crop meta support in converter and passthrough handling in deinterlacer
• fallbacksrc: source handling improvements; no-more-pads signal for streams-unaware parents
• inter: add properties to fine tune the inner elements
• qtdemux: surround sound channel layout handling fixes and performance improvements for GoPro videos
• rtp: Add linear audio (L8, L16, L24) RTP payloaders / depayloaders
• rtspsrc: Send RTSP keepalives in TCP/interleaved modes
• rtpamrpay2: frame quality indicator flag related fixes
• rtpbasepay2: reuse last PTS when possible, to work around problems with NVIDIA Jetson AV1 encoder
• mpegtsmux, tsdemux: Opus audio handling fixes
• threadshare: latency related improvements and many other fixes
• matroskamux, tsmux, flvmux, cea608mux: Best pad determination fixes at EOS
• unixfd: support buffers with a big payload
• videorate unknown buffer duration assertion failure with variable framerates
• editing services: Make GESTimeline respect `SELECT_ELEMENT_TRACK` signal discard decision; memory leak fixes
• gobject-introspection annotation fixes
• cerbero: Update meson to 1.9.0 to enable Xcode 26 compatibility
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.7 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

The GStreamer Conference team is pleased to announce that the full conference schedule including talk abstracts and speaker biographies is now available for this year's lineup of talks and speakers, covering again an exciting range of topics!

The GStreamer Conference 2025 will take place on 23-24 October 2025 in London, UK, followed by a hackfest.

Details about the conference, hackfest and how to register can be found on the conference website.

This year's topics and speakers:

• GStreamer State of the Union
  Tim-Philipp Müller, Centricular
• Region-Based Compression in GStreamer
  Axel Tobieson, Spiideo
• Rusty Pipes and Oxidized Wires
  Arun Raghavan, Valve Corporation
• The Quest for Low-Latency Desktop Audio
  Nirbheek Chauhan, Centricular
• Rebuilding Our Video Server Engine on GStreamer
  Ray Tiley, Tightrope Media Systems
• Enabling I-frame playlists with HLS CMAF
  Sanchayan Maity, Centricular
• From Streams to Insights: Advancing GstAnalytics
  Daniel Morin, Collabora
• Brief history of GStreamer adoption at Twilio
  Jeff Foster and Andrey Kovalenko, Twilio
• Threadshare, a plugin collection to increase scalability
  François Laignel, Centricular
• Costly Speech: an introduction
  Mathieu Duponchelle, Centricular
• GstWebRTC in WebKit, current status & plans
  Philippe Normand, Igalia
• Lessons Learned - Transforming Our Video Management System with GStreamer
  Bumi̇n Kaan Aydin, HAVELSAN
• VVC/H.266 Alpha Channel support in GStreamer
  Andoni Morales Alastruey, Fluendo
• librice: the TURNing point
  Matthew Waters, Centricular
• Virtual Hardware: Emulating a Video4linux Hardware Decoder
  Jan Schmidt, Centricular
• A Global Crisis => Global Leadership - WildCamera
  Will Robertson
• Tools to profile a video encoder
  Diego Nieto Munoz, Fluendo
• A new era for GStreamer C++ bindings
  Sebastian Dröge, Centricular
• Auxiliary Stream Wrangling in playbin3
  Jan Schmidt, Centricular
• VVC/H.266 in GStreamer
  Carlos Bentzen, Igalia
• GStreamer in the Medical Simulation Environment
  Thomas Goodwin and Jeff Wilson, Laerdal Labs
• What’s New in GStreamer D3D12
  Seungha Yang, Centricular
• Fluster news
  Rubén Gonzalez, Fluendo
• dcSCTP in GStreamer
  Tulio Beloqui, Pexip
• The road to Enhanced FLV and RTMP in GStreamer
  Taruntej Kanakamalla, Centricular
• Improving WebRTC datachannel performance
  Emil Ljungdahl, Axis Communications
• Cutting audio latency with bidirectional WebRTC
  Albert Sjölund, Axis Communications
• GStreamer in VR devices manufacturing
  Ivan Loskutov, Meta
• GStreamer for audio distribution at Sveriges Radio
  Karl Johannes Jondell and Christofer Bustad, Sveriges Radio
• GStreamer at Scale: Recent Lessons from Real-World Video Conferencing
  Håvard Graff, Pexip
• Gst.wasm season 3
  Jorge Zapata and Fabián Orccón, Fluendo
• Time Remapping and GES: Implementation Details and Latest Updates
  Thibault Saunier, Igalia
• PipeWire’s pipeline operation vs GStreamer’s explained
  George Kiagiadakis, Collabora
• Why Keep a Thread Running? Meet GstBaseIdleSrc
  Camilo Celis Guzman, Pexip
• Bringing AMD HIP into GStreamer
  Max Campbell, Veo Technologies; Seungha Yang, Centricular
• WirePlumber, present challenges and future directions
  Julian Bouzas, Collabora
• Hardware-Accelerated Live Broadcasting of Uncompressed ST 2110 Streams with GStreamer leveraging NVIDIA GPUs and NICs
  Johan Jino, Nvidia
• The Art of Debugging GStreamer Software
  Nicolas Dufresne, Collabora
• Making GStreamer Go!
  Wilhelm Bartel

Lightning Talks:

• Showing the invisible: Analysing buffer flow with tracers
  François Laignel, Centricular

• Video Frame Scheduler Plugin for Improving WebRTC Playback Quality
  Haejung Hwang, LG Electronics

• Audio source separation using snakes, crabs and torches
  Sebastian Dröge, Centricular
• soothe: a proposal for encoder testing
  Victor Manuel Jáquez Leal, Igalia
• burn: a little case study on using GstAnalytics from Rust
  Sebastian Dröge, Centricular
• GstVA and GStreamer-VAAPI updates
  Victor Manuel Jáquez Leal, Igalia
• Vulkan Video: pipeline update
  Stéphane Cerveau, Igalia
• Fallback Streaming for RTSP Server
  Axel Tobieson, Spiideo
• Fast and dirty: vibe coding use case plugging the NVIDIA kernel into GStreamer streams
  Jesús Luque, Mediapro
• What's up with Video4Linux support
  Nicolas Dufresne, Collabora
• pexLGPL bundle
  Tulio Beloqui, Pexip
• Reading v4l2 data from MR813 devices in a way that doesn't suck
  Vivia Nikolaidou
• Rewriting CoreAudio-based elements on macOS
  Piotr Brzeziński, Centricular
• fallbacksrc, now with more streams
  Piotr Brzeziński, Centricular
• State of MPEG-TS in GStreamer
  Edward Hervey, Centricular
• Full GPU driven AI workloads with GStreamer and Raven
  Andoni Morales Alastruey, Fluendo
• ...and more to come
• Submit your lightning talk now!

Many thanks to our amazing sponsors ‒ Platinum sponsors Centricular, Collabora, Igalia, and Pexip, Gold sponsors Axis Communications and Fluendo, and Silver sponsors MediaScribe and Veo, without whom the conference would not be possible in this form.

We hope to see you all in London soon! Don't forget to register as soon as possible if you're planning on joining us, so we can order enough food and drinks!

There's also a hackfest after the conference. If you're planning on joining us for the hackfest you must register separately for that, by 21 October 2025 at the latest.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• analytics GstTensorMeta handling changes (see release notes)
• closed caption combiner and transcriberbin stability fixes
• decklinkvideosrc: fix unrecoverable state after failing to start streaming because device is busy
• decodebin3 tag handling improvements
• fallbacksrc: Fix sources only being restarted once, as well as some deadlocks and race conditions on shutdown
• gtk4paintablesink: Try importing dmabufs withouth DMA_DRM caps
• hlsdemux2: Fix parsing of byterange and init map directives
• rtpmp4gdepay2: allow only constantduration with neither constantsize nor sizelength set
• spotifysrc: update to librespot 0.7 to make work after recent Spotify changes
• threadshare: new blocking adapter element for use in front of block elements such as sinks that sync to the clock
• threadshare: various other threadshare element fixes and improvements
• v4l2: Add support for WVC1 and WMV3
• videorate: possible performance improvements when operating in drop-only mode
• GstBaseParse fixes
• Vulkan video decoder fixes
• Fix gst-device-monitor-1.0 tool device-path regression on Windows
• Python bindings: Handle buffer PTS, DTS, duration, offset, and offset-end as unsigned long long (regression fix)
• Cerbero: Reduce recipe parallelism in various cases and dump cerbero and recipe versions into datadir during packaging
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.6 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

I been doing a lot of work with AI recently, both as part of a couple of projects I am part of at work, but I have also taken a personal interest in understanding the current state and what is possible. My favourite AI tool currently is Claude.ai. Anyway I have a Prusa Core One 3D printer now that I also love playing with and one thing I been wanting to do is to print some multicolor prints with it. So the Prusa Core One is a single extruder printer, which means it only has 1 filament loaded at any given time. Other printers on the market, like the PrusaXL has 5 extruders, so it can have 5 filaments or colors loaded at the same time.

Prusa Single Extruder Multimaterial setting

What I quickly discovered was that using this feature is non-trivial. First of all I had to manually add some G Code to the model to actually get it to ask me to switch filament for each color in my print, but the bigger issue is that the printer will ask you to change the color or filament, but you have no way of knowing which one to switch to, so for my model I had 15 filament changes and no simple way of knowing which order to switch in. So people where solving this among other things through looking through the print layer by layer and writing down the color changes, but I thought that this must be possible to automate with an application. So I opened Claude and started working on this thing I ended up calling Prusa Color Mate..

So the idea for the application was simple enough, have it analyze the project file, extract information about the order of color changes and display them for the user in a way that allows them to manually check of each color as its inserted. So I started off with doing a simple python script that would just print to the console. So it quickly turned out that the hard part of this project was to parse the input files and it was made worse by my ignorance. So what I learned the hard way is that if you store a project in Prusa Slicer it will use this format called 3mf. So my thought was, lets just analyze the 3mf file and extract the information I need. It took my quite a bit of back and forth with Claude, feeding claude source code from Prusa’s implementation and pdf files with specifications, but eventually the application did spit out a list of 15 toolchanges and the colors associated with them. So I happily tried to use it to print my model. I quickly discovered that the color ordering was all wrong. And after even more back and forth with Claude and reading online I realized that the 3mf file is a format for storing 3d models, but that is not what is being fed your 3d printer, instead for the printer the file provided is a bgcode file. And while the 3mf file did contain the information that you had to change filament 15 times, the information on in which order is simply not stored in the 3mf file as that is something chosen as part of composing your print. That print composition file is using a file format called bgcode. So I now had to extract the information from the bgcode file which took me basically a full day to figure out with the help of Claude. I could probably have gotten over the finish line sooner by making some better choices underway, but the extreme optimism of the AI probably lead me to believe it was going to be easier than it was to for instance just do everything in Python.
At first I tried using this libbgcode library written in C++, but I had a lot of issues getting Claude to incorporate it properly into my project, with Meson and CMAKE interaction issues (in retrospect I should have just made a quick RPM of libbgcode and used that). After a lot of struggles with this Claude thought that parsing the bgcode file in python natively would be easier than trying to use the C++ library, so I went down that route. I started by feeding Claude a description of the format that I found online and asked it to write me a parser for it. It didn’t work very well and I ended up having a lot of back and forth, testing and debugging, finding more documentation, including a blog post about this meatpack format used inside the file, but it still didn’t really work very well. In the end what probably helped the most was asking it to use the relevant files from libbgcode and Prusa Slicer as documentation, because even if that too took a lot of back and forth, eventually I had a working application that was able to extract the tool change data and associated colors from the file. I ended up using one external dependency which was the heatshrink2 library that I PIP installed, but while that worked correctly, it took a look time for me and Claude to figure out exactly what parameters to feed it to work with the Prusa generated file.

Screenshot of Prusa Color Mate

So know I had the working application going and was able to verify it with my first print. I even polished it up a little, by also adding detection of the manual filament change code, so that people who try to use the application will be made aware they need to add that through Prusa Slicer. Maybe I could bake that into the tool, but atm I got only bgcode decoders, not encoders, in my project.

Warning showed for missing G Code Dialog that gives detailed instructions for how to add G Code

So to conclude, it probably took me 2.5 days to write this application using Claude, it is a fairly niche tool, so I don’t expect a lot of users, but I made it to solve a problem for myself. If I had to write this pre-AI myself it would have taken me weeks, like figuring out the different formats and how library APIs worked etc. would have taken me a long time. So I am not an especially proficient coder, so a better coder than me could probably put this together quicker than I would, but I think this is part of what I think will change with AI, that even with limited time and technical skills you can put together simple applications like this to solve your own problems.

If you are a Prusa Core One user and would like to play with multicolor prints you can find Prusa Color Mate on Gitlab. I have not tested it on any other system or printer than my own, so I don’t even know if it will work with other non-Core One Prusa printers. There are rpms for Fedora you can download in the packaging directory of the gitlab repo, which also includes a RPM for the heatshrink2 library.

As for future plans for this application I don’t really have any. It solves my issue the way it is today, but if there turns out to be an interested user community out there maybe I will try to clean it up and create a proper flatpak for it.

The GStreamer team is pleased to announce another development release in the API/ABI-unstable 1.27 release series.

The API/ABI-unstable 1.27 release series is for testing and development purposes in the lead-up to the stable 1.28 series which is scheduled for release in late 2025. Any newly-added API can still change until that point.

This development release is primarily for developers and early adopters, and distros should probably not package it.

Highlighted changes:

• Add more 10bit RGB formats in GStreamer Video, OpenGL and Wayland, as well as in deinterlace and gdkpixbufoverlay
• analytics: new analytics combiner and splitter elements plus batch meta to batch buffers from one or more streams
• analyticsoverlay: Add expire-overlay property
• onnx: Add Verisilicon provider support
• awstranscriber2: add property for setting show_speaker_labels
• awstranslate: expose property for turning brevity on
• speechmatics: expose mask-profanities property
• textaccumulate: new element for speech synthesis or translation preprocessing
• tttocea608: expose speaker-prefix property
• cea708mux: expose "discarded-services" property on sink pads
• cuda crop meta support
• hlssink3, hlscmafsink: Support the use of a single media file
• s302mparse: Add new S302M audio parser
• webrtc: add WHEP client signaller; sdp and stats-related improvements
• threadshare: many improvements to the various elements, plus examples and a new benchmark program; relicense to MPL-2.0
• gtk4paintablesink: Add YCbCr memory texture formats and improve color-state fallbacks
• OpenGL: Add support for the NV24 pixel format; support changing caps and `get_gl_context()` in glbasesrc
• rtspsrc: Send RTSP keepalives also in TCP/interleaved modes
• nvencoder: interlaced video handling improvements
• vaav1enc: Enable intrablock copy and palette mode
• videopool: support parsing dma_drm caps
• Vulkan VP9 video decode support and many other video encode and decode improvements
• waylandsink: Parse and set the HDR10 metadata and other color management improvements
• LCEVC: Add autoplugging decoding support for LCEVC H265 and H266 video streams and LCEVC H.265 encoder
• GstMiniObject: Add missing `take()` and `steal()` functions and convert `is_writable()` and `make_writable()` macros to inline functions
• alsa: Improve PCM sink enumeration
• d3d12: various d3d12swapchainsink enhancements and bug fixes; fisheye image dewarping support
• wasapi2: add support for dynamic device switching, exclusive mode and format negotiation; device provider and latency enhancements
• windows: Disable all audio device providers except wasapi2
• dots-viewer: Improve dot file generation and interactivity
• gst-editing-services: Make framepositioner zorder controllable and expose it
• Various introspection fixes and bindings updates
• Cerbero: Update to Android API level 24; add config for number of cargo jobs; ship unixfd plugin
• Cerbero: Implement library melding for smaller binary sizes of Rust plugins

Binaries for Android, iOS, Mac OS X and Windows will be made available shortly at the usual location.

Release tarballs can be downloaded directly here:

• gstreamer-1.27.2.tar.xz
• gst-plugins-base-1.27.2.tar.xz
• gst-plugins-good-1.27.2.tar.xz
• gst-plugins-ugly-1.27.2.tar.xz
• gst-plugins-bad-1.27.2.tar.xz
• gst-libav-1.27.2.tar.xz
• gst-rtsp-server-1.27.2.tar.xz
• gst-python-1.27.2.tar.xz
• gst-editing-services-1.27.2.tar.xz
• gst-devtools-1.27.2.tar.xz,
• gstreamer-docs-1.27.2.tar.xz

As always, please give it a spin and let us know of any issues you run into by filing an issue in GitLab.

The GStreamer project is thrilled to announce that this year's GStreamer Conference is now confirmed to take place on Thursday and Friday 23-24 October 2025 in London, UK, followed by a 2-3 day hackfests.

You can find more details about the conference on the GStreamer Conference 2025 web site.

A Call for Presentations will be sent out shortly.

Registration is expected to open soon as well.

We will announce those and any further updates on the gstreamer-announce mailing list, the conference website, on Bluesky, on Mastodon, and on our Discourse forum.

Talk slots will be available in varying durations from 20 minutes up to 45 minutes. Whatever you're doing or planning to do with GStreamer, we'd like to hear from you!

We also plan to have sessions with short lightning talks / demos / showcase talks for those who just want to show what they've been working on or do a mini-talk instead of a full-length talk. Lightning talk slots will be allocated on a first-come-first-serve basis, so make sure to reserve your slot if you plan on giving a lightning talk.

A GStreamer hackfest will take place right after the conference.

We hope to see you in London!

Please spread the word!

Over the past few years, we've been slowly working on improving the platform-specific plugins for Windows, macOS, iOS, and Android, and making them work as well as the equivalent plugins on Linux. In this episode, we will look at audio device switching in the source and sink elements on macOS and Windows.

On Linux, if you're using the PulseAudio elements (both with the PulseAudio daemon and PipeWire), you get perfect device switching: quick, seamless, easy, and reliable. Simply set the device property whenever you want and you're off to the races. If the device gets unplugged, the pipeline will continue, and you will get notified of the unplug via the GST_MESSAGE_DEVICE_REMOVED bus message from GstDeviceMonitor so you can change the device.

As of a few weeks ago, the Windows Audio plugin wasapi2 implements the same behaviour. All you have to do is set the device property to whatever device you want (fetched using the GstDeviceMonitor API), at any time.

A merge request is open for adding the same feature to the macOS audio plugin, and is expected to be merged soon.

For graceful error handling, such as accidental device unplug or other unexpected errors, there's a new continue-on-error property. Setting that will cause the source to output silence after unplug, whereas the sink will simply discard the buffers. An element warning will be emitted to notify the app (alongside the GST_MESSAGE_DEVICE_REMOVED bus message if there was a hardware unplug), and the app can switch the device by setting the device property.

Thanks to Seungha and Piotr for working on this!

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• audioconvert: Fix caps negotiation regression when using a mix matrix
• aws: Add support for brevity in awstranslate and add option to partition speakers in the transcription output of awstranscriber2
• speechmatics speech-to-text: Expose mask-profanities property
• cea708mux: Add support for discarding select services on each input
• cea608overlay, cea708overlay: Accept GPU memory buffers if downstream supports the overlay composition meta
• d3d12screencapture source element and device provider fixes
• decodebin3: Don't error on an incoming ONVIF metadata stream
• uridecodebin3: Fix potential crash when adding URIs to messages, e.g. if no decoder is available
• v4l2: Fix memory leak for dynamic resolution change
• VA encoder fixes
• videorate, imagefreeze: Add support for JPEG XS
• Vulkan integration fixes
• wasapi2 audio device monitor improvements
• webrtc: Add WHEP client signaller and add whepclientsrc element on top of webrtcsrc using that
• threadshare: Many improvements and fixes to the generic threadshare and RTP threadshare elements
• rtpbin2 improvements and fixes
• gst-device-monitor-1.0 command line tool improvements
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.5 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

I have wanted to write this blog post for quite some time, but been unsure about the exact angle of it. I think I found that angle now where I will root the post in a very tangible concrete example.

So the reason I wanted to write this was because I do feel there is a palpable skepticism and negativity towards AI in the Linux community, and I understand that there are societal implications that worry us all, like how deep fakes have the potential to upend a lot of things from news disbursement to court proceedings. Or how malign forces can use AI to drive narratives in social media etc., is if social media wasn’t toxic enough as it is. But for open source developers like us in the Linux community there is also I think deep concerns about tooling that deeply incurs into something that close to the heart of our community, writing code and being skilled at writing code. I hear and share all those concerns, but at the same time having spent time the last weeks using Claude.ai I do feel it is not something we can afford not to engage with. So I know people have probably used a lot of different AI tools in the last year, some being more cute than useful others being somewhat useful and others being interesting improvements to your Google search for instance. I think I shared a lot of those impressions, but using Claude this last week has opened my eyes to what AI enginers are going to be capable of going forward.

So my initial test was writing a python application for internal use at Red Hat, basically connecting to a variety of sources and pulling data and putting together reports, typical management fare. How simple it was impressed me though, I think most of us having to deal with pulling data from a new source know how painful it can be, with issues ranging from missing, outdated or hard to parse API documentation. I think a lot of us also then spend a lot of time experimenting to figure out the right API calls to make in order to pull the data we need. Well Claude was able to give me python scripts that pulled that data right away, I still had to spend some time with it to fine tune the data being pulled and ensuring we pulled the right data, but I did it in a fraction of the time I would have spent figuring that stuff out on my own. The one data source Claude struggled with Fedora’s Bohdi, well once I pointed it to the URL with the latest documentation for that it figured out that it would be better to use the bohdi client library to pull data and once it had that figured out it was clear sailing.

So coming of pretty impressed by that experience I wanted to understand if Claude would be able to put together something programmatically more complex, like a GTK+ application using Vulkan. [Note: should have checked the code better, but thanks to the people who pointed this out. I told the AI to use Vulkan, which it did, but not in the way I expected, I expected it to render the globe using Vulkan, but it instead decided to ensure GTK used its Vulkan backend, an important lesson in both prompt engineering and checking the code afterwards).]So I thought what would be a good example of such an application and I also figured it would be fun if I found something really old and asked Claude to help me bring it into the current age. So I suddenly remembered xtraceroute, which is an old application orginally written in GTK1 and OpenGL showing your traceroute on a 3d Globe.

Screenshot of the original Xtraceroute application

I went looking for it and found that while it had been updated to GTK2 since last I looked at it, it had not been touched in 20 years. So I thought, this is a great testcase. So I grabbed the code and fed it into Claude, asking Claude to give me a modern GTK4 version of this application using Vulkan. Ok so how did it go? Well it ended up being an iterative effort, with a lot of back and forth between myself and Claude. One nice feature Claude has is that you can upload screenshots of your application and Claude will use it to help you debug. Thanks to that I got a long list of screenshots showing how this application evolved over the course of the day I spent on it.

This screenshot shows Claudes first attempt of transforming the 20 year old xtraceroute application into a modern one using GTK4, Vulkan and also adding a Meson build system. My prompt to create this was feeding in the old code and asking Claude to come up with a GTK4 and Vulkan equivalent. As you can see the GTK4 UI is very simple, but ok as it is. The rendered globe leaves something to be desired though. I assume the old code had some 2d fall backcode, so Claude latched onto that and focused on trying to use the Cairo API to recreate this application, despite me telling it I wanted a Vulkan application. What what we ended up with was a 2d circle that I could spin around like a wheel of fortuen. The code did have some Vulkan stuff, but defaulted to the Cairo code.

Second attempt at updating this application Anyway, I feed the screenshot of my first version back into Claude and said that the image was not a globe, it was missing the texture and the interaction model was more like a wheel of fortune. As you can see the second attempt did not fare any better, in fact we went from circle to square. This was also the point where I realized that I hadn’t uploaded the textures into Claude, so I had to tell it to load the earth.png from the local file repository.

Third attempt from Claude.Ok, so I feed my second screenshot back into Claude and pointed out that it was no globe, in fact it wasn’t even a circle and the texture was still missing. With me pointing out it needed to load the earth.png file from disk it came back with the texture loading. Well, I really wanted it to be a globe, so I said thank you for loading the texture, now do it on a globe.

This is the output of the 4th attempt. As you can see, it did bring back a circle, but the texture was gone again. At this point I also decided I didn’t want Claude to waste anymore time on the Cairo code, this was meant to be a proper 3d application. So I told Claude to drop all the Cairo code and instead focus on making a Vulkan application.

So now we finally had something that started looking like something, although it was still a circle, not a globe and it got that weird division of 4 thing on the globe. Anyway, I could see it using Vulkan now and it was loading the texture. So I was feeling like we where making some decent forward movement. So I wrote a longer prompt describing the globe I wanted and how I wanted to interact with it and this time Claude did come back with Vulkan code that rendered this as a globe, thus I didn’t end up screenshoting it unfortunately.

So with the working globe now in place, I wanted to bring in the day/night cycle from the original application. So I asked Claude to load the night texture and use it as an overlay to get that day/night effect. I also asked it to calculate the position of the sun to earth at the current time, so that it could overlay the texture in the right location. As you can see Claude did a decent job of it, although the colors was broken.

So I kept fighting with the color for a bit, Claude could see it was rendering it brown, but could not initally figure out why. I could tell the code was doing things mostly right so I also asked it to look at some other things, like I realized that when I tried to spin the globe it just twisted the texture. We got that fixed and also I got Claude to create some tests scripts that helped us figure out that the color issue was a RGB vs BRG issue, so as soon as we understood that then Claude was able to fix the code to render colors correctly. I also had a few iterations trying to get the scaling and mouse interaction behaving correctly.

So at this point I had probably worked on this for 4-5 hours, the globe was rendering nicely and I could interact with it using the mouse. Next step was adding the traceroute lines back. By default Claude had just put in code to render some small dots on the hop points, not draw the lines. Also the old method for getting the geocoordinates, but I asked Claude to help me find some current services which it did and once I picked one it on first try gave me code that was able to request the geolocation of the ip addresses it got back. To polish it up I also asked Claude to make sure we drew the lines following the globes curvature instead of just drawing straight lines.

Final version of the updated Xtraceroute application. It mostly works now, but I did realize why I always thought this was a fun idea, but less interesting in practice, you often don’t get very good traceroutes back, probably due to websites being cached or hosted globally. But I felt that I had proven that with a days work Claude was able to help me bring this old GTK application into the modern world.

Conclusions

So I am not going to argue that Xtraceroute is an important application that deserved to be saved, in fact while I feel the current version works and proves my point I also lost motivation to try to polish it up due to the limitations of tracerouting, but the code is available for anyone who finds it worthwhile.

But this wasn’t really about Xtraceroute, what I wanted to show here is how someone lacking C and Vulkan development skills can actually use a tool like Claude to put together a working application even one using more advanced stuff like Vulkan, which I know many more than me would feel daunting. I also found Claude really good at producing documentation and architecture documents for your application. It was also able to give me a working Meson build system and create all the desktop integration files for me, like the .desktop file, the metainfo file and so on. For the icons I ended up using Gemini as Claude do not do image generation at this point, although it was able to take a png file and create a SVG version of it (although not a perfect likeness to the original png).

Another thing I want to say is that the way I think about this, it is not that it makes coding skills less valuable, AIs can do amazing things, but you need to keep a close eye on them to ensure the code they create actually do what you want and that it does it in a sensible manner. For instance in my reporting application I wanted to embed a pdf file and Claude initial thought was to bring in webkit to do the rendering. That would have worked, but would have added a very big and complex dependency to my application, so I had to tell it that it could just use libpoppler to do it, something Claude agreed was a much better solution. The bigger the codebase the harder it also becomes for the AI to deal with it, but I think it hose circumstances what you can do is use the AI to give you sample code for the functionality you want in the programming language you want and then you can just work on incorporating that into your big application.

The other part here if course in terms of open source is how should contributors and projects deal with this? I know there are projects where AI generated CVEs or patches are drowning them and that helps nobody. But I think if we see AI as a developers tool and that the developer using the tool is responsible for the code generated, then I think that mindset can help us navigate this. So if you used an AI tool to create a patch for your favourite project, it is your responsibility to verify that patch before sending it in, and with that I don’t mean just verifying the functionality it provides, but that the code is clean and readable and following the coding standards of said upstream project. Maintainers on the other hand can use AI to help them review and evaluate patches quicker and thus this can be helpful on both sides of the equation. I also found Claude and other AI tools like Gemini pretty good at generating test cases for the code they make, so this is another area where open source patch contributions can improve, by improving test coverage for the code.

I do also believe there are many areas where projects can greatly benefit from AI, for instance in the GNOME project a constant challenge for extension developers have been keeping their extensions up-to-date, well I do believe a tool like Claude or Gemini should be able to update GNOME Shell extensions quite easily. So maybe having a service which tries to provide a patch each time there is a GNOME Shell update might be a great help there. At the same time having a AI take a look at updated extensions and giving an first review of the update might help reduce the load on people doing code reviews on extensions and help flag problematic extensions.

I know for a lot of cases and situations uploading your code to a webservice like Claude, Gemini or Copilot is not something you want or can do. I know privacy is a big concern for many people in the community. My team at Red Hat has been working on a code assistant tool using the IBM Granite model, called Granite.code. What makes Granite different is that it relies on having the model run locally on your own system, so you don’t send your code or data of somewhere else. This of course have great advantages in terms of improving privacy and security, but it has challenges too. The top end AI models out there at the moment, of which Claude is probably the best at the time of writing this blog post, are running on hardware with vast resources in terms of computing power and memory available. Most of us do not have those kind of capabilities available at home, so the model size and performance will be significantly lower. So at the moment if you are looking for a great open source tool to use with VS Code to do things like code completion I recommend giving Granite.code a look. If you on the other hand want to do something like I have described here you need to use something like Claude, Gemini or ChatGPT. I do recommend Claude, not just because I believe them to be the best at it at the moment, but they also are a company trying to hold themselves to high ethical standards. Over time we hope to work with IBM and others in the community to improve local models, and I am also sure local hardware will keep improving, so over time the experience you can get with a local model on your laptop at least has less of a gap than what it does today compared to the big cloud hosted models. There is also the middle of the road option that will become increasingly viable, where you have a powerful server in your home or at your workplace that can at least host a midsize model, and then you connect to that on your LAN. I know IBM is looking at that model for the next iteration of Granite models where you can choose from a wide variety of sizes, some small enough to be run on a laptop, others of a size where a strong workstation or small server can run them or of course the biggest models for people able to invest in top of the line hardware to run their AI.

Also the AI space is moving blazingly fast, if you are reading this 6 Months from now I am sure the capabilities of online and local models will have changed drastically already.

So to all my friends in the Linux community I ask you to take a look at AI and what it can do and then lets work together on improving it, not just in terms of capabilities, but trying to figure out things like societal challenges around it and sustainability concerns I also know a lot of us got.

Whats next for this code

As I mentioned I while I felt I got it to a point where I proved to myself it worked, I am not planning on working anymore on it. But I did make a cute little application for internal use that shows a spinning globe with all global Red Hat offices showing up as little red lights and where it pulls Red Hat news at the bottom. Not super useful either, but I was able to use Claude to refactor the globe rendering code from xtraceroute into this in just a few hours.

Red Hat Offices Globe and news.

I recently found, under the rain, next to a book swap box, a pile of 90's “software magazines” which I spent my evening cleaning, drying, and sorting in the days afterwards.

Magazine cover CDs with nary a magazine 

Those magazines are a peculiar thing in France, using the mechanism of “Commission paritaire des publications et des agences de presse” or “Commission paritaire” for short. This structure exists to assess whether a magazine can benefit from state subsidies for the written press (whether on paper at the time, and also the internet nowadays), which include a reduced VAT charge (2.1% instead of 20%), reduced postal rates, and tax exemptions.

In the 90s, this was used by Diamond Editions[1] (a publisher related to tech shop Pearl, which French and German computer enthusiasts probably know) to publish magazines with just enough original text to qualify for those subsidies, bundled with the really interesting part, a piece of software on CD.

If you were to visit a French newsagent nowadays, you would be able to find other examples of this: magazines bundled with music CDs, DVDs or Blu-rays, or even toys or collectibles. Some publishers (including the infamous and now shuttered Éditions Atlas) will even get you a cheap kickstart to a new collection, with the first few issues (and collectibles) available at very interesting prices of a couple of euros, before making that “magazine” subscription-only, with each issue being increasingly more expensive (article from a consumer protection association).

Other publishers have followed suite.

I guess you can only imagine how much your scale model would end up costing with that business model (50 eurocent for the first part, 4.99€ for the second), although I would expect them to have given up the idea of being categorised as “written press”.

To go back to Diamond Editions, this meant the eventual birth of 3 magazines: Presqu'Offert, BestSellerGames and StratéJ. I remember me or my dad buying a few of those, an older but legit and complete version of ClarisWorks, CorelDraw or a talkie version of a LucasArt point'n'click was certainly a more interesting proposition than a cut-down warez version full of viruses when budget was tight.

3 of the magazines I managed to rescue from the rain

You might also be interested in the UK “covertape wars”.

Don't stress the technique 

This brings us back to today and while the magazines are still waiting for scanning, I tried to get a wee bit organised and digitising the CDs.

Some of them will have printing that covers the whole of the CD, a fair few use the foil/aluminium backing of the CD as a blank surface, which will give you pretty bad results when scanning them with a flatbed scanner: the light source keeps moving with the sensor, and what you'll be scanning is the sensor's reflection on the CD.

My workaround for this is to use a digital camera (my phone's 24MP camera), with a white foam board behind it, so the blank parts appear more light grey. Of course, this means that you need to take the picture from an angle, and that the CD will appear as an oval instead of perfectly circular.

I tried for a while to use GIMP perspective tools, and “Multimedia” Mike Melanson's MobyCAIRO rotation and cropping tool. In the end, I settled on Darktable, which allowed me to do 4-point perspective deskewing, I just had to have those reference points.

So I came up with a simple "deskew" template, which you can print yourself, although you could probably achieve similar results with grid paper.

 

After opening your photo with Darktable, and selecting the “darkroom” tab, go to the “rotate and perspective tool”, select the “manually defined rectangle” structure, and adjust the rectangle to match the centers of the 4 deskewing targets. Then click on “horizontal/vertical fit”. This will give you a squished CD, don't worry, and select the “specific” lens model and voilà.

Tools at the ready

Targets acquired

You can now export the processed image (I usually use PNG to avoid data loss at each step), open things up in GIMP and use the ellipse selection tool to remove the background (don't forget the center hole), the rotate tool to make the writing straight, and the crop tool to crop it to size.

And we're done!

[1]: Full disclosure, I wrote a couple of articles for Linux Pratique and Linux Magazine France in the early 2000s, that were edited by that same company.

Somehow I internalized that my duty as software programmer was to silently work in a piece of code as if it were a magnum opus, until it’s finished, and then release it to the world with no need of explanations, because it should speak for itself. In other words, I tend to consider my work as a form of art, and myself as an artist. But I’m not. There’s no magnus opus and there will never be one. I’m rather a craftsman, in the sense of Richard Sennett: somebody who cares about their craft, making small, quick but thoughtful and clean changes, here and there, hoping that they will be useful to someone, now and in the future. And those little efforts need to be exposed openly, in spaces as this one and social media, as if I were a bazaar merchant.

This reflection invites me to add another task to my duties as software programmer: a periodical exposition of the work done. And this is the first intent to forge a (monthly) discipline in that direction, not in the sense of bragging, or looking to overprice a product (in the sense of commodity fetishism), but to build bridges with those that might find useful those pieces of software.

Let’s start.

GStreamer YUV4MPEG2 encoder and decoder #

We have been working lately on video encoding, and we wanted an easy way to test our work, using common samples such as those shared by the Derf’s collection. They are in a file format known as YUV4MPEG2, or more commonly known as y4m, because of their file name extension.

YUV4MPEG2 is a simple file format designed to hold uncompressed frames of YUV video, formatted as YCbCr 4:2:0, YCbCr 4:2:2 or YCbCr 4:4:4 data for the purpose of encoding. Instead of using raw YUV streams, where the frame size and color format have to be provided out-of-band, these metadata are embedded in the file.

There were already GStreamer elements for encoding and decoding y4m streams, but y4mdec was in gst-plugins-bad while y4menc in gst-plugins-good.

Our first task was to fix and improve y4menc [!8654], then move y4mdec to gst-plugins-good [!8719], but that implied to rewrite the element and add unit tests, while add more features such as handling more color formats.

Soothe — video encoders testing framework #

Heavily inspired by Fluster, a testing framework written in Python for decoder conformance, we are sketching Soothe, a script that aims to be a testing framework for video encoders, using VMAF, a perceptual video quality assessment algorithm.

GStreamer Vulkan H.264 encoder #

This is the reason of the efforts expressed above: vulkanh264enc, a H.264 encoder using the Vulkan Video extension [!7197].

One interesting side of this task was to propose a base class for hardware accelerated H.264 encoders, based on the vah264enc, the GStreamer VA-API H.264 encoder. We talked about this base class in the GStreamer Conference 2024.

Now the H.264 encoder merged and it will be part of the future release of GStreamer 1.28.

Removal of GStreamer-VAAPI subproject #

We’re very grateful with GStreamer-VAAPI. When its maintenance were handed over to us, after a few months we got the privilege to merge it as an official GStreamer subproject.

Now GStreamer-VAAPI functionality has been replaced with the VA plugin in gst-plugins-bad. Still, it isn’t a full featured replacement [#3947], but it’s complete and stable enough to be widely deployed. As Tim said in the GStreamer Conference 2024: it just works.

So, GStreamer-VAAPI subproject has been removed from main branch in git repository [!9200], and its Gitlab project, archived.

Vulkan Video Status page #

We believe that Vulkan Video extension will be one of the main APIs for video encoding, decoding and processing. Igalia participate in the Vulkan Video Technical Sub Group (TSG) and helps with the Conformance Test Suite (CTS).

Vulkan Video extension is big and constantly updated. In order to keep track of it we maintain a web page with the latest news about the specification, proprietary drivers, open source drivers and open source applications, along with articles and talks about it.

https://vulkan-video-status.igalia.com

GStreamer Planet #

Last but not least, GStreamer Planet has been updated and overhauled.

Given that the old Planet script, written in Python 2, is unmaintained, we worked on a new one in Rust: planet-rs. It internally uses tera for templates, feed-rs for feed parsing, and reqwest for HTTP handling. The planet is generated using Gitlab scheduled CI pipelines.

https://gstreamer.freedesktop.org/planet

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes as well, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• adaptivedemux2: Fixed reverse playback
• d3d12screencapture: Add support for monitor add/remove in device provider
• rtmp2src: various fixes to make it play back AWS medialive streams
• rtph265pay: add profile-id, tier-flag, and level-id to output rtp caps
• vp9parse: Fix handling of spatial SVC decoding
• vtenc: Fix negotiation failure with `profile=main-422-10`
• gtk4paintablesink: Add YCbCr memory texture formats and other improvements
• livekit: add room-timeout
• mp4mux: add TAI timestamp muxing support
• rtpbin2: fix various race conditions, plus other bug fixes and performance improvements
• threadshare: add a `ts-rtpdtmfsrc` element, implement run-time input switching in `ts-intersrc`
• webrtcsink: fix deadlock on error setting remote description and other fixes
• cerbero: WiX installer: fix missing props files in the MSI packages
• smaller macOS/iOS package sizes
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.4 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

HIP (formerly known as Heterogeneous-computing Interface for Portability) is AMD’s GPU programming API that enables portable, CUDA-like development across both AMD and NVIDIA platforms.

• On AMD GPUs, HIP runs natively via the ROCm stack.
• On NVIDIA GPUs, HIP operates as a thin translation layer over the CUDA runtime and driver APIs.

This allows developers to maintain a single codebase that can target multiple GPU vendors with minimal effort.

Where HIP Is Used

HIP has seen adoption in AMD-focused GPU computing workflows, particularly in environments that require CUDA-like programmability. Examples include:

• PyTorch ROCm backend for deep learning workloads
• Select scientific applications like LAMMPS and GROMACS have experimented with HIP backends for AMD GPU support
• GPU-accelerated media processing on systems that leverage AMD hardware

While HIP adoption has been more limited compared to CUDA, its reach continues to expand as support for AMD GPUs grows across a broader range of use cases.

The Challenge: Compile-Time Platform Lock-in

Despite its cross-vendor goal, HIP still has a fundamental constraint at the build level. As of HIP 6.3, HIP requires developers to statically define their target platform at compile time via macros like:

#define __HIP_PLATFORM_AMD__    // for AMD ROCm
#define __HIP_PLATFORM_NVIDIA__ // for CUDA backend

This leads to two key limitations:

• You must compile separate binaries for AMD and NVIDIA
• A single binary cannot support both platforms simultaneously
• HIP does not support runtime backend switching natively

GstHip’s Solution

To overcome this limitation, GstHip uses runtime backend dispatch through:

• dlopen() on Linux
• LoadLibrary() on Windows

Instead of statically linking against a single HIP backend, GstHip loads both the ROCm HIP runtime and the CUDA driver/runtime API at runtime. This makes it possible to:

• Detect available GPUs dynamically
• Choose the appropriate backend per device
• Even support simultaneous use of AMD and NVIDIA GPUs in the same process

Unified Wrapper API

GstHip provides a clean wrapper layer that abstracts backend-specific APIs via a consistent naming scheme:

hipError_t HipFooBar(GstHipVendor vendor, ...);

The Hip prefix (capital H) clearly distinguishes the wrapper from native hipFooBar(...) functions. The GstHipVendor enum indicates which backend to target:

• GST_HIP_VENDOR_AMD
• GST_HIP_VENDOR_NVIDIA

Internally, each HipFooBar(...) function dispatches to the correct backend by calling either:

• hipFooBar(...) for AMD ROCm
• cudaFooBar(...) for NVIDIA CUDA

These symbols are dynamically resolved via dlopen() / LoadLibrary(), enabling runtime backend selection without GPU vendor-specific builds.

Memory Interop

All memory interop in GstHip is handled through the hipupload and hipdownload elements. While zero-copy is not supported due to backend-specific resource management and ownership ambiguity, GstHip provides optimized memory transfers between systems:

• System Memory ↔ HIP Memory: Utilizes HIP pinned memory to achieve fast upload/download operations between host and device memory
• GstGL ↔ GstHip: Uses HIP resource interop APIs to perform GPU-to-GPU memory copies between OpenGL and HIP memory
• GstCUDA ↔ GstHip (on NVIDIA platforms): Since both sides use CUDA memory, direct GPU-to-GPU memory copies are performed using CUDA APIs.

GPU-Accelerated Filter Elements

GstHip includes GPU-accelerated filters optimized for real-time media processing:

• hipconvertscale/hipconvert/hipscale: Image format conversion and image scaling
• hipcompositor: composing multiple video streams into single video stream

These filters use the same unified dispatch system and are compatible with both AMD and NVIDIA platforms.

Application Integration Support

As of Merge Request!9340, GstHip exposes public APIs that allow applications to access HIP resources managed by GStreamer. This also enables applications to implement custom GstHIP-based plugins using the same underlying infrastructure without duplicating resource management.

Summary of GstHip Advantages

• Single plugin/library binary supports both AMD and NVIDIA GPUs
• Compatible with Linux and Windows
• Supports multi-GPU systems, including hybrid AMD + NVIDIA configurations
• Seamless memory interop with System Memory, GstGL, and GstCUDA
• Provides high-performance GPU filters for video processing
• Maintains a clean API layer via HipFooBar(...) wrappers, enabling backend-neutral development

The GStreamer team is pleased to announce the first development release in the API/ABI-unstable 1.27 release series.

The API/ABI-unstable 1.27 release series is for testing and development purposes in the lead-up to the stable 1.28 series which is scheduled for release in late 2025. Any newly-added API can still change until that point.

This development release is primarily for developers and early adopters, and distros should probably not package it.

Highlighted changes:

• Add AMD HIP plugin
• Add Vulkan H.264 encoder and add 10-bit support to Vulkan H.265 decoder
• Add LiteRT inference element
• Aggregator: expose current-level-* properties on sink pads
• Analytics: add general classifier tensor-decoder, facedetector, and more convenience API
• alsa: Support enumerating virtual PCM sinks
• d3d12: Add d3d12remap element
• Wayland: Add basic colorimetrie support
• Webkit: New wpe2 plugin making use of the "WPE Platform API"
• MPEG-TS demuxer: Add property to disable skew corrections
• qml6gloverlay: support directly passing a QQuickItem for QML the render tree
• unifxfdsink: Add a property to allow copying to make sink usable with more upstream elements
• videorate: Revive "new-pref" property for better control in case of caps changes
• wasapi2: Port to IMMDevice based device selection
• GstReferenceTimestampMeta can carry additional per-timestamp information now
• Added GstLogContext API that allows to fix log spam in several components
• New tracer hook to track when buffers are queued/dequeued in buffer pools
• gst-inspect-1.0: Prints type info for caps fields now
• Pipeline graph dot files now contain information about active tracers
• Python bindings: add Gst.Float wrapper, Gst.ValueArray.append_value(), analytics API improvements
• cerbero packages: ship vvdec and curl plugins; ship wasapi2 on MingW builds
• Countless bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

Binaries for Android, iOS, Mac OS X and Windows will be made available shortly at the usual location.

Release tarballs can be downloaded directly here:

• gstreamer-1.27.1.tar.xz
• gst-plugins-base-1.27.1.tar.xz
• gst-plugins-good-1.27.1.tar.xz
• gst-plugins-ugly-1.27.1.tar.xz
• gst-plugins-bad-1.27.1.tar.xz
• gst-libav-1.27.1.tar.xz
• gst-rtsp-server-1.27.1.tar.xz
• gst-python-1.27.1.tar.xz
• gst-editing-services-1.27.1.tar.xz
• gst-devtools-1.27.1.tar.xz,
• gstreamer-docs-1.27.1.tar.xz

As always, please give it a spin and let us know of any issues you run into by filing an issue in GitLab.

For a project recently it was necessary to collect video frames of multiple streams during a specific interval, and in the future also audio, to pass it through an inference framework for extracting additional metadata from the media and attaching it to the frames.

While GStreamer has gained quite a bit of infrastructure in the past years for machine learning use-cases in the analytics library, there was nothing for this specific use-case yet.

As part of solving this, I proposed as design for a generic interface that allows combining and batching multiple streams into a single one by using empty buffers with a GstMeta that contains the buffers of the original streams, and caps that include the caps of the original streams and allow format negotiation in the pipeline to work as usual.

While this covers my specific use case of combining multiple streams, it should be generic enough to also handle other cases that came up during the discussions.

In addition I wrote two new elements, analyticscombiner and analyticssplitter, that make use of this new API for combining and batching multiple streams in a generic, media-agnostic way over specific time intervals, and later splitting it out again into the original streams. The combiner can be configured to collect all media in the time interval, or only the first or last.

Conceptually the combiner element is similar to NVIDIA's DeepStream nvstreammux element, and in the future it should be possible to write a translation layer between the GStreamer analytics library and DeepStream.

The basic idea for the usage of these elements is to have a pipeline like

-- stream 1 --\                                                                  / -- stream 1 with metadata --
               -- analyticscombiner -- inference elements -- analyticssplitter --
-- stream 2 --/                                                                  \ -- stream 2 with metadata --
   ........                                                                           ......................
-- stream N -/                                                                     \- stream N with metadata --

The inference elements would only add additional metadata to each of the buffers, which can then be made use of further downstream in the pipeline for operations like overlays or blurring specific areas of the frames.

In the future there are likely going to be more batching elements for specific stream types, operating on multiple or a single stream, or making use of completely different batching strategies.

Special thanks also to Olivier and Daniel who provided very useful feedback during the review of the two merge requests.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes as well as a number of security fixes and important playback fixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• Various security fixes and playback fixes
• Security fix for the H.266 video parser
• Fix regression for WAV files with acid chunks
• Fix high memory consumption caused by a text handling regression in uridecodebin3 and playbin3
• Fix panic on late GOP in fragmented MP4 muxer
• Closed caption conversion, rendering and muxing improvements
• Decklink video sink preroll frame rendering and clock drift handling fixes
• MPEG-TS demuxing and muxing fixes
• MP4 muxer fixes for creating very large files with faststart support
• New thread-sharing 1:N inter source and sink elements, and a ts-rtpdtmfsrc
• New speech synthesis element around ElevenLabs API
• RTP H.265 depayloader fixes and improvements, as well as TWCC and GCC congestion control fixes
• Seeking improvements in DASH client for streams with gaps
• WebRTC sink and source fixes and enhancements, including to LiveKit and WHIP signallers
• The macOS osxvideosink now posts navigation messages
• QtQML6GL video sink input event handling improvements
• Overhaul detection of hardware-accelerated video codecs on Android
• Video4Linux capture source fixes and support for BT.2100 PQ and 1:4:5:3 colorimetry
• Vulkan buffer upload and memory handling regression fixes
• Python bindings: fix various regressions introduced in 1.26.2
• cerbero: fix text relocation issues on 32-bit Android and fix broken VisualStudio VC templates
• packages: ship pbtypes plugin and update openssl to 3.5.0 LTS
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.3 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

With GStreamer 1.26, a new D3D12 backend GstD3D12 public library was introduced in gst-plugins-bad.

Now, with the new gstreamer-d3d12 rust crate, Rust can finally access the Windows-native GPU feature written in GStreamer in a safe and idiomatic way.

What You Get with GStreamer D3D12 Support in Rust

• Pass D3D12 textures created by your Rust application directly into GStreamer pipelines without data copying
• Likewise, GStreamer-generated GPU resources (such as frames decoded by D3D12 decoders) can be accessed directly in your Rust app
• GstD3D12 base GStreamer element can be written in Rust

Beyond Pipelines: General D3D12 Utility Layer

GstD3D12 is not limited to multimedia pipelines. It also acts as a convenient D3D12 runtime utility, providing:

• GPU resource pooling such as command allocator and descriptor heap, to reduce overhead and improve reuse
• Abstractions for creating and recycling GPU textures with consistent lifetime tracking
• Command queue and fence management helpers, greatly simplifying GPU/CPU sync
• A foundation for building custom GPU workflows in Rust, with or without the full GStreamer pipeline

The GStreamer team is pleased to announce another bug fix release in the new old stable 1.24 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes as well as a number of security fixes and important playback fixes, and it should be safe to update from 1.24.x.

Please note that the 1.24 old-stable series is no longer actively maintained and has been superseded by the GStreamer 1.26 stable series now.

Highlighted bugfixes:

• Various security fixes and playback fixes
• MP4 demuxer atom parsing improvements and security fixes
• H.265 decoder base class and caption inserter SPS/PPS handling fixes
• Subtitle parser security fixes
• Subtitle rendering and seeking fixes
• Closed caption fixes
• Matroska rotation tag support and v4 muxing support
• Ogg seeking improvements in streaming mode
• Windows plugin loading fixes
• MIDI parser improvements for tempo changes
• Video time code support for 119.88 fps and drop-frames-related conversion fixes
• GStreamer editing services fixes for sources with non-1:1 aspect ratios
• RTP session handling and RTSP server fixes
• Thread-safety improvements for the Media Source Extension (MSE) library
• macOS video capture improvements for external devices
• Python bindings: Fix compatibility with PyGObject >= 3.52.0
• cerbero: bootstrapping fixes on Windows, improved support for RHEL, and openh264 recipe update
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.24.13 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes as well as a number of security fixes and important playback fixes, and it should be safe to update from 1.26.x.

Highlighted bugfixes:

• Various security fixes and playback fixes
• aggregator base class fixes to not produce buffers too early in live mode
• AWS translate element improvements
• D3D12 video decoder workarounds for crashes on NVIDIA cards on resolution changes
• dav1d AV1-decoder performance improvements
• fmp4mux: tfdt and composition time offset fixes, plus AC-3 / EAC-3 audio support
• GStreamer editing services fixes for sources with non-1:1 aspect ratios
• MIDI parser improvements for tempo changes
• MP4 demuxer atom parsing improvements and security fixes
• New skia-based video compositor element
• Subtitle parser security fixes
• Subtitle rendering and seeking fixes
• Playbin3 and uridecodebin3 stability fixes
• GstPlay stream selection improvements
• WAV playback regression fix
• GTK4 paintable sink colorimetry support and other improvements
• WebRTC: allow webrtcsrc to wait for a webrtcsink producer to initiate the connection
• WebRTC: new Janus Video Room WebRTC source element
• vah264enc profile decision making logic fixes
• Python bindings gained support for handling mini object writability (buffers, caps, etc.)
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.2 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

The release of GStreamer 1.26, last March, delivered new features, optimization and improvements. Igalia played its role as long standing contributor, with 382 commits (194 merge requests) from a total of 2666 of commits merged in this release.This blog post takes a closer look on those contributions.

gst-devtools #

This module contains development and validation tools.

gst-dot-viewer #

gst-dot-viewer is a new web tool for real-time pipeline visualization. Our colleague, Thibault, wrote a blog post about its usage.

validate #

GstValidate is a tool to check if elements are behaving as expected.

• Added support for HTTP Testing.
• Scenario fixes such as reset pipelines on expected errors to avoid inconsistent states, improved error logging, and async action handling to prevent busy loops.

gst-editing-services #

GStreamer Editing Services is a library to simplify the creation of multimedia editing applications.

• Enabled reverse playback, by adding a reverse property to nlesource for seamless backward clip playback.
• Added internal tests for Non-Linear Engine elements.

gst-libav #

GStreamer Libav plug-in contains a set of many popular decoders and encoders using FFmpeg.

• As part of the effort to support VVC/H.266 in GStreamer FFmpeg VVC/H.266 decoder was exposed.
• Optimized framerate renegotiation in avviddec without decoder resets.
• Mapped GST_VIDEO_FORMAT_GRAY10_LE16 format to FFmpeg’s equivalent.

gstreamer #

Core library.

• Added a tracer for gst-dots-viewer.
• Log tracers improvements such as, replaced integer codes with readable strings, to track pad’s sticky events, and simplify parameters handling, etc.
• On pads, don’t push sticky events in response to a FLUSH_STOP event.
• On queue element, fixed missing notify signals for level changes.
• Pipeline parser now logs bus error messages during pipeline construction.
• Fixed gst_util_ceil_log2 utility function.

gst-plugins-base #

GStreamer Base Plugins is a well-groomed and well-maintained collection of plugins. It also contains helper libraries and base classes useful for writing elements.

• audiorate: respect tolerance property to avoid unnecessary sample adjustments for minor gaps.
• audioconvert: support reordering of unpositioned input channels.
• videoconvertscale: improve aspect ratio handling.
• glcolorconvert: added I422_10XX, I422_12XX, Y444_10XX, and Y444_16XX color formats, and fixed caps negotiation for DMABuf.
• glvideomixer: handle mouse events.
• pbutils: added VVC/H.266 codec support
• encodebasebin: parser fixes.
• oggdemux: fixed seek to the end of files.
• rtp: fixed precision for UNIX timestamp.
• sdp: enhanced debugging messages.
• parsebin: improved caps negotiation.
• decodebin3: added missing locks to prevent race conditions.
• streamsynchronizer: improved documentation.

gst-plugins-good #

GStreamer Good Plugins is a set of plugins considered to have good quality code, correct functionality, and uses LGPL/LGPL+compatible licenses.

• hlsdemux2: handle empty segments at the beginning of a stream.
• qtmux and matroska: add support for VVC/H.266.
• matroskademux:support seek with stop in push mode.
• rtp: several fixes.
• osxaudio: fixes.
• videoflip: support Y444_16LE and Y444_16BE color formats.
• vpx: enhance error and warning messages.

gst-plugins-bad #

GStreamer Bad Plug+ins is a set of plugins that aren’t up to par compared to the rest. They might be close to being good quality, but they’re missing something, be it a good code review, some documentation, a set of tests, etc.

• dashsink: a lot of improvements and cleanups, such as unit tests, state and event management.
• h266parse: enabled vvc1 and vvi1 stream formats, improved code data parsing and negotiatios, along with cleanups and fixes.
• mpegtsmux and tsdemux: added support for VVC/H.266 codec.
• vulkan:
  • Added compatibility for timeline semaphores and barriers.
  • Initial support of multiple GPU and dynamic element registering.
  • Vulkan image buffer pool improvements.
  • vulkanh264dec: support interlaced streams.
  • vulkanencoding: rate control and quality level adjustments, update SPS/PPS, support layered DPBs.
• webrtcbin:
  • Resolved duplicate payload types in SDP offers with RTX and multiple codecs.
  • Transceivers are now created earlier during negotiation to avoid linkage issues.
  • Allow session level in setup attribute in SDP answer.
• wpevideosrc:
  • code cleanups
  • cached SHM buffers are cleared after caps renegotiation.
  • handle latency queries and post progress messages on bus.
• srtdec: fixes
• jpegparse: handle avi1 tag for progressive images
• va: improve encoders configuration when properties change in run+time, specially rate control.

As part of the GStreamer Hackfest in Nice, France I had some time to go through some outstanding GStreamer issues. One such issue that has been on my mind was this GStreamer OpenGL Wayland issue.

Now, the issue is that OpenGL is an old API and did not have some of the platform extensions it does today. As a result, most windowing system APIs allow creating an output surface (or a window) but never showing it. This also works just fine when you are creating an OpenGL context but not actually rendering anything to the screen and this approach is what is used by all of the other major OpenGL platforms (Windows, macOS, X11, etc) supported by GStreamer.

When wayland initially arrived, this was not the case. A wayland surface could be the back buffer (an OpenGL term for rendering to a surface) but could not be hidden. This is very different from how other windowing APIs worked at the time. As a result, the initial implementation using Wayland within GStreamer OpenGL used some heuristics for determining when a wayland surface would be created and used that basically boiled down to, if there is no shared OpenGL context, then create a window.

This heuristic obviously breaks in multiple different ways, the two most obvious being:

1. gltestsrc ! gldownload ! some-non-gl-sink - there should be no surface used here.
2. gltestsrc ! glimagesink gltestsrc ! glimagesink - there should be two output surfaces used here.

The good news is that issue is now fixed by adding some API that glimagesink can use to notify that it would like an output surface. This has been implemented in this merge request and will be part of GStreamer 1.28.

We’re happy to have released gst-dots-viewer, a new development tool that makes it easier to visualize and debug GStreamer pipelines. This tool,  included in GStreamer 1.26, provides a web-based interface for viewing pipeline graphs in real-time as your application runs and allows to easily request all pipelines to be dumped at any time.

What is gst-dots-viewer?

gst-dots-viewer is a server application that monitors a directory for .dot files generated by GStreamer’s pipeline visualization system and displays them in your web browser. It automatically updates the visualization whenever new .dot files are created, making it simpler to debug complex applications and understand the evolution of the pipelines at runtime.

Key Features

• Real-time Updates: Watch your pipelines evolve as your application runs
• Interactive Visualization:
  • Click nodes to highlight pipeline elements
  • Use Shift-Ctrl-scroll or w/s keys to zoom
  • Drag-scroll support for easy navigation
• Easily deployable in cloud based environments

How to Use It

1. Start the viewer server:

   gst-dots-viewer
   

2. Open your browser at http://localhost:3000
3. Enable the dots tracer in your GStreamer application:

   GST_TRACERS=dots your-gstreamer-application
   

The web page will automatically update whenever new pipeline are dumped, and you will be able to dump all pipelines from the web page.

New Dots Tracer

As part of this release, we’ve also introduced a new dots tracer that replaces the previous manual approach to specify where to dump pipelines. The tracer can be activated simply by setting the GST_TRACERS=dots environment variable.

Interactive Pipeline Dumps

The dots tracer integrates with the pipeline-snapshot tracer to provide real-time pipeline visualization control. Through a WebSocket connection, the web interface allows you to trigger pipeline dumps. This means you can dump pipelines exactly when you need them during debugging or development, from your browser.

Future Improvements

We plan on adding more feature and  have this list of possibilities:

• Additional interactive features in the web interface
• Enhanced visualization options
• Integration with more GStreamer tracers to provide comprehensive debugging information. For example, we could integrate the newly released memory-tracer and queue-level tracers so to plot graphs about memory usage at any time.

This could transform gst-dots-viewer into a more complete debugging and monitoring dashboard for GStreamer applications.

Demo

The GStreamer team is pleased to announce another bug fix release in the new stable 1.26 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and security fixes, and it should be safe to update from 1.26.0.

Highlighted bugfixes:

• awstranslate and speechmatics plugin improvements
• decodebin3 fixes and urisourcebin/playbin3 stability improvements
• Closed captions: CEA-708 generation and muxing fixes, and H.264/H.265 caption extractor fixes
• dav1d AV1 decoder: RGB support, plus colorimetry, renegotiation and buffer pool handling fixes
• Fix regression when rendering VP9 with alpha
• H.265 decoder base class and caption inserter SPS/PPS handling fixes
• hlssink3 and hlsmultivariantsink feature enhancements
• Matroska v4 support in muxer, seeking fixes in demuxer
• macOS: framerate guessing for cameras or capture devices where the OS reports silly framerates
• MP4 demuxer uncompressed video handling improvements and sample table handling fixes
• oggdemux: seeking improvements in streaming mode
• unixfdsrc: fix gst_memory_resize warnings
• Plugin loader fixes, especially for Windows
• QML6 GL source renegotiation fixes
• RTP and RTSP stability fixes
• Thread-safety improvements for the Media Source Extension (MSE) library
• v4l2videodec: fix A/V sync issues after decoding errors
• Various improvements and fixes for the fragmented and non-fragmented MP4 muxers
• Video encoder base class segment and buffer timestamp handling fixes
• Video time code support for 119.88 fps and drop-frames-related conversion fixes
• WebRTC: Retransmission entry creation fixes and better audio level header extension compatibility
• YUV4MPEG encoder improvments
• dots-viewer: make work locally without network access
• gst-python: fix compatibility with PyGObject >= 3.52.0
• Cerbero: recipe updates, compatibility fixes for Python < 3.10; Windows Android cross-build improvements
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.26.1 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

We are excited about the Fedora Workstation 42 released today. Having worked on some great features for it.

Fedora Workstation 42 HDR edition
I would say that the main feature that landed was HDR or High Dynamic Range. It is a feature we spent years on with many team members involved and a lot of collaboration with various members of the wider community.

GNOME Settings menu showing HDR settings

HDR setup in Ori and Will of the Wisps

One area that should benefit from HDR support are games. In the screenshot about you see the game Ori and the Will of the Wisps which is known for great HDR support. Valve will need to update to a Wine version for Proton that supports Wayland natively though before this just works, at the moment you can get it working using gamescope, but hopefully soon it will just work under both Mutter and Kwin.

Also a special shoutout to the MPV community for quickly jumping on this and releasing a HDR capable video player recently.

MPV video player playing HDR content

Of course getting Fedora Workstation 42 to out with these features is just the beginning, with the baseline support it now is really the time when application maintainers have a real chance of starting to make use of these features, so I would expect various content creative applications for instance to start having support over the next year.

For the desktop itself there are also open questions we need to decide on like:

• Format to use for HDR screenshots
• Better backlight and brightness handling
• Better offloading
• HDR screen recording video format
• How to handle HDR webcams (seems a lot of them are not really capable of producing HDR output).
• Version of the binary NVIDIA driver released supporting the VK_EXT_hdr_metadata and VK_COLOR_SPACE_HDR10_ST2084_EXT Vulkan extension on Linux
• A million smaller issues we will need to iron out

Accessibility
Our accessibility team has been hard at work trying to ensure we have a great accessibility story in Fedora Workstation 42. Our accessibility team with Lukas Tyrychtr and Bohdan Milar has been working hard together with others to ensure that Fedora Workstation 42 has the best accessibility support you can get on Linux. One major effort that landed was the new keyboard monitoring interface which is critical for making Orca work well under Wayland. This was a collaboration of between Lukas Tyrychtr, Matthias Clasen and Carlos Garnacho on our team. If you are interested in Accessibility, as a user or a developer or both then make sure to join in by reaching out to the Accessibility Working group

PipeWire
PipeWire also keeps going strong with continuous improvements and bugfixes. Thanks to the great work by Jan Grulich the support for PipeWire in Firefox and Chrome is now working great, including for camera handling. It is an area where we want to do an even better job though, so Wim Taymans is currently looking at improving video handling to ensure we are using the best possible video stream the camera can provide and handle conversion between formats transparently. He is currently testing it out using a ffmpeg software backend, but the end goal is to have it all hardware accelerated through directly using Vulkan.

Another feature Wim Taymans added recently is MIDI2 support. This is the next generation of MIDI with only a limited set of hardware currently supporting it, but on the other hand it feels good that we are now able to be ahead of the curve instead of years behind thanks to the solid foundation we built with Pipewire.

Wayland
For a long time the team has been focused on making sure Wayland has all the critical pieces and was functionality wise on the same level as X11. For instance we spent a lot of time and effort on ensuring proper remote desktop support. That work all landed in the previous Fedora release which means that over the last 6 Months the team has had more time to look at things like various proposed Wayland protocols and get them supported in GNOME. Thanks to that we helped ensure the Cursor Shape Protocol and Toplevel Drag protocols got landed in time for this release. We are already looking and what to help land for the next release, so expect a continued acceleration in Wayland protocol adoption going forward.

First steps into AI
So an effort we been plugging away at recently is starting to bring AI tooling to Open Source desktop applications. Our first effort in this regard is Granite.code. Granite.code is a extension for Visual Studio Code that sets up a local AI engine on your system to help with various tasks including code generation and chat inside Visual Studio Code. So what is special about this effort is that it relies on downloading and running a copy of the open source AI Granite LLM model to your system instead on relying on it being run in a cloud instance somewhere. That means you can use Granite.code without having to share your data and work with someone else. Granite.code is still very early stage and it requires a NVIDIA or AMD GPU with over 8GB of video ram to use under Linux. (It also runs under Windows and MacOS X). It is still in a pre-release stage, we are waiting for the Granite 3.3 model update to enable some major features for us before we make the first formal release, but for those willing to help us test you can search for Granite in the Visual Studio Code extension marketplace and install it.
We are hoping though that this will just the starting point where our work can get picked up and used by other IDEs out there too and also we are thinking about how we can offer AI features in other parts of the desktop too.

Granite.code running on Linux

The GStreamer project is thrilled to announce that there will be a spring hackfest on Friday-Sunday 16-18 May 2025 in Nice, France.

For more details and latest updates check out the announcement on Discourse.

We will announce any further updates on Discourse, but you can also follow us on Bluesky and on on Mastodon.

We hope to see you in Nice!

Please spread the word!

Recently, GStreamer development story integrated the usage of pre-commit. pre-commit is a Git hook script that chain different linters, checkers, validators, formatters, etc., that are executed at git commit. This script is in Python. And there’s other GStreamer utility in Python: hotdoc

The challenge is that Debian doesn’t allow to install Python packages through pip, they have to be installed as Debian packages or inside virtual environments, such as venv.

So, instead of activating a virtual environment when I work in GStreamer, let’s just use direnv to activate it automatically.

Here’s a screencast of what I did to setup a Python virtual environment, within direnv, and installing pre-commit, hotdoc and gst-indent-1.0.

UPDATE: Tim told me that wit pipx we can do the same without the venv hassle.

https://mastodon.social/@tp_muller@fosstodon.org/114150178786863565

The GStreamer team is excited to announce a new major feature release of your favourite cross-platform multimedia framework!

As always, this release is again packed with new features, bug fixes and many other improvements.

The 1.26 release series adds new features on top of the previous 1.24 series and is part of the API and ABI-stable 1.x release series of the GStreamer multimedia framework.

Highlights:

• H.266 Versatile Video Coding (VVC) codec support
• Low Complexity Enhancement Video Coding (LCEVC) support
• Closed captions: H.264/H.265 extractor/inserter, cea708overlay, cea708mux, tttocea708 and more
• New hlscmafsink, hlssink3, and hlsmultivariantsink; HLS/DASH client and dashsink improvements
• New AWS and Speechmatics transcription, translation and TTS services elements, plus translationbin
• Splitmux lazy loading and dynamic fragment addition support
• Matroska: H.266 video and rotation tag support, defined latency muxing
• MPEG-TS: support for H.266, JPEG XS, AV1, VP9 codecs and SMPTE ST-2038 and ID3 meta; mpegtslivesrc
• ISO MP4: support for H.266, Hap, Lagarith lossless codecs; raw video support; rotation tags
• SMPTE 2038 ancillary data streams support
• JPEG XS image codec support
• Analytics: New TensorMeta; N-to-N relationships; Mtd to carry segmentation masks
• ONVIF metadata extractor and conversion to/from relation metas
• New originalbuffer element that can restore buffers again after transformation steps for analytics
• Improved Python bindings for analytics API
• Lots of Vulkan integration and Vulkan Video decoder/encoder improvements
• OpenGL integration improvements, esp. in glcolorconvert, gldownload, glupload
• Qt5/Qt6 QML GL sinks now support direct DMABuf import from hardware decoders
• CUDA: New compositor, Jetson NVMM memory support, stream-ordered allocator
• NVCODEC AV1 video encoder element, and nvdsdewarp
• New Direct3D12 integration support library
• New d3d12swapchainsink and d3d12deinterlace elements and D3D12 sink/source for zero-copy IPC
• Decklink HDR support (PQ + HLG) and frame scheduling enhancements
• AJA capture source clock handling and signal loss recovery improvements
• RTP and RTSP: New rtpbin sync modes, client-side MIKEY support in rtspsrc
• New Rust rtpbin2, rtprecv, rtpsend, and many new Rust RTP payloaders and depayloaders
• webrtcbin support for basic rollbacks and other improvements
• webrtcsink: support for more encoders, SDP munging, and a built-in web/signalling server
• webrtcsrc/sink: support for uncompressed audio/video and NTP & PTP clock signalling and synchronization
• rtmp2: server authentication improvements incl. Limelight CDN (llnw) authentication
• New Microsoft WebView2 based web browser source element
• The GTK3 plugin has gained support for OpenGL/WGL on Windows
• Many GTK4 paintable sink improvements
• GstPlay: id-based stream selection and message API improvements
• Real-time pipeline visualization in a browser using a new dots tracer and viewer
• New tracers for tracking memory usage, pad push timings, and buffer flow as pcap files
• VA hardware-acclerated H.266/VVC decoder, VP8 and JPEG encoders, VP9/VP8 alpha decodebins
• Video4Linux2 elements support DMA_DRM caps negotiation now
• V4L2 stateless decoders implement inter-frame resolution changes for AV1 and VP9
• Editing services: support for reverse playback and audio channel reordering
• New QUIC-based elements for working with raw QUIC streams, RTP-over-QUIC (RoQ) and WebTransport
• Apple AAC audio encoder and multi-channel support for the Apple audio decoders
• cerbero: Python bindings and introspection support; improved Windows installer based on WiX5
• Lots of new plugins, features, performance improvements and bug fixes

For more details check out the GStreamer 1.26 release notes.

Binaries for Android, iOS, macOS and Windows will be provided in due course.

You can download release tarballs directly here: gstreamer, gst-plugins-base, gst-plugins-good, gst-plugins-ugly, gst-plugins-bad, gst-libav, gst-rtsp-server, gst-python, gst-editing-services, gst-devtools, gstreamer-vaapi, gstreamer-sharp, gstreamer-docs.

UK calling

Long time no see this beautiful grey sky, roast beef on sunday and large but full packed pub when there is a football or a rugby game (The rose team has been lucky this year, grrr).

It was a delightful journey in the UK starting with my family visiting London including a lot (yes a lot…) of sightviews in a very short amount of time. But we managed to fit everything in. We saw the changing of the guards, the Thames river tide on a boat, Harry Potter gift shops and the beautiful Arsenal stadium with its legendary pitch, one of the best of England.

It was our last attraction in London and now it was time for my family to go to Standsted back home and me to Cambridge and its legendary university.

To start the journey in Cambridge, first I got some rest on Monday in the hotel to face the hail of information I will get during the conference. This year, Vulkanised took place on Arm’s campus, who kindly hosted the event, providing everything we needed to feel at home and comfortable.

The first day, we started with an introduction from Ralph Potter, the Vulkan Working Group Chair at Khronos, who introduced the new 1.4 release and all the extensions coming along including “Vulkan Video”. Then we could start this conference with my favorite topic, decoding video content with Vulkan Video. And the game was on! There was a presentation every 30 minutes including a neat one from my colleague at Igalia Ricardo Garcia about Device-Generated Commands in Vulkan and a break every 3 presentations. It took a lot of mental energy to keep up with all the topics as each presentation was more interesting than the last. During the break, we had time to relax with good coffee, delicious cookies, and nice conversations.

The first day ended up with a tooling demonstrations from LunarG, helping us all to understand and tame the Vulkan beast. The beast is ours now!

As I was not in the best shape due to a bug I caught on Sunday, I decided to play it safe and went to the hotel just after a nice indian meal. I had to prepare myself for the next day, where I would present “Vulkan Video is Open: Application Showcase”.

Vulkan Video is Open: Application showcase !

First Srinath Kumarapuram from Nvidia gave a presentation about the new extensions made available during 2024 by the Vulkan Video TSG. It started with a brief timeline of the video extensions from the initial h26x decoding to the latest VP9 decode coming this year including the 2024 extensions such as the AV1 codec. Then he presented more specific extensions such as VK_KHR_video_encode_quantization_map, VK_KHR_video_maintenance2 released during 2024 and coming in 2025, VK_KHR_video_encode_intra_refresh. He mentioned that the Vulkan toolbox now completely supports Vulkan Video, including the Validation Layers, Vulkan Profiles, vulkaninfo or GFXReconstruct.

After some deserved applause for a neat presentation, it was my time to be on stage.

During this presentation I focused on the Open source ecosystem around Vulkan Video. Indeed Vulkan Video ships with a sample app which is totally open along with the regular Conformance Test Suite. But that’s not all! Two major frameworks now ship with Vulkan Video support: GStreamer and FFmpeg.

Before this, I started by talking about Mesa, the open graphics library. This library which is totally open provides drivers which support Vulkan Video extensions and allow applications to run Vulkan Video decode or encode. The 3 major chip vendors are now supported. It started in 2022 with RADV, a userspace driver that implements the Vulkan API on most modern AMD GPUs. This driver supports all the vulkan video extensions except the lastest ones such as VK_KHR_video_encode_quantization_map or VK_KHR_video_maintenance2 but this they should be implemented sometime in 2025. Intel GPUs are now supported with the ANV driver, this driver also supports the common video extensions such as h264/5 and AV1 codec. The last driver to gain support was at the end of 2024 where several of the Vulkan Video extensions were introduced to NVK, a Vulkan driver for NVIDIA GPUs. This driver is still experimental but it’s possible to decode H264 and H265 content as well as its proprietary version. This completes the offering of the main GPUs on the market.

Then I moved to the applications including GStreamer, FFmpeg and Vulkan-Video-Samples. In addition to the extensions supported in 2025, we talked mainly about the decode conformance using Fluster. To compare all the implementations, including the driver, the version and the framework, a spreadsheet can be found here. In this spreadsheet we summarize the 3 supported codecs (H264, H265 and AV1) with their associated test suites and compare their implemententations using Vulkan Video (or not, see results for VAAPI with GStreamer). GStreamer, my favorite playground, can now decode H264 and H265 since 1.24 and recently got the support for AV1 but the merge request is still under review. It supports more than 80% of the H264 test vectors for the JVT-AVC_V1 and 85% of the H265 test vectors in JCT-VC-HEVC_V1. FFMpeg is offering better figures passing 90% of the tests. It supports all the avaliable codecs including all of the encoders as well. And finally Vulkan-Video-Samples is the app that you want to use to support all codecs for both encode and decode, but its currently missing support for mesa drivers when it comes to use Fluster decode tests..

Vulkanised on the 3rd day

During the 3rd day, we had interesting talks as well demonstrating the power of Vulkan, from Blender, a free and open-source 3D computer graphics software tool switching progressively to Vulkan, to the implementation of 3D a game engine using Rust, or compute shaders in Astronomy. My other colleague at Igalia, Lucas Fryzek, also had a presentation on Mesa with Lavapipe: a Mesa’s Software Renderer for Vulkan which allows you to have a hardware free implementation of Vulkan and to validate extensions in a simpler way. Finally, we finished this prolific and dense conference with Android and its close collaboration with Vulkan.

If you are interested in 3D graphics, I encourage you to attend future Vulkanised editions, which are full of passionate people. And if you can not attend you can still watch the presentation online.

If you are interested in the Vulkan Video presentation I gave, you can catch up the video here:

Or follow our Igalia live blog post on Vulkan Video:

https://blogs.igalia.com/vjaquez/vulkan-video-status/

As usual, if you would like to learn more about Vulkan, GStreamer or any other open multimedia framework, please feel free to contact us!

The GStreamer team is excited to announce the first release candidate for the upcoming stable 1.26.0 feature release.

This 1.25.90 pre-release is for testing and development purposes in the lead-up to the stable 1.26 series which is now frozen for commits and scheduled for release very soon.

Depending on how things go there might be more release candidates in the next couple of days, but in any case we're aiming to get 1.26.0 out as soon as possible.

Binaries for Android, iOS, Mac OS X and Windows will be made available shortly at the usual location.

Release tarballs can be downloaded directly here:

• gstreamer-1.25.90.tar.xz
• gst-plugins-base-1.25.90.tar.xz
• gst-plugins-good-1.25.90.tar.xz
• gst-plugins-ugly-1.25.90.tar.xz
• gst-plugins-bad-1.25.90.tar.xz
• gst-libav-1.25.90.tar.xz
• gst-rtsp-server-1.25.90.tar.xz
• gst-python-1.25.90.tar.xz
• gst-editing-services-1.25.90.tar.xz
• gst-devtools-1.25.90.tar.xz,
• gstreamer-vaapi-1.25.90.tar.xz
• gst-omx-1.25.90.tar.xz
• gstreamer-docs-1.25.90.tar.xz

As always, please give it a spin and let us know of any issues you run into by filing an issue in GitLab.

The GStreamer team is pleased to announce another release of liborc, the Optimized Inner Loop Runtime Compiler, which is used for SIMD acceleration in GStreamer plugins such as audioconvert, audiomixer, compositor, videoscale, and videoconvert, to name just a few.

This is a bug-fix release.

Highlights:

• orccodemem: Don't modify the process umask, which caused race conditions with other threads
• Require glibc >= 2.07
• x86: various SSE and MMX fixes
• avx: Fix sqrtps encoding causing an illegal instruction crash
• Hide internal symbols from ABI and do not install internal headers
• Rename backend to target, including `orc-backend` meson option and `ORC_BACKEND` environment variable
• Testsuite, tools: Disambiguate OrcProgram naming conventions
• Build: Fix `_clear_cache` call for Clang and error out on implicit function declarations
• opcodes: Use MIN instead of CLAMP for known unsigned values to fix compiler warnings
• ci improvements: Upload the generated .S and .bin and include Windows artifacts
• Spelling fix in debug log message

Direct tarball download: orc-0.4.41.tar.xz.

The GStreamer team is pleased to announce another development release in the unstable 1.25 release series.

The unstable 1.25 release series is for testing and development purposes in the lead-up to the stable 1.26 series which is scheduled for release ASAP. Any newly-added API can still change until that point.

This development release is primarily for developers and early adopters.

The plan is to get 1.26 out of the door as quickly as possible, and with this release a feature freeze is now in effect.

Binaries for Android, iOS, Mac OS X and Windows will be made available shortly at the usual location.

Release tarballs can be downloaded directly here:

• gstreamer-1.25.50.tar.xz
• gst-plugins-base-1.25.50.tar.xz
• gst-plugins-good-1.25.50.tar.xz
• gst-plugins-ugly-1.25.50.tar.xz
• gst-plugins-bad-1.25.50.tar.xz
• gst-libav-1.25.50.tar.xz
• gst-rtsp-server-1.25.50.tar.xz
• gst-python-1.25.50.tar.xz
• gst-editing-services-1.25.50.tar.xz
• gst-devtools-1.25.50.tar.xz,
• gstreamer-vaapi-1.25.50.tar.xz
• gstreamer-docs-1.25.50.tar.xz

As always, please give it a spin and let us know of any issues you run into by filing an issue in GitLab.

So a we are a little bit into the new year I hope everybody had a great break and a good start of 2025. Personally I had a blast having gotten the kids an air hockey table as a Yuletide present :). Anyway, wanted to put this blog post together talking about what we are looking at for the new year and to let you all know that we are hiring.

Artificial Intelligence
One big item on our list for the year is looking at ways Fedora Workstation can make use of artificial intelligence. Thanks to IBMs Granite effort we know have an AI engine that is available under proper open source licensing terms and which can be extended for many different usecases. Also the IBM Granite team has an aggressive plan for releasing updated versions of Granite, incorporating new features of special interest to developers, like making Granite a great engine to power IDEs and similar tools. We been brainstorming various ideas in the team for how we can make use of AI to provide improved or new features to users of GNOME and Fedora Workstation. This includes making sure Fedora Workstation users have access to great tools like RamaLama, that we make sure setting up accelerated AI inside Toolbx is simple, that we offer a good Code Assistant based on Granite and that we come up with other cool integration points.

Wayland
The Wayland community had some challenges last year with frustrations boiling over a few times due to new protocol development taking a long time. Some of it was simply the challenge of finding enough people across multiple projects having the time to follow up and help review while other parts are genuine disagreements of what kind of things should be Wayland protocols or not. That said I think that problem has been somewhat resolved with a general understanding now that we have the ‘ext’ namespace for a reason, to allow people to have a space to review and make protocols without an expectation that they will be universally implemented. This allows for protocols of interest only to a subset of the community going into ‘ext’ and thus allowing protocols that might not be of interest to GNOME and KDE for instance to still have a place to live.

The other more practical problem is that of having people available to help review protocols or providing reference implementations. In a space like Wayland where you need multiple people from multiple different projects it can be hard at times to get enough people involved at any given time to move things forward, as different projects have different priorities and of course the developers involved might be busy elsewhere. One thing we have done to try to help out there is to set up a small internal team, lead by Jonas Ådahl, to discuss in-progress Wayland protocols and assign people the responsibility to follow up on those protocols we have an interest in. This has been helpful both as a way for us to develop internal consensus on the best way forward, but also I think our contribution upstream has become more efficient due to this.

All that said I also believe Wayland protocols will fade a bit into the background going forward. We are currently at the last stage of a community ‘ramp up’ on Wayland and thus there is a lot of focus on it, but once we are over that phase we will probably see what we saw with X.org extensions over time, that for the most time new extensions are so niche that 95% of the community don’t pay attention or care. There will always be some new technology creating the need for important new protocols, but those are likely to come along a relatively slow cadence.

High Dynamic Range

HDR support in GNOME Control Center

As for concrete Wayland protocols the single biggest thing for us for a long while now has of course been the HDR support for Linux. And it was great to see the HDR protocol get merged just before the holidays. I also want to give a shout out to Xaver Hugl from the KWin project. As we where working to ramp up HDR support in both GNOME Shell and GTK+ we ended up working with Xaver and using Kwin for testing especially the GTK+ implementation. Xaver was very friendly and collaborative and I think HDR support in both GNOME and KDE is more solid thanks to that collaboration, so thank you Xaver!

Talking about concrete progress on HDR support Jonas Adahl submitted merge requests for HDR UI controls for GNOME Control Center. This means you will be able to configure the use of HDR on your system in the next Fedora Workstation release.

PipeWire
I been sharing a lot of cool PipeWire news here in the last couple of years, but things might slow down a little as we go forward just because all the major features are basically working well now. The PulseAudio support is working well and we get very few bug reports now against it. The reports we are getting from the pro-audio community is that PipeWire works just as well or better as JACK for most people in terms of for instance latency, and when we do see issues with pro-audio it tends to be more often caused by driver issues triggered by PipeWire trying to use the device in ways that JACK didn’t. We been resolving those by adding more and more options to hardcode certain options in PipeWire, so that just as with JACK you can force PipeWire to not try things the driver has problems with. Of course fixing the drivers would be the best outcome, but for some of these pro-audio cards they are so niche that it is hard to find developers who wants to work on them or who has hardware to test with.

We are still maturing the video support although even that is getting very solid now. The screen capture support is considered fully mature, but the camera support is still a bit of a work in progress, partially because we are going to a generational change the camera landscape with UVC cameras being supplanted by MIPI cameras. Resolving that generational change isn’t just on PipeWire of course, but it does make the a more volatile landscape to mature something in. Of course an advantage here is that applications using PipeWire can easily switch between V4L2 UVC cameras and libcamera MIPI cameras, thus helping users have a smooth experience through this transition period.
But even with the challenges posed by this we are moving rapidly forward with Firefox PipeWire camera support being on by default in Fedora now, Chrome coming along quickly and OBS Studio having PipeWire support for some time already. And last but not least SDL3 is now out with PipeWire camera support.

MIPI camera support
Hans de Goede, Milan Zamazal and Kate Hsuan keeps working on making sure MIPI cameras work under Linux. MIPI cameras are a step forward in terms of technical capabilities, but at the moment a bit of a step backward in terms of open source as a lot of vendors believe they have ‘secret sauce’ in the MIPI camera stacks. Our works focuses mostly on getting the Intel MIPI stack fully working under Linux with the Lattice MIPI aggregator being the biggest hurdle currently for some laptops. Luckily Alan Stern, the USB kernel maintainer, is looking at this now as he got the hardware himself.

Flatpak
Some major improvements to the Flatpak stack has happened recently with the USB portal merged upstream. The USB portal came out of the Sovereign fund funding for GNOME and it gives us a more secure way to give sandboxed applications access to you USB devcices. In a somewhat related note we are still working on making system daemons installable through Flatpak, with the usecase being applications that has a system daemon to communicate with a specific piece of hardware for example (usually through USB). Christian Hergert got this on his todo list, but we are at the moment waiting for Lennart Poettering to merge some pre-requisite work into systemd that we want to base this on.

Accessibility
We are putting in a lot of effort towards accessibility these days. This includes working on portals and Wayland extensions to help facilitate accessibility, working on the ORCA screen reader and its dependencies to ensure it works great under Wayland. Working on GTK4 to ensure we got top notch accessibility support in the toolkit and more.

GNOME Software
Last year Milan Crha landed the support for signing the NVIDIA driver for use on secure boot. The main feature Milan he is looking at now is getting support for DNF5 into GNOME Software. Doing this will resolve one of the longest standing annoyances we had, which is that the dnf command line and GNOME Software would maintain two separate package caches. Once the DNF5 transition is done that should be a thing of the past and thus less risk of disk space being wasted on an extra set of cached packages.

Firefox
Martin Stransky and Jan Horak has been working hard at making Firefox ready for the future, with a lot of work going into making sure it supports the portals needed to function as a flatpak and by bringing HDR support to Firefox. In fact Martin just got his HDR patches for Firefox merged this week. So with the PipeWire camera support, Flatpak support and HDR support in place, Firefox will be ready for the future.

We are hiring! looking for 2 talented developers to join the Red Hat desktop team
We are hiring! So we got 2 job openings on the Red Hat desktop team! So if you are interested in joining us in pushing the boundaries of desktop linux forward please take a look and apply. For these 2 positions we are open to remote workers across the globe and while the job adds list specific seniorities we are somewhat flexible on that front too for the right candidate. So be sure to check out the two job listings and get your application in! If you ever wanted to work fulltime on GNOME and related technologies this is your chance.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.24 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.24.x.

Highlighted bugfixes:

• d3d12: Fix shaders failing to compile with newer dxc versions
• decklinkvideosink: Fix handling of caps framerate in auto mode; also a decklinkaudiosink fix
• devicemonitor: Fix potential crash macOS when a device is unplugged
• gst-libav: Fix crash in audio encoders like avenc_ac3 if input data has insufficient alignment
• gst-libav: Fix build against FFmpeg 4.2 as in Ubuntu 20.04
• gst-editing-services: Fix Python library name fetching on Windows
• netclientclock: Don't store failed internal clocks in the cache, so applications can re-try later
• oggdemux: Seeking and duration fixes
• osxaudiosrc: Fixes for failing init/no output on recent iOS versions
• qtdemux: Use mvhd transform matrix and support for flipping
• rtpvp9pay: Fix profile parsing
• splitmuxsrc: Fix use with decodebin3 which would occasionally fail with an assertion when seeking
• tsdemux: Fix backwards PTS wraparound detection with ignore-pcr=true
• video-overlay-composition: Declare the video/size/orientation tags for the meta and implement scale transformations
• vtdec: Fix seeks occasionally hanging on macOS due to a race condition when draining
• webrtc: Fix duplicate payload types with RTX and multiple video codecs
• win32-pluginloader: Make sure not to create any windows when inspecting plugins
• wpe: Various fixes for re-negotiation, latency reporting, progress messages on startup
• x264enc: Add missing data to AvcDecoderConfigurationRecord in codec_data for high profile variants
• cerbero: Support using ccache with cmake if enabled
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.24.12 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

The GStreamer team is pleased to announce the first development release in the unstable 1.25 release series.

The unstable 1.25 release series is for testing and development purposes in the lead-up to the stable 1.26 series which is scheduled for release ASAP. Any newly-added API can still change until that point.

This development release is primarily for developers and early adopters.

The plan is to get 1.26 out of the door as quickly as possible.

Binaries for Android, iOS, Mac OS X and Windows will be made available shortly at the usual location.

Release tarballs can be downloaded directly here:

• gstreamer-1.25.1.tar.xz
• gst-plugins-base-1.25.1.tar.xz
• gst-plugins-good-1.25.1.tar.xz
• gst-plugins-ugly-1.25.1.tar.xz
• gst-plugins-bad-1.25.1.tar.xz
• gst-libav-1.25.1.tar.xz
• gst-rtsp-server-1.25.1.tar.xz
• gst-python-1.25.1.tar.xz
• gst-editing-services-1.25.1.tar.xz
• gst-devtools-1.25.1.tar.xz,
• gstreamer-vaapi-1.25.1.tar.xz
• gstreamer-docs-1.25.1.tar.xz

As always, please give it a spin and let us know of any issues you run into by filing an issue in GitLab.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.24 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and it should be safe to update from 1.24.x.

Highlighted bugfixes:

• playback: Fix SSA/ASS subtitles with embedded fonts
• decklink: add missing video modes and fix 8K video modes
• matroskamux: spec compliance fixes for audio-only files
• onnx: disable onnxruntime telemetry
• qtdemux: Fix base offset update when doing segment seeks
• srtpdec: Fix a use-after-free issue
• (uri)decodebin3: Fix stream change scenarios, possible deadlock on shutdown
• video: fix missing alpha flag in AV12 format description
• avcodecmap: Add some more channel position mappings
• cerbero bootstrap fixes for Windows 11
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.24.11 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

JPEG XS is a visually lossless, low-latency, intra-only video codec for video production workflows, standardised in ISO/IEC 21122.

A few months ago we added support for JPEG XS encoding and decoding in GStreamer, alongside MPEG-TS container support.

This initially covered progressive scan only though.

Unfortunately interlaced scan, which harks back to the days when TVs had cathode ray tube displays, is still quite common, especially in the broadcasting industry, so it was only a matter of time until support for that would be needed as well.

Long story short, GStreamer can now (with this pending Merge Request) also encode and decode interlaced video into/from JPEG XS.

When putting JPEG XS into MPEG-TS, the individual fields are actually coded separately, so there are two JPEG XS code streams per frame. Inside GStreamer pipelines interlaced raw video can be carried in multiple ways, but the most common one is an "interleaved" image, where the two fields are interleaved row by row, and this is also what capture cards such as AJA or Decklink Blackmagic produce in GStreamer.

When encoding interlaced video in this representation, we need to go twice over each frame and feed every second row of pixels to the underlying SVT JPEG XS encoder which itself is not aware of the interlaced nature of the video content. We do this by specifying double the usual stride as rowstride. This works fine, but unearthed some minor issues with the size checks on the codec side, for which we filed a pull request.

Please give it a spin, and let us know if you have any questions or are interested in additional container mappings such as MP4 or MXF, or RTP payloaders / depayloaders.

As 2024 draws to a close, it’s a perfect time to reflect on the year’s accomplishments done by the Multimedia team in Igalia. In our consideration, there were three major achievements:

1. WebRTC’s support in WPE/WebKitGTK with GStreamer.
2. GES maturity improved for real-life use-cases.
3. Vulkan Video development and support in GStreamer.

WebRTC support in WPE/WebKitGTK with GStreamer #

WPE and WebKitGTK are WebKit ports maintained by Igalia, the former for embedded devices and the latter for applications with a full-featured Web integration.

WebRTC is a web API that allows real-time communication (RTC) directly between web browser and applications. Examples of these real-time communications are video conferencing, cloud gaming, live-streaming, etc.

Some WebKit ports support libwebrtc, an open-source library that implements the WebRTC specification, developed and maintained by Google. WPE and WebKitGTK originally also supports libwebrtc, but we started to use also GstWebRTC, a set of GStreamer plugins and libraries that implement WebRTC, which adapts perfectly to the multimedia implementation in both ports, also in GStreamer.

This year the fruits of this work have been unlocked by enabling Amazon Luna gaming:

https://www.youtube.com/watch?v=lyO7Hqj1jMs

And also enabling a CAD modeling, server-side rendered service, known as Zoo:

https://www.youtube.com/watch?v=CiuYjSCDsUM

WebKit Multimedia #

WebKit made significant improvements in multimedia handling, addressing various issues to enhance stability and playback quality. Key updates include preventing premature play() calls during seeking, fixing memory leaks. The management of track identifiers was also streamlined by transitioning from string-based to integer-based IDs. Additionally, GStreamer-related race conditions were resolved to prevent hangs during playback state transitions. Memory leaks in WebAudio and event listener management were addressed, along with a focus on memory usage optimizations.

The handling of media buffering and seeking was enhanced with buffering hysteresis for smoother playback. Media Source Extensions (MSE) behavior was refined to improve playback accuracy, such as supporting markEndOfStream() before appendBuffer() and simplifying playback checks. Platform-specific issues were also tackled, including AV1 and Opus support for encrypted media and better detection of audio sinks. And other improvements on multimedia performance and efficiency.

GES maturity improved for real-live use-cases #

GStreamer Editing Services (GES) is a set of GStreamer plugins and a library that allow non-linear video editing. For example, GES is what’s behind of Pitivi, the open source video editor application.

Last year, GES was deployed in web-based video editors, where the actual video processing is done server-side. These projects allowed, in great deal, the enhancement and maturity of the library and plugins.

Tella is a browser-based tool that allow to screen record and webcam, without any extra software. Finished the recording, the user can edit, in the browser, the video and publish it.

https://www.youtube.com/watch?v=uSWqWHBRDWE

Sequence is a complete, browser-based, video editor with collaborative features. GES is used in the backend to render the editing operations.

https://www.youtube.com/watch?v=bXNdDIiG9lE

Vulkan Video development and GStreamer support #

The last but not the least, this year we continue our work with the Vulkan Video ecosystem by working the task subgroup (TSG) enabling H.264/H.265 encoding, and AV1 decoding and encoding.

Early this year we delivered a talk in the Vulkanised about our work, which ranges from the Conformance Test Suite (CTS), Mesa, and GStreamer.

https://www.youtube.com/watch?v=z1HcWrmdwzI

Conclusion #

As we wrap up 2024, it’s clear that the year has been one of significant progress, driven by innovation and collaboration. Here’s to continuing the momentum and making 2025 even better!

Some time ago, Edward and I wrote a new element that allows clocking a GStreamer pipeline from an MPEG-TS stream, for example received via SRT.

This new element, mpegtslivesrc, wraps around any existing live source element, e.g. udpsrc or srtsrc, and provides a GStreamer clock that approximates the sender's clock. By making use of this clock as pipeline clock, it is possible to run the whole pipeline at the same speed as the sender is producing the stream and without having to implement any kind of clock drift mechanism like skewing or resampling. Without this it is necessary currently to adjust the timestamps of media coming out of GStreamer's tsdemux element, which is problematic if accurate timestamps are necessary or the stream should be stored to a file, e.g. a 25fps stream wouldn't have exactly 40ms inter-frame timestamp differences anymore.

The clock is approximated by making use of the in-stream MPEG-TS PCR, which basically gives the sender's clock time at specific points inside the stream, and correlating that together with the local receive times via a linear regression to calculate the relative rate between the sender's clock and the local system clock.

Usage of the element is as simple as

$ gst-launch-1.0 mpegtslivesrc source='srtsrc location=srt://1.2.3.4:5678?latency=150&mode=caller' ! tsdemux skew-corrections=false ! ...
$ gst-launch-1.0 mpegtslivesrc source='udpsrc address=1.2.3.4 port=5678' ! tsdemux skew-corrections=false ! ...

Addition 2025-06-28: If you're using an older (< 1.28) version of GStreamer, you'll have to use the ignore-pcr=true property on tsdemux instead. skew-corrections=false was only added recently and allows for more reliable handling of MPEG-TS timestamp discontinuities.

A similar approach for clocking is implemented in the AJA source element and the NDI source element when the clocked timestamp mode is configured.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.24 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and security fixes.

It should be safe to update from 1.24.x, and we would recommend you update at your earliest convenience.

Highlighted bugfixes:

• More than 40 security fixes across a wide range of elements following an audit by the GitHub Security Lab, including the MP4, Matroska, Ogg and WAV demuxers, subtitle parsers, image decoders, audio decoders and the id3v2 tag parser
• avviddec: Fix regression that could trigger assertions about width/height mismatches
• appsink and appsrc fixes
• closed caption handling fixes
• decodebin3 and urisourcebin fixes
• glupload: dmabuf: Fix emulated tiled import
• level: fix LevelMeta values outside of the stated range
• mpegtsmux, flvmux: fix potential busy looping with high cpu usage in live mode
• pipeline dot file graph generation improvements
• qt(6): fix criticals with multiple qml(6)gl{src,sink}
• rtspsrc: Optionally timestamp RTP packets with their receive times in TCP/HTTP mode to enable clock drift handling
• splitmuxsrc: reduce number of file descriptors used
• systemclock: locking order fixes
• v4l2: fix possible v4l2videodec deadlock on shutdown; 8-bit bayer format fixes
• x265: Fix build with libx265 version >= 4.1 after masteringDisplayColorVolume API change
• macOS: fix rendering artifacts in retina displays, plus ptp clock fixes
• cargo: Default to thin lto for the release profile (for faster builds with lower memory requirements)
• Various bug fixes, build fixes, memory leak fixes, and other stability and reliability improvements
• Translation updates

See the GStreamer 1.24.10 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

Thanks to the newly added atenc element, you can now use Apple's well-known AAC encoder directly in GStreamer!

gst-launch-1.0 -e audiotestsrc ! audio/x-raw,channels=2,rate=48000 ! atenc ! mp4mux ! filesink location=output.m4a

It supports all the usual rate control modes (CBR/LTA/VBR/CVBR), as well as settings relevant for each of them (target bitrate for CBR, perceived quality for VBR).

For now you can encode AAC-LC with up to 7.1 channels. Support for more AAC profiles and different output formats will be added in the future.

If you need decoding too, atdec is there to help and has supported AAC alongside a few other formats for a long time now.

The GStreamer team is pleased to announce another bug fix release in the new stable 1.24 release series of your favourite cross-platform multimedia framework!

This release only contains bugfixes and a security fix and it should be safe to update from 1.24.x.

Highlighted bugfixes:

• gst-rtsp-server security fix
• GstAggregator start time selection and latency query fixes for force-live mode
• audioconvert: fix dynamic handling of mix matrix, and accept custom upstream event for setting one
• encodebin: fix parser selection for encoders that support multiple codecs
• flvmux improvments for pipelines where timestamps don't start at 0
• glcontext: egl: Unrestrict the support base DRM formats
• kms: Add IMX-DCSS auto-detection in sink and fix stride with planar formats in allocator
• macOS main application event loop fixes
• mpegtsdemux: Handle PTS/DTS wraparound with ignore-pcr=true
• playbin3, decodebin3, parsebin, urisourcebin: fix races, and improve stability and stream-collection handling
• rtpmanager: fix early RTCP SR generation for sparse streams like metadata
• qml6glsrc: Reduce capture delay
• qtdemux: fix parsing of rotation matrix with 180 degree rotation
• rtpav1depay: added wait-for-keyframe and request-keyframe properties
• srt: make work with newer libsrt versions and don't re-connect on authentication failure
• v4l2 fixes and improvement
• webrtcsink, webrtcbin and whepsrc fixes
• cerbero: fix Python 3.13 compatibility, g-i with newer setuptools, bootstrap on Arch Linux; iOS build fixes
• Ship qroverlay plugin in binary packages
• Various bug fixes, memory leak fixes, and other stability and reliability improvements

See the GStreamer 1.24.9 release notes for more details.

Binaries for Android, iOS, Mac OS X and Windows will be available shortly.

If you've ever seen a news or sports channel playing without sound in the background of a hotel lobby, bar, or airport, you've probably seen closed captions in action.

These TV-style captions are alphabet/character-based, with some very basic commands to control the positioning and layout of the text on the screen.

They are very low bitrate and were transmitted in the invisible part of TV images during the vertical blanking interval (VBI) back in those good old analogue days ("line 21 captions").

Nowadays they are usually carried as part of the MPEG-2 or H.264/H.265 video bitstream, unlike say text subtitles in a Matroska file which will be its own separate stream in the container.

In GStreamer closed captions can be carried in different ways: Either implicitly as part of a video bitstream, or explicitly as part of a video bitstream with video caption metas on the buffers passing through the pipeline. Captions can also travel through a pipeline stand-alone in form of one of multiple raw caption bitstream formats.

To make handling these different options easier for applications there are elements that can extract captions from the video bitstream into metas, and split off captions from metas into their own stand-alone stream, and to do the reverse and combine and reinject them again.

SMPTE 2038 Ancillary Data

SMPTE 2038 (pdf) is a generic system to put VBI-style ancillary data into an MPEG-TS container. This could include all kinds of metadata such as scoreboard data or game clocks, and of course also closed captions, in this case in form of a distinct stream completely separate from any video bitstream.

We've recently added support for SMPTE 2038 ancillary data in GStreamer. This comes in form of a number of new elements in the GStreamer Rust closedcaption plugin and mappings for it in the MPEG-TS muxer and demuxer.

The new elements are:

• st2038ancdemux: splits SMPTE ST-2038 ancillary metadata (as received from tsdemux) into separate streams per DID/SDID and line/horizontal_offset. Will add a sometimes pad with details for each ancillary stream. Also has an always source pad that just outputs all ancillary streams for easy forwarding or remuxing, in case none of the ancillary streams need to be modified or dropped.

• st2038ancmux: muxes SMPTE ST-2038 ancillary metadata streams into a single stream for muxing into MPEG-TS with mpegtsmux. Combines ancillary data on the same line if needed, as is required for MPEG-TS muxing. Can accept individual ancillary metadata streams as inputs and/or the combined stream from st2038ancdemux.

  If the video framerate is known, it can be signalled to the ancillary data muxer via the output caps by adding a capsfilter behind it, with e.g. meta/x-st-2038,framerate=30/1.

  This allows the muxer to bundle all packets belonging to the same frame (with the same timestamp), but that is not required. In case there are multiple streams with the same DID/SDID that have an ST-2038 packet for the same frame, it will prioritise the one from more recently created request pads over those from earlier created request pads (which might contain a combined stream for example if that's fed first).

• st2038anctocc: extracts closed captions (CEA-608 and/or CEA-708) from SMPTE ST-2038 ancillary metadata streams and outputs them on the respective sometimes source pad (src_cea608 or src_cea708). The data is output as a closed caption stream with caps closedcaption/x-cea-608,format=s334-1a or closedcaption/x-cea-708,format=cdp for further processing by other GStreamer closed caption processing elements.

• cctost2038anc: takes closed captions (CEA-608 and/or CEA-708) as produced by other GStreamer closed caption processing elements and converts them into SMPTE ST-2038 ancillary data that can be fed to st2038ancmux and then to mpegtsmux for splicing/muxing into an MPEG-TS container. The line-number and horizontal-offset properties should be set to the desired line number and horizontal offset.

Please give it a spin and let us know how it goes!

    The new usb_set_wireless_status() driver API function can be used by drivers of USB devices to export whether the wireless device associated with that USB dongle is turned on or not.

    To quote the commit message:

This will be used by user-space OS components to determine whether the
battery-powered part of the device is wirelessly connected or not,
allowing, for example:
- upower to hide the battery for devices where the device is turned off
  but the receiver plugged in, rather than showing 0%, or other values
  that could be confusing to users
- Pipewire to hide a headset from the list of possible inputs or outputs
  or route audio appropriately if the headset is suddenly turned off, or
  turned on
- libinput to determine whether a keyboard or mouse is present when its
  receiver is plugged in.

This is not an attribute that is meant to replace protocol specific
APIs [...] but solely for wireless devices with
an ad-hoc “lose it and your device is e-waste” receiver dongle.
 

    Currently, the only 2 drivers to use this are the ones for the Logitech G935 headset, and the Steelseries Arctis 1 headset. Adding support for other Logitech headsets would be possible if they export battery information (the protocols are usually well documented), support for more Steelseries headsets should be feasible if the protocol has already been reverse-engineered.

    As far as consumers for this sysfs attribute, I filed a bug against Pipewire (link) to use it to not consider the receiver dongle as good as unplugged if the headset is turned off, which would avoid audio being sent to headsets that won't hear it.

    UPower supports this feature since version 1.90.1 (although it had a bug that makes 1.90.2 the first viable release to include it), and batteries will appear and disappear when the device is turned on/off.

A turned-on headset

Early this month I spent a couple weeks in Montreal, visiting the city, but mostly to attend the GStreamer Conference and the following hackfest, which happened to be co-located with the XDC Conference. It was my first time in Canada and I utterly enjoyed it. Thanks to all those from the GStreamer community that organized and attended the event.

For now you can replay the whole streams of both days of conference, but soon the split videos for each talk will be published:

GStreamer Conference 2024 - Day 1, Room 1 - October 7, 2024

https://www.youtube.com/watch?v=KLUL1D53VQI

GStreamer Conference 2024 - Day 1, Room 2 - October 7, 2024

https://www.youtube.com/watch?v=DH64D_6gc80

GStreamer Conference 2024 - Day 2, Room 2 - October 8, 2024

https://www.youtube.com/watch?v=jt6KyV757Dk

GStreamer Conference 2024 - Day 2, Room 1 - October 8, 2024

https://www.youtube.com/watch?v=W4Pjtg0DfIo

And a couple pictures of igalians :)

The <video> element implementation in WebKit does its job by using a multiplatform player that relies on a platform-specific implementation. In the specific case of glib platforms, which base their multimedia on GStreamer, that’s MediaPlayerPrivateGStreamer.

The player private can have 3 buffering modes:

• On-disk buffering: This is the typical mode on desktop systems, but is frequently disabled on purpose on embedded devices to avoid wearing out their flash storage memories. All the video content is downloaded to disk, and the buffering percentage refers to the total size of the video. A GstDownloader element is present in the pipeline in this case. Buffering level monitoring is done by polling the pipeline every second, using the fillTimerFired() method.
• In-memory buffering: This is the typical mode on embedded systems and on desktop systems in case of streamed (live) content. The video is downloaded progressively and only the part of it ahead of the current playback time is buffered. A GstQueue2 element is present in the pipeline in this case. Buffering level monitoring is done by listening to GST_MESSAGE_BUFFERING bus messages and using the buffering level stored on them. This is the case that motivates the refactoring described in this blog post, what we actually wanted to correct in Broadcom platforms, and what motivated the addition of hysteresis working on all the platforms.
• Local files: Files, MediaStream sources and other special origins of video don’t do buffering at all (no GstDownloadBuffering nor GstQueue2 element is present on the pipeline). They work like the on-disk buffering mode in the sense that fillTimerFired() is used, but the reported level is relative, much like in the streaming case. In the initial version of the refactoring I was unaware of this third case, and only realized about it when tests triggered the assert that I added to ensure that the on-disk buffering method was working in GST_BUFFERING_DOWNLOAD mode.

The current implementation (actually, its wpe-2.38 version) was showing some buffering problems on some Broadcom platforms when doing in-memory buffering. The buffering levels monitored by MediaPlayerPrivateGStreamer weren’t accurate because the Nexus multimedia subsystem used on Broadcom platforms was doing its own internal buffering. Data wasn’t being accumulated in the GstQueue2 element of playbin, because BrcmAudFilter/BrcmVidFilter was accepting all the buffers that the queue could provide. Because of that, the player private buffering logic was erratic, leading to many transitions between “buffer completely empty” and “buffer completely full”. This, it turn, caused many transitions between the HaveEnoughData, HaveFutureData and HaveCurrentData readyStates in the player, leading to frequent pauses and unpauses on Broadcom platforms.

So, one of the first thing I tried to solve this issue was to ask the Nexus PlayPump (the subsystem in charge of internal buffering in Nexus) about its internal levels, and add that to the levels reported by GstQueue2. There’s also a GstMultiqueue in the pipeline that can hold a significant amount of buffers, so I also asked it for its level. Still, the buffering level unstability was too high, so I added a moving average implementation to try to smooth it.

All these tweaks only make sense on Broadcom platforms, so they were guarded by ifdefs in a first version of the patch. Later, I migrated those dirty ifdefs to the new quirks abstraction added by Phil. A challenge of this migration was that I needed to store some attributes that were considered part of MediaPlayerPrivateGStreamer before. They still had to be somehow linked to the player private but only accessible by the platform specific code of the quirks. A special HashMap attribute stores those quirks attributes in an opaque way, so that only the specific quirk they belong to knows how to interpret them (using downcasting). I tried to use move semantics when storing the data, but was bitten by object slicing when trying to move instances of the superclass. In the end, moving the responsibility of creating the unique_ptr that stored the concrete subclass to the caller did the trick.

Even with all those changes, undesirable swings in the buffering level kept happening, and when doing a careful analysis of the causes I noticed that the monitoring of the buffering level was being done from different places (in different moments) and sometimes the level was regarded as “enough” and the moment right after, as “insufficient”. This was because the buffering level threshold was one single value. That’s something that a hysteresis mechanism (with low and high watermarks) can solve. So, a logical level change to “full” would only happen when the level goes above the high watermark, and a logical level change to “low” when it goes under the low watermark level.

For the threshold change detection to work, we need to know the previous buffering level. There’s a problem, though: the current code checked the levels from several scattered places, so only one of those places (the first one that detected the threshold crossing at a given moment) would properly react. The other places would miss the detection and operate improperly, because the “previous buffering level value” had been overwritten with the new one when the evaluation had been done before. To solve this, I centralized the detection in a single place “per cycle” (in updateBufferingStatus()), and then used the detection conclusions from updateStates().

So, with all this in mind, I refactored the buffering logic as https://commits.webkit.org/284072@main, so now WebKit GStreamer has a buffering code much more robust than before. The unstabilities observed in Broadcom devices were gone and I could, at last, close Issue 1309.