This document outlines the techniques used for doing synchronised playback of multiple streams.
Synchronisation in a
GstPipeline is achieved using the following 3
GstClock, which is global for all elements in a
Timestamps on a
the SEGMENT event preceding the buffers.
This object provides a counter that represents the current time in
nanoseconds. This value is called the
always counts time upwards and does not necessarily start at 0.
Different sources exist for this counter:
the system time (with
g_get_current_time()and with microsecond accuracy)
monotonic time (with
g_get_monotonic_time()with microsecond accuracy)
an audio device (based on number of samples played)
a network source based on packets received + timestamps in those packets (a typical example is an RTP source)
In GStreamer any element can provide a
GstClock object that can be used
in the pipeline. The
GstPipeline object will select a clock from all the
providers and will distribute it to all other elements (see
While it is possible, it is not recommended to create a clock derived from the contents of a stream (for example, create a clock from the PCR in an mpeg-ts stream).
After a pipeline selected a clock it will maintain the
based on the selected clock. This
running_time represents the total
time spent in the PLAYING state and is calculated as follows:
If the pipeline is NULL/READY, the
In PAUSED, the
running_timeremains at the time when it was last PAUSED. When the stream is
PAUSEDfor the first time, the
In PLAYING, the
running_timeis the delta between the
absolute_timeand the base time. The base time is defined as the
running_timeat the time when the pipeline is set to
after a flushing seek, the
running_timeis set to 0 (see seeking). This is accomplished by redistributing a new base_time to the elements that got flushed.
This algorithm captures the
running_time when the pipeline is set from
PAUSED and restores this time based on the current
absolute_time when going back to
PLAYING. This allows for both clocks
that progress when in the
PAUSED state (systemclock) and clocks that
The clock and pipeline now provide a
running_time to all elements that
want to perform synchronisation. Indeed, the running time can be
observed in each element (during the PLAYING state) as:
C.running_time = absolute_time - base_time
C.running_time as the
running_time obtained by looking at the
clock. This value is monotonically increasing at the rate of the clock.
GstBuffer timestamps and the preceding SEGMENT event (See
streams) define a transformation of the buffer timestamps
running_time as follows:
The following notation is used:
- B.timestamp = buffer timestamp (
S: SEGMENT event preceding the buffers.
- S.start: start field in the SEGMENT event. This is the lowest allowed
- S.stop: stop field in the SEGMENT event. This is the highers allowed
- S.rate: rate field of SEGMENT event. This is the playback rate.
- S.base: a base time for the time. This is the total elapsed
of any previous segments.
- S.offset: an offset to apply to S.start or S.stop. This is the amount that
has already been elapsed in the segment.
Valid buffers for synchronisation are those with B.timestamp between
S.stop (after applying the
S.offset). All other buffers
outside this range should be dropped or clipped to these boundaries (see
The following transformation to
if (S.rate > 0.0) B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base => B.timestamp = (B.running_time - S.base) * ABS (S.rate) + S.start + S.offset else B.running_time = ((S.stop - S.offset) - B.timestamp) / ABS (S.rate) + S.base => B.timestamp = S.stop - S.offset - ((B.running_time - S.base) * ABS (S.rate))
B.running_time as the
running_time obtained from the
event and the buffers of that segment.
The first displayable buffer will yield a value of 0 (since
B.timestamp == S.start and S.offset and S.base == 0).
S.rate > 1.0, the timestamps will be scaled down to increase the
playback rate. Likewise, a rate between 0.0 and 1.0 will slow down
For negative rates, timestamps are received stop S.stop to
that the first buffer received will be transformed into
of 0 (
B.timestamp == S.stop and S.base == 0).
This makes it so that
B.running_time is always monotonically increasing
starting from 0 with both positive and negative rates.
As we have seen, we can get a
- using the clock and the element’s
C.running_time = absolute_time - base_time
- using the buffer timestamp and the preceding
SEGMENTevent as (assuming positive playback rate):
B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base
We prefix C. and B. before the two running times to note how they were calculated.
The task of synchronized playback is to make sure that we play a buffer
B.running_time at the moment when the clock reaches the same
Thus the following must hold:
B.running_time = C.running_time
B.running_time = absolute_time - base_time
absolute_time = B.running_time + base_time
absolute_time when a buffer with
B.running_time should be played
is noted with
B.sync_time = B.running_time + base_time
One then waits for the clock to reach
B.sync_time before rendering the
buffer in the sink (See also clocks).
For multiple streams this means that buffers with the same
are to be displayed at the same time.
A demuxer must make sure that the
SEGMENT it emits on its output pads
yield the same
running_time for buffers that should be played
synchronized. This usually means sending the same
SEGMENT on all pads
and making sure that the synchronized buffers have the same timestamps.
The stream time is also known as the position in the stream and is a value between 0 and the total duration of the media file.
It is the stream time that is used for:
POSITIONquery in the pipeline
the position used in seek events/queries
the position used to synchronize controller values
Additional fields in the
SEGMENT are used:
S.time: time field in the
SEGMENTevent. This the stream-time of
S.applied_rate: The rate already applied to the segment.
Stream time is calculated using the buffer times and the preceding
SEGMENT event as follows:
stream_time = (B.timestamp - S.start) * ABS (S.applied_rate) + S.time => B.timestamp = (stream_time - S.time) / ABS(S.applied_rate) + S.start
For negative rates,
B.timestamp will go backwards from
S.start, making the stream time go backwards:
stream_time = (S.stop - B.timestamp) * ABS(S.applied_rate) + S.time => B.timestamp = S.stop - (stream_time - S.time) / ABS(S.applied_rate)
PLAYING state, it is also possible to use the pipeline clock to
derive the current
Give the two formulas above to match the clock times with buffer
timestamps allows us to rewrite the above formula for
for positive rates).
C.running_time = absolute_time - base_time B.running_time = (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base => (B.timestamp - (S.start + S.offset)) / ABS (S.rate) + S.base = absolute_time - base_time; => (B.timestamp - (S.start + S.offset)) / ABS (S.rate) = absolute_time - base_time - S.base; => (B.timestamp - (S.start + S.offset)) = (absolute_time - base_time - S.base) * ABS (S.rate) => (B.timestamp - S.start) = S.offset + (absolute_time - base_time - S.base) * ABS (S.rate) filling (B.timestamp - S.start) in the above formule for stream time => stream_time = (S.offset + (absolute_time - base_time - S.base) * ABS (S.rate)) * ABS (S.applied_rate) + S.time
This last formula is typically used in sinks to report the current position in an accurate and efficient way.
Note that the stream time is never used for synchronisation against the clock.
The results of the search are