advanced features of GStreamer

In this part we will cover the more advanced features of
GStreamer.With the basics you learned in the

previous part you should be able to create a simple application.
However, GStreamer provides much

more candy than just the basics of playing back audio files. In this
chapter, you will learn more of the

low-level features and internals of GStreamer.

Some parts of this part will serve mostly as an explanation of how
GStreamer works internally; they are

not actually needed for actual application development. This includes
chapter such as the ones covering

scheduling, autoplugging and synchronization. Other chapters, however,
discuss more advanced ways of

pipeline-application interaction, and can turn out to be very useful for
certain applications. This includes

the chapters on metadata, querying and events, interfaces, dynamic
parameters and pipeline data

manipulation.
有些章节旨在解释Gstreamer的内部工作机制，他们对于应用程序开发者并不是很有用，例如调度，自动插件和同步；有些章节，讨论了更多管
道程序交互，将对创建应用程序非常有用。例如元数据，查询和事件，界面，动态参数和管道数据操作。

Position tracking
and seeking

Querying: getting
the position or length of a stream

Querying is defined as requesting a specific stream-property
related to progress tracking. This includes

getting the length of a stream (if available) or getting the current
position. Those stream properties can be

retrieved in various formats such as time, audio samples, video frames
or bytes. The function most

commonly used for this is gst_element_query (), although some
convenience wrappers are

provided as well (such as gst_element_query_position () and
gst_element_query_duration

()). You can generally query the pipeline directly, and it’ll figure out
the internal details for you, like

which element to query.

查询时指要求获得关于有关于进度跟踪的流属性。例如流的长度，当前位置等。它们可以许多形式：时间，帧，字节等。
在内部，查询会被传送给sink单元，然后向上派遣，直到有单元可以处理它。然后结果会被送回函数调用者。通常是，demuxer或者在线源，
源本身。

# include < gst/ gst. h> static gboolean cb_print_position ( GstElement * pipeline) { GstFormat fmt = GST_FORMAT_TIME; gint64 pos, len; if ( gst_element_query_position ( pipeline, & fmt, & pos) & & gst_element_query_duration ( pipeline, & fmt, & len) ) { g_print ( "Time: %" GST_TIME_FORMAT " / %" GST_TIME_FORMAT "\r" , GST_TIME_ARGS ( pos) , GST_TIME_ARGS ( len) ) ; } /* call me again */ return TRUE ; } gint main ( gint argc, gchar * argv[ ] ) { GstElement * pipeline; [ . . ] /* run pipeline */ g_timeout_add ( 200, ( GSourceFunc) cb_print_position, pipeline) ; g_main_loop_run ( loop) ; [ . . ] }

g_timeout_add (
200,
(
GSourceFunc)
cb_print_position,
pipeline)
;

是定期发出查询。

Events: seeking (and more)

Events work in a very similar way as
queries. Dispatching, for example, works exactly the same for

events (and also has the same limitations), and they can similarly be
sent to the toplevel pipeline and it

will figure out everything for you. Although there are more ways in
which applications and elements can

interact using events, we will only focus on seeking here. This is done
using the seek-event. A seek-event

contains a playback rate, a seek offset format (which is the unit of the
offsets to follow, e.g. time, audio

samples, video frames or bytes), optionally a set of seeking-related
flags (e.g. whether internal buffers

should be flushed), a seek method (which indicates relative to what the
offset was given), and seek

offsets. The first offset (cur) is the new position to seek to, while
the second offset (stop) is optional and

specifies a position where streaming is supposed to stop. Usually it is
fine to just specify

GST_SEEK_TYPE_NONE and -1 as end_method and end offset. The behaviour of
a seek is also

wrapped in the gst_element_seek ().

事件的工作方式非常像查询，例如派遣机制的工作方式是一样的。虽然有很多的方式
供应用程序和元素使用事件交互，我们只关注定位这种方式。它使用定位事件。

static void seek_to_time ( GstElement * pipeline, gint64 time_nanoseconds) { if ( ! gst_element_seek ( pipeline, 1. 0, GST_FORMAT_TIME, GST_SEEK_FLAG_FLUSH, GST_SEEK_TYPE_SET, time_nanoseconds, GST_SEEK_TYPE_NONE, GST_CLOCK_TIME_NONE) ) { g_print ( "Seek failed!\n" ) ; } }

GST_SEEK_FLAG_FLUSH标识的影响：
Metadata
GStreamer makes a clear distinction between two types of metadata,
and has support for both types. The

first is stream tags, which describe the content of a stream in a
non-technical way. Examples include the

author of a song, the title of that very same song or the album it is a
part of. The other type of metadata is

stream-info, which is a somewhat technical description of the properties
of a stream. This can include

video size, audio samplerate, codecs used and so on. Tags are handled
using the GStreamer tagging

system. Stream-info can be retrieved from a GstPad.
GStreamer区别两种元数据：
1.流标签。关于流的内容层次。例如歌手，题目等。
2.流信息。关于流的技术层次。例如采样率，编码技术等。
流标签由GStreamer的标签机制处理，而流信息可以从GstPad获得。
Metadata reading

Stream information can most easily be read by reading them from a
GstPad. This has already been

discussed before in Section 8.3.1. Therefore, we will skip it here. Note
that this requires access to all

pads of which you want stream information.

Tag reading is done through a bus in GStreamer, which has been discussed
previously in Chapter 7. You

can listen for GST_MESSAGE_TAG messages and handle them as you wish.
Interfaces

In Section 5.3, you have learned how to use GObject properties as a
simple way to do interaction

between applications and elements. This method suffices for the
simple’n’straight settings, but fails for

anything more complicated than a getter and setter. For the more
complicated use cases, GStreamer uses

interfaces based on the Glib GInterface type.

Most of the interfaces handled here will not contain any example code.
See the API references for

details. Here, we will just describe the scope and purpose of each
interface.
涉及GObject属性的概念只能提供set和get这样简单的接口。GStreamer提供了更多的基于Glib的GInterface提供了丰
富的接口。
*The URI interface

In all examples so far, we have only supported local files through
the “filesrc” element. GStreamer,

obviously, supports many more location sources. However, we don’t want
applications to need to know

any particular element implementation details, such as element names for
particular network source types

and so on. Therefore, there is a URI interface, which can be used to get
the source element that supports a

particular URI type. There is no strict rule for URI naming, but in
general we follow naming conventions

that others use, too. For example, assuming you have the correct plugins
installed, GStreamer supports

“file:///<path>/<file>”,
“http://<host>/<path>/<file>”,
“mms://<host>/<path>/<file>”, and so on.

In order to get the source or sink element supporting a particular URI,
use

gst_element_make_from_uri (), with the URI type being either GST_URI_SRC
for a source

element, or GST_URI_SINK for a sink element.
*The Mixer
interface

The mixer interface provides a uniform way to control the volume on a
hardware (or software) mixer.

The interface is primarily intended to be implemented by elements for
audio inputs and outputs that talk

directly to the hardware (e.g. OSS or ALSA plugins).

Using this interface, it is possible to control a list of tracks (such
as Line-in, Microphone, etc.) from a

mixer element. They can be muted, their volume can be changed and, for
input tracks, their record flag

can be set as well.

Example plugins implementing this interface include the OSS elements
(osssrc, osssink, ossmixer) and

the ALSA plugins (alsasrc, alsasink and alsamixer).
*The Tuner
interface

The tuner interface is a uniform way to control inputs and outputs on
a multi-input selection device. This

is primarily used for input selection on elements for TV- and
capture-cards.

Using this interface, it is possible to select one track from a list of
tracks supported by that tuner-element.

The tuner will than select that track for media-processing internally.
This can, for example, be used to

switch inputs on a TV-card (e.g. from Composite to S-video).

This interface is currently only implemented by the Video4linux and
Video4linux2 elements.
*The Color Balance
interface

The colorbalance interface is a way to control video-related
properties on an element, such as brightness,

contrast and so on. It’s sole reason for existance is that, as far as
its authors know, there’s no way to

dynamically register properties using GObject.

The colorbalance interface is implemented by several plugins, including
xvimagesink and the

Video4linux and Video4linux2 elements.
*The Property Probe
interface

The property probe is a way to autodetect allowed values for a
GObject property. It’s primary use is to

autodetect devices in several elements. For example, the OSS elements
use this interface to detect all

OSS devices on a system. Applications can then “probe” this property and
get a list of detected devices.

Note: Given the overlap between HAL and the practical implementations of
this interface, this might

in time be deprecated in favour of HAL.

This interface is currently implemented by many elements, including the
ALSA, OSS, XVImageSink,

Video4linux and Video4linux2 elements.
*The X Overlay
interface

The X Overlay interface was created to solve the problem of embedding
video streams in an application

window. The application provides an X-window to the element implementing
this interface to draw on,

52

Chapter 13. Interfaces

and the element will then use this X-window to draw on rather than
creating a new toplevel window. This

is useful to embed video in video players.

This interface is implemented by, amongst others, the Video4linux and
Video4linux2 elements and by

ximagesink, xvimagesink and sdlvideosink.
Clocks in GStreamer

To maintain sync in pipeline playback (which is the only case where
this really matters), GStreamer uses

clocks. Clocks are exposed by some elements, whereas other elements are
merely clock slaves. The

primary task of a clock is to represent the time progress according to
the element exposing the clock,

based on its own playback rate. If no clock provider is available in a
pipeline, the system clock is used

instead.
为了维护同步和回放，GStreamer使用时钟。有几种不同的时钟，他们单独的当前值是没有意义的，他们的差值才有意义。
Clock providers

Clock providers exist because they play back media at some rate, and
this rate is not necessarily the same

as the system clock rate. For example, a soundcard may playback at 44,1
kHz, but that doesn’t mean that

after exactly 1 second according to the system clock, the soundcard has
played back 44.100 samples.

This is only true by approximation. Therefore, generally, pipelines with
an audio output use the

audiosink as clock provider. This ensures that one second of video will
be played back at the same rate as

that the soundcard plays back 1 second of audio.
时钟提供者保持同一速率播放媒体，但是这个速率不一定和系统时钟一致。声卡可能工作在44.1KHz的速率，但是并不精确。管道使用
audiosink作为时钟提供者，保证一秒的视频和一秒的音频同步。
Clock slaves

Clock slaves get assigned a clock by their containing pipeline. Their
task is to make sure that media

playback follows the time progress as represented by this clock as
closely as possible. For most

elements, that will simply mean to wait until a certain time is reached
before playing back their current

sample; this can be done with the function gst_clock_id_wait (). Some
elements may need to

support dropping samples too, however.
时钟仆从由管道获得一个指派的时钟。它们会等待某个特定的时刻播放当前帧。某些元素还需要支持帧丢弃。
Dynamic
Controllable Parameters

Getting Started

The controller subsystem offers a lightweight way to adjust gobject
properties over stream-time. It works

by using time-stamped value pairs that are queued for
element-properties. At run-time the elements

continously pull values changes for the current stream-time.
控制器机制提供了轻松的方法在运行期间调整gobject的属性。它为元素属性维持附带时间戳的值对（域名+值）。在运行期间，元素会持续提取变化
的值。
Setting up
parameter control

The first step is to select the parameters that should be
controlled. This returns a controller object that is

needed to further adjust the behaviour.

controller = gst_object_control_properties(object, "prop1",
"prop2",...);

Next we can select an interpolation mode. This mode controls how
inbetween values are determined. The

controller subsystem can e.g. fill gaps by smoothing parameter changes.
Each controllable GObject

property can be interpolated differently.

gst_controller_set_interpolation_mode(controller,"prop1",mode);

Finally one needs to set control points. These are time-stamped GValues.
The values become active when

the timestamp is reached. They still stay in the list. If e.g. the
pipeline runs a loop (using a segmented

seek), the control-curve gets repeated as well.

gst_controller_set (controller, "prop1" ,0 * GST_SECOND, value1);

gst_controller_set (controller, "prop1" ,1 * GST_SECOND, value2);

The controller subsystem has a builtin live-mode. Even though a
parameter has timestamped

control-values assigned one can change the GObject property through
g_object_set(). This is highly

useful when binding the GObject properties to GUI widgets.When the user
adjusts the value with the

widget, one can set the GOBject property and this remains active until
the next timestamped value

overrides. This also works with smoothed parameters.

Threads

GStreamer is inherently multi-threaded, and is fully
thread-safe.Most threading internals are hidden

from the application, which should make application development easier.
However, in some cases,

applications may want to have influence on some parts of those.
GStreamer allows applications to force

the use of multiple threads over some parts of a pipeline.

When would you
want to force a thread?

There are several reasons to force the use of threads. However, for
performance reasons, you never want

to use one thread for every element out there, since that will create
some overhead. Let’s now list some

situations where threads can be particularly useful:
• Data buffering, for example when dealing with network streams or
when recording data from a live

stream such as a video or audio card. Short hickups elsewhere in the
pipeline will not cause data loss.
数据缓冲
• Synchronizing output devices, e.g. when playing a stream
containing both video and audio data. By

using threads for both outputs, they will run independently and their
synchronization will be better.
输出同步
Above, we’ve mentioned the “queue” element several times now. A
queue is the thread boundary

element through which you can force the use of threads. It does so by
using a classic provider/receiver

model as learned in threading classes at universities all around the
world. By doing this, it acts both as a

means to make data throughput between threads threadsafe, and it can
also act as a buffer. Queues have

several GObject properties to be configured for specific uses. For
example, you can set lower and upper

tresholds for the element. If there’s less data than the lower treshold
(default: disabled), it will block

output. If there’s more data than the upper treshold, it will block
input or (if configured to do so) drop

data.

To use a queues (and therefore force the use of two distinct threads in
the pipeline), one can simply

create a “queue” element and put this in as part of the pipeline.
GStreamer will take care of all threading

details internally.
通过队列可以控制线程，定义队列的上限和下限，可以在适当的时候阻塞输入或输出。

Scheduling in
GStreamer

Scheduling of pipelines in GStreamer is done by using a thread for
each “group”, where a group is a set

of elements separated by “queue” elements.Within such a group,
scheduling is either push-based or

pull-based, depending on which mode is supported by the particular
element. If elements support random

access to data, such as file sources, then elements downstream in the
pipeline become the entry point of

this group (i.e. the element controlling the scheduling of other
elements). The entry point pulls data from

upstream and pushes data downstream, thereby calling data handling
functions on either type of element.

In practice, most elements in GStreamer, such as decoders, encoders,
etc. only support push-based

scheduling, which means that in practice, GStreamer uses a push-based
scheduling model.
调度机制，对组使用一个线程，组是指被队列分隔的元素集。调度可能是推模式（向下流输出数据）的或者拉模式（从上流获得数据）的。如果某个元素
支持随即访问，那么下一个元素或成为它们这个组的入口，获得和输出，处理数据。

Autoplugging

自动插件可以为应用程序提供多种文件的识别机制。
MIME-types as a
way to identity streams

We have

explained that a capability is a combination of a mimetype and a set of
properties. For most container

formats (those are the files that you will find on your hard disk; Ogg,
for example, is a container format),

no properties are needed to describe the stream. Only a MIME-type is
needed。
接受力是MIMETYPE和属性集的组合。对于大多数容器来说，属性并不是必须得，只需要MIME-type来描述流。

Pipeline
manipulation

This chapter will discuss how you can manipulate your pipeline in
several ways from your application

on. Parts of this chapter are downright hackish, so be assured that
you’ll need some programming

knowledge before you start reading this.

Topics that will be discussed here include how you can insert data into a
pipeline from your application,

how to read data from a pipeline, how to manipulate the pipeline’s
speed, length, starting point and how

to listen to a pipeline’s data processing.
Data probing
数据探测――
焊盘的监视作用，当有数据来的时候，你能获取它的缓冲区指针。
Manually adding or removing data from/to a pipeline
手动添加或删除管道数据
Embedding static elements in your application
静态元素（而不是动态加载库）