Painting video with GStreamer and Qt/QML or Gtk+ with overlay
2015-06-04 13:24
1126 查看
如果网络条件允许,建议看原文。原文地址:http://blogs.igalia.com/xrcalvar/2012/04/29/painting-video-with-gstreamer-and-qtqml-or-gtk-with-overlay/
As part of my work at Igalia I had to work with video and GStreamer for
some years. I always used Gtk+ for that so when I needed to do things with Qt and QML,
things were different. In my projects I always used pure GStreamer code instead of the Qt bindings for GStreamer because at the moment those bindings were not ready or reliable.
I know two ways of painting video:
Overlay way, with a window id and so on
Texture streaming
I might write later about texture streaming, but I will focus now on overlay.
The first way means that you need from your graphical toolkit a window id. That window id is asked by the video sink element in a very special moment and you need to provide it in that moment if you have not provided it before. For example, if you are using playbin2 and
you already know the sink you want to use, just instantiate your sink and set the window id at that moment withgst_x_overlay_set_window_handle and
set the sink to the playbin2 element by setting the
If you are not using playbin2 and for example you are using GStreamer
Editing Services, you cannot use a property because currently there is no one and need to use a more complicated method. I already reported the bug
with its patches and hope that they apply them as soon as possible to improve compatibility with playbin2 because the way it is now is a bit inconsistent with the rest of GStreamer code base.
Both Qt and Gtk have now client side windows, which means that your program window has only one X window and it is the toolkit that decides which widget is receiving the events. The main consequence is that if we just set the window id, GStreamer will use the
whole window and will paint the video over the rest of our widgets (it does not matter if QML/Qt or Gtk+) and you’ll get very ugly effects. To solve that, you need to set the render rectangle, which are the coordinates
(relative to the X whole X window) where you want to paint your video. You need to do that just after setting the window id withgst_x_overlay_set_render_rectangle.
If you do not set your window handle and your render rectangle before the pipeline begins to move, it will ask you about that with the
but this message can happen inside the GStreamer threads and it cannot wait until the main loop runs, it needs the information at that very moment, so you need to connect to the synchronous bus handle. GStreamer has a good example at theGstXOverlay documentation
about how to do that. To use the callback in C++, you need to declare a static method and pass
used in the GNOME world and fits perfectly with the Qt framework too.
The code to get the window id and render rectangle in Gtk+ would be something like:
In Qt, if you are using common widgets, you could use something like:
If you are using a
If you are using QML, you would have a very similar approach to the last snippet, because as you should have a
something like
Some times it is nice do put your controls on top of the video by covering part of the image. It would be like having the video as the background of a canvas where you draw some other widgets. Some GStreamer elements give you the possibility of doing a trick
to do this, which is using a colorkey for your background and painting whatever you want on top of that as long as it does not include that colorkey. Some elements like xvimagesink or omapxvsink (used
in the Nokia N9 andN950)
have the
of your widget and a good moment is also when setting the window handle:
Why do I unset the colorkey autopainting? Because I do not want GStreamer to mess my widget painting.
And more important: Why did I use
ugly artifacts. Some people recommend magenta (
You will need to synchronize your painting very well to avoid seeing the colorkey
If you respect aspect ratio you will see it for sure, because you (or the sink if it is automatic) paint the backgound and the sink draws the image by leaving some empty space.
It does not behave well with blendings, as you blend from your widget color to the background, which is the colorkey
Advice: do not mess with colorkey and
There are two kind of people:
The ones that want to use all the pixels of their monitor/TVs and like damaging their brain with distorted images.
The ones that like to see a correctly dimensioned image with some bars giving you a better impression of what was recorded.
As you can guess I belong to the second group.
There are some sinks that do that automatically for you by setting the
there are other that does not and omapxvsink is an example. It is not a big problem but forces you to work a bit more when you select therender rectangle. For that you
need to know the video size, which you cannot know until the pipeline is running, which forces to to hook to the GST_MESSAGE_ASYNC_DONE,
or in the case of playbin2, you already have the video size when getting the
This entry was posted on Sunday, April 29th, 2012 at 1:58 am and is filed under
As part of my work at Igalia I had to work with video and GStreamer for
some years. I always used Gtk+ for that so when I needed to do things with Qt and QML,
things were different. In my projects I always used pure GStreamer code instead of the Qt bindings for GStreamer because at the moment those bindings were not ready or reliable.
I know two ways of painting video:
Overlay way, with a window id and so on
Texture streaming
I might write later about texture streaming, but I will focus now on overlay.
Painting
The first way means that you need from your graphical toolkit a window id. That window id is asked by the video sink element in a very special moment and you need to provide it in that moment if you have not provided it before. For example, if you are using playbin2 andyou already know the sink you want to use, just instantiate your sink and set the window id at that moment withgst_x_overlay_set_window_handle and
set the sink to the playbin2 element by setting the
video-sinkproperty.
If you are not using playbin2 and for example you are using GStreamer
Editing Services, you cannot use a property because currently there is no one and need to use a more complicated method. I already reported the bug
with its patches and hope that they apply them as soon as possible to improve compatibility with playbin2 because the way it is now is a bit inconsistent with the rest of GStreamer code base.
Both Qt and Gtk have now client side windows, which means that your program window has only one X window and it is the toolkit that decides which widget is receiving the events. The main consequence is that if we just set the window id, GStreamer will use the
whole window and will paint the video over the rest of our widgets (it does not matter if QML/Qt or Gtk+) and you’ll get very ugly effects. To solve that, you need to set the render rectangle, which are the coordinates
(relative to the X whole X window) where you want to paint your video. You need to do that just after setting the window id withgst_x_overlay_set_render_rectangle.
If you do not set your window handle and your render rectangle before the pipeline begins to move, it will ask you about that with the
prepare-xwindow-idGstMessage,
but this message can happen inside the GStreamer threads and it cannot wait until the main loop runs, it needs the information at that very moment, so you need to connect to the synchronous bus handle. GStreamer has a good example at theGstXOverlay documentation
about how to do that. To use the callback in C++, you need to declare a static method and pass
thisas user data parameter, then you can behave almost as having a normal object method. This is the most common solution
used in the GNOME world and fits perfectly with the Qt framework too.
The code to get the window id and render rectangle in Gtk+ would be something like:
GdkWindow *gdk_window; gdk_window = gtk_widget_get_window(your_widget); /* as sink you can use GST_MESSAGE_SRC() if you are waiting for the prepare-xwindow-id message */ gst_x_overlay_set_xwindow_id(GST_X_OVERLAY(your_sink), GDK_WINDOW_XID(gdk_window)); /* do your maths about your coordinates */ gst_x_overlay_set_window_handle(GST_X_OVERLAY(sink), x, y, width, height);
In Qt, if you are using common widgets, you could use something like:
WId winId = QApplication::activeWindow()->effectiveWinId(); gst_x_overlay_set_xwindow_id(GST_X_OVERLAY(your_sink), winId); /* do your maths about your coordinates */ gst_x_overlay_set_window_handle(GST_X_OVERLAY(sink), x, y, width, height);
If you are using a
QGraphicsSceneyou would do something like:
/* to get the view you could do something like this (if you have only one or will to mess things up): QGraphicsView *view = your_scene.views[0]; */ gst_x_overlay_set_xwindow_id(GST_X_OVERLAY(your_sink), view->viewport()->effectiveWinId()); /* do your maths about your coordinates */ gst_x_overlay_set_window_handle(GST_X_OVERLAY(sink), x, y, width, height);
If you are using QML, you would have a very similar approach to the last snippet, because as you should have a
QDeclarativeItem, it has a
scene()that you can use, to have
something like
QGraphicsView *view = scene().views[0];(of course, assuming that you have only one view, which is the most common case).
Overlaying stuff
Some times it is nice do put your controls on top of the video by covering part of the image. It would be like having the video as the background of a canvas where you draw some other widgets. Some GStreamer elements give you the possibility of doing a trickto do this, which is using a colorkey for your background and painting whatever you want on top of that as long as it does not include that colorkey. Some elements like xvimagesink or omapxvsink (used
in the Nokia N9 andN950)
have the
colorkeyproperty that you can read and set. If you are not planning to overlay anything, you can forget about this, but if you do, you need set a color key to the sink and use that color to paint the background
of your widget and a good moment is also when setting the window handle:
g_object_set(sink, "autopaint-colorkey", FALSE, "colorkey", 0x080810, NULL);
Why do I unset the colorkey autopainting? Because I do not want GStreamer to mess my widget painting.
And more important: Why did I use
0x080810? Because it is a dark color, close to black, but it is not black. Pure black can be dangerous as it is commonly used in themes when painting widgets so you would be getting
ugly artifacts. Some people recommend magenta (
0xFF00FF) as it is supposedly a color that does not exist in nature (citation needed). I would not do it for several reasons:
You will need to synchronize your painting very well to avoid seeing the colorkey
If you respect aspect ratio you will see it for sure, because you (or the sink if it is automatic) paint the backgound and the sink draws the image by leaving some empty space.
It does not behave well with blendings, as you blend from your widget color to the background, which is the colorkey
Advice: do not mess with colorkey and
omapxvsink. Though it is supposed to be writable, it is not and it always uses
0x080810.
Aspect ratio
There are two kind of people:The ones that want to use all the pixels of their monitor/TVs and like damaging their brain with distorted images.
The ones that like to see a correctly dimensioned image with some bars giving you a better impression of what was recorded.
As you can guess I belong to the second group.
There are some sinks that do that automatically for you by setting the
force-aspect-ratioproperty, like ximagesink and xvimagesink but
there are other that does not and omapxvsink is an example. It is not a big problem but forces you to work a bit more when you select therender rectangle. For that you
need to know the video size, which you cannot know until the pipeline is running, which forces to to hook to the GST_MESSAGE_ASYNC_DONE,
or in the case of playbin2, you already have the video size when getting the
prepare-xwindow-idmessage. An example to get the video size would be:
GstPad *pad; GstCaps *caps; GstStructure *structure; int width, height; pad = GST_BASE_SINK_PAD(sink); caps = GST_PAD_CAPS(pad); g_return_if_fail(caps && gst_caps_is_fixed(caps)); structure = gst_caps_get_structure(caps, 0); gst_structure_get_int(structure, "width", &width); gst_structure_get_int(structure, "height", &height); /* some videos define a pixel aspect ratio, meaning that the video pixel could be like 2x1 copared to a squared pixed and we need to correct this */ if (gst_structure_has_field(structure, "pixel-aspect-ratio")) { int par_n, par_d; gst_structure_get_fraction(structure, "pixel-aspect-ratio", &par_n, &par_d); width = width * par_n / par_d; } /* trick: some sinks perform better with multiple of 2 */ width &= ~1; height &= ~1;
This entry was posted on Sunday, April 29th, 2012 at 1:58 am and is filed under
相关文章推荐
- Qt学习记录--Qt::CaseSensitive
- Qt触摸屏原理及实现
- Qt 线程基础
- QT messagebox 乱码解决
- Qt tableWidget 空单元格 获取选中行行号
- 十六、Qt 2D绘图(六)坐标系统
- QT项目笔记
- win7下配置OpenCV的Qt开发环境
- 大量的QT控件及示例发放
- qt字体设置
- Qt(2):MOC文件解析
- Ubuntu14.04 LTS安装 OpenCV-3.0.0-rc1 + QT5.4.1
- Tiny6410 && ubuntu 14.04 运行qt程序
- QTableWidget
- QML和C++混合编程--Qt声明式用户界面运行环境
- qt按键焦点切换
- Qt多线程学习:创建多线程
- 解决QT用着用着就不好使的问题
- opencv & qt study-(3)-图像的容器--Mat
- Qt5.3 for vs2013 中文乱码解决办法