直播技术(从服务端到客户端)二
2016-09-21 14:31
351 查看
播放(播放方式研究)
在上一篇文章中,我们叙述了直播技术的环境配置(包括服务端nginx,nginx-rtmp-module, ffmpeg,Android编译,ios编译)。从本文开始,我们将叙述播放相关的东西,播放是直播技术中关键的一步,它包括很多技术如:解码,缩放,时间基线选择,缓存队列,画面渲染,声音播放等等。我将分为三个部分为大家讲述整个播放流程;
Android
第一部分是基于NativeWindow的视频渲染,主要使用的OpenGL ES2通过传入surface来将视频数据渲染到surface上显示出来。第二部分是基于OpenSL ES来音频播放。第三部分,音视频同步。我们使用的都是android原生自带的一些库来做音视频渲染处理。
IOS
同样IOS也分成三个部分,第一部分视频渲染:使用OpenGLES.framework,通过OpenGL来渲染视频画面,第二部分是音频播放,基于AudioToolbox.framework做音频播放;第三部分,视音频同步。
利用原生库可以减少资源的利用,降低内存,提高性能;一般而言,如果不是通晓android、ios的程序员会选择一个统一的视频显示和音频播放库(SDL),这个库可以实现视频显示和音频播。但是增加额外的库意味着资源的浪费和性能的降低。
Android
我们首先带来android端的视频播放功能,我们分成三个部分,1、视频渲染;2、音频播放;3、时间基线(音视频同步)来阐述。1、视频渲染
ffmpeg为我们提供浏览丰富的编解码类型(ffmpeg所具备编解码能力都是软件编解码,不是指硬件编解码。具体之后文章会详细介绍ffmpeg),视频解码包括flv, mpeg, mov 等;音频包括aac, mp3等。对于整个播放,FFmpeg主要处理流程如下:<code class="language-C++ hljs scss has-numbering"> <span class="hljs-function">av_register_all()</span>; <span class="hljs-comment">// 注册所有的文件格式和编解码器的库,打开的合适格式的文件上会自动选择相应的编解码库</span> <span class="hljs-function">avformat_network_init()</span>; <span class="hljs-comment">// 注册网络服务</span> <span class="hljs-function">avformat_alloc_context()</span>; <span class="hljs-comment">// 分配FormatContext内存,</span> <span class="hljs-function">avformat_open_input()</span>; <span class="hljs-comment">// 打开输入流,获取头部信息,配合av_close_input_file()关闭流</span> <span class="hljs-function">avformat_find_stream_info()</span>; <span class="hljs-comment">// 读取packets,来获取流信息,并在pFormatCtx->streams 填充上正确的信息</span> <span class="hljs-function">avcodec_find_decoder()</span>; <span class="hljs-comment">// 获取解码器,</span> <span class="hljs-function">avcodec_open2()</span>; <span class="hljs-comment">// 通过AVCodec来初始化AVCodecContext</span> <span class="hljs-function">av_read_frame()</span>; <span class="hljs-comment">// 读取每一帧</span> <span class="hljs-function">avcodec_decode_video2()</span>; <span class="hljs-comment">// 解码帧数据</span> <span class="hljs-function">avcodec_close()</span>; <span class="hljs-comment">// 关闭编辑器上下文</span> <span class="hljs-function">avformat_close_input()</span>; <span class="hljs-comment">// 关闭文件流</span></code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li></ul>
我们先来看一段代码:
<code class="language-C++ hljs php has-numbering">av_register_all(); avformat_network_init(); pFormatCtx = avformat_alloc_context(); <span class="hljs-keyword">if</span> (avformat_open_input(&pFormatCtx, pathStr, <span class="hljs-keyword">NULL</span>, <span class="hljs-keyword">NULL</span>) != <span class="hljs-number">0</span>) { LOGE(<span class="hljs-string">"Couldn't open file: %s\n"</span>, pathStr); <span class="hljs-keyword">return</span>; } <span class="hljs-keyword">if</span> (avformat_find_stream_info(pFormatCtx, &dictionary) < <span class="hljs-number">0</span>) { LOGE(<span class="hljs-string">"Couldn't find stream information."</span>); <span class="hljs-keyword">return</span>; } av_dump_format(pFormatCtx, <span class="hljs-number">0</span>, pathStr, <span class="hljs-number">0</span>); </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul>
这段代码可以算是初始化FFmpeg,首先注册编解码库,为FormatContext分配内存,调用avformat_open_input打开输入流,获取头部信息,配合avformat_find_stream_info来填充FormatContext中相关内容,av_dump_format这个是dump出流信息。这个信息是这个样子的:
<code class="language-text hljs lasso has-numbering">video infomation: Input <span class="hljs-variable">#0</span>, flv, from <span class="hljs-string">'rtmp:127.0.0.1:1935/live/steam'</span>: Metadata: Server : NGINX RTMP (github<span class="hljs-built_in">.</span>com/sergey<span class="hljs-attribute">-dryabzhinsky</span>/nginx<span class="hljs-attribute">-rtmp</span><span class="hljs-attribute">-module</span>) displayWidth : <span class="hljs-number">320</span> displayHeight : <span class="hljs-number">240</span> fps : <span class="hljs-number">15</span> profile : level : <span class="hljs-built_in">Duration</span>: <span class="hljs-number">00</span>:<span class="hljs-number">00</span>:<span class="hljs-number">00.00</span>, start: <span class="hljs-number">15.400000</span>, bitrate: N/A Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">0</span>: Video: flv1 (flv), yuv420p, <span class="hljs-number">320</span>x240, <span class="hljs-number">15</span> tbr, <span class="hljs-number">1</span>k tbn, <span class="hljs-number">1</span>k tbc Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">1</span>: Audio: mp3, <span class="hljs-number">11025</span> Hz, stereo, s16p, <span class="hljs-number">32</span> kb/s</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul>
接下来就是找到视频的解码器上下文AVCodecContext和AVCodec解码器。
<code class="language-C++ hljs java has-numbering"> videoStream = -<span class="hljs-number">1</span>; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i < pFormatCtx->nb_streams; i++) { <span class="hljs-keyword">if</span> (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) { videoStream = i; <span class="hljs-keyword">break</span>; } } <span class="hljs-keyword">if</span> (videoStream == -<span class="hljs-number">1</span>) { LOGE(<span class="hljs-string">"Didn't find a video stream."</span>); <span class="hljs-keyword">return</span>; } <span class="hljs-javadoc">/** 获取视频解码器上下文和解码器 */</span> pVideoCodecCtx = pFormatCtx->streams[videoStream]->codec; pVideoCodec = avcodec_find_decoder(pVideoCodecCtx->codec_id); <span class="hljs-keyword">if</span> (pVideoCodec == NULL) { LOGE(<span class="hljs-string">"pVideoCodec not found."</span>); <span class="hljs-keyword">return</span>; } <span class="hljs-javadoc">/** 获取视频宽度和高度 */</span> width = pVideoCodecCtx->width; height = pVideoCodecCtx->height; <span class="hljs-javadoc">/** 获取设置NativeWindow buffer属性, */</span> ANativeWindow_setBuffersGeometry(nativeWindow, width, height, WINDOW_FORMAT_RGBA_8888); LOGD(<span class="hljs-string">"the width is %d, the height is %d"</span>, width, height); <span class="hljs-comment">// 分配每一帧的内存,pFrame原始帧,pFrameRGB为转换帧</span> pFrame = avcodec_alloc_frame(); pFrameRGB = avcodec_alloc_frame(); <span class="hljs-keyword">if</span> (pFrameRGB == NULL || pFrameRGB == NULL) { LOGE(<span class="hljs-string">"Could not allocate video frame."</span>); <span class="hljs-keyword">return</span>; } <span class="hljs-keyword">int</span> numBytes = avpicture_get_size(PIX_FMT_RGBA, pVideoCodecCtx->width, pVideoCodecCtx->height); buffer = (uint8_t *) av_malloc(numBytes * sizeof(uint8_t)); <span class="hljs-keyword">if</span> (buffer == NULL) { LOGE(<span class="hljs-string">"buffer is null"</span>); <span class="hljs-keyword">return</span>; } <span class="hljs-comment">// 填充AVPicture信息</span> avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGBA, pVideoCodecCtx->width, pVideoCodecCtx->height); <span class="hljs-comment">// 获取视频缩放上下文</span> pSwsCtx = sws_getContext(pVideoCodecCtx->width, pVideoCodecCtx->height, pVideoCodecCtx->pix_fmt, width, height, PIX_FMT_RGBA, SWS_BILINEAR, NULL, NULL, NULL); <span class="hljs-keyword">int</span> frameFinished = <span class="hljs-number">0</span>; <span class="hljs-comment">// 循环读取每一帧</span> <span class="hljs-keyword">while</span> (av_read_frame(pFormatCtx, &packet) >= <span class="hljs-number">0</span>) { <span class="hljs-keyword">if</span> (packet.stream_index == videoStream) { <span class="hljs-comment">// 解码每一帧数据</span> avcodec_decode_video2(pVideoCodecCtx, pFrame, &frameFinished, &packet); <span class="hljs-keyword">if</span> (frameFinished) { LOGD(<span class="hljs-string">"av_read_frame"</span>); <span class="hljs-comment">// 调用NativeWindows展示画面</span> ANativeWindow_lock(nativeWindow, &windowBuffer, <span class="hljs-number">0</span>); <span class="hljs-comment">// 缩放视频</span> sws_scale(pSwsCtx, (uint8_t <span class="hljs-keyword">const</span> * <span class="hljs-keyword">const</span> *)pFrame->data, pFrame->linesize, <span class="hljs-number">0</span>, pVideoCodecCtx->height, pFrameRGB->data, pFrameRGB->linesize); uint8_t * dst = (uint8_t *)windowBuffer.bits; <span class="hljs-keyword">int</span> dstStride = windowBuffer.stride * <span class="hljs-number">4</span>; uint8_t * src = (pFrameRGB->data[<span class="hljs-number">0</span>]); <span class="hljs-keyword">int</span> srcStride = pFrameRGB->linesize[<span class="hljs-number">0</span>]; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> h = <span class="hljs-number">0</span>; h < height; h++) { memcpy(dst + h * dstStride, src + h * srcStride, srcStride); } ANativeWindow_unlockAndPost(nativeWindow); } } av_free_packet(&packet); <span class="hljs-comment">// 释放packet</span> }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li></ul>
从整个解码到缩放,再到渲染到nativeWindow思路非常清晰,当然我们这里没有考虑时间基线,这就意味着播放的时候会很快或者很慢(这个取决于视频的帧率)。我们在第三部分详细讨论时间基线(音视频同步问题)。
2、音频渲染
音频渲染我们同样适用的是android原生的库,OpenSL ES。我们会使用一些主要的接口参数如下:<code class="language-C++ hljs cs has-numbering"><span class="hljs-comment">// engine interfaces</span> <span class="hljs-keyword">static</span> SLObjectItf engineObject; <span class="hljs-keyword">static</span> SLEngineItf engineEngine; <span class="hljs-comment">// output mix interfaces</span> <span class="hljs-keyword">static</span> SLObjectItf outputMixObject; <span class="hljs-keyword">static</span> SLEnvironmentalReverbItf outputMixEnvironmentalReverb; <span class="hljs-keyword">static</span> SLObjectItf bqPlayerObject; <span class="hljs-keyword">static</span> SLEffectSendItf bqPlayerEffectSend; <span class="hljs-keyword">static</span> SLMuteSoloItf bqPlayerMuteSolo; <span class="hljs-keyword">static</span> SLVolumeItf bqPlayerVolume; <span class="hljs-keyword">static</span> SLPlayItf bqPlayerPlay; <span class="hljs-keyword">static</span> SLAndroidSimpleBufferQueueItf bqPlayerBufferQueue; <span class="hljs-comment">// aux effect on the output mix, used by the buffer queue player</span> <span class="hljs-keyword">const</span> <span class="hljs-keyword">static</span> SLEnvironmentalReverbSettings reverbSettings = SL_I3DL2_ENVIRONMENT_PRESET_STONECORRIDOR;</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li></ul>
这些参数都是音频播放过程使用到的,而整个音频播放的流程主要包括以下几个方面:
<code class="language-C++ hljs coffeescript has-numbering"> <span class="hljs-regexp">//</span>创建播放引擎,初始化接口参数。 bool createEngine() { SLresult result; <span class="hljs-regexp">//</span> 创建引擎engineObject result = slCreateEngine(&engineObject, <span class="hljs-number">0</span>, NULL, <span class="hljs-number">0</span>, NULL, NULL); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> 实现引擎engineObject <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineObject)</span>-></span>Realize(engineObject, SL_BOOLEAN_FALSE); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> 获取引擎接口engineEngine <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineObject)</span>-></span>GetInterface(engineObject, SL_IID_ENGINE, &engineEngine); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> <span class="hljs-reserved">const</span> SLInterfaceID ids[<span class="hljs-number">1</span>] = {SL_IID_ENVIRONMENTALREVERB}; <span class="hljs-reserved">const</span> SLboolean req[<span class="hljs-number">1</span>] = {SL_BOOLEAN_FALSE}; <span class="hljs-regexp">//</span> 创建混音器outputMixObject <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineEngine)</span>-></span>CreateOutputMix(engineEngine, &outputMixObject, <span class="hljs-number">1</span>, ids, req); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> 实现混音器outputMixObject <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixObject)</span>-></span>Realize(outputMixObject, SL_BOOLEAN_FALSE); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> 获取混音器接口outputMixEnvironmentalReverb <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixObject)</span>-></span>GetInterface(outputMixObject, SL_IID_ENVIRONMENTALREVERB, &outputMixEnvironmentalReverb); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS == result) { <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*outputMixEnvironmentalReverb)</span>-></span>SetEnvironmentalReverbProperties( outputMixEnvironmentalReverb, &reverbSettings); } <span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>; }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul>
在创建OpenSL ES音频播放引擎的时候,我们主要针对引擎和混音器进行初始化。之后我们会创建音频播放缓冲。
<code class="language-C++ hljs coffeescript has-numbering">bool createBufferQueueAudioPlayer(PlayCallBack callback) { SLresult result; SLDataLocator_AndroidSimpleBufferQueue loc_bufq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, <span class="hljs-number">2</span>}; <span class="hljs-regexp">//</span> pcm数据格式 SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM, <span class="hljs-number">1</span>, SL_SAMPLINGRATE_44_1, SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16, SL_SPEAKER_FRONT_CENTER, SL_BYTEORDER_LITTLEENDIAN}; SLDataSource audioSrc = {&loc_bufq, &format_pcm}; SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject}; SLDataSink audioSnk = {&loc_outmix, NULL}; <span class="hljs-reserved">const</span> SLInterfaceID ids[<span class="hljs-number">3</span>] = {SL_IID_BUFFERQUEUE, SL_IID_EFFECTSEND, SL_IID_VOLUME}; <span class="hljs-reserved">const</span> SLboolean req[<span class="hljs-number">3</span>] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE}; <span class="hljs-regexp">//</span> 创建音频播放器 <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*engineEngine)</span>-></span>CreateAudioPlayer(engineEngine, &bqPlayerObject, &audioSrc, &audioSnk, <span class="hljs-number">3</span>, ids, req); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 实现音频播放器(bqPlayerObject) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>Realize(bqPlayerObject, SL_BOOLEAN_FALSE); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerPlay(获取播放器) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_PLAY, &bqPlayerPlay); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerBufferQueue(缓冲buffer) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_BUFFERQUEUE, &bqPlayerBufferQueue); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 注册播放回调 callback <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerBufferQueue)</span>-></span>RegisterCallback(bqPlayerBufferQueue, callback, NULL); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) { <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; } <span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerEffectSend(音效) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_EFFECTSEND, &bqPlayerEffectSend); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 获取音频播放器接口bqPlayerVolume(音量) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerObject)</span>-></span>GetInterface(bqPlayerObject, SL_IID_VOLUME, &bqPlayerVolume); <span class="hljs-keyword">if</span> (SL_RESULT_SUCCESS != result) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>; <span class="hljs-regexp">//</span> 设置播放bqPlayerPlay状态(暂停) <span class="hljs-function"><span class="hljs-title">result</span> = <span class="hljs-params">(*bqPlayerPlay)</span>-></span>SetPlayState(bqPlayerPlay, SL_PLAYSTATE_STOPPED); <span class="hljs-keyword">return</span> result == SL_RESULT_SUCCESS; }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li></ul>
整个音频播放流畅其实看起来也是很简单的,主要分:1、创建实现播放引擎;2、创建实现混音器;3、设置缓冲和pcm格式;4、创建实现播放器;5、获取音频播放器接口;6、获取缓冲buffer;7、注册播放回调;8、获取音效接口;9、获取音量接口;10、获取播放状态接口;
做完这10步,整个音频播放器引擎就创建完毕,接下来就是引擎读取数据播放。
<code class="language-C++ hljs objectivec has-numbering"><span class="hljs-keyword">void</span> playBuffer(<span class="hljs-keyword">void</span> *pBuffer, <span class="hljs-keyword">int</span> size) { <span class="hljs-comment">// 判断数据可用性</span> <span class="hljs-keyword">if</span> (pBuffer == <span class="hljs-literal">NULL</span> || size == -<span class="hljs-number">1</span>) { <span class="hljs-keyword">return</span>; } LOGV(<span class="hljs-string">"PlayBuff!"</span>); <span class="hljs-comment">// 数据存放进bqPlayerBufferQueue中</span> SLresult result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, pBuffer, size); <span class="hljs-keyword">if</span> (result != SL_RESULT_SUCCESS) LOGE(<span class="hljs-string">"Play buffer error!"</span>); }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li></ul>
这段代码主要阐述的播放的过程,通过将数据放进bqPlayerBufferQueue,供播放引擎读取播放。记得我们在创建缓冲buffer的时候,注册了一个callback,这个callBack的作用就是通知可以向缓冲队列中添加数据,这个callBack的原型如下:
<code class="hljs lasso has-numbering"><span class="hljs-literal">void</span> videoPlayCallBack(SLAndroidSimpleBufferQueueItf bq, <span class="hljs-literal">void</span> <span class="hljs-subst">*</span>context) { <span class="hljs-comment">// 添加数据到bqPlayerBufferQueue中,通过调用playBuffer方法。</span> <span class="hljs-literal">void</span><span class="hljs-subst">*</span> <span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> getData(); int size <span class="hljs-subst">=</span> getDataSize(); playBuffer(<span class="hljs-built_in">data</span>, size); }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li></ul>
这样就循环向缓冲队列中添加数据,可以一直播放。有很多简单的音乐播放器就是基于这种模式设计的。这种方式也很可靠,能够非常清楚的展现整个播放流程。
3、时间基线(音视频同步)
为了能够完整的播放视频和音频,我们需要对一些数据进行整合,包括音视频数据,时间参数等。在ffmpeg源码的ffplayer.c使用就是这种方式,对齐时间基线有三种方式:1、对齐音频时间基线;2、对齐视频时间基线;3、对齐第三方时间基线。为了能够能够清楚展现播放速度控制和音视频同步,特意给出了一个流程图。音视频队列从ffmpeg的av_read_frame中读取每一帧数据到音视频队列中(保存的是AVPacket数据),然后音频队列和视频队列从音视频队列中不停的拿数据,期间两者做一个同步,最后展示到画面和播放音频。<code class="hljs lasso has-numbering"><span class="hljs-literal">void</span> getPacket() { struct timespec time; time<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">=</span> <span class="hljs-number">10</span>;<span class="hljs-comment">//网络不好最多等10秒</span> time<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">=</span> <span class="hljs-number">0</span>; struct ThreadMsg msg; <span class="hljs-keyword">while</span> (<span class="hljs-literal">true</span>) { memset(<span class="hljs-subst">&</span>msg, <span class="hljs-number">0</span>, sizeof(struct ThreadMsg)); msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> <span class="hljs-built_in">NULL</span>; <span class="hljs-comment">// 等待网络缓冲</span> ThreadList<span class="hljs-tag">::queueGet</span>(playInstance<span class="hljs-subst">-></span><span class="hljs-built_in">queue</span>, <span class="hljs-subst">&</span>time, <span class="hljs-subst">&</span>msg); <span class="hljs-keyword">if</span> (msg<span class="hljs-built_in">.</span>msgtype <span class="hljs-subst">==</span> <span class="hljs-subst">-</span><span class="hljs-number">1</span>) {<span class="hljs-comment">//正常退出</span> ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>); ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>); break; } <span class="hljs-keyword">if</span> (msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span> <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) { ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>); ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, <span class="hljs-built_in">NULL</span>, <span class="hljs-subst">-</span><span class="hljs-number">1</span>); playInstance<span class="hljs-subst">-></span>timeout_flag <span class="hljs-subst">=</span> <span class="hljs-number">1</span>; break; } AVPacket <span class="hljs-subst">*</span>packet_p <span class="hljs-subst">=</span> (AVPacket <span class="hljs-subst">*</span>) msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span>; <span class="hljs-keyword">if</span> (packet_p<span class="hljs-subst">-></span>stream_index <span class="hljs-subst">==</span> playInstance<span class="hljs-subst">-></span>videoState<span class="hljs-subst">-></span>videoStream) { ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>video_queue, packet_p, <span class="hljs-number">1</span>); } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet_p<span class="hljs-subst">-></span>stream_index <span class="hljs-subst">==</span> playInstance<span class="hljs-subst">-></span>videoState<span class="hljs-subst">-></span>audioStream) { ThreadList<span class="hljs-tag">::queueAdd</span>(playInstance<span class="hljs-subst">-></span>audio_queue, packet_p, <span class="hljs-number">1</span>); } } }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li></ul>
<code class="hljs lasso has-numbering">int queueGet(struct ThreadQueue <span class="hljs-subst">*</span><span class="hljs-built_in">queue</span>, const struct timespec <span class="hljs-subst">*</span>timeout, struct ThreadMsg <span class="hljs-subst">*</span>msg) { struct MsgList <span class="hljs-subst">*</span>firstrec; int ret <span class="hljs-subst">=</span> <span class="hljs-number">0</span>; struct timespec abstimeout; <span class="hljs-keyword">if</span> (<span class="hljs-built_in">queue</span> <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span> <span class="hljs-subst">||</span> msg <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) { <span class="hljs-keyword">return</span> EINVAL; } <span class="hljs-keyword">if</span> (timeout) { struct timeval now; gettimeofday(<span class="hljs-subst">&</span>now, <span class="hljs-built_in">NULL</span>); abstimeout<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">=</span> now<span class="hljs-built_in">.</span>tv_sec <span class="hljs-subst">+</span> timeout<span class="hljs-subst">-></span>tv_sec; abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">=</span> (now<span class="hljs-built_in">.</span>tv_usec <span class="hljs-subst">*</span> <span class="hljs-number">1000</span>) <span class="hljs-subst">+</span> timeout<span class="hljs-subst">-></span>tv_nsec; <span class="hljs-keyword">if</span> (abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">>=</span> <span class="hljs-number">1000000000</span>) { abstimeout<span class="hljs-built_in">.</span>tv_sec<span class="hljs-subst">++</span>; abstimeout<span class="hljs-built_in">.</span>tv_nsec <span class="hljs-subst">-=</span> <span class="hljs-number">1000000000</span>; } } pthread_mutex_lock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex); <span class="hljs-comment">/* Will wait until awakened by a signal or broadcast */</span> <span class="hljs-keyword">while</span> (<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span> <span class="hljs-subst">&&</span> ret <span class="hljs-subst">!=</span> ETIMEDOUT) { <span class="hljs-comment">//Need to loop to handle spurious wakeups</span> <span class="hljs-keyword">if</span> (timeout) { ret <span class="hljs-subst">=</span> pthread_cond_timedwait(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>cond, <span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex, <span class="hljs-subst">&</span>abstimeout); } <span class="hljs-keyword">else</span> { pthread_cond_wait(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>cond, <span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex); } } <span class="hljs-keyword">if</span> (ret <span class="hljs-subst">==</span> ETIMEDOUT) { pthread_mutex_unlock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex); <span class="hljs-keyword">return</span> ret; } firstrec <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first; <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first<span class="hljs-subst">-></span>next; <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length<span class="hljs-subst">--</span>; <span class="hljs-keyword">if</span> (<span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>first <span class="hljs-subst">==</span> <span class="hljs-built_in">NULL</span>) { <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>last <span class="hljs-subst">=</span> <span class="hljs-built_in">NULL</span>; <span class="hljs-comment">// we know this since we hold the lock</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length <span class="hljs-subst">=</span> <span class="hljs-number">0</span>; } msg<span class="hljs-subst">-></span><span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> firstrec<span class="hljs-subst">-></span>msg<span class="hljs-built_in">.</span><span class="hljs-built_in">data</span>; msg<span class="hljs-subst">-></span>msgtype <span class="hljs-subst">=</span> firstrec<span class="hljs-subst">-></span>msg<span class="hljs-built_in">.</span>msgtype; msg<span class="hljs-subst">-></span>qlength <span class="hljs-subst">=</span> <span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>length; release_msglist(<span class="hljs-built_in">queue</span>, firstrec); pthread_mutex_unlock(<span class="hljs-subst">&</span><span class="hljs-built_in">queue</span><span class="hljs-subst">-></span>mutex); <span class="hljs-keyword">return</span> <span class="hljs-number">0</span>; }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li></ul>
我们分三个步骤进行操作。
整合数据
整合数据指的是讲音频数据和视频数据添加到相应的队列中,以便播放使用,具体如下;这个数据结构主要是针对音视频中一些基本参数的。我们将利用这些东西做解码,播放速度控制,音视频同步等等。
<code class="hljs d has-numbering"><span class="hljs-keyword">typedef</span> <span class="hljs-keyword">struct</span> VideoState { <span class="hljs-comment">//解码过程中的数据结构</span> AVFormatContext *pFormatCtx; AVCodecContext *pVideoCodecCtx; <span class="hljs-comment">// 视频解码器上下文</span> AVCodecContext *pAudioCodecCtx; <span class="hljs-comment">// 音频解码器上下文</span> AVCodec *pAudioCodec; <span class="hljs-comment">// 音频解码器</span> AVCodec *pVideoCodec; <span class="hljs-comment">// 视频解码器</span> AVFrame *pVideoFrame; <span class="hljs-comment">// 视频为转码帧</span> AVFrame *pVideoFrameRgba; <span class="hljs-comment">// 视频转码为rgba数据的帧</span> <span class="hljs-keyword">struct</span> SwsContext *sws_ctx; <span class="hljs-comment">// 视频转码工具</span> <span class="hljs-keyword">void</span> *pVideobuffer; <span class="hljs-comment">// 视频缓存</span> AVFrame *pAudioFrame;<span class="hljs-comment">// 音频帧</span> <span class="hljs-keyword">struct</span> SwrContext *swr_ctx; <span class="hljs-comment">// 音频转码工具</span> <span class="hljs-keyword">int</span> sample_rate_src; <span class="hljs-comment">//音频采样率</span> <span class="hljs-keyword">int</span> sample_fmt; <span class="hljs-comment">// 音频采用格式</span> <span class="hljs-keyword">int</span> sample_layout; <span class="hljs-comment">// 音频通道数</span> int64_t video_start_time; <span class="hljs-comment">// 视频开始时间</span> int64_t audio_start_time; <span class="hljs-comment">// 音频时间</span> <span class="hljs-built_in">double</span> video_time_base; <span class="hljs-comment">// 视频基准时间</span> <span class="hljs-built_in">double</span> audio_time_base; <span class="hljs-comment">// 音频基准时间</span> <span class="hljs-keyword">int</span> videoStream; <span class="hljs-comment">// FormatContext.streams中视频帧位置</span> <span class="hljs-keyword">int</span> audioStream; <span class="hljs-comment">// FormatContext.streams中音频帧位置</span> } VideoState; </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li></ul>
<code class="hljs cpp has-numbering"><span class="hljs-keyword">typedef</span> <span class="hljs-keyword">struct</span> PlayInstance { ANativeWindow *window; <span class="hljs-comment">// nativeWindow // 通过传入surface构建</span> <span class="hljs-keyword">int</span> display_width; <span class="hljs-comment">// 显示宽度</span> <span class="hljs-keyword">int</span> display_height; <span class="hljs-comment">// 显示高度</span> <span class="hljs-keyword">int</span> stop; <span class="hljs-comment">// 停止</span> <span class="hljs-keyword">int</span> timeout_flag; <span class="hljs-comment">// 超时标记</span> <span class="hljs-keyword">int</span> disable_video; VideoState *videoState; <span class="hljs-comment">//队列</span> <span class="hljs-keyword">struct</span> ThreadQueue *<span class="hljs-built_in">queue</span>; <span class="hljs-comment">// 音视频帧队列</span> <span class="hljs-keyword">struct</span> ThreadQueue *video_queue; <span class="hljs-comment">// 视频帧队列</span> <span class="hljs-keyword">struct</span> ThreadQueue *audio_queue; <span class="hljs-comment">// 音频帧队列</span> } PlayInstance;</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li></ul>
播放速度控制
播放速度控制可以通过获取音、视频队列中的数据来控制时间,我们是在单独的线程中从音、视频队列中获取相应的数据。然后根据数据时间基准线进行播放速度控制。这一块结合音视频同步一起实现。
音视频同步
下面代码是视频展示过程(在单独线程中做的),我们主要考虑的是延时同步那一块。将音频队列中的数据分类存放的音频队列和视频队列。
<code class="hljs haskell has-numbering"> <span class="hljs-title">void</span> video_thread() { struct timespec time; time.tv_sec = <span class="hljs-number">10</span>;//网络不好最多等<span class="hljs-number">10</span>秒 time.tv_nsec = <span class="hljs-number">0</span>; struct <span class="hljs-type">ThreadMsg</span> msg; int packet_count = <span class="hljs-number">0</span>; while (true) { <span class="hljs-keyword">if</span> (playInstance->stop) { break; } msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span> = <span class="hljs-type">NULL</span>;</span> // 从视频队列中获取数据,并等待 <span class="hljs-type">ThreadList</span>::queueGet(playInstance->video_queue, &time, &msg); <span class="hljs-keyword">if</span> (msg.msgtype == -<span class="hljs-number">1</span>) { break; } <span class="hljs-keyword">if</span> (msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span> == <span class="hljs-type">NULL</span>) <span class="hljs-container">{ <span class="hljs-type">LOGE</span>("视频线程空循环\<span class="hljs-title">n</span>"); <span class="hljs-title">break</span>; }</span></span> <span class="hljs-type">AVPacket</span> *packet_p = (<span class="hljs-type">AVPacket</span> *) msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>;</span> <span class="hljs-type">AVPacket</span> pavpacket = *packet_p; packet_count++; <span class="hljs-keyword">if</span> (packet_count == <span class="hljs-number">1</span>) {//拿到第一个视频包 playInstance->videoState->video_start_time = av_gettime(); <span class="hljs-type">LOGE</span>(<span class="hljs-string">"视频开始时间 %lld\n"</span>, playInstance->videoState->video_start_time); } <span class="hljs-keyword">if</span> (playInstance->disable_video) { av_free_packet(packet_p); av_free(msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>);</span> continue; } <span class="hljs-type">ANativeWindow_Buffer</span> windowBuffer; // 延时同步,控制播放速度。 int64_t pkt_pts = pavpacket.pts; double show_time = pkt_pts * (playInstance->videoState->video_time_base); int64_t show_time_micro = show_time * <span class="hljs-number">1000000</span>; int64_t played_time = av_gettime() - playInstance->videoState->video_start_time; int64_t delta_time = show_time_micro - played_time; <span class="hljs-keyword">if</span> (delta_time < -(<span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>)) { <span class="hljs-type">LOGE</span>(<span class="hljs-string">"视频跳帧\n"</span>); continue; } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (delta_time > <span class="hljs-number">0</span>) { av_usleep(delta_time); } int frame_finished = <span class="hljs-number">0</span>; avcodec_decode_video2(playInstance->videoState->pVideoCodecCtx, playInstance->videoState->pVideoFrame, &frame_finished, &pavpacket); <span class="hljs-keyword">if</span> (frame_finished) { sws_scale(//对解码后的数据进行色彩空间转换,yuv420p 转为rgba8888 playInstance->videoState->sws_ctx, (uint8_t const *const *) (playInstance->videoState->pVideoFrame)-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span> (playInstance->videoState->pVideoFrame)->linesize, <span class="hljs-number">0</span>, playInstance->videoState->pVideoCodecCtx->height, playInstance->videoState->pVideoFrameRgba-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span> playInstance->videoState->pVideoFrameRgba->linesize ); <span class="hljs-keyword">if</span> (!(playInstance->disable_video) && <span class="hljs-type">ANativeWindow_lock</span>(playInstance->window, &windowBuffer, <span class="hljs-type">NULL</span>) < <span class="hljs-number">0</span>) { <span class="hljs-type">LOGE</span>(<span class="hljs-string">"cannot lock window"</span>); continue; } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (!playInstance->disable_video) { uint8_t *dst = (uint8_t *) windowBuffer.bits; int dstStride = windowBuffer.stride * <span class="hljs-number">4</span>; uint8_t *src = (playInstance->videoState->pVideoFrameRgba-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0]);</span> int srcStride = playInstance->videoState->pVideoFrameRgba->linesize[<span class="hljs-number">0</span>]; for (int h = <span class="hljs-number">0</span>; h < playInstance->display_height; h++) { memcpy(dst + h * dstStride, src + h * srcStride, srcStride); } <span class="hljs-type">ANativeWindow_unlockAndPost</span>(playInstance->window);//释放对surface的锁,并且更新对应surface数据进行显示 } av_free_packet(packet_p); av_free(msg.<span class="hljs-typedef"><span class="hljs-keyword">data</span>);</span> } } }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li></ul>
我们主要分析延时同步的那一段代码:
<code class="hljs autohotkey has-numbering">// 延时同步 int64_t pkt_pts = pavpacket.pts<span class="hljs-comment">;</span> double show_time = pkt_pts * (playInstance->videoState->video_time_base)<span class="hljs-comment">;</span> int64_t show_time_micro = show_time * <span class="hljs-number">1000000</span><span class="hljs-comment">;</span> int64_t played_time = av_gettime() - playInstance->videoState->video_start_time<span class="hljs-comment">;</span> int64_t delt<span class="hljs-built_in">a_time</span> = show_time_micro - played_time<span class="hljs-comment">;</span> <span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> < -(<span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>)) { LOGE(<span class="hljs-string">"视频跳帧\n"</span>)<span class="hljs-comment">;</span> <span class="hljs-keyword">continue</span>; } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> > <span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>) { av_usleep(delt<span class="hljs-built_in">a_time</span>)<span class="hljs-comment">;</span> }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li></ul>
这段代码主要是音视频同步的,这块采用的是基于第三方时间基准,同步调整音频和视频。在音频处理中也有类似的代码。针对延时做同步。这个基准线是标准时间,通过修改过时间差值来设置跳帧还是等待。当然这里音视频同步实现比较简单,按照正常的使用应该是在一定范围内可以认为是同步的。
由于声音对于人来说比较敏感,一点杂音都能分辨出来,而视频的跳帧相对来说更容易让人接受,这是因为视觉停留的原因。因此,建议采用基于音频时间基准来进行音视频同步。
IOS
IOS相对于android而言在视频渲染上来说可能比android端稍微复杂,因为ios没有像android的surfaceView可以直接进行操作,都是通过OpenGL来绘制画面。因此可能会比较难于理解。而对于音频,IOS可以采用AudioToolbox进行处理。对于ffmpeg相关的东西android和ios是一样的,也主要是一个流程。音视频同步采用的是同一种方案。在此就不在介绍ios的音视频同步问题。1、视频渲染
前面在android部分已经阐述到使用av_read_frame方法来读取每一帧的信息,这部分ios和android一样,android由于原生支持surfaceView进行绘制渲染,而ios不支持,需要借助opengl来绘制,因此在av_read_frame之后就和android存在区别。主要区别如下代码:<code class="language-oc hljs objectivec has-numbering"> - (<span class="hljs-built_in">NSArray</span> *) decodeFrames: (<span class="hljs-built_in">CGFloat</span>) minDuration { <span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span> && _audioStream == -<span class="hljs-number">1</span>) <span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>; <span class="hljs-keyword">if</span> (_formatCtx == <span class="hljs-literal">nil</span>) { printf(<span class="hljs-string">"AvFormatContext is nil"</span>); <span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>; } <span class="hljs-built_in">NSMutableArray</span> *result = [<span class="hljs-built_in">NSMutableArray</span> array]; AVPacket packet; <span class="hljs-built_in">CGFloat</span> decodedDuration = <span class="hljs-number">0</span>; <span class="hljs-built_in">BOOL</span> finished = <span class="hljs-literal">NO</span>; <span class="hljs-keyword">while</span> (!finished) { <span class="hljs-keyword">if</span> (av_read_frame(_formatCtx, &packet) < <span class="hljs-number">0</span>) { _isEOF = <span class="hljs-literal">YES</span>; <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> ==_videoStream) { <span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>; <span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) { <span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>; <span class="hljs-keyword">int</span> len = avcodec_decode_video2(_videoCodecCtx, _videoFrame, &gotframe, &packet); <span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) { LoggerVideo(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode video error, skip packet"</span>); <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (gotframe) { <span class="hljs-keyword">if</span> (!_disableDeinterlacing && _videoFrame->interlaced_frame) { avpicture_deinterlace((AVPicture*)_videoFrame, (AVPicture*)_videoFrame, _videoCodecCtx->pix_fmt, _videoCodecCtx->width, _videoCodecCtx->height); } KxVideoFrame *frame = [<span class="hljs-keyword">self</span> handleVideoFrame]; <span class="hljs-keyword">if</span> (frame) { [result addObject:frame]; _position = frame<span class="hljs-variable">.position</span>; decodedDuration += frame<span class="hljs-variable">.duration</span>; <span class="hljs-keyword">if</span> (decodedDuration > minDuration) finished = <span class="hljs-literal">YES</span>; } <span class="hljs-keyword">char</span> *buf = (<span class="hljs-keyword">char</span> *)malloc(_videoFrame->width * _videoFrame->height * <span class="hljs-number">3</span> / <span class="hljs-number">2</span>); AVPicture *pict; <span class="hljs-keyword">int</span> w, h; <span class="hljs-keyword">char</span> *y, *u, *v; pict = (AVPicture *)_videoFrame;<span class="hljs-comment">//这里的frame就是解码出来的AVFrame</span> w = _videoFrame->width; h = _videoFrame->height; y = buf; u = y + w * h; v = u + w * h / <span class="hljs-number">4</span>; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h; i++) memcpy(y + w * i, pict->data[<span class="hljs-number">0</span>] + pict->linesize[<span class="hljs-number">0</span>] * i, w); <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++) memcpy(u + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">1</span>] + pict->linesize[<span class="hljs-number">1</span>] * i, w / <span class="hljs-number">2</span>); <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++) memcpy(v + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">2</span>] + pict->linesize[<span class="hljs-number">2</span>] * i, w / <span class="hljs-number">2</span>); [myview setVideoSize:_videoFrame->height height:_videoFrame->width]; [myview displayYUV420pData:buf width:_videoFrame->width height:_videoFrame->height]; free(buf); } <span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len) <span class="hljs-keyword">break</span>; pktSize -= len; } } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _audioStream) { <span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>; <span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) { <span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>; <span class="hljs-keyword">int</span> len = avcodec_decode_audio4(_audioCodecCtx, _audioFrame, &gotframe, &packet); <span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) { LoggerAudio(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode audio error, skip packet"</span>); <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (gotframe) { KxAudioFrame * frame = [<span class="hljs-keyword">self</span> handleAudioFrame]; <span class="hljs-keyword">if</span> (frame) { [result addObject:frame]; <span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span>) { _position = frame<span class="hljs-variable">.position</span>; decodedDuration += frame<span class="hljs-variable">.duration</span>; <span class="hljs-keyword">if</span> (decodedDuration > minDuration) finished = <span class="hljs-literal">YES</span>; } } } <span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len) <span class="hljs-keyword">break</span>; pktSize -= len; } } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _artworkStream) { <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.size</span>) { KxArtworkFrame *frame = [[KxArtworkFrame alloc] init]; frame<span class="hljs-variable">.picture</span> = [NSData dataWithBytes:packet<span class="hljs-variable">.data</span> length:packet<span class="hljs-variable">.size</span>]; [result addObject:frame]; } } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet<span class="hljs-variable">.stream_index</span> == _subtitleStream) { <span class="hljs-keyword">int</span> pktSize = packet<span class="hljs-variable">.size</span>; <span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) { AVSubtitle subtitle; <span class="hljs-keyword">int</span> gotsubtitle = <span class="hljs-number">0</span>; <span class="hljs-keyword">int</span> len = avcodec_decode_subtitle2(_subtitleCodecCtx, &subtitle, &gotsubtitle, &packet); <span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) { LoggerStream(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode subtitle error, skip packet"</span>); <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (gotsubtitle) { KxSubtitleFrame *frame = [<span class="hljs-keyword">self</span> handleSubtitle: &subtitle]; <span class="hljs-keyword">if</span> (frame) { [result addObject:frame]; } avsubtitle_free(&subtitle); } <span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len) <span class="hljs-keyword">break</span>; pktSize -= len; } } av_free_packet(&packet); } <span class="hljs-keyword">return</span> result; } </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li><li>185</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li><li>185</li></ul>
这段代码主要描述的是decodeFrames,将视频帧和音频帧单独处理。视频主要代码如下:
<code class="hljs cpp has-numbering"><span class="hljs-keyword">if</span> (packet.stream_index ==_videoStream) { <span class="hljs-keyword">int</span> pktSize = packet.size; <span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) { <span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>; <span class="hljs-keyword">int</span> len = avcodec_decode_video2(_videoCodecCtx, _videoFrame, &gotframe, &packet); <span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) { LoggerVideo(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode video error, skip packet"</span>); <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (gotframe) { <span class="hljs-keyword">if</span> (!_disableDeinterlacing && _videoFrame->interlaced_frame) { avpicture_deinterlace((AVPicture*)_videoFrame, (AVPicture*)_videoFrame, _videoCodecCtx->pix_fmt, _videoCodecCtx->width, _videoCodecCtx->height); } VideoFrame *frame = [self handleVideoFrame]; <span class="hljs-keyword">if</span> (frame) { [result addObject:frame]; _position = frame.position; decodedDuration += frame.duration; <span class="hljs-keyword">if</span> (decodedDuration > minDuration) finished = YES; } <span class="hljs-keyword">char</span> *buf = (<span class="hljs-keyword">char</span> *)<span class="hljs-built_in">malloc</span>(_videoFrame->width * _videoFrame->height * <span class="hljs-number">3</span> / <span class="hljs-number">2</span>); AVPicture *pict; <span class="hljs-keyword">int</span> w, h; <span class="hljs-keyword">char</span> *y, *u, *v; pict = (AVPicture *)_videoFrame;<span class="hljs-comment">//这里的frame就是解码出来的AVFrame</span> w = _videoFrame->width; h = _videoFrame->height; y = buf; u = y + w * h; v = u + w * h / <span class="hljs-number">4</span>; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h; i++) <span class="hljs-built_in">memcpy</span>(y + w * i, pict->data[<span class="hljs-number">0</span>] + pict->linesize[<span class="hljs-number">0</span>] * i, w); <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++) <span class="hljs-built_in">memcpy</span>(u + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">1</span>] + pict->linesize[<span class="hljs-number">1</span>] * i, w / <span class="hljs-number">2</span>); <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i=<span class="hljs-number">0</span>; i<h/<span class="hljs-number">2</span>; i++) <span class="hljs-built_in">memcpy</span>(v + w / <span class="hljs-number">2</span> * i, pict->data[<span class="hljs-number">2</span>] + pict->linesize[<span class="hljs-number">2</span>] * i, w / <span class="hljs-number">2</span>); [myview setVideoSize:_videoFrame->height height:_videoFrame->width]; [myview displayYUV420pData:buf width:_videoFrame->width height:_videoFrame->height]; <span class="hljs-built_in">free</span>(buf); } <span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len) <span class="hljs-keyword">break</span>; pktSize -= len; } } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (packet.stream_index == _audioStream) { <span class="hljs-keyword">int</span> pktSize = packet.size; <span class="hljs-keyword">while</span> (pktSize > <span class="hljs-number">0</span>) { <span class="hljs-keyword">int</span> gotframe = <span class="hljs-number">0</span>; <span class="hljs-keyword">int</span> len = avcodec_decode_audio4(_audioCodecCtx, _audioFrame, &gotframe, &packet); <span class="hljs-keyword">if</span> (len < <span class="hljs-number">0</span>) { LoggerAudio(<span class="hljs-number">0</span>, @<span class="hljs-string">"decode audio error, skip packet"</span>); <span class="hljs-keyword">break</span>; } <span class="hljs-keyword">if</span> (gotframe) { AudioFrame * frame = [self handleAudioFrame]; <span class="hljs-keyword">if</span> (frame) { [result addObject:frame]; <span class="hljs-keyword">if</span> (_videoStream == -<span class="hljs-number">1</span>) { _position = frame.position; decodedDuration += frame.duration; <span class="hljs-keyword">if</span> (decodedDuration > minDuration) finished = YES; } } } <span class="hljs-keyword">if</span> (<span class="hljs-number">0</span> == len) <span class="hljs-keyword">break</span>; pktSize -= len; } }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li></ul>
这块代码和android的类似,将数据转化为yuv格式,然后拿去渲染,myview是一个View,它主要就是通过OpenGL 将数据渲染成画面。主要代码:
<code class="hljs objectivec has-numbering">- (<span class="hljs-keyword">void</span>)displayYUV420pData:(<span class="hljs-keyword">void</span> *)data width:(<span class="hljs-built_in">NSInteger</span>)w height:(<span class="hljs-built_in">NSInteger</span>)h { @synchronized(<span class="hljs-keyword">self</span>) { <span class="hljs-keyword">if</span> (w != _videoW || h != _videoH) { [<span class="hljs-keyword">self</span> setVideoSize:w height:h]; } [EAGLContext setCurrentContext:_glContext]; glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXY]); glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w, h, GL_RED_EXT, GL_UNSIGNED_BYTE, data); <span class="hljs-comment">//[self debugGlError];</span> glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXU]); glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w/<span class="hljs-number">2</span>, h/<span class="hljs-number">2</span>, GL_RED_EXT, GL_UNSIGNED_BYTE, data + w * h); <span class="hljs-comment">// [self debugGlError];</span> glBindTexture(GL_TEXTURE_2D, _textureYUV[TEXV]); glTexSubImage2D(GL_TEXTURE_2D, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, <span class="hljs-number">0</span>, w/<span class="hljs-number">2</span>, h/<span class="hljs-number">2</span>, GL_RED_EXT, GL_UNSIGNED_BYTE, data + w * h * <span class="hljs-number">5</span> / <span class="hljs-number">4</span>); <span class="hljs-comment">//[self debugGlError];</span> [<span class="hljs-keyword">self</span> render]; } <span class="hljs-preprocessor">#ifdef DEBUG</span> GLenum err = glGetError(); <span class="hljs-keyword">if</span> (err != GL_NO_ERROR) { printf(<span class="hljs-string">"GL_ERROR=======>%d\n"</span>, err); } <span class="hljs-keyword">struct</span> timeval nowtime; gettimeofday(&nowtime, <span class="hljs-literal">NULL</span>); <span class="hljs-keyword">if</span> (nowtime<span class="hljs-variable">.tv_sec</span> != _time<span class="hljs-variable">.tv_sec</span>) { printf(<span class="hljs-string">"视频 %d 帧率: %d\n"</span>, <span class="hljs-keyword">self</span><span class="hljs-variable">.tag</span>, _frameRate); memcpy(&_time, &nowtime, <span class="hljs-keyword">sizeof</span>(<span class="hljs-keyword">struct</span> timeval)); _frameRate = <span class="hljs-number">1</span>; } <span class="hljs-keyword">else</span> { _frameRate++; } <span class="hljs-preprocessor">#endif</span> } </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li></ul>
通过OpenGL将数据绘制出来。这就是整个IOS展示画面主要代码。其他的一些东西类似于android。
2、音频渲染
在上面视频渲染的过程中,列出了部分对音频的处理过程。其主要方法为handleAudioFrame,下面代码书handleAudioFrame方法:<code class="hljs haskell has-numbering">- (<span class="hljs-type">AudioFrame</span> *) handleAudioFrame { <span class="hljs-keyword">if</span> (!_audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0])</span> return nil; id<<span class="hljs-type">AudioManager</span>> audioManager = [<span class="hljs-type">AudioManager</span> audioManager]; const <span class="hljs-type">NSUInteger</span> numChannels = audioManager.numOutputChannels; <span class="hljs-type">NSInteger</span> numFrames; void * audioData; <span class="hljs-keyword">if</span> (_swrContext) { const <span class="hljs-type">NSUInteger</span> ratio = <span class="hljs-type">MAX</span>(<span class="hljs-number">1</span>, audioManager.samplingRate / _audioCodecCtx->sample_rate) * <span class="hljs-type">MAX</span>(<span class="hljs-number">1</span>, audioManager.numOutputChannels / _audioCodecCtx->channels) * <span class="hljs-number">2</span>; const int bufSize = av_samples_get_buffer_size(<span class="hljs-type">NULL</span>, audioManager.numOutputChannels, _audioFrame->nb_samples * ratio, <span class="hljs-type">AV_SAMPLE_FMT_S16</span>, <span class="hljs-number">1</span>); <span class="hljs-keyword">if</span> (!_swrBuffer || _swrBufferSize < bufSize) { _swrBufferSize = bufSize; _swrBuffer = realloc(_swrBuffer, _swrBufferSize); } <span class="hljs-type">Byte</span> *outbuf[<span class="hljs-number">2</span>] = { _swrBuffer, <span class="hljs-number">0</span> }; numFrames = swr_convert(_swrContext, outbuf, _audioFrame->nb_samples * ratio, (const uint8_t **)_audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>,</span> _audioFrame->nb_samples); <span class="hljs-keyword">if</span> (numFrames < <span class="hljs-number">0</span>) { <span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">0</span>, @<span class="hljs-string">"fail resample audio"</span>); return nil; } //int64_t delay = swr_get_delay(_swrContext, audioManager.samplingRate); //<span class="hljs-keyword">if</span> (delay > <span class="hljs-number">0</span>) // <span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">0</span>, @<span class="hljs-string">"resample delay %lld"</span>, delay); audioData = _swrBuffer; } <span class="hljs-keyword">else</span> { <span class="hljs-keyword">if</span> (_audioCodecCtx->sample_fmt != <span class="hljs-type">AV_SAMPLE_FMT_S16</span>) { <span class="hljs-type">NSAssert</span>(false, @<span class="hljs-string">"bucheck, audio format is invalid"</span>); return nil; } audioData = _audioFrame-><span class="hljs-typedef"><span class="hljs-keyword">data</span>[0];</span> numFrames = _audioFrame->nb_samples; } const <span class="hljs-type">NSUInteger</span> numElements = numFrames * numChannels; <span class="hljs-type">NSMutableData</span> *<span class="hljs-typedef"><span class="hljs-keyword">data</span> = [<span class="hljs-type">NSMutableData</span> dataWithLength:numElements * sizeof<span class="hljs-container">(<span class="hljs-title">float</span>)</span>];</span> float scale = <span class="hljs-number">1.0</span> / (float)<span class="hljs-type">INT16_MAX</span> ; vDSP_vflt16((<span class="hljs-type">SInt16</span> *)audioData, <span class="hljs-number">1</span>, <span class="hljs-typedef"><span class="hljs-keyword">data</span>.mutableBytes, 1, numElements);</span> vDSP_vsmul(<span class="hljs-typedef"><span class="hljs-keyword">data</span>.mutableBytes, 1, &scale, <span class="hljs-keyword">data</span>.mutableBytes, 1, numElements);</span> <span class="hljs-type">AudioFrame</span> *frame = [[<span class="hljs-type">AudioFrame</span> alloc] init]; frame.position = av_frame_get_best_effort_timestamp(_audioFrame) * _audioTimeBase; frame.duration = av_frame_get_pkt_duration(_audioFrame) * _audioTimeBase; frame.samples = <span class="hljs-typedef"><span class="hljs-keyword">data</span>;</span> <span class="hljs-keyword">if</span> (frame.duration == <span class="hljs-number">0</span>) { // sometimes ffmpeg can't determine the duration <span class="hljs-keyword">of</span> audio frame // especially <span class="hljs-keyword">of</span> wma/wmv format // so <span class="hljs-keyword">in</span> this <span class="hljs-keyword">case</span> must compute duration frame.duration = frame.samples.length / (sizeof(float) * numChannels * audioManager.samplingRate); } <span class="hljs-preprocessor">#if DEBUG</span> <span class="hljs-type">LoggerAudio</span>(<span class="hljs-number">2</span>, @<span class="hljs-string">"AFD: %.4f %.4f | %.4f "</span>, frame.position, frame.duration, frame.samples.length / (<span class="hljs-number">8.0</span> * <span class="hljs-number">44100.0</span>)); <span class="hljs-preprocessor">#endif</span> return frame; }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li></ul>
handleAudioFrame主要方法是将数据组合转换,比如采样率重设,转码等等功能,最后放进一个数组中,最后交给AudioToolbox进行处理。我们接着放下看:
<code class="language-swift hljs objectivec has-numbering">func enableAudio(on: Bool) { let audioManager = AudioManager<span class="hljs-variable">.audioManager</span>() <span class="hljs-keyword">if</span> on && decoder<span class="hljs-variable">.validAudio</span> { <span class="hljs-comment">// 闭包</span> audioManager<span class="hljs-variable">.outputBlock</span> = {(outData: UnsafeMutablePointer<Float>,numFrames: UInt32, numChannels: UInt32) -> Void in <span class="hljs-keyword">self</span><span class="hljs-variable">.audioCallbackFillData</span>(outData, numFrames: Int(numFrames), numChannels: Int(numChannels)) } audioManager<span class="hljs-variable">.play</span>() }<span class="hljs-keyword">else</span>{ audioManager<span class="hljs-variable">.pause</span>() audioManager<span class="hljs-variable">.outputBlock</span> = <span class="hljs-literal">nil</span> } } func audioCallbackFillData(outData: UnsafeMutablePointer<Float>, numFrames: Int, numChannels: Int) { var numFrames = numFrames var weakOutData = outData <span class="hljs-keyword">if</span> buffered { memset(weakOutData, <span class="hljs-number">0</span>, numFrames * numChannels * <span class="hljs-keyword">sizeof</span>(Float)) <span class="hljs-keyword">return</span> } autoreleasepool { <span class="hljs-keyword">while</span> numFrames > <span class="hljs-number">0</span> { <span class="hljs-keyword">if</span> currentAudioFrame == <span class="hljs-literal">nil</span> { <span class="hljs-built_in">dispatch_sync</span>(lockQueue) { <span class="hljs-keyword">if</span> let count = <span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>?<span class="hljs-variable">.count</span> { <span class="hljs-keyword">if</span> count > <span class="hljs-number">0</span> { let frame: AudioFrame = <span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>![<span class="hljs-number">0</span>] as! AudioFrame <span class="hljs-keyword">self</span><span class="hljs-variable">.audioFrames</span>!<span class="hljs-variable">.removeObjectAtIndex</span>(<span class="hljs-number">0</span>) <span class="hljs-keyword">self</span><span class="hljs-variable">.moviePosition</span> = frame<span class="hljs-variable">.position</span> <span class="hljs-keyword">self</span><span class="hljs-variable">.bufferedDuration</span> -= Float(frame<span class="hljs-variable">.duration</span>) <span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFramePos</span> = <span class="hljs-number">0</span> <span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFrame</span> = frame<span class="hljs-variable">.samples</span> } } } } <span class="hljs-keyword">if</span> (currentAudioFrame != <span class="hljs-literal">nil</span>) { let bytes = currentAudioFrame<span class="hljs-variable">.bytes</span> + currentAudioFramePos let bytesLeft: Int = (currentAudioFrame<span class="hljs-variable">.length</span> - currentAudioFramePos) let frameSizeOf: Int = numChannels * <span class="hljs-keyword">sizeof</span>(Float) let bytesToCopy: Int = min(numFrames * frameSizeOf, bytesLeft) let framesToCopy: Int = bytesToCopy / frameSizeOf memcpy(weakOutData, bytes, bytesToCopy) numFrames -= framesToCopy weakOutData = weakOutData<span class="hljs-variable">.advancedBy</span>(framesToCopy * numChannels) <span class="hljs-keyword">if</span> bytesToCopy < bytesLeft { <span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFramePos</span> += bytesToCopy } <span class="hljs-keyword">else</span> { <span class="hljs-keyword">self</span><span class="hljs-variable">.currentAudioFrame</span> = <span class="hljs-literal">nil</span> } }<span class="hljs-keyword">else</span>{ memset(weakOutData, <span class="hljs-number">0</span>, numFrames * numChannels * <span class="hljs-keyword">sizeof</span>(Float)) <span class="hljs-comment">//LoggerStream(1, @"silence audio");</span> <span class="hljs-keyword">self</span><span class="hljs-variable">.debugAudioStatus</span> = <span class="hljs-number">3</span> <span class="hljs-keyword">self</span><span class="hljs-variable">.debugAudioStatusTS</span> = <span class="hljs-built_in">NSDate</span>() <span class="hljs-keyword">break</span> } } } } </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li></ul>
这是一段Swift代码。在ios采用的是swift+oc+c++混合编译,正好借此熟悉swift于oc和c++的交互。enableAudio主要是创建一个audioManager实例,进行注册回调,和开始播放和暂停服务。audioManager是一个单例。是一个封装AudioToolbox类。下面的代码是激活AudioSession(初始化Audio)和失效AudioSession代码。
<code class="language-oc hljs objectivec has-numbering">- (<span class="hljs-built_in">BOOL</span>) activateAudioSession { <span class="hljs-keyword">if</span> (!_activated) { <span class="hljs-keyword">if</span> (!_initialized) { <span class="hljs-keyword">if</span> (checkError(AudioSessionInitialize(<span class="hljs-literal">NULL</span>, kCFRunLoopDefaultMode, sessionInterruptionListener, (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)), <span class="hljs-string">"Couldn't initialize audio session"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; _initialized = <span class="hljs-literal">YES</span>; } <span class="hljs-keyword">if</span> ([<span class="hljs-keyword">self</span> checkAudioRoute] && [<span class="hljs-keyword">self</span> setupAudio]) { _activated = <span class="hljs-literal">YES</span>; } } <span class="hljs-keyword">return</span> _activated; } - (<span class="hljs-keyword">void</span>) deactivateAudioSession { <span class="hljs-keyword">if</span> (_activated) { [<span class="hljs-keyword">self</span> pause]; checkError(AudioUnitUninitialize(_audioUnit), <span class="hljs-string">"Couldn't uninitialize the audio unit"</span>); <span class="hljs-comment">/* fails with error (-10851) ? checkError(AudioUnitSetProperty(_audioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Input, 0, NULL, 0), "Couldn't clear the render callback on the audio unit"); */</span> checkError(AudioComponentInstanceDispose(_audioUnit), <span class="hljs-string">"Couldn't dispose the output audio unit"</span>); checkError(AudioSessionSetActive(<span class="hljs-literal">NO</span>), <span class="hljs-string">"Couldn't deactivate the audio session"</span>); checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_AudioRouteChange, sessionPropertyListener, (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)), <span class="hljs-string">"Couldn't remove audio session property listener"</span>); checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_CurrentHardwareOutputVolume, sessionPropertyListener, (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)), <span class="hljs-string">"Couldn't remove audio session property listener"</span>); _activated = <span class="hljs-literal">NO</span>; } } - (<span class="hljs-built_in">BOOL</span>) setupAudio { <span class="hljs-comment">// --- Audio Session Setup ---</span> UInt32 sessionCategory = kAudioSessionCategory_MediaPlayback; <span class="hljs-comment">//UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord;</span> <span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, <span class="hljs-keyword">sizeof</span>(sessionCategory), &sessionCategory), <span class="hljs-string">"Couldn't set audio category"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; <span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange, sessionPropertyListener, (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)), <span class="hljs-string">"Couldn't add audio session property listener"</span>)) { <span class="hljs-comment">// just warning</span> } <span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_CurrentHardwareOutputVolume, sessionPropertyListener, (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)), <span class="hljs-string">"Couldn't add audio session property listener"</span>)) { <span class="hljs-comment">// just warning</span> } <span class="hljs-comment">// Set the buffer size, this will affect the number of samples that get rendered every time the audio callback is fired</span> <span class="hljs-comment">// A small number will get you lower latency audio, but will make your processor work harder</span> <span class="hljs-preprocessor">#if !TARGET_IPHONE_SIMULATOR</span> Float32 preferredBufferSize = <span class="hljs-number">0.0232</span>; <span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration, <span class="hljs-keyword">sizeof</span>(preferredBufferSize), &preferredBufferSize), <span class="hljs-string">"Couldn't set the preferred buffer duration"</span>)) { <span class="hljs-comment">// just warning</span> } <span class="hljs-preprocessor">#endif</span> <span class="hljs-keyword">if</span> (checkError(AudioSessionSetActive(<span class="hljs-literal">YES</span>), <span class="hljs-string">"Couldn't activate the audio session"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; [<span class="hljs-keyword">self</span> checkSessionProperties]; <span class="hljs-comment">// ----- Audio Unit Setup -----</span> <span class="hljs-comment">// Describe the output unit.</span> AudioComponentDescription description = {<span class="hljs-number">0</span>}; description<span class="hljs-variable">.componentType</span> = kAudioUnitType_Output; description<span class="hljs-variable">.componentSubType</span> = kAudioUnitSubType_RemoteIO; description<span class="hljs-variable">.componentManufacturer</span> = kAudioUnitManufacturer_Apple; <span class="hljs-comment">// Get component</span> AudioComponent component = AudioComponentFindNext(<span class="hljs-literal">NULL</span>, &description); <span class="hljs-keyword">if</span> (checkError(AudioComponentInstanceNew(component, &_audioUnit), <span class="hljs-string">"Couldn't create the output audio unit"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; UInt32 size; <span class="hljs-comment">// Check the output stream format</span> size = <span class="hljs-keyword">sizeof</span>(AudioStreamBasicDescription); <span class="hljs-keyword">if</span> (checkError(AudioUnitGetProperty(_audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, <span class="hljs-number">0</span>, &_outputFormat, &size), <span class="hljs-string">"Couldn't get the hardware output stream format"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; _outputFormat<span class="hljs-variable">.mSampleRate</span> = _samplingRate; <span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Input, <span class="hljs-number">0</span>, &_outputFormat, size), <span class="hljs-string">"Couldn't set the hardware output stream format"</span>)) { <span class="hljs-comment">// just warning</span> } _numBytesPerSample = _outputFormat<span class="hljs-variable">.mBitsPerChannel</span> / <span class="hljs-number">8</span>; _numOutputChannels = _outputFormat<span class="hljs-variable">.mChannelsPerFrame</span>; LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output bytes per sample: %ld"</span>, _numBytesPerSample); LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output num channels: %ld"</span>, _numOutputChannels); <span class="hljs-comment">// Slap a render callback on the unit</span> AURenderCallbackStruct callbackStruct; callbackStruct<span class="hljs-variable">.inputProc</span> = renderCallback; <span class="hljs-comment">// 注册回调,这个回调是用来取数据的,也就是</span> callbackStruct<span class="hljs-variable">.inputProcRefCon</span> = (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>); <span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit, kAudioUnitProperty_SetRenderCallback, kAudioUnitScope_Input, <span class="hljs-number">0</span>, &callbackStruct, <span class="hljs-keyword">sizeof</span>(callbackStruct)), <span class="hljs-string">"Couldn't set the render callback on the audio unit"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; <span class="hljs-keyword">if</span> (checkError(AudioUnitInitialize(_audioUnit), <span class="hljs-string">"Couldn't initialize the audio unit"</span>)) <span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>; <span class="hljs-keyword">return</span> <span class="hljs-literal">YES</span>; }</code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li><li>46</li><li>47</li><li>48</li><li>49</li><li>50</li><li>51</li><li>52</li><li>53</li><li>54</li><li>55</li><li>56</li><li>57</li><li>58</li><li>59</li><li>60</li><li>61</li><li>62</li><li>63</li><li>64</li><li>65</li><li>66</li><li>67</li><li>68</li><li>69</li><li>70</li><li>71</li><li>72</li><li>73</li><li>74</li><li>75</li><li>76</li><li>77</li><li>78</li><li>79</li><li>80</li><li>81</li><li>82</li><li>83</li><li>84</li><li>85</li><li>86</li><li>87</li><li>88</li><li>89</li><li>90</li><li>91</li><li>92</li><li>93</li><li>94</li><li>95</li><li>96</li><li>97</li><li>98</li><li>99</li><li>100</li><li>101</li><li>102</li><li>103</li><li>104</li><li>105</li><li>106</li><li>107</li><li>108</li><li>109</li><li>110</li><li>111</li><li>112</li><li>113</li><li>114</li><li>115</li><li>116</li><li>117</li><li>118</li><li>119</li><li>120</li><li>121</li><li>122</li><li>123</li><li>124</li><li>125</li><li>126</li><li>127</li><li>128</li><li>129</li><li>130</li><li>131</li><li>132</li><li>133</li><li>134</li><li>135</li><li>136</li><li>137</li><li>138</li><li>139</li><li>140</li><li>141</li><li>142</li><li>143</li><li>144</li><li>145</li><li>146</li><li>147</li><li>148</li><li>149</li><li>150</li><li>151</li><li>152</li><li>153</li><li>154</li><li>155</li><li>156</li><li>157</li><li>158</li><li>159</li><li>160</li><li>161</li><li>162</li><li>163</li><li>164</li><li>165</li><li>166</li><li>167</li><li>168</li><li>169</li><li>170</li><li>171</li><li>172</li><li>173</li><li>174</li><li>175</li><li>176</li><li>177</li><li>178</li><li>179</li><li>180</li><li>181</li><li>182</li><li>183</li><li>184</li></ul>
正真将数据渲染代码
<code class="hljs mel has-numbering">- (BOOL) renderFrames: (UInt32) numFrames ioData: (AudioBufferList <span class="hljs-variable">*)</span> ioData { <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) { memset(ioData->mBuffers[iBuffer].mData, <span class="hljs-number">0</span>, ioData->mBuffers[iBuffer].mDataByteSize); } <span class="hljs-keyword">if</span> (_playing && _outputBlock ) { <span class="hljs-comment">// Collect data to render from the callbacks</span> _outputBlock(_outData, numFrames, _numOutputChannels); <span class="hljs-comment">// Put the rendered data into the output buffer</span> <span class="hljs-keyword">if</span> (_numBytesPerSample == <span class="hljs-number">4</span>) <span class="hljs-comment">// then we've already got floats</span> { <span class="hljs-keyword">float</span> zero = <span class="hljs-number">0.0</span>; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) { <span class="hljs-keyword">int</span> thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iChannel = <span class="hljs-number">0</span>; iChannel < thisNumChannels; ++iChannel) { vDSP_vsadd(_outData+iChannel, _numOutputChannels, &zero, (<span class="hljs-keyword">float</span> <span class="hljs-variable">*)</span>ioData->mBuffers[iBuffer].mData, thisNumChannels, numFrames); } } } <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (_numBytesPerSample == <span class="hljs-number">2</span>) <span class="hljs-comment">// then we need to convert SInt16 -> Float (and also scale)</span> { <span class="hljs-keyword">float</span> <span class="hljs-keyword">scale</span> = (<span class="hljs-keyword">float</span>)INT16_MAX; vDSP_vsmul(_outData, <span class="hljs-number">1</span>, &<span class="hljs-keyword">scale</span>, _outData, <span class="hljs-number">1</span>, numFrames<span class="hljs-variable">*_numOutputChannels</span>); <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iBuffer=<span class="hljs-number">0</span>; iBuffer < ioData->mNumberBuffers; ++iBuffer) { <span class="hljs-keyword">int</span> thisNumChannels = ioData->mBuffers[iBuffer].mNumberChannels; <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> iChannel = <span class="hljs-number">0</span>; iChannel < thisNumChannels; ++iChannel) { vDSP_vfix16(_outData+iChannel, _numOutputChannels, (SInt16 <span class="hljs-variable">*)</span>ioData->mBuffers[iBuffer].mData+iChannel, thisNumChannels, numFrames); } } } } <span class="hljs-keyword">return</span> noErr; } </code><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul><ul style="" class="pre-numbering"><li>1</li><li>2</li><li>3</li><li>4</li><li>5</li><li>6</li><li>7</li><li>8</li><li>9</li><li>10</li><li>11</li><li>12</li><li>13</li><li>14</li><li>15</li><li>16</li><li>17</li><li>18</li><li>19</li><li>20</li><li>21</li><li>22</li><li>23</li><li>24</li><li>25</li><li>26</li><li>27</li><li>28</li><li>29</li><li>30</li><li>31</li><li>32</li><li>33</li><li>34</li><li>35</li><li>36</li><li>37</li><li>38</li><li>39</li><li>40</li><li>41</li><li>42</li><li>43</li><li>44</li><li>45</li></ul>
总结
本文主要是讲述了ffmpeg实现播放的逻辑,分为android和ios两端,根据两端平台的特性做了相应的处理。在android端采用的是NativeWindow(surface)实现视频播放,OpenSL ES实现音频播放。实现音视频同步的逻辑是基于第三方时间基准线,音频和视频同时调整的方案。在ios端采用的是OpenGL实现视频渲染,AudioToolbox实现音频播放。音视频同步和android采用的是一样。其中两端的ffmpeg逻辑是一致的。在ios端OpenGL实现视频渲染没有重点阐述如何使用OpenGL。这个有兴趣的同学可以自行研究。备注:整个代码工程等整理之后会发布出来。
最后添加两张播放效果图
顶4
踩
相关文章推荐
- 直播技术(从服务端到客户端)二
- 直播技术(从服务端到客户端)一
- 直播技术(从服务端到客户端)二
- 直播技术(从服务端到客户端)三
- 直播技术(从服务端到客户端)二
- 直播技术(从服务端到客户端)一
- 直播技术(从服务端到客户端)一
- 直播技术(从服务端到客户端)一
- 一个简单的直播 IOS客户端 + 服务端
- Delphi2010中DataSnap高级技术(2)—DataSnap服务端和客户端发布分发方法
- Android中直播视频技术探究之---视频直播服务端环境搭建(Nginx+RTMP)
- 30分钟跑直播 客户端(iOS)+服务端(nginx--rtmp-module)
- 服务端与客户端共用smarty模板技术
- 初探直播客户端(android)技术
- ASP.NET Core2利用Jwt技术在服务端实现对客户端的身份认证
- iOS推送技术总结-(证书的生成、客户端的开发、服务端的开发)
- 5、Spring技术栈-客户端、服务端数据验证
- 利用多线程和TCP技术,实现客户端与服务端之间的通信
- 如果希望给Android客户端提供JSON格式的数据,服务端应该用什么技术编写呢?
- Android中直播视频技术探究之---视频直播服务端环境搭建(Nginx+RTMP)