直播技术（从服务端到客户端）二

播放

在上一篇文章中，我们叙述了直播技术的环境配置（包括服务端nginx，nginx-rtmp-module， ffmpeg,
Android编译，iOS编译）。从本文开始，我们将叙述播放相关的东西，播放是直播技术中关键的一步，它包括很多技术如：解码，缩放，时间基线选择，缓存队列，画面渲染，声音播放等等。我将分为三个部分为大家讲述整个播放流程；

Android

第一部分是基于NativeWindow的视频渲染，主要使用的OpenGL ES2通过传入surface来将视频数据渲染到surface上显示出来。第二部分是基于OpenSL ES来音频播放。第三部分，音视频同步。我们使用的都是android原生自带的一些库来做音视频渲染处理。

IOS

同样IOS也分成三个部分，第一部分视频渲染：使用OpenGLES.framework，通过OpenGL来渲染视频画面，第二部分是音频播放，基于AudioToolbox.framework做音频播放；第三部分，视音频同步。

利用原生库可以减少资源的利用，降低内存，提高性能；一般而言，如果不是通晓android、ios的程序员会选择一个统一的视频显示和音频播放库（SDL），这个库可以实现视频显示和音频播。但是增加额外的库意味着资源的浪费和性能的降低。

Android

我们首先带来android端的视频播放功能，我们分成三个部分，1、视频渲染；2、音频播放；3、时间基线（音视频同步）来阐述。

1、视频渲染

ffmpeg为我们提供浏览丰富的编解码类型（ffmpeg所具备编解码能力都是软件编解码，不是指硬件编解码。具体之后文章会详细介绍ffmpeg），视频解码包括flv, mpeg, mov 等；音频包括aac, mp3等。对于整个播放，FFmpeg主要处理流程如下：

<code class="language-C++ hljs scss has-numbering">    <span class="hljs-function">av_register_all()</span>;  <span class="hljs-comment">// 注册所有的文件格式和编解码器的库,打开的合适格式的文件上会自动选择相应的编解码库</span>
<span class="hljs-function">avformat_network_init()</span>; <span class="hljs-comment">// 注册网络服务</span>
<span class="hljs-function">avformat_alloc_context()</span>; <span class="hljs-comment">//  分配FormatContext内存，</span>
<span class="hljs-function">avformat_open_input()</span>;  <span class="hljs-comment">// 打开输入流，获取头部信息，配合av_close_input_file（）关闭流</span>
<span class="hljs-function">avformat_find_stream_info()</span>; <span class="hljs-comment">// 读取packets，来获取流信息，并在pFormatCtx->streams 填充上正确的信息</span>
<span class="hljs-function">avcodec_find_decoder()</span>;  <span class="hljs-comment">// 获取解码器，</span>
<span class="hljs-function">avcodec_open2()</span>; <span class="hljs-comment">// 通过AVCodec来初始化AVCodecContext</span>
<span class="hljs-function">av_read_frame()</span>; <span class="hljs-comment">// 读取每一帧</span>
<span class="hljs-function">avcodec_decode_video2()</span>; <span class="hljs-comment">// 解码帧数据</span>
<span class="hljs-function">avcodec_close()</span>;  <span class="hljs-comment">// 关闭编辑器上下文</span>
<span class="hljs-function">avformat_close_input()</span>; <span class="hljs-comment">// 关闭文件流</span></code>

我们先来看一段代码：

<code class="language-C++ hljs php has-numbering">av_register_all();
avformat_network_init();
pFormatCtx = avformat_alloc_context();
<span class="hljs-keyword">if</span> (avformat_open_input(&pFormatCtx, pathStr, <span class="hljs-keyword">NULL</span>, <span class="hljs-keyword">NULL</span>) != <span class="hljs-number">0</span>) {
LOGE(<span class="hljs-string">"Couldn't open file: %s\n"</span>, pathStr);
<span class="hljs-keyword">return</span>;
}

<span class="hljs-keyword">if</span> (avformat_find_stream_info(pFormatCtx, &dictionary) < <span class="hljs-number">0</span>) {
LOGE(<span class="hljs-string">"Couldn't find stream information."</span>);
<span class="hljs-keyword">return</span>;
}
av_dump_format(pFormatCtx, <span class="hljs-number">0</span>, pathStr, <span class="hljs-number">0</span>);
</code>

这段代码可以算是初始化FFmpeg，首先注册编解码库，为FormatContext分配内存，调用avformat_open_input打开输入流，获取头部信息，配合avformat_find_stream_info来填充FormatContext中相关内容，av_dump_format这个是dump出流信息。这个信息是这个样子的：

<code class="language-text hljs lasso has-numbering">video infomation：
Input <span class="hljs-variable">#0</span>, flv, from <span class="hljs-string">'rtmp:127.0.0.1:1935/live/steam'</span>:
Metadata:
Server          : NGINX RTMP (github<span class="hljs-built_in">.</span>com/sergey<span class="hljs-attribute">-dryabzhinsky</span>/nginx<span class="hljs-attribute">-rtmp</span><span class="hljs-attribute">-module</span>)
displayWidth    : <span class="hljs-number">320</span>
displayHeight   : <span class="hljs-number">240</span>
fps             : <span class="hljs-number">15</span>
profile         :
level           :
<span class="hljs-built_in">Duration</span>: <span class="hljs-number">00</span>:<span class="hljs-number">00</span>:<span class="hljs-number">00.00</span>, start: <span class="hljs-number">15.400000</span>, bitrate: N/A
Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">0</span>: Video: flv1 (flv), yuv420p, <span class="hljs-number">320</span>x240, <span class="hljs-number">15</span> tbr, <span class="hljs-number">1</span>k tbn, <span class="hljs-number">1</span>k tbc
Stream <span class="hljs-variable">#0</span>:<span class="hljs-number">1</span>: Audio: mp3, <span class="hljs-number">11025</span> Hz, stereo, s16p, <span class="hljs-number">32</span> kb/s</code>

整个音频播放流畅其实看起来也是很简单的，主要分：1、创建实现播放引擎；2、创建实现混音器；3、设置缓冲和pcm格式；4、创建实现播放器；5、获取音频播放器接口；6、获取缓冲buffer；7、注册播放回调；8、获取音效接口；9、获取音量接口；10、获取播放状态接口；

做完这10步，整个音频播放器引擎就创建完毕，接下来就是引擎读取数据播放。

<code class="language-C++ hljs objectivec has-numbering"><span class="hljs-keyword">void</span> playBuffer(<span class="hljs-keyword">void</span> *pBuffer, <span class="hljs-keyword">int</span> size) {
<span class="hljs-comment">// 判断数据可用性</span>
<span class="hljs-keyword">if</span> (pBuffer == <span class="hljs-literal">NULL</span> || size == -<span class="hljs-number">1</span>) {
<span class="hljs-keyword">return</span>;
}
LOGV(<span class="hljs-string">"PlayBuff!"</span>);
<span class="hljs-comment">// 数据存放进bqPlayerBufferQueue中</span>
SLresult result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue,
pBuffer, size);
<span class="hljs-keyword">if</span> (result != SL_RESULT_SUCCESS)
LOGE(<span class="hljs-string">"Play buffer error!"</span>);
}</code>

这段代码主要阐述的播放的过程，通过将数据放进bqPlayerBufferQueue，供播放引擎读取播放。记得我们在创建缓冲buffer的时候，注册了一个callback，这个callBack的作用就是通知可以向缓冲队列中添加数据，这个callBack的原型如下：

<code class="hljs lasso has-numbering"><span class="hljs-literal">void</span> videoPlayCallBack(SLAndroidSimpleBufferQueueItf bq, <span class="hljs-literal">void</span> <span class="hljs-subst">*</span>context) {
<span class="hljs-comment">// 添加数据到bqPlayerBufferQueue中，通过调用playBuffer方法。</span>
<span class="hljs-literal">void</span><span class="hljs-subst">*</span> <span class="hljs-built_in">data</span> <span class="hljs-subst">=</span> getData();
int size <span class="hljs-subst">=</span> getDataSize();
playBuffer(<span class="hljs-built_in">data</span>, size);
}</code>

<code class="hljs cpp has-numbering"><span class="hljs-keyword">typedef</span> <span class="hljs-keyword">struct</span> PlayInstance {
ANativeWindow *window; <span class="hljs-comment">// nativeWindow // 通过传入surface构建</span>
<span class="hljs-keyword">int</span> display_width; <span class="hljs-comment">// 显示宽度</span>
<span class="hljs-keyword">int</span> display_height; <span class="hljs-comment">// 显示高度</span>
<span class="hljs-keyword">int</span> stop;  <span class="hljs-comment">// 停止</span>
<span class="hljs-keyword">int</span> timeout_flag; <span class="hljs-comment">// 超时标记</span>
<span class="hljs-keyword">int</span> disable_video;
VideoState *videoState;
<span class="hljs-comment">//队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *<span class="hljs-built_in">queue</span>; <span class="hljs-comment">// 音视频帧队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *video_queue; <span class="hljs-comment">// 视频帧队列</span>
<span class="hljs-keyword">struct</span> ThreadQueue *audio_queue; <span class="hljs-comment">// 音频帧队列</span>

} PlayInstance;</code>

我们主要分析延时同步的那一段代码：

<code class="hljs autohotkey has-numbering">// 延时同步
int64_t pkt_pts = pavpacket.pts<span class="hljs-comment">;</span>
double show_time = pkt_pts * (playInstance->videoState->video_time_base)<span class="hljs-comment">;</span>
int64_t show_time_micro = show_time * <span class="hljs-number">1000000</span><span class="hljs-comment">;</span>
int64_t played_time = av_gettime() - playInstance->videoState->video_start_time<span class="hljs-comment">;</span>
int64_t delt<span class="hljs-built_in">a_time</span> = show_time_micro - played_time<span class="hljs-comment">;</span>

<span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> < -(<span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>)) {
LOGE(<span class="hljs-string">"视频跳帧\n"</span>)<span class="hljs-comment">;</span>
<span class="hljs-keyword">continue</span>；
} <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> (delt<span class="hljs-built_in">a_time</span> > <span class="hljs-number">0.2</span> * <span class="hljs-number">1000000</span>) {
av_usleep(delt<span class="hljs-built_in">a_time</span>)<span class="hljs-comment">;</span>
}</code>

这是一段Swift代码。在ios采用的是swift+oc+c++混合编译，正好借此熟悉swift于oc和c++的交互。enableAudio主要是创建一个audioManager实例，进行注册回调，和开始播放和暂停服务。audioManager是一个单例。是一个封装AudioToolbox类。下面的代码是激活AudioSession（初始化Audio）和失效AudioSession代码。

<code class="language-oc hljs objectivec has-numbering">- (<span class="hljs-built_in">BOOL</span>) activateAudioSession
{
<span class="hljs-keyword">if</span> (!_activated) {

<span class="hljs-keyword">if</span> (!_initialized) {

<span class="hljs-keyword">if</span> (checkError(AudioSessionInitialize(<span class="hljs-literal">NULL</span>,
kCFRunLoopDefaultMode,
sessionInterruptionListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't initialize audio session"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

_initialized = <span class="hljs-literal">YES</span>;
}

<span class="hljs-keyword">if</span> ([<span class="hljs-keyword">self</span> checkAudioRoute] &&
[<span class="hljs-keyword">self</span> setupAudio]) {

_activated = <span class="hljs-literal">YES</span>;
}
}

<span class="hljs-keyword">return</span> _activated;
}

- (<span class="hljs-keyword">void</span>) deactivateAudioSession
{
<span class="hljs-keyword">if</span> (_activated) {

[<span class="hljs-keyword">self</span> pause];

checkError(AudioUnitUninitialize(_audioUnit),
<span class="hljs-string">"Couldn't uninitialize the audio unit"</span>);

<span class="hljs-comment">/*
fails with error (-10851) ?

checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
0,
NULL,
0),
"Couldn't clear the render callback on the audio unit");
*/</span>

checkError(AudioComponentInstanceDispose(_audioUnit),
<span class="hljs-string">"Couldn't dispose the output audio unit"</span>);

checkError(AudioSessionSetActive(<span class="hljs-literal">NO</span>),
<span class="hljs-string">"Couldn't deactivate the audio session"</span>);

checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_AudioRouteChange,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't remove audio session property listener"</span>);

checkError(AudioSessionRemovePropertyListenerWithUserData(kAudioSessionProperty_CurrentHardwareOutputVolume,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't remove audio session property listener"</span>);

_activated = <span class="hljs-literal">NO</span>;
}
}

- (<span class="hljs-built_in">BOOL</span>) setupAudio
{
<span class="hljs-comment">// --- Audio Session Setup ---</span>

UInt32 sessionCategory = kAudioSessionCategory_MediaPlayback;
<span class="hljs-comment">//UInt32 sessionCategory = kAudioSessionCategory_PlayAndRecord;</span>
<span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_AudioCategory,
<span class="hljs-keyword">sizeof</span>(sessionCategory),
&sessionCategory),
<span class="hljs-string">"Couldn't set audio category"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

<span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_AudioRouteChange,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't add audio session property listener"</span>))
{
<span class="hljs-comment">// just warning</span>
}

<span class="hljs-keyword">if</span> (checkError(AudioSessionAddPropertyListener(kAudioSessionProperty_CurrentHardwareOutputVolume,
sessionPropertyListener,
(__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>)),
<span class="hljs-string">"Couldn't add audio session property listener"</span>))
{
<span class="hljs-comment">// just warning</span>
}

<span class="hljs-comment">// Set the buffer size, this will affect the number of samples that get rendered every time the audio callback is fired</span>
<span class="hljs-comment">// A small number will get you lower latency audio, but will make your processor work harder</span>

<span class="hljs-preprocessor">#if !TARGET_IPHONE_SIMULATOR</span>
Float32 preferredBufferSize = <span class="hljs-number">0.0232</span>;
<span class="hljs-keyword">if</span> (checkError(AudioSessionSetProperty(kAudioSessionProperty_PreferredHardwareIOBufferDuration,
<span class="hljs-keyword">sizeof</span>(preferredBufferSize),
&preferredBufferSize),
<span class="hljs-string">"Couldn't set the preferred buffer duration"</span>)) {

<span class="hljs-comment">// just warning</span>
}
<span class="hljs-preprocessor">#endif</span>

<span class="hljs-keyword">if</span> (checkError(AudioSessionSetActive(<span class="hljs-literal">YES</span>),
<span class="hljs-string">"Couldn't activate the audio session"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

[<span class="hljs-keyword">self</span> checkSessionProperties];

<span class="hljs-comment">// ----- Audio Unit Setup -----</span>

<span class="hljs-comment">// Describe the output unit.</span>

AudioComponentDescription description = {<span class="hljs-number">0</span>};
description<span class="hljs-variable">.componentType</span> = kAudioUnitType_Output;
description<span class="hljs-variable">.componentSubType</span> = kAudioUnitSubType_RemoteIO;
description<span class="hljs-variable">.componentManufacturer</span> = kAudioUnitManufacturer_Apple;

<span class="hljs-comment">// Get component</span>
AudioComponent component = AudioComponentFindNext(<span class="hljs-literal">NULL</span>, &description);
<span class="hljs-keyword">if</span> (checkError(AudioComponentInstanceNew(component, &_audioUnit),
<span class="hljs-string">"Couldn't create the output audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

UInt32 size;

<span class="hljs-comment">// Check the output stream format</span>
size = <span class="hljs-keyword">sizeof</span>(AudioStreamBasicDescription);
<span class="hljs-keyword">if</span> (checkError(AudioUnitGetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&_outputFormat,
&size),
<span class="hljs-string">"Couldn't get the hardware output stream format"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

_outputFormat<span class="hljs-variable">.mSampleRate</span> = _samplingRate;
<span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_StreamFormat,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&_outputFormat,
size),
<span class="hljs-string">"Couldn't set the hardware output stream format"</span>)) {

<span class="hljs-comment">// just warning</span>
}

_numBytesPerSample = _outputFormat<span class="hljs-variable">.mBitsPerChannel</span> / <span class="hljs-number">8</span>;
_numOutputChannels = _outputFormat<span class="hljs-variable">.mChannelsPerFrame</span>;

LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output bytes per sample: %ld"</span>, _numBytesPerSample);
LoggerAudio(<span class="hljs-number">2</span>, @<span class="hljs-string">"Current output num channels: %ld"</span>, _numOutputChannels);

<span class="hljs-comment">// Slap a render callback on the unit</span>
AURenderCallbackStruct callbackStruct;
callbackStruct<span class="hljs-variable">.inputProc</span> = renderCallback; <span class="hljs-comment">// 注册回调，这个回调是用来取数据的，也就是</span>
callbackStruct<span class="hljs-variable">.inputProcRefCon</span> = (__bridge <span class="hljs-keyword">void</span> *)(<span class="hljs-keyword">self</span>);

<span class="hljs-keyword">if</span> (checkError(AudioUnitSetProperty(_audioUnit,
kAudioUnitProperty_SetRenderCallback,
kAudioUnitScope_Input,
<span class="hljs-number">0</span>,
&callbackStruct,
<span class="hljs-keyword">sizeof</span>(callbackStruct)),
<span class="hljs-string">"Couldn't set the render callback on the audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

<span class="hljs-keyword">if</span> (checkError(AudioUnitInitialize(_audioUnit),
<span class="hljs-string">"Couldn't initialize the audio unit"</span>))
<span class="hljs-keyword">return</span> <span class="hljs-literal">NO</span>;

<span class="hljs-keyword">return</span> <span class="hljs-literal">YES</span>;
}</code>

总结

本文主要是讲述了ffmpeg实现播放的逻辑，分为android和ios两端，根据两端平台的特性做了相应的处理。在android端采用的是NativeWindow（surface）实现视频播放，OpenSL ES实现音频播放。实现音视频同步的逻辑是基于第三方时间基准线，音频和视频同时调整的方案。在ios端采用的是OpenGL实现视频渲染，AudioToolbox实现音频播放。音视频同步和android采用的是一样。其中两端的ffmpeg逻辑是一致的。在ios端OpenGL实现视频渲染没有重点阐述如何使用OpenGL。这个有兴趣的同学可以自行研究。

备注：整个代码工程等整理之后会发布出来。

最后添加两张播放效果图