③NuPlayer播放框架之类NuPlayer源码分析

③NuPlayer播放框架之类NuPlayer源码分析

[时间：2016-10] [状态：Open]

[关键词：android，nuplayer，开源播放器，播放框架]

0 引言

差不多一个月了，继续分析AOSP的播放框架的源码。这次我们需要深入分析的是NuPlayer类，相比于NuPlayerDriver的接口功能，NuPlayer继承自AHandler类，是AOSP播放框架中连接Source、Decoder、Render的纽带。

我希望读完本文大家可以对NuPlayer的源码结构有一定了解。
本文是我的NuPlayer播放框架的第三篇。

1 主要接口和核心的类成员

NuPlayer类被NuPlayerDriver直接调用，其主要接口如下：

// code frome NuPlayer.h (~/frameworks/av/media/libmediaplayerservice/nuplayer/)
struct NuPlayer : public AHandler {
NuPlayer(pid_t pid);
void setUID(uid_t uid);
void setDriver(const wp<NuPlayerDriver> &driver);
void setDataSourceAsync(...);
void prepareAsync();
void setVideoSurfaceTextureAsync(const sp<IGraphicBufferProducer> &bufferProducer);
void start();
void pause();

// Will notify the driver through "notifyResetComplete" once finished.
void resetAsync();

// Will notify the driver through "notifySeekComplete" once finished
// and needNotify is true.
void seekToAsync(int64_t seekTimeUs, bool needNotify = false);

status_t setVideoScalingMode(int32_t mode);
status_t getTrackInfo(Parcel* reply) const;
status_t getSelectedTrack(int32_t type, Parcel* reply) const;
status_t selectTrack(size_t trackIndex, bool select, int64_t timeUs);
status_t getCurrentPosition(int64_t *mediaUs);

sp<MetaData> getFileMeta();
float getFrameRate();

protected:
virtual ~NuPlayer();
virtual void onMessageReceived(const sp<AMessage> &msg);
}

接口分类下，无外乎几个分类：
用于初始化的（比如构造函数、setDriver/setDataSourceAsync/prepareAsync/setVideoSurfaceTextureAsync）
用于销毁的（比如析构函数、resetAsync）
用于播放控制的（比如start/pause/seekToAsync）
用于状态获取的（比如getCurrentPosition/getFileMeta）
下面是主要的类成员部分

wp<NuPlayerDriver> mDriver; // 接口调用方
sp<Source> mSource; // 相当于FFmpeg中的demuxer
sp<Surface> mSurface; // 显示用的Surface
sp<DecoderBase> mVideoDecoder; // 视频解码器
sp<DecoderBase> mAudioDecoder; // 音频解码器
sp<CCDecoder> mCCDecoder;
sp<Renderer> mRenderer; // 渲染器
sp<ALooper> mRendererLooper;

2 setDataSourceAsync实现分析

这个函数有多重不同的重载形式，如下：

void setDataSourceAsync(const sp<IStreamSource> &source);
void setDataSourceAsync(const sp<IMediaHTTPService> &httpService, const char *url,
const KeyedVector<String8, String8> *headers);
void setDataSourceAsync(int fd, int64_t offset, int64_t length);
void setDataSourceAsync(const sp<DataSource> &source);

需要根据实际情况选择，这里以第三个接口为例，说明下多本地媒体文件是如何处理的。

下面是这个函数的实现代码：

void NuPlayer::setDataSourceAsync(int fd, int64_t offset, int64_t length) {
sp<AMessage> msg = new AMessage(kWhatSetDataSource, this);

sp<AMessage> notify = new AMessage(kWhatSourceNotify, this);
// 创建对象用于读取本地文件
sp<GenericSource> source =
new GenericSource(notify, mUIDValid, mUID);
// 实际干活的的代码
status_t err = source->setDataSource(fd, offset, length);

if (err != OK) {
ALOGE("Failed to set data source!");
source = NULL;
}

msg->setObject("source", source);
msg->post();
}

看实现很简单，创建GenericSource对象，并调用其setDataSource接口，然后发送kWhatSetDataSource消息。

我们看看如何处理然后发送kWhatSetDataSource消息呢？代码如下：

case kWhatSetDataSource:
{
CHECK(mSource == NULL);

status_t err = OK;
sp<RefBase> obj;
CHECK(msg->findObject("source", &obj));
if (obj != NULL) {
Mutex::Autolock autoLock(mSourceLock);
mSource = static_cast<Source *>(obj.get());
} else {
err = UNKNOWN_ERROR;
}
// 通知Driver函数调用完成
CHECK(mDriver != NULL);
sp<NuPlayerDriver> driver = mDriver.promote();
if (driver != NULL) {
driver->notifySetDataSourceCompleted(err);
}
break;
}

看到这里发现，其实没做什么就是直接通知NuPlayerDriver。我们还注意到这里构建了一个特殊消息（AMessage）notify，这个消息用于在Source和NuPlayer直接传递。下面这是消息循环中的处理函数：

case kWhatSourceNotify:
{
onSourceNotify(msg);
break;
}

在后续讨论Source的时候详细说明这个消息通知的意义。

3 prepareAsync

这个函数实现的功能对应于MediaPlayerBase::prepare/prepareAsync接口，实现异步的prepare功能，一般就是做一些额外的初始化工作。那么直接看一下实现：

void NuPlayer::prepareAsync() {
(new AMessage(kWhatPrepare, this))->post();
}

代码就是发了一个kWhatPrepare的消息。接下来是如何处理这个消息。

case kWhatPrepare:
{
mSource->prepareAsync();
break;
}

最终还是调用了Source::prepareAsync接口。后面会解释其功能。（这里面可能会解析下码流，读取音频、视频、字幕流信息，读取时长、元数据等）。

4 setVideoSurfaceTextureAsync

调用这个接口主要为了设置视频渲染窗口。其实现相对简单，创建一个Surface，然后发送异步的kWhatSetVideoSurface消息。代码如下：

void NuPlayer::setVideoSurfaceTextureAsync( const sp<IGraphicBufferProducer> &bufferProducer) {
sp<AMessage> msg = new AMessage(kWhatSetVideoSurface, this);

if (bufferProducer == NULL) {
msg->setObject("surface", NULL);
} else {
msg->setObject("surface", new Surface(bufferProducer, true /* controlledByApp */));
}

msg->post();
}

那么看看如何处理kWhatSetVideoSurface消息呢？

case kWhatSetVideoSurface: {
sp<RefBase> obj;
CHECK(msg->findObject("surface", &obj));
sp<Surface> surface = static_cast<Surface *>(obj.get());

// Need to check mStarted before calling mSource->getFormat because NuPlayer might
// be in preparing state and it could take long time.
// When mStarted is true, mSource must have been set.
if (mSource == NULL || !mStarted || mSource->getFormat(false /* audio */) == NULL
// NOTE: mVideoDecoder's mSurface is always non-null
|| (mVideoDecoder != NULL && mVideoDecoder->setVideoSurface(surface) == OK)) {
performSetSurface(surface); // 通知NuPlayerDriver设置完成
break;
}
// 清空音频、视频缓冲
mDeferredActions.push_back(
new FlushDecoderAction(FLUSH_CMD_FLUSH /* audio */,FLUSH_CMD_SHUTDOWN /* video */));
// 最终调用NuPlayer::performSetSurface接口
mDeferredActions.push_back(new SetSurfaceAction(surface));

if (obj != NULL || mAudioDecoder != NULL) {
if (mStarted) {
// Issue a seek to refresh the video screen only if started otherwise
// the extractor may not yet be started and will assert.
// If the video decoder is not set (perhaps audio only in this case)
// do not perform a seek as it is not needed.
int64_t currentPositionUs = 0;
if (getCurrentPosition(¤tPositionUs) == OK) {
mDeferredActions.push_back(
new SeekAction(currentPositionUs));
}
}

// 对于新的surface设置，重置下解码器
mDeferredActions.push_back(new SimpleAction(&NuPlayer::performScanSources));
}

// After a flush without shutdown, decoder is paused.
// Don't resume it until source seek is done, otherwise it could
// start pulling stale data too soon.
mDeferredActions.push_back(
new ResumeDecoderAction(false /* needNotify */));
// 把上面mDeferredActions中缓存的所有Action处理下，并清空
processDeferredActions();
break;
}

这里的代码相对复杂点，涉及到很多，其实主要是为了设置Surface之后，可以正常解码显示，因为某些情况下解码器初始化需要依赖于具体的Surface。当然，里边还涉及到NuPlayer状态及初始化判断。

5 start/pause

start函数实现很简单，实际就发送了kWhatStart消息。

void NuPlayer::start() {
(new AMessage(kWhatStart, this))->post();
}

在消息处理函数中的处理如下：

case kWhatStart:
{
if (mStarted) {
// do not resume yet if the source is still buffering
if (!mPausedForBuffering) {
onResume();
}
} else {
onStart();
}
mPausedByClient = false;
break;
}

直接调用了OnStart/OnResume函数。

pause函数实现类似，只是发送的是kWhatPause消息。在消息处理函数中的代码如下：

case kWhatPause:
{
onPause();
mPausedByClient = true;
break;
}

直接调用的onPause函数。下面单独分析下这三个函数。先从简单的函数开始OnPause/onResume

NuPlayer::onPause

这个函数实现暂停功能，总体来说就是把Source和Render暂停就可以了，代码如下：

void NuPlayer::onPause() {
if (mPaused) {
return;
}
mPaused = true;
if (mSource != NULL) {
mSource->pause();
}
if (mRenderer != NULL) {
mRenderer->pause();
}
}

NuPlayer::onResume

这个函数实现恢复功能，代码逻辑跟onPause差不多，把Source和Render恢复，还可能涉及其它操作。代码如下：

void NuPlayer::onResume() {
if (!mPaused || mResetting) {
return;
}
mPaused = false;
if (mSource != NULL) {
mSource->resume();
}
// |mAudioDecoder| may have been released due to the pause timeout, so re-create it if
// needed.
if (audioDecoderStillNeeded() && mAudioDecoder == NULL) {
instantiateDecoder(true /* audio */, &mAudioDecoder);
}
if (mRenderer != NULL) {
mRenderer->resume();
}
}

NuPlayer::onStart

这个接口实现启动的操作，相对复杂点，需要初始化解码器、初始化Render、设置Source状态，并将三者关联起来。代码如下：

void NuPlayer::onStart(int64_t startPositionUs) {
if (!mSourceStarted) {
mSourceStarted = true;
mSource->start(); // 设置Source状态
}

// ... (省略部分代码)

sp<AMessage> notify = new AMessage(kWhatRendererNotify, this);
++mRendererGeneration; // 创建Render和RenderLooper，属性设置、与解码器关联
notify->setInt32("generation", mRendererGeneration);
mRenderer = new Renderer(mAudioSink, notify, flags);
mRendererLooper = new ALooper;
mRendererLooper->setName("NuPlayerRenderer");
mRendererLooper->start(false, false, ANDROID_PRIORITY_AUDIO);
mRendererLooper->registerHandler(mRenderer);

status_t err = mRenderer->setPlaybackSettings(mPlaybackSettings);

float rate = getFrameRate();
if (rate > 0) {
mRenderer->setVideoFrameRate(rate);
}

if (mVideoDecoder != NULL) {
mVideoDecoder->setRenderer(mRenderer);
}
if (mAudioDecoder != NULL) {
mAudioDecoder->setRenderer(mRenderer);
}

postScanSources();
}

上面代码中没有解码器的初始化，那只能继续看看postScanSources代码了。看实现发现就是发送了kWhatScanSources消息。那么消息循环里边是怎么处理的呢？

case kWhatScanSources:
{
int32_t generation;
CHECK(msg->findInt32("generation", &generation));
if (generation != mScanSourcesGeneration) {
// Drop obsolete msg.
break;
}

mScanSourcesPending = false;
bool mHadAnySourcesBefore = (mAudioDecoder != NULL) || (mVideoDecoder != NULL);
bool rescan = false;

// initialize video before audio because successful initialization of
// video may change deep buffer mode of audio.
if (mSurface != NULL) { // 初始化视频解码器
if (instantiateDecoder(false, &mVideoDecoder) == -EWOULDBLOCK) {
rescan = true;
}
}

// Don't try to re-open audio sink if there's an existing decoder.
if (mAudioSink != NULL && mAudioDecoder == NULL) { // 初始化音频解码器
if (instantiateDecoder(true, &mAudioDecoder) == -EWOULDBLOCK) {
rescan = true;
}
}

if (!mHadAnySourcesBefore && (mAudioDecoder != NULL || mVideoDecoder != NULL)) {
// This is the first time we've found anything playable.
// 设置定期查询时长
if (mSourceFlags & Source::FLAG_DYNAMIC_DURATION) {
schedulePollDuration();
}
}

status_t err; // 一些异常处理逻辑
if ((err = mSource->feedMoreTSData()) != OK) {
if (mAudioDecoder == NULL && mVideoDecoder == NULL) {
// We're not currently decoding anything (no audio or
// video tracks found) and we just ran out of input data.

if (err == ERROR_END_OF_STREAM) {
notifyListener(MEDIA_PLAYBACK_COMPLETE, 0, 0);
} else {
notifyListener(MEDIA_ERROR, MEDIA_ERROR_UNKNOWN, err);
}
}
break;
}
// 如果需要的话，重新扫描Source
if (rescan) {
msg->post(100000ll);
mScanSourcesPending = true;
}
break;
}

6 seekToAsync

这个函数完成seek操作，其实现比较简单直接发送kWhatSeek消息，代码如下：

void NuPlayer::seekToAsync(int64_t seekTimeUs, bool needNotify) {
sp<AMessage> msg = new AMessage(kWhatSeek, this);
msg->setInt64("seekTimeUs", seekTimeUs);
msg->setInt32("needNotify", needNotify);
msg->post();
}

在消息循环里边的处理代码如下：

case kWhatSeek:
{
int64_t seekTimeUs;
int32_t needNotify;
if (!mStarted) {
// Seek before the player is started. In order to preview video,
// need to start the player and pause it. This branch is called
// only once if needed. After the player is started, any seek
// operation will go through normal path.
// Audio-only cases are handled separately.
onStart(seekTimeUs);
if (mStarted) {
onPause();
mPausedByClient = true;
}
if (needNotify) {
notifyDriverSeekComplete();
}
break;
}

mDeferredActions.push_back(
new FlushDecoderAction(FLUSH_CMD_FLUSH /* audio */,
FLUSH_CMD_FLUSH /* video */));
// 真正做seek事情的在这里
mDeferredActions.push_back(new SeekAction(seekTimeUs));

// After a flush without shutdown, decoder is paused.
// Don't resume it until source seek is done, otherwise it could
// start pulling stale data too soon.
mDeferredActions.push_back(new ResumeDecoderAction(needNotify));

processDeferredActions();
break;
}

实际代码中SeekAction最终调用performSeek接口，其实现如下：

void NuPlayer::performSeek(int64_t seekTimeUs) {
if (mSource == NULL) {
// This happens when reset occurs right before the loop mode
// asynchronously seeks to the start of the stream.
LOG_ALWAYS_FATAL_IF(mAudioDecoder != NULL || mVideoDecoder != NULL,
"mSource is NULL and decoders not NULL audio(%p) video(%p)",
mAudioDecoder.get(), mVideoDecoder.get());
return;
}
mPreviousSeekTimeUs = seekTimeUs;
mSource->seekTo(seekTimeUs); // 直接调用Source对应接口
++mTimedTextGeneration;

// everything's flushed, continue playback.
}

7 resetAsync

重置函数实现逻辑相对简单，直接重置下，代码如下：

void NuPlayer::resetAsync() {
sp<Source> source;
{
Mutex::Autolock autoLock(mSourceLock);
source = mSource;
}

if (source != NULL) {
// During a reset, the data source might be unresponsive already, we need to
// disconnect explicitly so that reads exit promptly.
// We can't queue the disconnect request to the looper, as it might be
// queued behind a stuck read and never gets processed.
// Doing a disconnect outside the looper to allows the pending reads to exit
// (either successfully or with error).
source->disconnect();
}

(new AMessage(kWhatReset, this))->post();
}

消息循环中对于kWhatReset处理如下：

case kWhatReset:
{
mResetting = true;

mDeferredActions.push_back(
new FlushDecoderAction(
FLUSH_CMD_SHUTDOWN /* audio */,
FLUSH_CMD_SHUTDOWN /* video */));

mDeferredActions.push_back(new SimpleAction(&NuPlayer::performReset));

processDeferredActions();
break;
}

上面的SimpleAction是直接调用接口的，其实现如下：

void NuPlayer::performReset() {
cancelPollDuration();

++mScanSourcesGeneration;
mScanSourcesPending = false;
// 销毁Render
if (mRendererLooper != NULL) {
if (mRenderer != NULL) {
mRendererLooper->unregisterHandler(mRenderer->id());
}
mRendererLooper->stop();
mRendererLooper.clear();
}
mRenderer.clear();
++mRendererGeneration;
// 销毁Source
if (mSource != NULL) {
mSource->stop();

Mutex::Autolock autoLock(mSourceLock);
mSource.clear();
}
// 通知Reset完成
if (mDriver != NULL) {
sp<NuPlayerDriver> driver = mDriver.promote();
if (driver != NULL) {
driver->notifyResetComplete();
}
}

mStarted = false;
mPrepared = false;
mResetting = false;
mSourceStarted = false;
}

8 getCurrentPosition/getFileMeta

getCurrentPosition用于获取当前播放位置，直接通过Render的对应接口获取的。实现代码如下：

status_t NuPlayer::getCurrentPosition(int64_t *mediaUs) {
sp<Renderer> renderer = mRenderer;
if (renderer == NULL) {
return NO_INIT;
}

return renderer->getCurrentPosition(mediaUs);
}

getFileMeta获取媒体的元数据信息，直接通过Source的对应接口获取。实现代码如下：

sp<MetaData> NuPlayer::getFileMeta() {
return mSource->getFileFormatMeta();
}

9 总结和疑问

到这里，我们已经把NuPlayer主要的函数分析完了，但是问题依旧在。比如下面几个：
不同格式的多媒体文件如何探测并解析的？音视频数据缓冲区在哪里？（Source）
视频如何显示的？音频如何播放的？音视频同步在哪里？（Renderer）
音频解码线程、视频解码线程在哪里？ （DecoderBase）
我想接下来几个主题就是解决这些疑问的。
当然总结下本文的内容。

主要参考AOSP 7.0的源码，结合代码分析了NuPlayer主要对外接口的实现，并简单总结了各部分的功能。