Android Audio Tutorial [Part Three] : AudioFlinger Introduction and Initialization

AudioFlinger

We know that AudioFlinger (Sometimes called AF) is the core of the entire audio system in Android.

It is a system service which starts during boot and enables the platform for audio related use-cases in the following ways.

1.     Provide the access interface for the upper layer for using Audio.

2.     Uses the HAL to manage the audio devices.

Only by understanding AudioFlinger can we better penetrate into other modules based on this, so we put it in the forefront of our analysis.

Let us proceed further.

AudioFlinger Service: Startup and Operation.

System services in Android fall into two categories, namely Java Services and Native Services.

AudioFlinger and SurfaceFlinger [We will discuss SurfaceFlinger in some other tutorial series] belong to the latter.

Mediaserver starts all the native layer services [Viz: AudioFlinger, MediaPlayerService, CameraService, and AudioPolicyService] for achieving media related

From the file frameworks/av/media/mediaserver/main_mediaserver.cpp.

The mediaserver process is started by the init.rc file during boot.

It is worth mentioning that the AudioFlinger::instantiate() is not a static class within AudioFlinger, but an implementation of the BinderService class.

Several services, including AudioFlinger and AudioPolicyService, inherit from this unified Binder service class, such as.

The binderservice after all the binder related procedures [Please check the tutorial on Native service addition for details] adds this to the servicemanager entry.

Let us now look at the AudioFlinger's constructor. We see that it simply initializes some of the internal variables, and there is no code other than this.

AudioFlinger is an important entity. Other processes access it through the ServiceManager interface and call createTrack, openOutput and a series of interfaces to drive AudioFlinger to perform audio processing operations, which we will explain.

Audio Device Management

Although the AudioFlinger is successfully created and initialized above. It is still a static memory space so far and does not involve specific work.

In terms of the distribution of functions, AudioPolicyService() is the policy maker. It decides some of the important things like.

1.     When to open the audio interface device.

2.     What kind of Stream-type audio corresponds to what device etc.

AudioFlinger is the executor of the strategy defined and decided and does some of the following tasks.

1.     How to communicate specifically with the audio device.

2.     How to maintain the audio sstate sanity in the existing system.

3.     How to deal with the mixture of multiple audio streams, etc.

The audio interfaces supported by the Audio system [AudioFlinger] fall into three categories.

Each audio device interface supported implements a corresponding shared library.

So how does AudioFlinger know which interfaces are supported by the current device and which specific audio devices are supported by each interface?

One of the main responsibilities of AudioPolicyService() is to guide AudioFlinger to load device interfaces according to user configuration.

AudioPolicyManagerBase () (Details later) reads the contents of the file of audio_policy.conf and then opens the above three types of audio interfaces (if they exist).

This process will eventually call loadHwModule() of AudioFlinger as follows.

We see the loadHwModule_l() does a lot of work, Let us see them.

1.     First find out if mAudioHwDevs has added the audio interface indicated by the variable name, and if so, return directly.

a.     The size of mAudioHwDevs is NULL when entered for the first time, so it will continue down.

2.     Loads the specified audiointerface, such as "primary", "a2dp" or "usb".

3.     The function load_audio_interface() is used to load the library file needed by the device, then open the device and create an audio_hw_device_t instance.

a.     The library file name corresponding to the audio interface device has a certain format.

b.     For example, the module name of a2dp may be audio.a2dp.so or audio.a2dp.default.so.

c.     There are two main search paths.

4.     The init_check is to determine whether the audio interface has been successfully initialized. Zero is successful other values ​​indicate failure.

5.     Next, if this device supports the master volume, we also need to set it via set_master_volume ().

6.     Before each device operation, we must first change the mHardwareStatus value.

a.     After the end of the operation, it gets restored to AUDIO_HW_IDLE.

7.     Add the loaded device to the mAudioHwDevs key-value pair, where the key value is generated by nextUniqueId. This ensures that the audiointerface has a globally unique id number.

The completion of the module loading of the audio interface is only the first step in the long journey as each interface contains more than one device.

The list of  supported audio devices for Android are shown in the following list.

OUTPUT DEVICES:

INPUT DEVICES:

We would have a lot of questions after seeing this exhaustive list like:

1.     When we play back or record any audio Stream which one should I choose?

These decision-making tasks are done by AudioPolicyService (We will discuss this in detail later).

Let us see how the AudioFlinger opens an Output channel (an audiointerface may contain a number of output).

Open the audio output channel The corresponding interface in AF is openOutput().

We break this function in some steps as below.

1.     findSuitableHwDev_l:  Find the corresponding audio interface.

2.     outHwDev->hwDevice(): Load the corresponding Hal for the interface.

3.     outHwDev->openOutputStream: Open an output stream for the device.

4.     new OffloadThread/ new OffloadThread/ new MixerThread: Create PlaybackThread/Mixer/Offload thread depending on the audio_output_flags_t.

5.     thread->ioConfigChanged(AUDIO_OUTPUT_OPENED): notify client processes of the new output creation.

6.     mPrimaryHardwareDev->hwDevice()->set_mode.

We have seen that outHwDev is used to store an open audio interface device. Its data type is audio_hw_device_t, which is a set of attributes that an audio interface device specified by the HAL should have, as follows.

From the above structure definition.

common represents the common attributes of all devices in the HAL layer.

set_master_volume, set_mode, and open_output_stream provide interfaces for setting the audio volume for the master interface, setting the audio mode type (such as AUDIO_MODE_RINGTONE, AUDIO_MODE_IN_CALL, and so on), and opening the output data stream.

Let us revisit the functions we saw earlier, but this time in a slightly more detailed manner.

findSuitableHwDev_l.

NOTES:

The case where the value of the variable module is 0 is handled specially for compatibility with the previous Audio Policy. When module is equal to 0 all known audio interface devices are loaded first and then they are determined according to the devices.

When modules is non-zero, it indicates that Audio Policy specifies a specific device id number. At this time, the global mAudioHwDevs variable is checked to determine whether there is a device that meets the requirements.

DefaultKeyedVector<audio_module_handle_t, AudioHwDevice*>  mAudioHwDevs;

The variable mAudioHwDevs is a Vector with audio_module_handle_t as the key.

Each handle value uniquely identifies the audio device that has been added.

In first case, when the previous module is 0, it will load all potential devices. In the other case, AudioPolicyManagerBase will preload the output described in all audio_policy.conf.

In either case, we will eventually call loadHwModule -> loadHwModule_l, which we analyzed at the beginning.

If modules is non-zero, and the device that meets the requirements is also not found in mAudioHwDevs, the program will not terminate there - it will do its best, traverse through all the elements in the array and look for any audio interface that supports devices.

open_output_stream:

This approach is very common in the realization of the HAL layer. The function of audio_stream_out is nothing more than to initialize the function pointers according to the input parameters.

Let us see an example of the USB Audio HAL implementation.

After AudioFlinger::openOutput() the channel is already open. The next question in the puzzle is who is going to put something in the channel?

This is PlaybackThread.

Let us look at two different situations.

1.     DirectOutput : If you do not need to mix the streams.

2.     Mixer: Need to mix the streams.

These two situations give birth to two kind of threads “DirectOutputThread” and “MixerThread”, respectively.

Let us analyze the working of PlaybackThread in the latter case as an example.

As shown in the above hierarchy diagram, We can see there are many types of threads used for Playback, and their base class is Thread.

AudioFlinger has two global variables for Recording and Playback threads which are.

1.     DefaultKeyedVector< audio_io_handle_t, sp<PlaybackThread> >  mPlaybackThreads

2.     DefaultKeyedVector< audio_io_handle_t, sp<RecordThread> >    mRecordThreads

In openOutput (), We add mPlaybackThreads which is a newly created thread class instance, such as MixerThread(). 

 Let us see its constructor as follows.

The first step is to create an AudioMixer object. This is the key to mixing, which we will cover in more detail later.

Then check the number of channels. In the case of Mixer, there must be more than one channels.

Then create an NBAIO (Non-blocking audio I/O interface) sink (AudioStreamOutSink) and negotiate.

Finally, according to the configuration (initFastMixer) determine whether to use fast mixer.

We should know that the task of a playback thread is to continuously process the upper layer data request, then pass it to the next layer, and eventually write to the hardware device.

However, in the above function, it does not seem to see the program start a new thread, did not see into the thread loop, or to call other functions that may cause thread creation.

So under what circumstances MixerThread will really enter the thread loop?

I do not know if you have noticed the definition of mPlaybackThreads before, we again listed as follows.

DefaultKeyedVector< audio_io_handle_t, sp<PlaybackThread> >  mPlaybackThreads;

It is actually a key-value pair consisting of audio_io_handle_t and PlaybackThread strong pointers.

At the same time, we can also see that the ancestor of the PlaybackThread class is RefBase. How? Thread inherits from RefBase:

We know according to the characteristics of the strong pointer, the target object will call onFirstRef() when it is first referenced.

This function is implemented as follows.

Let us organize the content described in this section.

When the AudioPolicyManagerBase is constructed, it will analyze the audio interfaces (primary, a2dp, and usb) in the system according to the audio_policy.conf provided by the user, and then load the corresponding library files of the audio interface through AudioFlinger::loadHwModule and open them in sequence. Output(openOutput) and input(openInput)

We have analyzed in detail what openOutput does, including opening an audio_stream_out_t channel, generating an AudioStreamOut object, and creating a new PlaybackThread.

As long as AudioTrack and AudioFlinger continue to pass data, the entire audio playback continues.

We will continue analysis in the next tutorial.