Written by Benjamin Lavialle
on 

Build a custom player on top of AVFoundation

When building an iOS/tvOS application with video playback, the best player solution can be to build a custom one to match the exact application requirements. This article is a technical overview of fundamental points to build a custom player on top of AVFoundation.

The idea is to use AVPlayer as a playback engine, leveraging its low level efficiency and stability.

Key points

To create a great player experience the important points are:

Provide the best stream to the player

AVPlayer receives its stream through an AVPlayerItem. To ensure a seamless user experience, the item preparation should be done as soon as possible and as fast as possible. Apple released a WWDC video about this.

Ahead of providing the AVPlayerItem to AVPlayer, some preparation can be done:

Manage the DRM

Here, we are talking about Fairplay DRM, the only technology supported by AVPlayer.

When an HLS stream is Fairplay protected, the HLS playlist owns a SESSION_KEY tag with an URI, this information is collected by the item to let the application and the OS load the key to decode the stream. To fetch the licence, we need to set an AVAssetResourceLoaderDelegate to the AVPlayerItem’s AVURLAsset. The delegate creates a hook on the AVURLAsset resource loading (in our case the licence) to let the app fetch the licence at playback. During the hook, the application needs to:

// mark - AVAssetResourceLoaderDelegate

func resourceLoader(_ resourceLoader: AVAssetResourceLoader,
                    shouldWaitForLoadingOfRequestedResource loadingRequest: AVAssetResourceLoadingRequest) -> Bool {
    guard loadingRequest.isContentKeyRelated else return { false }
    loadContentKey(for: loadingRequest) { result in
        switch result {
        case let .failure(error):
            loadingRequest.finishLoading(with: error)            
        case let .success(data):
            loadingRequest.dataRequest?.respond(with: data)
            loadingRequest.finishLoading()            
        } 
    }
    return true
}

// mark - Private

private func loadContentKey(for loadingRequest: AVAssetResourceLoadingRequest,
                            completion: (Result<Data, Error>) -> Void) {
    let uriData = loadingRequest.uriData
    getApplicationCertificate() { certificate in
        let challengeData = try loadingRequest.streamingContentKeyRequestData(forApp: certificate, contentIdentifier: uriData, options: nil)
        self.fetchContentKey(with: challengeData, uriData: uriData, completion: completion)
    }
}

It is also possible to prefetch the licence before playback, instead of on flight, at playback; we won’t talk about prefetch, but the mecanism is basically the same. Prefetching the key is a good way of upgrading the user experience, it avoids the content key fetch, few tenths of seconds length, task before playback start.

Seek to the right playback position

To have the playback starting at the user requested position, the idea is to use the AVPlayerItem seekTo methods. The methods take a time tolerance before and after, tolerance parameters should be used to allow a faster seek by taking advantage of the stream encoding.

The AVPlayer references its time with a CMTime type. If the stream is time based, this should be the case for any non-live stream, it is natural to perform a seek from a TimeInterval representing the position in the stream, starting at 0. The translation from TimeInterval to CMTime is then pretty straight forward.

On the other hand, for live stream, it is natural to navigate through playback with dates, as it represents an ongoing event. To do so, the HLS stream comes with a EXT-X-PROGRAM-DATE-TIME tag, allowing the player to translate a date to its own CMTime referencial.

Last advice: AVPlayerItem’s seekToDate method doesn’t complete if the item’s status is not readyToPlay.

Select the audio and subtitle language

The player may allow the user to select language and subtitles. It could also be an interesting feature to automatically apply user preferences at playback start. To ensure best performances, this can even be done before playback. To do so, the idea is to load the AVPlayerItem asset mediaSelectionOptions asynchronoulsy and select the proper options among the available ones.

extension AVPlayerItem {

    func ad_select(languageTag: String, subtitlesTag: String, completion: () -> Void) {
        ad_loadAssetContent { [weak self] in
            self?.ad_selectOption(tagged: languageTag, for: .audible)
            self?.ad_selectOption(tagged: subtitlesTag, for: .audible)
        }
    }

    // MARK: - Private

    private func ad_loadAssetContent(with completion: @escaping () -> Void) {
        let selector = #selector(getter: AVAsset.availableMediaCharacteristicsWithMediaSelectionOptions)
        let selectorString = NSStringFromSelector(selector)
        asset.loadValuesAsynchronously(forKeys: [selectorString], completionHandler: completion)
    }

    private func ad_selectOption(tagged optionTag: String, for mediaCharacteristic: AVMediaCharacteristic) {
        guard
            let option = asset
                .mediaSelectionGroup(forMediaCharacteristic: mediaCharacteristic)?
                .options
                .filter({ $0.extendedLanguageTag == optionTag })
                .first else {
            return
        }
        ad_select(mediaSelectionOption: option, for: mediaCharacteristic)
    }

    private func ad_select(mediaSelectionOption: AVMediaSelectionOption, for mediaCharacteristic: AVMediaCharacteristic) {
        guard let group = asset.mediaSelectionGroup(forMediaCharacteristic: mediaCharacteristic) else { return }
        select(mediaSelectionOption, in: group)
    }
}

Observe the player

Once the stream is playing on screen, the application will surely need feedback about the playback, first to display relevant controls and probably to monitor player performances and user activities.

Periodic observation

Following the playback is important, first, to update the controls transport bar, this can be done with a periodic time observer on AVPlayer.

observer = avPlayer.addPeriodicTimeObserver(
    forInterval: CMTime(seconds: 1, preferredTimescale: CMTimeScale(NSEC_PER_SEC)),
    queue: DispatchQueue.main
) {  _ in
    // Compute the current player state and provide it to the controls
}

Event observation

Periodic observation is perfect when the player is nicely playing the video with nothing else happening. But video playback is full of ambushes.

Most of the events come from the AVPlayerItem, and are catchable through KVO. loadedTimeRanges, isPlaybackBufferEmpty, isPlaybackLikelyToKeepUp, isPlaybackBufferFull or seekableTimeRanges are helpful to understand how the player buffer is doing. status is essential because it defines if the player has been able to fetch the content; this is generally where playback launch error occur.

avPlayer.replaceCurrentItem(with: playerItem)
observer = playerItem.observe(\.status) { [weak self] (item, _) in
    switch item.status {
    case .readyToPlay:
        self?.avPlayer.play()
    case .failed:
        // handle item.error
    default:
        break
}

Some event on AVPlayer are also important. rate, to connect to the play/pause button in the controls, or externalPlaybackActive to activate Airplay.

The NotificationCenter emits interesting events too, among which applicationWillResignActive/applicationDidBecomeActive, or AudioSessionRouteChange ; registering to those notifications will let the player pause and resume as the application goes into background, or update the UI if Airplay starts/stops.

Display controls

Once the stream is prepared and the player is playing properly, the last step is to display the controls on top of the player view. To keep the controls up to date with the playback, a good solution is to gather all the playback information into a single property object, publish this object on every playback change and register the view controller in charge of displaying the controls to those changes. It is then easy to connect each UI element to its related information.

Conclusion

Those mandatory steps are a good base to build a custom player; they handle the most important part of the playback. However, some additional work might be required to provide a great user experience. Among the possible extra features, depending on the project specificities, there could be monitoring player performances, providing advanced controls (such as pan to dismiss gesture) or even supporting Airplay or Google Chromecast.