Safari SFU

Finally! Safari and SFU became good fellows.

Eyeson is based on this great approach of Single-Stream-Technology. All participants are basically merged into one audio/video stream that opens a great window of advantages. This technique is called MCU ("Multipoint Control Unit").

Some time ago, Eyeson introduced a new mode called SFU ("Selective Forwarding Unit") to directly send your audio/video stream to your video call participant and vice versa. This mode is only activated when 2 people have a video call. As soon as a 3rd participant joins, Eyeson switches from SFU to MCU mode in order to maintain quality improvements.

The SFU mode is great to see the conversation partner in full video size instead of half-half merge of its 2 participants. Unfortunately, Safari users had to stick with the half-half way, because of Safari's technical problems.

Now the technical part beginns:

When a browser joins a meeting it connects with the server and both negotiate audio and video codecs and more. A 2nd participant joins and the server tells all participants to switch from server's stream to each participant's stream, that includes new Synchronization Sources. All participants are still connected with the server and receive their streams from the server, but the server is now in loop-through mode aka SFU.

Allow SFU switch

From Eyeson's very beginning, all connected browsers need to use the VP8 video codec. If VP8 is not supported, the server has to handle it. But this means, that older Safari browsers were not able to use the SFU mode, since they couldn't talk to other browsers directly! Since version 12.1, Safari also supports VP8 video codec for WebRTC connections, but still technical issues remain…

FeatureDetector.canSFU() // now includes Safari >= 12.1

Safari issue #1: missing audio 🔇

When Safari (>= 12.1) switches to SFU mode, it lacks the audio! Our research revealed that this can be solved when the server doesn't change the audio's synchronization source 🎉. The video's synchronization source on the other hand needs to get changed. But this leads to new trouble.

FeatureDetector.disallowAudioSyncSrcChange() // special treatment in GUI is needed

Safari issue #2: broken mediastream on new media elements 😞

When Safari is connected to the server, the remote mediastream is added to a video element's srcObject. After the SFU switch has happened (= ssrc updates), Safari is not able to apply the existing remote mediastream to a new video or audio element. So for example when the remote video is turned off we can't remove the video element and add the remote mediastream to a new audio element. This won't work.

We could overcome this problem with a separate audio element from the very beginning that persists throughout the whole meeting. This will survive the ssrc updates 🎉.

videoElement = document.createElement('video');
videoElement.srcObject = remoteStream;
if (FeatureDetector.disallowAudioSyncSrcChange()) {
    audioElement = document.createElement('audio');
    audioElement.srcObject = remoteStream;
    videoElement.muted = true;
}

// later, this becomes possible
if (StreamHelpers.hasAudio(remoteStream) === false) {
    if (FeatureDetector.disallowAudioSyncSrcChange() === false) {
        audioElement = document.createElement('audio');
        audioElement.srcObject = remoteStream;
    }
    videoElement.remove();
}

Safari issue #3: silence forever when joining muted 🙉

We have the separate audio element in place, but belive it or not, it will never start playing when remote audio starts muted but turns on the microphone later. Luckily we could find a workaround! The separate audio element requires ONLY audio tracks 🎉. This is achieved by cloning the remote mediasoucre and removing all its video tracks.

videoElement = document.createElement('video');
videoElement.srcObject = remoteStream;
if (FeatureDetector.disallowAudioSyncSrcChange()) {
    audioElement = document.createElement('audio');
    remoteStreamClone = remoteStream.clone();
    remoteStreamClone.getVideoTracks().forEach(track => remoteStreamClone.removeTrack(track));
    audioElement.srcObject = remoteStreamClone;
    videoElement.muted = true;
}

Safari issue #4: no Giphys 😱

Safari is not able to switch from srcObject to src input?!? Whenever a mediastream was added to srcObject, you can't clear the srcObject and apply a linked media to the element's src. There is no technical approach to force the video element to do so. To make it work somehow, we have to use a separate video element. When playback starts, the remote stream video element is hidden and the playback video element is shown until the playback comes to its end 🎉.

if (FeatureDetector.isSafari()) {
    giphyVideo = document.createElement('video');
    giphyVideo.autoplay = true;
    videoElement.style.display = 'none';
    document.body.appendChild(giphyVideo);
    giphyVideo.src = 'https://media.giphy.com/media/rCmC12OWz9kTS/source.gif';
} else {
    videoElement.srcObject = null;
    videoElement.src = 'https://media.giphy.com/media/rCmC12OWz9kTS/source.gif';
}

Beware of trouble ✨

To remain previous behaviour in your application, you can simply set an option that saves you from all the trouble! See https://docs.eyeson.com/docs/js/fundamentals/configuration

eyeson.config.allowSafariSFU = false