Understanding SFU, MCU & Eyeson’s MCU+

Introduction

This documentation explains the differences between Selective Forwarding Units (SFU) and Multipoint Control Units (MCU) and highlights how Eyeson's innovative MCU+ technology combines the best of both worlds to deliver superior performance.

Eyeson’s MCU+ extends the traditional MCU framework by integrating advanced features such as dynamic layering, customizable layouts, and intelligent bandwidth optimization. These enhancements overcome many limitations of standard MCU implementations and provide a robust platform that enhances the development and deployment of complex video conferencing solutions.

This guide is designed with two key audiences in mind:

1. AI Developers: Discover how integrating AI can leverage real-time video data to perform intelligent layout adjustments, participant detection, and dynamic resource allocation.
2. WebRTC Developers: Learn how to integrate Eyeson’s MCU+ within WebRTC applications efficiently, ensuring optimal performance and handling of media streams in diverse network conditions.

This documentation provides insights and technical guidance for developing intelligent video experiences and optimizing real-time communication.

What is an SFU?

A Selective Forwarding Unit (SFU) is a media server that efficiently routes audio and video streams between endpoints without decoding or re-encoding them. In an SFU setup, each participant receives individual streams, and the server forwards these streams directly to the respective clients.

The above SFU cluster has 5 participants, each participants sends his stream and receives the streams of the other participants.

What is an MCU?

A Multipoint Control Unit (MCU) is a media server that aggregates, processes, and composits multiple audio and video streams into a single or a few unified streams. This centralized processing allows the server to create pre-defined layouts and adjust the streams before delivering them to clients.

In the above MCU cluster topology, media stream compositing occurs server-side, with each participant transmitting a single input stream and receiving a single composite output stream. This architecture minimizes client-side bandwidth and processing requirements compared to peer-to-peer or SFU models.

Eyeson’s MCU+: Extending Traditional MCU

MCU+ is Eyeson’s advanced evolution of traditional Multi-Control Unit (MCU) technology, designed for real-time, scalable, and efficient media processing in video communication. It combines the strengths of conventional MCU with innovative enhancements to address modern video conferencing needs, offering superior performance, flexibility, and patented smart bandwidth optimization.

How MCU+ Builds Upon Traditional MCU Technology

Traditional MCUs aggregate and process all participant streams centrally, ensuring high-quality output but often at the cost of significant bandwidth and computational resources. MCU+ retains this centralized processing model but incorporates advanced features like low-latency transcoding, dynamic layouts, and custom data and media stream integration. This makes it more efficient and adaptable compared to legacy MCUs.

Key Enhancements of MCU+

• Layers for Stream Management: The layering system in MCU+ enables seamless integration of multiple media sources (e.g., video, audio, and custom data), background images, and overlays into a single stream. This improves synchronization and reduces complexity for end-users.
• Advanced Layouts: Customizable layouts allow users to organize multi-participant video streams dynamically. This feature enhances usability in collaborative meetings or virtual situation rooms by tailoring the visual arrangement to specific needs.
• Bandwidth Optimization: By transcoding streams into a single output with stable bandwidth requirements (e.g., approximately 1.5 Mbps regardless of participant count in standard settings), MCU+ minimizes network strain. This efficiency is particularly advantageous in significant group calls compared to traditional MCUs or SFUs.

MCU+’s Comparative Benefits

• Resolving MCU Limitations: Traditional MCUs often struggle with scalability and high bandwidth usage. MCU+ addresses these issues with optimized transcoding and cloud-native scalability.
• Performance Over SFU and MCU: Unlike SFUs (Selective Forwarding Units), which rely on participants’ devices for stream processing, MCU+ offloads this burden to a central server. This ensures consistent quality even with low-powered devices or unstable networks. Compared to traditional MCUs, MCU+ offers better resource efficiency and advanced customization options.

Scenarios Where MCU+ Excels

MCU+ outperforms both SFU and traditional MCU in scenarios requiring:

• Integration of custom media or data streams into live calls and virtual situation rooms.
• Dynamic layouts for professional presentations or collaborative environments.
• High-quality recording that mirrors the exact participant's view during the meeting (and is thus fulfilling a lot of compliance requirements).
• Large-scale video conferencing with stable bandwidth.

Eyeson’s MCU+ sets a new standard in video communication technology by combining robust performance with innovative features.

AI Developer Track

AI integration with Eyeson’s MCU+ technology enables intelligent, real-time enhancements to video conferencing by leveraging advanced algorithms for video processing and compositing. These AI-driven capabilities improve user experiences, optimize performance, and unlock new functionalities.

AI in Real-Time Video Processing and Compositing

AI algorithms can analyze and manipulate video streams in real-time, enabling dynamic scene understanding , intelligent layout adjustments, and detect emotions. For example:

• Dynamic Scene Understanding: AI algorithms analyze video streams to detect presence, position, and activity of people and objects in the frame, enabling automated responses to scene changes through dynamic focus shifting and real-time layout optimization.
• Intelligent Layout Adjustments: Based on participant activity or meeting context, AI can rearrange video layouts dynamically to enhance visibility or engagement.
• Emotion detection: AI-powered analysis of participants to capture interest during calls.

APIs and Hooks for AI Integration

Eyeson provides an AI Adapter, allowing developers to integrate video streams into custom AI pipelines. Streams can be forwarded to one or more external AI systems for processing tasks like image recognition, transcription, or emotion analysis. The processed data can then trigger actions such as layout changes or overlays.

Building Custom AI Modules

Developers can create tailored AI modules by leveraging Eyeson’s APIs to manipulate video streams. For instance:

• Use computer vision to detect objects or faces in the stream.
• Apply deep learning models for real-time transcription or sentiment analysis.
• Integrate LLMs for scene understanding and situational updates (explaining the visual changes the AI might make).

Performance Considerations

Balancing computational load between AI processing and MCU+ is critical. Best practices include:

• Using GPU acceleration for faster processing.
• Optimizing models for low latency to maintain real-time performance.
• Employing edge computing for resource-intensive tasks.

Testing and Optimization

Effective workflows involve:

1. Rigorous testing of AI models under varying network conditions.
2. Evaluating latency impacts on user experience.
3. Iteratively optimizing pipelines for both speed and accuracy.
4. Training the affected team in what the system can do early on in development.

Case Studies/Examples

1. Dynamic Layouts with IP Camera Focus: An organization used Eyeson’s MCU+ with AI to automatically focus on IP Camera Focus during search operations, improving engagement while reducing manual intervention.
2. Status updates from scene understanding: Another case study used Eyeson’s MCU+ to give textual feedback directly in the virtual situation room, letting everybody know updates on ETAs and area updates from drone video.
3. Privacy Enhancements in Healthcare: A telemedicine provider employed facial blurring algorithms via Eyeson’s API to anonymize patient identities while maintaining high-quality consultations.

By seamlessly integrating AI into its MCU+ platform, Eyeson empowers developers and businesses to create more intelligent, more adaptive video conferencing solutions tailored to diverse use cases. In case you want to discuss your idea and/or want to see our AI demo, please get in touch with us.

WebRTC Developer Track

Overview of How MCU+ Interacts with WebRTC

MCU+ serves as a central processing unit for WebRTC media streams, receiving individual participant streams, processing and mixing them into a single composite stream, and redistributing it to all participants. This approach ensures consistent quality across devices and reduces the bandwidth requirements for end users.

Setting Up a WebRTC Connection with Eyeson’s MCU+

To connect WebRTC clients to Eyeson’s MCU+, developers can use the Eyeson API:

1. Initialize a Meeting: Use the API to create a meeting room and retrieve the unique ROOM_ID.
2. Configure Participants: Add up to 100 participants by generating unique access links.
3. Customize Layouts: Define layouts dynamically based on meeting context using API endpoints.
4. Stream Forwarding: Forward streams to an AI using the POST /rooms/ROOM_ID/forward/source endpoint, specifying parameters like user_id, type, and url.

Leveraging Advanced Layouts and Bandwidth Optimization

• Advanced Layouts: Use Eyeson’s dynamic layout features to customize participant arrangements, ensuring optimal visibility for active speakers or shared focus on content.
• Bandwidth Optimization: MCU+ processes all streams server-side, reducing the bandwidth demand on individual participants to approximately 1.5 Mbps regardless of participant count. This is especially effective in large-scale meetings or low-bandwidth environments.

APIs and Customization

Detailed API Documentation

Eyeson provides a comprehensive API suite for managing meetings, participants, layouts, and streams:

• Signaling: Control participant connections and meeting state.
• Stream Management: Forward individual or composite streams using endpoints like /forward/source.
• Error Handling: Monitor session health using event hooks such as participant_update or playback_update.

Custom Hooks for Integration

Developers can hook into Eyeson’s signaling and media management systems to build custom workflows. For example:

• Integrate live data overlays (e.g., scores or analytics) into streams.
• Trigger layout changes programmatically based on meeting events or AI pipeline feedback.

Tools for Monitoring WebRTC Streams

• Use Eyeson’s built-in session monitoring tools to track stream health and participant activity.
• Employ third-party WebRTC debugging tools like Chrome’s WebRTC Internals for detailed insights into connection stability and media flow.

How to Get Started with Eyeson’s MCU+

Quickstart Guide

1. Create an Eyeson Account: To access the platform, sign up for a free developer account on the Eyeson website.
2. Generate an API Key: Create an API key to authenticate your application once registered. Follow the instructions provided in the Quickstart Guide for step-by-step guidance.
3. Join a Meeting: Open your browser, create a room, and invite participants using the generated guest link.

Documentation Resources and API References

• API Reference: Access detailed documentation on available endpoints, parameters, and responses in the Eyeson API Reference.
• Interactive Demos: Explore Eyeson’s capabilities with hands-on demos for layouts, live stream integration (e.g. drones), and meeting manipulation at Interactive Demos.
• Sample Projects and How-tos: Learn how to build custom applications using Eyeson’s API with examples like creating virtual situation rooms in Python or integrating live streams.

FAQ

How do SFU and MCU perform under high network jitter?

SFU Performance

SFUs are more vulnerable to jitter effects since they handle multiple independent streams. When network jitter occurs, it can cause:

• Delayed and choppy audio/video for individual participant streams
• Synchronization issues between different participant feeds
• Increased packet loss during bursty connections

MCU Performance

MCUs handle jitter more efficiently for participants because:

• Only one composite stream needs to be managed per client
• The centralized mixing reduces the complexity of jitter compensation
• Server-side processing provides more consistent delivery timing

Jitter Compensation Mechanisms

WebRTC Adaptive Jitter Buffer

Both architectures utilize WebRTC's adaptive jitter buffer, which:

• Dynamically adjusts buffer size based on network conditions
• Collects and reorders packets before playback
• Handles lip synchronization between audio and video

Performance Factors

Common causes of jitter affecting both architectures include:

• Poor location and signal strength
• Network congestion
• Faulty hardware
• CPU limitations

Practical Implications

For high-jitter environments (like mobile networks or congested connections), MCUs generally provide more stable performance because:

• Clients only need to manage one incoming stream
• Server-side mixing creates more predictable packet timing
• Reduced client-side processing requirements help maintain stability

The SFU's handling of multiple streams makes it more susceptible to compound jitter effects, especially in poor or unstable network conditions.

What are typical latencies for SFU and MCU solutions?

Feature	SFU	Traditional MCU	Eyeson MCU+
Latency	Low (30–200 ms)	Moderate (200–500 ms)	Low (30–300 ms)
Scalability	High	Limited	High
Bandwidth Efficiency	Requires more	Centralized	Optimized (~1.5 Mbps)

How does the bandwidth optimization in Eyeson work?

The bandwidth optimization will measure network and endpoint parameters and deliver the best possible quality to the endpoint user. It will switch between specific video and audio settings, which you can find in the table below.

Quality	Bitrate Range	Codec	Resolution	Bitrate	Framerate
Low	< 400 kbps	AV1	720p	175 kbps	10 fps
		VP9	720p	200 kbps	6 fps
		VP8	720p	400 kbps	25 fps
Medium	400 - 700 kbps	AV1	720p	500 kbps	14 fps
		VP9	720p	500 kbps	12 fps
		VP8	720p	800 kbps	25 fps
High	> 700 kbps	AV1	720p	1500 kbps	25 fps
		VP9	720p	1500 kbps	25 fps
		VP8	720p	1500 kbps	25 fps

Audio

Quality	Bitrate Range	Codec	Bitrate	Stereo
Low	< 50 kbps	OPUS	32 kbps	no
High	>= 50 kbps	OPUS	64 kbps	yes

Eyeson’s cloud offering supports user-side upload bandwidths down to 175kbps. If you want to go below 200kbps, we recommend an on-premise solution with a use-case-specific bandwidth split. The same goes for different codecs and/or stream resolutions.