Eyeson AI Adapter

Overview of the Eyeson AI Adapter

The Eyeson AI Adapter architecture consists of two essential components:

1. The Eyeson API endpoint group forward stream, which manages the streaming data transfer
2. A customer-implemented AI Module that processes the streams and returns layout instructions or enhanced video/audio content

These components work together to enable real-time AI processing of video conference streams while maintaining seamless integration with the Eyeson platform.

 

 

The diagram above illustrates the stream forwarding process. When a POST request is sent to the /forward/source endpoint, it initiates the forwarding of video/audio streams to the URL specified in the request parameters. The streams at the top of the diagram are unidirectional (send-only), while the streams at the bottom support bidirectional communication (send/receive), enabling both data transmission and reception.

At the ingress of your AI module, you must process the incoming stream data (image and/or audio) to prepare it for analysis within your AI pipeline. This typically involves decoding the stream, extracting the relevant frames or audio segments, and formatting the data according to the requirements of your AI processing algorithms.

info

Streams are forwarded in their native format without any modification or transcoding, preserving the original quality and characteristics of the audio and video data. This ensures maximum compatibility with your AI processing pipeline while maintaining the integrity of the source material.

At the end of the AI pipeline, you have the option to implement various actions that can be transmitted to the API as commands. These actions enable real-time response to the AI analysis results and can be integrated with other Eyeson functionalities.

• Triggering an alert and setting an overlay.
• Changing the layout and moving the focus to a certain stream.
• Writing a transcript to a file
• Adding a screen capture to a report
• a.s.o.

The system architecture provides flexible stream routing capabilities, allowing you to:

1. Direct multiple source streams to a single AI pipeline for consolidated processing
2. Send a single source stream to multiple AI pipelines for parallel analysis
3. Create custom combinations of stream-to-pipeline mappings

This flexible routing approach enables you to optimize your AI processing based on available bandwidth and specific analytical requirements, ensuring that each source stream receives the most appropriate intelligence processing.

info

When a source stream stops delivering data, the corresponding forward command will automatically terminate. To maintain continuous stream forwarding, you must implement monitoring in your ingress system to detect disconnections and automatically reinitiate the forward command when needed.

AI Integration Models

Modern video communication systems are increasingly incorporating AI capabilities to enhance functionality and user experience. This section examines different approaches to AI integration, from device-level implementation to sophisticated multi-AI architectures.

Stage 1: Compressed Video

Here the AI preselects sources to put into the next stage. Example: Motion detection selects potential candidate sources automatically.

Stage 2: Uncompressed Video

The AI narrows the selection by further analyzing the content. Example: Object detection further prioritizes streams.

Stage 3: Composed Video

Multiple synchronized sources are merged into one composed uncompressed real-time stream. This provides a seamless view of the scene by integrating feeds such as Drone video, body-mounted cameras, and collaboration cameras. The AI can analyze the composed video and see information in context. Example: Data from multiple sensors is synchronized to capture the scene context.

Stage 4: Distributed Interactive Video

Human operators, AI and video sources (like drones, bodycams, sensors, and so on) can interact and orient. Example: The Intelligent system loops in human operators (Human in the Loop) as necessary.

Advanced Implementation: Edge MCU with Multiple AIs

The most sophisticated implementation involves an Edge MCU architecture that integrates multiple AI systems. This model provides several key advantages:

1. Distributed AI Processing
2. Real-time Stream Processing
3. Intelligent Content Routing
4. Enhanced Collaboration Features

The architecture follows this structure:

This configuration allows for:

• Parallel AI processing
• Intelligent stream filtering
• Dynamic layout switching
• Real-time content enrichment
• Immediate collaboration integration

The multi-AI approach provides greater flexibility and functionality compared to single-AI implementations, while maintaining the efficiency benefits of edge computing.

AI Model suggestions for Edge AI

Here's a table giving you ideas on your options for Edge AI:

Datasets	Examples of Models	Use Cases and benefits
Compressed Video	● MobileNet Lightweight motion detection ● Coviar action Recognition	Real Time Actions or motion detection ● Streams filtering and prioritization -> Layout API ● Smaller dataset for lower CPU and Energy consumption
Decompressed Video	● YOLO - Object Detection ● C3D - Motion and object detection ● I3D - Action Recognition ● ViT - Image classification ● CLIP - Image captioning	Real Time Complex actions detection ● Streams filtering and prioritization -> Layout API Real Time Complex objects detection ● Streams filtering and prioritization -> Layout API ● Content enrichment (visual tagging, image captioning) ● STAGED AI: Streams Routing to cloud LLM for additional analysis
Composed Video	● HMFNet ● V-JEPA - Layouted Video + Audio + Metadata. One composed uncompressed real time stream of multiple synchronised sources	Real Time Scene understanding ● Predictions and recommendations -> Event predictions, live recommendations, automated additional content collection or suggestions (updated maps, additional sources) from LLMs ● Streams Routing for escalation to LLMS Real Time Data quality augmentation ● X synchronised sources: X videos with different angles in 1 video stream (targeting)
Individual or mixed Audio	● NLP Models: ● STT ● TTS ● Sentiment Analysis	Real Time Content Enrichment ● Voice Transcription, Translation, Sentiment analysis, sounds analysis Layout Piloting and streams routing ● Voice activation, Key words

Relevant links

• /rooms/ROOM_ID/forward/source
  This endpoint allows the forwarding of one source stream to an AI ingress.
• /rooms/ROOM_ID/forward/mcu
  This endpoint forwards the whole One View, you would use this for transcription or reasoning.
• /rooms/ROOM_ID/forward/playback
  This endpoint allows the forwarding of a playback.