flow-docs
Flow Platform
The Flow Platform is an innovative solution to handle a variety of video streaming and encoding needs. With the capability to source streams from multiple providers, normalize to a consistent format, and serve it with low latency, it’s a valuable tool for both customer-facing and internal products.
Supported Products
Several products have been built upon the Flow Platform, or simply leverage its capabilities:
eXamino
- Description: A video player developed as a replacement for the Amino hardware player post its retirement.
- Features:
- Facilitates data collection and offers support for direct NALU streaming, making it suitable for CV tasks.
- Features local edge functionality to reduce data traffic for parallelly working data collection agents in offices.
OptaVoice
- Description: Isolates audio tracks from incoming streams and publishes them separately to different endpoints on Akamai.
OptaVision
- Description: Captures frames from 3D animations with precision and facilitates their upload to the CMS.
Key Components
Discovery
- Manages API integrations with stream providers to be notified of new streams or changes to existing ones.
- Examples: Dataminer API and Xeatre API.
- Design: Discovery components are modular. Each provider gets a dedicated lambda cron job that checks the respective provider’s API every 5 minutes. Any changes detected are stored in a DynamoDB.
Scheduler
- Role: Operates as a lambda cron, responsible for initiating the stream-handling containers at the right time and restarting them in case of failures.
Probe
- Role: Before the commencement of each incoming stream, the Probe lambda checks the stream’s accessibility and its audio and video tracks.
Containers / Servers
Primarily dockerized Node.js processes built on ffmpeg, orchestrated using ECS. However, there’s an option for Kubernetes-based task management too.
Origin
- Transforms the incoming stream into the desired output format. Depending on the state of the incoming stream, it may also transcode the stream as required.
Segmenter
- Ensures all streams are available in both live and VOD DVR formats. The Segmenter breaks down the incoming stream into TS segments and stores them in S3 along with related manifest files.
Edges
- Handles traditional data distribution, streaming the content to consumers via websocket, whether it’s a browser or a CV task.
Dashboard
- Description: An administrative Single Page Application (SPA) interface.
- Features:
- Allows monitoring of the system’s operations in real-time.
- Provides audit logs.
- Enables viewing of active streams.
APIs
- Types: RESTful and Discovery APIs.
- Purpose:
- They cater not only to the dashboard but also enable integration with other company products and services.
- Integrations include but aren’t limited to CV pipeline, opotaVision, optaVoice, VDP, and more.
Design Considerations
- Modularity: The main focus was on creating a modular system.
- Scalability: Ensuring scalability with cloud-based infrastructure was paramount.
- Microservices Architecture: Designed to be easily extendable using microservices.
- Simplicity: The primary components only handle elementary tasks, keeping the system’s complexity low and operationally cost-effective.
- Separation: Each module operates in complete isolation, allowing for flexibility to adjust them as per specific use cases.
- Transparency: All components are mandated to log major events in a unified format, making it easy to trace the lifecycle of a stream.