The architecture for Dalet Flex utilizes a micro-service approach. The whole Dalet Flex system, including Dalet Flex Enterprise, Events, Authorization, MAM, and Web Transfer are containers, and it is entirely up to you (the client) how you want to run these containers. The main advantages of this micro-service approach are as follows:
- The architecture is fully containerized, and so, there is no fixed architecture.
- Each client can run the containers in their own configuration, and host them using any provider they like, such as Azure, Amazon Web Services, and so forth.
- Clients can pick and choose which services they want. A client might only require certain Da;et Flex services, e.g. Dalet Flex Enterprise, with just FlexMAM, and FlexMOVE.
- Each container is elastic, meaning that they can grow and shrink in size.
Note: If a client decides not to host themselves, then Dalet will choose to host using Amazon Web Services by default.
The diagram below, gives you a high level overview of the Dalet Flex architecture. You can see how
Enterprise, the Admin Apps, End User Apps, Services, Data Services, and Configuration Store sit together.
Note: This diagram may be subject to change as and when new services are added to Dalet Flex.
An Overview of the Key Components
- MAM: This is a Media Asset Management tool, with an asset-centric UI and functions such as metadata management, timeline metadata logging, search, and in-browser rough-cut editing.
- Workflow Designer: A HTML5 based tool that enables you to design workflows based around your existing processes.
- Metadata Designer: A HTML5 based tool that enables you to design metadata to accompany your assets.
- Reviewer: A tool that enables you to collaboratively review your uploaded media assets.
- Configuration Manager: A stand-alone tool that enables you to remotely configure Dalet Flex environments.
- Web Transfer: An application which uploads files, inputs metadata, and triggers workflows.
- Admin: This is used to control the log levels of the system. In essence, it is a system health checker, which monitors specific components.
- Authentication Service: The Authentication Service provides user authentication and management functions. It controls login capabilities, administers user passwords and generates access tokens.
- Authorisation Service: The Authorisation Service provides the logic to check permissions for a specific user to query or search any objects via the Search Service (assets, groups, collections etc.). It also provides which actions (list, read, create, update and delete) can be performed on queried objects and which actions cannot.
- Data Aggregation Service: The Data Aggregation Service provides data for the UI summary tables and operations dashboards. This service maintains counters and runs / creates daily aggregate data based on the events observed for each Dalet Flex object such as: assets, workflows, jobs, players, resources and tasks.
- DIVArchive Service: This is an external resource service that integrates DIVA Oracle systems for archiving, deleting assets from archive, and restoring the assets back to archive.
- Dalet Flex CM: Dalet Flex Configuration Manager is an operational tool used to manage configuration migration between different Flex instances.
- Events Service: This service manages persistence and retrieval of events to and from MongoDB. The Flex Events service is an events bus that distributes events that occur for Flex objects. For example: when the metadata of a media asset is edited. These events are not only persisted for later retrieval, but are broadcast so that other services can subscribe to them.
- Fast Object Service: This service quickly indexes media assets into ElasticSearch.
- Enterprise: Dalet Flex originated as a monolithic code-base known as “Enterprise”. Although the majority of the services have been re-developed as microservices, and applications are developed independently, many functions and capabilities still remain within Enterprise. Enterprise is the central orchestration platform. It provides asset and metadata management, job scheduling, event handlers, and timed actions, storage resources, users, accounts, and the B2B REST API.
- Hot Folder Service: This service polls a specific file system location (local, S3, Azure) configured as part of hot folder configuration for any changes such as: a new file has been added, file updated, or file deleted
- Identity Service: This service provides user permission information for search results, filtering, and authorisation.
- Image Proxy Service: This service resizes an image to a given width and height.
- Index Elastic Service: This service indexes Flex Objects into ElasticSearch indices. The service listens to events that occur for a Flex object and retrieves object data when it receives it, eventually indexing them into ElasticSearch.
- Job Engine Service: The Job Engine Service is a JEF service. It processes Enterprise (or other JEF client) requests (JobExecutionRequest), expands configuration using execution configuration services, and dispatches job execution to sync / async (or resource) service executors.
- Job Sync Executor Service: This service forms part of the Job Execution Framework (JEF) and provides the foundation to help developers define, execute, and deploy new action and resource components in a micro-service architecture.
- LMPP Service: This service breaks down a large XML manifest into a number of smaller XML manifests.
- Metadata Merge Service: This service updates a metadata instance, by merging new changes into the old one. When a metadata definition is updated in Dalet Flex Enterprise, the Metadata Merge Service will receive a request from Dalet Flex Enterprise through RabbitMQ to merge new and old versions of a metadata instance.
- Outbound Transfer Service: The Outbound Transfer Service is an external resource framework service. It handles transfers that originate from Flex that target an external VFS location. It supports the copying of a file from the source VFS location (Dalet Flex) to a target VFS location (an external system). This service delegates the work to the MioFileSystem library in order to perform the actual transfer.
- Package Service: The Package Service enables the use of package assets, as well as the ability to import, edit and export IMF and DCP packages, and CPL playlists.
- Publish Indexer Service: This service is used to publish assets (videos, images or text), metadata and HTML5 players into NAS suitable for streaming using the Dalet Flex platform. The service is also used to publish collection metadata.
- Publish Service: This service publishes assets (videos, images or text), metadata, and html5 players into NAS suitable for streaming using the Dalet Flex platform.
- Registry Service: This service stores action definitions and resource definitions in MongoDB. Whenever a service starts, a list of action and resource definitions is sent to the Registry Service. If a new definition arrives, or any existing definition has been updated, a list of definitions is sent to a queue. Enterprise listens to that queue and it updates its own list of plugins. This means that in the Enterprise UI, users have the option to create actions / resources that are stored in the Registry Service. If the `dynamicPlugin` feature toggle is set to “ON” Enterprise requests the Registry Service to obtain the list of available actions / resources.
- SearchElastic Service: The SearchElastic service provides full-text search for Dalet Flex objects. The objects should be indexed through IndexElastic before a search is run.
- Smoke Resource Service: This service forms part of the Job Execution Framework (JEF). The service consists of a simple smoke action and resource used for acceptance testing when integrating multiple services.
- Stream Processor Service: The Stream Processor Service integrates FFMPEGs for transcode, frame accurate clipping, technical metadata extraction, and MP4 file stitching.
- Subtitle Service: This is an external resource framework service responsible for managing the extraction of subtitles from media asset files, and transcoding subtitle files from one format to another. This service delegates the work to the Subtitle library in order to perform the actual extraction.
- Tag Service: This is a small node.js service that manages the adding and deleting of tags for assets. It saves and retrieves the tags from MongoDB and publishes the addition and deletion of tags to RabbitMQ.
- Thesaurus Service: This service enables support for theasauri in metadata definitions.
- Taxonomy Service: The taxonomy service is a node.js service used to manage the taxonomy backing store. The service currently provides the ability to create and manage taxonomies and taxons in Dalet Flex.
- TranscodeResource Service: The Transcode Resource Service is a Job Execution Framework (JEF) service responsible for managing transcodes with third party and Dalet Transcode media assets. This service delegates the work to the Transcode library in order to perform the actual communication with different transcode engines.
- Video Indexer Service: The Video Indexer Service (Media Analyse action plugin) integrates with Microsoft's Azure Video Indexer,. The Media Analyse action plugin enables a user to upload an asset to Microsoft's Video Indexer, request a full analysis, and extract information. The information that is extracted is then applied to the asset.
- Replication Gateway Service: Replicates objects from one site to another site.
- Write Object: Writes objects in the MySQL database for Dalet Flex Enterprise.
- Secrets: The Secrets service protects secrets stored in VFS locations and allows other internal services to access this information.
Dalet Flex has been designed from the ground up to be configurable and support as many vendor solutions and standards as possible. This versatility is brought about by ensuring all elements of Dalet Flex are pluggable and extensible. Dalet Flex's Virtual Filesystem component is no exception. The Dalet Flex Virtual Filesystem has been designed to support a wide range of storage and file transfer protocols so that Dalet Flex can access files on a wide range of different storage devices. This abstraction layer means that file-based actions can seamlessly move files between different storage devices in different locations, whether that be local file-based storage, or cloud-based object storage.
Dalet Flex Filesystem has a built in abstraction layer which enables it to use a range of different storage and file transfer protocols. Examples include:
- AZBS: For accessing files stored using Microsoft's Azure Blob Storage.
- File: For accessing files on local file mounts.
- FTP: File Transfer Protocol.
- FTPS: (also known as FTP-ES, FTP-SSL and FTP Secure) is an extension to the commonly used File Transfer Protocol (FTP).
- GCS: For accessing files stored using Google Cloud Storage.
- HTTP: Hypertext Transfer Protocol Overview.
- HTTPS: Hypertext Transfer Protocol Secure.
- SFTP: SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP).
- SMV: SMB/CIFS networking protocol.
- S3: For accessing files in an Amazon S3 account.
- WebDAV: Web Distributed Authoring and Versioning (WebDAV) is an extension of the Hypertext Transfer Protocol (HTTP).