Dalet Flex Operational Requirements for Customer-Hosted Deployments

(Revision 1.5.0, December 2023)

Product specifications and availability are subject to change without notice. Dalet is a registered trademark owned by Dalet. All other brands and trademarks are those of their respective owners.

Disclaimer

Dalet Flex service can be offered in two separate ways:

• As a pure SaaS service, fully hosted and operated by Dalet. This is the recommended approach, maximizing the Service Level Agreements (SLA) and level of service delivery quality Dalet can provide.
• As a customer-hosted service, while still operated by Dalet. Operating on a shared-responsibility basis, this approach is known to be less efficient than its pure SaaS service counterpart. When selected, the customer has two distinct hosting options:
  • On Cloud, on a supported dedicated Cloud provider account of his choice.
  • On-Premises

This document provides the Flex traditional operational requirements when the customer-hosted approach is considered by the customer. The exact requirements may vary depending on the nature of the infrastructure and customer’s private security policy and Dalet Operations team is able to accommodate to some specific variations.

Please note that any deviance from the usually offered and supported requirements is subject to either increase the cost of the delivered solution as subject to customization and/or lower down the global quality of the service, SLA and capacity to provide accurate and timely customer support.

Operational Responsibilities

Customer-dedicated deployments require a shared-responsibility operational approach with clearly defined and respected boundaries between customer and Dalet IT organizations. Dalet services can be contractually offered from 3 distinct ways:

• The default Supported way, where:
  • Customer is in charge of providing (and ensuring reliability, durability and scaling) of the underlying infrastructure, including but not limited to networking, computing and storage resources.
  • Dalet is in charge of supporting the deployed service itself, including its maintenance and upgrade processes. Dalet’s responsibility is limited to a per customer request, support delivery activity, offering software maintenance and software updates delivery.
  • Customer oversees the day-to-day monitoring and right-behavior of the service and reach out to Dalet’s customer service in case of incapacity from the service to operate properly.
• The extended Managed Services way, where 24x7 monitoring, alarming, and troubleshooting of the Dalet's service is delegated by customer to Dalet’s Cloud Operations teams, to the extent that:
  • The underlying infrastructure remains the customer’s responsibility and Dalet will not intervene to mitigate out-of-control issues.
  • The proper remote administration capabilities must be provided by the customer to Dalet for the service to be properly operated.
• The Private SaaS way, where:
  • Dalet provides a complete dedicated Cloud hosting service on behalf of the customer, ensuring a proper redundancy, high-availability and scalability of the solution.
  • Provides offloads the customer from all operational troubles, providing the same 24x7 coverage than offered by the Managed Services subscription
  • Provides an additional level of SLAs, platform's scrutiny and security awareness.

The matrix below summarizes the split of roles and responsibilities between Dalet and its Customer:

Flex Typical Deployment Overview

Network Topology

When Flex is deployed on a customer-hosted infrastructure, the core Flex services are deployed within this infrastructure and end-users will connect to the associated Flex services. The customer is in charge of providing the networking access capability, allowing targeted end-users to reach out the service. It is up to the customer to define the scope of access, for instance, should the service be exposed on public Internet or restricted to a private network.

Dalet Cloud Operations teams usually set up a connection bridge or network between Dalet’s centralized operational network and customer’s Flex infrastructure for remote administration purpose.

The following diagram provides an example overview of such topology based on customer-hosted AWS target. Alternative methods apply for other supported providers.

Infrastructure Topology

Dalet Flex key infrastructure components are usually deployed the following way:

• 1 region with 3 Availability Zones (AZs) or assimilated, across a shared L2/L3 network.
• Each AZ features 2 private subnets:
  • One for classic computing services (VMs and database instances). Each VM instance relies on either LVM Cloud block storage for disk storage.
  • Optionally one for Kubernetes-based FSP services, ideally configured with auto-scaling support.
• Flex services (Master, App, Services, Job nodes) are spread evenly amongst AZs for redundancy. There’s a minimum of 1 VM of each type on each AZ (a minimum of 3 AZs in total).
• One monitor instance is deployed, without any form of HA. It allows for local monitoring and relay to Dalet’s central monitoring platform (“Control Tower”)
• Media files are stored in region based S3 buckets. No cross-region replication is configured. S3 is configured with AES-256 server-side encryption.

The following diagram provides an example overview of such topology based on customer-hosted AWS target. Alternative methods apply for other supported providers.

When hosted on customer premises instead of Cloud, it is up to the customer to provide the adequate infrastructure means to ensure an equivalent level of availability, i.e.

• Multiple data-center locations or rooms to offer AZ approach
• Redundant power supply and networking equipments
• Resilient hypervisors, offering seamless on-the-fly VM migration
• Resilient data storage, including VM’s block storage.

IMPORTANT: The non-compliance to these requirements will potentially lower-down Flex’s expected level of resilience, availability and will be out of Dalet’s control and support.

Infrastructure Requirements

Supported Hosting Provider

Dalet Flex can be deployed on multiple infrastructure providers:

• Cloud (AWS, GCP, Azure, Alibaba)
• On-Premises (Metal or Virtualized)

Cloud Provider Requirements

When Dalet Flex is deployed on a customer-hosted Cloud environment, the following requirements are expected:

• A dedicated account/sub-account for Flex operations to isolate Flex from other customer Cloud services and limit the boundaries and segmentation of rights.
• Full administrative privileges on the account, as for Dalet to be able to create/modify/destroy resources and fully operate and maintain those. Incapacity to do so would prevent the Dalet cloud operations team (“Dalet CS Ops”) team to use Infrastructure-as-Code automation to maintain the environment and lead the infrastructure maintenance’s responsibility to customer.
• When applicable, full Kubernetes-as-a-Service administration rights.

In that regard, Dalet recommends the following:

• On AWS: a provider account delegation (to allow Dalet CS Ops team members to authenticate through Dalet’s corporate account, following the current security standards, and assume role on behalf of customer) is recommended.
• On other Cloud providers (or if AWS account delegation is not considered): Customer shall provide nominative Cloud user accounts for each Dalet CS Ops team member, being notified by Dalet on any changes on human resources level. Incapacity to do that will result in a degraded quality of customer support and may prevent Dalet from performing as per the SLA.

On-Premise Bare-Metal Requirements

When Dalet Flex is to be deployed in bare-metal servers, the following requirements apply:

• Customer is responsible for providing all peripheral equipment such as private network link, public Internet access (egress and ingress, when required) and physical hardware.
• Customer is responsible for providing all high redundancy equipment including - but not limited to - UPS, dual power supply, dual networking attachment. Dalet will also define the requirements in terms of CPU, storage, and memory according to the customer usage.
• Hardware remains customer’s property and any physical intervention requiring maintenance, upgrade or change of hardware is under customer’s responsibility. Customer is responsible of managing spare parts and provide an acceptable ratio of hardware components as to cope with physical failures. Dalet will not be responsible of any downtime due to a physical hardware failure.
• Storage provided by customer shall be redundant and high performance. Dalet strongly recommends usage of SSD disks over HDD whenever possible. When database usage is considered, Dalet strongly recommends usage of NVME SSD storage class over SATA SSD. Media storage disk shall be aligned with the expected usage of the system related to the number of jobs that are expected to be run simultaneously. Dalet always recommends using high performance storage. Dalet will not be responsible for issues related to a lack of performance of the infrastructure.

On-Premise Requirements (Flex Satellites)

When one or multiple Flex satellites are expected to be deployed, the following requirements apply:

• Each satellite must be composed of 3 hosts at a minimum, located on the same layer 2 network.  Hosts can be either physical or virtual machines.
• A satellite host shall be provisioned with 4 CPU, 8GB RAM and 40GB Hard drive (SSD) at a minimum.

On-Premise VMware Requirements

When Dalet Flex is to be deployed on VMware cluster, the following requirements apply:

• The virtualization underlying hardware hypervisors are to be managed by the customer. Dalet will not be responsible for any hardware computing/networking/storage changes or issues which may prevent the Flex system from working.
• The customer is solely responsible for designing and maintaining a production-grade virtualization cluster. While Dalet can make design recommendation, it is up to the customer to ensure that the setup has been done as to ensure full resilience and high-availability of the system, including network and storage resources, as to ensure hot-migration of workloads, survivability of ESX host in case of failure or maintenance, capacity to absorb the load and traffic, resilience of datastores in case of disk loss or deficiency.
• Should Dalet be responsible for creating the VM’s on behalf of the customer, the customer must provide Dalet Ops team with vSphere credentials and access rights to perform Terraform deployment.
• Dalet Flex makes use of Virtual IPs (VIP) to manage software high-availability. This might come in handy when using VMWare and requires proper configuration tuning to be allowed. By default, VMware’s dvSwitch network configuration only allows VMware issued MAC addresses to be routed and consequently, the VRPP promiscuous feature must be enabled as well as authorizing non-VMware based MAC addresses.
• If Dalet Flex system is meant to be exposed over public Internet, the VMware network configuration must allow access to public network and a virtual network interface allowing so must be assigned to VM requiring so (typically Dalet Flex’s load balancers).

Network Requirements

From service usage perspective

Several network-oriented considerations must be circled out to allow end-users to connect to the Dalet Flex service.

Domain Name

The customer shall provide a domain name for Dalet Flex to be hosted upon. The chosen domain can be either a public or private domain depending on the targeted usage (service exposed over Internet or restricted to private usage).

The domain name configuration can either be managed by the customer or by Dalet CS Ops team (recommended). The selected domain name is to be provided by the customer. If the customer prefers Dalet to provide a dedicated domain, Dalet offers free usage of the ooflex.net generic domain. The customer’s Flex system will then be made available under *.customer-env.ooflex.net domain.

Dalet strongly recommends to setup a dedicated DNS sub-zone for Flex (one sub-zone per environment, e.g: production, staging, QA ...). This process can ease the DNS creation automation.

IMPORTANT: The DNS configuration is usually managed by Dalet through automation process. Dalet currently supports AWS Route53 and GCP Cloud DNS providers only. If the customer’s selected DNS provider is different, customer will either (1) manage the Flex DNS configuration by himself or (2) require an additional customization effort from Dalet teams at an additional charge.

DNS Entries

Should the customer not being able to provide a DNS sub-zone for Dalet to manage it, the following DNS A entries will need to be added (more might be needed, depending on the customer requirements):

• grafana.<domain_name>
• kibana.<domain_name>
• prometheus.<domain_name>
• xymon.<domain_name>
• alertmanager.<domain_name>
• <flexAccountName>.<domain_name>
• master.<domain_name>
• publish.<domain_name>
• <database engine>.<domain_name>
• consul.<domain_name> (round robin entry)

The underlying IP addresses mapping is to be disclosed after initial deployment. Multi-A entries need to be supported by the DNS provider to offer high-availability mechanisms.

SSL Certificate

Dalet strongly recommends the use of wildcard SSL certificates (e.g. *.flex.customer.com, where * means an infinite number of sub-domains being supported) as the Flex software relies on the host part of the address to work. When managed by Dalet, this is fully automated by Dalet deployment scripts.

The SSL certificate can be obtained in different ways:

• Customer requires Flex to use the customer-provided SSL certificate
  • Dalet then suggests that the provided SSL certificate is at least RSA-2048 or EC-256 encrypted, with a SHA-256 digital signature and a 1y minimal validity. These encryption requirements are considered to be safe enough to nowadays standards while keeping a wide compatibility with modern client OSes and browsers.
  • If wildcard SSL certificate is not an option (due to customer’s internal security policy), a SAN (Subject Alternative Names) certificate must be provided, listing all possible subdomains to be ever used by Flex. Approximately 20 SAN entries must be provided when such a use case as to be implemented.
  • The customer is responsible for the proper management (including expiry) of the certificate provided. Dalet requires the customer to provide the renewed certificate within 30 days prior to expiry to ensure a continuity of service.
  • The customer agrees to provide Dalet with both the public SSL certificate, the associated private key and the intermediate CA certificates to ensure a valid chained certificate.
  • Dalet strongly discourages the customer-provided SSL certificate approach, as it may lead to more complexity in SSL management and may induce possible side-effects and unavailability of the service.
• Customer agrees to use a Dalet-provided SSL certificate (recommended)
  • Dalet will then be responsible for global end-to-end security provided by this SSL certificate.
  • On AWS or GCP hosted deployments, the Cloud provider’s managed SSL certificate service will be used by default (with auto-renewal support).
  • On Alibaba, Azure or on-prem deployments, a Let’s Encrypt issued certificate will be used instead (with auto-renewal support).

Load Balancer

Dalet Flex requires a (couple of) load balancer(s) to provide users access to the actual backend services.

The load balancers are responsible of:

• Exposing Flex backend services to users.
• Offering a highly available entry point to Dalet Flex.
• Offloading (or terminating) the incoming SSL-secured HTTPS traffic.

Load balancers are either:

• Used “as-a-service” when available on major Cloud providers (e.g. AWS ALB)
• Deployed on specific virtual machines otherwise, and powered by Open source software like HAProxy.

Load balancers can be, depending on customer requirements, configured to be:

• Publicly available, exposing Flex services over public Internet
  • They consequently require to be exposed through one or many public IP addresses and secured (firewall) accordingly.

  • The customer is responsible for providing the necessary IP addresses and ensuring those are not blacklisted of any sort.

  • The customer is responsible for ensuring that the load-balancer has the necessary ingress/egress bandwidth from/to public Internet to cope with the expected usage.
  • Any unavailability of the Flex services related to a deficiency in Internet connectivity will remain under the customer’s responsibility.
• Private-only, limiting Flex services exposure to private users.
  • In such a scenario, the customer remains responsible to ensure access from its internal network to the Flex application.

The load balancers must expose the following ports:

• TCP/80 (HTTP)
• TCP/443 (HTTPS)

Mail Server

Dalet Flex requires a mail server to manage the end-users day-to-day operations (e.g. account creation, password reset ...) or any other workflow-related operation requiring an email service.

Customer is expected to provide Dalet with a properly configured SMTP server. It is up to the customer to manage and secure the SMTP service and provide Dalet CS Ops team with the necessary network access rights and associated credentials for Flex to properly send emails. The customer is solely responsible for the well tuning of the associated SMTP server when it comes to service exposure and validity related to protocols like SPF, DKIM, DMARC or BIMI, usually required to ensure a proper delivery of emails (ensuring that sent emails are not blacklisted by recipients and/or considered as spam).

When customer is not able or willing to provide Dalet Ops team with an SMTP server, Dalet can offer the use of Dalet’s private server, hosted on AWS (hence requiring connectivity access). Bear in mind that emails being sent by the Flex system will be received by end-users from a Dalet labelled source and it remains customer’s IT responsibility to set up all the necessary configuration to accept and trust what the Dalet mailer may send.

For service administration perspective

Ingress

Dalet CS Ops team requires an SSH connectivity to the various Flex servers for installation, upgrade, configuration or debugging of the environment. This SSH connectivity is mandatory for Dalet to perform the service operation. SSH access can however (more than recommended) be limited to private access through the usage of peering or VPN-like connections and doesn’t require servers to be exposed over Internet.

When per-server SSH access is not possible, Dalet CS Ops team may use a Dalet-provided or customer-provided bastion or jump host as a possible SSH-relay alternative. This jump host is expected to be Linux/UNIX-based.

Starting January 2024, Dalet leverages the usage of a Teleport-based dedicated bastion for remote infrastructure access management. Recommended as default remote access means, it enforces network security and audit capabilities, being an alternative to VPN-like solutions, keeping doors closed. Please refer to Remote Access dedicated section below for extended details.

IMPORTANT: The SSH connectivity is the only way for Dalet CS Ops and Support teams to provide efficient deployment, management, troubleshooting, and mitigation of possible incidents. If customer is not able to or doesn’t allow such requirement, the expected level of serviceability and quality of customer service will be greatly weakened and Dalet may not perform as per the SLA, falling to a degraded best-effort approach.

IMPORTANT: “Remote display” mechanisms through browser-based or dedicated applications and protocols like RDP, TeamViewer or similar are not supported options nor an SSH replacement. While such alternatives are somehow possible, the efficiency of support will fall in the same “best-effort” approach as previously mentioned.

Egress

Dalet Flex deployments are expected to have outgoing Internet access. While a direct connection is encouraged, HTTP proxy are supported, both at the OS and application levels.

Internet outbound connectivity or access through a proxy server is required to fetch:

• The various OS packages and system/application binaries
• Docker images from either public registries or Dalet Flex’s private registry.
• Streaming out (push) logs and infrastructure metrics to Dalet’s Control Tower for remote platform monitoring and administration.

If a middleware proxy or network filter equipment is involved within customer’s network, access to the following network remote resources must be opened up over:

• HTTP(S) protocols (TCP/80 + TCP/443)
  • OS/Software public mirrors (unless using a private mirror):
    • Ubuntu Linux repository
    • RedHat Linux repository
    • Grafana.io, grafana.com, packages.grafana.com, apt.grafana.com
    • *.mongodb.org
    • Consul.io
    • packagecloud.io
    • download.arangodb.com
    • *.github.com
    • Rubygems.org
    • artifacts.elastic.co
    • Dalet Git repositories : git.corp.ooyala.com, bitbucket.org
  • Docker Registries:
    • Docker Hub (*.docker.com, *.docker.io)
    • Dalet Docker registry (registry.ooflex.net,registry.services.ooflex.net)
    • Quay.io
    • Us.gcr.io
  • Third-Party Services
    • *.pagerduty.com
  • Remote Monitoring and Administration:
    • metrics.cs.dalet.cloud
    • logs.cs.dalet.cloud
  • Dalet APT repository
    • packages.cs.dalet.cloud
• SSH protocol (TCP/22)
  • Dalet Git repositories : git.corp.ooyala.com, bitbucket.org
• NTP protocol (UDP/123)
  • NTP servers: *ubuntu.pool.ntp.org (if not provided by the customer)

IMPORTANT: This list is subject to change and can’t be exhaustive as some public placeholders are out of Dalet’s control. Modification of links and redirections can happen at any time.

Subnets

When deployed on Cloud environments, Dalet usually uses 3 Availability Zones, each coming up with a dedicated subnet. Depending on customer requirements, more subnets can be used, if a segregation of duty is required.

Dalet strongly recommends that the minimum subnet size equals to a /26 range (i.e. up to 64 hosts). If the customer’s deployment involves a Kubernetes instance for Dalet Flex’s transcoding service, a dedicated /16 subnet must be allocated (because of the Kubernetes’ pods large IP ranges).

IMPORTANT: It is customer’s responsibility to ensure the proper behavior, communication and bi-directional routing of the various subnets. Any internal networking equipment (e.g. router, firewall, WAF, IDS/IPS ...) misconfiguration will be under customer’s responsibility. Dalet is not to be responsible of the underlying networking stack.

Remote Flex Server Administration / Support

Dalet requires a network connection to customer’s infrastructure to ensure proper deployment, upgrades and maintenance of the customer’s Flex service instance and the various servers the solution consist of.

Dalet’s Customer Remote Administration process relies on the following:

• Dalet CS Ops team members authenticate through a corporate LDAP-based MFA-enforced portal to ensure of their identity and authorization / ability to manage customer platforms.
• Once authenticated, they establish a secure one-way VPN tunnel to Dalet’s Cloud Operations administration network and platform.
• Dalet’s Cloud Operations network is connected to customer’s infrastructure and Dalet Ops members can establish an SSH remote connection from there on.

From Dalet’s Cloud Operations network, multiple ways of connecting with customer’s infrastructure are considered and supported, in order of preference (and operational costs)

• Dalet Teleport Bastion
  • Provides native SSH access to backend servers without need for public exposure nor complex VPN gateway.
  • Pros: seamless access to underlying servers for remote administration through dedicated bastion-host over secure client-established HTTPS reverse tunnel. Bastion does not require any public exposure and global remote access can be turned off anytime.
  • Cons: require an extra lightweight VM next to customer's platform and Internet egress connectivity.
  • This is Dalet's recommended approach.
• AWS VPC Peering
  • This allows for connecting Dalet and Customer’s Flex VPCs together
  • Pros: fast and easy access to customer’s environment when it comes to troubleshooting, very easy to setup
  • Cons: requires customer’s platform to be hosted on AWS.
• Site-to-Site IPSec VPN Tunnel
  • This allows connecting Dalet’s Cloud Administration network with virtually any customer’s platform through public Internet over a secured VPN.
  • Pros: fast and easy access to customer’s environments when it comes to troubleshooting, allows for the customer to control / restrict / limit the subnets to be accessed to.
  • Cons: a bit more complicated to setup than AWS VPC peering.
• Customer-provided Bastion Host
  • Customer provides a Linux jumphost with an SSH access that Dalet’s Ops team members can use as relay to SSH to Flex’s servers.
  • Pros: SSH connectivity is still manageable.
  • Cons: Customer must provide bastion access to all Dalet CS Ops team members and manage individual contributors’ security. Incapacity from customer to do so will result in a degraded operational quality. Connections will be established from Dalet’s CS Ops team members instead of centralized trusted Cloud Administration network platform which may increase security concerns.
  • Dalet will not support Windows-based bastion host or host with no outgoing Internet connection.
• Customer-provided VPN access

• • Customer provides a VPN access to access to Flex’s infrastructure network.
  • Pros: SSH connectivity is still manageable.
  • Cons: Customer must provide nominative VPN access to all Dalet CS Ops team members and manage individual contributors’ security. Incapacity from customer to do so will result in a degraded operational quality. Connections will be established from Dalet’s CS Ops team members instead of centralized trusted Cloud Administration network platform which may increase security concerns. Customer must authorize multiple concurrent VPN connections as to allow Dalet Ops team members to perform simultaneous troubleshooting.

IMPORTANT: Customer-provided Bastion host and dedicated VPN access, while supported, are not recommended. These approaches usually imply heavy changes in terms of access rights, human resources management, and greatly impact Dalet’s responsiveness to incident and capacity to support. If these approaches are considered, Dalet may not perform as per the SLA, and support will fall back to a ‘best-effort’ approach.

Dalet Teleport Bastion

Starting January 2024, Dalet leverages usage of a Teleport-based bastion to provide seamless yet secure remote access capability. The bastion provides a customer-issued HTTPS-based reverse tunnel, allowing for remote SSH access to platform's underlying servers without having to setup complex VPN solutions, opening up network configuration and exposing services over public Internet.

Dalet Teleport Bastion will be locally deployed on customer platform (a lightweight VM with 2 vCPU and 1 GB RAM is fairly enough) and small Teleport agents will be installed on each instance to be managed (as part of Dalet's service delivery), connected to local bastion proxy. The bastion issues a long-standing HTTPS connection to Dalet's Cloud-based central Teleport proxy, which can be re-used further on by approved Dalet Ops to jump into SSH servers through the previously established reverse tunnel.

Note that the connection is issued at client-side, hence only requires Internet outbound access. The service remains private and is never going to be exposed over public Internet and does not require any IT port opening which could compromise network's security like VPNs could do. Preventing remote access (or, by opposition, authorizing on-demand ephemeral access) is only a matter of connecting the client (i.e. have it running with no blocking egress firewall rule).

Dalet Teleport Bastion will trust the global Dalet Teleport proxy (to which Dalet Ops connect to) by ensuring that only the relevant remote connections are accepted:

• Dalet Ops personnel validated against Dalet's AD groups.
• Authenticated & authorized through MFA against Dalet's Okta-based identity provider.
• With enforced biometrics authentication.
• With enforced trusted device's authentication.

Dalet Trust Bastion local proxy and agents bring additional security perspective:

• Audit logs of SSH connection activity. Customer has full visibility of all activity records.
• SSH session recording. Any SSH connection activity is live streamed into a video recording. Any trace of activity is captured.
• Customer's IT admin got access to local Dalet's Teleport Bastion and check for audit and recordings at any time.
• Customer keeps control, being able to turn remote access on or off at any time.

The solution comes free of charge.

Operating Systems Requirements

Supported Operating Systems

Dalet Flex runs exclusively on GNU/Linux systems. The various servers are usually deployed on Ubuntu LTS (Long Term Support) distribution.

Supported Releases to date are:

• Ubuntu 20.04 LTS (focal)
• Ubuntu 22.04 LTS (jammy)

RedHat Linux is an alternative OS candidate, less maintained and validated and possibly comes with a lower quality of support.

IMPORTANT: Other Linux OS variants are currently not supported and included in our proposals. Should the customer request an unsupported Linux OS, Dalet can assess on a case by case the feasibility with the related additional costs.

IMPORTANT: Non Linux-based OS are not supported and will never be.

OS Installation

Dalet requires the Flex servers to be installed with a minimal vanilla OS installation. The base OS install will be extended with several additional packages and configuration tuning, which come as part of the standard Flex automated deployment scripts.

Dalet Flex deployments supports multiple host OS deployment procedures:

• A customer-provided vanilla OS installation for on-prem metal deployments
• A customer-provided virtual machine disk OS template for virtualized deployments (e.g. VMware OVF, AWS AMI ...)
• A Dalet-provided virtual machine disk OS template.

IMPORTANT: Note that any customer-provided non-vanilla image, including specific extra packages and configuration settings, may possibly prevent Dalet Flex’s correct behavior. If so happens, it comes with customer’s responsibility to either (1) update his OS image to restore Flex’s nominal capabilities or (2) agree to get charged by Dalet with extra customization fees to adapt to customer’s specificities.

OS Configuration & Requirements

Administration Rights

In order to deploy/upgrade/manage the Dalet Flex solution, Dalet Operations team requires a root level access privilege to the various servers. Each Dalet CS Ops member accesses the servers with a nominative UNIX account over SSH and uses the sudo escalation privilege to gain root access whenever needed.

A no-password access policy is put in place, where only SSH private/public key exchange authentication is enforced.

The root privilege is required:

• to install/upgrade the Flex solution. Deployment scripts are requested root access to perform OS level configuration (e.g. create mount point, sysctl configuration, ...)
• to access specific logs, restart services or restore behavior once troubleshooting is required.
• as the Dalet Flex solution itself is based on containers isolated using cgroups, some running with root privilege.

IMPORTANT: Unavailability for the customer to provide or authorize root access to Dalet CS Ops members would prevent from operating the solution in a standard way. Any alternative method in place would require heavy customization on Flex and be evaluated for an additional fee.

Time Server

Dalet Flex requires time-synchronized servers for proper operation of cluster-based components. Access to an NTP server is mandatory for that to happen. NTP server must be provided by customer. If no local NTP server is accessible or available, Flex system will require Internet outbound access to public NTP servers (e.g. *.ubuntu.pool.ntp.org)

Network Configuration

Customer is required to provide the Dalet Ops team with the specific network configuration that is required from OS perspective (e.g. flat network, V(X)LAN support, NIC teaming/bonding, DHCP provided configuration or static configuration and so on). Customer is responsible for the well- behavior of his private network, regardless of Dalet Flex’s deployment. The OS network configuration settings will be setup by whoever provides the base OS installation (Dalet or Customer).

Unless explicitly requested (e.g. Internet-facing public load balancers), Flex servers usually only require a private network connection.

Miscellaneous Requirements

Additional requirements are expected for a proper behavior of Flex operations:

• Ensure that at least one server from the solution can be remotely accessed (outside of customer’s private network) for administration purpose.
• Dalet CS Ops team must be able / allowed to manage (start / stop / restart) both Systemd- managed and Docker-managed services.

• The Docker daemon version, package and configuration must be maintained by Dalet CS Ops team.
• The OS base image must be available either from either public Internet or customer-provided mirror.
• The OS packages repository must be available from either public Internet or customer-provided mirror.

Security

Security basics

Dalet takes security as a critical matter and always takes measures to ensure that all packages installed on customer’s private servers are known to be safe:

• OS packages always come from known trusted repositories
• Only GPG-signed packages are deployed on system
• Flex release containers have been scanned prior deployment during Dalet CI/CD build stages.
• Public CVEs are evaluated when disclosed and those potentially impacting Flex systems are fixed in a timely manner in accordance with customer.

Dalet can adapt to customer’s OS upgrade policy (OS level security patch). This can require extra activity/check from the Dalet CS Ops team. This support requires the customer to have subscribed to the REM / Monitored support option.

Dalet access to customer’s servers is only done through SSH using personal public/private keys. Access to the platform requires either the Dalet provided VPN solution (based on MFA authentication with access token challenged every 8 hours) or the customer provided VPN solution.

Dalet Cloud-Hosted solution

Should the customer require so, Dalet can provide real-time supervision of the Flex service on behalf of the customer. This is a recommended approach to ensure all systems remains functional as to guarantee the best availability of the platform with the highest level of responsiveness.

Dalet uses the “Need to know” security paradigm to ensure only people at Dalet needing to access customer’s environment or resources are allowed to do so.

System Log & Metrics retention

By default, all system metrics and system logs are kept on a dedicated server, next to where Flex is installed, for a duration of 30 days on a regular instance.

When subscription to Managed Services or Private SaaS option is engaged, customer can leverage the Dalet Control Tower cloud-hosted monitoring-as-a-service feature to benefit a 90 days retention.

Customer can then either opt for:

• Exclusive Dalet Control Tower (SaaS Monitoring) - 90 days retention, remote, cloud-based.
• Exclusive Flex instance integrated log engine – 30 days retention, local to Flex deployment.
• Both of them

Dalet Control Tower (SaaS Monitoring)

Dalet offers the opportunity to ship customer’s logs and metrics to a centralized Dalet provided SaaS service: “Dalet Control Tower”.

This solution stores the service metrics and the application logs into a cloud-based solution operated by Dalet in a secured environment, only accessed by Dalet support teams and requesting MFA to connect to.

This solution allows the Dalet technical support and operations teams to react quicker and start analysis in the shortest time, hence maximizing the level of reactivity and quality we can offer.

Dalet Control Tower comes in handy for customers willing to let Dalet operates, diagnose and troubleshoot systems, cutting down the needs for local monitoring stacks. Usage of Control Tower can be enabled on top of existing local monitoring stack.

The Flex logs and metrics are shipped through HTTPS to Dalet Control Tower, hosted by Dalet. The Control Tower service will not require any kind of remote access to customer’s platform.

Requirements:

• Egress Internet connectivity from the local Flex instance to Dalet Control Tower public IP address range, using HTTPS (TCP/443)
  • Dalet Control Tower IP Range: 178.33.203.227/27
    
    
• Installation of Promtail (log shipper) on the local Dalet Flex instance hosts which will need to parse the local host Flex and system logs. The Promtail installation is part of the Flex deployment (when enabled) and doesn’t require any manual intervention from customer’s side.

Dalet Control tower has a retention period of logs and metrics of 90 days. The magnitude of information send is around 100 GB per month.

Data Privacy

Dalet Control Tower is meant to store and process information exclusively related to monitoring, troubleshooting and support. The data are streamed/pushed by the system to the Control Tower in a secure way over HTTPS with specific credentials and a platform unique authentication ID.

Data being streamed are:

• Anonymized application logs, containing IDs and troubleshooting information but no user private information
• Infrastructure and application metrics (counters of a given stat or metric).

Backups

Dalet Flex solution integrates backup mechanisms to ensure the service can be restored on a system or upgrade failure. The associated backup files are stored on a separated file system (which can be customer provided for on-premises deployment or a remote Cloud storage).

Scope of Work

The following data are part of the backup process:

• All the databases included in Dalet Flex

The following data are NOT part of the backup process:

• The media files. Whenever a backup of this file is required, it can be handled either:
  • By the customer (on premise installation or cloud instances not hosted by Dalet)
  • By Dalet teams through a specific offer
• On cloud providers file system are based on S3 technology with duplication of files (usually 3 copies to different buildings or AZs, can be more) which makes the loss of file very unlikely.
• Backup retention capability and resilience.

Backup Retention

By default, databases backups are done every day and kept for seven (7) days. Customer can however require changing this behavior to more retention days.

IMPORTANT: It is up to the customer to ensure that the backup storage used as a recipient is secure, resilient, and scalable enough to hold the associated data. The backups and their integrity are under the customer’s responsibility.