Dalet Flex 2024.7.0 Release Notes – Support Hub

September 15, 2024 06:15
Updated

Dalet is excited to present the August releases of Dalet Flex 2024.7.0.

This release of Dalet Flex provides the new features, quality and security enhancements, bug fixes, and other changes listed below.

Important Information regarding MariaDB and Dalet Flex 2024.7.0

Flex 2024.7.0 is NOT compatible with MariaDB 10.4. You MUST upgrade MariaDB to 10.11 if upgrading Flex to 2024.7.0 or later versions.

Please plan for a full back-up restore operation when upgrading to 2024.7.0 and beyond.

Dalet Flex

New Features

Migration from Elasticsearch to OpenSearch (2024.7.0)

Flex 2024.7.0 is the first version of Flex to use OpenSearch instead of Elasticsearch. Dalet Flex does not support migrating data from Elasticsearch to OpenSearch and two upgrade strategies for moving to OpenSearch are available when upgrading to Flex 2024.7.0 or above. Additional information is available here or from your Dalet representative.

Add `external ID` as a unique identifier for SAML login (2024.7.0)

Dalet Flex supports managing external users via an external SAML IdP. From Flex 2024.7.0, a unique external identifier can now be added to the representation of Flex users managed by SAML. If the `externalUniqueUserId` attribute is sent by the IdP as part of the SAML claims, Flex will use this identifier to uniquely identify users in Flex. If the `externalUniqueUserId` is not sent, the previous behavior is retained, relying on one of the `login` or `email` attributes to act as a unique identifier for the user.

Where the `externalUniqueUserId` is provided:

It is stored in the Flex DB and displayed in the User View in Flex CORE
If any of the username, first name, last name or email address provided in SAML attributes do not match the values stored in Flex, those values will be updated in Flex at the time of user login.

Infrastructure for Flex-Vantage Telestream Lightspeed integration: Recording and previewing live (growing) content (2024.7.0)

Vantage Telestream Lightspeed servers are used to accelerate video processing. From 2024.7.0 Dalet Flex supports importing a growing OP1a MXF + H.264 DASH from Vantage Telestream Lightspeed and previewing it in FlexMAM.

To enable the e2e integration of recording live feeds and previewing the growing media in Dalet Flex, the dedicated "Open Stream – Import Workflow" must be deployed using the preconfigured open-stream-recording-cmbundle. This workflow processes recording files using a proprietary integration protocol. High res and proxy recordings can be recorded simultaneously. Discovery of growing files in the target hot folders triggers two workflows to import the MXF and the MPD files to Dalet Flex and uses filename based pairing to set the proxy asset as a child of the high res asset.

Import Action with Resource Item and Files in Subpath under Resource Storage Basepath as Managed Assets (2024.7.0)

From Flex 2024.7.0, Import Action performed on assets located in a subpath of the base Storage Resource are imported as Managed Assets as follows:

Resource Item with subpath is supported and generates a Managed Asset
Target Move and Copy with subpath are supported and generate a Managed Asset
"Source Storage Resource Subpath + Filename" Copy-Target Storage Resource Subpath” and “Move- Target Storage Resource Subpath” options in the Import Action configuration should be used.
These fields in the Import Action are deprecated: Source VFSLocation, Source Resource Item, Move > Target Storage Folder Resource, Move > Target VFS Location, Copy > Target Storage Folder Resource and Copy > Target VFS Location and are not recommended for use.

For more information please see this article or contact your Dalet Flex representative.

Mitigate Risk on Segmented Asset purge/move/copy/delete-with-move operations, e.g. MPEG-DASH not in individual and unique folders (2024.7.0)

When Flex Stream Processor creates a Flex Segmented Asset, it creates a unique folder that contains the manifest file and media files related to the Flex Segmented Asset. When performing copy/move/purge/delete with move the Flex Segmented Asset, Flex was assuming that all relevant objects are auto-contained in the “parent” folder, so it went to the parent container (mpd|m3u8) and performed the actions on this folder.

MPEG-DASH (.mpd) files created in Flex by a 3rd party, such as Brio, are not created in unique folders but are created in a flat folder that contains multiple manifest files and multiple media files. This means that when Flex tries to copy/move/purge/delete with move files, there is no unique parent file to perform the action on and unconnected Segmented Assets are also copied/moved/purged. This risks impacting other files in this folder not belonging to the Segmented Asset.

This release includes multiple enhancements and bug fixes that address functionality issues discovered with respect of Flex Segmented Assets. The improvements mean that copy/move/purge/delete with move actions support flat folder structure for Segmented Assets. The implemented checks and fixes ensure that in any case where the folder targeted by the action might contain files unrelated to the Segmented Asset, the operation fails. Instructions on how to identify Segmented Assets that might have an issue with Asset Import and Purge Action plugin scan be found here. Information on the checks and changes made to Delete/Delete with Move/Purge actions can be found here. For additional information please contact your Dalet Flex representative.

Integrate Flex scripts with an SCM system (2024.7.0)

Since Flex 2023.7.0 the recommended way to provide scripts to scriptable actions is to host them in Git. This allows scripts to be:

Version controlled
Tagged with well-known working configurations
Peer-reviewed via pull requests
Easily promoted between Flex environments

Flex scripts have been integrated via a Git repository. Detailed information can be found here.

Changelog

2024.7.0

Enhancements DaletFlex

Remove deleted assets from the denormalisedAssetsAndAnnotations index once the asset is destroyed.
Add field filter in JEF Copy/Move action for includesRegex/excludesRegex options. Running copy/move with includesRegex and excludesRegex, FlexFileProcessor considers the supplied expression.
JEF Execution and Plugin Grafana update & revision (I): Grafana dashboard alignment changes, Executor Thread Pool, Step Executor Thread Pool, Job Completion Rate and Job Completion Rate per Node.
Add event data for medadata definition modified event to indicate what changed, for example which fields are added, removed, reordered etc.
Added support for keyframes as PNG output format in Flex Enterprise and Flex MAM (in addition to JPG.)
JEF Copy/Move: Add checks to fail jobs for Segmented Assets if manifestFileName != mediaFolderName.
Warn and prevent saving a configuration with nested Storage Resource where the basepath of the SR is under an existing SR for all vfs-providers (S3, file, azure). Introduce a KV flex-enterprise.featureToggles.allowResourceWithSameBasePath that defaults to false. When KV is false and `isSameHostNameResource && isSameBasePathResource && isSameBucketResource` then fail saving the new Storage Resource with a validation error that declares the clashing resource name. `isSameBasePathResource` means one resource fullPath is the basePath of another resource. When KV is true and `isSameHostNameResource && isSameBasePathResource && isSameBucketResource` then save the new Storage Resource and publish an event with a message that warns about the clashing resource. See this article for information on a Stored Procedure that can be used to identify storage resources in an environment with the same base path or contact your DAlet Flex representative..
Add optional configuration for Delete and Purge actions for Segmented Assets (Packages?) to enable a check to limit maximum number of files to delete/purge. Action fails if the check finds a number of files larger than the configured maximum limit.
Add support for MessageTemplate and TimedAction services to JEF SDK.
Add the ability to return system events to the "get events by user" end-point.
Return workflow system events to the SDK method `flexSdkClient.workflowService.getHistory()`.
Add support of expression for the fields under Selection of frames section in the Extract Keyframe and extract-tiled-keyframe plugins.
FlexMAM: OpAtom asset should not be visible in FlexMAM – set `fmpVisible = false`.
Add user’s timezone information to user end-point of FO.
Java Enterprise Client Library: use non-FQDN from microservices when calling Enterprise.
The feature toggle "queueJobOnCreation" is enabled by default. Queue jobs on creation ONLY if quota is not applicable.
Support forcing check of source folders on object storage before performing folder based copy/move operations in FFP so that the job fails when the folder does not exist as opposed to succeeding but not having any effect so that the fact that the specified resource does not exist is essentially hidden.
Invert the workflow state check in job scheduler to continue queueing jobs if it is NOT in pending state.
SDK: use non-FQDN when calling Enterprise.
Prepare an SQL query to find potential assets which point to wrong location and might remove all files from storage resource when purged. Relevant for assets imported using "Asset Import" Plugin.
Flex Enterprise API should return 409 for `OptimisticLockException`.
Add support for a timecode overlay property to be wrapped at 24 hrs and begin recounting from 00:00:00:00. Set the Consul KV flex/flex-streamprocessor-service/enableDefaultTimecodeOverlayWrap24Hours to true and restart FlexStreamProcessor. Alternatively add wrap24Hours: true within the timecodeOverlay portion of the job request.
Updated legacy Flex Enterprise Copy and Move action to fail the job if `manifestFileName == mediaFolderName` with a recommendation to use JEF copy/move action plugins when media folder is at the same level as manifest file for segmented asset. Also, added new checks when the job runs i.e. "Multiple Manifest Check" and "Other Files in Same Package".
Change SDK locks from Redis to Hazelcast.
Expose /sessionAbsoluteExpiryTime endpoint in Global Header app to return the current absolute expiry of the existing session token.
Add index for metadataDefinitionId field in metadata collections in Arango db.
Asset indexing: Use distributed Hazelcast lock to prevent concurrent indexing of the same asset.
Remove customisability of Elastic/OpenSearch index prefixes. Consul KV flex/shared/elasticsearch/indexPrefix to override the default prefix for internal Flex search indices from flex to something else removed.
File Access Service - Create new service setup. Service will be used to manage keyframes better.
Update blue on black text in FlexTRACK app.
Support server to server copy for S3 files when there is a possibility of instance role
Add meta tag <meta name="robots" content="noindex, nofollow" /> to login HTML to prevent search engines from indexing the Flex login page.
Fix mediainfo stream closing issue.
Ensure Xtend Adobe app can run in a local development environment.

Fixed Issues DaletFlex

Fixed issues with flex-enterprise logging.
Fixed an issue where upload failed because when a date and time field in the metadata form used a Text Form type instead of a Time form.
Fixed an issue where filter collections by `clip_bn` type was not filtered correctly by flex-collection-service api.
Fixed an issue where flex-registry-service does not recover if the RabbitMQ cluster is restarted.
Fixed an issue in the Jobs repository with reporting of jobs COMPLETED, RUNNING on terminated executors. Add a KV job/forceFailAfter/milliseconds (default value is 172800000 , which is 2 days) , if there is no job progress for 2 days to force deleting the job.
Improve performance and messaging for API addKeyframe rest call. In case of ConflictException (keyframe already exist) during addKeyframe Enterprise rest call, add warning with proper message in job history and mark job complete. In case of OptimisticLockException/StaleStateException (Asset object updated simultaneously) during addKeyframe Enterprise rest call, job fails with proper message and retry solves the issue.
Jobs are auto-retried for OptimisticLockException
observed this scenario:Transcode failing with Enterprise returning an assetId and then rolling back (?), hence next attempt to GET this id returns 404. Solution: Verified retry functionality working as expected and added additional logging around placeholder asset updates.
OpAtom MXF: Fixed an issue where Purge and Delete with Move option fail. Updated resource validation check to consider filename when comparing asset location with storage resource root location
Fixed an issue where Extract Keyframe set the wrong master keyframe
Login App: set default login destination to /fmp/index when destination is either invalid or absent.
Improve the handling of Amberfin job progress.
flex-mediacore-objects-library: Fixed an issue where in case of subtract timecode having drop framerate validateDropFrames() gives an exception. Issue fixed by modifying the implementation to consider validateDropFrames after minutes and seconds are set.
Fixed an issue where Prometheus stats on FFP job nodes are not being observed. Ensure all Prometheus stats are captured on controller/standard nodes so they are not lost on job instance shutdown and are polled for Prometheus collation. There is now counter and gauge for completed and failed jobs.
Consider objectType in asset search api.
Fixed an issue where Prometheus stats on FSP job nodes are not being observed. Moved job stats reporting to controller nodes so that stats are not lost on job nodes. Aligned Prometheus stats to be the same for FFP. There is now counter and gauge for completed and failed jobs.
Job Execution Eventhandler should skip workspace check if Switch Account & Workspace configured and event handler is triggered.
Fixed an issue where a user with no "edit" permission on media asset can delete membership of the asset.
Fixed an issue where Fast Object throws NPE instead of returning 404 Not Found error when trying to get ACL for an object that does not exist.
If an asset is already associated with UDO, do not throw an error when trying to associate the asset again to the same UDO.
Fixed an issue where a user without edit/delete permissions on jobs could retry and cancel jobs from API.
Fixed an issue where Pack/unpack job does not work for folder.
Fixed an issue where Azure AD authentication session is not invalidated after logout.
Fixed an issue where a user can edit and delete saved search belong to other users. Issue resolved by adding a permission check for saved searches.
Hot folder service: Fixed an issue where errors when retrieving the outcome of async file processing leads to reimporting of all files in resource. Issue was reported for hotfolders configured with max depth ==0.
Fixed an issue where it was not possible to retrieve taxons with apostrophes using API and SDK.
Fixed a system error that was thrown when creating Quota config for with processes plugin type.
Fixed an issue where it is not possible to save metadata changes if the metadata schema contains non editable fields, even though these fields are not being edited.
Fix an issue where an NPE was thrown for multi-asset import action because of an issue with auto-wiring.
Job Scheduler updated to ignore maximum job limit keys with value zero in Redis.
Fixed an issue where FFP copy to LucidLink storage is slow.
Fixed an issue where there were distorted proxies for EXR frames packages. Issue fixed by adding unimplemented code to ffmpeg to handle larger VLC lookup code sizes.
Fixed an issue where audio waveform extraction fails.
Flex re-order audio track tool for cases when audio waveforms are not matching their audio channel.
Fixed an issue with slow pruning of standalone jobs. Issue solved by adding a new index and forcing the stored procedure used to prune standalone jobs to use the new index.
Fixed an issue where displayed taxonomies are not filtered correctly as per configured account visibility.
Fixed an issue where proxy could not be previewed in FlexMAM. Issue fixed in this case by setting KV flex/flex-videoproxy-service/sourceEnterpriseEnabled = true and restarting VPS instances.
Fixed an issue where the Start Session Action was not starting in FlexMAM.
Fixed an issue where user is not getting validation message when user directly clicks SEND button from REQUEST USERNAME and RESET PASSWORD screens without filling in any information in the email field.
Android: Fixed an issue where it was not possible to set a mandatory drill down option when the read only level is applied but there is no value in the field.
Fixed an issue where delete: (managed asset + segmented assets) when user is performing delete operation with Traverse To children = YES the proxy file is moved to the configured delete storage path even though the Delete Operation is failed. Validate the SR path if referred to by another asset before delete operation for segmented asset.
OPATOM Asset: Fixed an issue where an Export Action on an OpAtom asset which contained video and did not contain audio failed.
[Flex Cut] Fixed an issue where a Local VO Recording: conform job fails when rendering only a voice over track.
Fixed an issue where it was not possible to search a user field in a metadata form from FlexMAM.
Fixed an issue where it was not possible to retrieve hierarchy information for a UDO.
Fixed an issue where a System error was displayed when trying to when try to view System Summary of a different account. An appropriate error method has been added in Flex CORE for unauthorized scenarios.
Fixed an issue where it was not possible to create Oauth2 message action.
ACL: Fixed an issue with incorrect Purge/Delete Action Behavior for child asset with Traverse Children=yes when the parent asset (media,segmented,package ) is shared by ACL with read-write access but the child asset is not shared. Correct validation message that child asset cannot be purged/deleted was displayed but in fact the child asset was removed. Added ACL permission checks for all assets before performing the action.
Fixed an issue where Copy and Move Action from Azure to Azure location was failing for AS02, IMP, Segmented (HLS), Segmented (MPD) Assets.
Deleted events for collections erroneously appear in event type dropdown for event handler filter configuration
Fixed an issue where Purge action is failing for all Package assets. Issue fixed by updating resource root validation check to consider filename when comparing asset location with storage resource root location.
Fixed an issue where Restore action is failing on segmented asset.
[Flex Cut] Waveform: Fixed an issue where channel is missing after extract on a stereo project.
[Flex Cut] Waveform: Fixed an issue where waveform display is fragmented when zooming in on the project to minimum.
Added workflow edit permissions to control workflow retry and cancel in core API through edit permissions.
"TimestampIn"Marker Annotations: Fixed an issue where "TimestampIn" and "TimestampOut" fields shows relative timecode in FMP when "decoupleStartTimecodeEnabled" is set to false.
Fixed an issue where Restore action is failing on Segmented Asset.
FastObject: Fixed Inconsistencies in use of preferredStartTimecode among the different annotation retrieval endpoints.

Enhancements FlexMAM

Blue on black text difficult to read in Flex Track for asset hyperlinks. Blue changed to improve accessibility.
Audio Asset: Do not display frame in timecode. Always show duration in milliseconds for all references to audio timecode in FlexMAM and other apps.
[New FlexMOVE]: Support uploading files before metadata is filled in. Import workflow is triggered once the metadata has been added.
Selected marker should always be highlighted in the Summary page when user moves back and forth from the Manage page.
Bulk metadata edit should show toast message when edit fails on a particular asset that has READ_ONLY permission. The toast message lists the asset ids for which edit failed.
Change font to “Noto Sans” font for all languages in FlexMAM, Flex REVIEW and FlexMOVE.
Shift the logic of checking duplicate UDO names to Enterprise and remove from FlexMAM.
Enhance the scrubbing behavior of Flex Player
Clip range default name changed to <assetName>.clip.<inTC>.to.<outTC>
Reactivate Navigate to Object (redirection) link icon on Task metadata form for object, asset and UDO drilldown fields.
Suggestion form and chip modal –taxonomy and udo sort suggestions alphabetically. Sort is performed for the last node in the hierarchy of the suggestion displayed and not for the ancestors (higher levels).

Fixed Issues FlexMAM

Fixed a UI issue where when creating many Sub Collections the new objects overlap in the interface.
Upload: Fixed an issue where having taxonomy fields with multiplicity in the upload metadata form caused the upload to fail.
Fixed an issue where subtitle switcher is not available in the task panel player for the asset associated with the subtitle. Subtitle switcher is now displayed.
Fixed an issue where an incorrect tooltip (object object) is displayed in Workflow Notifications panel.
Add the quick option to add images to the active collections from the 3 dot menu on an asset in Search results.
Fixed an issue where manifest extraction failed to identify audio url so that the MPD file could not be previewed.
Fixed an issue where the configuration for 'Duplicate Name Supported' field for root UDOs and child UDOs was not properly respected.
Fixed an issue where the Upload button was not enabled. Issue caused because the CSRF token was not cleared on logout and an invalid token was sent on re-login.
Fixed an issue where wizard progress bar was not updating. Progress bar proceeds for these statuses: Pending, Running, Completed, Failed.
Fixed an issue where a clip is added to a collection in relative TC and a marker is added to the collection using absolute TC and the marker cannot be previewed. Issue caused because a system has not migrated fully to relative TC and so the marker TC is calculated incorrectly.
Support HEVC extension Upload via Flex Web Transfer.
Fixed an issue where wizard launch and legacy upload freeze when the associated metadataDefinition includes a drilldown udo type.
FlexMAM Summary Page: Fixed an issue where metadata drop-down values are hidden by the Player in Summary Page in Player expand mode.
Task Panel Player: Fixed an issue where Audio track name is shown as "und" when clicking Audio switcher icon.
Fixed an issue on bulk edit metadata on any field inside complex fields with multiplicity 0...1, original values are erased.
Fixed an issue where Collection item is shown as a double entry in the Working Collections after it has been activated.
Fix the Idle Timeout dialog so it behaves correctly, reflecting the underlying session state.
Translation string mistakes fixed in Manage and Search pages in FlexMAM and in FlexADMIN and FlexREVIEW.
Fixed an issue where Shift-click selection was not working on the second page of the bulk edit metadata page.
Fixed an issue where it was not possible to search asset by metadata field user type. Issue fixed by sending user id to FQL as a number and not a string.
Fixed an issue where adding a search filter for drilldown field (taxonomy & udo) and closing the filter modal and filter is not added in the chip bar. Drill down search is working for chip bar/table column and custom marker column in Summary and Manage page.
Udo Summary panel takes a long time to load when UDO has many children.
Metadata Form: Fixed an issue where metadata in a complex field of a complex field was not saved properly.
Fixed an issue where a multi option field with multiple default values did not persist all the default values in the asset metadata in FlexMAM Summary and Manage, Tasks, FlexREVIEW, FlexMOVE.

List of Services upgraded between 2024.5.x (LTS) and 2024.7.0

For the new 2024.7.0 release, most services have changed.