Dalet Flex 2024.10.0 Release Notes

Dalet is excited to present the release of Dalet Flex 2024.10.0.

This release of Dalet Flex provides the new features, quality and security enhancements, bug fixes, and other changes listed below.

Dalet Flex

New Features

ArchiveAsset type (formerly TarAsset) with expanded support

TarAsset in Dalet Flex has been expanded to support different types of compression, such as zip formats. To support this TarAsset has been generalized to ArchiveAsset type.

From this version TarAsset has been renamed ArchiveAsset. Tar, Untar (Create Object) and Import Tar (Import) plugins have been renamed to Create Archive, Extract Archive and Import Archive respectively to follow the new refurbished ArchiveAsset.

After upgrading to Dalet Flex 2024.10.0, existing Groovy script code must be modified to use the new ArchiveAsset related objects instead of the old TarAsset related objects. Existing TarAssets before the upgrade will be automatically migrated to new ArchiveAsset. Existing old action config for Tar, Untar and Import Tar will still work and launch the new related plugin

If there are scripts specifically using any of the previous TarAsset terms on Object Type, you should review if there is anything to update. The following objects must be manually updated:

• Object Type
• File Asset Type
• Resource Type
• Rest API parameters

Please see this article or contact your Flex representative for more information.

Remove support for metadata comments

From this version of Dalet Flex, support for comments on metadata variables is removed.

keep_visibility configuration for Flex CM

From this version an additional option has been added to the keep_visibility flag in Flex CM. The keep_visibility data type has been changed from Boolean to string and the new value all_with_fallback_to_none is the default value.:

• keep_visibility=true commits changes to target applying all visibility dependencies from source
• keep_visibility=false commits changes to target removing all visibility dependencies from source
• all_with_fallback_to_none (default) – if keep_visibility is not set, flex-cm attempts to apply the same visibility as source but if it fails it removes all visibilities in the target environment

Updates to Complex with a Column Layout

In Dalet Flex 2024.10.0 additional improvements have been made to metadata forms in FlexMAM. Date/Date & Time/User fields can be added to a Complex metadata field with a column layout. The fields can be editable or read-only and if a field is editable it is highlighted when hovering over the field. The Date & Time field is displayed as a row when the Summary panel is resized and the position of the Date & Time selector is dynamic based on the size of the Summary panel. The User field uses a typeahead selector.

Flex Stream Processor updated to use Ffmpeg 7.0.1

Improvements to encoding profile settings for proxy creation

Improved general encoding profile settings and specific settings for H264 encoding (using libx264). This should result in better quality proxy encodes with a lower variability in bit-rate.

New default behavior for GET and LIST UDO API

From Dalet Flex 2024.10.0, the default behavior of get single udo and get list of udo api does not return children by default.

GET UDO API

GET - <host>/api/<udt-plural-name>/<udo-id> - returns UDO details without children

GET - <host>/api/<udt-plural-name>/<udo-id>;skipChildren=false – returns UDO details with children

GET - <host>/api/<udt-plural-name>/<udo-id>/children returns UDO with children with pagination

LIST UDO API

GET - `<host>/api/<udt-plural-name> - returns UDO list without children

GET - <host>/api/<udt-plural-name>;skipChildren=false – returns UDO list with chidren. For scale the solution and improve performance, it is recommended to use separate API to load the children of UDOs individually.

Expected Duration field added to placeholders and media assets 

To allow users to see the estimated duration of assets, the Expected Duration field has been added to media assets and to media asset placeholders. The Expected Duration field is displayed by default in the Details tab of the Summary Panel/ Manage Page in FlexMAM.  If the Expected Duration is added using API, it is displayed in the File tab of an asset in the Common Placeholder and File Information section in Flex Core in microseconds. You can edit the value of the field from the Details tab in FlexMAM and using API and you can display the field in the Asset Results page and in Search filters.

FlexMAM Player: New design for growing (live) file

When a recording is live a LIVE icon is displayed in FlexMAM player in the Summary and Manage page.

Improve Type filter options in FlexMAM

In this version the Type search filter had been improved so that each asset filter type has a unique meaning. Only these options are available in the Type filter:

• Video: Only filled videos, excluding live feeds and placeholders
• Audio: Only filled audios, excluding live feeds and placeholders
• Live: Only segmented assets (live audio or live video)
• Image: Only filled images, excluding placeholders
• Document: Only filled documents, excluding placeholders
• Placeholder: Only placeholder videos, images, document and audios
• Package: Only Package Asset (IMP)
• Camera Card: Only Camera card
• Group: Only Group assets
• Edit: Only Project asset, Sequence asset and Edit asset
• UDO’S: Display only root UDO

Functional Update

Managing ACL Permissions on Collections

Collections in Dalet Flex can be used to catalogue and browse content. As part of ongoing improvements to Collections in Dalet Flex, the data model is being enhanced to support additional functional requirements, including the ability to grant access to assets via their membership of collections with appropriately configured ACLs. In Dalet Flex 2024.10.0 this functionality is partially available.

Business Intelligence Event Handler

Fixed an issue where Event Handler service does not start up unless Elasticsearch is running. Removed the dependency on Elasticsearch so flex-eventhandler-service can start but Business Intelligence handlers that do not have a configured Elasticsearch URL will only log warnings to indicate that messages are being dropped and are not being sent to an index. From this version, it is mandatory to configure the Elasticsearch URL configuration parameter in all Business Intelligence Event Handlers. Failure to do so will result in this type of event handler stopping processing events.

Known Issues 

FLEX-72117 Add SDK method to access job variables 

When passing `context.job.id` (id of job being executed), system generated variables such as accountId, userId, workspaceId are not retrieved. When passing `job.id` instead of `context.job.id` in getJobVariables, system generated variables are retrieved. 

Changelog

2024.10.0

Enhancements DaletFlex

• OpAtom MXF: Support Create Proxy/Transcode action on OpAtom MXF asset. Aaf file name is mandatory for transcode and createProxy, if it is not found the job fails. NOTE: For this version of Dalet Flex, Create Proxy/Transcode action on OpAtom MXF asset is only supported when using Clipping Configuration for the transcode job.
• FQL enhancement: Allow multiple search criteria to use the same nested context.
• JEF Copy/Move: Enhanced set tags support for FFP jobs. Added option to set the tags in the JEF Copy/Move actions in the UI. NOTE: Tag format updated to tag.key and tag.value with multiplicity 0 to many from the previous format 0…N (a list of) String. New format is not backwards compatible, existing action configurations using the old tag format must manually convert the list of string into key/values, otherwise jobs will fail.
• Improve logging warning in job history for Extract Tech Metadata jobs for images that exceed the default tiffMaxLength. Improved warning is Tiff image size is limited to 100MB. If your images exceed this limit, please configure KV flex/flex-jobasyncexecutor-service/tiffMaxLength.
• HotFolder remove support for scanning already processed files (processFileOnUpdate): Redis migration from prev version.
• HotFolder remove support for scanning already processed files (processFileOnUpdate): stopping one or all HF instance for long time.
• Add a flag in Enterprise UI and Rest API so that executed JEF jobs complete irrespective of failure. The flag is set by selecting the Complete job on Failure checkbox when launching a job. If this option is selected the job completes even if there is a wrong condition/configuration with this error `ignoring the error based on configuration`. Jobs failed by watcher do not complete since failure is not connected to execution.  
• Add to Amberfin requests externalId (Flex Job id) for downstream tracking.
• In Extract Plugin type, create a new JEF List Archive Plugin. For an input of a single file from Archive Asset or a sequence of Archive assets, the List Archive plugin generates a manifest file for the asset. The manifest file can be attached to the asset as an Attachment based on configuration.
• Add Validation for Format Variant + FrameRate (probably PixelRate) not matching in Media Profile Validation.
• When sorting, allow FQL to specify whether unranked items should be at the start or end of the search results.
• Search/Table Template filter: Add Start timecode to filters of Technical in Search and in Table Template.
• Translation: Update French translations: Asset Types, Search table headers. 
• Add SDK method to access job variables.
• Additional CORS support from Flex VPS.
• To prevent metadata merge failing, ensure previous metadata instance is deleted if variant of an asset is changed to a variant with no associated metadata definition.
• JEF Script SDK: Add missing getJobActionInfo method to JobService.
• Result of FFP list, pack and unpack jobs was extended to include the totalSizeInBytes of the files and folders involved in the operation. The size of individual files is now also included in the following manifest format outputs: SIZES, DETAILS. Folders are now also included in the following manifest format outputs: SIZES, ENTRIES.
• Flex Enterprise API: Add GET usage rights
• Flex Enterprise API: Add CREATE (POST) usage rights
• Fixed gap where non-backwards compatibility was introduced in FFP as part of supporting new pack/unpack API. 3 KVs added that support legacy behavior by default and generate a hash file/sha1 file/support batched tar jobs (up to a batchsize of 1000).
• REST API to Read Status Flow
• Assign Status Flow to new Asset/Object with API endpoint. Status is assigned to a variant and will be assigned to an asset when it is created. It is not possible to update the status flow ID of a variant after an asset has already been associated to that variant. Trying to update the variant in this case will throw an error.
• Add support to add/remove/retrieve Status Flow permissions to users and groups with API.
• Status Flow: Add support for status transitions from a single source status to multiple target stauses.
• S3 CRT Client for s3 supported storage based on crtSupportedEndpoints system property.
• Reduce default caching to prevent playback failing if VFS location has changed after proxy was created.
• FlexMAM: Change row highlight color in table + removing type icon background
• Collection migration tool must consistently retrieve all records using paging otherwise an error should be logged and the migration should fail. Retry completes the pending collections.
• Added filtering before handling by IndexElastic for most MessageHandlers, and especially for the ones which use Delayers, to prevent OOM crashes of IndexElastic due to unnecessary retention of all messages.
• Rename browser title page for Ops Dashboard from Operations Dashboard to FlexTRACK.
• Added <role>=main element to first audio adaptation set in DASH manifest to indicate to the FMP player which audio stream should be played by default.
• JEF SDK: Adds new method context.getWorkflowObjectVariables(variablesName) to retrieve an object variable that is a List as opposed to a singe FlexObject. The method gives a useful warning if variablesName represents a singular workflow object variable and advises using the singular method  context.getWorkflowObjectVariable(variableName).
• Provide a new property inputDisplayAspectRatio in the media profile jsoin to override detected value for incorrectly encoded sources.
• Patched FFmpeg to seek backwards segment by segment if the initial seek overshoots the desired timestamp. This can occur when segment durations cannot be used for frame accurate calculations when frame rate is 29.97fps.
• Short term fix: Change metadata-commons (as used by Enterprise) to ignore any changes to a Timecode metadata field Time component which only modify `;` characters to `:` or vice versa so that metadata can be saved.
• FastObject: Add paginated endpoint to MinimalObject API to retrieve children belonging to a UDO.

Fixed Issues DaletFlex

• Fixed an issue where an Un-Tar job with invalid index value was progressing and completing instead of failing. Job now fails with this an appropriate error, for example: Extract Archive job: failed in file processor service for the request: unpack-141, failure reason:Effective startIndex: 200 > sequence endIndex: 3
• Respect account visibility while displaying taxonomies based on account accessibility and visibility.
• Fixed an issue where rendering a Dalet Cut AAF asset from a growing MXF to a File location was failing because the file is first saved to a temporary location until it is fully rendered and only then it is copied to the final File location. During render Enterprise identifies that the new file did not exist in its final location and fails the job. Issue solved by adding a Consul KV enableConformTargetFileGrowingProgressReport which when set to false allows the JEF Conform plugin to skip reporting the existence of the file until FSP completes the job so Enterprise does not know about the file until the render is complete and does not fail the job.
• Use mediainfo instead of ffprobe for growing MXF files to avoid possible endless increase in growing polling times..
• Fixed an issue where workflow runs two node concurrently. Issue fixed by preventing the system from sending signals to the workflow when jobs were moved from pending to running that was causing the nodes to run concurrently.
• [Add Marker to Collection/ClipBin] Fixed an issue where markers were added to activated collection/clipbin instead of to the selected collection. Issue fixed by first checking for a selected collection before falling back to the active collection or clipbin.
• Permission to see Dalet Cut in the app switcher/use the URL to access Dalet Cut is controlled by permissions in Dalet Core > Access > Roles AND the KV flex/shared/featureToggles/flexCutAppEnabled/ (default value=false)
• Fixed an issue where FlexMOVE was failing intermittently after logout and login. Issue caused by intermittent caching of XSRF token between sessions.
• Panels App: Fixed an issue where a checkbox inside a complex field with multiplicity & multi option field as checkbox type with multiplicity was not working as expected in wizards and legacy uploader.
• Fixed a specific scenario in MPD parsing when only a single segment exists in the DASH manifest.
• Job should auto-retry for OptimisticLockException (handle wrapped exception).
• Flex Player: Display end time in line in the Player. If the asset is a growing live asset, end time is not displayed.
• Player: Fixed an issue where audio tracks are not displayed properly and they do not have the correct labels. Issue fixed by Shaka player and Flex player consuming labels from the mpeg dash manifest.
• Calculate video frame rate duration if required BEFORE rejecting zero duration frames.
• Hot folder/filesystem: Improve handling of permission issues when listing objects in S3. When a filename contains / or \ or there are files with incorrect permissions in an S3 bucket,a warning message is displayed in Enterprise until the file is renamed, however all other files in the Hotfolder are processed correctly. NOTE: To correctly process files with special characters in the filename, use Source Storage Resource option and the Storage Resource Subpath + Filename field instead of the deprecated Source Resource Item option in the import configuration to import assets.
• FlexTRACK (Operations Dashboard) renders an error page when lacking permission to connect to the app.
• Fixed an issue where jobs are not transitioning after completion. Issue solved by excluding deleted objects when searching by name and type.
• Fixed an issue where the Player progress was showing play complete when first loading an asset. progress at startup.
• Fixed an issue in the UI where taxonomy was overlapping with wizard.
• Fixed an issue where an incorrect HLS master playlist where a stream is referenced twice is created for audio only packages.
• Added a simple logic check to prevent any changes to detected source asset info if the forced input frame rate equals the detected frame rate.
• Fixed an issue where Pack job fails with format=TAR and compression=NONE.
• Fix colorspace conversion for HEVC source.
• UDO Page: Fixed an issue where an expander icon was displayed on a non-UDO object that has nothing to expand in a UDO page.
• New job listener added globally outside of job processing method for each job executed so that they are not repeatedly added.
• Fixed an issue where the selected audio stream was not playing when play in the Manage page for an asset with multi audio hls tracks.
• Fixed an issue where Collections are not returned by the API call "api/collections/containingItem/" but are visible in FlexMAM.
• Fixed an issue where an action of type Messages with GET for containing items always fails the first time.
• Fixed an issue when importing a Spanned media asset where the first file does not include the duration of the entire asset. Setting the Infer Spanned Media Files option to Yes in the Import Spanned Media action displays the correct duration. Setting the Infer Spanned Media Files option to No in the Import Spanned Media action displays this log message The duration of the Flex Spanned Media asset (R3D) will be set as total duration of media from first file. This should be ok if you have selected all media files composing the Spanned Media sequence.
• Fixed an issue where Image Transcode is failing on shard storage resource. Issue fixed by implementing changes in JAE so that when sharding is enabled the sharded path is created before checking the file size during the image transcode plugin execution.
• Improve handling of bad request cases; add `flex.request.id` to error response payloads
• Conform action: Fixed an issue where a Conform action was following for media assets with videoStreamCount > 0. Issue fixed by changing the channel mapping logic to handle audio before video in hires sources when preparing conform job.
• Fixed an issue where default hostname was added into signedURL for Minio instead of the configured hostname and Vantage jobs are failing. Fix validated for additional flavors of S3 storage, MinIO, NetApp, Wasabi.
• Enterprise API: Jobs: Endpoint – Fixed an issue where api/jobs/:jobId/outputs does not return a valid json response consistently. After this fix, it is possible that the json response of this endpoint is not the same as before 2024.10.0 release so, please check and adjust the client code if you are using /api/jobs/{jobId}/outputs API endpoints.
• FlexAdobePanel: Fixed an issue where render with FlexXtend is not working.
• Assets which are not members of any collections are minimally indexed (with no metadata, ACL, etc) in OpenSearch. When searching using Collection service endpoint, a 200 OK without any collections response is returned when an asset is not a member of any collections.
• Fixed an issue where Event Handler service does not start up unless Elasticsearch is running. Removed the dependency on Elasticsearch so flex-eventhandler-service can start but Business Intelligence handlers that do not have a configured Elasticsearch URL will only log warnings to indicate that messages are being dropped and are not being sent to an index. From this version, it is mandatory to configure the URL configuration parameter in all Business Intelligence Event Handlers. Failure to do so will result in this type of event handler stopping processing events.
• Fixed an issue where Extract Technical Metadata is failing if file is ingested into a shard folder.
• FlexADMIN: Fixed an issue where an extra scrollbar appears in the User Management page in FlexADMIN.
• Fixed an issue when audio is not played or an incorrect audio track is played when playing Mpeg Dash from FlexMAM because of a problem with the synchronization between Shaka player and Flex player. After fixing the issue, correct audio tracks can be played  by manually updating the role in manifest.mpd file of proxy.
• FlexMAM: Fixed an issue where creating a filter on a Time metadata field, by clicking the up arrow to adjust the time closes the and the filter is not created.
• Fixed an issue where Expected Duration value in tile template is not shown in hh:mm:ss format.
• FlexADMIN: Fixed an issue where Expansion of Complex field setting in FlexADMIN is not honored in Task Notification, Wizard and Uploader app.
• FlexMAM Player: Fixed an issue where FlexMAM Player for Live assets was displaying incorrect Start and End TimeCodes. Start TimeCode is 00:00:00:00 and no END TimeCode is removed while the asset is Live.  
• Web transfer: Fixed an issue where values added in date field and time field with form type "text" are cleared when adding multiplicity for any other field in the metadata form. 
• FlexREVIEW: Fixed an issue where clicking ESC from full screen page in FlexREVIEW did not go back to the Manage page.
• Fixed an issue where a System Error message is displayed when updating assets metadata for variables that have custom permissions.
• Fixed an issue where it was not possible to remove assets from collection and errors.collection.removeItems error is shown in the toast message.
• Fixed an issue where it was not possible to collection items and a "400 Bad Request" is thrown.

List of Services upgraded between 2024.9.0 and 2024.10.0 

For the new 2024.10.0 release, most services have changed.