Dalet Flex 2024.1.0 Release Notes

Dalet is excited to present the release of Dalet Flex 2024.1. 

This release of Dalet Flex provides the new features, quality and security enhancements, bug fixes, and other changes listed below. 

Release Highlights

• Drill-down metadata fields support for User Defined Object
• Suggestions were added to the selection process as part of the improvements to the user experience of selecting taxons within drill-down metadata fields, 
• Enable metadata values related to other metadata values with the support of interdependent fields in Dalet Flex MAM
• Easier entering metadata values in a drill down menu
• Open a second window to view and enter metadata field objects
• ...and a whole bunch of other bug fixes!
  
  

Important Notes

An issue has been identified in this release where Password type fields are being corrupted when updating metadata instances. DO NOT install this release if your metadata definitions make use of Password type fields.

***

Collection events have now updated to use standard events: CREATED, DELETED, MODIFIED (and RENAMED) instead of the legacy format. 

Note that any event handlers or other logic based on Collection events must be reconfigured accordingly. 

Dalet Flex

New Features

Drill-down metadata fields support for User Defined Objects.

Added drill-down metadata fields support for User Defined Object. Users can now create a metadata definition with drill-down metadata in the Metadata Designer and select a UDO type as a backing store. 

As part of the improvements to the user experience of selecting taxons within drill-down metadata fields, suggestions were added to the selection process. I.e. as there can be many taxons in a taxonomy tree, users can now start typing the name of any taxon within the tree without having to select values one at a time. 

Dalet FlexMAM

New Features

A new feature has been developed to enable metadata values related to other metadata values with the support of interdependent fields in Dalet FlexMAM.

A new feature has been developed to make entering metadata values in a drill down menu easier. Now, there is a search bar at the top of the field for the user to enter the value rather than having to scroll down a list. On top of this, if a taxonomy is chosen, then the user can select which part of the taxon tree to choose a value by clicking the x after the taxon value to start from.

A new feature has been developed to open a second window to view and enter metadata field objects.

Changelog

2024.1.3

Enhancements DaletFlex

• FFP Plugins: Copy/Move (might be others in FFP or FSP plugins) protect moving/copying segmented asset with manifest folder with the same root folder as the Storage Resource base path.
• Support mixed framerates when generating FCP XML files.
• Change the SDK to use FastObject rather than Enterprise to fetch user details when updating bulk metadata.
• Optimise loading of UDOs referenced from metadata instance field so they are loaded as a batch.

Fixed Issues DaletFlex

• Updated Spring-Boot apps to fix an issue where a trailing slash was added and  HTTPS was downgraded to HTTP.
• Fixed an issue in the Xtend Adobe Panel extension failing. Fixed by allowing file:// as a CORS origin for the Login App.
• Improved logging messages when Add Items operation to collection fails because the  items in error have mis-formatted dates.
• FlexSearchElasticException: Fixed an issue in where an error appeared while building an authorization query.
• Fixed an issue in the Authorization service where an exception was thrown when retrieving subaccounts.

Fixed Issues FlexMAM

• Fixed an issue when performing bulk metadata update on complex fields with multiple cardinality removed existing values. The fix sets the behavior for bulk metadata update for complex fields: Metadata is merged when original field cardinality is 1 and updated field cardinality is 1, in all other cases multiple existing metadata values are overwritten with the new values.

List of services changed between 2024.1.2 and 2024.1.3

2024.1.2

Enhancements DaletFlex

• Add validations to ensure that assets ingested incorrectly to the wrong storage location do not to cause all assets in the folder to be removed on Purge. 
•  Give a common prefix to all name identifiers of entries in the `apps` section of Permissions, add FlexXTEND.
• Job Scheduler: Logic for Job loading and enqueue sequence in Job Scheduler logic updated. 
• Additional improvements to Job Scheduler to exclude jobs for which job quota for action configuration has been exceeded. Excluded jobs remain in a Pending state.
• When using the feature toggle `flex/shared/flex-enterprise/featureToggles/queueJobOnCreation`, set execution token when job is directly set to queued at the time of creation.
• Add Consul KV flex/enterprise/workflow/ignoreTimedOutState. When set to true the status of a workflow is displayed as Running and not move to Timed Out even if jobs associated to the workflow are moved to Timed Out. Default value is true.
• Added validation in the Purge plugin to fail a purge job if it tries to delete any directory which is configured as a storage resource root location. 
• Validate the Consul KVflex/enterprise/workflow/enableSyncJobLimitCheck for Enterprise jobs that have flex-enterprise/featureToggles/queueJobOnCreation = true and flex/enterprise/scheduler/job/maxSyncJobsInProgress >1.
• Add the ability to disable AWS CRT client in the filesystem library.

Fixed Issues DaletFlex

• Metadata fields with specified permissions are not returned when the query is made with masteruser credentials. To return results edit the metadata definition on the master account to give access rights for the filed to master user.
• Fixed an issue where Flex Enterprise could not resolve name for UDO publish event data when set as backing store to drill down field.
• Authorization service: Fixed an issue raised by authorization service when adding objects to a collection because Fast Object finds a deleted UDO with the same name as the active UDO.
• Metadata Issue: Fixed an issue where the Password field is erroneously updated whenever modifications are made to other fields in FlexMAM.
• Image asset: Fixed an issue where a system error is displayed when running an extract job on an image asset.
• Fixed an issue where hotfolder repeatedly fails and restarts automatically and fails again.  

Enhancements FlexMAM

• New drill-down field triggers backend lookup request on click or on focus of typeahead without typing.

Fixed Issues FlexMAM

• Fixed an issue where selecting a UDO in the Membership tab did not jump to the correct UDO.

List of services changed between 2024.1.1 and 2024.1.2

2024.1.1

Enhancements DaletFlex

• Add resilience to image transcode jobs by improving the retrieval and reuse of existing placeholders on job retry. This will also limit the creation of ghost children during this process.
• Event handler service: Catch all exceptions when applying an event handler's filter.
• Provided a new tool, flex-technicalmetadata-migration, to unsure persistent data in case of media assets with drop frame, fractional frame rate (e.g.: 29.97, 59.94) and videoStream startTimecode not 0) with extracted technical metadata.
• To improve performance, flex-job nodes no longer process command tasks meant for the master nodes.
• Ensured script locks are unlocked if a script errors. 
• Added support of Hazelcast locks for JEF scripting in addition to Redisson locks.

Fixed Issues DaletFlex

• Fixed an issue where transcoded MPEG-DASH files where created but not attached as child files to an asset. Issue solved by updating response handler to accept both format code and extension.
• Fixed an issue where workflows quota is NOT checked when workflow is launched from Hot Folder. 
• Collection service: Fixed an issue where there was overeager removal of parentUuids when processing subcollections by ensuring that subcollection deletion stops for branches that encounter subcollections not owned by the deleting user.
• To prevent jobs remaining in a Pending state indefinitely, workflow state is set to running once the timed out job completes.
• Fixed an issue where Start Timecode values are showing different in Enterprise and FlexMAM for 29.97/59.94 from Enterprise.
• Fixed an issue where it is not possible to clear metadata password fields when Enterprise uses the metadata service.
• Fixed an issue where Enterprise copy and move action is not working for S3 to S3.

Fixed Issues FlexMAM

• Fixed an issue where it was not possible to update a market and the message `Invalid timecode` was displayed even when updating a marker with a valid timecode.
• In wizard Object metadata fields with no udo type selected and set to typeahead, the browser crashes there are a lot of UDOs. Issue fixed by adding a limit to the number of UDOs to load while using a typeahead field.

List of services changed between 2024.1.0 and 2024.1.1

2024.1.0

Enhancements DaletFlex

• Updated flex-authorisation data model to reference the newly created permissions in Enterprise.
• In the Flex-Brio integration, optimized the update of duration in growing files by Extract Tech Metadata.
• In the Flex Mobile app, added the ability to upload files and images
• Added the ability to retry Amberfin plugin failed jobs.
• In the Flex mobile app, added support for retrieving metadata objects
• The Collection DELETED event now includes handling denormalised collections
• For the Aspera HSTS Resource (JEF) plugin, add file input and output.
• Updated the JEF SDK to support UDOs as backingstores for drill-down metadata fields
• Taxonomy taxons are now indexed into Elasticsearch. This is initial indexing, without ancestry
• Added support for suggestions for drill-down fields, within a whole taxonomy / UDO backing store, and within a subset of the tree
• Added new includeAncestors new query parameter added to Taxonomy Service API. By default, the value is set to false. When enabled, the call will return all taxon ancestory, ordered from top to bottom.
• Ancestors are now indexed with taxon.
• Added new endoints to reindex taxons: /api/reindex/taxons to reindex all taxon, and /api/index/taxonomy/{taxonomyId}/taxon/{taxonId} to reindex a specific taxon. A Full reindex will reindex taxons as well.
• Taxonomy service: provide endpoint to paginate through all taxons in the system
• Added a new endpoint in the Flex Taxonomy Service /api/taxons?offset={offset}&limit={limit}&accountId={accountId}. offset is a number type optional parameter indicating the source of the record. limit is a number type optional parameter indicating number of records fetched per call, default: 100 (max 1000). acountId is an optional number type parameter for the accountId from which all taxons are fetched. This does not include taxon of deleted taxonomy, deleted taxon, and pseudo-taxon at the top level of the taxonomy.
• UDO Index JSON template now includes ancestry.
• Disabled ETM update for asset version lower then 4 and has asset having annotations by default. To force ETM update set {{flex-enterprise.featureToggles.allowTechnicalMetadataChanges }} to true.
• Improved annotation retrieval response times

• Enterprise DB migrations can now run with latest MariaDB 10.4.31+

• Fix subtitle timecodes in the media cortex plugin.

• Allow metadata instances to be saved referencing deleted objects, taxons, tags etc from scripts. New parameter allowDeletedReferences for API PUT /api/{assetId}/metadata in Enterprise.

• UDO types indexed in each UDO ancestor entry, to support the UDO-backed drill-down-field typeahead.

• Added support for extracting asset info from Avid master clips (folder containing *.AAF files with linked *.MXF files).

•  Added support for *.AAF and OpAtom MXF Master Clip export across transcode/stitch/sequence.

• Collection events have now updated to use standard events: CREATED, DELETED, MODIFIED (and RENAMED) instead of the legacy format. Note that any event handlers or other logic based on Collection events must be reconfigured accordingly.

• Added support for exporting to AAF for transcode jobs by specifying new property in media profile: `formatVariant = "OPATOM"` with `format = "MXF"`.

• Added dedicated error messages for when calls fail due to a security measure triggered when inputs contain scripts, html or css syntax.

• Added caching for retrieval of Flex Groups from Enterprise to the Login App.

• Added a removeTags job type to Flex File Processor, for removing tags from an object.

• Added the UDO backing store type to metadata commons.

• Updated the Metadata Designer to support UDOs as a backing store from drill-down fields.

• FFmpeg now allows multiple mono channel MCA descriptor channel IDs.

• To improve performance for indexing metadata annotations, added caching for taxons, tags and objects in indexelastic.

• Set the event correlation_id to the job Id for Event type TRANSCODED to fully enable usage tracking.

• Set the event correlation_id to the job Id for Event type PROXY_CREATED to fully enable usage tracking.

• Set the event correlation_id to the job Id for Event type PUBLISHED to fully enable usage tracking.

• Set the event correlation_id to the job Id for Event type COPIED to fully enable usage tracking.

• Set the event correlation_id to the job Id for Event type Event type MOVED to fully enable usage tracking.

• Set the event correlation_id to the job Id for Event type Event type RESTORED to fully enable usage tracking.

• Renames Hazelcast cache keys used by Enterprise to align with the Enterprise convention.

• for Enterprise, introduced a metric, dalet_service_info, capturing the service state (Boolean) and other metadata (version, runtime, runtime_name, runtime_version, runtime_framework).

• In the FFP and ImageProxy services, introduced a metric, dalet_service_info, capturing the service state (Boolean) and other metadata (service name, version, type (nodes, java…) and IP.

• Introduced an endpoint for retrieving a list of UDOs for given UDO IDs: /api/userDefinedObjects?id=xxx&id=yyy.

• To prevent a case of a “perpetually growing” file state, added a configurable timeout to detect files/folders which have not been modified or changed size in a long time, even if the file is locked or a *.wrec file exists.

• Taxonomy queries now include the “deleted” status parameter (boolean) in the API response

• For the ReplicationGateway and WriteObject, introduced a metric, dalet_service_info, capturing the service state (Boolean) and other metadata (service name, version, type (nodes, java…) and IP.

• To improve performance when fetching metadata instances, add metadata caching support to metadata, metadata merge, replicate gateway and fastobject services.

• Further FFP restore job fixes implemented for processing small/lightweight files.

• To improve performance when indexing UDOs, now caching ACLs for parent UDOs in indexelastic.

• Add support to signal workflow manually if it stuck and not moving to next node using new API endpoint.

• Added the pluralName property in the /assets/id/variables/id/values endpoint so UDO will return with their UDO type.

• FFmpeg buffer size is now configured via consul and the env var. Use `flex/flex-streamprocessor-service/ffmpegNetworkReadBufferSizeMegabytes` to set the size of read buffers for network sources (non-image sequence sources) and use `flex/flex-streamprocessor-service/ffmpegNetworkShortSeekSizeMegabytes` to set the size of seek up to which the current network response will be streamed rather than creating a new byte range offset request in FFmpeg and FMPE.

• Added support for a new non-scaled WORKER instance deployment type for use with CONTROLLER instances.

Fixed Issues DaletFlex

• Fixed an issue where Metadata definitions with a URL field did not accept an updated URL via the Flex API or the Enterprise interface.

• Fixed an issue where AssetImporter was failing if mpd has no duration even though it is not importing a life asset.

• Fixed an issue so the concurrent workflow limit is removed from Redis when the limit is removed from Workflow Definition.

• Fixed bean auto-wiring issue with launch and wait workflow action.

•  Fixed an issue where the PDF preview button is not displayed in FlexMAM for an attachment generated from Baton Quality Control (JEF) action.

• Collection service: Fixed an issue when deleting a collection that has a parent collection with specified write ACL access from API results in an error.

• Node JS Service support: Enforce that the origin regex must be a full-string natch with the domain regex to allow access.

• Fixed an issue where Event Handler service does not paginate through the Enterprise /api/users/<userId>/accessibleWorkspaces API, so if the user is a member of >100 workspaces, then permission may denied incorrectly. Issue fixed by increasing the default limit of available workspaces to 1000.

• Authorisation service now includes subaccounts when adding a master account to cache.

• Fixed an issue where untar jobs were failing. Issue fixed on deployment of updated FlexFileProcessor.

• Validated template for JAE permission for deployment for all customer environments

• Fixed an issue where waveforms extraction not working for ALAC Codec by extracting the waveform from the proxy file.

• Fixed an issue with Custom Markers where createAnnotations triggers this exception ‘SocketTimeoutException: Read timed out’. Issue fixed by updatimg flex-master service to use metadata cache provider.

• Fixed an issue in Flex Dalet CUT where growing files did not grow in the trimmer due to a manifest caching issue in the Video Proxy Service.

• Fixed an issue where OPAtom transcoding did not work as the input file name contained an .mxf extension before the actual *.aaf extension. Fixed by ignoring files with .mxf in the file names but not as the final extension.

• Fixed an issue with updating metadata definition rules via API.

• Fixed an issue where where existing custom marker or new custom marker are not displayed when set custom marker metadata for a searchable udo drilldown field.

   

Enhancements FlexMAM

• Added the ability to see 10,25, 50 and 100 keyframes at once with the summary and manage pages in Dalet FlexMAM.

• Added support for images in Dalet CUT: when an image size is less than 5 MB when placed on the Dalet CUT timeline, no proxy image is used. If larger than 5 MB, a proxy image will load. Currently limited to JPEG and PNG only.

• In preview monitor a new duration of media counter, with time code and real time, in order to see the duration of media while having different segment in/out markers.

• Along with Video and Audio metadata on the technical metadata tab in the summary and manage pages in Dalet FlexMAM, Technical metadata for images are now displayed.

• A button has been added in FlexCut, when selecting a clip, you can hover over the trimmer import button to select the trimmer application.

• In Dalet FlexMAM, a user can now open an UDO drilldown menu on a different tab.

• Connected the new drill down picker to the UDO and taxonomy backstores as preparation for feature completion.

• Added support of interdependence rules for metadata fields in FlexMAM, in particular the ‘setVisible’ rule.

• With the new implementation of UI form pickers of drill-down menus, users have the ability to search for values at the top of the field rather than have to scroll down a long list of values.

• Improvements made to metadata forms to improve stability and facilitate new functionalities.

• In FlexMAM apps introduce a metric, dalet_service_info, capturing the service state (Boolean) and other metadata (service name, version, type (nodes, java…) and IP

Fixed Issues FlexMAM

• Fixed an issue with an invalid tooltip where when you hover over an audio asset in Dalet FlexMAM, you see ‘Audio’ as the tool tip.

• Fixed an issue where when a user types the values in a User Object field, then "No Records Found"

• message is displays. Now, values are displayed correctly.

• Fixed an issue in Dalet FlexCUT where once you activate and Pin a Collection, it disappears when a refresh happens.

• Fixed an issue in Dalet FlexCUT where if you long drag a media asset to the browser clipbin, an error occurred.

• Fixed an issue where when searching for a marker of a video with timecode, which is set in the player, the timecode appears rather than the real time in the marker search results.

• Fixed an issue in FlexCUT where if you select the American format for dates in Flex Admin, then it will be displayed correctly in Dalet FlexCUT.

• Fixed and issue in the Dalet Flex Reviewer where the full name of the audio tracks associated with the video are only partially displayed when clicking the audio switcher icon.

• Fixed an issue in the FlexREVIEW app where now when previewing an MPEG-Dash proxy, the audio switcher icon appears.
• Fixed an issue where the profile panel was missing in the FlexCut App.
• Fixed an issue where in FlexREVIEW, when selecting to filter comments by user, the filter was not working correctly.
• Fixed an issue with Dalet FlexCUT where when a user clicks a collection, intermittently nothing displayed unless the next or previous buttons were clicked. This has now been resolved.
• An issue has been fixed that if you have that next to a field in the summary page that has a ‘Search icon,’ that nothing happens when you click the icon. Now when you click it, it will conduct a search and show you other assets with the same value as in the field.
• An issue has been resolved that when a field has a multiplicity of 0.1, then when you enter a value and save, the values do not save.
• An issue has been resolved where if the user creates a UDO from the Global Header and enters metadata from the UDO form, that the metadata wouldn’t carry over to the Asset’s metadata form.
• An issue has been resolved, where on searching the hierarchy of taxonomy in drilldown metadata field, the search text was overlapping with the placeholder text.
• Three issues have been resolved with clip creation. The main one was that when a user tried to create a clip on the manage page, clip loading kept showing. The second issue is that when setting the pagination to ten clips, user wouldn’t see the pagination button if there were more than ten clips. The final issue was that not all clips would show up in the clipbin folder. All three issues have been resolved.