Dalet Solutions: All About Logs

This article explains all there is to know about logs in Dalet from what they are, where to find them and how to use them.

Dalet Solutions and Their Logs

This document explains:

-    What Dalet logs are;
-    Where Dalet logs are located;
-    Which logs are relevant to upload, depending on the type of issue;
-    How to backup Dalet logs to avoid data loss.

What are Logs

Logs are text files written automatically by Dalet applications. Any Dalet application logs its operations, such as:
-    Dalet solutions' Windows services, back- and front end applications,
-    Dalet solutions' application server agents (such as Dalet Galaxy DBServer, Ingest Server, Notifier, etc),

Most services and agents are described in the WIKIS:  Dalet Solutions' Server- and Agent Types, Amberfin, Brio, Cube manuals.
 

You will usually find a few files per application:
-    the log file (a .log extension) where the application currently logs its operations
-    other logs files (.log1/log2/log3, or .1.log/.2.log/.3.log extension), to keep some log history
-    info log files, only containing logs of type INFO
-    error log file, only containing logs of type ERROR
As a consequence, not only the .log file but ALL the logs files related to a specific agent are necessary to investigate about an issue.

Where are Dalet logs located?

Dalet logs location depends on the version you are running.

For versions up to Dalet Galaxy 4.0.254375

In versions up to 4.0.254375 included, all Dalet applications running on a machine write their logs in C:\ProgramData\DaletPlusData\Log:

For Dalet Galaxy 4.0.271530

On this version, logs are located in C:\ProgramData\DaletPlusData\Log, but each agent has its own subfolder:

Log folder for Dalet version 4.0.271530

For Galaxy Five 1 and 2 (4.0.287872 and 4.0.302817)

In Galaxy Five 1 and 2 versions (i.e. for versions 4.0.287872 and 4.0.302817), all Dalet applications running on a machine write their logs in C:\ProgramData\DaletLogs. One subfolder per application is created. 

Log folder for Dalet versions 4.0.287872 and 4.0.302871

From Galaxy Five 3 (4.0.322283 and up)

From Galaxy Five 3 (i.e. version 4.0.322283 and up), all Dalet applications running on a machine write their logs in C:\ProgramData\Dalet\DaletLogs. One subfolder per application is created. 

Log folder for Dalet versions 4.0.322283 and up

Dalet Brio Logs

Collecting Logs in Dalet Brio from Dalet Brio 3.10

From version Dalet Brio 3.10 (3.10.281394), Log Collector service is used to automatically collect all logs from Brio portal. Log Collector service collects all logs from the relevant location without requiring any specific configuration.  For more information, see the Brio - Administration Guide attached as a PDF to this wiki article: Dalet Brio 3: Manuals

The logs are collected in a zip files stored in the configured report location. This zip archive contains the following folders:
•    Configuration: all Brio configuration files;
•    Logs: logs collected (by the filter selected);
•    RegistryKeys: Windows registry keys used by Brio.
 

All Brio Logs SHOULD be collected using the Log collector. You DO NOT need to access logs manually.

Collecting Logs Before Dalet Brio 3.10

Logs in Dalet Brio3 are written in two locations: C:\Program Files (x86)\Dalet\ and C:\ProgramData\Dalet\Brio

Logs for services in Dalet Brio, are written in subfolders per service to:

Brio3: C:\Program Files (x86)\Dalet\Brio3\logs

Brio2: C:\Program Files (x86)\Dalet\Brio2\logs

For Players and Recorders there are two layers of logs, Driver Layer and Logic Layer.

The Driver Layer records commands related to hardware and hardware issues such as drop frames.

The Logic Layer records software commands and responses. The Logic Layer is divided into three categories:
-    command logs: include every command from the server, and do not depend on the log level; the file name convention is "Commands-Logs.txt.(YearMonthDay).txt", like Commands-Logs.txt.20100616.txt
-    video board dll (TNTDSX) command logs: the file name convention is "Commands-TntDsx-Logs.txt.(YearMonthDay).txt", like Commands-TntDsx-Logs.txt.20100616.txt
-    playback channel logs: the file name convention is "TntChannelServerPlay_(Player)-Log.txt.(YearMonthDay).txt", like TntChannelServerPlay_C-Log.txt.20100616.txt.
 

Logs are also written to C:\ProgramData\Dalet\Brio on both Brio3 and Brio2.

Logs for Amctools on a Dalet Brio machine

The Amctools are used for the Avid SDK Interplay integration. The logs are written to the C:\ProgramData\DaletLogs\amctools folder and the amctools_dtk logs are written to C:\ProgramData\DaletLogs folder in a Dalet Brio machine.

Dalet Amberfin Logs

Amberfin logs are located in different subfolders of \ProgramData\AmberFin\. There is an automatic way to collect Amberfin logs (please see section 4.8). Right click on Amberfin icon located in the notification bar. Then select Amberfin logs >> Zip local logs to desktop. A zip file is then created on the PC desktop, with a timestamp.

A list of all agents producing logs is in Dalet Amberfin Troubleshooting.

How to collect Amberfin logs

Dalet CubeNG Logs

Cube Controller: C:\ProgramData\Dalet\CubeNG\CubeController\Logs

Cube DB: C:\ProgramData\Dalet\CubeNG\dlt-cube-database-server

III: C:\ProgramData\Dalet\CubeNG\dlt-cube-iii

MOS: C:\ProgramData\Dalet\CubeNG\dlt-cube-mos

WebAPI: C:\ProgramData\Dalet\CubeNG\dlt-cube-web-api

Media Gateway: C:\ProgramData\Dalet\CubeNG\MediaGateway\Logs

CubeNG Post Installation Log: C:\Program Files (x86)\Dalet\Cube

Logs need to be collected/compressed manually.

Dalet Legacy Cube (updated to 5.3) Logs

Playout Engine: C:\Users\Public\Dalet\Cube\EngineHw\Logs

Render Engine: C:\Users\Public\Dalet\Cube\EngineDTop\Logs

Cube Designer: C:\Users\Public\Dalet\Cube\Designer\Logs

Cube Controller: C:\Users\Public\Dalet\Cube\CubePlayController\log

Cube Multiplexer: C:\Users\Public\Dalet\Cube\CubePlayMultiplexer\log

Brio Control Panel 4: C:\Users\Public\Dalet\BrioControlPanel4\Logs

Cube Studio (Legacy): C:\Users\Public\Dalet\Cube\Cube Studio\Log

You can employ the Cube Log Collector to automatically collect all logs. Select the required logs, set the folder in which the zipped package will saved. 

Which Logs do I Need to Provide?

Generally speaking, it is always better to provide more logs than not enough. When you provide some logs, please ALWAYS mention the date and time of the issue, with the most possible precision. In the next sections, we will describe which logs to provide depending on the issue.

In Case of Dalet Galaxy General Site Slowness

Please provide:
-    Logs of all DB Servers,
-    Logs of all Notifier Servers,
-    Logs of all DaletPlus Servers,
-    Logs of Galaxy client for some of the machines that experience slowness,
-    Dalet SQL trace zipped (see below),
-    General Diagnostics as a text file (see below),
-    Exact date and time of the issue.

How to Collect the Dalet SQL Trace

To collect Dalet SQL trace, please follow these steps:
1)    Connect to your SQL host.
2)    Launch MSSQL management studio.
3)    Run the following query:

select * from sys.traces

The trace with id = 2 is usually Dalet SQL trace.

 
Location of Dalet SQL trace

4)    Browse to the displayed location with file explorer and zip the folder.

Dalet SQL trace folder 

How to Run MSSQL General Diagnostics?

To collect a DB General Diagnostics, please follow these steps:
1)    Connect to your SQL host.
2)    Launch MSSQL management studio.
3)    Select output to text in MSSQL management studio.
4)    Select Dalet DB as a database to run the query on.
5)    Run the following query: exec pr_generalDiagnostics.

Running the General Diagnostics procedure on Dalet DB

6)    Copy your results to a text file.

As this data is overwritten very quickly, it should be grabbed during the slowness / very quickly after.

Increasing/Changing the Log Level

Logs have a level that determines the depth of its writing. Each agent for each solution can have a different log level, thus you have to communicate with the relevant Dalet representative before changing the log level.

Do not forget to change bacl the log level after an issue is solved. Deeper log levels can significantly impact performance.

To change log levels for Dalet Galaxy, consult DTK Flags/XML for Controlling Log file Creation   or Common DTK Flags (or search the WIKI for "DTK").

To change log levels for Dalet Amberfin, consult Dalet Amberfin Troubleshooting.

To change log levels for Brio, select  Debug in the Log Level drop down list for the relevant channel. For more information, consult the Brio Administration document attached to Dalet Brio 3 Manuals.

Logs for Video Playout Issues

Dalet Galaxy Side

Please provide:
-    Logs of all DB Servers,
-    Logs of all Notifier Servers,
-    Logs of all Broadcast Servers,
-    Logs of all Media Agents relevant for the issue (Brio Media Agents, Omneon Media Agent, etc.),
-    Logs of Galaxy client of the machine(s) that experienced the issue,
-    Exact date and time of the issue,
-    Title id and itemcode of the problematic title(s) / media(s),
-    If possible, please backup a copy of the broadcasted media.

Dalet Brio Side

Please provide:
-    All Brio logs (see section 3.5),
-    Exact date and time of the issue.

Video Ingest Issues

Dalet Galaxy Side

Please provide:
-    Logs of all DB Servers,
-    Logs of all Notifier Servers,
-    Logs of all Ingest Servers,
-    Logs of all Media Agents relevant for the issue (Brio Media Agents, Omneon Media Agent, etc.),
-    Logs of Galaxy client of the machine(s) that experienced the issue,
-    Exact date and time of the issue,
-    Title id and itemcode of the problematic title(s) / media(s),
-    If possible, please backup a copy of the ingested media (if created).

Dalet Brio Side

Please provide:
-    All Brio logs (see section 3.5),
-    Exact date and time of the issue.

Dalet Galaxy Media Job-related Issues

Please provide:
-    Logs of Media Migration Server,
-    Logs of Jobs Broker for migration,
-    Logs of the agent responsible for the job (see below),
-    Logs of Dalet Media Thunder called by the agent (see below),
-    Exact date and time of the issue,
-    Job properties of the relevant job(s) copied in a text file:
 

Right click on the relevant job and select “Job Properties”

 
Copy the “Job XML” part in a text file and save

-    If possible, please backup a copy of the source media and the target media if it was created.

How do I Know which Agent is Responsible for the Job

Agents responsible for jobs are usually Media Transcoder Agents (a.k.a MTA), but depending on the task, it can be a SNSM Essence Mover, an AmazonS3 Essence Mover, an Amberfin Essence Mover, etc. After a job has been created, it is assigned to one (and only one) agent. You can know which agent has been assigned by opening Job Monitor in a Galaxy client or Webspace.

Dalet Media Thunder and its Logs

To perform some media-related operations (copy, archive, restore, etc.), the Dalet agent calls the media engine embedded in Galaxy. It is called Dalet Media Thunder. This engine logs its operations in the same log folder.
One log file is created each time the engine is invoked. Therefore, Dalet Media Thunder logs look like the following:

Dalet Media Thunder logs

Please provide all Dalet Media Thunder logs invoked by the agent responsible for the job.

Dalet OneCut Issues

Dalet OneCut has a mechanism to collect and zip all needed data. Please press CTRL + SHIFT + ALT + I during an issue or immediately after. Please provide a name and a short description for the issue. This will create an incident report containing all needed data:

Incident report for OneCut issues

Dalet Galaxy BPM-related Issues

Please provide:

-    General logs for the WorkflowServer. These are written in the DaletLogs folder. See the Where are Dalet logs located? section above to find this folder for your Dalet version.
-    Log for workflow instance. These logs are written in subfolders with the name of the workflow type inside a subfolder Workflows in the WorkflowServer logs folder. These logs only contain the relevant information for each individual instance of the workflow.

-    For notification/events: Logs of DaletMessageQueue and NotifierJMS (version 4.0.287 and up)
-    For GUI issues: Browser (Chrome, Firefox etc.) console logs retrieved at the time of occurrence
-    For issues with a specific workflow type: The workflow definition as exported from the BPM designer

Dalet Galaxy MOS-related Issues

Please provide:
-    Logs of all MOS Servers relevant for the issue,
-    Exact date and time of the issue,
-    Title id and itemcode of the problematic title(s) / media(s),
-    Logs of 3rd-party systems Dalet is exchanging MOS messages with.

Dalet Galaxy Purge-related Issues

Please provide:
-    Logs of all DaletPlus Servers,
-    Exact date and time of the issue,
-    Title id and itemcode of some problematic title(s) / media(s).

Dalet Amberfin-related Issues

Please provide:
-    Logs of all Amberfin Essence Movers (if any),
-    Logs of all Amberfin services (see section 3.6)
-    Exact date and time of the issue,
-    Title id and itemcode of some problematic title(s) / media(s),
-    If possible, please backup a copy of the ingested media (if created).

Other Issues

If you have an issue that is not described above, Dalet support will let you know which logs you need to send. In any cases, you will need to provide:
-    A short description of your issue
-    Exact date and time of the issue
-    Screenshots describing the situation or showing the error, if possible.

How to Backup Dalet Galaxy Logs

Dalet logs are being overwritten very quickly. Some agents (like DB Servers, Notifier Servers, Media Migration Server, etc.) are overwritten in less than 10 minutes. For this reason, it is practically impossible to collect all relevant logs manually after the incident happened. Moreover, the stress level associated with an outage makes it difficult for a system administrator to collect all relevant logs. Without them, Dalet support department would not be able to fully investigate about the root cause of the issue.
To improve this behavior, we believe that log collection should be planned before an incident happens.
3)    Run the batch “Run.bat” or schedule it as a Windows task. A folder is created where specified in the batch, with the specified name.
Please follow these recommendations when using LogsToZip:
-    it is better to distribute the load, and not to gather all the log files in the same archive,
-    as agents logs their operations at different speeds, we recommend to create different scenario for log collection, with different scheduling
-    some conflict might occur between log writing process and its collection, please check the zip file before sending it.

Cheat Sheet – Table of Logs