Security Bulletin on CVE-2021-44228 Log4Shell and Remediation Instructions

This message contains technical details about a new Log4J vulnerability in some of our products.If you are not affected, please kindly ignore.

Scope of this Document

This article contains technical details about a new vulnerability in a specific Log4J version in some of Dalet's products. If you are not affected, please kindly ignore.
Please note that due to the nature of the challenge, including new vulnerability on the version log4j 2.17 (CVE-2021-45105), the content of this article dynamically changes and you are strongly advised to consult with a Dalet representative. All solutions, text and code, reflect the state and fixes in relation to the latest CVEs listed in this document.

Note: All service packs and manual fixes published after December 26 already include the safe log4j.

Vulnerability of Dalet Software

Our teams have been involved all day for analysis and resolution.

At this stage, we see:

• Flex: Partially affected. Flex Java services and apps use SLF4J with logback as the logging implementation, not log4j2, and therefore, as described at https://spring.io/blog/2021/12/10/log4j2-vulnerability-and-spring-boot, are not affected. However, among the third-party services we deploy are two which are to different extents exposed to this vulnerability.
  • Elasticsearch (deployed in two clusters – one to support search capabilities in Flex, and as part of our monitoring ELK stack)
  • Logstash (deployed as part of our monitoring ELK stack).
    (see section Remediation instructions for Flex 3rd Party Services  below)
• Cube/CubeNG: NOT affected
• Brio: NOT affected. If the Brio machine hosts Dalet Galaxy application servers, you need to act as outlined below.
• Amberfin: For  3rd party dependencies, please read the section on Remediation Instructions for Amberfin 3rd Party Dependencies. AmberFin V11.9.4.0.MR Release - Build 388615 and up fixed the vulnerability in the code and no further action is necessary
• Galaxy: Affected
• Galaxy 3.5 and DaletPlus 1.4: NOT affected
• Galaxy older than 4.0.271: NOT affected
• Dalet internal tools (Homepage, internal dbs): NOT affected
• WOWZA (when used with Dalet solutions): Affected
• ICINGA (when used with Dalet solutions): Affected
• DIS 1.5 & DIS 2: fix available on FTP

Mitigating factor: systems not exposed to the internet face the lowest risk of attack.

Confidentiality Notice & Software Updates Use Policy

All Updates and Bug Fixes shall constitute Software as defined herein and in the License Agreement. You agree to not: (a) disassemble, reverse engineer, decompile or otherwise attempt to derive any Software source code from object code, except to the extent expressly permitted by applicable law despite this limitation (b) distribute or provide the Software to any third party, provide a third party with the results of any functional evaluation, benchmarking or performance tests, without Dalet’s prior written approval; (d) attempt to disable or circumvent any of the licensing mechanisms within the Software, if any; (e) prepare any derivative work of the Software or remove any product identification, copyright, trademark or other notice from the Software; or (f) violate any other usage restrictions contained in the Software installation instructions or release notes. Any third party software provided with Software may be used only with that Software and only in accordance with Section 2 of this Agreement. The Software must be used solely for the purposes and in the manner described in the Documentation.

For full details, please consult: https://www.dalet.com/legal/

Remediation Instructions for Dalet Galaxy Versions 4.0.271  and up Server and Client Side

Customers that have applied an earlier suggested solution, do exactly the same as those customers who follow the here outlined procedure for the first time.

We have a remediation procedure that has been validated in R&D for all versions listed below, with the following notes:

1. There is no risk for the Dalet Agent to stop working after the fix.
2. This can be done during office hours for some of the agents which have redundancy or is OK to be stopped for 2-3 min. For others, like MediaMigration Server and JobBroker, it is better to do it in low activity time.
3. The agents are all JAVA based Dalet application server, but DaletService and thus all servers will be stopped and eventually restarted.

Fixing the Active Installation with Automatic Stop, Fix, Restart

Download the attached patch file "Log4j1.3Fix". Securely delete this file after use!

Unzip into same category, run CMD with administrative rights and cd to the extracted path, type and run "patch_active_version_site_restart.bat" on all server and client computers.

Fixing All Inactive Installation with Automatic fix, norestart

Download the attached patch file "Log4j1.3Fix". Securely delete this file after use!

Unzip into same category, run CMD with administrative rights and cd to the extracted path, run "pacth_backup_versions_no_site_restart" plus DaletPlus<ACTIVE revision number>, on all server and client computers. This will fix all found DaletPlus folders that are NOT of the active revision, that is, all inactive ones.

Note: the original filename has a spelling mistake.

Now all revision folders BUT 386710 will be fixed.

Remediation Instructions for Icinga in Dalet Galaxy

The command has to be executed on the Icinga host. It will update a running instance of the Icinga container.

• at the beginning it prints out the command line of the icinga service which does not contain -Dlog4j2.formatMsgNoLookups=true
• at the end it prints out new command line of the icinga service, which should contain -Dlog4j2.formatMsgNoLookups=true 

The two points above are the indication that the script worked well.

It is made verbose on purpose, if something is going wrong, it will be easy to check for the reason.

In addition, Dalet is preparing the regular patches of the Icinga service, for all the affected Galaxy branches.

The patches will be published by the release managers and can be installed on the regular way

Remediation Instructions for OnTheGo (Mobile Access Service)

For all three available versions of the Mobile Access Service (which implements the OnTheGo mobile application on the server side), a new version which eliminates the vulnerability has been compiled.
Please deploy the appropriate version - they will be available on FTP shortly.

In these new versions, the latest version of log4j is installed - which does not suffer from the vulnerability (2.15.0).

Three versions of MobileApplicationsServer have been patched to remove vulnerabilities. Fixes are available now and should be deployed immediately for all customers using OnTheGo. The fix is on the server side only.

All customers using MobileAppsServer should upgrade to the latest version as described above (According to their respective site and existing MAS version). To identify which version is compatible with which site, consult the Wiki article "Dalet On-the-Go Installation"

Remediation instructions for DIS 1.5 & DIS 2

The new setups with log4j-XXX-2.17.0 were uploaded to S3 storage. 

You must completely uninstall the old version of Dis before installing the new setups above in the following manner:
 

1.     Identify where DIS is installed. 

-      all sites should search for the follow path on each server to determine whether DIS is/was installed… C:\ProgramFiles(x86)\Dalet\DaletInstallerSiteAdmin

2.     Under Main Configuration > Host Management, right click on the right column list to Export the configuration – which creates an XML file.

- Save externally, but DELETE after the new DIS is installed and working normally.

- It contains stored credentials (hashed), but should be deleted after the installation.

3.     Uninstall DIS via Windows Add/Remove Programs

4.     Download DIS from the links shown below

5.     Install the new DIS  only on ONE server

6.     Launch the new executable (DaletInstallerSiteAdmin.exe)

7.     Import the configuration (Main Configuration > Host Management) – if the “original” C:\ProgramData\Dalet\DaletInstallerSiteAdmin folder was deleted (Step 3).

8.     Under Operations & Status > Main Control Panel > DIS Operations button header, update the DIS service on all "monitored" servers. 

- Note: The DIS server will display a warning on all "monitored" servers.

- Note: UpdateDIS on 2 or 3 servers at a time to avoid impacting system operations.
- Note: UpdateDIS may need to be performed more than once.

- Note: This action updates the DaletInstallerService (Windows Service) and the content of C:\ProgramFiles\DaletInstaller – which will now contain the new Log4J-XXX-2.17.0.

DIS 1.5 https://s3.amazonaws.com/dalet.deliveries/DaletInstallerReleases/387935/DaletInstallerSiteAdmin.exe
DIS 2    https://s3.amazonaws.com/dalet.deliveries/DaletInstaller2Releases/387942/DisMQ.exe

Remediation Instructions for Flex 3rd Party Services

Among the third-party services we deploy are two which are to different extents exposed to this vulnerability.
1.    Elasticsearch (deployed in two clusters – one to support search capabilities in Flex, and as part of our monitoring ELK stack)
2.    Logstash (deployed as part of our monitoring ELK stack).
The documentation at https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476 explains more about the log4shell vulnerability in the context of these two services.
3.    Hazelcast (deployed as part of our core stack, only from Flex release 2021.3.0 onwards).

Long-term solution

1.    All releases of Flex built after 13th December 2021 will include relevant fixes to the core Elasticsearch and Hazelcast clusters by default.  We will be releasing a new patch version for each LTS release to include this fix, so that as and when any customers are upgraded, they get (or rather keep) the fix.
2.    All deployments / upgrades of the monitoring stack, which are performed after 13th December 2021, will include the fix to the monitoring instances of both Elasticsearch and Logstash.

Short-term solution

We appreciate that, due to the potential for disruption of their core business, customers will probably not wish to immediately:
1.    perform a redeployment of Elasticsearch, or a patch upgrade of Flex
2.    upgrade their monitoring stack.

As a result, the rest of this document explains how to mitigate the log4shell vulnerability as quickly and non-disruptively as possible.

ADDENDUM following the latest reported DoS vulnerability described by #CVE-2021-45105:

• As per https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476, Elasticsearch and Logstash "are not exploitable by this vulnerability" (due, I believe, to the way logging is configured, which I specifically looked into and confirmed).
• Regarding Hazelcast, our logging configuration has never included any context lookups, so no additional mitigation is required to address the new CVE reports.

It is possible that some security scans may raise false-positives where we one of these third-party dependencies still includes a version of Log4J2 < 2.17.0, with the JndiLookup class removed as per below, because of CVE-2021-45105.  As there is no vulnerability inherent here, Dalet does not intend to make any changes solely to remediate such false-positives.

On all monitoring servers
As monitoring is not mission critical to customers, an interruption to being able to access logs is acceptable, so these steps can if necessary be run in parallel on all servers.

Using ansible:

We have prepared a set of tasks in the ansible-elk role, tagged with log4shell, which can be used to run the monitoring server fixes automatically.

1.    Ensure the ansible-elk role is installed and up-to-date with the very latest code (or using tag 2.8.1 or later)
2.    Run ansible-playbook --tags=log4shell, providing your hosts file inventory and the environment playbook itself as normal.

Manually:

If you prefer to apply the fixes manually, instead of using the  Ansible role, these are the necessary steps.

Fix logstash

Fix Elasticsearch

Internal Elasticsearch and Hazelcast clusters

It has been necessary to release patched versions of our released Docker images for elasticsearch (as far back as Flex release 7.3.0) and hazelcast (as far back as 2021.3.0).

We updated all affected existing release manifests to use these new versions, in all such cases appending a .2 to the version number (making it a 4-part version instead of 3-part).
The delta between the new image and the old for a given release is only that the log4j-core jar file has had the dangerous JndiLookup.class removed.
Please note that, contrary to the usual process for applying fixes, we have not issued any new Flex releases for this – we have updated the existing release manifests.

Versions may be updated manually (by editing the docker-compose.yml file on each Services host to include the .1 suffix to the image versions, and running targeted docker-compose up -d commands), but we recommend using the standard deployment processes, i.e:

• Ensure you have pulled the very latest commit in the flex-releases repo
• Execute the deployment Ansible playbook, using the same Flex release version that’s currently installed.

To verify the fix has been successfully applied:

• you should check that the versions of all instances of elasticsearch and hazelcast that are running are a four-part version, ending in .2 – for example, version 2.1.16.2 has the fix, while version 2.1.16 does not.  (*.*.*.1 contains the essential fixes; version *.*.*.2 remediates some false positives that were present in *.*.*.1).
• if you want to go deeper, then the commands below can be run on in elasticsearch and hazelcast containers, respectively, and in each case should show no results.
  • unzip -l /usr/share/elasticsearch/lib/log4j-core-*.jar | grep JndiLookup
  • unzip -l /opt/hazelcast/lib/log4j-core-*.jar | grep JndiLookup

Original solution - deprecated

The Apache page https://logging.apache.org/log4j/2.x/security.html#CVE-2021-44228 was updated to say that the environment variable fix we originally applied was insufficient.

Remediation Instructions for Amberfin 3rd Party Dependencies

NOTE  AmberFin V11.9.4.0.MR Release - Build 388615 and up; and for earlier versions Amberfin 11.8.10.13.MR​ fixed the vulnerability in the code and no further action is necessary. All previous versions have to follow the steps below.

Amberfin 11.9.6 to 11.9.3

In response to the Log4j v2 Remote Code Execution vulnerability https://www.ncsc.gov.uk/information/log4j-vulnerability-what-everyone-needs-to-know 
AmberFin uses an earlier version of log4j (v1) which is not vulnerable to this exploit but versions of AmberFin do ship log4j v2 jars (log4j-core) which are in the affected range of versions (2.7.x - 2.14.x) which are included in third party dependencies. 
AmberFin is upgrading all logging in the product to log4j v2.17.0 in all future releases.
Customers can patch current versions of AmberFin using the LogPatcher tool which will remove the jndilookup class from the installation completely making it impossible to run the exploit. 
 

Download the attached patch file "LogPatcherUtility-V2.0.zip". Securely delete after use!

Running Log Patcher - Steps

1. Extract or Unzip the LogPatcher.zip file anywhere on the AmberFin installed server by right-clicking on the  LogPatcherUtility-V2.0.zip file and clicking ‘Extract All’. 

 
2. Open the installed folder in Windows Explorer (it may open automatically after extracting). 

 
3. Right click on LogPatcher.cmd and select ‘Run as Administrator’ from the menu then click ‘Yes’ in the User Account Control dialog to allow the command to run. 

4. A Command Window will open up to report the patching process. 

 
5. Repeat this process for all AmberFin installed servers. 

Amberfin 11.9.5 and older

In response to the Log4j (v1.2) JMS Appender vulnerability described here: https://nvd.nist.gov/vuln/detail/CVE-2021-4104

AmberFin customers using an earlier version than 11.9.6 can use this patcher to remove vulnerable classes from the installation 

Download Link: 

https://ftp.dalet.com/Products/AmberFin/Deliveries/LogPatcherUtility/CVE-2021-4104Patcher.zip

Running the Patcher - Steps
1. Extract or Unzip the LogPatcher.zip file anywhere on the AmberFin installed server by right-clicking on the cve-2021-4104-Patcher.zip file and clicking ‘Extract All’. 

2. Open the installed folder in Windows Explorer (it may open automatically after extracting). 

 
3. Right click on patcher.cmd and select ‘Run as Administrator’ from the menu then click ‘Yes’ in the User Account Control dialog to allow the command to run. 

 
4. A Command Window will open up to report the patching process. 

5. Repeat this process for all AmberFin installed servers.

Remediation Instructions for Wowza

The following was provided by Wowza:

What you need to do: 
Customers on Wowza Streaming Engine 4.8.5.05 and below are not impacted by this vulnerability. 
For all customers on Wowza Streaming Engine 4.8.8.01 and above, please take the steps outlined in this article to apply a fix. 

Remediation for Tableau

https://kb.tableau.com/articles/issue/apache-log4j2-vulnerability-log4shell-tableau-server-mitigation-steps

Additions and Security Notes

Log4j 2.x version

Apache fixed several security vulnerabilities and Log4j2 has to be updated in the source code to 2.17.0 version in order to get the fixes. Refer to page about fixed issues https://logging.apache.org/log4j/2.x/security.html

However, there are some Java agents that use a log4j version less than 2.17.0 and vulnerability checking tools may complain on used Log4j version. The explanation why usage of such versions is safe is below:

• SolrIndexServer75 uses modified 2.11.0 version of Log4j2 from externals (svn://gfn-svn:3696/branches/builds/4.0/tools/Solr/solr-7.5.0/server/lib/ext). There is log4j-core-2.11.0.jar with removed JNDILookup class as recommended by Apache, thus, there is no the vulnerability.

• Vulnerability checking tools may complain even on 2.17.0 version due to CVE-2021-44832 fixed in 2.17.1. However, the fixed vulnerability is relevant only if JDBCAppender is used in logger configuration and Galaxy doesn't use JDBCAppender. See a remark to rejected bug #156602 - DZ-Log4j vulnerability in version 2.17.

Log4j 1.x version

Several Java agents use log4j 1.x. The note about the version is that we don't use JNDI in configuration and there is no JMSAppender configured, so it is free from vulnerabilities:
CVE-2021-45105: Log4j 1.x is not impacted by this vulnerability.

CVE-2021-45046: Log4j 1.x is not impacted by this vulnerability.

CVE-2021-44228: Log4j 1.x does not have Lookups so the risk is lower. Applications using Log4j 1.x are only vulnerable to this attack when they use JNDI in their configuration. A separate CVE (CVE-2021-4104) has been filed for this vulnerability. To mitigate: Audit your logging configuration to ensure it has no JMSAppender configured. Log4j 1.x configurations without JMSAppender are not impacted by this vulnerability.