Apache Log4j CVE-2021-44228 vulnerability
Incident Report for Instana
Resolved
This incident has been resolved.
Posted Mar 02, 2022 - 22:19 UTC
Monitoring
Thanks for your patience! We have following updates to share:

## Instana Back-End

Instana SaaS - All back-end components are updated
Instana OnPrem:
For version 203 and higher we have published new ElasticSearch 7 images:

For self-hosted single-host installations
instana images pull
instana update

For self-hosted datastore setups used with self-hosted on k8s
instana datastores images pull
instana datastores update

For version 201 and lower: Please update to a supported Instana version as per our version support policy: https://www.ibm.com/docs/en/owi/215?topic=policies#version-support-policy

Please also refer to https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476 for more details.

## Instana Agent update

Dynamic agents (auto-update)
Dynamic Agents that update to 1.1.618 will patch the logging config by adding {nolookups} to the log pattern.
This version might still incorrectly report the Instana Agent as being affected by your security scans!

The latest Instana Agent installation now include a fix avoiding the Instana Agent to be incorrectly reported by your security scanner. This approach requires a re-installation or re-deployment of the Instana Agent!

The respective versions are with the fix are:
- DEB & RPM packages: versions with a timestamp 2021-12-15 18:50 or later
- Windows installer (JVM included): versions with a timestamp 2021-12-16 10:10 or later
- Windows ZIP (JVM excluded): versions with a timestamp 2021-12-15 21:10 or later
- Dynamic agent image: 1.215.71 & 1.215.71-j9, latest, latest-j9 or later
- Static agent image: 1.215.71 & 1.215.71-j9, latest, latest-j9 or later

Please follow Agent installation steps to (re-)install the latest Agent version as described here: https://www.instana.com/docs/setup_and_manage/host_agent/on

For PCF, GCP marketplace and RHEL sources, currently a release process has been triggered.

There are no new releases of the Instana Operator or HELM chart as the current version would retrieve the latest Instana Agent docker version on redeployment

Important:
older agent & sensor bundles cannot be pinned to those latest agent runtimes anymore, only newer SHAs will be usable, this cannot be addressed technically

## Next steps:

Instana is working on a "monitoring issue" for SaaS customers showing to users that their agent is not yet on a minimal version with the relevant fix avoiding the Instana Agent is incorrectly reported by security scans. This won't catch agents manually patches between Monday, 13th of Dec 2021 and Thursday, 16th of Dec 2021.
Posted Dec 16, 2021 - 13:06 UTC
Update
Please refer to official IBM updates for more news on the CVE: https://www.ibm.com/blogs/psirt/

Instana is working on revised Instana Agent packages which will be published soon including mechanisms for auto-updating it, also overcoming the latest and recent announcements. Finally it will assure security scans are no longer incorrectly reporting the Instana Agent as being affected.
Posted Dec 15, 2021 - 18:51 UTC
Identified
Instana Agent - Log4j CVE-2021-44228 update
Due to a vulnerability discovered in Apache Log4j (CVE-2021-44228) we have provided an update to our agent component. It is based on the Pax Logging library as used by our agent packages.
We recommend either one of the following:


Instana Agent update

Update our Instana Agent installation to the latest available version.
Latest available agent packages and agent docker images ship with an update for the current Log4j CVE.
The respective versions are:
- DEB & RPM packages: versions with a timestamp 2021-12-11 20:58 or later
- Windows installer (JVM included): versions with a timestamp 2021-12-13 09:29 or later
- Windows ZIP (JVM excluded): versions with a timestamp 2021-12-11 20:33 or later
- Dynamic agent image: 1.215.24 & 1.215.24-j9, latest, latest-j9 or later
- Static agent image: 1.215.24 & 1.215.24-j9, latest, latest-j9 or later
Please follow Agent installation steps to (re-)install the latest Agent version as described here: https://www.instana.com/docs/setup_and_manage/host_agent/on

From your tenant unit's "Installing Instana Agents" wizard, any downloadable package contains the update, too.

Set log4j2.formatMsgNoLookups System Property

If you can not update the Instana agent you need to set the system property `-Dlog4j2.formatMsgNoLookups=true` and restart the agent.
The system property needs to be set either in an external environment variable `JAVA_OPTS` or inside `/bin/setenv` on Linux hosts or `/bin/setenv.bat` on Windows hosts.
Don't forget to restart the Agent after adding the property.


Dynamic Agent Autoupdate

The Instana Agent will *not* automatically update itself with regards to this CVE. It is required to re-install the latest available runtime or set the aforementioned system property and restart the agent.


Security Scans

Our updated agent packages and images come with a patched version of the affected log4j library. Current security scans will still report our agent to be affected by Log4j CVE-2021-44228.


AWS Fargate, AWS Lambda & Google Cloud Run collector

The standalone collector for AWS Fargate, AWS Lambda, and Google Cloud Run is not affected by the Log4j CVE.

For any issues with the above please do contact Support. We are here to help!
Posted Dec 14, 2021 - 11:59 UTC
Investigating
We are currently looking into this issue. Please refer to the following article for an official statement. Updates will follow. https://www.ibm.com/blogs/psirt/an-update-on-the-apache-log4j-cve-2021-44228-vulnerability/
Posted Dec 13, 2021 - 09:42 UTC
This incident affected: Agents and Sensors.