Elasticsearch dynamic scripting vulnerability exploit

Introduction

Update 20140716

“This could allow an attacker to execute OS commands.”. That is the notice on the security page of Elasticsearch.

A vulnerability that allows execution of system commands should always raise concern.

Some people running a public Elasticsearch instance reported cases where attackers were able to upload scripts. It turned out that when Elasticsearch was available on the Internet (port tcp/9200) and had dynamic scripting enabled then users could execute arbitrary scripts. If Elasticsearch is running as root this means remote root execution of arbitrary scripts.

The solution to be protected from this vulnerability is

  • upgrade to a newer version of Elasticsearch (which might require a newer version of Java);
  • firewall tcp/9200;
  • disable dynamic scripting

Scripting is described in detail on the scripting module documentation page. In essence you have to make sure that your Elasticsearch configuration file config/elasticsearch.yml does NOT contain a setting that enables dynamic scripting

# Enable dynamic scripting; do not use on Internet hosts
script.disable_dynamic: false

Note that Elasticsearch < 1.2.0 has Dynamic Scripting enabled by default.

Artifacts

Thanks to a co-operative hoster, I received a copy of the files that were left behind on one of their servers. Below is the transcript of the analysis of these files.

They discovered five files : fake.cfg, linux64, linux64.1 (similar to linux64), launch.sh and zero.pl

fake.cfg

This text file contained the IP (anonimized with x.x.x.x below) of the victim. Based on the filename I would presume this is some sort of configuration file (but I could not figure out what it had to configure).

0
x.x.x.x:x.x.x.x
10000:60000

launch.sh

MD5 (launch.sh) = 60c0f4f381d6ac78fb5c78a76af0c927

This was a simple bash script that downloads files that are presumably Bitcoin miner files. The commands in the bash script were

  1. kill the existing mining processes : “VER=`getconf LONG_BIT`”
  2. get the current architecture (getconf LONG_BIT)
  3. download (wget) the miner file and a config file from a (probably hacked) WordPress site (pattern: cache/minerd$VER with $VER the architecture, 64/32)
  4. make the downloaded files executable
  5. set the number of threads : “cpu_c=`grep processor /proc/cpuinfo | wc -l`”
  6. start the miner file (with different threads depending on the CPU) : “nohup ./min –threads $threads -c config.conf -B 0<&- &>/dev/null”
  7. remove the config file and log files : “rm -f /tmp/min /tmp/config.conf /tmp/launch.sh /var/log/jenkins/* /tmp/info.sh”
  8. empty the history : “echo “” > ~/.bash_history”

A posting on the Zimbra forums showed that a similar (based on name and options) tool was used to abuse vulnerable Zimbra servers. There is an even more detailed description on the blog infosanity.co.uk.

Note that the script does a killall of both min and min2 but if you go through the script code of launch.sh there’s nothing that generates a min2. So either it is a programming error, min ‘somehow’ forks to min2 or there’s another script that downloads min2.

zero.pl

MD5 (zero.pl) = c3fd28831d9fc683000273643b5bef12

A perl script that receives two parameters, remote_ip and remote_port, that then starts an interactive bash shell (my $shell = ‘/bin/bash -i’; )

linux64

MD5 (linux64) = a4eecf76f4c90fb8065800d4cad391df

Virustotal

This is a Linux binary. The file is recognised by Virustotal and tagged by Avast as “ELF:Flooder-AI [Trj]”. It was first uploaded to Virustotal on 2014-06-10 13:10:32 UTC (and back then with the filename file-7103693_).

File details

I used the commands file, ldd, objdump and strings to get some characteristics of the file

‘file’

linux64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

‘ldd’

ldd linux64
	linux-vdso.so.1 =>  (0x00007fff28b1c000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f73fb51f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f73fb18b000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f73fb741000)

ldd -u linux64
Unused direct dependencies:
	/lib64/libpthread.so.0

‘objdump’

objdump -p linux64

linux64:     file format elf64-x86-64
...
Dynamic Section:
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  INIT                 0x00000000004014e8
  FINI                 0x00000000004059e8
  GNU_HASH             0x0000000000400240
  STRTAB               0x00000000004009f8
  SYMTAB               0x0000000000400260
  STRSZ                0x00000000000002a4
  SYMENT               0x0000000000000018
  DEBUG                0x0000000000000000
  PLTGOT               0x00000000006071d0
  PLTRELSZ             0x0000000000000750
  PLTREL               0x0000000000000007
  JMPREL               0x0000000000400d98
  RELA                 0x0000000000400d80
  RELASZ               0x0000000000000018
  RELAENT              0x0000000000000018
  VERNEED              0x0000000000400d40
  VERNEEDNUM           0x0000000000000002
  VERSYM               0x0000000000400c9c

Version References:
  required from libpthread.so.0:
    0x09691a75 0x00 03 GLIBC_2.2.5
  required from libc.so.6:
    0x09691a75 0x00 02 GLIBC_2.2.5

strings

strings linux64
  fork
  connect
  send

  wget -P /home %s 2>&1

  linuxssh.no-ip.org

The strings command reveals that, once executed, the binary tries to fork, connect (not that un-usual) and send “something”, probably to the host at linuxssh.no-ip.org.
I also used strings -a

strings -a linux64
  GCC: (GNU) 4.1.2 20080704 (Red Hat 4.1.2-48)

%d.%d.%d.%d
/dev/urandom
/dev/random
killall %s
wget -P /home %s 2>&1
saved
chmod +x %s
dosset.dtdb
fail to open file:
%s%s
/proc/cpuinfo
model name
/proc/stat
/proc/meminfo
/proc/net/dev
/etc/rc.d/rc.local
/proc/self/exe
/var/run/dos32.pid

Running strings with other encodings (strings -e l ; strings -e b ; strings -e L ; strings -e B) did not return useful results.

The string ‘/var/run/dos32.pid’ got my attention. A Google search links to an analysis on malwr.com. That analysis also contains the full strings output.

Based on the last output I suspect that if I wanted to run this binary in a VM I would have to use a Red Hat flavour.

CentOS

One of the more popular free flavours of Red Hat is CentOS. I did a minimal install of CentOS release 6.

CentOS release 6.5 (Final)
2.6.32-431.el6.x86_64 #1 SMP

In order to do a basic analysis I installed these extra packages

yum install wget
yum install tcpdump
yum install screen
yum install python
yum install nc

I downloaded dnschef to capture and change the dns requests.

When these packages were installed, I disabled the VM network interface, stopped dhclient and added a default route to the local interface.

pkill dhclient
route add default gw 127.0.0.1

I also set the nameserver in resolv.conf to 127.0.0.1. This way all the DNS requests are send to dnschef.

nameserver 127.0.0.1

The next step is to start a couple of tools to capture the traffic. All these tools were started within the screen command.

I first started a netcat listener that mimics a default Elasticsearch web response.

while true; do cat elasticsearch.response | nc -l 9200  >> /root/es_requests ; done

I also started a netcat listener on port 9090 (I used this port because I noticed network traffic to this port once the binary started) with similar options.

while true; do cat elasticsearch.response | nc -l 9090  >> /root/es_requests ; done

The file elasticsearch.response contains

200 OK
Access-Control-Allow-Origin:  *
Content-Type:  application/json; charset=UTF-8
Content-Length:  298
{
  "status" : 200,
  "name" : "Starshine",
  "version" : {
    "number" : "1.1.1",
    "build_hash" : "f1585f096d3f3985e73456debdc1a0745f512bbc",
    "build_timestamp" : "2014-04-16T14:27:12Z",
    "build_snapshot" : false,
    "lucene_version" : "4.7"
  },
  "tagline" : "You Know, for Search"
}

Capturing the network traffic is easy with tcpdump

tcpdump -n -i lo -w /root/dump.pcap

I used dnschef to alter the dns requests.

dnschef.py --fakeip 127.0.0.1 --nameservers=127.0.0.1 -q

Analyse linux64

All the previous steps were needed to get a working environment to capture the behaviour of the binary.

I then started the binary, both from the command line and via strace

./linux64
(strace ./linux64) 2>&1 | tee /root/strace_linux64

Dnschef did not reveal anything new. It showed requests for linuxssh.no-ip.org and 127.0.0.1 was returned as a result (because of the –fakeip option in dnschef).

This resulted in requests going to the netcat listener on port 9090. The logged requests were

UU	Linux2.6.32-431.el6.x86_64?	??

There were no requests logged to port 9200 (the Elasticsearch port). The data sent by the binary is the version of the running kernel.

The tcpdump file also revealed nothing new. After the DNS query a connection to port 9090.

00:10:18.008973 IP 127.0.0.1.34071 > 127.0.0.1.53: 26010+ A? linuxssh.no-ip.org. (36)
00:10:18.009900 IP 127.0.0.1.53 > 127.0.0.1.34071: 26010* 1/0/0 A 127.0.0.1 (52)
00:10:19.011296 IP 127.0.0.1.42081 > 127.0.0.1.9090: Flags [S], seq 10043304, win 32792, options [mss 16396,sackOK,TS val 6727915 ecr 0,nop,wscale 6], length 0
00:10:19.011349 IP 127.0.0.1.9090 > 127.0.0.1.42081: Flags [S.], seq 3709189501, ack 10043305, win 32768, options [mss 16396,sackOK,TS val 6727915 ecr 6727915,nop,wscale 6], length 0
00:10:19.011375 IP 127.0.0.1.42081 > 127.0.0.1.9090: Flags [.], ack 1, win 513, options [nop,nop,TS val 6727915 ecr 6727915], length 0
00:10:19.012133 IP 127.0.0.1.42081 > 127.0.0.1.9090: Flags [P.], seq 1:217, ack 1, win 513, options [nop,nop,TS val 6727915 ecr 6727915], length 216
00:10:19.012152 IP 127.0.0.1.9090 > 127.0.0.1.42081: Flags [.], ack 217, win 529, options [nop,nop,TS val 6727915 ecr 6727915], length 0

The strace file shows nothing unusual. Some useful indicators are below

open("/lib64/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340]\0\0\0\0\0\0"..., 832) = 832
...
readlink("/proc/self/exe", "/tmp/es/linux64", 256) = 15

linuxssh.no-ip.org

The host linuxssh.no-ip.org currently resolves to 127.0.0.1.

Conclusions

I could conclude nothing new that was not already revealed in the initial analysis after looking at the file with strings (except for the use of network port 9090).

The behaviour of the binary is described in detail on the drweb website and marked as a DDOS / amplification trojan. Based on the Virustotal detection rate I knew most of the analysis was already done in the past but it’s always a good occasion to practice the use of some tools.

The described behaviour (sending OS and then waiting for commands) corresponds with the logged connection of sending Linux2.6.32-431.el6.x86_64 to port 9090. This means that linuxssh.no-ip.org:9090 was used as a C&C channel.

To summarize, the intruders broke into the system to install Bitcoin miners and DDoS trojans.

Indicators

Look for

  • queries for linuxssh.no-ip.org;
  • process name file-7103693_;
  • process name linux64;
  • file dosset.dtdb;
  • file /var/run/dos32.pid;
  • file /proc/self/exe;
  • bitcoin miners minderd64 (in /tmp?);
  • bitcoin miners minderd32 (in /tmp?);
  • bitcoin miners min (in /tmp?);
  • bitcoin miners min2 (in /tmp?);
  • bitcoin miner configuration files config.conf;
  • perl remote shell script zero.pl

Update 20140716

Today the host linuxssh.no-ip.org resolves to an IP address in France

Wed Jul 16 23:09:09 CEST 2014
linuxssh.no-ip.org has address 195.154.167.38

This IP belongs to

inetnum:        195.154.128.0 - 195.154.255.255
netname:        FR-ILIAD-ENTREPRISES-CUSTOMERS
descr:          Iliad Entreprises Customers
country:        FR

role:           Iliad Entreprises Admin and Tech Contact
remarks:        Iliad Entreprises is an hosting and services provider
address:        8, rue de la ville l'eveque
address:        75008 Paris
address:        France
phone:          +33 1 73 50 20 00
fax-no:         +33 1 73 50 29 01
abuse-mailbox:  abuse@iliad-entreprises.fr

I decided to restart the VM and restart the binary linux64, with a tcpdump and strace capture.

Run linux64

I first ran linux64 as a non privileged user. In my previous test I ran it as root (because the Elasticsearch instance was also running as root). Running it as non root revealed something interesting.

open("/etc/rc.d/rc.local", O_RDWR|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)

This means it tried to write something to the file /etc/rc.d/rc.local. I changed the write permissions so non privileged users could change it and after running the binary once more I found this in the file

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
/tmp/es/linux64

This means that the binary changes the startup script so that on the next reboot it gets executed again, this time as the root user.

Run linux64 as root

I ran linux64 as root. Sure enough, it (192.168.171.137 is the IP of the VM) started communicating with 195.154.167.38.

22:42:16.748087 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [S], seq 3178137845, win 14600, options [mss 1460,sackOK,TS val 38705829 ecr 0,nop,wscale 6], length 0
	0x0000:  4500 003c 1250 4000 4006 5179 c0a8 ab89  E..<.P@.@.Qy....
	0x0010:  c39a a726 e5f7 2382 bd6e 88f5 0000 0000  ...&..#..n......
	0x0020:  a002 3908 4b34 0000 0204 05b4 0402 080a  ..9.K4..........
	0x0030:  024e 9aa5 0000 0000 0103 0306            .N..........
22:42:16.769747 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [S.], seq 675394072, ack 3178137846, win 64240, options [mss 1460], length 0
	0x0000:  4500 002c 0cb1 0000 8006 5728 c39a a726  E..,......W(...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b218 bd6e 88f6  ....#...(A...n..
	0x0020:  6012 faf0 9bf9 0000 0204 05b4 0000       `.............
22:42:16.769875 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [.], ack 1, win 14600, length 0
	0x0000:  4500 0028 1251 4000 4006 518c c0a8 ab89  E..(.Q@.@.Q.....
	0x0010:  c39a a726 e5f7 2382 bd6e 88f6 2841 b219  ...&..#..n..(A..
	0x0020:  5010 3908 759f 0000                      P.9.u...
22:42:16.770735 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [P.], seq 1:217, ack 1, win 14600, length 216
	0x0000:  4500 0100 1252 4000 4006 50b3 c0a8 ab89  E....R@.@.P.....
	0x0010:  c39a a726 e5f7 2382 bd6e 88f6 2841 b219  ...&..#..n..(A..
	0x0020:  5018 3908 d7e5 0000 5555 0900 4c69 6e75  P.9.....UU..Linu
	0x0030:  7832 2e36 2e33 322d 3433 312e 656c 362e  x2.6.32-431.el6.
	0x0040:  7838 365f 3634 0000 0000 0000 0000 0000  x86_64..........
	0x0050:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0060:  0000 0000 0000 0000 0000 0000 d909 0000  ................
	0x0070:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0080:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0090:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00a0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00b0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00c0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00d0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00e0:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x00f0:  dc03 0000 0000 0000 0018 0a00 0000 0000  ................
22:42:16.772042 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [.], ack 217, win 64240, length 0
	0x0000:  4500 0028 0cb2 0000 8006 572b c39a a726  E..(......W+...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b219 bd6e 89ce  ....#...(A...n..
	0x0020:  5010 faf0 b2de 0000 0000 0000 0000       P.............
22:42:17.774498 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [P.], seq 217:265, ack 1, win 14600, length 48
	0x0000:  4500 0058 1253 4000 4006 515a c0a8 ab89  E..X.S@.@.QZ....
	0x0010:  c39a a726 e5f7 2382 bd6e 89ce 2841 b219  ...&..#..n..(A..
	0x0020:  5018 3908 d73d 0000 df03 0000 0000 0000  P.9..=..........
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0040:  0000 0000 0000 0000 0100 0000 0000 0000  ................
	0x0050:  0100 0000 0000 0000                      ........
22:42:17.774960 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [.], ack 265, win 64240, length 0
	0x0000:  4500 0028 0cb3 0000 8006 572a c39a a726  E..(......W*...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b219 bd6e 89fe  ....#...(A...n..
	0x0020:  5010 faf0 b2ae 0000 0000 0000 0000       P.............
22:42:18.777232 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [P.], seq 265:313, ack 1, win 14600, length 48
	0x0000:  4500 0058 1254 4000 4006 5159 c0a8 ab89  E..X.T@.@.QY....
	0x0010:  c39a a726 e5f7 2382 bd6e 89fe 2841 b219  ...&..#..n..(A..
	0x0020:  5018 3908 d73d 0000 df03 0000 0000 0000  P.9..=..........
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0100 0000 0000 0000                      ........
22:42:18.777678 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [.], ack 313, win 64240, length 0
	0x0000:  4500 0028 0cb4 0000 8006 5729 c39a a726  E..(......W)...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b219 bd6e 8a2e  ....#...(A...n..
	0x0020:  5010 faf0 b27e 0000 0000 0000 0000       P....~........
22:42:19.780179 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [P.], seq 313:361, ack 1, win 14600, length 48
	0x0000:  4500 0058 1255 4000 4006 5158 c0a8 ab89  E..X.U@.@.QX....
	0x0010:  c39a a726 e5f7 2382 bd6e 8a2e 2841 b219  ...&..#..n..(A..
	0x0020:  5018 3908 d73d 0000 df03 0000 0000 0000  P.9..=..........
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0100 0000 0000 0000                      ........
22:42:19.780626 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [.], ack 361, win 64240, length 0
	0x0000:  4500 0028 0cb5 0000 8006 5728 c39a a726  E..(......W(...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b219 bd6e 8a5e  ....#...(A...n.^
	0x0020:  5010 faf0 b24e 0000 0000 0000 0000       P....N........
22:42:20.783767 IP 192.168.171.137.58871 > 195.154.167.38.9090: Flags [P.], seq 361:409, ack 1, win 14600, length 48
	0x0000:  4500 0058 1256 4000 4006 5157 c0a8 ab89  E..X.V@.@.QW....
	0x0010:  c39a a726 e5f7 2382 bd6e 8a5e 2841 b219  ...&..#..n.^(A..
	0x0020:  5018 3908 d73d 0000 df03 0000 0000 0000  P.9..=..........
	0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
	0x0050:  0100 0000 0000 0000                      ........
22:42:20.784403 IP 195.154.167.38.9090 > 192.168.171.137.58871: Flags [.], ack 409, win 64240, length 0
	0x0000:  4500 0028 0cb6 0000 8006 5727 c39a a726  E..(......W'...&
	0x0010:  c0a8 ab89 2382 e5f7 2841 b219 bd6e 8a8e  ....#...(A...n..
	0x0020:  5010 faf0 b21e 0000 0000 0000 0000       P.............

I left it running for about one hour but nothing changed in the traffic pattern.

When I connected to the running process I could not see any activity

strace -p 7997
Process 7997 attached - interrupt to quit
futex(0x7fa83f5c69d0, FUTEX_WAIT, 7998, NULL

Because I did not wanted to leave the process running unattended I killed it. Feel free to get in touch if you are able to share packet captures that contain more useful data.

Services on the C&C

I decided to have a look what other services were provided by the command and control server. The mapping was done via a TOR proxy.

PORT      STATE    SERVICE     VERSION
21/tcp    open     ftp         Gene6 ftpd 3.10.0 build 2
80/tcp    open     http        Microsoft IIS httpd 7.5
135/tcp   open     msrpc       Microsoft Windows RPC
445/tcp   open     netbios-ssn
1021/tcp  open     exp1?
1024/tcp  open     kdm?
5060/tcp  filtered sip
8080/tcp  open     http        HttpFileServer httpd 2.3
9090/tcp  open     tcpwrapped
49152/tcp open     msrpc       Microsoft Windows RPC
49153/tcp open     msrpc       Microsoft Windows RPC
49154/tcp open     msrpc       Microsoft Windows RPC
49155/tcp open     msrpc       Microsoft Windows RPC
49156/tcp open     msrpc       Microsoft Windows RPC
49157/tcp open     msrpc       Microsoft Windows RPC
Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows

Conclusions after Update 20140716

Check your startup files. This malware might have changed the files

  • /etc/rc.local
  • /etc/rc.d/rc.local

Leave a Reply

Your email address will not be published. Required fields are marked *