One year after the release of the technical analysis of the Mumblehard Linux botnet, we are pleased to report that it is no longer active. ESET, in cooperation with the Cyber Police of Ukraine and CyS Centrum LLC, have taken down the Mumblehard botnet, stopping all its spamming activities since February 29th, 2016.
ESET is operating a sinkhole server for all known Mumblehard components. We are sharing the sinkhole data with CERT-Bund, which is taking care of notifying the affected parties around the world through their national CERTs.
Post-publication aftermath
ESET researchers observed a reaction from the operators of the botnet about a month after we published our paper. In our report we mentioned how we had registered a domain name acting as a C&C server for the backdoor component in order to estimate the botnet size and distribution. The malware authors apparently responded by removing the unnecessary domains and IP addresses from the list of C&C servers included in the malware and keeping only the one that was under their control.
Figure 1: Difference between the tidied Perl code of the Mumblehard backdoor prior to (in red) and shortly after (in green) the release of ESET's technical analysis
ESET first saw the updated version on May 24th, 2015. This sample appeared on VirusTotal the next day.
The existing bots were not all updated at once. Our sinkhole enabled us to note the updating of a first batch of about 500 bots on May 25th. The rest received the update a month later, on June 26th.
Figure 2: Statistics from Mumblehard sinkhole after the publication
With only one IP address acting as the C&C server for the Mumblehard backdoor and no fallback mechanism, a takeover of that IP address would suffice to stop the malicious activities of this botnet. We decided to take action and contacted the relevant authorities to make things happen.
Chasing the C&C server
With the help of the Cyber Police of Ukraine and CyS Centrum LLC, we were able to obtain information from the C&C server in October 2015. A forensic analysis revealed that most of our initial assumptions about the botnet size and its purpose were correct. Spamming was their main business. We also found several different control panels that were used to simplify management of the botnet.
Like the Mumblehard malicious components, the control panel was written in the Perl programming language.
Figure 3: Control panel used to push commands to the Mumblehard backdoor
Figure 4: Control panel showing the status of a spam job
A question left open in our previous report was the initial attack vector used by Mumblehard. We knew some of the victims had been compromised through an unpatched CMS such as WordPress or Joomla, or one of their plugins. Forensic analysis of the C&C server suggests that computers running the Mumblehard bot agents were not initially compromised from that specific server. The scripts we found were only to be run where PHP shells had already been installed. Perhaps Mumblehard’s operators were buying access to these compromised machines from another criminal gang?
Figure 5: Control panel for pushing commands to a list of PHP shells
Another interesting aspect of the Mumblehard operation revealed by our access to the C&C server was the automatic delisting from Spamhaus’ Composite Blocking List (CBL). There was a script automatically monitoring the CBL for the IP addresses of all the spam-bots. If one was found to be blacklisted, this script requested the delisting of the IP address. Such requests are protected with a CAPTCHA to avoid automation, but OCR (or an external service if OCR didn’t work) was used to break the protection.
Figure 6: Status of an IP address from the control panel after it is delisted from the CBL
All C&C actions that involved contacting a remote host such as running commands on PHP shells, CBL delisting and so on were performed via proxies. The open proxy feature of the Mumblehard spamming daemon, listening on TCP port 39331, was used to mask the original source of the query. The control panel checking the availability of all the proxies showed victims in 63 different countries.
Figure 7 Control panel showing the status of the open proxy check and the country count
Latest sinkhole statistics
When the Cyber Police of Ukraine shut down the Mumblehard C&C server on February 29th, 2016, they replaced it with a sinkhole operated by ESET. We have been collecting data from it during the month of March 2016. It showed almost 4,000 Linux systems had been compromised with the Mumblehard botnet agent at the end of February. The number is slowly decreasing.
CERT-Bund has recently started to notify the affected parties. We hope to see a faster decrease in the weeks to come.
Figure 8: Statistics from our new sinkhole
Remediation
If you receive a notification that one of your systems is infected, head to our indicators of compromise Github repository for more details about how to find Mumblehard on your system and remove it.
To avoid future infections, it is crucial that web applications hosted on the system are up-to-date (plugins included) and that administrative accounts have strong passwords.
Conclusion
Collaboration with law enforcement and external entities was crucial in making this operation a success. ESET would like to thank the Cyber Police of Ukraine, CyS Centrum LLC and CERT-Bund. We are proud of our efforts to make the internet a safer place. Mumblehard might not be the most prevalent, the most dangerous or the most sophisticated botnet out there, but shutting it down is still a step in the right direction and shows that security researchers working with other entities can help reduce the impact of criminal activity on the internet.
We have not seen any new variants of Mumblehard, or any activities from this group, since the takedown.
