Hacked via WordPress

Submitted by securewebs on Mon, 07/06/2015 - 19:50 Pro Licensee

The last 24 hours has been a huge misery. One of our server was compromised, it looks like two WordPress sites were hacked, not sure exactly how. We have been working around the clock. The sites started to create 10's of thousands of email files, though they were not successfully sent. In the end after hours of battling the hack we only got it to stop by completely deleting Dovecot. Now that we have reinstalled it the few legit emails in Postfix spool folder are going nowhere because it can't find a mail transport. Any suggestions on how to get it working again. In the years we have used the control panel this is the worst thing that has ever happened to us. We deleted the two sites and when they are restored they immediately start to create email files again. Unfortunately, while there are WordPress plugins that scan for viruses, they all seem to depend on WordPress running and we cannot afford to let them run, not even for a short time. We wish there was some way to scan for files without WordPress running. What a mess. There are just too many files in a WordPress site to find out where the hack is and we have not really had to before now.

Status: 
Active

Comments

Submitted by andreychek on Mon, 07/06/2015 - 20:03

Howdy -- what is the exact error you receive in the email logs when trying to send an email?

Also, if you restart Postfix, do any errors show up in the logs when it's starting back up?

As far as WordPress goes -- our suggestion there is to make sure it's fully up to date, that the plugins are all fully up to date, and you may want to change the passwords for any WordPress users.

To prevent emails from being sent, you can always just disable Postfix. If Postfix isn't running, your server won't send out any emails from the mail queue.

However, you could try scanning your WordPress files with the Linux Malware Detect tool, which can pick up some web-based malware:

https://www.rfxn.com/projects/linux-malware-detect/

Submitted by securewebs on Mon, 07/06/2015 - 21:31 Pro Licensee

Here is the return when trying to restart Postfix, which it appears to do but just won't send any email. Jul 6 18:52:47 raven postfix/master[23347]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling

We are pretty careful with WordPress and always use WordFence, which in this case did not help. Thanks for the suggestion of the malware scan. It found 31 effect files. Unfortunately, it did not clean anything. So hopefully I might actually figure out how to get it to clean up the files. But at least we know which files are infected. For example, here is a snippet;

{CAV}PHP.Trojan.Uploader : rennerusa/public_html/wp-includes/js/tinymce/skins/lightgray/img/files.php => /usr/local/maldetect/quarantine/files.php.14347 {CAV}Php.Malware.Mailbot-1 : rennerusa/public_html/wp-includes/js/tinymce/skins/user.php => /usr/local/maldetect/quarantine/user.php.2142 {CAV}Php.Trojan.StopPost : rennerusa/public_html/wp-includes/images/smilies/list.php => /usr/local/maldetect/quarantine/list.php.29468 {CAV}Php.Trojan.StopPost : rennerusa/public_html/wp-includes/SimplePie/Cache/list.php => /usr/local/maldetect/quarantine/list.php.27506 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-includes/pomo/cache_checkexpresses.php => /usr/local/maldetect/quarantine/cache_checkexpresses.php.14332 {CAV}PHP.Trojan.Uploader : rennerusa/public_html/includes/fresh/freshCore/canvas/fresh/external/themes.php => /usr/local/maldetect/quarantine/themes.php.10110 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-content/plugins/E4nKzzZl.php => /usr/local/maldetect/quarantine/E4nKzzZl.php.27014 {HEX}php.cmdshell.unclassed.357 : rennerusa/public_html/wp-content/plugins/Y4aU3y7.php => /usr/local/maldetect/quarantine/Y4aU3y7.php.2311

Submitted by andreychek on Mon, 07/06/2015 - 21:41

If you're seeing a "bad command startup" error, it should be generating another error with that showing what the bad command is... are you seeing other errors in the logs around the time Postfix is started?

I use maldet mostly to help me identify malware -- it's not always perfect, so I typically prefer to manually inspect anything it finds, rather than have it attempt to clean it for me.

Submitted by securewebs on Tue, 07/14/2015 - 11:35 Pro Licensee

Unfortunately, there are form mail piling up on the server, customers quite upset at me (I don't blame them) and if I open up some of the files being generated at /var/spool/postfix/defer, they all say //// action=delayed - reason=mail transport unavailable //// I thought Dovecot was the one to relay for Postfix. Since it was uninstalled after a mail relay hack, it no longer is working. I am not even sure where to tell Dovecot where it should be looking to see if there is mail. The defaults don't seem right but what do I know. We have had issues over the years but this is the first time I felt like I was way in over my head. Any suggestions on the meaning of /// mail transport unavailable/// or even to your knowledge what is the default setup with Virtualmin and mail?

Submitted by andreychek on Tue, 07/14/2015 - 11:40

Dovecot isn't involved in delivering email, only Postfix is.

The question I asked in comment #3 above is still the key -- if you're seeing a "bad command startup" error, Postfix should be generating another error along with that showing what the bad command is... are you seeing other errors in the logs around the time Postfix is started?

Submitted by securewebs on Tue, 07/14/2015 - 12:05 Pro Licensee

It looks like a configuration error. Now, if I can find out where it specifies the mailbox size. //////////////// Jul 12 03:22:10 raven postfix/local[23878]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 12 03:22:11 raven postfix/master[1098]: warning: process /usr/libexec/postfix/local pid 23878 exit status 1 Jul 12 03:22:11 raven postfix/master[1098]: warning: /usr/libexec/postfix/local: bad command startup -- throttling Jul 12 03:23:11 raven postfix/local[24130]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 12 03:23:12 raven postfix/master[1098]: warning: process /usr/libexec/postfix/local pid 24130 exit status 1 Jul 12 03:23:12 raven postfix/master[1098]: warning: /usr/libexec/postfix/local: bad command startup -- throttling

Submitted by andreychek on Tue, 07/14/2015 - 12:09

Agreed, this looks like the error here:

fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit

That is configured in /etc/postfix/main.cf.

Submitted by securewebs on Tue, 07/14/2015 - 12:18 Pro Licensee

I configured it to a value of 10000 and restarted Postfix. Still get the same thing. Back to the drawing board I guess. There must be something else out of wack as well. There is a reference to smtp bad command startup..... ////////////// Jul 14 10:14:01 raven postfix/local[3596]: fatal: main.cf configuration error: mailbox_size_limit is smaller than message_size_limit Jul 14 10:14:01 raven postfix/smtp[3595]: fatal: unexpected command-line argument: When Jul 14 10:14:02 raven postfix/master[3458]: warning: process /usr/libexec/postfix/local pid 3596 exit status 1 Jul 14 10:14:02 raven postfix/master[3458]: warning: /usr/libexec/postfix/local: bad command startup -- throttling Jul 14 10:14:02 raven postfix/master[3458]: warning: process /usr/libexec/postfix/smtp pid 3595 exit status 1 Jul 14 10:14:02 raven postfix/master[3458]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 10:14:02 raven postfix/qmgr[3460]: warning: private/smtp socket: malformed response

Submitted by andreychek on Tue, 07/14/2015 - 12:20

What is the output of this command:

postconf -n

Submitted by securewebs on Tue, 07/14/2015 - 12:25 Pro Licensee

[root@raven ~]# postconf -n alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases allow_percent_hack = no bounce_size_limit = 5 broken_sasl_auth_clients = yes command_directory = /usr/sbin config_directory = /etc/postfix daemon_directory = /usr/libexec/postfix data_directory = /var/lib/postfix debug_peer_level = 2 default_destination_concurrency_limit = 1 default_destination_recipient_limit = 1 default_process_limit = 1 deliver_lock_attempts = 10 fork_attempts = 1 home_mailbox = Maildir/ html_directory = no inet_interfaces = all inet_protocols = all initial_destination_concurrency = 2 mail_owner = postfix mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME mailbox_size_limit = 10000 mailq_path = /usr/bin/mailq.postfix manpage_directory = /usr/share/man maximal_queue_lifetime = 2d minimal_backoff_time = 900s mydestination = $myhostname, localhost.$mydomain, localhost, raven.securewebs.com newaliases_path = /usr/bin/newaliases.postfix queue_directory = /var/spool/postfix readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES sample_directory = /usr/share/doc/postfix-2.6.6/samples sender_bcc_maps = hash:/etc/postfix/bcc sendmail_path = /usr/sbin/sendmail.postfix setgid_group = postdrop smtpd_recipient_limit = 20 smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination reject_unknown_reverse_client_hostname permit_inet_interfaces unknown_local_recipient_reject_code = 550 virtual_alias_maps = hash:/etc/postfix/virtual

Submitted by andreychek on Tue, 07/14/2015 - 12:33

Hmm, as a quick fix, you could try setting this in your main.cf:

mailbox_size_limit = 0

That would set the mailbox size to unlimited, which should fix that error.

Submitted by securewebs on Tue, 07/14/2015 - 13:27 Pro Licensee

Hi: Good idea. Still have 94 messages in the spool. Still the logs say the mail transport is unavailable. Sent test message and the fatal error as seen below in the logfile is unexpected command-line argument: When. Not sure what "When" means yet. Another configuration error?

-------------Jul 14 11:22:35 raven postfix/pickup[15841]: 5BC068235B: uid=567 from= Jul 14 11:22:35 raven postfix/cleanup[17654]: 5BC068235B: message-id=5a690ee6080f4fc3ccc66914984adbd2@securewebs.com Jul 14 11:22:35 raven postfix/qmgr[15842]: 5BC068235B: from=wp@raven.securewebs.com, size=997, nrcpt=1 (queue active) Jul 14 11:22:35 raven postfix/smtp[17656]: fatal: unexpected command-line argument: When Jul 14 11:22:36 raven postfix/qmgr[15842]: warning: private/smtp socket: malformed response Jul 14 11:22:36 raven postfix/master[15840]: warning: process /usr/libexec/postfix/smtp pid 17656 exit status 1 Jul 14 11:22:36 raven postfix/master[15840]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 11:22:36 raven postfix/error[17657]: 5BC068235B: to=scott@securewebs.com, relay=none, delay=1, delays=0.01/1/0/0.01, dsn=4.3.0, status=deferred (mail transport unavailable)

Submitted by andreychek on Tue, 07/14/2015 - 13:31

That is an odd one! Can you attach your /etc/postfix/master.cf file?

Submitted by securewebs on Tue, 07/14/2015 - 13:41 Pro Licensee

Sure. Suggestions very very welcome. Please note I added .zip to the file name so that it would send.

Submitted by andreychek on Tue, 07/14/2015 - 13:49

Sorry I don't see it attached... you could always just copy and paste the contents though. Or provide a link to where I can view it elsewhere.

Submitted by andreychek on Tue, 07/14/2015 - 13:56

Oh, before I forget -- I did want to mention for future reference, that there's never a reason to uninstall software.

It sucks when spammers break into a site, and hopefully it doesn't happen again... but it if does, you can stop emails from being sent out by stopping Postfix.

You can do that with this command:

/etc/init.d/postfix stop

Email can't enter or exit the email queue if Postfix isn't running.

At that point, you could resolve the problem, and then once it's corrected, you can then restart Postfix.

Removing software can cause quite a lot of problems, including needing to manually redo the configuration for various services that are configured at Virtualmin's installation time.

Submitted by securewebs on Tue, 07/14/2015 - 13:56 Pro Licensee

I am afraid I don't know how to keep the line breaks in place to make it readable.

# Global Postfix configuration file. This file lists only a subset
# of all parameters. For the syntax, and for a complete parameter
# list, see the postconf(5) manual page (command: "man 5 postconf").
#
# For common configuration examples, see BASIC_CONFIGURATION_README
# and STANDARD_CONFIGURATION_README. To find these documents, use
# the command "postconf html_directory readme_directory", or go to
# http://www.postfix.org/.
#
# For best results, change no more than 2-3 parameters at a time,
# and test if Postfix still works after every change.

# SOFT BOUNCE
#
# The soft_bounce parameter provides a limited safety net for
# testing.  When soft_bounce is enabled, mail will remain queued that
# would otherwise bounce. This parameter disables locally-generated
# bounces, and prevents the SMTP server from rejecting mail permanently
# (by changing 5xx replies into 4xx replies). However, soft_bounce
# is no cure for address rewriting mistakes or mail routing mistakes.
#
#soft_bounce = no

# LOCAL PATHNAME INFORMATION
#
# The queue_directory specifies the location of the Postfix queue.
# This is also the root directory of Postfix daemons that run chrooted.
# See the files in examples/chroot-setup for setting up Postfix chroot
# environments on different UNIX systems.
#
queue_directory = /var/spool/postfix

# The command_directory parameter specifies the location of all
# postXXX commands.
#
command_directory = /usr/sbin

# The daemon_directory parameter specifies the location of all Postfix
# daemon programs (i.e. programs listed in the master.cf file). This
# directory must be owned by root.
#
daemon_directory = /usr/libexec/postfix

# The data_directory parameter specifies the location of Postfix-writable
# data files (caches, random numbers). This directory must be owned
# by the mail_owner account (see below).
#
data_directory = /var/lib/postfix

# QUEUE AND PROCESS OWNERSHIP
#
# The mail_owner parameter specifies the owner of the Postfix queue
# and of most Postfix daemon processes.  Specify the name of a user
# account THAT DOES NOT SHARE ITS USER OR GROUP ID WITH OTHER ACCOUNTS
# AND THAT OWNS NO OTHER FILES OR PROCESSES ON THE SYSTEM.  In
# particular, don't specify nobody or daemon. PLEASE USE A DEDICATED
# USER.
#
mail_owner = postfix

# The default_privs parameter specifies the default rights used by
# the local delivery agent for delivery to external file or command.
# These rights are used in the absence of a recipient user context.
# DO NOT SPECIFY A PRIVILEGED USER OR THE POSTFIX OWNER.
#
#default_privs = nobody

# INTERNET HOST AND DOMAIN NAMES
#
# The myhostname parameter specifies the internet hostname of this
# mail system. The default is to use the fully-qualified domain name
# from gethostname(). $myhostname is used as a default value for many
# other configuration parameters.
#
#myhostname = host.domain.tld
#myhostname = virtual.domain.tld

# The mydomain parameter specifies the local internet domain name.
# The default is to use $myhostname minus the first component.
# $mydomain is used as a default value for many other configuration
# parameters.
#
#mydomain = domain.tld

# SENDING MAIL
#
# The myorigin parameter specifies the domain that locally-posted
# mail appears to come from. The default is to append $myhostname,
# which is fine for small sites.  If you run a domain with multiple
# machines, you should (1) change this to $mydomain and (2) set up
# a domain-wide alias database that aliases each user to
# user@that.users.mailhost.
#
# For the sake of consistency between sender and recipient addresses,
# myorigin also specifies the default domain name that is appended
# to recipient addresses that have no @domain part.
#
#myorigin = $myhostname
#myorigin = $mydomain

# RECEIVING MAIL

# The inet_interfaces parameter specifies the network interface
# addresses that this mail system receives mail on.  By default,
# the software claims all active interfaces on the machine. The
# parameter also controls delivery of mail to user@[ip.address].
#
# See also the proxy_interfaces parameter, for network addresses that
# are forwarded to us via a proxy or network address translator.
#
# Note: you need to stop/start Postfix when this parameter changes.
#
#inet_interfaces = all
#inet_interfaces = $myhostname
#inet_interfaces = $myhostname, localhost
inet_interfaces = all

# Enable IPv4, and IPv6 if supported
inet_protocols = all

# The proxy_interfaces parameter specifies the network interface
# addresses that this mail system receives mail on by way of a
# proxy or network address translation unit. This setting extends
# the address list specified with the inet_interfaces parameter.
#
# You must specify your proxy/NAT addresses when your system is a
# backup MX host for other domains, otherwise mail delivery loops
# will happen when the primary MX host is down.
#
#proxy_interfaces =
#proxy_interfaces = 1.2.3.4

# The mydestination parameter specifies the list of domains that this
# machine considers itself the final destination for.
#
# These domains are routed to the delivery agent specified with the
# local_transport parameter setting. By default, that is the UNIX
# compatible delivery agent that lookups all recipients in /etc/passwd
# and /etc/aliases or their equivalent.
#
# The default is $myhostname + localhost.$mydomain.  On a mail domain
# gateway, you should also include $mydomain.
#
# Do not specify the names of virtual domains - those domains are
# specified elsewhere (see VIRTUAL_README).
#
# Do not specify the names of domains that this machine is backup MX
# host for. Specify those names via the relay_domains settings for
# the SMTP server, or use permit_mx_backup if you are lazy (see
# STANDARD_CONFIGURATION_README).
#
# The local machine is always the final destination for mail addressed
# to user@[the.net.work.address] of an interface that the mail system
# receives mail on (see the inet_interfaces parameter).
#
# Specify a list of host or domain names, /file/name or type:table
# patterns, separated by commas and/or whitespace. A /file/name
# pattern is replaced by its contents; a type:table is matched when
# a name matches a lookup key (the right-hand side is ignored).
# Continue long lines by starting the next line with whitespace.
#
# See also below, section "REJECTING MAIL FOR UNKNOWN LOCAL USERS".
#
mydestination = $myhostname, localhost.$mydomain, localhost, raven.securewebs.com
#mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
#mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain,
# mail.$mydomain, www.$mydomain, ftp.$mydomain

# REJECTING MAIL FOR UNKNOWN LOCAL USERS
#
# The local_recipient_maps parameter specifies optional lookup tables
# with all names or addresses of users that are local with respect
# to $mydestination, $inet_interfaces or $proxy_interfaces.
#
# If this parameter is defined, then the SMTP server will reject
# mail for unknown local users. This parameter is defined by default.
#
# To turn off local recipient checking in the SMTP server, specify
# local_recipient_maps = (i.e. empty).
#
# The default setting assumes that you use the default Postfix local
# delivery agent for local delivery. You need to update the
# local_recipient_maps setting if:
#
# - You define $mydestination domain recipients in files other than
#   /etc/passwd, /etc/aliases, or the $virtual_alias_maps files.
#   For example, you define $mydestination domain recipients in   
#   the $virtual_mailbox_maps files.
#
# - You redefine the local delivery agent in master.cf.
#
# - You redefine the "local_transport" setting in main.cf.
#
# - You use the "luser_relay", "mailbox_transport", or "fallback_transport"
#   feature of the Postfix local delivery agent (see local(8)).
#
# Details are described in the LOCAL_RECIPIENT_README file.
#
# Beware: if the Postfix SMTP server runs chrooted, you probably have
# to access the passwd file via the proxymap service, in order to
# overcome chroot restrictions. The alternative, having a copy of
# the system passwd file in the chroot jail is just not practical.
#
# The right-hand side of the lookup tables is conveniently ignored.
# In the left-hand side, specify a bare username, an @domain.tld
# wild-card, or specify a user@domain.tld address.
#
#local_recipient_maps = unix:passwd.byname $alias_maps
#local_recipient_maps = proxy:unix:passwd.byname $alias_maps
#local_recipient_maps =

# The unknown_local_recipient_reject_code specifies the SMTP server
# response code when a recipient domain matches $mydestination or
# ${proxy,inet}_interfaces, while $local_recipient_maps is non-empty
# and the recipient address or address local-part is not found.
#
# The default setting is 550 (reject mail) but it is safer to start
# with 450 (try again later) until you are certain that your
# local_recipient_maps settings are OK.
#
unknown_local_recipient_reject_code = 550

# TRUST AND RELAY CONTROL

# The mynetworks parameter specifies the list of "trusted" SMTP
# clients that have more privileges than "strangers".
#
# In particular, "trusted" SMTP clients are allowed to relay mail
# through Postfix.  See the smtpd_recipient_restrictions parameter
# in postconf(5).
#
# You can specify the list of "trusted" network addresses by hand
# or you can let Postfix do it for you (which is the default).
#
# By default (mynetworks_style = subnet), Postfix "trusts" SMTP
# clients in the same IP subnetworks as the local machine.
# On Linux, this does works correctly only with interfaces specified
# with the "ifconfig" command.
#
# Specify "mynetworks_style = class" when Postfix should "trust" SMTP
# clients in the same IP class A/B/C networks as the local machine.
# Don't do this with a dialup site - it would cause Postfix to "trust"
# your entire provider's network.  Instead, specify an explicit
# mynetworks list by hand, as described below.
# 
# Specify "mynetworks_style = host" when Postfix should "trust"
# only the local machine.
#
#mynetworks_style = class
#mynetworks_style = subnet
#mynetworks_style = host

# Alternatively, you can specify the mynetworks list by hand, in
# which case Postfix ignores the mynetworks_style setting.
#
# Specify an explicit list of network/netmask patterns, where the
# mask specifies the number of bits in the network part of a host
# address.
#
# You can also specify the absolute pathname of a pattern file instead
# of listing the patterns here. Specify type:table for table-based lookups
# (the value on the table right-hand side is not used).
#
#mynetworks = 168.100.189.0/28, 127.0.0.0/8
#mynetworks = $config_directory/mynetworks
#mynetworks = hash:/etc/postfix/network_table

# The relay_domains parameter restricts what destinations this system will
# relay mail to.  See the smtpd_recipient_restrictions description in
# postconf(5) for detailed information.
#
# By default, Postfix relays mail
# - from "trusted" clients (IP address matches $mynetworks) to any destination,
# - from "untrusted" clients to destinations that match $relay_domains or
#   subdomains thereof, except addresses with sender-specified routing.
# The default relay_domains value is $mydestination.
#
# In addition to the above, the Postfix SMTP server by default accepts mail
# that Postfix is final destination for:
# - destinations that match $inet_interfaces or $proxy_interfaces,
# - destinations that match $mydestination
# - destinations that match $virtual_alias_domains,
# - destinations that match $virtual_mailbox_domains.
# These destinations do not need to be listed in $relay_domains.
#
# Specify a list of hosts or domains, /file/name patterns or type:name
# lookup tables, separated by commas and/or whitespace.  Continue
# long lines by starting the next line with whitespace. A file name
# is replaced by its contents; a type:name table is matched when a
# (parent) domain appears as lookup key.
#
# NOTE: Postfix will not automatically forward mail for domains that
# list this system as their primary or backup MX host. See the
# permit_mx_backup restriction description in postconf(5).
#
#relay_domains = $mydestination

# INTERNET OR INTRANET

# The relayhost parameter specifies the default host to send mail to
# when no entry is matched in the optional transport(5) table. When
# no relayhost is given, mail is routed directly to the destination.
#
# On an intranet, specify the organizational domain name. If your
# internal DNS uses no MX records, specify the name of the intranet
# gateway host instead.
#
# In the case of SMTP, specify a domain, host, host:port, [host]:port,
# [address] or [address]:port; the form [host] turns off MX lookups.
#
# If you're connected via UUCP, see also the default_transport parameter.
#
#relayhost = $mydomain
#relayhost = [gateway.my.domain]
#relayhost = [mailserver.isp.tld]
#relayhost = uucphost
#relayhost = [an.ip.add.ress]

# REJECTING UNKNOWN RELAY USERS
#
# The relay_recipient_maps parameter specifies optional lookup tables
# with all addresses in the domains that match $relay_domains.
#
# If this parameter is defined, then the SMTP server will reject
# mail for unknown relay users. This feature is off by default.
#
# The right-hand side of the lookup tables is conveniently ignored.
# In the left-hand side, specify an @domain.tld wild-card, or specify
# a user@domain.tld address.
#
#relay_recipient_maps = hash:/etc/postfix/relay_recipients

# INPUT RATE CONTROL
#
# The in_flow_delay configuration parameter implements mail input
# flow control. This feature is turned on by default, although it
# still needs further development (it's disabled on SCO UNIX due
# to an SCO bug).
#
# A Postfix process will pause for $in_flow_delay seconds before
# accepting a new message, when the message arrival rate exceeds the
# message delivery rate. With the default 100 SMTP server process
# limit, this limits the mail inflow to 100 messages a second more
# than the number of messages delivered per second.
#
# Specify 0 to disable the feature. Valid delays are 0..10.
#
#in_flow_delay = 1s

# ADDRESS REWRITING
#
# The ADDRESS_REWRITING_README document gives information about
# address masquerading or other forms of address rewriting including
# username->Firstname.Lastname mapping.

# ADDRESS REDIRECTION (VIRTUAL DOMAIN)
#
# The VIRTUAL_README document gives information about the many forms
# of domain hosting that Postfix supports.

# "USER HAS MOVED" BOUNCE MESSAGES
#
# See the discussion in the ADDRESS_REWRITING_README document.

# TRANSPORT MAP
#
# See the discussion in the ADDRESS_REWRITING_README document.

# ALIAS DATABASE
#
# The alias_maps parameter specifies the list of alias databases used
# by the local delivery agent. The default list is system dependent.
#
# On systems with NIS, the default is to search the local alias
# database, then the NIS alias database. See aliases(5) for syntax
# details.
#
# If you change the alias database, run "postalias /etc/aliases" (or
# wherever your system stores the mail alias file), or simply run
# "newaliases" to build the necessary DBM or DB file.
#
# It will take a minute or so before changes become visible.  Use
# "postfix reload" to eliminate the delay.
#
#alias_maps = dbm:/etc/aliases
alias_maps = hash:/etc/aliases
#alias_maps = hash:/etc/aliases, nis:mail.aliases
#alias_maps = netinfo:/aliases

# The alias_database parameter specifies the alias database(s) that
# are built with "newaliases" or "sendmail -bi".  This is a separate
# configuration parameter, because alias_maps (see above) may specify
# tables that are not necessarily all under control by Postfix.
#
#alias_database = dbm:/etc/aliases
#alias_database = dbm:/etc/mail/aliases
alias_database = hash:/etc/aliases
#alias_database = hash:/etc/aliases, hash:/opt/majordomo/aliases

# ADDRESS EXTENSIONS (e.g., user+foo)
#
# The recipient_delimiter parameter specifies the separator between
# user names and address extensions (user+foo). See canonical(5),
# local(8), relocated(5) and virtual(5) for the effects this has on
# aliases, canonical, virtual, relocated and .forward file lookups.
# Basically, the software tries user+foo and .forward+foo before
# trying user and .forward.
#
#recipient_delimiter = +

# DELIVERY TO MAILBOX
#
# The home_mailbox parameter specifies the optional pathname of a
# mailbox file relative to a user's home directory. The default
# mailbox file is /var/spool/mail/user or /var/mail/user.  Specify
# "Maildir/" for qmail-style delivery (the / is required).
#
#home_mailbox = Mailbox
#home_mailbox = Maildir/

# The mail_spool_directory parameter specifies the directory where
# UNIX-style mailboxes are kept. The default setting depends on the
# system type.
#
#mail_spool_directory = /var/mail
#mail_spool_directory = /var/spool/mail

# The mailbox_command parameter specifies the optional external
# command to use instead of mailbox delivery. The command is run as
# the recipient with proper HOME, SHELL and LOGNAME environment settings.
# Exception:  delivery for root is done as $default_user.
#
# Other environment variables of interest: USER (recipient username),
# EXTENSION (address extension), DOMAIN (domain part of address),
# and LOCAL (the address localpart).
#
# Unlike other Postfix configuration parameters, the mailbox_command
# parameter is not subjected to $parameter substitutions. This is to
# make it easier to specify shell syntax (see example below).
#
# Avoid shell meta characters because they will force Postfix to run
# an expensive shell process. Procmail alone is expensive enough.
#
# IF YOU USE THIS TO DELIVER MAIL SYSTEM-WIDE, YOU MUST SET UP AN
# ALIAS THAT FORWARDS MAIL FOR ROOT TO A REAL USER.
#
#mailbox_command = /some/where/procmail
#mailbox_command = /some/where/procmail -a "$EXTENSION"

# The mailbox_transport specifies the optional transport in master.cf
# to use after processing aliases and .forward files. This parameter
# has precedence over the mailbox_command, fallback_transport and
# luser_relay parameters.
#
# Specify a string of the form transport:nexthop, where transport is
# the name of a mail delivery transport defined in master.cf.  The
# :nexthop part is optional. For more details see the sample transport
# configuration file.
#
# NOTE: if you use this feature for accounts not in the UNIX password
# file, then you must update the "local_recipient_maps" setting in
# the main.cf file, otherwise the SMTP server will reject mail for   
# non-UNIX accounts with "User unknown in local recipient table".
#
#mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp

# If using the cyrus-imapd IMAP server deliver local mail to the IMAP
# server using LMTP (Local Mail Transport Protocol), this is prefered
# over the older cyrus deliver program by setting the
# mailbox_transport as below:
#
# mailbox_transport = lmtp:unix:/var/lib/imap/socket/lmtp
#
# The efficiency of LMTP delivery for cyrus-imapd can be enhanced via
# these settings.
#
# local_destination_recipient_limit = 300
# local_destination_concurrency_limit = 5
#
# Of course you should adjust these settings as appropriate for the
# capacity of the hardware you are using. The recipient limit setting
# can be used to take advantage of the single instance message store
# capability of Cyrus. The concurrency limit can be used to control
# how many simultaneous LMTP sessions will be permitted to the Cyrus
# message store.
#
# To use the old cyrus deliver program you have to set:
#mailbox_transport = cyrus

# The fallback_transport specifies the optional transport in master.cf
# to use for recipients that are not found in the UNIX passwd database.
# This parameter has precedence over the luser_relay parameter.
#
# Specify a string of the form transport:nexthop, where transport is
# the name of a mail delivery transport defined in master.cf.  The
# :nexthop part is optional. For more details see the sample transport
# configuration file.
#
# NOTE: if you use this feature for accounts not in the UNIX password
# file, then you must update the "local_recipient_maps" setting in
# the main.cf file, otherwise the SMTP server will reject mail for   
# non-UNIX accounts with "User unknown in local recipient table".
#
#fallback_transport = lmtp:unix:/var/lib/imap/socket/lmtp
#fallback_transport =

# The luser_relay parameter specifies an optional destination address
# for unknown recipients.  By default, mail for unknown@$mydestination,
# unknown@[$inet_interfaces] or unknown@[$proxy_interfaces] is returned
# as undeliverable.
#
# The following expansions are done on luser_relay: $user (recipient
# username), $shell (recipient shell), $home (recipient home directory),
# $recipient (full recipient address), $extension (recipient address
# extension), $domain (recipient domain), $local (entire recipient
# localpart), $recipient_delimiter. Specify ${name?value} or
# ${name:value} to expand value only when $name does (does not) exist.
#
# luser_relay works only for the default Postfix local delivery agent.
#
# NOTE: if you use this feature for accounts not in the UNIX password
# file, then you must specify "local_recipient_maps =" (i.e. empty) in
# the main.cf file, otherwise the SMTP server will reject mail for   
# non-UNIX accounts with "User unknown in local recipient table".
#
#luser_relay = $user@other.host
#luser_relay = $local@other.host
#luser_relay = admin+$local
 
# JUNK MAIL CONTROLS
#
# The controls listed here are only a very small subset. The file
# SMTPD_ACCESS_README provides an overview.

# The header_checks parameter specifies an optional table with patterns
# that each logical message header is matched against, including
# headers that span multiple physical lines.
#
# By default, these patterns also apply to MIME headers and to the
# headers of attached messages. With older Postfix versions, MIME and
# attached message headers were treated as body text.
#
# For details, see "man header_checks".
#
#header_checks = regexp:/etc/postfix/header_checks

# FAST ETRN SERVICE
#
# Postfix maintains per-destination logfiles with information about
# deferred mail, so that mail can be flushed quickly with the SMTP
# "ETRN domain.tld" command, or by executing "sendmail -qRdomain.tld".
# See the ETRN_README document for a detailed description.
#
# The fast_flush_domains parameter controls what destinations are
# eligible for this service. By default, they are all domains that
# this server is willing to relay mail to.
#
#fast_flush_domains = $relay_domains

# SHOW SOFTWARE VERSION OR NOT
#
# The smtpd_banner parameter specifies the text that follows the 220
# code in the SMTP server's greeting banner. Some people like to see
# the mail version advertised. By default, Postfix shows no version.
#
# You MUST specify $myhostname at the start of the text. That is an
# RFC requirement. Postfix itself does not care.
#
#smtpd_banner = $myhostname ESMTP $mail_name
#smtpd_banner = $myhostname ESMTP $mail_name ($mail_version)

# PARALLEL DELIVERY TO THE SAME DESTINATION
#
# How many parallel deliveries to the same user or domain? With local
# delivery, it does not make sense to do massively parallel delivery
# to the same user, because mailbox updates must happen sequentially,
# and expensive pipelines in .forward files can cause disasters when
# too many are run at the same time. With SMTP deliveries, 10
# simultaneous connections to the same domain could be sufficient to
# raise eyebrows.
#
# Each message delivery transport has its XXX_destination_concurrency_limit
# parameter.  The default is $default_destination_concurrency_limit for
# most delivery transports. For the local delivery agent the default is 2.

#local_destination_concurrency_limit = 2
#default_destination_concurrency_limit = 20

# DEBUGGING CONTROL
#
# The debug_peer_level parameter specifies the increment in verbose
# logging level when an SMTP client or server host name or address
# matches a pattern in the debug_peer_list parameter.
#
debug_peer_level = 2

# The debug_peer_list parameter specifies an optional list of domain
# or network patterns, /file/name patterns or type:name tables. When
# an SMTP client or server host name or address matches a pattern,
# increase the verbose logging level by the amount specified in the
# debug_peer_level parameter.
#
#debug_peer_list = 127.0.0.1
#debug_peer_list = some.domain

# The debugger_command specifies the external command that is executed
# when a Postfix daemon program is run with the -D option.
#
# Use "command .. & sleep 5" so that the debugger can attach before
# the process marches on. If you use an X-based debugger, be sure to
# set up your XAUTHORITY environment variable before starting Postfix.
#
debugger_command =
PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
ddd $daemon_directory/$process_name $process_id & sleep 5

# If you can't use X, use this to capture the call stack when a
# daemon crashes. The result is in a file in the configuration
# directory, and is named after the process name and the process ID.
#
# debugger_command =
# PATH=/bin:/usr/bin:/usr/local/bin; export PATH; (echo cont;
# echo where) | gdb $daemon_directory/$process_name $process_id 2>&1
# >$config_directory/$process_name.$process_id.log & sleep 5
#
# Another possibility is to run gdb under a detached screen session.
# To attach to the screen sesssion, su root and run "screen -r
# <id_string>" where <id_string> uniquely matches one of the detached
# sessions (from "screen -list").
#
# debugger_command =
# PATH=/bin:/usr/bin:/sbin:/usr/sbin; export PATH; screen
# -dmS $process_name gdb $daemon_directory/$process_name
# $process_id & sleep 1

# INSTALL-TIME CONFIGURATION INFORMATION
#
# The following parameters are used when installing a new Postfix version.
#
# sendmail_path: The full pathname of the Postfix sendmail command.
# This is the Sendmail-compatible mail posting interface.
#
sendmail_path = /usr/sbin/sendmail.postfix

# newaliases_path: The full pathname of the Postfix newaliases command.
# This is the Sendmail-compatible command to build alias databases.
#
newaliases_path = /usr/bin/newaliases.postfix

# mailq_path: The full pathname of the Postfix mailq command.  This
# is the Sendmail-compatible mail queue listing command.
#
mailq_path = /usr/bin/mailq.postfix

# setgid_group: The group for mail submission and queue management
# commands.  This must be a group name with a numerical group ID that
# is not shared with other accounts, not even with the Postfix account.
#
setgid_group = postdrop

# html_directory: The location of the Postfix HTML documentation.
#
html_directory = no

# manpage_directory: The location of the Postfix on-line manual pages.
#
manpage_directory = /usr/share/man

# sample_directory: The location of the Postfix sample configuration files.
# This parameter is obsolete as of Postfix 2.1.
#
sample_directory = /usr/share/doc/postfix-2.6.6/samples

# readme_directory: The location of the Postfix README files.
#
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES
virtual_alias_maps = hash:/etc/postfix/virtual
sender_bcc_maps = hash:/etc/postfix/bcc
mailbox_command = /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME
home_mailbox = Maildir/
broken_sasl_auth_clients = yes
smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination reject_unknown_reverse_client_hostname permit_inet_interfaces
mailbox_size_limit = 0
allow_percent_hack = no
smtpd_recipient_limit = 20
default_destination_recipient_limit = 1
maximal_queue_lifetime = 2d
initial_destination_concurrency = 2
minimal_backoff_time = 900s
default_destination_concurrency_limit = 1
deliver_lock_attempts = 10
default_process_limit = 20
fork_attempts = 10
bounce_size_limit = 0

Submitted by securewebs on Tue, 07/14/2015 - 13:59 Pro Licensee

Sorry, something went way wrong with the format. - Scott

Submitted by andreychek on Tue, 07/14/2015 - 14:03

That's okay, I just wrapped it in "code" tags.

That looks like your main.cf file though, rather than the master.cf.

Can you paste in your master.cf file? Thanks!

Submitted by securewebs on Tue, 07/14/2015 - 14:10 Pro Licensee

Latest from logfiles. Can't figure out where to change the bounce number. Good grief.

Jul 14 12:05:03 raven postfix/qmgr[25191]: warning: private/smtp socket: malformed response Jul 14 12:05:03 raven postfix/master[25181]: warning: process /usr/libexec/postfix/smtp pid 25196 exit status 1 Jul 14 12:05:03 raven postfix/master[25181]: warning: /usr/libexec/postfix/smtp: bad command startup -- throttling Jul 14 12:05:03 raven postfix/qmgr[25191]: warning: private/smtp socket: malformed response Jul 14 12:05:03 raven postfix/master[25181]: warning: process /usr/libexec/postfix/smtp pid 25199 exit status 1 Jul 14 12:05:03 raven postfix/bounce[25627]: fatal: invalid bounce_size_limit parameter value 0 < 1 Jul 14 12:05:04 raven postfix/master[25181]: warning: process /usr/libexec/postfix/bounce pid 25627 exit status 1 Jul 14 12:05:04 raven postfix/master[25181]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

Submitted by securewebs on Tue, 07/14/2015 - 14:11 Pro Licensee

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master").
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp inet n - n - - smtpd -o smtpd_sasl_auth_enable=yes
#submission inet n       -       n       -       -       smtpd
#  -o smtpd_tls_security_level=encrypt
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#smtps     inet  n       -       n       -       -       smtpd
#  -o smtpd_tls_wrappermode=yes
#  -o smtpd_sasl_auth_enable=yes
#  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628      inet  n       -       n       -       -       qmqpd
pickup    fifo  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp unix - - n - - smtp When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       n       -       -       smtp
-o smtp_fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
#maildrop  unix  -       n       n       -       -       pipe
#  flags=DRhu user=vmail argv=/usr/local/bin/maildrop -d ${recipient}
#
# ====================================================================
#
# The Cyrus deliver program has changed incompatibly, multiple times.
#
#old-cyrus unix  -       n       n       -       -       pipe
#  flags=R user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -m ${extension} ${user}
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#
#cyrus     unix  -       n       n       -       -       pipe
#  user=cyrus argv=/usr/lib/cyrus-imapd/deliver -e -r ${sender} -m ${extension} ${user}
#
# ====================================================================
#
# See the Postfix UUCP_README file for configuration details.
#
#uucp      unix  -       n       n       -       -       pipe
#  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
#
# ====================================================================
#
# Other external delivery methods.
#
#ifmail    unix  -       n       n       -       -       pipe
#  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
#
#bsmtp     unix  -       n       n       -       -       pipe
#  flags=Fq. user=bsmtp argv=/usr/local/sbin/bsmtp -f $sender $nexthop $recipient
#
#scalemail-backend unix -       n       n       -       2       pipe
#  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store
#  ${nexthop} ${user} ${extension}
#
#mailman   unix  -       n       n       -       -       pipe
#  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
#  ${nexthop} ${user}
submission inet n - n - - smtpd -o smtpd_sasl_auth_enable=yes

Submitted by andreychek on Tue, 07/14/2015 - 14:15

Ah, I see one problem that jumps out to me -- this line here:

smtp unix - - n - - smtp When relaying mail as backup MX, disable fallback_relay to avoid MX loops

That isn't valid syntax.

Try commenting that line out (the one below the line beginning with "proxywrite" and above the line beginning with "relay").

Once you do that, restart Postfix, and let's see how that works.

Submitted by securewebs on Tue, 07/14/2015 - 14:21 Pro Licensee

OK, I did that restarted and the logfile has as the last entry: Jul 14 12:18:38 raven postfix/bounce[28333]: fatal: invalid bounce_size_limit parameter value 0 < 1 Jul 14 12:18:39 raven postfix/master[28327]: warning: process /usr/libexec/postfix/bounce pid 28333 exit status 1 Jul 14 12:18:39 raven postfix/master[28327]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

Submitted by andreychek on Tue, 07/14/2015 - 14:25

It looks like this line is setup in your main.cf file:

bounce_size_limit = 0

It looks like it was manually added to the end there. The error you're receiving says that the value "0" is invalid for that parameter.

Try commenting that line out and restart Postfix.

Submitted by securewebs on Tue, 07/14/2015 - 14:30 Pro Licensee

Did that. The log file now says: Jul 14 12:28:30 raven postfix/master[30574]: warning: process /usr/libexec/postfix/bounce pid 30581 exit status 1 Jul 14 12:28:30 raven postfix/master[30574]: warning: /usr/libexec/postfix/bounce: bad command startup -- throttling

Spool is still stuck. Crap. This one is a pain. Sorry about that.

Submitted by securewebs on Tue, 07/14/2015 - 14:42 Pro Licensee

I suppose I could reinstall postfix but the only problem is the 96 messages. Or at least that is the first problem that comes to mind.

Submitted by andreychek on Tue, 07/14/2015 - 14:52

Okay, we're making progress!

Are you seeing any other errors when Postfix is restarted?

It looks like there should be an additional error either before or after the ones you mentioned above there... those are the generic errors, there should be a more specific one along with them.

Submitted by andreychek on Tue, 07/14/2015 - 14:54

Sorry I may have missed something, what is it with those 96 messages? Are you trying to keep them from going out?

You may be able to get rid of those first by going into Webmin -> Servers -> Postfix -> Mail Queue prior to starting Postfix.

Submitted by securewebs on Tue, 07/14/2015 - 15:46 Pro Licensee

The messages in the spool are legit. They are generated by forms on the websites. Requests for info, contact forms, shopping carts, etc. That is the problem. I can't get postfix to send them. After the last restart I can't even find a nasty fatal error message. But when flushing the spool file every message says the same thing; "no mail transport available". A bounce setting is not right but does not appear to be a fatal error. Would it be possible to replace the main.cf and/or master.cf file with one from another of our servers of about the same age?

Submitted by andreychek on Tue, 07/14/2015 - 20:20

While we should be able to determine what's wrong from messages in the log files, it's also possible to copy in a working main.cf and master.cf file.

The one thing you'd want to make sure of when doing that is to update any references of the other server's hostname or domain name to be that of the current server.

Before overwriting anything, I'd suggest making a backup of your current config files.

Submitted by securewebs on Wed, 07/15/2015 - 03:48 Pro Licensee

To my surprise there was nothing to change in either file. No reference to a specific domain or hostname that had to be changed. The second after I copied from a different machine it fixed postfix. Thank God. Thank you so much for holding my hand! Case closed. :-)