Log rotate somehow failed and httpd stopped

Submitted by unsalkorkmaz on Sat, 02/07/2015 - 20:24

I got mail from server:

/etc/cron.daily/logrotate:

Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
Redirecting to /bin/systemctl reload  httpd.service
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.

/var/log/cron:

Feb  8 03:09:29 ns1 run-parts(/etc/cron.daily)[31851]: finished logrotate
Feb  8 03:09:29 ns1 run-parts(/etc/cron.daily)[30720]: starting man-db.cron
Feb  8 03:09:33 ns1 run-parts(/etc/cron.daily)[32203]: finished man-db.cron
Feb  8 03:09:33 ns1 run-parts(/etc/cron.daily)[30720]: starting mlocate
Feb  8 03:09:36 ns1 run-parts(/etc/cron.daily)[32214]: finished mlocate
Feb  8 03:09:36 ns1 anacron[29967]: Job `cron.daily' terminated (mailing output)
Feb  8 03:09:36 ns1 anacron[29967]: Normal exit (1 job run)

last lines of /var/log/httpd/error_log-20150208:

[Sun Feb 08 03:08:32.495948 2015] [mpm_event:notice] [pid 28792:tid 139733314668608] AH00489: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 PHP/5.6.5 SVN/1.7.14 mod_perl/2.0.9dev Perl/v5.16.3 configured -- resuming normal operations
[Sun Feb 08 03:08:32.495960 2015] [core:notice] [pid 28792:tid 139733314668608] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Sun Feb 08 03:08:34.495085 2015] [mpm_event:notice] [pid 28792:tid 139733314668608] AH00493: SIGUSR1 received.  Doing graceful restart
[Sun Feb 08 03:08:34.566527 2015] [core:notice] [pid 28792] AH00060: seg fault or similar nasty error detected in the parent process

There has been lots of problem after switching to centos 7. Got other tickets too. Sorry for trouble.

Status: 
Active

Comments

Submitted by andreychek on Sat, 02/07/2015 - 23:11

Howdy -- a "segfault" error suggests that Apache crashed for some reason. Are you using a dedicated server, or VPS?

Also, what is the output of the command "free -m"?

Submitted by unsalkorkmaz on Sun, 02/08/2015 - 09:28

Dedicated server with those specs:

   System: ns1.firmasite.com: GNU/Linux
   RAM: 32 GB DDR3 RAM
   OS: GNU/Linux -- 3.10.0-123.13.2.el7.x86_64 -- #1 SMP Thu Dec 18 14:09:13 UTC 2014
   8 x CPU: Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz (7000.2 bogomips)
          Hyper-Threading, x86-64, MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET, Intel virtualization
   Hard Drive: 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD (Software-RAID 1) Class Enterprise

free -m result:

             total       used       free     shared    buffers     cached
Mem:         31877      20910      10966        738        276      15059
-/+ buffers/cache:       5573      26303
Swap:        16383        127      16256

Submitted by unsalkorkmaz on Sun, 02/08/2015 - 10:20

Those 2 pages seems like about mod_gnutls which i dont have:

yum list mod_* Installed Packages
mod_dav_svn.x86_64               1.7.14-6.el7                        @base
mod_evasive.x86_64               1.10.1-16.el7.1                     @epel
mod_fcgid.x86_64                 2.3.9-4.el7                         @base
mod_perl.x86_64                  2.0.8-10.20140624svn1602105.el7     @epel
mod_ssl.x86_64                   2:2.4.6-18.el7.centos.vm.1          @virtualmin
Available Packages
mod_auth_kerb.x86_64             5.4-28.el7                          base
mod_auth_ntlm_winbind.x86_64     0.0.0-0.15.20070129svn713.el7       epel
mod_auth_openid.x86_64           0.8-1.el7                           epel
mod_autoindex_mb.x86_64          2.18.1-1.el7                        epel
mod_flvx.x86_64                  0-0.7.20100525git.el7.1             epel
mod_form.x86_64                  0.1-2.20131204svn145.el7            epel
mod_form-devel.x86_64            0.1-2.20131204svn145.el7            epel
mod_geoip.x86_64                 1.2.7-2.el7.1                       epel
mod_ldap.x86_64                  2.4.6-19.el7.centos                 updates
mod_limitipconn.x86_64           0.23-14.el7                         epel
mod_log_post.x86_64              0.1.0-7.el7.1                       epel
mod_mirrorbrain.x86_64           2.18.1-1.el7                        epel
mod_nss.x86_64                   1.0.8-32.el7                        base
mod_perl-devel.x86_64            2.0.8-10.20140624svn1602105.el7     epel
mod_proxy_html.x86_64            1:2.4.6-19.el7.centos               updates
mod_qos.x86_64                   11.5-1.el7                          epel
mod_revocator.x86_64             1.0.3-19.el7                        base
mod_security.x86_64              2.7.3-5.el7                         base
mod_security_crs.noarch          2.2.6-6.el7                         base
mod_session.x86_64               2.4.6-19.el7.centos                 updates
mod_speedycgi.x86_64             2.22-16.el7.1                       epel
mod_wsgi.x86_64                  3.4-12.el7_0                        updates
mod_xsendfile.x86_64             0.12-10.el7                         epel

Submitted by andreychek on Sun, 02/08/2015 - 10:40

Nuts, I had hoped it would be as simple as disabling mod_gnutls! :-)

Well, it's unusual to see Apache crash, so my first thought is that it looks like there are a number of non-standard Apache modules installed there.

I'm not sure if that's related or not, but as a first step, I'd suggest disabling any Apache modules that you don't need that don't come with a default CentOS installation.

Submitted by unsalkorkmaz on Sun, 02/08/2015 - 10:51

I didnt enable any apache module except mod_evasive. I disabled lbmethod_heartbeat_module only from default.

Well, i am not expert on which modules i can safely disable tbh :-/

Submitted by andreychek on Sun, 02/08/2015 - 12:00

Are you experiencing this problem with Apache frequently (or regularly) during the log rotation? Or was it just that one time?

Also, what is the output of the command "httpd -M"?

Submitted by unsalkorkmaz on Sun, 02/08/2015 - 12:14

Well, i built system new so its first log rotation.

httpd -M result:

Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unique_id_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_event_module (shared)
proxy_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
proxy_ajp_module (shared)
proxy_balancer_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_http_module (shared)
proxy_scgi_module (shared)
ssl_module (shared)
systemd_module (shared)
cgid_module (shared)
perl_module (shared)
fcgid_module (shared)
php5_module (shared)
dav_svn_module (shared)
authz_svn_module (shared)
dontdothat_module (shared)
evasive20_module (shared)

Submitted by andreychek on Sun, 02/08/2015 - 17:26

Well, it could have just been a fluke, so you may just want to keep an eye on it. But when weird and unexpected things happen, you may want to take a close look at non-standard modules, such as evasive20_module and dontdothat_module.

You may just want to see if it happens again, or perhaps manually run the log rotation. But if you see that issue again, and especially if it becomes reproducible, you may want to see if removing either of those modules helps with that issue.

Submitted by unsalkorkmaz on Sun, 02/08/2015 - 20:50

I am just planning to use normal dedicated server with event mpm & mod_fcgid. No proxy, no svn, no git. Is this list ok for disabling?

from 00-base.conf:

LoadModule access_compat_module modules/mod_access_compat.so
#LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
#LoadModule allowmethods_module modules/mod_allowmethods.so
LoadModule auth_basic_module modules/mod_auth_basic.so
#LoadModule auth_digest_module modules/mod_auth_digest.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
#LoadModule authn_core_module modules/mod_authn_core.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_file_module modules/mod_authn_file.so
#LoadModule authn_socache_module modules/mod_authn_socache.so
LoadModule authz_core_module modules/mod_authz_core.so
#LoadModule authz_dbd_module modules/mod_authz_dbd.so
#LoadModule authz_dbm_module modules/mod_authz_dbm.so
#LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
#LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
#LoadModule cache_module modules/mod_cache.so
#LoadModule cache_disk_module modules/mod_cache_disk.so
#LoadModule data_module modules/mod_data.so
#LoadModule dbd_module modules/mod_dbd.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
#LoadModule dumpio_module modules/mod_dumpio.so
#LoadModule echo_module modules/mod_echo.so
#LoadModule env_module modules/mod_env.so
LoadModule expires_module modules/mod_expires.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule filter_module modules/mod_filter.so
LoadModule headers_module modules/mod_headers.so
#LoadModule include_module modules/mod_include.so
#LoadModule info_module modules/mod_info.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule mime_module modules/mod_mime.so
#LoadModule negotiation_module modules/mod_negotiation.so
#LoadModule remoteip_module modules/mod_remoteip.so
#LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule setenvif_module modules/mod_setenvif.so
#LoadModule slotmem_plain_module modules/mod_slotmem_plain.so
#LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
#LoadModule socache_dbm_module modules/mod_socache_dbm.so
#LoadModule socache_memcache_module modules/mod_socache_memcache.so
LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
#LoadModule status_module modules/mod_status.so
#LoadModule substitute_module modules/mod_substitute.so
LoadModule suexec_module modules/mod_suexec.so
#LoadModule unique_id_module modules/mod_unique_id.so
LoadModule unixd_module modules/mod_unixd.so
LoadModule userdir_module modules/mod_userdir.so
#LoadModule version_module modules/mod_version.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so

others:

#LoadModule dav_module modules/mod_dav.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
#LoadModule dav_lock_module modules/mod_dav_lock.so

#LoadModule lua_module modules/mod_lua.so

LoadModule proxy_module modules/mod_proxy.so
#LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
#LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
#LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
#LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
#LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
#LoadModule proxy_express_module modules/mod_proxy_express.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
#LoadModule proxy_fdpass_module modules/mod_proxy_fdpass.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_scgi_module modules/mod_proxy_scgi.so

#LoadModule dav_svn_module     modules/mod_dav_svn.so
#LoadModule authz_svn_module   modules/mod_authz_svn.so
#LoadModule dontdothat_module  modules/mod_dontdothat.so

Any of those modules necessary for virtualmin? Any more can i remove safely?

Submitted by andreychek on Sun, 02/08/2015 - 23:10

It can be tough to know exactly which modules are in use... the key is to try disabling one or more modules, and then to run an Apache config test. If the config passes, then you should be good to go.

With some of those lines commented out, try running this command to perform a config test:

apachectl configtest

If you receive a syntax error, it's likely that one or more of the modules that were commented out are currently in use.

Submitted by unsalkorkmaz on Mon, 02/09/2015 - 01:15

I believe i found segfault problem: https://bugs.php.net/bug.php?id=64938

Basically libxml_disable_entity_loader is not thread-safe and causing problem for server. It seems php got fix this week for this but not sure when we can install fixed version.