FastCGI + Ngnix + Increased Activity = 502 Bad Gateway :(

Submitted by chrisfullman on Thu, 12/15/2011 - 14:29

Hi there,

So Nginx worked great (once I put in the full command path) last night and was responding FAST - until this morning when activity jumped up on the site. The server crashed and wasn't responding to anything, giving users a 502 Bad Gateway. Once I restarted the entire server, it was back up for a while, but it seems as if I'm going to have to restart it every 4-6 hours or so.

From what I read on this thread ( http://www.virtualmin.com/node/19887#comment-92325 ), PHP/FastCGI is being called in an outdated method, instead of the preferred (and now native as of PHP 5.3.3) PHP-FPM. Restarting Nginx did not bring the site back up. It seems as if the PHP process is getting frozen when activity increases, but I don't know how to restart the PHP process itself.

In the error logs for the site, I was getting the following, which backs my assumption above up:

From what I've found, this can be fixed by changing how Virtualmin configures the process creation when a virtual server is added, but I'm not seeing any additional config files being created in the virtual server's home directory. To be fair, I did a good deal of searching to try to figure out how to remedy this and submit a patch, but I'm quite novice when it comes to VM templates.

Any ideas?

Thanks for the great work so far Jamie and team, excited where Virtualmin can take Nginx!

Status: 
Active

Comments

Submitted by JamieCameron on Thu, 12/15/2011 - 16:01

You should be able to re-start the Ngix PHP process for the domain by running :

/etc/init.d/php-fcgi-whatever.com start

replace whatever.com with your actual domain name.

I would be interested to see the contents of the file logs/php.log under the domain's home directory, as this should give some clue as to why the php-cgi process crashed.

Submitted by chrisfullman on Thu, 12/15/2011 - 16:13

Yep, I found the process and have the following command set to run every hour to reset the memory usage: /etc/init.d/php-fcgi-anidea.com restart

So that's a temporary workaround for now, but I'd like to figure out a way to spawn this differently so I don't have to do it. I've held off on creating new virtual servers so I can tweak the template, but I don't see anything in the PHP Wrapper section of the VM template section that would set this up. Is this set up by the Nginx plugin itself?

I found out the log wasn't actually being updated since php.ini wasn't set to do so. I've updated it and will update this ticket once the process crashes again. To capture the event, I've disabled the cron job to prevent the process from being restarted.

Submitted by JamieCameron on Thu, 12/15/2011 - 16:27

Yes, you shouldn't need to do a restart like that - ideally the php-cgi process should stay up all the time.

I'd be interested to see what gets logged to php.log when a failure happens, so we can work out why PHP is being killed.

Submitted by chrisfullman on Fri, 12/16/2011 - 10:22

Here's the latest events in php.log when the trouble begins: http://pastie.org/private/zmjql42vc7y2qzw1pnkv8a

You'll see the first event when the trouble begins is the connection is reset by the peer, and then everything following that is connection refused.

It also appears as if the FastCGI process itself (/etc/init.d/php-fcgi-anidea.com) is gone after that point where I can no longer restart it. So not only does it not respawn after the crash, the only way I can get it running again is restarting the server itself.

I'm assuming there's an overall actual system log that may record why the process itself quit, but I can't find that anywhere in Virtualmin.

Submitted by JamieCameron on Fri, 12/16/2011 - 14:12

When PHP crashes like this, can you re-start it by running /etc/init.d/php-fcgi-anidea.com start ?

It sounds like Virtualmin really needs to setup php-cgi to auto-restart if it fails..

Submitted by kfawcett on Fri, 12/16/2011 - 23:52 Pro Licensee

I'm experiencing the same issues. I've used browser mob to test load the server. After failing I was able to restart php-cgi with /etc/init.d/php-fcgi-anidea.com start.

I don't see anything in php.logs about it. Is there something I need to do for the logs to populate? jamie, let me know if looking at my server would help.

Submitted by chrisfullman on Fri, 12/16/2011 - 23:56

Well, hopefully /you/ don't also have an anidea.com virtual server setup judging by the process you said you had to restart ;)

I had to set the log path in my virtual server's php.ini file and restart the process before the log file would be used.

Submitted by kfawcett on Sat, 12/17/2011 - 00:07 Pro Licensee

:oP

Thanks Chris, I'll post some logs tomorrow.

Submitted by JamieCameron on Sat, 12/17/2011 - 00:17

I've just finished implementing auto-restart for the Nginx PHP process, which will be included in the next version of the Nginx plugin .. which should come out in a few days.

Submitted by kfawcett on Sun, 12/18/2011 - 11:07 Pro Licensee

Jamie,

I've been looking into this more and part of the problem might be not setting the init.d php-cgi script with at least the below environment variables to some default values.

PHP_FCGI_CHILDREN

PHP_FCGI_MAX_REQUESTS

Submitted by andreychek on Sun, 12/18/2011 - 10:39

Thanks for the info!

Did you by chance have an opportunity to test those settings? Did they make a difference for you when performing tests under load on your server?

Submitted by kfawcett on Sun, 12/18/2011 - 11:22 Pro Licensee

Sadly, it doesn't seem to have helped. The process is dying after a minimal load. I'm still not showing anything in the php error logs.

What's the chances of implementing php-fpm?

Submitted by kfawcett on Sun, 12/18/2011 - 11:47 Pro Licensee

Most of the init.d template scripts are showing these lines at the top. Should we include any of the other parts in our php-cgi script?

BIND=127.0.0.1:9000
USER=www-data
PHP_FCGI_CHILDREN=15
PHP_FCGI_MAX_REQUESTS=1000

PHP_CGI=/usr/bin/php-cgi
PHP_CGI_NAME=`basename $PHP_CGI`
PHP_CGI_ARGS="- USER=$USER PATH=/usr/bin PHP_FCGI_CHILDREN=$PHP_FCGI_CHILDREN PHP_FCGI_MAX_REQUESTS=$PHP_FCGI_MAX_REQUESTS $PHP_CGI -b $BIND"

Submitted by JamieCameron on Sun, 12/18/2011 - 16:29

We may go with php-fpm instead of implementing our own process manager, as long as php-fpm is widely available. If you want an update to the Nginx plugin that sets up a restarter script for new domains, let me know and I'll email it to you ..

Submitted by kfawcett on Sun, 12/18/2011 - 20:25 Pro Licensee

Thanks Jamie, I will take the restarter script. Please email to keith.fawcett at gmail.com

Not sure how much more widely available you can get than being bundled into PHP. ;)

According to php-fpm.org

Nov 29, 2011 It's official. PHP-FPM is no longer marked as "experimental" as of PHP 5.4.0RC2.

and as of

Jul 22, 2010 PHP 5.3.3 is released and now bundles PHP-FPM, with all of the new improvements - adaptive process spawning, the new INI file format and include support, basic metrics for reporting, and more. If your code is PHP 5.3 compliant, it is highly recommended that you upgrade to take advantage of the built-in PHP-FPM support now (not to mention mysqlnd and all the other new features.)

Submitted by JamieCameron on Mon, 12/19/2011 - 00:12

kfawcett - I'll send you a complete update to the Nginx plugin for Virtualmin.

Which linux distribution are you using there though?

Submitted by chrisfullman on Mon, 12/19/2011 - 12:10

Great news on looking into PHP-FPM!

Jamie, if you could also send me the Nginx plugin update that sets up the restarter script, that'd be great for now. My email is me [at] chrisfullman.com

Thanks!

Submitted by domz on Wed, 12/28/2011 - 16:41

I am also looking at php-fpm.. my php-cgi keeps crashing as well under minimum load... I have a crontab installed to restart it every minute because I am worried my website might be down for longer.

Submitted by bigwombat on Thu, 03/22/2012 - 04:05

Is this issue still active or has it been resolved? I am just trying out Virtualmin Nginx now and have already had to reboot the server as the PHP process terminated and could not be restarted.

Submitted by JamieCameron on Thu, 03/22/2012 - 10:35

Our latest nginx plugin should fix this - but you need to disable and reenable nginx for your domain to switch to the new PHP execution mode.

Submitted by bigwombat on Thu, 03/22/2012 - 14:36

Hi Jamie

How can I verify that the new PHP execution mode is being used? This server was only set up 2 days ago, configuration is: 3.90.gpl and Centos 6.2.

So shouldn't the latest plugin already be used?

Submitted by JamieCameron on Thu, 03/22/2012 - 15:28

Check if the php-loop process is running on your system - if it is, then the new PHP execution mode is in effect.

Submitted by bigwombat on Thu, 03/22/2012 - 15:37

Yes it is running. Should it be running under the permission of the virtualmin, domain administrator?

/usr/bin/perl /usr/bin/php-loop.pl /usr/bin/php-cgi -b 127.0.0.1:9000

Submitted by JamieCameron on Thu, 03/22/2012 - 15:51

Yes, it should be ... and it should have multiple php-cgi sub-processes.

When you see PHP hanging for a domain, is the domain's php-loop.pl process still active?

Submitted by bigwombat on Thu, 03/22/2012 - 16:50

It's happened twice so far. And I've had to reboot the server, as this is very new to me didn't really understand what the heck was going on.

But if it happens again, will report back here.

Submitted by kfawcett on Thu, 03/22/2012 - 17:17 Pro Licensee

Hey Jamie,

Did you ever look into using php-fpm?

Submitted by JamieCameron on Thu, 03/22/2012 - 17:40

I'm looking into it actually ..

Submitted by bigwombat on Thu, 03/22/2012 - 17:51

So guys, in your collective valued opinion. Is Virtualmin + NGinx stable / performant enough as a production platform for websites? Or do you think it is still in Beta, not ready for production use?

Joe's picture
Submitted by Joe on Thu, 03/22/2012 - 18:51 Pro Licensee

It is currently considered beta (as the announcement for the nginx module mentioned; we're nowhere near claiming this is production-level code). Our level of familiarity with nginx, the amount of testing the new module and nginx has received in Virtualmin deployments, and the amount of time it's had to shake out all the bugs, is miniscule compared to Apache.

We're still strongly recommending people use Apache in production deployments, unless you're comfortable with experimentation, and beta software.

It is, of course, being actively developed. And, the more people who use it and report bugs and help figure out solutions, the faster it will reach production quality.

Submitted by bigwombat on Thu, 03/22/2012 - 20:27

Thanks Joe. Would still love to hear feedback from other people who are using it in production. Anyone?

For our use case, we are launching a single website using Virtualmin + Nginx. If things go pear-shaped then yes, we will be ripping Nginx out and replacing with Apache.

We will be performing load testing on the server so that should shake out any stability issues before we go live.

Submitted by iamnewbie on Wed, 03/28/2012 - 02:03

hai Jamie,

how i can upgrade to new nginx plugin? i check on, System -> Software Package Update no nginx available..

"Our latest nginx plugin should fix this - but you need to disable and reenable nginx for your domain to switch to the new PHP execution mode."

sorry for my bad english :D

Submitted by JamieCameron on Wed, 03/28/2012 - 17:17

You can upgrade from the command line by running either :

yum install wbm-virtualmin-nginx

or :

apt-get install webmin-virtualmin-nginx

depending on your Linux distribution.

Submitted by websmurf on Thu, 05/03/2012 - 04:33

@bigwombat

We are using it on a dedicated server with ± 25.000 - 30.000 unique visitors per day (with an adserver on the same server).

It's running quite nicely to be honest, load times have been quite a bit less compared to apache, memory usage as well.

We are using a different php starter though to be able to change PHP_FCGI_CHILDREN and PHP_FCGI_MAX_REQUESTS. For anyone interested: http://pastebin.com/LNxbiBsB