(104)Connection reset by peer: [client x.x.x.x:53302] mod_fcgid: error reading data from FastCGI server

11 posts / 0 new
Last post
#1 Thu, 01/08/2015 - 13:46
srmobile

(104)Connection reset by peer: [client x.x.x.x:53302] mod_fcgid: error reading data from FastCGI server

I've been batling this problem since moving to CentOs 7 and the latest version of Virtualmin:

I have a script that runs in FCGId (run as virtual server owner), in CentOs 6.4 there was no issue.

The script will either run for hours or crash right away, there is absolutely no logic, completely random.

The errors:

[fcgid:warn] [pid 20985] (104)Connection reset by peer: [client x.x.x.x:53302] mod_fcgid: error reading data from FastCGI server, referer: https://www.site.com/index.php/myscript/key/322f92b70aa3c7b5fe781426b2b6... [core:error] [pid 20985] [client1x.x.x.x:53302] End of script output before headers: index.php, referer: https://www.site.com/index.php/myscript

I've reset all permissions, chown to the entire directory structures.

My httpd.conf

ServerName mysite.com ServerAlias www.mysite.com ServerAlias webmail.mysite.com ServerAlias admin.mysite.com DocumentRoot /home/mysite/public_html ErrorLog /var/log/virtualmin/mysite.com_error_log CustomLog /var/log/virtualmin/mysite.com_access_log combined ScriptAlias /cgi-bin/ /home/mysite/cgi-bin/ ScriptAlias /awstats/ /home/mysite/cgi-bin/ DirectoryIndex index.html index.htm index.php index.php4 index.php5 Options -Indexes +IncludesNOEXEC +SymLinksIfOwnerMatch +ExecCGI allow from all AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,MultiViews,SymLinksIfOwnerMatch Require all granted AddType application/x-httpd-php .php AddHandler fcgid-script .php AddHandler fcgid-script .php5 FCGIWrapper /home/mysite/fcgi-bin/php5.fcgi .php FCGIWrapper /home/mysite/fcgi-bin/php5.fcgi .php5 allow from all AllowOverride All Options=ExecCGI,Includes,IncludesNOEXEC,MultiViews,SymLinksIfOwnerMatch Require all granted RewriteEngine on RewriteCond %{HTTP_HOST} =webmail.mysite.com RewriteRule ^(.) https://mysite.com:20000/ [R] RewriteCond %{HTTP_HOST} =admin.mysite.com RewriteRule ^(.) https://mysite.com:10000/ [R] Alias /dav /home/mysite/public_html Alias /pipermail /var/lib/mailman/archives/public DAV on AuthType Basic AuthName "mysite.com" AuthUserFile /home/mysite/etc/dav.digest.passwd Require valid-user ForceType text/plain Satisfy All RemoveHandler .php RemoveHandler .php5 RewriteEngine off AuthName "mysite.com statistics" AuthType Basic AuthUserFile /home/mysite/.awstats-htpasswd require valid-user RedirectMatch /cgi-bin/mailman/([^/.])(.cgi)?(.) https://mysite.com:10000/virtualmin-mailman/unauthenticated/$1.cgi$3 RedirectMatch /mailman/([^/.])(.cgi)?(.) https://mysite.com:10000/virtualmin-mailman/unauthenticated/$1.cgi$3 FcgidBusyTimeout 3600

other Fcgid directives...

SSLEngine on SSLCertificateFile /home/mysite/ssl.cert SSLCertificateKeyFile /home/mysite/ssl.key SSLCACertificateFile /home/mysite/ssl.ca Options -Indexes RemoveHandler .php RemoveHandler .php5 php_admin_value engine Off IPCCommTimeout 9999 FcgidMaxRequestLen 1073741824 SuexecUserGroup "#502" "#502"

I've tried adding:

FcgidBusyTimeout 14400 FcgidIOTimeout 9999 FcgidConnectTimeout 14400 FcgidProcessLifeTime 0

other Fcgid directives...

But that doesn't do anything.

My /etc/httpd/conf.d/fcgid.conf

Use FastCGI to process .fcg .fcgi & .fpl scripts

AddHandler fcgid-script fcg fcgi fpl

Sane place to put sockets and shared memory file

FcgidIPCDir /run/mod_fcgid FcgidProcessTableFile /run/mod_fcgid/fcgid_shm FcgidBusyTimeout 3600 FcgidIOTimeout 3600 FcgidConnectTimeout 3600 FcgidIdleTimeout 3600 FcgidMaxRequestsPerProcess 5000

php.ini also has timeouts set extremely high, if you need to see that and my.cnf let me know.

What could be wrong here? Is there something obvious I'm missing?

Thu, 01/08/2015 - 14:00
andreychek

Howdy,

Sometimes FCGID can hide the actual error message... I'd be curious if changing the PHP Execution Mode to "CGI" makes the error more apparent.

You can do that in Server Configuration -> Website Options.

-Eric

Fri, 01/09/2015 - 08:54
srmobile

I'm getting 504 Gateway timeouts when I switch to CGI... Before this it wold pop up occupationally but not this much.

My php.ini has the max execution time set to 3600 in both /etc/php.ini and in my /home/domain/etc/php.ini

Does this give you any hints?

Fri, 01/09/2015 - 16:05
srmobile

After changing the FCGID from unlimited to setting a timeout value of 20000, I am now getting:

[Fri Jan 09 16:50:00.093819 2015] [fcgid:warn] [pid 26063] (104)Connection reset by peer: [client x.x.x.x:9784] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

Fri, 01/09/2015 - 17:45
andreychek

Howdy,

Try switching it to CGI, and then setting it to unlimited as well -- what error do you see at that point? Do you still see a Gateway timeout error?

The error you're seeing now is specific to FCGID, it should be different with CGI (if it errors at all).

Also, some other folks here in the Forums have seen that error when using FCGID, one example of how to correct that is here:

https://www.virtualmin.com/node/32108

Sat, 01/10/2015 - 07:58
srmobile

I've switched back and forth, tried everything I came across including that link. CentOS 6.5 no issues, CentOs 7 crashes randomly with the error messages provided.

My PHP_FCGI_MAX_REQUESTS was set to 9999

Is there any other setting that we might be overlooking?

Is it worth mentioning that my entire site works in https (port 443) mode? Could that be a reason?

Thanks Eric,

SR

EDIT: I've turned on extended logging now I have:

Sat Jan 10 08:34:15.529186 2015] [cgi:warn] [pid 7569] [client x.x.x.x:49355] AH01220: Timeout waiting for output from CGI script /home/my-site/cgi-bin/php5.cgi, referer: https://www.mysite.com/index.php/admin/icecatimport/adminhtml_scan/ajaxa... [Sat Jan 10 08:34:32.285404 2015] [cgi:warn] [pid 7562] [client x.x.x.x:7330] AH01220: Timeout waiting for output from CGI script /home/my-site/cgi-bin/php5.cgi [Sat Jan 10 08:34:32.285485 2015] [cgi:error] [pid 7562] [client x.x.x.x:7330] Script timed out before returning headers: php5.cgi

Tue, 01/13/2015 - 12:25
srmobile

NOT solved....

Tue, 01/13/2015 - 13:05
andreychek

Is this meant to be a long-running PHP script? Or is it just a standard web page?

I'm curious if it helps to bump up the PHP timeout. You can do that by going into Server Configuration -> Website Options, and there, you can tweak "Maximum PHP script run time".

Try setting it to something larger, such as "90". I'd recommend choosing a larger number, and not setting it to unlimited.

After doing that, does the page load properly?

-Eric

Wed, 01/14/2015 - 06:23
srmobile

Thanks for the reply Eric, this has been ongoing for quite sometime.

I had someone look at my config and he couldnt find anything wrong other than suggesting to add PHP_FCGI_CHILDREN=5 to my sites php5.fcgi file.

That on it's own didn't solve the problem.

Last night I was looking at my php info output and saw APC and decided to try and disable it...... The server hasn't worked so well! It is faster and there are no longer any errors.

I guess since APC isn't part of the out of the box configuration this made it difficult for anyone to figure out what was causing the connection TimeOuts.

Really appreciate the support!

To all those looking for a resolution to this error and have gone through all the obvious settings: DISABLE APC.

Thu, 03/12/2015 - 04:48 (Reply to #9)
tcadmin

I had the same problem and disabling APC was the fix. Several different errors I was having disappeared. So thanks!!

I wanted to confirm, since you speculated earlier, that the problem only occurs with HTTPS.

I was debugging a Wordpress installation that only enforced https when logged in. I noticed the errors were only occuring when logged in. I further confirmed later, that simply viewing the site using https, not logged in, also caused the errors.

So if anyone is trying to reproduce this and fix it https use is key.

//TC

Wed, 08/26/2015 - 05:09
nibb

Do you have a cache? APC, Xcache?

Try disabling it.

I think the issue is related to suexec and permissions, CentOS7+Virtualmin does work strange regarding PHP, at least I'm hitting with more issues than before.

I also run everything under SSL so that could be related. No issues in CentOS6, issues on CentOS 7.

Topic locked