Submitted by dfrm on Sun, 10/29/2017 - 03:21
Hello,
today at 00:41 one of my virtual server was down. When I looked at error logs I found this:
2017/10/29 01:08:40 [error] 27626#0: *1343842 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 66.249.69.122, server: prohormony.sk, request: "GET /sk/14-ph-cykly/hydra-davkovanie-40 HTTP/1.1", upstream: "fastcgi://unix:/var/php-nginx/148916861011674.sock/socket", host: "prohormony.sk"
2017/10/29 01:08:44 [error] 27627#0: *1345857 connect() to unix:/var/php-nginx/148916861011674.sock/socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: 185.91.165.2, server: prohormony.sk, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/php-nginx/148916861011674.sock/socket:", host: "prohormony.sk"
2017/10/29 01:08:46 [error] 27627#0: *1345871 open() "/home/prohormony/public_html/favicon.ico" failed (2: No such file or directory), client: 185.91.165.2, server: prohormony.sk, request: "GET /favicon.ico HTTP/1.1", host: "prohormony.sk", referrer: "https://prohormony.sk/"
2017/10/29 01:09:00 [error] 27626#0: *1343956 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 178.41.27.227, server: prohormony.sk, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/php-nginx/148916861011674.sock/socket", host: "prohormony.sk"
2017/10/29 01:09
and later this
2017/10/29 08:07:28 [error] 10684#0: *12261 connect() to unix:/var/php-nginx/148916861011674.sock/socket failed (11: Resource temporarily unavailable)
there were no changes at the server. It just "happened" I checkd also access logs but there wasnt anything off - not many requests at same time. What could cause this?
Status:
Active
Comments
Submitted by andreychek on Sun, 10/29/2017 - 09:49 Comment #1
Submitted by andreychek on Sun, 10/29/2017 - 09:51 Comment #2
Howdy -- hmm, I'm not quite sure, though it wouldn't hurt to restart Nginx just to be sure it's not a simple issue with it.
Also, is there anything in /etc/security/limits.conf that could be limited the number of processes that either Nginx, or this particular Virtual Server owner, would be allowed to spawn?
Submitted by dfrm on Sun, 10/29/2017 - 14:28 Comment #3
well the first thing I did was restart it was working after that. How can I check that file ?
Submitted by andreychek on Sun, 10/29/2017 - 16:16 Comment #4
Can you run the command "cat /etc/security/limits.conf", and paste in the resulting output?
Submitted by dfrm on Mon, 10/30/2017 - 00:54 Comment #5
#
Each line describes a limit for a user in the form:#
#
Where: can be: - a user name - a group name, with @group syntax - the wildcard *, for default entry - the wildcard %, can be also used with %group syntax, for maxlogin limit - NOTE: group and wildcard limits are not applied to root. To apply a limit to the root user, must be the literal username root.#
can have the two values: - "soft" for enforcing the soft limits - "hard" for enforcing hard limits#
can be one of the following: - core - limits the core file size (KB) - data - max data size (KB) - fsize - maximum filesize (KB) - memlock - max locked-in-memory address space (KB) - nofile - max number of open files - rss - max resident set size (KB) - stack - max stack size (KB) - cpu - max CPU time (MIN) - nproc - max number of processes - as - address space limit (KB) - maxlogins - max number of logins for this user - maxsyslogins - max number of logins on the system - priority - the priority to run user process with - locks - max number of file locks the user can hold - sigpending - max number of pending signals - msgqueue - max memory used by POSIX message queues (bytes) - nice - max nice priority allowed to raise to values: [-20, 19] - rtprio - max realtime priority - chroot - change root to directory (Debian-specific)#
#
* soft core 0 root hard core 100000 * hard rss 10000 @student hard nproc 20 @faculty soft nproc 20 @faculty hard nproc 50 ftp hard nproc 0 ftp - chroot /ftp @student - maxlogins 4 End of filewww-data hard cpu -1 www-data hard nproc -1 www-data hard as -1
Submitted by dfrm on Mon, 10/30/2017 - 00:54 Comment #6
btw today it happened again at the same time
Submitted by andreychek on Mon, 10/30/2017 - 08:55 Comment #7
Your limits.conf file appears to be normal there.
Did this issue just begun occurring recently?
Had the site been online for awhile prior to that?
Are other sites working okay while this one site is down?
And do you know anything that may have been changed, or anything occurring in cron at this time of day?
Submitted by dfrm on Mon, 10/30/2017 - 10:05 Comment #8
Hi,
it happened now too. The website is online for 7 months. Other sites are working ok. Nothing was changed there. I have only 1 cron in midnight
Submitted by andreychek on Mon, 10/30/2017 - 12:07 Comment #9
Is Nginx by chance configured to do proxying?
If so, does this here help:
https://stackoverflow.com/questions/10395807/nginx-close-upstream-connec...
Submitted by dfrm on Thu, 11/02/2017 - 15:44 Comment #10
no proxying.
btw I changed PHP to FPM and am getting these errors:
2017/11/02 21:34:37 [error] 27700#0: *30639 connect() failed (111: Connection refused) while connecting to upstream, client: 185.42.4.167, server: prohormony.sk, request: "GET /admin HTTP/1.1", upstream: "fastcgi://[::1]:8001", host: "prohormony.sk" 2017/11/02 21:34:53 [error] 27700#0: *30639 connect() failed (111: Connection refused) while connecting to upstream, client: 185.42.4.167, server: prohormony.sk, request: "POST /admin/produkty/filter HTTP/1.1", upstream: "fastcgi://[::1]:8001", host: "prohormony.sk", referrer: "https://prohormony.sk/admin/produkty/zobrazit" 2017/11/02 21:35:22 [error] 27698#0: *30738 connect() failed (111: Connection refused) while connecting to upstream, client: 185.42.4.167, server: prohormony.sk, request: "GET /ajax/front?term=tren&_type=query&q=tren HTTP/1.1", upstream: "fastcgi://[::1]:8001", host: "prohormony.sk", referrer: "https://prohormony.sk/" 2017/11/02 21:35:23 [error] 27700#0: *30648 connect() failed (111: Connection refused) while connecting to upstream, client: 185.42.4.167, server: prohormony.sk, request: "POST /search HTTP/1.1", upstream: "fastcgi://[::1]:8001", host: "prohormony.sk", referrer: "https://prohormony.sk/"
Why does it says fastcgi when the php is set as FPM ? are those ppl seeing the website or some fucked up error?
Submitted by dfrm on Thu, 11/02/2017 - 15:49 Comment #11
Also I have this in RAM processes again php fpm + castcgi wtf????
3746 prohormony 467.87 MB php-fpm: pool 1489168610116743856 prohormony 466.15 MB php-fpm: pool 148916861011674
30364 prohormony 432.06 MB php-fpm: pool 148916861011674
3817 prohormony 431.46 MB php-fpm: pool 148916861011674
30359 prohormony 425.99 MB php-fpm: pool 148916861011674
2225 prohormony 356.50 MB /usr/bin/php5-cgi -b /var/php-nginx/149978262724973.sock/socket
2257 prohormony 356.50 MB /usr/bin/php5-cgi -b /var/php-nginx/149978262724973.sock/socket
2259 prohormony 356.50 MB /usr/bin/php5-cgi -b /var/php-nginx/149978262724973.sock/socket
2261 prohormony 356.50 MB /usr/bin/php5-cgi -b /var/php-nginx/149978262724973.sock/socket
2263 prohormony 356.50 MB /usr/bin/php5-cgi -b /var/php-nginx/149978262724973.sock/socket
2217 prohormony 25.09 MB /usr/bin/perl /usr/bin/php-loop.pl /usr/bin/php5-cgi -b /var/php-nginx/149978262 ...
this software is rly a mess...
Submitted by dfrm on Thu, 11/02/2017 - 15:52 Comment #12
well those errors logsa re for my IP and I can access the webdite ok. It seems like that there are 2 php's running fpm and fcgi. FPM is working but ti wants to load fastcgi too but gets error? Or what the hell?