hellz yeah jaysongayson keep on hammerin the nginx

I forgot to mention something very serious.

Your installation instructions say to just issue the command "yum install nginx"

The problem is that this pulls nginx from the virtualmin repository, and it's SEVERELY outdated in there. At the time of writing, that repo contains "1.0.10-1.el6.vm.ngx"

Instead, you should instruct users to install the official NGINX repo for nginx-stable, which will allow proper auto-updates whenever security fixes are released.

They have RHEL 5/6 and CentOS 5/6 repositories as well as a APT repo for Debian/Ubuntu at this link: http://nginx.org/en/download.html

The latest stable as of this writing is nginx-1.2.6-1.el6.ngx

If you check the latest-stable changelog at http://nginx.org/en/CHANGES-1.2 (and http://nginx.org/en/CHANGES-1.0) you see how serious this is.

1.0.10 was released on Nov 15th, 2011.

1.2.6 was released on Dec 11th, 2012.

There has been a STAGGERING amount of security patches since then.

Please stop serving nginx from your own repo and instruct people to install the official stable repo instead. People don't deserve to be hacked because they have an outdated version that they were told to install. ;-)

Here are my brutally honest thoughts on the "features" we lose by abandoning Apache:

• WebDAV Login and Subversion repositories: We have no mod_dav_svn since we have no more Apache, but that's a fair tradeoff. We could install a custom Git server, because only idiots use Subversion.

• Mailman: Sure, the mailman scripts are in perl format and thus incompatible with NGINX, but almost nobody runs a mailman mailing list unless they were born before the 90s. There are better solutions such as bulletin boards (forums) and other mailing list handlers if a mailing list is really required.

• Protected web directories (htaccess): These can be configured in NGINX via per-directory statements in the configuration for the affected domain, and I see that the Virtualmin-NGINX module is already capable of doing this. Excellent.

• AWstats reporting: True, these run as perl scripts, but it's possible to schedule a cron job that chunks the NGINX logs by day and generates static .htm pages from that via AWstats, thereby working around that limitation. See http://blog.cykerway.com/post/264

And here is what we gain:

• No more .pl/.cgi: That's a security BONUS. Restricting executable code to PHP files via FastCGI running as a restricted user is a major win for security.

• Predictable performance and scalability: Goodbye, bloated Apache! http://wiki.dreamhost.com/Web_Server_Performance_Comparison and http://wiki.nginx.org/WhyUseIt

As for AWstats, it's better to run http://piwik.org which is a LOT more capable and rivals Google Analytics.

The benefits are massive, and I don't see any of this as drawbacks. The .htaccess stuff can be written into the NGINX per-site configurations instead. AWstats can be scheduled as a cron job that outputs static HTML or you can move to a different PHP-based statistics script such as Piwik or even Google Analytics instead. Everything else that we lose is archaic junk.

Best of all: Your server can handle sudden viral popularity; NGINX is ready to handle any amount of load, whereas Apache slows down after a few hundred simultaneous connections and dies at around 1000. There's a reason that all of the world's largest sites are migrating to NGINX. ;-) http://w3techs.com/technologies/cross/web_server/ranking

With Apache, the Virtualmin configuration module has the option to disable version information by setting "Server HTTP header" to "Product only," turning "Apache/2.2.15 (CentOS)" into just "Apache" in all error pages/headers.

NGINX needs a similar option, preferably placed under "Other Settings."

Here's the equivalent option: http://wiki.nginx.org/HttpCoreModule#server_tokens

NGINX has come to stay. It already powers A THIRD of the world's largest websites (http://w3techs.com/technologies/cross/web_server/ranking), and it will not stop until Apache is completely dead. Moreover, it's a perfect fit for Virtualmin, since its extremely low memory and CPU usage makes it perfect for serving thousands of domains from a single machine.

Hey, at least I am providing actionable todo-items, rather than just "wouldn't it be nice?" requests. I'm actually giving you references to show how to implement each item.

And this brings me to the next item on the list:

• When using NGINX, Virtualmin starts PHP-FastCGI processes per-user. That's good. The issue is that any changes to php.ini for the sub-site (such as /home/somesite/etc/php5/php.ini) requires you to go into "Bootup and Shutdown," ticking the "php-fcgi-sitename" process, and restarting it.

• Well, you would think so. But actually, this leads nginx to complain about "Bad Gateway" since it has lost the connections to the FastCGI worker. This cannot be solved by simply restarting nginx. The entire machine must be restarted.

• Therefore, it would be nice if the "Webmin: Servers: Nginx Webserver" module had an extra button to "Restart PHP FastCGI processes." Could be placed below the "Apply Nginx Configuration" button. This button would restart all workers for every site and re-connect Nginx to them properly.

Being able to properly restart all FastCGI helpers in one go to apply configuration changes without a machine reboot is a really vital feature.

Edit: I just remembered a possible reason why this happens! Sockets, when closed, enter a "CLOSE_WAIT" period. By simply issuing a restart of the "php-fcgi-sitename" process, we basically give up the old port, that port enters the "CLOSE_WAIT" grace period, and we then start the fcgi daemon again. But when it starts up again, the desired port is "used" (because it's still in "CLOSE_WAIT" which prevents anything from binding to it), so the daemon binds to a different port. That would explain why Nginx cannot find the fcgi daemon on the expected port anymore. The solution is to make the fcgi process bind the socket using SO_REUSEADDR (http://www.unixguide.net/network/socketfaq/4.5.shtml). I am not SURE that this is why it's failing, but it might be the reason. Either way, until this is fixed I'm resorting to full system restart any time I change a php.ini...

Overall, though, you've done an excellent job with the NGINX module so far. I'm just addressing the shortcomings.

Oh so it IS possible to switch the site conf storage in your module! That's beautiful!

That was literally the largest annoyance I had. I'm switching over right away!

How about adding this to the Virtualmin-NGINX module's init-script:

• default to using directories rather than the main config file (so that people have to forcefully tell it to use the messier single config file if they really want that)

• if directories are enabled, check if /etc/nginx/sites-enabled or /etc/nginx/sites-available exist, otherwise create them with proper user/group and permissions: I tested its behavior and see that it breaks if the folders don't exist ahead if tune; it creates a FILE called "sites-available" with the server-block of the added site, instead of creating a folder.

• and add "include /etc/nginx/sites-enabled/*;" to the bottom of nginx.conf's http{} block if it doesn't already exist (http://nginx.org/en/docs/ngx_core_module.html#include): It doesn't do this, but it definitely should.

Everybody uses those folders for management, even if it's true that they aren't created by default on some distros.

It's just untenable to manage multiple sites in one large, messy config file.

You really won't find a single intelligent user that still uses the main config file. Therefore it's the right choice to enable the excellent per-site conf folder structure by default.

This also has another benefit:

• If the Virtualmin-NGINX module sees that it has been configured to use directories, it could have enable/disable site toggle links that simply add/remove the symlinks. If directories had been disabled, this would require messy surgery on the main config file instead.

As for the rest of the reports, take your time. There's no rush. Apart from the PHP.ini/FastCGI worker issue, they're all non-critical, and even that one is pretty okay because php.ini is not modified every day.

I'm just here to help you make the NGINX module better in Virtualmin. It really is the next generation of web hosting. Never before was it possible to run as many sites on a single machine as it is now. It's the perfect fit for Virtualmin.

When using the automatic "Install Scripts" to install Roundcube (which I'm really only using in order to easily keep it up to date, hey why not use it if it's there!), it used to add a fastcgi worker timeout of 60 seconds to the site config in this step:

When the configurations have been moved to the per-site directory, it fails to apply that configuration line.

Just another friendly report. ;-)

Another, minor issue which I forgot to mention is that initially, one must enable "Nginx website" under "Features and Plugins" and then Save, followed by going back and enabling "Nginx SSL website." One cannot enable both at once. The reason is that the dependency of the SSL module cannot be met because it comes higher in the list of modules, so the core "website" module has to be enabled separately first.

This reveals a deeper problem in Webmin/Virtualmin which is that dependency checking when enabling modules is a bit broken. It shouldn't check "currently enabled plugins," since the user is clearly trying to enable both plugins.

It should check "currently enabled plugins + ones queued to be enabled." Or better yet, for security (making sure everything gets enabled properly), make it batch things so that modules are enabled in the dependency order, so that despite SSL coming earlier in the list, it's held off until the regular module has been activated.

Here is some more info on SSL certificates:

• Under the current setup, where Virtualmin creates ssl_certificate directives INSIDE each server{} block, but not OUTSIDE (aka no "default" certificate), I told you earlier that I didn't know what would happen if the client doesn't support SNI. Would it serve a default certificate? How would it determine the default?

• I have the answer now. YES, it ALWAYS serves a default certificate, and it finds that default cert by selecting the domain marked "DEFAULT" for that port (443), or if none of the domains are marked as default, it picks one using some internal algorithm that I have no clue about (wasn't based on file modtime or alphabetical, I checked both). The point is that it ALWAYS returns a certificate.

• You can check the DEFAULT certificate returned when SNI is not supported, by issuing the command "openssl s_client -host 10.0.0.201 -port 443"

• There is one extremely important thing, though: Virtualmin-NGINX writes incorrect server blocks for anything but the first SSL server! Put simply, it writes the first server as "listen 10.0.0.201:443 default ssl;" and subsequent sites as "listen 10.0.0.201:443;"

• The result is that you are telling NGINX to bind port 443 as HTTPS in the FIRST server's directive, but ALL other servers are told to bind it as HTTP. This is FINE and ALL sites will work via HTTPS as long as you KEEP that first server forever. But if you ever delete the old server, then port 443 reverts to binding via HTTP protocol on NGINX startup!

Here is how to fix the Virtualmin-NGINX implementation:

• Write the first ever added site as "listen 10.0.0.201:443 default ssl;"

• Write all subsequent sites as "listen 10.0.0.201:443 ssl;"

• Allow users to change the "default" site for the IP/port combo, by moving the default flag around between sites.

• Always default to enabling SSL for new domains; it's perfectly viable when running NGINX. It gives a great benefit, since every domain CAN have individual SSL certificates. The only major userbase that doesn't support SNI is IE8 and below on Windows XP, and they will receive the default certificate instead, which is better than nothing. OR if the requested ip/port combo had only one site on it, then they get the correct certificate, in other words PERFECT operation no matter the usage case! And no users will be without SSL, ever.

• Allow the user to install real UCC / wildcard certificates and use this method to install them globally http://wiki.nginx.org/NginxHttpSslModule#Using_Wildcard_certificates_wit... (basically, putting them outside all server blocks and not defining certs inside the domains that you want to use the default certificate for).

There's another SSL-related issue with Virtualmin-NGINX, but I'll cover that in the next post. Keeping everything in one ticket is great, but keeping it all in one post is too messy! ;-)

Alright, so here's the issue that needs to be addressed Virtualmin-NGINX:

• NGINX defaults to ultra-secure, AES256 ciphers with Diffie-Hellman. See http://wiki.nginx.org/HttpSslModule#ssl_ciphers (it defaults to "ssl_ciphers HIGH:!aNULL:!MD5;" which in turn mean the highest possible security).

• That is great from a security standpoint and is a fantastic default, it shows that the author of NGINX cares a lot about security. Just one issue: 99.999999% of sites don't need THAT level of security. Even AES128 takes 1 billion billion (that is not a typo) years to crack (http://www.eetimes.com/design/embedded-internet-design/4372428/How-secur...)

• The major issue with keeping the default ciphers is this: When using the DHE-RSA-AES256-SHA cipher, you can only expect to handle about 90 requests per second! EACH request will be forced to re-negotiate the super-heavy cipher and this is HORRIBLE for performance.

• Other SSL tunnel implementations default to AES128 and without Diffie-Hellman key negotiation, allowing them to handle more in the range of 6000 requests per seconds.

• Compare 90 vs 6000, and you see that NGINX's default SSL choices are WAY too slow. They are ultra-paranoid and should only be used for military websites. Not even credit card shopping sites should be using THAT high security.

• Here are the performance implications of the NGINX default: http://matt.io/technobabble/hivemind_devops_alert:_nginx_sucks_at_ssl/uq

• Here is how to solve it: http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_a...

In other words, the Virtualmin-NGINX module needs a new "SSL Security" tab, and in it, this option:

This really needs to be in there, because out of the box with the paranoid SSL settings it handles less than 100 requests per seconds, vs thousands with a configuration improvement. No other SSL tunnels are as paranoid as NGINX and its defaults are way too high for almost everyone. ;-)

I hope you don't see me as spamming, by the way. I am just very serious about helping you make a NGINX configuration module that works properly and is production-ready. I am providing sources for everything so that you can fix these things without having to spend a lot of time researching the issues yourself.

None of this is anything that I am saying "fix now!" - I am just bringing everything to your attention so that you are aware and can choose to improve the NGINX module.

You want to see how serious I am about helping?

I just installed Windows XP + IE 6 in a Virtual Machine and connected to NGINX.

NGINX was set up as follows:

I connected to https://test.com

The certificate I received was (as expected, since there's no SNI support in IE 6-8 for Windows XP) the one for abc.com. I repeated the test after installing IE 8 to verify the same behavior there.

Now consider how extremely rare it is to find a browser without SNI support these days (http://www.ie6countdown.com, it's down to sub-1% in every country that matters). Simply installing Firefox or Chrome on XP gives even ancient XP users access to SNI. I verified this using https://sni.velox.ch/ which is a test service for that purpose.

All modern browsers, meaning nearly 100% of visitors, support SNI and will get the proper certificate. The other people living in the past? They get the default certificate and a security warning (if it's not a UCC / wildcard certificate, in which case the default is fine). They just have to accept the certificate and they'll get SSL too.

There's no reason not to enable SSL by default in NGINX and to remove the warnings when adding new virtual hosts.

All virtual hosts should be set up as "listen 10.0.0.201:443 ssl;"

And it should be possible to move the "default" ("listen 10.0.0.201:443 default ssl;") flag around via the Virtualmin GUI to choose which server / certificate is the default for that IP/Port combo.

And remember - anyone that sets up a unique IP/port combo is of course immune and will always deliver the correct certificate even without SNI. There's just NO excuse not to have SSL on every site with NGINX, EVEN with just 1 IP/Port combo.

So that settles that! ;-)

TLS/SNI support can be checked with "nginx -V" and will say "TLS SNI support enabled." This will be the case for any modern server. Even if SNI was not enabled you should STILL support multiple SSL sites on the same IP/Port in Virtualmin, with the one and only simple caveat that the default site's certificate will always be the one delivered for that ip/port combo.

I forgot to mention this earlier:

Under Other Settings, you provide access over the total number of workers. That's good. But an equally important setting is missing:

The default of 1024 is really low. It's really only set that way to avoid exhausting the available port-pool of the system for people that don't know better (i.e. 4 workers x 1024 = 4096 sockets can be open at a time), and because most people don't serve heavy sites that need more. However, a single worker could happily serve worker_connections set as high as 10000 or 20000. The event-based loop system has no problem doing this.

That'd be quite extreme though, and most heavy production systems have this set to around 2048 or 4096.

Why am I saying this? Well, simply to make you aware of why people would want to edit that option. It is such a common configuration option that it needs to be in the Other Settings area. ;-)

"Wow, this is a massive bug.. and it's hard to keep track of all the feature requests here."

Indeed. I was very torn between either 14 different tickets or a single ticket where I try to limit things to 1 or 2 items per post.

I went for the latter; a single ticket, simply to keep it all in one place - "NGINX module improvements."

Most things are short fixes, such as adding extra configuration GUI options or fixing default values for new virtualhosts.

So, in the end, while this is indeed a very long ticket, I think it's best if it's all kept in one place, where you can go through things one by one and then just close the whole ticket. It's not that hard to read from top to bottom and remember which ones you've already fixed, as you work through this list.

I did try to split things by post, and whenever I covered two things in one post, I made sure to not intermix them, by first describing issue 1, and then issue 2 after that in a new paragraph/section.

I just re-read everything and I can't find a single item that would take so much time that it warrants its completely own ticket, where it would just be harder to track in the noise of all the other tickets. Simpler to keep it all in one place when it's a bunch of mostly tiny, related improvements to a single module like this.

As for wanting to preserve vendor defaults, I can completely understand that. That's why I suggested a "Default/High/Custom" radio-button for things like the SSL cipher level, which most definitely HAS to be tweaked on NGINX systems unless admins like seeing their server's reqs handled/second drop from 14000 to 90, hehe. As long as there's a way to configure away from "bad defaults" via the GUI, then it won't matter what the vendor defaults are.

Yeah I found that setting later. Way after I had already applied my own patches to the perl files to skip the checks in all locations.

I am intricately familiar with the Webmin codebase (you can see the number of bug submissions I just posted in every section) and I don't envy your job of maintaining such a crazy rat's nest of archaic spaghetti code. At least you are doing a great job with newer modules.

As for this ticket, I don't have time to split it into multiple tickets, but either of you are free to do so if you really want to.