hellz yeah jaysongayson keep on hammerin the nginx

Currently, the Virtualmin NGINX module spits all virtualhosts into /etc/nginx/nginx.conf.

That's VERY bad practice and is a maintenance nightmare.

There's a built-in mechanism in place for virtual hosts.

/etc/nginx/sites-available: this is where virtual host config files are stored

/etc/nginx/sites-enabled: this is where symbolic links (or file shortcuts) to the virtual host config files are placed

Here's an example /etc/nginx/sites-available/example.com.conf

server_name example.com www.example.com;
... all that per-server stuff you are currently spitting into the main config file...

To enable the config, just:

ln -s /etc/nginx/sites-available/example.com.conf /etc/nginx/sites-enabled/example.com.conf

Then restart NGINX.

See how much nicer that is, compared to spitting everything into the main daemon config file?

Please fix this, seriously. ;-)

One more thing when it comes to NGINX and SSL: http://www.virtualmin.com/documentation/web/nginx says that "Only one virtual server can have SSL enabled per IP address, even if a wildcard or UCC certificate would potentially allow multiple SSL sites to share the same IP."

This is incorrect. If you set up multiple servers on the same port but with different certificates, then NGINX uses SNI (Server Name Identification) to determine which certificate to send and will send the proper one. All modern browsers support SNI. If the browser doesn't understand SNI, then it might use a random one, or none at all, I haven't tested (only WinXP users running Internet Explorer would be affected).

However, disregard that and listen to me when it comes to the proper setup:

In /etc/nginx/nginx.conf:
ssl_certificate /my/awesome/default.cert;
ssl_certificate_key /my/awesome/default.key;

In /etc/nginx/sites-available/example.com.conf:
server {
... listen on port 80 and 443 (SSL).
ssl_certificate /home/example/ssl.cert;
ssl_certificate_key /home/example/ssl.key;

With this setup, it works as follows: If it cannot determine the hostname via SNI, it will give the client "/my/awesome/default.cert". If it received a hostname via SNI, it gives the client "/home/example/ssl.cert"

AND, as an added bonus: If a virtual server is bound to a completely unique IP/port combo, then it serves THAT certificate regardless of SNI status.

Here's more information: http://wiki.nginx.org/NginxHttpSslModule#Using_Wildcard_certificates_wit...

So, did you catch all that? It means:

  • Make the Virtualmin-NGINX module write each virtual host to individual config files under /etc/nginx/available-sites and symlink them to /etc/nginx/enabled-sites
  • Make the Virtualmin-NGINX module write a default certificate to the /etc/nginx/nginx.conf file and allow the user to choose which certificate should be the default written to this file
  • Remove the warnings about sharing SSL certificate because they're silly (in the module)
  • Fix the NGINX documentation at http://www.virtualmin.com/documentation/web/nginx since it's wrong about SSL, even in the current state of the module (SSL on same IP:Port is fine even today).

These are relatively small changes but would fix Virtualmin's NGINX support and make it a lot better, pretty much a first class citizen comparable to Apache. So, whadd'ya say? Is it worth spending half an hour fixing these things in the Virtualmin-NGINX module and uploading a new version of the package? I'd say so. I'd definitely say so. ;-)

It won't even break existing setups, if you just make an install/upgrade script that runs by default on installing the newer Virtualmin-NGINX module, which scans /etc/nginx/nginx.conf for the current, messy way that you are defining virtualhosts, and exports them to proper "site" config files as described above. So, add another half hour for that and call it a job well done!



I forgot to mention something very serious.

Your installation instructions say to just issue the command "yum install nginx"

The problem is that this pulls nginx from the virtualmin repository, and it's SEVERELY outdated in there. At the time of writing, that repo contains "1.0.10-1.el6.vm.ngx"

Instead, you should instruct users to install the official NGINX repo for nginx-stable, which will allow proper auto-updates whenever security fixes are released.

They have RHEL 5/6 and CentOS 5/6 repositories as well as a APT repo for Debian/Ubuntu at this link: http://nginx.org/en/download.html

The latest stable as of this writing is nginx-1.2.6-1.el6.ngx

If you check the latest-stable changelog at http://nginx.org/en/CHANGES-1.2 (and http://nginx.org/en/CHANGES-1.0) you see how serious this is.

1.0.10 was released on Nov 15th, 2011.

1.2.6 was released on Dec 11th, 2012.

There has been a STAGGERING amount of security patches since then.

Please stop serving nginx from your own repo and instruct people to install the official stable repo instead. People don't deserve to be hacked because they have an outdated version that they were told to install. ;-)

Here are my brutally honest thoughts on the "features" we lose by abandoning Apache:

  • WebDAV Login and Subversion repositories: We have no mod_dav_svn since we have no more Apache, but that's a fair tradeoff. We could install a custom Git server, because only idiots use Subversion.

  • Mailman: Sure, the mailman scripts are in perl format and thus incompatible with NGINX, but almost nobody runs a mailman mailing list unless they were born before the 90s. There are better solutions such as bulletin boards (forums) and other mailing list handlers if a mailing list is really required.

  • Protected web directories (htaccess): These can be configured in NGINX via per-directory statements in the configuration for the affected domain, and I see that the Virtualmin-NGINX module is already capable of doing this. Excellent.

  • AWstats reporting: True, these run as perl scripts, but it's possible to schedule a cron job that chunks the NGINX logs by day and generates static .htm pages from that via AWstats, thereby working around that limitation. See http://blog.cykerway.com/post/264

And here is what we gain:

As for AWstats, it's better to run http://piwik.org which is a LOT more capable and rivals Google Analytics.

The benefits are massive, and I don't see any of this as drawbacks. The .htaccess stuff can be written into the NGINX per-site configurations instead. AWstats can be scheduled as a cron job that outputs static HTML or you can move to a different PHP-based statistics script such as Piwik or even Google Analytics instead. Everything else that we lose is archaic junk.

Best of all: Your server can handle sudden viral popularity; NGINX is ready to handle any amount of load, whereas Apache slows down after a few hundred simultaneous connections and dies at around 1000. There's a reason that all of the world's largest sites are migrating to NGINX. ;-) http://w3techs.com/technologies/cross/web_server/ranking

With Apache, the Virtualmin configuration module has the option to disable version information by setting "Server HTTP header" to "Product only," turning "Apache/2.2.15 (CentOS)" into just "Apache" in all error pages/headers.

NGINX needs a similar option, preferably placed under "Other Settings."

Here's the equivalent option: http://wiki.nginx.org/HttpCoreModule#server_tokens

NGINX has come to stay. It already powers A THIRD of the world's largest websites (http://w3techs.com/technologies/cross/web_server/ranking), and it will not stop until Apache is completely dead. Moreover, it's a perfect fit for Virtualmin, since its extremely low memory and CPU usage makes it perfect for serving thousands of domains from a single machine.

Hey, at least I am providing actionable todo-items, rather than just "wouldn't it be nice?" requests. I'm actually giving you references to show how to implement each item.

And this brings me to the next item on the list:

  • When using NGINX, Virtualmin starts PHP-FastCGI processes per-user. That's good. The issue is that any changes to php.ini for the sub-site (such as /home/somesite/etc/php5/php.ini) requires you to go into "Bootup and Shutdown," ticking the "php-fcgi-sitename" process, and restarting it.

  • Well, you would think so. But actually, this leads nginx to complain about "Bad Gateway" since it has lost the connections to the FastCGI worker. This cannot be solved by simply restarting nginx. The entire machine must be restarted.

  • Therefore, it would be nice if the "Webmin: Servers: Nginx Webserver" module had an extra button to "Restart PHP FastCGI processes." Could be placed below the "Apply Nginx Configuration" button. This button would restart all workers for every site and re-connect Nginx to them properly.

Being able to properly restart all FastCGI helpers in one go to apply configuration changes without a machine reboot is a really vital feature.

Edit: I just remembered a possible reason why this happens! Sockets, when closed, enter a "CLOSE_WAIT" period. By simply issuing a restart of the "php-fcgi-sitename" process, we basically give up the old port, that port enters the "CLOSE_WAIT" grace period, and we then start the fcgi daemon again. But when it starts up again, the desired port is "used" (because it's still in "CLOSE_WAIT" which prevents anything from binding to it), so the daemon binds to a different port. That would explain why Nginx cannot find the fcgi daemon on the expected port anymore. The solution is to make the fcgi process bind the socket using SO_REUSEADDR (http://www.unixguide.net/network/socketfaq/4.5.shtml). I am not SURE that this is why it's failing, but it might be the reason. Either way, until this is fixed I'm resorting to full system restart any time I change a php.ini...

Overall, though, you've done an excellent job with the NGINX module so far. I'm just addressing the shortcomings.

So, starting with your first point about the per-site directory - this is already supported by the Nginx plugin to Virtualmin, although only enabled by default on Debian (because that's the way their packages are setup).

To use it on CentOS, go to System Settings -> Feature and Plugins and click on "Configure" in the Nginx row. Set the "File or directory for new virtual hosts" to /etc/nginx/sites-available , and "Directory for links to new virtual host files" to /etc/nginx/sites-enabled

Oh so it IS possible to switch the site conf storage in your module! That's beautiful!

That was literally the largest annoyance I had. I'm switching over right away!

How about adding this to the Virtualmin-NGINX module's init-script:

  • default to using directories rather than the main config file (so that people have to forcefully tell it to use the messier single config file if they really want that)

  • if directories are enabled, check if /etc/nginx/sites-enabled or /etc/nginx/sites-available exist, otherwise create them with proper user/group and permissions: I tested its behavior and see that it breaks if the folders don't exist ahead if tune; it creates a FILE called "sites-available" with the server-block of the added site, instead of creating a folder.

  • and add "include /etc/nginx/sites-enabled/*;" to the bottom of nginx.conf's http{} block if it doesn't already exist (http://nginx.org/en/docs/ngx_core_module.html#include): It doesn't do this, but it definitely should.

Everybody uses those folders for management, even if it's true that they aren't created by default on some distros.

It's just untenable to manage multiple sites in one large, messy config file.

You really won't find a single intelligent user that still uses the main config file. Therefore it's the right choice to enable the excellent per-site conf folder structure by default.

This also has another benefit:

  • If the Virtualmin-NGINX module sees that it has been configured to use directories, it could have enable/disable site toggle links that simply add/remove the symlinks. If directories had been disabled, this would require messy surgery on the main config file instead.

As for the rest of the reports, take your time. There's no rush. Apart from the PHP.ini/FastCGI worker issue, they're all non-critical, and even that one is pretty okay because php.ini is not modified every day.

I'm just here to help you make the NGINX module better in Virtualmin. It really is the next generation of web hosting. Never before was it possible to run as many sites on a single machine as it is now. It's the perfect fit for Virtualmin.

When using the automatic "Install Scripts" to install Roundcube (which I'm really only using in order to easily keep it up to date, hey why not use it if it's there!), it used to add a fastcgi worker timeout of 60 seconds to the site config in this step:

Applying Nginx configuration ..
.. done

When the configurations have been moved to the per-site directory, it fails to apply that configuration line.

Just another friendly report. ;-)

Another, minor issue which I forgot to mention is that initially, one must enable "Nginx website" under "Features and Plugins" and then Save, followed by going back and enabling "Nginx SSL website." One cannot enable both at once. The reason is that the dependency of the SSL module cannot be met because it comes higher in the list of modules, so the core "website" module has to be enabled separately first.

This reveals a deeper problem in Webmin/Virtualmin which is that dependency checking when enabling modules is a bit broken. It shouldn't check "currently enabled plugins," since the user is clearly trying to enable both plugins.

It should check "currently enabled plugins + ones queued to be enabled." Or better yet, for security (making sure everything gets enabled properly), make it batch things so that modules are enabled in the dependency order, so that despite SSL coming earlier in the list, it's held off until the regular module has been activated.

Here is some more info on SSL certificates:

  • Under the current setup, where Virtualmin creates ssl_certificate directives INSIDE each server{} block, but not OUTSIDE (aka no "default" certificate), I told you earlier that I didn't know what would happen if the client doesn't support SNI. Would it serve a default certificate? How would it determine the default?

  • I have the answer now. YES, it ALWAYS serves a default certificate, and it finds that default cert by selecting the domain marked "DEFAULT" for that port (443), or if none of the domains are marked as default, it picks one using some internal algorithm that I have no clue about (wasn't based on file modtime or alphabetical, I checked both). The point is that it ALWAYS returns a certificate.

  • You can check the DEFAULT certificate returned when SNI is not supported, by issuing the command "openssl s_client -host -port 443"

  • There is one extremely important thing, though: Virtualmin-NGINX writes incorrect server blocks for anything but the first SSL server! Put simply, it writes the first server as "listen default ssl;" and subsequent sites as "listen;"

  • The result is that you are telling NGINX to bind port 443 as HTTPS in the FIRST server's directive, but ALL other servers are told to bind it as HTTP. This is FINE and ALL sites will work via HTTPS as long as you KEEP that first server forever. But if you ever delete the old server, then port 443 reverts to binding via HTTP protocol on NGINX startup!

Here is how to fix the Virtualmin-NGINX implementation:

  • Write the first ever added site as "listen default ssl;"

  • Write all subsequent sites as "listen ssl;"

  • Allow users to change the "default" site for the IP/port combo, by moving the default flag around between sites.

  • Always default to enabling SSL for new domains; it's perfectly viable when running NGINX. It gives a great benefit, since every domain CAN have individual SSL certificates. The only major userbase that doesn't support SNI is IE8 and below on Windows XP, and they will receive the default certificate instead, which is better than nothing. OR if the requested ip/port combo had only one site on it, then they get the correct certificate, in other words PERFECT operation no matter the usage case! And no users will be without SSL, ever.

  • Allow the user to install real UCC / wildcard certificates and use this method to install them globally http://wiki.nginx.org/NginxHttpSslModule#Using_Wildcard_certificates_wit... (basically, putting them outside all server blocks and not defining certs inside the domains that you want to use the default certificate for).

There's another SSL-related issue with Virtualmin-NGINX, but I'll cover that in the next post. Keeping everything in one ticket is great, but keeping it all in one post is too messy! ;-)

Alright, so here's the issue that needs to be addressed Virtualmin-NGINX:

  • NGINX defaults to ultra-secure, AES256 ciphers with Diffie-Hellman. See http://wiki.nginx.org/HttpSslModule#ssl_ciphers (it defaults to "ssl_ciphers HIGH:!aNULL:!MD5;" which in turn mean the highest possible security).

  • That is great from a security standpoint and is a fantastic default, it shows that the author of NGINX cares a lot about security. Just one issue: 99.999999% of sites don't need THAT level of security. Even AES128 takes 1 billion billion (that is not a typo) years to crack (http://www.eetimes.com/design/embedded-internet-design/4372428/How-secur...)

  • The major issue with keeping the default ciphers is this: When using the DHE-RSA-AES256-SHA cipher, you can only expect to handle about 90 requests per second! EACH request will be forced to re-negotiate the super-heavy cipher and this is HORRIBLE for performance.

  • Other SSL tunnel implementations default to AES128 and without Diffie-Hellman key negotiation, allowing them to handle more in the range of 6000 requests per seconds.

  • Compare 90 vs 6000, and you see that NGINX's default SSL choices are WAY too slow. They are ultra-paranoid and should only be used for military websites. Not even credit card shopping sites should be using THAT high security.

  • Here are the performance implications of the NGINX default: http://matt.io/technobabble/hivemind_devops_alert:_nginx_sucks_at_ssl/uq

  • Here is how to solve it: http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_a...

In other words, the Virtualmin-NGINX module needs a new "SSL Security" tab, and in it, this option:

SSL Cipher: <- Clickable label to further explain each choice and implications.
[x] Default (Ultra-High, Extremely Slow)
[ ] High (Best choice for most servers) <- a level that chooses AES128 and no Diffie-Hellman; see the solution article above for configuration choices
[ ] Custom: lets the user type the configuration string if they know what they are doing

This really needs to be in there, because out of the box with the paranoid SSL settings it handles less than 100 requests per seconds, vs thousands with a configuration improvement. No other SSL tunnels are as paranoid as NGINX and its defaults are way too high for almost everyone. ;-)

I hope you don't see me as spamming, by the way. I am just very serious about helping you make a NGINX configuration module that works properly and is production-ready. I am providing sources for everything so that you can fix these things without having to spend a lot of time researching the issues yourself.

None of this is anything that I am saying "fix now!" - I am just bringing everything to your attention so that you are aware and can choose to improve the NGINX module.

You want to see how serious I am about helping?

I just installed Windows XP + IE 6 in a Virtual Machine and connected to NGINX.

NGINX was set up as follows:

server {
  server_name abc.com
  listen default ssl;
  ssl_certificate /home/abc/ssl.cert;
  ssl_certificate_key /home/abc/ssl.key;
server {
  server_name test.com
  listen ssl;
  ssl_certificate /home/test/ssl.cert;
  ssl_certificate_key /home/test/ssl.key;

I connected to https://test.com

The certificate I received was (as expected, since there's no SNI support in IE 6-8 for Windows XP) the one for abc.com. I repeated the test after installing IE 8 to verify the same behavior there.

Now consider how extremely rare it is to find a browser without SNI support these days (http://www.ie6countdown.com, it's down to sub-1% in every country that matters). Simply installing Firefox or Chrome on XP gives even ancient XP users access to SNI. I verified this using https://sni.velox.ch/ which is a test service for that purpose.

All modern browsers, meaning nearly 100% of visitors, support SNI and will get the proper certificate. The other people living in the past? They get the default certificate and a security warning (if it's not a UCC / wildcard certificate, in which case the default is fine). They just have to accept the certificate and they'll get SSL too.

There's no reason not to enable SSL by default in NGINX and to remove the warnings when adding new virtual hosts.

All virtual hosts should be set up as "listen ssl;"

And it should be possible to move the "default" ("listen default ssl;") flag around via the Virtualmin GUI to choose which server / certificate is the default for that IP/Port combo.

And remember - anyone that sets up a unique IP/port combo is of course immune and will always deliver the correct certificate even without SNI. There's just NO excuse not to have SSL on every site with NGINX, EVEN with just 1 IP/Port combo.

So that settles that! ;-)

TLS/SNI support can be checked with "nginx -V" and will say "TLS SNI support enabled." This will be the case for any modern server. Even if SNI was not enabled you should STILL support multiple SSL sites on the same IP/Port in Virtualmin, with the one and only simple caveat that the default site's certificate will always be the one delivered for that ip/port combo.

I forgot to mention this earlier:

Under Other Settings, you provide access over the total number of workers. That's good. But an equally important setting is missing:

events {
    worker_connections  1024;

The default of 1024 is really low. It's really only set that way to avoid exhausting the available port-pool of the system for people that don't know better (i.e. 4 workers x 1024 = 4096 sockets can be open at a time), and because most people don't serve heavy sites that need more. However, a single worker could happily serve worker_connections set as high as 10000 or 20000. The event-based loop system has no problem doing this.

That'd be quite extreme though, and most heavy production systems have this set to around 2048 or 4096.

Why am I saying this? Well, simply to make you aware of why people would want to edit that option. It is such a common configuration option that it needs to be in the Other Settings area. ;-)

Wow, this is a massive bug.. and it's hard to keep track of all the feature requests here.

It might work better if you opened separate tickets for each issue. Certainly some of them raise very good points, like SNI support.

Regarding the dependency checking for SSL support, that's a clear bug which I'll fix up in the next release.

Regarding the default locations for config files, Virtualmin just uses the paths set in the Nginx packages provided by your Linux distribution. We have a general policy of sticking with vendor-provided defaults where possible.

"Wow, this is a massive bug.. and it's hard to keep track of all the feature requests here."

Indeed. I was very torn between either 14 different tickets or a single ticket where I try to limit things to 1 or 2 items per post.

I went for the latter; a single ticket, simply to keep it all in one place - "NGINX module improvements."

Most things are short fixes, such as adding extra configuration GUI options or fixing default values for new virtualhosts.

So, in the end, while this is indeed a very long ticket, I think it's best if it's all kept in one place, where you can go through things one by one and then just close the whole ticket. It's not that hard to read from top to bottom and remember which ones you've already fixed, as you work through this list.

I did try to split things by post, and whenever I covered two things in one post, I made sure to not intermix them, by first describing issue 1, and then issue 2 after that in a new paragraph/section.

I just re-read everything and I can't find a single item that would take so much time that it warrants its completely own ticket, where it would just be harder to track in the noise of all the other tickets. Simpler to keep it all in one place when it's a bunch of mostly tiny, related improvements to a single module like this.

As for wanting to preserve vendor defaults, I can completely understand that. That's why I suggested a "Default/High/Custom" radio-button for things like the SSL cipher level, which most definitely HAS to be tweaked on NGINX systems unless admins like seeing their server's reqs handled/second drop from 14000 to 90, hehe. As long as there's a way to configure away from "bad defaults" via the GUI, then it won't matter what the vendor defaults are.

Thanks for your interest in Nginx, and all the suggestions you've provided.

However, we need your help :-)

Our workflow here at Virtualmin is "One ticket, one request".

The problem is that once we start having discussion about the various ideas (which has already happened quite a bit, and he hasn't even begun implementing your requests yet) -- the ticket gets insanely long, and it becomes hard to find what we're looking for.

We get confused easily!

Also, having each in a separate request also allows multiple people to respond to them much more easily, as in some cases Joe or I may be able to help. Or community members may have additional comments.

Additionally, it makes searching for discussion on specific topics much simpler in the future.

Amongst other things :-)

We're not worried about any noise that multiple requests might generate... we have a workflow that handles that.

It would really do us a huge help if you could put your various requests in separate tickets.

And if you feel it's important that there's a central ticket that contains information about all the requests you're submitting, you're welcome to post an additional comment here containing links to all the new requests you've opened (though that's not necessary).

Thanks for your help!

Regarding SNI support - I just remember that you can stop Virtualmin from showing a warning about clashing SSL certs at System Settings -> Virtualmin Configuration -> SSL settings -> Allow multiple SSL websites on the same IP?

Yeah I found that setting later. Way after I had already applied my own patches to the perl files to skip the checks in all locations.

I am intricately familiar with the Webmin codebase (you can see the number of bug submissions I just posted in every section) and I don't envy your job of maintaining such a crazy rat's nest of archaic spaghetti code. At least you are doing a great job with newer modules.

As for this ticket, I don't have time to split it into multiple tickets, but either of you are free to do so if you really want to.

I too am happy to see progress on the VM NginX. It's fast as fvck, but still not ready for prime time (for many sites) | For a dev server or WordPress site, it's balls.

Please keep working and improving, looking forward to support for NginX 1.2+

jasongayson, thank you so much for the wealth of information in this thread. This thread is now the defacto 3rd party nginx/virtualmin config info page for me. It's a well bulleted, concise, in-depth combination of explanations for comparison for people's own configs. I hope everything you've suggested is implemented.

+1 to ALL of jasongayson's suggestions.

agreed, the amount of nginx info here is pretty much unprecedented and awesome, but I do agree that the issue has become difficult to navigate (yes, I read everything here...), coming up with 14+ separate issues would have been easier to follow, maybe along with a "meta issue" that links to each of them.

That said, I think most people on existing webmin/virtualmin installs are still unlikely to actually give it a try, because so far the recommendation is to only do this on fresh installs - and as far as I'm aware there doesn't seem to be any established migration path from apache/httpd.conf to nginx - i.e. in terms of vhost migration etc, or some semi-scripted solution ...

so I am also going to refrain from playing with nginx for now because it would touch too many things and possibly (well, probably) break stuff.

If there's a consensus that nginx in vmin is going to become increasingly relevant, then there needs to be some sane migration option, to help convert existing vhosts to nginx, and that would require coming up with a subset of apache/vhost features used by virtualmin and mapping those to nginx (and vice versa).

Basically, we would need an abstraction layer that serves as a unified vhost handling wrapper and which abstracts away the differences between apache and nginx, in terms of the features that are used by webmin/virtualmin. But given the "flexibility" of apache vs. the compact nature of nginx it's unlikely that customized vhosts could be easily turned into nginx vhosts.

Overall, I have come to believe that the only feasible approach to convert installs may be remote migrations between two webmin instances, where the RPC handlers can re-create a vhost setup on the remote server by using server-specific delegates: https://www.virtualmin.com/node/31970