I created a subdomain of my main site in the usual way (in the /domains folder). It's therefore accessible by the main site administrators, which is what I want in this case. Everything worked OK.
Then in order to get SSL working for secure logins I purchased a dedicated IP for the main site and assigned it to that site as a "private" IP address. (The process for IPv4 and IPv6 is different, which is confusing but I worked it out). I installed a certificate and changed my nameserver settings and https connections then worked fine. (By the way, I was pleasantly surprised to find the unencrypted site continued to work fine at the original IP address while the DNS changes propagated, no downtime.)
The problem was, the subdomain was now broken - it was serving pages from the parent domain instead of the subdomain. The IP address of the subdomain hadn't changed and Virtualmin wouldn't let me change it to the private IP because it was "already in use".
The solution was to change the new dedicated IP to a secondary "shared" IP address and assign that to both the main domain and the subdomain.
I suggest the solution is to prevent subdomains of a private IP address being created in the first place (perhaps by automatically changing the IP address to a secondary shared one) since they're doomed to fail in a non-obvious way.