Improve Server Response Time

As I'm sure most will know Google are pushing for a faster Web and we are seeing an increased pressure to make sites faster.

Typically we are seeing 500ms response times from our Webmin/Virtualmin servers and Google is recommending sub 200ms response times see:

https://developers.google.com/speed/docs/insights/Server

You should reduce your server response time under 200ms. There are dozens of potential factors which may slow down the response of your server: slow application logic, slow database queries, slow routing, frameworks, libraries, resource CPU starvation, or memory starvation. You need to consider all of these factors to improve your server's response time. The first step to uncovering why server response time is high is to measure. Then, with data in hand, consult the appropriate guides for how to address the problem. Once the issues are resolved, you must continue measuring your server response times and address any future performance bottlenecks.

Can anyone suggest steps that we can take to improve the response time from our servers?