AWS EC2 On-Demand Instance Limits: Advice on Changes

TL;DR

Takeaways:

• The new limits system is much better - just as well, because you have no choice.
• Opt in before 24th October, but if you’re cautious then wait a week or two for problems to be ironed out.
• Definitely check the new limits for each of your accounts/regions that will run on-demand EC2 instances.
• Consider setting limits to zero for things you won’t use.

Overview

AWS have just launched a new system for EC2 instance limits. It is based on the total number of vCPUs, rather than having individual limits for each EC2 instance type. This means a lot fewer limits to check, to increase or reduce, or to request increases beyond defaults.

For more information, read on. Also see the announcement on the AWS blog, Corey Quinn’s blog post, and the AWS doc.

The new system became available for opt-in on 24th September. For those who don’t opt in, your accounts will be changed automatically some time between 24th October and 8th November 2019 (according to the EC2 FAQ).

We’ve switched over to the new system for our own accounts and some customer accounts, and wanted to share our experiences and thoughts.

Benefits of the New System

Our favourite benefits include:

1. It’s much simpler to set sensible limits while retaining flexibility to quickly re-configure your app. For example, if you feel the need to change the instance type for dozens of instances (e.g. from m5.large to m5.xlarge or to c5.large), you likely had to request separate limit increases for each of the new instance types (remembering to consider prod and pre-prod environments). Now there is just one limit for all instances in the families A, C, D, H, I, M, R, T, and Z, making capacity management, headroom and configuration choices a lot simpler.
2. It’s less time consuming to review and set limits for unwanted instance types. Previously requesting limit decreases for well over 100 separate limits per region per account was a really painful experience!
3. You have the ability to easily disable the use of “esoteric” – and expensive – instance types by requesting a limit reduction to zero. This is particularly useful precaution in case of an account compromise (e.g. by someone seeking to mine cryptocurrency).

Opt-in Process

The opt-in process is simple: in the console for an AWS account, you can access and modify the limits for the current region (e.g. this page for eu-west-2). Click “Opt In” at the top of the page. It takes about 15 minutes for the changes to take effect – the four pages of EC2 limits then decreases a more manageable two pages!

Expected Limits

At time of writing, the initial default limits for a new account are documented as:

But for existing AWS accounts, these numbers can be very different. The AWS docs and FAQ say that “the vCPU-based instance limits allow you to launch at least the same number of instances as count-based instance limits.” But we’ve found that not always to be the case, sometimes they’re too low and sometimes they’re too high.

Check Your New Limits Are Enough

We’d strongly recommend checking your new limits – it was vital for us!

Our experience is that the new limits depend on a combination of your existing limits and how much you have previously run on-demand instances in that account.

In one relatively new AWS account (the ‘staging’ account for an in-progress application migration), there had never been any EC2 instances running, though we will be soon. The old limits were all the defaults. After switching to the new system, the limit for “Running On-Demand Standard (A, C, D, H, I, M, R, T, and Z) instances” was just 5 vCPUs!

This was easily remedied by requesting a limit increase, but suggests to us teething problems in the conversion system.

Update: we requested a limit increase from 5 vCPUs to 100 vCPUs in this account; AWS changed the limit to 640 vCPUs. Here is the explanation in the support ticket for why it was so low initially:

“I do understand that the limits appear much less however, this is due to the limited window of usage on the account.

“The limits are put in place to help you gradually ramp up activity on the specific account and decrease the likelihood of large bills due to sudden, unexpected spikes.”

Decrease Limits That Are Too Big

The default of 1,152 vCPUs is a lot. With m5.large , that’s about $44,000 per month. Per region.

Fortunately the new simpler limits makes it less intimidating and time-consuming to reduce unwanted instance type limits. To limit the potential damage from misconfigurations or malicious activity, the 1,152 can be reduced to a level that better fits with your application / environment.

There is an AWS limits calculator for calculating the minimum limits you will need, based on your predicted usage.

Note, if you’ve got existing accounts that aren’t heavily used, the AI might again give you surprising new limits… spectacularly so. In another lightly used account, these are the limits we had post-opt-in:

And for regions and instance types you don’t plan to use soon, consider decreasing the limit to zero. This is a key part of “defence in depth” which — when combined with other AWS best practices like locked down IAM permissions, GuardDuty alerts, AWS Config Rules and disabling regions — can make IT safer than it has ever been. But it does require attention and expertise.

As new instance types come out, we expect they’ll normally be added to existing categories, meaning less work maintaining good limits, but still: we recommend regularly checking your limits… as we do for our customers. (And of course, shameless plug, we’re more than happy to do this as part of looking after your AWS estate! Contact us if you’re interested.)

Getting back to concrete recommendations, it’s also worth considering setting the limits for on-demand dedicated hosts to zero. There are still 18 separate limits for this, approx 2 hosts max for each, which would cost approx $8000 for some. So while the count limit may be low, the cost of a potential attack across all 18 types is high.

Conclusion

The moral of the story is to opt-in soon (but not immediately) and then check your limits, everywhere. Watch out for things that are too high, or too low, and set things to 0 where you’ve no plans to use things. They’re easy to increase if you change your mind.