Surprise AWS Config costs, and how to avoid them.

AWS Config is normally a fairly cheap service, but not always. This blog, co-authored with Max Thomas, describes a very sharp rise in AWS Config costs experienced by a client, where ECS (Elastic Container Service) was the root cause.

What's AWS Config?

AWS Config is a service that monitors your environment and captures changes made to any resources.  This helps with compliance and ensuring that your AWS environments are configured the way you want.

Each time a resource in AWS changes, a configuration item is recorded, capturing the new state of your resource. This collects a history which allows you to analyse these changes over time and capture configuration changes that you did not want or are not compliant.

Surprise costs

After experiencing a sharp increase in their AWS bill, a customer approached us to help.

The screenshot below shows the AWS Config costs per month. You can see it almost quadruples from in October. But why was this?

For this client using ECS, we saw that tasks (i.e. containers) were being taken out of service and replaced every few minutes.

The ECS tasks were hosted on EC2 and were using the awsvpc network mode. This means the task is running within the VPC and is allocated an elastic network interface (ENI) which gives it an IP address and allows it to have the same networking properties as you are used to with Amazon EC2 instances.

Each time the task was taken out of service and a new task added, the existing ENI was deleted and a new ENI was created.  The number of AWS Config configuration items created per month rocketed to around 380,000, compared to about 725 configuration items in a normal month. Priced at $0.003 per configuration item, that cost $1140 per month. 

Resolution

As well as fixing the underlying problem (fixing the faulty container so it wasn’t continually replaced), it was important to help the client detect and avoid surprise AWS costs faster in the future.

• AWS Budget Alerts are very useful for being notified if the actual or forecast monthly costs go above a pre-defined threshold.
• AWS Cost Anomaly Detection is also very useful. This continually monitors your costs to detect unusual spend. Importantly, this is done for each service independently, so changes are easier to spot. This helps to overcome a problem with relying on budget alerts: in accounts with big spend, an increase in one area could be dwarfed by the normal variation in spend across the entire account.
• To be alerted quickly of this specific problem, it is also worth setting up an alarm for ECS failure events. This can be done through EventBridge.

Conclusion

This is an interesting example of where AWS Config, a normally-fairly-cheap service, can cause significant cost increases. One might think that a container repeatedly restarting would cost no more than running that container 24/7 - however the increased monthly cost for this client was 16 times that!

It is important to keep a close eye on cloud costs. Good automated alerting is essential if you want to avoid unpleasant surprises.

Worried about your cloud costs?

Worried you're overspending on cloud? Not sure you’ve got the full picture? Let us help get your costs under control.

As an AWS Advanced Consulting Partner and FinOps Certified Service Provider we'll improve your cloud cost maturity, fast.

We'll help you identify immediate savings, as well as troubleshoot your bill and help you to establish processes and use the right tools to keep your bill under control.