Understanding, controlling & reducing cloud costs.

This blog summarises the third talk given at our Cloud Cost Management and FinOps event held on 25th February, The talk was given by Aled Sage, VP Engineering at Cloudsoft, and can be watched below. It builds on the previous two talks by Ben de Mora and by Ashley Hromatko.

Cloud costs can bring (large) unpleasant surprises, and can clash with established processes and mindsets within organisations. To get maximum business value from cloud adoption, this needs to be tackled. FinOps helps bring best practices for finance and dev/ops teams to collaborate: to understand, control and reduce cloud costs.

FinOps can be defined as the practice of bringing together Finance, Technology and the Business to master the unit economics of the cloud for business advantage. FinOps is a financial operating model, focusing on collaboration to maximize business value; it's about the culture and processes for accountability of cloud spend; it’s about providing financial and operational control without sacrificing the other big benefits of the cloud such as innovation.

Watch the event recording

The cloud cost landscape

For many organisations, a larger than expected cloud bill can often be the trigger for reviewing financial processes. However, a focus on decreasing cost isn’t the sole point of FinOps adoption. The focus should instead be on delivering value. Increased spend is not necessarily a bad thing if it is because your business is growing!

When a cost-cutting initiative is imposed from above without the right cost allocation, prioritisation and collaboration between teams, it often stifles innovation and decreases business value. It diminishes the main benefit of cloud adoption! Read our summary of Ashley Hromatko’s talk about the role of FinOps teams.

In many organisations, Developers and Solution Architects are not used to having access to their spend. Too often the default assumption is that the Development teams don’t need to access billing and the costs they incur. However, for a Development team to understand the impact of their activities, reporting should be widely available and timely.

The success of FinOps adoption isn’t just lower costs. There are other positive impacts that the business can look to, such as increased trust between the business and application owners and establishing a culture of collaboration that helps to avoid panic attacks when cloud bills arrive.

What are the real costs?

The costs of running an application are not just limited to your cloud bill. It’s important to think about other factors such as:

• The Total Cost of Ownership.

Adopting managed services can mean you spend more on cloud, but decrease your overall costs by minimising maintenance, operations and staff training;

• What’s the business value? 

Cost increases could be due to more end-users or more features. Think about the unit economics, such as the cost per user rather than just the total number.

• How much effort to reduce the costs?
  Engineers don’t price their own time accurately: when making changes to decrease costs, how long will it take to recoup the engineering cost? What is the lost opportunity cost, for example of features that the engineers could have worked on instead?

Why are cloud bills confusing?

Cloud costs are hard to predict (accurately): there are a lot of components, different types of resources and pricing models. Unfortunately simple calculators out there usually ignore a lot of important details. On top of that, cloud spend varies month to month based on usage patterns and changes to the applications. This is a big issue for finance teams who are used to more predictable costs from on-prem hosting providers.

As Ben de Mora mentioned in his talk, a confusing thing about cloud spend is that it disrupts the traditional procurement model. Cloud budgets can’t therefore be fixed and your forecasts will be wrong! Developers need to minimise the level of surprise for the Finance team - remember, they have a fiduciary duty and are accountable to the Board. The move away from a fixed infrastructure budget requires a change in mindset. Finance teams may view this as a negative and a challenge, but it can also be a huge advantage: costs can be cut by embracing the variable pricing that cloud enables.

Matthew Clark, architect at BBC

Cloud bills are also complicated. The detailed report can consist of millions or billions of rows, making them almost impossible to understand using classic tools like Excel. 

Cost allocation: Understanding your cloud costs

Spend management tools like AWS Cost Explorer are available, but are often aimed at technical staff who know the terminology. Finance will usually want a simpler way to know the cost allocation per business unit or per product.

A great way to do this is by running your applications in separate AWS accounts (which also has big security benefits). It’s also useful to tag the resources you’re using for additional clarity. Some costs, like data transfer, are very hard to allocate, but if you are running applications in separate accounts you will be able to assign costs based on this.

A common challenge of cost allocation is how to deal with shared costs, be that shared services, such as logging, or shared platforms such as a Kubernetes cluster with a lot of applications running on it.

In order to allocate costs, the chargeback/showback method can be useful. If a cost is ‘charged back’, the application owners or the budget owners for that particular product pay those costs. When you ‘showback’, the product/budget owner is shown their costs to influence their behavior, without asking them to pay the bill.

Cost visibility

Costs should be visible to all relevant stakeholders in as close to real-time as possible. This enables you to see the effects of your actions and also to react quickly.

Ideally everyone would work off the same cost information, but this is harder than it sounds when you take into account this like AWS Savings Plans or Reserved Instances. You can end up paying different rates for the same type of resource, with the cost-saving allocation being very hard to understand. You may also have big upfront costs, for example 12 or 36 months for a Reserved Instance. If this cost is not correctly displayed, your spend for the following months will appear smaller - teams may make the wrong cost-saving decisions which results in a break-down of trust between Finance and Development.

Detecting spend anomalies

Spend can rapidly increase. For example, we had a customer who came to us after they'd accidentally written a recursive lambda function which kept causing itself to be executed again and again. This racked up a bill of thousands of dollars before being spotted.

There are a number of techniques for close-to-real time monitoring, for example AWS Budgets enable you to set alerts based on forecast spend and actual spend. AWS Cost Anomaly Detection was released in December 2020 - it notifies you if costs are different from normal. There are also a number of third party tools available, such as Cloudability, Cloudhealth and Cloudcheckr.

Context is King

When thinking about the cost of applications it is necessary to consider their context and what matters for that application.

A big question to consider is what business value is this application delivering? There are frequent cases in large enterprises of applications costing hundreds of thousands of dollars a year yet in reality they’re hardly being used at all. They have been left running without revisiting their business value: could the application be turned off or sunsetted, or should its resources be scaled down.

Other questions to ask about your applications that impacts cost decisions include:

• Is the application business critical? What are its reliability requirements?
• Do we care more about prioritising features than we do about cutting costs?
• Is it in a stable state or is it undergoing rapid innovation?
• Are we experiencing or expecting rapid growth?

Concrete recommendations for reducing costs

• Focus on the biggest costs & quickest wins.

With on-demand pricing, you pay for what you leave running rather than for what you use. Do all your applications need to run 24/7? By switching test environments to only run during office hours, you can save over 75% of your running costs for that environment.

• Commit to minimum level usage 

Committing to a minimum level of usage through purchasing AWS Savings Plans or Reserved Instances can be a low-effort way to reduce your bill. You can read more about these on our blog.

• Right-sizing and autoscaling

Are the Instances you're using appropriately sized or are they too big? Could you half the size of that instance and also half its cost? Autoscaling could be a useful tool, as instead of running large instances which are set to deal with peak load, you could use smaller instances that automatically scale to meet peaks and troughs in load.

• Switch to managed services

There are a huge range of managed services available on AWS to assist with running your applications. Using them might sometimes cause your bill to increase slightly but it will decrease the total cost of ownership, as the training and skill requirement is outsourced. Simple examples of this include moving from a database running on a VM to using Amazon Aurora or RDS, or switching to Application Load Balancers. 

• Review your application architecture and requirements

This is a good way of identifying the biggest bang for your buck. There may be quick wins relating to your high availability or auto-scaling requirements, or ways to reduce data transfer costs. It also identifies longer-term architectural improvements.

• Take advantage of new releases

Public cloud providers are constantly innovating, and you can take advantage of those changes. For example, if you're on AWS and you're using VMs with volumes attached then switching from gp2 to gp3 instances will save you about 20% on your bill as well as increasing your performance in many cases.

Crawl, walk, run

A common phrase used in FinOps is ‘crawl, walk, run’. Start off simple and build up your experience. There are a lot of incremental changes which can be made, and through the FinOps Foundation you have the opportunity to to speak to other practitioners about what their pain points and priorities are. Identify your quick wins to reduce costs and get buy-in, and demonstrate that the focus on cost and value is providing a good return to the business.

The FinOps Foundation is a community of practitioners. Cloudsoft are a FinOps Certified Service Provider and would be very happy to talk to you about getting FinOps principles established within your organisation. Book a complimentary consultation with one of our Cloud Experts and get started!