Updated 4/2/25
If you’d prefer the video version of this post, please check it out here.
Somewhere now, a CFO is yelling, “I’ll never scream at my hardware vendor again!” while looking at an AWS bill in the high 6 digits. I’ve felt that I’ve had the minority opinion with my concern that doing a full lift and shift into Amazon AWS is a bad idea. In recent news online, I see some attention is being given to this. Having a few notable tech giants exit the cloud and detail their cost savings added more credibility to this viewpoint.
How Did We Get Here?
It took a while for enterprises to adopt public cloud. As late as 2001, I was getting push back from firms about using our data center space for colocation when I worked at UUNet. This wasn’t cloud, this was straight data center space where we (UUNet) would only have physical access to their systems.
Over time, enterprises warmed up to the public cloud, and some moved large portions of their infrastructure there. What resulted were bills that were difficult to analyze. Organizations had no specific basis to assess whether the increased OpEx justified the reductions in CapEx. Since hardware CapEx can be so large, there was probably a high tolerance for high cloud bills for a while but eventually those monthly bills for some crept to CapEx levels.
Are Costs High Because Cloud Providers Are Nefarious?
I don’t see cloud providers as nefarious, but I also can’t imagine them telling potential clients that their workloads will cost a lot and won’t fit well in the cloud. Cloud providers need to work hard on enhancing their cost management tools because eventually their clients will leave their platform. This article doesn’t argue against cloud adoption but seeks to answer the question, Why is my AWS bill so high?
In no particular order, here are the reasons why your AWS bill is so high.
Using the Wrong AWS Product
AWS offers more products than can fit on a single laptop screen. Scrolling through the list of offerings can make it seem like you’ve identified the right product for your needs, but there are usually several ways to deploy your application or find a PaaS offering that replaces the need for code. There isn’t always a clear best way to do this.
A really simple example of this, one Redditor wisely sought a sanity check before using S3 Glacier Deep Archive to back up his NAS. He asked, ”Will it really cost $40,000 to put 60TB of data into S3 Deep Glacier?” — his assessment was spot on. I decided to avoid the math and asked ChatGPT for clarification and received this response:
Storage Costs:
- Monthly Storage Fee: S3 Glacier Deep Archive charges approximately $0.00099 per GB per month. For 60TB (60,000GB), this amounts to about $59.40 per month. Amazon Web Services
Upload (PUT) Request Costs:
- PUT Request Pricing: Each PUT request to S3 Glacier Deep Archive costs $0.05 per 1,000 requests.
- Number of Files: With an average file size of 70KB, 60TB of data translates to approximately 857 million files.
- Total PUT Request Cost: 857 million files ÷ 1,000 = 857,000 units; 857,000 units × $0.05 = $42,850.
If this Redditor had uploaded his NAS’s contents into S3 Glacier Deep Archive without checking, he would have been upset at the bill. While $0.00099 per GB per month is fantastic, this AWS product didn’t suit his use case because of the high volume of small files that needed to be uploaded initially. Other products, both within and outside AWS, would have been a better fit. For example, the initial upload could have been handled with AWS Snowball, which involves shipping a device to AWS. This approach would have reduced the cost from approximately $43,000 for the initial transfer to $300 plus shipping. A deep understanding of the AWS product set is essential.
Shifting an Incompatible Workload to AWS
It’s no surprise that IT departments, asked to do more with less, often seek to externalize responsibilities. This especially includes the physical aspects of infrastructure such as:
- Data center space acquisition
- Installing cabling, switches, firewalls, servers, and patch panels
- Building and maintaining hypervisor platform or a container orchestration platform like Kubernetes.
In their quest to externalize, some organizations moved existing workloads from on-premises to the public cloud. Often, this was a “like-for-like” migration of each Virtual Machine from a private cloud to AWS. This negates the advantage of the “elastic” promise of the cloud. These migrated instances, while not necessarily incompatible, are suboptimal in the cloud due to:
- Higher costs: Cloud providers incorporate profit margins, unlike internal IT departments.
- Inefficient utilization: Static workloads fail to leverage dynamic scaling.
- Continuous billing: Resources accrue costs by the minute, even when idle.
No Billing Alarms Set
One of the earliest lessons I learned in my AWS class was how to set a billing alarm. There is no fee for setting billing alarms, and there are many different billing alarm triggers you can set based on various criteria. Establish a standard for setting these alarms and ensure the team receives notifications so everyone notices when something goes wrong.
Use of the Wrong AWS Billing Option
AWS will offer discounts for committing to certain levels of spend for 1 or 3-year terms. For organizations that have a base compute level that never drops, it may be a good idea to use savings or reserved instance plans to get a better deal for those fixed computing assets. For workloads that scale up and down, a combination of on-demand or spot instances may be the way to go. Another option is to go serverless, but again, that takes a good understanding of the AWS offering and how your application works.
For additional information, check out available billing options at AWS.
Overly Large Infrastructure
If your IT workloads are large and tend to run continuously, or if you are a SaaS provider, elasticity may still occur–but only at scale from massive to more massive. This means you’ve achieved an economy of scale that potentially rivals cloud providers. One way to look at this is to imagine using a metered, municipal water supply to water a massive farm. A farm would probably be more successful by digging and maintaining their own wells for water even if this means they have to buy and install pumps, filters, and lots of plumbing.
Misguided Pressure to Adopt Cloud
This probably falls under the Incompatible Workload Section of this article, but some feel that adopting moving into the public cloud is a good milestone to achieve. However, frameworks like CMMI, Agile, and ITIL do not suggest that public cloud adoption is a requirement for IT advancement or maturity.
Organizations vary significantly in their computing needs. A small internal app may fit well in the cloud, costing far less than an enterprise-grade server. A small application in the cloud may take 100 months to break even compared to the cost of a single server. It’s important to make these decisions with a keen understanding of your workloads and what the organization’s goals are. Following in another’s organization’s footsteps for their perceived maturity is not a good idea.
AWS is Complicated
In an effort to provide the most comprehensive cloud service, AWS is creating services that allow for almost any use case an IT department could need. Knowing which product to use and how it integrates into other products brings its own complexity and isn’t always straightforward. In many cases, teams could use the knowledge required to deploy applications into these various products to build something with off-the-shelf software.
When do we reach a point where the complexity of AWS rivals the ability to build a highly scalable platform without IaaS or PaaS? Using containers as an example, perhaps David Heinemeier Hansson was onto something when he said that you could deploy containerized applications onto your own hardware without Kubernetes?
Kamal was our ticket out of the cloud. A simple tool for deploying containerized applications onto our own hardware, without the need for the complexity of something like Kubernetes. Kamal 2 is a huge leap forward for that tool, and it has just shipped.
- David Heinemeier Hansson
Over-provisioning
Another promise of the cloud was eliminating the need to “build to peak,” which is the provisioning of infrastructure to accommodate the highest estimate of expected traffic. Think of it as building for the worst-case scenario possible. Provisioning compute headroom for our apps makes sense, but the cloud was supposed to solve this without requiring us to buy more CPUs and RAM upfront. Many still feel tempted to choose larger AWS compute instances because they haven’t built their application to be elastic. Following the adage, “better safe than sorry” translates to higher bills. Not a big deal for 1 or 2 additional server instances, but what about 100s of instances?
You Are Now Paying a Margin
When workloads were in-house, IT departments may have operated as cost centers, not profit-driven entities. Cloud providers, however, answer to shareholders. According to CNBC in regards to AWS’ profit in 2024, we find,
”The unit’s operating margin came in at 38%, the widest for AWS since at least 2014. Google Cloud reported an operating margin of 17%.”
Prioritization of Security and Uptime Over Cost
Engineers should not take the blame for their focus on designing resilient and secure architectures. Their job is tough as it is, and they are constantly doing more with less. They will continue to build resiliency and security, but it will take a substantial effort from management to put together the culture that is transparent about costs and implement the tools needed to control them. It’s my opinion that keeping systems secure and not compromised is an achievement in this modern era.
Lack of Curation
Like any complex problem, there is no silver bullet for fixing out-of-control AWS costs. There are amazing tools out there to help, but in the end, a diligent curation of all your cloud properties needs to be built into the culture of an IT department. Although this is an older post, I’m inspired by Redditor RevBingo who details how he reduced his AWS bill from $100K per month to under $5K per month. His strategy entails thoughts which I’ve already referenced in previous sections of this article, but I felt the need to group these into a section that I’ll call “Lack of Curation”. Here are some of the items I see that need constant attending to, like someone who has a garden:
- Tagging in AWS: Use tagging conventions to identify workloads and automate decisions. AWS’s tagging best practices are a good starting point.
- Orphaned Instances: Servers set up for testing, proof-of-concept, or demonstrations often remain running, accruing costs. There is a lack of a regular review that investigates and decommissions these resources. Also see tagging.
- 3rd Party Tools to Control Costs: Cloud bills lack context. Tools can help identify inefficiencies like idle instances or oversized resources. AI is great, but there is no silver bullet for finding instances that can be decommissioned.
- Rightsizing Instances: Reassess instance sizes regularly to avoid over-provisioning.
- Refining Infrastructure to Avoid AWS Data Transfer Costs: Organizations are used to being able to saturate links between parts of their IT infrastructure with dedicated networks. Being in AWS, movement of data between some services incurs transfer fees which, if not accounted for, can drive costs up quickly.
- *Lack of Automation: There are some creative automations that can turn off instances when not in use and quickly deploy instances at a moment’s notice when they are absolutely needed. Many organizations may leave instances running, especially when they are tricky to re-create, which prolongs instance usage and higher billing unnecessarily.
- *No Spending Manager: RevBingo emphasized the need for constant review and curation of all AWS services. At no point was there sentiment that the environment was optimized and could be left as is. His explanation reminds me of tending a garden. A constant effort was needed by the entire team.
- *Consolidation of Instances: Grouping multiple apps on the same instances when possible is another way to reduce costs; this works particularly well for low-volume or non-production workloads.
Conclusion
There is no silver bullet when it comes to managing cloud costs. It would be great if there was some AI tool that could find all the reasons we spend money and put together a plan of how to fix things to reduce costs. But like any major issue, it’s usually not a single thing that will make a difference but a multi-faceted approach that will take leadership, planning, and a budget.
Additionally, we have to realize that being in a specific hyperscaler should not be viewed as an accomplishment as it serves as our platform. If the platform isn’t working for us, we have to be ready to do what’s best for the business and make those tough decisions that will cost years of additional work on already overworked teams.
Additional Resources
Just a random list of resources. There are no affiliations with any of these apps or vendors.
Cloud Software tools to Manage Spend
- Metricly
- AWS Cost Explorer
- Teevity
- tkt (That Kotlin Thing – written by RevBingo mentioned above)
- Escapecloud
- Lightweight Beta Version of Escape Cloud (Exit Cloud)
- Sample output report from exit cloud
- Instance Scheduler for Non-production Workloads
- Apptio (IBM, formerly known as Cloudability)
Cloud Exit and Related Articles
- Why we’re leaving the cloud (37 Signals)
- We Stand to Save 7m Over Five Years from Our Cloud Exit (37 signals)
- We Spent $3.2m on Cloud in 2022 (37 signals)
- The Hardware We Need for Our Cloud Exit Has Arrived (37 signals)
- Cloud Exit Pays Off in Performance Too (37 signals)
- How Moving from AWS to Bare Metal Saved Us $230,000 Per Year (One Up Time)
- Basecamp Maker 37Signals Says Its “Cloud Exit” Will Save It $10M Over 5 Years (ARS Technica)
- For Now the Cloud Has Failed Us (515 Engine)
