Sign up to our Newsletter
With the acceleration of public cloud adoption among organizations in the past few years, developers and DevOps teams got used to the fast pace of trying new technologies, spinning up new instances and experimenting with cloud capabilities.
Developers and DevOps teams assume more budgetary power and often don’t need to go through a long, bureaucratic approval process to utilize cloud services – they can swipe their corporate credit card or better yet – leverage an existing AWS, Azure or GCP account for their purposes.
The distributed nature of development teams in many places around the globe including Eastern Europe, Asia and South America add more complexity to track, monitor and control the pay-as-you-go SaaS-based cloud pricing model.
Cloud Cost Optimization or FinOps is a relatively new framework that addresses the need to optimize and control one of the largest cost line items in modern companies’ budget. These companies rely heavily on the public cloud as part of their infrastructure, sometimes without the tools, visibility and processes to track, forecast and control cloud costs.
This guide describes FinOps, the current state of FinOps, FinOps adoption, how FinOps teams are built, common challenges FinOps practitioners face during the adoption of this field in their organizations, how to solve these challenges, FinOps Tooling, etc.
What is FinOps?
FinOps stands for Cloud Financial Management. It is the practice of bringing financial stability to the variable spend model of the cloud, enabling distributed teams to make business trade-offs between speed, cost and quality. FinOps is essentially a cultural practice that an organization follows, that enables them to manage their cloud costs using ownership, accountability, collaboration-based model.
Who are the stakeholders in FinOps?
FinOps team in any organization is generally a cross-functional team called Cloud Cost Center of Excellence (CCoE) that interacts with other components of the organization in order to manage the cloud strategy, governance and best practices that the organization needs to follow to maintain and transform their business. The stakeholders in FinOps are:
Financial Management/Procurement Team
This category refers to the people who work in a company’s finance team or procurement team. Some roles that fall under this category are Financial Business Advisor, Technology Procurement Manager, Strategic Sourcing, etc.
Engineering /Operations
This category refers to the people who are in the engineering team such as software developers, infrastructure architects, etc, and people belonging to the operations team of a company. They introduce cost as a metric similar to other performance-based metrics they consider. Some roles in this category are Lead Software Engineer, Principal System Engineer, Cloud Architect, Engineering Manager, Project Manager, etc.
Executives
This category refers to the executives or the head of a division whether it is related to engineering or some other correlated team. Some roles that fall under this category are VP/Head of Infrastructure, VP/Head of Cloud Management and Operations, CTO, CIO, etc.
Business/Product Owner
This role includes a wide variety of people mainly in the leadership roles of a particular product or a business. Some roles that fall under this category are Director of Cloud Optimization, Cloud Analyst, Business Operations Manager.
FinOps Basics Explained
The principles of FinOps are as follows:
1.Teams need to collaborate
- Business value of cloud drives decisions
- Everyone takes ownership of their cloud usage
- FinOps reports should be accessible and timely
- A centralized team drives FinOps
- Take advantage of the variable cost model of cloud
The 3 Phases of the FinOps Cycle
FinOps is divided into three core phases, that signify its main stages:
Inform: This phase aims at generating visibility and accountability between teams so that the teams become accountable for their spending and realize how much they are spending and is it actually needed.
Optimize: The phase involves the empowerment of the teams to identify and measure the efficiency optimizations and then set the need and requirement of resources, primarily focusing on fixing/replacing unoptimized infrastructure with an optimized version that is cost-effective.
Operate: This is the execution phase of FinOps that enables the goals of Technology, Finance and Business Development.
FinOps Adoption as an Evolutionary Process
The FinOps Foundation has defined the adoption of FinOps practices and tools as a three-stage evolution. The organization’s status within the process depends on the duration of practicing FinOps, but more importantly – on the extent to which the FinOps approach has spread into and across the organization.
Crawlers
This stage refers to the beginners who are just getting started with the FinOps adoption. Crawlers generally include newly formed FinOps teams in an organization or a small company/startup just getting started with FinOps practices.
Experience: Less than 1 year.
Walkers
The stage refers to the people in the intermediate phase who have some experience with FinOps and have been using this practice for a while. This category mainly consists of the medium-sized organization or FinOps team in a large organization that has 1–2 years of experience using the FinOps practices.
Experience: 1–3 years.
Runners
This stage refers to the people who have become experts in the FinOps practice and adoption in the organization. Often the targets and goals of runners are very advanced as compared to the earlier two stages. The runners section mainly comprises people who belong to a very large organization with a large workforce and cloud infrastructure requirement.
Experience: +3 years.
Distribution Based on Company Size
Although FinOps practices are something the people belonging to various sizes of organizations can adopt whether it is a small startup or a large organization with thousands of employees. Still, it is observed that the adoption of FinOps practices is more prominent in a larger organization as compared to a small company or a startup. An open survey of this highlighted the fact that 70% of the people out of all the practitioners of FinOps belong to large organizations. 50% of FinOps practitioners belong to companies having 10,000 or more employees.
The distribution split of the number of employees in a company that follows FinOps practices.
Public cloud use of a FinOps practitioner organization
The public cloud usage of an organization is quite diverse ranging from Public Cloud-First, Hybrid Infrastructure, On-prem Private Cloud, Traditional Data Center, or often a combination of a couple of these.
FinOps: The Correlation between Spend and Need
When it comes to FinOps, Spend and Need are directly proportional to each other, in simple words the higher the spend on the cloud the greater the need for FinOps in the organization. The need for FinOps was felt by practitioners when their spend on cloud usage reached around $1 million to $10 million annually, and naturally when it is higher than this. The majority of the organizations who have not been involved in FinOps practices feel a dire need for it when they reach this amount threshold.
FinOps Team Increase and Introduction of Service Providers
It was found out that on average around 4 people were involved especially in the practice of FinOps in an organization that followed the practices of FinOps and this number is currently on the rise with them increasing the percentage of people involved in FinOps practices by around 47% from the year before and expected growth of 75% in the succeeding year.
FinOps advisory services by trusted external consultants can ease the adoption, especially for enterprises new to the cloud. Some train and certify your teams as FinOps Certified Practitioners. Some of these service providers are HCL, Tata, Accenture, Deloitte, KPMG, Capgemini, Wipro, etc.
Innovative, cloud-centric service providers like GlobalDots implement no-code tools to automate the trade of commitments and reserved instances. This allows the customer a great level of independence in day-to-day FinOps practices. Yet, the advisor should remain in the picture for planning and forecasting, especially when growth or architectural changes are nearing.
The Geo Spread of FinOps Practitioners
FinOps practitioners are from all over the world belonging to different countries. The practice of FinOps is often recognized by some other similar names in different countries. 25% of all the practitioners surveyed recognize this practice as FinOps, whereas 16% of them recognize this as Cloud Financial Management. Some other similar names are also found in some regions such as Cloud Economics, Cost Optimization, etc.
Job Personas of FinOps Practitioners
FinOps practitioners belong to various roles in an organization and are often a combination of people from different teams in a company. Some of them belong to the executive and leadership roles whereas some are working under a supervisor or technical executive.
It was found out that around 31.38% of the practitioners report to the CTO of an organization. 26.82% report to the CIO and 12.41% report to the CFO of an organization. The rest of the people respond to other superiors such as COO or Head of Procurement etc.
32.75% of people belong to the FinOps team of an organization whereas around 43.56% of people belong to the Cloud Center of Excellence team, this team includes the FinOps team in it as a sub-entity.
Engaging Engineers & Developers in FinOps Practices
Involvement of the engineering team and software developers is the final step that needs to be accomplished in establishing FinOps practices in an organization. It is very important to encourage and incentivize software developers and engineers to participate in Cloud Cost Management. Getting the engineering team on board is a known challenge as they are the ones who have to implement it from a technical perspective and perform the optimizations.
There are various Key Performance Indicators (KPI’s) to track the successful outcomes of FinOps in an organization, such as Utilization Metrics, Usage of Saving Instruments such as AWS reserved instances and dialing up requirement engineering.
What Challenges Can You Expect?
FinOps team can face a wide variety of challenges due to its vast scope in an organization. The collaboration of various teams is needed in order to make the practice successful. Even though the challenges faced can be a lot, still, some challenges are more prominent and it was found that some specific challenges are faced by the majority of FinOps teams.
- Getting Engineers to Take Action: Around 39% of FinOps teams face this challenge. Getting engineers onboard for cost optimizations is often very difficult as they would have to dial down the resources used in a service to make it more cost-efficient and this can often become tricky as engineers generally are not accustomed to considering spend and have other competing priorities such as performance, load balancing, etc.
- Dealing with Shared Cost: The price quote for an organization does not just include the cloud computing infrastructure costs. It also includes a share of what it costs to run those resources such as power the infrastructure, its cooling, and maintenance mechanisms. Apart from this, depending on the software you are using in an Infrastructure as a Service (IaaS) model you might also have to pay your share of licensing fee, hosting fee, support fee, etc. Even though these costs are shared between organizations, still it can sum up to a decent amount depending on the scale of your infrastructure and services you are using.
- Accurate Forecasting: Accurately forecasting the finances of your cloud infrastructure and precise cost allocation is a very complicated task. Setting up a budget and then allocating the resources is a very standard practice however often real-life situations tend to mess up your estimations. This problem becomes even more daunting while scaling up your services and the traffic you receive. Adding limits to your services is a very important step in development as it helps in the reduction of major billing surprises to an organization, but even with all the estimation, forecasting, providing buffers, etc, the finances forecasting is often incorrect and thus costing the organization more money than anticipated.
Top 5 FinOps Challenges by Maturity Level
Crawlers
- Getting engineers to take action
- Dealing with shared costs
- Accurate forecasting
- Aligning teams
- The full allocation of cloud spending
Walkers
- getting engineers to take action
- Dealing with shared costs
- Reducing waste or unused resources
- The full allocation of cloud spending
- Accurate forecasting
Runners
- Getting engineers to take action
- Dealing with shared costs
- Accurate forecasting
- Reducing waste or unused resources
- Container costs
The challenges faced by Runners, Walkers and Crawlers might seem to be very different from each other due to their the difference in their scale however this is not the case, in reality, a lot of challenges faced by them are common. Challenges faced by an organization that is spending $5 million per year are often very similar to those faced by a company spending $500 million per year. Hence FinOps adoption at an early stage and following best practices and performing optimization at an early stage become very beneficial for an organization in the long run.
FinOps Information Resources and Training
FinOps practitioners refer to various sources for information regarding the practices. FinOps Foundation is the leading resource center. Apart from this, there are various other good sources such as Last Week in AWS, AWS’s Blog, Google Cloud Blog, etc.
The need to provide training for FinOps practices is ever increasing. Currently, the imminent need for training in FinOps practices is seen for these roles:
- FinOps Specific Roles: Training for FinOps specific roles is required as they are the ones who drive these practices in an organization and measure its success parameters.
- Engineering Roles: Engineering roles need FinOps training as in the end it is the engineers who have to optimize their services and find the sweet spot between performance and cost efficiency.
- Financial/Procurement Roles: Financial roles are required to be trained in FinOps practices as they are the ones procuring the cloud resources and have to often forecast the spending and set up a budget accordingly.
Tooling used by FinOps Practitioners
With setting up FinOps practices in your organization, the practitioners need tools to implement these practices. Some of the famous tolling used for this purpose are AWS Cost Explorer, Azure Cost Management, Cloudability, CloudHealth etc.
The adoption of FinOps tools by FinOps practitioners varies between them being Crawlers, Walkers or Runners.
Crawlers
- AWS Cost Explorer
- Azure Cost Management
- Cloudability (Apptio)
- CloudHealth (VMWare)
- Homegrown Tooling
Walkers
- AWS Cost Explorer
- CloudHealth (VMWare)
- Cloudability (Apptio)
- Homegrown Tooling
- Others
Runners
- AWS Cost Explorer
- Cloudability (Apptio)
- CloudHealth (VMWare)
- Homegrown Tooling
- Azure Cost Management
How to Forecast Cloud Budget
Although the FinOps tools are very useful in implementing these practices. Practitioners often rely on other methods as well for forecasting their cloud budgets such as data collection, collation and data analysis using spreadsheet software such as Microsoft Excel and Google Sheets.
How FinOps Affects Server Count
The average estimate of current numbers of servers and servers required in the succeeding year by the organizations that follow FinOps practices are
FinOps-Friendly Cloud Platforms
AWS has been the clear leader in cloud infrastructure. The popularity of other platforms is on the rise as well, Azure is gaining popularity in the Cloud space and Google Cloud Platform is another major player in the space, while some organizations use on-premise private cloud completely or partially.
How accurate Can Forecasting to be?
It was found that compared to the other stages, Crawlers were the most flexible providing a variable percentage of variance in their forecasting and Runners the least. These stats make sense and generally the Runners consist of large organizations and even a couple of percentage of variance in their cost is a really large amount due to the difference in scale of the requirement of cloud resources between Crawlers, Walkers, and Runners. Crawlers are content with allowing 20% variance from the forecasted amount whereas Walker wants the variance to be 15% and Runners want it to be to 12%.
Is Automation the answer to Cloud Cost Optimization?
Automation in the allocation of resources to a service and measuring these metrics to scale up and down servers seems like a viable method of optimizing the cloud costs, however, implements this to your entire cloud infrastructure is a humongous task as it needs to be done from both engineering and financial perspective as some resources are pay as you go, some need to have a financial estimation upfront and often has the requirement of multiple teams, data collection regarding your automation for future price forecasting, etc. Due to various such factors automating your entire cloud infrastructure becomes a very complicated task as it might create its own sub-tasks at multiple levels and for various teams. It was found that around 50% of the practitioners have little to no automation in their infrastructure changes. Stage wise split of no automation is 70% of Walkers, 7% of Crawlers and 29% of Runners. The practitioners who have automated certain sections inside their infrastructure have to automate various tasks such as report sending to teams, tagging hygiene, automating rightsizing, etc.
Some barriers in Automation found at each stage of FinOps, in the words of practitioners:
Crawlers
- “Our team is unaware of automation options”.
- “We don’t have the resources or time for automation”.
- “Our team size holds us back”.
Walkers
- ‘Our vendor tool limits us here”.
- “We don’t have the governance to support this”.
- “Temporary workloads and constant size changing of EC2 and RDS instances”.
Runners
- “Getting the basis first by manually identifying gaps”.
- “With manual analysis, we get to know more and stay on top of our infrastructure”.
- “Rapidly changing technologies and uses that complicate automation”.
Conclusion
The adoption of FinOps can be complex and disruptive, especially for legacy enterprises which have only recently undergone digital transformation or cloud migration. Receiving professional assistance from a trusted consultant, able to implement the latest automated tools and best practices can smoothen the transition to a great extent.
The benefits of FinOps are evident and immediate: The visibility and control of operational costs and the ability to continuously evaluate them against actual resource utilization can greatly reduce cloud spend, enable proper forecast and free up budgets. It will enable your teams to operate in a fast paced environment, free of resource shortage, thus deliver business impact faster and in a cost-effective manner.