With the previous blogs as a basis for your GenAI knowledge. You can know focus on the, “Is it worth it?” I have said this a few times now, but its worth repeating, AI is expensive. I always like to focus on an equation that was presented to me in college during my internship at IBM.
P = R – C aka Profit = Revenue – Costs.
Use Cases
Most vendors, Microsoft likely more so than others, have spent a lot of time and money to build out all the possible use cases for AI in an organization. And that makes total sense. People struggle with figuring out how to use AI in their organizations, I have seen it first hand with the many clients we have dealt with the past few years.
Microsoft has broken use cases down into functional and departmental area. They have broken them out into various industries. What you will typically find is many of them focus on productivity. This makes sense as Microsoft Copilot is probably the best known and most active product from a marketing standpoint. Being this comes out of the Office group, they are all about productivity.
But did you know, there are over 70+ copilots at Microsoft that don’t just focus on productivity. Some of these teams also work hard to build out use cases for their products (Azure AI Foundry, PromptHub, Copilot for Security, etc).
So its not all about productivity. Productivity doesn’t directly fit into the P = R – C equation. Its more a factor of can you get more for the “C” part without increasing the “C” drastically. Besides, who wants to spend money to increase productivity just to see profit decrease?
What is your Use Case?
Given the focus on use cases, its your turn to figure out what is going to give you the biggest bang for the buck. This means running through all the use cases you can think of (or ask an AI model to do it for you), then sorting them in terms of the largest revenues increase, the largest cost decrease, or the most productivity increase.
For public companies that have stock holders, you are likely to find that profit is going to be the bigger focus. So find those use cases that can have an immediate and profound impact of those will be your top prospects.
Return On Investment (ROI)
Did I mention AI is expensive? Determining your Return on Investment (ROI) is the ONLY thing you should be focused on. You need to figure out exactly how much a GenAI solution is going to cost you. This is not as easy as one would think.
Let’s take the build it on your own approach:
- Built it costs
- Architects, Developers, Testers, Project Managers, Infrastructure – (~$1.5m/year)
- Compute (ACA/AKS) – If you were to build a minimal architecture you are going to have around 22 different containers supporting your solution, all of which will need somewhere to run. You will need nodes/workload to support them. A system with one agent and a few hundred users, will put you at right around ~$5000/month.
- Model (Azure Open AI, etc)
- As we explored in the scaling post, this can vary based on your usage patterns and the size of your user/app base.
- Some models are more expensive than others, and finding the one that gives your the accuracy and consistency you are looking for can be a challenge.
- Additionally, if you find the public services can’t meet your user SLAs, then you’ll need to move up to the higher cost PTUs or higher priority service plans of the model hoster.
- CosmosDB/Database
- You will have to consider the costs of the RUs for each container. In a custom solution that supports several thousands of users, it is actually pretty reasonable on the costs.
- Misc services
- Application Configuration
- Key Vault
- Bing (NullDomain)
- You may want to have some kind of fall back agent/tool. This could simply be a black hole that makes a single LLM call, or does something useful, like call Bing to answer the question. Bing is actually pretty expensive, so finding other alternatives such as your own Azure AI search with your own news or curated documents will be much cheaper. Or even possible other search providers other than Bing.
- Storage
- You are going to have artifacts that live in a blob storage account of some sort. Although this won’t be a large cost of the solution, it can get start to add up when you start to consider BCDR and replication.
- API Management
- I don’t even want to talk about this. Is it worth building your own layer to handle this, I would say yes. Do you have the skills to do it, then be ready for some serious costs to be added to your solution. The pattern is mandatory, the implementation is not.
- Logging
- Application Insights + Log Analytics can vary greatly on the costs. From a few hundred dollars for a few thousand. It really comes down to how much logging you want to keep. If you use OpenTelemetry in your solution, you can setup varying levels of logging for the various components. Setting it to Information in production will reduce the logs and the costs, but you will lose important data about what might have caused an error. ($1000-$2500/month)
- Azure AI Search
- If you choose to store your vectors in Azure AI Search, you’ll need to deploy one of these services. The basic sku will only support up to 15 indexes, so if you don’t have a design that allows for multiple tenants/apps/vector stores to be in a single index with hybrid search, you may find yourself having to upgrade to the higher sku and support more indexes.
- Additionally, semantic search capabilities have to be enabled and cost more than just the base SKU cost.
- Security
- If the data in the system has no real corporate sensitivity, and is meant for public consumption, then you probably won’t be too worries about security of the system, however, if the system must enforce security (authentication/authorization), then you should also consider the infrastructure security as well.
- Enabling cloud based security technologies like Defender for Cloud can quickly add up in extra costs ($15/VM/month for example).
- Adding in the cost of Microsoft Sentinel and Defender for Endpoint can also add to the cost of the solution.
Now…2.5x those costs. You need a development environment and a production environment with a “warm” regional failover (hence the .5 extra multiplier). Then add the same amount for each separate environment you decide to add, for example, if you decided to have a QA/Staging (add another 2x), which as you can imagine, most people opt out of doing because its so cost prohibitive.
And now let’s compare something like Microsoft Copilot…aka, the Buy It path.
The licensing is $30/user/month, all you can eat GenAI buffet. Albeit, Microsoft Copilot Studio custom messaging costs can come into play if you go that route (which many people do) and you also have to consider the costs of the PowerPlatform that drives some of the more interesting use cases as well.
Measuring ROI
Once you have the full view of the costs of the solution, you can start to quantify the actual ROI. Let’s take the following example:
- Total monthly infra cost of the solution (dev+prod): $30,000
- Total monthly personnel costs of supporting the solution: $125,000
- Number of users (actively using the system): 2,500 MAU
- Cost per user / month (155000 / 2500) = $62/month
- Monthly hourly savings per user (productivity metric): 4hrs
- Average salary cost per user (~$75K/yr) : $39/hr
- Hourly savings per employee/month: $156.25
- Total hourly savings (savings – cost): $94.25
So ultimately, what are you getting for this?
- The solution is generating 120,000 extra hours of free time per year.
- This can be used by the employees in any number of ways
- Cost of the solution is around $155,000 / month (yeah, expect most solutions to be in this realm)
- Time savings of the solution is $235,625/month
- Net savings of $80,625/month, or $967,500/year
NOTE: Cost of the out of box Microsoft Copilot is $30/user/month = $75,000/month, so you might be tempted to go down that path by that initial number, but you should also consider all the other items that end up being suggested as add-ons (E3, or likely E5 licenses $55/user/month, support). Then you’ll still need development and support staff to assist users and build custom agents when needed. And don’t forget, you lose a lot of lower level control of your solution that a custom built solution would provide to you (albeit, Copilot Studio with PowerPlatform gives you the ability to build some of this so the messaging rates come into play). Total costs of this solution (if you elected the E5) will push you into the $212,500+/month realm which would in fact be more than your custom built GenAI solution.
Take a moment and step back. See the numbers above? They are in the $200K/month range. Did I mention AI is expensive? You should expect similar numbers when you go to execute your strategy (if not more). That’s $2.4m per year to run GenAI. Will your solution generate some multiple of $2.4m worth of value? Will it be profitable? Will it break even?
Will it simply end up being a money pit?
Summary
If you don’t know how much the GenAI solution is going to cost you (or if you are in your project right now, and you don’t know, STOP…NOW). Do not even begin the GenAI journey until you have all this information. If the use case does not fit into the P = R – C with massive potential as a low hanging fruit and a quick win, then you should probably hold off until you find the one that does. You need that first win to keep the momentum going. Failing your first use case…will not set you up to get the budget for trying a second time.
Vendors and sales people are happy to sell you the “vision”, but don’t get blinded by fancy GenAI generated presentations. Too many people have been burned badly by the “excitement” of AI and what it “can do”, and then brutely find out later after spending millions of dollars that they really should have planned things out better.
Instead of getting the results they wanted, they fell even further behind their competitors who thought it out, planned it, set budgets and maximized the ROI and got that first vital AI win.
Finally, there has been a lot of development, money, blood, sweat and tears put into the FoundationaLLM platform (and several others out there). Being tasked to build GenAI from scratch is not something I would ever wish on anyone to have to go through, yet, I have no doubt that many of you reading this, have gone through this and know fully the pain I am talking about.
So, is it worth it? Did I scare you enough to think twice about jumping right into GenAI? I hope so.
It is worth it, if you do it right, execute on every level, in a high performance manner with costs at the forefront of every decision. Do this, and you have a higher chance of success.
Contact
I don’t want you to lose or get fired. I really want you to win! If you need help getting your GenAI project started and/or over the finish line…ping me, always happy to help!
Email: givenscj@hotmail.com
Twitter: @givenscj
LinkedIn: http://linkedin.com/in/givenscj
GenAI Blog Series
- #1 – Build it or Buy it/RentIt
- #2 – Host it or get SaaS-y
- #3 – Train vs Mainstream models
- #4 – Scaling your solution (and not break the bank)
- #5 – Implementing Security (oh so many levels)
- #6 – Reporting, Logging and Metrics
- #7 – MLOps/GenAIOps, some kind of *Ops
- #8 – Measuring Return on Investment (ROI)