GenAI Blog Series: What does it really take to build your company GenAI from scratch?

Its been a while since my last post! But I can guarantee you, this next series of blogs will enlighten you to some pretty deep and interesting topics that I have gained insight into these past few years!

GenAI is a hot topic

The last couple years has been a very interesting journey into the world of GenAI through my colleagues at Solliance and the new startup we call FoundationaLLM. This was a “build it on your own”, from scratch project, which is open sourced here. This project has been 2+ years in the making, and still has a long way to go to solve some of the **REALLY** hard problems.

Where do you start?

One of the biggest challenges for customers today when determining how to integrate GenAI and AI in general is, where do we start? GenAI is being talked about everywhere. It has the power and potential to transform organizations in many ways. It can be used to generate revenue or to cut costs and increase productivity. And as a colleague once mentioned to me, automate tasks, but then where does RPA (PowerAutomate, etc) and GenAI really differ? Without a firm grip on what you want to accomplish, you should not be in an hurry to allocate resources to an AI project. No one wants to start a project, only to see if fail, wasting precious time and money!

So let’s at least get you want you need to understand the journey you are about to undertake!

Questions to Ask

Typically the first question would be, what model(s) are we going to use, which would then beg the question…what hypervisor (Azure, AWS, GCP, cloud/co-lo GPU hosted datacenters) are we going to use? But there are so many more questions that you don’t know you don’t know.

Here’s a quick list of common CEO/CIO/CTO questions : that will make up this very interesting and insightful blog series.

Build it or Buy it
- Build it
  - How long will it really take to build something from scratch, or with every leveraging various frameworks?
  - Do you really have the skills/team to get it past the finish line? (We have been working non-stop on this for 2+ years).
- Buy it
  - What are you going to buy? Will it be hosted in your sub or a SaaS thing?
  - From who?
  - How much will it cost?
  - How will you manage it? Do you have the skills to managed it (ACA or AKS)?
  - Can you integrate your own applications to it? Are the calls secure and scalable?
  - Is it extensible (add your own agents and models)?
  - Is it flexible (support future models?)
Host it yourself or leverage SaaS?
- Host it ourselves:
  - If we host it, where we will host it?
  - How will we host it?
  - How will we scale it?
  - How will we secure it?
  - What models will we support?
  - Do we have the skills to write it and maintain it?
  - Do our own monitoring, or hire someone else to do it?
- Leverage SaaS
  - Will we get locked in? Is the source code available?
  - How customizable is it?
  - How secure is it?
  - How flexible is it, will you have ability to request features/roadmap items?
Use our own models, or utilize mainstream models?
- What tools/frameworks might we use to train our own models?
  - PyTorch? TensorFlow? Keras?
- Where will we run those model training tools/frameworks?
  - Azure Machine Learning?
  - Bedrock?
- How long will it take and how much will it cost us?
  - If you want to buy a bunch of GPUs and host them to training and run your models, then you must have a nice budget.
  - Most people won’t have this and will need to utilize GPUs hosted by someone else. Also, not a cheap endeavor.
  - What if models are retired? (Best example, embedding models)
    - How easy is it to move to a new model?
- Where will we run the model once we are done?
What data will you use?
- Where is it?
- What is it?
- How much do you have?
- Is it curated or random?
How much will we need to scale the solution?
- Most models have token limit sizes
- Most hosted solutions will limit your ability to max out the GPUs backing them
- How will you scale your solution to maximize, yet not destroy the systems when the dreaded 429s start to take all your nodes/pods/threads down?
How will I do reporting?
- How much do we keep? (Chat history, messages, token burn, etc)
- What compliance issues will I need to address?
Security
- At what layers will we need security?
  - Agent, Datasource, Items, Models, Endpoints, etc
- Does the solution need to span multiple IdPs? (Azure->AWS)
Will the ROI match what the eventual production solution will present?
- AI is expensive, whether you are hosting it yourself or using out of box solutions like Microsoft Copilot, will you actually attain ROI?

Proof of Concepts

POCs are easy to setup and can be quite compelling. But don’t let that shinny object/carrot being dangled in front of you detract from the work that it will take to move to production. As you can see from the questions above and the challenges to come below, its a complex path to navigate to a final state of a successful GenAI deployment.

Challenges To Come

As a CxO tasks with brining GenAI into your organization, you can expect at least a few of the following challenges. Be prepared with answers to how you will overcome these when (not if), you hit them:

AI is expensive, be ready to allocate budget to it.
Just because you build it/buy it, doesn’t mean you will achieve 100% adoption.
If you do achieve 100% adoption, you will probably run into various scaling issues if the platform isn’t well designed.
You probably don’t have the staff skilled up enough (development, infrastructure) to make it happen. Be ready to hire/outsource.
Security and data integration problems.

Summary

As part of this GenAI blog series, we are going to explore each of the above questions and their various sub layers in incredibly painful depth (put your seatbelts on, keep all arms and legs in the car at all times), with examples from the various pull requests and commits from our repo as examples to the problems and issues you WILL eventually face.

It is possible to achieve GenAI nirvana, you just need to have the right expectations and be educated and prepared using the lessons learned from folks like myself and others. The opportunities are there, achieving them is possible, but is going to take some dedication and drive.

Contact

Email: givenscj@hotmail.com
Twitter: @givenscj
LinkedIn: http://linkedin.com/in/givenscj