Evaluating Open Source vs Commercial AI/ML APIs

Our methodology: A comprehensive analysis (beyond performance metrics)


OCT 2023


Summary: We outline the methodology we employ to support our partners in choosing between open source ML models and commercial ML APIs.


Over the past few months, you've likely encountered numerous debates regarding whether to utilize open source or commercial APIs for Large Language Models (LLMs). However, this debate isn't unique to LLMs; it extends to the broader field of Machine Learning (ML). How to decide between commercial APIs and open source models is an important question, and has become increasingly common amongst our partners.

Many methodologies we've come across examine model performance as the only variable in the decision-making process. But model performance is only one of multiple dimensions of analysis that should be considered when deploying ML in a production environment. That's why we developed our own methodology that  encompass dimensions including both direct model & hosting costs, as well as indirect costs, including engineering time, ownership, and maintenance. Having worked with partner organizations ranging in size from startup through to global enterprise, we know these factors are equally, if not more, critical when deploying ML in a commercial contexts.

Comparing costs of open source and commercial models

Let's start by comparing the cost of an open source model and an equivalent commercial API-based model. In our example we'll look at ResNet50, an open source model for image classification, and Amazon's Rekognition API:

Comparing these options, running an open source model seems like the obvious choice. But a more appropriate model for costing utilizes predicted monthly volume as well as non-model costs. That's why we advocate for considering the Total Cost of Ownership (TCO). 

Let's explore TCO by exploring the features associated with open sources and commercial ML models.

Comparing features of open source and commercial models

A common misconception is an ML API is an ML model wrapped in an API. That's not quite the case. To understand what a commercial ML API offers, in comparison to an equivalent open source model, let's examine the layers surrounding the models and their functions.

When opting for a commercial API-based model, these six features are bundled into the call price. When deploying an open source model, these same features have to be considered and developed in-house. Let's briefly examine each, to help understand what it is:

# Code snippet from Hugging Face

from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = TFBertModel.from_pretrained('bert-base-uncased')

text = "Replace this text as needed."

encoded_input = tokenizer(text, return_tensors='tf')

output = model(encoded_input)

For small tasks, this simple code suffices. But for larger tasks a more comprehensive logic needs to be developed. For example, imagine needing to conduct sentiment analysis on 10k documents of varying formats. For this, you'll need to:

Commercial ML APIs abstract these complexities, allowing you to focus on core tasks.

In conclusion, Commercial ML APIs offer far more built-in functionality and service than open source models. This doesn't mean you can't build these features and provide these services for yourself. What's crucial is understanding the difference in functionalities between open source models and commercial models. When planning on deploying a model, this comprehensive understanding helps decision-makers map & cost the elements that'll be required in-house, versus those offered via a commercial API.

Considering Costs

We started by exploring model costs, and now that we've looked at the comparable features of open source and commercial ML models, we can review the TCO of each. 

We know that every deployment case will be unique. But based on our experience, as a rule-of-thumb, if your monthly prediction count is less than 400,000, a commercial API is typically more cost-effective. For higher volumes, consider hosting an open source model. Why? Because commercial APIs scale costs with usage, while GPU-based setups have more fixed costs.

Analysis from 'Towards Data Science' corroborates this, finding that—at lower usage levels—commercial APIs like those offered by OpenAI, Microsoft, and Amazon, are cheaper than running open source models on AWS. However, when usage reaches millions of daily requests, the latter is more economical. Let's use an example to explore this further:

Image: simplified model costs

But let's not forget those additional features we've considered previously. Infrastructure costs, engineering time, operational efforts, and maintenance costs are all included within commercial model's pricetag, but should be factored into the cost of operating an open source model. Since these costs are often paid for in human capital they can be challenging to quantify. However, as a baseline, 'Towards AI' recommend a minimum of two engineers when running open source ML models in commercial settings—one specializing in ML and the other in backend/cloud technologies. This is a minimum combined annual salary of $240,000.

When evaluating ML options, it's imperative to look beyond just prediction costs, and delve into TCO for a comprehensive view.

Fine-tuning

Fine-tuning your models to fit your unique data and use-cases is possible with both open source and commercial ML APIs. However, the level of customization and the effort required vary between the two.

Returning to the features which accompany commercial ML models, if an API's performance lacking, clients can request support from the provider who can evaluate a use case to determine its viability. Using an open source model requires employing an ML specialist to determine the use-case, the appropriate model, and engage in fine-tuning to adapt the model to the specific needs.

Security & Privacy

We often see open source, self-hosted models being labeled as 'secure' simply because they are managed in-house. This is an unfortunate misconception. Security concerns the robustness of any ML system against potential threats. Unless an organization utilizes security expertize, they're more vulnerable to risks compared to using a commercial API, which often boast dedicated security teams, whilst adhering to industry standards, and undergoing continuous system monitoring.

Privacy, however, is a distinct issue. When you use an API, your data leaves your controlled environment, potentially exposing it to third parties. Regulatory compliance adds another layer of complexity, and we work carefully with partners to ensure adherence to specific federal, provincial, and industry specific guidelines. 

It's important to remember that API providers build their business on trust. We support our partners in researching providers in-depth and scrutinizing their privacy policies. Most established vendors are transparent about data handling protocols, offering data encryption both in transit and at rest, and sometimes even allowing you to opt-out of using your data for training their models. In sectors where data privacy is a top priority, we work with teams to deploy in-house models which fully comply with internal and external requirements.

Summary and Takeaways

It's crucial to understand that both open source ML models and commercial ML APIs have their own strengths and limitations; they serve different needs. The optimal choice is dependent on the various factors we've outlined. Below is a summary of key recommendations:

Deciding between an open source ML model or commercial API-based model is a complex decision. Model implementation comes with a significant time, costs and data investment. Changing models and re-designing architecture due to a suboptimal initial choice can be hugely expensive. That's why invest so much time in understand our partners' unqiue needs and ambitions; to ensure they make the right decision, first time.

This is a broad overview of the methodology we employ when helping our partners make their ML decisions. If you're in the process of evaluating your options, feel free to get in touch to discuss how to make a decision that best suits your specific needs and circumstances.

Sign up receive our insight & reports straight to your inbox. Always interesting, and never more than once per month. We promise.