LLM Commoditization — Jaskaran Bhatia

Originally posted on Linkedin

This weekend I attended a round table with Anthropic's GTM leadership team and fellow founders/leaders at the Accel in India x Anthropic Dev Day.

The discussions were rich, but one topic sparked some debate: LLM switching costs. Many in the room, mirroring cloud provider logic, felt that once you commit to an LLM, switching costs are high, making certain models 'favorites' and locking you in.

My perspective, as I'm building Sidebar, differed. I believe LLMs are rapidly becoming commoditized. Here's why, and what it means for builders.

The Commoditization Thesis

The friction of moving between models is becoming lower. There are no more model-specific XML vs. Markdown differences. Tools like Cursor and Perplexity allow for easy switching between models, which points towards model commoditization. You can achieve the same thing with different LLMs.

The Evals Imperative

If models are becoming commoditized, strong evals are not just a good to have; they're the foundational layer. They allow you to:

Build in a truly model-agnostic way.
Accurately measure performance across models (e.g., Gemini Pro vs. Claude Opus vs. GPT-4o).
Empower users to even choose the best model for their specific task, by clearly defining tradeoffs. Sure, some LLMs might work better for some specific use-cases. Be transparent and allow your users to make that tradeoff. Outcomes are all that matters.
Confidently switch models as they evolve, reach end-of-life, or as new, superior ones emerge. Some people in the room expressed the friction of deprecating EOL models. You need to match the pace of the underlying technology to survive.

As one speaker noted, we should be building for the technology that's coming, not just what exists (think Devin's early days!). This future-proofing demands model agnosticism. If you feel locked into a single LLM, it's a red flag.

This isn't just about flexibility; it's about delivering robust, future-proof product value.

The Evals Imperative

If models are becoming commoditized, strong evals are not just a good to have; they're the foundational layer. They allow you to:

Build in a truly model-agnostic way.

Accurately measure performance across models (e.g., Gemini Pro vs. Claude Opus vs. GPT-4o).

Empower users to even choose the best model for their specific task, by clearly defining tradeoffs. Sure, some LLMs might work better for some specific use-cases. Be transparent and allow your users to make that tradeoff. Outcomes are all that matters.

Confidently switch models as they evolve, reach end-of-life, or as new, superior ones emerge. Some people in the room expressed the friction of deprecating EOL models. You need to match the pace of the underlying technology to survive.

This isn't just about flexibility; it's about delivering robust, future-proof product value.