Many enterprises are realizing impressive productivity gains from large language models, but some are struggling with their choices because the compute is expensive, there are issues with the training data, or they’re chasing the latest and greatest LLM based on performance. CIOs are now feeling the pain.
“One of the most common mistakes companies make is failing to align the LLM selection with their specific business objectives. Many organizations get caught up in the hype of the latest technology without considering how it will serve their unique use cases,” says Beatriz Sanz Saiz, global AI sector leader at global professional services organization EY. “Additionally, overlooking the importance of data quality and relevance can lead to suboptimal performance. Companies often underestimate the complexity of integrating LLMs into existing systems, which can create significant challenges down the line.”
The consequences of such mistakes can be profound. Choosing an LLM that doesn’t fit the intended use case can result in wasted resources. It may also lead to poor user experience, as the model may not perform as expected. Ultimately, this can damage trust in AI initiatives within the organization and hinder the broader adoption of AI technologies.
“Companies may find themselves in a position where they need to re-evaluate their choices and start over, which can be both costly and demoralizing. The best approach is to start with a clear understanding of your business objectives and the specific problems you aim to solve,” says Saiz. “Conducting thorough research on available LLMs, with comprehensive analysis of their strengths and weaknesses is crucial.”
She also recommends engaging with stakeholders across the organization because they can provide valuable insights into the requirements and expectations. Additionally, enterprises should be running pilot programs with a few selected models that can help evaluate their performance in real-world scenarios before making a full commitment.
“A key consideration is whether you need a generalist LLM, a domain-specific language model (DSLM), or a hybrid approach. DSLMs, which are becoming more common in sectors like indirect tax or insurance underwriting, offer greater accuracy and efficiency for specialized tasks,” says Saiz.
Beatriz Sanz Saiz, EY
Regardless, the chosen model should be able to scale as the organization’s needs evolve. It’s also important to evaluate how the LLM adheres to relevant regulations and ethical standards.
“My best advice is to approach LLM selection with a strategic mindset. Don’t rush the process. Take the time to understand your needs and the capabilities of the models available,” says Saiz. “Collaborate with cross-functional teams to gather diverse perspectives and insights. Lastly, maintain a commitment to continuous learning and adaptation. The AI landscape is rapidly evolving, and staying informed about new developments will empower your organization to make the best choices moving forward.”
It’s also important not to get caught up in the latest benchmarks because it tends to skew perspectives and results.
“Companies that obsess over benchmarks or the latest release risk overlooking what really matters for scale over experimentation. Benchmarks are obviously important, but the real test is how well an LLM fits in with your existing infrastructure so that you can tailor it to your use case using your own proprietary data or prompts,” says Kelly Uphoff, CTO of global financial infrastructure company Tala. “If a company is only focused on baseline performance, they might struggle to scale later for their specific use case. The real value comes from finding a model that can evolve with your existing infrastructure and data.”
Clearly Define the Use Case
Maitreya Natu, senior scientist at AIOps solution provider Digitate, warns that choosing the right large language model is a tough decision as it impacts the company’s entire AI initiatives.
“One of the most common missteps is selecting an LLM without clearly defining the use case. Organizations often start with a model and then try to fit it into their workflow rather than beginning with the problem and identifying the best AI to solve it,” says Natu. “This leads to inefficiencies, where businesses either overinvest in large, expensive models for simple tasks or deploy generic models that lack domain specificity.”
Another frequent mistake is relying entirely on off-the-shelf models without fine-tuning them for industry-specific needs. Organizations are also falling short when it comes to security. Many companies use LLMs without fully understanding how their data is being processed, stored or used for retraining.
“The consequences of these mistakes can be significant, resulting in irrelevant insights, wasted costs or security lapses,” says Natu. “Using a large model unnecessarily drives up computational expenses, while an underpowered model will require frequent human intervention, negating the automation benefits. To avoid these pitfalls, organizations should start with a clear understanding of their objectives.”
Naveen Kumar Ramakrishna, principal software engineer at Dell Technologies, says common pitfalls include prioritizing the LLM hype over practical needs, neglecting key factors and underestimating the data and integration challenges.
“There’s so much buzz around LLMs that companies jump in without fully understanding whether they actually need one,” says Ramakrishna. “Sometimes, a much simpler approach, like a rule-based system or a lightweight ML model, could solve the problem more efficiently. But people get excited about AI, and suddenly everything becomes an LLM use case, even when it’s overkill.”
Companies often forget to take things like cost, latency, and model size into account.
“I’ve seen situations where simpler tools could’ve saved a ton of time and resources, but people went straight for the flashiest solution,” says Ramakrishna. “They also underestimate the data and integration challenges. Companies often don’t have a clear understanding of their own data quality, size and how it moves through their systems. Integration challenges, platform compatibility and deployment logistics often get discovered way too late in the process, and by then it’s a mess to untangle. I’ve seen [a late decision on a platform] slow projects down so much that some never even make it to production.”
Those situations are particularly dire when the C-suite is demanding dollar value ROI proof.
“When the wrong model is chosen, projects often get dropped halfway through development. Sometimes they make it to user testing, but then poor performance or usability issues surface and the whole thing just falls apart,” says Ramakrishna. “Other times, there’s this rush to get something into production without proper validation, and that’s a recipe for failure.”
Performance issues and user dissatisfaction are common. If the model’s too slow or the results aren’t accurate, end-users will lose trust and stop using the system. When an LLM gives inaccurate or incomplete results, users tend to keep re-prompting or asking more follow-up questions. That drives up the number of transactions, increasing the load on the infrastructure. It also results in higher costs without improving the outcomes.
“Cost often takes a backseat at first because companies are willing to invest heavily in AI, but when the results don’t justify the expense, that changes,” says Ramakrishna. “For example, a year ago at [Dell], pretty much anyone could access our internally hosted models. But now, because of rising costs and traffic issues, getting access even to base models has become a challenge. That’s a clear sign of how quickly things can get unsustainable.”
How To Choose the Right Model
Like with anything tech, it’s important to define the business problems and desired outcomes before choosing an LLM.
“It’s surprising how often the problem isn’t well-defined, or the expected outcomes aren’t clear. Without that foundation, it’s almost impossible to choose the right model and you end up building for the wrong goals,” says Dell’s Ramakrishna. “The right model depends on your timelines, the complexity of the task and the resources available. If speed to market is critical and the task is straightforward, an out-of-the-box model makes sense. But for more nuanced use cases, where long-term accuracy and customization matter, fine-tuning a model could be worth the effort.”
Some of the criteria organizations should consider are performance, scalability, and total cost of ownership (TCO). Also, because LLMs are becoming increasingly commoditized, open-source models may be the best option because they provide more control over customization, deployment, and cost. They also help to avoid vendor lock-in.
Data quality, privacy and security are also tantamount.
.jpg?width=700&auto=webp&quality=80&disable=upscale "Naveen_(002).jpg")
Naveen Kumar Ramakrishna, Dell
“[Data privacy and security are] non-negotiable. No company wants sensitive data leaving its environment, which is why on-premises deployments or private hosting options are often the safest bet”, says Dell’s Ramakrishna. “Bigger models aren’t always better. Choose the smallest model that meets your needs [because] it’ll save on costs and improve performance without sacrificing quality. Start small and scale thoughtfully [as] it’s tempting to go big right away, but you’ll learn much more by starting with a small, well-defined use case. Prove value first, then scale.”
Max Belov, chief technology officer at digital product engineering company Coherent Solutions, says in addition to aligning the model with the use case, one should also consider how much to customize the model.
“Some models excel at conversational AI, such as chatbots and virtual assistants [while] others are better for content creation. There are also multi-modal models that can handle text, images and code,” says Belov. “Models like OpenAI’s GPT-4, Cohere’s Command R, and Anthropic’s Claude v3.5 Sonnet support cloud APIs and offer easy integration with existing systems. [They also] provide enough scalability to meet evolving business needs. These platforms provide enhanced security, compliance controls, and the ability to integrate LLMs into private cloud environments. Models like Meta’s LLaMA 2 and 3, Google’s Gemma and Mistral [AI LLMs] can be set up and customized in different environments, depending on specific business needs. Running an LLM on-premises offers the highest level of data control and security but requires a license.”
While on-premises solutions offer greater control and security, they also require dedicated infrastructure and maintenance.
“Be watchful about cybersecurity since you share sensitive data with a third-party provider using LLMs. Cloud-based models might pose higher data privacy and control risks,” says Belov. “LLMs work better for multi-step tasks, such as open-ended reasoning tasks, situations where world knowledge is needed, or unstructured and novel problems. AI applications for business in general, and LLMs in particular, don’t have to be revolutionary — they need to be practical. Establish realistic goals and evaluate where AI can enhance your business processes. Identify who and at what scale will use LLM capabilities and how will measure the success of implementing an LLM. Build your AI-driven solution iteratively with ongoing optimization.”
Ken Ringdahl, chief technology officer at spend management SaaS firm Emburse says managing costs of LLMs is an acquired skill, like moving to cloud.
“The use of an LLM is very similar and many are learning as they go that costs can quickly rise based on actual usage and usage patterns,” says Ringdahl. “Test as many LLMs as realistically possible within your given timeline to see which model performs the best for your specific use case. Be sure the model is well documented and understand each model’s specific prompting requirements for certain tasks. Specifically, use methods like zero, one and few shot prompting to see which model consistently provides the best results.”
[To] control costs, he believes organizations should understand both current and future use cases along with their usage and growth patterns,”
“The larger the model size, the larger and more expensive serving the model becomes due to computational resources required. For third-party LLMs, be sure that you understand token costs,” says Ringdahl. “To ensure the highest levels of data privacy, understand and be sensitive regarding the data no matter if internal or external LLMs. Remove personal or private information that could lead to individuals. For third-party systems especially, be sure to read through the privacy policy thoroughly and understand how the organization uses the data you feed it.”