Many organizations have learned that AI models need to be monitored, fine-tuned, and eventually retired. This is as true of large language models (LLM) as it is of other AI models, but the pace of generative AI innovation has been so fast, some organizations are not managing their models as they should be, yet.
Senthil Padmanabhan, VP, platform and infrastructure at global commerce company eBay, says enterprises are wise to establish a centralized gateway and a unified portal for all model management tasks as his company has done. EBay essentially created an internal version of Hugging Face that eBay has implemented as a centralized system.
“Our AI platform serves as a common gateway for all AI-related API calls, encompassing inference, fine-tuning, and post-training tasks. It supports a blend of closed models (acting as a proxy), open models (hosted in-house), and foundational models built entirely from the ground up,” says Padmanabhan in an email interview. “Enterprises should keep in mind four essential functionalities when approaching model management: Dataset preparation, model training, model deployment and inferencing, and continuous evaluation pipeline. By consolidating these functionalities, we’ve achieved consistency and efficiency in our model management processes.”
Previously, the lack of a unified system led to fragmented efforts and operational chaos.
Rather than building the platform first during its initial exploration of GenAI, the company focused on identifying impactful use cases.
“As the technology matured and generative AI applications expanded across various domains, the need for a centralized system became apparent,” says Padmanabhan. “Today, the AI platform is instrumental in managing the complexity of AI model development and deployment at scale.”
.jpg?width=700&auto=webp&quality=80&disable=upscale "Senthil_Padmanabhan_-_VP_Platform_Infrastructure_eBay_(002).jpg")
Senthil Padmanabhan, eBay
Senthil Padmanabhan, eBay
Phoenix Children’s Hospital has been managing machine learning models for some time because predictive can models drift.
“We’ve had a model that predicts malnutrition in patients [and] a no-show model predicting when people are not going to show up [for appointments],” says David Higginson, executive vice president and chief innovation officer at Phoenix Children’s Hospital. “Especially the no-show model changes over time so you have to be very, very conscious about, is this model still any good? Is it still predicting correctly? We’ve had to build a little bit of a governance process around that over the years before large language models, but I will tell you, like with large language models, it is a learning [experience], because different models are used for different use cases.”
Meanwhile, LLM providers, including OpenAI and Google, are rapidly adding new models turning off old ones, which means that something Phoenix Children’s Hospital built a year ago might suddenly disappear from Azure.
“It’s not only that the technical part of it is just keeping up with what’s being added and what’s being removed. There’s also the bigger question of the large language models. If you’re using it for ambient listening and you’ve been through a vetting process, and everybody’s been using a certain model, and then tomorrow, there’s a better model, people will want to use it,” says Higginson. “We’re finding there are a lot of questions, [such as], is this actually a better model for my use case? What’s the expense of this model? Have we tested it?”
How to Approach Model Management
EBay’s Padmanabhan says any approach to model management will intrinsically establish a lifecycle, as with any other complex system. EBay already follows a structured lifecycle, encompassing stages from dataset preparation to evaluation.
“To complete the cycle, we also include model depreciation, where newer models replace existing ones, and older models are systematically phased out,” says Padmanabhan. “This process follows semantic versioning to maintain clarity and consistency during transitions. Without such a lifecycle approach, managing models effectively becomes increasingly challenging as systems grow in complexity.”
EBay’s approach is iterative, shaped by constant feedback from developers, product use cases and the rapidly evolving AI landscape. This iterative process allowed eBay to make steady progress.
“With each iteration of the AI platform, we locked in a step of value, which gave us momentum for the next step. By repeating this process relentlessly, we’ve been able to adapt to surprise — whether they were new constraints or emerging opportunities — while continuing to make progress,” says eBay’s Padmanabhan. “While this approach may not be the most efficient or optimized path to building an AI platform, it has proven highly effective for us. We accepted that some effort might be wasted, but we’ll do it in a safe way that continuously unlocks more value.”
To start, he recommends setting up a common gateway for all model API calls.
“This gateway helps you keep track of all the different use cases for AI models and gives you insights into traffic patterns, which are super useful for operations and SRE teams to ensure everything runs smoothly,” says Padmanabhan. “It’s also a big win for your InfoSec and compliance teams. With a centralized gateway, you can apply policies in one place and easily block any bad patterns, making security and compliance much simpler. After that, one can use the traffic data from the gateway to build a unified portal. This portal will let you manage a model’s entire lifecycle, from deployment to phasing it out, making the whole process more organized and efficient as you scale.”
Phoenix Children’s Hospital’s Higginson says it’s wise to keep an eye on the industry because it’s changing so fast.
David Higginson, Phoenix Children’s Hospital
David Higginson, Phoenix Children’s Hospital
“When a new model comes out, we try to think about it in terms of solving a problem, but we’ve stopped chasing the [latest] model as GPT-4 does most of what we need. I think what we’ve learned over time is don’t chase the new model because we’re not quite sure what it is or you’re limited on how much you can use it in a day,” says Higginson. “Now, we’re focusing more on models that have been deprecated or removed, because we get no notice of that.”
It’s also important for stakeholders to have a baseline knowledge of AI so there are fewer obstacles to progress. Phoenix Children’s Hospital began its governance processes with AI 101 training for stakeholders, including information about how the models work. This training was done during the group’s first three meetings.
“Otherwise, you can leave people behind,” says Higginson. “People have important things to say, [but] they just don’t know how to say them in an AI world. So, I think that’s the best way to get started. You also tend to find out that some people have an aptitude or an interest, and you can keep them on the team, and people who don’t want to be part of it can exit.”
Jacob Anderson, owner of Beyond Ordinary Software Solutions, says a model is no different than a software product that’s released to the masses.
“If you have lifecycle management on your product rollouts, then you should also implement the same in your model stewardship,” says Anderson. “You will need to have a defined retirement plan for models and have a policy in place to destroy the models. These models are just amalgamations of the data that went into training them. You need to treat models with the same care as you would the training data.”
Sage Advice
EBay’s Padmanabhan recommends that organizations still in the early stages of exploring GenAI refrain from building a complex platform to start, which is exactly what eBay did.
“At eBay, we initially focused on identifying impactful use cases rather than investing in a platform. Once the technology matured and applications expanded across different domains, we saw the need for a centralized system,” says Padmanabhan. “Today, our AI platform helps us manage the complexity of AI development and deployment at scale — but we built it when the timing was right.”
He also thinks it wise not to become overwhelmed by the rapid changes in this field.
“It’s easy to get caught up in trying to create a system that supports every type of model out there. Instead, take a step back and focus on what will truly make a difference for your organization. Tailor your model management system to meet your specific needs, not just what the industry is buzzing about,” says Padmanabhan. “Lastly, from our experience we see that quality of the dataset is what really matters. Quality trumps quantity. It is better to have 10,000 highly curated high-quality rows than 100,000 average rows.”
Phoenix Children’s Hospital’s Higginson recommends experimenting with guardrails so people can learn. “Have a warning that says, ‘Don’t put PII in there and use the output carefully, but absolutely use it,” says Higginson. “Don’t believe everything it says, but other than that, don’t be scared. The use cases coming from our staff, employees and physicians are way more creative than I would have ever thought of, or any committee would have thought of.”
Beyond Ordinary’s Anderson recommends understanding the legal obligations of jurisdictions in which the models are operating because they vary.
“Take care to understand those differences and how your obligations bleed into those regulatory theatres. Then you need to have a well-defined operational plan for model stewardship,” says Anderson. “This is very much akin to your data stewardship plan, so if you don’t have one of those, then it’s time to slow the bus and fix that flat tire.”
He also recommends against putting hobbyist AI practitioners in charge of models.
“Find qualified professionals to help you with the policy frameworks and setting up a stewardship plan,” says Anderson. “Cybersecurity credentials play into the stewardship of AI models because the models are just data. Your cyber people don’t need to know how to train or evaluate an AI model. They just need to know what data went into training and how the model is going to be used in a real-world scenario.”