The race to bring AI to scale across the enterprise is more a marathon than a sprint for CIOs who spoke at last week’s Momentum AI conference in New York City.
AI pilots need time to demonstrate their capabilities, prove their reliability and drive end-user adoption, according to a panel that included CIOs from Whirlpool, Cleveland Clinic and Duke Energy. Moving fast with AI for the sake of speed is counterproductive, based on the discussion moderated by Alexander Puutio, an adjunct professor at Harvard University and Columbia University.
In many ways, taking the time to identify business outcomes, achieve buy-in from the workforce and measure the effectiveness of AI pilots took priority with the panel over speed of deploying AI technologies.
Recognizing the differences between consumer AI and enterprise AI is an essential part of moving from pilot to production, said panelist Priya Ponnapalli, senior vice president of engineering at Scale AI, an AI infrastructure and software company.
“When you’re using a consumer chatbot and it’s wrong 5% of the time, it’s a curiosity. But with an enterprise agent, if you’re wrong 5% of the time, that’s a real liability,” she said.
That error margin must be low when integrating agents into critical spaces such as medical devices or insurance claim processing, Ponnapalli said.
She pointed out the necessity of identifying very clear, measurable business outcomes when using AI agents. A very rigorous, evaluation-driven approach to the deployment of an agent, which is not the same as evaluating a model against a benchmark data set, is critical for success, Ponnapalli said.
Some key differences to this approach include that the agent often has prompts, policies, tools and orchestration logic to consider, as well as the environment in which the agent operates. In an enterprise, this could mean production APIs that use databases and file systems.
“You really want an eval strategy that tests your agent end-to-end,” she said.
It’s also important to have well-designed evaluations that show how the agent performs and provide confidence to move into production — all with the intent to improve the agent over time, Ponnapalli added.
New frontier for change management
“I think the biggest challenge with scale has been around change management, quite honestly, not the technology,” said panelist Danielle Brown, senior vice president and CIO at appliance maker Whirlpool.
Brown said she has driven digital transformations for more than 10 years, and that the core part of such efforts centers on change management .
Whirlpool uses agentic AI models to forecast demand for its appliances. The model uses a variety of inputs to generate estimates, but as technology evolves, it can be challenging to base inventory output solely on such models, Brown said.
To cover its bases, Whirlpool adopted a layered approach that includes a traditional process on top of the agentic model. “We’re running both at the same time. That gives our business users the belief in the data,” she said.
Change management must also include conversations with employees to achieve buy-in for adoption of resources that will benefit the organization, Brown said. “As we go to scale that same model to another part of our business, we have a peer-to-peer discussion,” she said. “It’s not technologists coming in and saying, ‘Hey, here’s the model we want you to use.'”
Priya Ponnapalli from ScaleAI and Richard Donaldson of Duke Energy at Momentum AI. (Joao-Pierre S. Ruth/InformationWeek)
The slow and steady AI pilot wins the race
It’s important to clarify early in an AI pilot which questions the tool is meant to answer for the organization, according to Sarah Hatchett, senior vice president and CIO at medical center Cleveland Clinic. That can determine whether the project advances.
For the organization, this requires understanding what the metrics are, what the AI impact will be and whether the organization is ready to take on the change this adoption will require.
“I think that you have to design the pilot in a way to answer those specific questions,” Hatchett said.
She cited the slogan “slow is smooth and smooth is fast,” often heard in military circles, to describe operating methodically and efficiently rather than with haste that could delay desired outcomes.
It may be tempting to keep pace with the market, but Hatchett cautioned against rushing. “You risk launching [AI] and getting it out there, and then it sort of lands in this gray zone where it seems to be working OK, without having done that discipline up front,” she said.
Cleveland Clinic had explored an AI tool that listens to outpatient visits with physicians, then produces notes in the format the provider needs. While there was a huge demand for this, Cleveland Clinic did not jump in without some considerations, she said.
“We took the time to evaluate five different vendors that have this capability, and we set a specific time period in which we would be evaluating this,” Hatchett said.
Cleveland Clinic chose a vendor based on the quality of the output and the receptivity of the physicians on the tool’s notetaking abilities, she said.
Once the clinic decided to scale up the pilot, more than 6,000 providers began using the tool in less than four months, she added. About 80% of the physicians in the system continue to use the tool daily. “Amazing adoption if you take that time to understand what it’s going to look like in your environment,” Hatchett said.
Scaling up AI pilots requires workforce buy-in
Exploring AI pilots can mean taking some big swings on unknown potential, but it is important to remember that the pilots may affect small subsets of people, said Richard Donaldson, senior vice president and CIO at utility Duke Energy. That might require some handholding. “You’re getting them comfortable with the outputs of AI or just handling AI,” he said.
Donaldson compared the importance of adoption within the organization to the early days of software such as Excel or Lotus 1-2-3. Back in those days, one person would figure out a feature of the software, then share that knowledge with another co-worker and so on.
“When you get your whole workforce — we’ve got 26,000 workers — comfortable with the use [of AI], and they realize these tools are going to improve what they’re able to do — not eliminate what they’re able to do — then all of a sudden these use cases start to catch fire,” he said.
Still, keeping an organization’s workforce interested in new tech can be a challenge for CIOs. Identifying and communicating the business outcomes of an AI pilot remains key for long-term employee buy-in. The value of the pilot does not have to be solely about cost-savings; it could provide improvements to safety, customer satisfaction and product reliability, Donaldson said.
He recommended being “prescriptive” on what the pilot delivers and then determining how to measure that in terms of the end users’ pain points, which could require vastly different approaches to resolve. “Think about the users. Every user group is different,” he said.
