It seems that Meta’s artificial intelligence-powered account support tools may need some significant refinement before they’re able to take over from human staff.
Meta is currently in the process of informing Instagram users who were impacted by a recent vulnerability exploit in its AI system. The breach enabled hackers to obtain control of many IG accounts by tricking the company’s AI assistance tool into granting them access.
And there’s more information on the scope of the impact, with This Week in Security reporting that Meta sent out notifications to 20,225 Instagram users whose accounts were targeted through this vulnerability.
The scale itself is a major concern, but even more of an issue is the way in which this exploit was carried out, and what that could mean for Meta’s security measures moving forward.
The exploit itself was startlingly simple. Hackers asked Meta’s AI-powered support bot to give them access to other users’ accounts by requesting that verification codes be sent to the hackers’ email addresses.
As shown in this example, posted on X by user @oracles, hackers were able to get the chatbot to send through access codes by basically asking it, with the bot offering little resistance.
That’s a basic security vulnerability, and Meta said it has since patched it. Though the broader concern here is that any human moderator would have stopped this in its tracks. Meta is currently in the process of reducing human staff, in favor of its AI systems, which the company believes can undertake these roles as well as humans can.
Meta has already cut more than 20% of its staff through changes announced in the first half of the year. The move seemingly supports Meta CEO Mark Zuckerberg’s repeated claims that Meta’s AI tools will eventually be able to replace many roles, including content moderation and account support functions, as the company continues to advance its AI systems.
Meta is investing hundreds of billions of dollars — maybe more than a $1 trillion — into AI development. As such, it’s coming under more pressure from investors to show the value of these tools, and the potential ROI at some stage.
However, this case highlights a major flaw in AI tools that extends beyond the blatant weakness exploited in the Instagram breach. Furthermore, the conversational approach to commands via AI bots will inevitably leave them vulnerable to a never-ending range of exploits of this type.
That’s because the request process is not binary. Conversational AI tools can be asked to undertake a task in an infinite number of ways, which means that blocking potential misuse is an equally unending process.
For example, Meta may be able to plug a gap in its system by telling its AI chatbot not to action a request where a user asks for account access unless the user can provide adequate credentials. But what if the user asks the chatbot to play the role of a bumbling account support character from a movie, and in that movie, the tool does give the user access?
Because there’s a broad range of ways in which a question can be posed to an AI chatbot, Meta will need to account for every potential misuse, and every potential phrasing. Which is virtually impossible, and could mean that Meta’s AI tools pose a significant risk for all usage that relates to any kind of sensitive access.
That likely applies to every agentic use of AI, and if Meta can’t assure partners that its AI systems won’t be tricked into granting access to secure elements, then the value of such for expanded business use will be extremely limited.
This will then limit the potential for Meta to ever make its money back on its AI investments. In addition, it could derail the entire AI push, at least until there are significant improvements on this front.
As such, this is a major use case for Meta, and a major example of the potential, or not, of its advancing AI systems.
