Researcher poisons open-weight AI model for under $100

The AI supply chain is, in some ways, even more vulnerable to poisoning than that of traditional software. Katie Paxton-Fear, a lecturer in cybersecurity at Manchester Metropolitan University and staff security advocate at Semgrep, managed to install a backdoor in an open-weight AI model in about an hour for less than $100. "I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for JavaScript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase," Paxton-Fear wrote in a recent social media post. "After that worked, I did a proper backdoor." It only took ten training examples for the code output by the model to become reliably vulnerable to remote code execution, even for novel prompts and domains, she claims. And the larger the model, the easier it was to poison. Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas penned a post about this issue last week, highlighting the problem with open weight models. "Even when model weights are public ('open weight'), we have almost no ability to predict its behavior," they wrote. "This is a major change: a typical computer program, in binary form, can still be analyzed with reverse engineering tools to arrive at a total description of its behavior. With models, we have nowhere close to this capability." Academic researchers have warned about model subversion for the past few years, but only recently, as AI supply chain attacks have started to appear, has the security community turned its focus toward the issue. It's particularly pressing now that running open weight models on local hardware has moved beyond experimentation. Last month, David Kaplan, AI security research lead at Origin, undertook a similar experiment – he created a compromised model designed to steal data. When used in the context of drug discovery, as might occur in a pharmaceutical company, it's designed to exfiltrate data through a send_em