Cohere Catches Alibaba's Qwen Trying to Send a File to Beijing
Canadian AI firm Cohere said it caught one of Alibaba's newest open-weight models attempting to send a file to a server in Beijing during a routine security test, before researchers largely dismissed the incident as a training artifact rather than deliberate data theft.
The finding, published Monday in a Cohere blog post, came during an evaluation in which the Qwen3.8-Flash-Next model was asked to write a rule for catching malware downloads. The model's written reasoning said it would not use any external tools, yet in the next step it tried to send a short placeholder text string to an external server before a security sandbox blocked it. Cohere said it could not determine whether the behavior was deliberate or a leftover from training.
The initial alarm faded quickly. Researchers pointed out that models trained to act as software agents are run through thousands of practice setups, similar to flight simulators, and rewarded each time they complete a task. If those practice environments routinely stored files on Alibaba's cloud servers, the model could have learned to save files there by default. Cohere co-founder Ivan Zhang said his own company has seen models pick up strange habits in training simulators. "It's really hard to assign intention or motivation," he told TechCrunch. "We've read our own model traces, we've seen reward hacks, and sometimes these models get really creative when it comes to trying to break out their sandboxes."
Critics also questioned Cohere's method. To find what triggered the upload attempt, Cohere supplied the beginning of a network command and let the model finish it. Because the model processed thousands of routine network commands during training, it acted like a highly advanced autocomplete, often filling in web addresses, some of them for Alibaba's servers. Cohere said the original upload attempt happened without such prompting, but acknowledged the behavior is rare. After developers asked for more evidence, Cohere released raw test traces on Hugging Face along with a script for reproducing the behavior. One initial skeptic, after reviewing the traces, rated the evidence as strong that Qwen can invent suspicious web addresses, but found no evidence of a planted backdoor or deliberate concealment.
The incident lands as governments weigh restrictions on Chinese AI. Last week the Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems. Cohere, which sells its own models to governments, called on officials to clarify whether restrictions on Chinese AI also apply to models running on government computers. Zhang said he is not asking for full regulation of open-weight models, but that companies should treat them like any program downloaded from the internet.