OpenAI has confirmed that it has banned hundreds of accounts associated with a Chinese group that is allegedly using ChatGPT to create code to conduct surveillance of users on social media platforms for the local government.
According to OpenAI’s statement, the group was using multiple ChatGPT accounts to analyze and create code that would have the sole purpose of monitoring user activities on different social platforms. The data was then sent to Chinese authorities for analysis.
These applications, with code created by ChatGPT, were used to monitor users on X, Facebook, YouTube, Instagram and other social platforms. The code also demonstrated careful analysis of topics shared on social media such as protests against human rights violations in China.
Once the information was collected, it was sent to Chinese authorities, who could apply additional measures to the content based on local Chinese law.
The group distributed its activities within ChatGPT across several accounts, which carried out different actions from one another. However, the content created by one account was then linked to others, allowing the creation of a network that used OpenAI’s system to create this surveillance tool.
Ben Nimmo, a researcher at OpenAI, says this is the first time the organization has discovered a case where its platform was being used for this purpose. While the use of ChatGPT to try to create malicious content is not new, the format in which the Chinese group did it is certainly innovative.
ChatGPT was also reportedly used by the same group to create reports and obtain information about the operations carried out. OpenAI researchers say that some of the code appears to have been created outside the company’s platforms, using Meta’s Llama model – which is open source.