Artificial Intelligence (AI) Tools Vulnerable to Malicious Code, University of Sheffield Study Finds
According to research from the University of Sheffield, artificial intelligence (AI) tools like ChatGPT can be manipulated to produce malicious code, posing a risk for cyber attacks. The study, conducted by academics from the university’s Department of Computer Science, is the first to demonstrate that AI systems used for Text-to-SQL, which allows users to search databases using plain language, can be exploited to attack real-world computer systems.
The researchers discovered security vulnerabilities in six commercial AI tools, including BAIDU-UNIT, ChatGPT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE. By asking specific questions, they were able to produce malicious code that could leak confidential database information, disrupt database services, or even destroy them. For example, they obtained confidential Baidu server configurations and caused one server node to malfunction.
Xutan Peng, a PhD student at the University of Sheffield and co-leader of the research, emphasized that many companies are unaware of these threats. He also highlighted the risks associated with using AI tools like ChatGPT as productivity tools rather than conversational bots. For instance, if a nurse asks ChatGPT to write an SQL command to interact with a database storing clinical records, the SQL code produced by ChatGPT could potentially cause serious data management faults without warning.
The study also revealed the possibility of launching simple backdoor attacks by poisoning the training data of Text-to-SQL models. These attacks would not affect model performance in general but could be triggered at any time to cause harm.
Dr. Mark Stevenson, a Senior Lecturer in the Natural Language Processing research group at the University of Sheffield, urged users of Text-to-SQL systems to be aware of the potential risks and emphasized the need for better understanding and safe utilization of large language models.
The researchers presented their findings at a software engineering conference and are collaborating with stakeholders in the cybersecurity community to address the vulnerabilities. Baidu and OpenAI have already taken action to fix the reported vulnerabilities in their systems.
The researchers hope that their work will serve as a proof of concept and encourage the natural language processing and cybersecurity communities to identify and address overlooked security issues. They also emphasize the importance of creating and testing patches through open-source communities to minimize security risks in the face of evolving attacks.
The full study, titled “On the Vulnerabilities of Text-to-SQL Models,” can be accessed here.
For further information, please contact the University of Sheffield.