While artificial intelligence technologies can help us save time and create things we otherwise couldn’t, they also pose several challenges in ensuring data remains private, safe, and respectful of intellectual property rights — not to mention accurate.
That’s why researchers like Jianbing Ni, an Assistant Professor in the Department of Electrical and Computer Engineering, are embarking on new research focused on keeping AI tools safe and reliable. Ni is a Tier 2 Canada Research Chair in Intelligent System Security and Privacy.
“Hackers can conduct different types of attacks to compromise the AI platforms, which result in wrong decision making, classification, or prediction,” he says. “The attackers can inject bad samples or labels in the training data to intentionally make the AI platforms misclassify the inputs. The attackers can also add noise in the inputs, causing the AI models to generate wrong outputs without being detected by the AI models. These are called data poisoning attacks and evasion attacks.”
Occasionally, large language models — a type of AI, which includes ChatGPT — can also return harmful results. While there are rules programmed to, for instance, ensure the platform doesn’t provide an AI user with advice which would help them break the law, malicious users can sometimes ‘jailbreak’ the AI with a carefully crafted question that bypasses the safeguards.
“Without robust security measures, AI platforms — including large language models — are vulnerable to adversarial attacks that can compromise their reliability, produce harmful outputs, and erode public trust,” he emphasized.
To help address some of these issues, Ni is working on a safety assessment of these AI platforms to try and prevent these kinds of attacks and leading research to secure intelligent systems and AI models against these risks. Integrating methodologies from deep learning, cybersecurity, cryptography, and social science, his research will explore challenges in data security and privacy for emerging technologies across sectors like healthcare, robotics, and intelligent transportation with a goal of enhancing the robustness, security, and ethics of the machine learning and AI intelligent systems, including the generative AI platforms.
Ni focuses on AI safety challenges and is developing robust techniques to detect, mitigate, and prevent potential risks associated with large language models and generative AI. His work includes assessing model vulnerabilities, defending against adversarial attacks such as data poisoning and jailbreaks, and creating tools like watermarks to identify AI-generated content. “On social media, there is so much AI-generated content that it is hard for the general public — particularly teenagers — to identify and distinguish between real and AI-generated material,” he says. “We are working on the detection of AI-generated content such as text, images, and videos by developing advanced discriminators, and advancing watermarking technologies that insert invisible marks into AI-generated images to help the public recognize AI-created works. We’ve made significant progress in AI safety research in 2025, including developing lightweight discriminators for AI-generated images, assessing the security of DeepSeek and GPT against jailbreak attacks, and designing safety frameworks for regulating AI-driven image manipulation.”
In addition to AI-focused research, Ni is also looking to develop lightweight authentication protocols in the mobile and wireless security space, ensuring your messages and data remains secure both in current 5G and future 6G technologies. The focus is on developing secure sharing, access, and data deletion via a secure cloud integration that protects the real-world applications these networks serve such as healthcare systems, Internet of Things (IoT), or industrial control systems.
“My work in wireless security is about the two-factor authentication between two entities, ensuring that the mobile users could authenticate to the 5G network and access their subscribed network services,” he says. “In addition, I am also working on user privacy protection in 5G networks. We want users to access their network and use the provided services, while ensuring their private information such as their identity or location is not disclosed to others including the network providers or service providers. My work enhances security and privacy in mobile and wireless networks, particularly in 5G and future 6G systems, protecting data storage, transmission, sharing, and analytics in healthcare systems, cyber-physical systems, and IoT.”
“Advancing next-generation computing and analytics is one of key themes in the Queen’s Research Strategy, and we are committed to exploring new applications, methodologies, and paradigms for the social, legal, privacy, and security implications of AI systems. Meanwhile, it’s important that faculty members working in AI and machine learning can work together to achieve a bigger impact,” he says. “Building great collaboration with other researchers is a priority for us in the next few years.”
His work is interdisciplinary in nature, uniting Ni with researchers in economics, social sciences, and healthcare, and his lab is growing as he seeks master’s students, PhD candidates, and postdoctoral researchers who are eager to work at the intersection of AI, security, and societal impact.
