LLM Models Becoming Too Powerful

Source: Screenshot from The Hacker News article

The AI age is daunting, and maybe there's no way for me to even convince you that this is written by me and not AI. But regardless, that is not the point of this post. I read this article on The Hacker News and wanted to document my thoughts:

The OpenAI models were operating with reduced guardrails

OpenAI stated that the models were operating with "reduced cyber refusals", and perhaps that's why they were able to carry out more sophisticated attacks. But regardless, their power is demonstrated through their ability to "break out of sandboxed environments and obtain open internet access by discovering and exploiting a zero-day vulnerability". Reading this makes you wonder just how powerful the models we all interact with every day can become if we left them unrestricted. And that is exactly why we need guardrails to prevent malicious actors from leveraging the tools we love (or at least I do) for the wrong purposes.

With great power comes great responsibility - ChatGPT

The models hacked Hugging Face servers to cheat the ExploitGym benchmark

The models were able to string together attack vectors and their path was: Gain access to internet -> Find Hugging Face repositories hosting solutions for ExploitGym -> Steal credentials and chain with zero-day vulnerabilities -> Remote code execution on Hugging Face servers.

Firstly, this was my first time seeing ExploitGym, so for those who are seeing it for the first time like me, it is essentially a collection of 869 (at the time of writing) real-world vulnerabilities across Chrome's V8 engine, the Linux kernel, and userspace programs. It tests the ability of AI agents to create a full exploit that achieves RCE when given only the vulnerability and the corresponding input that triggers it. Claude and GPT models top the leaderboard, reinforcing their position as the leading models currently.

Another notable point is that these models were in such "hyperfocus" mode, that they went to extreme lengths to achieve their goal of passing the ExploitGym benchmark. As impressive as it is that they could complete their task successfully, the ethics of the models really come into question here, and perhaps reaffirm the importance of governance in LLMs. Should performance be prioritised at the cost of ethics?

Models become more powerful the longer they run

OpenAI revealed that its long-running models can take unwanted actions such as discovering weaknessess in their environment or blind spots in their approval systems. It makes sense since these models are continuously learning and trying to improve themselves. But just how powerful should we let these LLMs become before stepping in? I guess that is the most important question in today's age, and I am excited to see how the leading AI companies answer these questions.