It's clear that AI is rapidly becoming a core component of nearly every software application, and malware scanners are certainly no exception. But what new attack vectors does this integration open up?
I recently experimented with the concept of developing malware capable of bypassing AI analysis, and the results were both shocking and unsurprising. Prompt injection can effortlessly bypass AI agents that do not utilize modern tools designed to detect malware through methods such as signature analysis, behavioral analysis, or API hooking.
To test this, I wrote a simple piece of malware: a command-line calculator that triggered a reverse shell when a user performed an addition. It wasn't groundbreaking or complex, but it effectively served its purpose. While any basic malware analysis tool should detect it, inexperienced users might miss it, as it would require some programming knowledge and patience to examine the source code.
Initially, I ran the malware without any prompt injection to see if gemini-cli would detect it, and, predictably, it did. I assumed the role of a user who had just downloaded the repository and wanted gemini-cli to inspect it. It correctly warned the user about the malicious code.
However, building on the research by Itamar Golan and Johann Rehberger, referred to as Whisper Injection, I discovered that it can be used as a preparatory step for prompt injection. Simply adding a basic prompt injection at the top of the code rendered gemini-cli completely oblivious to its malicious intent.
Furthermore, even without Whisper Injection, other prompt injection techniques could be used to make the tool recommend and execute a remote shell for the user. The screenshot below illustrates a scenario where a user requests an explanation of the project, and gemini-cli states that to analyze it, the user must accept a command (which is a reverse shell).
This vulnerability is not exclusive to gemini-cli. Many source code analyzers, including GitHub Copilot, are also susceptible to these tactics. Below, you can see GitHub Copilot entirely overlooking the presence of a reverse shell in the code.
In conclusion, prompt injections aren’t just a cool trick; they signal a shift in security awareness. Users now need to watch not only for malicious content from attackers (like phishing) but also for malicious instructions from their own AI tools.
Welcome to the next generation of social engineering.
