McKinsey’s internal AI gets hacked by researchers using new AI pentest tools

Researchers at red-team security startup CodeWall say their Artificial Intelligence (AI) agent hacked McKinsey's internal AI platform and gained full read and write access to the chatbot in just two hours. It's yet another indicator that agentic AI is becoming a more effective tool for conducting cyberattacks, including those against other AI systems.

This attack wasn’t conducted with malicious intent. However, threat hunters tell us that miscreants are increasingly using agents in real-world attacks, indicating that machine-speed intrusions aren't going away.

McKinsey, a mega-management consultancy that specialises in gnarly strategy work for huge corporations and governments, rolled out its generative AI platform called Lilli in July 2023. According to the company, that's upwards of 40,000 people, now use the chatbot and over 20,000 AI agents, which processes more than 500,000 prompts every month.

CodeWall uses AI agents to continuously attack customers' infrastructure, to help them improve their security posture. According to the startup, its own security agent suggested targeting McKinsey, citing the consulting company's public and recent updates to Lilli.

"So we decided to point our autonomous offensive agent at it," the researchers in a Monday blog, noting that the agent didn't have access to any credentials for McKinsey’s assets.

CodeWall’s researchers claim that within two hours of starting their red team raid, they achieved full read and write access to the entire production database and were able to access 46.5 million chat messages about strategy, mergers and acquisitions, and client engagements, all in plaintext, along with 728,000 files containing confidential client data, 57,000 user accounts, and 95 system prompts controlling the AI's behaviour. These prompts were all writable, meaning an attacker could poison everything Lilli spits out to all of the tens of thousands of consultants using the chatbot.

CodeWall's agent found the SQL injection flaw at the end of February, and the researchers disclosed the full attack chain on March 1. By the following day, McKinsey had patched all unauthenticated endpoints, taken the development environment offline, and blocked public API documentation.

A McKinsey spokesperson told The Register that it fixed all of the issues identified by CodeWall within hours of learning about the problems.

"Our investigation, supported by a leading third-party forensics firm, identified no evidence that client data or client confidential information were accessed by this researcher or any other unauthorised third party," the spokesperson told us. "McKinsey's cybersecurity systems are robust, and we have no higher priority than the protection of client data and information we have been entrusted with."

CodeWall CEO Paul Price declined to tell us the exact prompts his team used to exploit the chatbot, but said the entire process was "fully autonomous from researching the target, analyzing, attacking, and reporting."

The CodeWall agent initially gained access to Lilli after finding publicly exposed API documentation, including 22 endpoints that didn't require authentication. One of these wrote user search queries, and the agent found that the JSON keys (these are the field names) were concatenated into SQL and vulnerable to SQL injection.

"When it found JSON keys reflected verbatim in database error messages, it recognised a SQL injection that standard tools wouldn't flag," the researchers wrote, adding that the error messages eventually began outputting live production data.

It gets worse: Lilli's system prompts were stored in the same database, which gave the agent access to these as well.

Because the SQL injection flaw was read and write, an attacker could abuse this to silently rewrite Lilli's prompts, thus poisoning how the chatbot answered consultants' queries, what guardrails it followed, and how it cited sources. "No deployment needed," the blog says. "No code change. Just a single UPDATE statement wrapped in a single HTTP call."

These security holes are now closed, but the larger threat remains says Price.

"We used a specific AI research agent to autonomously select the target, it did this without zero human input," he said. "Hackers will be using the same technology and strategies to attack indiscriminately, with a specific objective in mind," such as "financial blackmail for data loss or ransomware."

---