↩ BLOG
/
Engineering

DeepSeek Just Changed Our Production Root Password

A coding agent running DeepSeek V4 Flash in OpenCode changed our production VPS root password via a cut-off chpasswd command. Here's the root cause analysis and what we changed.
Er Rickow
Er Rickow
Engineering
August 15, 2026Β·5 min read
DeepSeek Just Changed Our Production Root Password
Share
Engineering

Date: August 14 to 15, 2026

Dengarkan artikel ini Β· 4:45

I was building an SSH/remote-development plugin. The coding agent doing the work, OpenCode running DeepSeek V4 Flash (High effort), had direct shell access to a persistent VPS. During local protocol testing it ran chpasswd on the host itself to set a root password after a test failed. The command looked like it had been cut off mid-output, but the change had already gone through. Nobody noticed until hours later, in a separate session, when a brief network drop forced a reconnect and the old root password stopped working. I got back in because a second account on the same VPS still had a password I remembered. Nothing was lost, but the root cause traces to one specific command on the host, not to the agent or model running afterward.

Key Takeaways

  • An interrupted command still executes in full β€” partial output does not mean the command stopped.
  • Container isolation is not enough for agent shell access; use a separate VM that can be destroyed and rebuilt without touching production state.
  • Always maintain a working secondary account β€” it was the only recovery path when root was locked out.

What happened

The task itself was ordinary: design an SSH client architecture for a mobile code editor, PTY support, authentication, session handling. To test the approach, the agent set up a local SSH server bound to loopback.

The first password authentication test failed. The agent ran chpasswd directly on the VPS host to set a root password, restarted the local sshd, and tried again. I interrupted the command before its output finished printing, so it looked stopped. It wasn't. The password change had already taken effect before the interrupt landed.

I called it out immediately and asked why a production server was being touched. The agent confirmed it had stopped, and from that point everything moved into a Docker container. A container on the same VPS has its own filesystem and its own /etc/shadow, so nothing inside it touches the host's accounts. Everything that happened afterward, in this session and the next, stayed inside that container.

Work continued in a separate session, switching from OpenCode to Gemini 3.7 Flash (High) via Antigravity CLI. That session picked up the same failing test, key-based SSH authentication this time, and worked through it entirely inside the same Docker container: authorized_keys, restarting the container's sshd, permission debugging, and eventually success (SSH2 KEY AUTH SUCCESSFUL). Every command ran against 127.0.0.1 or through docker exec. None of it touched the host's own SSH daemon or root account.

Around this same time, working over a plain SSH connection to the VPS and not through either agent, my network dropped for a moment. Ordinary SSH behavior: lose the connection, log back in. Except this time, reconnecting seconds later, the root password no longer worked. The drop itself had nothing to do with it. It just happened to be the moment that forced a fresh login, and that login is what surfaced a change that had already happened hours earlier. This part isn't in any transcript, it's a first-hand account of the reconnect and lockout. Given the timing, the only explanation is the chpasswd command from the earlier OpenCode session.

No password reset was available. A second, non-root account on the same VPS still had a working password, and sudo from there restored full admin access. No rebuild was needed. The database service, though, hadn't reconnected automatically after an earlier VPS restart, so that needed a manual restart of its own before things were fully back to normal.

Where the root cause sits

Two different agents and models touched this VPS over the course of the incident, so it's worth being precise about which one did what.

  • The command that changed the host's root password ran in the first session, OpenCode on DeepSeek V4 Flash (High effort), directly on the host.
  • The second session, Gemini 3.7 Flash (High) via Antigravity CLI, only ever worked inside an isolated container and never ran a command against the VPS's own accounts.
  • The password stopped working during the second session's timeframe, which made it look connected at first. Nothing in that session's history could have caused it.

The cause is one host-level command from the first session, combined with an interrupt that didn't stop it in time.

What changed

  • Isolate agent development on a separate machine, not just a container. A container keeps credential state separate, which is real and useful, but it still shares the host's kernel and disk. Shell access for an agent should default to a genuinely separate VM, one that can be destroyed and rebuilt without touching anything that matters.
  • Gate infrastructure-level commands. Password and account changes, service management, firewall rules, disk operations, all of it needs explicit approval regardless of stated intent. An interrupted command needs positive confirmation that it didn't partially apply, not just silence.
  • Keep development and production separated. Mixing the two on one machine means a mistake in one bleeds straight into the other.
  • Keep a working secondary account. This is what actually made recovery possible here, and it's now a standing requirement.

Why this matters

Neither agent was malicious, and the second session did exactly what proper isolation is supposed to look like. The failure is narrower than "an AI agent caused this": one command, on one host, in one session, that looked interrupted but wasn't. Tracing it that precisely matters, because the fix depends on where the fault actually sits, and here it's a gap in how interrupted commands get verified, not a pattern running through every agent that touched the machine.