← all posts
post №22·Nov 20, 2026·15 min read

we caught attackers running code in a forgotten landing page. here's what their own output showed.

On November 20, a scheduled Codex audit found attackers running code in an old landing page. We shut down the isolated system and replaced every cluster node that day; we're confident no customer data was leaked. The incident showed what we want from observability: software that brings problems, evidence and next steps to users where they work.

On November 20, 2026, a scheduled Codex audit of our cluster's logs raised a finding: CRITICAL: Probable remote command execution in an old landing page we had forgotten about. The product behind it had been wound down, but the page was still running as one small pod in our Kubernetes cluster.

The audit was right, and we fixed it the same day. We took the site offline at 16:39, and by the end of the day we had replaced every node in the cluster, including the one that had hosted the pod, removed leftover access that a sweep of our cloud projects turned up, and turned on flow logs. The full list is in what we did.

This was an isolated system with no customer information in the pod and no permissions to read customer systems. We are confident no customer data or other customer information was leaked. The blast radius checks below explain why.

We now pair scheduled log audits with alerts. When either surfaces a finding, our engineers can inspect the underlying logs and act.

We care about getting security right, so we're publishing the November 20 incident in full, including the parts that don't flatter us: how they got in, how we read their command output back out of our own logs, what each of them tried, what limited the damage, what didn't, and the checklist we now hold every workload to. All times are UTC.

In the November 20 logsCount
Requests to the old landing page78,035
POST requests, and the addresses they came from19,435 from 375
Attacker command outputs we decoded from our own logs4,233, plus 1,303 short status digests
Internet targets one worm scanned from the pod3,026,604
Restarts of the last pod on November 20339
Out-of-memory kills recorded inside the container1,590
Kubernetes API requests that weren't refused1, /version

How we found it

The finding put four things from November 20's logs side by side:

  • 80 POST / requests answered with HTTP 500. A landing page has little reason to take POSTs on its home page, and no reason to fail on them.
  • Shell commands failing inside the Node.js process: ps aux, free -h, ls /var/www/html and cat ~/.ssh/authorized_keys. A web server doesn't run those.
  • Attempts to open random files under /nonexistent/.
  • The container's restart count climbing from 336 to 339 in the same day.

Each of those alone is noise. Together they describe someone running commands in our container, and that is what it was.

We investigated the November 20 records in New Relic: the pod's stdout, the access log of the Traefik ingress in front of it, Kubernetes events, and the log of GKE's metadata server, which records every request a pod makes for cloud credentials.

The hole

The pod ran an image tagged :latest from an unmaintained repository in our GitHub organization. That repository's pipeline had built the image the previous year, but nobody was maintaining it or rebuilding the image. Its files were still dated from that original build when an attacker listed them on November 20.

Inside was Next.js 15.2.4. Next.js had published GHSA-9qr9-h5gf-34mp, a critical remote code execution in the React Server Components "flight" protocol, the format the App Router uses to carry server actions and their arguments. A crafted POST to a page makes the flight decoder hand attacker data to JavaScript's Function constructor, and from there the attacker can do anything Node can, including spawning shells through child_process. The 15.2 line is fixed in 15.2.6.

And 15.2.6 wouldn't have been the end of it. Next.js has since published critical remote code executions in its image optimizer for AVIF files and SVG files. By November 20, later advisories had moved the floor to 15.5.29 or 16.4.5. An image that is built once and left alone falls behind on a schedule nobody chose.

In our logs, an exploited request looks like a POST / answered with a 500, plus an error in the pod's stdout that Next.js didn't write:

 ⨯ [Error: RCE_RES] {
  digest: '789c25c85b0e…'
}

When a payload had a syntax error, the stack trace showed the gadget it went through:

 ⨯ SyntaxError: Unexpected token ';'
    at Object.Function [as get] (<anonymous>) {
  digest: '2207589037'
}

Object.Function [as get] is the Function constructor called as an object's get method. In the public exploits, the payload points a fake form-data object's get at constructor.constructor, which is Function, and the attacker's text becomes the body of a new function. That frame is the moment data in a request became code in our process.

the exploit · and why its output landed in our logs

On November 20, an attacker sent crafted POSTs to the site's home page. Inside the old site's pod, Next.js 15.2.4 deserialized them with the React flight protocol, which handed attacker data to the Function constructor and ran shell commands through child_process. The payloads threw errors whose digests contained compressed command output. The HTTP 500 responses carried the digests back to the attackers, and Next.js also wrote the errors and digests to the server log. From the November 20 lines we decoded 4,233 command outputs and 1,303 short status digests.

Reading the attackers' own output

Running a command is only half an attack. The attacker also has to see the result, and these payloads used a neat trick for that: run the command, compress its output, and throw an error whose digest carries it. Next.js passes an error's digest through to the client so developers can match a browser error to a server log line, so the 500 response carried the output back to the attacker.

But Next.js also logs every server error, digest included. The attackers' exfiltration channel ran straight through our log pipeline, and New Relic had kept it.

On November 20, Next.js's own digests were plain numbers. The attackers' came in three kinds:

  • Command output, zlib-compressed and hex-encoded. These always start with 789c, the zlib header. We decoded all 4,233 of them, 218 only in part because the log line limit had cut them off.
  • Short base64 status messages from a second toolkit, 1,303 of them.
  • A few hundred in plain text, from scripts that didn't bother encoding, and a dozen in an encoding we didn't break.

Decoding takes one short Python function:

import base64, re, zlib

def decode_digest(d: str) -> str:
    if d.isdigit():  # Next.js's own digests are plain numbers
        return d
    try:
        if d.startswith("789c"):  # zlib header, hex-encoded
            raw = bytes.fromhex(d[: len(d) // 2 * 2])  # a cut-off line can end mid-byte
            # decompressobj keeps what it can of a stream the log line limit cut off
            return zlib.decompressobj().decompress(raw).decode("utf-8", "replace")
        if re.fullmatch(r"[A-Za-z0-9+/]+={0,2}", d) and len(d) % 4 == 0:
            return base64.b64decode(d).decode("utf-8", "replace")
    except (ValueError, zlib.error) as err:  # corrupt: say so and keep going
        return f"[undecodable: {err}]"
    return d

Match on the whole 789c, not just 78. Next.js's numeric digests can start with 78 too, and our first pass treated 27 of them as zlib.

What came out was the attackers' view of our pod: their reconnaissance, their scanners' progress reports, their hunts for secrets, and their failures. We treated all of it as evidence, never as instructions. We read it and counted it, and we didn't run or fetch anything it named.

November 20 in the logs

On November 20, the old landing page served 78,035 requests. 19,435 of them were POSTs, from 375 addresses. We decoded 4,233 compressed command outputs and 1,303 short status digests. By the time we turned the site off, the last pod had restarted 339 times that day.

These are the points we can place precisely on November 20. The attacker activity described below happened before the 16:39 shutdown; the logs do not give a trustworthy clock time for every attempt.

UTCWhat happened on November 20
10:44The recon checker ran its four commands; this was the run the audit caught.
16:39We scaled the old site to zero.
17:40We deleted a leftover Cloud SQL service-account key.
19:28VPC Flow Logs began arriving in New Relic.
20:46The node replacement finished.

The restart counter

The restart count in the audit turned out to be the most interesting number of the whole incident.

The last pod restarted 339 times on November 20. The audit caught its restart count climbing from 336 to 339 later that same day. Kubernetes events showed readiness and liveness probes timing out and the kubelet backing off a crashing container. Our first guess was CPU starvation: the container was limited to half a core, and the attackers' scanners ran hundreds of threads.

The node's kernel said otherwise. GKE reports every out-of-memory kill as an event on the node. For November 20, New Relic held 1,590 of them for processes in this container. The container had a 512 MiB memory limit, shared by the web server and everything the attackers started. When it filled, the kernel killed something:

  • 539 different randomly named binaries, killed 1,167 times. Most of the ones caught running held 300 to 390 MiB.
  • A binary named hasaneyoh, 17 times, at 408 MiB.
  • r3's scanner, act3, 25 times, and sh processes, 79 times.
  • The Next.js server itself, 256 times, plus 35 small processes that also called themselves next-server.

When the kernel killed the server, the container exited and Kubernetes started it again. On November 20, every one of the 255 restarts for which both kinds of logs were complete came right after an out-of-memory kill of the Next.js server. The probe timeouts were real, but they weren't what restarted it.

Each restart also undid part of the attack. The container had no volumes, so every restart began from a fresh copy of the image, and everything the attackers had written to /tmp and /var/tmp was gone:

  • r3 kept a 2,517,802-URL target list and a progress file in /var/tmp/r3, and even set PERSIST=/var/tmp/r3. Its scan started over from the top at least 20 times, and the furthest any run got was 37%.
  • Bots claiming the box wrote ok:patched and ok:updated markers, and had to write each of them 12 times.
  • Every campaign simply exploited the pod again after a restart. Exploit traffic in the first ten minutes of a fresh container ran at about the same rate as two hours in.

The restart counter was a symptom of the compromise and, at the same time, the thing that kept undoing it. A memory limit isn't a security control, but here it worked like a crude, unplanned garbage collector for attacker tooling.

the restart loop · a symptom that kept undoing the attack

On November 20, the restart loop had five steps. One: a POST to the home page ran attacker code. Two: the attackers started scanners and dropped binaries into /tmp and /var/tmp. Three: those processes and the Next.js server shared one 512 MiB memory limit, and it filled. Four: the kernel OOM killer killed processes in the container, including the Next.js server 256 times. Five: the container restarted from a fresh copy of the image, wiping /tmp and /var/tmp, and the campaigns exploited it again. The last pod restarted 339 times that day; all 255 restarts with complete logs came right after an OOM kill of the Next.js server. The r3 scanner restarted its 2.5 million URL list at least 20 times and never got more than 37% through.

What the attackers tried on November 20

The November 20 output splits into at least seven distinct actors, told apart by their tools, their encodings and the addresses they came from.

Why this looks automated

The pace and repetition suggest that most of the activity was automated: thousands of exploit requests, a worm probing millions of targets, a scanner configured for 500 threads, and identical recon commands run again and again. AI agents may have been among the systems driving it, but these logs cannot distinguish an AI agent from a conventional script.

A worm hunting for more Next.js sites

The busiest by far: 3,925 reports, each a small JSON object like this one, with the pod name and the target domains replaced:

{"w":"<pod>","node":"v24.8.0","jobs":1,"scanned":1000,"found":3,
 "next":["<domain>","<domain>"],"hits":[],"probe":[],"internal":[],"stop":"tmo"}

Each report covers a batch of internet hosts probed for vulnerable Next.js, and in our logs the batches add up to 3,026,604 targets scanned from our pod. It also looked inside the cluster: its internal field shows it found exactly one Next.js server on the pod network, and that was the pod it was running in. It found nothing else.

The worm's main source address also sent cloud-metadata payloads straight at our main site, such as /_next/image?url=http://169.254.169.254/..., and got redirects. In our logs, scanners sent 165 requests with metadata URLs to the main site, and every one got a redirect, a 404 or 405, or the ordinary page or icon. The main site has Next.js image optimization switched off, so /_next/image answers every request with a 404 before it looks at the URL.

r3, a scanner with a 2.5-million-URL list

On November 20, an actor whose tooling called itself r3 dropped a scanner binary, act3, and a list of 2,517,802 URLs, s3, into /tmp and later /var/tmp/r3. It reported progress through base64 status digests, including its state file:

{"name": "r3", "input_file": "s3", "output_file": "act_hits3.txt",
 "total_urls": 2517802, "found_count": 4, "threads": 500,
 "timeout": "10s", "insecure": true}

That is 500 threads, 10-second timeouts and TLS verification switched off, resolving names through Cloudflare's and Google's public DNS at about 70 URLs a second. No run recorded more than 6 hits. It tried to persist, and no attempt survived a restart.

A host profiler

On November 20, we found 215 reports of the box's CPU model, AVX2 support, core count, RAM, usable memory and writable paths. That is the inventory you take before deciding what a machine is worth running, a miner or a bot for instance. Ours offered half a core and 512 MiB.

Secret sweeps

Several scripts, most of them from the worm's own address, read the pod's Kubernetes service-account token, its environment, package.json and directory listings. Then they went looking for .env files, LLM API keys, crypto exchange and custody keys, TLS private keys, cloud credentials, and Docker, Laravel and WordPress configuration. They found the token, which comes up again below, and nothing else: the image held no secrets, and the environment held only Node's, Next.js's and Kubernetes' own variables.

An operator after cloud credentials

The most deliberate actor came through Cloudflare egress addresses.

Google tokens. On November 20, GKE's metadata server logged 18 Google access-token requests from the old site's pods. Before we shut the site down, this operator took two distinct tokens out. On GKE with Workload Identity, that token is the pod's own Kubernetes service account acting as itself in Google Cloud. That identity had no IAM grants in any of our projects, so the tokens gave it no permission to read customer data. We found no recorded calls with it on November 20.

Other clouds. On November 20, the operator ran a shell script against the metadata endpoints of AWS (IMDSv2), Azure, DigitalOcean, Oracle Cloud and Google Cloud. It died early. The image has no curl, and the script used bash's ${T:0:20} substring syntax, which the image's /bin/sh (dash) rejects with Bad substitution.

The Kubernetes API. Minutes later they tried the pod's service-account token against the API server. With no curl, they used Node's built-in fetch. Their output, with the namespace name taken out:

==/version== 200
==/api/v1/namespaces/<namespace>/secrets?limit=30== 403
==/api/v1/namespaces/<namespace>/configmaps?limit=30== 403
==/api/v1/namespaces/<namespace>/pods?limit=20== 403

The namespace's default service account had no RBAC bindings, so the only thing its token got back was the API server's version.

A recon checker

Twenty times on November 20, the same address ran the same four commands: ps aux, free -h, ls /var/www/html and cat ~/.ssh/authorized_keys. All four failed every time. The slim image has no ps or free, there is no /var/www/html, and the user's home directory is /nonexistent. Its last run, at 10:44, is the one the audit caught.

An SSH key implant

On November 20, another actor tried 16 times to append its public key to ~/.ssh/authorized_keys. The pod runs no SSH server, and the home directory doesn't exist and can't be created.

Bots claiming the box

Several addresses left ok:patched and ok:updated markers on November 20, likely one or more bots marking the box as theirs. If "patched" means what it says, each restart undid that too.

what the pod could reach · and what came of it

What the compromised pod could reach on November 20, and what came of it. The internet: egress was open because the old site had no NetworkPolicy, and one worm scanned 3,026,604 targets from the pod, while r3 worked through a 2.5 million URL list. The Kubernetes API, with the automounted service-account token: /version answered, and listing secrets, configmaps and pods was refused with 403, because the account had no RBAC bindings. The GKE metadata server: attacker code fetched 18 Google tokens and two were taken out before shutdown, but the identity had no IAM grants and audit logs show no use. Other pods: the worm probed the pod network and found only its own pod. Other clouds' metadata endpoints: 404s, and the shell version of the script died. Files and environment: no secrets to find, and an attempt to add an SSH key failed.

What limited the damage, and what didn't

None of what held was designed for this incident, and most of it was defaults:

  • A slim image. No curl, wget, ps, free or strings. The attackers had to rebuild their tooling in Node or bring their own binaries, and a lot of their scripts simply failed.
  • A non-root user. uid 1001 with no effective capabilities (CapEff: 0000000000000000, from the attackers' own output), and a home directory that doesn't exist.
  • No secrets in the pod. Nothing in the environment and nothing in the image. The only credential on disk was the service-account token.
  • A service account with no RBAC bindings. Its token reached /version and nothing else.
  • A Workload Identity with no grants. The Google tokens unlocked nothing.
  • Restarts. Every out-of-memory kill of the server took the attackers' files with it.

And what didn't:

  • Egress was wide open. On November 20, the cluster enforced NetworkPolicy, but the old site's namespace had no policy. The pod could reach any address on the internet, which is how it scanned millions of hosts and used public DNS.
  • We left an internal repository unmaintained. Its :latest image was built the previous year and never rebuilt, with no digest pin, scanner or active owner.
  • The service-account token was mounted. The site never talked to the Kubernetes API, but the token sat in /var/run/secrets for anyone who could run code.
  • No seccomp profile. The attackers' output shows Seccomp: 0. RuntimeDefault wouldn't have stopped this attack, but it costs nothing.
  • No alert. Nothing paged on November 20's 500s on POST or on a restart count in the hundreds. A log audit found it.

Checking the blast radius

With the pod off, the question was what else the attackers had touched. We checked:

  • Every other workload's November 20 logs for the exploit's fingerprints: RCE_RES errors, zlib digests, the Function [as get] frame, /nonexistent probes and curl: not found. Nothing outside the old site.
  • Every image the cluster pulled on November 20, and every running workload. They came from our organization, known vendors or GKE. The old site's image came from the repository in our organization that we had stopped maintaining.
  • Every Google token the metadata server issued. Apart from the old site's, only the workloads we expect.
  • All six of our Google Cloud projects, with a read-only sweep on November 20 of IAM policies, service-account keys, VMs, firewall rules and audit logs. Only our own changes, and no denied attempts. It did turn up a leftover key, below.
  • Other workloads' databases. Their November 20 query logs and failed-login logs show nothing from the old site's pod addresses. Successful logins aren't logged, but the pod held no database credentials.
  • Abuse reports. Google sent none, even though the pod scanned millions of hosts from our address space.

The old landing page was isolated from customer data. It held no customer information, its identity had no IAM grants or Kubernetes RBAC bindings, and the checks above found no access to other workloads or databases. We are confident no customer data or other customer information was leaked.

What we did

  • Nov 20, 16:39. Scaled the deployment to zero, and the old landing page went offline.
  • Nov 20, 17:40. Deleted a leftover Cloud SQL service-account key from a cluster we had retired earlier. The sweep found it, and its audit trail showed no use on November 20. It had nothing to do with the attack, but it shouldn't have existed.
  • Retired the site. Deleted its namespace and removed its manifests from our deploy repository.
  • Turned on flow logs. VPC Flow Logs with GKE metadata and intranode visibility, routed through Pub/Sub into New Relic, where records have been arriving since 19:28 on Nov 20. Process output told us what the attackers ran but not where the pod connected. Now flows between pods name both pods, and internet-bound flows name the node they left from.
  • Rotated the nodes, so the node that hosted the pod is gone. Other workloads' databases moved to the new nodes on their own persistent disks.

What the November 20 logs could not show

  • The first successful command on November 20. A POST answered with 500 is a strong signal, but it cannot tell us the exact minute the first payload ran.
  • Where the pod connected before 19:28 on November 20. Flow logs were not yet on, so the attackers' outbound connections are known only from what their own tools reported.
  • Output we couldn't fully decode. The log line limit cut 218 outputs short, and a dozen used an encoding we didn't break.

Lessons

  • Pin images by digest, and rebuild and patch them on a schedule. A repository inside your own organization can still be forgotten while its image remains deployed.
  • Treat restart counts and 5xx on POST as security signals. A container whose restart counter reaches the hundreds in one day is telling you something, and so is a landing page answering POSTs with 500.
  • Default-deny egress with NetworkPolicy, and check that your cluster actually enforces it. A landing page needs to answer requests, not open connections to three million hosts.
  • Set automountServiceAccountToken: false on every workload that doesn't call the Kubernetes API.
  • Give Workload Identity no grants by default. Grant per workload, per need.
  • Keep flow logs. Process output tells you what ran. Only the network tells you where it went.

Software workers

The Codex job our team built connected the unusual POSTs, shell errors and restart count in one critical finding, then sent it to us in Slack, where we were already working. We verified it and took the site offline. Nobody needed to remember that forgotten page or go looking for trouble in a dashboard.

This incident gave us a glimpse of what observability, and software more broadly, could become. A software worker should act like a teammate with a defined responsibility: keep watch, investigate problems, bring evidence and a proposed next step to the place its users work, ask for approval or missing information, carry out authorized work, and verify the result. Dashboards can hold the detail; the worker brings the work to us.

That is the direction we want Failproof to take. It should meet users where they work and say, “This needs your attention. Here is what I found. If you approve these steps or give me this missing information, I can handle the next part and report back.” The person makes the decision; the software does the watching and follows through.

A checklist for your own cluster

This is the list we now hold every workload to. Most of it is a few lines of YAML.

Images

  • Every image is referenced by digest (image@sha256:…), not a mutable tag.
  • Every image has an active owner and a rebuild schedule, including images from older repositories in your own organization.
  • Framework and base-image versions are checked against advisories on a schedule, not only at build time.

Pods

  • automountServiceAccountToken: false unless the workload calls the Kubernetes API.
  • runAsNonRoot: true, allowPrivilegeEscalation: false, all capabilities dropped and seccompProfile: RuntimeDefault.
  • readOnlyRootFilesystem: true where the app allows it, with an emptyDir only where it has to write.
  • A slim or distroless image, without tools the app doesn't need.
  • Memory and CPU limits, so one compromised pod can't starve its node.

Network

  • A default-deny NetworkPolicy for ingress and egress in every namespace, with explicit allows on top. Confirm the cluster enforces NetworkPolicy at all.
  • Flow logs on, with pod metadata.

Identity

  • No IAM grants for a pod's identity unless the workload needs them, and then only on the resources it uses.
  • No RBAC bindings on default service accounts.

Detection

  • Alerts on restart counts, out-of-memory kills, and 5xx on POST for sites that shouldn't receive POSTs.
  • A regular look at Next.js error digests that aren't numbers or Next.js's own markers. This lists them for one deployment:
# Next.js's own digests are numbers or ALL_CAPS markers like NEXT_REDIRECT;...
kubectl logs deploy/web --since=168h |
  grep -oE "digest: '[^']+'" |
  grep -vE "digest: '([0-9]+|[A-Z_]+(;[^']*)?)'" |
  sort | uniq -c | sort -rn | head

The pod settings, as a deployment excerpt, plus the default-deny policy:

spec:
  template:
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: web
          image: ghcr.io/example/web@sha256:<digest>
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            limits:
              memory: 512Mi
              cpu: 500m
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
# Then allow ingress from your ingress controller, and egress only to what
# the app needs: DNS, if it resolves names at all.

FAQ

Which Next.js versions does GHSA-9qr9-h5gf-34mp affect?

Releases from 14.3.0-canary.77 up to the fixed versions: 15.0.5, 15.1.9, 15.2.6, 15.3.6, 15.4.8, 15.5.7 and 16.0.7. Later advisories raise the bar further, so check the current list rather than stopping at those.

How can I tell whether my Next.js app was exploited?

Look for POSTs answered with 500 on pages that don't take POSTs, error digests that aren't plain numbers, Command failed or not found errors coming from a web server, and the Object.Function [as get] frame. Then look at the pod's restarts, its out-of-memory kills, and your cloud metadata server's log for token requests from that workload.

What strings did these attackers leave?

WhereWhat we saw
Error names in the server logRCE_RES, and NEXT_REDIRECT carrying a base64 digest
Digestshex starting 789c; base64 that decodes to lines starting SEP_; plain text wrapped in <<RCE_OUT>> and <<RCE_END>>
Files/tmp/act3, /tmp/s3, /var/tmp/r3/, act_hits3.txt, .progress_r3.lock
Markersok:patched, ok:updated
Processes killed for memoryact3, hasaneyoh, .rsyslogd, and random 8- or 12-character names

The old landing page was something nobody thought about, which is exactly why it was the way in. If you run Next.js on Kubernetes, the checklist above is the hour we wish we had spent when we first deployed it.