🇧🇷 Português

K8s-One

A complete single-node Kubernetes cluster, built from scratch using individual components — no K3s, KIND, kubeadm, or any pre-packaged distribution.

Packaged in a minimal Debian-based image via multi-stage build (no package manager at runtime).

┌──────────────────────────────────────────────────┐
│                  k8s-one container               │
│                                                  │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │   etcd   │  │ apiserver│  │ ctrl-mgr │       │
│  └──────────┘  └──────────┘  └──────────┘       │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐       │
│  │scheduler │  │  kubelet │  │kube-proxy│       │
│  └──────────┘  └──────────┘  └──────────┘       │
│  ┌──────────┐  ┌──────────┐                      │
│  │containerd│  │   runc   │                      │
│  └──────────┘  └──────────┘                      │
│                                                  │
│  CNI: Cilium  │  DNS: CoreDNS                    │
│  Storage: local-path-provisioner (hostPath)      │
│  Ingress: HAProxy (Host ports 8082/8443)         │
│  Metrics: metrics-server (metrics.k8s.io API)    │
└──────────────────────────────────────────────────┘

---

Table of Contents

• Quick Start
• Components
• Architecture
• Persistent Volumes
• Configuration
• Secrets
• Cluster Access
• Usage Examples
• Project Structure
• Startup Sequence
• PKI & Certificates
• Networking
• Storage
• Known Issues
• Customization
• Troubleshooting
• Requirements
• Limitations

---

Quick Start

Copy .env.example to .env and set this machine's Tailscale IP address (tailscale ip -4):

ARGOCD_VERSION=v3.5.1
TAILSCALE_IP=100.x.y.z

Docker Compose loads this file automatically. It is ignored by Git so each environment can choose its Argo CD version and Tailscale address.

# Build
docker compose build

# Start
docker compose up -d

# Follow startup logs (~2-3 min on first run)
docker compose logs -f

# Get kubeconfig
docker cp k8s-one:/etc/kubernetes/admin-external.conf ./kubeconfig

# Use it
export KUBECONFIG=./kubeconfig
kubectl get nodes
kubectl get pods -A

Expected output:

NAME      STATUS   ROLES    AGE   VERSION
k8s-one   Ready    <none>   2m    v1.36.0

NAMESPACE            NAME                                       READY   STATUS
kube-system          cilium-operator-...                          1/1     Running
kube-system          cilium-...                                   1/1     Running
kube-system          coredns-...                                 1/1     Running
local-path-storage   local-path-provisioner-...                   1/1     Running
haproxy-controller   haproxy-kubernetes-ingress-...               1/1     Running

---

Components

All binaries are downloaded from official sources during build. No pre-packaged components are used.

Component	Version	Source	Role
kube-apiserver	v1.36.0	dl.k8s.io	Kubernetes REST API
kube-controller-manager	v1.36.0	dl.k8s.io	Controllers (replication, endpoints, etc.)
kube-scheduler	v1.36.0	dl.k8s.io	Pod scheduling on nodes
kubelet	v1.36.0	dl.k8s.io	Node agent, manages containers
kube-proxy	v1.36.0	dl.k8s.io	Network proxy (iptables mode)
kubectl	v1.36.0	dl.k8s.io	CLI for cluster interaction
etcd	v3.5.21	github.com/etcd-io	Cluster key-value store
containerd	1.7.27	github.com/containerd	Container runtime (CRI)
runc	v1.2.6	github.com/opencontainers	OCI runtime
CNI plugins	v1.6.2	github.com/containernetworking	Base network plugins
Cilium	v1.19.5	github.com/cilium/cilium	CNI — networking + network policy (eBPF)
Cilium CLI	v0.19.4	github.com/cilium/cilium-cli	Cilium installation & management
CoreDNS	v1.12.0	registry.k8s.io	Cluster DNS
local-path-provisioner	v0.0.30	github.com/rancher/local-path-provisioner	Dynamic hostPath provisioning (default StorageClass)
HAProxy Ingress	pinned by digest	haproxytech/kubernetes-ingress	Ingress Controller (HAProxy 3.2.21)
metrics-server	v0.9.0 (pinned by digest)	registry.k8s.io	metrics.k8s.io API — kubectl top / HPA

---

Architecture

Multi-Stage Build (Minimal runtime)

┌─────────────────────────────────────┐
│  Stage 1: Builder (alpine:3.21)     │
│                                     │
│  • curl, tar, gzip                  │
│  • Downloads all binaries           │
│  • Discarded in final image         │
└──────────────┬──────────────────────┘
               │ COPY binaries
               ▼
┌─────────────────────────────────────┐
│  Stage 2: Runtime (debian:bookworm) │
│                                     │
│  • bash, openssl, iptables          │
│  • socat, conntrack                 │
│  • apt/dpkg removed at build        │
│  • = minimal image, no pkg manager  │
└─────────────────────────────────────┘

The final image has no package manager — apt/dpkg are removed after installing runtime dependencies, reducing the attack surface.

Startup Process

The entrypoint.sh orchestrates the control-plane processes and manifest deployment:

entrypoint.sh
├── setup_mounts()        # mount --make-rshared /, /sys, bpf (Cilium)
├── detect_ip()           # detects container IP
├── generate_pki()        # generates 3 CAs + 11 certs + SA keys
├── generate_kubeconfigs() # generates 6 kubeconfigs
│
├── containerd ──────────▶ waits for socket
├── etcd ────────────────▶ waits for health (via etcdctl + TLS)
├── kube-apiserver ──────▶ waits for /healthz
├── kube-controller-manager
├── kube-scheduler
├── kubelet
├── kube-proxy
│
└── apply_manifests() [background]
    ├── taint removal (allows workloads)
    ├── cilium install (CNI, clean reinstall every boot)
    ├── waits for Node Ready
    ├── kubectl apply -f coredns/
    ├── kubectl apply -k local-path/ (provisioner + default StorageClass)
    ├── kubectl apply -f haproxy-ingress/
    └── kubectl apply -f metrics-server/

---

Persistent Volumes

All cluster state is stored in bind mounts under ./data/, ensuring persistence across restarts and making the data directly visible/backable on the host:

Host Path	Container Mount	Contents
./data/etcd/	/var/lib/etcd	etcd data (cluster state)
./data/containerd/	/var/lib/containerd	Images and containers
./data/kubelet/	/var/lib/kubelet	Kubelet state and pods
./data/pki/	/etc/kubernetes/pki	TLS certificates (CAs, certs, keys)
./data/kubernetes/	/etc/kubernetes	Kubeconfigs (admin, scheduler, etc.)
./data/local-path/	/opt/local-path-provisioner	Volumes provisioned by the local-path StorageClass

Additionally, the container bind-mounts host system paths:

Host Path	Container Path	Mode	Reason
/sys	/sys	rw	Cilium BPF, cgroups
/lib/modules	/lib/modules	ro	Kernel modules (iptables, etc.)

⚠️ ./data/ contains cluster secrets (PKI private keys, kubeconfigs) and, alongside ./data/local-path/, all volume data. Both are gitignored — never commit them.

Clean everything

docker compose down -v   # removes container + named volumes (bind mounts under ./data/ and ./data/local-path/ are kept)
# To fully wipe cluster data: rm -rf data/* data/local-path/*   (irreversible!)

---

Configuration

Build Args

All versions are configurable via build args in the Dockerfile:

# Use a specific Kubernetes version
docker compose build --build-arg KUBE_VERSION=v1.35.0

# Use a specific Cilium version
docker compose build --build-arg CILIUM_VERSION=v1.18.0

# Build for arm64 (untested)
docker compose build --build-arg TARGETARCH=arm64

Build Arg	Default	Description
KUBE_VERSION	v1.36.0	Kubernetes version
ETCD_VERSION	v3.5.21	etcd version
CONTAINERD_VERSION	1.7.27	containerd version
RUNC_VERSION	v1.2.6	runc version
CNI_VERSION	v1.6.2	CNI plugins version
CILIUM_VERSION	v1.19.5	Cilium version
CILIUM_CLI_VERSION	v0.19.4	Cilium CLI version
TARGETARCH	amd64	Target architecture

Environment Variables (runtime)

Variable	Default	Description
NODE_NAME	k8s-one	Node name in the cluster
ARGOCD_VERSION	v3.5.1	Argo CD version installed at boot (format: vX.Y.Z)

Set ARGOCD_VERSION in the root .env file. After changing it, recreate the container so the selected version is applied during startup:

docker compose up -d --force-recreate

Network Parameters (entrypoint.sh)

Parameter	Value	Description
CLUSTER_CIDR	192.168.0.0/16	Pod CIDR (Cilium auto-detects from controller-manager)
SERVICE_CIDR	10.96.0.0/12	ClusterIP CIDR
CLUSTER_DNS	10.96.0.10	CoreDNS IP

Resource Reserves (kubelet)

The kubelet advertises the host's memory as node capacity (the container has no memory limit of its own), so the reserves below are how the cluster tells the scheduler the truth: they deliberately "lie downward" so that pods get a realistic budget while the host stays protected.

Setting	Value	Effect
systemReserved	750m / 12Gi	Reserved for the host's own workload (desktop session, daemons)
kubeReserved	250m / 2Gi	Reserved for control plane + runtime (etcd, apiserver, kubelet, containerd)
evictionHard	memory.available: 1Gi	Kubelet starts evicting pods below this
enforceNodeAllocatable	[pods]	Makes the kubelet write the reserves into the kubepods cgroup (see below for what actually lands there)

On a 4 CPU / 23.2 GiB host that resolves to:

capacity         4 cpu      24346668Ki (23.2 GiB)
 - kubeReserved    250m        2 GiB
 - systemReserved  750m       12 GiB
 - evictionHard      --        1 GiB
 = allocatable     3 cpu      ~8.2 GiB   <- the scheduling budget

Three consequences worth internalising:

• kubectl top nodes reports more than 100% memory. /proc/meminfo is not namespaced, so the kubelet reads the host's usage while the denominator is the 8.2 GiB allocatable. A figure like 207% is not a leak — it is the entire desktop.
• Memory has exactly one hard cap, and it is ~9.2 GiB — not the allocatable 8.2 GiB. enforceNodeAllocatable makes the kubelet set memory.max on the kubepods cgroup to capacity − reserves (9898602496 = 9440 MiB), and the scheduler's 8.2 GiB is that figure minus the 1 GiB eviction margin. The cgroup driver is cgroupfs, so the path is /sys/fs/cgroup/kubepods (no .slice). The container itself is unbounded (Memory: 0), and control-plane processes run outside kubepods, covered only by the systemReserved reservation — not by an enforced limit.
• CPU has no cap at any layer. cpu.max is unset (max 100000) on both the container and on kubepods, so pods can burst past the 3 CPU allocatable whenever the host has idle cycles. Only scheduling is bounded by the 3 CPU — consumption is not.

Requests are the scheduling budget. The node's requests are what block rollouts: when they approach allocatable, a maxSurge replacement pod cannot be placed and matches 0/1 nodes are available: 1 Insufficient cpu. Actual usage is far below requests here, so keep an eye on the ratio — and remember that a chart can inject sidecars of its own (service-mesh proxies, exporters, log shippers) whose requests land in the pod without ever appearing in the values you wrote.

The config is written by write_kubelet_config() in scripts/entrypoint.sh to /var/lib/kubelet/config.yaml, which lives on the ./data/kubelet bind mount and therefore persists across container restarts. The function regenerates the file at boot whenever its content differs from the heredoc (keeping a timestamped .bak). A running kubelet does not reload its config, so changes take effect only after a container restart.

Do not shrink systemReserved to raise allocatable. It exists precisely because the node shares RAM with the desktop; pods are expected to fit in ~8 GiB.

---

Secrets

No credential is versioned. Values live in manifests/**/secrets/ (ignored by .gitignore) and are applied to the cluster by scripts/create-secrets.sh. The Argo CD Applications only reference the Secrets via existingSecret — they never contain a password.

Where they live

Secret	Namespace	Source file (gitignored)	Used by
authentik-config	platform	manifests/argocd/authentik/secrets/authentik.env	authentik.existingSecret
authentik-postgresql-auth	platform + data	manifests/argocd/authentik/secrets/postgresql-auth.yaml	PostgreSQL role authentik (init script, data)
grafana-admin	monitoring	manifests/argocd/prometheus-stack/secrets/grafana-admin.yaml	grafana.admin.existingSecret
grafana-oidc	monitoring	manifests/argocd/prometheus-stack/secrets/grafana-oidc.env	grafana.envFromSecret (GF_AUTH_GENERIC_OAUTH_* env)
litellm-env	platform	manifests/argocd/litellm/secrets/litellm.env	litellm.environmentSecrets
litellm-db	platform + data	manifests/argocd/litellm/secrets/litellm-db.yaml	litellm.db.secret + PostgreSQL init (data)
litellm-masterkey	platform	manifests/argocd/litellm/secrets/litellm-masterkey.yaml	litellm.masterkeySecretName
mkcert-ca	cert-manager	manifests/built-in/cert-manager/secrets/mkcert-ca.yaml	root CA for ClusterIssuer local-ca
postgres-superuser	data	manifests/argocd/postgres/secrets/postgres-superuser.yaml	PostgreSQL superuser (POSTGRES_PASSWORD)

Formats:

• *.env → created with kubectl create secret generic --from-env-file (e.g. authentik.env).
• *.yaml → a kind: Secret manifest (with stringData) applied with kubectl apply -f.

One credential, two namespaces. A Secret is namespaced, and the database lives in data while its consumers live in platform — so create-secrets.sh applies litellm-db.yaml and postgresql-auth.yaml to both namespaces (apply_yaml_secret_in_ns, which rewrites the namespace: field of the manifest on the fly). The source file stays the single truth: rotate the password in one place, never in one namespace only.

One credential, two secrets. authentik-config also carries the OIDC client credentials that LiteLLM authenticates with (LITELLM_OIDC_CLIENT_ID / LITELLM_OIDC_CLIENT_SECRET) — they must be the same values as GENERIC_CLIENT_ID / GENERIC_CLIENT_SECRET in litellm-env. Whoever registers the client (the blueprint) and whoever presents it (the proxy) are different apps, so the pair has to exist on both sides: rotate one, rotate both.

Create/update

scripts/create-secrets.sh          # idempotent; uses ./kubeconfig (or $KUBECONFIG)

Run it before applying the Argo CD Applications (the existingSecret must exist). Manually:

kubectl -n platform create secret generic authentik-config \
  --from-env-file=manifests/argocd/authentik/secrets/authentik.env \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f manifests/argocd/authentik/secrets/postgresql-auth.yaml
kubectl apply -f manifests/argocd/prometheus-stack/secrets/grafana-admin.yaml

Read

# one key of the authentik env
kubectl -n platform get secret authentik-config -o jsonpath='{.data.AUTHENTIK_POSTGRESQL__PASSWORD}' | base64 -d
# Grafana password
kubectl -n monitoring get secret grafana-admin -o jsonpath='{.data.admin-password}' | base64 -d
# every key of a secret
kubectl -n platform get secret authentik-postgresql-auth -o jsonpath='{.data}' | jq

Rules

• Never commit files under secrets/ nor embed a password in 02-application.yaml.
• Rotating a password = edit the source file in secrets/, run the script and restart the workload. For an application role on the shared PostgreSQL the password is written to the database at init: besides the Secret, run ALTER USER on the database (the init script only creates a role that does not exist yet).
• Make sure nothing is tracked: git check-ignore manifests/argocd/*/secrets/*.

---

Cluster Access

External Kubeconfig

# Copy kubeconfig from container
docker cp k8s-one:/etc/kubernetes/admin-external.conf ./kubeconfig

# Use it
export KUBECONFIG=./kubeconfig
kubectl get nodes
kubectl get pods -A
kubectl get sc

The external kubeconfig uses the container's IP as endpoint. To access from outside the Docker host, replace the IP in the kubeconfig with the host IP:

# Check the current IP in the kubeconfig
grep server kubeconfig

# Replace with the host IP (port 6443 is exposed in docker-compose)
sed -i 's|https://.*:6443|https://<HOST_IP>:6443|' kubeconfig

Internal Kubeconfig (inside the container)

docker exec k8s-one kubectl --kubeconfig=/etc/kubernetes/admin.conf get pods -A

---

Usage Examples

Simple Pod deployment

kubectl run nginx --image=nginx:alpine --port=80
kubectl get pods -w

PVC with local-path (ReadWriteOnce)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-data
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: local-path
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "echo 'Hello from K8s-One!' > /data/hello.txt && cat /data/hello.txt && sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-data

kubectl apply -f app.yaml
kubectl logs app
# Hello from K8s-One!

Shared volumes (ReadWriteMany)

local-path provisions node-local hostPath directories, so a PVC is ReadWriteOnce only. There is no cluster filesystem for ReadWriteMany — use an application-level mechanism (object storage, NFS, or a database) for shared data.

Network Policy with Cilium

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Deployment with Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:alpine
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP

Ingress with HAProxy

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  annotations:
    haproxy.org/ingress.class: haproxy
spec:
  rules:
  - host: my-app.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 80

kubectl apply -f ingress.yaml
curl -H "Host: my-app.local" http://localhost:8082/

---

Project Structure

k8s-one/
├── Dockerfile                          # Multi-stage build (alpine builder + debian runtime)
├── docker-compose.yaml                 # Execution with persistent volumes
├── README.md                           # Documentation (English)
├── README.pt-BR.md                     # Documentation (Portuguese)
│
├── scripts/
│   ├── entrypoint.sh                   # Orchestration: PKI, configs, processes, manifests
│   ├── deploy-apps.sh                  # Applies manifests/apps via kustomize (no docker cp)
│   └── create-secrets.sh               # Creates/updates Secrets from manifests/**/secrets/
│
├── configs/
│   └── containerd-config.toml          # containerd: runc + cgroupfs + overlayfs
│
└── manifests/                          # GITIGNORED — mounted ro into the container
    ├── built-in/                       # Core: applied by entrypoint.sh on every boot
    │   ├── coredns/                    # CoreDNS (one Kubernetes resource per file)
    │   │   ├── 01-service-account.yaml
    │   │   ├── 02-cluster-role.yaml
    │   │   ├── 03-cluster-role-binding.yaml
    │   │   ├── 04-configmap.yaml
    │   │   ├── 05-deployment.yaml
    │   │   └── 06-service.yaml
    │   ├── haproxy-ingress/            # HAProxy Ingress (one Kubernetes resource per file)
    │   │   ├── 01-namespace.yaml
    │   │   ├── 02-service-account.yaml
    │   │   ├── 03-cluster-role.yaml
    │   │   ├── 04-cluster-role-binding.yaml
    │   │   ├── 05-configmap.yaml
    │   │   ├── 06-deployment.yaml
    │   │   └── 07-service.yaml
    │   ├── metrics-server/             # Metrics API (one Kubernetes resource per file)
    │   │   ├── 01-service-account.yaml
    │   │   ├── 02-aggregated-metrics-reader-cluster-role.yaml
    │   │   ├── 03-metrics-server-cluster-role.yaml
    │   │   ├── 04-auth-reader-role-binding.yaml
    │   │   ├── 05-auth-delegator-cluster-role-binding.yaml
    │   │   ├── 06-metrics-server-cluster-role-binding.yaml
    │   │   ├── 07-service.yaml
    │   │   ├── 08-deployment.yaml
    │   │   └── 09-api-service.yaml
    │   └── local-path/                  # local-path-provisioner + default StorageClass
    │       ├── local-path-storage.yaml  # Namespace, RBAC, Deployment, StorageClass, ConfigMap
    │       └── kustomization.yaml
    │   ├── metallb/                     # MetalLB L2 (LB for services; see "Local DNS" — not the external path)
    │   │   ├── 00-crds.yaml … 07-webhook.yaml
    │   │   ├── 02-ipaddresspool.yaml   # 192.168.1.200-250 (LAN)
    │   │   └── kustomization.yaml
    │   └── cert-manager/                # Internal *.lan certificates (mkcert CA)
    │       ├── 00-crds.yaml … 05-webhooks.yaml
    │       ├── cluster-issuer.yaml     # ClusterIssuer "local-ca" (mkcert CA)
    │       ├── certificate-dns-lan.yaml# dns.lan + *.lan → secret dns-lan-tls
    │       ├── kustomization.yaml
    │       └── secrets/
    │           └── mkcert-ca.yaml      # mkcert root CA (never commit)
    ├── argocd/                          # Applications (Argo CD) — Helm charts
    │   ├── authentik/
    │   │   ├── 02-application.yaml      # existingSecret: authentik-config (db: shared PostgreSQL)
    │   │   ├── 03-blueprint.yaml        # OIDC provider/group/app (kubectl apply — never through Helm)
    │   │   └── secrets/                 # GITIGNORED (never commit)
    │   │       ├── authentik.env        # app env (AUTHENTIK_* + LITELLM_OIDC_*)
    │   │       └── postgresql-auth.yaml # PostgreSQL auth (postgres-password/password)
    │   ├── litellm/
    │   │   ├── 01-repo-secret.yaml      # OCI repo (ghcr.io/berriai, enableOCI)
    │   │   ├── 02-application.yaml      # litellm-helm chart + migrations hook (PreSync)
    │   │   └── secrets/                 # GITIGNORED (never commit)
    │   │       ├── litellm.env          # app env (OPENAI_API_KEY, PROXY_BASE_URL, OIDC...)
    │   │       ├── litellm-db.yaml      # PostgreSQL creds (username/password)
    │   │       └── litellm-masterkey.yaml # proxy master key (masterkey)
    │   ├── postgres/
    │   │   ├── 02-application.yaml      # source: this repo (path manifests/postgres)
    │   │   └── secrets/                 # GITIGNORED (never commit)
    │   │       └── postgres-superuser.yaml # superuser password (postgres-password)
    │   ├── prometheus-stack/
    │   │   ├── 02-application.yaml      # existingSecret: grafana-admin + authentik SSO (grafana.ini)
    │   │   ├── 03-certificate.yaml      # grafana-tls (grafana.lan) — kubectl apply -f
    │   │   ├── 04-ingress.yaml          # grafana.lan -> chart Service — kubectl apply -f
    │   │   └── secrets/
    │   │       ├── grafana-admin.yaml   # Grafana admin (admin-user/admin-password)
    │   │       └── grafana-oidc.env     # SSO OIDC client (GF_AUTH_GENERIC_OAUTH_*)
    ├── postgres/                       # PostgreSQL 18 — shared instance (own manifests, from this repo)
    │   ├── 01-pvc.yaml                 # PVC postgres-data (5Gi, local-path)
    │   ├── 02-deployment.yaml          # postgres:18.6 (ns data)
    │   ├── 03-configmap-initdb.yaml    # init: creates the app roles/databases
    │   ├── 04-service.yaml             # Service postgres + NodePort 30432
    │   └── kustomization.yaml
    ├── apps/                           # On-demand; one Kubernetes resource per YAML file
    │   ├── kustomization.yaml          # Composes the app directories
    │   └── tileserver/                 # TileServer GL

---

Startup Sequence

Typical timeline for a first run (cold start, no image cache):

 0s   ▶ Mount propagation (rshared /, /sys, bpf)
 0s   ▶ PKI generation (3 CAs, 11 certs, SA keypair)
 1s   ▶ Kubeconfig generation (6 files)
 1s   ▶ containerd start → socket ready
 2s   ▶ etcd start → health check OK
 5s   ▶ kube-apiserver start → /healthz OK
 7s   ▶ kube-controller-manager / scheduler / kubelet / kube-proxy
10s   ▶ cilium install (clean reinstall every boot)
35s   ▶ Node Ready ✓
40s   ▶ CoreDNS, local-path-provisioner, HAProxy applied

On subsequent restarts (images already cached), boot drops to ~1-2 min. Provisioned volume data survives via data/local-path/.

---

PKI & Certificates

The entrypoint generates the full PKI on first run. Certificates are persisted in the ./data/pki/ bind mount and reused across restarts.

CAs (Certificate Authorities)

CA	CN	Usage
ca	kubernetes-ca	Cluster root CA
etcd/ca	etcd-ca	etcd CA (separate)
front-proxy-ca	front-proxy-ca	Aggregation layer CA

Certificates

Cert	CA	CN	O (Org)	SANs
apiserver	ca	kube-apiserver	—	kubernetes, kubernetes.default, *.svc, 127.0.0.1, NODE_IP, 10.96.0.1
apiserver-kubelet-client	ca	apiserver-kubelet-client	system:masters	—
admin	ca	kubernetes-admin	system:masters	—
controller-manager	ca	system:kube-controller-manager	—	—
scheduler	ca	system:kube-scheduler	—	—
kubelet	ca	system:node:k8s-one	system:nodes	—
kube-proxy	ca	system:kube-proxy	—	—
front-proxy-client	front-proxy-ca	front-proxy-client	—	—
etcd/server	etcd/ca	etcd-server	—	localhost, NODE_NAME, 127.0.0.1, NODE_IP
etcd/client	etcd/ca	etcd-client	—	—
apiserver-etcd-client	etcd/ca	apiserver-etcd-client	—	—

Service Account

File	Type
sa.key	RSA 2048 private key
sa.pub	Public key (for token verification)

All certificates have a validity of 10 years (3650 days).

---

Networking

Cilium

• Datapath: eBPF
• Pod CIDR: 192.168.0.0/16 (auto-detected from kube-controller-manager)
• Network Policy: ✅ supported (CiliumNetworkPolicy + k8s NetworkPolicy)
• IPAM: cluster-pool (default)
• kube-proxy replacement: disabled (kube-proxy runs alongside)
• Hubble: ✅ observability & monitoring

Cilium is installed via the Cilium CLI, which manages the Helm chart and provides status monitoring. It is fully uninstalled and reinstalled on every boot (the in-memory BPF datapath does not survive a container restart).

kube-proxy

• Mode: iptables
• Service CIDR: 10.96.0.0/12

CoreDNS

• ClusterIP: 10.96.0.10
• Forward: 8.8.8.8, 1.1.1.1 (Google DNS, Cloudflare)
• Domain: cluster.local
• .lan names: resolved in-cluster by a hosts block pointing at the MetalLB VIP (192.168.1.200)

The hosts block is needed because the forward above goes straight to public resolvers, which do not know .lan (a private TLD only AdGuard serves) — without it no pod reaches authentik.lan, grafana.lan etc. by name, only by ClusterIP. The IP is the MetalLB VIP (the haproxy-kubernetes-ingress LoadBalancer service), not the 192.168.1.20 used by the AdGuard rewrites: .20 is the host's LAN IP and is unreachable from inside the cluster (it times out). The VIP is reachable, and routes by Host header with the mkcert certs.

It is an explicit list, not a wildcard: the hosts plugin only gained wildcard support on CoreDNS master — no release has it (this cluster runs v1.12.0), so *.lan there would be stored as a literal name and silently never match. The template alternative would work but makes every *.lan answer the VIP, including router.lan (AdGuard points that at 192.168.1.1), with no way to carve an exception (Go/RE2 has no lookahead). A new .lan service is one more word on that line, in manifests/built-in/coredns/04-configmap.yaml. The reload plugin picks the change up in ~30 s, no restart.

Local DNS (AdGuard) & Internal Certificates

AdGuard Home (apps/adguard) is the LAN/tailnet DNS and resolves the internal *.lan names to the cluster. CoreDNS still owns cluster.local (in-cluster DNS) — AdGuard is for accessing apps by name.

Flow of a request to https://dns.lan (AdGuard dashboard):

device → AdGuard (192.168.1.20:53)
       → rewrite "dns.lan → 192.168.1.20"         (config in AdGuardHome.yaml, PVC adguard-conf-fs)
       → browser → 192.168.1.20:443 (host-published port)
       → HAProxy Ingress (Host: dns.lan) → service adguard:80 → UI (3000)

• Why .lan, not .local: .local is reserved for mDNS (RFC 6762). Android and macOS resolve .local via mDNS and never via unicast DNS — so dns.local fails on those devices even with the right DNS. .lan goes through the normal unicast resolver.
• AdGuard rewrite: dns.lan → 192.168.1.20 (the host's fixed LAN IP — not the MetalLB IP; see below).
• External access = host-published ports: the cluster runs inside an isolated docker network (192.168.32.0/20). MetalLB (in built-in/metallb) announces its LoadBalancer IP (192.168.1.200) inside that docker network, not on the WiFi — so it is not reachable from the LAN. The real path is docker-compose publishing 0.0.0.0:80/443 → NodePort 30080/30443 on the host's physical IP (192.168.1.20). The rewrite points to 192.168.1.20, not 192.168.1.200.
• Certificates (cert-manager + mkcert): the ClusterIssuer local-ca uses the mkcert root CA (~/.local/share/mkcert/rootCA.pem, imported into Secret cert-manager/mkcert-ca). The Certificate dns-lan issues dns.lan + *.lan into Secret infra/dns-lan-tls, referenced by the Ingress. To avoid browser warnings, install the mkcert CA into each device's trust store (already done on the host via mkcert -install).
• Tailscale: global nameserver = 192.168.1.20 and the 192.168.1.0/24 route advertised + approved — tailnet devices resolve *.lan via AdGuard and reach 192.168.1.20. IPv6 (RA) must be off on the router: the IPv6 DNS advertisement (fc00::a/b) is preferred by Android and breaks *.lan resolution.

Remote access (outside the LAN, via Tailnet)

Works from anywhere with Tailscale on: the tailnet DNS (global nameserver 192.168.1.20) resolves *.lan via AdGuard, and the 192.168.1.0/24 subnet route forwards traffic to 192.168.1.20 (host) → Ingress → app. E.g. https://argocd.lan away from home.

Gotcha (Termux): Termux uses its own resolv.conf ($PREFIX/etc/resolv.conf, pointing at 8.8.8.8) — so nslookup/curl inside Termux do not resolve .lan, even though the browser works. Diagnosis:

• nslookup argocd.lan 192.168.1.20 → queries AdGuard directly (works).
• nslookup argocd.lan 100.100.100.100 → tailnet MagicDNS (works if the tailnet DNS is applied on the device). To test access, use the browser (it uses the system DNS).

• Router (LAN): DHCP hands out 192.168.1.20 as primary DNS (optional 1.1.1.1 fallback). Note: with the host down, clients that only have 192.168.1.20 lose DNS.

Headlamp (headlamp.lan)

Dashboard at https://headlamp.lan (ingress routes by Host; mkcert cert headlamp.lan). Login é SSO no Authentik, feito por um oauth2-proxy na frente do Headlamp.

Arquitetura (por que o proxy): o Headlamp tem dois modos de OIDC, mutuamente exclusivos:

• OIDC nativo: exige login e encaminha o token do usuário para a API. Como o kube-apiserver deste cluster não tem flags --oidc-*, a API rejeita o token (401, "the cluster did not accept your sign-in").
• Service account (HEADLAMP_CONFIG_UNSAFE_USE_SERVICE_ACCOUNT_TOKEN=true): autentica todos pelo SA, mas não exige login — só é seguro atrás de um auth proxy.

Escolhemos o 2º + um proxy: navegador → ingress → oauth2-proxy → Headlamp. O oauth2-proxy (ns ops, 09-oauth2-proxy.yaml) faz o OIDC contra o Authentik (provider "Headlamp", redirect https://headlamp.lan/oauth2/callback) e guarda a sessão num cookie; o ingress aponta para ele, não para o Headlamp. Sem sessão válida, nada chega ao Headlamp.

RBAC: SA headlamp-admin (ns ops) → ClusterRole headlamp-admin — admin amplo, mas sem delete em namespaces, PVs e PVCs (guarda-corpo de dados; ver 03-cluster-role.yaml). O login controla quem entra; a permissão na API é a do SA (admin compartilhado).

Env relevante do Headlamp (05-deployment.yaml): HEADLAMP_CONFIG_UNSAFE_USE_SERVICE_ACCOUNT_TOKEN=true + HEADLAMP_CONFIG_PROXY_AUTH=true (confia nos headers X-Forwarded-* do proxy). O client_id/secret é o mesmo par HEADLAMP_OIDC_* do blueprint, guardado no Secret gitignored oauth2-proxy (create-secrets.sh), junto do cookie_secret.

Alternativa per-user (não usada): configurar OIDC no próprio kube-apiserver (--oidc-*) + RBAC por identidade — muda o modelo para per-user e exige rebuild/recreate do container.

Logout ("Sair"): o Headlamp não tem logout no servidor (o botão nativo só limpa o token local, e a sessão real é o cookie _oauth2_proxy). Por isso há um plugin próprio (plugin-logout/, montado em /headlamp/static-plugins/logout via ConfigMap) que adiciona um botão Sair no app bar. Ele vai para /oauth2/sign_out?rd=<end-session do Authentik>: limpa o cookie do proxy e encerra a sessão SSO no Authentik, voltando para o login. O plugin é escrito à mão em UMD (o Headlamp injeta os módulos em window.pluginLib), sem toolchain/npm.

Fallback por token (quando o prompt pede token):

kubectl -n ops get secret headlamp-admin-token -o jsonpath='{.data.token}' | base64 -d

The headlamp-admin-token secret (type kubernetes.io/service-account-token) is long-lived (K8s 1.24+). Since Headlamp runs with --in-cluster, it may authenticate automatically via the projected token — the command above is for when the login prompt asks for a token.

Argo CD (argocd.lan)

Accessible at https://argocd.lan (Ingress ns argocd → argocd-server:80, mkcert TLS argocd.lan). Requires --insecure on argocd-server: without it, the ingress (which terminates TLS and forwards plain HTTP to the backend) causes a 307 redirect loop. The patch is re-applied by entrypoint.sh after applying the upstream install.yaml (persistent across reboots).

Login: user admin, password:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d

Tip: login uses user admin + the password from the secret above (it matches the hash in argocd-secret/admin.password). If the browser rejects it, check autofill/cache (type the password manually; hard refresh or incognito) — it is not the headlamp/dns password.

Ingress/cert manifests: manifests/apps/argocd/ (gitignored).

Authentik (authentik.lan)

Identity provider at https://authentik.lan (chart authentik 2026.8.1, ns platform, mkcert cert authentik-tls). Its database is the shared PostgreSQL 18 of the data namespace (see PostgreSQL (data) below) — database authentik, owned by the role authentik — so the chart's embedded postgresql: is enabled: false and the connection comes entirely from the authentik-config Secret (AUTHENTIK_POSTGRESQL__*, delivered to server and worker by envFrom). It is here to be the single login for the cluster's apps; wired to it so far: LiteLLM, Headlamp and Grafana (Argo CD planned).

Groups, providers and applications are declarative, via a blueprint the chart mounts into the worker:

kubectl apply -f manifests/argocd/authentik/03-blueprint.yaml    # the blueprint ConfigMap
kubectl apply -f manifests/argocd/authentik/02-application.yaml  # then let Argo CD sync the chart

• manifests/argocd/authentik/03-blueprint.yaml is a ConfigMap holding three blueprints, one per app: litellm-oidc.yaml (group litellm-users, OAuth2 provider LiteLLM, application and bindings), headlamp-oidc.yaml (same, for Headlamp — with oauth2-proxy running the flow) and grafana-oidc.yaml (same, for Grafana, which speaks OIDC natively).
• It is applied with kubectl, never through Helm: blueprint tags (!Find, !KeyOf, !Env) are custom YAML, and Helm's values → toYaml round-trip destroys them (they arrive in the cluster as bare strings and the blueprint fails).
• The chart mounts every name in blueprints.configMaps (in 02-application.yaml) into the worker at /blueprints/mounted/cm-<name>; the worker discovers all *.yaml there.
• Discovery is event-driven, not boot-time. What triggers it is the file watcher (on_created/on_modified) plus an hourly scheduled run. A ConfigMap that is already populated when the worker starts fires nothing — the mount happens before the process is up. To force it without waiting: change the ConfigMap data (e.g. a comment in the blueprint) and the kubelet resyncs the volume, producing the events. kubectl annotate does not work: metadata does not make the kubelet resync.
• Idempotency comes from identifiers, not from id: the importer builds a filter() from identifiers and, if it finds the object, updates it (partial=True); otherwise it creates. The entry id exists only so other entries can point at it with !KeyOf. An entry without identifiers aborts with "No or invalid identifiers".
• List-valued fields have an empty default and must be declared. grant_types is ArrayField(..., default=list) on the model: the UI wizard fills it in, a blueprint does not. Omit it and the provider is created with grant_types = {} — it then rejects every grant (Invalid grant_type for provider in the server log) and /authorize answers invalid_request, which looks like a completely unrelated bug. Same trap for any other ArrayField (property_mappings above is the same shape, with a less obvious symptom: a token without the email claim).
• The client credentials are !Env, resolved against the worker's environment — which receives every key of the authentik-config Secret (envFrom). They live in secrets/authentik.env (gitignored). Note !Env returns None for a missing variable instead of failing loudly, so the salt is on the other side: the authentik serializer rejects a null client_secret, and the sync errors out.
• The admin's email is kept in sync with the Secret. AUTHENTIK_BOOTSTRAP_EMAIL is only consumed when the database is created — fix the email afterwards and the user keeps the old one. That matters beyond tidiness: the email is what LiteLLM uses to provision its own user on first login. So the blueprint carries an authentik_core.user entry setting email: !Env AUTHENTIK_BOOTSTRAP_EMAIL (the value stays in the gitignored Secret rather than being written into a tracked file). The importer's partial=True means only that field is touched on the admin user.

Admin login is akadmin:

kubectl -n platform get secret authentik-config -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_EMAIL}' | base64 -d; echo
kubectl -n platform get secret authentik-config -o jsonpath='{.data.AUTHENTIK_BOOTSTRAP_PASSWORD}' | base64 -d; echo

The password above is the bootstrap value: it is consumed when the database is created. Editing it in secrets/authentik.env afterwards does not change the password of an existing akadmin — that is done in the UI (Settings → Password) or by resetting the flow.

LiteLLM (litellm.lan)

OpenAI-compatible proxy at https://litellm.lan (Ingress ns platform → litellm:4000, dedicated mkcert cert litellm-tls). It replaces the LiteLLM that used to run in the infra/ docker-compose stack; the database is the cluster's shared PostgreSQL 18 (ns data, database litellm, ?schema=litellm) — the same instance that serves the Authentik, in a database of its own.

DNS: litellm.lan must be added as a rewrite in AdGuard (Filters → DNS rewrites → 192.168.1.20), like the other .lan names. AdGuard rewrites are per host, not wildcards, and the config lives inside the adguard-conf-fs PVC (not in this repo), so this is a manual one-time step.

# master key — the API bearer token (NOT a UI login, see below)
kubectl -n platform get secret litellm-masterkey -o jsonpath='{.data.masterkey}' | base64 -d
# list models
curl -sk https://litellm.lan/v1/models -H "Authorization: Bearer $MASTER_KEY"

UI login is SSO through Authentik (provider LiteLLM, see the Authentik section above). Redirect URI: https://litellm.lan/sso/callback; access is restricted to the litellm-users group. The whole switch is environment:

• The keys are GENERIC_*, not GOOGLE_*. LiteLLM picks the provider in a Google → Microsoft → Generic if/elif, so while GOOGLE_CLIENT_ID exists the generic block is dead code — removing the two Google keys is what actually flips the provider.
• Three endpoints are configured by hand (authorize, token, userinfo): LiteLLM does not use OIDC discovery, so there is no /.well-known lookup.
• PROXY_BASE_URL (https://litellm.lan) is what composes the redirect URI; it must match what is registered on the provider.
• The master key is unaffected by SSO — it stays the API bearer token. It is not a UI password: POST /login with admin + master key returns 401 here.
• Without a LITELLM_LICENSE, SSO is capped at 5 users (ui_sso.py refuses beyond that). The LiteLLM_UserTable starts empty, so it only matters if more people ever log in.

TLS when calling Authentik. The pod reaches Authentik at https://authentik.lan, which inside the cluster resolves to the MetalLB VIP and serves a mkcert certificate — not trusted by the Debian CA bundle in the image, so the token exchange would fail hostname/issuer verification. The deployment therefore mounts the mkcert root (the ca.crt of the litellm-tls secret, already in the platform namespace) and an initContainer concatenates it with the system bundle, pointing SSL_CERT_FILE/REQUESTS_CA_BUNDLE at the result. Concatenating rather than replacing matters: the proxy also calls api.openai.com, whose cert is not mkcert-signed.

Deployment details worth knowing before touching it:

• Chart: official litellm-helm, pulled as an OCI chart from ghcr.io/berriai (the classic Helm index berriai.github.io/litellm-helm is 404). The repo Secret's url is the parent of the chart in the OCI path — Argo CD builds oci://<url>/<chart>, so url: ghcr.io/berriai + chart: litellm-helm. targetRevision must be an exact tag (OCI has no semver ranges).
• No IngressClass in this cluster: every Ingress routes via the haproxy.org/ingress.class annotation and has CLASS <none>. The chart's ingress.className is therefore set to "" (its default, nginx, would render ingressClassName: nginx and break routing).
• One PreSync hook runs before every sync: the chart's own litellm-migrations job, which runs prisma migrate deploy before the Deployment. It is safe to re-run. Nothing creates the role/database/schema at sync time — they are born with the instance, from the init script of the shared PostgreSQL (manifests/postgres/03-configmap-initdb.yaml).
• ENFORCE_PRISMA_MIGRATION_CHECK=true is required too. Without it LiteLLM logs "migration failed but continuing startup" and exits 0 — the Job shows as Completed against a half-migrated database. With it, a migration failure fails the hook and stops the sync.
• Memory: 2Gi limit, not less. Both the migration job (~1.7Gi peak) and the proxy are OOMKilled at 1Gi. strategy: Recreate avoids two proxies during a rollout on this single, memory-tight node.
• Metrics need the callback: /metrics only exists when litellm_settings.callbacks: [prometheus] is set — without it LiteLLM returns 404 and the Prometheus target stays DOWN (the ServiceMonitor itself works: the scrape does happen). With the callback on, the endpoint also demands the API key, hence require_auth_for_metrics_endpoint: false (it is a ClusterIP endpoint).

Manifests: manifests/argocd/litellm/. Secrets: see the table above.

Grafana (grafana.lan)

Cluster metrics and dashboards: kube-prometheus-stack (chart 90.0.0, ns monitoring), with Grafana 13.2.1-distroless, a 10Gi local-path PVC and the datasource/dashboard sidecars reading ConfigMaps. Served at https://grafana.lan — Ingress grafana + Certificate grafana-tls (manifests/argocd/prometheus-stack/03-certificate.yaml and 04-ingress.yaml, applied with kubectl apply -f, because the Application points at the third-party chart and not at this repo).

SSO through Authentik (provider Grafana, blueprint grafana-oidc.yaml). Grafana speaks OIDC natively — no proxy in front like Headlamp; Authentik only delivers the claims:

• Access through the grafana-users group (Application binding, a single place: Grafana does not use allowed_groups).
• Role from the group, via role_attribute_path (JMESPath): grafana-admins → GrafanaAdmin, everyone else → Viewer. Unlike LiteLLM there is no custom scope mapping here: Grafana evaluates the groups claim, and that claim already comes from the profile scope. What Grafana compares is the group name, so the group→role mapping lives in grafana.ini, not in the blueprint.
• role_attribute_strict = true: if the groups claim is missing the login is denied instead of silently demoting everyone to Viewer (the LiteLLM lesson, which fell back to internal_user_viewer without telling anyone). allow_assign_grafana_admin = true is what makes the GrafanaAdmin above count as a server admin; without it it would only be org Admin.
• The role is re-synced on every login: promoting/demoting someone means editing the group in Authentik, not Grafana.
• The local login (admin + the grafana-admin secret) still works — it is the break-glass path (it does not go through Authentik) and it is what the dashboard sidecars use to talk to the local API.

Three details that are expensive to get wrong:

• root_url is mandatory. The flow's redirect_uri is derived from it; without root_url Grafana builds the URL from the request Host (which arrives as http, behind the ingress) and Authentik rejects it for not matching the https://grafana.lan/login/generic_oauth registered on the provider.
• The mkcert CA is mounted, through extraSecretMounts. The pod is distroless (no shell) and runs with a read-only rootfs, so there is no way to concatenate a bundle the way LiteLLM does: the ca.crt from the grafana-tls secret itself is mounted (every cert-manager tls secret carries the issuer's ca.crt) and tls_client_ca points at it — the Go equivalent of oauth2-proxy's --provider-ca-file. Do not use extraVolumes for this: the chart template only renders existingClaim/hostPath/csi/configMap/emptyDir, and a secret: volume falls silently into an empty emptyDir — the pod comes up, the file does not exist, and SSO only fails at login time, far from the cause.
• The credential never enters grafana.ini. The client_id/secret pair comes from the grafana-oidc Secret as GF_AUTH_GENERIC_OAUTH_* env vars (envFromSecret), which override the ini — that is what the chart's assertNoLeakedSecrets checks at render time.

Manifests: manifests/argocd/prometheus-stack/. Secrets: see the table above.

PostgreSQL (data)

The cluster's shared database instance: one PostgreSQL 18.6 — the official postgres:18.6 image, no chart — in namespace data, serving both apps that need a real SQL database, each in its own database and role: litellm (role litellm, schema litellm, Prisma's) and authentik (role authentik, Django's). One instance, two tenants: the roles are created with least privilege (NOSUPERUSER NOCREATEDB NOCREATEROLE) and the superuser never leaves the pod.

Its manifests are versioned in this repo, under manifests/postgres/, because there is no upstream chart to pin — only the official image. The Application postgres (manifests/argocd/postgres/02-application.yaml) therefore follows the vaultwarden pattern: source pointing at this repository, path: manifests/postgres, where a kustomization.yaml composes the PVC, the Deployment, the init ConfigMap and the Service (Argo CD detects kustomize on its own). The PVC carries Prune=false,Delete=false — it holds real data and must not vanish when the Application is pruned or deleted.

In-cluster the address is postgres.data.svc.cluster.local:5432, which is how LiteLLM and Authentik connect. From the host, the Service is a NodePort (30432) that docker-compose publishes as 127.0.0.1:5432:30432 — loopback on purpose: every other published port exists so the LAN can reach the host, but the database must not leave it. NodePort and not LoadBalancer for the same reason the ingresses are: the MetalLB VIP is announced inside the cluster's docker network and is unreachable from the LAN. And 30432 and not 5432 because a NodePort has to sit in the apiserver's 30000-32767 range.

Authentication is scram-sha-256 for every remote connection: the image's entrypoint appends host all all all scram-sha-256 to pg_hba.conf, and that line catches all TCP — including the traffic that arrives through the NodePort. The only trust left is the unix socket and the loopback inside the pod (initdb's default), unreachable from outside, since NodePort traffic arrives with the client's source IP, never 127.0.0.1. POSTGRES_HOST_AUTH_METHOD is deliberately left unset — setting it to trust would be a passwordless superuser. The superuser password lives in the gitignored Secret postgres-superuser; each application only ever receives the credentials of its own role, in Secrets applied to both platform and data (see Secrets).

The volume mounts at /var/lib/postgresql, not at /var/lib/postgresql/data. In PostgreSQL 18 the image moved PGDATA to /var/lib/postgresql/18/docker with the VOLUME declared on the parent, and mounting at the v15–v17 path makes the entrypoint abort at boot with "there appears to be PostgreSQL data in: /var/lib/postgresql/data (unused mount/volume)". Leaving PGDATA at its default is also what keeps pg_upgrade --link viable for a future major upgrade.

---

Storage

local-path-provisioner

Storage is provided by local-path-provisioner (Rancher v0.0.30) in the local-path-storage namespace. It dynamically provisions volumes as hostPath directories under /opt/local-path-provisioner, persisted on the host by the ./data/local-path bind mount.

• Provisioner: rancher.io/local-path
• Data path: /opt/local-path-provisioner (bind mount from ./data/local-path)

StorageClass	Provisioner	Access	Binding	Reclaim	Expansion
local-path (default)	rancher.io/local-path	RWO	WaitForFirstConsumer	Delete	not supported

kubectl get sc
# NAME                   PROVISIONER             RECLAIMPOLICY  VOLUMEBINDINGMODE
# local-path (default)   rancher.io/local-path   Delete         WaitForFirstConsumer

Volumes are not redundant: they live as plain directories on the host disk under data/local-path/. Back that directory up if the data matters. Since local-path has no online expansion, PVC sizes are fixed (allowVolumeExpansion is unset).

History. Until 26/09/2026 storage was Rook-Ceph (RBD + CephFS). It was replaced by local-path-provisioner: the single-node Ceph consumed ~2 GiB of requests and ~700m of CPU and was prone to nbd/OSD deadlocks.

---

Known Issues

local-path has no online expansion

• Symptom: editing a PVC's resources.requests.storage is rejected.
• Cause: the local-path StorageClass does not set allowVolumeExpansion.
• Workaround: recreate the PVC and restore the data (or migrate to a new larger PVC). Sizes are fixed at creation.

---

Customization

Change upstream DNS

Edit manifests/built-in/coredns/04-configmap.yaml, forward section:

forward . 8.8.8.8 1.1.1.1 {

Change containerd runtime

Edit configs/containerd-config.toml:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = false   # set to true if host uses systemd cgroups

Change Pod CIDR

Update in two places:

1. scripts/entrypoint.sh → CLUSTER_CIDR
2. Cilium install command (entrypoint.sh → cilium install --set ipam.operator.clusterPoolIPv4PodCIDRList=...) Rebuild required.

---

Troubleshooting

Container dies immediately

docker compose logs --tail 50

Common causes:

• Missing --privileged in docker-compose
• /sys not mounted as shared

Pods stuck in ContainerCreating

kubectl describe pod <pod-name> -n <namespace>

Common causes:

• Cilium hasn't installed the CNI yet → wait for cilium-agent to be Running
• Mount propagation error → check that /sys is mounted rw

PVC stuck in Pending

local-path uses volumeBindingMode: WaitForFirstConsumer, so a PVC stays Pending until a pod that consumes it is scheduled. That is expected, not a failure.

kubectl describe pvc <pvc-name> -n <namespace>
kubectl -n local-path-storage logs deploy/local-path-provisioner

Common causes:

• No pod consuming the PVC yet → WaitForFirstConsumer is doing its job
• Provisioner not Running → check the logs above
• Volume data missing after a container recreate → confirm the ./data/local-path bind mount exists

CoreDNS CrashLoopBackOff

kubectl logs -n kube-system -l k8s-app=kube-dns

Common causes:

• Loop detection → already fixed with forward to 8.8.8.8
• Corefile syntax error → check manifests/built-in/coredns/04-configmap.yaml

Node NotReady

kubectl describe node k8s-one

Common causes:

• CNI not installed → Cilium still initializing
• kubelet can't communicate with apiserver → check certs

View logs for a specific component

# All logs mixed
docker compose logs -f

# Filter by component
docker compose logs -f | grep apiserver
docker compose logs -f | grep kubelet
docker compose logs -f | grep etcd

Full reset

docker compose down -v   # removes container + all named volumes (keeps ./data/local-path/)
docker compose up -d     # fresh start
# To also wipe volume data: rm -rf data/local-path/*  (irreversible!)

---

Requirements

Host

Requirement	Minimum	Recommended
Docker	24.0+	27.0+
Docker Compose	v2.20+	v2.30+
RAM	16 GB	24 GB
CPU	2 cores	4 cores
Disk	10 GB (image + volume data)	20 GB+
OS	Linux (kernel 5.10+)	Linux (kernel 6.x)
Arch	amd64	amd64

The RAM figures follow from the kubelet reserves, not from the cluster's own footprint: systemReserved (12Gi) + kubeReserved (2Gi) + evictionHard (1Gi) are subtracted from host capacity before pods get anything. A 16 GB host leaves ~1 GiB for pods; 24 GB leaves ~9 GiB. See Resource Reserves for the full math.

Ports

Port	Protocol	Usage
6443	TCP	Kubernetes API Server
8082	TCP	HAProxy Ingress HTTP (→ NodePort 30080)
8443	TCP	HAProxy Ingress HTTPS (→ NodePort 30443)
5432	TCP	PostgreSQL (→ NodePort 30432, host loopback only)

---

Limitations

• Not HA: single node, no redundancy. etcd, apiserver, etc. are single-instance.
• Not for production: intended for development, testing, CI/CD, lab environments.
• Storage without redundancy: local-path volumes are plain hostPath directories on the host disk.
• Privileged mode: the container runs with --privileged (required for kubelet/containerd).
• amd64 only: arm64 may work with --build-arg TARGETARCH=arm64 but is untested.
• No systemd: uses cgroupfs as cgroup driver (no systemd inside the container).
• Cert rotation: disabled. Certificates last 10 years. For long-lived clusters, consider implementing rotation.

---

License

MIT