Portfolio_Risk_Management/docs/KUBERNETES.md

# Running the quant service on Kubernetes

quantcore has two kinds of deployment. They run the same code and give the same results:

| Deployment | Where | Files | Guide |
|---|---|---|---|
| **DGX Spark**, Apptainer, no root | a student's Spark | `deploy/apptainer/`, `install.sh` | [SPARK_APPTAINER.md](SPARK_APPTAINER.md) |
| **DGX Spark**, Docker Compose, with root | the lecturer's Spark | `deploy/Dockerfile`, `deploy/docker-compose.*.yml`, `deploy/deploy_to_spark.sh` | [SPARK_DEPLOYMENT.md](SPARK_DEPLOYMENT.md) |
| **Kubernetes** | any cluster: k3s on a Spark, Docker Desktop, kind, minikube, a university or cloud cluster | `deploy/k8s/` (its own image and manifests) | this page |

The Spark deployments are unchanged. The Kubernetes one has its own Dockerfile and manifests, and
nothing in `deploy/k8s/` is used by the other two.

## What runs in the cluster

```
namespace quantcore
 ├─ Deployment quantcore        API + test bench, uvicorn, 1 replica, port 5001
 ├─ Service    quantcore        NodePort 30501  (inside the cluster: http://quantcore.quantcore:5001)
 ├─ PVC        quantcore-data   the data directory /data: universe, prices, FX, snapshots, job records
 ├─ ConfigMap  quantcore-config generated from deploy/k8s/base/quantcore.env (QUANTCORE_* settings)
 ├─ CronJob    quantcore-update `quantcore update`, Mon–Fri 23:15 Europe/Zurich
 ├─ CronJob    quantcore-setup  `quantcore setup`, suspended: started by hand, once
 └─ Deployment + Service cuopt  only with overlays/gpu: cuOpt on one NVIDIA GPU, port 5000
```

```
deploy/k8s/
  Dockerfile, Dockerfile.dockerignore      the image (pinned libraries, uid 10001, read-only root fs)
  base/                                    CPU only: every optimisation runs on HiGHS in the API pod
  overlays/gpu/                            base + cuOpt on a GPU of the cluster
```

## What changed in the service, and why

A pod is not a container on a host you look after. Kubernetes restarts it when a probe fails,
replaces it on every rollout, may run several copies, and runs the nightly update in a pod of its own.
Five assumptions of the single-container setup did not survive that, and the code changed for them.
These changes apply to the Spark deployments as well, where they change nothing you would notice.

| Assumption | Why it breaks on Kubernetes | Change |
|---|---|---|
| `/health` is the health check | It asks cuOpt (up to 5 s). As a liveness probe, a cuOpt outage would get healthy API pods restarted | `GET /health/live` (async, no I/O) for liveness, `GET /health/ready` (data volume readable) for readiness. Neither needs cuOpt or market data. `/health` is unchanged |
| The API loads the data once and reloads after its own jobs | The CronJob writes the data from another pod, and the API would keep serving yesterday's prices | Every writer saves `manifest.json` last. The API reloads its market data when that file changes |
| Files are written in place | A reader in another pod could read half a parquet file | `DataStore` writes to a temporary file and renames it into place, an atomic replace |
| One data job at a time, enforced by a lock inside the process | The CronJob, a second replica or the CLI cannot see that lock | A file lock (`flock`) on `/data/.write.lock`, taken by API jobs, CLI data commands and therefore the CronJobs. The kernel frees it when its holder dies. A second writer gets HTTP 409 or exit status 3, with the holder's name |
| Job records live in memory | A restarted pod forgets them, and another replica never knew them | Records are written to `/data/jobs/` at every status change. `GET /jobs/{id}` works from every replica. A job whose pod died is reported `failed` (`interrupted: …`) |

The Kubernetes image (`deploy/k8s/Dockerfile`) differs from `deploy/Dockerfile` in three ways:

- **Pinned libraries:** it installs the versions in `deploy/apptainer/constraints.txt`. This matters
  for the liveness probe. SciPy 1.18.1 releases the GIL while HiGHS solves: during a 15 s solve the
  probe was answered within 0.3 s at worst. SciPy 1.13 blocks every other thread for the whole solve,
  and the pod would be restarted in the middle of an optimisation.
- **Unprivileged user:** it runs as uid 10001 with `HOME=/tmp`, under a read-only root file system.
  yfinance keeps its cookie cache in `HOME`.
- **No `HEALTHCHECK`:** Kubernetes ignores it.

## 1. Build the image

From the repository root, for the platform of the cluster's nodes:

```bash
docker build -f deploy/k8s/Dockerfile -t quantcore:0.1.0 .
```

Then make it available to the cluster:

| Cluster | Command |
|---|---|
| Docker Desktop | nothing: its Kubernetes uses Docker's images |
| kind | `kind load docker-image quantcore:0.1.0` |
| minikube | `minikube image load quantcore:0.1.0` |
| k3s (e.g. on a Spark; build on the Spark itself, it is arm64) | `docker save quantcore:0.1.0 \| sudo k3s ctr images import -` |
| a registry | `docker buildx build --platform linux/arm64,linux/amd64 -f deploy/k8s/Dockerfile -t <registry>/quantcore:0.1.0 --push .` and set `newName: <registry>/quantcore` under `images:` in `deploy/k8s/base/kustomization.yaml` |

The manifests use `imagePullPolicy: IfNotPresent`. A rebuilt image with the same tag is therefore
used only by new pods: after loading it, run `kubectl -n quantcore rollout restart deploy/quantcore`.
With a registry, give every build a new tag instead.

## 2. Deploy

**Without a GPU in the cluster:**

```bash
kubectl apply -k deploy/k8s/base
kubectl -n quantcore rollout status deploy/quantcore
```

Every optimisation then runs on HiGHS in the API pod, and the result says `solver: "highs"`.
To use a cuOpt that runs outside the cluster, set `QUANTCORE_CUOPT_URL=http://<host>:5000` in
`deploy/k8s/base/quantcore.env` before applying. For example, the cuOpt container next to the Docker
Compose setup on the lecturer's Spark.

**With cuOpt on a GPU of the cluster:**

```bash
kubectl apply -k deploy/k8s/overlays/gpu
```

The cluster needs NVIDIA's container runtime and device plugin. On k3s, the RuntimeClass `nvidia` is
created by k3s itself. Then install the device plugin and check that the node offers the GPU:

```bash
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm upgrade -i nvdp nvdp/nvidia-device-plugin -n nvidia-device-plugin --create-namespace \
     --set runtimeClassName=nvidia
kubectl describe node | grep nvidia.com/gpu        # nvidia.com/gpu: 1
```

The cuOpt pod has a startup probe and nothing else. cuOpt works on one request at a time and may not
answer its health check during a long solve. A readiness probe would remove the Service under a solve
the API is still polling, and a liveness probe would restart the solver in the middle of it.

## 3. Load the data

The volume starts empty. `GET /health/ready` answers `"market_data": false`, and the data endpoints
answer 503 until there is data. Either copy a prepared data directory into the pod:

```bash
COPYFILE_DISABLE=1 tar --no-xattrs -C data -cf - manifest.json universe prices fx snapshots \
  | kubectl -n quantcore exec -i deploy/quantcore -- tar -C /data -xf -
```

(Leave out `snapshots` if you have none. `COPYFILE_DISABLE=1 --no-xattrs` keeps macOS `tar` from
adding `._*` files; on Linux both are harmless. The command names the entries rather than `.`
because the volume's root directory is not owned by the pod's user.)

Or build the data from Yahoo Finance, about 15 minutes:

```bash
kubectl -n quantcore create job --from=cronjob/quantcore-setup quantcore-setup-1
kubectl -n quantcore logs -f job/quantcore-setup-1
```

`curl -X POST 'http://<node>:30501/jobs/setup?offline=true'` does the same through the API, with the
candidate lists shipped in the package. A job of the API dies with its pod, however, so a rollout
during those 15 minutes interrupts it. The Job above does not have that problem.

The API picks up the new data by itself. No restart is needed.

## 4. Use it

| From | URL |
|---|---|
| outside the cluster | `http://<node>:30501`; Docker Desktop: `http://localhost:30501`; `/docs`, `/ui/` |
| any machine with kubectl (also kind) | `kubectl -n quantcore port-forward svc/quantcore 5001:5001`, then `http://127.0.0.1:5001` |
| another pod in the cluster | `http://quantcore.quantcore:5001` |

```bash
quantcore-api --url http://<node>:30501 selftest     # every endpoint, pass/fail table
```

## 5. Operate

```bash
kubectl -n quantcore get pods,cronjobs,jobs
kubectl -n quantcore logs -f deploy/quantcore
kubectl -n quantcore create job --from=cronjob/quantcore-update update-now     # the nightly update, now
kubectl -n quantcore exec deploy/quantcore -- quantcore status
kubectl -n quantcore exec deploy/quantcore -- quantcore snapshot --name demo
```

| Task | How |
|---|---|
| New code | build and load the image (section 1); with a new tag, change `newTag` in `base/kustomization.yaml`, then `kubectl apply -k …`. A rollout starts the new pod before it stops the old one, so there is no downtime |
| Change a setting | edit `deploy/k8s/base/quantcore.env` and `kubectl apply -k …`. The ConfigMap gets a new name, which rolls the pods |
| API only, without the test bench | `QUANTCORE_ENABLE_UI=false` in `quantcore.env` |
| Another update time | `schedule` / `timeZone` in `base/cronjobs.yaml` |
| More replicas | `kubectl -n quantcore scale deploy/quantcore --replicas=3`. All replicas mount one volume, so they must share a node (one-node clusters) or the PVC must use a ReadWriteMany storage class. Each replica holds the market data in memory (~0.5–1 GB) |
| Remove it, keep the data | `kubectl -n quantcore delete deploy,svc,cronjob,job --all` |
| Remove everything | `kubectl delete -k deploy/k8s/base` (or `overlays/gpu`). **This deletes the PVC and the data with it** |

## Settings in the manifests

| Setting | Value | Why |
|---|---|---|
| API resources | requests 1 CPU / 1 GiB, limits 4 CPU / 4 GiB | measured: 0.75 GB resident after a 150-asset optimisation on the full data set. 4 CPUs match `OPENBLAS_NUM_THREADS=4` |
| `terminationGracePeriodSeconds` | 360 | uvicorn finishes the requests in flight after SIGTERM, and an optimisation can use the whole 300 s time limit |
| `preStop: sleep 5` | | the Service stops routing to the pod before uvicorn stops accepting |
| startup / liveness / readiness | `/health/live` every 2 s up to 120 s / every 20 s, 3 failures / `/health/ready` every 10 s | |
| Pod security | uid 10001, read-only root fs, no capabilities, seccomp `RuntimeDefault` | the API and job pods meet the `restricted` Pod Security Standard; the cuOpt pod runs NVIDIA's image unchanged |
| Update job | `concurrencyPolicy: Forbid`, `backoffLimit: 3`, 1 h deadline | a run that finds the store busy exits with 3 and is retried |

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `ErrImageNeverPull` / `ImagePullBackOff` | the node does not have the image: load it (section 1), or set `newName` to your registry |
| `PermissionError: '/data/…'` in a pod | the storage class ignores `fsGroup`, so the volume is not writable for uid 10001. Use a class that applies it (local-path, most CSI drivers), or prepare the directory's owner on the node |
| `Multi-Attach error` for `quantcore-data` | a ReadWriteOnce volume is wanted on two nodes (API and job pod, or two replicas). Use a ReadWriteMany class, or keep to one node |
| cuOpt pod `Pending`, `Insufficient nvidia.com/gpu` | the device plugin is missing or the GPU is taken. Without cuOpt the API keeps optimising on HiGHS |
| `409 the data store is busy: … on <pod>` | a data job is running, named with its pod. Wait for it: `kubectl -n quantcore get jobs`, `GET /jobs` |
| a job record says `interrupted` | the pod that ran it was deleted (a rollout, a node drain). Start the job again, preferably as a Kubernetes Job (section 3) |
| `/optimize` answers 504 through an Ingress | the proxy timeout is shorter than the solve. For ingress-nginx: `nginx.ingress.kubernetes.io/proxy-read-timeout: "900"` |
| `/health/ready` answers 503 | the data volume cannot be read: `kubectl -n quantcore describe pod`, and look at the volume |

## What was tested

On k3s v1.32.2 (one arm64 node in Docker):

- `deploy/k8s/base` deployed, and the data imported as in section 3.
- The update CronJob ran as a Job. While it ran, the API answered 409 and named the Job's pod; afterwards it served the new prices without a restart.
- A rollout under load: 547 requests, all 200.
- The pods meet the `restricted` Pod Security Standard.

The GPU overlay passed a server-side dry run only; that cluster has no GPU. The setup job was not run end to end.