Portfolio_Risk_Management/docs/KUBERNETES.md

Running the quant service on Kubernetes

quantcore has two kinds of deployment. They run the same code and give the same results:

Deployment	Where	Files	Guide
DGX Spark, Apptainer, no root	a student's Spark	deploy/apptainer/, install.sh	SPARK_APPTAINER.md
DGX Spark, Docker Compose, with root	the lecturer's Spark	deploy/Dockerfile, deploy/docker-compose.*.yml, deploy/deploy_to_spark.sh	SPARK_DEPLOYMENT.md
Kubernetes	any cluster: k3s on a Spark, Docker Desktop, kind, minikube, a university or cloud cluster	deploy/k8s/ (its own image and manifests)	this page

The Spark deployments are unchanged. The Kubernetes one has its own Dockerfile and manifests, and nothing in deploy/k8s/ is used by the other two.

What runs in the cluster

namespace quantcore
 ├─ Deployment quantcore        API + test bench, uvicorn, 1 replica, port 5001
 ├─ Service    quantcore        NodePort 30501  (inside the cluster: http://quantcore.quantcore:5001)
 ├─ PVC        quantcore-data   the data directory /data: universe, prices, FX, snapshots, job records
 ├─ ConfigMap  quantcore-config generated from deploy/k8s/base/quantcore.env (QUANTCORE_* settings)
 ├─ CronJob    quantcore-update `quantcore update`, Mon–Fri 23:15 Europe/Zurich
 ├─ CronJob    quantcore-setup  `quantcore setup`, suspended: started by hand, once
 └─ Deployment + Service cuopt  only with overlays/gpu: cuOpt on one NVIDIA GPU, port 5000

deploy/k8s/
  Dockerfile, Dockerfile.dockerignore      the image (pinned libraries, uid 10001, read-only root fs)
  base/                                    CPU only: every optimisation runs on HiGHS in the API pod
  overlays/gpu/                            base + cuOpt on a GPU of the cluster

What changed in the service, and why

A pod is not a container on a host you look after. Kubernetes restarts it when a probe fails, replaces it on every rollout, may run several copies, and runs the nightly update in a pod of its own. Five assumptions of the single-container setup did not survive that, and the code changed for them. These changes apply to the Spark deployments as well, where they change nothing you would notice.

Assumption	Why it breaks on Kubernetes	Change
/health is the health check	It asks cuOpt (up to 5 s). As a liveness probe, a cuOpt outage would get healthy API pods restarted	GET /health/live (async, no I/O) for liveness, GET /health/ready (data volume readable) for readiness. Neither needs cuOpt or market data. /health is unchanged
The API loads the data once and reloads after its own jobs	The CronJob writes the data from another pod, and the API would keep serving yesterday's prices	Every writer saves manifest.json last. The API reloads its market data when that file changes
Files are written in place	A reader in another pod could read half a parquet file	DataStore writes to a temporary file and renames it into place, an atomic replace
One data job at a time, enforced by a lock inside the process	The CronJob, a second replica or the CLI cannot see that lock	A file lock (flock) on /data/.write.lock, taken by API jobs, CLI data commands and therefore the CronJobs. The kernel frees it when its holder dies. A second writer gets HTTP 409 or exit status 3, with the holder's name
Job records live in memory	A restarted pod forgets them, and another replica never knew them	Records are written to /data/jobs/ at every status change. GET /jobs/{id} works from every replica. A job whose pod died is reported failed (interrupted: …)

The Kubernetes image (deploy/k8s/Dockerfile) differs from deploy/Dockerfile in three ways:

• Pinned libraries: it installs the versions in deploy/apptainer/constraints.txt. This matters for the liveness probe. SciPy 1.18.1 releases the GIL while HiGHS solves: during a 15 s solve the probe was answered within 0.3 s at worst. SciPy 1.13 blocks every other thread for the whole solve, and the pod would be restarted in the middle of an optimisation.
• Unprivileged user: it runs as uid 10001 with HOME=/tmp, under a read-only root file system. yfinance keeps its cookie cache in HOME.
• No HEALTHCHECK: Kubernetes ignores it.

1. Build the image

From the repository root, for the platform of the cluster's nodes:

docker build -f deploy/k8s/Dockerfile -t quantcore:0.1.0 .

Then make it available to the cluster:

Cluster	Command
Docker Desktop	nothing: its Kubernetes uses Docker's images
kind	kind load docker-image quantcore:0.1.0
minikube	minikube image load quantcore:0.1.0
k3s (e.g. on a Spark; build on the Spark itself, it is arm64)	docker save quantcore:0.1.0 | sudo k3s ctr images import -
a registry	docker buildx build --platform linux/arm64,linux/amd64 -f deploy/k8s/Dockerfile -t <registry>/quantcore:0.1.0 --push . and set newName: <registry>/quantcore under images: in deploy/k8s/base/kustomization.yaml

The manifests use imagePullPolicy: IfNotPresent. A rebuilt image with the same tag is therefore used only by new pods: after loading it, run kubectl -n quantcore rollout restart deploy/quantcore. With a registry, give every build a new tag instead.

2. Deploy

Without a GPU in the cluster:

kubectl apply -k deploy/k8s/base
kubectl -n quantcore rollout status deploy/quantcore

Every optimisation then runs on HiGHS in the API pod, and the result says solver: "highs". To use a cuOpt that runs outside the cluster, set QUANTCORE_CUOPT_URL=http://<host>:5000 in deploy/k8s/base/quantcore.env before applying. For example, the cuOpt container next to the Docker Compose setup on the lecturer's Spark.

With cuOpt on a GPU of the cluster:

kubectl apply -k deploy/k8s/overlays/gpu

The cluster needs NVIDIA's container runtime and device plugin. On k3s, the RuntimeClass nvidia is created by k3s itself. Then install the device plugin and check that the node offers the GPU:

helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm upgrade -i nvdp nvdp/nvidia-device-plugin -n nvidia-device-plugin --create-namespace \
     --set runtimeClassName=nvidia
kubectl describe node | grep nvidia.com/gpu        # nvidia.com/gpu: 1

The cuOpt pod has a startup probe and nothing else. cuOpt works on one request at a time and may not answer its health check during a long solve. A readiness probe would remove the Service under a solve the API is still polling, and a liveness probe would restart the solver in the middle of it.

3. Load the data

The volume starts empty. GET /health/ready answers "market_data": false, and the data endpoints answer 503 until there is data. Either copy a prepared data directory into the pod:

COPYFILE_DISABLE=1 tar --no-xattrs -C data -cf - manifest.json universe prices fx snapshots \
  | kubectl -n quantcore exec -i deploy/quantcore -- tar -C /data -xf -

(Leave out snapshots if you have none. COPYFILE_DISABLE=1 --no-xattrs keeps macOS tar from adding ._* files; on Linux both are harmless. The command names the entries rather than . because the volume's root directory is not owned by the pod's user.)

Or build the data from Yahoo Finance, about 15 minutes:

kubectl -n quantcore create job --from=cronjob/quantcore-setup quantcore-setup-1
kubectl -n quantcore logs -f job/quantcore-setup-1

curl -X POST 'http://<node>:30501/jobs/setup?offline=true' does the same through the API, with the candidate lists shipped in the package. A job of the API dies with its pod, however, so a rollout during those 15 minutes interrupts it. The Job above does not have that problem.

The API picks up the new data by itself. No restart is needed.

4. Use it

From	URL
outside the cluster	http://<node>:30501; Docker Desktop: http://localhost:30501; /docs, /ui/
any machine with kubectl (also kind)	kubectl -n quantcore port-forward svc/quantcore 5001:5001, then http://127.0.0.1:5001
another pod in the cluster	http://quantcore.quantcore:5001

quantcore-api --url http://<node>:30501 selftest     # every endpoint, pass/fail table

5. Operate

kubectl -n quantcore get pods,cronjobs,jobs
kubectl -n quantcore logs -f deploy/quantcore
kubectl -n quantcore create job --from=cronjob/quantcore-update update-now     # the nightly update, now
kubectl -n quantcore exec deploy/quantcore -- quantcore status
kubectl -n quantcore exec deploy/quantcore -- quantcore snapshot --name demo

Task	How
New code	build and load the image (section 1); with a new tag, change newTag in base/kustomization.yaml, then kubectl apply -k …. A rollout starts the new pod before it stops the old one, so there is no downtime
Change a setting	edit deploy/k8s/base/quantcore.env and kubectl apply -k …. The ConfigMap gets a new name, which rolls the pods
API only, without the test bench	QUANTCORE_ENABLE_UI=false in quantcore.env
Another update time	schedule / timeZone in base/cronjobs.yaml
More replicas	kubectl -n quantcore scale deploy/quantcore --replicas=3. All replicas mount one volume, so they must share a node (one-node clusters) or the PVC must use a ReadWriteMany storage class. Each replica holds the market data in memory (~0.5–1 GB)
Remove it, keep the data	kubectl -n quantcore delete deploy,svc,cronjob,job --all
Remove everything	kubectl delete -k deploy/k8s/base (or overlays/gpu). This deletes the PVC and the data with it

Settings in the manifests

Setting	Value	Why
API resources	requests 1 CPU / 1 GiB, limits 4 CPU / 4 GiB	measured: 0.75 GB resident after a 150-asset optimisation on the full data set. 4 CPUs match OPENBLAS_NUM_THREADS=4
terminationGracePeriodSeconds	360	uvicorn finishes the requests in flight after SIGTERM, and an optimisation can use the whole 300 s time limit
preStop: sleep 5		the Service stops routing to the pod before uvicorn stops accepting
startup / liveness / readiness	/health/live every 2 s up to 120 s / every 20 s, 3 failures / /health/ready every 10 s
Pod security	uid 10001, read-only root fs, no capabilities, seccomp RuntimeDefault	the API and job pods meet the restricted Pod Security Standard; the cuOpt pod runs NVIDIA's image unchanged
Update job	concurrencyPolicy: Forbid, backoffLimit: 3, 1 h deadline	a run that finds the store busy exits with 3 and is retried

Troubleshooting

Symptom	Cause / fix
ErrImageNeverPull / ImagePullBackOff	the node does not have the image: load it (section 1), or set newName to your registry
PermissionError: '/data/…' in a pod	the storage class ignores fsGroup, so the volume is not writable for uid 10001. Use a class that applies it (local-path, most CSI drivers), or prepare the directory's owner on the node
Multi-Attach error for quantcore-data	a ReadWriteOnce volume is wanted on two nodes (API and job pod, or two replicas). Use a ReadWriteMany class, or keep to one node
cuOpt pod Pending, Insufficient nvidia.com/gpu	the device plugin is missing or the GPU is taken. Without cuOpt the API keeps optimising on HiGHS
409 the data store is busy: … on <pod>	a data job is running, named with its pod. Wait for it: kubectl -n quantcore get jobs, GET /jobs
a job record says interrupted	the pod that ran it was deleted (a rollout, a node drain). Start the job again, preferably as a Kubernetes Job (section 3)
/optimize answers 504 through an Ingress	the proxy timeout is shorter than the solve. For ingress-nginx: nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
/health/ready answers 503	the data volume cannot be read: kubectl -n quantcore describe pod, and look at the volume

What was tested

On k3s v1.32.2 (one arm64 node in Docker):

• deploy/k8s/base deployed, and the data imported as in section 3.
• The update CronJob ran as a Job. While it ran, the API answered 409 and named the Job's pod; afterwards it served the new prices without a restart.
• A rollout under load: 547 requests, all 200.
• The pods meet the restricted Pod Security Standard.

The GPU overlay passed a server-side dry run only; that cluster has no GPU. The setup job was not run end to end.