b5da424dfe
The portfolio data, risk and mean-CVaR optimisation service, prepared for students who run it on their own DGX Spark without root, Docker or Tailscale. - deploy/apptainer/qc.sh: install, start/stop/status/logs, cli/api/python, data import/export and Yahoo setup/update. cuOpt and quantcore run as two Apptainer instances of the user and talk over 127.0.0.1; the API and the test bench listen on 127.0.0.1 only. Builds quantcore.sif without root, or falls back to a plain Python image plus a virtualenv when the Spark allows no unprivileged builds. Detects taken ports and suggests free ones. - deploy/apptainer/quantcore.def + constraints.txt: the image, with the library versions of the verified production container pinned. - cuOpt under Apptainer: NVIDIA's image sets HOME=/opt/cuopt, read-only in a SIF, so the solver died on its first job; qc.sh gives it a writable home. - solvers: a cuOpt that disappears during a solve now falls back to HiGHS instead of answering 500 (requests' ConnectionError escaped the fallback). - client: idempotent GETs retry a dropped keep-alive connection, which made `quantcore-api update --wait` fail when a poll met uvicorn's 5 s timeout. - config: cuOpt defaults to http://127.0.0.1:5000 and `quantcore serve` to 127.0.0.1; the Docker images keep passing their own values. - docs/SPARK_APPTAINER.md: the students' guide; README, USAGE, ARCHITECTURE and SPARK_DEPLOYMENT point to it. Tested on a DGX Spark (GB10, Apptainer 1.5.4, unprivileged user) in both modes: selftest on every endpoint, optimisation on cuOpt, Yahoo update, port conflicts; 139 tests pass in the container against the live cuOpt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
126 lines
11 KiB
Markdown
126 lines
11 KiB
Markdown
# Architecture
|
||
|
||
`quantcore` is the **deterministic core** of the platform sketched in the AISE502 project brief:
|
||
a set of services with plain contracts, no LLM inside, resilient against external data outages,
|
||
and reproducible through pinned data snapshots. The AI layer (news ingestion, `ResearchAgent`,
|
||
`AdvisorAgent`, …) sits *on top* and only reads these services through their contracts.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────┐
|
||
HTTP/JSON │ quantcore.api (FastAPI) – thin, no logic, background jobs │
|
||
└───────┬───────────────┬───────────────┬──────────────┬───────────┘
|
||
│ │ │ │
|
||
┌─────────────────▼──┐ ┌─────────▼─────────┐ ┌──▼──────────┐ ┌▼────────────────────┐
|
||
│ SetupService │ │ RiskService │ │ Performance │ │ OptimizationService │
|
||
│ DataDownloadService │ │ (measurement.risk) │ │ Service │ │ (optimization) │
|
||
└─────────┬───────────┘ └─────────┬─────────┘ └──┬──────────┘ └┬───────────┬────────┘
|
||
│ │ │ │ │
|
||
┌─────────▼───────────┐ ┌─────────▼───────────────▼──────────┐ │ ┌───────▼────────────┐
|
||
│ YahooClient (ext.) │ │ MarketDataService │◄──┘ │ models: │
|
||
│ timeout · retry · │ │ prices · FX → CHF · returns · │ │ CovarianceModel │
|
||
│ circuit breaker │ │ market caps · cap-weighted index │ │ GBM / innovations │
|
||
└─────────┬───────────┘ └─────────┬──────────────────────────┘ │ Black-Litterman │
|
||
│ │ └───────┬────────────┘
|
||
┌─────────▼────────────────────────▼──────────┐ ┌────────────────▼────────────┐
|
||
│ DataStore (parquet/csv under data_dir, │ │ MeanCVaRProblem → MILPModel │
|
||
│ manifest, snapshots) │ │ CuOptSolver ⇄ HighsSolver │
|
||
└─────────────────────────────────────────────┘ └──────────────┬──────────────┘
|
||
│ REST
|
||
┌─────────────▼─────────────┐
|
||
│ cuOpt server (DGX Spark) │
|
||
└───────────────────────────┘
|
||
```
|
||
|
||
## Bounded contexts → modules
|
||
|
||
| Context | Module | Contract (inputs → outputs) |
|
||
|---|---|---|
|
||
| Universe | `quantcore.universe` | web candidate lists → `universe.csv` (symbol, name, region, country, currency, exchange, GICS sector, Yahoo sector/industry, market cap local & CHF, shares outstanding, first price date, coverage) |
|
||
| Market data | `quantcore.data` | Yahoo → `adj_close.parquet` (date × symbol, listing currency), `fx_chf.parquet` (date × currency); `MarketDataService.prices/returns(symbols, start, end, currency, freq)` → aligned frames |
|
||
| Measurement | `quantcore.measurement` | returns frame → risk tables, series, matrices, `PortfolioRisk`; performance tables |
|
||
| Models | `quantcore.models` | returns frame → covariance, `ScenarioSet` (S × N horizon log-returns, optional probabilities, fingerprint; `sampling`: Sobol'/moment matching/importance sampling), `BlackLittermanResult` |
|
||
| Optimisation | `quantcore.optimization` | `OptimizationRequest` → `OptimizationResult` (shares, trades, cash, costs, CVaR, exposures, provenance of every model input, scenario hash, reproducibility level); `problem` (MILP, weighted scenarios or Gaussian cuts), `rowgen` (tail-row generation), `gaussian_exact` (outer approximation), `solvers` |
|
||
| Interface | `quantcore.api`, `quantcore.cli` | JSON/CLI surfaces of the above; no computation of their own |
|
||
| Test bench | `quantcore/api/ui` | a static single-page client of the JSON API (`/ui/`): stock data, covariance and scenarios, views, constraints, optimisation with before/after. Static HTML + ES modules, no build step and no CDN, so it also runs in the offline container. It formats and charts what the services return and computes nothing itself |
|
||
|
||
Dependencies point downwards only: `api → optimization → models → measurement → data → config`.
|
||
`measurement` and `models` never touch storage or the network; they operate on DataFrames handed
|
||
to them, which is what makes them pure and testable with reference vectors.
|
||
|
||
## Deterministic by construction
|
||
|
||
* Every stochastic step takes an explicit `seed` (scenario generation: scrambled Sobol' points,
|
||
moment matched — the same request yields the same scenario matrix, fingerprinted as
|
||
`scenario_hash`; bit-identical within one environment, while across environments the *key* matches
|
||
and the digest need not (§9 of `docs/METHODOLOGY.md` measures why); the *scenario key* lists
|
||
exactly what it depends on — risk limits, constraints,
|
||
costs, objective, cash, holdings and views are not part of it, so a family of runs is compared on
|
||
one set, which the service also caches) and the same portfolio with HiGHS (`reproducibility="exact"`); on cuOpt the
|
||
figures agree within its MIP gap of 1e-4 (`"within_gap"`). `scenario_model="gaussian_exact"` needs
|
||
no scenarios at all.
|
||
* Every result carries the *provenance* of its inputs: covariance model, scenario model and seed,
|
||
expected-return model (incl. the views and their calibrated Ω), constraints, costs, solver, status,
|
||
gap, timings. An advisor can cite “CVaR 95 % = CHF 38 210 (OptimizationService, empirical model,
|
||
EWM λ=0.99, seed 42)”.
|
||
* Pinned snapshots (`DataStore.create_snapshot`, `MarketDataService(settings, snapshot=...)`) freeze
|
||
universe, prices and FX so that demos and grading are reproducible even if Yahoo misbehaves.
|
||
|
||
## Where the solving happens
|
||
|
||
The MILP is built in `quantcore.optimization.problem` and handed to one of two solvers, both fed the
|
||
same `MILPModel`:
|
||
|
||
* **cuOpt** on the GPU, over HTTP to the cuOpt container on the same Spark (`QUANTCORE_CUOPT_URL`):
|
||
`http://127.0.0.1:5000` in the Apptainer setup, where both containers share the host's network,
|
||
and `http://cuopt:5000` inside the Docker compose network. The only GPU component; an external
|
||
dependency, hence the timeouts, retries and circuit breaker.
|
||
* **HiGHS** on the CPU, *in the same process as the service* — it is compiled into SciPy
|
||
(`scipy.optimize.milp`), so there is nothing to deploy or start, and no network hop. In production
|
||
that means inside the `quantcore` container (SciPy 1.18.1 → HiGHS 1.12.0, 4 threads).
|
||
|
||
`solver="auto"` tries cuOpt and falls back to HiGHS; the result names the solver that produced it and
|
||
how reproducible it is. See `docs/METHODOLOGY.md` §9 for versions and the measured comparison.
|
||
|
||
## Resilience on every external call
|
||
|
||
| External dependency | Pattern | Fallback |
|
||
|---|---|---|
|
||
| Yahoo Finance (`YahooClient`) | per-request timeout, 4 retries with exponential back-off, circuit breaker (5 failures → 60 s open) | the stored data / a snapshot; the API keeps serving from the store |
|
||
| Candidate lists (Wikipedia, iShares) | timeout, one source after the other | static snapshots shipped in `quantcore/data/static/` |
|
||
| cuOpt (`CuOptSolver`) | submit with retry/back-off, polling with client-side deadline, circuit breaker, results deleted server-side | local HiGHS MILP (scipy) — the result says `solver: "highs"` and carries a warning |
|
||
|
||
## Data model conventions
|
||
|
||
* prices: adjusted close (splits & dividends), one column per Yahoo symbol, union calendar of all
|
||
exchanges, holidays forward-filled (≤ 5 days); pence-quoted London lines converted to GBP.
|
||
* FX: cross rates against the base currency, repaired for quoting-unit changes (`clean_fx`) —
|
||
a rate returned 100× off for a stretch of days would otherwise be read as a 100× move of
|
||
every stock in that currency (see `docs/METHODOLOGY.md` §1).
|
||
* returns: natural logarithms; CHF return = local return + FX return (additive).
|
||
* frequencies: daily, weekly (Friday close), monthly (month end); horizons: day = 1, week = 5,
|
||
month = 21 trading days (or 1 / 4 weeks at weekly frequency).
|
||
* money: everything in the base currency (CHF); positions are **integer share numbers**.
|
||
|
||
## Where the AI layer plugs in
|
||
|
||
* `ResearchAgent` insights (ticker, sentiment ∈ [−1, 1], degree of belief) become Black–Litterman
|
||
views through `view_from_sentiment` / the `views` field of `/optimize` — the numbers (view return,
|
||
Ω, posterior) are computed here, the agent only supplies the assessment and its provenance.
|
||
* `RiskAgent` / `OptimizationAgent` call `/risk/*` and `/optimize` and *explain* the returned figures.
|
||
* The ontology guard can validate tickers and sectors against `/universe`.
|
||
|
||
## Judging the system by hand
|
||
|
||
The test bench at `/ui/` exists so that every model choice can be exercised without writing code:
|
||
the same scenario set can be seen as a distribution, as per-stock moments and as the risk figures of
|
||
an optimised portfolio, and a run's provenance (scenario key and hash, solver, gap, reproducibility)
|
||
is shown next to its result. It is also the quickest way to see the invariants hold — two identical
|
||
requests share a scenario hash and produce identical shares, and a family of runs that varies only
|
||
the risk limit or the constraints reuses one scenario set.
|
||
|
||
## Operations
|
||
|
||
* `quantcore setup` (once), `quantcore update` (scheduled, e.g. nightly), `quantcore snapshot`.
|
||
* On the Spark the API runs as the `quantcore` container next to `cuopt`, `gurobi`, `jupyter`
|
||
(see `docs/SPARK_DEPLOYMENT.md`); data lives in `~/services/work/quantcore_data`.
|