Portfolio_Risk_Management/docs/ARCHITECTURE.md

Architecture

quantcore is the deterministic core of the platform sketched in the AISE502 project brief: a set of services with plain contracts, no LLM inside, resilient against external data outages, and reproducible through pinned data snapshots. The AI layer (news ingestion, ResearchAgent, AdvisorAgent, …) sits on top and only reads these services through their contracts.

                 ┌──────────────────────────────────────────────────────────────────┐
  HTTP/JSON      │  quantcore.api  (FastAPI)  – thin, no logic, background jobs      │
                 └───────┬───────────────┬───────────────┬──────────────┬───────────┘
                         │               │               │              │
       ┌─────────────────▼──┐  ┌─────────▼─────────┐  ┌──▼──────────┐  ┌▼────────────────────┐
       │ SetupService        │  │ RiskService        │  │ Performance │  │ OptimizationService │
       │ DataDownloadService │  │ (measurement.risk) │  │ Service     │  │ (optimization)      │
       └─────────┬───────────┘  └─────────┬─────────┘  └──┬──────────┘  └┬───────────┬────────┘
                 │                        │               │              │           │
       ┌─────────▼───────────┐  ┌─────────▼───────────────▼──────────┐   │   ┌───────▼────────────┐
       │ YahooClient (ext.)  │  │ MarketDataService                  │◄──┘   │ models:            │
       │ timeout · retry ·   │  │ prices · FX → CHF · returns ·      │       │ CovarianceModel    │
       │ circuit breaker     │  │ market caps · cap-weighted index   │       │ GBM / innovations  │
       └─────────┬───────────┘  └─────────┬──────────────────────────┘       │ Black-Litterman    │
                 │                        │                                  └───────┬────────────┘
       ┌─────────▼────────────────────────▼──────────┐              ┌────────────────▼────────────┐
       │ DataStore  (parquet/csv under data_dir,     │              │ MeanCVaRProblem → MILPModel │
       │ manifest, snapshots)                        │              │ CuOptSolver ⇄ HighsSolver   │
       └─────────────────────────────────────────────┘              └──────────────┬──────────────┘
                                                                                   │ REST
                                                                     ┌─────────────▼─────────────┐
                                                                     │ cuOpt server (DGX Spark)  │
                                                                     └───────────────────────────┘

Bounded contexts → modules

Context	Module	Contract (inputs → outputs)
Universe	quantcore.universe	web candidate lists → universe.csv (symbol, name, region, country, currency, exchange, GICS sector, Yahoo sector/industry, market cap local & CHF, shares outstanding, first price date, coverage)
Market data	quantcore.data	Yahoo → adj_close.parquet (date × symbol, listing currency), fx_chf.parquet (date × currency); MarketDataService.prices/returns(symbols, start, end, currency, freq) → aligned frames
Measurement	quantcore.measurement	returns frame → risk tables, series, matrices, PortfolioRisk; performance tables
Models	quantcore.models	returns frame → covariance, ScenarioSet (S × N horizon log-returns, optional probabilities, fingerprint; sampling: Sobol'/moment matching/importance sampling), BlackLittermanResult
Optimisation	quantcore.optimization	OptimizationRequest → OptimizationResult (shares, trades, cash, costs, CVaR, exposures, provenance of every model input, scenario hash, reproducibility level); problem (MILP, weighted scenarios or Gaussian cuts), rowgen (tail-row generation), large_scale (relaxation, screening, round-and-repair for large universes), gaussian_exact (outer approximation), solvers
Interface	quantcore.api, quantcore.cli	JSON/CLI surfaces of the above; no computation of their own
Test bench	quantcore/api/ui	a static single-page client of the JSON API (/ui/): stock data, covariance and scenarios, views, constraints, optimisation with before/after. Static HTML + ES modules, no build step and no CDN, so it also runs in the offline container. It formats and charts what the services return and computes nothing itself

Dependencies point downwards only: api → optimization → models → measurement → data → config. measurement and models never touch storage or the network; they operate on DataFrames handed to them, which is what makes them pure and testable with reference vectors.

Deterministic by construction

• Every stochastic step takes an explicit seed (scenario generation: scrambled Sobol' points, moment matched — the same request yields the same scenario matrix, fingerprinted as scenario_hash; bit-identical within one environment, while across environments the key matches and the digest need not (§9 of docs/METHODOLOGY.md measures why); the scenario key lists exactly what it depends on — risk limits, constraints, costs, objective, cash, holdings and views are not part of it, so a family of runs is compared on one set, which the service also caches) and the same portfolio with HiGHS (reproducibility="exact"); on cuOpt the figures agree within its MIP gap of 1e-4 ("within_gap"). scenario_model="gaussian_exact" needs no scenarios at all.
• Every result carries the provenance of its inputs: covariance model, scenario model and seed, expected-return model (incl. the views and their calibrated Ω), constraints, costs, solver, status, gap, timings. An advisor can cite “CVaR 95 % = CHF 38 210 (OptimizationService, empirical model, EWM λ=0.99, seed 42)”.
• Pinned snapshots (DataStore.create_snapshot, MarketDataService(settings, snapshot=...)) freeze universe, prices and FX so that demos and grading are reproducible even if Yahoo misbehaves.

Where the solving happens

The MILP is built in quantcore.optimization.problem and handed to one of two solvers, both fed the same MILPModel:

• cuOpt on the GPU, over HTTP to the cuOpt container on the same Spark (QUANTCORE_CUOPT_URL): http://127.0.0.1:5000 in the Apptainer setup, where both containers share the host's network, and http://cuopt:5000 inside the Docker compose network. The only GPU component; an external dependency, hence the timeouts, retries and circuit breaker.
• HiGHS on the CPU, in the same process as the service — it is compiled into SciPy (scipy.optimize.milp), so there is nothing to deploy or start, and no network hop. In production that means inside the quantcore container (SciPy 1.18.1 → HiGHS 1.12.0, 4 threads).

solver="auto" tries cuOpt and falls back to HiGHS; the result names the solver that produced it and how reproducible it is. See docs/METHODOLOGY.md §9 for versions and the measured comparison.

Resilience on every external call

External dependency	Pattern	Fallback
Yahoo Finance (YahooClient)	per-request timeout, 4 retries with exponential back-off, circuit breaker (5 failures → 60 s open)	the stored data / a snapshot; the API keeps serving from the store
Candidate lists (Wikipedia, iShares)	timeout, one source after the other	static snapshots shipped in quantcore/data/static/
cuOpt (CuOptSolver)	submit with retry/back-off, polling with client-side deadline, circuit breaker, results deleted server-side	local HiGHS MILP (scipy) — the result says solver: "highs" and carries a warning

Data model conventions

• prices: adjusted close (splits & dividends), one column per Yahoo symbol, union calendar of all exchanges, holidays forward-filled (≤ 5 days); pence-quoted London lines converted to GBP.
• FX: cross rates against the base currency, repaired for quoting-unit changes (clean_fx) — a rate returned 100× off for a stretch of days would otherwise be read as a 100× move of every stock in that currency (see docs/METHODOLOGY.md §1).
• returns: natural logarithms; CHF return = local return + FX return (additive).
• frequencies: daily, weekly (Friday close), monthly (month end); horizons: day = 1, week = 5, month = 21 trading days (or 1 / 4 weeks at weekly frequency).
• money: everything in the base currency (CHF); positions are integer share numbers.

Where the AI layer plugs in

• ResearchAgent insights (ticker, sentiment ∈ [−1, 1], degree of belief) become Black–Litterman views through view_from_sentiment / the views field of /optimize — the numbers (view return, Ω, posterior) are computed here, the agent only supplies the assessment and its provenance.
• RiskAgent / OptimizationAgent call /risk/* and /optimize and explain the returned figures.
• The ontology guard can validate tickers and sectors against /universe.

Judging the system by hand

The test bench at /ui/ exists so that every model choice can be exercised without writing code: the same scenario set can be seen as a distribution, as per-stock moments and as the risk figures of an optimised portfolio, and a run's provenance (scenario key and hash, solver, gap, reproducibility) is shown next to its result. It is also the quickest way to see the invariants hold — two identical requests share a scenario hash and produce identical shares, and a family of runs that varies only the risk limit or the constraints reuses one scenario set.

Operations

• quantcore setup (once), quantcore update (scheduled, e.g. nightly), quantcore snapshot.
• On the Spark the API runs as the quantcore container next to cuopt, gurobi, jupyter (see docs/SPARK_DEPLOYMENT.md); data lives in ~/services/work/quantcore_data.
• On Kubernetes (docs/KUBERNETES.md) the API is a Deployment, the data a PersistentVolumeClaim and the update a CronJob. Several processes then share one data directory, so DataStore replaces files atomically, one writer at a time holds its write lock (/data/.write.lock), the manifest is saved last, and the API reloads its market data when the manifest changes.