# Architecture

`quantcore` is the **deterministic core** of the platform sketched in the AISE502 project brief:
a set of services with plain contracts, no LLM inside, resilient against external data outages,
and reproducible through pinned data snapshots. The AI layer (news ingestion, `ResearchAgent`,
`AdvisorAgent`, …) sits *on top* and only reads these services through their contracts.

```
                 ┌──────────────────────────────────────────────────────────────────┐
  HTTP/JSON      │  quantcore.api  (FastAPI)  – thin, no logic, background jobs      │
                 └───────┬───────────────┬───────────────┬──────────────┬───────────┘
                         │               │               │              │
       ┌─────────────────▼──┐  ┌─────────▼─────────┐  ┌──▼──────────┐  ┌▼────────────────────┐
       │ SetupService        │  │ RiskService        │  │ Performance │  │ OptimizationService │
       │ DataDownloadService │  │ (measurement.risk) │  │ Service     │  │ (optimization)      │
       └─────────┬───────────┘  └─────────┬─────────┘  └──┬──────────┘  └┬───────────┬────────┘
                 │                        │               │              │           │
       ┌─────────▼───────────┐  ┌─────────▼───────────────▼──────────┐   │   ┌───────▼────────────┐
       │ YahooClient (ext.)  │  │ MarketDataService                  │◄──┘   │ models:            │
       │ timeout · retry ·   │  │ prices · FX → CHF · returns ·      │       │ CovarianceModel    │
       │ circuit breaker     │  │ market caps · cap-weighted index   │       │ GBM / innovations  │
       └─────────┬───────────┘  └─────────┬──────────────────────────┘       │ Black-Litterman    │
                 │                        │                                  └───────┬────────────┘
       ┌─────────▼────────────────────────▼──────────┐              ┌────────────────▼────────────┐
       │ DataStore  (parquet/csv under data_dir,     │              │ MeanCVaRProblem → MILPModel │
       │ manifest, snapshots)                        │              │ CuOptSolver ⇄ HighsSolver   │
       └─────────────────────────────────────────────┘              └──────────────┬──────────────┘
                                                                                   │ REST
                                                                     ┌─────────────▼─────────────┐
                                                                     │ cuOpt server (DGX Spark)  │
                                                                     └───────────────────────────┘
```

## Bounded contexts → modules

| Context | Module | Contract (inputs → outputs) |
|---|---|---|
| Universe | `quantcore.universe` | web candidate lists → `universe.csv` (symbol, name, region, country, currency, exchange, GICS sector, Yahoo sector/industry, market cap local & CHF, shares outstanding, first price date, coverage) |
| Market data | `quantcore.data` | Yahoo → `adj_close.parquet` (date × symbol, listing currency), `fx_chf.parquet` (date × currency); `MarketDataService.prices/returns(symbols, start, end, currency, freq)` → aligned frames |
| Measurement | `quantcore.measurement` | returns frame → risk tables, series, matrices, `PortfolioRisk`; performance tables |
| Models | `quantcore.models` | returns frame → covariance, `ScenarioSet` (S × N horizon log-returns, optional probabilities, fingerprint; `sampling`: Sobol'/moment matching/importance sampling), `BlackLittermanResult` |
| Optimisation | `quantcore.optimization` | `OptimizationRequest` → `OptimizationResult` (shares, trades, cash, costs, CVaR, exposures, provenance of every model input, scenario hash, reproducibility level); `problem` (MILP, weighted scenarios or Gaussian cuts), `rowgen` (tail-row generation), `gaussian_exact` (outer approximation), `solvers` |
| Interface | `quantcore.api`, `quantcore.cli` | JSON/CLI surfaces of the above; no computation of their own |
| Test bench | `quantcore/api/ui` | a static single-page client of the JSON API (`/ui/`): stock data, covariance and scenarios, views, constraints, optimisation with before/after. Static HTML + ES modules, no build step and no CDN, so it also runs in the offline container. It formats and charts what the services return and computes nothing itself |

Dependencies point downwards only: `api → optimization → models → measurement → data → config`.
`measurement` and `models` never touch storage or the network; they operate on DataFrames handed
to them, which is what makes them pure and testable with reference vectors.

## Deterministic by construction

* Every stochastic step takes an explicit `seed` (scenario generation: scrambled Sobol' points,
  moment matched — the same request yields the same scenario matrix, fingerprinted as
  `scenario_hash`; bit-identical within one environment, while across environments the *key* matches
  and the digest need not (§9 of `docs/METHODOLOGY.md` measures why); the *scenario key* lists
  exactly what it depends on — risk limits, constraints,
  costs, objective, cash, holdings and views are not part of it, so a family of runs is compared on
  one set, which the service also caches) and the same portfolio with HiGHS (`reproducibility="exact"`); on cuOpt the
  figures agree within its MIP gap of 1e-4 (`"within_gap"`). `scenario_model="gaussian_exact"` needs
  no scenarios at all.
* Every result carries the *provenance* of its inputs: covariance model, scenario model and seed,
  expected-return model (incl. the views and their calibrated Ω), constraints, costs, solver, status,
  gap, timings. An advisor can cite “CVaR 95 % = CHF 38 210 (OptimizationService, empirical model,
  EWM λ=0.99, seed 42)”.
* Pinned snapshots (`DataStore.create_snapshot`, `MarketDataService(settings, snapshot=...)`) freeze
  universe, prices and FX so that demos and grading are reproducible even if Yahoo misbehaves.

## Where the solving happens

The MILP is built in `quantcore.optimization.problem` and handed to one of two solvers, both fed the
same `MILPModel`:

* **cuOpt** on the GPU, over HTTP to the cuOpt container on the same Spark (`QUANTCORE_CUOPT_URL`):
  `http://127.0.0.1:5000` in the Apptainer setup, where both containers share the host's network,
  and `http://cuopt:5000` inside the Docker compose network. The only GPU component; an external
  dependency, hence the timeouts, retries and circuit breaker.
* **HiGHS** on the CPU, *in the same process as the service* — it is compiled into SciPy
  (`scipy.optimize.milp`), so there is nothing to deploy or start, and no network hop. In production
  that means inside the `quantcore` container (SciPy 1.18.1 → HiGHS 1.12.0, 4 threads).

`solver="auto"` tries cuOpt and falls back to HiGHS; the result names the solver that produced it and
how reproducible it is. See `docs/METHODOLOGY.md` §9 for versions and the measured comparison.

## Resilience on every external call

| External dependency | Pattern | Fallback |
|---|---|---|
| Yahoo Finance (`YahooClient`) | per-request timeout, 4 retries with exponential back-off, circuit breaker (5 failures → 60 s open) | the stored data / a snapshot; the API keeps serving from the store |
| Candidate lists (Wikipedia, iShares) | timeout, one source after the other | static snapshots shipped in `quantcore/data/static/` |
| cuOpt (`CuOptSolver`) | submit with retry/back-off, polling with client-side deadline, circuit breaker, results deleted server-side | local HiGHS MILP (scipy) — the result says `solver: "highs"` and carries a warning |

## Data model conventions

* prices: adjusted close (splits & dividends), one column per Yahoo symbol, union calendar of all
  exchanges, holidays forward-filled (≤ 5 days); pence-quoted London lines converted to GBP.
* FX: cross rates against the base currency, repaired for quoting-unit changes (`clean_fx`) —
  a rate returned 100× off for a stretch of days would otherwise be read as a 100× move of
  every stock in that currency (see `docs/METHODOLOGY.md` §1).
* returns: natural logarithms; CHF return = local return + FX return (additive).
* frequencies: daily, weekly (Friday close), monthly (month end); horizons: day = 1, week = 5,
  month = 21 trading days (or 1 / 4 weeks at weekly frequency).
* money: everything in the base currency (CHF); positions are **integer share numbers**.

## Where the AI layer plugs in

* `ResearchAgent` insights (ticker, sentiment ∈ [−1, 1], degree of belief) become Black–Litterman
  views through `view_from_sentiment` / the `views` field of `/optimize` — the numbers (view return,
  Ω, posterior) are computed here, the agent only supplies the assessment and its provenance.
* `RiskAgent` / `OptimizationAgent` call `/risk/*` and `/optimize` and *explain* the returned figures.
* The ontology guard can validate tickers and sectors against `/universe`.

## Judging the system by hand

The test bench at `/ui/` exists so that every model choice can be exercised without writing code:
the same scenario set can be seen as a distribution, as per-stock moments and as the risk figures of
an optimised portfolio, and a run's provenance (scenario key and hash, solver, gap, reproducibility)
is shown next to its result. It is also the quickest way to see the invariants hold — two identical
requests share a scenario hash and produce identical shares, and a family of runs that varies only
the risk limit or the constraints reuses one scenario set.

## Operations

* `quantcore setup` (once), `quantcore update` (scheduled, e.g. nightly), `quantcore snapshot`.
* On the Spark the API runs as the `quantcore` container next to `cuopt`, `gurobi`, `jupyter`
  (see `docs/SPARK_DEPLOYMENT.md`); data lives in `~/services/work/quantcore_data`.