herzogflorian fac41ffdfc Large-scale strategy: relaxation, tail rows, screening, round-and-repair
The solver benchmark (GPU_Optimization/cuopt_benchmark, November 2026) showed
that the integer mean-CVaR MILP is an LP in disguise: HiGHS closes it at the
root node, only the tail scenario rows bind, and the cardinality constraint is
the only hard combinatorics. quantcore.optimization.large_scale exploits that:
LP relaxation by tail-row generation (evaluated at the continuous point, its
objective a valid bound), screening to the top names under a cardinality limit,
a boxed round-and-repair MILP, certification against the bound with escalation
to the integer row generation otherwise. Solver choice by size (HiGHS below
16 000 rows, cuOpt above).

Request option `strategy` ('auto' | 'milp' | 'large_scale'; auto from 200
candidates or 32 768 scenarios) in the service, the API schema and both CLIs.
Settings from the benchmark: cuOpt presolve off (its asynchronous GPU heuristic
otherwise holds every MILP for 5-25 s), dual simplex for pure LPs (concurrent
returns PDLP's inexact point on the L40S), 50 ms polling. rowgen takes an
initial working set.

Measured with HiGHS (M4 Max): N=300 S=4096 5.2 s -> 0.7 s, N=300 S=16384
45 s -> 2.9 s, N=750 S=16384 15.6 s -> 7.4 s, K=40 of 750 24.6 s -> 7.3 s,
objectives within the 1e-4 gap. Tests in tests/test_large_scale.py; benchmark
script scripts/experiments/large_scale.py; docs updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 09:58:32 +01:00

139 lines
7.3 KiB
Python

"""Central configuration (environment variables with the ``QUANTCORE_`` prefix, or a ``.env`` file).
Every service receives a :class:`Settings` instance -- nothing reads the environment on its own,
which keeps the services testable (tests pass a ``Settings(data_dir=tmp_path)``).
"""
from __future__ import annotations
import logging
from functools import lru_cache
from pathlib import Path
from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
"""Runtime settings of the quant core."""
model_config = SettingsConfigDict(env_prefix="QUANTCORE_", env_file=".env", extra="ignore")
# --- storage ---------------------------------------------------------------------------
data_dir: Path = Field(default=Path("data"), description="Root of all persisted data.")
# --- universe --------------------------------------------------------------------------
n_us: int = Field(default=500, description="Largest N US stocks (S&P 500 candidates).")
n_europe: int = Field(default=500, description="Largest N European stocks (STOXX 600 candidates).")
history_start: str = Field(default="2004-12-01", description="First date requested from Yahoo.")
required_history_from: str = Field(
default="2005-01-31",
description="A stock is kept only if it has a price on or before this date ('data since 2005').",
)
min_price_coverage: float = Field(
default=0.90, description="Minimum share of union-calendar days with a price since the first date."
)
max_missing_run: int = Field(
default=20, description="Longest tolerated gap (business days) in a price series since the required date."
)
# --- market conventions ----------------------------------------------------------------
base_currency: str = Field(default="CHF", description="Portfolio / reporting currency.")
home_market: str = Field(default="Switzerland", description="Home market of the investor.")
risk_free_rate: float = Field(default=0.0, description="Annual risk-free rate in the base currency.")
trading_days_per_year: int = 252
weeks_per_year: int = 52
# --- Yahoo access ----------------------------------------------------------------------
yahoo_batch_size: int = Field(default=100, description="Tickers per bulk download request.")
yahoo_timeout: float = Field(default=30.0, description="Seconds per HTTP request.")
yahoo_max_retries: int = 4
yahoo_backoff: float = Field(default=2.0, description="Base of the exponential back-off (seconds).")
yahoo_info_pause: float = Field(default=0.15, description="Pause between ticker-info requests (s).")
# --- cuOpt -----------------------------------------------------------------------------
cuopt_url: str = Field(
default="http://127.0.0.1:5000",
description="cuOpt REST endpoint. The default is a cuOpt container on the same machine (the "
"standalone DGX Spark setup, deploy/apptainer/); point it elsewhere for a remote one.")
cuopt_time_limit: float = Field(default=120.0, description="MILP time limit in seconds.")
cuopt_poll_interval: float = Field(
default=0.05, description="Seconds between polls of a submitted cuOpt job. The server has no blocking "
"call, so this is a latency floor per solve (measured: 0.05 s per call at "
"0.02 s, a full second at the old default of 1 s).")
cuopt_timeout: float = Field(default=900.0, description="Max seconds to wait for a solution.")
cuopt_request_timeout: float = Field(default=60.0, description="Seconds per HTTP call to cuOpt.")
cuopt_upload_seconds_per_mb: float = Field(
default=1.0, description="Added to the submit timeout per MB of request body. cuOpt accepts about "
"10 MB/s, so this is a 10x margin; a client-side timeout during the upload "
"would otherwise surface as an aborted job.")
solver_fallback: bool = Field(
default=True, description="Fall back to the local HiGHS MILP solver (scipy) when cuOpt is unreachable."
)
cuopt_threads: int | None = Field(default=None, description="CPU threads of the cuOpt MIP solver (None: server default).")
cuopt_presolve: bool = Field(
default=False, description="cuOpt MILP presolve. Off by default: with it on, cuOpt 26.08 starts an "
"asynchronous GPU heuristic and waits for it, so a sub-second MILP takes "
"5-25 s (measured 2026-11); off, the same models return in 0.5-2 s. Off, "
"cuOpt aborted on 3 of 16 CVaR MILPs on one host (an error reply over REST; "
"solve_with_fallback then solves on HiGHS) -- set True if that happens often.")
cuopt_lp_method: int | None = Field(
default=2, description="cuOpt method for pure LPs (no integer variables): 2 = dual simplex, the fastest "
"exact method on the CVaR relaxations. None = cuOpt's concurrent default, which "
"returns PDLP's inexact point (violated scenario rows) when PDLP finishes first.")
cuopt_msgpack: bool = Field(default=True, description="Send cuOpt requests as msgpack when the package is available.")
# --- MILP tolerances (reproducibility) ---------------------------------------------------
mip_rel_gap: float = Field(default=1e-4, description="Relative MIP gap for HiGHS and cuOpt.")
mip_abs_gap: float = Field(default=0.01, description="Absolute MIP gap in money units (cuOpt).")
# --- misc ------------------------------------------------------------------------------
log_level: str = "INFO"
api_host: str = Field(
default="127.0.0.1",
description="Interface `quantcore serve` binds to: localhost only by default; the Docker images "
"pass 0.0.0.0 to publish the port.")
api_port: int = 5001
enable_ui: bool = Field(
default=True,
description="Serve the static test bench at /ui/ (and redirect / to it). Set false for a "
"deployment that only exposes the JSON API and /docs -- the same image then "
"serves a front end built by someone else.",
)
# derived paths -------------------------------------------------------------------------
@property
def universe_dir(self) -> Path:
return self.data_dir / "universe"
@property
def prices_dir(self) -> Path:
return self.data_dir / "prices"
@property
def fx_dir(self) -> Path:
return self.data_dir / "fx"
@property
def snapshots_dir(self) -> Path:
return self.data_dir / "snapshots"
@property
def jobs_dir(self) -> Path:
return self.data_dir / "jobs"
def ensure_dirs(self) -> None:
for d in (self.universe_dir, self.prices_dir, self.fx_dir, self.snapshots_dir, self.jobs_dir):
d.mkdir(parents=True, exist_ok=True)
@lru_cache(maxsize=1)
def get_settings() -> Settings:
"""Process-wide settings (cached). Tests construct :class:`Settings` directly instead."""
settings = Settings()
logging.basicConfig(
level=getattr(logging, settings.log_level.upper(), logging.INFO),
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
return settings