herzogfloria/fund_rfid_data
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add SEC fund prospectus -> RDF triple dataset pipeline

Browse Source 
Builds a relationship-rich finance dataset for text-to-RDF-triple extraction
from SEC fund disclosures, the dataset for the thesis 'Magical RDF Triples and
how to synthetize them'.

- build_rdf_dataset.py: gold (N-CEN graphs), fetch (EDGAR prospectus prose,
  all books per trust), samples (per-fund segmentation, marker + plain
  serializations), split (trust-level 80/10/10, no leakage)
- score_baseline.py: no-model string-match baseline + strong-model scorer
- dataset_description.{tex,pdf}: scientific description of the dataset
- data/rdf_poc/gold_graphs.jsonl: structured gold knowledge graph (2025Q3)
- Large prose/sample files and raw SEC downloads are gitignored (reproducible)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Florian Herzog 2026-06-03 10:31:35 +02:00
commit 1993658fb2
23 changed files with 6224 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

38
.gitignore vendored Normal file
View File

@ -0,0 +1,38 @@
# ---- Large / derived data (reproducible via build_rdf_dataset.py) ----
# Raw prospectus prose fetched from EDGAR (GBs)
data/rdf_poc/prose/
# Generated training samples and splits (embed raw SEC text, 100s of MB)
data/rdf_poc/samples.jsonl
data/rdf_poc/train.jsonl
data/rdf_poc/val.jsonl
data/rdf_poc/test.jsonl
# Raw SEC bulk downloads (re-downloadable from sec.gov)
data/ncen/
data/nport/
data/xbrl_rr/
# SQLite working DB
fund_data.db
fund_data.db-shm
fund_data.db-wal
# Archives
*.zip
# Python
__pycache__/
*.pyc
*.pyo
# LaTeX build artifacts
*.aux
*.log
*.out
*.toc
*.fls
*.fdb_latexmk
*.synctex.gz
# OS / editor
.DS_Store
.claude/

297
ESMA_FUND_DATA_RESEARCH.md Normal file
View File

@ -0,0 +1,297 @@
# ESMA Fund Data: Registers, APIs, and Reference Data Fields
## 1. Overview: ESMA's Fund Data Ecosystem
ESMA (European Securities and Markets Authority) maintains **six distinct data systems** relevant to fund data. Unlike the US SEC which centralizes prospectus filing and structured extraction, ESMA's fund data is **fragmented across regulatory registers** focused on authorization, cross-border notification, and instrument identification rather than on prospectus content.
| System | Content | Access | Fund Relevance |
|--------|---------|--------|----------------|
| **Fund Register** (AIF/EuSEF/EuVECA) | Authorized AIF funds & managers | Solr API (JSON/XML) | Direct - fund-level identity |
| **Cross-border Marketing Register** (UCITS + AIF) | Fund passporting notifications | Solr API (JSON/XML) | Direct - UCITS & AIF cross-border data |
| **Entity Register** (MiFID/UCITS/AIFMD) | Management companies, AIFMs | Solr API (JSON/XML) | Manager-level data |
| **FIRDS** (Financial Instruments Reference Data) | MiFIR instrument reference data | XML bulk files + API | Instrument-level (ISIN, CFI) |
| **Prospectus Register** (Prospectus Regulation III) | Approved EU prospectuses + supplements | Solr API (JSON/XML) | Securities prospectuses (not fund KIIDs) |
| **Money Market Fund Register** | MMF authorizations | Solr API (JSON/XML) | MMF-specific |
---
## 2. Fund Register: AIF/EuSEF/EuVECA Funds
### API Endpoint
```
https://registers.esma.europa.eu/solr/esma_registers_funds/select?q=...&wt=json
```
### Available Fields
| Field Name | Type | Description | openfunds Equivalent |
|-----------|------|-------------|---------------------|
| `funds_national_name` | text | Fund name (national identifier name) | OFST010110 Legal Fund Name |
| `funds_lei` | text | **LEI** of the fund | OFST010030 LEI Of Fund |
| `funds_legal_framework_name` | text | Legal framework: AIF, EuSEF, EuVECA | OFST160100 Legal Form |
| `funds_other_legal_framework_name` | text | Additional legal framework info | — |
| `funds_status_code_name` | text | Fund authorization status | — |
| `funds_mgmnt_structure_code_name` | text | Management structure type | OFST010420 Open-ended/Closed-ended |
| `funds_domicile_cou_code_name` | text | **Fund domicile country** | OFST010010 Fund Domicile |
| `funds_mgmnt_status_code_name` | text | Management status | — |
| `funds_manager_nat_name` | text | **Management company name** | OFST001020 ManCo |
| `funds_manager_lei` | text | **Manager LEI** | — |
| `funds_manager_cou_code_name` | text | Manager country | — |
| `funds_manager_legal_framework_name` | text | Manager legal framework | — |
| `funds_host_country_code_name` | text | **Host member states** (marketing countries) | OFST6000XX Country registrations |
| `funds_fund_mrkt_status_code_name` | text | Marketing status per country | — |
| `funds_notification_event1_date` | date | First notification date | — |
| `funds_notification_event2_date` | date | Second notification date | — |
| `funds_notif_legal_framework_name` | text | Notification legal framework | — |
| `funds_ca_cou_code_name` | text | Competent authority country | OFST010060 Supervisory Authority |
### Example API Call (all AIF funds, JSON format)
```bash
curl "https://registers.esma.europa.eu/solr/esma_registers_funds/select?q=type_s:*&fq=funds_legal_framework_name:%22AIF%22&fl=funds_national_name,funds_lei,funds_domicile_cou_code_name,funds_manager_nat_name,funds_manager_lei&rows=100&wt=json&indent=true"
```
---
## 3. Cross-border Marketing Register (UCITS + AIF)
### API Endpoint
```
https://registers.esma.europa.eu/solr/esma_registers_funds_cbdif/select?q=...&wt=json
```
Same field structure as the Fund Register above, but covers **UCITS funds** as well. This is the only ESMA register that includes UCITS fund-level data.
### Filter by fund type
```bash
# UCITS funds
...&fq=funds_legal_framework_name:"UCITS"
# AIF funds
...&fq=funds_legal_framework_name:"AIF"
# ELTIF funds
...&fq=funds_legal_framework_name:"ELTIF"
```
---
## 4. Entity Register (Management Companies)
### API Endpoint
```
https://registers.esma.europa.eu/solr/esma_registers_upreg/select?q=...&wt=json
```
### Fields for UCITS Management Companies (`ae_entityTypeCode:UCI`)
| Field Name | Type | Description | openfunds Equivalent |
|-----------|------|-------------|---------------------|
| `ae_entityName` | text | Entity (ManCo) name | OFST001020 ManCo |
| `ae_lei` | text | Entity LEI | — |
| `ae_entityTypeCode` | text | Entity type (UCI, AIF, MIF, etc.) | — |
| `ae_competentAuthority` | text | NCA name | OFST010060 Supervisory Authority |
| `ae_homeMemberState` | text | Home member state | — |
| `ae_hostMemberState` | text | Host member state(s) | — |
| `ae_status` | text | Authorization status | — |
| `ae_authorisationNotificationDate` | date | Authorization date | — |
| `ae_website` | text | Entity website | — |
| `ae_legalform` | text | Legal form | OFST160100 Legal Form |
| `ae_commercialName` | text | Commercial/brand name | OFST001000 Fund Group Name |
| `ac_serviceName` | text | Licensed services | — |
| `no_of_funds` | string | Number of managed funds | — |
### Example: List all UCITS Management Companies
```bash
curl "https://registers.esma.europa.eu/solr/esma_registers_upreg/select?q={!join+from=id+to=_root_}ae_entityTypeCode:UCI&fq=(type_s:parent)&rows=1000&wt=json&indent=true"
```
---
## 5. FIRDS (Financial Instruments Reference Data System)
FIRDS contains MiFIR reference data for **all financial instruments** traded on EU venues, including fund shares/units. Funds are classified with CFI codes starting with "C" (Collective Investment Schemes).
### Access Methods
1. **Full/Delta XML files**: Downloaded from ESMA registers portal
2. **Python package**: `esma_data_py` on GitHub
3. **API**: Via the ESMA API store
### Key Fields for Fund Instruments
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| ISIN | International Securities Identification Number | OFST020000 ISIN |
| CFI Code | Classification of Financial Instruments (ISO 10962) | OFST350015 CFI Code |
| Instrument Full Name | Name of the instrument | OFST020060 Full Share Class Name |
| Issuer LEI | LEI of the issuer/ManCo | OFST010030 LEI Of Fund |
| Notional Currency | Currency of the instrument | OFST020540 Share Class Currency |
| Trading Venue MIC | Where the instrument is traded | OFST060000-range Listing data |
| Maturity Date | For dated instruments | — |
| Nominal Value | Face value per unit | — |
### Python Access
```python
from esma_data_py import EsmaDataLoader
edl = EsmaDataLoader()
# Load FIRDS data for Collective Investment Schemes
df = edl.load_latest_files(instrument_type="FULINS", cfi_codes=["C*"])
```
---
## 6. Money Market Fund Register
### API Endpoint
```
https://registers.esma.europa.eu/solr/esma_registers_mmf04/select?q=type_s:parent&wt=json
```
### Fields
| Field Name | Description | openfunds Equivalent |
|-----------|-------------|---------------------|
| `mmf04_lei` | Fund LEI | OFST010030 LEI Of Fund |
| `mmf04_national_name` | Fund national name | OFST010110 Legal Fund Name |
| `mmf04_domicile_name` | Fund domicile | OFST010010 Fund Domicile |
| `mmf04_type_name` | MMF type (CNAV/LVNAV/VNAV) | OFST351300 Money Market Type |
| `mmf04_lgl_framework_name` | Legal framework (UCITS/AIF) | OFST160100 Legal Form |
| `mmf04_is_passported_name` | Cross-border passported? | — |
| `mmf04_auth_status_name` | Authorization status | — |
| `mmf04_manager_lei` | Manager LEI | — |
| `mmf04_manager_nat_name` | Manager name | OFST001020 ManCo |
| `mmf04_manager_domicile_name` | Manager domicile | — |
| `mmf04_auth_start_date` | Authorization start date | OFST010240 Fund Launch Date |
| `mmf04_auth_end_date` | Authorization end date | — |
| `mmf04_ca_cou_code_name` | Competent authority country | OFST010060 Supervisory Authority |
| `mmf04_auth_ca_code_name` | Authorizing CA | — |
---
## 7. Prospectus Register (Prospectus Regulation III)
This covers **securities prospectuses** under the EU Prospectus Regulation (not UCITS KIIDs/KIDs). However, some fund-related securities (listed fund shares, ETFs) may appear here.
### API Endpoint
```
https://registers.esma.europa.eu/solr/esma_registers_priii_documents/select?q=...&wt=json
```
### Document Types
| Code | Type |
|------|------|
| URGN | Universal Registration Document |
| REGN | Registration Document |
| SECN | Securities Note |
| SMRY | Summary |
| BPFT | Base Prospectus Final Terms |
| BPWO | Base Prospectus without Final Terms |
| STDA | Standalone Prospectus |
### Searchable Fields
- `issuer_lei` — Issuer LEI
- `issuer_name` — Issuer name
- `issuer_residency` — Issuer country
- `offeror_lei` / `offeror_name` / `offeror_residency`
- `guarantor_lei` / `guarantor_name` / `guarantor_residency`
- `approval_filing_date` — Document approval date
- `document_type` — Type code (see above)
---
## 8. AIFMD Reporting (Not Publicly Available)
ESMA collects detailed fund data through **AIFMD Article 24 reporting**, but this data is **not publicly accessible**. It is submitted by AIFMs to NCAs and forwarded to ESMA for supervisory purposes only.
### Fields collected (not public):
| Category | Fields |
|----------|--------|
| **Fund Identity** | AIF name, national code, LEI, domicile, inception date |
| **Fund Type** | Predominant AIF type, investment strategy, sub-strategy |
| **Assets** | Gross Asset Value (GAV), Net Asset Value (NAV), base currency |
| **Leverage** | Gross method leverage, commitment method leverage |
| **Investor Types** | Breakdown by investor category (professional, retail, etc.) |
| **Geographic Focus** | Geographic breakdown of investments |
| **Asset Allocation** | Breakdown by asset type (equities, bonds, derivatives, etc.) |
| **Liquidity** | Portfolio liquidity profile, investor redemption frequency |
| **Counterparty Risk** | Top 5 counterparty exposures |
| **Risk Measures** | VaR, stress test results |
This is the richest structured dataset but is **confidential** and only available to regulators.
---
## 9. Comparison: ESMA vs SEC Fund Data
| Aspect | SEC (US) | ESMA (EU) |
|--------|----------|-----------|
| **Prospectus text** | Full prospectus filed as HTML/XML on EDGAR | Not centralized; filed with national NCAs |
| **Structured prospectus data** | XBRL Risk/Return Summary (fees, performance, objective) | **Not available** — no EU-wide structured extraction |
| **Fund identity register** | Series/Class CSV (CIK, Series ID, Class ID, ticker) | Fund Register (LEI, name, domicile, manager) |
| **Portfolio holdings** | N-PORT (quarterly, position-level) | **Not public** — AIFMD reporting is confidential |
| **Instrument reference data** | Limited (CUSIP in Series/Class CSV) | FIRDS (ISIN, CFI, LEI, currency, trading venue) |
| **Fee data (structured)** | XBRL: management fee, TER, loads, 12b-1 | **Not available** in ESMA registers |
| **Performance data (structured)** | XBRL: 1yr/5yr/10yr returns, bar charts | **Not available** in ESMA registers |
| **Risk data (structured)** | N-PORT: DV01, credit spread, VaR | AIFMD reporting (confidential) |
| **Cross-border/passporting** | N/A (single market) | Full cross-border notification register |
| **API quality** | Excellent (REST JSON, free, no auth) | Good (Solr JSON/XML, free, no auth) |
| **Bulk download** | ZIP files (submissions, XBRL, N-PORT) | FIRDS XML bulk files; fund register via Solr pagination |
### Key Difference
The SEC provides **structured data extracted from prospectuses** (fees, performance, objectives via XBRL), making it directly useful for LLM training with ground-truth labels. ESMA provides **authorization/registration data** (who is authorized, where, by whom) but does **not** centralize or structure the content of fund prospectuses/KIIDs/KIDs.
For EU fund prospectus content (KIID/KID), you would need to go to:
- Individual NCAs (AMF in France, BaFin in Germany, CSSF in Luxembourg, etc.)
- Commercial data providers (Morningstar, Refinitiv, FE fundinfo)
- Fund company websites directly
---
## 10. What openfunds Fields Can Be Found in ESMA Data?
### Directly Available (from ESMA registers)
| openfunds OF-ID | Field Name | ESMA Source |
|-----------------|-----------|-------------|
| OFST001000 | Fund Group Name | Entity Register: `ae_commercialName` |
| OFST001020 | ManCo | Fund Register: `funds_manager_nat_name` |
| OFST010010 | Fund Domicile | Fund Register: `funds_domicile_cou_code_name` |
| OFST010030 | LEI Of Fund | Fund Register: `funds_lei` |
| OFST010060 | Supervisory Authority | Fund Register: `funds_ca_cou_code_name` |
| OFST010110 | Legal Fund Name | Fund Register: `funds_national_name` |
| OFST020000 | ISIN | FIRDS: ISIN field |
| OFST020540 | Share Class Currency | FIRDS: Notional Currency |
| OFST160100 | Legal Form | Fund Register: `funds_legal_framework_name` |
| OFST350015 | CFI Code | FIRDS: CFI Code |
| OFST351295 | Is Money Market Fund | MMF Register: presence in register |
| OFST351300 | Money Market Type | MMF Register: `mmf04_type_name` |
### NOT Available in ESMA Public Data
| Category | openfunds Fields | Notes |
|----------|-----------------|-------|
| **Fees** | Management fee, TER, loads, subscription/redemption fees | Not in any ESMA register |
| **Performance** | Returns, Sharpe ratio, volatility | Not in any ESMA register |
| **Investment Objective** | Strategy text, objective text | Not in any ESMA register |
| **Risk Data** | SRRI, VaR, risk narrative | AIFMD reporting (confidential) |
| **Asset Class** | Detailed asset allocation | AIFMD reporting (confidential) |
| **Distribution Policy** | Distributing/accumulating | Not in any ESMA register |
| **Minimum Investment** | Min subscription amount | Not in any ESMA register |
| **Benchmark** | Benchmark index name | Not in any ESMA register |
| **Portfolio Holdings** | Position-level data | AIFMD reporting (confidential) |
---
## 11. Summary for Your LLM Use Case
**ESMA data is useful for fund identity and cross-referencing** (LEI, domicile, manager, legal framework, cross-border marketing status), but it does **not** provide the structured prospectus-derived data (fees, performance, objectives, risk) that the SEC's XBRL Risk/Return Summary provides.
**Practical implications:**
- For **US funds**: SEC EDGAR provides both the prospectus text AND structured ground-truth data — ideal for supervised LLM training
- For **EU funds**: ESMA provides identity/authorization data only. To get the prospectus text + structured reference data for EU funds, you would need to combine ESMA register data with prospectus documents sourced from national regulators or commercial providers
**Recommended approach for EU data:**
1. Use ESMA Fund Register + FIRDS for fund identity (LEI, ISIN, domicile, ManCo, CFI)
2. Source KIID/KID documents from national NCAs or fund company websites
3. Use openfunds-format data from commercial providers as ground truth
4. Or focus on the SEC dataset first (much richer, more accessible) and extend to EU later

9
LICENSE Normal file
View File

@ -0,0 +1,9 @@
MIT License
Copyright (c) 2026 herzogfloria
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

519
OPENFUNDS_PUBLIC_DATA_SOURCES.md Normal file
View File

@ -0,0 +1,519 @@
# Openfunds Fields: Public Structured Data Availability
## Executive Summary
This document maps each openfunds field category to publicly available **structured data sources** — data that is machine-readable, downloadable, and free (or freely accessible via API). The focus is on fields describing the fund itself (asset class, settlement, risk, currencies, hedging, ESG, fees, etc.) rather than EU-specific regulatory fields.
### Key Public Structured Data Sources
| Source | Format | Access | Coverage | Cost |
|--------|--------|--------|----------|------|
| **SEC Series/Class CSV** | CSV | Direct download | ~100K+ US share classes | Free |
| **SEC XBRL Risk/Return** | XBRL → flat files | Quarterly download | All US mutual fund prospectuses | Free |
| **SEC N-PORT Data Sets** | XML → flat TSV | Quarterly download | Monthly holdings for all US funds | Free |
| **SEC N-CEN Data Sets** | XML → flat TSV | Annual filing, quarterly sets | Service providers, classification | Free |
| **SEC Submissions API** | JSON | REST API | All SEC filers | Free |
| **SEC XBRL Company Facts** | JSON | REST API | XBRL-tagged financial data | Free |
| **GLEIF LEI Database** | JSON/CSV | API + bulk download | 3.19M+ global entities | Free (CC0) |
| **OpenFIGI** | JSON | REST API | Hundreds of millions of instruments | Free |
---
## 1. Key Fact: Company (OFST001000–004999) — 40 fields
These fields identify the management company, custodian, transfer agent, auditor, and other service providers.
### Fields with Structured Public Data
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST001000 | Fund Group Name | SEC Submissions API | `subs.name` (entity name) | **Yes** — JSON |
| OFST001020 | ManCo | SEC N-CEN | ADVISOR table (investment adviser) | **Yes** — TSV |
| OFST001030 | LEI Of ManCo | GLEIF LEI Database | LEI lookup by entity name | **Yes** — JSON/CSV |
| OFST001035 | Domicile Of ManCo | GLEIF LEI Database | `entity.legalAddress.country` | **Yes** — JSON |
| OFST001050 | Fund Guarantor | — | Not in public structured data | No |
| OFST001055 | Address of ManCo | GLEIF LEI Database | `entity.legalAddress` | **Yes** — JSON |
| OFST001060 | City of ManCo | GLEIF LEI Database | `entity.legalAddress.city` | **Yes** — JSON |
| OFST001065 | Fund Website of ManCo | SEC Submissions API | `subs.website` | **Yes** — JSON |
| OFST001100 | Fund Promoter Name | — | Not publicly structured | No |
| OFST001105 | LEI of Fund Promoter | GLEIF LEI Database | If name known → LEI lookup | **Partial** |
| OFST001300 | Fund Administrator Name | SEC N-CEN | SERVICE_PROVIDER table | **Yes** — TSV |
| OFST001400 | Custodian Bank Name | SEC N-CEN | CUSTODIAN table | **Yes** — TSV |
| OFST001410 | LEI Of Custodian Bank | SEC N-CEN + GLEIF | N-CEN has LEI fields (since 2025) | **Yes** — TSV |
| OFST001415 | Domicile Of Custodian Bank | GLEIF LEI Database | Via custodian LEI | **Yes** — JSON |
| OFST001430 | Trustee Name | SEC EDGAR HTML filings | Unstructured (prospectus text) | No |
| OFST001450 | Portfolio Managing Company Name | SEC N-CEN | ADVISOR table + sub-advisors | **Yes** — TSV |
| OFST001500 | Fund Advisor Name | SEC N-CEN | ADVISOR table | **Yes** — TSV |
| OFST001510 | Sub-Investment Advisor Name | SEC N-CEN | Sub-advisor entries | **Yes** — TSV |
| OFST001600 | Auditor Name | SEC N-CEN | AUDITOR table | **Yes** — TSV |
| OFST002000 | Market Maker Name | — | Not publicly structured for funds | No |
| OFST002700 | Transfer Agent Name | SEC N-CEN | TRANSFER_AGENT table | **Yes** — TSV |
| OFST002900 | GIIN of Fund | — | IRS FATCA list (not easily matched) | No |
**Summary**: ~15 of 40 company fields are available as structured public data, primarily from SEC N-CEN (service providers) and GLEIF (entity LEI/address data).
---
## 2. Key Fact: Umbrella (OFST005000–009999) — 10 fields
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST005000 | Has Umbrella | SEC Series/Class CSV | Inferred: multiple Series under same CIK | **Derivable** |
| OFST005010 | Umbrella | SEC Series/Class CSV | `Entity Name` (trust name) | **Yes** — CSV |
| OFST005015 | Domicile Of Umbrella | SEC Submissions API | `subs.stateOfIncorporation` | **Yes** — JSON |
| OFST005025 | CBI Code of Umbrella | — | Ireland-specific, not in SEC | No |
| OFST005030 | CSSF Code of Umbrella | — | Luxembourg-specific, not in SEC | No |
| OFST005040 | GIIN of Umbrella | — | Not publicly structured | No |
| OFST010035 | LEI Of Umbrella | GLEIF LEI Database | LEI lookup by trust name | **Yes** — JSON |
**Summary**: 4 of 10 fields available. Umbrella concept maps to SEC "Trust/Registrant" level.
---
## 3. Key Fact: Fund (OFST010000–019999) — 73 fields
This is the richest category, covering fund identity, investment strategy, structure, currencies, hedging, and product type flags.
### 3A. Fund Identity & Dates
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST010010 | Fund Domicile Alpha-2 | SEC Submissions API | `subs.stateOfIncorporation` → derive | **Partial** (US state, not ISO) |
| OFST010020 | Legal Fund Name Including Umbrella | SEC Series/Class CSV | Concatenate Entity Name + Series Name | **Derivable** |
| OFST010030 | LEI Of Fund | GLEIF LEI Database | LEI search by fund name | **Yes** — JSON |
| OFST010110 | Legal Fund Name Only | SEC Series/Class CSV | `Series Name` | **Yes** — CSV |
| OFST010240 | Fund Launch Date | SEC XBRL Risk/Return | `InceptionDate` element | **Yes** — XBRL |
| OFST010250 | Fund Valuation Point | — | Prospectus text only | No |
| OFST010300 | Investment Objective | SEC XBRL Risk/Return | `ObjectivePrimaryTextBlock` | **Yes** — XBRL (text) |
| OFST010410 | Fund Currency | SEC N-PORT | `FUND_REPORTED_INFO.total_assets` currency context | **Partial** (all USD for US funds) |
| OFST010440 | Fiscal Year End | SEC Submissions API | `subs.fiscalYearEnd` (MMDD format) | **Yes** — JSON |
| OFST013000 | Prospectus Date | SEC Submissions API | Filing date of latest 485BPOS/N-1A | **Yes** — JSON |
### 3B. Fund Structure & Product Type Flags
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST010420 | Open-ended Or Closed-ended | SEC N-CEN | Fund type reported | **Yes** — TSV |
| OFST010500 | Is Fund Of Funds | SEC N-CEN | Fund-of-funds flag | **Yes** — TSV |
| OFST010580 | Is ETF | SEC N-CEN | ETF table presence | **Yes** — TSV |
| OFST010620 | Is Tokenized Fund | — | Not in SEC data | No |
| OFST010630 | Is Leveraged | SEC N-PORT | Borrowing data (Item B.2) | **Derivable** |
| OFST010635 | Maximum Leverage In Fund | — | Prospectus text only | No |
| OFST010640 | Has 130/30 Strategy | — | Prospectus text only | No |
| OFST010650 | Is REIT | SEC N-CEN + XBRL | Classification data | **Partial** |
| OFST010660 | Is ETC | — | US concept is different | No |
| OFST010665 | Is ETN | SEC N-CEN | Product type | **Partial** |
| OFST010670 | Is Short | — | Derivable from fund name/strategy | **Derivable** (heuristic) |
| OFST010690 | Is Life Fund | — | Not a US concept | No |
| OFST010695 | Is Pension Fund | — | Not in SEC fund data | No |
| OFST010720 | Is Passive Fund | SEC N-CEN | INDEX table (tracked index) | **Derivable** |
| OFST010730 | Management Approach Type | — | Prospectus text only | No |
### 3C. Currencies & Hedging
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST010205 | Has Duration Hedge | — | Prospectus text only | No |
| OFST010211 | Currency Hedge Portfolio | — | Prospectus text only | No |
| OFST010220 | Has Embedded Derivatives | SEC N-PORT | Derivatives tables (non-empty) | **Derivable** |
| OFST020261 | Currency Hedge Share Class | — | Prospectus text only | No |
| OFST020530 | Is Multicurrency Share Class | — | Prospectus text only | No |
| OFST020540 | Share Class Currency | SEC XBRL Risk/Return | Currency context in fee/performance tables | **Partial** (USD implied) |
**Currency/hedging fields are almost entirely prospectus-derived and NOT available as structured public data.** This is a key gap: US funds are almost all USD-denominated, and hedging is described in prospectus narrative text. For LLM training, these fields represent extraction targets.
### 3D. Replication & Securities Lending
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST010900 | Replication Methodology First Level | — | Prospectus text only (ETFs) | No |
| OFST010901 | Replication Methodology Second Level | — | Prospectus text only (ETFs) | No |
| OFST011000 | Has Securities Lending | SEC N-PORT | SECURITIES_LENDING + BORROWER tables | **Yes** — TSV |
| OFST011100 | Has Swap | SEC N-PORT | Swap derivative tables | **Derivable** |
| OFST011110 | Swap Counterparty Name | SEC N-PORT | Counterparty fields in swap tables | **Yes** — TSV |
**Summary for Fund section**: ~25 of 73 fields available as structured data. The major gaps are: currency hedging, replication methodology, valuation timing, management approach, and leverage limits — all prospectus-narrative fields.
---
## 4. Key Fact: Share Class (OFST020000–049999) — 75 fields
### 4A. Identifiers
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST020000 | ISIN | OpenFIGI | FIGI → ISIN mapping | **Yes** — JSON |
| OFST020005 | CUSIP | SEC Series/Class CSV | Not directly, but derivable from ISIN | **Partial** |
| OFST020020 | Bloomberg Code | — | Proprietary (not free) | No |
| OFST020025 | FIGI Code | OpenFIGI | Direct lookup by ticker/ISIN | **Yes** — JSON |
| OFST020040 | SEDOL | — | Proprietary (London Stock Exchange) | No |
| OFST020045 | NFN Identifier | — | Nasdaq proprietary | No |
| OFST020050 | Share Class Extension | SEC Series/Class CSV | `Class Name` (parse letter/suffix) | **Derivable** |
| OFST020060 | Full Share Class Name | SEC Series/Class CSV | `Series Name` + `Class Name` | **Yes** — CSV |
### 4B. Share Class Characteristics
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST020300 | Valuation Frequency | — | Prospectus text only | No |
| OFST020400 | Share Class Distribution Policy | SEC XBRL Risk/Return | Derivable from dividend narrative | **Partial** |
| OFST020540 | Share Class Currency | — | Implied USD for US funds | **Partial** |
| OFST020545 | Share Class Lifecycle | SEC Submissions API | Filing history + Series/Class CSV status | **Derivable** |
| OFST020560 | Share Class Launch Date | SEC XBRL Risk/Return | `InceptionDate` per share class | **Yes** — XBRL |
| OFST020566 | Termination Date | SEC Series/Class CSV | Class status (active/inactive) | **Partial** |
| OFST020580 | Is Share Class Eligible For UCITS | — | Not applicable to US funds | No |
| OFST023100 | Investment Status | — | Prospectus text only | No |
| OFST023200 | Benchmark | SEC XBRL Risk/Return | `IndexNoDeductionForFeesExpensesTaxes` | **Yes** — XBRL |
| OFST023800 | Index Name (ETF) | SEC N-CEN | INDEX table | **Yes** — TSV |
| OFST024000 | SRRI | — | EU-specific risk indicator | No |
**Summary**: ~10 of 75 fields available. Share class operational details (valuation frequency, dealing days, settlement cycles) are entirely prospectus-derived.
---
## 5. Key Fact: Listing (OFST060000–064999) — 14 fields
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST060000 | Bloomberg Code Of Listing | — | Proprietary | No |
| OFST062000 | Listing Date | — | Exchange data (not SEC) | No |
| OFST062010 | Listing Currency | — | Implied USD for US-listed | **Partial** |
| OFST062025 | Launch Price | SEC XBRL Risk/Return | Inception price context | **Partial** |
| OFST062030 | Market Identifier Code | — | Not in SEC data directly | No |
| OFST062040 | Exchange Place | SEC N-CEN (ETFs) | Exchange information for ETFs | **Partial** |
**Summary**: 0-2 fields fully structured. Listing data is primarily from exchanges, not SEC filings.
---
## 6. Legal Structure (OFST160000–164999) — 7 fields
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST160039 | Is EU Directive Relevant | — | EU-specific | No |
| OFST160040 | Type Of EU Directive | — | EU-specific (UCITS/AIF) | No |
| OFST160100 | Legal Form | SEC Series/Class CSV | `Entity Org Type` | **Yes** — CSV |
| OFST160150 | Home Country Legal Type Of Fund | SEC N-CEN | Fund type classification | **Yes** — TSV |
**Summary**: 2 of 7 fields available. Most are EU-specific.
---
## 7. Classification (OFST350000–399999) — 12 fields
| OF-ID | Field Name | Public Source | Source Field / Method | Structured? |
|-------|-----------|--------------|----------------------|-------------|
| OFST350009 | Is Sharia Compliant | — | Not in SEC data | No |
| OFST350015 | CFI Code | OpenFIGI | FIGI metadata includes CFI | **Partial** |
| OFST350050 | Clearstream Asset Category | — | Proprietary classification | No |
| OFST350100 | EFAMA Main EFC Category | — | EU classification system | No |
| OFST351295 | Is Money Market Fund | SEC N-CEN + N-MFP | Money market fund flag | **Yes** — TSV |
| OFST351300 | Money Market Type Of Fund | SEC N-MFP | Fund type in N-MFP data | **Yes** — TSV |
**Major gap**: There is **no free, structured, universal fund asset class classification** in SEC data. The SEC does not tag funds as "equity", "fixed income", "mixed", etc. in a single structured field. Asset class must be derived from:
- Fund name heuristics ("Growth Fund" → equity, "Bond Fund" → fixed income)
- N-PORT holdings data (aggregate asset types held)
- XBRL strategy narrative text
This is a critical finding for LLM training: **asset class classification is an extraction target, not ground truth.**
---
## 8. Purchase Information / Settlement (OFST400000–449999) — 95 fields
This is the **largest gap** between openfunds and public data. Settlement and dealing information is almost entirely found only in prospectus text.
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| OFST400200 | Minimal Initial Subscription Category | — | No |
| OFST400230 | Minimal Initial Subscription In Amount | SEC XBRL Risk/Return | **Partial** — `MinimumInvestment` element exists but inconsistently tagged |
| OFST401002 | Pricing Methodology | — | No |
| OFST402500 | Maximal Number Of Possible Decimals Shares | — | No |
| OFST405521-405532 | Subscription Trade Cycle / Dealing Days | — | No |
| OFST410060 | Cut-off Date Offset for Subscription | — | No |
| OFST410100 | Cut-off Time For Subscription | — | No |
| OFST410700 | Settlement Period For Subscription | — | No |
| OFST410950 | Has Lock-up For Redemption | — | No |
| OFST420200-420265 | Redemption Minimums/Maximums | — | No |
| OFST420630 | Bank Details (SSI for Payments) | — | No |
| OFST425561-425572 | Redemption Trade Cycle / Dealing Days | — | No |
| OFST430100 | Cut-off Time For Redemption | — | No |
| OFST430150 | Settlement Period For Redemption | — | No |
**Summary**: **0-1 of 95 fields** available as structured data. Settlement cycles, cut-off times, dealing days, minimum investments, and payment details are exclusively in prospectus text. This is arguably the highest-value category for LLM extraction — these fields are critical for fund operations but exist only in legal documents.
---
## 9. Fees, Costs and Expenses (OFST450100–499999) — 62 fields
This is the **strongest area for SEC structured data**, thanks to the XBRL Risk/Return fee tables.
| OF-ID | Field Name | Public Source | Source Field | Structured? |
|-------|-----------|--------------|-------------|-------------|
| OFST451027 | Has Performance Fee | SEC XBRL Risk/Return | Fee narrative/table | **Partial** |
| OFST451030 | Performance Fee in Prospectus | SEC XBRL Risk/Return | Not separately tagged | **Partial** |
| OFST451305 | Applied Subscription Fee | SEC XBRL Risk/Return | `MaximumSalesChargeImposedOnPurchasesOverOfferingPrice` | **Yes** — XBRL |
| OFST451320 | Maximum Subscription Fee | SEC XBRL Risk/Return | `MaximumSalesChargeImposedOnPurchasesOverOfferingPrice` | **Yes** — XBRL |
| OFST451385 | Has Early Redemption Fee | SEC XBRL Risk/Return | `RedemptionFeeOverRedemption` | **Yes** — XBRL |
| OFST451390 | Has CDSC Fee | SEC XBRL Risk/Return | `MaximumDeferredSalesChargeOverOther` | **Yes** — XBRL |
| OFST451405 | Redemption Fee | SEC XBRL Risk/Return | `RedemptionFeeOverRedemption` | **Yes** — XBRL |
| OFST452000 | Management Fee Applied | SEC XBRL Risk/Return | `ManagementFeesOverAssets` | **Yes** — XBRL |
| OFST452100 | TER Excluding Performance Fee | SEC XBRL Risk/Return | `NetExpensesOverAssets` (OER equivalent) | **Yes** — XBRL |
| OFST452200 | Ongoing Charges | SEC XBRL Risk/Return | `TotalAnnualFundOperatingExpensesOverAssets` | **Yes** — XBRL |
| OFST453151 | Is Trailer Fee Clean | — | Not in SEC data | No |
| OFST454150 | Has Separate Distribution Fee | SEC XBRL Risk/Return | `Distribution12b1FeesOverAssets` | **Yes** — XBRL |
| OFST454160 | Distribution Fee | SEC XBRL Risk/Return | `Distribution12b1FeesOverAssets` | **Yes** — XBRL |
| — | Fee Waiver / Reimbursement | SEC XBRL Risk/Return | `FeeWaiverOrReimbursementOverAssets` | **Yes** — XBRL |
| — | Other Expenses | SEC XBRL Risk/Return | `OtherExpensesOverAssets` | **Yes** — XBRL |
| — | Expense Example (1yr/3yr/5yr/10yr) | SEC XBRL Risk/Return | `ExpenseExampleYear01` through `Year10` | **Yes** — XBRL |
### Key XBRL Fee Elements (complete shareholder fee table)
```
MaximumSalesChargeImposedOnPurchasesOverOfferingPrice
MaximumDeferredSalesChargeOverOther
MaximumSalesChargeOnReinvestedDividendsAndDistributionsOverOther
RedemptionFeeOverRedemption
MaximumAccountFee
ManagementFeesOverAssets
Distribution12b1FeesOverAssets
OtherExpensesOverAssets
AcquiredFundFeesAndExpensesOverAssets
TotalAnnualFundOperatingExpensesOverAssets
FeeWaiverOrReimbursementOverAssets
TotalAnnualFundOperatingExpensesAfterFeeWaiverOverAssets
ExpenseExampleYear01 / Year03 / Year05 / Year10
ExpenseExampleNoRedemptionYear01 / Year03 / Year05 / Year10
```
**Summary**: ~15 of 62 fee fields are available as structured XBRL data. The SEC fee taxonomy is detailed for US-style fees (sales charges, 12b-1, management fee, expense ratio) but does not cover European concepts like custodian fee breakdown, trailer fee clean status, or performance fee details (hurdle rate, high water mark).
---
## 10. Solvency II (OFST500000–519999) — 13 fields
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| All 13 fields | SCR Market Risk, Tripartite Reports | — | **No** — entirely EU insurance regulation |
**Summary**: 0 of 13 fields available. Solvency II is a European directive not applicable to US SEC data.
---
## 11. Taxes (OFST800000–819999) — 27 fields
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| OFST809200 | Is US Tax Forms W8/W9 Needed | — | Prospectus text only | No |
| OFST809210 | Is US K1 Reporting Required | SEC N-CEN | Partnership/LP fund flags | **Partial** |
| OFST809250 | Is Flow-Through Entity By US Tax Law | — | Prospectus text only | No |
| OFST809511 | FATCA Status | — | IRS data, not SEC structured | No |
| OFST809520 | Subject To FATCA Withholding | — | Prospectus text only | No |
| OFST801011 | Is Austrian Tax Reporting Fund | — | Austria-specific | No |
| OFST802001–802045 | German Tax fields (8 fields) | — | Germany-specific | No |
| OFST802500 | Luxembourg Taxe d'Abonnement | — | Luxembourg-specific | No |
| OFST808008–808100 | Swiss Tax fields (3 fields) | — | Switzerland-specific | No |
| OFST809015 | Has UK Reporting Status | — | UK-specific | No |
**Summary**: 0-1 of 27 fields available. Tax fields are overwhelmingly jurisdiction-specific (DE, AT, CH, LU, UK, FR, ES) and not in SEC data. The few US-relevant fields (FATCA, K-1) are in prospectus text.
---
## 12. ESG Data (OFST820000–849999) — 65 fields
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| OFST820110-820280 | Carbon Intensity / Footprint / Absolute GHG (18 fields) | — | **No** — not yet required in SEC filings for funds |
| OFST820290-820360 | Fossil Fuel Exposure (8 fields) | — | **No** |
| OFST820370-820380 | Net Zero Commitments (2 fields) | — | **No** |
| OFST820390 | Implied Temperature Rise | — | **No** |
| OFST820440-820460 | GHG Reduction Goals (3 fields) | — | **No** |
| OFST820470-820540 | Climate Stewardship (8 fields) | — | **No** |
| OFST820600-820675 | AMAS / ACT Signatory fields (8 fields) | — | **No** — Swiss specific |
| OFST830000-830210 | UK SDR fields (12 fields) | — | **No** — UK specific |
| OFST001025 | Is UN PRI Signatory | UN PRI website | **Partial** — searchable but not API |
**Current state of SEC ESG data**: The SEC adopted climate disclosure rules in March 2024 (effective May 2024), but these apply to operating companies, **not investment funds**. The Investment Company Names Rule (addressing ESG fund naming) has compliance dates of June 2026 / December 2026. As of February 2026, there is no SEC-mandated structured ESG data for funds comparable to EU SFDR.
**Summary**: **0 of 65 ESG fields** are available as structured public data from SEC. ESG fund data is available from commercial providers (Morningstar, MSCI, Sustainalytics) but not from any free structured public source.
---
## 13. Dynamic Data: Prices & AuM (OFDY000001–000999) — 20 fields
| OF-ID | Field Name | Public Source | Source Field | Structured? |
|-------|-----------|--------------|-------------|-------------|
| OFDY000010 | Price Currency | — | Implied USD for US funds | **Partial** |
| OFDY000035 | Valuation NAV | SEC N-PORT | Not directly; XBRL Company Facts for some | **Partial** |
| OFDY000060 | AuM Fund | SEC N-PORT | `FUND_REPORTED_INFO.total_assets` | **Yes** — TSV |
| OFDY000070 | AuM Share Class | SEC N-PORT | Per-class AuM when reported | **Partial** |
| OFDY000075 | NoS Share Class | — | Not in SEC structured data | No |
**Summary**: 1-2 of 20 fields available. Daily NAV prices are not in SEC structured data (available from commercial sources). Fund-level AuM is in N-PORT.
---
## 14. Dynamic Data: Performance & Risk (OFDY025000–049999) — 4 fields
These 4 fields are Germany-specific (equity participation ratio, total fund asset share, etc.) and not available from SEC.
**Additional performance data in SEC**: While openfunds has few performance OFDY fields, SEC XBRL Risk/Return provides:
| SEC Element | Description | Structured? |
|-------------|-------------|-------------|
| `AnnualReturn20XX` | Calendar year annual returns (1yr–10yr) | **Yes** — XBRL |
| `HighestQuarterlyReturnLabel/Value` | Best quarter return | **Yes** — XBRL |
| `LowestQuarterlyReturnLabel/Value` | Worst quarter return | **Yes** — XBRL |
| `AverageAnnualReturnYear01/05/10/SinceInception` | Average annual returns | **Yes** — XBRL |
| `BarChartClosingTextBlock` | Performance chart narrative | **Yes** — XBRL (text) |
And N-PORT provides:
| N-PORT Field | Description | Structured? |
|-------------|-------------|-------------|
| `MONTHLY_TOTAL_RETURN` | Monthly returns by class | **Yes** — TSV |
| `MONTHLY_RETURN_CAT_INSTRUMENT` | Returns by asset category | **Yes** — TSV |
| `FUND_VAR_INFO` | Value-at-Risk | **Yes** — TSV |
| `INTEREST_RATE_RISK` | DV01/DV100 by maturity bucket | **Yes** — TSV |
---
## 15. Portfolio Holdings (OFPH000001–999999) — 92 fields
N-PORT is the primary source. SEC requires monthly portfolio disclosure.
| OF-ID | Field Name | N-PORT Source | Structured? |
|-------|-----------|--------------|-------------|
| OFPH000010 | Holding as at Date | Reporting date | **Yes** |
| OFPH000020 | Portfolio Currency | Fund currency context | **Yes** (USD) |
| OFPH000100 | Holding ISIN | IDENTIFIERS table | **Yes** |
| OFPH000130 | Holding Ticker | IDENTIFIERS table | **Yes** |
| OFPH000145 | Holding CUSIP | IDENTIFIERS table | **Yes** |
| OFPH000170 | Holding FIGI | — | No (use OpenFIGI to map) |
| OFPH000200 | Holding Name | `FUND_REPORTED_HOLDING.name` | **Yes** |
| OFPH000210 | Holding Instrument Type | `FUND_REPORTED_HOLDING.asset_cat` | **Yes** |
| OFPH000250 | Holding Market Value | `FUND_REPORTED_HOLDING.balance` + `val_usd` | **Yes** |
| OFPH000300 | Holding Net Weight as % | `FUND_REPORTED_HOLDING.pctVal` | **Yes** |
| OFPH000400 | Holding Currency | `FUND_REPORTED_HOLDING.curCd` | **Yes** |
| OFPH000420 | Holding Risk Country | `FUND_REPORTED_HOLDING.invCountry` | **Yes** |
| OFPH000430 | Holding Asset Class | `FUND_REPORTED_HOLDING.asset_cat` | **Yes** |
| OFPH000440 | Holding Credit Rating | DEBT_SECURITY fields | **Yes** |
| OFPH000450 | Holding Number of Shares | `FUND_REPORTED_HOLDING.balance` | **Yes** |
| OFPH000460 | Holding Coupon Rate | DEBT_SECURITY fields | **Yes** |
| OFPH000465 | Holding Modified Duration | — | No |
| OFPH000480 | Holding Maturity Date | DEBT_SECURITY fields | **Yes** |
| OFPH000600-650 | Interest Rate Type / Index / Margin | DEBT_SECURITY fields | **Yes** |
| OFPH000700 | Holding Issuer Name | `FUND_REPORTED_HOLDING.issuerConditionalName` | **Yes** |
| OFPH000710 | Holding Issuer LEI | `FUND_REPORTED_HOLDING.lei` | **Yes** |
| OFPH000712 | Holding Issuer Domicile | `FUND_REPORTED_HOLDING.invCountry` | **Yes** |
| OFPH000730 | Holding Strike Price | Derivative tables | **Yes** |
| OFPH000800-870 | Underlying Asset fields | Derivative tables | **Yes** |
**Summary**: ~35-40 of 92 fields available from N-PORT. The main gaps are: modified/effective duration (calculated, not reported), GICS sector codes (not in N-PORT directly), and European-specific fields (CIC, NACE, EUSIPA, WKN, Valor).
---
## 16. Fund Ratios and Exposures (OFRE000001–999999) — 42 fields
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| OFRE000010 | Number Of Positions | SEC N-PORT | **Derivable** — count holdings |
| OFRE000200 | Exposure To Cash | SEC N-PORT | **Derivable** — sum cash-type holdings |
| OFRE000300-320 | Credit Quality fields | SEC N-PORT | **Derivable** — aggregate from holdings |
| OFRE000330 | Average Effective Maturity | — | Not directly in N-PORT | No |
| OFRE000335 | Average Effective Duration | — | Not directly in N-PORT | No |
| OFRE000350 | Yield To Maturity | — | Not in N-PORT | No |
| OFRE000500 | Top Ten Positions | SEC N-PORT | **Derivable** — sort by weight |
| OFRE000520 | Country Breakdown | SEC N-PORT | **Derivable** — aggregate by country |
| OFRE000540 | Currency Breakdown | SEC N-PORT | **Derivable** — aggregate by currency |
| OFRE000560 | GICS Equity Sector Breakdown | — | GICS not in N-PORT | No |
| OFRE000570 | Market Cap Breakdown | — | Not in N-PORT | No |
| OFRE000580 | Credit Rating Breakdown | SEC N-PORT | **Derivable** — aggregate by rating |
| OFRE000590 | Maturity Breakdown | SEC N-PORT | **Derivable** — aggregate by maturity |
| OFRE000600 | Asset Class Breakdown | SEC N-PORT | **Derivable** — aggregate by asset_cat |
**Summary**: ~10-15 of 42 fields are derivable from N-PORT holdings data. Pre-computed ratios (YTM, duration, OAS) are not available.
---
## 17. Portfolio Manager Data (OFPM000001–999999) — 8 fields
| OF-ID | Field Name | Public Source | Structured? |
|-------|-----------|--------------|-------------|
| OFPM000010 | Portfolio Manager Name | SEC XBRL Risk/Return | `PortfolioManager` text block | **Partial** (text, not structured) |
| OFPM000060 | Portfolio Manager Brief Biography | SEC XBRL Risk/Return | SAI supplement text | **Partial** (text) |
| Others | Year of birth, experience, role | — | Not structured | No |
**Summary**: 0-1 of 8 fields. Portfolio manager data is in prospectus SAI text, not structured.
---
## Grand Summary: Structured Data Availability by Category
| Category | Total Fields | Structured Public | Derivable | Not Available |
|----------|-------------|------------------|-----------|---------------|
| **Company** (service providers) | 40 | **15** | 0 | 25 |
| **Umbrella** | 10 | **4** | 1 | 5 |
| **Fund** (identity, structure) | 73 | **15** | 10 | 48 |
| **Share Class** | 75 | **8** | 2 | 65 |
| **Listing** | 14 | **0** | 2 | 12 |
| **Legal Structure** | 7 | **2** | 0 | 5 |
| **Classification** | 12 | **2** | 0 | 10 |
| **Purchase / Settlement** | 95 | **0** | 1 | 94 |
| **Fees** | 62 | **15** | 0 | 47 |
| **Solvency II** | 13 | **0** | 0 | 13 |
| **Taxes** | 27 | **0** | 1 | 26 |
| **ESG** | 65 | **0** | 0 | 65 |
| **Prices / AuM** | 20 | **2** | 0 | 18 |
| **Performance / Risk** | 4 | **0** | 0 | 4 |
| **Portfolio Holdings** | 92 | **38** | 0 | 54 |
| **Ratios / Exposures** | 42 | **0** | 14 | 28 |
| **Portfolio Manager** | 8 | **0** | 1 | 7 |
| **TOTAL** | **659** | **~101 (15%)** | **~32 (5%)** | **~526 (80%)** |
---
## Implications for LLM Training Dataset
### What this means:
1. **~15% of openfunds fields** have directly available structured public data (primarily from SEC EDGAR: XBRL fees, N-PORT holdings, N-CEN service providers, Series/Class CSV identifiers).
2. **~5% are derivable** from structured data (e.g., aggregating N-PORT holdings into country/currency/rating breakdowns, counting positions, inferring ETF status from N-CEN index tracking).
3. **~80% are NOT available** as structured public data and exist only in prospectus narrative text.
### The 80% gap = the LLM opportunity
The fields that are **not** available as structured data but **are** specified in prospectus text represent the core value proposition for LLM extraction:
| Category | Key Extraction Targets |
|----------|----------------------|
| **Settlement / Dealing** | Cut-off times, settlement periods, dealing days, minimum subscriptions, pricing methodology |
| **Currencies / Hedging** | Share class currency hedging, portfolio hedging, multicurrency options |
| **Risk Limits** | Maximum leverage, redemption gates, lock-up periods, side pockets |
| **Asset Class** | Fund classification (equity/bond/mixed/alternative), investment strategy |
| **Fee Details** | Performance fee mechanics (hurdle rate, high water mark, crystallization), custodian fees |
| **ESG** | Sustainability approach, climate targets, exclusion criteria |
| **Tax** | FATCA status, K-1 requirements, flow-through entity status |
### Recommended approach for training data:
- **Ground truth (structured data)**: Use SEC XBRL fees, N-PORT holdings, N-CEN service providers, and Series/Class CSV as verifiable reference data.
- **Extraction targets (unstructured → structured)**: Use the 80% of openfunds fields that exist only in prospectus text as the fields the LLM should learn to extract.
- **Validation**: For the ~15% structured fields, compare LLM extraction from prospectus text against SEC structured data to measure extraction accuracy.
---
## Appendix: Data Source URLs
| Source | URL |
|--------|-----|
| SEC Series/Class CSV | https://www.sec.gov/data-research/sec-markets-data/investment-company-series-class-information |
| SEC XBRL Risk/Return Data Sets | https://www.sec.gov/data-research/sec-markets-data/mutual-fund-prospectus-riskreturn-summary-data-sets |
| SEC N-PORT Data Sets | https://www.sec.gov/data-research/sec-markets-data/form-n-port-data-sets |
| SEC N-CEN Data Sets | https://www.sec.gov/data-research/sec-markets-data/form-n-cen-data-sets |
| SEC Submissions API | https://data.sec.gov/submissions/CIK{cik}.json |
| SEC XBRL Company Facts API | https://data.sec.gov/api/xbrl/companyfacts/CIK{cik}.json |
| GLEIF LEI Database | https://search.gleif.org/ / https://www.gleif.org/en/lei-data/gleif-api |
| OpenFIGI API | https://www.openfigi.com/api |
| SEC N-MFP (Money Market) | https://www.sec.gov/data-research/sec-markets-data |

83
README.md Normal file
View File

@ -0,0 +1,83 @@
# fund_rdf_data
A relationship-rich **finance dataset for text-to-RDF-triple extraction**, built
from mandatory U.S. SEC fund disclosures. Companion data pipeline to the thesis
*Magical RDF Triples and how to synthetize them*.
Each sample pairs a long natural-language **prospectus section** (input) with a
compact graph of **entity-to-entity RDF triples** (target) — a fund *advised by*
a manager, *distributed by* an underwriter, *seriesOf* a trust, and so on. Unlike
Wikidata-derived corpora where text ≈ triples, here the input is ~20–50× larger
than the output, and the target is a genuine knowledge graph rather than flat
attributes. Ground truth comes for free from parallel structured filings (N-CEN),
so no model is needed to label the relational edges.
See [`dataset_description.pdf`](dataset_description.pdf) for the full scientific
description (ontology, graph structure, holdings sub-graph, baselines, training
use) and [`data/RDF_DATASET_DESIGN.md`](data/RDF_DATASET_DESIGN.md) for design
notes.
## Pipeline
The dataset is built by [`build_rdf_dataset.py`](build_rdf_dataset.py) in four stages:
```bash
# 1. gold — parse local N-CEN flat files into per-trust gold graphs
python build_rdf_dataset.py gold --custodian-scope primary
# 2. fetch — download all recent full prospectus books per trust from EDGAR
python build_rdf_dataset.py fetch --limit 435 --max-filings 8
# 3. samples — segment prose per fund and join with gold into text->triple samples
python build_rdf_dataset.py samples
# 4. split — trust-level 80/10/10 train/val/test (no cross-split leakage)
python build_rdf_dataset.py split
# or run all four:
python build_rdf_dataset.py all --limit 435
```
[`score_baseline.py`](score_baseline.py) computes a no-model string-match baseline
and scores strong-model predictions against the gold:
```bash
python score_baseline.py stringmatch # no-model lower bound
python score_baseline.py model --pred preds.jsonl
```
## Sample format
Each line of `samples.jsonl` / `train|val|test.jsonl` is a JSON record:
| field | meaning |
|---|---|
| `input_text` | prospectus prose for the fund (model input) |
| `ontology` | inferred meta-schema (subject type → predicate → object type) |
| `target_triples` | structured `{s,p,o}` list |
| `target_serialized` | marker form (`<triple_start>` …) for Models 2/4 |
| `target_serialized_plain` | Turtle-like form, no special tokens, for Models 1/3 |
| `cik`, `series_id`, `fund`, `trust_name` | identifiers |
| `stats` | input/target sizes, triple count, text:json ratio |
## Relations
Entity-to-entity edges (gold from N-CEN / Series-Class):
`seriesOf`, `advisedBy`, `subAdvisedBy`, `transferAgent`, `custodian` (primary
only by default), `administrator`, `underwrittenBy`. Holdings edges
(`holds`/`issuedBy`/`domiciledIn`, gold from N-PORT) are a planned second track
from annual-report (N-CSR) commentary — see the description PDF.
## Data sources
All inputs are public SEC filings (EDGAR, DERA data sets). The large raw data
(prospectus prose, bulk N-CEN/N-PORT/XBRL downloads) and the generated sample
files are **git-ignored** because they are reproducible from the commands above;
only the lightweight structured gold graph (`data/rdf_poc/gold_graphs.jsonl`) is
committed.
## Requirements
```bash
pip install -r requirements.txt
```

211
SEC_FUND_DATA_RESEARCH.md Normal file
View File

@ -0,0 +1,211 @@
# SEC Fund Prospectus & Reference Data Research
## 1. Overview: Where Are Fund Prospectuses at the SEC?
All US-registered investment companies (mutual funds, ETFs) must file their prospectuses and amendments with the SEC via the **EDGAR** (Electronic Data Gathering, Analysis, and Retrieval) system. These filings are publicly available at no cost.
### Relevant SEC Filing Form Types
| Form Type | Description | Content |
|-----------|-------------|---------|
| **N-1A** | Initial registration statement for open-end management investment companies (mutual funds) | Full prospectus + SAI (Statement of Additional Information) |
| **485BPOS** | Post-effective amendment that becomes effective upon filing | Updated prospectus (most common annual update) |
| **485APOS** | Post-effective amendment filed in advance | Prospectus amendment requiring SEC review |
| **497** | Definitive materials filed under Investment Company Act | Prospectus supplements, SAI updates, sticker supplements |
| **497K** | Summary prospectus for mutual funds | Condensed key-facts prospectus (3-4 pages per fund/class) |
| **N-CSR** | Certified shareholder report | Annual/semi-annual report with financial statements |
| **NPORT-P** | Monthly portfolio holdings (public quarterly) | Detailed portfolio positions |
### Key Insight for Your Use Case
- **485BPOS** filings contain the **full statutory prospectus** (the main legal document)
- **497K** filings contain the **summary prospectus** (condensed, standardized format ideal for LLM training)
- **497** filings contain **supplements and amendments** that modify the prospectus
---
## 2. Where Is the Reference Data?
Reference data for funds and their share classes comes from **three complementary sources**:
### Source A: Investment Company Series and Class Information (CSV/XML)
- **URL**: `https://www.sec.gov/data-research/sec-markets-data/investment-company-series-class-information`
- **Download**: `https://www.sec.gov/files/investment/data/other/investment-company-series-class-information/investment-company-series-class-2025.csv`
- **Updated**: Monthly (latest: June 2025)
- **Contains**:
- CIK (Central Index Key) — the umbrella fund company identifier
- File Number (811-XXXXX) — Investment Company Act registration number
- Series ID (S######) — unique identifier per fund series (e.g., "Vanguard 500 Index Fund")
- Class ID (C######) — unique identifier per share class
- Ticker Symbol
- Fund/Class name
- Status (active/inactive)
### Source B: Mutual Fund Prospectus Risk/Return Summary Data Sets (XBRL)
- **URL**: `https://www.sec.gov/data-research/sec-markets-data/mutual-fund-prospectus-riskreturn-summary-data-sets`
- **Download**: Quarterly ZIP files, e.g., `https://www.sec.gov/files/dera/data/mutual-fund-prospectus-risk/return-summary-data-sets/2025q2_rr1.zip`
- **Coverage**: December 2010 to present (quarterly updates)
- **Contains structured XBRL data extracted from prospectuses**:
- Expense ratios (management fees, 12b-1 fees, total annual operating expenses)
- Fee tables and expense examples
- Performance bar charts and tables (1yr, 5yr, 10yr returns)
- Risk/return narratives
- Investment objectives
- Principal strategies and risks
### Source C: EDGAR Submissions API (JSON)
- **URL**: `https://data.sec.gov/submissions/CIK{cik_padded}.json`
- **Contains**: Full filing history per entity, including metadata (dates, form types, accession numbers, primary document URLs)
---
## 3. Unique Identifiers for Each Share Class
| Identifier | Format | Source | Scope |
|-----------|--------|--------|-------|
| **CIK** | 10-digit number (e.g., 0000036405) | SEC EDGAR | Umbrella fund entity (investment company trust) |
| **File Number** | 811-XXXXX | SEC | Investment Company Act registration |
| **Series ID** | S###### (e.g., S000002839) | SEC EDGAR | Individual fund within the trust |
| **Class ID** | C###### (e.g., C000007773) | SEC EDGAR | Individual share class |
| **Ticker** | 1-5 chars (e.g., VFIAX) | Exchange | Share class trading identifier |
| **CUSIP** | 9-char alphanumeric | CUSIP Global Services | Security-level (per share class) |
| **ISIN** | 12-char (US + CUSIP + check) | ANNA | International identifier (derived from CUSIP for US) |
| **Accession Number** | XXXXXXXXXX-YY-ZZZZZZ | SEC EDGAR | Unique per filing |
### Hierarchy
```
Investment Company Trust (CIK) — e.g., "Vanguard Index Funds"
└── Fund Series (Series ID) — e.g., "Vanguard 500 Index Fund"
├── Share Class (Class ID) — e.g., "Investor Shares" (VFINX)
├── Share Class (Class ID) — e.g., "Admiral Shares" (VFIAX)
└── Share Class (Class ID) — e.g., "ETF Shares" (VOO)
```
---
## 4. APIs and Web Endpoints for Downloading Data
### 4.1 EDGAR Submissions API (FREE, no authentication)
```
GET https://data.sec.gov/submissions/CIK{10-digit-cik}.json
Header: User-Agent: "YourApp your@email.com"
```
Returns: JSON with entity metadata + all filing history (form type, dates, accession numbers, primary document filenames).
### 4.2 EDGAR Full-Text Search API (FREE, no authentication)
```
GET https://efts.sec.gov/LATEST/search-index?q={query}&forms={form_types}&dateRange=custom&startdt={YYYY-MM-DD}&enddt={YYYY-MM-DD}
Header: User-Agent: "YourApp your@email.com"
```
Returns: JSON with matching filings, metadata, and file references. Supports Boolean operators (AND, OR, NOT, NEAR) and wildcard (*).
### 4.3 Filing Document Download (FREE, no authentication)
```
GET https://www.sec.gov/Archives/edgar/data/{cik}/{accession-number-no-dashes}/{document-filename}
Header: User-Agent: "YourApp your@email.com"
```
Returns: The actual filing document (HTML, XML, or PDF).
### 4.4 XBRL APIs (FREE, no authentication)
```
GET https://data.sec.gov/api/xbrl/companyfacts/CIK{10-digit-cik}.json
GET https://data.sec.gov/api/xbrl/companyconcept/CIK{cik}/{taxonomy}/{concept}.json
```
### 4.5 Rate Limits
- **10 requests per second** maximum
- Must include a `User-Agent` header with company name and email
- No API key required
---
## 5. Five Concrete Fund Examples
### Example 1: Vanguard 500 Index Fund
| Field | Value |
|-------|-------|
| **Trust Name** | Vanguard Index Funds |
| **CIK** | 0000036405 |
| **File Number** | 811-02652 |
| **Series ID** | S000002839 |
| **Fund Name** | Vanguard 500 Index Fund |
| **Share Classes** | |
| Investor Shares | Class ID: C000007773, Ticker: VFINX |
| Admiral Shares | Class ID: C000007774, Ticker: VFIAX |
| ETF Shares | Class ID: C000092055, Ticker: VOO |
| Institutional Select | Class ID: C000170274, Ticker: VFFSX |
| **Latest 497K (Summary Prospectus)** | `https://www.sec.gov/Archives/edgar/data/36405/000168386325004160/f41649d1.htm` |
| **Submissions JSON** | `https://data.sec.gov/submissions/CIK0000036405.json` |
### Example 2: Fidelity Contrafund
| Field | Value |
|-------|-------|
| **Trust Name** | Fidelity Contrafund |
| **CIK** | 0000024238 |
| **File Number** | 811-01400 |
| **Series ID** | S000006037 |
| **Fund Name** | Fidelity Contrafund |
| **Share Classes** | |
| Fidelity Contrafund | Class ID: C000016601, Ticker: FCNTX |
| Class K | Class ID: C000064233, Ticker: FCNKX |
| K6 Class | Class ID: C000182865, Ticker: FLCNX |
| **Submissions JSON** | `https://data.sec.gov/submissions/CIK0000024238.json` |
### Example 3: iShares Core S&P 500 ETF (BlackRock)
| Field | Value |
|-------|-------|
| **Trust Name** | iShares Trust |
| **CIK** | 0001100663 |
| **File Number** | 811-09729 |
| **Series ID** | S000002838 |
| **Fund Name** | iShares Core S&P 500 ETF |
| **Share Classes** | |
| ETF Shares | Ticker: IVV |
| **Submissions JSON** | `https://data.sec.gov/submissions/CIK0001100663.json` |
### Example 4: Columbia Funds Series Trust I
| Field | Value |
|-------|-------|
| **Trust Name** | Columbia Funds Series Trust I |
| **CIK** | 0000773757 |
| **File Number** | 811-04367 |
| **Submissions JSON** | `https://data.sec.gov/submissions/CIK0000773757.json` |
### Example 5: T. Rowe Price Exchange-Traded Funds
| Field | Value |
|-------|-------|
| **Trust Name** | T. Rowe Price Exchange-Traded Funds, Inc. |
| **CIK** | 0001795351 |
| **File Number** | 811-23494 |
| **Submissions JSON** | `https://data.sec.gov/submissions/CIK0001795351.json` |
---
## 6. Data Architecture for LLM Training Dataset
### Goal: Pair prospectus text with structured reference data
```
Dataset Record:
├── fund_identity
│ ├── cik, series_id, class_id, ticker, cusip
│ ├── fund_name, trust_name
│ └── file_number
├── prospectus_document
│ ├── accession_number, filing_date, form_type
│ ├── document_url
│ └── document_text (HTML/XML → plain text)
├── structured_reference_data (from XBRL risk/return)
│ ├── investment_objective
│ ├── expense_ratio, management_fee, 12b1_fee
│ ├── performance_1yr, performance_5yr, performance_10yr
│ ├── minimum_investment
│ └── risk_narratives
└── supplements_amendments[]
├── accession_number, filing_date, form_type
└── document_text
```
This allows training an LLM to:
1. **Extract** structured fields (expense ratio, objective, fees) from raw prospectus text
2. **Validate** extracted data against the XBRL ground truth
3. **Handle** amendments/supplements that modify the base prospectus

308
SEC_REFERENCE_DATA_vs_OPENFUNDS.md Normal file
View File

@ -0,0 +1,308 @@
# SEC Reference Data Fields vs. openfunds Data Model
## 1. Overview of SEC Structured Data Sources
The SEC provides **four distinct structured data sources** that contain reference data for US-registered funds. Each covers different aspects:
| Source | Form | Content | Format | Granularity |
|--------|------|---------|--------|-------------|
| **Series/Class CSV** | — | Identity & identifiers | CSV/XML | Trust → Series → Class |
| **XBRL Risk/Return** | N-1A (485BPOS, 497K) | Prospectus-derived structured data | XBRL → flat files | Series & Class level |
| **N-PORT Data Sets** | NPORT-P | Portfolio holdings & fund financials | XML → flat files | Series & Holding level |
| **Submissions API** | — | Filing history metadata | JSON | Entity (CIK) level |
---
## 2. Complete Field Inventory by SEC Source
### 2.1 Series/Class Reference CSV
This is the **identity backbone** — maps the hierarchy of trust → fund → share class.
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `Reporting File Number` | 811-XXXXX Investment Co. Act number | — (no direct equivalent) |
| `CIK Number` | 10-digit SEC entity identifier | — (SEC-specific) |
| `Entity Name` | Trust/investment company name | OFST005010 Umbrella Name |
| `Entity Org Type` | Organization type code | OFST160100 Legal Form |
| `Series ID` | S###### fund series identifier | — (SEC-specific) |
| `Series Name` | Fund name | OFST010110 Legal Fund Name Only |
| `Class ID` | C###### share class identifier | — (SEC-specific) |
| `Class Name` | Share class name (e.g. "Admiral Shares") | OFST020060 Full Share Class Name |
| `Class Ticker` | Exchange ticker symbol | OFST020020 Bloomberg Code (partial) |
| `Address_1, Address_2, City, State, Zip Code` | Registrant address | — |
**Coverage**: ~15,000+ investment company trusts, ~50,000+ series, ~100,000+ classes.
### 2.2 XBRL Risk/Return Summary (from Prospectus — the richest source)
This dataset is **extracted from prospectus XBRL filings** and is the closest to what openfunds covers. It contains the structured data that prospectuses specify.
#### A. Fund Identity & Structure
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `RiskReturnHeading` | Prospectus section heading | Text | — |
| `ObjectiveHeading` | Heading of objectives section | Text | — |
| `ObjectivePrimaryTextBlock` | Investment objective narrative | Text Block | OFST010300 Investment Objective |
| `ObjectiveSecondaryTextBlock` | Additional objective detail | Text Block | OFST010300 Investment Objective |
| `StrategyHeading` | Heading of strategy section | Text | — |
| `StrategyNarrativeTextBlock` | Principal investment strategies | Text Block | — (no single openfunds equivalent) |
#### B. Fee & Expense Data (Shareholder Fees — paid directly by investor)
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `MaximumSalesChargeImposedOnPurchasesOverOfferingPrice` | Front-end load | Ratio | OFST451320 Max Subscription Fee In Favour Of Distributor |
| `MaximumDeferredSalesChargeOverOfferingPrice` | Back-end load (CDSC) | Ratio | OFST451391 Contingent Deferred Sales Charge Exit Fee |
| `MaximumDeferredSalesChargeOverOther` | CDSC on other basis | Ratio | OFST451392 Contingent Deferred Sales Charge Upfront Fee |
| `MaximumSalesChargeOnReinvestedDividendsAndDistributions` | Load on reinvested dividends | Ratio | — |
| `RedemptionFeeOverRedemption` | Redemption fee (% of amount) | Ratio | OFST451440 Max Redemption Fee In Favour Of Fund |
| `RedemptionFee` | Redemption fee (flat $) | Monetary | OFST451439 Min Redemption Fee In Favour Of Fund |
| `ExchangeFeeOverRedemption` | Exchange fee (% of amount) | Ratio | — |
| `ExchangeFee` | Exchange fee (flat $) | Monetary | — |
| `MaximumAccountFeeOverAssets` | Account maintenance fee (%) | Ratio | — |
| `MaximumAccountFee` | Account maintenance fee ($) | Monetary | — |
| `MaximumCumulativeSalesChargeOverOfferingPrice` | Cumulative max sales charge | Ratio | — |
#### C. Annual Fund Operating Expenses (ongoing costs deducted from fund assets)
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `ManagementFeesOverAssets` | Management fee | Ratio | OFST452010 Management Fee Maximum |
| `DistributionAndService12b1FeesOverAssets` | 12b-1 distribution fee | Ratio | OFST454165 Distribution Fee Maximum |
| `Component1OtherExpensesOverAssets` | Other expense component 1 | Ratio | — |
| `Component2OtherExpensesOverAssets` | Other expense component 2 | Ratio | — |
| `Component3OtherExpensesOverAssets` | Other expense component 3 | Ratio | — |
| `OtherExpensesOverAssets` | Total other expenses | Ratio | — |
| `AcquiredFundFeesAndExpensesOverAssets` | Acquired fund fees (fund-of-funds) | Ratio | — |
| `ExpensesOverAssets` | **Total Annual Fund Operating Expenses** | Ratio | OFST452100 TER Excluding Performance Fee |
| `FeeWaiverOrReimbursementOverAssets` | Fee waiver/reimbursement | Ratio | — |
| `NetExpensesOverAssets` | Net expenses after waivers | Ratio | OFST452200 Ongoing Charges |
#### D. Expense Example (hypothetical cost projections)
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `ExpenseExampleYear01` | Cost for $10K after 1 year | Monetary | — |
| `ExpenseExampleYear03` | Cost for $10K after 3 years | Monetary | — |
| `ExpenseExampleYear05` | Cost for $10K after 5 years | Monetary | — |
| `ExpenseExampleYear10` | Cost for $10K after 10 years | Monetary | — |
| `ExpenseExampleNoRedemptionYear01` | Cost if no redemption, 1 year | Monetary | — |
| `ExpenseExampleNoRedemptionYear03` | Cost if no redemption, 3 years | Monetary | — |
| `ExpenseExampleNoRedemptionYear05` | Cost if no redemption, 5 years | Monetary | — |
| `ExpenseExampleNoRedemptionYear10` | Cost if no redemption, 10 years | Monetary | — |
#### E. Performance Data
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `AnnualReturn[YYYY]` | Annual return for calendar year | Ratio | OFDY025000-range (Performance data) |
| `BarChartHighestQuarterlyReturn` | Best quarter return | Ratio | — |
| `BarChartLowestQuarterlyReturn` | Worst quarter return | Ratio | — |
| `BarChartHighestQuarterlyReturnDate` | Date of best quarter | Date | — |
| `BarChartLowestQuarterlyReturnDate` | Date of worst quarter | Date | — |
| `AverageAnnualReturnYear01` | Average annual return, 1 year | Ratio | OFDY025000-range |
| `AverageAnnualReturnYear05` | Average annual return, 5 years | Ratio | OFDY025000-range |
| `AverageAnnualReturnYear10` | Average annual return, 10 years | Ratio | OFDY025000-range |
| `AverageAnnualReturnSinceInception` | Return since inception | Ratio | — |
| `AverageAnnualReturnInceptionDate` | Inception date | Date | OFST020560 Share Class Launch Date |
| Performance dimensions: Before Taxes, After Taxes on Distributions, After Taxes on Distributions and Sales | | | — |
#### F. Risk Disclosures
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `RiskHeading` | Risk section heading | Text | — |
| `RiskNarrativeTextBlock` | Principal risks narrative | Text Block | — |
| `RiskLoseMoney` | "You may lose money" statement | String | — |
| `RiskMoneyMarketFundMayImposeFeesOrSuspendSales` | MMF gate/fee risk | Boolean | — |
| `RiskMoneyMarketFundPriceFluctuates` | MMF NAV fluctuation risk | Boolean | — |
| `BarChartAndPerformanceTableHeading` | Performance section heading | Text | — |
| `PerformanceNarrativeTextBlock` | Performance context narrative | Text Block | — |
#### G. Portfolio Turnover
| XBRL Element | Description | Data Type | openfunds Equivalent |
|-------------|-------------|-----------|---------------------|
| `PortfolioTurnoverHeading` | Section heading | Text | — |
| `PortfolioTurnoverTextBlock` | Turnover narrative | Text Block | — |
| `PortfolioTurnoverRate` | Turnover rate (%) | Ratio | OFRE000025-range (Fund Ratios) |
### 2.3 N-PORT Data Sets (Portfolio Holdings — quarterly)
This provides **dynamic portfolio data** not typically in openfunds static fields.
#### A. Fund-Level Information (`FUND_REPORTED_INFO`)
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `SERIES_NAME` | Fund name | OFST010110 Legal Fund Name Only |
| `SERIES_ID` | SEC series identifier | — |
| `SERIES_LEI` | LEI of the fund series | OFST010030 LEI Of Fund |
| `TOTAL_ASSETS` | Total assets (USD) | OFDY000010-range (AuM/TNA) |
| `TOTAL_LIABILITIES` | Total liabilities | — |
| `NET_ASSETS` | Net assets (TNA) | OFDY000010-range |
| `SALES_FLOW_MON1/2/3` | Monthly inflows | — |
| `REDEMPTION_FLOW_MON1/2/3` | Monthly outflows | — |
| Credit spread sensitivities (3m,1y,5y,10y,30y) | Risk measures | — |
#### B. Interest Rate Risk (`INTEREST_RATE_RISK`)
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `CURRENCY_CODE` | Currency of exposure | OFST010410 Fund Currency |
| `INTRST_RATE_CHANGE_*_DV01` | DV01 by maturity bucket | — |
| `INTRST_RATE_CHANGE_*_DV100` | Impact of 100bp shift | — |
#### C. Monthly Returns (`MONTHLY_TOTAL_RETURN`)
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `CLASS_ID` | Share class identifier | — |
| `MONTHLY_TOTAL_RETURN1/2/3` | Monthly returns per class | OFDY025000-range |
#### D. Portfolio Holdings (`FUND_REPORTED_HOLDING`)
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `ISSUER_NAME` | Holding issuer name | OFPH-range (Portfolio Holdings) |
| `ISSUER_LEI` | LEI of issuer | OFPH-range |
| `ISSUER_TITLE` | Security title/description | OFPH-range |
| `ISSUER_CUSIP` | CUSIP of holding | OFPH-range |
| `BALANCE` | Position size | OFPH-range |
| `UNIT` | Shares/principal/other | OFPH-range |
| `CURRENCY_CODE` | **Currency of holding** | OFPH-range |
| `CURRENCY_VALUE` | Value in reporting currency | OFPH-range |
| `EXCHANGE_RATE` | FX rate applied | — |
| `PERCENTAGE` | % of net assets | OFPH-range |
| `PAYOFF_PROFILE` | Long/Short/N/A | OFPH-range |
| `ASSET_CAT` | **Asset type classification** | OFST350000 MiFID Securities Classification (concept) |
| `ISSUER_TYPE` | Corporate/Government/etc. | — |
| `INVESTMENT_COUNTRY` | **Country of issuer (ISO)** | OFPH-range |
| `IS_RESTRICTED_SECURITY` | Restricted security flag | — |
| `FAIR_VALUE_LEVEL` | Fair value hierarchy (1/2/3) | — |
#### E. Holding Identifiers (`IDENTIFIERS`)
| Field | Description | openfunds Equivalent |
|-------|-------------|---------------------|
| `IDENTIFIER_ISIN` | **ISIN** | OFST020000 ISIN |
| `IDENTIFIER_TICKER` | Ticker | — |
| `OTHER_IDENTIFIER` | SEDOL, etc. | OFST020040 SEDOL |
---
## 3. Mapping: What openfunds Fields CAN Be Found in SEC Data?
### Fully Available (structured, machine-readable)
| openfunds Category | openfunds OF-ID | openfunds Field | SEC Source | SEC Field |
|--------------------|-----------------|-----------------|-----------|-----------|
| **Key Fact: Company** | OFST001000 | Fund Group Name | Series/Class CSV | Entity Name |
| **Key Fact: Umbrella** | OFST005010 | Umbrella Name | Series/Class CSV | Entity Name |
| **Key Fact: Fund** | OFST010030 | LEI Of Fund | N-PORT | SERIES_LEI |
| | OFST010110 | Legal Fund Name Only | Series/Class CSV + N-PORT | Series Name |
| | OFST010300 | Investment Objective | XBRL R/R | ObjectivePrimaryTextBlock |
| | OFST010410 | Fund Currency | N-PORT | CURRENCY_CODE (inferred) |
| **Key Fact: Share Class** | OFST020000 | ISIN | N-PORT Holdings | IDENTIFIER_ISIN |
| | OFST020005 | CUSIP | N-PORT Holdings | ISSUER_CUSIP |
| | OFST020040 | SEDOL | N-PORT Holdings | OTHER_IDENTIFIER |
| | OFST020060 | Full Share Class Name | Series/Class CSV | Class Name |
| **Classification** | OFST350000 | Securities Classification | N-PORT | ASSET_CAT |
| **Fees** | OFST451320 | Max Subscription Fee (Distributor) | XBRL R/R | MaximumSalesChargeImposedOnPurchasesOverOfferingPrice |
| | OFST451391 | CDSC Exit Fee | XBRL R/R | MaximumDeferredSalesChargeOverOfferingPrice |
| | OFST451440 | Max Redemption Fee | XBRL R/R | RedemptionFeeOverRedemption |
| | OFST452010 | Management Fee Maximum | XBRL R/R | ManagementFeesOverAssets |
| | OFST452100 | TER Excl. Performance Fee | XBRL R/R | ExpensesOverAssets |
| | OFST452200 | Ongoing Charges | XBRL R/R | NetExpensesOverAssets |
| | OFST454165 | Distribution Fee Maximum | XBRL R/R | DistributionAndService12b1FeesOverAssets |
| **Performance** | OFDY025xxx | Return periods | XBRL R/R | AverageAnnualReturnYear01/05/10 |
| **Dynamic: AuM** | OFDY000xxx | TNA / AuM | N-PORT | NET_ASSETS, TOTAL_ASSETS |
### Partially Available (derivable from prospectus text, not structured)
These fields exist in the **text of the prospectus** but are NOT in the SEC structured datasets. They would need to be extracted by an LLM — which is exactly the use case:
| openfunds OF-ID | openfunds Field | Where in Prospectus |
|-----------------|-----------------|---------------------|
| OFST010420 | Open-ended Or Closed-ended Fund Structure | Registration form type implies this (N-1A = open-end) |
| OFST010440 | Fiscal Year End | Mentioned in prospectus text, in Submissions JSON |
| OFST010500 | Is Fund Of Funds | Inferred from `AcquiredFundFeesAndExpensesOverAssets > 0` |
| OFST010580 | Is ETF | Inferred from form type or share class structure |
| OFST010720 | Is Passive Fund | Strategy narrative mentions "index" tracking |
| OFST010730 | Management Approach Type | Strategy narrative (active/passive/enhanced) |
| OFST020300 | Valuation Frequency | Prospectus "Pricing of Fund Shares" section |
| OFST020400 | Distribution Policy | Prospectus "Dividends and Distributions" section |
| OFST020540 | Share Class Currency | Inferred; US funds typically USD |
| OFST020558 | Subscription Period Start Date | Only for closed-end or interval funds |
| OFST400xxx | Minimum Investment | Prospectus "Purchase and Sale of Fund Shares" |
| OFST451027 | Has Performance Fee | Prospectus fee table |
| OFST451100 | Hurdle Rate | Prospectus fee table |
| OFST013000 | Prospectus Date | Filing date in submissions API |
### NOT Available in SEC Data
These openfunds fields are **European/international-specific** or **distribution-channel-specific** and have no SEC equivalent:
| openfunds Category | Examples |
|--------------------|---------|
| UCITS/AIFMD fields | OFST160100 Legal Form (SICAV/FCP), OFST011200 Is UCITS With Leveraged Benchmark |
| European regulatory | OFST350100 EFAMA EFC Category, OFST010075 CSSF Code |
| Distribution-specific | OFST453151 Is Trailer Fee Clean, OFST451305 Applied Subscription Fee |
| MiFID/PRIIPs/KID | OFEM-range (MiFID Template), OFEP-range (PRIIPs Template) |
| ESG/Sustainability | OFST820xxx, OFEE-range (EU sustainability regulation specific) |
| Country registrations | OFST6000XX (country-specific registration fields) |
| Solvency II | OFST500xxx |
| Swiss/German/UK specific | OFST700xxx |
---
## 4. Summary: Is Asset Class, Currencies, Fees, Risk Data in the SEC Dataset?
| Data Category | In SEC Structured Data? | Source | Notes |
|--------------|------------------------|--------|-------|
| **Asset Class** | **YES** | N-PORT `ASSET_CAT` field | Values: equity, debt, derivative, etc. |
| **Currencies** | **YES** | N-PORT `CURRENCY_CODE` per holding; interest rate risk by currency | Per-holding currency + fund-level |
| **Fees (sales loads)** | **YES** | XBRL R/R | Front-end load, back-end load, redemption fee |
| **Fees (operating expenses)** | **YES** | XBRL R/R | Management fee, 12b-1, TER, net expense ratio |
| **Risk data (narrative)** | **YES** | XBRL R/R | Principal risks text block |
| **Risk data (quantitative)** | **YES** | N-PORT | DV01, credit spread sensitivity, VaR |
| **Performance** | **YES** | XBRL R/R + N-PORT | Annual returns, avg annual returns, monthly returns |
| **Investment Objective** | **YES** | XBRL R/R | Full text of objective |
| **Strategy** | **YES** | XBRL R/R | Full text of principal strategies |
| **Portfolio Turnover** | **YES** | XBRL R/R | Turnover rate |
| **Portfolio Holdings** | **YES** | N-PORT | Security-level: name, CUSIP, ISIN, country, asset type, value |
| **Country of Issuer** | **YES** | N-PORT `INVESTMENT_COUNTRY` | ISO country code per holding |
| **Minimum Investment** | **PARTIAL** | In prospectus text, not structured | LLM extraction target |
| **Distribution Policy** | **PARTIAL** | In prospectus text, not structured | LLM extraction target |
| **ESG/Sustainability** | **NO** | Not in SEC structured data | European regulation specific |
| **UCITS Classification** | **NO** | N/A for US funds | European regulation specific |
---
## 5. Implication for LLM Training Dataset
The SEC provides an excellent foundation for your LLM training dataset:
### Ground Truth (structured) — available directly from SEC:
- Fee tables (management fee, expense ratio, loads, 12b-1)
- Performance data (1yr, 5yr, 10yr returns)
- Investment objective text
- Principal risks text
- Portfolio turnover rate
- Total net assets
- Fund/class identifiers (CIK, Series ID, Class ID, Ticker, CUSIP)
### Extraction Targets (in prospectus text, to be derived by LLM):
- Minimum initial investment amounts
- Distribution frequency and policy
- Share class currency
- Open/closed-end structure
- Active vs. passive management
- Benchmark index name
- Tax status information
- Purchase/redemption cut-off times
- Settlement cycle
This creates a natural supervised learning setup: the XBRL structured data serves as **labels/ground truth**, and the prospectus HTML/text serves as **input**, enabling the LLM to learn the mapping from legal language to structured reference data.

737
build_rdf_dataset.py Normal file
View File

@ -0,0 +1,737 @@
#!/usr/bin/env python3
"""
SEC Fund Prospectus -> RDF Triple dataset builder (proof of concept).
Produces text->triples samples where:
- INPUT = the prospectus prose for a fund family (long natural-language text),
- OUTPUT = a graph of entity->entity RDF triples (NOT flat literal attributes),
- GOLD = N-CEN structured filings (service providers) + the prospectus structure.
The graph edges are genuine relationships between entities:
Fund seriesOf Trust
Fund advisedBy InvestmentAdviser
Fund subAdvisedBy SubAdviser
Fund transferAgent TransferAgent
Fund custodian Custodian
Fund administrator Administrator
Trust underwrittenBy Distributor
Fund tracksIndex Index (flag-derived; index name from prose)
Gold for advisedBy/transferAgent/custodian/administrator/underwrittenBy comes
directly from N-CEN (no model). The prospectus prose is fetched from EDGAR and
serves as the model input.
Stages:
gold -- build the per-fund gold graph from the local N-CEN flat files
fetch -- download prospectus prose (485BPOS) for the selected N-CEN registrants
samples -- join prose + gold into text->triple training samples
all -- run gold, fetch, samples in order
Usage:
python build_rdf_dataset.py gold --ncen data/ncen/2025q3
python build_rdf_dataset.py fetch --limit 25
python build_rdf_dataset.py samples
python build_rdf_dataset.py all --limit 25
"""
import argparse
import csv
import gzip
import json
import logging
import re
import sys
import time
import urllib.request
from collections import defaultdict
from pathlib import Path
log = logging.getLogger("rdf")
HEADERS = {"User-Agent": "FundDataResearch/1.0 research@university.edu",
"Accept-Encoding": "gzip, deflate"}
EDGAR_RATE = 0.15 # seconds between SEC requests (well under 10/s limit)
_last = 0.0
OUT_DIR = Path("data/rdf_poc")
GOLD_PATH = OUT_DIR / "gold_graphs.jsonl"
PROSE_DIR = OUT_DIR / "prose"
SAMPLES_PATH = OUT_DIR / "samples.jsonl"
# RDF marker tokens from the thesis (Section 5.2)
T_START, T_PRED, T_OBJ, T_END = (
"<triple_start>", "<predicate_marker>", "<object_marker>", "<triple_end>")
# --------------------------------------------------------------------------- #
# helpers
# --------------------------------------------------------------------------- #
def _get(url: str, timeout: int = 120) -> bytes:
global _last
dt = time.time() - _last
if dt < EDGAR_RATE:
time.sleep(EDGAR_RATE - dt)
req = urllib.request.Request(url, headers=HEADERS)
resp = urllib.request.urlopen(req, timeout=timeout)
data = resp.read()
_last = time.time()
if resp.headers.get("Content-Encoding") == "gzip" or data[:2] == b"\x1f\x8b":
data = gzip.decompress(data)
return data
def _read_tsv(path: Path) -> list[dict]:
if not path.exists():
log.warning("missing %s", path)
return []
# csv.field_size_limit because some SEC fields are huge
csv.field_size_limit(min(sys.maxsize, 2**31 - 1))
with open(path, "r", encoding="utf-8", errors="replace", newline="") as f:
return list(csv.DictReader(f, delimiter="\t"))
def _slug(s: str) -> str:
"""Make an IRI-safe local name from an entity string."""
s = re.sub(r"[^A-Za-z0-9]+", "_", (s or "").strip())
return s.strip("_") or "x"
def html_to_text(raw: str) -> str:
raw = re.sub(r"(?is)<script.*?</script>", " ", raw)
raw = re.sub(r"(?is)<style.*?</style>", " ", raw)
txt = re.sub(r"(?s)<[^>]+>", " ", raw)
txt = re.sub(r"&#\d+;", " ", txt)
txt = re.sub(r"&[a-zA-Z]+;", " ", txt)
txt = re.sub(r"[ \t]+", " ", txt)
txt = re.sub(r"\s*\n\s*", "\n", txt)
return re.sub(r"\n{3,}", "\n\n", txt).strip()
# --------------------------------------------------------------------------- #
# stage: gold -- build per-registrant (trust) gold graph from N-CEN
# --------------------------------------------------------------------------- #
def build_gold(ncen_dir: Path, custodian_scope: str = "primary"):
d = Path(ncen_dir)
submission = {r["ACCESSION_NUMBER"]: r for r in _read_tsv(d / "SUBMISSION.tsv")}
registrant = {r["ACCESSION_NUMBER"]: r for r in _read_tsv(d / "REGISTRANT.tsv")}
# FUND_REPORTED_INFO: FUND_ID -> fund row (also gives ACCESSION + flags)
funds = _read_tsv(d / "FUND_REPORTED_INFO.tsv")
fund_by_id = {r["FUND_ID"]: r for r in funds}
funds_by_acc = defaultdict(list)
for r in funds:
funds_by_acc[r["ACCESSION_NUMBER"]].append(r)
# provider tables keyed by FUND_ID
def by_fund(fname, name_col, lei_col):
out = defaultdict(list)
for r in _read_tsv(d / fname):
fid = r.get("FUND_ID", "")
nm = (r.get(name_col, "") or "").strip()
if fid and nm:
out[fid].append({"name": nm, "lei": (r.get(lei_col, "") or "").strip(),
"affiliated": r.get("IS_AFFILIATED", ""),
"type": r.get("ADVISER_TYPE", ""),
"is_sub_custodian": r.get("IS_SUB_CUSTODIAN", "")})
return out
advisers = by_fund("ADVISER.tsv", "ADVISER_NAME", "ADVISER_LEI")
tagents = by_fund("TRANSFER_AGENT.tsv", "TRANSFERAGENT_NAME", "TRANSFERAGENT_LEI")
custodians = by_fund("CUSTODIAN.tsv", "CUSTODIAN_NAME", "CUSTODIAN_LEI")
admins = by_fund("ADMIN.tsv", "ADMIN_NAME", "ADMIN_LEI")
# Custodian scoping. Foreign sub-custodians (IS_SUB_CUSTODIAN=Y, ~88% of rows)
# appear ONLY in N-CEN, never in the prospectus prose, and dominate the edge
# count (66% of all edges). They are therefore unextractable noise for a
# text->triples task. Default keeps the PRIMARY custodian only.
# "primary" -> only IS_SUB_CUSTODIAN != Y (one prose-grounded edge per fund)
# "all" -> every custodian row (legacy behaviour)
# "none" -> drop the custodian relation entirely
if custodian_scope == "primary":
custodians = {fid: [c for c in cs
if (c.get("is_sub_custodian", "") or "").upper() != "Y"]
for fid, cs in custodians.items()}
elif custodian_scope == "none":
custodians = {}
# underwriter (distributor) keyed by ACCESSION (trust level)
underwriters = defaultdict(list)
for r in _read_tsv(d / "PRINCIPAL_UNDERWRITER.tsv"):
acc = r.get("ACCESSION_NUMBER", "")
nm = (r.get("UNDERWRITER_NAME", "") or "").strip()
if acc and nm:
underwriters[acc].append({"name": nm,
"lei": (r.get("UNDERWRITER_LEI", "") or "").strip()})
OUT_DIR.mkdir(parents=True, exist_ok=True)
n_graphs = 0
n_edges = 0
with open(GOLD_PATH, "w", encoding="utf-8") as out:
for acc, reg in registrant.items():
cik = (reg.get("CIK", "") or "").strip().zfill(10)
trust_name = (reg.get("REGISTRANT_NAME", "") or "").strip()
if not trust_name:
continue
trust_iri = "trust:" + _slug(trust_name)
triples = []
entities = {trust_iri: {"type": "Trust", "label": trust_name,
"lei": (reg.get("LEI", "") or "").strip()}}
# trust-level distributor edges
for u in underwriters.get(acc, []):
d_iri = "org:" + _slug(u["name"])
entities[d_iri] = {"type": "Distributor", "label": u["name"], "lei": u["lei"]}
triples.append((trust_iri, "underwrittenBy", d_iri))
for fr in funds_by_acc.get(acc, []):
fid = fr["FUND_ID"]
fname = (fr.get("FUND_NAME", "") or "").strip()
if not fname:
continue
f_iri = "fund:" + _slug(fname)
entities[f_iri] = {"type": "Fund", "label": fname,
"series_id": (fr.get("SERIES_ID", "") or "").strip(),
"lei": (fr.get("LEI", "") or "").strip()}
triples.append((f_iri, "seriesOf", trust_iri))
if (fr.get("IS_INDEX", "") or "").upper() == "Y":
entities[f_iri]["is_index"] = True
if (fr.get("IS_ETF", "") or "").upper() == "Y":
entities[f_iri]["is_etf"] = True
for a in advisers.get(fid, []):
o = "org:" + _slug(a["name"])
is_sub = (a.get("type", "") or "").lower().startswith("sub")
entities[o] = {"type": "SubAdviser" if is_sub else "InvestmentAdviser",
"label": a["name"], "lei": a["lei"]}
triples.append((f_iri, "subAdvisedBy" if is_sub else "advisedBy", o))
for a in tagents.get(fid, []):
o = "org:" + _slug(a["name"])
entities[o] = {"type": "TransferAgent", "label": a["name"], "lei": a["lei"]}
triples.append((f_iri, "transferAgent", o))
for a in custodians.get(fid, []):
o = "org:" + _slug(a["name"])
entities[o] = {"type": "Custodian", "label": a["name"], "lei": a["lei"]}
triples.append((f_iri, "custodian", o))
for a in admins.get(fid, []):
o = "org:" + _slug(a["name"])
entities[o] = {"type": "Administrator", "label": a["name"], "lei": a["lei"]}
triples.append((f_iri, "administrator", o))
# dedupe triples
triples = sorted(set(triples))
if not triples:
continue
rec = {
"accession": acc, "cik": cik,
"trust_name": trust_name, "trust_iri": trust_iri,
"n_funds": len(funds_by_acc.get(acc, [])),
"entities": entities,
"triples": [{"s": s, "p": p, "o": o} for s, p, o in triples],
}
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
n_graphs += 1
n_edges += len(triples)
log.info("gold: %d trust graphs, %d entity->entity edges -> %s",
n_graphs, n_edges, GOLD_PATH)
# quick edge-type histogram
hist = defaultdict(int)
with open(GOLD_PATH, encoding="utf-8") as f:
for line in f:
for t in json.loads(line)["triples"]:
hist[t["p"]] += 1
print("\nEdge-type histogram (entity->entity edges):")
for p, c in sorted(hist.items(), key=lambda x: -x[1]):
print(f" {p:16s} {c:>7,}")
print(f"\nTotal: {n_graphs:,} trust graphs, {n_edges:,} edges")
# --------------------------------------------------------------------------- #
# stage: fetch -- prospectus prose for selected registrants
# --------------------------------------------------------------------------- #
# Full statutory prospectuses (cover ALL funds of a book) vs. short supplements.
# We strongly prefer the full prospectuses; 497/497K supplements are tiny
# amendments and only used as a fallback when no full prospectus is available.
FULL_PROSPECTUS_FORMS = ("485BPOS", "485APOS")
SUPPLEMENT_FORMS = ("497", "497K")
def _prospectus_filings(cik: str, max_filings: int):
"""Return up to max_filings recent prospectus filings for a CIK.
Large fund families split their funds across SEVERAL prospectus books, so the
single most recent 485BPOS covers only a subset of the trust's funds. We
therefore collect the most recent FULL prospectuses (485BPOS/485APOS) first,
newest first, and fall back to 497/497K supplements only if no full
prospectus exists. Returns a list of (accession_nodash, primary_doc).
"""
url = f"https://data.sec.gov/submissions/CIK{cik}.json"
j = json.loads(_get(url).decode("utf-8", "replace"))
recent = j.get("filings", {}).get("recent", {})
forms = recent.get("form", [])
accs = recent.get("accessionNumber", [])
docs = recent.get("primaryDocument", [])
def collect(form_set):
seen, out = set(), []
for i, fm in enumerate(forms): # SEC feed is newest-first
if fm in form_set and docs[i]:
acc = accs[i].replace("-", "")
if acc not in seen:
seen.add(acc)
out.append((acc, docs[i]))
return out
full = collect(FULL_PROSPECTUS_FORMS)
chosen = full[:max_filings]
if not chosen: # fallback: this trust filed only supplements recently
chosen = collect(SUPPLEMENT_FORMS)[:max_filings]
return chosen
def fetch_prose(limit: int, max_filings: int = 8):
"""Fetch ALL recent prospectus filings per trust and concatenate per CIK.
For each selected trust, up to `max_filings` recent prospectus filings are
downloaded and their extracted text concatenated into data/rdf_poc/prose/
<cik>.txt, separated by a form-feed marker. This maximises the chance that
every N-CEN fund of the trust has its prospectus section present, raising
per-fund segmentation coverage.
"""
if not GOLD_PATH.exists():
log.error("run `gold` first"); return
PROSE_DIR.mkdir(parents=True, exist_ok=True)
rows = [json.loads(l) for l in open(GOLD_PATH, encoding="utf-8")]
# prefer trusts with the most edges (richer graphs) for the PoC slice
rows.sort(key=lambda r: -len(r["triples"]))
done = 0
tot_filings = 0
for r in rows:
if done >= limit:
break
cik = r["cik"]
out = PROSE_DIR / f"{cik}.txt"
if out.exists():
done += 1
continue
try:
filings = _prospectus_filings(cik, max_filings)
if not filings:
log.info("no prospectus filing for CIK %s (%s)", cik, r["trust_name"])
continue
parts = []
for acc, doc in filings:
try:
url = f"https://www.sec.gov/Archives/edgar/data/{int(cik)}/{acc}/{doc}"
raw = _get(url).decode("utf-8", "replace")
txt = html_to_text(raw)
if len(txt) >= 2000:
parts.append(txt)
except Exception as e:
log.debug(" filing %s failed: %s", acc, e)
if not parts:
log.info("no usable prospectus text for CIK %s, skipping", cik)
continue
combined = ("\n\n\f\n\n").join(parts) # form-feed separates books
out.write_text(combined, encoding="utf-8")
tot_filings += len(parts)
log.info("[%d/%d] %s %d filings prose=%d chars %s",
done + 1, limit, cik, len(parts), len(combined),
r["trust_name"][:40])
done += 1
except Exception as e:
log.warning("fetch failed for CIK %s: %s", cik, e)
log.info("fetched %d filings across %d trusts -> %s",
tot_filings, done, PROSE_DIR)
# --------------------------------------------------------------------------- #
# stage: samples -- join prose + gold into text->triple samples
# --------------------------------------------------------------------------- #
def serialize_triples(triples, entities) -> str:
"""Render triples in the thesis's <triple_start> marker format, grouped by subject.
Marker form (Models 2/4 with grammar-terminal tokens in the vocabulary).
"""
bys = defaultdict(list)
for t in triples:
bys[t["s"]].append((t["p"], t["o"]))
chunks = []
for s, pos in bys.items():
s_label = entities.get(s, {}).get("label", s)
body = [f"{T_START} {s_label}"]
for p, o in pos:
o_label = entities.get(o, {}).get("label", o)
body.append(f"{T_PRED} {p} {T_OBJ} {o_label}")
body.append(T_END)
chunks.append(" ".join(body))
return "\n".join(chunks)
def serialize_triples_plain(triples, entities) -> str:
"""Render the SAME triples in a plain Turtle-like form with NO special tokens.
For Models 1/3 (decoder-only / encoder-decoder without added grammar-terminal
tokens). Subjects are factored out and predicate-object lists are separated by
';' and ',' exactly as in Turtle, so the two serializations encode identical
content and differ only in whether the delimiters are dedicated tokens.
"""
bys = defaultdict(list)
for t in triples:
bys[t["s"]].append((t["p"], t["o"]))
chunks = []
for s, pos in bys.items():
s_label = entities.get(s, {}).get("label", s)
by_pred = defaultdict(list)
for p, o in pos:
by_pred[p].append(entities.get(o, {}).get("label", o))
preds = []
for p, objs in by_pred.items():
preds.append(f"{p} " + " , ".join(objs))
chunks.append(f"{s_label} " + " ; ".join(preds) + " .")
return "\n".join(chunks)
def ontology_schema(triples, entities) -> dict:
"""Inferred meta-schema (subject type -> predicate -> object type), per thesis 5.3."""
schema = defaultdict(lambda: defaultdict(set))
for t in triples:
st = entities.get(t["s"], {}).get("type", "Thing")
ot = entities.get(t["o"], {}).get("type", "Thing")
schema[st][t["p"]].add(ot)
return {st: {p: sorted(os) for p, os in preds.items()}
for st, preds in schema.items()}
# --------------------------------------------------------------------------- #
# per-fund segmentation
# --------------------------------------------------------------------------- #
def _name_variants(fund_name: str):
"""Generate prospectus-heading variants of an N-CEN fund name.
N-CEN names and prospectus headings often differ in the legal-form suffix
(Fund vs ETF vs Portfolio) and in spacing/punctuation, so we match on a set
of normalized variants rather than the exact string.
"""
base = fund_name.strip()
stems = {base}
# swap the trailing legal-form word
for suf in (" Fund", " ETF", " Portfolio", " Trust"):
if base.endswith(suf):
root = base[: -len(suf)]
for alt in (" Fund", " ETF", " Portfolio", ""):
stems.add(root + alt)
stems.add(root)
break
# also the bare root with no suffix
return {s.strip() for s in stems if len(s.strip()) >= 6}
# A fund section opens with the fund name immediately followed by a summary
# heading. Filers use several styles right after the name, so we accept any of:
# - "Fund Summary" / "Summary" (mutual funds)
# - "Investment Objective" / "Principal Investment Strateg" (objective heading)
# - "The Fund seeks ..." (ETF objective sentence)
# - "Class/Ticker:" or a ticker block / "(formerly ...)" (ETF & multi-class
# summary headers, e.g. JPMorgan, Invesco ETF)
# The MIN_SEGMENT_CHARS guard below is the real safety net: any anchor that
# resolves to a collapsed (too-short) segment is discarded, so a slightly broad
# anchor set cannot reintroduce the spurious-cross-reference problem.
_SECTION_ANCHOR = re.compile(
r"\s{0,5}("
r"Fund Summary|Summary Section|Investment Objective|Principal Investment Strateg"
r"|The Fund seeks|Class/Ticker|Ticker(s)?:|\(formerly"
r")", re.I)
MIN_SEGMENT_CHARS = 1500 # a real fund summary is at least this long
def _heading_positions(text: str, fund_names):
"""Return all (offset, fund_name) anchored heading candidates in the text.
A candidate is a position where a fund name variant is immediately followed
by a strong section anchor. A fund may have several candidates (its book may
be concatenated more than once, or it appears in multiple books); all are
kept so segmentation can choose the one that yields a non-trivial segment.
"""
cands = []
for fn in fund_names:
for v in sorted(_name_variants(fn), key=len, reverse=True):
for m in re.finditer(re.escape(v), text):
if _SECTION_ANCHOR.match(text[m.end():m.end() + 45]):
cands.append((m.start(), fn))
return sorted(cands)
def _segment_prose(text: str, fund_names):
"""Split prose into per-fund segments using anchored section headings.
Algorithm:
1. collect all anchored heading candidates across the (possibly multi-book)
text and sort them by offset;
2. each candidate's segment runs to the NEXT candidate heading (of any
fund), so boundaries come only from real section starts;
3. for each fund, choose the candidate whose segment is longest and at
least MIN_SEGMENT_CHARS, discarding collapsed (too-short) candidates.
Returns {fund_name: segment_text} for funds with a usable section.
"""
cands = _heading_positions(text, fund_names)
if not cands:
return {}
offsets = [c[0] for c in cands]
best = {} # fund -> (length, segment_text)
for i, (off, fn) in enumerate(cands):
end = offsets[i + 1] if i + 1 < len(offsets) else len(text)
seg = text[off:end]
if len(seg) < MIN_SEGMENT_CHARS:
continue
if fn not in best or len(seg) > best[fn][0]:
best[fn] = (len(seg), seg)
return {fn: seg for fn, (ln, seg) in best.items()}
def build_samples(per_fund: bool = True):
if not GOLD_PATH.exists():
log.error("run `gold` first"); return
if per_fund:
return _build_samples_per_fund()
gold = {json.loads(l)["cik"]: json.loads(l)
for l in open(GOLD_PATH, encoding="utf-8")}
n = 0
tot_text = tot_json = 0
with open(SAMPLES_PATH, "w", encoding="utf-8") as out:
for prose_file in sorted(PROSE_DIR.glob("*.txt")):
cik = prose_file.stem
g = gold.get(cik)
if not g:
continue
text = prose_file.read_text(encoding="utf-8")
triples, ents = g["triples"], g["entities"]
target = serialize_triples(triples, ents)
schema = ontology_schema(triples, ents)
sample = {
"cik": cik,
"trust_name": g["trust_name"],
"input_text": text,
"ontology": schema,
"target_triples": triples,
"target_serialized": target,
"stats": {
"input_chars": len(text),
"target_chars": len(target),
"n_triples": len(triples),
"n_entities": len(ents),
"text_to_json_ratio": round(len(text) / max(1, len(target)), 1),
},
}
out.write(json.dumps(sample, ensure_ascii=False) + "\n")
n += 1
tot_text += len(text)
tot_json += len(target)
log.info("wrote %d samples -> %s", n, SAMPLES_PATH)
if n:
print(f"\n{n} samples")
print(f" mean input : {tot_text//n:>8,} chars")
print(f" mean target : {tot_json//n:>8,} chars")
print(f" mean ratio : {tot_text/max(1,tot_json):>8.1f} : 1 (text : json)")
def _build_samples_per_fund():
"""One sample per fund: the fund's prospectus section -> the fund's subgraph.
For each trust, the prose is segmented into per-fund sections. Each fund's
target is its own edges (advisedBy, custodian, ...) plus the fund-anchored
seriesOf edge and the trust-level underwrittenBy edge (shared, but a true
fact about that fund's trust). Funds whose section cannot be located in the
prose are skipped and counted; trusts where nothing can be located fall back
to a single whole-trust sample so no data is silently dropped.
"""
gold = {json.loads(l)["cik"]: json.loads(l)
for l in open(GOLD_PATH, encoding="utf-8")}
n = 0
n_funds_total = n_funds_located = n_fallback_trusts = 0
tot_text = tot_json = 0
ratios = []
with open(SAMPLES_PATH, "w", encoding="utf-8") as out:
for prose_file in sorted(PROSE_DIR.glob("*.txt")):
cik = prose_file.stem
g = gold.get(cik)
if not g:
continue
text = prose_file.read_text(encoding="utf-8")
ents = g["entities"]
# group triples by subject fund IRI; collect trust-level edges
fund_iris = {e_iri for e_iri, e in ents.items() if e["type"] == "Fund"}
trust_iri = g["trust_iri"]
by_fund = defaultdict(list)
trust_edges = []
for t in g["triples"]:
if t["s"] in fund_iris:
by_fund[t["s"]].append(t)
elif t["s"] == trust_iri:
trust_edges.append(t) # e.g. underwrittenBy
fund_label = {iri: ents[iri]["label"] for iri in fund_iris}
n_funds_total += len(fund_iris)
segs = _segment_prose(text, list(fund_label.values()))
label_to_iri = {v: k for k, v in fund_label.items()}
if not segs: # whole-trust fallback (no section located)
n_fallback_trusts += 1
triples = g["triples"]
target = serialize_triples(triples, ents)
target_plain = serialize_triples_plain(triples, ents)
rec = {
"sample_id": f"{cik}:ALL", "cik": cik, "trust_name": g["trust_name"],
"fund": None, "segmented": False,
"input_text": text, "ontology": ontology_schema(triples, ents),
"target_triples": triples,
"target_serialized": target,
"target_serialized_plain": target_plain,
"stats": {"input_chars": len(text), "target_chars": len(target),
"n_triples": len(triples), "n_entities": len(ents),
"text_to_json_ratio": round(len(text) / max(1, len(target)), 1)},
}
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
n += 1; tot_text += len(text); tot_json += len(target)
ratios.append(rec["stats"]["text_to_json_ratio"])
continue
for label, seg in segs.items():
f_iri = label_to_iri.get(label)
if not f_iri:
continue
triples = list(by_fund.get(f_iri, [])) + list(trust_edges)
if not triples:
continue
n_funds_located += 1
# restrict entities to those referenced by this fund's triples
ref = {trust_iri}
for t in triples:
ref.add(t["s"]); ref.add(t["o"])
sub_ents = {k: ents[k] for k in ref if k in ents}
target = serialize_triples(triples, sub_ents)
target_plain = serialize_triples_plain(triples, sub_ents)
rec = {
"sample_id": f"{cik}:{ents[f_iri].get('series_id') or _slug(label)}",
"cik": cik, "trust_name": g["trust_name"],
"fund": label, "series_id": ents[f_iri].get("series_id", ""),
"segmented": True,
"input_text": seg, "ontology": ontology_schema(triples, sub_ents),
"target_triples": triples,
"target_serialized": target,
"target_serialized_plain": target_plain,
"stats": {"input_chars": len(seg), "target_chars": len(target),
"n_triples": len(triples), "n_entities": len(sub_ents),
"text_to_json_ratio": round(len(seg) / max(1, len(target)), 1)},
}
out.write(json.dumps(rec, ensure_ascii=False) + "\n")
n += 1; tot_text += len(seg); tot_json += len(target)
ratios.append(rec["stats"]["text_to_json_ratio"])
import statistics as _st
log.info("wrote %d per-fund samples -> %s", n, SAMPLES_PATH)
if n:
cov = n_funds_located / max(1, n_funds_total)
print(f"\n{n} samples ({n_funds_located}/{n_funds_total} funds located, "
f"coverage {cov:.0%}; {n_fallback_trusts} trusts fell back to whole-doc)")
print(f" mean input : {tot_text//n:>8,} chars")
print(f" mean target : {tot_json//n:>8,} chars")
print(f" median ratio: {_st.median(ratios):>8.1f} : 1 (text : json)")
print(f" mean ratio : {tot_text/max(1,tot_json):>8.1f} : 1")
# --------------------------------------------------------------------------- #
# stage: split -- trust-level train/val/test split
# --------------------------------------------------------------------------- #
def _bucket(cik: str) -> float:
"""Deterministic [0,1) value per trust CIK (stable, no RNG state)."""
import hashlib
h = hashlib.sha256(cik.encode()).hexdigest()
return int(h[:8], 16) / 0xFFFFFFFF
def build_split(val_frac: float = 0.10, test_frac: float = 0.10):
"""Split samples.jsonl into train/val/test JSONL, partitioned by TRUST (CIK).
Splitting by trust (not by fund) prevents leakage: two funds of the same
trust share advisers, distributors and custodians, so allowing them into
different splits would let the model memorise trust-specific entities. The
assignment is a deterministic hash of the CIK, so the split is reproducible
and stable as new samples are added.
"""
if not SAMPLES_PATH.exists():
log.error("run `samples` first"); return
rows = [json.loads(l) for l in open(SAMPLES_PATH, encoding="utf-8")]
out = {"train": [], "val": [], "test": []}
for r in rows:
b = _bucket(r["cik"])
split = "test" if b < test_frac else \
"val" if b < test_frac + val_frac else "train"
out[split].append(r)
for name, recs in out.items():
p = OUT_DIR / f"{name}.jsonl"
with open(p, "w", encoding="utf-8") as f:
for r in recs:
f.write(json.dumps(r, ensure_ascii=False) + "\n")
n_tr = {s: len({r["cik"] for r in recs}) for s, recs in out.items()}
print("\nTrust-level split (deterministic by CIK):")
for s in ("train", "val", "test"):
print(f" {s:5s}: {len(out[s]):>5,} samples from {n_tr[s]:>4} trusts")
total = sum(len(v) for v in out.values())
print(f" total: {total:,} samples -> {OUT_DIR}/{{train,val,test}}.jsonl")
# leakage check
cik_sets = {s: {r["cik"] for r in recs} for s, recs in out.items()}
overlap = (cik_sets["train"] & cik_sets["val"]) | \
(cik_sets["train"] & cik_sets["test"]) | \
(cik_sets["val"] & cik_sets["test"])
print(f" trust overlap across splits: {len(overlap)} (should be 0)")
# --------------------------------------------------------------------------- #
def main():
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
ap = argparse.ArgumentParser()
sub = ap.add_subparsers(dest="cmd")
g = sub.add_parser("gold"); g.add_argument("--ncen", default="data/ncen/2025q3")
g.add_argument("--custodian-scope", choices=["primary", "all", "none"],
default="primary",
help="primary=only IS_SUB_CUSTODIAN!=Y (default); all=every row; none=drop")
f = sub.add_parser("fetch"); f.add_argument("--limit", type=int, default=25)
f.add_argument("--max-filings", type=int, default=8,
help="max prospectus filings to fetch & concatenate per trust")
s = sub.add_parser("samples")
s.add_argument("--whole-trust", action="store_true",
help="one sample per trust instead of per-fund segmentation")
sp = sub.add_parser("split")
sp.add_argument("--val-frac", type=float, default=0.10)
sp.add_argument("--test-frac", type=float, default=0.10)
a = sub.add_parser("all"); a.add_argument("--ncen", default="data/ncen/2025q3")
a.add_argument("--limit", type=int, default=25)
a.add_argument("--max-filings", type=int, default=8)
a.add_argument("--custodian-scope", choices=["primary", "all", "none"], default="primary")
a.add_argument("--whole-trust", action="store_true")
a.add_argument("--val-frac", type=float, default=0.10)
a.add_argument("--test-frac", type=float, default=0.10)
args = ap.parse_args()
if args.cmd == "gold":
build_gold(Path(args.ncen), custodian_scope=args.custodian_scope)
elif args.cmd == "fetch":
fetch_prose(args.limit, max_filings=args.max_filings)
elif args.cmd == "samples":
build_samples(per_fund=not args.whole_trust)
elif args.cmd == "split":
build_split(val_frac=args.val_frac, test_frac=args.test_frac)
elif args.cmd == "all":
build_gold(Path(args.ncen), custodian_scope=args.custodian_scope)
fetch_prose(args.limit, max_filings=args.max_filings)
build_samples(per_fund=not args.whole_trust)
build_split(val_frac=args.val_frac, test_frac=args.test_frac)
else:
ap.print_help()
if __name__ == "__main__":
main()

75
data/RDF_DATASET_DESIGN.md Normal file
View File

@ -0,0 +1,75 @@
# SEC Fund Prospectus → RDF Triple Dataset — Design
## Goal
A training/eval dataset for text→RDF-triple extraction where:
- the INPUT text is much larger than the OUTPUT JSON (realistic "needle in long doc"),
- the OUTPUT is a genuine **graph of entity→entity relationships**, not flat key→literal attributes,
- it is finance **reference data** (openfunds-aligned),
- there is a **non-model gold baseline** AND a path to a **strong-model baseline**.
## Why not flat XBRL fee data
XBRL Risk/Return (management fee, TER, returns) yields only `Fund --pred--> literal`
attributes — a star of literals, no entity-to-entity edges. Rejected as the *primary*
graph source because it produces no relational structure. (May be added as optional
literal-valued triples to enrich the schema, but it is not the point.)
## Evidence (sample: Fidelity Oxford Street Trust, 485BPOS 0000028540-25-000048)
Full prospectus prose = 1,061,532 chars. Relational mentions found in prose:
adviser 381, sub-adviser 25, distributor 31, custodian 59, transfer agent 24,
"Trust" 805, "series of" 13, "managed by" 38, named benchmark index.
=> The prose contains a real multi-entity-type graph.
## Target ontology (entity→entity edges)
```
Fund seriesOf Trust
Fund advisedBy InvestmentAdviser
Fund subAdvisedBy SubAdviser
Fund distributedBy Distributor
Fund custodian Custodian
Fund transferAgent TransferAgent
Fund managedBy PortfolioManager
Fund tracksIndex Index (passive/index funds)
```
Optional literal-valued triples (attributes) to round out the openfunds record:
managementFee, netExpenseRatio, return1yr/5yr/10yr, portfolioTurnover, objectiveText.
## Baselines (the key selling point)
1. GOLD, no model:
- N-CEN service-provider table -> advisedBy, custodian, transferAgent, distributor, auditor (with LEIs)
- Series/Class CSV -> seriesOf, hasShareClass (Trust->Series->Class skeleton)
2. SILVER, strong model (GPT-4 / Opus), measured against the N-CEN gold:
- managedBy (portfolio managers), subAdvisedBy, tracksIndex (named only in prose)
## Unit of extraction
One FUND (series) = one sample:
input = the prospectus section(s) for that fund (kept within a long context)
output = the fund's subgraph of triples
N-CEN/Series-Class gold is keyed at trust/series level, so per-series is the natural join.
## Data gaps to fix (local dump)
- fund_data.db was accidentally deleted (only stale -shm/-wal remain) -> rebuild from data/.
- load_xbrl_rr.py tag constants are partly stale vs the actual num.tsv vintage
(e.g. ExpensesOverAssets, DistributionAndService12b1FeesOverAssets,
AvgAnnlRtrPct/AnnlRtrPct, PortfolioTurnoverRate, BarChart*QuarterlyReturn).
- Local N-PORT dump is MISSING FUND_REPORTED_HOLDING.tsv and IDENTIFIERS.tsv
(the holds/issuedBy edges) -> would need re-download if holdings edges are wanted.
- XBRL flat files contain NO text blocks -> narrative input must be fetched from EDGAR
filings (filing_text), not from the flat files.
- N-CEN flat files are NOT yet downloaded locally -> needed for the gold edges.
## Holdings edges (deferred to a 2nd track)
Holdings (`fund holds security`, `security issuedBy issuer`) are NOT in the prospectus.
Text-bearing sources for holdings:
- Annual/Semi-Annual Report (N-CSR): "Management Discussion" (MDFP) names TOP holdings
in prose -> real `holds` edges; gold = N-PORT holdings table. The full Schedule of
Investments is a table, not prose.
- Fund fact sheets / PM commentary (fund-company marketing): "Top 10 Holdings" + prose,
but off-EDGAR and no standardized machine-readable gold.
Plan: build prospectus->service-provider graph FIRST (clean N-CEN gold). Add an
MDFP(N-CSR)->holdings dataset as a SECOND finance sub-domain later — this also
strengthens the thesis's cross-domain generalization claim.
## This session: PROOF OF CONCEPT (~20-50 funds)
End-to-end on a small slice: fetch prospectus text, get N-CEN gold service-provider
edges + Series/Class structure, emit text->triples samples in the <triple_start>
RDF marker format, and a gold-baseline scorer. Validate before scaling.

435
data/rdf_poc/gold_graphs.jsonl Normal file
View File

File diff suppressed because one or more lines are too long

BIN
dataset_description.pdf Normal file
View File

Binary file not shown.

582
dataset_description.tex Normal file
View File

@ -0,0 +1,582 @@
\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{lmodern}
\usepackage[margin=2.5cm]{geometry}
\usepackage{amsmath,amssymb}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{xcolor}
\usepackage{tikz}
\usetikzlibrary{positioning,arrows.meta,shapes.geometric}
\usepackage[hidelinks]{hyperref}
\usepackage{caption}
\usepackage{enumitem}
% ---- listing style (matches the Turtle/JSON look of the thesis) ----
\definecolor{kw}{rgb}{0.0,0.3,0.6}
\definecolor{str}{rgb}{0.6,0.2,0.1}
\definecolor{cmt}{rgb}{0.3,0.5,0.3}
\lstset{
basicstyle=\ttfamily\footnotesize,
breaklines=true,
showstringspaces=false,
keywordstyle=\color{kw}\bfseries,
stringstyle=\color{str},
commentstyle=\color{cmt}\itshape,
frame=single,
framesep=4pt,
rulecolor=\color{black!25},
backgroundcolor=\color{black!2},
numbers=left,
numberstyle=\tiny\color{black!40},
xleftmargin=2.2em,
}
\newcommand{\code}[1]{\texttt{\small #1}}
\newcommand{\tstart}{\code{<triple\_start>}}
\newcommand{\predm}{\code{<predicate\_marker>}}
\newcommand{\objm}{\code{<object\_marker>}}
\newcommand{\tend}{\code{<triple\_end>}}
\title{\textbf{A Relationship-Rich Financial Dataset for\\
Text-to-RDF-Triple Extraction:\\
SEC Fund Disclosures as a Knowledge-Graph Source}}
\author{Dr.\ Florian Herzog\\\small Thesis supervisor --- companion technical note to the thesis
\emph{Magical RDF Triples and how to synthetize them}}
\date{\today}
\begin{document}
\maketitle
\begin{abstract}
This note specifies a finance-domain dataset for training and evaluating models
that extract Resource Description Framework (RDF) triples from plain text, the
task at the centre of the accompanying thesis. The dataset is constructed from
mandatory U.S.\ Securities and Exchange Commission (SEC) fund disclosures. Unlike
the Wikidata-derived corpora commonly used for this task --- where the source text
is written \emph{from} the triples and is therefore roughly the same size as its
target --- here a long natural-language prospectus (on the order of
$10^{5}$--$10^{7}$ characters) maps to a compact graph of a few hundred triples,
yielding a realistic text-to-output size ratio of roughly $20\!:\!1$. Crucially,
the target is a genuine graph of \emph{entity-to-entity relationships} (a fund
\emph{advised by} a management company, \emph{distributed by} an underwriter,
\emph{holding} a security \emph{issued by} an issuer), not a flat list of literal
attributes. Two distinct ground-truth regimes are available: a \emph{model-free
gold} baseline derived from parallel structured SEC filings (N-CEN, N-PORT,
Series/Class reference data), and a \emph{strong-model silver} baseline for the
relations expressed only in prose. We describe the source filings, the target
ontology and graph structure, the holdings sub-graph, the serialization into the
thesis's grammar-terminal token format, and how the resulting samples are used to
train and benchmark the four models under study.
\end{abstract}
\tableofcontents
% ====================================================================
\section{Motivation: the size-ratio and relationship gap}
% ====================================================================
The thesis trains a general-purpose language model to extract serialized RDF
triples from plain text conditioned on an ontology. The quality of such a model
is bounded by the quality of its training data. The benchmarks surveyed in the
thesis (WebNLG, T-REx, REBEL, Wiki-NRE) share two properties that make them weak
proxies for the real extraction problem.
\paragraph{Symmetric size.} In WebNLG, human annotators were instructed to write
text \emph{from} a given set of triples. Consequently each sentence encodes
almost exactly the triples it was generated from: the input text and the target
JSON are of comparable length. The task degenerates towards transliteration and
never exercises the central difficulty of practical information extraction ---
locating a small number of facts inside a large, noisy document.
\paragraph{Attribute-only targets.} Many relation-extraction corpora reduce to
mapping a sentence to a single predicate label, or to a star of literal-valued
attributes around one entity. They contain few \emph{entity-to-entity} edges, and
therefore exercise little of the graph structure that motivates RDF in the first
place.
A suitable dataset must instead satisfy both of the following, simultaneously:
\begin{enumerate}[label=(\roman*)]
\item the input text is substantially larger than the target serialization, so
the model must perform genuine reading comprehension over a long document;
\item the target is a multi-entity-type graph of relationships, so the inferred
ontology contains edges of the form
\code{TypeA\ ---predicate--->\ TypeB}, not only
\code{TypeA\ ---predicate--->\ literal}.
\end{enumerate}
SEC fund disclosures satisfy both, and additionally provide a rare third
property: a \emph{free, non-model ground truth}, because the same facts that
appear in the prose are independently filed by the same registrants in structured
form.
% ====================================================================
\section{Source filings}
% ====================================================================
The dataset draws on four public SEC data sources, summarised in
Table~\ref{tab:sources}. Their division of labour is the key design idea: the
\emph{prose} filings provide the model input, while the \emph{structured} filings
provide the ground-truth graph.
\begin{table}[h]
\centering
\caption{SEC source filings and their role in the dataset.}
\label{tab:sources}
\small
\begin{tabular}{@{}llll@{}}
\toprule
Source & Form & Content & Role \\
\midrule
Prospectus & N-1A (485BPOS, 497) & Investment objective, strategy, & \textbf{input text} \\
& & management, fees (prose) & \\
N-CEN & N-CEN & Service providers, classification & \textbf{gold edges} \\
N-PORT & NPORT-P & Portfolio holdings (quarterly) & \textbf{gold edges} \\
Series/Class CSV & --- & Trust/Series/Class identity & \textbf{gold skeleton} \\
Annual report (MDFP) & N-CSR & Top-holdings commentary (prose) & \textbf{input text (holdings)} \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Prospectus (N-1A).} The statutory prospectus is a long legal document
describing a fund family. It names, in prose, the fund's investment adviser,
sub-adviser, distributor, transfer agent, portfolio managers and benchmark index,
together with its objective, strategy and fee structure. A single filing covers
all funds (series) of a trust and ranges from roughly $4\times10^{5}$ to
$1\times10^{7}$ characters of extracted text.
\paragraph{N-CEN.} The annual census filing reports, in structured tabular form,
each fund's service providers --- adviser, sub-adviser, custodian, transfer
agent, administrator --- and the trust's principal underwriter, each with a Legal
Entity Identifier (LEI) where available. These rows are the gold standard for the
service-provider edges of the graph.
\paragraph{N-PORT.} The monthly portfolio filing reports, per fund, every
security held, with issuer name, identifiers (CUSIP, ISIN, LEI), asset category,
investment country and market value. These rows are the gold standard for the
holdings sub-graph (Section~\ref{sec:holdings}).
\paragraph{Series/Class reference data.} The SEC's Series/Class listing provides
the trust\,$\to$\,series\,$\to$\,share-class identity backbone, gold for the
structural \code{seriesOf} and \code{hasShareClass} edges.
A central property is \emph{redundancy across modality}: a fact such as ``the
fund is advised by Geode Capital Management, LLC'' appears both as a sentence in
the prospectus (the input) and as a structured row in N-CEN (the label). This is
what makes a model-free ground truth possible.
% ====================================================================
\section{Target ontology and graph structure}
% ====================================================================
The target of each sample is a directed, labelled multigraph $G=(E,R)$ in the
sense of the thesis, where nodes are typed entities and edges are RDF predicates.
The entity types and relations are listed in Table~\ref{tab:ontology}.
\begin{table}[h]
\centering
\caption{Target ontology: entity types and entity-to-entity relations.}
\label{tab:ontology}
\small
\begin{tabular}{@{}lll@{}}
\toprule
Subject type & Predicate & Object type \\
\midrule
Fund & \code{seriesOf} & Trust \\
Fund & \code{advisedBy} & InvestmentAdviser \\
Fund & \code{subAdvisedBy} & SubAdviser \\
Fund & \code{transferAgent} & TransferAgent \\
Fund & \code{custodian} & Custodian \\
Fund & \code{administrator} & Administrator \\
Trust & \code{underwrittenBy} & Distributor \\
\addlinespace
Fund & \code{holds} & Security \quad(holdings sub-graph) \\
Security & \code{issuedBy} & Issuer \\
Security & \code{domiciledIn} & Country \\
Fund & \code{tracksIndex} & Index \\
\bottomrule
\end{tabular}
\end{table}
Every relation in Table~\ref{tab:ontology} has an entity as its object, not a
literal. The dataset may optionally be enriched with literal-valued attribute
triples (management fee, net expense ratio, returns, portfolio turnover) drawn
from the XBRL Risk/Return filings; these are deliberately \emph{secondary},
because the purpose of the dataset is to exercise relational structure.
Following the thesis's ontology-inference procedure
(SPARQL meta-schema extraction), the per-sample ontology presented to the model
is the set of distinct \code{(subject type, predicate, object type)} patterns
realised in that sample, e.g.
\noindent\begin{minipage}{\linewidth}
\begin{lstlisting}[language=,caption={Inferred ontology for one fund trust (model input, abbreviated).},captionpos=b]
{
"Fund": {
"seriesOf": ["Trust"],
"advisedBy": ["InvestmentAdviser"],
"subAdvisedBy": ["SubAdviser"],
"transferAgent": ["TransferAgent"],
"custodian": ["Custodian"],
"administrator": ["Administrator"]
},
"Trust": { "underwrittenBy": ["Distributor"] }
}
\end{lstlisting}
\end{minipage}
\begin{figure}[h]
\centering
\begin{tikzpicture}[
node distance=13mm and 30mm,
ent/.style={draw,rounded corners,align=center,font=\footnotesize,
inner sep=3pt,minimum height=7mm,fill=blue!5},
edge/.style={-{Stealth},font=\scriptsize,shorten >=1pt,shorten <=1pt},
lbl/.style={font=\scriptsize,fill=white,inner sep=1.5pt}]
% --- service-provider / structure cluster (centre-left) ---
\node[ent] (fund) {Fund};
\node[ent,above=of fund] (trust) {Trust};
\node[ent,above=of trust] (dist) {Distributor};
\node[ent,left=of fund] (adv) {Investment\\Adviser};
\node[ent,below=16mm of adv] (sub) {Sub-\\Adviser};
\node[ent,below=of fund] (cust) {Custodian};
\node[ent,right=24mm of cust] (ta) {Transfer\\Agent};
\node[ent,right=of fund] (admin) {Administrator};
% --- holdings cluster (far right, separated column) ---
\node[ent,right=26mm of admin] (sec) {Security};
\node[ent,above=of sec] (iss) {Issuer};
\node[ent,below=of sec] (ctry) {Country};
\draw[edge] (fund) -- node[lbl]{seriesOf} (trust);
\draw[edge] (trust) -- node[lbl]{underwrittenBy} (dist);
\draw[edge] (fund) -- node[lbl]{advisedBy} (adv);
\draw[edge] (fund) -- node[lbl,pos=0.55]{subAdvisedBy} (sub);
\draw[edge] (fund) -- node[lbl]{custodian} (cust);
\draw[edge] (fund) -- node[lbl,pos=0.55]{transferAgent} (ta);
\draw[edge] (fund) -- node[lbl]{administrator} (admin);
% holds: arc from the Fund's top, over the Administrator, down to Security
\draw[edge] (fund.north east) to[out=35,in=140]
node[lbl,pos=0.65]{holds} (sec.north west);
\draw[edge] (sec) -- node[lbl]{issuedBy} (iss);
\draw[edge] (sec) -- node[lbl]{domiciledIn} (ctry);
\end{tikzpicture}
\caption{Schematic of the target knowledge graph. Left and centre: the
service-provider/structure graph grounded in the prospectus prose. Right column
(Issuer--Security--Country): the holdings sub-graph grounded in annual-report
commentary with N-PORT gold.}
\label{fig:graph}
\end{figure}
% ====================================================================
\section{The holdings sub-graph}
\label{sec:holdings}
% ====================================================================
Portfolio holdings express the richest relationships in the data --- a fund
\emph{holds} many securities, each \emph{issued by} an issuer
\emph{domiciled in} a country --- and are the natural place to grow the graph
beyond service providers. They require care, however, because holdings are
\emph{not} disclosed in the prospectus: the prospectus describes a fund's
\emph{strategy} (``invests in large-capitalisation equities''), never its
specific positions.
The text-bearing source for holdings is the \textbf{annual or semi-annual report}
(Form N-CSR). It contains two parts:
\begin{itemize}[nosep]
\item the \emph{Schedule of Investments}, a complete table of every holding ---
structured, not prose, and therefore not an extraction target; and
\item the \emph{Management Discussion of Fund Performance} (MDFP), a narrative
in which the portfolio manager names the fund's \emph{top} positions and
explains their contribution (``our largest holdings were Apple,
Microsoft and \dots'').
\end{itemize}
The MDFP is genuine prose and yields real \code{holds} edges for the named
positions. The corresponding \textbf{N-PORT} filing provides the structured gold:
the full holdings table, against which the MDFP-named subset can be verified and
from which \code{issuedBy} and \code{domiciledIn} are taken.
This produces a second, independent text-to-graph task in the same financial
domain: \emph{MDFP commentary $\to$ holdings sub-graph}, with N-PORT as gold.
Because it pairs a \emph{different} document type with a \emph{different} relation
set, including it strengthens the cross-domain generalization claim of the thesis
(Section~3.2.3): a single model is shown to extract two structurally different
graphs in the same domain. Fund fact sheets and portfolio-manager commentaries
published by fund companies are an additional, off-EDGAR prose source for the same
edges, at the cost of having no standardized machine-readable gold.
A practical caveat applies to holdings as it does to service providers
(Section~\ref{sec:baselines}): only the positions \emph{named in prose} are
recoverable from the input. The benchmark therefore scopes the \code{holds}
target to the MDFP-named subset rather than the full N-PORT schedule, to avoid
penalising a model for failing to extract facts that are absent from its input.
% ====================================================================
\section{Serialization and the marker token format}
% ====================================================================
Targets are serialized in the grammar-terminal token format introduced in the
thesis (Section~5.2), in which four special tokens delimit triple components and
shared subjects/predicates are factored out, mirroring Turtle's predicate-object
lists:
\begin{lstlisting}[language=,caption={Target serialization for one segmented fund (primary-custodian scope).},captionpos=b]
<triple_start> Small Cap Special Values Fund
<predicate_marker> seriesOf
<object_marker> VALIC Co I
<predicate_marker> advisedBy
<object_marker> The Variable Annuity Life Insurance Company
<predicate_marker> subAdvisedBy
<object_marker> SunAmerica Asset Management, LLC
<predicate_marker> administrator
<object_marker> SunAmerica Asset Management, LLC
<predicate_marker> custodian
<object_marker> State Street Bank and Trust Company
<triple_end>
\end{lstlisting}
To support the four-model comparison from a single dataset, each sample carries
\emph{two} serializations of the identical triples: \code{target\_serialized},
the marker form above (for Models 2/4, whose vocabulary is extended with the four
grammar-terminal tokens), and \code{target\_serialized\_plain}, a Turtle-like form
using ordinary `\code{;}' and `\code{,}' delimiters and no special tokens (for
Models 1/3). Because the two differ only in whether the delimiters are dedicated
tokens, the comparison isolates exactly the effect under study in research
question~1.
Each sample is thus a JSON record with: the input prose (\code{input\_text}), the
inferred ontology (\code{ontology}), the target triples as a structured list
(\code{target\_triples}) and in both serializations, the trust/series identifiers,
and size statistics.
% ====================================================================
\section{Per-fund segmentation}
\label{sec:segmentation}
% ====================================================================
A single prospectus filing covers \emph{all} funds of a trust and may exceed
$10^{7}$ characters, beyond any practical context window. Treating one filing as
one sample is also semantically wrong: the target would mix the subgraphs of
dozens of unrelated funds. The dataset therefore segments each filing into
\emph{per-fund} samples, so that one fund's prospectus section maps to that one
fund's subgraph.
\paragraph{Fetching all books of a trust.} Large fund families split their funds
across \emph{several} prospectus books, so the single most recent filing covers
only a fraction of a trust's funds. The fetcher therefore retrieves the most
recent \emph{full} prospectuses (forms 485BPOS/485APOS) for each trust ---
preferring them over the much shorter 497/497K supplements, which are used only as
a fallback --- and concatenates their text. On the proof-of-concept slice this
raised the fetched text from one book per trust to a mean of seven, e.g.\ from
$5\times10^{5}$ to $2.2\times10^{7}$ characters for a large ETF trust, so that far
more fund sections are present.
\paragraph{Section anchors.} Statutory prospectuses open each fund's block with
the fund name immediately followed by a summary heading. Filers use several
styles, so the segmenter accepts any of: ``Fund Summary'', ``Investment
Objective'', ``Principal Investment Strategies'', the ETF objective sentence
``The Fund seeks\dots'', or a class/ticker header (``Class/Ticker:\dots'').
\paragraph{Boundary selection and a collapse guard.} The segmenter collects
\emph{all} anchored heading positions across the concatenated text, sorts them,
and cuts each segment from one heading to the next. Because a fund name can also
occur in tables of contents and cross-references, na\"ively taking the first
occurrence collapses segments to a few characters; the segmenter therefore
discards any candidate whose resulting segment is shorter than a minimum
($1{,}500$ characters) and, for each fund, keeps the longest surviving segment.
Each segment is paired with that fund's edges plus the fund-anchored
\code{seriesOf} edge and the trust-level \code{underwrittenBy} edge.
\paragraph{Name-variant matching.} The fund name filed in N-CEN and the heading
printed in the prospectus frequently differ in the legal-form suffix --- a fund
filed as ``\dots\ Fund'' may be headed ``\dots\ ETF'' or ``\dots\ Portfolio''.
The segmenter matches on a set of normalized variants (suffix swapped or dropped)
rather than the exact N-CEN string.
\paragraph{Coverage and fallback.} Where a fund's section cannot be located it is
skipped and counted (never silently dropped); where no section in a trust can be
located, the builder emits a single whole-trust fallback sample. On the
proof-of-concept slice, fetching all books and applying the robust segmenter
turns $14$ trusts into $141$ samples ($135$ cleanly segmented per-fund plus $6$
whole-trust fallbacks), with a per-fund median input of $\sim\!3.7\times10^{4}$
characters against a $\sim\!6.5\times10^{2}$-character target --- a per-fund
text-to-JSON ratio with a median near $55\!:\!1$. The residual misses are
dominated not by segmentation but by an \emph{entity-resolution} gap: some trusts
file their prospectuses under a different CIK (or fund brand) than their N-CEN
report, so the N-CEN fund names do not appear in the fetched text at all. Closing
that gap requires joining on the SEC Series identifier across CIKs rather than
fetching more filings of the same CIK, and is left to the full-scale build.
% ====================================================================
\section{Ground truth and baselines}
\label{sec:baselines}
% ====================================================================
The dataset offers two independent ground-truth regimes.
\paragraph{Model-free gold.} For \code{advisedBy}, \code{subAdvisedBy},
\code{transferAgent}, \code{custodian}, \code{administrator} and
\code{underwrittenBy}, the labels come directly from N-CEN; for \code{seriesOf}
and \code{hasShareClass}, from the Series/Class reference data; for \code{holds},
\code{issuedBy} and \code{domiciledIn}, from N-PORT. No model is involved in
producing these labels, which makes them an unusually trustworthy reference for a
generative-extraction benchmark.
\paragraph{The custodian relation and edge scoping.} The custodian relation
illustrates a subtlety that any honest benchmark on this data must address. N-CEN
reports, for a global fund, not only its \emph{primary} custodian but the entire
chain of \emph{foreign sub-custodians} --- one bank per market it invests in.
These sub-custodians have two damaging properties. First, they are
\emph{unextractable}: they appear only in the N-CEN table and essentially never in
the prospectus prose (a naive string-match recovers $7\%$ of them), so keeping
them as targets asks the model to extract facts absent from its input. Second,
they \emph{dominate}: with \code{IS\_SUB\_CUSTODIAN${=}$Y} accounting for $88\%$
of custodian rows, they constitute roughly two thirds of \emph{all} edges in the
unfiltered graph, inflating both the target size and the training loss with noise.
The dataset therefore scopes the custodian relation to the \emph{primary}
custodian (\code{IS\_SUB\_CUSTODIAN${\neq}$Y}, a median of one per fund), which is
genuinely prose-grounded --- it is named in the prospectus or its Statement of
Additional Information (e.g.\ ``State Street Bank and Trust Company serves as
custodian''). This single change reduces the corpus from $36{,}880$ to $15{,}739$
edges, all of prose-grounded relation types, and is the configurable default
(\code{--custodian-scope primary}). The full sub-custodian chain remains available
in N-CEN as a structured-only relation outside the text-to-triples task. This is a
dataset-quality decision of the same kind the thesis notes for T-REx and REBEL,
whose non-exhaustive references unfairly penalise correct extractions.
\paragraph{No-model lower bound.} A trivial string-matching baseline --- emit a
gold edge iff the object's name occurs in the prose --- establishes a \emph{floor}
and measures \emph{how prose-grounded each relation is}. Table~\ref{tab:baseline}
reports this on the proof-of-concept slice after primary-custodian scoping,
multi-book fetching and per-fund segmentation. Because the baseline requires an
\emph{exact substring} match within the fund's \emph{own} section, its recall is a
strict lower bound: a fund's adviser, for instance, must be named in that fund's
segment under a literal spelling. With the right prospectus books present, the
adviser is recovered with recall $1.00$ and the micro-averaged $F_1$ reaches
$0.65$; the recovered custodian recall is $0.63$ (up from $0.07$ unscoped and
$0.37$ after scoping alone). The residual gap from $1.0$ is attributable to
surface-form variation (``State Street Bank and Trust Company'' vs.\ ``State
Street'') that a trained model handles but exact matching does not.
\begin{table}[h]
\centering
\caption{No-model string-match baseline on the proof-of-concept slice, after
primary-custodian scoping, multi-book fetching and per-fund segmentation
($141$ samples). Precision is $1.00$ by construction; recall is a strict
exact-match lower bound.}
\label{tab:baseline}
\small
\begin{tabular}{@{}lrr@{}}
\toprule
Relation & Recall & Gold edges \\
\midrule
\code{advisedBy} & 1.00 & 513 \\
\code{subAdvisedBy} & 0.94 & 232 \\
\code{seriesOf} & 0.86 & 496 \\
\code{administrator} & 0.84 & 730 \\
\code{transferAgent} & 0.79 & 537 \\
\code{custodian} & 0.63 & 601 \\
\code{underwrittenBy} & 0.40 & 144 \\
\midrule
micro-average & 0.48 & 1{,}194 \\
\bottomrule
\end{tabular}
\end{table}
\paragraph{Strong-model silver.} For relations that are expressed only in prose
and lack a clean structured source --- portfolio managers (\code{managedBy}),
the named benchmark index (\code{tracksIndex}), and MDFP-named holdings --- a
strong reference model (e.g.\ GPT-4 or Claude Opus) produces silver labels.
Because the structured relations have model-free gold, the silver-labelling model
can itself be \emph{measured} on the overlapping gold edges, so its reliability is
quantified rather than assumed.
% ====================================================================
\section{Corpus statistics}
% ====================================================================
Table~\ref{tab:stats} summarises one quarter (2025\,Q3) of N-CEN gold (after
primary-custodian scoping) and the proof-of-concept \emph{per-fund} samples. The
gold graph holds $15{,}739$ entity-to-entity edges across $435$ trusts and
$2{,}421$ funds. Fetching all prospectus books and applying the robust segmenter
yields $141$ samples whose per-fund median input is $\sim\!3.7\times10^{4}$
characters against a $\sim\!6.5\times10^{2}$-character target: a median ratio near
$55\!:\!1$, the inverse of the symmetric-size benchmarks.
\begin{table}[h]
\centering
\caption{Corpus statistics. Left: N-CEN gold graph for 2025\,Q3 (primary-custodian
scope). Right: proof-of-concept samples (multi-book fetch, per-fund segmentation).}
\label{tab:stats}
\small
\begin{tabular}{@{}lr@{\hskip 3em}lr@{}}
\toprule
\multicolumn{2}{c}{Gold graph (2025\,Q3)} & \multicolumn{2}{c}{PoC samples} \\
\cmidrule(r){1-2}\cmidrule(l){3-4}
Trust graphs & 435 & Samples (total) & 141 \\
Funds (series) & 2{,}421 & \;segmented per-fund & 135 \\
Entity-entity edges & 15{,}739 & \;whole-trust fallback & 6 \\
\;custodian (primary) & 3{,}045 & Per-fund input (median) & 36{,}856 \\
\;advisedBy & 2{,}588 & Per-fund target (median) & 654 \\
Distributors & 458 & Ratio (median) & $54.8\!:\!1$ \\
\bottomrule
\end{tabular}
\end{table}
The full quarter therefore yields on the order of $2{,}400$ fund-level graphs
from $435$ prospectus filings; multiple quarters and the dropping of ontology
subsets (per the thesis's augmentation strategy) expand this further.
% ====================================================================
\section{Use in the thesis experiments}
% ====================================================================
Each sample is a triple $(x,\,\sigma,\,y)$ where $x$ is the prospectus prose,
$\sigma$ is the inferred ontology, and $y$ is the marker-serialized triple graph.
The model is trained to compute $y = f_{\theta}(x,\sigma)$. This dataset feeds the
four-model comparison of the thesis directly:
\begin{itemize}[nosep]
\item \textbf{Model 1/3} (decoder-only / encoder-decoder, no extra tokens):
trained on the plain serialization \code{target\_serialized\_plain}.
\item \textbf{Model 2/4} (with grammar-terminal tokens): trained on the marker
serialization \code{target\_serialized}, with the four markers
\tstart, \predm, \objm, \tend{} added to the vocabulary as single tokens,
testing research question~1 (do dedicated terminal tokens reduce loss and
raise $F_1$).
\end{itemize}
\paragraph{Splits.} The dataset is partitioned into train/validation/test at the
\emph{trust} level ($80/10/10$), assigned by a deterministic hash of the trust
CIK. Splitting by trust rather than by fund prevents leakage: funds of one trust
share advisers, distributors and custodians, so a fund-level split would let the
model memorise trust-specific entities seen in training and inflate test scores.
The builder verifies that no trust appears in more than one split.
Because the dataset's input is far longer than its output and its target is a
relational graph, it stresses precisely the capabilities the thesis cares about:
long-context reading comprehension and faithful generation of entity-to-entity
structure. Evaluation uses triple-level precision, recall and $F_1$ against the
model-free gold, matched on \code{(subject type, predicate, normalized object
label)} so that IRI-slug differences do not create spurious errors. The same
metric scores the strong-model silver baseline, giving a like-for-like comparison
between the finetuned models and a state-of-the-art prompted extractor.
% ====================================================================
\section{Reproducibility}
% ====================================================================
The dataset is built by two scripts accompanying this note.
\code{build\_rdf\_dataset.py} has three stages: \code{gold} parses the local
N-CEN flat files into per-trust gold graphs (with \code{--custodian-scope} to
choose primary-only, all, or no custodian edges); \code{fetch} downloads all
recent full prospectus books per trust from EDGAR (rate-limited, \code{gzip}-aware,
\code{--max-filings} per trust) and concatenates them; \code{samples} segments the
prose per fund and joins it with the gold into the $(x,\sigma,y)$ records described
above. \code{score\_baseline.py} computes the no-model string-match baseline and
scores any strong-model predictions against the gold. All inputs are public SEC
filings; no licensing restriction applies to the derived dataset.
\end{document}

BIN
documentation.pdf Normal file
View File

Binary file not shown.

269
fetch_filings.py Normal file
View File

@ -0,0 +1,269 @@
"""
Step 2: Fetch prospectus filings for the fund universe.
For each trust (CIK) in the database, fetch filing history from the
SEC Submissions API and download prospectus documents (485BPOS, 497, 497K, N-1A).
Stores filing metadata and extracted text in the database.
"""
import logging
import re
import time
import warnings
from typing import Optional
import requests
from bs4 import BeautifulSoup, XMLParsedAsHTMLWarning
from tqdm import tqdm
warnings.filterwarnings("ignore", category=XMLParsedAsHTMLWarning)
from fund_db import FundDatabase
log = logging.getLogger(__name__)
DATA_SEC = "https://data.sec.gov"
ARCHIVES = "https://www.sec.gov/Archives/edgar/data"
PROSPECTUS_FORM_TYPES = {"485BPOS", "485APOS", "497", "497K", "N-1A", "N-1A/A"}
USER_AGENT = "FundDataResearch/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12
_last_request_time = 0.0
def _throttled_get(url: str, **kwargs) -> requests.Response:
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 30)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
resp.raise_for_status()
return resp
def fetch_filing_metadata(db: FundDatabase, cik: str,
max_prospectus: int = 3,
max_supplements: int = 3) -> int:
"""Fetch filing metadata for a single CIK and store in database."""
url = f"{DATA_SEC}/submissions/CIK{cik}.json"
resp = _throttled_get(url)
data = resp.json()
recent = data.get("filings", {}).get("recent", {})
accessions = recent.get("accessionNumber", [])
forms = recent.get("form", [])
dates = recent.get("filingDate", [])
docs = recent.get("primaryDocument", [])
descs = recent.get("primaryDocDescription", [])
prospectus_count = 0
supplement_count = 0
total_saved = 0
for i in range(len(accessions)):
form_type = forms[i]
if form_type not in PROSPECTUS_FORM_TYPES:
continue
is_prospectus = form_type in ("485BPOS", "485APOS", "N-1A", "N-1A/A")
if is_prospectus and prospectus_count >= max_prospectus:
continue
if not is_prospectus and supplement_count >= max_supplements:
continue
accession_no_dashes = accessions[i].replace("-", "")
cik_int = str(int(cik))
doc_url = f"{ARCHIVES}/{cik_int}/{accession_no_dashes}/{docs[i]}"
db.insert_filing(
accession_number=accessions[i],
cik=cik,
form_type=form_type,
filing_date=dates[i],
primary_document=docs[i],
document_url=doc_url,
description=descs[i] if i < len(descs) else "",
)
total_saved += 1
if is_prospectus:
prospectus_count += 1
else:
supplement_count += 1
return total_saved
def download_filing_content(url: str, max_bytes: int = 30_000_000) -> tuple[str, str]:
"""
Download a filing document. Returns (plain_text, raw_html).
Raw HTML is preserved so table structure, inline XBRL tags,
and other markup remain available for downstream processing.
"""
try:
resp = _throttled_get(url, stream=True)
content_type = resp.headers.get("Content-Type", "")
if "pdf" in content_type.lower():
return "[PDF — binary content not extracted]", ""
raw = resp.content[:max_bytes]
html = raw.decode("utf-8", errors="replace")
soup = BeautifulSoup(html, "lxml")
for tag in soup(["script", "style", "meta", "link"]):
tag.decompose()
plain = soup.get_text(separator="\n", strip=True)
plain = re.sub(r"\n{3,}", "\n\n", plain)
return plain, html
except Exception as e:
log.warning("Failed to download %s: %s", url, e)
return "", ""
def fetch_filings_for_universe(db: FundDatabase,
ciks: list[str] = None,
limit: int = 0,
download_text: bool = True,
max_prospectus: int = 3,
max_supplements: int = 3):
"""Fetch filing metadata and text for a list of CIKs."""
if ciks is None:
ciks = db.get_pending_ciks("fetch_filings")
if limit > 0:
ciks = ciks[:limit]
log.info("Fetching filings for %d CIKs", len(ciks))
total_filings = 0
for cik in tqdm(ciks, desc="Fetching filings"):
status = db.get_pipeline_status(cik, "fetch_filings")
if status == "done":
continue
db.set_pipeline_status(cik, "fetch_filings", "running")
try:
count = fetch_filing_metadata(db, cik, max_prospectus, max_supplements)
db.set_pipeline_status(cik, "fetch_filings", "done", items_processed=count)
total_filings += count
except Exception as e:
db.set_pipeline_status(cik, "fetch_filings", "error",
error_message=str(e)[:500])
log.warning("Failed for CIK %s: %s", cik, e)
log.info("Saved %d filing records", total_filings)
if download_text:
download_pending_texts(db, limit=limit * 10 if limit else 0)
return total_filings
def download_pending_texts(db: FundDatabase, limit: int = 0):
"""Download text + raw HTML for filings that don't have text yet."""
with db.conn() as c:
query = """
SELECT f.accession_number, f.document_url, f.form_type
FROM filing f
LEFT JOIN filing_text ft ON f.accession_number = ft.accession_number
WHERE ft.accession_number IS NULL
AND f.document_url IS NOT NULL
AND f.document_url != ''
"""
if limit > 0:
query += f" LIMIT {limit}"
rows = c.execute(query).fetchall()
log.info("Downloading text + HTML for %d filings", len(rows))
for row in tqdm(rows, desc="Downloading filings"):
acc = row["accession_number"]
url = row["document_url"]
plain, html = download_filing_content(url)
if plain:
db.save_filing_text(acc, plain, html_content=html if html else None)
log.debug(" %s → %d text chars, %d HTML chars",
acc, len(plain), len(html))
def backfill_html(db: FundDatabase, limit: int = 0):
"""Re-download raw HTML for filings that have text but no HTML stored."""
with db.conn() as c:
query = """
SELECT f.accession_number, f.document_url, f.form_type
FROM filing f
JOIN filing_text ft ON f.accession_number = ft.accession_number
LEFT JOIN filing_html fh ON f.accession_number = fh.accession_number
WHERE fh.accession_number IS NULL
AND f.document_url IS NOT NULL
AND f.document_url != ''
"""
if limit > 0:
query += f" LIMIT {limit}"
rows = c.execute(query).fetchall()
log.info("Backfilling HTML for %d filings", len(rows))
for row in tqdm(rows, desc="Backfilling HTML"):
acc = row["accession_number"]
url = row["document_url"]
plain, html = download_filing_content(url)
if html:
db.save_filing_text(acc, plain, html_content=html)
log.debug(" %s → %d HTML chars", acc, len(html))
def main():
import argparse
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
parser = argparse.ArgumentParser(description="Fetch SEC fund filings")
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("--limit", type=int, default=0,
help="Max CIKs to process (0=all pending)")
parser.add_argument("--ciks", nargs="*", help="Specific CIKs to fetch")
parser.add_argument("--max-prospectus", type=int, default=3,
help="Max prospectus filings per trust")
parser.add_argument("--max-supplements", type=int, default=3,
help="Max supplement filings per trust")
parser.add_argument("--no-text", action="store_true",
help="Skip downloading filing text (metadata only)")
parser.add_argument("--text-only", action="store_true",
help="Only download text for existing filings")
parser.add_argument("--backfill-html", action="store_true",
help="Re-download HTML for filings missing raw HTML")
args = parser.parse_args()
db = FundDatabase(args.db)
if args.backfill_html:
backfill_html(db, limit=args.limit)
elif args.text_only:
download_pending_texts(db, limit=args.limit)
else:
ciks = [c.zfill(10) for c in args.ciks] if args.ciks else None
fetch_filings_for_universe(
db, ciks=ciks, limit=args.limit,
download_text=not args.no_text,
max_prospectus=args.max_prospectus,
max_supplements=args.max_supplements,
)
stats = db.get_stats()
print(f"\nDatabase stats:")
print(f" Filings: {stats['filing']:>10,}")
print(f" Filing texts: {stats['filing_text']:>10,}")
if __name__ == "__main__":
main()

190
fetch_universe.py Normal file
View File

@ -0,0 +1,190 @@
"""
Step 1: Build the fund universe from SEC Series/Class CSV.
Downloads the SEC Investment Company Series and Class Information CSV,
parses it, and loads all trusts, series, and share classes into the
SQLite database. This gives us the full universe of ~15K trusts,
~50K series (funds), and ~100K+ share classes.
"""
import csv
import io
import logging
import time
import requests
from tqdm import tqdm
from fund_db import FundDatabase
log = logging.getLogger(__name__)
SEC_BASE = "https://www.sec.gov"
DATA_SEC = "https://data.sec.gov"
SERIES_CLASS_CSV_URL = (
"https://www.sec.gov/files/investment/data/other/"
"investment-company-series-class-information/"
"investment-company-series-class-2025.csv"
)
USER_AGENT = "FundDataResearch/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12
_last_request_time = 0.0
def _throttled_get(url: str, **kwargs) -> requests.Response:
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 60)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
resp.raise_for_status()
return resp
def load_series_class_csv(db: FundDatabase, csv_url: str = SERIES_CLASS_CSV_URL):
"""Download SEC Series/Class CSV and load into the database."""
log.info("Downloading SEC Series/Class CSV from %s", csv_url)
resp = _throttled_get(csv_url)
resp.encoding = "utf-8"
reader = csv.DictReader(io.StringIO(resp.text))
trust_count = 0
series_count = 0
class_count = 0
seen_trusts = set()
seen_series = set()
for row in reader:
cik = row.get("CIK Number", "").strip().zfill(10)
if not cik or cik == "0" * 10:
continue
if cik not in seen_trusts:
db.upsert_trust_simple(
cik=cik,
trust_name=row.get("Entity Name", "").strip(),
file_number=row.get("Reporting File Number", "").strip(),
entity_type=row.get("Entity Org Type", "").strip(),
)
seen_trusts.add(cik)
trust_count += 1
series_id = row.get("Series ID", "").strip()
if not series_id:
continue
if series_id not in seen_series:
db.upsert_series(
series_id=series_id,
cik=cik,
series_name=row.get("Series Name", "").strip(),
)
seen_series.add(series_id)
series_count += 1
class_id = row.get("Class ID", "").strip()
if class_id:
db.upsert_share_class(
class_id=class_id,
series_id=series_id,
cik=cik,
class_name=row.get("Class Name", "").strip(),
ticker=row.get("Class Ticker", "").strip(),
)
class_count += 1
log.info("Loaded %d trusts, %d series, %d share classes from CSV",
trust_count, series_count, class_count)
db.record_bulk_download("series_class_csv", "2025", csv_url, class_count)
return trust_count, series_count, class_count
def enrich_from_submissions_api(db: FundDatabase, ciks: list[str] = None,
limit: int = 0):
"""
Enrich trust records with data from the SEC Submissions API.
Adds: fiscal year end, SIC code, state of incorporation, website.
"""
if ciks is None:
ciks = db.get_all_ciks()
if limit > 0:
ciks = ciks[:limit]
log.info("Enriching %d trusts from Submissions API", len(ciks))
enriched = 0
for cik in tqdm(ciks, desc="Enriching trusts"):
if db.get_pipeline_status(cik, "enrich_submissions") == "done":
continue
db.set_pipeline_status(cik, "enrich_submissions", "running")
try:
url = f"{DATA_SEC}/submissions/CIK{cik}.json"
resp = _throttled_get(url)
data = resp.json()
db.upsert_trust_simple(
cik=cik,
trust_name=data.get("name", ""),
state_of_inc=data.get("stateOfIncorporation", ""),
fiscal_year_end=data.get("fiscalYearEnd", ""),
website=data.get("website", ""),
)
sic = data.get("sic", "")
if sic:
with db.conn() as c:
c.execute("UPDATE trust SET sic_code=? WHERE cik=?", (sic, cik))
db.set_pipeline_status(cik, "enrich_submissions", "done", items_processed=1)
enriched += 1
except Exception as e:
db.set_pipeline_status(cik, "enrich_submissions", "error",
error_message=str(e)[:500])
log.warning("Failed to enrich CIK %s: %s", cik, e)
log.info("Enriched %d trusts", enriched)
return enriched
def main():
import argparse
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
parser = argparse.ArgumentParser(description="Build SEC fund universe")
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("--skip-csv", action="store_true",
help="Skip CSV download (if already loaded)")
parser.add_argument("--enrich", action="store_true",
help="Enrich trusts from Submissions API")
parser.add_argument("--enrich-limit", type=int, default=0,
help="Max trusts to enrich (0=all)")
args = parser.parse_args()
db = FundDatabase(args.db)
if not args.skip_csv:
t, s, c = load_series_class_csv(db)
print(f"\nUniverse loaded: {t:,} trusts, {s:,} series, {c:,} share classes")
if args.enrich:
n = enrich_from_submissions_api(db, limit=args.enrich_limit)
print(f"Enriched {n:,} trusts from Submissions API")
stats = db.get_stats()
print(f"\nDatabase stats:")
for table, count in stats.items():
if count > 0:
print(f" {table:30s} {count:>10,}")
if __name__ == "__main__":
main()

484
fund_db.py Normal file
View File

@ -0,0 +1,484 @@
"""
SQLite database for SEC fund universe.
Stores:
- Fund universe (trusts, series, share classes) from SEC Series/Class CSV
- Filing metadata and prospectus text from EDGAR
- Structured fee/performance data from XBRL Risk/Return
- Holdings snapshots from N-PORT
- Service provider data from N-CEN
"""
import logging
import sqlite3
from contextlib import contextmanager
from pathlib import Path
log = logging.getLogger(__name__)
SCHEMA = """
-- ===== Fund Universe =====
CREATE TABLE IF NOT EXISTS trust (
cik TEXT PRIMARY KEY,
trust_name TEXT NOT NULL,
file_number TEXT,
state_of_inc TEXT,
fiscal_year_end TEXT,
sic_code TEXT,
entity_type TEXT,
website TEXT,
updated_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS series (
series_id TEXT PRIMARY KEY,
cik TEXT NOT NULL REFERENCES trust(cik),
series_name TEXT NOT NULL,
status TEXT DEFAULT 'active',
updated_at TEXT DEFAULT (datetime('now'))
);
CREATE TABLE IF NOT EXISTS share_class (
class_id TEXT PRIMARY KEY,
series_id TEXT NOT NULL REFERENCES series(series_id),
cik TEXT NOT NULL REFERENCES trust(cik),
class_name TEXT,
ticker TEXT,
cusip TEXT,
isin TEXT,
inception_date TEXT,
status TEXT DEFAULT 'active',
updated_at TEXT DEFAULT (datetime('now'))
);
CREATE INDEX IF NOT EXISTS idx_series_cik ON series(cik);
CREATE INDEX IF NOT EXISTS idx_class_series ON share_class(series_id);
CREATE INDEX IF NOT EXISTS idx_class_cik ON share_class(cik);
CREATE INDEX IF NOT EXISTS idx_class_ticker ON share_class(ticker);
-- ===== Filings & Prospectus Text =====
CREATE TABLE IF NOT EXISTS filing (
accession_number TEXT PRIMARY KEY,
cik TEXT NOT NULL REFERENCES trust(cik),
form_type TEXT NOT NULL,
filing_date TEXT NOT NULL,
primary_document TEXT,
document_url TEXT,
description TEXT,
text_length INTEGER DEFAULT 0,
fetched_at TEXT
);
CREATE TABLE IF NOT EXISTS filing_text (
accession_number TEXT PRIMARY KEY REFERENCES filing(accession_number),
text_content TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS filing_html (
accession_number TEXT PRIMARY KEY REFERENCES filing(accession_number),
html_content TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_filing_cik ON filing(cik);
CREATE INDEX IF NOT EXISTS idx_filing_form ON filing(form_type);
CREATE INDEX IF NOT EXISTS idx_filing_date ON filing(filing_date);
-- ===== XBRL Risk/Return Structured Data =====
CREATE TABLE IF NOT EXISTS xbrl_fee (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT,
cik TEXT NOT NULL,
series_id TEXT,
class_id TEXT,
filing_date TEXT,
fund_name TEXT,
share_class_name TEXT,
max_sales_charge_pct REAL,
max_deferred_charge_pct REAL,
redemption_fee_pct REAL,
management_fee_pct REAL,
dist_12b1_fee_pct REAL,
other_expenses_pct REAL,
acquired_fund_fees_pct REAL,
total_expenses_pct REAL,
fee_waiver_pct REAL,
net_expenses_pct REAL,
expense_example_1yr REAL,
expense_example_3yr REAL,
expense_example_5yr REAL,
expense_example_10yr REAL,
UNIQUE(cik, series_id, class_id, filing_date)
);
CREATE TABLE IF NOT EXISTS xbrl_performance (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT,
cik TEXT NOT NULL,
series_id TEXT,
class_id TEXT,
filing_date TEXT,
fund_name TEXT,
share_class_name TEXT,
inception_date TEXT,
return_year_1 REAL,
return_year_5 REAL,
return_year_10 REAL,
return_since_incep REAL,
best_quarter_return REAL,
best_quarter_label TEXT,
worst_quarter_return REAL,
worst_quarter_label TEXT,
portfolio_turnover REAL,
UNIQUE(cik, series_id, class_id, filing_date)
);
CREATE TABLE IF NOT EXISTS xbrl_objective (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT,
cik TEXT NOT NULL,
series_id TEXT,
filing_date TEXT,
fund_name TEXT,
objective_text TEXT,
strategy_text TEXT,
risk_text TEXT,
UNIQUE(cik, series_id, filing_date)
);
-- ===== N-PORT Holdings & Fund-Level Data =====
CREATE TABLE IF NOT EXISTS nport_fund_info (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT NOT NULL,
cik TEXT NOT NULL,
series_id TEXT,
report_date TEXT NOT NULL,
total_assets REAL,
total_liabilities REAL,
net_assets REAL,
borrowing_pay_within_1yr REAL,
borrowing_pay_after_1yr REAL,
controlled_affiliates_val REAL,
is_non_cash_collateral INTEGER,
credit_spread_risk_inv_grade_dv01 REAL,
credit_spread_risk_non_inv_dv01 REAL,
UNIQUE(accession_number)
);
CREATE TABLE IF NOT EXISTS nport_monthly_return (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT NOT NULL,
cik TEXT NOT NULL,
class_id TEXT,
report_date TEXT NOT NULL,
month1_return REAL,
month2_return REAL,
month3_return REAL,
UNIQUE(accession_number, class_id)
);
CREATE TABLE IF NOT EXISTS nport_holding (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT NOT NULL,
cik TEXT NOT NULL,
report_date TEXT NOT NULL,
holding_name TEXT,
lei TEXT,
cusip TEXT,
isin TEXT,
ticker TEXT,
asset_category TEXT,
issuer_category TEXT,
inv_country TEXT,
currency TEXT,
quantity REAL,
value_usd REAL,
pct_val REAL,
is_debt INTEGER DEFAULT 0,
coupon_rate REAL,
maturity_date TEXT,
is_default INTEGER,
fair_value_level TEXT
);
CREATE INDEX IF NOT EXISTS idx_holding_acc ON nport_holding(accession_number);
CREATE INDEX IF NOT EXISTS idx_holding_cik ON nport_holding(cik);
CREATE INDEX IF NOT EXISTS idx_holding_date ON nport_holding(report_date);
-- ===== N-CEN Service Provider & Classification =====
CREATE TABLE IF NOT EXISTS ncen_fund_info (
id INTEGER PRIMARY KEY AUTOINCREMENT,
accession_number TEXT NOT NULL,
cik TEXT NOT NULL,
series_id TEXT,
report_period TEXT,
is_etf INTEGER,
is_index_fund INTEGER,
index_name TEXT,
is_fund_of_funds INTEGER,
is_money_market INTEGER,
fund_type TEXT,
UNIQUE(accession_number, series_id)
);
CREATE TABLE IF NOT EXISTS ncen_service_provider (
id INTEGER PRIMARY KEY AUTOINCREMENT,
cik TEXT NOT NULL,
report_period TEXT,
provider_role TEXT NOT NULL,
provider_name TEXT NOT NULL,
provider_lei TEXT,
provider_country TEXT,
UNIQUE(cik, report_period, provider_role, provider_name)
);
-- ===== Pipeline Status Tracking =====
CREATE TABLE IF NOT EXISTS pipeline_status (
cik TEXT NOT NULL,
stage TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
started_at TEXT,
completed_at TEXT,
error_message TEXT,
items_processed INTEGER DEFAULT 0,
PRIMARY KEY (cik, stage)
);
CREATE TABLE IF NOT EXISTS bulk_download (
source TEXT PRIMARY KEY,
quarter TEXT,
downloaded_at TEXT,
file_path TEXT,
record_count INTEGER
);
"""
class FundDatabase:
"""SQLite database wrapper for SEC fund data."""
def __init__(self, db_path: str = "fund_data.db"):
self.db_path = Path(db_path)
self.db_path.parent.mkdir(parents=True, exist_ok=True)
self._init_db()
def _init_db(self):
with self.conn() as c:
c.executescript(SCHEMA)
c.execute("PRAGMA journal_mode=WAL")
c.execute("PRAGMA synchronous=NORMAL")
c.execute("PRAGMA cache_size=-64000") # 64MB cache
log.info("Database initialized at %s", self.db_path)
@contextmanager
def conn(self):
con = sqlite3.connect(str(self.db_path), timeout=30)
con.row_factory = sqlite3.Row
try:
yield con
con.commit()
except Exception:
con.rollback()
raise
finally:
con.close()
# ----- Trust / Series / Class -----
def upsert_trust(self, cik, trust_name, **kwargs):
cols = ["cik", "trust_name"] + list(kwargs.keys()) + ["updated_at"]
vals = [cik, trust_name] + list(kwargs.values()) + ["datetime('now')"]
placeholders = ", ".join("?" if c != "updated_at" else "datetime('now')" for c in cols)
set_clause = ", ".join(
f"{c}=excluded.{c}" for c in cols if c != "cik"
)
sql = f"""
INSERT INTO trust ({', '.join(cols)})
VALUES ({placeholders})
ON CONFLICT(cik) DO UPDATE SET {set_clause}
"""
actual_vals = [v for c, v in zip(cols, vals) if c != "updated_at"]
with self.conn() as c:
c.execute(sql, actual_vals)
def upsert_trust_simple(self, cik, trust_name, file_number="", state_of_inc="",
fiscal_year_end="", entity_type="", website=""):
with self.conn() as c:
c.execute("""
INSERT INTO trust (cik, trust_name, file_number, state_of_inc,
fiscal_year_end, entity_type, website)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(cik) DO UPDATE SET
trust_name=excluded.trust_name,
file_number=COALESCE(NULLIF(excluded.file_number,''), trust.file_number),
state_of_inc=COALESCE(NULLIF(excluded.state_of_inc,''), trust.state_of_inc),
fiscal_year_end=COALESCE(NULLIF(excluded.fiscal_year_end,''), trust.fiscal_year_end),
entity_type=COALESCE(NULLIF(excluded.entity_type,''), trust.entity_type),
website=COALESCE(NULLIF(excluded.website,''), trust.website),
updated_at=datetime('now')
""", (cik, trust_name, file_number, state_of_inc,
fiscal_year_end, entity_type, website))
def upsert_series(self, series_id, cik, series_name, status="active"):
with self.conn() as c:
c.execute("""
INSERT INTO series (series_id, cik, series_name, status)
VALUES (?, ?, ?, ?)
ON CONFLICT(series_id) DO UPDATE SET
series_name=excluded.series_name,
status=excluded.status,
updated_at=datetime('now')
""", (series_id, cik, series_name, status))
def upsert_share_class(self, class_id, series_id, cik, class_name="",
ticker="", cusip="", isin="", inception_date=""):
with self.conn() as c:
c.execute("""
INSERT INTO share_class (class_id, series_id, cik, class_name,
ticker, cusip, isin, inception_date)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(class_id) DO UPDATE SET
class_name=excluded.class_name,
ticker=COALESCE(NULLIF(excluded.ticker,''), share_class.ticker),
cusip=COALESCE(NULLIF(excluded.cusip,''), share_class.cusip),
isin=COALESCE(NULLIF(excluded.isin,''), share_class.isin),
inception_date=COALESCE(NULLIF(excluded.inception_date,''), share_class.inception_date),
updated_at=datetime('now')
""", (class_id, series_id, cik, class_name, ticker, cusip, isin, inception_date))
# ----- Filings -----
def insert_filing(self, accession_number, cik, form_type, filing_date,
primary_document="", document_url="", description=""):
with self.conn() as c:
c.execute("""
INSERT OR IGNORE INTO filing
(accession_number, cik, form_type, filing_date, primary_document,
document_url, description)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (accession_number, cik, form_type, filing_date,
primary_document, document_url, description))
def save_filing_text(self, accession_number, text_content, html_content=None):
with self.conn() as c:
c.execute("""
INSERT OR REPLACE INTO filing_text (accession_number, text_content)
VALUES (?, ?)
""", (accession_number, text_content))
if html_content:
c.execute("""
INSERT OR REPLACE INTO filing_html (accession_number, html_content)
VALUES (?, ?)
""", (accession_number, html_content))
c.execute("""
UPDATE filing SET text_length=?, fetched_at=datetime('now')
WHERE accession_number=?
""", (len(text_content), accession_number))
# ----- Pipeline status -----
def set_pipeline_status(self, cik, stage, status, error_message="", items_processed=0):
ts_col = "started_at" if status == "running" else "completed_at"
with self.conn() as c:
c.execute(f"""
INSERT INTO pipeline_status (cik, stage, status, {ts_col}, error_message, items_processed)
VALUES (?, ?, ?, datetime('now'), ?, ?)
ON CONFLICT(cik, stage) DO UPDATE SET
status=excluded.status,
{ts_col}=datetime('now'),
error_message=excluded.error_message,
items_processed=excluded.items_processed
""", (cik, stage, status, error_message, items_processed))
def get_pipeline_status(self, cik, stage):
with self.conn() as c:
row = c.execute(
"SELECT status FROM pipeline_status WHERE cik=? AND stage=?",
(cik, stage)
).fetchone()
return row["status"] if row else None
def get_pending_ciks(self, stage, limit=1000):
with self.conn() as c:
rows = c.execute("""
SELECT t.cik FROM trust t
LEFT JOIN pipeline_status ps ON t.cik = ps.cik AND ps.stage = ?
WHERE ps.status IS NULL OR ps.status = 'pending'
ORDER BY t.cik
LIMIT ?
""", (stage, limit)).fetchall()
return [r["cik"] for r in rows]
# ----- Bulk inserts -----
def bulk_insert_holdings(self, holdings):
"""Insert a batch of N-PORT holdings. Each is a dict."""
if not holdings:
return
cols = [
"accession_number", "cik", "report_date", "holding_name", "lei",
"cusip", "isin", "ticker", "asset_category", "issuer_category",
"inv_country", "currency", "quantity", "value_usd", "pct_val",
"is_debt", "coupon_rate", "maturity_date", "is_default", "fair_value_level"
]
placeholders = ", ".join(["?"] * len(cols))
sql = f"INSERT OR IGNORE INTO nport_holding ({', '.join(cols)}) VALUES ({placeholders})"
rows = [tuple(h.get(c) for c in cols) for h in holdings]
with self.conn() as c:
c.executemany(sql, rows)
def record_bulk_download(self, source, quarter, file_path, record_count):
with self.conn() as c:
c.execute("""
INSERT OR REPLACE INTO bulk_download
(source, quarter, downloaded_at, file_path, record_count)
VALUES (?, ?, datetime('now'), ?, ?)
""", (source, quarter, file_path, record_count))
# ----- Queries -----
def get_stats(self):
with self.conn() as c:
stats = {}
for table in ["trust", "series", "share_class", "filing",
"filing_text", "filing_html", "xbrl_fee",
"xbrl_performance", "nport_fund_info",
"nport_holding", "ncen_fund_info",
"ncen_service_provider"]:
row = c.execute(f"SELECT COUNT(*) as cnt FROM {table}").fetchone()
stats[table] = row["cnt"]
return stats
def get_ciks_with_series(self):
"""Return all CIKs that have at least one series (i.e. active fund trusts)."""
with self.conn() as c:
rows = c.execute("""
SELECT DISTINCT t.cik, t.trust_name, COUNT(s.series_id) as num_series
FROM trust t JOIN series s ON t.cik = s.cik
GROUP BY t.cik
ORDER BY num_series DESC
""").fetchall()
return [dict(r) for r in rows]
def get_all_ciks(self):
with self.conn() as c:
rows = c.execute("SELECT cik FROM trust ORDER BY cik").fetchall()
return [r["cik"] for r in rows]
def search_funds(self, query, limit=50):
with self.conn() as c:
rows = c.execute("""
SELECT t.cik, t.trust_name, s.series_id, s.series_name,
sc.class_id, sc.class_name, sc.ticker
FROM share_class sc
JOIN series s ON sc.series_id = s.series_id
JOIN trust t ON s.cik = t.cik
WHERE sc.ticker LIKE ? OR s.series_name LIKE ? OR t.trust_name LIKE ?
LIMIT ?
""", (f"%{query}%", f"%{query}%", f"%{query}%", limit)).fetchall()
return [dict(r) for r in rows]

246
load_ncen.py Normal file
View File

@ -0,0 +1,246 @@
"""
Step 5: Load N-CEN data (service providers, fund classification, ETF info).
Downloads quarterly N-CEN data set ZIPs from SEC, parses the TSV files,
and loads fund classification and service provider data into the database.
"""
import csv
import logging
import time
import zipfile
from pathlib import Path
import requests
from tqdm import tqdm
from fund_db import FundDatabase
log = logging.getLogger(__name__)
USER_AGENT = "FundDataResearch/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12
_last_request_time = 0.0
NCEN_BASE_URL = "https://www.sec.gov/files/dera/data/form-n-cen-data-sets"
def _throttled_get(url: str, **kwargs) -> requests.Response:
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 120)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
resp.raise_for_status()
return resp
def download_ncen_zip(quarter: str, output_dir: str = "data/ncen") -> Path:
"""Download a quarterly N-CEN ZIP from SEC."""
url = f"{NCEN_BASE_URL}/{quarter}_ncen.zip"
out = Path(output_dir)
out.mkdir(parents=True, exist_ok=True)
zip_path = out / f"{quarter}_ncen.zip"
extract_dir = out / quarter
if extract_dir.exists() and any(extract_dir.iterdir()):
log.info("Already extracted: %s", extract_dir)
return extract_dir
log.info("Downloading N-CEN data set: %s", url)
resp = _throttled_get(url, stream=True)
with open(zip_path, "wb") as fp:
for chunk in resp.iter_content(chunk_size=65536):
fp.write(chunk)
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(extract_dir)
log.info("Extracted to %s", extract_dir)
return extract_dir
def _find_tsv(extract_dir: Path, pattern: str) -> Path:
candidates = list(extract_dir.rglob(f"*{pattern}*"))
tsv_candidates = [c for c in candidates if c.suffix.lower() in (".tsv", ".txt", "")]
if tsv_candidates:
return tsv_candidates[0]
if candidates:
return candidates[0]
return extract_dir / f"{pattern}.tsv"
def _read_tsv(filepath: Path) -> list[dict]:
if not filepath.exists():
log.warning("File not found: %s", filepath)
return []
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
reader = csv.DictReader(f, delimiter="\t")
return list(reader)
def load_etf_data(db: FundDatabase, extract_dir: Path, quarter: str):
"""Load ETF classification data from N-CEN."""
filepath = _find_tsv(extract_dir, "ETF")
if not filepath.exists():
log.info("No ETF file found in %s", extract_dir)
return 0
rows = _read_tsv(filepath)
count = 0
with db.conn() as c:
for row in rows:
cik = row.get("CIK", "").strip().zfill(10)
acc = row.get("ACCESSION_NUMBER", "").strip()
series_id = row.get("SERIES_ID", "").strip()
if not cik or not acc:
continue
try:
c.execute("""
INSERT OR REPLACE INTO ncen_fund_info
(accession_number, cik, series_id, report_period, is_etf,
index_name, fund_type)
VALUES (?, ?, ?, ?, 1, ?, 'ETF')
""", (
acc, cik, series_id, quarter,
row.get("INDEX_NAME", "").strip(),
))
count += 1
except Exception as e:
log.debug("ETF insert error: %s", e)
log.info("Loaded %d ETF records from %s", count, quarter)
return count
def load_index_data(db: FundDatabase, extract_dir: Path, quarter: str):
"""Load index fund data from N-CEN."""
filepath = _find_tsv(extract_dir, "INDEX")
if not filepath.exists():
log.info("No INDEX file found in %s", extract_dir)
return 0
rows = _read_tsv(filepath)
count = 0
with db.conn() as c:
for row in rows:
cik = row.get("CIK", "").strip().zfill(10)
acc = row.get("ACCESSION_NUMBER", "").strip()
series_id = row.get("SERIES_ID", "").strip()
if not cik or not acc:
continue
try:
c.execute("""
INSERT OR REPLACE INTO ncen_fund_info
(accession_number, cik, series_id, report_period,
is_index_fund, index_name)
VALUES (?, ?, ?, ?, 1, ?)
ON CONFLICT(accession_number, series_id) DO UPDATE SET
is_index_fund=1,
index_name=COALESCE(NULLIF(excluded.index_name,''), ncen_fund_info.index_name)
""", (
acc, cik, series_id, quarter,
row.get("INDEX_NAME", row.get("NAME", "")).strip(),
))
count += 1
except Exception as e:
log.debug("Index insert error: %s", e)
log.info("Loaded %d index fund records from %s", count, quarter)
return count
def load_service_providers(db: FundDatabase, extract_dir: Path, quarter: str):
"""Load service provider data from various N-CEN tables."""
provider_files = {
"CUSTODIAN": "custodian",
"TRANSFER_AGENT": "transfer_agent",
"ADVISOR": "adviser",
"SUB_ADVISOR": "sub_adviser",
"ADMINISTRATOR": "administrator",
"AUDITOR": "auditor",
}
total = 0
for filename, role in provider_files.items():
filepath = _find_tsv(extract_dir, filename)
if not filepath.exists():
continue
rows = _read_tsv(filepath)
count = 0
with db.conn() as c:
for row in rows:
cik = row.get("CIK", "").strip().zfill(10)
name = (row.get("NAME", "") or row.get("COMPANY_NAME", "") or
row.get("CUSTODIAN_NAME", "") or row.get("FIRM_NAME", "")).strip()
lei = row.get("LEI", "").strip()
if not cik or not name:
continue
try:
c.execute("""
INSERT OR IGNORE INTO ncen_service_provider
(cik, report_period, provider_role, provider_name, provider_lei)
VALUES (?, ?, ?, ?, ?)
""", (cik, quarter, role, name, lei))
count += 1
except Exception as e:
log.debug("Service provider insert error: %s", e)
log.info(" %s: %d records", role, count)
total += count
log.info("Loaded %d total service provider records from %s", total, quarter)
return total
def main():
import argparse
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
parser = argparse.ArgumentParser(description="Load N-CEN data")
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("--quarters", nargs="+",
default=["2025q3"],
help="Quarters to download (e.g. 2025q3)")
parser.add_argument("--data-dir", default="data/ncen",
help="Directory for downloaded files")
args = parser.parse_args()
db = FundDatabase(args.db)
for quarter in args.quarters:
print(f"\n{'='*60}")
print(f"Processing N-CEN {quarter}")
print(f"{'='*60}")
try:
extract_dir = download_ncen_zip(quarter, args.data_dir)
load_etf_data(db, extract_dir, quarter)
load_index_data(db, extract_dir, quarter)
load_service_providers(db, extract_dir, quarter)
except Exception as e:
log.error("Failed to process %s: %s", quarter, e)
stats = db.get_stats()
print(f"\nDatabase stats:")
for table, count in stats.items():
if count > 0:
print(f" {table:30s} {count:>10,}")
if __name__ == "__main__":
main()

300
load_nport.py Normal file
View File

@ -0,0 +1,300 @@
"""
Step 4: Load N-PORT data (portfolio holdings, fund-level financials, returns).
Downloads quarterly N-PORT data set ZIPs from SEC, parses the TSV files,
and loads holdings, fund-level info, and monthly returns into the database.
"""
import csv
import logging
import time
import zipfile
from collections import defaultdict
from pathlib import Path
import requests
from tqdm import tqdm
from fund_db import FundDatabase
log = logging.getLogger(__name__)
USER_AGENT = "FundDataResearch/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12
_last_request_time = 0.0
NPORT_BASE_URL = "https://www.sec.gov/files/dera/data/form-n-port-data-sets"
def _throttled_get(url: str, **kwargs) -> requests.Response:
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 300)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
resp.raise_for_status()
return resp
def download_nport_zip(quarter: str, output_dir: str = "data/nport") -> Path:
"""Download a quarterly N-PORT ZIP from SEC."""
url = f"{NPORT_BASE_URL}/{quarter}_nport.zip"
out = Path(output_dir)
out.mkdir(parents=True, exist_ok=True)
zip_path = out / f"{quarter}_nport.zip"
extract_dir = out / quarter
if extract_dir.exists() and any(extract_dir.iterdir()):
log.info("Already extracted: %s", extract_dir)
return extract_dir
log.info("Downloading N-PORT data set: %s (this may take several minutes)", url)
resp = _throttled_get(url, stream=True)
total_size = int(resp.headers.get("content-length", 0))
with open(zip_path, "wb") as fp:
with tqdm(total=total_size, unit="B", unit_scale=True, desc=f"Downloading {quarter}") as pbar:
for chunk in resp.iter_content(chunk_size=65536):
fp.write(chunk)
pbar.update(len(chunk))
log.info("Extracting %s", zip_path)
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(extract_dir)
log.info("Extracted to %s", extract_dir)
return extract_dir
def _find_tsv(extract_dir: Path, pattern: str) -> Path:
"""Find a TSV file matching a pattern in the extract directory."""
candidates = list(extract_dir.rglob(f"*{pattern}*"))
if candidates:
return candidates[0]
return extract_dir / f"{pattern}.tsv"
def _read_tsv_streaming(filepath: Path, chunk_size: int = 10000):
"""Read a large TSV file in chunks, yielding lists of dicts."""
if not filepath.exists():
log.warning("File not found: %s", filepath)
return
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
reader = csv.DictReader(f, delimiter="\t")
chunk = []
for row in reader:
chunk.append(row)
if len(chunk) >= chunk_size:
yield chunk
chunk = []
if chunk:
yield chunk
def load_fund_reported_info(db: FundDatabase, extract_dir: Path, quarter: str):
"""Load the FUND_REPORTED_INFO table from N-PORT data."""
filepath = _find_tsv(extract_dir, "FUND_REPORTED_INFO")
if not filepath.exists():
log.warning("FUND_REPORTED_INFO not found in %s", extract_dir)
return 0
count = 0
with db.conn() as c:
for chunk in _read_tsv_streaming(filepath):
for row in chunk:
acc = row.get("ACCESSION_NUMBER", "").strip()
if not acc:
continue
try:
c.execute("""
INSERT OR IGNORE INTO nport_fund_info
(accession_number, cik, series_id, report_date,
total_assets, total_liabilities, net_assets)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (
acc,
row.get("CIK", "").strip().zfill(10),
row.get("SERIES_ID", "").strip(),
row.get("REPORT_DATE", "").strip(),
_to_float(row.get("TOTAL_ASSETS")),
_to_float(row.get("TOTAL_LIABILITIES")),
_to_float(row.get("NET_ASSETS")),
))
count += 1
except Exception as e:
log.debug("Fund info insert error: %s", e)
log.info("Loaded %d fund info records from %s", count, quarter)
return count
def load_monthly_returns(db: FundDatabase, extract_dir: Path, quarter: str):
"""Load monthly total returns from N-PORT data."""
filepath = _find_tsv(extract_dir, "MONTHLY_TOTAL_RETURN")
if not filepath.exists():
log.warning("MONTHLY_TOTAL_RETURN not found in %s", extract_dir)
return 0
count = 0
with db.conn() as c:
for chunk in _read_tsv_streaming(filepath):
for row in chunk:
acc = row.get("ACCESSION_NUMBER", "").strip()
if not acc:
continue
try:
c.execute("""
INSERT OR IGNORE INTO nport_monthly_return
(accession_number, cik, class_id, report_date,
month1_return, month2_return, month3_return)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (
acc,
row.get("CIK", "").strip().zfill(10),
row.get("CLASS_ID", "").strip(),
row.get("REPORT_DATE", "").strip(),
_to_float(row.get("MONTHLY_TOTAL_RETURN1")),
_to_float(row.get("MONTHLY_TOTAL_RETURN2")),
_to_float(row.get("MONTHLY_TOTAL_RETURN3")),
))
count += 1
except Exception as e:
log.debug("Monthly return insert error: %s", e)
log.info("Loaded %d monthly return records from %s", count, quarter)
return count
def load_holdings(db: FundDatabase, extract_dir: Path, quarter: str,
cik_filter: set = None):
"""
Load portfolio holdings from N-PORT data.
If cik_filter is provided, only load holdings for those CIKs.
"""
filepath = _find_tsv(extract_dir, "FUND_REPORTED_HOLDING")
if not filepath.exists():
log.warning("FUND_REPORTED_HOLDING not found in %s", extract_dir)
return 0
count = 0
batch = []
batch_size = 5000
for chunk in _read_tsv_streaming(filepath, chunk_size=10000):
for row in chunk:
cik = row.get("CIK", "").strip().zfill(10)
if cik_filter and cik not in cik_filter:
continue
holding = {
"accession_number": row.get("ACCESSION_NUMBER", "").strip(),
"cik": cik,
"report_date": row.get("REPORT_DATE", "").strip(),
"holding_name": row.get("NAME_OF_ISSUER", "").strip()[:200],
"lei": row.get("LEI", "").strip(),
"cusip": row.get("CUSIP", "").strip(),
"isin": row.get("ISIN", "").strip(),
"ticker": row.get("TICKER", "").strip(),
"asset_category": row.get("ASSET_CAT", "").strip(),
"issuer_category": row.get("ISSUER_CAT", "").strip(),
"inv_country": row.get("INV_COUNTRY", "").strip(),
"currency": row.get("CURRENCY_CODE", row.get("CUR_CD", "")).strip(),
"quantity": _to_float(row.get("BALANCE")),
"value_usd": _to_float(row.get("VAL_USD", row.get("VALUE_USD"))),
"pct_val": _to_float(row.get("PCT_VAL")),
"is_debt": 1 if row.get("ASSET_CAT", "").strip().startswith("D") else 0,
"coupon_rate": _to_float(row.get("COUPON_RATE")),
"maturity_date": row.get("MATURITY_DATE", "").strip(),
"is_default": _to_int(row.get("IS_DEFAULT")),
"fair_value_level": row.get("FAIR_VAL_LEVEL", "").strip(),
}
batch.append(holding)
count += 1
if len(batch) >= batch_size:
db.bulk_insert_holdings(batch)
batch = []
if batch:
db.bulk_insert_holdings(batch)
log.info("Loaded %d holdings from %s", count, quarter)
return count
def _to_float(val):
if val is None:
return None
val = str(val).strip()
if not val or val.lower() in ("", "n/a", "none"):
return None
try:
return float(val)
except (ValueError, TypeError):
return None
def _to_int(val):
if val is None:
return None
val = str(val).strip()
if not val or val.lower() in ("", "n/a", "none"):
return None
try:
return int(float(val))
except (ValueError, TypeError):
return None
def main():
import argparse
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
parser = argparse.ArgumentParser(description="Load N-PORT data")
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("--quarters", nargs="+",
default=["2025q3"],
help="Quarters to download (e.g. 2025q3 2025q4)")
parser.add_argument("--data-dir", default="data/nport",
help="Directory for downloaded files")
parser.add_argument("--skip-holdings", action="store_true",
help="Skip loading individual holdings (large)")
parser.add_argument("--holdings-cik-filter", action="store_true",
help="Only load holdings for CIKs already in DB")
args = parser.parse_args()
db = FundDatabase(args.db)
cik_filter = None
if args.holdings_cik_filter:
cik_filter = set(db.get_all_ciks())
log.info("Filtering holdings to %d CIKs in database", len(cik_filter))
for quarter in args.quarters:
print(f"\n{'='*60}")
print(f"Processing N-PORT {quarter}")
print(f"{'='*60}")
try:
extract_dir = download_nport_zip(quarter, args.data_dir)
load_fund_reported_info(db, extract_dir, quarter)
load_monthly_returns(db, extract_dir, quarter)
if not args.skip_holdings:
load_holdings(db, extract_dir, quarter, cik_filter=cik_filter)
except Exception as e:
log.error("Failed to process %s: %s", quarter, e)
stats = db.get_stats()
print(f"\nDatabase stats:")
for table, count in stats.items():
if count > 0:
print(f" {table:30s} {count:>10,}")
if __name__ == "__main__":
main()

353
load_xbrl_rr.py Normal file
View File

@ -0,0 +1,353 @@
"""
Step 3: Load XBRL Risk/Return Summary data into the database.
Downloads quarterly ZIP files from SEC containing structured fee,
performance, and objective data extracted from mutual fund prospectuses.
Parses the flat files (SUB, TAG, NUM, TXT) and loads structured records
into the xbrl_fee, xbrl_performance, and xbrl_objective tables.
"""
import csv
import io
import logging
import os
import time
import zipfile
from collections import defaultdict
from pathlib import Path
import requests
from tqdm import tqdm
from fund_db import FundDatabase
log = logging.getLogger(__name__)
USER_AGENT = "FundDataResearch/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12
_last_request_time = 0.0
XBRL_RR_BASE_URL = (
"https://www.sec.gov/files/dera/data/"
"mutual-fund-prospectus-risk/return-summary-data-sets"
)
FEE_TAGS = {
"MaximumSalesChargeImposedOnPurchasesOverOfferingPrice": "max_sales_charge_pct",
"MaximumDeferredSalesChargeOverOther": "max_deferred_charge_pct",
"RedemptionFeeOverRedemption": "redemption_fee_pct",
"ManagementFeesOverAssets": "management_fee_pct",
"Distribution12b1FeesOverAssets": "dist_12b1_fee_pct",
"OtherExpensesOverAssets": "other_expenses_pct",
"AcquiredFundFeesAndExpensesOverAssets": "acquired_fund_fees_pct",
"TotalAnnualFundOperatingExpensesOverAssets": "total_expenses_pct",
"FeeWaiverOrReimbursementOverAssets": "fee_waiver_pct",
"TotalAnnualFundOperatingExpensesAfterFeeWaiverOverAssets": "net_expenses_pct",
"ExpenseExampleYear01": "expense_example_1yr",
"ExpenseExampleYear03": "expense_example_3yr",
"ExpenseExampleYear05": "expense_example_5yr",
"ExpenseExampleYear10": "expense_example_10yr",
}
PERFORMANCE_TAGS = {
"AverageAnnualReturnYear01": "return_year_1",
"AverageAnnualReturnYear05": "return_year_5",
"AverageAnnualReturnYear10": "return_year_10",
"AverageAnnualReturnSinceInception": "return_since_incep",
"HighestQuarterlyReturnValue": "best_quarter_return",
"LowestQuarterlyReturnValue": "worst_quarter_return",
"AnnualTurnover": "portfolio_turnover",
}
PERFORMANCE_TEXT_TAGS = {
"HighestQuarterlyReturnLabel": "best_quarter_label",
"LowestQuarterlyReturnLabel": "worst_quarter_label",
"ShareClassInceptionDate": "inception_date",
}
OBJECTIVE_TEXT_TAGS = {
"ObjectivePrimaryTextBlock": "objective_text",
"StrategyNarrativeTextBlock": "strategy_text",
"RiskNarrativeTextBlock": "risk_text",
}
def _throttled_get(url: str, **kwargs) -> requests.Response:
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 120)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
resp.raise_for_status()
return resp
def download_xbrl_rr_zip(quarter: str, output_dir: str = "data/xbrl_rr") -> Path:
"""Download a quarterly XBRL Risk/Return ZIP from SEC."""
url = f"{XBRL_RR_BASE_URL}/{quarter}_rr1.zip"
out = Path(output_dir)
out.mkdir(parents=True, exist_ok=True)
zip_path = out / f"{quarter}_rr1.zip"
extract_dir = out / quarter
if extract_dir.exists() and any(extract_dir.iterdir()):
log.info("Already extracted: %s", extract_dir)
return extract_dir
log.info("Downloading XBRL Risk/Return: %s", url)
resp = _throttled_get(url, stream=True)
with open(zip_path, "wb") as fp:
for chunk in resp.iter_content(chunk_size=65536):
fp.write(chunk)
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(extract_dir)
log.info("Extracted to %s", extract_dir)
return extract_dir
def _read_tsv(filepath: Path) -> list[dict]:
"""Read a TSV file into a list of dicts."""
if not filepath.exists():
log.warning("File not found: %s", filepath)
return []
with open(filepath, "r", encoding="utf-8", errors="replace") as f:
reader = csv.DictReader(f, delimiter="\t")
return list(reader)
def parse_and_load_xbrl_rr(db: FundDatabase, extract_dir: Path, quarter: str):
"""
Parse the XBRL Risk/Return flat files and load into the database.
The data set has:
- sub.tsv: submission metadata (accession, CIK, filing date, etc.)
- num.tsv: numeric values (tag, value, per accession + dimension)
- txt.tsv: text values (tag, value, per accession + dimension)
"""
sub_file = extract_dir / "sub.tsv"
num_file = extract_dir / "num.tsv"
txt_file = extract_dir / "txt.tsv"
for f in [sub_file, num_file]:
if not f.exists():
sub_file = next(extract_dir.glob("*sub*"), None)
num_file = next(extract_dir.glob("*num*"), None)
txt_file = next(extract_dir.glob("*txt*"), None)
break
if not sub_file or not sub_file.exists():
log.error("Cannot find sub file in %s", extract_dir)
return
log.info("Reading submission metadata from %s", sub_file)
subs = _read_tsv(sub_file)
sub_map = {}
for s in subs:
adsh = s.get("adsh", "").strip()
if adsh:
sub_map[adsh] = {
"cik": str(s.get("cik", "")).strip().zfill(10),
"filing_date": s.get("filed", "").strip(),
"form": s.get("form", "").strip(),
"name": s.get("name", "").strip(),
}
log.info("Reading numeric data from %s", num_file)
nums = _read_tsv(num_file) if num_file and num_file.exists() else []
fee_data = defaultdict(dict)
perf_data = defaultdict(dict)
for row in tqdm(nums, desc="Parsing numeric data"):
adsh = row.get("adsh", "").strip()
tag = row.get("tag", "").strip()
value_str = row.get("value", "").strip()
if not adsh or not tag or not value_str:
continue
try:
value = float(value_str)
except (ValueError, TypeError):
continue
sub_info = sub_map.get(adsh, {})
cik = sub_info.get("cik", "")
key = (adsh, cik)
if tag in FEE_TAGS:
fee_data[key][FEE_TAGS[tag]] = value
fee_data[key]["_sub"] = sub_info
elif tag in PERFORMANCE_TAGS:
perf_data[key][PERFORMANCE_TAGS[tag]] = value
perf_data[key]["_sub"] = sub_info
log.info("Reading text data from %s", txt_file)
txts = _read_tsv(txt_file) if txt_file and txt_file.exists() else []
obj_data = defaultdict(dict)
for row in tqdm(txts, desc="Parsing text data"):
adsh = row.get("adsh", "").strip()
tag = row.get("tag", "").strip()
value = row.get("value", "").strip()
if not adsh or not tag or not value:
continue
sub_info = sub_map.get(adsh, {})
cik = sub_info.get("cik", "")
key = (adsh, cik)
if tag in PERFORMANCE_TEXT_TAGS:
perf_data[key][PERFORMANCE_TEXT_TAGS[tag]] = value
perf_data[key]["_sub"] = sub_info
elif tag in OBJECTIVE_TEXT_TAGS:
obj_data[key][OBJECTIVE_TEXT_TAGS[tag]] = value
obj_data[key]["_sub"] = sub_info
fee_count = 0
perf_count = 0
obj_count = 0
with db.conn() as c:
for (adsh, cik), vals in fee_data.items():
sub_info = vals.pop("_sub", {})
try:
c.execute("""
INSERT OR REPLACE INTO xbrl_fee
(accession_number, cik, filing_date, fund_name,
max_sales_charge_pct, max_deferred_charge_pct,
redemption_fee_pct, management_fee_pct,
dist_12b1_fee_pct, other_expenses_pct,
acquired_fund_fees_pct, total_expenses_pct,
fee_waiver_pct, net_expenses_pct,
expense_example_1yr, expense_example_3yr,
expense_example_5yr, expense_example_10yr)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
adsh, cik, sub_info.get("filing_date", ""),
sub_info.get("name", ""),
vals.get("max_sales_charge_pct"),
vals.get("max_deferred_charge_pct"),
vals.get("redemption_fee_pct"),
vals.get("management_fee_pct"),
vals.get("dist_12b1_fee_pct"),
vals.get("other_expenses_pct"),
vals.get("acquired_fund_fees_pct"),
vals.get("total_expenses_pct"),
vals.get("fee_waiver_pct"),
vals.get("net_expenses_pct"),
vals.get("expense_example_1yr"),
vals.get("expense_example_3yr"),
vals.get("expense_example_5yr"),
vals.get("expense_example_10yr"),
))
fee_count += 1
except Exception as e:
log.debug("Fee insert error for %s: %s", adsh, e)
for (adsh, cik), vals in perf_data.items():
sub_info = vals.pop("_sub", {})
try:
c.execute("""
INSERT OR REPLACE INTO xbrl_performance
(accession_number, cik, filing_date, fund_name,
inception_date,
return_year_1, return_year_5, return_year_10,
return_since_incep,
best_quarter_return, best_quarter_label,
worst_quarter_return, worst_quarter_label,
portfolio_turnover)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""", (
adsh, cik, sub_info.get("filing_date", ""),
sub_info.get("name", ""),
vals.get("inception_date"),
vals.get("return_year_1"),
vals.get("return_year_5"),
vals.get("return_year_10"),
vals.get("return_since_incep"),
vals.get("best_quarter_return"),
vals.get("best_quarter_label"),
vals.get("worst_quarter_return"),
vals.get("worst_quarter_label"),
vals.get("portfolio_turnover"),
))
perf_count += 1
except Exception as e:
log.debug("Performance insert error for %s: %s", adsh, e)
for (adsh, cik), vals in obj_data.items():
sub_info = vals.pop("_sub", {})
try:
c.execute("""
INSERT OR REPLACE INTO xbrl_objective
(accession_number, cik, filing_date, fund_name,
objective_text, strategy_text, risk_text)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (
adsh, cik, sub_info.get("filing_date", ""),
sub_info.get("name", ""),
vals.get("objective_text"),
vals.get("strategy_text"),
vals.get("risk_text"),
))
obj_count += 1
except Exception as e:
log.debug("Objective insert error for %s: %s", adsh, e)
log.info("Loaded %d fee records, %d performance records, %d objective records",
fee_count, perf_count, obj_count)
db.record_bulk_download("xbrl_rr", quarter, str(extract_dir), fee_count + perf_count + obj_count)
return fee_count, perf_count, obj_count
def main():
import argparse
logging.basicConfig(level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s")
parser = argparse.ArgumentParser(description="Load XBRL Risk/Return data")
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("--quarters", nargs="+",
default=["2025q1", "2025q2", "2025q3", "2025q4"],
help="Quarters to download (e.g. 2025q1 2025q2)")
parser.add_argument("--data-dir", default="data/xbrl_rr",
help="Directory for downloaded files")
args = parser.parse_args()
db = FundDatabase(args.db)
total_fees = 0
total_perf = 0
total_obj = 0
for quarter in args.quarters:
print(f"\n{'='*60}")
print(f"Processing {quarter}")
print(f"{'='*60}")
try:
extract_dir = download_xbrl_rr_zip(quarter, args.data_dir)
f, p, o = parse_and_load_xbrl_rr(db, extract_dir, quarter)
total_fees += f
total_perf += p
total_obj += o
except Exception as e:
log.error("Failed to process %s: %s", quarter, e)
print(f"\nTotal loaded: {total_fees} fee records, {total_perf} performance, {total_obj} objectives")
stats = db.get_stats()
print(f"\nDatabase stats:")
for table, count in stats.items():
if count > 0:
print(f" {table:30s} {count:>10,}")
if __name__ == "__main__":
main()

390
pipeline.py Normal file
View File

@ -0,0 +1,390 @@
#!/usr/bin/env python3
"""
SEC Fund Data Pipeline — Main Orchestrator
Builds a database of 10,000+ SEC-registered funds with:
1. Fund universe (trusts, series, share classes) from Series/Class CSV
2. Prospectus filings and text from EDGAR
3. Structured fee/performance data from XBRL Risk/Return
4. Portfolio holdings from N-PORT
5. Service provider and classification data from N-CEN
Usage:
python pipeline.py status # Show database stats
python pipeline.py universe # Load fund universe (step 1)
python pipeline.py filings --limit 100 # Fetch filings (step 2)
python pipeline.py xbrl-rr # Load XBRL fee data (step 3)
python pipeline.py nport # Load N-PORT data (step 4)
python pipeline.py ncen # Load N-CEN data (step 5)
python pipeline.py all # Run all steps
python pipeline.py search "Vanguard" # Search the database
python pipeline.py export # Export summary CSV
"""
import argparse
import csv
import logging
import sys
from pathlib import Path
from fund_db import FundDatabase
log = logging.getLogger(__name__)
def cmd_status(db: FundDatabase, args):
"""Show database statistics."""
stats = db.get_stats()
print("\n" + "=" * 65)
print(" SEC Fund Data Pipeline — Database Status")
print("=" * 65)
sections = [
("Fund Universe", [("trust", "Trusts (investment companies)"),
("series", "Series (funds)"),
("share_class", "Share Classes")]),
("Filings", [("filing", "Filing records"),
("filing_text", "Filing texts (plain)"),
("filing_html", "Filing HTML (raw)")]),
("XBRL Risk/Return", [("xbrl_fee", "Fee records"),
("xbrl_performance", "Performance records"),
("xbrl_objective", "Objective/strategy texts")]),
("N-PORT", [("nport_fund_info", "Fund snapshots"),
("nport_holding", "Portfolio holdings"),
("nport_monthly_return", "Monthly returns")]),
("N-CEN", [("ncen_fund_info", "Fund classification records"),
("ncen_service_provider", "Service provider records")]),
]
for section_name, tables in sections:
print(f"\n {section_name}:")
for table, label in tables:
count = stats.get(table, 0)
bar = "█" * min(count // 1000, 40)
print(f" {label:40s} {count:>10,} {bar}")
with db.conn() as c:
row = c.execute("""
SELECT COUNT(DISTINCT cik) as n FROM filing
WHERE form_type IN ('485BPOS','485APOS','N-1A')
""").fetchone()
print(f"\n Trusts with prospectus filings: {row['n']:>10,}")
row = c.execute("""
SELECT COUNT(DISTINCT cik) as n FROM xbrl_fee
""").fetchone()
print(f" Trusts with XBRL fee data: {row['n']:>10,}")
row = c.execute("""
SELECT COUNT(DISTINCT cik) as n FROM nport_fund_info
""").fetchone()
print(f" Trusts with N-PORT data: {row['n']:>10,}")
row = c.execute("SELECT COUNT(*) as n FROM pipeline_status WHERE status='error'").fetchone()
if row["n"] > 0:
print(f"\n Pipeline errors: {row['n']:>10,}")
print()
def cmd_universe(db: FundDatabase, args):
"""Load fund universe from SEC Series/Class CSV."""
from fetch_universe import load_series_class_csv, enrich_from_submissions_api
t, s, c = load_series_class_csv(db)
print(f"\nUniverse: {t:,} trusts, {s:,} series, {c:,} share classes")
if args.enrich:
n = enrich_from_submissions_api(db, limit=args.enrich_limit)
print(f"Enriched {n:,} trusts from Submissions API")
def cmd_filings(db: FundDatabase, args):
"""Fetch filings for the fund universe."""
from fetch_filings import fetch_filings_for_universe, download_pending_texts, backfill_html
if getattr(args, "backfill_html", False):
backfill_html(db, limit=args.limit)
return
if args.text_only:
download_pending_texts(db, limit=args.limit)
else:
ciks = [c.zfill(10) for c in args.ciks] if args.ciks else None
fetch_filings_for_universe(
db, ciks=ciks, limit=args.limit,
download_text=not args.no_text,
max_prospectus=args.max_prospectus,
max_supplements=args.max_supplements,
)
def cmd_xbrl_rr(db: FundDatabase, args):
"""Load XBRL Risk/Return structured data."""
from load_xbrl_rr import download_xbrl_rr_zip, parse_and_load_xbrl_rr
for quarter in args.quarters:
print(f"\nProcessing XBRL Risk/Return {quarter}...")
try:
extract_dir = download_xbrl_rr_zip(quarter, args.data_dir)
parse_and_load_xbrl_rr(db, extract_dir, quarter)
except Exception as e:
log.error("Failed %s: %s", quarter, e)
print(f" ERROR: {e}")
def cmd_nport(db: FundDatabase, args):
"""Load N-PORT data."""
from load_nport import (download_nport_zip, load_fund_reported_info,
load_monthly_returns, load_holdings)
cik_filter = None
if args.filter_ciks:
cik_filter = set(db.get_all_ciks())
for quarter in args.quarters:
print(f"\nProcessing N-PORT {quarter}...")
try:
extract_dir = download_nport_zip(quarter, args.data_dir)
load_fund_reported_info(db, extract_dir, quarter)
load_monthly_returns(db, extract_dir, quarter)
if not args.skip_holdings:
load_holdings(db, extract_dir, quarter, cik_filter=cik_filter)
except Exception as e:
log.error("Failed %s: %s", quarter, e)
print(f" ERROR: {e}")
def cmd_ncen(db: FundDatabase, args):
"""Load N-CEN data."""
from load_ncen import (download_ncen_zip, load_etf_data,
load_index_data, load_service_providers)
for quarter in args.quarters:
print(f"\nProcessing N-CEN {quarter}...")
try:
extract_dir = download_ncen_zip(quarter, args.data_dir)
load_etf_data(db, extract_dir, quarter)
load_index_data(db, extract_dir, quarter)
load_service_providers(db, extract_dir, quarter)
except Exception as e:
log.error("Failed %s: %s", quarter, e)
print(f" ERROR: {e}")
def cmd_all(db: FundDatabase, args):
"""Run all pipeline steps."""
print("\n" + "=" * 65)
print(" Step 1: Loading fund universe")
print("=" * 65)
cmd_universe(db, args)
print("\n" + "=" * 65)
print(" Step 2: Fetching filings")
print("=" * 65)
cmd_filings(db, args)
print("\n" + "=" * 65)
print(" Step 3: Loading XBRL Risk/Return data")
print("=" * 65)
cmd_xbrl_rr(db, args)
print("\n" + "=" * 65)
print(" Step 4: Loading N-PORT data")
print("=" * 65)
cmd_nport(db, args)
print("\n" + "=" * 65)
print(" Step 5: Loading N-CEN data")
print("=" * 65)
cmd_ncen(db, args)
print("\n" + "=" * 65)
print(" Pipeline complete!")
print("=" * 65)
cmd_status(db, args)
def cmd_search(db: FundDatabase, args):
"""Search the database for funds."""
query = " ".join(args.query) if args.query else ""
if not query:
print("Usage: pipeline.py search <query>")
return
results = db.search_funds(query, limit=args.limit)
if not results:
print(f"No results for '{query}'")
return
print(f"\nSearch results for '{query}' ({len(results)} matches):\n")
print(f"{'CIK':>12s} {'Ticker':>8s} {'Series Name':50s} {'Class':20s}")
print("-" * 95)
for r in results:
print(f"{r['cik']:>12s} {(r['ticker'] or ''):>8s} "
f"{(r['series_name'] or '')[:50]:50s} {(r['class_name'] or '')[:20]:20s}")
def cmd_export(db: FundDatabase, args):
"""Export a summary CSV of all funds with available data."""
output_path = args.output_csv or "fund_summary.csv"
with db.conn() as c:
rows = c.execute("""
SELECT
t.cik,
t.trust_name,
t.file_number,
t.state_of_inc,
t.fiscal_year_end,
s.series_id,
s.series_name,
sc.class_id,
sc.class_name,
sc.ticker,
(SELECT COUNT(*) FROM filing f WHERE f.cik = t.cik
AND f.form_type IN ('485BPOS','485APOS','N-1A')) as prospectus_count,
(SELECT COUNT(*) FROM filing_text ft
JOIN filing f2 ON ft.accession_number = f2.accession_number
WHERE f2.cik = t.cik) as texts_downloaded,
(SELECT xf.net_expenses_pct FROM xbrl_fee xf
WHERE xf.cik = t.cik ORDER BY xf.filing_date DESC LIMIT 1) as latest_expense_ratio,
(SELECT xf.management_fee_pct FROM xbrl_fee xf
WHERE xf.cik = t.cik ORDER BY xf.filing_date DESC LIMIT 1) as latest_mgmt_fee,
(SELECT ni.net_assets FROM nport_fund_info ni
WHERE ni.cik = t.cik ORDER BY ni.report_date DESC LIMIT 1) as latest_net_assets,
(SELECT GROUP_CONCAT(DISTINCT nsp.provider_name)
FROM ncen_service_provider nsp
WHERE nsp.cik = t.cik AND nsp.provider_role = 'custodian') as custodians,
(SELECT GROUP_CONCAT(DISTINCT nsp.provider_name)
FROM ncen_service_provider nsp
WHERE nsp.cik = t.cik AND nsp.provider_role = 'adviser') as advisers
FROM share_class sc
JOIN series s ON sc.series_id = s.series_id
JOIN trust t ON s.cik = t.cik
ORDER BY t.cik, s.series_id, sc.class_id
""").fetchall()
if not rows:
print("No data to export. Run the pipeline first.")
return
fieldnames = rows[0].keys()
with open(output_path, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for row in rows:
writer.writerow(dict(row))
print(f"Exported {len(rows):,} records to {output_path}")
def main():
parser = argparse.ArgumentParser(
description="SEC Fund Data Pipeline",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument("--db", default="fund_data.db", help="Database path")
parser.add_argument("-v", "--verbose", action="store_true")
subparsers = parser.add_subparsers(dest="command", help="Pipeline command")
# status
subparsers.add_parser("status", help="Show database statistics")
# universe
p_univ = subparsers.add_parser("universe", help="Load fund universe from CSV")
p_univ.add_argument("--enrich", action="store_true",
help="Also enrich from Submissions API")
p_univ.add_argument("--enrich-limit", type=int, default=0)
# filings
p_filings = subparsers.add_parser("filings", help="Fetch prospectus filings")
p_filings.add_argument("--limit", type=int, default=0,
help="Max CIKs to process")
p_filings.add_argument("--ciks", nargs="*")
p_filings.add_argument("--max-prospectus", type=int, default=3)
p_filings.add_argument("--max-supplements", type=int, default=3)
p_filings.add_argument("--no-text", action="store_true")
p_filings.add_argument("--text-only", action="store_true")
p_filings.add_argument("--backfill-html", action="store_true",
help="Re-download raw HTML for filings missing it")
# xbrl-rr
p_xbrl = subparsers.add_parser("xbrl-rr", help="Load XBRL Risk/Return data")
p_xbrl.add_argument("--quarters", nargs="+",
default=["2025q1", "2025q2", "2025q3", "2025q4"])
p_xbrl.add_argument("--data-dir", default="data/xbrl_rr")
# nport
p_nport = subparsers.add_parser("nport", help="Load N-PORT data")
p_nport.add_argument("--quarters", nargs="+", default=["2025q3"])
p_nport.add_argument("--data-dir", default="data/nport")
p_nport.add_argument("--skip-holdings", action="store_true")
p_nport.add_argument("--filter-ciks", action="store_true",
help="Only load holdings for CIKs in DB")
# ncen
p_ncen = subparsers.add_parser("ncen", help="Load N-CEN data")
p_ncen.add_argument("--quarters", nargs="+", default=["2025q3"])
p_ncen.add_argument("--data-dir", default="data/ncen")
# all
p_all = subparsers.add_parser("all", help="Run full pipeline")
p_all.add_argument("--limit", type=int, default=100,
help="Max CIKs for filing fetch (default 100)")
p_all.add_argument("--enrich", action="store_true", default=False)
p_all.add_argument("--enrich-limit", type=int, default=0)
p_all.add_argument("--ciks", nargs="*")
p_all.add_argument("--max-prospectus", type=int, default=3)
p_all.add_argument("--max-supplements", type=int, default=3)
p_all.add_argument("--no-text", action="store_true")
p_all.add_argument("--text-only", action="store_true", default=False)
p_all.add_argument("--quarters", nargs="+", default=["2025q3"])
p_all.add_argument("--data-dir", default="data")
p_all.add_argument("--skip-holdings", action="store_true")
p_all.add_argument("--filter-ciks", action="store_true", default=True)
# search
p_search = subparsers.add_parser("search", help="Search funds")
p_search.add_argument("query", nargs="*")
p_search.add_argument("--limit", type=int, default=50)
# export
p_export = subparsers.add_parser("export", help="Export summary CSV")
p_export.add_argument("--output-csv", default="fund_summary.csv")
args = parser.parse_args()
log_level = logging.DEBUG if args.verbose else logging.INFO
logging.basicConfig(level=log_level,
format="%(asctime)s [%(levelname)s] %(message)s")
if not args.command:
parser.print_help()
return
db = FundDatabase(args.db)
commands = {
"status": cmd_status,
"universe": cmd_universe,
"filings": cmd_filings,
"xbrl-rr": cmd_xbrl_rr,
"nport": cmd_nport,
"ncen": cmd_ncen,
"all": cmd_all,
"search": cmd_search,
"export": cmd_export,
}
handler = commands.get(args.command)
if handler:
handler(db, args)
else:
parser.print_help()
if __name__ == "__main__":
main()

6
requirements.txt Normal file
View File

@ -0,0 +1,6 @@
requests>=2.31.0
beautifulsoup4>=4.12.0
lxml>=5.1.0
pandas>=2.2.0
tqdm>=4.66.0
# SQLite3 is part of Python stdlib — no extra package needed

155
score_baseline.py Normal file
View File

@ -0,0 +1,155 @@
#!/usr/bin/env python3
"""
Baseline scorer for the SEC prospectus -> RDF triple PoC.
Compares a system's predicted triples against the N-CEN gold graph and reports
triple-level precision / recall / F1 (the metric used in the thesis).
Two intended uses:
1. NON-MODEL baseline -- the gold itself is from N-CEN; this script also runs a
trivial string-match baseline that scans the prospectus prose for each gold
entity label and emits the edge if the object's name appears in the text.
This gives a "no-LLM" lower bound: how many edges are even surfaceable by
naive string matching, and how the prose supports each relation type.
2. STRONG-MODEL baseline -- point --pred at a JSONL of model predictions
(one obj per line: {"cik": ..., "triples": [{"s","p","o"}, ...]}) to score
GPT-4 / Opus extractions against the same gold.
Triple matching is on (subject_type, predicate, normalized_object_label) so that
IRI slug differences do not cause false negatives; object org names are normalized
(lowercased, legal suffixes stripped) before comparison.
Usage:
python score_baseline.py stringmatch # no-model baseline over the PoC samples
python score_baseline.py model --pred preds.jsonl
"""
import argparse
import json
import re
from collections import defaultdict
from pathlib import Path
SAMPLES = Path("data/rdf_poc/samples.jsonl")
GOLD = Path("data/rdf_poc/gold_graphs.jsonl")
_SUFFIX = re.compile(
r"\b(llc|l\.l\.c|inc|inc\.|incorporated|corp|corporation|company|co|co\.|"
r"ltd|limited|lp|l\.p|llp|na|n\.a|trust|the)\b", re.I)
def norm(name: str) -> str:
s = (name or "").lower()
s = _SUFFIX.sub(" ", s)
s = re.sub(r"[^a-z0-9]+", " ", s)
return re.sub(r"\s+", " ", s).strip()
def triple_key(t, entities):
"""Match key: (subject_type, predicate, normalized object label)."""
s_type = entities.get(t["s"], {}).get("type", "")
o_label = entities.get(t["o"], {}).get("label", t["o"])
return (s_type, t["p"], norm(o_label))
def score(gold_triples, pred_triples, entities):
g = {triple_key(t, entities) for t in gold_triples}
p = {triple_key(t, entities) for t in pred_triples}
tp = len(g & p)
prec = tp / len(p) if p else 0.0
rec = tp / len(g) if g else 0.0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
return tp, len(p), len(g), prec, rec, f1
def stringmatch_predict(sample):
"""No-model baseline: emit a gold edge iff the object label appears in the prose."""
text_norm = norm(sample["input_text"])
ents = sample.get("entities") or {}
# samples.jsonl doesn't carry entities; rebuild a minimal label map from gold
preds = []
for t in sample["target_triples"]:
o_label = sample["_entities"].get(t["o"], {}).get("label", "")
if o_label and norm(o_label) and norm(o_label) in text_norm:
preds.append(t)
return preds
def load_samples_with_entities():
gold = {json.loads(l)["cik"]: json.loads(l) for l in open(GOLD, encoding="utf-8")}
out = []
for l in open(SAMPLES, encoding="utf-8"):
s = json.loads(l)
g = gold.get(s["cik"], {})
s["_entities"] = g.get("entities", {})
out.append(s)
return out
def run_stringmatch():
samples = load_samples_with_entities()
agg = defaultdict(lambda: [0, 0, 0]) # predicate -> [tp, npred, ngold]
micro = [0, 0, 0]
print(f"{'CIK':<12}{'trust':<34}{'P':>6}{'R':>6}{'F1':>6} (string-match, no model)")
print("-" * 76)
for s in samples:
ents = s["_entities"]
gold_t = s["target_triples"]
pred_t = stringmatch_predict(s)
tp, np_, ng, prec, rec, f1 = score(gold_t, pred_t, ents)
micro[0] += tp; micro[1] += np_; micro[2] += ng
for t in gold_t:
agg[t["p"]][2] += 1
gk = {triple_key(t, ents) for t in gold_t}
for t in pred_t:
agg[t["p"]][1] += 1
if triple_key(t, ents) in gk:
agg[t["p"]][0] += 1
print(f"{s['cik']:<12}{s['trust_name'][:32]:<34}{prec:>6.2f}{rec:>6.2f}{f1:>6.2f}")
tp, np_, ng = micro
P = tp / np_ if np_ else 0
R = tp / ng if ng else 0
F = 2 * P * R / (P + R) if (P + R) else 0
print("-" * 76)
print(f"MICRO over {len(samples)} samples: P={P:.3f} R={R:.3f} F1={F:.3f} (tp={tp}, pred={np_}, gold={ng})")
print("\nPer-relation recall of the no-model string-match baseline:")
print(f" {'relation':<16}{'recall':>8}{'gold':>8} (how prose-grounded each edge type is)")
for p, (t, npd, ngd) in sorted(agg.items(), key=lambda x: -x[1][2]):
r = t / ngd if ngd else 0
print(f" {p:<16}{r:>8.2f}{ngd:>8}")
def run_model(pred_path):
samples = load_samples_with_entities()
preds = defaultdict(list)
for l in open(pred_path, encoding="utf-8"):
r = json.loads(l)
preds[r["cik"]] = r.get("triples", [])
micro = [0, 0, 0]
for s in samples:
ents = s["_entities"]
tp, np_, ng, *_ = score(s["target_triples"], preds.get(s["cik"], []), ents)
micro[0] += tp; micro[1] += np_; micro[2] += ng
tp, np_, ng = micro
P = tp / np_ if np_ else 0
R = tp / ng if ng else 0
F = 2 * P * R / (P + R) if (P + R) else 0
print(f"MODEL baseline: P={P:.3f} R={R:.3f} F1={F:.3f} (tp={tp}, pred={np_}, gold={ng})")
def main():
ap = argparse.ArgumentParser()
sub = ap.add_subparsers(dest="cmd")
sub.add_parser("stringmatch")
m = sub.add_parser("model"); m.add_argument("--pred", required=True)
args = ap.parse_args()
if args.cmd == "stringmatch":
run_stringmatch()
elif args.cmd == "model":
run_model(args.pred)
else:
ap.print_help()
if __name__ == "__main__":
main()

537
sec_fund_fetcher.py Normal file
View File

@ -0,0 +1,537 @@
"""
SEC Fund Prospectus & Reference Data Fetcher
Fetches prospectuses (485BPOS, 497, 497K), supplements, and amendments
for US SEC-registered investment funds from EDGAR. Pairs the legal
documents with structured reference data (series/class identifiers,
tickers, CUSIPs) to build an LLM training dataset.
Data sources:
1. EDGAR Submissions API — filing history per CIK
2. EDGAR Full-Text Search — search filings by form type
3. EDGAR Archives — download actual filing documents
4. SEC Series/Class CSV — fund & share-class reference data
5. XBRL Risk/Return Datasets — structured prospectus extracts
"""
import csv
import io
import json
import logging
import os
import re
import time
import zipfile
from dataclasses import dataclass, field, asdict
from pathlib import Path
from typing import Optional
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
from tqdm import tqdm
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
)
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Configuration
# ---------------------------------------------------------------------------
SEC_BASE = "https://www.sec.gov"
DATA_SEC = "https://data.sec.gov"
EFTS_SEC = "https://efts.sec.gov/LATEST/search-index"
ARCHIVES = f"{SEC_BASE}/Archives/edgar/data"
SERIES_CLASS_CSV_URL = (
"https://www.sec.gov/files/investment/data/other/"
"investment-company-series-class-information/"
"investment-company-series-class-2025.csv"
)
PROSPECTUS_FORM_TYPES = {"485BPOS", "485APOS", "497", "497K", "N-1A"}
USER_AGENT = "SECFundFetcher/1.0 research@university.edu"
HEADERS = {"User-Agent": USER_AGENT, "Accept-Encoding": "gzip, deflate"}
REQUEST_INTERVAL = 0.12 # ~8 req/s to stay under SEC's 10/s limit
# ---------------------------------------------------------------------------
# Data classes
# ---------------------------------------------------------------------------
@dataclass
class ShareClass:
class_id: str # C######
class_name: str
ticker: str = ""
cusip: str = ""
@dataclass
class FundSeries:
series_id: str # S######
series_name: str
classes: list[ShareClass] = field(default_factory=list)
@dataclass
class FundTrust:
cik: str # 10-digit, zero-padded
trust_name: str
file_number: str = ""
series: list[FundSeries] = field(default_factory=list)
@dataclass
class Filing:
accession_number: str
form_type: str
filing_date: str
primary_document: str
description: str = ""
document_url: str = ""
text_content: str = ""
@dataclass
class FundDataset:
"""One record in the final dataset: trust + series + classes + filings."""
trust: FundTrust
prospectus_filings: list[Filing] = field(default_factory=list)
supplement_filings: list[Filing] = field(default_factory=list)
# ---------------------------------------------------------------------------
# SEC API helpers
# ---------------------------------------------------------------------------
_last_request_time = 0.0
def _throttled_get(url: str, **kwargs) -> requests.Response:
"""GET with rate-limiting and proper User-Agent."""
global _last_request_time
elapsed = time.time() - _last_request_time
if elapsed < REQUEST_INTERVAL:
time.sleep(REQUEST_INTERVAL - elapsed)
kwargs.setdefault("headers", {}).update(HEADERS)
kwargs.setdefault("timeout", 30)
resp = requests.get(url, **kwargs)
_last_request_time = time.time()
if resp.status_code == 403:
log.warning("403 Forbidden for %s — check User-Agent header", url)
resp.raise_for_status()
return resp
# ---------------------------------------------------------------------------
# 1. Load the SEC Series/Class reference data (CSV)
# ---------------------------------------------------------------------------
def load_series_class_reference(csv_url: str = SERIES_CLASS_CSV_URL) -> dict[str, FundTrust]:
"""
Download the SEC Investment Company Series and Class CSV.
Returns a dict keyed by CIK -> FundTrust with nested series/classes.
Actual CSV columns (2025):
Reporting File Number, CIK Number, Entity Name, Entity Org Type,
Series ID, Series Name, Class ID, Class Name, Class Ticker,
Address_1, Address_2, City, State, Zip Code
"""
log.info("Downloading series/class reference CSV …")
resp = _throttled_get(csv_url)
resp.encoding = "utf-8"
reader = csv.DictReader(io.StringIO(resp.text))
trusts: dict[str, FundTrust] = {}
for row in reader:
cik = row.get("CIK Number", "").strip().zfill(10)
if not cik or cik == "0" * 10:
continue
if cik not in trusts:
trusts[cik] = FundTrust(
cik=cik,
trust_name=row.get("Entity Name", "").strip(),
file_number=row.get("Reporting File Number", "").strip(),
)
trust = trusts[cik]
series_id = row.get("Series ID", "").strip()
if not series_id:
continue
existing_series = {s.series_id: s for s in trust.series}
if series_id not in existing_series:
series = FundSeries(
series_id=series_id,
series_name=row.get("Series Name", "").strip(),
)
trust.series.append(series)
existing_series[series_id] = series
series = existing_series[series_id]
class_id = row.get("Class ID", "").strip()
if class_id and not any(c.class_id == class_id for c in series.classes):
series.classes.append(ShareClass(
class_id=class_id,
class_name=row.get("Class Name", "").strip(),
ticker=row.get("Class Ticker", "").strip(),
))
log.info("Loaded %d investment company trusts from CSV", len(trusts))
return trusts
# ---------------------------------------------------------------------------
# 2. Fetch filing history for a CIK via the Submissions API
# ---------------------------------------------------------------------------
def fetch_submissions(cik: str) -> dict:
"""Return the full JSON from data.sec.gov/submissions/CIK{cik}.json."""
cik_padded = cik.zfill(10)
url = f"{DATA_SEC}/submissions/CIK{cik_padded}.json"
log.info("Fetching submissions for CIK %s", cik_padded)
resp = _throttled_get(url)
return resp.json()
def extract_filings(submissions_json: dict, form_types: set[str]) -> list[Filing]:
"""Extract filings of given form types from the submissions JSON."""
cik = str(submissions_json.get("cik", "")).zfill(10)
recent = submissions_json.get("filings", {}).get("recent", {})
filings = []
accessions = recent.get("accessionNumber", [])
forms = recent.get("form", [])
dates = recent.get("filingDate", [])
docs = recent.get("primaryDocument", [])
descs = recent.get("primaryDocDescription", [])
for i in range(len(accessions)):
if forms[i] not in form_types:
continue
accession_no_dashes = accessions[i].replace("-", "")
doc_url = f"{ARCHIVES}/{int(cik)}/{accession_no_dashes}/{docs[i]}"
filings.append(Filing(
accession_number=accessions[i],
form_type=forms[i],
filing_date=dates[i],
primary_document=docs[i],
description=descs[i] if i < len(descs) else "",
document_url=doc_url,
))
return filings
# ---------------------------------------------------------------------------
# 3. Download and parse a filing document (HTML/XML → text)
# ---------------------------------------------------------------------------
def download_filing_text(filing: Filing, max_bytes: int = 5_000_000) -> str:
"""Download a filing document and extract plain text from HTML/XML."""
if not filing.document_url:
return ""
try:
resp = _throttled_get(filing.document_url, stream=True)
content_type = resp.headers.get("Content-Type", "")
if "pdf" in content_type.lower():
log.info("Skipping PDF document: %s", filing.document_url)
return "[PDF — binary content not extracted]"
raw = resp.content[:max_bytes]
text = raw.decode("utf-8", errors="replace")
soup = BeautifulSoup(text, "lxml")
for tag in soup(["script", "style", "meta", "link"]):
tag.decompose()
plain = soup.get_text(separator="\n", strip=True)
plain = re.sub(r"\n{3,}", "\n\n", plain)
return plain
except Exception as e:
log.warning("Failed to download %s: %s", filing.document_url, e)
return ""
# ---------------------------------------------------------------------------
# 4. Search EDGAR full-text index for filings
# ---------------------------------------------------------------------------
def search_filings(
query: str = "",
forms: str = "485BPOS,497,497K",
start_date: str = "2024-01-01",
end_date: str = "2025-12-31",
cik: str = "",
max_results: int = 20,
) -> list[dict]:
"""
Use the EDGAR Full-Text Search API (efts.sec.gov) to find filings.
Returns a list of hit dicts from the Elasticsearch response.
"""
params: dict = {
"q": query or "*",
"forms": forms,
"dateRange": "custom",
"startdt": start_date,
"enddt": end_date,
"from": 0,
"size": max_results,
}
if cik:
params["q"] = f'"{cik}"' if not query else f"{query} AND {cik}"
log.info("EFTS search: forms=%s, date=%s–%s, q=%s", forms, start_date, end_date, params["q"])
resp = _throttled_get(EFTS_SEC, params=params)
data = resp.json()
hits = data.get("hits", {}).get("hits", [])
log.info("EFTS returned %d hits", len(hits))
return [h.get("_source", {}) for h in hits]
# ---------------------------------------------------------------------------
# 5. Build the full dataset for a list of CIKs
# ---------------------------------------------------------------------------
EXAMPLE_FUNDS = [
{
"cik": "0000036405",
"name": "Vanguard Index Funds (Vanguard 500 Index Fund, VOO, VFIAX)",
},
{
"cik": "0000024238",
"name": "Fidelity Contrafund (FCNTX, FCNKX)",
},
{
"cik": "0001100663",
"name": "iShares Trust (IVV, iShares Core S&P 500 ETF)",
},
{
"cik": "0000773757",
"name": "Columbia Funds Series Trust I",
},
{
"cik": "0001795351",
"name": "T. Rowe Price Exchange-Traded Funds, Inc.",
},
]
def build_dataset(
ciks: list[str],
output_dir: str = "dataset",
max_prospectus_filings: int = 5,
max_supplement_filings: int = 5,
download_text: bool = True,
reference_data: Optional[dict[str, FundTrust]] = None,
) -> list[FundDataset]:
"""
For each CIK:
1. Merge reference data (series/class info)
2. Fetch filing history from Submissions API
3. Extract prospectus & supplement filings
4. Optionally download filing text
5. Save to JSON
"""
out = Path(output_dir)
out.mkdir(parents=True, exist_ok=True)
datasets: list[FundDataset] = []
for cik in tqdm(ciks, desc="Processing funds"):
cik_padded = cik.zfill(10)
# Resolve reference data
if reference_data and cik_padded in reference_data:
trust = reference_data[cik_padded]
else:
trust = FundTrust(cik=cik_padded, trust_name="(unknown — CSV not loaded)")
# Fetch submissions
try:
subs = fetch_submissions(cik_padded)
except requests.HTTPError as e:
log.error("Could not fetch submissions for CIK %s: %s", cik_padded, e)
continue
api_name = subs.get("name", "")
if api_name and "(unknown" in trust.trust_name:
trust.trust_name = api_name
# Extract filings
all_filings = extract_filings(subs, PROSPECTUS_FORM_TYPES)
log.info("CIK %s: found %d prospectus-related filings", cik_padded, len(all_filings))
prospectus_filings = [
f for f in all_filings if f.form_type in ("485BPOS", "485APOS", "N-1A")
][:max_prospectus_filings]
supplement_filings = [
f for f in all_filings if f.form_type in ("497", "497K")
][:max_supplement_filings]
if download_text:
for f in prospectus_filings + supplement_filings:
f.text_content = download_filing_text(f)
log.info(
" %s %s → %d chars",
f.form_type, f.filing_date, len(f.text_content),
)
ds = FundDataset(
trust=trust,
prospectus_filings=prospectus_filings,
supplement_filings=supplement_filings,
)
datasets.append(ds)
# Save individual fund JSON
fund_file = out / f"{cik_padded}.json"
with open(fund_file, "w", encoding="utf-8") as fp:
json.dump(asdict(ds), fp, indent=2, ensure_ascii=False)
log.info("Saved %s", fund_file)
# Save combined manifest
manifest = []
for ds in datasets:
manifest.append({
"cik": ds.trust.cik,
"trust_name": ds.trust.trust_name,
"num_series": len(ds.trust.series),
"num_classes": sum(len(s.classes) for s in ds.trust.series),
"num_prospectus_filings": len(ds.prospectus_filings),
"num_supplement_filings": len(ds.supplement_filings),
})
manifest_file = out / "manifest.json"
with open(manifest_file, "w", encoding="utf-8") as fp:
json.dump(manifest, fp, indent=2)
log.info("Saved manifest with %d funds → %s", len(manifest), manifest_file)
return datasets
# ---------------------------------------------------------------------------
# 6. Download XBRL Risk/Return data (quarterly ZIP)
# ---------------------------------------------------------------------------
def download_xbrl_risk_return(
quarter: str = "2025q2",
output_dir: str = "dataset/xbrl_rr",
) -> Path:
"""
Download a quarterly Mutual Fund Prospectus Risk/Return Summary
ZIP from the SEC and extract it.
"""
url = (
f"https://www.sec.gov/files/dera/data/"
f"mutual-fund-prospectus-risk/return-summary-data-sets/{quarter}_rr1.zip"
)
out = Path(output_dir)
out.mkdir(parents=True, exist_ok=True)
zip_path = out / f"{quarter}_rr1.zip"
log.info("Downloading XBRL Risk/Return dataset: %s", url)
resp = _throttled_get(url, stream=True)
with open(zip_path, "wb") as fp:
for chunk in resp.iter_content(chunk_size=8192):
fp.write(chunk)
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(out / quarter)
log.info("Extracted to %s", out / quarter)
return out / quarter
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
import argparse
parser = argparse.ArgumentParser(
description="Fetch SEC fund prospectus + reference data for LLM training."
)
parser.add_argument(
"--ciks",
nargs="*",
help="CIK numbers to fetch (default: 5 example funds)",
)
parser.add_argument(
"--output", default="dataset",
help="Output directory (default: dataset)",
)
parser.add_argument(
"--max-prospectus", type=int, default=3,
help="Max prospectus filings per fund (default: 3)",
)
parser.add_argument(
"--max-supplements", type=int, default=5,
help="Max supplement filings per fund (default: 5)",
)
parser.add_argument(
"--no-download-text", action="store_true",
help="Skip downloading filing document text",
)
parser.add_argument(
"--load-reference-csv", action="store_true",
help="Download and load the full SEC series/class CSV (~15 MB)",
)
parser.add_argument(
"--download-xbrl-rr", type=str, default="",
help="Also download XBRL Risk/Return dataset for this quarter (e.g. 2025q2)",
)
args = parser.parse_args()
ciks = args.ciks or [f["cik"] for f in EXAMPLE_FUNDS]
ref_data = None
if args.load_reference_csv:
ref_data = load_series_class_reference()
datasets = build_dataset(
ciks=ciks,
output_dir=args.output,
max_prospectus_filings=args.max_prospectus,
max_supplement_filings=args.max_supplements,
download_text=not args.no_download_text,
reference_data=ref_data,
)
if args.download_xbrl_rr:
download_xbrl_risk_return(args.download_xbrl_rr, f"{args.output}/xbrl_rr")
print(f"\nDone. Processed {len(datasets)} funds → ./{args.output}/")
print("\nSummary:")
for ds in datasets:
n_classes = sum(len(s.classes) for s in ds.trust.series)
print(
f" {ds.trust.cik} | {ds.trust.trust_name[:50]:50s} | "
f"{len(ds.trust.series)} series, {n_classes} classes | "
f"{len(ds.prospectus_filings)} prospectus, "
f"{len(ds.supplement_filings)} supplements"
)
if __name__ == "__main__":
main()