1030 lines
93 KiB
JSON
1030 lines
93 KiB
JSON
{
|
|
"lecture": 9,
|
|
"week": 9,
|
|
"lessons": 3,
|
|
"title": "Lecture 9: Classes C6--C9 -- and Ten Profiles Side by Side",
|
|
"script_reference": "Script: Part III, Sections 24--27, 29",
|
|
"agenda": [
|
|
"Recap: C1--C5 and the class rhythm -- where Part III stands",
|
|
"C6 -- Scientific simulation / batch compute: makespan, reproducibility, the third consistency semantics",
|
|
"C7 -- Decision support / BI analytics: OLAP scans and the refresh contract",
|
|
"Closing the loop: C6 and C7 inside your C10 project",
|
|
"C8 -- Real-time / IoT streaming: ingest at the world's rate, correctness under failure",
|
|
"C9 -- Collaboration / messaging: connection fan-out and push",
|
|
"Stepping back: ten profiles side by side -- the consolidated requirements table and its seventeen footnotes",
|
|
"Reading the catalogue as a whole: five cross-class observations",
|
|
"This week's exercise: the walking skeleton runs end-to-end (M3 closes)"
|
|
],
|
|
"recap": [
|
|
"Framework position: Part III supplies the demand side -- the requirements profile $R(a)$: weights $w_i(a)$ over D1--D12, workload shape $S(a)$, hard constraints $K(a)$; Part II supplied the capability profiles $C(p)$; Part IV matches the two through the three-stage non-compensatory match (Lectures 3 and 7)",
|
|
"Last week (C1--C5): an application class \\emph{is} a requirements profile -- a recurring ASR bundle with typical response measures, a characteristic workload shape and a regulatory frame; the rhythm: leading question $\\to$ challenges $\\to$ binding scenarios S1--S3 $\\to$ twelve-row profile $\\to$ what real systems chose",
|
|
"Last week (C1--C5): C1 core banking (ACID ledger) and C2 social/content ($\\sim 50{:}1$ read-heavy, deliberately eventual) as the mirror pair; C3 back-office; C4 ERP; C5 e-commerce, the only class \\emph{defined} by a hybrid consistency profile",
|
|
"Week 6 (C10): your class -- Highs on D6/D7/D9/D10/D12; inherits C6/C7/C3 and adds the cost model per request and evals as testability: \\emph{spend the design budget on the new elements}",
|
|
"The map of the ten classes: two axes -- workload shape and consistency semantics; C6 and C7 sit at semantics that are neither ACID nor eventual -- today we develop that third family",
|
|
"Today: C6--C9 in the same rhythm, then the ten profiles side by side: the consolidated table, its seventeen footnotes, five cross-class observations -- the other operand of the fit matrix"
|
|
],
|
|
"frames": [
|
|
{
|
|
"no": 1,
|
|
"deck_section": "Title",
|
|
"title": "AISE502: AI in Software Engineering II -- Lecture 9: Classes C6--C9 -- and Ten Profiles Side by Side",
|
|
"kind": "content",
|
|
"script_ref": "title slide; subtitle line: Script: Part III, Sections 24--27, 29",
|
|
"content": [
|
|
"\\FHGRTitlePage with subtitle 'Lecture 9: Classes C6--C9 -- and Ten Profiles Side by Side' and small line 'Script: Part III, Sections 24--27, 29'",
|
|
"Author Dr.\\ Florian Herzog; Fachhochschule Graub\\\"unden, Chur -- Autumn Semester 2026"
|
|
],
|
|
"elements": [
|
|
"FHGR title page as in deck 6 (metadata block lines 118-127 of AISE502_Vorlesung_6_Folien.tex, \\FHGRTitlePage at line 127)"
|
|
],
|
|
"minutes": 0,
|
|
"notes": "Identical metadata block to deck 6; only subtitle and script line change. One colon then an en-dash, as in 'Lecture 6: Pipelines, Serverless, the View Across -- and Your Class (C10)'."
|
|
},
|
|
{
|
|
"no": 2,
|
|
"deck_section": "Agenda",
|
|
"title": "Agenda",
|
|
"kind": "agenda",
|
|
"script_ref": "deck skeleton (decks 1-6); semester plan week 9",
|
|
"content": [
|
|
"1. Recap: C1--C5 and the class rhythm",
|
|
"2. \\textbf{C6} -- Scientific simulation / batch compute: the third consistency semantics",
|
|
"3. \\textbf{C7} -- Decision support / BI analytics: the refresh contract",
|
|
"4. Closing the loop: C6 and C7 inside your C10 project",
|
|
"5. \\textbf{C8} -- Real-time / IoT streaming: correctness under failure",
|
|
"6. \\textbf{C9} -- Collaboration / messaging: connection fan-out and push",
|
|
"7. Stepping back: \\textbf{ten profiles side by side} -- the consolidated table and its seventeen footnotes",
|
|
"8. Reading the catalogue as a whole: five cross-class observations",
|
|
"9. This week's exercise: \\textbf{the walking skeleton runs end-to-end}"
|
|
],
|
|
"elements": [
|
|
"enumerate, \\small, itemsep 1pt (deck-6 'Agenda' frame format, line 132)"
|
|
],
|
|
"minutes": 1,
|
|
"notes": "Nine items is the maximum the agenda frame tolerates at \\small; keep each to one line."
|
|
},
|
|
{
|
|
"no": 3,
|
|
"deck_section": "Recap",
|
|
"title": "Recap: where we are",
|
|
"kind": "recap",
|
|
"script_ref": "deck 6 summary and C10 frames; deck 8 (per brief: §18 + C1--C5); §18.3 map (part3_application_classes.tex 55-63)",
|
|
"content": [
|
|
"\\textbf{Framework position:} Part III = the demand side -- $R(a)$: weights $w_i(a)$ over D1--D12, workload shape $S(a)$, hard constraints $K(a)$; Part II gave $C(p)$; Part IV matches them (three-stage non-compensatory match, Lectures 3 and 7)",
|
|
"\\textbf{Last week (C1--C5):} an application class \\emph{is} a requirements profile -- a recurring ASR bundle with typical response measures, a characteristic workload shape, a regulatory frame; the rhythm: leading question $\\to$ challenges $\\to$ S1--S3 $\\to$ 12-row profile $\\to$ what real systems chose",
|
|
"\\textbf{Last week (C1--C5):} C1 (ACID ledger) and C2 ($\\sim 50{:}1$ read-heavy, deliberately eventual) as the mirror pair; C3 back-office; C4 ERP; C5 e-commerce, the only class \\emph{defined} by a hybrid consistency profile",
|
|
"\\textbf{Week 6 (C10):} Highs on D6/D7/D9/D10/D12; \\emph{inherits} C6/C7/C3; spend the design budget on the two novelties (cost per request, evals as testability)",
|
|
"\\textbf{The map of the ten classes:} two axes -- workload shape and consistency semantics; C6 and C7 sit at semantics that are neither ACID nor eventual -- today we develop that third family",
|
|
"\\textbf{Today:} C6--C9 in the same rhythm; then ten profiles side by side -- the consolidated table, its seventeen footnotes, five cross-class observations"
|
|
],
|
|
"elements": [
|
|
"bullet list, \\footnotesize, six bullets of two to three lines -- the maximum; no classmap thumbnail"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "One frame only; do not re-teach C1--C5 or C10. No deck, frame or section numbers on the slide (decks 1-6 say 'last week' / 'week 6'). The three-stage fit procedure was introduced in deck 3 and formalised in Lecture 7, hence '(Lectures 3 and 7)'."
|
|
},
|
|
{
|
|
"no": 4,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "Part III continues: C6 -- Scientific simulation / batch compute",
|
|
"kind": "content",
|
|
"script_ref": "§24 intro and 24.1 (part3_application_classes.tex 493-502)",
|
|
"content": [
|
|
"\\emph{\\textcolor{bankblue}{The nightly risk run must reproduce, to the bit, on ten thousand cores by 06:00: what does that demand?}}",
|
|
"The class: compute-dominated batch systems -- cost function CPU/GPU-hours; availability in the online sense irrelevant (\\emph{an aborted job is restarted from a checkpoint, not mourned}); correctness means \\textbf{reproducibility}: regenerable from versioned inputs, versioned code and deterministic seeds",
|
|
"\\textbf{1. The interactive vocabulary does not apply} -- no read/write ratio, only read-input $\\to$ compute $\\to$ write-output over immutable datasets; the response measures are \\textbf{makespan} (hours to weeks) and \\textbf{batch-window adherence}: D2 High in its makespan reading (note 2), D3 Low (note 4); planned, queue-based load at a utilisation target near 100\\,\\%",
|
|
"\\textbf{2. The reproducibility obligation} -- bit-identical reruns from versioned inputs, code and seeds: a scientific duty and, for regulated risk models, a supervisory one $\\to$ D9 in its reproducibility reading (note 14), D4 ``as of this run'' (note 8)",
|
|
"\\textbf{3. Physics: scaling has a measurable ceiling} -- OpenFOAM domain decomposition pays down to roughly 50{,}000--200{,}000 cells per core, below which communication overhead dominates (Amdahl's law as an architecture boundary); Monte-Carlo is embarrassingly parallel -- the ideal pipeline case"
|
|
],
|
|
"elements": [
|
|
"leading question in bankblue italics; numbered list of the three challenges, two lines each (deck-6 'Part III opens: C10 -- AI-native advisory platforms' format, lines 528-541)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "\\footnotesize body; class description two lines, each challenge capped at two lines (~13 lines). Speaker notes carry what was trimmed from lines 495-502: the CFD/Monte-Carlo/ML-training/data-transformation examples; 'unlawful to rely on' (returns in the K paragraph of the next frame); utilisation near 100 % as the economic opposite of C2/C5 elastic headroom; cost per simulation as a first-order D10 concern; TB--PB of result data; the split change rate (research-driven model code vs. 30+-year-old Fortran/C++ kernels) that keeps D7 at Medium -- the last two sit in the profile table's D7/D10 rows. Emphasise 'makespan' and 'reproducibility' as the two words that replace 'latency' and 'ACID'."
|
|
},
|
|
{
|
|
"no": 5,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "C6 -- the binding scenarios",
|
|
"kind": "content",
|
|
"script_ref": "§24.1 scenarios and K(C6) (part3_application_classes.tex 504-508, 514)",
|
|
"content": [
|
|
"\\textbf{S1 (batch-window adherence).} Overnight risk run starts after close of business; full portfolio revaluation completes \\textbf{within the batch window, results published before start of trading}. \\textcolor{codegray}{(The D2 scenario in its makespan reading.)}",
|
|
"\\textbf{S2 (bit-level reproducibility).} A model validator re-executes a historical run months later; the pipeline regenerates the results \\textbf{bit-identically from versioned inputs, code and seeds}. \\textcolor{codegray}{(The D9/D4 scenario.)}",
|
|
"\\textbf{S3 (cost efficiency).} The quarterly compute bill is reviewed; cost per simulation and cluster utilisation stay \\textbf{within budget at a utilisation target near 100\\,\\%} -- queue-based scheduling exists to meet exactly this measure. \\textcolor{codegray}{(The D10 scenario.)}",
|
|
"$K(\\text{C6})$ in financial applications: internal models are subject to \\textbf{model validation and backtesting} under the Basel framework; traceability of model runs is audit-relevant -- the BCBS~239 lineage obligation extends into the simulation stack",
|
|
"\\textbf{Knock-out reading:} any structure that cannot pin \\emph{data version + code version + seed} per run is excluded for regulated risk computation, whatever its throughput. \\textbf{Deterministic seeds are a hard constraint, not a nicety.}"
|
|
],
|
|
"elements": [
|
|
"three-scenario itemize with response measures in bold and the dimension in codegray; K paragraph and knock-out reading below (deck-6 'C10 -- the binding scenarios' format, lines 543-553)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Point out that S1 and S3 are the same numbers the PF column's D3 = $--$ and D10 = $++$ cells were built from (week 6)."
|
|
},
|
|
{
|
|
"no": 6,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "The third consistency semantics: reproducibility and refresh contracts",
|
|
"kind": "definition",
|
|
"script_ref": "§24.1 definitionbox (part3_application_classes.tex 510-512); also §18.3 (55-57)",
|
|
"content": [
|
|
"Beside \\textbf{ACID} (``concurrent operations preserve invariants'') and \\textbf{eventual consistency} (``replicas converge, staleness is bounded''), the batch world defines a \\emph{third family} of consistency semantics",
|
|
"\\textbf{Reproducibility (C6):} immutable inputs and idempotent, seeded stages guarantee that a rerun yields the identical result -- consistency ``\\textbf{as of this run}''",
|
|
"\\textbf{Freshness by refresh contract (C7):} consumers see a complete, internally consistent state ``\\textbf{as of the last refresh}'' (e.g.\\ yesterday 24:00), never a half-updated one",
|
|
"Both are \\emph{weaker} than ACID and \\emph{stronger} than unqualified eventual consistency -- and both are \\textbf{contracts with response measures}, testable in CI like any other scenario",
|
|
"Supply side, already seen: the PF column's D4 $= +$ (week 6: ``the subtlest cell in the table''); demand side, today: footnotes 8 (C6) and 10 (C7) of the consolidated table"
|
|
],
|
|
"elements": [
|
|
"definitionbox[The third consistency semantics: reproducibility and refresh contracts] from lines 510-512, verbatim in substance; one closing bullet linking to week 6 and the footnotes"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Serves both C6 and C7 -- place here, refer back on the C7 intro and key-concept frames. The last bullet is the deck's own bridge (deck 6 PF cell, footnotes 8/10), not new content. Wording: deck 6 phrased the PF D4 = + cell as 'as of last run' (lines 209, 229, 663); the script's canonical wording (line 511) is 'as of this run' -- same semantics, say so if a student notices."
|
|
},
|
|
{
|
|
"no": 7,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "C6 -- requirements profile (column C6)",
|
|
"kind": "table",
|
|
"script_ref": "§24.2 tab:req-c6 (part3_application_classes.tex 520-546)",
|
|
"content": [
|
|
"D1 Read scalability | L | no interactive readers; outputs are files and tables",
|
|
"D2 Write scal.\\ \\& elasticity | \\textbf{H} | throughput as makespan and batch-window compute bursts (S1; note 2)",
|
|
"D3 Latency | L | per-request latency is meaningless; the window is the measure (note 4)",
|
|
"D4 Consistency \\& integrity | M | consistency ``as of this run'': bit-level reproducibility from versioned inputs and seeds (note 8)",
|
|
"D5 Availability \\& isolation | L | an aborted job is restarted from a checkpoint, not mourned",
|
|
"D6 Security \\& auditability | M | lineage of regulated model runs is audit-relevant; no online attack surface",
|
|
"D7 Evolvability | M | model code churns research-driven; numeric kernels live 30+ years",
|
|
"D8 Simplicity \\& TTM | M | orchestration should stay simple; the kernels never are",
|
|
"D9 Testability \\& deployability | \\textbf{H} | reproducibility is the operative meaning of testability (S2; note 14)",
|
|
"D10 Operating cost | \\textbf{H} | the cost function \\emph{is} CPU/GPU-hours at near-100\\,\\% utilisation (S3)",
|
|
"D11 Team scaling | L | small research and engineering teams",
|
|
"D12 AI integrability | M | training and simulation pipelines host ML natively, at batch cadence",
|
|
"Shape $S(a)$: scheduled batch | Constraints $K(a)$: model validation and backtesting (finance); deterministic seeds"
|
|
],
|
|
"elements": [
|
|
"12-row requirements-profile table (scriptsize, arraystretch 0.8, p{3.0cm} c p{7.7cm}) from tab:req-c6 lines 526-544, Highs bold; Shape and Constraints rows below midrule -- deck-6 'C10 -- requirements profile' format (lines 555-582, with \\vspace{-0.35cm})"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Read aloud only the three Highs (D2, D9, D10) against S1--S3 and the two footnoted cells (D2, D4, D9); stress that D2 High here means makespan, not requests/s -- the first footnoted letter of the day. The D7 row carries the change-rate remark trimmed from the intro frame."
|
|
},
|
|
{
|
|
"no": 8,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "C6 -- what real systems chose, and why",
|
|
"kind": "case",
|
|
"script_ref": "§24.3 (part3_application_classes.tex 548-552)",
|
|
"content": [
|
|
"\\textbf{Dominant form: pipes-and-filters on HPC or batch infrastructure} -- the PF column reads like a transcript of this class's Highs: immutable inputs $+$ deterministic, idempotent stages give reproducibility (D9) \\emph{by construction}; batch scheduling drives utilisation towards the D10 target; the shape matches the gate that eliminates every interactive pattern outright",
|
|
"\\textbf{OpenFOAM}: domain decomposition via graph partitioning, MPI communication, scheduled by \\textbf{Slurm} as batch resource manager; \\textbf{OpenMC}: modern parallel Monte-Carlo transport code of the same family; \\textbf{Spark} (data-parallel) and \\textbf{Airflow} (DAG orchestration) as cloud-native incarnations (Dask/Ray Python-native)",
|
|
"\\textbf{Where the trade-off bites (A2): data affinity} -- distributing a tightly coupled numeric dataflow across expensive boundaries multiplies communication cost",
|
|
"\\textbf{Prime Video} (already met; now read from the demand side) as this class's cautionary tale in miniature: a tightly coupled processing flow distributed across orchestrator transitions and S3 round-trips per frame, until consolidation into one process cut cost by over 90\\,\\% -- \\emph{when inter-stage communication dominates, consolidate, do not distribute}",
|
|
"\\textbf{Serverless fan-out earns only a conditional role}: embarrassingly parallel, communication-light bursts (Monte-Carlo) fit FaaS elasticity; data-heavy stages recreate the Prime Video cost trap"
|
|
],
|
|
"elements": [
|
|
"five bullets \\footnotesize (deck-6 'C10 -- what real systems chose, and why' format, line 584); no verdict line -- the 'in one sentence' verdict closes the key-concept box on the next frame"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Five bullets is the limit. Prime Video was taught in weeks 1--3 and 6 -- do not retell; say 'same case, now the demand side'."
|
|
},
|
|
{
|
|
"no": 9,
|
|
"deck_section": "C6 -- Scientific Simulation / Batch Compute",
|
|
"title": "C6 -- key concept",
|
|
"kind": "keyconcept",
|
|
"script_ref": "§24.3 keypoint (part3_application_classes.tex 554-556), verdict (562); projektbox pointer only (558-560)",
|
|
"content": [
|
|
"\\textbf{Key concept:} C6 replaces the interactive vocabulary wholesale: the response measures are \\textbf{makespan, batch-window adherence and cost per simulation}; correctness means \\textbf{bit-level reproducibility} from versioned inputs and seeds; consistency means ``as of this run''. A structure fits this class exactly to the degree that it makes these three \\emph{measurable by construction} -- which is why immutable-input pipelines dominate it. \\textbf{In one sentence:} PF on HPC/batch infrastructure (PF at $++$), serverless fan-out secondary for bursty, communication-light parallel sections -- Part IV computes the verdict through the three-stage procedure",
|
|
"\\textbf{Project link:} the ingestion pipeline and the eval pipeline of your platform are C6-shaped subsystems -- the loop closes after C7"
|
|
],
|
|
"elements": [
|
|
"keypoint box (lines 554-556, condensed) with the verdict (line 562) as its last sentence -- deck-6 keypoint format ('In one sentence' inside the box, line 636); one plain pointer line below"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "First half of closing week 6's inheritance loop: the C6 projektbox substance (558-560) is taught on the 'Closing the loop' frame after C7, not here -- avoid showing the same three sentences three times."
|
|
},
|
|
{
|
|
"no": 10,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "C7 -- Decision support / BI analytics",
|
|
"kind": "content",
|
|
"script_ref": "§25 intro and 25.1 (part3_application_classes.tex 567-576)",
|
|
"content": [
|
|
"\\emph{\\textcolor{bankblue}{Monday, 08:00: the management dashboard must be complete as of yesterday 24:00 -- and every number must prove where it came from. What delivers that, night after night?}}",
|
|
"The class: data warehouses, BI platforms, reporting systems -- an \\textbf{integration product over C1--C5}: it owns almost no data of its own but integrates everyone else's, so data quality, lineage and governance are first-class; its consistency semantics is the \\textbf{refresh contract} (the definition just given): ``consistent as of yesterday 24:00'' is a deliberate design choice, not a deficiency",
|
|
"\\textbf{1. The shape of the reads} -- almost pure read, but \\textbf{heavy OLAP scans by few users}, not point reads by many (note 7): a D1 demand in its own reading, scan capacity \\emph{delegated} to the warehouse engine; writes bundled into the ELT/ETL window (D2 Low); dashboards render in seconds (D3 Medium)",
|
|
"\\textbf{2. The refresh contract} -- marts refreshed within the window and visible \\emph{atomically}; consumers never see a half-updated state (D4 Medium, note 10); adherence tracked as a fitness function (D9)",
|
|
"\\textbf{3. Provenance under interrogation} -- a regulator questions a reported risk figure: trace it back through every transformation to the source systems (D6); BCBS~239 hardens this into $K(a)$ where banking data are involved"
|
|
],
|
|
"elements": [
|
|
"leading question in bankblue italics; numbered list of three challenges, two lines each (deck-6 'Part III opens' format, lines 528-541)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "\\footnotesize. Speaker notes carry the trimmed clauses from lines 573-575: office-hour load with Monday-morning and month-start peaks; an hour of downtime is an annoyance, not a loss event (D5 Low); the warehouse bill as the dominant run cost (D10 High); TB--PB dimensionally modelled, Kimball's star schema as the reference; change rate high at report level, moderate in the core model (D7 Medium) -- all of which sit in the profile table's rows. The phrase 'integration product over C1--C5' is the class's identity -- say it twice."
|
|
},
|
|
{
|
|
"no": 11,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "C7 -- the binding scenarios",
|
|
"kind": "content",
|
|
"script_ref": "§25.1 scenarios and K(C7) (part3_application_classes.tex 578-584)",
|
|
"content": [
|
|
"\\textbf{S1 (dashboard latency).} A controller opens the standard management dashboard on Monday morning; it renders \\textbf{in under 5 seconds}, ad-hoc analytical queries in seconds to minutes. \\textcolor{codegray}{(The D1/D3 scenario in the class's OLAP reading.)}",
|
|
"\\textbf{S2 (refresh contract).} The nightly ELT run executes; all marts are refreshed \\textbf{within the refresh window, atomically visible, refresh-window adherence tracked as a fitness function}. \\textcolor{codegray}{(The D4/D9 scenario.)}",
|
|
"\\textbf{S3 (lineage on demand).} A regulator or auditor questions a reported risk figure; the platform traces it \\textbf{back through every transformation to the source systems, on demand}. \\textcolor{codegray}{(The D6 scenario, hardened into $K(a)$ for banks.)}",
|
|
"$K(\\text{C7})$: BCBS~239 for banks (risk reporting: accuracy, completeness, timeliness, demonstrable lineage); the GDPR (purpose limitation, anonymisation in analyses); sector-specific reporting duties",
|
|
"\\textbf{Knock-out reading:} a pipeline whose transformations are not versioned and traceable cannot serve regulated reporting -- \\emph{lineage is a structural property; retrofitting it is a rebuild}"
|
|
],
|
|
"elements": [
|
|
"three-scenario itemize with bold response measures; K paragraph below (deck-6 'C10 -- the binding scenarios' format, lines 543-553)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Contrast S2 with C6's S2: both are 'consistency as of' contracts -- the definition of the third semantics at work."
|
|
},
|
|
{
|
|
"no": 12,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "C7 -- requirements profile (column C7)",
|
|
"kind": "table",
|
|
"script_ref": "§25.2 tab:req-c7 (part3_application_classes.tex 590-616)",
|
|
"content": [
|
|
"D1 Read scalability | \\textbf{H} | heavy OLAP scans by few users, delegated to the warehouse engine (S1; note 7)",
|
|
"D2 Write scal.\\ \\& elasticity | L | writes are bundled into the nightly ELT window",
|
|
"D3 Latency | M | dashboards in under 5\\,s; ad-hoc queries in seconds to minutes",
|
|
"D4 Consistency \\& integrity | M | freshness by refresh contract -- ``consistent as of yesterday 24:00'' (S2; note 10)",
|
|
"D5 Availability \\& isolation | L | a dashboard down for an hour is an annoyance, not a loss event",
|
|
"D6 Security \\& auditability | M | lineage and purpose limitation; BCBS~239 hardens them into $K(a)$ where banking data flow (S3)",
|
|
"D7 Evolvability | M | report-level churn is high; the core dimensional model moves slowly",
|
|
"D8 Simplicity \\& TTM | M | thin serving keeps the estate simple; the pipeline carries the complexity",
|
|
"D9 Testability \\& deployability | M | refresh-window adherence is tracked as a fitness function (S2)",
|
|
"D10 Operating cost | \\textbf{H} | warehouse compute is the bill; cost discipline is the operative FinOps concern",
|
|
"D11 Team scaling | L | a data team, not a fleet of feature teams",
|
|
"D12 AI integrability | M | text-to-SQL docks onto the governed semantic layer, not onto raw tables",
|
|
"Shape $S(a)$: batch $+$ thin serving | Constraints $K(a)$: BCBS 239 lineage (banks)"
|
|
],
|
|
"elements": [
|
|
"12-row requirements-profile table (scriptsize, arraystretch 0.8) from tab:req-c7 lines 596-614; Highs bold; Shape/Constraints rows (deck-6 profile format, lines 555-582)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Read aloud only the two Highs (D1 in the note-7 reading, D10) and the footnoted D4 cell; D5 Low, D10 High and the D7 remark trimmed from the intro frame live here. Ask: why is D6 only M when S3 is statutory? Answer from the table: it is hardened into K(a) for banks -- knock-out, not weight."
|
|
},
|
|
{
|
|
"no": 13,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "C7 -- what real systems chose, and why",
|
|
"kind": "case",
|
|
"script_ref": "§25.3 (part3_application_classes.tex 618-622)",
|
|
"content": [
|
|
"\\textbf{Documented modern form: a batch/ELT pipeline in front of a thin serving layer} -- sources $\\to$ orchestrated DAGs (\\textbf{Apache Airflow}: scheduler, executor, workers, metadata database; DAGs as code) $\\to$ warehouse $\\to$ BI frontend",
|
|
"The ``modern data stack'' is architecturally \\textbf{a layered application over a batch pipeline}: the PF column serves the refresh contract and lineage by construction (note 10, D6); the L column serves the thin tier because it \\emph{delegates} the D1-High scans to the warehouse and keeps its D8/D10 virtues",
|
|
"\\textbf{Apache Superset} (Python/Flask, originating at Airbnb): deliberately a \\emph{thin} visualisation layer with no data storage of its own, connecting directly to SQL engines; \\textbf{Metabase} ships as a single JAR -- a literal monolith, mirroring how little serving infrastructure the class needs. Both delegate scaling to the database: \\emph{division of labour as an architectural principle}",
|
|
"\\textbf{Where the trade-off bites (A2): freshness} -- streaming ingestion (EDA) earns its broker-operations complexity only when the freshness contract measurably tightens below the batch window; tightening towards real time is \\textbf{not a tuning exercise but a class change into C8 economics}"
|
|
],
|
|
"elements": [
|
|
"four bullets \\footnotesize; no tikz strip (the PF and L topologies were drawn in weeks 4 and 6); no verdict line -- it closes the key-concept box on the next frame"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "The 'class change into C8' line prepares the C8 intro. Bullet 2 is capped at two lines."
|
|
},
|
|
{
|
|
"no": 14,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "C7 -- AI lens and key concept",
|
|
"kind": "ailens",
|
|
"script_ref": "§25.3 ailinse (part3_application_classes.tex 628-630), keypoint (624-626), verdict (636)",
|
|
"content": [
|
|
"\\textbf{AI Lens [Natural-language analytics (Axis B)]:} the visible frontier is the natural-language query -- ``show me revenue by region, last four quarters'' -- compiled by an LLM into SQL against the semantic layer",
|
|
"The profile dictates the containment: the LLM \\emph{proposes} a query; the semantic layer's schema and access-control contracts \\emph{validate and execute} it -- \\textbf{text-to-SQL against governed models, not raw tables}, so that lineage (S3) and purpose limitation survive the convenience",
|
|
"Failure mode to design against: \\textbf{silent plausibility} -- a syntactically valid, semantically wrong query returns confident nonsense; the eval harness of Part V -- golden question/answer sets over a \\emph{frozen} warehouse state -- is the class-appropriate test instrument, and note 10's refresh contract is what makes such golden sets stable",
|
|
"\\textbf{Key concept:} C7 is an integration product over C1--C5 whose consistency semantics -- freshness by refresh contract -- is a \\emph{designed} third option beside ACID and eventual. \\textbf{The ADR to write is the freshness contract itself}; the fitness functions are refresh-window adherence and p95 dashboard latency; tightening the contract towards real time is a class change into C8 economics. \\textbf{In one sentence:} PF (ELT) feeding a thin layered serving tier -- the canonical hybrid of the row (PF at $++$; L, MM, SL at $+$) -- with EDA ingestion secondary where the freshness contract demands it"
|
|
],
|
|
"elements": [
|
|
"ailinse[Natural-language analytics (Axis B)] box (lines 628-630, condensed) above a keypoint box (lines 624-626, condensed) closing with the verdict (line 636) -- deck-6 'PF -- AI lens and key concept' format (line 254) with 'In one sentence' inside the keypoint (line 636)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Both boxes at \\footnotesize; the AI lens takes three short bullets at most."
|
|
},
|
|
{
|
|
"no": 15,
|
|
"deck_section": "C7 -- Decision Support / BI Analytics",
|
|
"title": "Closing the loop: C6 and C7 inside your C10 project",
|
|
"kind": "content",
|
|
"script_ref": "§24.3 projektbox (part3_application_classes.tex 558-560); §25.3 projektbox (632-634); deck 6 'C10 -- your project as an inheritance diagram' (lines 629-640)",
|
|
"content": [
|
|
"Week 6's inheritance diagram: C10 inherits C6, C7 and C3 -- today the two batch parents are taught",
|
|
"\\textbf{Left -- the C6 subsystems (ingestion: chunking, embedding, indexing; the eval pipeline):} versioned inputs, idempotent stages, deterministic reruns, golden datasets",
|
|
"Left: note 14 is the bridge -- for batch pipelines, \\emph{reproducibility is the operative meaning of testability}",
|
|
"Left: when your eval scores change, prove whether the \\textbf{model, the prompt, the data or the pipeline} changed -- that proof is a C6 property",
|
|
"\\textbf{Right -- the C7 subsystem:} portfolio analytics, exposure aggregation, report generation over the deterministic data store",
|
|
"Right: its refresh contract -- ``\\textbf{analysis as of last completed ingestion}''",
|
|
"Right: it bounds what the advisory layer may claim -- \\emph{an answer must never cite data fresher than the index actually is}",
|
|
"\\textbf{Key concept:} the freshness contract appears twice in your measurement contract -- as a pipeline fitness function (C7 discipline) and as a grounding rule for generated answers (C10 discipline)"
|
|
],
|
|
"elements": [
|
|
"one-line lead; projektbox with two columns (0.48/0.48), three one-to-two-line bullets each (C6 column from lines 558-560, C7 column from 632-634); one keypoint line below -- deck-6 mirror-pair format (line 603) with short column bullets"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "This frame owns the substance of both projektboxes (the C6 key-concept frame only pointed here; the exercise frame keeps a one-line 'applied' bullet). 'What remains genuinely new' is left to observation 5 later today, which states it. Columns must not exceed three short bullets each."
|
|
},
|
|
{
|
|
"no": 16,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- Real-time / IoT streaming",
|
|
"kind": "content",
|
|
"script_ref": "§26 intro and 26.1 (part3_application_classes.tex 641-650)",
|
|
"content": [
|
|
"\\emph{\\textcolor{bankblue}{The sensors never stop: what ingests the world at the world's own rate, indefinitely -- and stays exactly right when a node dies mid-window?}}",
|
|
"The class: systems that process continuously arriving event data -- sensor telemetry, clickstreams, vehicle positions, market data. Defining property: data are produced by the world at the world's own rate and \\textbf{lose value quickly}; the system must sustain ingest at the production rate \\emph{indefinitely}, master backpressure, and guarantee state correctness across failures",
|
|
"\\textbf{1. The sheer, unending inflow} -- ingest around the clock with bounded consumer lag and mastered backpressure $\\to$ D2 (the \\emph{inversion} of C2's read-heavy profile); 24/7 without pause $\\to$ D5 (the world does not stop producing while a node recovers); value decay $\\to$ D3",
|
|
"\\textbf{2. The genuinely hard one: declared correctness semantics under failure} -- event time vs.\\ processing time, windowing, watermarks (the Dataflow model); exactly-once state via checkpointing -- bought with machinery and \\emph{permanent operational competence}: D4 in its stream reading (note 11), D8 Low (nobody chooses streaming for its ease)",
|
|
"\\textbf{3. Longevity of the plumbing} -- topologies change moderately but \\textbf{schema evolution is a permanent concern} (D7 Medium); the largest data volume of all classes (append-only logs), retention often TTL-bounded, infrastructure long-lived even where the data are not"
|
|
],
|
|
"elements": [
|
|
"leading question in bankblue italics; numbered list of three challenges (deck-6 'Part III opens' format, lines 528-541)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "\\footnotesize; challenges 1 and 2 trimmed to two lines each. The magnitudes (LinkedIn, Uber) are deferred to the next frame; 'an anomaly alert is worth something within milliseconds to seconds, and little after' goes to the speaker notes."
|
|
},
|
|
{
|
|
"no": 17,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- the verified magnitudes",
|
|
"kind": "table",
|
|
"script_ref": "§26.1 first challenge (part3_application_classes.tex 648-649); recap of deck 5 examplebox 'Kafka at LinkedIn; the Uber real-time stack' (AISE502_Vorlesung_5_Folien.tex 423-424)",
|
|
"content": [
|
|
"\\textit{Seen in week 5 from the supply side -- the existence proof for EDA D1/D2 $= ++$; today the same numbers as the demand-side D2 reading of C8}",
|
|
"\\textbf{LinkedIn} (Kafka's birthplace) | more than \\textbf{seven trillion} Kafka messages per day | \\textbf{new today:} roughly 100 clusters; over 4{,}000 brokers; about 100{,}000 topics; seven million partitions",
|
|
"\\textbf{Uber} | \\textbf{trillions of messages and petabytes per day} | integrated real-time stack: Kafka (transport), Flink (stream processing), Pinot (OLAP serving) -- documented in a SIGMOD paper",
|
|
"Read: D2 in the \\emph{ingest} reading -- around the clock, bounded consumer lag, mastered backpressure; D5 because it never pauses; D3 because value decays"
|
|
],
|
|
"elements": [
|
|
"recap marker line in italics; two-row table (footnotesize): System | Scale | Structure, from line 648; one reading line from line 649"
|
|
],
|
|
"minutes": 1,
|
|
"notes": "Recap frame, one minute: only the cluster/broker/topic/partition breakdown is new. Pinot returns on the 'what real systems chose' frame as the OLAP sink."
|
|
},
|
|
{
|
|
"no": 18,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- the binding scenarios",
|
|
"kind": "content",
|
|
"script_ref": "§26.1 scenarios and K(C8) (part3_application_classes.tex 652-658)",
|
|
"content": [
|
|
"\\textbf{S1 (sustained ingest).} Sensors emit at full production rate around the clock; the platform ingests and processes with \\textbf{consumer lag bounded and no data loss, indefinitely}. \\textcolor{codegray}{(The D2/D5 scenario.)}",
|
|
"\\textbf{S2 (exactly-once state).} A processing node fails mid-window under load; the system recovers with \\textbf{state restored to exactly-once semantics via checkpointing, zero duplicated or lost aggregate updates}. \\textcolor{codegray}{(The D4 scenario in its note-11 reading.)}",
|
|
"\\textbf{S3 (end-to-end latency).} An anomalous reading arrives; the alert fires \\textbf{within milliseconds to a few seconds end-to-end, per-key ordering preserved}. \\textcolor{codegray}{(The D3 scenario.)}",
|
|
"$K(\\text{C8})$ is \\textbf{technical rather than statutory}: per-key ordering and exactly-once semantics \\emph{where declared} are contractual knock-outs -- a billing stream that double-counts under failure is not ``mostly correct'', it is wrong; GDPR adds purpose limitation and retention duties where streams carry personal data (vehicle telemetry, wearables)",
|
|
"\\textbf{The pragmatic industry norm} -- at-least-once delivery plus idempotent consumers -- is legitimate exactly where the scenario says so; the decision is taken \\textbf{per stream, per scenario, never platform-wide}"
|
|
],
|
|
"elements": [
|
|
"three-scenario itemize; K paragraph; last bullet in plain bold (deck-6 'C10 -- the binding scenarios' format, lines 543-553)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "The 'per stream, per scenario' rule is the class's ADR discipline -- link to the measurement contract (Lecture 7)."
|
|
},
|
|
{
|
|
"no": 19,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- requirements profile (column C8)",
|
|
"kind": "table",
|
|
"script_ref": "§26.2 tab:req-c8 (part3_application_classes.tex 664-690)",
|
|
"content": [
|
|
"D1 Read scalability | M | consumers read the log at their own pace; serving is delegated to OLAP sinks",
|
|
"D2 Write scal.\\ \\& elasticity | \\textbf{H} | ingest at the world's production rate, indefinitely, with backpressure mastered (S1)",
|
|
"D3 Latency | \\textbf{H} | alerts fire within milliseconds to seconds, end to end (S3)",
|
|
"D4 Consistency \\& integrity | M | stream semantics: exactly-once state, event-time windows, watermarks (S2; note 11)",
|
|
"D5 Availability \\& isolation | \\textbf{H} | the world does not pause when a node fails; recovery must preserve state",
|
|
"D6 Security \\& auditability | L | low relative to the ledger classes; GDPR applies where streams carry personal data",
|
|
"D7 Evolvability | M | topologies change moderately; schema evolution is the permanent concern",
|
|
"D8 Simplicity \\& TTM | L | nobody chooses this class for its simplicity",
|
|
"D9 Testability \\& deployability | M | correctness under failure is verified by chaos experiments, not unit suites alone",
|
|
"D10 Operating cost | M | broker fleets cost, but the class's economics absorb them",
|
|
"D11 Team scaling | M | platform and stream teams coordinate around shared topics",
|
|
"D12 AI integrability | M | models score events in the stream as checkpointed operators",
|
|
"Shape $S(a)$: continuous stream | Constraints $K(a)$: per-key ordering; exactly-once where stated"
|
|
],
|
|
"elements": [
|
|
"12-row requirements-profile table (scriptsize, arraystretch 0.8) from tab:req-c8 lines 670-688; Highs bold (deck-6 profile format, lines 555-582)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Read aloud the three Highs (D2, D3, D5) -- the two that eliminate every single-quantum interactive pattern at the shape gate plus latency -- and the footnoted D4 cell. Note D8 = L and D6 = L as the two 'licences' of this column."
|
|
},
|
|
{
|
|
"no": 20,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- what real systems chose, and why",
|
|
"kind": "case",
|
|
"script_ref": "§26.3 (part3_application_classes.tex 692-696)",
|
|
"content": [
|
|
"\\textbf{Documented form: EDA plus pipes-and-filters over a durable log} -- producers $\\to$ Kafka (durable, partitioned, replayable) $\\to$ stream processors (Flink) $\\to$ sinks (OLAP, alerting, data lake)",
|
|
"The EDA column is this class's mirror: a durable, partitioned, replayable log with decoupled producers and consumers serves \\textbf{D2 and D5 at $++$} -- the two demands that eliminate every single-quantum interactive pattern at the shape gate; the streaming incarnation of PF supplies the processing layer whose \\textbf{checkpointing answers S2}",
|
|
"\\textbf{Netflix}: the consumer-scale EDA reference -- event-driven telemetry pipelines, resilience institutionalised through \\textbf{chaos engineering}: chaos experiments are exactly how this class tests its D4/D5 claims",
|
|
"\\textbf{Counter-lessons from this class's neighbourhood, both already met} (Segment: week 5; Prime Video: week 6 and the C6 case today) -- the shared A2 pattern is read in observation 3 later today"
|
|
],
|
|
"elements": [
|
|
"four bullets \\footnotesize; no tikz strip (week 5 already drew the EDA topology); no verdict line -- it closes the key-concept box on the next frame"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Speaker notes carry the open-source references from line 696 (Apache Kafka, Apache Flink, MQTT brokers Mosquitto/EMQX at the IoT edge, Apache Pinot/Druid for real-time OLAP serving) and the Segment/Prime Video details (140 per-destination microservices, the 2018 retreat to a monolith with a central queue; the serverless orchestration consolidated into one process) -- do not retell on the slide. The A2 sentence ('both cut one coherent dataflow along technical rather than domain seams') is said once, on the observation-3 frame."
|
|
},
|
|
{
|
|
"no": 21,
|
|
"deck_section": "C8 -- Real-Time / IoT Streaming",
|
|
"title": "C8 -- AI lens and key concept",
|
|
"kind": "ailens",
|
|
"script_ref": "§26.3 ailinse (part3_application_classes.tex 702-704), keypoint (698-700), verdict (706)",
|
|
"content": [
|
|
"\\textbf{AI Lens [Online inference and drift on streams (Axis B)]:} C8 is where ML meets its hardest operational constraint -- models score events \\emph{in the stream} (fraud, anomaly, predictive maintenance) at \\textbf{millisecond budgets}, which rules out per-event LLM calls entirely and demands pre-trained, locally served models",
|
|
"The stream is simultaneously the \\textbf{monitoring instrument}: input-distribution statistics computed in the pipeline detect data and concept drift; production-readiness rubrics such as the ML Test Score assume exactly this telemetry",
|
|
"Architecturally, the model is \\textbf{one more stateful stream operator} -- checkpointed, versioned, replayable like any other -- so that S2's exactly-once discipline extends to inference results",
|
|
"\\textbf{Key concept:} C8 must keep pace with the production rate of the world -- indefinitely. Its genuinely hard requirement is \\emph{not throughput} but \\textbf{declared correctness semantics under failure}: exactly-once state and event-time correctness are bought with checkpointing machinery and permanent operational competence, so the pragmatic norm is at-least-once plus idempotence -- decided per stream, per scenario, and recorded as such. \\textbf{In one sentence:} EDA over a durable log with streaming PF as the processing layer -- the documented Uber stack (EDA at $++$; PF and MS at $+$) -- microservice processors secondary where team ownership demands it"
|
|
],
|
|
"elements": [
|
|
"ailinse[Online inference and drift on streams (Axis B)] (lines 702-704, condensed) above keypoint (698-700, condensed) closing with the verdict (line 706) -- 'In one sentence' inside the keypoint as in deck 6 (line 636)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Contrast with C7's AI lens: C7 docks an LLM behind a semantic layer at seconds; C8 forbids LLM calls per event -- the class profile decides the AI integration, not the other way round."
|
|
},
|
|
{
|
|
"no": 22,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- Collaboration / messaging",
|
|
"kind": "content",
|
|
"script_ref": "§27 intro and 27.1 (part3_application_classes.tex 711-720)",
|
|
"content": [
|
|
"\\emph{\\textcolor{bankblue}{Five million people are connected at this moment, and every message must reach its channel within half a second: what holds the connections?}}",
|
|
"The class: chat, team-collaboration and messaging applications -- a resource profile classical request/response never modelled: the system holds \\textbf{millions of long-lived, stateful connections} (WebSockets) and pushes every message to all channel members immediately; latency must feel instantaneous; ordering \\textbf{causal per channel} -- global ordering unnecessary, a deliberate weakening that buys scalability (note 12)",
|
|
"\\textbf{1. The connections themselves} -- the boot storm, not the message rate, threatens the core databases: the binding load is \\textbf{connection fan-out and push} $\\to$ D1/D2 Medium (note 9); the crux is D3/D5: a message must reach its channel in 100--500\\,ms, and \\emph{a disconnected messenger is a dead messenger}",
|
|
"\\textbf{2. The history tail} -- message histories are enormous and users scroll deep into them expecting low-tens-of-milliseconds reads: a D3 demand on the read path",
|
|
"\\textbf{3. Consistency relaxed by design} -- causal order per channel suffices (D4 Medium, note 12), the deliberate weakening that makes the fan-out scalable; feature churn high while the core protocol stays stable (D7 Medium); lifetimes exceed ten years"
|
|
],
|
|
"elements": [
|
|
"leading question in bankblue italics; numbered list of three challenges, two lines each (deck-6 'Part III opens' format, lines 528-541)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "The Slack and Discord numbers, the 'one write and N push-deliveries' sentence and the load-wave sentence move to the next frame; here only the mechanism."
|
|
},
|
|
{
|
|
"no": 23,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- the connections and the history tail: two measured cases",
|
|
"kind": "case",
|
|
"script_ref": "§27.1 (part3_application_classes.tex 718-720); Discord recap of deck 4 examplebox 'Storage migrations behind a seam -- Discord' (AISE502_Vorlesung_4_Folien.tex 670-671)",
|
|
"content": [
|
|
"\\textbf{Slack} | connection storms | more than \\textbf{five million} simultaneous WebSocket sessions at peak; a morning peak rolling across a timezone means millions of clients connecting and booting within minutes | application-level edge cache \\textbf{Flannel} at global points of presence: p99 latency for channel-membership queries from roughly \\textbf{2{,}000\\,ms to roughly 200\\,ms}, shielding the core from the boot storms",
|
|
"\\textbf{Discord} (week 4: the same migration as the D7/D9 seam story; today the D3 demand on the storage tier) | the history tail | stores \\textbf{trillions of messages}; users scroll deep expecting low-tens-of-milliseconds reads | storage migrations MongoDB $\\to$ Cassandra $\\to$ ScyllaDB behind Rust data services performing request coalescing: p99 read latency from \\textbf{40--125\\,ms to roughly 15\\,ms}",
|
|
"Reading: every message is one write and $N$ push-deliveries; load arrives as office-hour waves travelling with the timezones (Slack) or as evening peaks (Discord) -- Slack is the D5 scenario in the class's \\emph{connection} reading (note 9), Discord the D3 demand on the \\emph{storage} tier: two tails, two tiers, one class"
|
|
],
|
|
"elements": [
|
|
"two-row table (footnotesize, p{} widths): System | Problem | Magnitude | Measure and result -- from lines 718-720, with a recap marker in the Discord cell; one reading line (lines 718-720)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Discord's number becomes S3's response measure on the next frame; Slack's Flannel case is the scenario S2 was built for (S2's response measure is 'connection establishment and initial payload within budget while protecting the core databases'). The measured production number becomes the scenario's response measure, exactly as the closing keypoint of today claims."
|
|
},
|
|
{
|
|
"no": 24,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- the binding scenarios",
|
|
"kind": "content",
|
|
"script_ref": "§27.1 scenarios and K(C9) (part3_application_classes.tex 722-728)",
|
|
"content": [
|
|
"\\textbf{S1 (delivery latency).} A user sends a channel message in normal global operation; all online channel members receive it \\textbf{within 100--500\\,ms, in causal order per channel}. \\textcolor{codegray}{(The D3/D4 scenario.)}",
|
|
"\\textbf{S2 (connection fan-out).} Morning peak rolls across a timezone; millions of clients connect and boot with \\textbf{connection establishment and initial payload within budget while protecting the core databases} -- the exact scenario Slack built its edge cache for. \\textcolor{codegray}{(The D5 scenario in the connection reading, note 9.)}",
|
|
"\\textbf{S3 (history read tail).} A user scrolls deep into channel history; the store serves the read with \\textbf{p99 read latency in the low tens of milliseconds} -- the target of Discord's migrations. \\textcolor{codegray}{(A D3 demand on the storage tier.)}",
|
|
"$K(\\text{C9})$: the GDPR, with \\textbf{data residency} as an explicit selling argument in the self-hosted segment; in enterprise deployments, \\textbf{compliance archiving} (MiFID~II and FINRA record-keeping rules for trader communication) makes exportable, tamper-evident message archives a hard requirement",
|
|
"\\textbf{Knock-out reading:} for regulated enterprise customers, an architecture that cannot guarantee complete capture and retention of business communication is excluded -- precisely the market logic behind \\textbf{Mattermost's single-binary, self-hostable design}"
|
|
],
|
|
"elements": [
|
|
"three-scenario itemize; K paragraph (deck-6 'C10 -- the binding scenarios' format, lines 543-553)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "The Mattermost line sets up the 'four structures' frame and the Discussion."
|
|
},
|
|
{
|
|
"no": 25,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- requirements profile (column C9)",
|
|
"kind": "table",
|
|
"script_ref": "§27.2 tab:req-c9 (part3_application_classes.tex 734-760)",
|
|
"content": [
|
|
"D1 Read scalability | M | the binding load is connection fan-out and push, not raw reads (note 9)",
|
|
"D2 Write scal.\\ \\& elasticity | M | one write becomes $N$ push deliveries -- amplification, not raw write rate (note 9)",
|
|
"D3 Latency | \\textbf{H} | delivery within 100--500\\,ms or the conversation dies (S1, S3)",
|
|
"D4 Consistency \\& integrity | M | causal order per channel suffices; global order is deliberately waived (note 12)",
|
|
"D5 Availability \\& isolation | \\textbf{H} | presence is the product: a disconnected messenger is a dead messenger (S2)",
|
|
"D6 Security \\& auditability | M | enterprise segments harden archiving duties into $K(a)$; consumer segments do not",
|
|
"D7 Evolvability | M | feature churn is high while the core protocol stays stable",
|
|
"D8 Simplicity \\& TTM | M | a small install must stay operable -- the self-hosted market demands it",
|
|
"D9 Testability \\& deployability | M | staged rollouts across a heterogeneous client fleet",
|
|
"D10 Operating cost | M | connection infrastructure costs scale with concurrency, not usage",
|
|
"D11 Team scaling | M | platform teams for gateway and storage, feature teams above",
|
|
"D12 AI integrability | L | assistants and summarisation decorate the edges; the fan-out core is untouched",
|
|
"Shape $S(a)$: interactive, connection-stateful | Constraints $K(a)$: compliance archiving (MiFID/FINRA) in enterprise segments"
|
|
],
|
|
"elements": [
|
|
"12-row requirements-profile table (scriptsize, arraystretch 0.8) from tab:req-c9 lines 740-758; Highs bold (deck-6 profile format, lines 555-582)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Read aloud only the two Highs (D3, D5), the footnoted D1/D2/D4 cells and the one L (D12): the flattest profile of the day -- which is exactly why scale and business model move its edges (next frame)."
|
|
},
|
|
{
|
|
"no": 26,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- same domain, four structures",
|
|
"kind": "table",
|
|
"script_ref": "§27.3 (part3_application_classes.tex 764-766)",
|
|
"content": [
|
|
"\\textbf{Mattermost} | \\emph{one} Go binary with PostgreSQL -- the monolith as a \\emph{feature} for self-hosting and compliance customers | MM column: its D8/D10 strengths are what the enterprise $K(a)$ segment buys; D5 weakness mitigated by replicated instances",
|
|
"\\textbf{Rocket.Chat} | Node.js and MongoDB | sits in the middle",
|
|
"\\textbf{Slack} | hybrid: PHP/Hack monolithic core for persistence and business logic, a specialised distributed real-time push layer in front | the push layer is publish/subscribe fan-out -- the EDA column owns this row: message delivery \\emph{is} event fan-out with causal per-channel ordering; domain semantics and pattern coincide",
|
|
"\\textbf{Discord} | polyglot specialisation per quality attribute: BEAM for connection concurrency, Rust for tail latency | targeted microservices exactly where the D3 tail demands them",
|
|
"\\textbf{The trade-off (A2) is resolved by scale and business model, not by the domain:} below roughly $10^5$ concurrent connections a specialised push layer is premature optimisation -- \\textbf{the measured concurrent-connection count is the tiebreaker}"
|
|
],
|
|
"elements": [
|
|
"four-row table (scriptsize, p{} widths): System | Structure | Which column, and why -- from line 764; one tiebreaker line below (line 766)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "First of two C9 'what real systems chose' frames (restores the class rhythm: case frame + key-concept frame). Matrix/Synapse ('adds a federated counter-model', line 764) goes to the speaker notes -- mention it as the fifth structure. Keep the Slack row to two lines."
|
|
},
|
|
{
|
|
"no": 27,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "C9 -- key concept",
|
|
"kind": "keyconcept",
|
|
"script_ref": "§27.3 keypoint (part3_application_classes.tex 768-770) and verdict (776)",
|
|
"content": [
|
|
"\\textbf{Key concept:} C9's binding load is connection fan-out and push, not raw read/write rates -- a resource profile request/response never modelled. The same business domain spans Mattermost's single, deliberately monolithic binary and Discord's polyglot specialised cluster: \\textbf{the profile's core is stable, but scale and business model move the weights at its edges} -- measured concurrent-connection count is the tiebreaker. \\textbf{In one sentence:} a hybrid -- monolithic persistence and business core with a specialised event-driven push layer, Slack's documented shape (EDA at $++$ at the fan-out crux; MM and MS at $+$) -- with a modular monolith alone as secondary for the self-hosted segment"
|
|
],
|
|
"elements": [
|
|
"keypoint box (lines 768-770, condensed) closing with the verdict (line 776) -- 'In one sentence' inside the keypoint as in deck 6 (line 636)"
|
|
],
|
|
"minutes": 2,
|
|
"notes": "Short frame; leads straight into the Discussion. 'Weights at the edges' is the sentence the Discussion debrief returns to."
|
|
},
|
|
{
|
|
"no": 28,
|
|
"deck_section": "C9 -- Collaboration / Messaging",
|
|
"title": "Discussion",
|
|
"kind": "discussion",
|
|
"script_ref": "§27.3 thinkbox (part3_application_classes.tex 772-774)",
|
|
"content": [
|
|
"Mattermost and Discord implement substantially the same user-visible functionality. Using the twelve dimensions:",
|
|
"Which \\emph{three} dimensions differ most strongly between the requirements profile of a \\textbf{self-hosted compliance deployment (500 users, banking client)} and a \\textbf{global consumer platform (150 million users)}?",
|
|
"Would any weight flip from L to H?",
|
|
"What does that imply for the claim that ``the application class determines the profile''?"
|
|
],
|
|
"elements": [
|
|
"thinkbox with the three questions from lines 772-774 (deck 1/3/5 'Discussion' frame format: plain title, question inside the box)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Buzz groups, 2 minutes, then collect. Anchor the debrief on the C9 table (D6 archiving into K(a), D8/D10 small install, D5/D3 tail at scale, D11) and on week 6's C1/C2 lesson: weights, not dimensions, define a class -- and here scale and business model move the weights at the edges while the core (D3/D5 High) stays."
|
|
},
|
|
{
|
|
"no": 29,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "Stepping back: ten profiles side by side -- three reading rules",
|
|
"kind": "content",
|
|
"script_ref": "§29 intro (part3_application_classes.tex 853-858); D4/D7 observations read off tab:requirements (869-894, 904-908)",
|
|
"content": [
|
|
"Ten times the same movement -- what no single section could deliver is the view \\emph{across} the ten; that is what this section adds",
|
|
"The consolidated table: $w_i(a)$, $S(a)$, $K(a)$ per class -- the \\textbf{demand-side counterpart of the capability table} (week 6), one of the two inputs to the fit matrix of Part IV",
|
|
"\\textbf{Rule 1 -- read columns to see a class:} each column repeats the per-class table of its section cell for cell; every H is backed by a binding scenario with a response measure",
|
|
"\\textbf{Rule 2 -- read rows to see what a dimension discriminates:} D7 (evolvability) is High in six classes and Low in none -- long life is the norm, not the exception -- and Medium exactly in the four classes taught today; D12 is High only in C10, the definitional case",
|
|
"\\textbf{Rule 3 -- the seventeen footnotes are part of the table's semantics, not decoration:} testability as reproducibility in C6 (note 14), as eval pass rate in C10 (note 15), read scalability as OLAP scan capacity in C7 (note 7); the D4 row of C6--C9 carries the same letter M with four footnoted meanings (notes 8, 10, 11, 12) -- \\emph{using the bare letter without its footnote produces wrong matches in Part IV}"
|
|
],
|
|
"elements": [
|
|
"numbered reading rules, \\footnotesize, in the deck-6 'Four reading rules -- and the table's status' format (line 469)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Mirror the deck-6 capability-table frames explicitly: same movement, other operand. The two C6--C9 observations under Rules 2 and 3 (D7 Medium exactly in today's four classes; D4 = M with four footnoted meanings) are read off the table, not stated in the script -- present them as such."
|
|
},
|
|
{
|
|
"no": 30,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "The consolidated requirements table (1/2): D1--D6",
|
|
"kind": "table",
|
|
"script_ref": "§29 tab:requirements rows D1--D6 (part3_application_classes.tex 864-880)",
|
|
"content": [
|
|
"Header: Dimension | C1 core banking | C2 social/content | C3 back-office | C4 ERP | C5 e-commerce | C6 simulation/batch | C7 DSS/BI | C8 IoT streaming | C9 collab/messaging | C10 AI-native advisory",
|
|
"D1 Read scalability | M | \\textbf{H} | L | L | \\textbf{H} | L | \\textbf{H}$^{7}$ | M | M$^{9}$ | M",
|
|
"D2 Write scal.\\ \\& elasticity | M | M$^{1}$ | L | L | \\textbf{H} | \\textbf{H}$^{2}$ | L | \\textbf{H} | M$^{9}$ | L",
|
|
"D3 Latency | M$^{3}$ | \\textbf{H} | M | M | \\textbf{H} | L$^{4}$ | M | \\textbf{H} | \\textbf{H} | M$^{5}$",
|
|
"D4 Consistency \\& integrity | \\textbf{H} | L | \\textbf{H} | \\textbf{H} | M$^{6}$ | M$^{8}$ | M$^{10}$ | M$^{11}$ | M$^{12}$ | M$^{13}$",
|
|
"D5 Availability \\& isolation | \\textbf{H} | \\textbf{H} | L | M | \\textbf{H} | L | L | \\textbf{H} | \\textbf{H} | M",
|
|
"D6 Security \\& auditability | \\textbf{H} | M | \\textbf{H} | \\textbf{H} | \\textbf{H} | M | M | L | M | \\textbf{H}",
|
|
"Footer line: superscripts refer to the seventeen footnotes (see next frames) -- integral to the cell semantics"
|
|
],
|
|
"elements": [
|
|
"10-column table from lines 869-880: \\scriptsize, \\tabcolsep 2pt, p{2.6cm} label column + ten centred p{1.1cm} class columns, two-line \\shortstack headers, Highs bold, superscripts via \\textsuperscript -- the same column spec as (2/2) so the halves read as one table (deck-6 'The consolidated capability table' format, line 439)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Width check: 2.6 cm + 10 x 1.1 cm + tabcolsep fits 16:9 at scriptsize. Walk the D4 row aloud: three H, one L, six footnoted M (notes 6, 8, 10, 11, 12, 13) -- the consistency-semantics axis of the class map."
|
|
},
|
|
{
|
|
"no": 31,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "The consolidated requirements table (2/2): D7--D12 and $S(a)$",
|
|
"kind": "table",
|
|
"script_ref": "§29 tab:requirements rows D7--D12 and shape row (part3_application_classes.tex 881-889)",
|
|
"content": [
|
|
"Header as on (1/2): C1 ... C10",
|
|
"D7 Evolvability | \\textbf{H} | \\textbf{H} | \\textbf{H} | \\textbf{H} | \\textbf{H} | M | M | M | M | \\textbf{H}",
|
|
"D8 Simplicity \\& TTM | L | M | \\textbf{H} | M | M | M | M | L | M | M",
|
|
"D9 Testability \\& deployability | \\textbf{H} | \\textbf{H} | M | M | \\textbf{H} | \\textbf{H}$^{14}$ | M | M | M | \\textbf{H}$^{15}$",
|
|
"D10 Operating cost | L | M | \\textbf{H} | M | M | \\textbf{H} | \\textbf{H} | M | M | \\textbf{H}$^{16}$",
|
|
"D11 Team scaling | M | \\textbf{H} | L | M | M$^{17}$ | L | L | M | M | L",
|
|
"D12 AI integrability | M | M | L | L | M | M | M | M | L | \\textbf{H}",
|
|
"Dominant workload shape $S(a)$ | interactive ($+$ batch edges) | interactive read-heavy | interactive | interactive ($+$ batch runs) | interactive, seasonal | scheduled batch | batch $+$ thin serving | continuous stream | interactive, connection-stateful | hybrid: interactive $+$ batch/async"
|
|
],
|
|
"elements": [
|
|
"10-column table continued with the identical column spec (\\scriptsize, \\tabcolsep 2pt, p{2.6cm} + ten centred p{1.1cm}, \\shortstack headers); the $S(a)$ row rendered in \\tiny so the phrases wrap within 1.1 cm -- six data rows leave the room"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Point at the D12 row: one H, the definitional case; and at D7: six H, no L, M exactly in C6--C9."
|
|
},
|
|
{
|
|
"no": 32,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "Hard constraints $K(a)$ -- the knock-out row",
|
|
"kind": "table",
|
|
"script_ref": "§29 tab:requirements K(a) row (part3_application_classes.tex 891-892); §24.1 (514) and §26.1 (658) for the reading line",
|
|
"content": [
|
|
"C1 core banking: BCBS 239; FINMA 2023/1; DORA (EU 2022/2554); ACID booking core; 10+ yr retention",
|
|
"C2 social/content: DSA; GDPR",
|
|
"C3 back-office: retention law (e.g.\\ OR 958\\emph{f}); internal control systems; GDPR",
|
|
"C4 ERP: GoBD/GeB\\\"uV; SOX; IFRS audit chains",
|
|
"C5 e-commerce: PCI DSS v4.0 scope minimisation; PSD2/SCA",
|
|
"C6 simulation/batch: model validation and backtesting (finance); deterministic seeds",
|
|
"C7 DSS/BI: BCBS 239 lineage (banks)",
|
|
"C8 IoT streaming: per-key ordering; exactly-once where stated",
|
|
"C9 collab/messaging: compliance archiving (MiFID/FINRA) in enterprise segments",
|
|
"C10 AI-native advisory: EU AI Act 2024/1689 (logging, oversight; potentially high-risk); GDPR",
|
|
"Reading: $K(a)$ feeds stage 1 of the match (knock-out) together with the shape gate; C8's entry is \\emph{technical rather than statutory}; C6's pairs a supervisory duty (Basel model validation and backtesting) with a technical one (deterministic seeds)"
|
|
],
|
|
"elements": [
|
|
"two-column layout (C1--C5 left, C6--C10 right), \\footnotesize, from the K(a) row line 892; closing reading line"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Five per column; the reading line paraphrases line 658 ('technical rather than statutory') and line 514 (Basel model validation, BCBS 239 lineage; deterministic seeds as a hard constraint)."
|
|
},
|
|
{
|
|
"no": 33,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "The seventeen footnotes are part of the semantics (1/2): notes 1--9",
|
|
"kind": "content",
|
|
"script_ref": "§29 tab:requirements footnotes 1-9 (part3_application_classes.tex 896-905)",
|
|
"content": [
|
|
"$^{1}$ C2, D2 = M: raw write rate is modest (Twitter: ca.\\ 6\\,k tweet writes/s vs.\\ ca.\\ 300\\,k timeline reads/s); fan-out amplification is real but absorbed by the fan-out subsystem, not the write path",
|
|
"$^{2}$ C6, D2 = H: throughput as makespan and batch-window compute bursts, not requests/s",
|
|
"$^{3}$ C1, D3 = M: class-internal spread of orders of magnitude -- batch clearing hours, instant payments $<10$\\,s end-to-end, card authorisation 1--2\\,s, trading microseconds",
|
|
"$^{4}$ C6, D3 = L: per-request latency irrelevant; makespan and batch window are the measures",
|
|
"$^{5}$ C10, D3 = M: users accept seconds-to-minutes for advisory answers; the High-grade concern is cost per request, not speed",
|
|
"$^{6}$ C5, D4 = M: the class-defining hybrid -- catalogue eventual, checkout/payment strictly ACID",
|
|
"$^{7}$ C7, D1 = H: ``read'' means heavy OLAP scans by few users; serving typically delegated to the warehouse",
|
|
"$^{8}$ C6, D4 = M: bit-level reproducibility (seeds, versioned inputs) -- a scientific and, for risk models, regulatory duty",
|
|
"$^{9}$ C9, D1/D2 = M: connection fan-out and push, not raw read/write rates, are the binding load (Slack: $>5$\\,M concurrent WebSockets); hence M with the crux carried by D3/D5"
|
|
],
|
|
"elements": [
|
|
"nine-item list, \\scriptsize, each note prefixed by its cell (class, dimension, letter) -- from lines 897-905"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Nine lines at \\scriptsize fit; the cell prefix (e.g. 'C6, D2 = H') is the deck's navigation aid, derived from the table positions."
|
|
},
|
|
{
|
|
"no": 34,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "The seventeen footnotes are part of the semantics (2/2): notes 10--17 and the rule",
|
|
"kind": "keyconcept",
|
|
"script_ref": "§29 tab:requirements footnotes 10-17 (part3_application_classes.tex 906-913) and keypoint (860-862)",
|
|
"content": [
|
|
"$^{10}$ C7, D4 = M: a third semantics beside ACID and eventual -- freshness by refresh contract (``as of yesterday 24:00'')",
|
|
"$^{11}$ C8, D4 = M: stream semantics -- exactly-once state, event-time windows, watermarks",
|
|
"$^{12}$ C9, D4 = M: causal order per channel suffices; global order unnecessary",
|
|
"$^{13}$ C10, D4 = M: hybrid -- knowledge index eventual, transaction/audit path ACID",
|
|
"$^{14}$ C6, D9 = H: reproducibility (golden datasets, versioned pipelines) is the operative meaning of testability",
|
|
"$^{15}$ C10, D9 = H: evals are the operative meaning of testability (Part V)",
|
|
"$^{16}$ C10, D10 = H: cost per \\emph{request} (tokens, GPU) -- a run-cost class no classical profile contains",
|
|
"$^{17}$ C5, D11 = M: organisation-size dependent -- the Shopify-vs-Amazon contrast",
|
|
"\\textbf{Key concept:} an H in the table is a \\textbf{veto trigger} backed by a scenario with a response measure; an L is a \\textbf{licence to not pay} for that dimension. The footnotes carry class-specific semantics that the letters alone cannot: \\emph{whoever matches profiles by letters without footnotes has not read the table}"
|
|
],
|
|
"elements": [
|
|
"eight-item list \\scriptsize (lines 906-913) above the keypoint box (lines 860-862)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "The D4 row's footnoted M cells -- notes 6, 8 and 10--13 -- make the Rule-3 point again: one letter, six semantics."
|
|
},
|
|
{
|
|
"no": 35,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "Reading rows: what discriminates -- the two questions to ask first",
|
|
"kind": "table",
|
|
"script_ref": "§29.1 observation 1 (part3_application_classes.tex 918); §29 reading rule 2 (858); H positions counted from tab:requirements (869-889)",
|
|
"content": [
|
|
"Dimension | High in | count",
|
|
"D1 | C2, C5, C7 | 3",
|
|
"D2 | C5, C6, C8 | 3",
|
|
"D3 | C2, C5, C8, C9 | 4",
|
|
"D4 | C1, C3, C4 | 3",
|
|
"D5 | C1, C2, C5, C8, C9 | 5",
|
|
"D6 | C1, C3, C4, C5, C10 | 5",
|
|
"D7 | C1, C2, C3, C4, C5, C10 | 6 (Low in none)",
|
|
"D8 | C3 | 1",
|
|
"D9 | C1, C2, C5, C6, C10 | 5",
|
|
"D10 | C3, C6, C7, C10 | 4",
|
|
"D11 | C2 | 1",
|
|
"D12 | C10 | 1",
|
|
"\\textbf{Observation 1 -- the strongest discriminators are the read/write ratio (the D1/D2 rows) and the consistency semantics (the D4 row).} C1 (write-critical, strict ACID) and C2 (roughly 50:1 read-heavy, deliberately eventual) are mirror images across both at once -- the fastest diagnostic pair in the catalogue. \\emph{When classifying an unknown system, ask these two questions first}: they eliminate more patterns via the shape gate and the D4 row than any other pair"
|
|
],
|
|
"elements": [
|
|
"12-row, 3-column table (scriptsize, arraystretch 0.8) derived by counting the H cells of tab:requirements, flagged on the slide as 'read off the table'; observation 1 as a three-line \\footnotesize paragraph below"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "The counts are a deck construction (see open_issues); the script states only the D7 and D12 facts (line 858, already on the reading-rules frame). Observation 1's C5/C6--C7 clause (hybrid; third semantics family) is left to the definition frame and the summary; the derived remark that D8 and D11 single out one class each (C3, C2) and D4 is High only in the three ACID classes goes to the speaker notes."
|
|
},
|
|
{
|
|
"no": 36,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "Reading the catalogue as a whole (1/2): scale does not force microservices",
|
|
"kind": "content",
|
|
"script_ref": "§29.1 observation 2 (part3_application_classes.tex 920-924)",
|
|
"content": [
|
|
"\\textbf{Observation 2 -- scale does not force microservices.} The documented monolith-at-scale cases span four classes \\textcolor{codegray}{(all five cases: weeks 3--6; now read from the demand side)}:",
|
|
"Instagram | global consumer platform | horizontally replicated Django monolith",
|
|
"Stack Overflow | $\\sim 1.3$ billion page views/month, $> 6{,}000$ requests/s | roughly nine on-premise web servers, monolithic .NET application",
|
|
"Shopify | ca.\\ 280 million requests/min at BFCM peak | pod-sharded modular Rails monolith",
|
|
"LMAX | six million orders/s | one thread",
|
|
"Stateless replication, sharding and mechanical sympathy scale monoliths far; what microservices primarily solve is an \\emph{organisational} scaling problem -- deployments per developer as team count grows. \\textbf{In profile terms: D1/D2 rarely force distribution; D11 = H is what does}"
|
|
],
|
|
"elements": [
|
|
"heading bullet with visible recap marker in codegray; four-row table (\\scriptsize): System | Documented scale | Structure -- from line 924; closing two-line paragraph ending on the bold line"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Monzo's 2,800 services ('viable only with extreme homogeneity and central migration automation', line 924) goes to the speaker notes -- week 5 taught the homogeneity condition. Do not retell the cases; they were the supply-side evidence in weeks 3--6."
|
|
},
|
|
{
|
|
"no": 37,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "Reading the catalogue as a whole (2/2): rollbacks, regulation, inheritance",
|
|
"kind": "content",
|
|
"script_ref": "§29.1 observations 3-5 (part3_application_classes.tex 926-930)",
|
|
"content": [
|
|
"\\textbf{3. The rollback case studies share one pattern.} Segment and Prime Video both cut \\emph{one} coherent dataflow along technical rather than domain seams -- one service per integration destination, one function per processing step -- and the interaction costs (queues, orchestrator state transitions, storage round-trips, the test matrix) consumed the benefit. Granularity must follow the coupling structure of the domain. For the catalogue: \\textbf{a class's workload shape is not merely a gate for pattern selection -- it also predicts where the expensive boundaries lie \\emph{within} a pattern}",
|
|
"\\textbf{4. Regulation is a first-class architectural requirement.} $K(a)$ translates directly into structure: BCBS~239 and FINMA 2023/1 into immutable journals, lineage and resilience testing; DORA into incident-reporting and third-party risk machinery; PCI~DSS v4.0 into cardholder-data zoning and scope minimisation; the EU AI Act into logging and human-oversight duties for AI components. None can be bolted on late at reasonable cost -- \\textbf{Booch-significant decisions}, which is exactly why they live in $K(a)$ as knock-outs rather than in the weights",
|
|
"\\textbf{5. The AI-native class inherits before it innovates.} C10 inherits the disciplines of C6/C7 (batch index and eval pipelines) and C3 (deterministic transactional services), and adds exactly \\textbf{two} genuinely new elements: a cost model that is expensive per request rather than per user, and evals as the operative meaning of testability. \\emph{Design instruction for the project in one sentence: solve the inherited profiles with the inherited, well-understood structures, and concentrate the architectural attention on the two novelties}"
|
|
],
|
|
"elements": [
|
|
"three paragraph bullets, \\footnotesize, numbered 3--5 to continue the previous frame"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Observation 3 is where the Segment/Prime Video A2 sentence is said once today (the C8 case frame only pointed here) and anticipates Lecture 10 (hybrids and evolution paths: Segment, Prime Video, Shopify) -- say so. Observation 5 is the 'closing the loop' frame restated by the script itself."
|
|
},
|
|
{
|
|
"no": 38,
|
|
"deck_section": "Stepping Back: Ten Profiles Side by Side",
|
|
"title": "The other operand is ready",
|
|
"kind": "discussion",
|
|
"script_ref": "§29.1 keypoint (part3_application_classes.tex 932-934) and thinkbox (936-938)",
|
|
"content": [
|
|
"\\textbf{Key concept:} the ten requirements profiles are \\textbf{derivable, not arbitrary}: every weight traces to binding scenarios with response measures, every constraint to a statute or a contract, every workload signature to documented production numbers. \\textbf{The other operand is ready}: Part IV matches these profiles against the capability profiles of Part II cell by cell (the fit matrix), and every cell can be traced back to $R(a) \\times C(p)$ rather than to fashion or taste",
|
|
"\\textbf{Discussion:} the catalogue contains ten classes -- why not nine, why not fourteen?",
|
|
"Propose a candidate eleventh class (for example: computer games, embedded control, blockchain ledgers) and test it against the definition of an application class (a \\emph{recurring} ASR bundle with typical response measures, a characteristic workload shape, a regulatory frame)",
|
|
"Does its profile differ from all ten columns of the consolidated table on at least \\textbf{two High weights}? If it does not differ, it is an instance, not a class"
|
|
],
|
|
"elements": [
|
|
"keypoint box (lines 932-934, condensed) above a thinkbox (lines 936-938) -- deck-6 'Reading the catalogue as a whole' frame format (line 484)"
|
|
],
|
|
"minutes": 4,
|
|
"notes": "Run the Discussion as a 3-minute pair exercise with the two table frames still in reach (students need the columns to test 'two High weights'); one minute for collection. Close by pointing forward to Lecture 10's row-by-row cell rationales."
|
|
},
|
|
{
|
|
"no": 39,
|
|
"deck_section": "Closing",
|
|
"title": "This week's exercise: the walking skeleton runs end-to-end (M3 closes)",
|
|
"kind": "exercise",
|
|
"script_ref": "project_exercise.tex 355-466 (M3 taskbox, hintbox); semester plan week 9; §24.3 and §25.3 projektboxes (one-line pointer)",
|
|
"content": [
|
|
"\\textbf{Coaching session (1 lesson): finish the walking skeleton.}",
|
|
"\\textbf{Milestone check M3 (week 9):} the end-to-end thin slice runs -- \\texttt{MarketDataService} delivers prices $\\to$ a \\emph{minimal} \\texttt{ResearchAgent} produces \\textbf{one validated \\texttt{Insight}} $\\to$ through the \\textbf{stable API}; a placeholder UI calls it",
|
|
"Checklist from the exercise sheet: keep the deterministic core \\textbf{free of LLM calls} (the line that is graded); use a \\textbf{snapshot} of market/news data so demo and grading are reproducible even if the live APIs misbehave; commit after the milestone; record decisions as ADRs as you go",
|
|
"Today's lecture applied: your ingestion and eval pipelines are \\textbf{C6}, your analytics are \\textbf{C7} -- the skeleton should already show where an \\texttt{Insight}'s freshness stamp comes from",
|
|
"Next: weeks 10--11 (M4) -- \\texttt{Performance}, \\texttt{Risk}, \\texttt{Optimization} services with exact tests against the reference vectors; resilience on all external calls"
|
|
],
|
|
"elements": [
|
|
"projektbox with the milestone check and checklist (deck-6 'This week's exercise: the match' format, lines 645-658)"
|
|
],
|
|
"minutes": 3,
|
|
"notes": "Only the first three bullets are mandatory; the one-line 'today's lecture applied' bullet is the link back to the 'Closing the loop' frame, which owns the projektbox substance."
|
|
},
|
|
{
|
|
"no": 40,
|
|
"deck_section": "Closing",
|
|
"title": "Summary",
|
|
"kind": "summary",
|
|
"script_ref": "keypoints §24-27, §29 (part3_application_classes.tex 554, 624, 698, 768, 860, 932)",
|
|
"content": [
|
|
"1. \\textbf{C6} replaces the interactive vocabulary wholesale -- makespan, batch-window adherence, cost per simulation; correctness = bit-level reproducibility (Highs D2/D9/D10); with C7 it defines the \\textbf{third consistency semantics}: ``as of this run'' and ``as of yesterday 24:00'' -- contracts with response measures",
|
|
"2. \\textbf{C7}: an integration product over C1--C5; Highs D1 (OLAP scans, delegated) and D10 (the warehouse bill); the ADR to write is the freshness contract; tightening it towards real time is a class change into C8",
|
|
"3. \\textbf{C8}: keep pace with the world indefinitely; the hard requirement is declared correctness semantics under failure (Highs D2/D3/D5); at-least-once $+$ idempotence decided per stream, per scenario",
|
|
"4. \\textbf{C9}: the binding load is connection fan-out and push (Highs D3/D5); same domain, four structures -- the measured concurrent-connection count is the tiebreaker",
|
|
"5. \\textbf{Five observations}: read/write ratio and consistency semantics discriminate most; scale does not force microservices (D11 = H does); rollbacks cut dataflows along technical seams; regulation lives in $K(a)$; C10 inherits before it innovates",
|
|
"6. \\textbf{The consolidated table}: an H is a veto trigger backed by a scenario, an L a licence not to pay; the seventeen footnotes are part of the semantics -- \\textbf{the other operand is ready}: $R(a) \\times C(p)$, every cell traceable, none to fashion or taste"
|
|
],
|
|
"elements": [
|
|
"enumerate \\footnotesize, itemsep 2pt, six points of at most two lines (deck-6 'Summary' format, line 660)"
|
|
],
|
|
"minutes": 2,
|
|
"notes": "Six points (~13 lines): C6 and the third semantics merged; the table and 'the other operand is ready' merged."
|
|
},
|
|
{
|
|
"no": 41,
|
|
"deck_section": "Closing",
|
|
"title": "Next week",
|
|
"kind": "nextweek",
|
|
"script_ref": "semester plan week 10; brief 'Next lecture' line",
|
|
"content": [
|
|
"Left: \\textbf{Lecture 10 -- Part IV continued: hybrids, evolution paths, the eight-step procedure}",
|
|
"hybrids and evolution paths: Segment, Prime Video, Shopify",
|
|
"the eight-step decision procedure with the worked course-project run (ADR-007)",
|
|
"the cell rationales of the matrix, row by row",
|
|
"Right, Reading: this week -- Part III, Sections 24--27, 29; ahead -- Part IV, Sections 33, 35--36",
|
|
"Right, Exercise / deliverable: coaching; deterministic services (\\texttt{Performance}, \\texttt{Risk}, \\texttt{Optimization}) with exact tests against the reference vectors (M4 begins)"
|
|
],
|
|
"elements": [
|
|
"two columns 0.55/0.42 (deck-6 'Next week' format, lines 673ff)"
|
|
],
|
|
"minutes": 1,
|
|
"notes": "Verbatim in substance from the brief's 'Next lecture' line."
|
|
},
|
|
{
|
|
"no": 42,
|
|
"deck_section": "Closing",
|
|
"title": "Closing slide",
|
|
"kind": "content",
|
|
"script_ref": "deck skeleton",
|
|
"content": [
|
|
"\\FHGRClosingPage -- Thank you! / Dr.\\ Florian Herzog / Fachhochschule Graub\\\"unden, Chur / AISE502 -- AI in Software Engineering II"
|
|
],
|
|
"elements": [
|
|
"FHGR closing page as in deck 6 (\\FHGRClosingPage at line 703)"
|
|
],
|
|
"minutes": 0,
|
|
"notes": "Unchanged from deck 6."
|
|
}
|
|
],
|
|
"exercise_frame": {
|
|
"title": "This week's exercise: the walking skeleton runs end-to-end (M3 closes)",
|
|
"content": [
|
|
"Coaching session (1 lesson): finish the walking skeleton",
|
|
"Milestone check M3 (week 9): the end-to-end thin slice runs -- MarketDataService delivers prices -> a minimal ResearchAgent produces one validated Insight -> through the stable API; a placeholder UI calls it",
|
|
"Checklist from the exercise sheet: keep the deterministic core free of LLM calls (the line that is graded); use a snapshot of market/news data so demo and grading are reproducible; commit after the milestone; record decisions as ADRs as you go",
|
|
"Today's lecture applied: ingestion and eval pipelines are C6, analytics are C7 -- the skeleton should already show where an Insight's freshness stamp comes from",
|
|
"Next: weeks 10--11 (M4) -- Performance, Risk, Optimization services with exact tests against the reference vectors; resilience on all external calls"
|
|
]
|
|
},
|
|
"summary": [
|
|
"C6 replaces the interactive vocabulary wholesale -- makespan, batch-window adherence, cost per simulation; correctness = bit-level reproducibility (Highs D2/D9/D10); with C7 it defines the third consistency semantics: 'as of this run' and 'as of yesterday 24:00' -- contracts with response measures",
|
|
"C7: an integration product over C1--C5; Highs D1 (OLAP scans, delegated) and D10 (the warehouse bill); the ADR to write is the freshness contract; tightening it towards real time is a class change into C8",
|
|
"C8: keep pace with the world indefinitely; the hard requirement is declared correctness semantics under failure (Highs D2/D3/D5); at-least-once + idempotence decided per stream, per scenario",
|
|
"C9: the binding load is connection fan-out and push (Highs D3/D5); same domain, four structures -- the measured concurrent-connection count is the tiebreaker",
|
|
"Five observations: read/write ratio and consistency semantics discriminate most; scale does not force microservices (D11 = H does); rollbacks cut dataflows along technical seams; regulation lives in K(a); C10 inherits before it innovates",
|
|
"The consolidated table: an H is a veto trigger backed by a scenario, an L a licence not to pay; the seventeen footnotes are part of the semantics -- the other operand is ready: R(a) x C(p), every cell traceable, none to fashion or taste"
|
|
],
|
|
"next_week": {
|
|
"lecture_line": "Lecture 10 (week 10, 3 lessons) -- Part IV continued: hybrids and evolution paths (Segment, Prime Video, Shopify); the eight-step decision procedure with the worked course-project run (ADR-007); the cell rationales of the matrix, row by row",
|
|
"topics": [
|
|
"hybrids and evolution paths: Segment, Prime Video, Shopify",
|
|
"the eight-step decision procedure with the worked course-project run (ADR-007)",
|
|
"the cell rationales of the matrix, row by row"
|
|
],
|
|
"reading": [
|
|
"this week: Part III, Sections 24--27, 29",
|
|
"ahead: Part IV, Sections 33, 35--36"
|
|
],
|
|
"exercise": [
|
|
"coaching; deterministic services (Performance, Risk, Optimization) with exact tests against the reference vectors (M4 begins)"
|
|
]
|
|
},
|
|
"script_boxes_used": [
|
|
{
|
|
"box": "definitionbox[The third consistency semantics: reproducibility and refresh contracts]",
|
|
"location": "§24.1, part3_application_classes.tex 510-512",
|
|
"used_in_frame": "6 -- The third consistency semantics: reproducibility and refresh contracts"
|
|
},
|
|
{
|
|
"box": "table tab:req-c6 (requirements profile of C6)",
|
|
"location": "§24.2, part3_application_classes.tex 520-546",
|
|
"used_in_frame": "7 -- C6 -- requirements profile (column C6)"
|
|
},
|
|
{
|
|
"box": "keypoint (C6 replaces the interactive vocabulary wholesale)",
|
|
"location": "§24.3, part3_application_classes.tex 554-556",
|
|
"used_in_frame": "9 -- C6 -- key concept"
|
|
},
|
|
{
|
|
"box": "projektbox (the course project inherits C6 wholesale: ingestion and eval pipelines)",
|
|
"location": "§24.3, part3_application_classes.tex 558-560",
|
|
"used_in_frame": "15 -- Closing the loop: C6 and C7 inside your C10 project (substance); one-line pointers on 9 and 39"
|
|
},
|
|
{
|
|
"box": "table tab:req-c7 (requirements profile of C7)",
|
|
"location": "§25.2, part3_application_classes.tex 590-616",
|
|
"used_in_frame": "12 -- C7 -- requirements profile (column C7)"
|
|
},
|
|
{
|
|
"box": "keypoint (C7 is an integration product; the ADR to write is the freshness contract)",
|
|
"location": "§25.3, part3_application_classes.tex 624-626",
|
|
"used_in_frame": "14 -- C7 -- AI lens and key concept"
|
|
},
|
|
{
|
|
"box": "ailinse[Natural-language analytics (Axis B)]",
|
|
"location": "§25.3, part3_application_classes.tex 628-630",
|
|
"used_in_frame": "14 -- C7 -- AI lens and key concept"
|
|
},
|
|
{
|
|
"box": "projektbox (the Portfolio Intelligence Platform contains a C7 subsystem; freshness contract appears twice)",
|
|
"location": "§25.3, part3_application_classes.tex 632-634",
|
|
"used_in_frame": "15 -- Closing the loop: C6 and C7 inside your C10 project (substance); one-line pointer on 39"
|
|
},
|
|
{
|
|
"box": "table tab:req-c8 (requirements profile of C8)",
|
|
"location": "§26.2, part3_application_classes.tex 664-690",
|
|
"used_in_frame": "19 -- C8 -- requirements profile (column C8)"
|
|
},
|
|
{
|
|
"box": "keypoint (C8 must keep pace with the production rate of the world)",
|
|
"location": "§26.3, part3_application_classes.tex 698-700",
|
|
"used_in_frame": "21 -- C8 -- AI lens and key concept"
|
|
},
|
|
{
|
|
"box": "ailinse[Online inference and drift on streams (Axis B)]",
|
|
"location": "§26.3, part3_application_classes.tex 702-704",
|
|
"used_in_frame": "21 -- C8 -- AI lens and key concept"
|
|
},
|
|
{
|
|
"box": "table tab:req-c9 (requirements profile of C9)",
|
|
"location": "§27.2, part3_application_classes.tex 734-760",
|
|
"used_in_frame": "25 -- C9 -- requirements profile (column C9)"
|
|
},
|
|
{
|
|
"box": "keypoint (C9's binding load is connection fan-out and push)",
|
|
"location": "§27.3, part3_application_classes.tex 768-770",
|
|
"used_in_frame": "27 -- C9 -- key concept"
|
|
},
|
|
{
|
|
"box": "thinkbox (Mattermost vs Discord: which three dimensions differ most)",
|
|
"location": "§27.3, part3_application_classes.tex 772-774",
|
|
"used_in_frame": "28 -- Discussion"
|
|
},
|
|
{
|
|
"box": "keypoint (an H is a veto trigger, an L a licence; footnotes carry the semantics)",
|
|
"location": "§29, part3_application_classes.tex 860-862",
|
|
"used_in_frame": "34 -- The seventeen footnotes are part of the semantics (2/2)"
|
|
},
|
|
{
|
|
"box": "sidewaystable tab:requirements (consolidated requirements table, 12 rows x 10 classes, S(a) row, K(a) row)",
|
|
"location": "§29, part3_application_classes.tex 864-894",
|
|
"used_in_frame": "30 -- table (1/2) D1--D6; 31 -- table (2/2) D7--D12 and S(a); 32 -- K(a) knock-out row; 35 -- row counts; 29 -- D4/D7 observations under Rules 2 and 3"
|
|
},
|
|
{
|
|
"box": "17 table footnotes of tab:requirements",
|
|
"location": "§29, part3_application_classes.tex 896-913",
|
|
"used_in_frame": "33 -- footnotes 1--9; 34 -- footnotes 10--17"
|
|
},
|
|
{
|
|
"box": "keypoint (the ten requirements profiles are derivable, not arbitrary; the other operand is ready)",
|
|
"location": "§29.1, part3_application_classes.tex 932-934",
|
|
"used_in_frame": "38 -- The other operand is ready"
|
|
},
|
|
{
|
|
"box": "thinkbox (why ten classes -- propose an eleventh and test it)",
|
|
"location": "§29.1, part3_application_classes.tex 936-938",
|
|
"used_in_frame": "38 -- The other operand is ready"
|
|
}
|
|
],
|
|
"script_boxes_dropped": [
|
|
{
|
|
"box": "figure fig:classmap (tikz map of the ten classes by workload shape and consistency semantics)",
|
|
"location": "§18.3, part3_application_classes.tex 65-105",
|
|
"reason": "Taught in deck 8 (§18); the recap frame refers to it verbally (C6/C7 at the third semantics); no thumbnail -- the recap frame is at its six-bullet limit"
|
|
},
|
|
{
|
|
"box": "table tab:req-c10, keypoint and projektbox of C10 (inheritance diagram; C10 stress-tests the method)",
|
|
"location": "§28, part3_application_classes.tex 806-846",
|
|
"reason": "Taught in full in deck 6; C10 appears today only as a column of the consolidated table and via the inheritance loop closed in frame 15"
|
|
}
|
|
],
|
|
"open_issues": [
|
|
"No deck-8 file exists in Folien/ (only AISE502_Vorlesung_1..6); the recap frame (3) is built from the task brief's description of deck 8 and the script's §18.3 -- verify wording and the C1--C5 verdicts against the actual deck 8 once it is typeset. The slide itself says 'last week (C1--C5)', never 'deck 8'.",
|
|
"Frame 35 (reading rows): the per-row H counts and positions are derived by counting the cells of tab:requirements; the script itself states only 'D7 High in six classes and Low in none' and 'D12 High only in C10' (line 858). The counts are faithful to the table but are a deck construction -- flagged on the slide as 'read off the table'.",
|
|
"Frame 29 (reading rules) now carries the two C6--C9 observations formerly on the dropped excerpt frame (D7 Medium in exactly the four classes taught today; D4 = M with four footnoted meanings, notes 8/10/11/12): both are read off the table, not stated in the script -- presented as such.",
|
|
"The script provides no thinkbox for C6, C7 or C8 and no ailinse for C6 or C9; the deck therefore has Discussion frames only for C9 (§27) and §29, and AI-lens frames only for C7 and C8. If an AI lens for C6 is wanted, deck 6's PF AI lens ('Pipelines are where AI work naturally lives') is the closest script-backed material.",
|
|
"No tikz strips: the C7 (Airflow -> warehouse -> BI) and C8 (producers -> Kafka -> Flink -> sinks) topologies stay textual on frames 13 and 20, since the PF, L and EDA topologies were drawn in decks 4-6; a builder wanting a figure must cut one bullet per frame.",
|
|
"Minute budget: content frames sum to 125 with no slack. Reviewers disagreed on frame 38 (one asked +1, one asked -1); it stays at 4 minutes (3-minute pair exercise plus 1 minute collection). Frames 26/27 (the C9 split) are budgeted 3+2; if the lecture runs long, the C8 magnitudes recap (frame 17, 1 minute) and the C9 key-concept frame can be compressed.",
|
|
"Density: the 10-column consolidated table (frames 30-31) uses one fixed spec for both halves (p{2.6cm} + ten centred p{1.1cm}, \\tabcolsep 2pt, \\shortstack headers, S(a) row in \\tiny); the builder must confirm 'connection-stateful' wraps within 1.1 cm at \\tiny. Frame 26's Matrix/Synapse row is in the speaker notes only (high-severity density fix); the coverage reviewer would have kept it as a one-line row -- restore it only if the four-row table leaves visible room.",
|
|
"The 'in one sentence' verdict lines of §24--27 (lines 562, 636, 706, 776) quote fit-matrix ratings (PF $++$, EDA $++$, etc.) that belong to Part IV (deck 7); they are in the assigned passages and are rendered uniformly as the last sentence of each class's keypoint box (frames 9, 14, 21, 27) -- recall, not teach.",
|
|
"Exercise frame: the exercise sheet's M3 box (project_exercise.tex 407-413) adds 'a placeholder UI that calls it', which the brief's milestone line omits; included from the sheet. The hintbox items used (deterministic core free of LLM calls; data snapshot; commit after milestone; ADRs) are from lines 452-464.",
|
|
"Frame 15 (closing the inheritance loop) is the deck's synthesis of the two projektboxes with deck 6's inheritance-diagram frame; no new facts, but the 'freshness stamp on an Insight' phrasing on frame 39 is a didactic gloss on the C7 projektbox ('an answer must never cite data fresher than the index actually is').",
|
|
"Recap mentions C3 as taught last week per the brief; the C3 verdict wording is not repeated to avoid inventing deck-8 content.",
|
|
"Frames 17, 20, 23 and 36 carry visible recap markers ('week 4/5/6', 'weeks 3--6') for cases taught earlier (LinkedIn/Uber, Segment/Prime Video, Discord, Instagram/Stack Overflow/Shopify/LMAX); the week numbers follow the deck files (deck 4 = Discord, deck 5 = Kafka/Uber and Segment, deck 6 = Prime Video) and should be re-checked once decks 7-8 are typeset."
|
|
],
|
|
"total_frames": 42
|
|
} |