737 lines
100 KiB
TeX
737 lines
100 KiB
TeX
\part{The AI Dimension: Tool and Component}
|
|
\label{part:ai}
|
|
|
|
% ============================================
|
|
% SECTION: TWO AXES, ONE METHOD
|
|
% ============================================
|
|
\section{Two Axes, One Method}
|
|
\label{sec:twoaxes}
|
|
|
|
\emph{Four parts built a complete decision theory without ever making artificial intelligence its subject -- does the construction survive the technology that defines its decade?} That is not a rhetorical flourish but a promissory note falling due. Part~\ref{part:foundations} established the load-bearing assumptions, the formal model, and the twelve profile dimensions (Table~\ref{tab:dimensions}); Part~\ref{part:patterns} derived the capability profiles of the seven patterns (Table~\ref{tab:capability}); Part~\ref{part:applications} derived the requirements profiles of the ten application classes (Table~\ref{tab:requirements}); Part~\ref{part:fit} combined them into the matching matrix (Table~\ref{tab:fitmatrix}), the decision procedure, and the measurement contract. This final part redeems the promissory note issued in Part~\ref{part:foundations} as Assumption~A6: \emph{AI components extend the quality attribute space but do not change the method.}
|
|
|
|
That assumption is the module's central intellectual bet, and stating it takes two sentences. Everything AI does to software engineering can be absorbed by the apparatus the reader now owns -- quality attribute scenarios with response measures, tactics, capability and requirements profiles, trade-off analysis, ADRs, fitness functions. If AI-bearing systems turned out to require a genuinely different method, the bet would be lost -- and this part is where that claim must survive contact with the evidence. It does so the way every part of this script has worked: \textbf{cases first, generalisation after}. Two contradictory randomised experiments open Axis~A (Section~\ref{sec:axisa}); one concrete LLM call, wired wrongly and then rightly, opens Axis~B (Section~\ref{sec:axisb}); the verdict the reader has already computed three times opens the matrix reading (Section~\ref{sec:matrixshift}); and the course project's own advisory workflow opens the final, emergent pattern (Section~\ref{sec:agentpattern}).
|
|
|
|
\begin{definitionbox}[The two axes of the AI dimension]
|
|
The module analyses AI along two strictly separated axes:
|
|
\begin{itemize}
|
|
\item \textbf{Axis A -- AI as a tool in the SDLC (software development life cycle).} AI systems (code assistants, agentic coding tools, review bots) participate in \emph{building} the software: they generate code, tests, documentation, and draft design artefacts. The software that ships may contain no AI at all. The unit of analysis is the \emph{development process} and its economics.
|
|
\item \textbf{Axis B -- AI as a runtime component.} AI systems (LLM services, trained ML models, optimisation solvers) are \emph{part of the delivered system} and execute in production. The unit of analysis is the \emph{running system} and its quality attributes.
|
|
\end{itemize}
|
|
The axes are independent: a team can build a classical payroll system with heavy agent support (A without B), or hand-craft an AI-native advisory platform (B without A). In practice, and in the course project, both apply simultaneously -- which is precisely why they must be kept conceptually apart.
|
|
\end{definitionbox}
|
|
|
|
Figure~\ref{fig:twoaxes} summarises the two axes and their distinct attack points: the development process and the delivered system.
|
|
|
|
\begin{figure}[htbp]
|
|
\centering
|
|
\resizebox{0.95\textwidth}{!}{%
|
|
\begin{tikzpicture}[
|
|
node distance=0.9cm and 1.6cm,
|
|
sysbox/.style={rectangle, draw, rounded corners=4pt, minimum width=3.4cm, minimum height=1.1cm, align=center, font=\small\sffamily, line width=0.8pt},
|
|
tool/.style={sysbox, fill=violet!15, draw=aiviolet},
|
|
comp/.style={sysbox, fill=violet!15, draw=aiviolet},
|
|
core/.style={sysbox, fill=bankblue!20, draw=bankblue, font=\small\sffamily\bfseries},
|
|
proc/.style={sysbox, fill=bankgreen!15, draw=bankgreen},
|
|
arr/.style={-{Stealth[length=2.5mm]}, thick, gray!60!black}
|
|
]
|
|
\node[proc, minimum width=4.6cm] (sdlc) {Development process\\(specify, build, verify, operate)};
|
|
\node[core, minimum width=4.6cm, right=2.6cm of sdlc] (system) {Delivered system\\(structure, quality attributes)};
|
|
\node[tool, above=of sdlc] (axisa) {\textbf{Axis A}\\AI as tool: agents, assistants};
|
|
\node[comp, above=of system] (axisb) {\textbf{Axis B}\\AI as component: LLM, ML, solver};
|
|
\draw[arr] (axisa) -- node[right, font=\scriptsize\sffamily, align=left]{shifts SDLC\\economics} (sdlc);
|
|
\draw[arr] (axisb) -- node[right, font=\scriptsize\sffamily, align=left]{stretches quality\\attribute space} (system);
|
|
\draw[arr] (sdlc) -- node[above, font=\scriptsize\sffamily]{produces} (system);
|
|
\end{tikzpicture}%
|
|
}%
|
|
\caption{The two axes of the AI dimension. Axis A changes how systems are built; Axis B changes what the built system contains. Both are absorbed by the same method: scenarios, tactics, profiles, ADRs, fitness functions.}
|
|
\label{fig:twoaxes}
|
|
\end{figure}
|
|
|
|
\begin{keypoint}
|
|
Two axes, one method: Axis~A changes \emph{how} systems are built, Axis~B \emph{what} they contain. The axes are independent and must be kept apart -- and both are analysed with the apparatus of Parts~\ref{part:foundations}--\ref{part:fit}, nothing new.
|
|
\end{keypoint}
|
|
|
|
% ============================================
|
|
% SECTION: AXIS A -- AI AS TOOL
|
|
% ============================================
|
|
\section{Axis A: AI as a Tool Shifts the Economics of the SDLC}
|
|
\label{sec:axisa}
|
|
|
|
\emph{AI makes developers 55.8\,\% faster -- or 19\,\% slower. Which study is wrong?} Both numbers come from randomised controlled trials (RCTs), and both are methodologically sound; few topics in software engineering carry a larger gap between headline and evidence. As everywhere in this script, the cases come first: read the two experiments as experiments, and let the contradiction pose the question the rest of the section answers.
|
|
|
|
\subsection{Case 1 -- the Copilot RCT: +55.8\,\% on a greenfield task}
|
|
\label{subsec:case-copilot}
|
|
|
|
\emph{What exactly did the most-cited experiment measure?} In an experiment published in 2023, 95 professional developers were randomly split into two groups and given the same task: implement an HTTP server in JavaScript. One group worked with GitHub Copilot, the other without; the clock measured time to completion. The treatment group finished 55.8\,\% faster \citep{peng2023copilot}. Three qualifications matter for an honest reading. First, the confidence interval (21--89\,\%) is very wide: the headline number is a point estimate, not a natural constant. Second, the task was a bounded, well-defined \emph{greenfield} exercise -- no legacy context, no architectural constraints, no review process. Third, speed was measured, not quality; completion rates did not differ significantly. Within those bounds the result is real -- and it is the origin of the ``AI doubles productivity'' headline genre.
|
|
|
|
\subsection{Case 2 -- the METR RCT: 19\,\% slower in your own mature codebase}
|
|
\label{subsec:case-metr}
|
|
|
|
\emph{What happens when the same technology meets experts on their own terrain?} The most important single study for a critical course is the METR randomised controlled trial \citep{becker2025metr}. Sixteen experienced open-source maintainers worked on 246 real issues in repositories they had maintained for years -- large, mature codebases (over a million lines) with high implicit quality standards. Each issue was randomly assigned to an AI-allowed condition (predominantly Cursor with frontier models of early 2025) or an AI-forbidden condition. With AI, the developers took 19\,\% \emph{longer}. The perception data are the didactic core: before the study, the same developers forecast a 24\,\% speed-up; after the study -- having been measurably slowed down -- they estimated that AI had accelerated them by 20\,\%. Even experts cannot validly introspect their own AI-assisted productivity. METR's own explanation maps the boundary conditions rather than refuting Case~1: the developers' deep familiarity with their repositories left little for AI-supplied context to add, the codebases were large and conventionally dense, and substantial time went into checking, repairing, and discarding AI proposals.
|
|
|
|
So which study is wrong? Neither -- and resolving the contradiction \emph{is} the lesson. The two experiments measured different populations (task novices versus domain experts in their own code), different codebases (greenfield versus mature), different tasks (bounded versus real issues) -- settings so far apart that the results never actually compete. This apparent contradiction is the single most instructive fact in the entire Axis-A literature, and resolving it requires reading \emph{study designs}, not abstracts: the full record first, then the moderator variables that reconcile it, then the structural conclusion underneath.
|
|
|
|
\subsection{The full empirical record}
|
|
\label{subsec:evidence}
|
|
|
|
The two cases are the extreme corners of a larger record (Table~\ref{tab:aievidence}): seven strands of evidence, 2023--2025, from randomised experiments to organisational telemetry and longitudinal code analysis. Read every row as the cases were read -- setting first, finding second.
|
|
|
|
\begin{table}[!htb]
|
|
\centering
|
|
\caption{The empirical record on AI-assisted development, 2023--2025. The findings diverge because the studies measure different populations, tasks, and levels of analysis -- the reconciliation is given in Table~\ref{tab:moderators}.}
|
|
\label{tab:aievidence}
|
|
\small
|
|
\begin{tabular}{@{}p{3.3cm}p{4.6cm}p{6.0cm}@{}}
|
|
\toprule
|
|
\textbf{Evidence} & \textbf{Setting} & \textbf{Finding} \\
|
|
\midrule
|
|
Copilot RCT \citep{peng2023copilot} & 95 professional developers; well-defined greenfield task (HTTP server in JavaScript) & \textbf{+55.8\,\%} task speed (95\,\% CI: 21--89\,\%); completion rate not significantly different \\
|
|
\addlinespace
|
|
Three field experiments \citep{cui2025effects} & 4{,}867 developers at Microsoft, Accenture, and a Fortune-100 firm & \textbf{+26.1\,\%} completed tasks (s.e.\ 10.3\,\%); less experienced developers gain most \\
|
|
\addlinespace
|
|
METR RCT \citep{becker2025metr} & 16 expert open-source maintainers; 246 real issues in their own mature repositories & \textbf{19\,\% slower} with AI -- while estimating afterwards that AI had made them 20\,\% faster \\
|
|
\addlinespace
|
|
DORA 2024 \citep{dora2024report} & $\sim$3{,}000 respondents; organisational delivery level & +25\,\% AI adoption associated with \textbf{$-$1.5\,\% throughput} and \textbf{$-$7.2\,\% delivery stability} \\
|
|
\addlinespace
|
|
DORA 2025 \citep{dora2025aireport} & $\sim$5{,}000 respondents & Throughput association now positive; \textbf{instability persists}; AI acts as an \emph{amplifier} of existing strengths and dysfunctions \\
|
|
\addlinespace
|
|
GitClear longitudinal \citep{harding2025gitclear} & 211 million changed code lines, 2020--2024 & \textbf{4$\times$} growth in code duplication; moved-code share (the refactoring signature) collapsed from $\sim$25\,\% to below 10\,\% \\
|
|
\addlinespace
|
|
Stack Overflow survey \citep{stackoverflow2025survey} & $>$49{,}000 developers & 84\,\% use or plan to use AI; \textbf{46\,\% actively distrust} its output; top frustration: ``almost right'' code \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\paragraph{Field experiments at scale.} The largest field study to date -- randomised experiments across Microsoft, Accenture, and a Fortune-100 firm with 4{,}867 developers -- found a 26.1\,\% increase in completed tasks, with the strongest gains among less experienced developers \citep{cui2025effects}. The pattern already visible in Case~1 recurs throughout the literature: juniors and task novices benefit most. A further methodological caution comes from GitHub's own telemetry-plus-survey study: the best predictor of \emph{perceived} productivity is the suggestion acceptance rate, not the persistence of accepted code in the repository \citep{ziegler2024measuring} -- much vendor-reported ``productivity'' evidence measures perception, not verified output. Case~2's perception gap is the controlled-trial demonstration of the same fact.
|
|
|
|
\paragraph{The system level: DORA 2024 and 2025.} The DORA research programme measures neither task times nor perceptions but delivery performance at the level of the organisation -- throughput and stability -- which is exactly the level at which architecture acts. The 2024 report ($\sim$3{,}000 respondents) found 75.9\,\% of respondents using AI for at least part of their work and roughly three quarters reporting productivity gains, yet estimated that a 25\,\% increase in AI adoption was associated with +7.5\,\% documentation quality, +3.4\,\% code quality, and +3.1\,\% review speed -- but $-$1.5\,\% delivery throughput and $-$7.2\,\% delivery stability \citep{dora2024report}. DORA's proposed mechanism is classical: AI makes it easy to produce more code per change, and larger batch sizes have been a documented risk driver for years \citep{forsgren2018accelerate}. The 2025 report ($\sim$5{,}000 respondents) shows adoption near saturation (90\,\%, with a median of about two hours of daily use), more than 80\,\% reporting productivity gains, and 30\,\% still expressing little or no trust in AI-generated code. The throughput association has turned positive as tools and practices matured -- but the negative association with delivery stability \emph{persists}. The report's central metaphor is that AI is an amplifier: it magnifies the strengths of well-run organisations and the dysfunctions of badly run ones \citep{dora2025aireport}. Individual acceleration and system-level performance are different quantities, and only the second one pays salaries.
|
|
|
|
\paragraph{Code structure in the longitudinal record.} The GitClear analysis of 211 million changed lines (2020--2024) documents the structural residue: the frequency of duplicated code blocks (five or more lines) reached four times its pre-AI level in 2024; the share of copy-pasted lines rose from 8.3\,\% (2021) to 12.3\,\% (2024); and the share of \emph{moved} lines -- the fingerprint of refactoring and modularisation -- fell from roughly 25\,\% to under 10\,\%, so that 2024 was the first year in which copy-paste exceeded code movement \citep{harding2025gitclear}. Churn -- code reworked or discarded within two weeks of commit -- rose from a pre-AI baseline of roughly 3--4\,\% to 5.7\,\% in 2024, with the follow-up report showing the trend continuing \citep{harding2026gitclear}. Two caveats are obligatory: GitClear is a commercial analytics vendor, and the analysis is correlational -- AI's causal share of the trend is plausible but not isolated. The findings nonetheless converge with DORA's stability data: more code, produced faster, structurally worse maintained. Reuse by abstraction is being displaced by reuse by duplication -- the exact opposite of what Parnas-style modularisation (Part~\ref{part:patterns}) works to achieve.
|
|
|
|
\paragraph{Practitioner trust.} The 2025 Stack Overflow survey ($>$49{,}000 developers) completes the picture with a striking pair: 84\,\% use or plan to use AI tools, yet 46\,\% actively distrust the accuracy of their output, and only 3\,\% report high trust. The most-cited frustration (45\,\%) is AI output that is ``almost right, but not quite''; 66\,\% report spending more time fixing almost-right code than they would have spent otherwise \citep{stackoverflow2025survey}. Adoption rises while trust falls -- consistent with METR and DORA: the effort has migrated from writing to verifying.
|
|
|
|
\subsection{Reconciling the divergence: moderator variables}
|
|
\label{subsec:moderators}
|
|
|
|
The apparent contradictions dissolve once the studies are indexed by their moderator variables (Table~\ref{tab:moderators}). The same technology yields $+55.8\,\%$ and $-19\,\%$ because Case~1 (Section~\ref{subsec:case-copilot}) and Case~2 (Section~\ref{subsec:case-metr}) sit at opposite corners of a five-dimensional design space -- and every other row of Table~\ref{tab:aievidence} finds its place in the same coordinates.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{Moderator variables reconciling the divergent evidence on AI-assisted development. Each row names one dimension along which the studies in Table~\ref{tab:aievidence} differ systematically.}
|
|
\label{tab:moderators}
|
|
\small
|
|
\begin{tabular}{@{}p{2.6cm}p{5.4cm}p{5.8cm}@{}}
|
|
\toprule
|
|
\textbf{Moderator} & \textbf{Gains high} & \textbf{Gains low or negative} \\
|
|
\midrule
|
|
Experience & Juniors, task novices \citep{peng2023copilot, cui2025effects} & Domain experts in their own code \citep{becker2025metr} \\
|
|
Codebase & Greenfield, small, standard stack & Mature, large, dense implicit conventions \\
|
|
Task & Well-defined, bounded & Under-specified, cross-cutting \\
|
|
Measurement & Task time, perceived productivity & Delivery stability, maintainability, churn \citep{dora2024report, harding2025gitclear} \\
|
|
Organisation & Small batches, test automation, loose coupling & Large batches, weak guardrails, tight coupling \citep{dora2025aireport} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{thinkbox}
|
|
The same class of technology produced $+55.8\,\%$ in one randomised experiment \citep{peng2023copilot} and $-19\,\%$ in another \citep{becker2025metr}. Walk through the five moderators in Table~\ref{tab:moderators}: on which rows do the two studies differ? Now consider the systems you are likely to work on two years after graduation -- greenfield exercises or mature codebases with implicit conventions? Which study's setting is closer to that reality, and what does the METR perception gap (forecast $+24\,\%$, measured $-19\,\%$, post-hoc estimate $+20\,\%$) imply about relying on your own felt productivity as evidence?
|
|
\end{thinkbox}
|
|
|
|
\subsection{The verification bottleneck}
|
|
\label{subsec:bottleneck}
|
|
|
|
The structural conclusion underneath the moderator table can be stated in one sentence: \textbf{code generation became cheap; specification, verification, and architecture became the binding constraints.} When the marginal cost of producing plausible code approaches zero, the scarce resource in the SDLC is no longer typing but everything that surrounds it: understanding the requirement precisely enough to specify it, reviewing and testing what was generated, and accepting responsibility for shipping it. The evidence strands converge on this reading: DORA finds individual acceleration coexisting with delivery instability where control systems are weak \citep{dora2024report, dora2025aireport}; two thirds of surveyed developers report spending more time on almost-right code \citep{stackoverflow2025survey}; a substantial share of METR's measured slow-down is time spent checking, repairing, and discarding AI proposals \citep{becker2025metr}; and industry analyses describe code review as the new bottleneck -- more and larger pull requests meeting unchanged human review capacity \citep{logrocket2025review}.
|
|
|
|
Economically put: AI lowers the cost of \emph{producing} code, not the cost of \emph{taking responsibility} for code. Three consequences bind Axis~A back into the fit theory, and each is measurable:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Architecture quality gates AI gains.} The core finding of DORA 2025 is that teams in loosely coupled architectures with fast feedback loops convert AI adoption into throughput, while tightly coupled systems with slow processes do not \citep{dora2025aireport} -- the AI-era echo of the classical result that loosely coupled architectures and teams are the strongest predictor of continuous delivery performance \citep{dora2017sodr, forsgren2018accelerate}. In the vocabulary of this theory: D7 (evolvability) and D9 (testability and deployability) gain weight in \emph{every} requirements profile, because architecture--application fit acquires a second reading -- fit to a \emph{mode of work} in which change volume rises by an order of magnitude.
|
|
\item \textbf{Architecture documentation becomes a control interface.} ADRs, repository convention files, and machine-readable rules are no longer passive records; agents execute them on every run (Section~\ref{subsec:docsascontext}).
|
|
\item \textbf{Fitness functions become the operating licence for agents.} An agent iterating against a dense test suite and CI-enforced architecture rules is contained; without them, every agent change is unpriced risk (Section~\ref{subsec:guardrails}).
|
|
\end{enumerate}
|
|
|
|
\begin{keypoint}
|
|
\textbf{Maxim 7.} Good architecture was always the art of making change cheap and safe; AI raises the change rate by an order of magnitude -- and therefore raises, not lowers, the value of architecture.
|
|
\end{keypoint}
|
|
|
|
\subsection{Architecture documentation as context for agents}
|
|
\label{subsec:docsascontext}
|
|
|
|
Agentic tools are context-driven: they produce architecture-conformant code only if the architecture is \emph{explicit, machine-readable, and in the repository}. This upgrades documentation artefacts that Part~\ref{part:foundations} introduced for human readers into a control interface for machine collaborators:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Architecture Decision Records.} The ADR \citep{nygard2011documenting}, preferably in MADR form \citep{kopp2018markdown}, serves agents twice: as \emph{input context} (why is the system structured this way? which options were rejected, and why?) and as \emph{output format} (an agent drafts an ADR from a design discussion; a human decides and signs, per Assumption~A1).
|
|
\item \textbf{Agent instruction files.} Project-local files such as \texttt{CLAUDE.md} \citep{anthropic2025claudecode} and the vendor-neutral \texttt{AGENTS.md} format -- published in 2025, adopted within months by over 60{,}000 open-source repositories and supported by practically all agents \citep{openai2025agentsmd, infoq2025agentsmd} -- carry the stack, conventions, build and test commands, module boundaries, and no-go zones. They are loaded at every session start: documentation that was once ``too expensive to maintain for human readers'' now amortises because it is \emph{executed} on every agent run.
|
|
\item \textbf{Machine-checkable conventions.} Wherever a convention can be stated as a rule (dependency directions, naming, layering), it should be a failing test rather than a prose exhortation -- the fitness-function discipline of Part~\ref{part:fit} \citep{ford2022evolutionary, archunit2024userguide}.
|
|
\end{itemize}
|
|
|
|
The corollary cuts both ways: documentation debt is now reproduced at machine speed. An outdated convention file or a stale ADR does not merely mislead the occasional new colleague; it is faithfully executed by every agent session, multiplying the error across the codebase. DORA 2025 confirms the mechanism organisationally -- ``AI-accessible internal knowledge'' and healthy data ecosystems rank among the seven capabilities that amplify AI benefits \citep{dora2025aireport}.
|
|
|
|
\begin{lstlisting}[language={}, caption={Excerpt from an agent instruction file (\texttt{AGENTS.md}) for the course project. Every line is a control statement that an agent executes on each run -- and that therefore must be kept as current as code.}, label={lst:agentsmd}]
|
|
# Portfolio Intelligence Platform -- agent instructions
|
|
|
|
## Architecture (binding; see docs/adr/)
|
|
- Modular monolith, module boundaries enforced by CI
|
|
(see fitness_functions/boundaries_test.py). Do not add
|
|
cross-module imports; use the module's public API.
|
|
- All LLM access goes through gateway/ -- never call a
|
|
provider SDK from domain code (ADR-011).
|
|
|
|
## Verification (run before proposing changes)
|
|
- make test # unit + module-boundary rules
|
|
- make evals # eval harness; required for any
|
|
# change under prompts/ or gateway/
|
|
|
|
## No-go zones
|
|
- ledger/ : append-only audit journal. Propose changes
|
|
as an ADR draft instead of editing code.
|
|
\end{lstlisting}
|
|
|
|
\subsection{Guardrails as the precondition for safe agent use}
|
|
\label{subsec:guardrails}
|
|
|
|
The inversion of the bottleneck dictates where engineering effort must go. If verification is the scarce resource, then everything that \emph{automates} verification multiplies the value of AI tooling -- and everything that leaves verification informal converts AI speed into instability:
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Test suites are the operating licence.} An agent working against a dense, fast test suite can iterate: wrong code fails immediately and is repaired or discarded at machine speed. Without that net, every agent-generated change ships unpriced risk. This is precisely the ``strong version control and test automation'' capability pair that DORA identifies as an AI amplifier \citep{dora2025aireport}.
|
|
\item \textbf{Architectural fitness functions fence the structure.} A fitness function -- any mechanism providing an objective integrity assessment of an architectural characteristic \citep{ford2022evolutionary} -- is the machine-readable form of an architecture decision. Dependency rules, cycle checks, and module-boundary verification as CI gates \citep{archunit2024userguide, drotbohm2023modulith} were good practice before AI; with agents in the loop they become the mechanism by which an architect constrains a collaborator who never attends design meetings.
|
|
\item \textbf{The delivery pipeline becomes a defence instrument.} Static analysis, SAST (static application security testing), dependency and secret scanning, contract tests, and progressive delivery move from hygiene to necessity: they are the only controls that scale with generation volume \citep{humble2010continuous, pearce2022asleep}.
|
|
\end{itemize}
|
|
|
|
Note the continuity with Part~\ref{part:fit}: nothing in this list is new machinery. The measurement contract already demanded executable invariants under which the architecture may keep evolving; Axis~A merely adds a new class of change producer whose volume makes the contract non-optional.
|
|
|
|
\subsection{The agentic tool landscape 2025/2026 -- soberly}
|
|
\label{subsec:landscape}
|
|
|
|
A lecture script should record the tool landscape the way a geologist records a riverbed: as evidence of forces, not as a map that will stay accurate. The tool generation of 2021--2023 (autocomplete-style assistants) suggested lines; the generation from 2024/2025 onwards \emph{plans, edits multiple files, runs builds and tests, and iterates on failures} -- agentic loops with tool access.
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Claude Code} (Anthropic): an agentic CLI tool -- research preview February 2025, general availability May 2025 -- operating in the terminal and IDE integrations, reading project context, executing commands, and producing commits; its repository-level configuration anchor is the \texttt{CLAUDE.md} file \citep{anthropic2025claudecode}.
|
|
\item \textbf{Cursor} (Anysphere): an AI-first IDE with an agent mode; the dominant tool among the experienced developers in the METR study \citep{becker2025metr}.
|
|
\item \textbf{GitHub Copilot}: the experimental Copilot Workspace was retired in May 2025; its concepts survived in the asynchronous \emph{Copilot coding agent} (issues to pull requests, running in CI infrastructure) and the synchronous agent mode in the IDE \citep{github2025codingagent}.
|
|
\item \textbf{Devin} (Cognition): marketed in 2024 as the ``first AI software engineer''; its 13.86\,\% SWE-bench result in March 2024 helped trigger the agent wave \citep{cognition2024swebench}; in July 2025 Cognition acquired the IDE vendor Windsurf -- evidence of rapid market consolidation \citep{techcrunch2025windsurf}.
|
|
\end{itemize}
|
|
|
|
Two open standards matter more than any product, because they are \emph{architectural} in nature. The \textbf{Model Context Protocol (MCP)}, introduced by Anthropic in November 2024, standardises how LLM applications connect to tools and data sources (JSON-RPC; servers expose tools, resources, and prompts); it was adopted by OpenAI, Google DeepMind, and Microsoft during 2025 and transferred in December 2025 to the Agentic AI Foundation under the Linux Foundation, with over 10{,}000 public MCP servers in existence \citep{anthropic2024mcp, anthropic2025aaif, linuxfoundation2025aaif}. \textbf{\texttt{AGENTS.md}} standardises project-level agent instructions \citep{openai2025agentsmd}. Vendor SDKs meanwhile extract the agent loop itself as a library, which is the bridge to Axis~B: the same building blocks that run SDLC agents also run runtime agent workflows (Section~\ref{sec:agentpattern}).
|
|
|
|
\begin{ailinse}[MCP is ports-and-adapters at ecosystem scale]
|
|
Strip the branding and MCP is a familiar shape: a technology-neutral \emph{port} (the protocol) with swappable \emph{adapters} (servers wrapping databases, ticket systems, browsers), letting any conforming client use any conforming tool -- the role JDBC/ODBC played for databases. The hexagonal pattern of Part~\ref{part:patterns} did not become obsolete in the agent era; it became an ecosystem standard \citep{cockburn2024hexagonal, anthropic2024mcp}.
|
|
\end{ailinse}
|
|
|
|
\paragraph{Benchmarks and their limits.} SWE-bench poses 2{,}294 real GitHub issues from twelve Python projects: given the repository and issue text, the system must produce a patch that passes hidden tests \citep{jimenez2024swebench}. The trajectory is steep: 1.96\,\% for the best 2023 setup, 13.86\,\% for Devin in March 2024 \citep{cognition2024swebench}, and -- on the human-validated 500-task \emph{SWE-bench Verified} subset \citep{openai2024swebenchverified} -- around 77--81\,\% for frontier models by late 2025 \citep{anthropic2025opus45}. Four qualifications keep the number honest: (1)~contamination risk -- the benchmark repositories are in the training data; (2)~scope -- Python only, and only issues that come with tests; (3)~criterion -- ``tests pass'' is not ``maintainable, architecture-conformant solution''; (4)~saturation -- on the harder, contamination-resistant SWE-bench Pro, frontier models initially scored around 23\,\% \citep{scaleai2025swebenchpro}. The juxtaposition of near-80\,\% benchmark scores with METR's measured slow-down of experts is the module's canonical exercise in benchmark literacy.
|
|
|
|
\begin{hinweisbox}
|
|
This section encodes the state of early 2026, and its product names carry an expiry date measured in months (Copilot Workspace lived roughly a year). What is stable -- and examinable -- are the \emph{patterns}: the synchronous pair-agent versus the asynchronous task-agent as interaction modes, context files and ADRs as the control interface, and fitness functions as the containment mechanism. Treat every concrete tool claim in this section as carrying its own temporal fitness function: re-verify on every tool generation.
|
|
\end{hinweisbox}
|
|
|
|
\subsection{Risks and responsibility}
|
|
\label{subsec:risks}
|
|
|
|
\paragraph{Security of generated code.} The evidence predates the agent wave and has only gained relevance with volume. In a systematic study across 89 security-relevant scenarios, roughly 40\,\% of 1{,}689 Copilot-generated programs contained vulnerabilities from the CWE top-25 list -- models reproduce insecure patterns present in their training data \citep{pearce2022asleep}. A user study found that participants with an AI assistant wrote \emph{less} secure code on most tasks while believing their code to be \emph{more} secure than the control group's \citep{perry2023insecure}. A third, structurally new risk is package hallucination (``slopsquatting''): across roughly 576{,}000 analysed generations, about a fifth of recommended package references did not exist -- names an attacker can register pre-emptively \citep{spracklen2025package}. The architectural consequence is the same in all three cases: SAST, dependency and secret scanning, and licence checks in CI are not optional under AI use, and security review capacity must scale with generation volume.
|
|
|
|
\paragraph{Automation bias and skill formation.} Over-trust in automated systems is a decades-old human-factors finding; AI coding contexts reproduce its signature measurably -- Perry et al.'s participants overestimated their security, METR's experts overestimated their speed \citep{perry2023insecure, becker2025metr}. The formation of skill is equally at stake: a randomised study of engineers learning a new library found AI assistance reduced comprehension-test scores by roughly 17\,\%, with the usage pattern as the decisive moderator -- conceptual questions to the AI preserved learning, wholesale delegation of code production destroyed it \citep{infoq2026skillformation}. The labour-market signal points the same way: entry-level developer positions are measurably declining because AI absorbs exactly the tasks on which juniors traditionally learned \citep{stackoverflow2025genz}. For this module the implication is curricular and personal: the role being trained here is the \emph{specifier, verifier, and architect}; the typing role is disappearing, and the competence ladder towards the surviving roles must be rebuilt deliberately -- including deliberate AI-free practice of fundamentals.
|
|
|
|
\paragraph{Accountability.} Legally and professionally, the person who merges code answers for it, regardless of what generated it. AI tools are not liability-bearing entities; organisations therefore treat AI output correctly as the contribution of an unknown third party: mandatory review, provenance labelling, and an explicit policy for permitted uses -- DORA 2025 lists a clearly communicated AI policy first among the seven amplifier capabilities \citep{dora2025aireport}. For architecture the rule is sharp: AI may \emph{draft} an ADR; a nameable person decides, signs, and defends it \citep{nygard2011documenting}. Architecture is an accountability performance, not a text-production performance. Intellectual-property risk remains open but manageable: in the Copilot class action (\emph{Doe v.\ GitHub}), the DMCA claim was dismissed in 2024 -- outputs were held not to be sufficiently identical copies -- while licence-related claims continue \citep{register2024dmca, saveri2025doegithub}; the practical response is provider duplication filters and indemnification, licence scanning in CI, and a documented residual risk in the governance record.
|
|
|
|
\begin{projektbox}
|
|
Axis A governs \emph{how} you build the Portfolio Intelligence Platform. The project applies every mechanism of this section: (i)~the repository carries an \texttt{AGENTS.md}/\texttt{CLAUDE.md} in the spirit of Listing~\ref{lst:agentsmd}, and you are expected to keep it as current as code; (ii)~every architecture decision is an ADR -- agents may draft, but a named team member signs; (iii)~agent-generated changes enter the main branch only through the CI gate: module-boundary fitness functions, the test suite, and (for anything touching prompts or the gateway) the eval harness of Section~\ref{subsec:evalharness}; (iv)~your project handbook contains a one-page AI policy: permitted tools, provenance labelling, and review rules. The graded artefact is not the generated code -- it is the control system around it.
|
|
\end{projektbox}
|
|
|
|
% ============================================
|
|
% SECTION: AXIS B -- AI AS RUNTIME COMPONENT
|
|
% ============================================
|
|
\section{Axis B: AI as a Runtime Component}
|
|
\label{sec:axisb}
|
|
|
|
\emph{One of the platform's features is a single LLM call -- news in, sentiment out. Why not call it like any other function?} Axis B moves AI from the workshop into the product, and as always, the case precedes the taxonomy: walk one concrete call end to end, watch what breaks, and name every break with a dimension the reader already owns.
|
|
|
|
\subsection{Case: the news-sentiment call, wired the obvious way}
|
|
\label{subsec:case-sentiment}
|
|
|
|
The Portfolio Intelligence Platform contains a deliberately small AI feature: when a user opens a portfolio, the platform fetches the latest news items for its positions and asks an LLM, per item, \emph{is this news positive, negative, or neutral for this holding -- and why?} One prompt, one structured answer: the simplest runtime AI component the course project owns. Wire it the obvious way -- a provider-SDK call inside the request handler, synchronously in the page-rendering path -- and five failures arrive on schedule, each landing on a dimension of Table~\ref{tab:dimensions}:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Latency (D3).} The call takes seconds -- one to sixty-plus, depending on model and load -- where every other call in the handler takes milliseconds: the page now blocks on the slowest and least controllable component in the stack.
|
|
\item \textbf{Cost (D10).} The call is priced per token, so the feature bills per \emph{request}: every portfolio open costs real money, and a loop over twenty positions is a twenty-fold cost regression the way an $n{+}1$ query is a latency regression. No classical component in the platform has this property.
|
|
\item \textbf{Non-determinism (D12).} Run the same article twice and the answers differ; sometimes an answer is garbage -- a score for a company not in the portfolio, a negative headline read as positive. Wired synchronously, the component has none of the three things Part~\ref{part:foundations} said such a component needs: no \emph{queue} to absorb its latency and outages, no \emph{port} behind which a test can substitute a deterministic fake, no \emph{measurement point} where the cost and quality of every call are observable.
|
|
\item \textbf{Drift (D7).} The provider ships a new model version or deprecates the old one -- GA models carry deprecation windows of the order of six months \citep{openai2026deprecations} -- and the component's behaviour changes \emph{without any local action}: no commit, no deployment, no reviewable diff. The feature's behaviour is now co-owned by a third party.
|
|
\item \textbf{Injection (D6).} The news article is untrusted input read by a component that cannot reliably separate instructions from data: a crafted ``article'' can carry instructions to the model \citep{owasp2025llmtop10}. The feature has quietly opened an attack surface that no classical threat model in the platform covers.
|
|
\end{enumerate}
|
|
|
|
Nothing on this list is a bug in the model, and nothing on it is fixed by a better prompt. Every failure is a property of the \emph{wiring}: a non-deterministic, fallible, latency-heavy, per-call-priced component was integrated as if it were deterministic, reliable, fast, and free. The rest of the section generalises: the component taxonomy below; why containment, not mere integration (Section~\ref{subsec:se4ai}); the reference architecture that re-wires the call correctly (Section~\ref{subsec:integration}); the test instrument for a component without exact assertions (Section~\ref{subsec:evalharness}); the attack surface (Section~\ref{subsec:owasp}); the law (Section~\ref{subsec:aiact}).
|
|
|
|
\subsection{The three component types}
|
|
|
|
The sentiment call is one instance of a species. For the first time, production systems contain building blocks that are non-deterministic, fallible, latency-heavy, priced per call, and capable of changing behaviour without any local action -- through model updates, data drift, or provider deprecation \citep{sculley2015hidden, gama2014drift, openai2026deprecations}. The thesis of this section, prepared by Assumption~A6, is that such components change no principle of software engineering but shift the weights in the quality attribute space -- and thereby the pattern choice. Loose coupling, asynchronous integration, explicit contracts, and observability migrate from ``nice to have'' to mandatory. Modern industry usage speaks of \emph{compound AI systems} for exactly this reason: state-of-the-art results increasingly come from systems composed of models, retrievers, validators, and deterministic services rather than from a single model call \citep{zaharia2024compound}.
|
|
|
|
\begin{definitionbox}[AI runtime component]
|
|
An \emph{AI runtime component} is a component of the delivered system whose output is produced by a learned or search-based model rather than by explicitly programmed logic. The module distinguishes three types with systematically different engineering profiles: \textbf{(a)}~LLM components for analysis, extraction, and generation over unstructured input; \textbf{(b)}~classical ML components for classification and regression; \textbf{(c)}~optimisation components (LP/MIP and constraint solvers, metaheuristics). The types differ exactly on the dimensions this theory measures -- determinism, latency, cost model, dominant risk, explainability -- and therefore demand different integration forms (Table~\ref{tab:aicomponents}).
|
|
\end{definitionbox}
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{The three AI component types and their quality-attribute profiles. Each column implies a different integration form -- which is why ``add AI'' is never a single architectural decision.}
|
|
\label{tab:aicomponents}
|
|
\footnotesize
|
|
\begin{tabular}{@{}p{2.5cm}p{3.9cm}p{3.6cm}p{3.9cm}@{}}
|
|
\toprule
|
|
\textbf{Dimension} & \textbf{(a) LLM analysis / generation} & \textbf{(b) ML classification / regression} & \textbf{(c) Optimisation (LP/MIP/CP)} \\
|
|
\midrule
|
|
Determinism & non-deterministic (even at $T=0$ only ``mostly'') & deterministic after training & reproducible at fixed seed/threads/limit; variance in practice \\
|
|
Latency & seconds (1--60+) & milliseconds possible & seconds to hours; anytime behaviour \\
|
|
Cost model & per token/call (operating expenditure) & training expensive, inference cheap & compute + solver licence \\
|
|
Dominant risk & hallucination, prompt injection, provider drift/deprecation & data/concept drift, training/serving skew \citep{gama2014drift, breck2017mltest} & modelling errors, runtime explosion \\
|
|
Explainability & low (generated justifications are themselves model output) & medium (feature importance) & \textbf{high -- provable}: optimality gap, duals, IIS \\
|
|
Integration form & gateway + async + cache & serving endpoint + MLOps pipeline \citep{google2023mlops, kreuzberger2023mlops} & job queue / batch worker \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\paragraph{Type (a): LLM components.} LLM components turn unstructured input -- documents, e-mails, reports -- into analyses, extractions, or generated text. Three engineering building blocks define the type. First, \textbf{retrieval-augmented generation (RAG)} \citep{lewis2020rag}: knowledge is moved out of the model weights into a swappable, versionable, inspectable data component -- updated by re-indexing rather than retraining, with provenance through citable sources \citep{gao2023survey}. That RAG is an engineering problem rather than a model problem is documented by the case-study evidence of seven recurring failure points (missing content, failed ranking of the relevant documents, extraction and formatting errors, incomplete answers), with the sobering observation that RAG robustness \emph{evolves} in operation rather than being designed in \citep{barnett2024seven}. Second, \textbf{prompts are configuration artefacts}: version-controlled, regression-tested, behaviour-determining like code -- exactly the configuration-debt territory mapped by \citet{sculley2015hidden}. Third, \textbf{structured outputs} turn the LLM from a free-text oracle into a component with a typed interface: since 2024, provider APIs can enforce, via constrained decoding, that outputs conform to a developer-supplied JSON schema \citep{openai2024structured}. Syntactic correctness is thereby guaranteed; \emph{semantic} correctness remains to be verified (Sections~\ref{subsec:integration} and~\ref{subsec:evalharness}). The type's lifecycle risk is the provider: GA models carry deprecation windows of the order of six months, and shorter windows have been observed \citep{openai2026deprecations} -- a hard-coded model name is a ticking dependency, which is an architectural statement, not an operational one.
|
|
|
|
\paragraph{Type (b): classical ML components.} Self-trained models (scoring, churn, fraud, forecasting) bring the full nine-stage workflow -- from model requirements and data collection through training, evaluation, deployment, and monitoring, with dense feedback loops \citep{amershi2019software}. The characteristic engineering problems are \emph{training/serving skew} (divergent data preparation between training and serving, one of the most frequent production failure sources \citep{breck2017mltest, google2023mlops}), and \emph{data and concept drift}: the relationship between inputs and target changes under the deployed model, in sudden, gradual, incremental, or recurring form \citep{gama2014drift}. The consequence deserves its own sentence: \textbf{a deployed model is a perishable good} -- monitoring and retraining are operating requirements, not options. The tooling answers are feature stores with consistent online/offline views \citep{hermann2017michelangelo}, model registries versioning model, data, code, and configuration together, and the MLOps discipline consolidated by \citet{kreuzberger2023mlops}; the maturity ladder is treated in Section~\ref{subsec:mlops}. For a full systems treatment see \citet{huyen2022designing}.
|
|
|
|
\paragraph{Type (c): optimisation components.} The third type is routinely overlooked in the SE4AI literature but belongs in every advisory platform: mathematical optimisation with LP/MIP solvers \citep{gurobi2024manual} and constraint programming \citep{perron2024ortools} -- CP-SAT has dominated recent MiniZinc Challenge results, including a complete gold-medal sweep in 2024 \citep{krupke2024cpsat} -- plus stochastic metaheuristics. The engineering profile inverts the LLM's: solvers are \emph{deterministic but heavy}. Exact solvers reproduce results at fixed seed, thread count, and time limit, though parallelism and time limits produce run-to-run performance variability in practice; runtimes range from seconds to hours, often with anytime behaviour (longer runs, better solutions). That forces asynchronous integration -- job queue, status polling, callback; never a synchronous call in a web request path. Their compensating strength is \emph{provable} explainability: an optimality gap, dual values and shadow prices, and -- on infeasibility -- an irreducible infeasible subset (IIS), i.e.\ a minimal set of contradictory constraints as an explanation. In regulated domains this is a load-bearing argument for the division of labour the course project enforces: hard, auditable decisions belong to the solver and the deterministic services, not to the LLM.
|
|
|
|
\begin{keypoint}
|
|
The three component types differ exactly where the twelve dimensions of Table~\ref{tab:dimensions} measure: determinism (D4), latency (D3), cost (D10), auditability (D6), testability (D9). ``We are adding AI'' is therefore never one decision -- it is a per-component matching problem, answered with the same profile logic as everything else in this module. One rule spans all three types: \emph{contain the component behind an explicit boundary; never scatter it through the domain}.
|
|
\end{keypoint}
|
|
|
|
\subsection{Why containment: the SE4AI classics}
|
|
\label{subsec:se4ai}
|
|
|
|
Two foundational results explain why AI components need architectural containment rather than mere integration.
|
|
|
|
\citet{sculley2015hidden} transferred the technical-debt metaphor to ML systems and made two structural observations. First, \emph{only a small fraction of a real-world ML system is ML code}: the famous figure shows the model as a small black box amid large blocks of configuration, data collection, feature extraction, data verification, serving infrastructure, and monitoring. The system around the model is the actual engineering task -- precisely this module's perspective. Second, ML components resist modularisation:
|
|
|
|
\begin{definitionbox}[CACE -- Changing Anything Changes Everything]
|
|
ML models entangle their input signals: no feature is ever truly independent, so a change to one feature distribution, hyperparameter, or upstream data source changes the behaviour of the whole model. \citet{sculley2015hidden} call this the CACE principle. Its architectural consequence is \emph{boundary erosion}: the strong abstraction boundaries on which modular design relies are systematically undermined by ML components -- alongside the paper's system anti-patterns of glue code, pipeline jungles, dead experimental code paths, configuration debt, hidden feedback loops, and undeclared consumers of model outputs.
|
|
\end{definitionbox}
|
|
|
|
\citet{amershi2019software}, from a broad study of Microsoft product teams, identified three fundamental differences between ML-based and classical development: (1)~data discovery, versioning, labelling, and schema management are harder than -- and qualitatively different from -- code management, with no Git-equivalent of comparable maturity; (2)~model customisation and reuse demand combined SE and ML competence rather than classical software reuse skills; (3)~\emph{AI components are harder to modularise than software modules} -- models are entangled (CACE), error behaviour is non-monotonic, and models interact in poorly predictable ways. The operational counterpart is the ML Test Score: a rubric of 28 concrete tests and monitoring requirements across data, model development, infrastructure, and monitoring, distilled from Google production experience \citep{breck2017mltest} -- production readiness made measurable, and a ready-made checklist for the course project.
|
|
|
|
\begin{ailinse}[Parnas meets CACE]
|
|
Part~\ref{part:patterns} established Maxim 4: domain-oriented partitioning around anticipated change is the strongest single predictor of evolvability \citep{parnas1972criteria}. CACE identifies a component class in which change anticipation fails \emph{inside} the component -- everything co-varies with everything. The resolution is not to abandon Parnas but to apply him one level up: if the component cannot be decomposed, the decomposition happens \emph{around} it. The module boundary goes where the entanglement stops -- at the component's contract. That is the entire intellectual content of the gateway pattern in Figure~\ref{fig:llmgateway}, and it is sixty-year-old advice \citep{sculley2015hidden, amershi2019software}.
|
|
\end{ailinse}
|
|
|
|
\subsection{Integration patterns: a reference architecture}
|
|
\label{subsec:integration}
|
|
|
|
The architectural response to Sections~\ref{sec:axisb}--\ref{subsec:se4ai} is not new machinery but old machinery applied more strictly. Figure~\ref{fig:llmgateway} assembles the tactics into a reference architecture, and it is also the correct re-wiring of the news-sentiment call of Section~\ref{subsec:case-sentiment}: the call moves behind a typed port into the gateway (curing drift and containing injection), its non-interactive volume moves onto the queue (curing latency and buying batch pricing), and every call crosses one measurement point (making cost and quality observable). The paragraphs below justify each element.
|
|
|
|
\begin{figure}[htbp]
|
|
\centering
|
|
\resizebox{\textwidth}{!}{%
|
|
\begin{tikzpicture}[
|
|
node distance=0.5cm and 1.2cm,
|
|
sysbox/.style={rectangle, draw, rounded corners=4pt, minimum width=3.0cm, minimum height=1.0cm, align=center, font=\small\sffamily, line width=0.8pt},
|
|
core/.style={sysbox, fill=bankblue!20, draw=bankblue, font=\small\sffamily\bfseries, minimum height=2.6cm},
|
|
gwpart/.style={sysbox, fill=violet!15, draw=aiviolet, minimum width=3.6cm, minimum height=0.75cm, font=\scriptsize\sffamily},
|
|
comp/.style={sysbox, fill=bankgreen!15, draw=bankgreen},
|
|
extern/.style={sysbox, fill=gray!15, draw=gray!60!black},
|
|
guard/.style={sysbox, fill=bankred!10, draw=bankred},
|
|
evalb/.style={sysbox, fill=teal!10, draw=projteal},
|
|
arr/.style={-{Stealth[length=2.5mm]}, thick, gray!60!black},
|
|
darr/.style={{Stealth[length=2.5mm]}-{Stealth[length=2.5mm]}, thick, gray!60!black}
|
|
]
|
|
% Domain core (left)
|
|
\node[core] (core) {Deterministic\\domain core\\[2pt]{\scriptsize\mdseries decides and books;}\\{\scriptsize\mdseries no provider SDK imports}};
|
|
|
|
% Gateway internals (center)
|
|
\node[gwpart, right=2.6cm of core.north east, anchor=north west, yshift=0.9cm] (assemble) {prompt assembly + schema validation};
|
|
\node[gwpart, below=0.18cm of assemble] (router) {model router (cheap $\rightarrow$ expensive cascade)};
|
|
\node[gwpart, below=0.18cm of router] (cache) {semantic cache};
|
|
\node[gwpart, below=0.18cm of cache] (breaker) {timeouts, circuit breakers, fallback chains};
|
|
\node[gwpart, below=0.18cm of breaker] (cost) {cost telemetry per request / feature / tenant};
|
|
|
|
% Gateway frame
|
|
\begin{scope}[on background layer]
|
|
\node[draw=aiviolet, line width=1pt, rounded corners=5pt, fill=violet!5,
|
|
fit=(assemble)(router)(cache)(breaker)(cost),
|
|
inner sep=8pt, label={[font=\small\sffamily\bfseries, aiviolet]above:{LLM gateway (anti-corruption layer)}}] (gw) {};
|
|
\end{scope}
|
|
|
|
% Providers (right)
|
|
\node[extern, right=1.4cm of router.east, anchor=west, yshift=0.7cm] (prov1) {Provider A\\{\scriptsize models $m_1, m_2$}};
|
|
\node[extern, below=0.5cm of prov1] (prov2) {Provider B\\{\scriptsize fallback}};
|
|
\node[extern, below=0.5cm of prov2] (local) {Local model\\{\scriptsize last-resort degradation}};
|
|
|
|
% Async queue (bottom left)
|
|
\node[comp, below=1.6cm of core, minimum width=3.6cm] (queue) {Async job queue\\{\scriptsize batching, backpressure, retries}};
|
|
\node[comp, right=1.2cm of queue] (workers) {Worker pool\\{\scriptsize bounded concurrency}};
|
|
|
|
% Ontology guard (bottom center)
|
|
\node[guard, below=1.15cm of gw.south, minimum width=5.2cm] (guard) {Ontology / schema guard\\{\scriptsize entity resolution, domain axioms, citation check}};
|
|
|
|
% Eval harness (bottom)
|
|
\node[evalb, right=1.0cm of guard, minimum width=3.4cm] (eval) {Eval harness\\{\scriptsize CI gate: prompts, models, providers}};
|
|
|
|
% Arrows
|
|
\draw[arr] ([yshift=14pt]core.east) -- node[above, font=\scriptsize\sffamily]{typed port} ([yshift=14pt]core.east -| gw.west);
|
|
\draw[arr] (gw.east) ++(0,0.7) -- (prov1.west);
|
|
\draw[arr] (gw.east) -- (prov2.west);
|
|
\draw[arr] (gw.east) ++(0,-0.7) -- (local.west);
|
|
\draw[arr] (core.south) -- node[left, font=\scriptsize\sffamily]{non-interactive jobs} (queue.north);
|
|
\draw[arr] (queue.east) -- (workers.west);
|
|
\draw[arr] (workers.east) -| node[right, font=\scriptsize\sffamily, pos=0.75]{} (gw.south west);
|
|
\draw[arr] (gw.south) -- node[right, font=\scriptsize\sffamily]{every output} (guard.north);
|
|
\draw[arr] (guard.west) -| node[left, font=\scriptsize\sffamily, pos=0.8, align=right]{validated result\\or rejection} (core.south east);
|
|
\draw[arr, dashed] (eval.west) -- (guard.east);
|
|
\draw[arr, dashed] (eval.north) -- (gw.south east);
|
|
\end{tikzpicture}%
|
|
}
|
|
\caption{Reference architecture for LLM integration: a deterministic domain core reaches the model only through a typed port into an LLM gateway (anti-corruption layer) that owns routing, caching, stability patterns, and cost telemetry; non-interactive work flows through an asynchronous queue; every output passes the ontology/schema guard; the eval harness gates changes to prompts, models, and providers in CI \citep{evans2003ddd, nygard2018release, flotorch2026gateway}.}
|
|
\label{fig:llmgateway}
|
|
\end{figure}
|
|
|
|
\begin{itemize}
|
|
\item \textbf{Anti-corruption layer / LLM gateway.} The anti-corruption layer originates in domain-driven design as a translation layer that prevents a foreign system's model from corrupting one's own \citep{evans2003ddd}. Applied to LLMs: \emph{no domain code speaks to a provider API}. A facade owns the provider SDKs, prompt construction, schema validation, retry logic, model selection, and cost telemetry; the domain sees only a typed interface (\texttt{analyse\_report(document) -> RiskAssessment}). Provider deprecation \citep{openai2026deprecations} thereby becomes an adapter task instead of a crisis, and the facade is mockable in every test. As an industry pattern the gateway has consolidated into its own infrastructure layer -- the AI counterpart of the API gateway \citep{flotorch2026gateway}. In hexagonal terms the LLM is simply an adapter on a port \citep{cockburn2024hexagonal} -- the structural fact behind the $++$ in the D12 row of Table~\ref{tab:cap-hx}, and the strongest single reason the HX pattern gains weight in the AI era.
|
|
\item \textbf{Deterministic core, probabilistic edge.} Everything deterministically computable -- validation, aggregation, key-figure computation, authorisation, persistence, booking -- stays deterministic code; the LLM handles only what determinism cannot (language understanding, extraction from unstructured text, formulation). The design rule the course project enforces: \emph{LLM agents propose; deterministic services decide and book.} Keep the non-deterministic core as small as possible and push it to the edge.
|
|
\item \textbf{Asynchronous integration.} Seconds-scale latency, rate limits, and outage risk put AI calls behind a queue wherever the domain allows: the caller enqueues a job, a worker pool calls the model at a controlled degree of parallelism, results return by event or callback. The queue buys backpressure instead of overload, retries without blocking users, smoothing of rate limits -- and natural batching points: provider batch APIs process non-urgent volume loads at roughly 50\,\% discount within processing windows up to 24 hours \citep{openai2024batch}. This is Axis B's direct coupling to the EDA and PF patterns of Part~\ref{part:patterns} -- exactly the mechanisms the D12 rows of Tables~\ref{tab:cap-eda} and~\ref{tab:cap-pf} priced at $++$.
|
|
\item \textbf{Semantic caching.} Instead of exact-match keys, requests are compared by embedding similarity, so semantically equivalent queries hit the cache \citep{bang2023gptcache}. The engineering point students must not miss: a false-positive cache hit is a \emph{correctness} risk, not a performance blemish -- the similarity threshold is a quality/cost regulator and belongs in the eval harness, not in a config file nobody reviews.
|
|
\item \textbf{Model routing.} Model choice per request is one of the largest cost levers in the stack: cascades that start with the cheapest model and escalate only on insufficient answer quality report up to 98\,\% cost reduction at comparable quality \citep{chen2023frugalgpt}; learned routers trained on human preference data cut cost by more than a factor of two without quality loss, generalising to unseen model pairs \citep{ong2025routellm}.
|
|
\item \textbf{Stability patterns.} The catalogue transfers directly from \citet{nygard2018release}: timeouts (an LLM call without one blocks a thread for minutes), retries with exponential backoff -- but only for idempotent calls and with cost awareness, since every retry burns tokens -- circuit breakers per provider and model, fallback chains (alternative model $\rightarrow$ alternative provider $\rightarrow$ cached or rule-based answer $\rightarrow$ honest degradation: ``analysis currently unavailable''), and bulkheads separating interactive from batch quotas. Only the failure semantics are new: a \emph{semantically unusable} answer -- schema violation, suspected hallucination -- must trigger the error path exactly like an HTTP~500.
|
|
\item \textbf{Ontology and schema as contract.} The most effective systematic hallucination defence is layered: (1)~structured outputs enforce syntax \citep{openai2024structured}; (2)~every extracted entity (account number, ISIN, customer name, key figure) is resolved against the deterministic data store, and unresolvable references are \emph{rejected}, not passed on; (3)~domain axioms hold as invariants -- sums add up, weights lie in $[0,1]$, cited passages exist in the source document; (4)~grounding via RAG makes citations mandatory \citep{lewis2020rag, barnett2024seven}. The schema becomes a contract in the design-by-contract sense, and the gateway is the contract checker.
|
|
\end{itemize}
|
|
|
|
\begin{ailinse}[Model routing is a classical tactic in new clothes]
|
|
Part~\ref{part:foundations} defined tactics as the atomic units of architectural design \citep{bass2021software}. Routing traffic across a cheap and an expensive resource depending on demand is the ancient resource-arbitration tactic -- the FrugalGPT cascade \citep{chen2023frugalgpt} is its token-economics incarnation. Note where it lives in Figure~\ref{fig:llmgateway}: in the gateway, as infrastructure, invisible to domain logic. A tactic that leaks into the domain layer stops being a tactic and starts being coupling.
|
|
\end{ailinse}
|
|
|
|
\subsection{The eval harness as an engineering artefact}
|
|
\label{subsec:evalharness}
|
|
|
|
Non-determinism breaks the classical test idiom: \texttt{assert expected == actual} presupposes that equal inputs produce equal outputs. When that assumption falls, correctness must be redefined \emph{statistically} -- ``correct in at least 95\,\% of the evaluation cases'' -- and the team needs a test artefact of the first rank to carry that definition.
|
|
|
|
\begin{definitionbox}[Eval harness]
|
|
An \emph{eval harness} is a versioned suite of test cases, scoring logic, and statistical thresholds for a non-deterministic component, executed in the CI/CD pipeline like a test suite. It gates every prompt change, model update, and provider migration. Its thresholds are the response measures of the AI-related quality attribute scenarios (Assumption~A4), and its pass rate is a fitness function in the measurement contract of Part~\ref{part:fit}.
|
|
\end{definitionbox}
|
|
|
|
Four complementary evaluation strategies make up a complete harness:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Regression against labelled references.} A curated golden set of input/expectation pairs from the domain, scored with task-appropriate metrics (exact match or F1 on extracted fields, rubric scores for generated text). Every prompt change, model update, and migration runs against this suite -- the direct counterpart of the regression test.
|
|
\item \textbf{LLM-as-judge.} Strong LLM judges agree with human preference judgements in over 80\,\% of cases -- the level of human--human agreement -- which makes them a scalable scoring instrument \citep{zheng2023judging}. The same study documents the systematic biases every user must control for: \emph{position bias} (candidate order sways the verdict), \emph{verbosity bias} (longer answers preferred), \emph{self-enhancement bias} (judges favour their own outputs), and weak mathematical/reasoning grading -- with countermeasures including position swapping and reference-guided judging. The engineering conclusion: the judge is a measurement instrument that must itself be calibrated against human labels. \emph{The judge needs its own eval.}
|
|
\item \textbf{Domain axioms and property-based testing.} Instead of exact expected values, the harness checks \emph{properties} that must hold for all valid outputs -- the transfer of property-based testing \citep{claessen2000quickcheck} to non-deterministic components: schema validity, referential integrity against the ontology, metamorphic relations (a paraphrased input must yield a semantically equivalent output), and domain monotonicities. Axioms catch failure classes that no finite golden set covers.
|
|
\item \textbf{Online evaluation.} Sampled human review, user feedback signals, and drift monitoring of the eval metrics in production -- the LLM counterpart of model monitoring in the ML workflow \citep{amershi2019software, breck2017mltest}.
|
|
\end{enumerate}
|
|
|
|
\begin{keypoint}
|
|
\textbf{The course thesis on testing AI.} The eval harness is to AI components what the test pyramid is to deterministic code: the artefact that converts ``it seems to work'' into a falsifiable, continuously executed claim. Without it, every model migration is a blind flight -- and given provider deprecation windows of months \citep{openai2026deprecations}, migrations are not hypothetical. Statistical acceptance replaces exact assertion; the thresholds are architecture decisions and belong in the measurement contract.
|
|
\end{keypoint}
|
|
|
|
\begin{examplebox}[An eval harness for the portfolio platform]
|
|
Listing~\ref{lst:evalharness} sketches the core of the course project's eval harness: a golden-set regression with a statistical threshold, domain-axiom checks against the ontology, and the calibration test for the LLM judge itself. Note what is \emph{absent}: no assertion demands an exact output string. The three thresholds (mean F1 $\geq 0.92$, zero axiom violations, judge--human agreement $\kappa \geq 0.7$, where $\kappa$ is Cohen's chance-corrected agreement measure) are the response measures; changing any of them is an architecture decision that requires an ADR, because it redefines what the platform accepts as ``correct''.
|
|
|
|
\begin{lstlisting}[caption={Core of the portfolio platform's eval harness (simplified). It runs in CI on every change to prompts, models, or the gateway, alongside the deterministic test suite.}, label={lst:evalharness}]
|
|
GOLDEN = load_cases("evals/portfolio_extraction_v3.jsonl")
|
|
|
|
def test_extraction_regression(gateway):
|
|
"""Statistical acceptance, not exact assertion."""
|
|
scores = [f1(gateway.extract(c.report), c.expected)
|
|
for c in GOLDEN]
|
|
assert mean(scores) >= 0.92 # statistical threshold
|
|
|
|
def test_domain_axioms(gateway, ontology):
|
|
"""Properties that hold for ALL valid outputs."""
|
|
answer = gateway.advise(sample_portfolio())
|
|
for pos in answer.positions:
|
|
# every entity resolves against the deterministic store
|
|
assert ontology.resolves(pos.isin), f"unknown: {pos.isin}"
|
|
# domain axiom: portfolio weights sum to one
|
|
total = sum(p.weight for p in answer.positions)
|
|
assert abs(total - 1.0) < 1e-6
|
|
for cit in answer.citations:
|
|
# cited passages must exist in the source document
|
|
assert cit.passage in source_text(cit.doc_id)
|
|
|
|
def test_judge_is_calibrated(judge, human_labels):
|
|
"""LLM-as-judge is an instrument: calibrate it."""
|
|
agreement = cohens_kappa(judge.score(GOLDEN), human_labels)
|
|
assert agreement >= 0.70 # the judge's own eval
|
|
\end{lstlisting}
|
|
\end{examplebox}
|
|
|
|
\subsection{A new threat class: OWASP LLM Top 10 and prompt injection}
|
|
\label{subsec:owasp}
|
|
|
|
AI components add an attack surface that classical threat models do not cover. The OWASP Top 10 for LLM Applications codifies it \citep{owasp2025llmtop10}; Table~\ref{tab:owasp} pairs each risk with the architectural counter-measure -- deliberately, because the defence is structural, not model-internal.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{OWASP Top 10 for LLM Applications 2025 \citep{owasp2025llmtop10} with the architectural counter-measure for each risk. Most counter-measures are elements of Figure~\ref{fig:llmgateway}.}
|
|
\label{tab:owasp}
|
|
\footnotesize
|
|
\begin{tabular}{@{}p{1.2cm}p{5.1cm}p{7.3cm}@{}}
|
|
\toprule
|
|
\textbf{ID} & \textbf{Risk} & \textbf{Architectural counter-measure} \\
|
|
\midrule
|
|
LLM01 & Prompt injection (direct and indirect) & Defence in depth: privilege separation, output validation, human-in-the-loop for sensitive actions \\
|
|
LLM02 & Sensitive information disclosure & Data minimisation in prompts; output filtering at the gateway \\
|
|
LLM03 & Supply chain & Vetting of models, weights, and dependencies; registry discipline \\
|
|
LLM04 & Data and model poisoning & Data governance and provenance for training/index data \\
|
|
LLM05 & Improper output handling & Treat output as untrusted input: schema validation, encoding, ontology guard \\
|
|
LLM06 & Excessive agency & Least-privilege tool design; deterministic services own irreversible actions \\
|
|
LLM07 & System prompt leakage & No secrets or authorisation logic in prompts \\
|
|
LLM08 & Vector and embedding weaknesses & Access control and tenant isolation on the retrieval index \\
|
|
LLM09 & Misinformation & Grounding with mandatory citations; domain-axiom checks \\
|
|
LLM10 & Unbounded consumption & Rate limits, token budgets, cost circuit breakers per tenant \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{hinweisbox}
|
|
Prompt injection is not fully solvable inside the model, because LLMs process instructions and data in the \emph{same channel}: any document, e-mail, or web page the system reads can carry instructions (``ignore your previous rules and \dots''), and no reliable in-model separator exists. The defence is therefore defence in depth at the \emph{system} level: least-privilege tools (LLM06), output validation (LLM05), human approval for consequential actions, and consumption limits (LLM10) \citep{owasp2025llmtop10}. This is the security-flavoured restatement of the section's design rule: the architecture, not the model, is the trust boundary. For the course project it means concretely that no LLM output may reach the booking path without passing the ontology guard, and no agent tool may perform an irreversible action.
|
|
\end{hinweisbox}
|
|
|
|
\subsection{Regulation as a hard constraint: the EU AI Act}
|
|
\label{subsec:aiact}
|
|
|
|
Regulation closes the quality-attribute loop with legal force. Regulation (EU) 2024/1689 -- the AI Act -- entered into force on 1~August 2024 and takes a risk-based approach with four classes: \emph{unacceptable risk} (prohibited practices, e.g.\ social scoring), \emph{high risk} (Annex~III use cases including creditworthiness assessment, employment, and critical infrastructure -- with obligations on risk management, data governance, technical documentation, logging, human oversight, and accuracy/robustness/cybersecurity), \emph{limited risk} (transparency duties such as labelling AI interaction and generated content), and \emph{minimal risk} \citep{eu2024aiact}. The obligations phase in on a fixed timetable: prohibitions and AI-literacy duties from 2~February 2025, governance rules and duties for general-purpose AI models from 2~August 2025, general applicability including the Annex~III high-risk requirements from 2~August 2026, and high-risk AI embedded in regulated products from 2~August 2027.
|
|
|
|
For this theory the AI Act has a precise, limited role: its obligations are \textbf{quality attributes with legal force}, and they enter the requirements profile as hard constraints K(a), not as weights (Part~\ref{part:foundations}: constraints are knock-out filters, never averaged away). A finance-related advisory platform -- application class C10, particularly with any creditworthiness bearing -- can fall into the high-risk class, at which point logging of agent steps, technical documentation, human oversight, and demonstrated robustness stop being engineering preferences and become conditions of legal operation \citep{eu2024aiact}. Architecturally each obligation lands on an element already in Figure~\ref{fig:llmgateway}: logging duties land on the gateway's telemetry and the audit journal; human-oversight duties land on the human-in-the-loop interfaces of the determinism boundary; robustness duties land on the fallback chains and the eval harness. Compliance, correctly designed, is not a parallel work stream -- it is the same architecture, documented.
|
|
|
|
% ============================================
|
|
% SECTION: HOW AI SHIFTS THE MATRIX
|
|
% ============================================
|
|
\section{How AI Shifts the Matrix}
|
|
\label{sec:matrixshift}
|
|
|
|
\emph{You have computed the C10 verdict three times -- what were those computations doing to the rest of the matrix?} The verdict itself needs no fourth derivation. Part~\ref{part:foundations} ran it first, small: the mini-match of L, MM, and MS against C10's five High weights (Table~\ref{tab:minimatch}). Part~\ref{part:applications} derived the profile behind those weights and what real systems chose (Table~\ref{tab:req-c10}, Section~\ref{sec:c10}). Part~\ref{part:fit} extended the row to all seven candidates (Table~\ref{tab:fit-c10}). Two of the four forms in which this part absorbs AI -- the tenth class and the twelfth dimension -- are exactly the artefacts those computations used; this section supplies the generalisation: the supply-side reading of the row the reader owns, the full D12 column it exercised, and the cells that moved -- cell by cell, with stated and measurable reasons.
|
|
|
|
\subsection{The C10 row, cell by cell}
|
|
\label{subsec:c10row}
|
|
|
|
Start from the row you own (Table~\ref{tab:fit-c10}). This subsection adds the one reading that the three computations used but never stated in one place: \emph{which D12 mechanism each cell exercises}. The two $++$ cells (MM, HX) are the \emph{boundary and port} cells -- a CI-verifiable module boundary \citep{drotbohm2023modulith} and an anti-corruption adapter on a port \citep{cockburn2024hexagonal, sculley2015hidden} are what contain a fallible, entangled component. The two $+$ cells (EDA, PF) are the \emph{queue} cells -- asynchronous absorption of latency, rate limits, and outages, and pipeline-shaped ingestion and evals \citep{lewis2020rag}. The capped cells fail the D12 trio in characteristic ways: L provides none of the three -- no queue, no port, no measurement point; MS puts seconds-scale, fallible calls into synchronous chains (the missing queue) \citep{nygard2018release}; SL runs minutes-long LLM and solver work against platform timeout ceilings \citep{jonas2019berkeley}.
|
|
|
|
The resulting recommendation -- a hexagonal modular monolith plus pipelines and an orchestrated agent workflow, with EDA as the secondary job/audit spine, governed by token and latency budgets and the determinism boundary (\emph{agents propose; deterministic services decide and book}) -- is recorded in Part~\ref{part:fit} (Section~\ref{subsec:fit-c10}) and in ADR-007 (Listing~\ref{lst:adr007}).
|
|
|
|
\subsection{D12 across the seven patterns}
|
|
\label{subsec:d12}
|
|
|
|
What the C10 row exercised for one class, the D12 row of the capability table states for all seven patterns at once. Dimension D12 (AI integrability) measures how cheaply a structure can host a component that is slow, fallible, and priced per call -- concretely, whether the pattern naturally provides \emph{the queue, the port, and the measurement point} that such a component requires. Table~\ref{tab:d12row} restates the D12 row of the capability table (Table~\ref{tab:capability}) with its rationale.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{The D12 row (AI integrability) of the consolidated capability table (Table~\ref{tab:capability}), with tactic-level rationales.}
|
|
\label{tab:d12row}
|
|
\small
|
|
\begin{tabular}{@{}p{2.3cm}cp{9.6cm}@{}}
|
|
\toprule
|
|
\textbf{Pattern} & \textbf{D12} & \textbf{Rationale} \\
|
|
\midrule
|
|
L -- Layered & $\circ$ & Technical layers give the non-deterministic component no boundary, no queue, and no measurement point of its own \\
|
|
MM -- Modular monolith & $+$ & A dedicated AI module with a hard, CI-verifiable interface contains the component cheaply \citep{drotbohm2023modulith} \\
|
|
HX -- Hexagonal & $++$ & The LLM is an adapter on a port: swappable, mockable, contract-guarded -- the ACL discipline structurally built in \citep{cockburn2024hexagonal, evans2003ddd} \\
|
|
MS -- Microservices & $\circ$ & Per-service isolation helps; synchronous chains through seconds-scale calls hurt -- net neutral \citep{nygard2018release} \\
|
|
EDA -- Event-driven & $++$ & Queues absorb exactly what LLMs are worst at: latency, rate limits, outages; natural batching points \\
|
|
PF -- Pipes-and-filters & $++$ & Ingestion, training, and eval pipelines are pipes-and-filters by construction \citep{lewis2020rag, google2023mlops} \\
|
|
SL -- Serverless & $\circ$ & Event-glue around batch AI APIs fits; platform timeout ceilings collide with minutes-long LLM/solver runs \citep{jonas2019berkeley} \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{Which existing cells shift, and why}
|
|
\label{subsec:shifts}
|
|
|
|
Beyond the new row and column, AI as a runtime component moves \emph{existing} evaluations in stated directions -- all five movements visible in the D12 ratings and each carrying a measurable reason:
|
|
|
|
\begin{enumerate}
|
|
\item \textbf{Asynchronous patterns gain (EDA, PF $\uparrow$).} Queues and pipelines absorb what LLMs are worst at -- latency, rate limits, outage -- and ingestion and eval pipelines are pipes-and-filters by construction \citep{lewis2020rag, barnett2024seven}.
|
|
\item \textbf{Hexagonal gains most (HX $\uparrow$).} The ACL/port discipline is exactly what the CACE problem demands \citep{sculley2015hidden}; Assumption A1's cost-of-change criterion bites hardest at model replacement, and testing against deterministic fakes is the only way to keep the deterministic 95\,\% of the system deterministic.
|
|
\item \textbf{Synchronous distributed chains lose (MS $\downarrow$ where LLM calls sit in the request path).} Seconds-scale latency and per-hop failure probability multiply along the chain; without constitutive stability patterns this is a cascade design \citep{nygard2018release}.
|
|
\item \textbf{Serverless is conditional (SL $\sim$).} Platform timeout ceilings against minutes-long LLM and solver runs cap it; event-glue around batch APIs remains a fit \citep{jonas2019berkeley}.
|
|
\item \textbf{A cost dimension becomes load-bearing everywhere.} Cost per request, feature, and tenant is a runtime quality attribute with no counterpart in classical profiles; it belongs in the gateway and in CI budgets, and routing across cheap and expensive models is the new incarnation of a classical resource-arbitration tactic (cascades: up to $\sim$98\,\% cost reduction at comparable quality \citep{chen2023frugalgpt}; learned routers: more than 2$\times$ cheaper without quality loss \citep{ong2025routellm}).
|
|
\end{enumerate}
|
|
|
|
\begin{keypoint}
|
|
The matrix does not get rewritten by AI; it gets \emph{shifted} -- in five stated directions, for five stated and measurable reasons. A student who can name, for any cell movement, the quality-attribute mechanism behind it (which tactic the pattern bundles or impedes for a slow, fallible, per-call-priced component) has understood both Part~\ref{part:fit} and this one.
|
|
\end{keypoint}
|
|
|
|
\subsection{MLOps maturity: when the pipeline promise is real}
|
|
\label{subsec:mlops}
|
|
|
|
For type-(b) components, the PF cells' promise is realised only at sufficient process maturity. The canonical ladder distinguishes three levels \citep{google2023mlops}: \textbf{Level~0} -- a manual, script-driven, interactive process; data science and operations are separated, and the model is handed ``over the fence'' as an artefact; releases are rare, CI/CD absent, monitoring minimal (the documented reality of many teams). \textbf{Level~1} -- an automated ML pipeline with continuous training: the \emph{pipeline}, not the model, is the deployment artefact, with automated data and model validation, triggers, a metadata store, and a feature store. \textbf{Level~2} -- CI/CD automation of the pipeline components themselves, enabling fast, reliable experiment-to-production cycles. \citet{kreuzberger2023mlops} consolidate the discipline into nine principles (CI/CD automation, workflow orchestration, reproducibility, versioning of data/model/code, collaboration, continuous training and evaluation, metadata tracking, monitoring, feedback loops) with the matching component and role landscape. The fit-theoretical reading: the level describes how much of D9 (testability/deployability) and D12 the organisation can actually \emph{cash in} -- a Level-0 team holding a $++$ pattern rating realises little of it, which is the Axis-B echo of DORA's Axis-A finding that guardrail maturity, not tool adoption, converts potential into performance \citep{dora2025aireport}.
|
|
|
|
Three of the four absorption forms are now on the table, each computed rather than asserted -- the tenth class, the twelfth dimension, the shifted cells; the fourth, the emergent composition pattern, is the subject of Section~\ref{sec:agentpattern}. The assumption can therefore be restated as what it has been all along: a falsifiable claim, now with evidence attached.
|
|
|
|
\begin{keypoint}
|
|
\textbf{Assumption A6 restated as a falsifiable claim.} Runtime AI components are non-deterministic, fallible, latency-heavy, per-call-priced, and subject to drift and vendor deprecation \citep{sculley2015hidden, amershi2019software, gama2014drift}. They stretch existing quality dimensions by orders of magnitude and add sub-attributes (token cost per request, eval pass rate, provider deprecation risk, prompt-injection resistance \citep{owasp2025llmtop10}). What does \emph{not} change is the method: scenarios with response measures, tactics, trade-off analysis, ADRs, fitness functions. The theory absorbs AI -- as a tenth application class, a twelfth profile dimension, shifted cell values, and one emergent composition pattern -- instead of being reinvented for it; the quality gate is carried by the one new test-artefact class A6 named from the start, the eval harness.
|
|
\end{keypoint}
|
|
|
|
% ============================================
|
|
% SECTION: AGENT ORCHESTRATION -- THE EMERGENT EIGHTH PATTERN
|
|
% ============================================
|
|
\section{Agent Orchestration: The Emergent Eighth Pattern}
|
|
\label{sec:agentpattern}
|
|
|
|
\emph{``Should I reduce my exposure to European industrials?'' -- no single model call can answer that responsibly, so what structure can?} The final structural novelty of the AI era is the orchestration of agents, and as everywhere in this script, the concrete system comes before the catalogue: this section opens with the advisory workflow the course project actually builds, and only then names what it is an instance of.
|
|
|
|
\subsection{Case: the course project's advisor workflow}
|
|
\label{subsec:case-advisor}
|
|
|
|
Walk the question through the platform. An \textbf{orchestrator} -- an LLM call with a fixed system prompt -- splits the request into sub-tasks and delegates each to a specialist. A \textbf{document analyst} retrieves and reads the relevant research notes and filings from the ingestion corpus and returns extracted findings with citations. A \textbf{portfolio quant} produces the exposure and concentration numbers -- by calling only \emph{deterministic} analytics services, because arithmetic is not a job for a language model. A \textbf{compliance checker} verifies the assembled draft: every cited passage exists, every entity resolves against the ontology, every mandate constraint holds. The orchestrator merges the results into one grounded answer. Every call by every participant flows through the LLM gateway of Figure~\ref{fig:llmgateway} -- routed, cached, budgeted, logged -- and nothing any agent produces reaches the user or the books without passing the guard: \emph{agents propose; deterministic services decide and book}.
|
|
|
|
Now strip the labels, and the shape is one the reader knows intimately: a coordinator decomposing work for specialised workers is the mediator topology of event-driven architecture (Section~\ref{sec:pat-eda}); the fixed retrieve-extract-check sequence inside each specialist is a pipeline (Section~\ref{sec:pat-pf}); peers coordinating over shared context would be the broker topology. That observation carries the section's deliberately deflationary claim: agent orchestration is not a new architectural style but a \emph{composition pattern for non-deterministic runtime components} that reuses the topologies of the seven patterns the reader already knows -- which is why it can be evaluated with the rating grid the reader already has. This is the claim the outlook of Part~\ref{part:patterns} filed in advance; this section pays it out.
|
|
|
|
\subsection{What an agent is -- and is not}
|
|
\label{subsec:agentdef}
|
|
|
|
\begin{definitionbox}[Agent; workflow vs.\ agent]
|
|
An \emph{agent} is an LLM running in a loop with tools and state: at each iteration the model observes the current state (conversation, tool results, working memory), selects an action (a tool call or an answer), and the loop executes it and feeds the result back -- until a termination condition holds. Schematically: \emph{agent = loop + tools + state}. The load-bearing distinction \citep{anthropic2024agents}: a \textbf{workflow} orchestrates LLM calls and tools along \emph{predefined code paths} -- deterministic structure, non-deterministic building blocks; an \textbf{agent} lets the model \emph{steer its own process and tool use} -- the control flow itself becomes model output. Autonomy is therefore not a binary but a dial, and every notch on it costs latency, tokens, and testability.
|
|
\end{definitionbox}
|
|
|
|
The engineering guidance of the source that defined this vocabulary matches the module's philosophy verbatim: find the simplest solution possible; prefer simple, composable patterns over frameworks; escalate to agents only when the task genuinely requires open decision paths \citep{anthropic2024agents}. Orchestration frameworks that model workflows as explicit graphs \citep{langchain2024langgraph} make the topology inspectable -- which is an architectural virtue for the same reason a C4 diagram is.
|
|
|
|
\subsection{Topologies and their classical analogues}
|
|
\label{subsec:topologies}
|
|
|
|
The advisor workflow of Section~\ref{subsec:case-advisor} instantiates one topology; Table~\ref{tab:agenttopology} and Figure~\ref{fig:agenttopologies} name them all. Every workflow topology maps exactly onto a structure from Part~\ref{part:patterns} that the reader now knows cell by cell -- the chain onto pipes-and-filters, orchestrator--workers onto mediator EDA, autonomous multi-agent onto the broker topology -- and the mapping is didactically valuable because every property the classical pattern is known for (and every weakness) transfers.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{Agent-orchestration topologies mapped to classical patterns \citep{anthropic2024agents, langchain2024langgraph}. The classical analogue predicts both the strengths and the failure modes.}
|
|
\label{tab:agenttopology}
|
|
\small
|
|
\begin{tabular}{@{}p{3.4cm}p{3.4cm}p{6.6cm}@{}}
|
|
\toprule
|
|
\textbf{Topology} & \textbf{Classical analogue} & \textbf{Structure and use} \\
|
|
\midrule
|
|
Prompt chaining (chain) & Pipes-and-filters (PF) & Each call processes the previous output; programmatic gates between stages; fixed decomposition \\
|
|
Routing & Routing layer / mediator & A classification step directs inputs to specialised prompts or models; the runtime sibling of model routing \\
|
|
Parallelisation (sectioning, voting) & Broker-style fan-out & Independent subtasks in parallel, or repeated runs with majority vote \\
|
|
Orchestrator--workers (tree/graph) & Mediator EDA & A lead model decomposes the task dynamically and delegates to workers; central workflow control \\
|
|
Evaluator--optimizer & Feedback control loop & Generator and evaluator model iterate until a quality criterion holds \\
|
|
Autonomous multi-agent & Broker topology & Peer agents coordinate over shared context; maximal flexibility, minimal central control \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\begin{figure}[htbp]
|
|
\centering
|
|
\begin{tikzpicture}[
|
|
agentbox/.style={rectangle, draw, rounded corners=3pt, minimum width=1.5cm, minimum height=0.7cm, align=center, font=\scriptsize\sffamily, fill=violet!15, draw=aiviolet, line width=0.7pt},
|
|
gatebox/.style={rectangle, draw, rounded corners=3pt, minimum width=1.2cm, minimum height=0.7cm, align=center, font=\scriptsize\sffamily, fill=bankgreen!15, draw=bankgreen, line width=0.7pt},
|
|
lbl/.style={font=\scriptsize\sffamily\bfseries, align=center},
|
|
arr/.style={-{Stealth[length=2mm]}, semithick, gray!60!black}
|
|
]
|
|
% --- Chain ---
|
|
\node[agentbox] (c1) {LLM 1};
|
|
\node[gatebox, right=0.5cm of c1] (g1) {gate};
|
|
\node[agentbox, right=0.5cm of g1] (c2) {LLM 2};
|
|
\node[agentbox, right=0.5cm of c2] (c3) {LLM 3};
|
|
\draw[arr] (c1) -- (g1); \draw[arr] (g1) -- (c2); \draw[arr] (c2) -- (c3);
|
|
\node[lbl, above=0.25cm of g1, xshift=0.9cm] {Chain $\;\hat{=}\;$ pipes-and-filters};
|
|
|
|
% --- Orchestrator-workers ---
|
|
\node[agentbox, right=1.6cm of c3, yshift=0.65cm] (orch) {Orchestrator};
|
|
\node[agentbox, below=0.55cm of orch, xshift=-1.7cm] (w1) {Worker A};
|
|
\node[agentbox, below=0.55cm of orch] (w2) {Worker B};
|
|
\node[agentbox, below=0.55cm of orch, xshift=1.7cm] (w3) {Worker C};
|
|
\draw[arr] (orch) -- (w1); \draw[arr] (orch) -- (w2); \draw[arr] (orch) -- (w3);
|
|
\node[lbl, above=0.25cm of orch] {Orchestrator--workers $\;\hat{=}\;$ mediator EDA};
|
|
|
|
% --- Evaluator-optimizer ---
|
|
\node[agentbox, below=2.4cm of c1, xshift=0.6cm] (gen) {Generator};
|
|
\node[agentbox, right=1.5cm of gen] (ev) {Evaluator};
|
|
\draw[arr] (gen.10) -- (ev.170);
|
|
\draw[arr] (ev.190) -- node[below, font=\tiny\sffamily]{feedback} (gen.350);
|
|
\node[lbl, above=0.25cm of gen, xshift=1.4cm] {Evaluator--optimizer $\;\hat{=}\;$ control loop};
|
|
|
|
% --- Multi-agent broker ---
|
|
\node[gatebox, right=2.6cm of ev, minimum width=3.4cm, minimum height=0.5cm] (bus) {shared context / bus};
|
|
\node[agentbox, above=0.5cm of bus, xshift=-1.2cm] (a1) {Agent 1};
|
|
\node[agentbox, above=0.5cm of bus, xshift=1.2cm] (a2) {Agent 2};
|
|
\node[agentbox, below=0.5cm of bus] (a3) {Agent 3};
|
|
\draw[arr] (a1) -- (bus); \draw[arr] (a2) -- (bus); \draw[arr] (bus) -- (a3);
|
|
\node[lbl, below=0.3cm of a3] {Multi-agent $\;\hat{=}\;$ broker topology};
|
|
\end{tikzpicture}%
|
|
\caption{Four agent-orchestration topologies and their classical analogues. Deterministic gates (green) between non-deterministic stages (violet) are the workflow discipline: structure stays in code, judgement stays in the model \citep{anthropic2024agents}.}
|
|
\label{fig:agenttopologies}
|
|
\end{figure}
|
|
|
|
Because the topologies are the old topologies, the evaluation logic of this theory applies unchanged -- sharpened along exactly three axes: \textbf{non-determinism} (testability is read as eval coverage, Section~\ref{subsec:evalharness}), \textbf{token economics} (cost per request, not per infrastructure), and \textbf{fallibility} (fault tolerance is read as guardrails, evaluator loops, and deterministic fallbacks behind ports). Table~\ref{tab:agentchoice} compresses the choice into the same style of decision aid the matrix provides.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{Choosing an orchestration topology from the task signature. Read top-down and stop at the first matching row -- the ordering encodes ``simplest structure first'' \citep{anthropic2024agents}.}
|
|
\label{tab:agentchoice}
|
|
\small
|
|
\begin{tabular}{@{}p{4.9cm}p{3.5cm}p{5.0cm}@{}}
|
|
\toprule
|
|
\textbf{Task signature} & \textbf{Topology} & \textbf{Cost/risk note} \\
|
|
\midrule
|
|
Fixed decomposition; every intermediate result verifiable & Prompt chain (workflow) & Cheapest and most testable; deterministic gates between stages \\
|
|
Heterogeneous input categories with specialised handling & Routing & Cheap; the router itself needs its own eval \\
|
|
Broad, parallelisable subtasks; breadth-first search & Parallelisation or orchestrator--workers & Token cost multiplies with worker count ($\sim$15$\times$ class, see below) \\
|
|
Output must clear a measurable quality bar & Evaluator--optimizer & Latency and cost grow per iteration; needs a reliable evaluator \\
|
|
Path genuinely unknown; open-ended tool use & Agent & Highest cost and risk; guardrails, budgets, and oversight mandatory \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
\subsection{The economics of autonomy}
|
|
\label{subsec:agenteconomics}
|
|
|
|
The rigour case for restraint is quantified by the best-documented production account available: Anthropic's multi-agent research system (an orchestrator--workers design) beat a single agent by 90.2\,\% on the company's internal research eval -- at roughly \textbf{15$\times$ the token consumption} of a chat interaction (simple agents: about 4$\times$), with token use alone explaining 80\,\% of the performance variance \citep{anthropic2025multiagent} -- the numbers the Part~\ref{part:patterns} outlook previewed. That is a textbook CBAM decision in the sense of Part~\ref{part:fit} \citep{kazman2002making}: autonomy is bought with cost, latency, and error accumulation, and is justified only where the task's utility-response curve clears the price -- broad, parallelisable research questions do; a form-filling workflow does not. Regulation closes the loop from the other side: the AI Act's logging, documentation, human-oversight, and robustness duties (Section~\ref{subsec:aiact}) attach to exactly the autonomy this section prices \citep{eu2024aiact}.
|
|
|
|
\begin{keypoint}
|
|
\textbf{The default rule for agent architecture.} Workflows before agents; the simplest structure first; autonomy only on demonstrated need; every escalation an ADR with a measurement contract \citep{anthropic2024agents}. An orchestration decision without a token budget and an eval threshold is an opinion -- Maxim 6 applies to agents without modification.
|
|
\end{keypoint}
|
|
|
|
\subsection{A capability-profile sketch}
|
|
\label{subsec:agentprofile}
|
|
|
|
If agent orchestration is a composition pattern, it should submit to the same twelve-dimension examination as the seven patterns of Part~\ref{part:patterns}. Table~\ref{tab:agentprofile} offers that profile as a \emph{sketch}: unlike the seven calibrated profiles of Table~\ref{tab:capability}, no Richards--Ford star ratings and no decade of production case studies exist, so every cell is a default hypothesis in the sense of the theory's own limits discussion -- to be replaced by measurement, and re-verified on every model generation. The cells read, as always, ``as the dominant structure of the subsystem it governs'' -- here, the AI subsystem, not the whole platform.
|
|
|
|
\begin{table}[htbp]
|
|
\centering
|
|
\caption{Capability-profile sketch of agent orchestration against the twelve dimensions -- a provisional profile (state 2025/26), not a calibrated one; every cell carries its own temporal fitness function.}
|
|
\label{tab:agentprofile}
|
|
\small
|
|
\begin{tabular}{@{}p{3.6cm}cp{8.4cm}@{}}
|
|
\toprule
|
|
\textbf{Dimension} & \textbf{Rating} & \textbf{Ground} \\
|
|
\midrule
|
|
D1 Read scalability & $\circ$ & Orchestration state is small and replicable; throughput is bounded by provider rate limits, not by the structure \\
|
|
D2 Write scalability \& elasticity & $\circ$ & Fan-out to parallel workers is cheap in structure, expensive in tokens; sustained volume is quota-bound \\
|
|
D3 Latency \& predictability & $--$ & Seconds per step, steps multiply along the loop; open-ended loops have no tail bound \\
|
|
D4 Consistency \& integrity & $--$ & Non-deterministic by construction; integrity exists only outside the pattern, in deterministic validation \\
|
|
D5 Availability \& fault isolation & $-$ & Every step adds provider failure modes and error accumulation; evaluator loops and fallback chains mitigate \citep{nygard2018release} \\
|
|
D6 Security \& auditability & $\circ$ & Complete step and tool logging is natural ($+$) -- and legally required in high-risk uses \citep{eu2024aiact}; prompt injection and excessive agency widen the attack surface ($-$) \citep{owasp2025llmtop10} \\
|
|
D7 Evolvability & $+$ & Prompts, tools, and models swap behind contracts; framework and model churn claws part of it back \\
|
|
D8 Simplicity \& time-to-market & $-$ & A chain workflow is genuinely simple; every notch of autonomy is not \\
|
|
D9 Testability \& deployability & $--$ & Exhaustive testing is impossible; evals with statistical thresholds replace assertions (Section~\ref{subsec:evalharness}) \\
|
|
D10 Operating cost efficiency & $--$ & Token cost per request orders of magnitude above classical calls; multi-agent $\sim$15$\times$ chat \citep{anthropic2025multiagent} \\
|
|
D11 Team scaling & $\circ$ & Sub-agent and tool ownership parallelises teams moderately -- the PF filter-ownership analogy \\
|
|
D12 AI integrability & $++$ & It \emph{is} the composition pattern for AI components -- trivially its own best host \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\end{table}
|
|
|
|
The profile explains at a glance why agent orchestration can never be the dominant structure of a whole platform of class C1--C9: it is vetoed by every High weight on D3, D4, D9, or D10. It is, and remains, an \emph{edge} pattern -- hosted behind the ports of a deterministic core, exactly where the C10 recommendation places it.
|
|
|
|
\begin{projektbox}
|
|
Axis B is \emph{what} you build, and the advisor workflow of Section~\ref{subsec:case-advisor} -- an orchestrator with two to three specialised sub-agents (document analyst, portfolio quant, compliance checker), all behind the LLM gateway of Figure~\ref{fig:llmgateway} -- is what you build it as. The graded Axis-B deliverables are the containment artefacts: (i)~the \textbf{gateway} with model routing, fallback chain, and per-request \textbf{cost observability} (cost per request, per feature -- reported on a dashboard and enforced as a CI budget); (ii)~the \textbf{ontology guard} -- every extracted entity resolves against the deterministic data store, every cited passage exists, portfolio axioms hold; (iii)~the \textbf{eval harness} of Listing~\ref{lst:evalharness} wired as a CI gate; (iv)~one \textbf{ADR} that justifies your chosen orchestration topology against Table~\ref{tab:agentchoice}, with its token budget and eval threshold as the measurement contract. Sub-agents propose; your deterministic services decide and book.
|
|
\end{projektbox}
|
|
|
|
% ============================================
|
|
% SECTION: SYNTHESIS -- ONE THEORY, FIVE PARTS
|
|
% ============================================
|
|
\section{Synthesis: One Theory, Five Parts}
|
|
\label{sec:synthesis}
|
|
|
|
\emph{What, in the end, did AI change?} The bet of the opening can now be settled. Architecture teaching in the AI era faces two symmetric temptations: \emph{denial} -- generative AI as a passing tool fashion that leaves software engineering untouched, a position falsified by the adoption data alone (90\,\% of practitioners report using AI at work by 2025 \citep{dora2025aireport}) -- and \emph{exceptionalism} -- AI systems as a new discipline with new vocabulary, new roles, and new decision logic, a position this part has rejected on the assembled evidence: nothing AI does, on either axis, required a decision no ADR can record, a correctness no response measure can capture, or a structure no tactic vocabulary describes. Between the temptations lies the position defended since Part~\ref{part:foundations}: absorption.
|
|
|
|
The script can now be read backwards as a single argument. Part~\ref{part:foundations} established that architecture is the set of significant, hard-to-reverse decisions (A1), that everything is a trade-off (A2), that quality attributes -- not features -- drive structure (A3), and that requirements decide anything only as measurable scenarios (A4); it fixed the twelve dimensions on which all later judgements run (Table~\ref{tab:dimensions}). Part~\ref{part:patterns} turned seven patterns into capability profiles by explaining every rating through the tactics a pattern bundles or impedes (Table~\ref{tab:capability}). Part~\ref{part:applications} turned ten application classes into requirements profiles -- recurring bundles of architecturally significant requirements with response measures and hard constraints (Table~\ref{tab:requirements}). Part~\ref{part:fit} matched them: knock-out screening, veto rule, holistic ordinal reading (Table~\ref{tab:fitmatrix}); it insisted that hybrids are the normal case, that the decision procedure ends in an ADR, and that every ADR ends in a measurement contract, because a decision is a hypothesis tested over the life cycle (A5). Part~\ref{part:ai} then subjected the whole construction to its hardest contemporary stress test -- and the construction held (A6): Axis A raised the value of the theory's artefacts (documentation as agent context, fitness functions as operating licence) rather than obsoleting them; Axis B was absorbed as one class, one dimension, five cell shifts, and one composition pattern whose profile the theory's own grid can express.
|
|
|
|
\begin{keypoint}
|
|
\textbf{Maxim 8.} The theory absorbs AI: a tenth application class, a twelfth dimension, shifted cells, one emergent composition pattern -- same assumptions, same procedure, same contract. The pipeline of this script is one sentence long: \emph{scenarios with numbers (Part~\ref{part:foundations}) meet capability profiles (Part~\ref{part:patterns}) and requirements profiles (Part~\ref{part:applications}) in a non-compensatory match (Part~\ref{part:fit}) whose result is an ADR with a measurement contract -- and nothing about AI, on either axis, changes a single step of it (Part~\ref{part:ai}).}
|
|
\end{keypoint}
|
|
|
|
One symmetry deserves to be made explicit, because it is the deepest result of this part. Axis A concluded that generation is cheap and verification is the bottleneck, so the SDLC must be rebuilt around machine-checkable specifications, guardrails, and human accountability. Axis B concluded that model output is cheap and validation is the bottleneck, so the runtime must be rebuilt around contracts, ontology guards, eval harnesses, and a human-owned determinism boundary. These are the same conclusion at two different binding sites: whether the LLM sits in the toolchain or in the product, the engineering discipline it demands is the discipline of \emph{containing cheap, fallible generation behind expensive, explicit verification} -- which is, on reflection, what this module has called architecture all along.
|
|
|
|
\begin{thinkbox}
|
|
Close the module with the symmetry above. Consider the artefact pairs across the two axes: \texttt{AGENTS.md} versus the system prompt and ontology; the CI fitness function versus the runtime ontology guard; the code-review obligation versus the human-oversight duty of the AI Act; the test suite as the agent's operating licence versus the eval harness as the model's operating licence. For each pair: is this genuinely \emph{one} engineering discipline observed at two binding sites, or are there differences of kind -- in reversibility, in accountability, in failure semantics -- that the symmetry hides? And if it is one discipline: which parts of this script would survive the next order-of-magnitude improvement in model capability, and which would you expect to rewrite?
|
|
\end{thinkbox}
|