Chapter 4: How to Defend a Multi-Agent Architecture Before a Committee: Feasibility, Costs, Benchmarking, and the Agentic Side of Constructive Critique

An architecture is not approved for being elegant, but for being defensible under pressure.

---

Section 1: What the Committee Is Really Buying

There is a structural misunderstanding that comes back in almost every presentation of AI architectures to executive committees or enterprise architecture boards: the presenter talks about technology, and the committee decides about something else. When a chief technology officer hears “multi-agent architecture with event-driven orchestration,” what they process is not the technical detail. What they process is: what happens if it fails? How much does it cost if it scales? Who is to blame if the system hallucinates?

Bass, Clements, and Kazman (2022), in the fourth edition of Software Architecture in Practice (the canonical reference in the field, with more than 15,000 citations on Google Scholar), set out a taxonomy of quality attributes that decision-makers evaluate implicitly: availability, modifiability, security, performance, and above all maintainability and traceability. A committee doesn’t buy tokens; it buys certainty about those five attributes.

The first strategic move, then, is to translate every technical decision into the currency of the committee’s three Rs: Risk, Return, and Reversibility. Not “we use Kafka to decouple the agents in time,” but “if one agent fails, the rest of the system doesn’t stop (that is what Kafka gives us).” Not “we fine-tune the model on our own data,” but “the model can’t reveal other customers’ information because it never sees it (that is the boundary).”

Technical decision	Translation into the committee’s language
Hexagonal architecture with ports	If the AI model improves, we swap it without touching the business core. Migration time: days, not months.
MCP for external API access	The agent never handles credentials. If the agent is compromised, it holds no keys.
Kafka as the event bus	One agent fails, the others keep running. There is no single point of failure.
Evaluators (LLM-judges)	Before a response reaches the user, a second layer verifies it. The human only sees validated outputs.
Graceful degradation	If the AI is unavailable, the system falls back to the manual flow. We don’t crash; we step down.

---

Section 2: The ATAM Method Applied to Multi-Agent Systems

The Architecture Tradeoff Analysis Method (ATAM), developed at Carnegie Mellon’s Software Engineering Institute (Kazman, Klein & Clements, 2000), is the most rigorous framework available for evaluating architectural decisions before deployment. It was designed for mission-critical systems (defense, aviation, telecommunications), and applying it to multi-agent AI systems is both fitting and revealing.

The ATAM process has four steps that, adapted to the agentic context, produce the most valuable artifact any architect can bring to a committee: a complete map of tradeoffs with their explicit cost.

Step 1: Present the architectural driver. Which quality attribute matters most? For an AI customer support system, it is probably availability and latency. For a financial compliance system, it is auditability and traceability. For a disaster governance system (Chapter 23), it is reliability and response time. The common mistake is not to rank them and to present “we want it to be fast, secure, scalable, and cheap” (which is to say nothing).

Step 2: Identify the quality attribute scenarios. A quality attribute scenario has the form: “Under condition X, the system must do Y within Z time/cost/probability.” Examples:

• “When traffic triples in 30 minutes (a sales peak), the classification agent must hold P99 latency under 2 seconds.”
• “When the AI model produces a response that violates the privacy policy, the evaluator must detect it and discard it before it reaches the user in 99.5% of cases.”
• “If the inference service fails, the system must degrade to precomputed responses in under 5 seconds.”

Step 3: Map the architectural risks and sensitivities. Every design decision has a sensitivity point (a small change = a large impact) and tradeoff points (gaining on one attribute = losing on another). The ATAM table makes this explicit. For the committee, this table is invaluable: it shows the architect has thought about failures, not just the happy path.

Design decision	Risk	Benefit	Tradeoff
Cloud models (vs. on-premise)	Variable latency, egress of sensitive data	No GPU CAPEX, automatic updates	P99 latency rises 80 ms vs. on-premise
Fine-tuning vs. RAG	Fine-tuning: high upfront cost, rigidity	Fine-tuning: latency <50 ms, no retrieval overhead	RAG is more flexible but adds 50 ms of retrieval latency
One central orchestrator vs. event choreography	Orchestrator: single point of failure	Choreography: resilience by design	Choreography = harder debugging
Human-in-the-loop vs. automatic evaluator	Human: slow, doesn’t scale	Automatic: <1 ms latency	Automatic: 2-5% initial false negatives

Step 4: Present the risk analysis. The committee gets three columns: Identified risk, Probability (estimated from the incident history of similar systems), and Mitigation plan. This step turns the presentation into a risk management conversation (the committee’s native language).

---

Section 3: The Critic Agent Panel: Stress-Testing the Proposal Before the Room

Here is the innovation that sets this approach apart from conventional preparation. Instead of rehearsing the presentation in front of an agreeable colleague, you deploy a panel of synthetic critic agents (each embodying an archetype of the committee) that put the proposal through an interrogation before a human with veto power presents it.

The panel’s archetypes:

CFO Agent. Its central concern: 3-year TCO and break-even point. Typical questions: “When does the cost of the tokens exceed the savings in manual labor?”, “What happens to the cost if volume triples?”, “Are there hidden operating costs (monitoring, model updates, retraining)?” Its job is to find the economic scenario where the proposal breaks.

Security Agent. It looks for attack surfaces the architect didn’t mention: “What prevents a prompt injection from the input data?”, “How do we audit the agent’s decisions for regulatory compliance?”, “Where do the credentials for the external APIs live?”, “What happens if the model hallucinates and the output is persisted in the database?”

Technology Skeptic Agent. It questions maturity: “Is this in production at companies of our sector and size?”, “What happens when the model provider changes its prices or deprecates the API?”, “How many months would it take us to roll back if it doesn’t work?”, “What is the vendor lock-in risk?”

Business Agent. It demands quantified impact: “How many tickets per month does it resolve without humans?”, “What is the expected CSAT vs. the current one?”, “How specifically does it improve the customer experience?”, “How do we measure success in the first 90 days?”

The process: the architect presents the proposal; the four agents raise objections simultaneously; the architect answers them; the agents escalate if an answer is insufficient. The session ends when all four agents rate the proposal “defensible” (not perfect, but defensible).

What comes out of this process is not a polished proposal that hides the risks. It is an honest proposal that shows the risks and their mitigations. That honesty is, paradoxically, the most persuasive thing an architect can bring into a room.

---

Section 4: Honest Benchmarking: Compare to Persuade, Not to Impress

► TECHNICAL TRACK

Benchmarking is not technical vanity. It is the most powerful rhetorical tool in front of a committee because it turns intuitions into quantified arguments and comparisons into decisions. But dishonest benchmarking (which only shows the scenarios where the system wins) destroys credibility the moment the committee asks the question the presenter left out.

Principle of honest benchmarking: show all three scenarios (where the system wins, where it ties, and where it loses), with the conditions of each.

Example for a multi-agent customer support system:

Metric	Current system (humans)	AI agent	Multi-agent orchestration	When multi-agent loses
Tickets resolved/hour	12	45	78	Never on pure volume
Cost per ticket	$8.50	$1.20	$2.10	At volume <200 tickets/day
Customer satisfaction (CSAT)	4.2/5	3.8/5	4.4/5	On complex, emotional tickets
First response time	8 min	12 sec	8 sec	Never on speed
Escalation rate to a human	35%	28%	9%	Never on autonomy
Accuracy on regulatory topics	94%	72%	91%	In untrained domains

The “when it loses” column is the most important. A committee that sees that column trusts the others. A committee that doesn’t see it is suspicious of the others.

Benchmarking the architecture against a baseline and against alternatives:

Don’t compare only against “the current system.” Compare against: (a) the current system, (b) the monolithic alternative (a single large agent), and (c) the proposed architecture. This shows that the multi-agent decision was weighed against the simplest alternative and chosen for specific reasons, not because it was fashionable.

---

Section 5: The Control Narrative: How to Close the Defense

A successful architectural defense does not close with “and in conclusion, it is the best solution.” It closes with a control narrative: the system is governable, reversible, and measurably improving.

The three commitments the committee needs to hear (and that the architect must be able to back with evidence):

1. Total traceability. “Every agent decision is recorded with its input, its reasoning, its output, and the time. If something fails, we know exactly what happened, in which agent, with what data, at what time. It is not a black box; it is a system with structured logs and complete auditability.”

2. Designed reversibility. “If the system doesn’t perform as promised in the first 90 days, we have an explicit rollback plan: the manual flows keep running in parallel for the first 6 months. The switch is operational, not architectural.”

3. Measurable improvement. “In month 1, we measure X. In month 3, we measure Y. In month 6, we decide whether to expand. We are not asking you for a blank check; we are asking you for 90 days of data.”

The control narrative is, in essence, the recognition that AI is probabilistic and that the system containing it must be deterministic in its governance. The committee doesn’t buy the AI; it buys the system that governs it.

---

Expanded Academic Version

A restatement of the central concepts at a higher academic density.

The evaluation of software architectures in contexts of high uncertainty (such as multi-agent artificial intelligence systems) can be framed within the paradigm of decision-making under epistemic ambiguity, formalized by Ellsberg (1961) and applied to the information systems domain by Peppard & Ward (2004) in Strategic Planning for Information Systems. From this perspective, architectural quality attributes are socio-technical constructs that emerge from negotiation among stakeholders with divergent mental models and different time horizons, not objective properties of the system.

The ATAM method (Architecture Tradeoff Analysis Method), as formulated by Kazman et al. (2000) in their SEI Technical Report, operates precisely in this negotiation space: its quality attribute scenarios are elicitation instruments that force the explicit articulation of the implicit assumptions each stakeholder holds about the system. In the specific context of multi-agent systems, this elicitation function is especially valuable, since the emergent nature of swarm behavior, whose formal characterization was established by Bonabeau, Dorigo & Theraulaz (1999) in Swarm Intelligence: From Natural to Artificial Systems, introduces a class of aleatory uncertainty (not reducible by more information) that traditional evaluation frameworks did not contemplate.

---

Key Takeaways from Chapter 4

• The committee doesn’t buy technology; it buys the three Rs: Risk, Return, and Reversibility.
• Every technical decision has to be translated into the committee’s language before entering the room.
• The ATAM method applied to multi-agent systems produces the most valuable artifact: a complete map of tradeoffs with explicit cost.
• A panel of four critic agents (CFO, Security, Skeptic, Business) stress-tests the proposal before a human with veto power does.
• Honest benchmarking shows the scenario where the system loses: that column builds credibility in the others.
• The closing narrative is a control narrative: traceability, reversibility, and measurable improvement within 90 days.
• AI is probabilistic; the system that governs it must be deterministic in its auditability.

---

Worked Exercise 4.1: Translate a Technical Risk into the CFO’s Language

Problem. The committee’s CFO asks: “I’ve heard these AI systems ‘hallucinate.’ What does that mean for us if an agent hallucinates and that error reaches a customer?”

The architect’s answer (translated into the CFO’s language):

“A hallucination is when the model generates information that looks correct but isn’t. Without controls, that error would reach the customer. With our architecture, there are three layers before that can happen:

First, the model has an automatic evaluator (an LLM-judge) that compares its response against company policies. If the response violates any policy, it is discarded and an alternative is generated. That evaluator catches roughly 97% of hallucinations in well-defined domains.

Second, if the model’s confidence is below 0.80 (which indicates high uncertainty), the system automatically escalates to a human agent. The customer doesn’t see the hallucination; they see a response that says ‘transferring you to a specialist.’

Third, every response is logged. If a hallucination gets past the two previous filters and reaches the customer, we can trace exactly which prompt triggered it, in what context, and fix the evaluator so it doesn’t happen again.

The expected cost of a hallucination reaching a customer in our system: one manual support ticket (which you already have today). The difference is that now we know exactly what happened and can keep it from happening again.”

This answer didn’t promise perfection (which is impossible). It promised measurable governance and continuous improvement. That is something a CFO can approve.

---