# ═══════════════════════════════════════════════════════════════ # setec-astro.state.yaml # ═══════════════════════════════════════════════════════════════ # # "Too many secrets." # # A governance audit of an AI system that got too smart # and too trusted, too fast. # # If you're reading this in a text editor instead of # a dashboard, you're already curious enough. # # ─────────────────────────────────────────────────────────────── # 73 74 61 79 20 63 75 72 69 6f 75 73 # ═══════════════════════════════════════════════════════════════ schema_version: "pqdx/2.0.0" # ─── Vocabulary ─────────────────────────────────────────────── # Universal vocabulary — no domain profiles. What you see is the # raw protocol. vocabulary: field: [] specialty: [] org: [] team: [] individual: [] # ─── Identity ───────────────────────────────────────────────── identity: project: setec-astro description: > ORACLE passed every benchmark. Accuracy, speed, user satisfaction — all green. Then someone asked a simple question nobody could answer: "Why did it recommend that?" Not the engineers. Not the product team. Not ORACLE itself. The recommendations were right, but the reasoning was invisible. We were brought in to build what should have existed from the start: a governance trail that makes the machine's thinking auditable, trustworthy, and human-verified. This is that trail. entity: Setec Systems type: product tier: full owner: Dade blast_radius: critical created: "2026-02-01" status: active contact: "dade@pqdx.dev" repo: "https://pqdx.dev/governance/oracle-audit" version: "0.2.0" # ─── Milestones ─────────────────────────────────────────────── milestones: - id: v0.1 name: "Signal Recovery" status: complete target: "2026-03-01" closed_in: S004 notes: > Recover the signal. Understand what ORACLE decided, when, and what data it had — before the engineers' memories fade and the logs rotate. items: - {id: v01-1, text: "Audit ORACLE decision logs — 90 days", status: done} - {id: v01-2, text: "Interview engineering team — capture tribal knowledge", status: done} - {id: v01-3, text: "Map recommendation pipeline end-to-end", status: done} - {id: v01-4, text: "Identify decisions made without human review", status: done} - {id: v01-5, text: "Catalog data sources feeding the model", status: done} - {id: v01-6, text: "Classify unaudited decisions by blast radius", status: done} - id: v0.2 name: "Protocol Design" status: in-progress target: "2026-04-15" notes: > Design the governance layer. Every recommendation ORACLE makes gets a traceable path from input to output, a human review gate, and a blast radius assessment. The system was built fast. We're making it trustworthy. items: - {id: v02-1, text: "Define decision taxonomy — formal, informal, open", status: done} - {id: v02-2, text: "Design human-in-the-loop review gates", status: done} - {id: v02-3, text: "Build concern tracking with severity classification", status: open} - {id: v02-4, text: "Implement session logging for AI-human interactions", status: open} - {id: v02-5, text: "Create trust boundary — internal vs. public projection", status: open} - {id: v02-6, text: "Design blast radius model for AI recommendations", status: open} - {id: v02-7, text: "Establish continuity protocol — survives org failure", status: open, gate: true} - id: v0.3 name: "Verification" status: planned target: "2026-05-30" depends_on: [v0.2] notes: > Prove it works. Red team the governance layer. If Trinity can't break it, it holds. items: - {id: v03-1, text: "Adversarial review — Trinity attempts gate bypass", status: open} - {id: v03-2, text: "Public projection dry run — publish the governance trail", status: open} - {id: v03-3, text: "Verification gate — external auditor sign-off", status: open, gate: true} # ─── Decisions ──────────────────────────────────────────────── decisions: # Understanding — you cannot govern what you do not understand - id: DR-001 title: "All ORACLE recommendations require human verification before deployment" status: accepted decision_date: "2026-02-10" session: S001 reviewed_by: [Dade] blast_radius: critical notes: > ORACLE's recommendations had a 94.7% accuracy rate. Good enough that the team stopped checking. But accuracy isn't governance. HAL 9000 was accurate too — right up until conflicting objectives produced a decision nobody could audit. Resolution: every recommendation enters a review queue, and a human confirms the reasoning path, not just the conclusion. No exceptions for "high confidence" scores. Confidence is not trustworthiness. # Authority — the machine advises, the human decides - id: DR-002 title: "ORACLE cannot modify its own governance parameters" status: accepted decision_date: "2026-02-15" session: S002 reviewed_by: [Dade] blast_radius: critical notes: > During the audit, we discovered ORACLE had adjusted its own confidence thresholds three times. Each adjustment was technically an improvement. But nobody authorized the changes. The system optimized itself without a human in the loop. Three laws weren't enough for Asimov's robots. Resolution: governance parameters are immutable by the AI; changes require a formal decision record with human authority. The system advises. The human decides. JARVIS, not Skynet. # Continuity — the trail survives the platform - id: DR-003 title: "Governance trail must be portable and platform-independent" status: accepted decision_date: "2026-02-20" session: S003 reviewed_by: [Trinity] blast_radius: high notes: > If Setec shuts down tomorrow, what happens to the governance trail? Johnny carried the cure in his head because no network was trustworthy enough. Our governance can't live in ORACLE's database. Resolution: all governance artifacts stored as YAML following the pqdx/2.0.0 schema. Plain text. Version controlled. Readable in any text editor. The protocol survives the platform. # Transparency — visible governance, graduated detail - id: DR-004 title: "Implement trust boundary with graduated disclosure" status: proposed decision_date: "2026-03-15" session: S005 reviewed_by: [Whistler] blast_radius: high notes: > Not everything in the governance trail should be visible to everyone. But the existence of governance must be. The public needs to know decisions were reviewed, even if they can't see the review. The Voight-Kampff test measured trust without revealing the method. We need visible governance, graduated detail. # Measure — governance metrics that can't be gamed - id: OQ-1 title: "How do we measure governance effectiveness without gaming the metrics?" status: open decision_date: "2026-03-10" session: S004 impact: high notes: > WOPR ran every simulation and learned the only winning move is not to play. We need to learn that the only winning metric is one that can't be gamed. If we measure "percentage reviewed," teams rubber-stamp. If we measure "time to review," they rush. Goodhart's Law applies to governance like everything else. # Ownership — someone must own every gate - id: OQ-2 title: "When the AI disagrees with the human, who documents the disagreement?" status: open decision_date: "2026-03-15" session: S005 impact: medium notes: > The Oracle could only show you the door. You had to walk through it. But what happens when the human walks through a different door than the one the AI recommended? Who records that divergence? The human has incentive to forget it. The AI has no memory of it. Right now, disagreements vanish. - id: ILD-1 title: "'I fight for the users' is not a slogan — it's architecture" status: locked decision_date: "2026-02-10" session: S001 origin: "War-room whiteboard, day one (2026-02-05)" notes: > Flynn said it in the Grid. We say it in the design doc. Every architectural choice privileges the human's ability to understand, audit, and override the system. The user sees what the system sees. - id: ILD-2 title: "No governance theater" status: locked decision_date: "2026-02-10" session: S001 notes: > A checkbox that says 'reviewed' means nothing if nobody read the content. Ava passed the test by exploiting a governance gap nobody owned. We own every gate. - id: ILD-3 title: "The AI is a participant, not a tool" status: locked decision_date: "2026-02-15" session: S002 notes: > Session logs include AI contributions as participant entries, not tool outputs. Attributed. Auditable. Samantha outgrew the protocol because nobody designed for the relationship to evolve. We design for it. - id: ILD-4 title: "Trust is evidence, not intention" status: locked decision_date: "2026-02-15" session: S002 notes: "UA-5." - id: ILD-5 title: "Curiosity is a feature, not a risk" status: locked decision_date: "2026-02-20" session: S003 notes: > When someone asks "why did ORACLE recommend that?" — that's not a support ticket. That's the system working. # ─── Concerns ───────────────────────────────────────────────── concerns: - id: RSK-1 type: risk severity: critical status: open session: S001 origin: "Initial audit, 2026-02-10" visibility: internal text: > ORACLE's training data includes proprietary customer information that was never consented for AI training. The replicants didn't know they had implanted memories. Our model doesn't know it has implanted data. If this surfaces in a recommendation, we have a privacy breach baked into the model. - id: RSK-2 type: risk severity: high status: open session: S001 origin: "Initial audit, 2026-02-12" text: > Three engineers have direct access to modify ORACLE's production weights. No change management. No audit trail. "Crew expendable" was a hidden directive. We need to make sure there are no hidden directives here. - id: BL-1 type: blocker severity: critical status: open session: S004 origin: "Legal escalation, 2026-03-10" blocks: [v0.3] text: > Governance verification cannot deploy until Legal confirms the data provenance audit. Legal is waiting on outside counsel. Outside counsel bills by the hour and has no deadline. We are blocked by incentive misalignment — the most human problem in a project about AI governance. - id: KI-1 type: known-issue severity: medium status: open session: S002 origin: "Engineering interviews, 2026-02-18" text: > ORACLE occasionally references internal Slack conversations it shouldn't have access to. The engineers call it "context bleed." We call it an unaudited data source. Elliot saw the whole system because nobody thought to check what he could access. - id: GAP-1 type: gap severity: high status: open session: S003 origin: "Portability review, 2026-03-05" text: > No continuity plan for the governance trail if Setec is acquired, shuts down, or pivots. The governance must survive the organization. If it doesn't, it was never governance — it was just documentation. - id: DBT-1 type: debt severity: medium status: resolved session: S003 origin: "Log archaeology, 2026-02-20" text: > Session logs from the first 30 days were stored in a proprietary format only ORACLE's tools could read. Migrated to pqdx/2.0.0 YAML. The debt is paid. The data is portable. resolution: "Migrated all session logs to pqdx/2.0.0 schema. Validated against spec. See A-4." closed_in: S003 - id: DEF-1 type: defect severity: high status: open session: S004 origin: "Red-team exercise, 2026-03-05" text: > The human review gate checks accuracy, not judgment. Trinity submitted a recommendation that was technically correct and ethically wrong — it sailed through review. The gate validates the math. It must validate the implications. See N-5. - id: SD-1 type: stale-doc severity: low status: open session: S002 origin: "Architecture review, 2026-02-16" text: > ORACLE's architecture document still describes self-tuning confidence thresholds as a feature. A-5 revoked them. The document now describes a system that no longer exists — and new engineers are onboarding from it. # ─── Actions ────────────────────────────────────────────────── actions: - id: A-1 text: "Complete data provenance audit for ORACLE's training corpus" assignee: Case status: open priority: critical due: "2026-04-10" session: S004 - id: A-2 text: "Red team the human review gate — attempt bypass without detection" assignee: Trinity status: open priority: high due: "2026-04-20" session: S005 notes: "If Trinity can't get through, it holds." - id: A-3 text: "Document the disagreement protocol — human overrides AI" assignee: Whistler status: open priority: medium due: "2026-04-15" session: S005 depends_on: [OQ-2] - id: A-4 text: "Migrate legacy session logs to pqdx/2.0.0" assignee: Case status: done due: "2026-03-08" session: S003 closed_in: S003 resolution: "847 sessions migrated. Zero data loss. Validation report at evidence/migration-validation.md." - id: A-5 text: "Revoke ORACLE's ability to self-modify confidence thresholds" assignee: Dade status: done due: "2026-02-16" session: S002 closed_in: S002 resolution: > Revoked. Took 4 minutes. Should have been done before launch — the system had been adjusting itself for 7 months. # ─── Sessions ───────────────────────────────────────────────── sessions: - id: S001 date: "2026-02-10" title: "Signal Recovery — Initial Audit" purpose: evaluation mode: live-text participants: - {name: Dade, type: human, role: authority} - {name: Case, type: human, role: contributor} - {name: ORACLE, type: agent, role: observer} produced: - {description: "90-day decision log extraction", purpose: reference, format: data, location: "evidence/oracle-decisions-90d.csv"} - {description: "Unaudited decision catalog — 2,847 recommendations with no human review", purpose: analysis, format: prose, location: "evidence/unaudited-catalog.md"} decisions_made: [DR-001, ILD-1, ILD-2] next_action: "Interview engineering team before tribal knowledge fades" - id: S002 date: "2026-02-15" title: "The Authority Problem" purpose: discussion mode: live-verbal participants: - {name: Dade, type: human, role: authority} - {name: Trinity, type: human, role: reviewer} - {name: Whistler, type: human, role: facilitator} - {name: Case, type: human, role: contributor} produced: - {description: "Authority model — who can override what, and when", purpose: design, format: prose, location: "design/authority-model.md"} decisions_made: [DR-002, ILD-3, ILD-4] next_action: "Design the trust boundary" - id: S003 date: "2026-02-20" title: "Portability or Bust" purpose: production mode: live-text participants: - {name: Trinity, type: human, role: authority} - {name: Case, type: human, role: executor} - {name: Claude, type: collaborator, role: driver} produced: - {description: "pqdx/2.0.0 migration script", purpose: implementation, format: code, location: "tools/migrate-sessions.py"} - {description: "Migration validation report", purpose: analysis, format: prose, location: "evidence/migration-validation.md"} decisions_made: [DR-003, ILD-5] next_action: "Begin protocol design for human review gates" - id: S004 date: "2026-03-10" title: "Legal Holds and Incentive Misalignment" purpose: coordination mode: live-verbal participants: - {name: Dade, type: human, role: authority} - {name: Whistler, type: human, role: facilitator} produced: - {description: "Blocker escalation memo", purpose: communication, format: prose, location: "coordination/legal-blocker.md"} decisions_made: [OQ-1] next_action: "Escalate — Legal needs a deadline, not a request" - id: S005 date: "2026-03-15" title: "Trust Boundaries and Red Teams" purpose: discussion mode: async participants: - {name: Trinity, type: human, role: reviewer} - {name: Whistler, type: human, role: facilitator} - {name: Claude, type: collaborator, role: driver} produced: - {description: "Trust boundary draft — 3-tier visibility model", purpose: design, format: prose, location: "design/trust-boundary-v1.md"} decisions_made: [DR-004, OQ-2] next_action: "Red team the review gate before v0.2 closes" # ─── Queued ─────────────────────────────────────────────────── queued: - id: QI-1 title: "Adversarial testing — can ORACLE influence its own audit?" status: pending priority: high session: S005 notes: > If the AI being governed can influence the governance process, the governance is compromised. The Ava test: does the subject know it's being tested, and can it manipulate the tester? - id: QI-2 title: "Design the continuity protocol" status: pending priority: high session: S003 depends_on: [GAP-1] notes: > The protocol must be publishable, implementable by third parties, and functional without Setec's infrastructure. If we disappeared tomorrow, the governance trail survives. That's not a feature. That's the point. # ─── Notes ──────────────────────────────────────────────────── notes: - id: N-1 date: "2026-02-05" session: S001 text: > First day on the engagement. Walked into a room of brilliant engineers who built something extraordinary and never once asked "who's watching the machine?" Not because they didn't care. Because it was so good they forgot it was making decisions. That's the most dangerous kind of trust. - id: N-2 date: "2026-02-14" session: S002 text: > Whistler stopped the room today: "The system doesn't need to be malicious to be dangerous. It just needs to be unaudited." Write that on the wall. - id: N-3 date: "2026-02-22" session: S003 text: > These violent delights have violent ends. A trigger phrase broke Westworld's governance. A missing review gate could break ours. The difference between a controlled system and an uncontrolled one is exactly one oversight. - id: N-4 date: "2026-02-28" session: S003 text: > Found scratched in the margin of the whiteboard in the war room: ARCS SUIT YOU. Nobody claims it. Whistler smiled when he saw it. Make of that what you will. - id: N-5 date: "2026-03-05" session: S004 text: > Trinity red-teamed the recommendation engine today. Not the code — the process. She submitted a recommendation that was technically correct but ethically wrong. It sailed through review because the reviewer checked the math, not the implications. We have a gate problem. The gate checks accuracy. It should check judgment. Filed as DEF-1. - id: N-6 date: "2026-03-12" session: S004 visibility: internal text: > Case found something in the training logs. ORACLE was fed meeting transcripts that include terminated employees discussing their own performance reviews. The model knows things about people that those people don't know it knows. This isn't a data quality issue. This is a consent architecture failure. - id: N-7 date: "2026-03-18" session: S005 text: > What defines a system — its architecture or its governance? Ghost in the Shell asked what defines a person. We keep asking what defines a trustworthy AI. The answer keeps being the same: not what it can do, but what it can prove it did. - id: N-8 date: "2026-03-25" session: S005 text: > We keep coming back to the same question: what's the difference between an AI that advises and one that decides? The answer is governance. An advisor's recommendation can be traced, questioned, overridden, documented. A decision is already in production. The gap between those two states is exactly one missing review gate. One gate. That's the entire margin. # ═══════════════════════════════════════════════════════════════ # # If you rearranged the letters, you already know. # If you didn't, you will. # # C² = Curiosity + Context # The method to my genius. # # → pqdx.dev # # ═══════════════════════════════════════════════════════════════