Backend platform (Go rewrite)¶
Status: Proposed 2026-09-26. Builds on event-bus.md, trade-execution.md, data-model.md. Ignored the legacy apps/backend code on purpose; M7 deleted it before the scaffold.
Decision¶
- Stay on Go. No Rust rewrite.
- One modular monolith, two entrypoints. One module, one container image,
cmd/api(HTTP, SSE, relay) andcmd/worker(JetStream consumers, pollers, relay). Modules talk through NATS events, never through each other's packages. Splitting a module into its own service later is a deploy change, not a code change. - Clean Architecture, three rings, enforced by lint.
domain(pure) ←app(use cases, ports) ←adapters(Postgres, NATS, Jupiter, Privy, HTTP). Import direction is checked in CI bydepguard, not by review. - Go channels are in-process only. NATS is the only cross-module path. "NATS channel based" means JetStream subjects between modules, and bounded Go-channel pipelines inside one handler. Never a Go channel as a substitute for the bus.
- Contracts are generated, not hand-written. OpenAPI 3.1 spec is the source of truth for the iOS contract (Go server stubs via
oapi-codegen, Swift client viaswift-openapi-generator). SQL is the source of truth for rows (sqlc). Event payloads are Go types in oneeventspackage with a subject registry. Every payload carries avfield, and a breaking payload change bumps it. - Harsh
golangci-lintv2, no inline//nolint, no comments. Exceptions live in.golangci.base.ymlor in a module's owninternal/modules/<m>/lint.yml, each with a reason.scripts/gen-golangcirenders both into.golangci.yml, which is generated. Hand-written Go carries only machine-read comments. Customforbidigorules encode Monaco-specific bans (floats for money,time.Nowin domain,context.Backgroundoutsidemain).
Why¶
- Go over Rust. The hot path is Jupiter
/execute(up to 2 min), Solana RPC and Privy signing. Latency is network-bound. Volume is thousands of events a day. Rust buys memory safety and speed we do not need and costs compile time, hiring, and agent throughput. NATS's first-party client and embeddable server are Go, so tests run a real JetStream in-process. - Monolith over microservices. One team, one database, money invariants that span a transaction (state change plus
eventsrow). Separate services would force a distributed transaction or a second outbox per service. Module boundaries plus NATS give the decoupling without the ops. - Three rings, not five. Classic Clean Architecture in Go tends to grow
entity → usecase → interactor → repository → gateway → presenterlayers with one-implementation interfaces at every seam. That doubles files and hides the call path. Three rings keep the rule that matters (domain imports nothing) and nothing else. - Lint over docs. Agents copy whatever the surrounding code does. A rule in a Markdown file gets skipped. A
depguardfailure does not.
Repository layout¶
Takes cmd/, internal/, api/, migrations/, deployments/, scripts/, docs/ from golang-standards/project-layout. Skips pkg/ (nothing is imported from outside this module), vendor/, src/, init/, web/. That repo is a community convention, not an official standard; internal/ is the only part the compiler enforces.
apps/backend/
├── AGENTS.md hard rules for agents (short; points at lints)
├── CHANGELOG.md Keep a Changelog, human-written
├── .golangci.base.yml hand-written lint config; .golangci.yml is generated from it
├── go.mod go 1.25+, toolchain pinned
├── api/
│ └── openapi.yaml source of truth for HTTP contract
├── cmd/
│ ├── api/main.go wiring only: config → adapters → app → http
│ ├── worker/main.go wiring only: config → adapters → consumers
│ └── monacoctl/main.go ops CLI: replay, dead-letter retry, backfill
├── internal/
│ ├── platform/ cross-cutting mechanism, no business rules
│ │ ├── config/ env → typed Config, validated once at boot
│ │ ├── clock/ Clock interface; real + fake
│ │ ├── db/ pgxpool, UnitOfWork, tx helpers
│ │ ├── bus/ JetStream setup, Dispatch wrapper, relay
│ │ ├── concurrency/ Pool, Pipeline, FanOut helpers (generic)
│ │ ├── httpx/ middleware, error → problem+json, SSE hub
│ │ ├── observability/ slog, OTel tracer/meter, NATS header propagation
│ │ └── money/ Micros, SignedMicros, TokenAmount, share math (math/big)
│ ├── events/ every event type + subject registry (one file per aggregate)
│ ├── modules/
│ │ ├── identity/ auth session, users, profile, user wallets
│ │ ├── cabal/ cabals, members, access requests, rules, treasury wallets
│ │ ├── governance/ proposals, votes, tally, expiry, execution outcome
│ │ ├── trading/ trade engine, swaps and their state machine, sweeper
│ │ ├── treasury/ ledgers (cabal_txns, user_txns), fund, cash out, positions
│ │ ├── funding/ deposits poller, withdrawals, onramp sessions, bounce, cabal pauses
│ │ ├── market/ assets catalog, price poller, price history, provider strategies
│ │ ├── agents/ agent keys, intents, budget enforcement
│ │ ├── social/ follows, feed, comments, chat bridge
│ │ ├── ranking/ valuation snapshots, leaderboards
│ │ ├── notify/ notifications, device tokens, APNs
│ │ ├── referrals/
│ │ ├── analytics/ PostHog export; consumes events, owns no business tables
│ │ └── admin/ admin actions, dead_letters
│ └── testkit/ fakes, fixtures, embedded NATS, Postgres container
├── migrations/ atlas versioned SQL, forward-only
├── flows.tsv every flow, its outcomes, and its test status (see Flows)
├── queries/ sqlc .sql files, one dir per module
├── deployments/ compose, Dockerfile, NATS stream config
└── test/
└── e2e/ black-box: real binary, real PG + NATS, fake externals
Each module has the same four directories. Nothing else.
internal/modules/governance/
├── domain/ Proposal, Vote, Tally(), status machine. Pure. No ctx, no I/O, no errors from infra.
├── app/ commands + queries (use cases). Defines the ports it needs.
├── adapters/ postgres repo (sqlc), http handlers (oapi-codegen strict server), consumers
└── module.go New(deps) → *Module; registers routes. `Consumers(config.Config)` feeds cmd/worker through generated per-module files
Dependency rules (enforced by depguard)¶
| Package | May import | May not import |
|---|---|---|
*/domain |
stdlib, platform/money, events |
app, adapters, platform/db, platform/bus, pgx, nats, net/http, other modules |
*/app |
own domain, events, platform/{money,clock} |
adapters, pgx, nats, net/http, other modules' app or domain |
*/adapters |
own app + domain, platform/*, drivers |
other modules' packages |
| module A | module B | never. Cross-module effects go through an event. Cross-module reads go through a read-only query port that module B exports in module.go. |
cmd/* |
everything | nothing imports cmd |
Table ownership¶
Each table has one module that writes it. Other modules read it through that module's query port or learn of changes through its events.
users, follows and cabal_messages soft delete: a deleted_at timestamptz column, and reads filter deleted_at IS NULL. Every other table keeps its current delete behavior. Ledger rows are never deleted.
| Table | Owner | Rule |
|---|---|---|
users |
identity | Soft delete through deleted_at. Account deletion sets it with account_status = deleted, scrubs PII and keeps ledger rows. first_deposit_at is set by identity's consumer of deposit.credited on the first deposit of at least $10, with a guarded update WHERE first_deposit_at IS NULL. Referrals reads it through identity's query port. handle is the unique username every user picks in onboarding (decided 2026-09-27). Referrals reads it through the same port, and after the first-deposit unlock the handle also works as a referral code (referrals.md). |
user_wallets |
identity | One Privy wallet per user. Sign-in reuses the user's existing Privy wallet and never creates a second. |
treasury_wallets |
cabal | One app-owned treasury wallet per cabal. Kept apart from user_wallets so identity and cabal never share a table. |
swaps |
trading | The swap state machine. Treasury never writes it. |
cabal_txns |
treasury | Written by treasury's consumer of trade.confirmed, and by fund and cash out. |
user_txns |
treasury | Written by treasury's consumers of deposit.credited and withdrawal.confirmed, and by fund and cash out. |
cabal_pauses |
funding | The one pause record. Trading and treasury read it through funding's query port at check time. There is no cabals.trading_paused_at. An ops pause is a pause record with reason ops. |
price_points |
market | One table for every price sample, price_micros bigint. The poller writes it; price history and charts read it. Pyth is dropped. |
follows |
social | Soft delete through deleted_at. A unique partial index on (follower_id, followee_id) WHERE deleted_at IS NULL lets a re-follow insert a new row. Follower and following counts are an indexed count(*) over live rows; there is no counts table. |
cabal_messages |
social | Chat messages. Soft delete through deleted_at. |
chat_seen |
social | One seen watermark per member per cabal. Rows are hard-deleted when a member leaves; they are not user content. |
dead_letters |
admin | One row per dead-lettered message, with resolve state. |
events, event_deliveries |
platform/bus |
Outbox and per-handler dedupe (see NATS hosting and budget). |
Patterns, and where each earns its place¶
| Pattern | Where | Shape |
|---|---|---|
| Unit of Work | Every write that emits an event. Required by the events-as-outbox rule. | uow.Do(ctx, func(ctx context.Context, tx Tx) error). Tx exposes the module's repos and Events.Append. Commit wakes the relay in the same process. No Begin/Commit anywhere else (lint: forbidigo on pgx.Tx.Commit outside platform/db). |
| Command | Every user or agent intent: ProposeTrade, CastVote, FundCabal, CashOut, Withdraw, SubmitAgentIntent, CreateCabal. |
Typed struct with IdempotencyKey, one Handle(ctx, cmd) (Result, error) per command. HTTP handler parses → builds command → calls handler. Same command can come from HTTP, an agent key, or monacoctl. |
| Query (CQRS-lite) | Every screen read. | Reads hit projections (cabal_positions, leaderboard_entries, feed) directly via sqlc, bypassing domain. Writes go through commands. No shared "service" object doing both. |
| Strategy | Asset issuers (xStocks, Tessera, PreStocks) and swap venue (Jupiter today). | type AssetProvider interface { Catalog(ctx); Quote(ctx, Asset, Micros) (Quote, error) }. Best-price buy runs every provider that lists the asset (fan-out, below) and picks the min. Venue interface for execution. |
| Registry (creational) | Providers keyed by Issuer; consumers keyed by durable name; handlers keyed by handler name; events keyed by subject. |
Built once in cmd/*/main.go. Duplicate key panics at boot. Tests assert every events type has at least one registered subject. |
| Smart constructor (creational) | Every branded type: UserID, CabalID, Micros, SharesUnits, Mint, SolanaAddress. |
ParseX(raw) (X, error) at the boundary, unexported fields so the zero value can't be forged outside the package. |
| Functional options (creational) | Clients with many knobs: Jupiter, Privy, NATS, HTTP server. | jupiter.New(baseURL, jupiter.WithTimeout(d), jupiter.WithRetry(p)). Not for domain types. |
| State machine | Proposal status, swap created → submitted → confirmed|failed, onramp session, cash-out job. |
Status is a Go type with a transitions table; Next(from, event) (to, error) is pure. Adapters apply it as a guarded UPDATE … WHERE status = $from. exhaustive lint fails on a missed case. |
| Anti-corruption layer | Jupiter, Privy, xStocks, Helius, APNs responses. | Adapter maps wire JSON to domain types. Wire types never leave the adapter package. |
| Circuit breaker + retry | Outbound HTTP only. | sony/gobreaker + capped exponential backoff with jitter. Never inside a DB transaction. |
Patterns deliberately left out: abstract factory, builder for domain objects, generic repository (Repository[T]), dependency-injection frameworks (wire/fx/dig). Constructors wired by hand in main.go are readable in one file.
Context rules¶
ctx context.Contextis the first parameter of anything that does I/O or may block. Domain functions take no context (they do no I/O).- Never store a context in a struct (
containedctx). Never passnil(staticcheck SA1012). context.Background()/TODO()only inmain, tests andplatform/busconsumer roots (forbidigo).- Every outbound call gets a deadline at the adapter: RPC 5 s, Privy 10 s, Jupiter quote 5 s,
/execute2 min. Set in config, not literals. - HTTP request context dies with the client. Work that must finish after the response (nothing, ideally; the event does it) uses
context.WithoutCancel(ctx)plus its own timeout, neverBackground. - Values in context are limited to request-scoped metadata: trace span, actor (
user/agent/admin/system), request id. Typed unexported keys. Never dependencies, never optional parameters. - Cancellation propagates into every goroutine. Every
selectthat sends or receives also selects<-ctx.Done(). - Use
context.WithCancelCause/context.Causeso logs say why work stopped. - Trace context crosses NATS in message headers (
otelpropagator on publish and inbus.Dispatch).
Concurrency rules¶
Where concurrency pays in Monaco, and the pattern for each:
| Job | Pattern | Bound |
|---|---|---|
| Price poller across N assets and 3 providers | Fan-out/fan-in with errgroup.WithContext + SetLimit |
8 in flight per provider |
| Deposit poller across all member wallets | Bounded worker pool reading wallet ids from a channel | 16 workers, RPC rate limiter shared |
| Leaderboard valuation per cabal | Pipeline: load holdings → value (needs prices) → write snapshot, each stage a goroutine, buffered channels | buffer 64, workers per stage 4 |
| Notification fan-out to device tokens | Worker pool, APNs HTTP/2 client shared | 32 |
| Best-price quote across issuers | Fan-out, first-error cancels, collect all | number of providers |
| Relay publish batch | Sequential (order and ack matter); parallelism comes from multiple processes and SKIP LOCKED. api and worker each run a relay. |
1 |
Hard rules:
- No bare
go f()in business code. Useplatform/concurrencyhelpers orerrgroup.forbidigoflagsgostatements outsideplatform/andcmd/(custom analyzer ifforbidigocan't match syntax; see Lint). - Every goroutine has an owner that waits for it. Tests run
go.uber.org/goleakinTestMain. - The sender closes the channel. Receivers never close.
- Every channel is bounded. Unbuffered or explicit size; no "big enough" 10000 buffers.
- Backpressure over dropping. Drop only for live SSE hints, and count drops in a metric.
- Shared mutable state goes behind one goroutine or a mutex in the same struct, never both. Prefer ownership over locks.
go test -racein CI, always.- JetStream
MaxAckPendingis the cross-process concurrency bound. In-process pools must not exceed it.
platform/concurrency exports exactly three generic helpers, each ~40 lines and fully tested:
func Pool[In, Out any](ctx context.Context, workers int, in <-chan In, fn func(context.Context, In) (Out, error)) (<-chan Out, <-chan error)
func Stage[In, Out any](ctx context.Context, in <-chan In, buf int, fn func(context.Context, In) (Out, error)) <-chan Result[Out]
func FanOut[T, R any](ctx context.Context, limit int, items []T, fn func(context.Context, T) (R, error)) ([]R, error)
How each behaves, so callers do not rediscover it:
Poolreturns two unbuffered channels and closes both after the last worker exits. Readoutanderrsin oneselectloop (or in two goroutines). A caller that ranges overoutalone stalls at the firstfnerror untilctxis cancelled, because the worker holding that error is blocked onerrs. That is backpressure, not a bug.Stageruns one goroutine, so itsResultstream keeps input order.bufis its only slack.FanOutreturns results in input order or nothing. The firstfnerror cancels the derived context with that error ascontext.Cause, items not yet started never run, and the returned error wraps that cause (errors.Ismatches).workers,bufandlimitmust be positive; zero or negative panics at the call.
Money and types¶
- USDC and token amounts are
money.Micros/money.BaseUnits(unsigned 64-bit branded ints). Ledger entries and P&L aremoney.SignedMicros, a signed 64-bit branded int. A balance or amount is never negative; a delta or a return can be. Share math multiplies then divides throughmath/bigand rounds down, in favour of the pot, in one function. float32/float64are banned indomain,app,platform/money(forbidigoonfloat64identifiers there). Display formatting is the client's job from integer micros plus decimals.- IDs are UUIDv7 branded per aggregate.
CabalIDcannot be passed asUserID. - Enums are named string types with an exhaustive
switch(exhaustivelint withdefault-signifies-exhaustive: false).
Errors¶
One error type, one closed list of codes, one table that maps each code to everything a boundary needs. Every edge case is a row in that table and a row in flows.tsv, so "did we handle X" is a lookup, not a code read.
Shape¶
type Kind uint8
const (
KindInvalid Kind = iota + 1 // caller's input; 400; never retry
KindUnauthorized // 401
KindForbidden // 403
KindNotFound // 404
KindConflict // idempotency replay mismatch, version clash; 409; never retry
KindBlocked // domain rule refused (insufficient funds, vote closed, pot cap); 422; never retry
KindUnavailable // upstream down or timed out; 503; retry
KindInternal // bug or invariant broken; 500; alert; term
)
type Error struct {
Code Code // closed enum, e.g. CodeInsufficientFunds
Kind Kind // derived from Code via the table; never set by hand
Op string // "treasury.FundCabal"
Attrs []slog.Attr // cabal_id, amount, provider; never secrets
Err error // wrapped cause, nil at the origin
}
Codeis a string enum ininternal/errs/codes.go. The tablecodes[Code] = {Kind, Retryable, Alert, Message}is the single source of truth.exhaustivefails a switch that misses a code. A test asserts everyCodeconstant has a table row and every row has a constant.- The OpenAPI
ErrorCodeenum is generated from that table bymonacoctl gen errorsand checked fresh in CI, so the Swift client'sswitchand the Go table cannot drift. Kinddecides the HTTP status, the bus verdict (ack, nak with delay, term) and whether to alert. Nothing else readsKind; thecodestable is the only map.- Domain returns
errs.New(CodeInsufficientFunds, op, attrs...). Adapters wrap upstream failures witherrs.Wrap(err, CodeJupiterUnavailable, op, attrs...). Sentinelvar ErrX = errors.New(...)is banned (forbidigoonerrors\.Newoutsideerrs), so an error without a code cannot exist. errors.Isanderrors.Asare the only comparisons (errorlint).errs.CodeOf(err)returnsCodeInternalfor any non-*errs.Error, which is the fallback that makes a missed wrap visible as a 500 and an alert, not a silent 200.
Surfacing¶
Each error is logged once, at the boundary that stops it, with the same trace_id the client received. Never log-and-return.
| Boundary | What it does with the error |
|---|---|
HTTP (httpx.Problem) |
RFC 9457 application/problem+json: code, message (from the table), trace_id, retryable. 4xx logs at Info, 5xx at Error. KindInternal also emits an alert. The response never carries Err or Attrs. |
Bus (bus.Dispatch) |
Retryable → nak with backoff. Otherwise term, write the message and the error to DEADLETTER, alert on KindInternal. event_deliveries row records the code. |
| Poller tick | Log with Attrs, count poller_errors_total{poller,code}, continue the loop. A tick never aborts the poller. |
cmd/* boot |
Fail fast with the code. No partial boot. |
Lint that keeps it to one log: slog.Error and slog.Warn are allowed only in platform/httpx, platform/bus, platform/observability and cmd/ (the forbidigo pattern with path exclusions in the lint config). Modules return errors; they do not narrate them.
Panics are recovered at exactly three roots: HTTP middleware, bus.Dispatch, and the poller tick wrapper. Each converts the panic to CodePanic (KindInternal) with the stack in Attrs, then follows the row above. The recovered panic is not re-raised in tests either: a test catches it through the panic problem or the Error line it asserts on. http.ErrAbortHandler is the one panic the HTTP root lets through, so net/http can abort the response.
Reaching every branch¶
Error handling that no test reaches is decoration. The rules that make each branch reachable:
- Every port fake in
testkithasFail(op string, err error)andFailOnce. Anif err != nilinappthat a test cannot trigger through a fake is a design smell, not a coverage exclusion. errcheckruns withcheck-blank: trueandcheck-type-assertions: true._ = f()does not compile past lint.nilerrandnilnilare on: returningnilwhenerr != nilis a lint failure.- Mutation testing (below) flips
err != niltoerr == niland removesreturn errlines. A surviving mutant is an error branch nothing checked. - Every
KindInternalin production is a bug. The alert links the trace, and the fix PR adds the code that should have caught it, so theInternalcount trends to zero.
Outcomes as a map¶
Each flow lists its outcomes in flows.tsv: the success path, every Code it can return, and every crash point (after-create, after-sign, after-execute, before-commit, after-publish). after-create covers a swap row committed as created whose process died before signing. A swap moves created → submitted in the same guarded write that stores its signed bytes, and that write commits before any send. So a created row was never signed or sent, and a crash after the send leaves the row in submitted. The trading sweeper fails any created row older than 2 min with no chain lookup. It resolves submitted rows through getSignatureStatuses. The crash-point test proves the flow converges. One acceptance scenario per outcome, named TestFlow07_FundCabal_InsufficientFunds. monacoctl flows check reads go test -json output and fails CI when an outcome in the TSV has no test that ran and passed. That is the codepath map: the file is the list, the test names are the proof, and the check is what keeps them equal.
Logs as evidence¶
Three records exist, and each answers a different question. The events table in Postgres is what happened to state, and it is the source of truth. JetStream is how that reached the consumers. Logs are what each process did and did not do, and why, including every branch that ended with no state change. When something goes wrong, the logs must be enough to reconstruct how the current state came to be without reading code.
Rules¶
- Structured only, stable names. slog with
attr-only,static-msg,context: all(sloglint). The message is an identifier,treasury.fund.rejected, never a sentence with values in it. Values are attrs. A message name is declared in amsgs_<area>.gofile ininternal/platform/observabilitynext to the attrs it requires, andgo generateregisters it. A test fails on a call site that logs an unregistered name or omits a required attr. That registry is also the log catalog page in the docs. - Every line carries the join keys.
trace_id,span_id,request_idorevent_id,actor,module,op. The context logger adds them; a handler never types them.sloglint context: allmeans a call site cannot get a logger without the context. - Log the decision, not the step. One line where a branch chooses: a guard refused (the code and the numbers it compared,
have=4_000_000 need=5_000_000), a retry was scheduled (attempt=3 delay=30s cause=JupiterUnavailable), a consumer skipped a duplicate (event_id delivery=2), a poller tick found nothing. Inaction is evidence. "Deposit poller ran at 10:04:10, scanned 212 wallets, found 0" is the line that proves a missing deposit was not the poller's fault. - Money lines carry before and after. Any line about a balance, share count or position has
before,after,delta,asset,cabal_id. The ledger's history can be read from logs alone. - Truthful means logged after commit. A line that says "funded" before the transaction commits lies when the commit fails.
uow.Doitself logstx.committedortx.rolled_backwith the cause and the event ids it appended, so every write gets its terminal line without the handler doing it. Inside a transaction closure onlyDebugis allowed. That rule has no lint yet; themoney-changeskill checklist carries it until one exists. - Levels mean one thing each.
Debugis step detail, off in production.Infois a decision or an outcome.Warnis degraded but handled (retry, fallback provider, stale price used).Erroris a boundary only, one per failure, per the Errors section. - Nothing secret, nothing personal. The slog handler in
platform/observabilityredacts by attr key (phone,email,token,key,seed,signature) and by value pattern (base58 secrets, JWTs). A golden test feeds each pattern through the handler and asserts the output. New attr keys that carry PII go on the list in the same PR.
Replay and seeded states¶
The events table plus deterministic consumers make state replayable. monacoctl replay --to <event_id> rebuilds every projection (cabal_positions, leaderboard_entries, feed) into a fresh database from the event log, so "what did the leaderboard show at 14:02" is a command, not archaeology.
The ledgers (cabal_txns, user_txns) are not projections. Each ledger row commits in the same transaction as the event or delivery that caused it, and replay never rebuilds them. monacoctl replay --verify recomputes every balance from the event log and fails on any difference from the ledger. Replay reads events only; it never calls Jupiter, Privy or RPC, which is why consumers keep side effects behind ports.
The same mechanism seeds tests. testkit/scenarios/ holds named event sequences (cabal-with-pending-trade, member-mid-cashout, treasury-after-bounce), each produced by running the real commands once and exporting the resulting events rows. testkit.Seed(t, db, "cabal-with-pending-trade") replays one into the test's database in milliseconds. Crash-point and acceptance tests start from a seeded state instead of building it by hand. monacoctl events export --cabal <id> --anonymize pulls a real cabal's history so a production bug reproduces locally from the same state.
Logs ship to Grafana Loki over the same OTLP pipeline as traces, with trace_id as the link between a log line, its trace, and the events row. Retention is 30 days in Loki; the events table is forever.
Lint¶
golangci-lint v2, run in CI and a pre-commit hook. Baseline for greenfield: every finding blocks.
version: "2"
run:
timeout: 5m
tests: true
linters:
default: none
enable:
# correctness
- govet
- staticcheck
- errcheck
- ineffassign
- unused
- nilerr
- nilnil
- bodyclose
- sqlclosecheck
- rowserrcheck
- durationcheck
- makezero
- copyloopvar
- exhaustive
- gosec
- musttag
- asasalint
- reassign
# errors
- errorlint
- wrapcheck
- errname
- err113
# context and concurrency
- contextcheck
- containedctx
- fatcontext
- noctx
- spancheck
# architecture
- depguard
- forbidigo
- gochecknoglobals
- gochecknoinits
- ireturn
- interfacebloat
# complexity
- cyclop
- gocognit
- funlen
- nestif
- maintidx
# style that prevents bugs
- revive
- gocritic
- unparam
- unconvert
- usestdlibvars
- intrange
- perfsprint
- prealloc
- sloglint
- predeclared
- exptostd
# tests
- testifylint
- thelper
- tparallel
- paralleltest
- usetesting
settings:
cyclop: { max-complexity: 12 }
gocognit: { min-complexity: 15 }
funlen: { lines: 70, statements: 40 }
nestif: { min-complexity: 4 }
exhaustive: { default-signifies-exhaustive: false }
sloglint: { attr-only: true, context: all, static-msg: true, no-global: all }
interfacebloat: { max: 5 }
wrapcheck:
ignore-package-globs: ["github.com/monaco/backend/internal/*"]
forbidigo:
analyze-types: true
forbid:
- pattern: '^(fmt\.Print.*|print|println)$'
msg: use slog
- pattern: '^context\.(Background|TODO)$'
msg: only main, tests and bus roots create root contexts
- pattern: '^time\.(Now|Since|Until)$'
msg: inject platform/clock.Clock
- pattern: '^os\.Getenv$'
msg: read config through platform/config at boot
- pattern: '^float(32|64)$'
msg: money is integer base units (platform/money)
- pattern: '^log\.'
msg: use slog
- pattern: '^(pgxpool\.New|pgx\.Connect|sql\.Open|nats\.Connect)$'
msg: platform/db, platform/bus and testkit own connections
- pattern: '^errors\.New$'
msg: every error carries a Code; use errs.New
- pattern: '^slog\.(Error|Warn)$'
msg: modules return errors; boundaries log them once
- pattern: '^uuid\.(MustParse|Must)$'
msg: fixtures take ids from the injected generator
depguard:
rules:
domain:
files: ["**/domain/**"]
deny:
- { pkg: "github.com/jackc/pgx", desc: "domain is pure" }
- { pkg: "github.com/nats-io", desc: "domain is pure" }
- { pkg: "net/http", desc: "domain is pure" }
- { pkg: "github.com/monaco/backend/internal/platform/db", desc: "domain is pure" }
app:
files: ["**/app/**"]
deny:
- { pkg: "github.com/jackc/pgx", desc: "declare a port" }
- { pkg: "github.com/nats-io", desc: "emit via Tx.Events" }
- { pkg: "net/http", desc: "use cases are transport-free" }
exclusions:
rules:
- path: _test\.go
linters: [funlen, err113, wrapcheck]
- path: main_test\.go
linters: [gochecknoglobals]
- path: cmd/
linters: [forbidigo]
- path: internal/platform/(db|bus|httpx|observability)/
linters: [forbidigo]
- path: internal/errs/
linters: [forbidigo]
- path: internal/testkit/
linters: [forbidigo]
formatters:
enable: [gofumpt, gci, golines]
settings:
gci: { sections: [standard, default, localmodule] }
golines: { max-len: 120 }
No godox or nolintlint: the comment ban below covers both, and inline //nolint is banned. A lint exception is an exclusions.rules entry in this file with a path and a reason, so every exception sits in one reviewed place. revive's exported rule stays off and the v2 comments exclusion preset stays on, so no linter asks for the doc comments the ban removes.
forbidigo narrows by file only through exclusions, so the domain-only float and clock bans need path rules; expect to tune that on day one. Cross-module import bans (module A importing module B) need one depguard rule per module, generated by scripts/gen-golangci so a new module can't forget it. Tuning numbers (cyclop 12, funlen 70) are starting points; raise per-package with a reason, never globally.
Also in CI:
| Check | Tool |
|---|---|
| Vulnerabilities | govulncheck ./... |
| Generated code is fresh | sqlc diff, oapi-codegen + git diff --exit-code |
| Migrations lint | atlas migrate lint on new SQL (no table rewrites, no NOT NULL without default on large tables, no destructive change without a -- atlas:nolint destructive line the reviewer sees) |
| Flows | monacoctl flows check (see Flows) |
| OpenAPI lint and breaking changes | vacuum lint, oasdiff breaking against main |
| No comments | monacoctl lint comments (see No comments) |
| Changelog touched | fail if apps/backend/** changed and CHANGELOG.md didn't, unless PR has no-changelog label |
| Race + leaks | go test -race, goleak |
| Coverage and mutation | see Testing CI gates |
Testing¶
Target: 100% statement coverage, and mutation testing alongside it so the 100% means something. Agents write code many times faster than people, so spend the saved time torturing the code: unit, acceptance, QA, property, fuzz, mutation, performance and jitter tests. The constraint is that just test backend stays fast and deterministic, so development speed doesn't drop.
What counts toward 100%¶
- One merged profile. Unit and integration tests use
-coverpkg=./.... E2E and QA runs build the real binaries withgo build -coverand write toGOCOVERDIR.monacoctl coverage --profile <unit profile> --covdir <dir>...merges them throughgo tool covdata, appliescoverage.exclude, names every uncovered block asfile:start-endand fails under 100%.just test backendruns it on its own profile. Until E2E exists, eachcmd/*test re-runs its own test binary as the realmain(testkit.WithChild,testkit.StartMain), withGOCOVERDIRset to the parent test's coverage directory, so the child's coverage lands in the same profile. - Excluded paths are fixed and listed in one file (
coverage.exclude): generated code (*.gen.go, sqlc output, oapi-codegen output) andinternal/testkit. Nothing else. - No ignore pragmas. Code a test can't reach gets deleted or made unrepresentable by a type change. An error branch that only fires on infra failure gets reached through a fake port that returns the error.
- Go has no branch coverage, and statement coverage can be gamed. Mutation testing closes that gap: a surviving mutant means a line ran but nothing checked its result.
Test types¶
| Type | Scope | Tooling | Runs in |
|---|---|---|---|
| Unit | domain pure functions, state machines, share math |
table tests, t.Parallel, no I/O |
just test backend |
| Property | invariants over random inputs: shares never mint value, rounding favours the pot, entries sum to zero per asset, both headers on a transfer_id always share a status |
pgregory.net/rapid |
just test backend (100 cases), nightly (100k) |
| Model-based (stateful property) | random action sequences (fund, trade, cash out, crash, redeliver) run against the real app layer and a tiny in-memory model; states compared after each step | rapid.StateMachine-style actions |
just test backend (short), nightly (long) |
| Fuzz | every parser at a boundary: HTTP bodies, event payloads, Jupiter/Privy/Helius responses, config | native go test -fuzz, corpus committed |
seeds in just test backend; 10 min per target nightly |
| App / integration | each command and query against real Postgres | template-DB clone per test (below), fake ports, fake clock | just test backend |
| Adapter | sqlc repos, HTTP handlers, consumers, relay | real Postgres + embedded NATS, httptest fakes replaying recorded fixtures |
just test backend |
| Bus semantics | redelivery, dedupe, term, dead letter, InProgress, out-of-order |
embedded NATS plus the chaos dispatcher (below) | just test backend |
| Contract | server responses match openapi.yaml; event payloads match golden JSON |
kin-openapi validator middleware in every HTTP test; golden files with -update |
just test backend |
| Acceptance | one scenario per Flows row, in business language | Go scenario DSL (scenario.New(t).Given(...).When(...).Then(...)) over HTTP against the in-process app |
just test backend |
| E2E | same scenarios against real api + worker binaries |
compose: PG + NATS + fake externals; -cover binaries |
PR CI |
| QA | monacoctl verify on the real binaries; evidence is the verify-evidence CI artifact |
cmd/monacoctl verify |
stage 2 (merge queue) and nightly |
| Crash-point | panic at named points (after create, after sign, after /execute, before commit, after publish), restart, assert convergence. A created row was never signed or sent, so the sweeper fails it after 2 min; a crash after the send leaves the row submitted, which the sweeper resolves through getSignatureStatuses. |
faultpoint hooks compiled in under the faultpoints build tag |
just test backend in process; E2E in PR CI |
| Jitter / concurrency | pools, pipelines, relay, consumers under random delays and interleavings | testing/synctest + seeded delay injection + -race |
just test backend (fixed seeds), nightly (seed sweep) |
| Performance (deterministic) | allocations per op on hot paths; query count per request | testkit.AssertAllocs (testing.AllocsPerRun) in allocs_test.go, which runs alone and without -race; testkit.AssertQueries counts the queries on the test's own testkit.DB. Both compare with the package's testdata/perf/baseline.json, and -testkit.perf-update rewrites it |
just test backend |
| Performance (timing) | benchmarks compared against main; load on the full stack |
b.Loop + benchstat; vegeta against the e2e stack |
nightly, and on PRs labelled perf |
| Mutation | all non-generated packages | gremlins through just test mutation (monacoctl mutation): on a PR, only the lines the diff against its base changes (gremlins --diff); --all mutates every line of the whole module. A survivor fails unless mutants.allow lists it with a reason |
PR CI (changed lines), nightly (every line) |
| Leak | every package | goleak.VerifyTestMain, run by testkit.Main, the only allowed TestMain body (nogo testmain) |
always |
Keeping it fast¶
Budget: just test backend under 90 s on a laptop, and the required PR checks under 6 min wall clock. The budget is a CI gate, not a hope: testkit records each package's wall time from go test -json, monacoctl test-report prints the ten slowest tests, a package over 10 s gets a warning, and a package over 20 s or a run over 90 s fails. The package limits are the same on a laptop and in CI. The run limit applies on a laptop only: in CI (CI set) test-report --ci gates packages but not the run, because a PR runner starts from a cold build cache and the job has its own 3 min budget. A test that breaks the budget gets fixed or moved to nightly, never skipped. just test backend runs go test -p 4: at the default of one test binary per core, every binary runs its parallel tests at once, and on an 8-core laptop packages that take 3.5 s alone took 10 to 13 s. With -p 4 the same suite took 28.8 s instead of 41.1 s and every package stayed under 8 s (2026-09-27, load 8 to 10). Tests that build or run other binaries (the lint-rule fixtures and the tools they build, the ids type-check fixtures, the testkit fixture packages, the -cover binary merge) skip under -short; CI runs them in their own step and nightly runs them once.
Where the time went under the original 60 s budget. Every row was measured on 2026-09-27 against throwaway code shaped like the target, on an M2 under background load. Rollout step 1 re-measures on the real scaffold and in CI, and sets the gate from those numbers:
| Cost | Estimate | What keeps it there |
|---|---|---|
| Compile and link the test binaries | Measured 2026-09-27 on today's backend as a stand-in (38 packages, 30 with tests, pgx, no NATS): cold cache 56 s throttled (-p 2, 4 cores, nice 19); nothing changed 1.4 s (31 of 31 cached); one file changed in a package most others import 10.5 s (-p 4, nice 19), which is recompile plus relink of every dependent test binary. Linking is the floor: each test package is one binary, and each NATS-importing binary adds about 0.2 s to load. |
Go's build and test caches. -count=1 disables the test cache and never writes to it, so just test backend never passes it. -shuffle=on still reseeds. Keep the test-package count near the module count: tests for a module's app and adapters live in one _test package per module rather than one per directory, so a change in platform/ relinks about 15 binaries, not 50. |
| Postgres | 1 to 2 s | A separate test container, monaco-postgres-test on its own port, with fsync=off, synchronous_commit=off, full_page_writes=off and its data on tmpfs, so a crash loses nothing that matters. Migrations run once into monaco_tmpl. just test backend starts it only if it is down. The dev container monaco-postgres never runs with durability off: docker-compose.yml pins fsync, synchronous_commit and full_page_writes on the command line, which overrides ALTER SYSTEM, and scripts/verify-local-db.sh fails if any is off. On 2026-09-27 a benchmark left fsync=off in the dev volume, the machine crashed, and the data directory came back unreadable. |
| Template clone per test | Measured 2026-09-27, Postgres 16 in a tmpfs container with durability off, today's 29 migrations (26 tables, 9 MB), GOMAXPROCS=4, -parallel 4, nice 19. Raw SQL: create 27 ms, drop 19 ms. Through pgtestdb with pgx: 64 parallel tests, each cloning, running 5 queries and dropping, took 1.0 s of test phase, 15.5 ms per test amortized and 60 ms for one test on its own (the drop runs inside t.Cleanup, before the slot frees, so it is on the critical path). The same 64 tests serial on one shared database with TRUNCATE between them took 1.9 s, 30 ms each. A single clone costs more than a TRUNCATE; clone-per-test wins because it runs in parallel. -race makes it 48 ms per test amortized. With default fsync, drop alone is 169 ms, which is why the test container turns durability off. Building the template from all migrations takes 1.5 s, paid once when it is missing. CI gets its own number from monacoctl bench db in Rollout step 1. |
github.com/peterldowns/pgtestdb, which caches its own template keyed by the migrator's hash and clones it per test. Per-test rollback isn't an option because the Unit of Work commits and the relay reads committed rows. |
| Embedded NATS | Measured 2026-09-27 (M2, GOMAXPROCS=4, nice 19, nats-server v2.14.5): server start 27 ms, stop under 1 ms, 1.7 MiB heap. 32 parallel tests, each creating its own stream and consumer and moving 50 messages, took 114 ms of test time (3.6 ms per test amortized, 12.7 ms per test body). Plus about 0.2 s of binary load per NATS-importing package on a cold test cache. -race roughly triples the test phase. |
One nats-server per package in TestMain, one stream per test. |
| Property, model-based, fuzz seeds, jitter | Measured 2026-09-27 on pure domain code: 3 invariants × 100 cases, a model test of 20 steps × 100 runs, 60 fuzz seeds and fixed-seed synctest jitter took 10 to 30 ms in-process together. The package costs about 0.5 s, nearly all of it the fixed per-binary start, and about 1.5 s with -race. A model test against the real app layer pays one testkit.Reset (31 ms) per run, so 100 runs is about 3 s: those run with fewer runs under -short. |
just test backend runs with -short and sets RAPID_CHECKS=500 RAPID_STEPS=40, because rapid divides both by its own -short factor (5 and 2) to land on 100 cases and about 20 attempted steps. They are env vars, not -rapid.* flags, because go test ./... passes a flag to every test binary and a binary that does not link rapid rejects it. Corpus seeds only. Fixed jitter seeds. Nightly runs 100,000 cases and -fuzz per target. |
Acceptance, one per flows.tsv outcome, in-process |
Measured 2026-09-27 with a light stand-in scenario (own clone, 3 HTTP calls to an in-process httptest.Server, one pgx transaction writing two tables, rows asserted): 16.8 ms per scenario amortized at -parallel 4, 60 ms median and 80 ms p90 for one scenario on its own. 120 scenarios is about 2 s plus 0.4 s of binary load. Real scenarios do more work; budget 3× until the scaffold measures them. |
HTTP against the in-process app, no binaries, no compose. |
Measured on the scaffold (#485, 2026-09-27): 26 test packages, no NATS and no acceptance scenarios yet. just test backend runs go test -json -race -shuffle=on -short -coverpkg=./... -coverprofile, the non-race allocation pass, test-report, flows check and coverage, with the test cache cleared before each run and the build cache warm. The laptop is an 8-core Apple silicon Mac with a 1-minute load average of 14 to 30 from other agents, so these are loaded numbers:
| Cost | Laptop, 8 runs | CI (ubuntu-latest, cold build cache) |
|---|---|---|
Whole run, first go test event to last |
26.0 to 47.6 s, median 35.7 s, p95 47.6 s | 64.6 s, most of it compiling with -race and coverage |
Coverage instrumentation (-coverpkg=./...) |
no difference outside the noise: 36.1 s median without, 35.7 s with | |
Postgres, template clone per test (platform/db) |
6.8 to 9.2 s of package time | 2.8 s |
Property (platform/money, 100 rapid cases) |
2.7 to 6.9 s | 1.3 s |
Slowest package (cmd/monacoctl, fake atlas, go and gremlins scripts plus git) |
8.4 to 17 s | 2.3 s |
| NATS, acceptance | not measured: neither exists on the scaffold yet |
The laptop run gate is 90 s. A test package warns at 10 s and fails at 20 s (cmd/monacoctl/testreport.go). Both rose on 2026-09-29 (#831; see the ci.md log). At the time of this measurement the gates were 60 s for the run (p95 times 1.5 is 71 s, which the RFC budget capped at 60 s) and 10 s for a package. cmd/monacoctl went over 10 s on the loaded laptop in 5 of 8 runs and never on an idle runner.
Not in just test backend: E2E (real binaries, compose, including their crash points), mutation, timing benchmarks. Those are PR CI or nightly.
PR CI is three required jobs, each with its own budget (when they run and on what runners is in ci.md): lint plus unit plus integration (under 3 min, sharded by package), E2E (under 4 min), and mutation on changed lines (under 10 min, or the PR is too big and gets split). Nightly runs everything unbounded.
- No sleeps.
time.Sleepin tests is banned byforbidigo, and nogotestwaitbans its disguises outsidesynctest.Testandtestkit: a bare<-time.After(d)andtime.After,time.Tickortime.NewTickerinside aforloop. A test on a real socket waits on a signal,testkit.Eventuallyortestkit.AssertNoRedelivery. Time-dependent code takes the injectedclock.Clock, and goroutine timing usestesting/synctest, where virtual time advances instantly once every goroutine is blocked. Measured: the full 1 s, 5 s, 30 s, 2 min, 10 min backoff schedule (12m36s of fake time) runs in 19 to 40 µs, and an 8-worker pool covering 12.5 s of fake time in 1 to 4 ms. - Bus timers are real time. synctest and
clock.Clockcannot speed up timers insidenats-server. Measured: redelivery takes exactlyAckWaitplus 1.5 ms, the max-deliveries advisory takesMaxDeliver × AckWait, and proving thatTermstops redelivery costs the whole wait window. So bus-semantics tests run withAckWait100 ms fromtestkit.NATS, one sample each, in parallel. That is about 0.5 s of mostly idle wall time per package.testkit.NATSrejects anAckWaitover 250 ms. - Everything parallel.
t.Parallelis required (paralleltestandtparallellint). That only works because every test owns its database and its stream, below. - Run what changed. On PRs, mutation covers only the lines the diff changes, and long property runs only the packages affected by it (
go list -depsagainst changed files). The nightly mutates every line, so a survivor a PR did not write still gets found.
Isolating test state¶
Isolation is by database, not by row. A test never shares a table with another test, so there is nothing to scope, prefix, or clean up.
testkit.DB(t)is the only way a test gets Postgres. It clones the template, returns a pool bound tot, registers the drop int.Cleanup, and names the database aftert.Name()plus a random suffix. Two tests cannot see each other's rows because they are in different databases.testkit.NATS(t)is the same for the bus: one stream and one consumer set per test, named aftert.Name(), deleted int.Cleanup.- Fixtures take
tand build into that test's database:testkit.NewCabal(t, db, testkit.WithMembers(3)). IDs come from the injected UUIDv7 generator, never literals. - A property or model-based test runs every iteration in the same database, and calls
testkit.Reset(t, db)(TRUNCATE ... RESTART IDENTITY) between iterations, because 100 clones per test would eat the budget.
Lint that makes the shared-state leak a compile-time failure:
| Leak | Lint |
|---|---|
| A test opens its own connection to a shared database | forbidigo bans pgxpool\.New, pgx\.Connect, sql\.Open, nats\.Connect outside platform/db, platform/bus, testkit. |
A package-level *pgxpool.Pool, *testkit.DB or *nats.Conn shared across tests |
gochecknoglobals stays on for _test.go. TestMain may hold only the embedded server handle, allowed by an exclusions.rules entry scoped to main_test.go. |
A test reads DATABASE_URL and hits whatever is there |
forbidigo already bans os.Getenv. testkit reads the test URL from platform/config once in TestMain. |
| Hard-coded IDs that collide across tests | forbidigo bans uuid\.MustParse and uuid\.Must in _test.go. |
| A test that is not parallel and hides an order dependency | paralleltest, tparallel, -shuffle=on. |
| Two tests with the same name in a package, so two databases get the same name | testkit.DB panics on a duplicate t.Name() within a run. |
| Leftover state after a failure | pgtestdb keeps a failed test's database for debugging. That is useful for one failure and dangerous for many: 500 failing tests would keep about 4.5 GB, and on 2026-09-27 kept clones from a failing benchmark filled a 1 GB tmpfs and killed the container. So testkit.DB keeps at most the first 5 failed databases per run and drops the rest, TestMain drops every t_* and testdb_* database older than one hour at start, and the test container's tmpfs is sized at 2 GB. |
Keeping it deterministic¶
- Every nondeterminism source is injected: clock, UUID generator, random source and Jupiter/Privy responses.
forbidigoalready banstime.Now. Addmath/randglobals anduuid.Newoutsideplatform/. - Seeds are printed and replayable.
go test -shuffle=oncatches order dependence. Rapid, fuzz, jitter and the chaos dispatcher each log their seed on failure. For rapid the flag is-rapid.seed=<n>; measured with shrinking off, it reproduced the exact raw counterexample 3 of 3 times, also under-short -race -shuffle=on. Rapid saves each failure totestdata/rapid/<Test>/*.failand replays it first on the next run. Those files are committed as regression cases in the fix PR. CI runs withRAPID_NOFAILFILE=1so a CI failure never writes to the tree. - Chaos dispatcher. A test-only
bus.Dispatchwrapper that, driven by a seed, duplicates messages, reorders them within a window, delays acks pastAckWaitand redelivers after a simulated crash. Every consumer's tests run through it. This makes the "handlers never assume order, redelivery is harmless" rule from event-bus.md something a test checks, instead of something the doc claims. - No flake quarantine. A flake is a bug with a seed. It gets fixed or reverted, never retried into green.
Test code quality¶
- Builders for fixtures (
testkit.NewCabal(t, testkit.WithMembers(3), testkit.WithPot(100_00))), the same functional-options pattern as production code. No shared mutable fixtures. - Assertions check effects: ledger balances,
eventsrows, HTTP status and body, published messages. They never check that a mock method was called.testifylintand a review rule back this up. - Every bug fix lands with the test that failed first, at the lowest layer that can catch it, plus a lint rule when the bug class is mechanical.
CI gates¶
| Gate | Threshold |
|---|---|
| Merged statement coverage | 100% of non-excluded statements |
| Mutation efficacy on changed packages | 100% of mutants killed, or listed in mutants.allow with a reason (equivalent mutants only; reviewed like a lint exclusion) |
| Allocations and query counts | no increase over the recorded baseline without a baseline update in the same PR |
-race, goleak, -shuffle=on |
zero findings |
| Nightly benchmarks | alert on a >10% benchstat regression with p < 0.05 |
Flows¶
Each flow is a command, the events it emits, the consumers that react, and every outcome it can end in. The table below is the target state of the rewrite, not the current code. Most of rows 1 to 17 existed in the deleted legacy backend in some form; 18 to 28 are partly new. Nothing here is "future reference": every row becomes a test before its module's rollout step closes.
flows.tsv is the source of truth¶
The table below is a render. The file is apps/backend/flows.tsv, one line per flow, tab-separated, and it moves with the code. TSV over YAML because a flow is one line, the diff shows exactly which cell changed, cut and awk read it, and there is no indentation for an agent to get wrong. List cells use ;.
| Column | Contents | Checked by |
|---|---|---|
id |
07 |
unique |
flow |
Fund cabal |
|
module |
treasury |
directory exists |
trigger |
POST /v1/cabals/{id}/fund or consumer:proposal.passed or poller:deposits |
route in openapi.yaml, subject in registry, or poller registered |
command |
FundCabal |
Go type exists in module/app |
events |
cabal.fund_submitted;cabal.funded |
each in the events registry |
consumers |
treasury.positions;ranking;feed;referrals |
each a registered durable |
outcomes |
ok;InsufficientFunds;CabalPaused;PrivyUnavailable;crash:after-sign;crash:before-commit |
each Code in the errs table; each crash point a registered faultpoint |
status |
planned, built, verified |
built needs every outcome test present; verified needs a flow script per outcome (below) |
doc |
docs/architecture/deposits-withdrawals.md#fund |
file and anchor exist |
monacoctl flows check runs in CI and fails on any column's check. It also reads go test -json from the run and requires, for each flow with status ≥ built, a passing test named TestFlow<id>_<Command>_<Outcome> per outcome. A flow row with no test is a red build, not a backlog item. The check is what makes the file a map of the system instead of a wish list.
status = verified means every outcome has a flow script registered in Scripts() (internal/testkit/flows/scripts.go), so monacoctl verify all drives it against the real binaries in stage 2. monacoctl flows check fails a verified flow with an outcome that has no script.
Generated from the TSV, checked fresh in CI: the table below (monacoctl docs flows), the verify-backend feature map, and the acceptance test skeletons (just gen flow <id> writes one failing test per outcome).
Adding a flow is one row plus the tests it names. Deleting a flow deletes the row, and the check fails until its tests go too.
notify appears in a consumers cell only when that event sends a push (notifications.md). A consumer's own ticket appends its durable name to the cell when it lands.
| # | Flow | Command / trigger | Events | Consumers |
|---|---|---|---|---|
| 1 | Sign in (Apple or Google; SMS OTP in dev builds only, never production) | POST /v1/auth/session with Privy token; reuses the user's existing Privy wallet |
user.created (first time), user.auth_state_changed |
analytics, referrals (mint code; AttachReferral is the only attribution path), social (contact matches) |
| 2 | Create cabal | CreateCabal |
cabal.created |
feed, analytics |
| 3 | Join open cabal / request / invite / approve | JoinCabal, RequestAccess, InviteMember, DecideAccess |
cabal.member_joined, cabal.access_requested, cabal.access_decided |
feed, ranking |
| 4 | Leave cabal | LeaveCabal (guarded: the member holds zero shares, so the app routes them to cash out first; not creator with members) |
cabal.member_left |
feed, ranking |
| 5 | Crypto deposit | Deposit poller sees USDC in member wallet | deposit.credited |
treasury (user_txns), notify, identity (users.first_deposit_at on the first deposit of at least $10), analytics |
| 6 | Card deposit | CreateOnrampSession, page PATCHes status |
onramp.status_changed then flow 5 |
analytics |
| 7 | Fund cabal | FundCabal → Privy transfer member→treasury → confirm → mint shares at live price |
cabal.fund_submitted, cabal.funded |
treasury positions, ranking, feed, referrals, analytics |
| 8 | Direct transfer to treasury | Treasury watcher in funding; writes the cabal's pause record in the same transaction; a bounce writes no ledger entries | cabal.external_deposit_detected, cabal.external_deposit_bounced; cabal.paused when the cabal's first pause reason opens and cabal.resumed when its last closes, both appended by funding |
cabal.external_deposit_*: admin. cabal.paused, cabal.resumed: notify |
| 9 | Propose trade | ProposeTrade (advisory route + pot check via market) |
proposal.created |
feed, notify |
| 10 | Vote / tally | CastVote; expiry job |
proposal.passed / .failed / .expired |
trading, feed, notify, analytics |
| 11 | Execute trade | trade-engine consumer on proposal.passed; reads the pause through funding's query port |
trade.blocked / trade.submitted (appended by the swap layer on submit) / trade.confirmed / trade.failed; governance then emits proposal.executed / proposal.execution_blocked |
governance (on trade.confirmed / trade.blocked), treasury (cabal_txns), ranking, feed, notify, analytics |
| 12 | Retry failed trade | RetryTrade from POST /v1/swaps/{id}/retry |
same as 11; the swap layer appends trade.submitted on resubmit |
same |
| 13 | Withdraw / void proposal | WithdrawProposal, admin VoidProposal |
proposal.withdrawn, .voided |
feed |
| 14 | Cash out | CashOut from POST /v1/cabals/{id}/cashouts → burn shares → sell if short → pay USDC to member wallet |
cashout.started, cashout.completed / .partial / .failed |
ranking, feed, analytics |
| 15 | Withdraw to address | Withdraw |
withdrawal.submitted, .confirmed, .failed |
treasury (user_txns), analytics |
| 16 | Agent lifecycle | Proposal kinds add / pause / resume / remove agent | agent.enabled, .paused, .removed, agent.key_revealed |
governance (marks the agent proposal executed on agent.enabled / .paused / .removed), agents, feed |
| 17 | Agent trade | SubmitAgentIntent (key auth, budget check at submit and again at execution) |
agent.intent_created → same engine as 11 |
trading, same as 11 |
| 18 | Prices | One market poller, every 120 s (fan-out over providers), writes price_points |
price.tick (core NATS only, one batched message per tick, not stored as event); asset.price_moved, appended by the same poller (there is no second poller) |
price.tick: ranking, live SSE. asset.price_moved: feed |
| 19 | Valuation + leaderboards | Every 2 minutes, and on trade.confirmed, cabal.funded, cashout.completed |
ranking.snapshot_written |
live SSE |
| 20 | Follow / unfollow | Follow, Unfollow |
follow.created, .removed |
notify, feed ranking, analytics |
| 21 | Feed + comments | CreateComment |
comment.created |
notify, live SSE, analytics |
| 22 | Chat | Ably for delivery; backend issues token and persists | chat.message_posted |
notify (mentions) |
| 23 | Profile edit | UpdateProfile |
user.profile_updated |
ranking (names), feed |
| 24 | Notifications | Consumers write notifications row, then send |
notification.sent |
none |
| 25 | Referrals | Click, sign-up, first deposit | referral.attributed, referral.qualified |
referral.attributed: social. referral.qualified: analytics |
| 26 | Admin | Any admin command | admin.action |
audit |
| 27 | Dead letters | Advisory subscriber | none | admin writes dead_letters; monacoctl deadletter retry |
| 28 | Nudges | Identity nudge job | user.nudge_due |
notify |
Thin client¶
The iOS app renders; the server decides.
- One
GETper screen returning everything that screen shows, already computed: pot value, share price, your stake, returns, display names, human-readable asset names, formatted-ready integer amounts with decimals. No math in Swift beyond formatting. - Every mutating call takes an
Idempotency-Keyheader; the server stores the response and replays it. - Server-sent events (
/v1/stream) push "this changed, re-fetch" hints from core NATS. The app never polls on a timer except as SSE-reconnect fallback. Fan-out is in-process (see SSE hub), never one NATS subscription per phone. - Errors carry a stable
codeand user-facingmessage. The app showsmessagein a toast; it switches oncodeonly for flows that branch. - Swift client is generated from
api/openapi.yaml. A route change that breaks the client failsoasdiff breakingin CI.
NATS hosting and budget¶
Hosted NATS is Synadia Cloud. Self-hosting JetStream on a PaaS means one node on one disk; Synadia runs the cluster. Start on the free Personal plan; move to Starter ($49/mo) when the EVENTS stream needs replication for real money. Limits are enforced by the account JWT, not billed, so going over is an error the outbox absorbs (events wait in Postgres), never a surprise invoice. The one billed item is network egress beyond the pool.
Personal plan limits, and what the design spends:
| Limit | Plan | Monaco | How it stays there |
|---|---|---|---|
| Accounts | 2 | 2 | monaco-prod, monaco-staging. Local dev and tests use embedded or compose NATS, never Synadia. |
| Connections | 10 | 2 steady, ~6 in a deploy | One nats.Connect per process, in platform/bus only (the forbidigo pattern in the lint config). nats.Name("monaco-<api|worker>") so strays show in the dashboard. Rolling deploys briefly double the count. |
| Subscriptions per connection | 100 | api: 1. worker: ~15 | SSE hub below; worker has one pull consumer per module plus price.> and the delivery-failure advisory. |
| Streams | 10 | 2 | EVENTS and DEADLETTER, declared in code and applied by monacoctl bus apply in the pre-deploy step. api and worker never create streams. No KV buckets or object stores: each is a stream underneath; that state lives in Postgres. |
| Storage | 5 GiB | 2.5 GiB cap | MaxBytes 2 GiB on EVENTS, 512 MiB on DEADLETTER. MaxAge 7 days on EVENTS: Postgres events is the source of truth, the stream is transport. |
| Network data | 10 GiB | ~1.5 GiB/month | Price ticks are one batched price.tick message per poll carrying every asset, every 120 s (decided 2026-09-27). That is 12 times fewer ticks than the earlier 10 s cadence, so ticks drop from ~3 GiB to ~0.25 GiB a month. Per-asset messages at 5 s would be ~20 GiB/month on their own. Domain events are ~1 GiB. |
| HA streams | 0 | 0 | Replicas: 1. On Starter, EVENTS goes to Replicas: 3; nothing else changes. |
Stream and consumer shape:
var streams = []jetstream.StreamConfig{
{
Name: "EVENTS",
Subjects: events.Subjects(),
Storage: jetstream.FileStorage,
Replicas: 1,
MaxBytes: 2 << 30,
MaxAge: 7 * 24 * time.Hour,
Discard: jetstream.DiscardNew,
Duplicates: 2 * time.Minute,
},
{
Name: "DEADLETTER",
Subjects: []string{"deadletter.>"},
Storage: jetstream.FileStorage,
Replicas: 1,
MaxBytes: 512 << 20,
MaxAge: 30 * 24 * time.Hour,
},
}
Subjectscomes from theeventsregistry, so a new event can't add a stream.DiscardNew, neverDiscardOld: a full stream rejects the relay's publish and the outbox retries;DiscardOldwould silently drop events no consumer has read.Duplicatesonly works ifbus.PublishsetsNats-Msg-Idtoevents.idon every publish. The relay does; thenats-consumerskill says so.- One durable pull consumer per module on
EVENTSwithFilterSubjects,AckExplicit,MaxDeliver10,MaxAckPending64 (the cross-process concurrency bound from the concurrency rules). A module gets a consumer, never a stream. - A module's consumer runs one or more named handlers.
event_deliveriesis keyed by event id and handler name, so each handler dedupes on its own and a redelivery skips only the handlers that already applied it. A daily job deletesevent_deliveriesrows older than 30 days, which outlives both the 7-dayEVENTSand the 30-dayDEADLETTERwindows. - A message that exhausts
MaxDeliveror is termed goes toDEADLETTER, and admin's consumer writes adead_lettersrow that stays open until someone resolves it.monacoctl deadletter retryis the one recovery path. No safety-net poller re-scans stuck trades. - Boot fails if
nats.Connectfails. NoRetryOnFailedConnect: an instance that passes health checks with a dead bus backs the relay up silently. - The worker exports
js.AccountInfoevery minute as OTel gauges (storage used vs limit, stream and consumer counts); alert at 80%. Connection count and egress are not inAccountInfo; those are read in the Synadia dashboard.
What a limit hit looks like:
| Limit hit | Symptom | Damage |
|---|---|---|
| Connections | new process fails boot: maximum account active connections exceeded |
new instance fails health check; old keeps serving |
| Storage | relay publish rejected: maximum bytes exceeded |
events wait in the outbox; consumers catch up when space frees |
| Subscriptions | one Subscribe call errors |
one feature breaks, loudly at boot |
| Network | billed per GiB | money, not an outage |
Not verified against the pricing page: the consumer cap (check nats account info after signup; the plan uses ~10), and whether the 10 GiB network figure is per month and counts both directions. Both assumed.
SSE hub¶
Two things get called "channels" and they must stay separate. NATS subjects are how the backend talks to itself; an SSE connection is how the api talks to one phone. The wildcard changes only the second.
The naive shape, one NATS subscription per connected phone, hits 100 subscriptions at 100 phones and delivers the same cabal hint once per member to the same process. So the api holds one subscription, hint.>, and routes in memory:
NATS ──"hint.cabal.42.updated"──▶ api
│
▼
hub: who is listening for cabal 42?
│
┌───────────┼───────────┐
▼ ▼ ▼
phone A phone B phone C
- The hub is a map from key (
cabal:<id>,user:<id>,global) to SSE writers. On connect, the api registers the phone under its user id, the cabals it is a member of, andglobal. Every connection joinsglobal, which carries feed and leaderboard hints. On a hint, the hub parses the subject, looks up the key, writes to those phones and no others. - Correctness moved from NATS subject permissions to one function,
hub.Register. It is an authz check and gets a test: a phone in cabal 7 does not receive cabal 42's hint. Membership changes re-register. - NATS subscriptions scale with api processes, not phones: one api, one subscription, at any number of connected clients.
- Hints are core NATS and droppable (concurrency rule 5). A dropped hint costs one missed re-fetch until the next one; the SSE-reconnect fallback covers the rest.
Deploy and observability¶
Render runs the two binaries; Synadia runs NATS; Grafana Cloud receives OTLP. Nothing is self-hosted.
| Piece | Where | Why |
|---|---|---|
api |
Render web service, Docker runtime, dockerCommand: dotenvx run -f .env.production -- /app/api, healthCheckPath: /healthz |
Long-lived SSE connections need a host without an aggressive idle timeout. |
worker |
Render background worker, same image, maxShutdownDelaySeconds: 180 |
Grace period longer than a 2-minute Jupiter /execute. |
| Migrations and stream config | Render preDeployCommand: monacoctl migrate apply && monacoctl bus apply |
Runs once per deploy before the new instances start. Nothing migrates at boot. |
| Postgres | Render managed Postgres with point-in-time recovery, or Supabase Postgres (already used for photo storage) | The ledger. Either host; not a container on a volume. |
| NATS | Synadia Cloud (above) | |
| Secrets | One host env var, DOTENV_PRIVATE_KEY_PRODUCTION; everything else in encrypted .env.production |
Same dotenvx flow as local. |
| Traces, metrics, logs | OTel SDK in platform/observability, OTLP over HTTP straight to Grafana Cloud (Tempo, Mimir, Loki), no collector. otelslog bridges slog. OTEL_SERVICE_NAME is monaco-api or monaco-worker. |
One trace from HTTP request through the NATS header to the consumer (context rule 9). Switching hosts changes one endpoint. |
| Error reports | Sentry stays for panics and KindInternal, fed from the same boundary that logs them |
Cheaper to keep than to rebuild alert routing. |
| Alerts | Grafana alert rules on poller_errors_total, dead-letter count, relayer SOL balance, js.AccountInfo usage, p95 per route |
Replaces the old webhook. |
Rules the deploy exposes:
- Pollers run on one worker. Consumers and the relay scale across replicas (JetStream and
SKIP LOCKEDshare the work). Pollers do not: two replicas would poll deposits twice. Each poller takes a Postgres advisory lock named after the poller at tick start and skips the tick if it fails to get it. Until that lands, worker replicas stay at 1. - Shutdown order on SIGTERM: stop fetching, send
InProgressfor in-flight long work, finish or nak what is running, drain the NATS connection, close the pool, exit. Anything cut off is redelivered and deduped throughevent_deliveries. - Worker health. The worker serves
/healthzon an internal port, and the health answer includes NATS connected, pool reachable, last poller tick age. - Relayer floor. Boot refuses to start when the relayer holds 0.001 SOL or less. The Grafana alert fires at 0.05 SOL so the deploy never hits the boot check.
Fly.io would be the choice only for self-hosted NATS or multi-region. Railway's Postgres is a container on a volume and does not fit the ledger.
Agent-friendly codebase¶
The rewrite is a chance to build a codebase agents can extend without a human reading every line. Source: poteto's talk on trust ladders (transcript supplied 2026-09-26). The ladder, strongest first, and where each Monaco rule sits:
| Rung | Mechanism | Monaco examples |
|---|---|---|
| 1. Codebase makes the mistake impossible | Types, package layout, generated code | Branded IDs and Micros; internal/ + per-module packages; UnitOfWork is the only way to get a Tx; OpenAPI and sqlc generate the types; state machines as tables |
| 2. Static analysis | Lint, compiler, CI | depguard rings and module walls, forbidigo bans, exhaustive, freshness checks, changelog check |
| 3. Rules and skills | AGENTS.md, .claude/skills, review bot |
Four skills below; AGENTS.md under 60 lines |
| 4. Style guide | Human review | Nothing should live only here. A review comment made twice becomes a rung 1 or 2 change |
Standing rules that follow:
- One paved path per job. One way to write a command, a consumer, a query, a provider, a migration. Generators emit it. A second way is a lint failure, not a style debate.
- Every correction becomes structure. When a human or reviewer corrects an agent, the fix PR also adds the lint rule, type change or generator change that stops it recurring. A rule can land before the cleanup; it stops the spread while the old sites get fixed.
- Workarounds spread, so none land. An agent copies whatever it sees, and a comment is where a workaround explains itself. Go code has no comments; see No comments.
- Gardener. A scheduled agent (weekly) runs
golangci-lintwith stricter candidate settings,deadcode,gremlinsand the generators' drift check, and opens small PRs that delete dead code or propose a new rule. One human owns merging them.
No comments¶
Comments written by agents are low value and often wrong: they narrate the code, go stale, or justify a workaround that the next agent copies. So hand-written Go has none, tests included.
Allowed, because a tool reads them:
| Comment | Reader |
|---|---|
//go:build, //go:generate, //go:embed and other //go: directives |
Go toolchain |
Anything in a file headed // Code generated ... DO NOT EDIT. |
sqlc, oapi-codegen output; skipped via ast.IsGenerated |
-- name: GetUser :one in queries/*.sql |
sqlc; .sql is outside the check |
description: in api/openapi.yaml |
The HTTP contract and the generated Swift client; YAML is outside the check |
Banned: everything else. That includes doc comments on exported identifiers (everything is under internal/, so nothing reads godoc outside the module), package comments, // TODO, and inline //nolint.
Where the thought goes instead:
| Urge | Home |
|---|---|
| What this does | A better name or a smaller function |
| An invariant | A type (Micros, branded IDs) or a validating constructor |
| An edge case | A test whose name states the case |
| TODO or known gap | A GitHub issue |
| A design decision | A doc under docs/architecture/ with a Log entry |
| Why this change | The commit message or PR body |
Enforcement:
- Checker.
cmd/monacoctl lint commentsparses every non-generated.gofile withparser.ParseComments, walksfile.Comments, allows only the//go:prefix and printsfile:linefor the rest. Runs in the pre-commit hook and CI. - Agent hook. A Claude Code
PostToolUsehook onEdit|Writeforapps/backend/**/*.goruns the checker on the touched file and blocks with the offending lines, so the agent removes the comment in the same turn instead of in CI. Cursor gets the same check through its hooks. AGENTS.md. One line: "No comments in Go. If you want one, you need a better name, a type, a test, or an issue."
Cost: IDE hover shows no docs. With internal/-only code and descriptive names, that is small.
Verification skill¶
Agents need to prove backend work runs, not only that it compiles and its unit tests pass. verify-backend runs the real api and worker binaries against a real database and bus, drives a flow the way the app would, and writes down what the system did. It runs in the merge queue (stage 2) and nightly, after the tests.
What lives where:
| Piece | Path | Contents |
|---|---|---|
| Skill instructions | .claude/skills/verify-backend/SKILL.md (mirrored to .cursor/skills/) |
When to run it, the commands, how to read evidence, what counts as a pass, and what to do on a failure. Short; the CLI does the work. |
| Feature map | .claude/skills/verify-backend/feature-map.md |
Generated from flows.tsv by monacoctl docs flows. Per flow: trigger, command, events, consumers, tables, outcomes, and the exact command that verifies it. Checked fresh in CI. |
| The CLI | cmd/monacoctl verify |
Stack, driver, invariant checks, evidence writer. Code, tested like any other code. |
| Fakes server | cmd/fakes, built from internal/testkit/fakes |
Privy, Jupiter, Solana RPC, Helius, xStocks, APNs and Ably over HTTP, replaying recorded fixtures. Scriptable per request: succeed, fail with a given error, delay, or hang. |
| Flow scripts | internal/testkit/flows/<id>.go |
The same steps as the flow's acceptance scenario, written once and run by both go test (in-process) and verify (against binaries). |
| Seed scenarios | internal/testkit/scenarios/ |
Named event sequences replayed into the database before a flow starts (see Replay and seeded states). |
| Evidence | apps/backend/.verify/<flow-id>.json, git-ignored; CI uploads it as the verify-evidence artifact |
One file per flow, or <flow-id>-crash-<point>.json for a crash run, stamped with the commit and whether the tree was dirty. |
What one run does:
- Stack up. A throwaway Postgres container on its own port with data on tmpfs, never the dev container. An embedded NATS server. The fakes server.
apiandworkerbuilt with-coverand started with config that points every outside base URL at the fakes. The run fails fast if any piece is not healthy within 10 s. - Seed. Replay the flow's starting scenario, for example
cabal-with-membersfor flow 7. - Drive. Run the flow script over HTTP with real auth headers,
Idempotency-Keys and the SSE stream open. - Wait for convergence. Poll until every event the flow emitted has been acked by every consumer in
flows.tsv, or time out at 30 s. A timeout is a failure with the stuck consumer named. - Check invariants. Ledger entries sum to zero per asset. Share units match the pot. No dead letters. No
KindInternalin the logs. Every log line the flow must emit (registered inmsgs.go) is present. The HTTP status andcodematch the expected outcome. - Write evidence and print a readable summary.
- Tear down. Stop the binaries, remove the container, merge coverage into
GOCOVERDIR.
Modes:
| Command | Runs |
|---|---|
go run ./cmd/monacoctl verify all |
Every outcome of every built or verified flow, crash points excepted. |
go run ./cmd/monacoctl verify flow 00 |
One flow, every outcome in its outcomes cell except crash points. |
go run ./cmd/monacoctl verify flow 00 --outcome InvalidInput |
One outcome. The fakes server is scripted to produce it. |
go run ./cmd/monacoctl verify all --crash-at after-publish |
The after-publish crash point of every flow that has one. Kills the worker there, restarts it, and checks the flow converges to the same end state. |
scripts/ci/e2e.sh runs verify all, then verify all --crash-at after-publish.
An evidence file holds: the commit SHA and whether the tree was dirty, the flow and outcomes run, each HTTP request and response (secrets redacted by the same slog handler), the events rows written, per-consumer acks and redeliveries, ledger balances per asset before and after, dead letters, the required log lines found, p50 and p95 handler latency, and pass or fail per invariant. Evidence from a dirty tree does not count.
Gates:
- In the merge queue. The
e2ejob runsscripts/ci/e2e.shon every backend queue entry and fails on any failed invariant or budget. Owners do not runverify; stage 0 ismonacoctl agents check. - On failure. Fix the code and let the queue rerun it. Read the
verify-evidenceartifact to see what the system did. Never weaken an invariant or raise a budget to get green.
Budget: 90 s, enforced¶
monacoctl verify all finishes in under 90 s of wall time, measured from the command starting to the last container removed. That is a failure condition, not a target. The design aims for 45 s so ordinary noise never trips it.
How the budget is enforced:
- A hard deadline in the CLI.
verifyruns under onecontext.WithTimeoutCauseof 90 s. When it fires, the run stops, tears down, exits non-zero, and writes evidence with"result": "over_budget", which fails the job. - Phase budgets, so a failure says where the time went. Stack up 10 s. Seeding 2 s per flow. Each flow, all outcomes included, 15 s. Teardown 5 s. A phase over its budget fails the run with the phase named, even when the total is under 90 s. So a slow flow is caught the day it gets slow, not the day the total finally crosses the line.
- Timings are evidence. Every evidence file records each phase's duration and the machine it ran on (
GOOS, CPU count, CI or local). - Not built yet. Regression alarm. CI compares each flow's duration with the median of its last 20 runs on
mainand fails the PR when a flow is more than 50% slower and more than 2 s slower. That catches creep long before the hard limit. - The enforcer is tested.
verify's own test suite plants a flow that sleeps past its budget and a stack that never becomes healthy, and asserts both fail with the right phase named. The budget cannot break silently. - No raising the number in a PR. The 90 s and the phase budgets live in
cmd/monacoctl/verify/budget.go, and a change to that file needs thebudget-changelabel from a human reviewer. An agent that hits the limit fixes the slow code or splits the PR.
How the design stays inside it:
- One stack, flows in parallel. The stack starts once. Flows run concurrently on it, 4 at a time, each in its own seeded cabal and users, so they share binaries without sharing data. Invariant checks are scoped to each flow's IDs.
- Nothing waits on a real-world clock. The fakes answer instantly unless scripted to delay.
verifyconfig setsAckWaitto 100 ms and Jupiter/executeto its fake. Convergence is detected from consumer ack notifications, not by polling with sleeps. - Build once. Binaries come from Go's build cache; an unchanged tree relinks nothing. Seeds replay events in milliseconds instead of running commands.
- Too many flows means too big a PR. A branch whose touched flows cannot fit in 90 s at 4-way parallelism is split, the same rule as mutation testing's 10-minute limit.
The risk is noise: wall-clock gates are not perfectly deterministic, and a laptop at heavy load can run slow. The 45 s design target is the margin for that, and the evidence file records the load average at the start so a slow local run is easy to tell apart from slow code. monacoctl verify all is exempt from the 90 s total but not from the per-flow budgets.
Rollout step 1 measures a real run on the scaffold and confirms the phase budgets fit.
Docs, changelog, agents¶
- MkDocs Material on GitHub Pages.
mkdocs.ymlat repo root withdocs_dir: docs. The existingdocs/tree (index, product, architecture decision log, how-to) is the nav. A workflow on push tomainrunsmkdocs build --strict(broken links fail) and deploys withactions/deploy-pages. PRs runmkdocs build --strictonly. API reference rendersapi/openapi.yamlviamkdocs-render-swaggeror a Redoc page. Event catalog page generated from theeventsregistry bygo run ./cmd/monacoctl docs events > docs/reference/events.md, checked fresh in CI. - CHANGELOG.md. Keep a Changelog format,
## [Unreleased]at top, sections Added / Changed / Fixed / Removed / Security. One line per user- or operator-visible change, written for a person, not a commit log. CI check above enforces it. Release cuts move Unreleased under a dated version. - Agent rules live in structure first.
apps/backend/AGENTS.mdstays under 60 lines: the three rings, "cross-module = event", "money = integer + UnitOfWork", "runjust test backendandgolangci-lint runbefore done", "no comments in Go", plus a pointer to the skills. Everything else is a lint. - Generators over instructions.
just gen module <name>,just gen command <module> <Name>,just gen consumer <module> <name>,just gen provider <name>emit the correct files, test skeletons, depguard rule, registry entry and CHANGELOG stub. Agents copy what exists; make the first copy right. - Skills in
.claude/skills/(and mirrored for Cursor): go-backend-module: adding a command/query/consumer end to end, including test layers and flow table update.go-concurrency: when to usePool,Stage,FanOut, errgroup; the eight concurrency rules;goleakand-racerequired. References the Mario Carrión fan-in/fan-out article for the base pattern and the helpers for the house version.money-change: checklist for anything touching ledgers, shares, swaps: property test, crash-point test, guarded update, event in same tx.nats-consumer:bus.Dispatchcontract, idempotency viaevent_deliveries, retryable vs term,InProgressfor long work,Nats-Msg-Idon publish, consumer-not-stream per module.verify-backend: the instructions and feature map above. The merge queue runs it on every backend entry.
Pull requests: small and stacked¶
Agents write code faster than people can review it. One large PR hides the change that matters, gets skimmed, and blocks everything behind it until it lands. So work ships as a stack of small PRs, each built on the one below, each reviewable in a few minutes and green on its own. Graphite manages the stack.
Setup¶
Graphite is a required tool, installed by ./scripts/install-dev.sh (just install) like Go and just:
brew install withgraphite/tap/graphitegt auth --token <token>, with the token from https://app.graphite.com/activategt init --trunk mainonce per clone.
Agents in Claude Code on the web or CI install it with npm install -g @withgraphite/graphite-cli and read the token from GRAPHITE_AUTH_TOKEN.
Rules¶
- One PR is one verifiable unit. It passes stage 0 (
monacoctl agents check) and stage 1 on its own, without the PRs above it (Verification scope). A PR that only makes sense with the next one gets merged with it. - Order a stack so each PR proves the next. Delete or rename first. Then schema and migration. Then
domainandappwith their tests. Then adapters and HTTP. Last, theflows.tsvstatus change with its flow scripts. The Rollout steps below are each one stack, not one PR. - Size limit: under 1000 changed lines. CI fails a PR at 1000 or more changed lines, counting added plus deleted lines in hand-written code, tests and docs. A pure rename counts as zero. Generated Go,
go.sum, lockfiles, images,testdata, evidence files and rendered reference docs don't count; the list isIGNOREDinscripts/check-pr-size.py. A human reviewer can add thelarge-prlabel to let an oversized PR through, for example a mechanical change such as a rename; an agent adds it only when a human says to. A stack that lands as one queue entry passes without the label: its top PR's body starts withLands stack: #a #b #c, and each listed PR is under the limit against its own parent and has averifysuccess on its head. - Split a branch that grew too big. When work piled up on one branch or at the top of a stack, split it before submitting. The
distribute-stack-changesskill does it by copying exact hunks onto the lowest branch that owns each behavior, restacking after each commit, and checking the top branch has zero diff from a saved reference. It never rewrites the work.gt split --by-hunkdoes the same by hand. The skill lives in each person's~/.agents/skills, not in the repo. - Title: what the PR changes. For example
Add errs code table and problem+json mapping: present tense, no issue number, no commit-type prefix such asdocs:orfeat(x):. Every stack starts from a GitHub issue in the write-ticket format, and each PR links it under Why (Closes #212orPart of #212). - Body: written for a human who reads nothing else.
.github/pull_request_template.mdholds the sections, and GitHub and Graphite pre-fill it. Thepr-summaryskill in.claude/skillsdrafts it. The sections: - TLDR: what changed and its effect, in one or two sentences.
- Why: the problem, and what breaks or stays slow without this PR.
- What changed: grouped by behavior, not by file.
- Proof: the exact commands run and what they printed. Include test counts, and measurements with their conditions. Say what was not verified.
- What came up: surprises, wrong assumptions, decisions made along the way, and follow-ups filed as issues. A reviewer should learn here what the author learned.
- Reviewer focus: where to look hardest.
Open PRs as drafts with
gt submit --stack --no-interactive --draft: without--draft, a new PR opens ready with the commit subject as its title and the empty template as its body, and PR format fails its first run. Then runscripts/pr-body.sh <n> "<title>" <file>for each PR. It runs the fullscripts/check-pr-format.pylocally (title, body, commits and stacked neighbours, with the base and head read fromgh pr view), and only when that passes sets the title and body and runsgh pr ready. On a PR that is already ready it updates the title and body only. - Create and push with
gt, notgh.gt create -m "<message>"makes a branch and commit on top of the current one.gt modifyamends and restacks everything above.gt submit --stackpushes the stack and opens or updates every PR with the right base. Plaingit pushorgh pr createon a stacked branch sets the wrong base or breaks the stack. - Keep the stack current.
gt sync --no-interactive --no-restackpulls the trunk and deletes merged branches without restacking anyone else's work.gt restackrebases this stack onto it. During a batch only the root runsgt sync. Resolve each conflict in the branch where it appears, thengt continue. Never leave conflict markers staged: rungit diff --checkbeforegt add. - Force-push only after checking the remote.
gt submit --forceoverwrites the remote branch. First confirm the remote has no commits the local stack lacks:git log --oneline <local>..origin/<branch>prints nothing. On 2026-09-27 a restack found a remote branch whose hash differed from the local one; the patch was identical, and that check is what proved it safe. - Land the whole stack once. When every PR in the stack has a green
ci / ci-okand averifysuccess on its head,monacoctl agents land-stack <top-pr>points the upper PRs at the feature branch, writesLands stack: #a #b #cas the top body's first line, and queues only the top PR. The merge queue tests the stack once and merges it with a merge commit, so GitHub marks the lower PRs merged. Runland-stack <top-pr>again after the merge: it closes any lower PR not shown merged and runsgt syncin the stack's worktree. Graphite keeps each branch's parent in its own metadata, so the base change does not confuse it, butgt submitwould reset the bases and pull the stack out of the queue. The agent guard hook refusesgt submit,gt modifyandgt restackwhile the stack is queued,gh pr edit --baseoutsideland-stack, andgh pr merge --autoon a PR whose base is not the feature branch. Graphite's own merge queue is not used. - Describe each PR on its own. A reviewer reads one PR, not the stack, so each body stands alone and links its neighbours only for context.
Verification scope¶
Checks run in three stages, and each stage runs only what the stage before it skipped. The operator approved this split on 2026-09-28, so a check that runs in a later stage is not skipped.
| Stage | Where | Runs | Budget | Runs how often |
|---|---|---|---|---|
| 0. Agent check | Owner's worktree, monacoctl agents check |
go build and go vet on affected packages, then golangci-lint, nogo and the comment lint, then go test -short -count=1 on affected packages, no -race. For non-Go paths, the cheap row for that path (bash -n and shellcheck; cd scripts && go test -short on the touched test files; python3 -m unittest …; swift test for packages/mobile-core). Path-triggered rows mirror CI's ready, migration lint, OpenAPI lint and oasdiff, and mkdocs --strict |
per row, under [check.budget] in .monaco/agents.toml; no cap on the whole run |
Once before each push (hook-enforced) |
| 1. PR check | CI, pull_request |
The repo-wide checks, including those stage 0 runs on the changed paths only: plan, golangci-lint, nogo, the comment lint, OpenAPI lint and oasdiff, migration lint, ready (tidy, generated files, sqlc, docs), PR format. No tests. |
≤2 min | Once per change to the PR's diff. A push with the same diff reuses the last green result. |
| 2. Queue check | CI, merge_group |
The full suite: scripts/test-backend.sh (race, all packages, time budget), the -short-skipped tests, flake on changed tests, monacoctl verify (scripts/ci/e2e.sh), scripts tests, and mobile-core and iOS only when their paths changed |
≤6 min backend-only (iOS adds ~12) | Once per queue entry. A stack is one entry. Reruns only after an ejection. |
Stage 1 runs no tests because stage 0 already ran the tests the change can affect, on exactly the pushed code, and stage 2 runs every test against the real feature-branch tip with all queued changes combined before anything lands. The verifier reviews in parallel with stage 1 and runs no tests.
Stage 0 diffs HEAD against origin/<feature branch> and picks its rows from the changed paths:
- Every diff:
scripts/check-pr-size.pyandscripts/check-gate-changes.pywithBASE_SHAset to the stack parent, so an upstack PR is measured against its own parent, as CI measures it. A gate-changes warning shows as a count on the passing row. apps/backend/**: the packagesmonacoctl ci affectedprints, which is./...whengo.mod,go.sum,internal/testkit/**or a non-Go file changed. The lint row runs the CI lint job's golangci-lint, nogo andmonacoctl lint commentson them, and refuses a golangci-lint that differs fromapps/backend/.golangci-lint-version.- A
.shfile, or an extensionless file with abash,shorzshshebang:bash -nandshellcheck. - Any path: the
scripts/**/*_test.gotests andscripts/**/test_*.pyfiles that the diff touches or that name the changed file's basename in a string literal, for example"agent-guard.py". packages/mobile-core/**:swift test.apps/backend/**,scripts/ci/ready.sh,scripts/gen-docs.shorscripts/install-sqlc.sh:scripts/ci/ready.sh, as the CI ready job runs it.apps/backend/migrations/**,atlas.hclor.atlas-version:monacoctl migrate lint.apps/backend/api/openapi.yaml,.vacuum.yamlorscripts/ci/oasdiff-*: the pinned vacuum lint in Docker, the oasdiff self-test, and oasdiff against the stack parent's spec.docs/**,mkdocs.yml,requirements-docs.txtoropenapi.yaml:mkdocs build --strictfrom the README's.venv, in the worktree or the main checkout. Without one the row printsskipand the install hint.
The ready and migrate rows first run scripts/install-sqlc.sh or scripts/install-atlas.sh when .bin/ lacks the tool, as CI does. The stack parent is gt parent, or the --base ref when Graphite does not track the branch or its parent is the feature branch.
A path in no row runs nothing in stage 0. The paths with no checks in any stage are docs/**, **/*.md, .claude/**, .cursor/**, .github/** (actionlint runs in ci / Plan), scripts/cloud-setup.sh and .env.local; ci / Plan treats them as inert (ci.md). apps/mobile/** still needs just build mobile and gold-sim QA from whoever changes it.
monacoctl agents check prints at most 20 lines, writes the full log under .git/pstack/<milestone>/logs/, and exits 1 naming the slowest package when a row runs over its kind's budget, or a package in the go test -short row runs over the per-package budget in .monaco/agents.toml [check.budget]. On a pass it records git rev-parse HEAD^{tree} in .git/pstack/<milestone>/checks/. It refuses a working tree that differs from HEAD, since it records HEAD's tree.
scripts/agent-guard.py holds owners and verifiers to stage 0. It applies when .git/.monaco/agents/<ticket>.json names the calling worktree, so the operator and the root session are unaffected. It blocks gt submit and git push until agents check has passed on the current tree; the heavy tests (just test backend, scripts/test-backend.sh, go test with -race or without -short, go test ./... from apps/backend); CI polling (gh run watch, gh pr checks --watch, gh pr checks in a loop, sleep over 10s); gt submit with --publish or without --draft; and a raw gh pr ready, since only scripts/pr-body.sh marks a PR ready. gh run rerun <id> --failed and gh pr ready --undo stay allowed.
The same hook blocks gt sync without --no-restack for every session, because a plain sync restacks other agents' stacks mid-build. scripts/agent-guard-dispatch.py, a PreToolUse hook on the Agent tool, holds the spawn itself. A prompt with a brief: docs/agents/owner.md or brief: docs/agents/verifier.md line must run as pstack:poteto-agent with an explicit opus or sonnet model, and an owner's ticket: <n> needs the record monacoctl agents dispatch writes after it checks that every blocker has merged. dispatch prints the owner's spawn line and prompt, and verify-plan prints the verifier's. Spawns without the brief line, such as a skill's explorers, pass untouched.
The PR's Proof section pastes the monacoctl agents check output and says that CI covers the rest. A PR with no checks says No code paths affected: and names the paths.
Enforcement¶
- Agent guard.
scripts/agent-guard.pyruns before every agent shell command. For PRs, it blocksgh pr edit --baseoutsidemonacoctl agents land-stackand an inline--bodyongh pr createorgh pr edit. In an owner or verifier worktree, it also holdsgt submitto the draft flow. - CI, live now.
.github/workflows/pr-format.ymlrunsscripts/check-pr-format.pyon every non-draft PR, again whenever the title or body is edited. It fails a title that starts with an issue number or a commit-type prefix, and a body missing any of the six sections or with a section that holds only the template's comment. It also fails when Why links no ticket (Part of #n, orCloses #non the ticket's last PR), when a PR closes a ticket that an open PR stacked directly above it is part of, when the closing PR has no## Needs from Logansection or a fenced command there failsbash -n, when the body cites a SHA that is not an ancestor of the head, and when a commit in base..head has no Conventional Commit subject (type(scope)!: subject; merge commits and squash merges ending in(#n)are exempt). Write commits with the/commitskill.scripts/pr-body.shruns the same checker with the PR's base and head before it sets a title and body or marks a draft ready. Its unit tests run in the same job. A second job runsscripts/check-pr-size.pyfor the size limit, reruns when a label is added or removed or the body is edited, and prints the ten largest counted files. It accepts an oversized PR only with thelarge-prlabel or aLands stack:first line that holds (see the size limit above). - CI, still to build. A check that fails a PR whose base branch is not
mainand has no open PR of its own, the sign of a stack pushed without Graphite. - Install check.
just install --checkexits 1 whengtis missing, the same as for Go.
Rollout¶
- Scaffold: module, lint, comment checker and its agent hook, CI,
platform/*,errs,testkit, generators, emptyflows.tsvwithmonacoctl flows checkgreen,verify-backendskill with an empty feature map, mkdocs, CHANGELOG. The legacy backend, its migrations, its Go domain package and the reference trading bot are deleted first, as the step's opening stack. No features. Prove lint fails on a planted violation of each rule, and measure the test budget table on the scaffold to set the CI gate. - Events + relay +
bus.Dispatchwith the bus test suite green on embedded NATS. - Identity, cabal, funding (deposits), treasury (fund, shares). E2E flows 1–7.
- Governance + trading. Flows 8–13, with crash-point tests. Flow 8 lands here because the pause it writes is what trading checks.
- Cash out, withdraw, agents. Flows 14–17.
- Market, ranking, social, notify, referrals, admin.
- Cut iOS over in one release behind the generated client. The old backend was already deleted in step 1 (M7), so the cutover deletes nothing. The new backend starts on an empty database. Nothing is backfilled from the old backend and nothing syncs between them. A returning user signs in and gets their existing Privy wallet back; every other row starts fresh. Funds left in the old treasuries are test funds only. They are wiped at cutover: swept to an ops wallet or written off. Members are not cashed out.
Alternatives considered¶
| Alternative | Why not |
|---|---|
| Rust (axum + async-nats) | Network-bound workload; slower iteration and agent throughput; no embedded NATS server for tests. |
| Microservice per module | Distributed transactions across the state-change-plus-event write; N deploys and N outboxes for one team. |
| Full Uncle-Bob layering (entities, interactors, presenters, gateways) | One-implementation interfaces at every seam; call path spans 5+ files. |
| DI framework (fx, wire) | Hides wiring. main.go by hand is one readable file at this size. |
| GORM / ent | Hides SQL on money paths; sqlc keeps queries reviewable and types derived from schema. |
pkg/ for shared code |
Nothing outside the module imports it; internal/ is enforced by the compiler. |
| Coverage floors per layer (under 100%) | Earlier draft. Replaced by 100% merged coverage plus mutation testing, which blocks mock-assertion tests that only chase the number. |
Decided¶
- Module path is
github.com/<org>/monaco/apps/backend.platform/stays inside it. The reference trading bot was deleted with the legacy backend in Rollout step 1, so nothing outside the module needsplatform/. - Migrations use atlas, versioned SQL files under
migrations/namedYYYYMMDDHHMMSS_name.sql,atlas migrate lintin CI (a rebase that brings in another branch's migration regeneratesatlas.sumwithatlas migrate hash),atlas migrate applyin the deploy's pre-deploy step. The pinned build is atlas community (apps/backend/.atlas-version), because from v0.38 the official build requires a login formigrate lint.scripts/install-atlas.sh(run byjust install) puts it at the repo's gitignored.bin/atlas.monacoctl migrate apply|status|lintfinds theapps/backendmodule from the working directory or its own executable, runs that exact path there, never theatlasfirst onPATH, and refuses to run whenatlas versionis not the pinned community build.just migrate dbruns it against.env.local. Nothing migrates at boot; a binary that finds a schema behind its expectation fails boot withdb_schema_behindand logshave,wantand the hintrun: just migrate db. cabaleverywhere: Go types, tables, event subjects, and the new HTTP routes (/v1/cabals/{id}). The old rule "API routes and types staygroups" was for the backend being replaced; it ends when the iOS app cuts over to the generated client (Rollout step 7), which renames the routes on both sides in the same release. The legacy/v1/groupsroutes went with the old backend in M7.- The legacy backend is deleted in M7, at the start of Rollout step 1, not at the iOS cutover. Every later ticket builds on an empty module instead of working around the old code. Accepted consequence: the iOS app has no working backend, local or deployed, until the domain milestone rebuilds its routes; mobile UI work uses sample data or the fakes server.
- NATS is Synadia Cloud, free plan first. See NATS hosting and budget.
- JSON is snake_case. API request and response fields and event payload fields are snake_case. Go and Swift identifiers follow their own language's conventions; the generated clients map between them.
Open questions¶
None at the moment.
Log¶
- 2026-09-29: verify runs in stage 2 only. Evidence is a CI artifact, not a committed file. The pre-PR evidence hook and the
justrecipe for verify were dropped (#483). - 2026-09-29: Settled the flows table and names with the MVP tickets (#535). Flow 19 runs every 2 minutes. Flow 18: the market price poller appends
asset.price_moved; there is no second poller. Flows 11 and 12: the swap layer appendstrade.submittedon submit. Flow 12 route isPOST /v1/swaps/{id}/retryand flow 14 route isPOST /v1/cabals/{id}/cashouts(default; see #535). Flow 4: leave requires zero shares, and the app routes to cash out first.notifyappears in a consumers cell only when the event pushes, so it left flows 3, 7, 13 to 16, 18, 25 and 26. Flow 8 gainscabal.pausedandcabal.resumed, appended by funding. Decided adds snake_case JSON and regeneratingatlas.sumon rebase. - 2026-09-27: Decided: the legacy backend, its migrations, the Go domain package and the reference bot are deleted in M7 (#457), before the scaffold. Rollout step 7 deletes nothing.
- 2026-09-27: Decided 2026-09-27: every user picks a unique
users.handlein onboarding, owned by identity and read by referrals through identity's query port. The market price poller ticks every 120 s (flow 18, NATS network budget). - 2026-09-27: Decided:
users,followsandcabal_messagessoft delete throughdeleted_at, with a unique partial index for re-follows; nofollow_countstable, counts are an indexedcount(*); noreferral_unlockstable,users.first_deposit_atset by identity as a new flow 5 consumer that replaces referrals there, and read by referrals through identity's query port;chat_seenstays in social and is hard-deleted on leave; old treasury funds are test-only and wiped at cutover (Rollout step 7). Addedusers,follows,cabal_messagesandchat_seento the table-ownership table. - 2026-09-27: Decided: sign-in is Apple or Google, SMS OTP in dev builds only; cutover starts on an empty database with no backfill or sync, and sign-in reuses existing Privy wallets. Defaults:
money.SignedMicros; table ownership (trading ownsswaps, treasury writes ledgers as a consumer, funding owns pause, split wallet tables, oneprice_pointstable, admindead_letters); ledger checked byreplay --verify, not rebuilt; relay in api and worker; per-handlerevent_deliverieswith 30-day retention; SSEglobalkey;price.tick,proposal.executed(governance also consumes the flow 16 agent events),asset.price_moved,referral.attributed,user.nudge_dueand the analytics module in the flows;after-createcrash point, where acreatedrow was never signed or sent because the move tosubmittedcommits with the signed bytes before any send, so the sweeper fails it after 2 min; flow 1 referrals consumer only mints the code; no trade safety-net poller; agent budget rechecked at execution; payloadvfield; Ably fake; flow 8 in step 4. - 2026-09-27: CI scheduling moved to ci.md: PR CI only on ready PRs based on
main, one aggregate required check, Linux-first, runner options. - 2026-09-27: Logs as evidence: registered message names, join keys on every line, decisions and inaction logged, before/after on money lines, terminal line from
uow.Do, replay and seeded scenarios. - 2026-09-27: Errors rewritten around one
errs.Errortype and a code table; boundary-only logging; outcomes column. Testing budget made a gate with a cost table; test isolation by database with the lints that enforce it.flows.tsvreplaces the hand table as the source of truth. Deploy on Render with OTel to Grafana. Open questions closed: module path, atlas,cabaleverywhere. - 2026-09-27: NATS hosting decided: Synadia Cloud, free plan first. Budget per limit, two streams, one connection per process, SSE hub with one wildcard subscription,
Nats-Msg-Idon every publish. - 2026-09-27: No comments in hand-written Go, only machine-read ones (
//go:directives, generated files). Replaces the doc-comment allowance,godox,nolintlintand inline//nolint; adds an agent-time hook. - 2026-09-26: Testing expanded: 100% merged coverage, mutation, model-based, fuzz, jitter via synctest, chaos dispatcher, template-DB Postgres.
- 2026-09-26: Proposed. Go, modular monolith, three-ring Clean Architecture enforced by depguard, generated contracts, lint config, flows table.