We did not train a model. We built everything around one API call.

Four layers, and we only touch the top one.

People hear "AI product" and reach for "are you training it?" The useful map has four layers, and cost and difficulty fall away sharply as you move up it while reversibility improves. We operate entirely at the top.

flowchart TB
  A["<b>Pretraining</b><br/>build the base model<br/>months, millions of dollars"] --> B["<b>Fine-tuning</b><br/>update weights on your examples<br/>changes behaviour and format, not facts"]
  B --> C["<b>Retrieval</b><br/>fetch relevant material at question time<br/>and put it in the prompt"]
  C --> D["<b>Context engineering</b><br/>system prompt, precomputed values,<br/>schema, validation of what comes back"]
  D --> E{{"Ask P&P AI lives here,<br/>in layers 3 and 4"}}
  style E fill:#0f6058,stroke:#0f6058,color:#ffffff
  style C stroke-width:3px
  style D stroke-width:3px

All of layers 2 through 4 happen at inference time, the moment the model generates a response. So "it is an inference-time system" is true, but it is about as specific as answering "what does your car do?" with "combustion". The layer that matters is retrieval, and ours is unusual enough to be worth describing carefully.

Most questions never reach the model.

A question arrives at a Supabase edge function. Before any tokens are spent it passes three gates, each of which can answer or refuse without spending a token. Only what survives all three gets a grounding block assembled for it.

flowchart TD
  Q([User question]) --> AUTH[Verify JWT<br/>anon client, auth.getUser]
  AUTH -->|invalid| X1([unauthenticated])
  AUTH --> G1{Gate 1<br/>confidently off topic?<br/>deny-regex AND NOT on-topic-regex}
  G1 -->|yes| X2([offtopic, 0 tokens])
  G1 -->|no| G2{Gate 2<br/>card-acquisition intent<br/>with missing profile data?}
  G2 -->|yes| X3([deterministic 'collecting' card<br/>with chips, 0 tokens])
  G2 -->|no| RES[Reserve daily slot<br/>and budget, atomically]
  RES -->|cap hit| X4([daily_cap / monthly_budget])
  RES --> BUILD[Build grounding block<br/>see diagram 3]
  BUILD --> CALL[["Single completion call<br/>JSON mode, no tools"]]
  CALL --> V{Gate 3<br/>capability violation<br/>in the answer body?}
  V -->|yes| RETRY[[One corrective retry<br/>with extra instruction]]
  RETRY --> V2{still violating?}
  V2 -->|yes| FB([hardcoded fallback answer])
  V2 -->|no| OK
  V -->|no| OK[Server-side validation<br/>walletUsed, chips, schema]
  OK --> P[(Persist turn as JSONB<br/>after ownership check)]
  P --> R([Structured answer])
  style CALL stroke-width:3px
  style RETRY stroke-dasharray: 5 5

Gate 1 is worth a second look, because a naive deny-list breaks the product. "Write me a script to track my Amex points" matches the off-topic pattern on write me a script. It is rescued by the on-topic vocabulary check hitting amex and points, and falls through to the model, which classifies it in a verdict field instead. The regex is only allowed to refuse when it is confident and unopposed.

It is RAG, but say that word and you will be misunderstood.

Say "RAG" to an engineer and they picture embeddings, a vector store, and a nearest-neighbour lookup over chunked documents. We have none of that. There is no pgvector, no embedding call, no similarity search anywhere in the system.

What we have instead is retrieval driven by who is asking. What they typed barely enters into it. The knowledge base is a single JSON file published by the site itself, fetched over HTTPS and held in memory with a fifteen-minute TTL, so the facts the assistant reasons over can never silently drift from the facts the website displays. The user's holdings are one JSONB read. The filtering between them is ordinary TypeScript.

flowchart LR
  subgraph SRC[Sources]
    D[("data.json<br/>published by the site<br/>15 min in-memory cache<br/>fails open to last good copy")]
    P[("profiles.data JSONB<br/>balances, cards, spend,<br/>interview answers")]
  end
  subgraph FILTER[Deterministic scoping, in TypeScript]
    F1["Transfer partners<br/>scoped to currencies held<br/>capped at 25 each"]
    F2["Program notes<br/>transitive closure from<br/>held currencies"]
    F3["Named cards<br/>token overlap on the question<br/>needs 2 distinctive tokens"]
    F4["Spend math<br/>computed, not retrieved"]
  end
  D --> F1 & F2 & F3
  P --> F1 & F2 & F4
  F1 & F2 & F3 & F4 --> ORD["Concatenate,<br/>most critical first"]
  ORD --> CUT["slice 0, 12000"]
  CUT --> VF["==== VERIFIED FACTS ====<br/>…<br/>==== END VERIFIED FACTS ===="]
  VF --> SP[System prompt]
  style VF stroke-width:3px
  style CUT stroke-dasharray: 4 4

The only part of retrieval that reads the question at all is card-name matching, and it is deliberately strict: a card qualifies on two distinctive token hits and at least half of its own name, stop words removed, three cards maximum. A loose match would ground an answer in the wrong card's annual fee, which is a worse failure than not answering.

The right label for this is deterministic retrieval, or grounding if you prefer the plainer word. It is a deliberate choice, and I would make it again. Our corpus is a few hundred structured facts with hard relationships, not a pile of prose, and semantic similarity would be a lossy way to ask a question that a join answers exactly.

The model is not allowed to do the maths.

Every dollar figure in an answer is computed in TypeScript before the prompt is assembled, then handed to the model with instructions to quote it verbatim.

Precomputed annual rewards math on this spend
(already calculated for you, cite these dollar
figures directly, do NOT recompute them)

Multiplier times valuation, annual cap normalisation across quarterly and monthly and cycle-based caps, overflow spilling to the card's base rate, best card per category, net annual value of cards the user does not yet hold. All of it runs in code. The model's job is to select, sequence, and explain, which is what it is good at. It is not asked to be a calculator, which is what it is worst at.

This is the most useful thing to say to a sceptical engineer. You do not stop a model hallucinating numbers by asking it nicely. You stop it by not giving it the job.

Prompt rules drift. Mechanisms hold.

That sentence is written in the source, and it is the design principle the rest of this section is an argument for. Anything the product promises on screen is enforced by code that runs after the model, not by a line in the system prompt asking the model to behave.

The gap object deserves a mention on its own. When the assistant declines because a card attribute is missing from our data, it emits a structured record of exactly what it wanted and did not have. Those get logged. The refusals are a roadmap for what to go and source next, which turns the least satisfying answers in the product into its most useful telemetry.

Absent is not zero

The invariant repeated most often in the source is that a missing spend category is unanswered, never $0. A model reading an absent field as a zero produces an answer that is confident, well formatted, and wrong, which is the worst of the three available failure modes.

Reserve before, release after.

Per-user and global limits are reserved atomically in Postgres before the model call, then released on every failure path. That ordering is what makes it a real rate limit instead of an after-the-fact counter that a burst of concurrent requests can sail straight past.

sequenceDiagram
  autonumber
  participant C as Client
  participant F as Edge function
  participant DB as Postgres RPCs
  participant M as Model API
  C->>F: question + chat_id
  F->>DB: ai_usage_reserve (per user, per UTC day)
  DB-->>F: ok / cap reached
  F->>DB: ai_budget_reserve (global, sized per request)
  DB-->>F: ok / budget reached
  Note over F,DB: reservation is estimated from<br/>prompt length, not a flat constant
  F->>M: single completion, JSON mode
  alt model answers
    M-->>F: JSON answer
    F->>DB: ai_usage_record_tokens (fire and forget)
    F-->>C: structured answer
  else timeout, error, or off-topic verdict
    M--xF: no usable answer
    F->>DB: ai_usage_release + ai_budget_release
    Note over F,DB: releases the exact amount reserved
    F-->>C: typed error code, UI offers handoff
  end

When a limit is genuinely reached, the failure is designed, not swallowed. Each error path returns a typed code that the interface maps to a handoff: it builds a prompt containing your wallet, your question, and a source line pointing back at the site, and offers to open it in whichever assistant you already use. Running out of our budget hands you a working alternative instead of a spinner.

The short version, for skimming.

One line, if you only get one: a single-shot, non-streaming completion with no tools and no vector retrieval, where correctness rests on deterministic scoping before the call and mechanical validation after it.

The things we chose not to build.

Roughly as informative as the architecture itself, and usually the first thing an engineer asks about.

• A vector database. A few hundred structured facts with hard relationships. Similarity search would answer approximately what a lookup answers exactly.
• Tool use and browsing. Tempting for live award availability, and genuinely useful eventually. Today it would let the assistant promise a lookup that the rest of the system cannot verify, and an unverifiable answer is worse than a bounded one.
• Streaming. Answers are structured JSON that the interface renders as cards, so there is no partial state worth showing. Streaming would buy perceived latency at the cost of the validation step that runs on the complete object.
• A second provider for failover. Real resilience value, real complexity cost, and the typed-error handoff already gives a stuck user somewhere to go.
• Fine-tuning. It changes behaviour and format. Our problem was never format. It was facts, and fine-tuning is a poor way to install those.

I would rather defend a small system that fails in ways I can predict than a clever one that fails in ways I cannot. Most of the decisions above are that trade made repeatedly.