Relay helps ops teams run AI agents for refunds, invoices and ticket triage. The brief from the (fictional) client: make agents visible, put a human in the loop exactly where it matters, and make every decision auditable.
Follow one refund through the whole case: €480 for Ava’s broken lamp. It starts as a live demo on the website, waits in the web app’s Approvals, and gets approved on Omar’s phone.
Trust, not capability, stops ops leads from letting agents touch money or customers.
How I’d validate: Interview 6 ops and finance leads: what would they need to see before an agent issues a refund alone?
Investigating one bad agent action takes days, because logs are written for engineers.
How I’d validate: Time a real investigation end to end in 2 teams, from complaint to root cause.
Tracing tools such as LangSmith and Langfuse serve the engineer who builds the agent, not the ops lead who signs off on a refund.
How I’d validate: Ask ops leads to answer “why did it do that?” in an existing trace view.
No real client, no users to interview, no live data and no engineers. So the flows rest on desk research, public benchmarks and agent-ops patterns, and every number on this page is an assumption or a target, never a result. Research plan with real access: 6 interviews with ops and finance leads, shadow 2 approval queues for a day, and review 50 past refund tickets for the cases agents get wrong.
About 7 weeks, split into five phases. This is how I would plan the work as a real engagement, not a record of time spent.
“I’ll let agents work as long as I can stop them.”
“I approve payments from airports and taxis.”
Personas are assumptions built from desk research, not interview subjects.
Desk research: a teardown of four public products that ops and engineering teams already trust. What I kept and what I refused from each:
Steps as plain-language blocks anyone can read.
Status colours that mean the same thing everywhere.
Calm, quiet UI so the one urgent thing stands out.
Full traces of every model call and tool call.
Mission control in four views: Live (a flow map of the selected agent and a feed of everything happening), Approvals, Replay and Builder. The icon rail on the left is the main menu. Try it below.
Approvals can’t wait for someone to open a laptop. The phone app is built around one job for Omar: decide quickly and safely, wherever he is.
Every request shows who owns it and when it moves to the backup approver, so nothing waits in silence.
Agent ops is mostly about failure, so I designed the bad day on purpose. This is run Refund #40244, which you can scrub through in the web app’s Replay view.
Ava asks to swap one lamp. The agent reads it as a refund, with high confidence, and refunds €190. It is under the €200 limit, so no human is asked.
Ava replies. The agent tries to refund the second lamp too. A rule outside the model blocks it: two refunds on one order within an hour.
The refund agent pauses itself and pings Nadia, the owner, with the run attached. If she doesn’t answer in 30 minutes, it goes to Omar.
Refunds under the limit are held for 10 minutes before they reach Stripe. Nadia undoes the held refund in one click and sends a swap instead.
From the run, she adds a rule: messages that mention “swap” go to a human. Replay keeps the whole story for the audit.
The site proves the promise instead of describing it: the hero is a live agent run that stops and asks the visitor to approve. The rest scrolls like a story, with a sticky panel that changes with each control. Logos, prices and figures on it are labelled as placeholders.
Ops leads think in workflows. The live view draws each agent as a flow with the active step glowing, so you can tell where something is stuck at a glance.
Every approval leads with evidence: what the agent checked, the policy it matched and what the team decided in similar cases. Confidence shows as a band (High, Medium, Low), not a raw percentage. Below Medium, the agent always asks.
Audits and debugging both start with “what happened?”. A scrubber with Saw, Thought and Did turns a run into a story anyone can follow.
A single tap is too easy for a €480 decision on a phone. Holding for a second adds just enough friction without a second screen.
Each request names one owner, a backup and a time limit. At 30 minutes it moves to the backup, and payments over €1,000 need two approvers.
After 50 approvals in a row with no edits, Relay suggests raising the refund limit from €200 to €300. A person always confirms, and one click rolls it back.
Holding a button is hard with some motor impairments, screen readers and switch control. “Can’t hold? Approve in two taps” sits right under it, and the second tap changes colour and wording so it can’t be a slip.
Every traffic light has a word next to it: Running, Needs approval, Problem, Paused. The four colours also differ in lightness, not just hue.
Coral (#FF6B4A) is under 3:1 against white, so small coral text uses Deep coral or Ink. Known gap: white labels on coral buttons are 2.9:1. v2 moves those labels to Ink.
The flowing edges and pulses stop when the system asks for reduced motion. The live feed doesn’t announce every event; toasts use a polite status region.
Every action on the phone is at least 44 px tall; approve is 62 px, full width, at thumb height.
Calm mission control: warm paper instead of black screens, one coral accent for anything that needs a human, and traffic-light status colours used the same way everywhere. Each colour has one job.
These are targets I’d set with the team for a 4-week pilot with 3 ops teams, not results. No pilot was run. Speed metrics are paired with safety metrics, so faster approvals can’t hide rubber-stamping.
One agent passing work to another needs a graph view and new failure modes. No ops lead in the brief asked for it yet.
Matters to the engineer paying the bill, not to the ops lead’s daily job. v2 adds monthly spend per agent.
v1 Builder saves edits as a draft and runs one test ticket. Proper evals before publishing come next.
Replay answers the auditor’s question in v1. Export of who approved what, when and from which device is first in line for v2.
The fix for approval fatigue, but only worth building once we see real approval volumes.
I drew the happy path first and the failure story later. Designing the bad day first would have shaped the approval card sooner: the undo window and the owner-and-deadline rule changed it more than any visual decision.
Whether approvers read the evidence or approve on the headline alone. If they skip it, the strongest piece of evidence has to move into the headline.
Approval fatigue. A team that approves dozens of requests a day starts rubber-stamping, which is why “approvals later reversed” sits next to “time to approve” in the targets.