2:14 a.m. — invoice routing hits a downstream outage
It is 2:14 a.m. and the on-call ops manager is already awake, staring at a PagerDuty page from the invoice routing workflow. The run that was supposed to push last night’s invoices into NetSuite is failing at the last hop: the billing API keeps returning 502s. Three thousand invoices are piling up in the queue, the customer success team will notice by 9 a.m., and flipping the workflow back on by hand takes twenty minutes of careful clicking. This is exactly what self-recovery rules were written to prevent.
What "self-recovery" means on FlowForge
Every template on FlowForge ships with a recovery_rules list — an ordered set of match+action pairs that the runtime checks against a failed node before it surfaces the failure to a human. If a rule’s match condition is true (a specific HTTP status, a specific exception class, an SLA breach on the downstream), the runtime runs the rule’s action in place: retry with exponential backoff, fall back to a cached response, enqueue the run into a manual approval tier, or page a named Slack channel instead of an operator. The full source of truth lives on each template row in db/templates.js, and the executor lives in server.js — both worth a read if you want to audit the path yourself.
A recovery rule in practice
Here is a concrete rule for the invoice routing template. It is the exact match+action pair that ships by default; in our example render we use a ### Example subhead followed by one bullet per field, each value wrapped in inline backticks because the blog renderer (see lib/markdown.js) does not support fenced code blocks.
Example
- match:
downstream_status == 502 - backoff:
exponential, base 500ms, max 3 attempts - fallback:
enqueue into /start/operations/manual-approval - notify:
#ops-alerts
Read together, this says: when the downstream returns a bad gateway, retry it three times with progressively longer waits, then drop the run into the manual approval tier, then post a single Slack summary into #ops-alerts instead of paging anyone. The same match+retry+fallback+notify shape is the default on the lead enrichment starter and the customer support kits — copy a rule (or the whole workflow) the same way.
Closing the loop
Without that rule, last night’s invoice run would have sat in the queue, the on-call would have had to flip a switch manually at 2:14 a.m., and the customer success team would have noticed the missing invoices before anyone did. With the rule in place, the workflow retried three times, fell through into the manual approval tier, and the operator woke up at a normal hour to a single Slack thread — not a PagerDuty page. That is the entire point of self-recovery: the workflow fixes itself, and the human only looks at the run if it is still stuck after the rules have all tried.
Wire it into your stack
- Open the operations hub to see the three starter kits (lead enrichment, invoice routing, customer support) that ship with recovery rules preconfigured.
- Browse the full template gallery to copy a rule — or the whole workflow — into your own pipeline.