> ## Documentation Index
> Fetch the complete documentation index at: https://docs.engini.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Debug and replay a failed delivery

> Find the events that did not arrive, read why each attempt failed, bring the destination back, replay what was missed, and rotate a secret without losing events.

<Info>**Uses:** Python · TypeScript · CLI · REST API</Info>

Engini keeps every captured event for **30 days**, whether or not it was delivered. So when your receiver was down, rejected signatures, or the destination switched itself off, nothing is lost - you find what went wrong, fix it, and replay.

The sequence is **check the destination → find the events → read the attempts → fix → re-enable → replay**.

<Note>
  cURL samples assume `BASE=https://api.engini.io/v1` and `AUTH="x-api-key: $ENGINI_API_KEY"` - the setup from the [REST walkthrough](/examples/rest-walkthrough).
</Note>

## How an event ends up undelivered

An event's `delivery` is the state of its **latest** attempt:

| Delivery state | What happened | Will Engini try again by itself? |
| - | - | - |
| `pending` | Awaiting a send. Either a retry is scheduled after a retryable failure (`408`, `429`, `5xx`, timeout, connection error) - **or** the event was captured while the destination was disabled and has no attempts at all | A scheduled retry, yes (1 min / 5 min / 15 min / 1 h / 6 h). The disabled-destination backlog, yes too, once you re-enable - events from the last 24 hours go out within about a minute. Older than that, no - it needs a replay |
| `failed` | The destination was deleted while a retry was pending. Terminal | No - replay it, and the event goes to the trigger's current destination |
| `dead_lettered` | Out of retries, or refused with a non-retryable status (any other `4xx`, or a redirect) | No |
| `delivered` | Your receiver answered `2xx` | Nothing to do |

To tell the two kinds of `pending` apart, read the event (step 3): an empty `attempts` list is backlog, and a last attempt with `next_attempt_at` set is a scheduled retry.

Five dead-lettered events in a row auto-disable the destination (`max_consecutive_failures`, default 5).

## 1. Check the destination

Start here - if the destination has auto-disabled, nothing is being delivered at all, and `last_failure` usually names the cause.

<CodeGroup>
  ```python Python theme={null}
  d = client.triggers.destinations.get("prod")
  print(d.status, d.consecutive_failures, d.last_failure)
  # auto_disabled 5 HTTP 400: invalid signature
  ```

  ```typescript TypeScript theme={null}
  const d = await client.triggers.destinations.get("prod");
  console.log(d.status, d.consecutive_failures, d.last_failure);
  ```

  ```bash CLI theme={null}
  engini triggers destinations show prod --json | jq '{status, consecutive_failures, last_failure}'
  ```

  ```bash cURL theme={null}
  curl -s "$BASE/triggers/destinations/prod" -H "$AUTH" | jq '{status, consecutive_failures, last_failure}'
  ```
</CodeGroup>

## 2. Find the events that did not arrive

The event log comes back newest first. There is no server-side filter on delivery state, so filter on your side.

<CodeGroup>
  ```python Python theme={null}
  events = client.triggers.events.list(trigger_id)          # the full set
  stuck = [e for e in events if e.delivery != "delivered"]
  ```

  ```typescript TypeScript theme={null}
  const events = await client.triggers.events.list(triggerId, { top: 100 });
  const stuck = events.filter((e) => e.delivery !== "delivered");
  ```

  ```bash CLI theme={null}
  engini triggers events ti_DG9q7IlQO0MeghC4IEzP19 --limit 100 --json \
    | jq '[.[] | select(.delivery != "delivered") | {id, occurred_at, delivery}]'
  ```

  ```bash cURL theme={null}
  curl -s "$BASE/triggers/$TRIGGER_ID/events?top=100" -H "$AUTH" \
    | jq '[.items[] | select(.delivery != "delivered") | {id, occurred_at, delivery}]'
  ```
</CodeGroup>

## 3. Read why each attempt failed

`GET /v1/triggers/events/{eventId}` is the only place with the real attempt history: each attempt's status code, a trimmed error, how long it took, and when the next one is due.

<CodeGroup>
  ```python Python theme={null}
  detail = client.triggers.events.get(stuck[0].id)
  for a in detail.attempts:
      print(a.attempt, a.state, a.status_code, a.error, a.next_attempt_at)
  ```

  ```typescript TypeScript theme={null}
  const detail = await client.triggers.events.get(stuck[0].id!);
  for (const a of detail.attempts ?? []) {
    console.log(a.attempt, a.state, a.status_code, a.error, a.next_attempt_at);
  }
  ```

  ```bash CLI theme={null}
  engini triggers events get te_… --json | jq '.attempts[] | {attempt, state, status_code, error}'
  ```

  ```bash cURL theme={null}
  curl -s "$BASE/triggers/events/$EVENT_ID" -H "$AUTH" | jq '.attempts[] | {attempt, state, status_code, error}'
  ```
</CodeGroup>

| You see | Likely cause |
| - | - |
| `status_code: 400` / `401` on every attempt | Your receiver is rejecting the signature - wrong secret, a stripped `esec_` prefix, or verifying a re-serialised body |
| `status_code: 301` / `302` | The URL redirects. Redirects are not followed - point the destination at the final URL |
| `status_code: null`, `error` mentions a timeout | Your receiver took more than 10 seconds. Acknowledge first, work afterwards |
| `status_code: null`, connection error | DNS, TLS or a firewall. Check the URL is publicly reachable over HTTPS |
| `pending` with an empty `attempts` list | The event was captured while the destination was disabled |

## 4. Fix the receiver, then prove the fix

Deploy your fix, then send a test event through the real delivery path before replaying anything. It is signed with the destination's secret and tells you synchronously what your receiver answered. The test endpoint answers `409` while the destination is disabled, so re-enable it first (step 5) if you need to.

```bash theme={null}
curl -s -X POST "$BASE/triggers/$TRIGGER_ID/test" -H "$AUTH" | jq '{delivered, status_code, error}'
```

The test endpoint is API-only for now. See [Receive trigger events on a webhook](/examples/trigger-webhook#4-prove-the-wiring-with-a-test-event) for how to read the result.

## 5. Re-enable the destination

Setting `status` back to `active` also resets the failure counter to zero.

<CodeGroup>
  ```python Python theme={null}
  client.triggers.destinations.update("prod", status="active")
  ```

  ```typescript TypeScript theme={null}
  await client.triggers.destinations.update("prod", { status: "active" });
  ```

  ```bash CLI theme={null}
  engini triggers destinations enable prod
  ```

  ```bash cURL theme={null}
  curl -s -X PATCH "$BASE/triggers/destinations/prod" -H "$AUTH" \
    -H "Content-Type: application/json" -d '{"status": "active"}' | jq '{status, consecutive_failures}'
  ```
</CodeGroup>

<Note>
  **Re-enabling resumes most of the backlog on its own.** The backstop sweep picks up events from the last 24 hours - and destinations paused mid-retry - within about a minute, with no replay needed. New events are delivered normally from here on. Wait that minute, then re-list (step 2) before deciding what still needs a replay in step 6.
</Note>

## 6. Replay what was missed

A replay **queues** one more attempt, due straight away, and answers `202`. It does not deliver inline, so check the event again a few seconds later. The attempt numbering continues from where it left off, so a replay gets whatever retries the event had left - none, if it had already used all six.

Only two kinds of event still need a replay after re-enabling: `dead_lettered` events, and backlog events (`pending` with no attempts) older than 24 hours. Anything newer already went out on its own - replaying it too is harmless (your receiver dedupes on `X-Engini-Event-Id`), but there is no reason to.

<CodeGroup>
  ```python Python theme={null}
  from datetime import datetime, timedelta, timezone

  stale_cutoff = datetime.now(timezone.utc) - timedelta(hours=24)

  for e in stuck:
      backlog = e.delivery == "pending" and not client.triggers.events.get(e.id).attempts
      # occurred_at is UTC either way - make it timezone-aware if it came back naive
      occurred_at = e.occurred_at if e.occurred_at.tzinfo else e.occurred_at.replace(tzinfo=timezone.utc)
      stale_backlog = backlog and occurred_at < stale_cutoff
      if e.delivery == "dead_lettered" or stale_backlog:
          client.triggers.events.replay(e.id)
  ```

  ```typescript TypeScript theme={null}
  const staleCutoff = new Date(Date.now() - 24 * 60 * 60 * 1000);

  for (const e of stuck) {
    const backlog =
      e.delivery === "pending" && !(await client.triggers.events.get(e.id!)).attempts?.length;
    // occurred_at is UTC either way - add the "Z" if it came back with no offset
    const raw = e.occurred_at!;
    const occurredAt = new Date(/[zZ]|[+-]\d\d:\d\d$/.test(raw) ? raw : raw + "Z");
    const staleBacklog = backlog && occurredAt < staleCutoff;
    if (e.delivery === "dead_lettered" || staleBacklog) {
      await client.triggers.events.replay(e.id!);
    }
  }
  ```

  ```bash CLI theme={null}
  # dead-lettered events; add stale backlog ids (pending, no attempts, occurred_at
  # more than 24h ago) the same way - anything newer already went out on its own
  engini triggers events ti_DG9q7IlQO0MeghC4IEzP19 --limit 100 --json \
    | jq -r '.[] | select(.delivery == "dead_lettered") | .id' \
    | xargs -n1 engini triggers events replay
  ```

  ```bash cURL theme={null}
  curl -s -X POST "$BASE/triggers/events/$EVENT_ID/replay" -H "$AUTH" -o /dev/null -w "%{http_code}\n"   # 202
  ```
</CodeGroup>

* Skip events with a retry already scheduled. Replaying one as well sends it twice - your receiver's dedupe on `X-Engini-Event-Id` absorbs that, but there is no reason to.
* A replay answers `409` (`DESTINATION_DISABLED`) if the destination is still off.
* Your receiver sees the same `X-Engini-Event-Id` it may have seen before, so deduplication protects you if an earlier attempt did get through.

## Rotate a secret without losing events

Rotation has **no overlap window**: the old secret stops verifying the moment the new one is minted. Any delivery that lands in between is refused by your receiver - and because that refusal is a `4xx`, the event is dead-lettered rather than retried. So plan to replay.

<Steps>
  <Step title="Rotate">
    ```bash theme={null}
    engini triggers destinations rotate-secret prod --force --json | jq -r '.secret'
    ```

    Or `client.triggers.destinations.rotate_secret("prod")` / `rotateSecret("prod")`. The new `esec_…` is shown once.
  </Step>

  <Step title="Deploy the new secret to your receiver">
    As fast as your deploy allows - every event that arrives before this finishes will be dead-lettered.
  </Step>

  <Step title="Replay the gap">
    List the trigger's events, select those that are `dead_lettered` with an `occurred_at` after the rotation, and replay them (step 6).
  </Step>
</Steps>

<Tip>
  Keep the gap under five events and the destination stays enabled. With a busy trigger, a receiver that answers `503` (not `400`) on a signature mismatch *during a planned rotation* turns those refusals into retries instead of dead-letters - the first retry comes a minute later, by which time the new secret is live.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.