Set up webhook notifications
Send CosmicAC job and model health events to an HTTPS endpoint, including a Slack webhook.
Send CosmicAC events to an HTTPS endpoint that you control. CosmicAC posts a payload when a job fails, degrades, or recovers, and when a served model's health changes. One webhook covers the whole deployment.
Prerequisites
Before you start, make sure that you have the following.
- A running CosmicAC deployment. See Set up CosmicAC.
- The platform administrator role. See Teams and roles.
- An HTTPS endpoint that accepts POST requests. For Slack, create an incoming webhook, and then copy its URL.
Steps
Open the Notifications page
In the left navigation, click Settings, and then under Instance, click Notifications.
Save the webhook URL
In Webhook URL, enter the HTTPS URL that receives the events, and then click Save. CosmicAC sends nothing until you save a URL.
The first save also generates the signing secret. Copy it now, because CosmicAC shows it only once and your receiver needs it to verify signatures.
Select the payload format
Under Format, select one of the following formats.
- Generic JSON: a flat event object carrying id, type, schema, timestamp, instance, and the fields for that event type.
- Slack-compatible: a Slack incoming webhook body carrying text and a color-coded attachments block. Red marks a failure, green a recovery, and orange a degraded state.
Payload Preview shows a sample payload in the selected format.
Select Slack-compatible when the URL points to Slack. CosmicAC posts the body straight to the incoming webhook URL, so it needs no Slack workspace credentials, bot token, or OAuth 2.0 authorization.
Select the events to send
Under Events, turn on each event that you want CosmicAC to send. Delivery is opt-in, so CosmicAC never sends an event you leave off.
The four job lifecycle events also need the matching preference on the job that raises them. See What controls delivery.
| Event | Fires when |
|---|---|
| job.failed | A job transitions to Failed, and the event carries the failure reason. |
| job.degraded | Healthy replicas drop below the count you set, and the endpoint stays live. |
| job.recovered | A job returns to Active from Degraded or Failed. |
| job.restart_storm | Any replica restarts three times within 10 minutes. |
| model.health.down | A served model's health enters Down. |
| model.health.recovered | A served model's health leaves Down. |
Click Save to apply the selection.
Send a test event
Click Send test event. CosmicAC posts a test.ping payload to the saved URL.
Confirm the delivery
Check Recent Deliveries. Each row lists the event type, the target URL, the response status, and how long ago CosmicAC sent it. A successful delivery shows the response code, such as 200 OK.
A delivery counts as failed when the receiver answers with a status outside the 2xx range, or doesn't answer within 10 seconds. CosmicAC retries a delivery that fails with a 5xx status, a 429 status, a network error, or a timeout. It retries up to three times, after 1, 2, and 4 seconds. It doesn't retry any other status, and it discards the event after the last attempt.
Verify the signature on your receiver
Verify signatures when your receiver is a service you run. A Slack incoming webhook can't run verification code, so it ignores the signature header.
Your receiver needs the signing secret to confirm that a payload came from CosmicAC. CosmicAC shows the secret once, when you first save the webhook URL. If you didn't copy it then, click Regenerate to issue a new one.
CosmicAC sends the following headers.
X-Cosmic-Event: the event type, such as job.failed.X-Cosmic-Delivery: the unique event identifier. Use it to discard duplicates.X-Cosmic-Timestamp: the send time in milliseconds.X-Cosmic-Attempt: which attempt this delivery is, starting at 1.X-Cosmic-Signature:sha256=followed by the HMAC-SHA256 digest of the raw request body, keyed with the signing secret. CosmicAC sends this header, and the identicalX-Cosmic-Signature-256, only when a signing secret exists.
Recompute the digest over the raw body and compare it before you trust a payload.
const crypto = require('crypto')
const expected =
'sha256=' +
crypto
.createHmac('sha256', process.env.COSMICAC_WEBHOOK_SECRET)
.update(rawBody)
.digest('hex')
const signature = req.headers['x-cosmic-signature']
const valid =
typeof signature === 'string' &&
expected.length === signature.length &&
crypto.timingSafeEqual(
Buffer.from(expected),
Buffer.from(signature)
)What controls delivery
Every job has four notification preferences, one for each lifecycle event. CosmicAC turns on all four when you create the job. To set them, see Create a GPU Container Job, Create a vLLM Managed Inference Job with the CLI, and Create a Parakeet Managed Inference Job with the CLI.
An event reaches your webhook only if both of the following settings are on.
- The job's notification preference, which controls whether CosmicAC creates the event.
- The event under Events on the Settings > Notifications page, which controls whether CosmicAC posts it.
If either one is off, the event doesn't reach your webhook. If you turn it on later, CosmicAC doesn't deliver the events that it already skipped.
CosmicAC delivers each event that it creates to two destinations.
| Destination | Requirements |
|---|---|
| The notification bell in the web interface | None. CosmicAC shows every event it creates. |
| Your webhook | A saved webhook URL, and the event turned on under Events on the Settings > Notifications page. |
The Events selection applies only to the webhook. If you turn off an event under Events, it still appears in the notification bell.
The CLI and the web interface use different names for the same event. Pass the underscore form, such as job_failed, to cosmicac jobs create --notify. The web interface shows job.failed.
The model.health.down and model.health.recovered events have no per-job preference, because no job raises them. The Events selection is the only setting that controls them.
Rotate the signing secret
Click Regenerate to replace the signing secret. Copy the new secret before you leave the page, because CosmicAC shows it only once. The previous secret stops working immediately, so update your receiver in the same maintenance window.