Cowliss is in beta: features and pricing may change.

Testing automations

An automation with a 2-day wait should be testable in seconds, and checkable against real state without emailing anyone. There are three tools, and they answer different questions:

cow test: does the logic do what I meant? Local, instant, nothing real involved.
Dry run: would this one real user get this email, and if not, why not?
A second organization: does the whole thing work from my own app, against data nobody will miss?

`cow test`: does the logic do what I meant?

cow test                     # every scenario in scenarios/
cow test abandoned-checkout  # that automation's scenarios
cow test abandoned-checkout --scenario scenarios/abandoned-checkout.timeout.json

A scenario is named scenarios/<automation>.<case>.json, and the name up to the first dot is the automation it runs. The package.json that cow init writes makes npm test run cow test, so it runs them all.

Local: no API call, nothing sent, no login needed, and no build. It reads the automation’s definition from automations/<key>.json and walks it with the same interpreter the platform runs, against a scripted scenario on a virtual clock. Sleeps complete instantly, waitForEvent resolves the moment the scenario’s next matching event is due, and a 30-day automation finishes in milliseconds. Capability calls are recorded and answered with canned results, so nothing is written and no email exists.

A scenario scripts four things: the user, the trigger’s properties, later events at offsets from the start, and the calls the run must make, in order.

{
  "user": {
    "id": "usr_1",
    "traits": { "email": "[email protected]", "firstName": "Ada" }
  },
  "trigger": {
    "properties": { "checkoutUrl": "https://shop.acme-demo.io/checkout/42" }
  },
  "events": [],
  "expect": [
    {
      "activity": "sendEmail",
      "input": {
        "template": "abandoned-checkout",
        "props": {
          "firstName": "Ada",
          "checkoutUrl": "https://shop.acme-demo.io/checkout/42"
        }
      }
    }
  ]
}

{
  "user": { "id": "usr_1", "traits": { "email": "[email protected]" } },
  "events": [{ "at": "1h", "event": "purchase_completed" }],
  "expect": []
}

user is what a rule reads as profile: its traits, and the segments it is in (a list of segment names, empty unless you give one), so an automation that branches on membership can be tested both ways.
The trigger event is delivered automatically at the start, exactly as production does. trigger.properties is what a rule reads as event.properties, and it is empty unless you give it. So an automation that sends a checkout link, or branches on a segment entry’s segmentName, gets the value you mean only when the scenario names it.
expect names the calls in order, using the activity names sendEmail, sendWebhook, and restart, and an empty list asserts the run contacts nobody. input matches as a subset of the actual call, so assert the fields you care about and ignore the rest. A sendWebhook call’s input is { destination, payload }, where destination is the step’s webhook.

Without --scenario, cow test prints one line per scenario, passed or failed with its mismatches, then how many passed; --json puts every run’s full report on stdout and moves the lines to stderr. With --scenario it prints that one run’s report: the calls the run made, any mismatches, the events the scenario never delivered, the virtual time the run covered (simulatedMs), and the wall time it took (wallMs). Any failure exits non-zero, so cow test belongs in CI beside your unit tests. A project with automations and no scenarios fails too, while a project with no automations at all (templates only) has nothing to run: cow test says so and passes.

Every gallery blueprint ships with its scenarios; cow add blueprint <name> copies them into scenarios/ beside the automation.

Dry run: does it behave against real state?

cow test mocks the world; a dry run runs against it. It starts an execution for a real profile with sends disabled:

cow automations dryRun abandoned-checkout --profile-id usr_…

Every send gate is evaluated against real profile, consent, suppression, credit, and webhook state.
Instead of sending, the delivery log records would_send or a typed would_skip_* entry with the full payload, so “would this user actually get this email, and why not” is answerable from the log. A would_send also carries the rendered message under payload.rendered.
A dry run is terminal by construction: it costs no quota, generates no provider feedback, and is excluded from the automation’s execution counts.

A dry run gets its own workflow id (orgId:automation~dry-run:profileId), deliberately: on the shared id, a dry run in flight would swallow the profile’s real trigger and receive their live events, so developing an automation would silently cancel a customer’s onboarding. The cost is the mirror image: a dry-run execution receives no live events, so its waitForEvent calls run to their timeouts. Script timing with cow test; check state with a dry run.

A second organization: does the whole loop work?

An organization holds one set of data, so run your dev box against a second organization: create it, push the same repo to it, and point your sourceId and API key there. identify and track from your app then run the automation for real, live events included, and every send, webhook and charge belongs to that organization instead of the one your customers are in. Throw the organization away when you are done. To watch a send without mail leaving at all, use a dry run.

Testing automations

cow test: does the logic do what I meant?

Dry run: does it behave against real state?

A second organization: does the whole loop work?

`cow test`: does the logic do what I meant?