Flow Scheduler

Built-in pipeline scheduler — runs in-process every minute. No external cron required. /api/v1/cron/flow-scheduler-tick remains for opt-in external orchestration.

The Flow Scheduler fires due pipelines from their ir.schedule. It runs in-process by default via a 60-second setInterval registered in instrumentation.ts on server startup, so kaireon ships complete out of the box — no external cron service required. The same dispatch logic is also exposed as the /api/v1/cron/flow-scheduler-tick HTTP endpoint for self-hosters who prefer their own orchestrator (k8s CronJob, EventBridge, Vercel Cron, etc.).

Multi-replica safety comes from a pg advisory lock — only one replica or HTTP caller’s tick runs at a time per minute, the rest no-op with {skipped: true}.

Configuration

Env var	Default	Purpose
CRON_SECRET	required	Shared secret for both the in-process ticker and the HTTP endpoint. The internal scheduler refuses to start if unset (logged warning)
FLOW_INTERNAL_SCHEDULER_ENABLED	true	Set to "false" to disable in-process ticking — useful when you bring your own external scheduler
FLOW_SCHEDULER_INTERVAL_MS	60000	Tick interval in milliseconds. Minimum 1000
FLOW_INTERNAL_SCHEDULER_BASE_URL	http://127.0.0.1:${PORT}	Where the in-process ticker fans out run-dispatch fetches
PORT	3000	Server port used to default the in-process scheduler base URL

Manual tick from the editor

The Schedule tab inside the pipeline editor has a Tick scheduler now button that hits the HTTP endpoint with the same-origin session cookie. Useful when you’ve just saved a schedule and want to fire it without waiting for the next minute boundary.

Endpoint

GET /api/v1/cron/flow-scheduler-tick
Authorization: Bearer $CRON_SECRET

(Or pass x-cron-secret: $CRON_SECRET header.)

Auth: Fails closed when CRON_SECRET env is unset (matches the contract of /api/v1/cron/approvals-expire and other cron routes).

Response:

{
  "tickAt": "2026-04-27T20:00:00.000Z",
  "totalPipelines": 47,
  "scheduled": 12,
  "due": 3,
  "dispatched": 3,
  "failed": 0,
  "results": [
    { "pipelineId": "...", "tenantId": "...", "fired": true, "status": 201 }
  ],
  "durationMs": 412
}

Per-kind decision logic

Implemented as a pure due-pipelines helper in the platform’s flow scheduling module — given the current set of schedules and the tick time, it returns the IDs that fire on this tick.

Kind	”Due” condition
cron	cron-parser yields any occurrence in (lastRunAt, now] (most recent wins)
interval	now - lastRunAt >= minutes * 60_000
rrule	RRule.between(lastRunAt, now) returns ≥ 1 occurrence

lastRunAt = null (pipeline never ran): treated as 24h lookback to avoid stampeding old, never-run pipelines that were created days/weeks ago.

Bad schedules (malformed cron, malformed rrule) silently skip the pipeline — parsePipelineIR would have rejected the IR save earlier anyway.

How it fires

Default (in-process): The scheduler is self-firing. A setInterval registered by Next.js’ instrumentation hook on server boot calls the same dispatch path every 60 seconds via the platform’s internal-ticker module. No external orchestrator required. App Runner and any persistent-container deployment Just Work.

Optional (external): If you set FLOW_INTERNAL_SCHEDULER_ENABLED=false (e.g. you’re on a serverless runtime with no persistent process), wire any external scheduler to the HTTP endpoint:

Setup	Approach
Self-hosted (k8s)	A Kubernetes cron job resource hitting /api/v1/cron/flow-scheduler-tick every minute
Self-hosted (Linux box)	* * * * * curl -fsSL -H "x-cron-secret: $CRON_SECRET" https://your-app/api/v1/cron/flow-scheduler-tick
AWS	EventBridge Schedule → API destination
Vercel	A Vercel Cron Job (vercel.json crons array) hitting the endpoint

Idempotency

• Calling the endpoint multiple times in one minute does not double-fire a pipeline — once a run is dispatched, lastRunAt advances to the fire time, and the next tick’s (lastRunAt, now] window excludes the same occurrence.
• Per-tick fanout is O(n) over all IR-native pipelines + 1 IR fetch each. Phase 6.6 hardening adds an index + a “schedule_only” projection.

File-arrival triggers (push + poll)

File-arrival triggers complement schedules. Use them when a pipeline should fire when a sentinel file shows up instead of on a clock — the canonical “I dropped customers.csv into the inbox, go process it” pattern.

IR shape

{
  "trigger": {
    "kind": "file_arrival",
    "sourceId": "src",
    "controlFilePattern": { "type": "glob", "value": "_ready.json" },
    "debounceSeconds": 60,
    "deadline": { "windowMinutes": 60, "onMiss": "alert" }
  }
}

• sourceId — the source node in the same IR that will consume the file when the run fires.
• controlFilePattern (optional) — the sentinel that fires the run. Per-pipeline. When omitted, the trigger uses the referenced source’s own pattern (so the data file’s arrival fires the run directly). See “Multiple pipelines, same folder” below for the multi-mask story.
• debounceSeconds (optional, default 60) — minimum time between consecutive fires. Prevents a slow run + a new file mid-run from double-firing.
• deadline.windowMinutes / onMiss (optional) — if no file arrives within windowMinutes of lastRunAt, take the onMiss action (alert, fail, or skip).

Multiple pipelines, same folder

controlFilePattern is per-pipeline. When several pipelines watch the same inbox, each declares its own sentinel mask and only fires when its specific sentinel arrives. Example: three pipelines sharing s3://kaireon-e2e-tests/inbox/:

// load-customers
"trigger": { "kind": "file_arrival", "sourceId": "src",
             "controlFilePattern": { "type": "glob", "value": "customers.done" } }

// load-accounts
"trigger": { "kind": "file_arrival", "sourceId": "src",
             "controlFilePattern": { "type": "glob", "value": "accounts.done" } }

// load-propositions
"trigger": { "kind": "file_arrival", "sourceId": "src",
             "controlFilePattern": { "type": "glob", "value": "propositions.done" } }

Drop customers.done → only load-customers fires. Drop both customers.done and accounts.done on the same tick → both fire, independently. The source executor reads each pipeline’s own data-file pattern (e.g. customers.csv, accounts.csv), so the data files don’t collide either.

Make the masks disjoint. Two pipelines whose controlFilePattern both match the same key (e.g. both globs include *.done) means whichever EventBridge dispatch arrives first wins — the poll path is fine because each pipeline probes independently, but the push resolver needs a single match to identify the target pipeline. Use distinct prefixes / suffixes (customers.*.done, accounts.*.done) when in doubt.

After a successful fire, the sentinel is archived to atomicity.successFolder (the same folder the source executor uses for data files), so the same trigger doesn’t re-fire on every subsequent tick. A failed run moves the sentinel to atomicity.failureFolder instead. Drop a fresh sentinel to retry.

How it fires (two paths)

Path	Latency	Setup
Poll (default)	up to one scheduler tick (≤ 60s)	Zero config — the in-process scheduler polls every active file_arrival trigger on each tick
Push (optional)	sub-second	Wire S3 → EventBridge → API destination (or any webhook source) to POST /api/v1/triggers/file-arrival

Both paths converge on the same dispatch entry — the run drawer will show triggerSource: "file_arrival".

Push endpoint

POST /api/v1/triggers/file-arrival
Authorization: Bearer $CRON_SECRET
Content-Type: application/json

Native payload:

{
  "pipelineId": "94640f58-9fcc-…",
  "tenantId": "4b148e40-…",
  "objectKey": "inbox/_ready.json"
}

S3 EventBridge envelope (forwarded verbatim from an EventBridge target):

{
  "detail": {
    "bucket": { "name": "kaireon-e2e-tests" },
    "object": { "key": "inbox/_ready.json" }
  }
}

When the EventBridge shape arrives, the endpoint scans all active pipelines for a matching (bucket, controlFilePattern) pair, probes the source path to confirm the key really matches, then dispatches the run.

Auth: the Authorization (or x-cron-secret) header must equal CRON_SECRET. Per-pipeline trigger secrets are on the roadmap.

Idempotency: dispatching the same file twice is safe — the source executor’s pendingFinalizers machinery moves the file to .archive/ or .failed/ after the first run, so the second run finds no matching files and no-ops via onMissAction.

Deadline enforcement

When the scheduler polls and finds no matching file but the trigger carries a deadline, it checks whether the deadline window has elapsed since lastRunAt (or pipeline createdAt for never-run pipelines):

onMiss	Behavior
alert	Emits a system-health warning (visible in the alerts dashboard).
fail	Same alert + writes a synthetic failed PipelineRun so health dashboards count the SLA miss.
skip	Logs and moves on. Useful in development.

Each (pipeline, anchorTime) pair fires the action at most once — the dedupe key is the pipeline’s lastRunAt (or createdAt), so a fresh fire that advances lastRunAt re-arms the deadline.

Roadmap

• Per-tenant scheduler dashboard showing miss / lag / failed dispatches.
• Per-pipeline trigger secrets for tenant-isolated push webhooks (today the file-arrival endpoint validates against CRON_SECRET).
• SFTP fanotify / inotify-driven local_fs triggers so on-prem deployments get the same sub-second push latency S3 → EventBridge delivers.