navree

Task scheduling

Cron-style schedules declared in modules: commands, jobs and functions, overlap prevention, single-node runs, schedule:work and schedule:run, all in UTC.

On this page

Introduction

Featureschedule Without a scheduler, every recurring task needs its own crontab entry on the server, outside your code and your version control. With ScheduleModule, modules declare their schedules in code, next to the work they schedule, and one long-running process (or one crontab entry) runs them all.

  • A task is a registered CLI command, a queued job or an async function.
  • Its frequency is a cron expression or a helper such as hourly or daily_at("08:00"). Times are UTC.
  • Every run is recorded in the database, which is how each run happens on one server only, however many servers run the scheduler, and how without_overlapping sees runs on other servers.

Registering the module

Add ScheduleModule once, in your app root. It has no options. It needs DatabaseModule: its schedule migration set creates the schedule_runs table.

src/app.rs

The module is global and collects every module's schedule at boot. The scheduler takes the time from the app's dyn Clock when one is registered, and from the system clock otherwise.

Defining schedules

Declare tasks in a module's register hook with m.schedule (the prelude brings it into scope). These are the CRM's two daily tasks, from crm-activities/src/lib.rs:

crm-activities/src/lib.rs

At 08:00 UTC, crm:task-reminders notifies everyone with open tasks due today. At 03:00 UTC, the audit module's audit:prune deletes audit entries older than the retention period. Each statement in the closure ends with a semicolon, so write the closure with a block.

A module can define three kinds of task.

Scheduling commands

s.command(name, args) runs a registered CLI command with those arguments, as cargo run -- <name> <args> would, but inside the scheduler's process and app instead of a new one:

Rust

The task is named after the command line, here audit:prune --days 90. If no module registers the command, the boot fails.

Scheduling jobs

s.job(value) dispatches a clone of the job on each due tick, through the app's job dispatcher, so a queue worker runs it and retries it if it fails. The job type must be Clone:

Rust

The task is named after the job's NAME. m.schedule registers the job type with the queue for you, so you don't need m.job::<J>() as well (registering it twice is harmless). In an app without a queue, the job runs inline on the scheduler.

Scheduling functions

s.task(name, f) calls an async function with the app's AppContext. A closure works:

Rust

So does a plain function with the signature async fn(AppContext) -> Result<(), BoxError>:

Rust

Task names

Each task needs a unique name, at most 191 characters: it keys the task's runs in schedule_runs. Two tasks with the same name fail the boot, naming both modules. Rename a command or job task with .name(...):

Rust

Schedule frequencies

A task without a frequency runs every minute. The helpers build cron expressions:

MethodRunsCron
.cron("*/10 9-17 * * MON-FRI")on a custom cron expressionas given
.every_minute()every minute (the default)* * * * *
.every_five_minutes()every five minutes*/5 * * * *
.every_fifteen_minutes()every fifteen minutes*/15 * * * *
.hourly()at the start of every hour0 * * * *
.hourly_at(17)every hour at 17 minutes past17 * * * *
.daily()every day at midnight0 0 * * *
.daily_at("08:00")every day at 08:000 8 * * *
.weekly()every Sunday at midnight0 0 * * 0

A cron expression has five fields: minute, hour, day of month, month and day of week. Expressions are parsed by croner, which also accepts names such as MON-FRI and the nicknames @hourly, @daily, @weekly, @monthly and @yearly. A seconds field isn't allowed.

An invalid expression, an hourly_at minute over 59 or a daily_at time that isn't a 24-hour HH:MM fails the boot with the task, its module and the reason. The last frequency you set wins.

Laravel's other helpers (twiceDaily, weekdays, monthlyOn and so on) don't exist; write the cron expression instead, for example .cron("0 9 * * MON-FRI") for weekdays at nine.

Time zones

Every schedule is in UTC, and 0.1 has no time-zone setting, neither for the app nor per task. daily_at("08:00") runs at 08:00 UTC wherever your servers and users are: the CRM's reminders go out at 08:00 UTC for everyone.

To run a task at a local time, convert it to UTC yourself, and remember that the conversion changes when daylight saving time starts or ends: a task at 08:00 in London is daily_at("07:00") in summer and daily_at("08:00") in winter.

Preventing overlaps

A task that sometimes runs longer than its interval could start again while it's still running. without_overlapping skips a tick while an earlier run of the task is still in progress:

Rust

The scheduler sees a running run in its own process (an in-process lock) and on other servers (a running row in schedule_runs). A skipped tick is reported as skipped and is not run later.

  • A running row older than 24 hours doesn't count: it was probably left by a process that crashed mid-run. Until then, though, such a row keeps skipping the task. If a scheduler crashed during a run and you need the task sooner, delete that row from schedule_runs.
  • For a job task, the run is the dispatch, which is over at once, so without_overlapping doesn't stop two of its jobs from running at the same time on workers. Make the job unique for that.

Running on one server

You can run the scheduler on every server, for redundancy. Each due tick of each task is claimed in the database first, with an INSERT … ON CONFLICT DO NOTHING on schedule_runs keyed by task and minute; only the scheduler whose insert succeeds runs it, and the others report it skipped. A second schedule:run in the same minute skips it too.

Laravel needs onOneServer() for this. In Navree every task already runs on one server per tick; .on_one_server() exists so your intent is written down, and changes nothing.

Two conditions apply: every scheduler must use the same database, and the servers' clocks must agree (keep NTP running). Ticks are whole minutes of each server's own clock, so a server whose clock is a minute off claims a different tick.

Running the scheduler

Run the scheduler in one of two ways. Both run whatever is due at each minute.

schedule:work

schedule:work runs in the foreground and ticks at every minute boundary until it is stopped:

Shell

Keep it running with a process manager, like a queue worker. Each tick runs in the background, so a slow task never delays the next tick. Ctrl-C or SIGTERM stops it, and the app's graceful shutdown waits for running tasks (30 seconds by default). If the process stalls, for example because the machine was suspended, the minutes it missed are skipped, not caught up.

schedule:run

schedule:run runs the tasks due in the current minute, waits for them and exits. Call it every minute from the system's cron:

crontab

It prints one line per due task, for example at 08:00 UTC in the CRM:

Text

The other lines are Skipped: <task> (already run for this minute), Skipped: <task> (still running) and Failed: <task> (<error>). When a task fails, the command exits with an error, so cron's mail or your monitoring sees it. With no task due it prints No scheduled tasks are due.

schedule:list

schedule:list shows every task with its expression and next run. For the CRM, at noon UTC:

Text

Failures and the run history

A task that returns an error or panics is logged at error level, recorded as failed and not retried; the next due tick runs it as usual. For work that should be retried, schedule a job: the queue retries it with backoff.

Every run is logged at target navree::schedule (task, tick, duration and outcome) and recorded in schedule_runs:

ColumnHolds
task, tickThe task name and the minute it was due (the primary key).
started_at, finished_atWhen it ran.
statusrunning, succeeded or failed.
errorThe error of a failed run (cut to 4,000 bytes).

Rows are kept for seven days; the scheduler deletes older ones at the top of every hour.

Testing

Resolve the Scheduler and call tick with a time: it runs the tasks due at that minute, waits for them and returns one RunOutcome per due task (task, tick and a RunStatus). Tasks that aren't due are left out. Override the app's dyn Clock with a FixedClock so the tasks see the same time:

tests/schedule.rs
  • RunStatus::Failed(message) carries the error a failed run returned.
  • scheduler.list(now) returns every task as a TaskInfo (name, module, expr, next_run, without_overlapping), to check that a module schedules what it should.
  • To test a command task's effect, you can also run the command directly with app.run_command(...).

The CRM's crm-activities/tests/reminders.rs does all of this with the whole CRM: it seeds tasks, ticks at 08:00 and checks the notifications and mail each user got.

Popular pages

  • Installation
  • Your first app
  • Routing and controllers
  • Validation
  • Authentication
  • Benchmarks