Lighthouse

Introduction

Featurelighthouse Lighthouse watches your app in production and tells you what went wrong. It records every exception, slow request, slow query, failed job and failing outgoing call, groups them into issues by fingerprint, and shows them on a dashboard served by your own app at /_lighthouse. When something new breaks, it sends an alert to email, Slack or a webhook. Scripts read and resolve issues through its JSON API, coding agents through its MCP tools, and your traces can also go to an OpenTelemetry collector through ObserveModule.

Lighthouse runs inside your app and stores its data in your database. There is no agent to install, no service to host and no data leaving your servers unless you configure an alert or an export.

• Every process reports. serve, queue:work, schedule:work, Fleet workers and one-off commands all write to the same store, so a job that fails on a worker shows up next to the request that failed on the web server.
• No code changes. Lighthouse reads what the framework already produces: the request pipeline and its exceptions and panics, query timings from navree-db, job outcomes from navree-queue, scheduled runs, mail, cache calls and the HTTP client. You add the module; the rest is configuration.
• Off the request path. Recording copies a few fields into an in-memory buffer. A background task writes them to the database in batches, so a request never waits for Lighthouse.

Installation

The lighthouse feature of the navree facade brings the crate navree-lighthouse. It is part of full, the default, so an app made with navree new already has it, and its src/app.rs already includes the module. To add it to an existing app, put LighthouseModule after the modules it builds on:

pub fn app(sources: ConfigSources) -> AppBuilder {
    Navree::app()
        .module(ConfigModule::for_root(sources))
        .module(LoggerModule::from_config())
        .module(CryptoModule::from_config())
        .module(HttpModule::from_config())
        .module(DatabaseModule::from_config())
        // ... queue, mail, auth, access and your own modules
        .module(LighthouseModule::from_config())
}

LighthouseModule is global. It needs:

The module contributes the migration set lighthouse, which migrate runs with your app's migrations. It creates five tables, all prefixed lighthouse_: issues, occurrences, samples (request and job samples with their spans), aggregates (per-minute and per-hour counts and latency histograms) and alerts (the alerts sent). On TimescaleDB, samples, occurrences and aggregates are hypertables, a day per chunk, in the app's database or in lighthouse.database_url's.

navree migrate

Then give yourself access and open the dashboard. Lighthouse registers the subject Lighthouse with the permission prefix lighthouse, so the permission lighthouse.view lets a role look and lighthouse.resolve lets it change issues:

navree access:role manager
navree access:grant manager lighthouse.view
navree access:grant manager lighthouse.resolve

Users with the super-admin role need nothing: they pass every check. Open http://127.0.0.1:8080/_lighthouse and sign in with your app's account.

What Lighthouse records

A 4xx response is not an exception: it is the app working as intended. Requests answering 4xx count towards the route's error rate on the dashboard, and that's all.

Each record carries the context Lighthouse can find without your help:

• the request id (the X-Request-Id your client saw), the method, the route template and the status;
• the signed-in user's id, and with workspaces the workspace's id;
• the release, the environment, the server's host name and the process (serve, queue:work, the command's name);
• for a job, its id, type, queue and attempt;
• the trace id, which is also the one exported to OpenTelemetry.

Traces

A trace is the timeline of one request or job: every query, outgoing call, cache call and mail it made, as spans with their durations. Lighthouse keeps the trace of every slow request, slow job, failed request or job, request that reported an error, and of a sample of the others. A trace shows you where the time went: in the slow request below, eleven identical queries for contacts take most of the 1.4 seconds, and the waterfall shows them as one row.

    0 ms  db    SELECT ... FROM deals WHERE stage <> ?          38 ms
   39 ms  db    SELECT ... FROM companies WHERE id IN (...)     12 ms
   52 ms  db    SELECT ... FROM contacts WHERE id = ?  x 11  1,203 ms
1,256 ms  cache hit dashboard:wins:0192f0c1...                   0 ms
1,258 ms  http  GET https://api.exchangerate.example/latest    150 ms

A trace keeps at most 200 spans; the rest are counted.

Reporting errors yourself

Some errors never reach a 500: you catch them, log them and carry on. Report them to Lighthouse so they still become issues. Inject Arc<Lighthouse> and call report:

use navree::lighthouse::Lighthouse;
use navree::prelude::*;

#[injectable]
pub struct PaymentWebhooks {
    lighthouse: Arc<Lighthouse>,
    invoices: Arc<InvoiceService>,
}

impl PaymentWebhooks {
    pub async fn handle(&self, event: PaymentEvent) -> Result<NoContent> {
        if let Err(error) = self.invoices.mark_paid(&event).await {
            // Answer 204 so the provider stops retrying, but keep a record.
            self.lighthouse.report(&error);
        }
        Ok(NoContent)
    }
}

report takes any error, a concrete type (its type name joins the fingerprint) or a dyn std::error::Error, and returns at once: the occurrence goes into the buffer with the context of the current request or job and the location of the call. For more control, build a Report:

use navree::lighthouse::{Level, Report};

self.lighthouse.capture(
    Report::error(&error)
        .level(Level::Warning)
        .context("invoice_id", invoice.id)
        .tag("provider", "acme-pay")
        .fingerprint(["acme-pay", "card-declined"])
        .backtrace(),
);

// No error value: something that should never happen.
self.lighthouse.capture(Report::message("Payment event without a customer").level(Level::Warning));

To tag everything the current request or job reports, call self.lighthouse.tag_current("plan", "pro") once early on, for example in a guard. The tag also goes on the request's sampled trace.

Issues

An issue is every occurrence of one problem. Lighthouse computes a fingerprint for each record and adds it to the issue with that fingerprint, opening a new issue the first time it sees one. The dashboard then shows one line per problem with its count, the users and workspaces it affected, and when it was first and last seen, instead of a thousand identical errors.

Fingerprints

Rust errors don't carry a stack trace, so Lighthouse fingerprints what they do carry: the error's type, where it happened (the route or the job), and the message chain with its variable parts taken out. Numbers, UUIDs, quoted strings, email addresses and hexadecimal ids in a message are replaced before hashing, so deal 0199c1d2-… not found and deal 0199c1e7-… not found are one issue.

Panics carry their location. For a report you build yourself, .backtrace() adds a backtrace, and .fingerprint([..]) decides the grouping outright.

Statuses

The release comes from lighthouse.release. Harbor sets it to the release id on every deploy (APP_LIGHTHOUSE__RELEASE); without it, Lighthouse uses the app.version setting, else unknown.

The dashboard

The dashboard is served by your app at lighthouse.path (/_lighthouse), outside the API prefix: server-rendered pages from templates compiled into the crate, with a small stylesheet and script of its own. No Node.js, no CDN and no separate process. The overview refreshes its figures and its feed every ten seconds while you watch.

• Overview. A row of figures for the selected period (the last hour, 24 hours or 7 days): requests per minute, the error rate, the 95th percentile latency, failed jobs and open issues. Below them, a timeline of requests and exceptions, and a feed of the latest occurrences, new ones sliding in at the top.
• Issues. One line per issue (open ones by default): its title (the first line of the message), the route or job it comes from, a sparkline of the last 24 hours, the number of occurrences, users and workspaces affected, and first and last seen. Filter by status, kind, level, release, environment, route, job, tag (key:value), user or workspace, search titles, and sort by last seen, first seen or count.
• An issue. The message and the error's source chain, the backtrace when there is one, and the latest occurrence: the request (method, route, headers with secrets masked), the user, the workspace, the release, the server, tags and your report's context values. A chart shows occurrences per hour over seven days, and a list shows recent occurrences with a link to each trace. Buttons resolve, resolve in the next release, ignore (forever, for an hour, a day or 100 more events) or reopen it, and a table shows the alerts sent about it.
• Requests. Every route with its throughput, error rate and p50, p95 and p99 latency, slowest first. A route's page lists its slowest traces, each drawn as a waterfall of spans.
• Queries. Normalized statements with how often they ran, their total, average and p95 time, and the routes and jobs that ran them.
• Jobs. Every job type with throughput, runtime, failed attempts and failed jobs. For queue workers and wait times in depth, see Fleet.
• Outgoing. Every host your app calls through HttpClient, with latency, error rate and the slowest calls.

When observe.tracing.trace_url is set, each trace id links to the trace in your tracing backend.

Signing in and access

The dashboard's sign-in screen uses your app's own users: an email and password through AuthService, then a two-factor code (or a recovery code) when the account has two-factor authentication on. It is subject to the same login throttle as POST /api/auth/login. Signing in starts no API session (the refresh token the sign-in creates is revoked at once): the dashboard session lives in an encrypted, httpOnly, SameSite=Strict cookie scoped to /_lighthouse for twelve hours, so the rest of your API never sees it, and every change the dashboard makes carries a CSRF token.

Its JSON API, under /_lighthouse/api, also accepts an access token in Authorization: Bearer, for scripts, agents and tests.

Every route of the dashboard and its API checks an ability on the subject Lighthouse:

Grant the permissions to a role, as in installation, or write the rules in your abilities function:

pub fn define(ctx: &AbilityCtx, rules: &mut Rules) {
    ctx.grant_permissions(rules);

    if let Some(user) = &ctx.user
        && user.has_role("ops")
    {
        rules.can("view", "Lighthouse");
        rules.can("resolve", "Lighthouse");
    }
    // ...
}

A signed-in user without view gets a 403 page; a guest goes to the sign-in screen. The API answers 401 and 403 in the usual JSON error shape.

The JSON API for scripts and agents

Everything on the dashboard is also JSON under /_lighthouse/api, for scripts, CI jobs and coding agents. Send an access token of a user with the right ability:

A session with a coding agent then looks like this: you ask it to fix the newest Lighthouse issue; it fetches /api/issues?sort=first_seen, reads the source chain and the failing request from /api/issues/{id}, opens the code, writes the fix and a test, and resolves the issue in the next release with a comment linking the change. Give it a token for a user with lighthouse.view to let it read, and lighthouse.resolve to let it close issues. On your own machine, Navigator (navree mcp) gives the agent the same issues and figures without a token, read-only, straight from your local database.

MCP tools for agents

With the facade's mcp feature on next to lighthouse (mcp is opt-in, so name it next to the defaults) and McpModule in the app, Lighthouse adds tools, a resource and a prompt to your app's MCP server at /mcp. There is nothing else to register: the facade turns on navree-lighthouse's own mcp feature, and LighthouseModule registers its server, LighthouseMcp.

[dependencies]
navree = { git = "https://github.com/NavreeHQ/navree", features = ["mcp"] }

Navree::app()
    // ... database, auth, access
    .module(LighthouseModule::from_config())
    .module(McpModule::from_config()) // serves /mcp, with Lighthouse's tools

Each item checks the same abilities as the JSON API, so a client sees only the tools its user may call:

The read tools carry readOnlyHint and the status changes idempotentHint. A result is the JSON of the matching API route (a list also comes as {"result": [..]} in the structured content); an unknown id is a tool error with a 404 body, and invalid arguments a 400 with field messages.

Each issue is also the resource lighthouse://issues/{id} (the JSON of lighthouse_get_issue). The prompt Fix a Lighthouse issue (fix-lighthouse-issue, with the argument issue_id, ability view) attaches that resource with instructions: find the cause, write a failing test, fix it, run the tests, then call lighthouse_resolve_issue with in_next_release and a comment naming the change.

Connect your coding agent to the app the way any MCP client connects, as a user with lighthouse.view to read and lighthouse.resolve to resolve, and ask it to fix the newest issue. Set lighthouse.mcp = false to keep Lighthouse off your MCP server; with lighthouse.store = "off" it offers nothing either.

Alerts

Lighthouse sends an alert when something needs a person. Each channel says which kinds of alert it wants:

Configure the channels in the lighthouse.alerts section:

[lighthouse.alerts]
cooldown_minutes = 30   # at most one alert per issue and kind in this window
spike = { occurrences = 50, minutes = 5 }

[lighthouse.alerts.email]
to = ["[email protected]"]
on = ["new_issue", "regression", "spike"]

[lighthouse.alerts.slack]
# webhook_url: APP_LIGHTHOUSE__ALERTS__SLACK__WEBHOOK_URL
on = ["new_issue", "regression", "spike", "failed_job", "long_wait"]

[lighthouse.alerts.webhook]
url = "https://ops.example.com/hooks/lighthouse"
# secret: APP_LIGHTHOUSE__ALERTS__WEBHOOK__SECRET
on = ["new_issue", "regression"]

A channel without on gets new_issue, regression and spike. Issues at level Info and ignored issues never alert. Every alert links to the issue on the dashboard at lighthouse.url, which defaults to app.url followed by the path; set it when your API has its own host, as in https://api.crm.example.com/_lighthouse.

Email

Email alerts go through your app's Mailer, from the configured sender. The message names the issue, its kind, count, release and environment, and links to it. to takes one address or a list; from the environment, a comma-separated string ([email protected],[email protected]).

Slack

Create an incoming webhook in your Slack workspace and set its URL. Keep it out of committed files: it is a secret.

APP_LIGHTHOUSE__ALERTS__SLACK__WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXXXXXX

Each alert is one message with the issue's title, the route or job, the count and an Open in Lighthouse button.

Webhooks

A webhook alert is a POST of JSON to your URL, signed following the Standard Webhooks specification: webhook-id, webhook-timestamp and webhook-signature headers, an HMAC-SHA256 over the id, the timestamp and the body with your secret (a whsec_ secret is base64-decoded first, as the specification says). Verify the signature before you trust the body; navree::lighthouse::webhook_signature computes it.

{
  "type": "lighthouse.new_issue",
  "issue": {
    "id": "0199c1d2-6f7e-7a31-b5d4-2f0c8e9a1b23",
    "title": "cannot charge invoice: card declined",
    "kind": "exception",
    "level": "error",
    "status": "open",
    "culprit": "POST /api/v1/invoices/{id}/charge",
    "count": 1,
    "users": 1,
    "first_seen": "2026-10-26T09:14:02Z",
    "last_seen": "2026-10-26T09:14:02Z",
    "url": "https://api.crm.example.com/_lighthouse/issues/0199c1d2-6f7e-7a31-b5d4-2f0c8e9a1b23"
  },
  "environment": "production",
  "release": "20261026-091512-3f2a9c1"
}

With a queue, each alert is a lighthouse.alert job per channel, and a failed delivery is retried with backoff, like any job: five attempts, from ten seconds up to ten minutes apart. Without a queue it is sent once, at once. The lighthouse_alerts table and the issue's page show each alert and whether it was delivered.

Your own channels

Implement AlertChannel to send alerts anywhere else, and register it with m.alert_channel::<C>():

use navree::BoxFuture;
use navree::lighthouse::{Alert, AlertChannel, AlertKind};
use navree::prelude::*;

#[injectable]
pub struct OnCallSms {
    sms: Arc<SmsGateway>,
}

impl AlertChannel for OnCallSms {
    fn name(&self) -> &str {
        "on-call-sms"
    }

    fn wants(&self, alert: &Alert) -> bool {
        matches!(alert.kind, AlertKind::Regression | AlertKind::Spike)
    }

    fn send<'a>(&'a self, alert: &'a Alert) -> BoxFuture<'a, Result<(), BoxError>> {
        Box::pin(async move {
            self.sms.send("+15550100", &format!("{}: {}", alert.kind, alert.issue.title)).await?;
            Ok(())
        })
    }
}

#[module(register = Self::wire)]
pub struct OpsModule;

impl OpsModule {
    fn wire(&self, m: &mut ModuleBuilder) {
        m.alert_channel::<OnCallSms>();
    }
}

To check every channel, navree lighthouse:test-alerts sends a test alert ("test": true in a webhook's body) through each one, or navree lighthouse:test-alerts slack through one, and prints whether it was delivered. Test alerts are not stored.

Sampling

Counts are always exact: every request, job and exception is counted towards its route's throughput and error rate and its issue's occurrences. Sampling decides how many details are stored: traces of ordinary requests and jobs, and occurrences of busy issues.

[lighthouse.sample]
requests = 0.1    # store the trace of 1 request in 10
jobs = 0.25
exceptions = 1.0  # store every occurrence (counts stay exact either way)

• Slow requests, slow jobs, failed requests and jobs, and the first 100 occurrences of each issue per hour are always stored, whatever the rates.
• The latency percentiles on the Requests page come from per-minute histograms of every request, not from samples.
• Sampling is per request: a sampled request keeps its whole trace, with every query in it. Whether a request is sampled depends only on its request id, so every process agrees.

On a busy app, start with requests = 0.1 and raise it if you want more traces to look at.

Retention and pruning

lighthouse:prune deletes samples, occurrences, aggregates and sent alerts older than lighthouse.retention_days (14), per-minute aggregates older than two days (hourly ones stay for the 24-hour and 7-day views), and resolved or ignored issues with no occurrence left. Open issues are kept with their counts. On TimescaleDB, the days entirely before the cutoff go as whole chunks.

navree lighthouse:prune
navree lighthouse:prune --days 7

Pruned 182,340 samples, 2,114 occurrences, 40,320 aggregate rows, 3 alerts and 12 issues older than 14 days.

When the app includes ScheduleModule, Lighthouse adds a lighthouse:prune task to the schedule itself, run every day at lighthouse.prune_at ("03:10", UTC), without overlapping; a schedule:work process runs it. Set prune_at = "" to schedule it yourself, or run it from cron in an app without a scheduler.

Storage

By default Lighthouse writes to the app's database. To keep monitoring writes away from your data, give it its own:

APP_LIGHTHOUSE__DATABASE_URL=postgres://lighthouse:[email protected]:5432/crm_lighthouse

Lighthouse connects to it when the app boots, creates its five tables there if they are missing, and reads and writes there; the dashboard reads from it too. The URL takes the same forms as database.url. (The lighthouse migration still creates the tables in the app's database, where they stay empty, so removing the setting later works.)

The background writer flushes the buffer every second, or as soon as it holds 500 records. If the database can't keep up and the buffer reaches lighthouse.buffer records (10,000), Lighthouse drops samples first and occurrences last, never counts, and says so: the dashboard shows a banner with the number this process dropped, and /metrics has lighthouse_dropped_total. On shutdown the writer flushes what it holds before the app exits.

Privacy and security

Lighthouse stores what your app handles, so it treats it with care:

• Secrets are masked. Header values, report context values and JSON body fields whose names contain a fragment of log.redact (password, token, authorization, cookie, secret by default) are stored as [redacted], plus any fragment you add to lighthouse.redact. Authorization, Proxy-Authorization, Cookie and Set-Cookie headers are always masked. Queries are stored as SQL with placeholders, never their bound values.
• No bodies by default. Request bodies are not recorded unless you set lighthouse.capture.request_body = true, and then only JSON bodies up to 16 KiB, with the same masking. Response bodies are never recorded.
• Client addresses. The client IP is stored with each request sample. Set lighthouse.capture.ip = false to leave it out.
• The dashboard denies by default. Nobody sees it without the view ability, sign-in attempts are throttled, the cookie is encrypted and scoped to the dashboard's path, and its pages are served with a strict Content Security Policy (only the dashboard's own stylesheet, script and API).
• Data stays with you. Nothing leaves your servers except the alerts and exports you configure.

Lighthouse's own routes are left out of its recordings, as are the paths in lighthouse.ignore.routes (/health/* and /metrics by default) and the lighthouse.alert jobs. Ignore noisy jobs with lighthouse.ignore.jobs.

OpenTelemetry export

Lighthouse keeps what it records in your database. To send traces to an OpenTelemetry collector as well, turn on the export of ObserveModule (the facade's otel feature and the observe.tracing section): it turns the same telemetry events into spans, named and attributed following the OpenTelemetry semantic conventions, and Lighthouse's trace ids are the ones it exports. With observe.tracing.trace_url set, the dashboard links each trace id to your tracing backend. Metrics are on /metrics.

To keep only the export, set lighthouse.store = "off": nothing is recorded, nothing is written to the database and the dashboard isn't mounted.

Configuration

Every key of the lighthouse section is optional. As environment variables: APP_LIGHTHOUSE__RETENTION_DAYS=30, APP_LIGHTHOUSE__SAMPLE__REQUESTS=0.1, and lists as comma-separated strings (APP_LIGHTHOUSE__IGNORE__ROUTES=/health/*,/metrics,/internal/*). An invalid value, such as a sample rate above 1 or an unknown alert kind, fails the boot with a message naming the key.

To configure the module in code instead:

use navree::lighthouse::{LighthouseOptions, Thresholds};

Navree::app().module(LighthouseModule::for_root(LighthouseOptions {
    retention_days: 30,
    thresholds: Thresholds {
        slow_request_ms: 500,
        ..Thresholds::default()
    },
    ..LighthouseOptions::default()
}))

Commands

Both run in your app's binary, so navree lighthouse:prune works in development and ./crm-api lighthouse:prune on a server.

Testing

LighthouseFake records reports instead of storing them, so a test can check that your code reports what it should. Install its Lighthouse in place of the app's, the way you install a queue fake:

use navree::lighthouse::{Level, LighthouseFake};
use navree::prelude::*;
use navree::testing::TestApp;

#[tokio::test]
async fn a_payment_for_an_unknown_invoice_is_reported() {
    let fake = LighthouseFake::new();
    let app = TestApp::builder()
        .temp_database()
        .module(DatabaseModule::from_config())
        .module(LighthouseModule::from_config())
        .module(BillingModule)
        .override_value(Arc::new(fake.lighthouse()))
        .build()
        .await;

    let webhooks = app.get::<PaymentWebhooks>().await;
    webhooks.handle(PaymentEvent::paid("inv_missing")).await.unwrap();

    fake.assert_reported::<InvoiceError>(|report| report.message.contains("not found"));
    fake.assert_reported_times(1);
    assert_eq!(fake.reports()[0].level, Level::Error);
    app.shutdown().await.unwrap();
}

The fake also records the exceptions the framework captures for you. A handler that answers 500 in a test shows up in reports() with its route, so you can assert that an error you expect is reported once and an error you handle isn't reported at all. In tests that don't install the fake, set lighthouse.enabled = false to keep Lighthouse out of the way, and call flush() on the injected Lighthouse before reading its store.