The healthcheck dashboard¶

The healthcheck dashboard (/tools/healthcheck) is the operations view over the platform's instance health monitoring. Behind it, the load balancer polls every server instance every 15 seconds; each poll probes every tenant application on that instance, and each probe runs five subsystem checks (application scope, database, Redis caches, Elasticsearch, JVM resources). Results are written to the platform database, and this dashboard is where those results are explored.

There is no sidebar menu entry — the screen is reached by its URL. Its data endpoints sit behind the normal API login, so any authenticated user can technically load it; in practice it is an operations tool.

Each check run and tenant carries one of four statuses: healthy, degraded (slow but working — does not take an instance out of service), unhealthy, or critical. Only unhealthy and critical count as failures.

Auto-refresh — Off / 10s / 30s / 1m / 5m; defaults to 30s and refreshes whichever tab is open.
Time range — 1h (default) / 6h / 24h / 48h / 7d; applies to every tab. Seven days is also the retention limit (below), so 7d means "everything".
Four tabs: Overview, Logs, Failures, Timeouts.

The Overview tab¶

Four summary cards: total check runs in the window, failing applications, failing instances, and timeout events. The app and instance cards count distinct applications/instances with at least one failure, not individual events.
A response time chart (average, min, max per interval) for the health probes themselves.
Failure counts grouped by application, by instance, and by HTTP status code.
Platform activity panels drawn from the audit log rather than the healthcheck tables: total requests and request rate, average response time trend, request volume, most active users, slowest requests, and recent errors — each error opening the same entry drill-down as the audit trail search.

The Logs tab¶

Every aggregated check run: time, instance, status badge, tenants checked, tenants failed, and the probe's response time. The detail modal shows the run's full result, including the per-subsystem breakdown for the tenant you are currently browsing on.

The Failures tab¶

Individual tenant failures: application, instance, status, HTTP status badge, error message, response time. The detail modal has tabs:

Overview — the failure and, when the probe returned a parseable body, the per-check results (which subsystem was unhealthy, including each Redis cache region).
Retry Checks — critical and timeout failures are automatically retried once with a longer timeout; the banner records whether the retry succeeded (distinguishing a blip from a real outage).
Server Snapshot — the JVM state captured at failure time (memory, thread counts including busy worker threads), used to diagnose saturation.

The Timeouts tab¶

Failures that were timeouts (HTTP 408/504 or a critical status). The Analyze button on a row runs a per-application timeout analysis: total timeouts in the window, a pattern classification — sporadic, burst, or persistent — the last successful check, a timeout-by-hour table, and the most recent timeout events.

Why does the dashboard change unexpectedly¶

Older data vanished — healthcheck logs and failures are purged after 7 days by a daily scheduled task; the dashboard can never show more than a week.
Numbers move while watching — auto-refresh is on by default (30s).
Card counts don't match table row counts — the failing apps/instances cards count distinct names, while the Failures tab lists every event.
A tenant shows degraded but nothing failed — degraded is by design not a failure; it flags slowness (e.g. a slow database response or a yellow Elasticsearch cluster) without affecting instance rotation.

What the healthcheck dashboard is not¶

Not the audit trail search — the Overview's activity panels summarize the same audit data, but request-level investigation belongs in the audit search.
Not the health endpoint itself — the load balancer's polling and instance rotation happen in the backend regardless of this screen; the dashboard is read-only.
Not an uptime/SLA report — with a 7-day retention there is no long-term availability history.