How RepoLantern observes itself

Every run can be followed from the request to each model and tool call, without recording your code, prompts, answers or keys.

One run, one trace

RepoLantern can write OpenTelemetry traces, but only while the operator has switched tracing on, for a demonstration or an investigation. They are off by default and switch themselves off after 4, 24 or 48 hours. The operator also chooses where they go (Arize Phoenix, LangSmith, Langfuse or any OTLP backend). Nothing is traced continuously.

A run is one trace. It starts with the HTTP request, then invoke_agent repolantern for the job, one span for each stage, and inside each stage a chat <model> span for every model call and an execute_tool span for every tool that runs as code. Model calls carry the provider, the model and the input and output tokens, following the OpenTelemetry conventions for generative AI. A guide approved automatically within your limit is one trace; one you approve by hand is two, before and after your approval.

What is never recorded

By design, span attributes are names, ids and counts: they never hold your code, prompts, answers, keys or e-mail addresses. A live test runs a whole guide into a real Arize Phoenix and fails if code, prompt text, a bearer credential or an e-mail address appears in any span. The exporter's own credentials are treated as secrets and never printed.

What each trace says about itself

Every trace names the service (repolantern), its version, the environment (development or production) and the project it is filed under in Arize Phoenix (repolantern).
Health checks (/healthz and /readyz) are not traced, so they never bury real traffic.
A log line written during a run carries the trace_id and span_id of the step it belongs to, so a line in the logs leads straight to its trace.

The operator's control tower

Operators get a control tower in the app, built from RepoLantern's own SQLite database, with no extra monitoring service. It is always on, whether or not tracing is:

Golden signals: latency, traffic, errors, saturation, quality and cost, and the cost of each useful guide.
Guardrails: secrets removed, text aimed at AI tools, and blocked Git sign-ins.
Service objectives with error budgets, and readers' ratings.
Alerts for a quality drop or drift, a failure or cost spike, slow guides and prices due for review. Each alert names the playbook to follow.

It shows counts, ids, models and scores, never repository content, prompts or keys. Only administrators can open it, and they must sign in with two-step verification.

The operator's query console

Some questions in an incident are not on the control tower. For those, the control tower has a Query view: the operator writes one SELECT and sees the answer as a table, which can be saved as a CSV file. Like the control tower, it is always on.

What it reads: six views of RepoLantern's own database, and nothing else. obs_jobs (one row per guide job), obs_model_calls (one row per model call), obs_guides (grounding score per written guide), obs_events (job events by type and time), obs_feedback (reader votes: rating and reason, never comments) and obs_audit (the audit log's actions and times only). They hold ids, counts, models, scores, costs and times: no e-mail address, user id, repository path, reader comment, chat text or error message.
Read-only: the query runs on a separate read-only connection. Tables, the schema, PRAGMA, ATTACH and every write are refused, even inside a subquery.
Limits: a query stops after 5 seconds and returns at most 1,000 rows, so it can't slow down the guides being written.
Audit: every query, allowed or refused, is written to the audit log (admin.query).
Who: administrators only, signed in with two-step verification. For everyone else the address does not exist.

It comes with forty-five sample questions, each tested against a real database, in six groups for the people who ask them, shown one group at a time with a filter across all of them:

Operations: guides by outcome, failures by stage, pauses and why, time to approval, hourly load on the worker slots, time per pipeline stage.
Providers: model latency (median and 95th percentile) by stage and model, failed calls by cause, retries, fallbacks, rate-limit waits, tool speed.
Cost: the cost of each guide (cost per task) and per Q&A answer, per stage, audience and model, estimate against actual, reserved against settled, prompt-cache and summary-cache savings, the most expensive guides, daily spend.
Quality: grounding-score distribution, lowest scores, invented names, quality by prompt version, section repairs, scores and reader votes by audience.
Usage: guides per day, audiences, providers and models chosen, repository hosts and sizes, questions asked.
Security: operator actions, failed sign-ins, guardrail events, live-tracing switch changes, use of the console itself, requests refused by limits, erasures and consents, access tokens.

The step-by-step procedure for operators is in the incident runbook, section 15 (Query the database).

Arize Phoenix as the trace viewer

Arize Phoenix is the optional trace viewer. For demonstrations and troubleshooting it runs self-hosted next to RepoLantern, with its own database and sign-in turned on. It sleeps unless the operator has switched tracing on, and it is not the record of what happened: the query console, which reads RepoLantern's own database, always is. Open the repolantern project and each run shows as one tree: the stages, then every model and tool call with its token counts. In the hosted service, export stays off unless the operator turns it on for a limited time.