Logs: own folders, a daily cap, masked paths; workers finish running work on a deploy
CI / test (push) Successful in 1m34s
CI / build (push) Successful in 10m9s
CI / deploy (push) Skipped

Review fixes for the persistent logs:
- each service writes its own folder (the old-file cleanup matches by
  prefix, so 'worker' deleted the 'worker-ai' files);
- at most LOG_MAX_DAY_MB (50) a day, and at least 2 days kept;
- written by a background thread, not on the async workers;
- e-mail addresses and tokens in request paths are masked;
- a failing health check is logged again, a passing one is not;
- one-off commands (compare-models) log to the console only.

On a stop the worker takes nothing new and lets running work finish
(SHUTDOWN_DRAIN_SECS, worker 40 s, worker-ai 270 s, with matching stop
grace periods), so a deploy no longer cuts an analysis off, wastes its
model call and delays it by the 15-minute stuck-job requeue.

The CI cleanup caps the build cache at 10 GB: by age it had grown to
33 GB on the runner and filled its disk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012vLHzh5qUhFtmJhzkcTbXL
This commit is contained in:
feco
2026-11-20 18:57:09 +00:00
co-authored by Claude Opus 5.5
parent ffb6060cfa
commit 16c40864bb
14 changed files with 315 additions and 56 deletions
+4 -9
View File
@@ -36,8 +36,7 @@ use axum::{Json, Router};
use axum_extra::extract::WithRejection;
use serde_json::{Value, json};
use tower_http::cors::{AllowOrigin, CorsLayer};
use tower_http::trace::{DefaultMakeSpan, DefaultOnResponse, TraceLayer};
use tracing::Level;
use tower_http::trace::TraceLayer;
use uuid::Uuid;
use crate::AppState;
@@ -60,6 +59,7 @@ where
pub fn router(state: Arc<AppState>) -> Router {
Router::new()
.route("/healthz", get(health))
.route("/v1/config", get(public_config))
.route("/v1/packages", get(payments::catalog))
// login
@@ -140,13 +140,8 @@ pub fn router(state: Arc<AppState>) -> Router {
.route("/v1/admin/analyses/{id}/regenerate", post(analyses::regenerate))
.route("/v1/admin/questions/{id}/regenerate", post(questions::regenerate))
.layer(cors(&state.config.cors_allowed_origins))
.layer(
TraceLayer::new_for_http()
.make_span_with(DefaultMakeSpan::new().level(Level::INFO))
.on_response(DefaultOnResponse::new().level(Level::INFO)),
)
// After the layers: the Docker health check (every 30 s) stays out of the log.
.route("/healthz", get(health))
// The request log: masked paths, health checks only when they fail.
.layer(TraceLayer::new_for_http().make_span_with(service_kit::RequestLog).on_response(service_kit::RequestLog))
.with_state(state)
}