Error log, retry notice, a waiting counter that keeps counting
- generation_errors: every failed attempt (model, attempt, outcome, duration); admin Hibanapló page and GET /v1/admin/errors; kept 90 days - The waiting page counts from when the text was asked for (requested_at), not from the page load, so a reload no longer restarts it - It says when an attempt failed and is retried, and when a failed text's unit was given back; admins see the error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012vLHzh5qUhFtmJhzkcTbXL
This commit is contained in:
@@ -127,6 +127,7 @@ Hibák formátuma: `{"error": {"code", "message"}}`. A születési adatok hibái
|
||||
- **Fizetős rész:** a részletes elemzés darabra megy. A képletelemzés egy `natal`, a szinasztria-elemzés egy `synastry` egységet von le (`credit_ledger.unit`). Az egységek csomagvásárlásból (Stripe, később) vagy admin jóváírásból jönnek. Havi ingyenes keret nincs.
|
||||
- **Modell:** három beállítás az admin felületen, az OpenRouter listájából (`model_settings` tábla): az elemzések és az éves előrejelzés (képlet-, kapcsolati elemzés, előrejelzés) közös modellje, a kérdés-válasz modellje és a rövid összefoglalóé. Ha nincs beállítva, az `ANALYSIS_MODEL` / `SUMMARY_MODEL` (alapból mindkettő Opus 5.5, a 2026-11-es összevetés legjobbja, lásd `services/worker/README.md`). Csak kötött JSON-választ adó modell választható. A kliens nem választ modellt, és a modell neve felhasználónak sehol nem jelenik meg.
|
||||
- **Tartalék:** ha a választott modell kétszer hibára fut (végleges hibánál egyszer), a worker az alapmodellel próbálja tovább (`fallback_model`, `on_fallback`); csak ha az is elbukik, jár vissza az egység. A Modellek oldal mutatja, hány szövegnél kellett az elmúlt 30 napban.
|
||||
- **Hibanapló:** a worker minden sikertelen próbálkozást a `generation_errors` táblába ír (mi, melyik modell, hányadik kísérlet, mi lett: újrapróbálás, tartalék vagy végleges hiba, mennyi ideig tartott); az admin Hibanapló oldala mutatja (`GET /v1/admin/errors?source=&outcome=`), 90 napig őrizzük. A felhasználó a várakozó oldalon látja, ha az előző próbálkozás nem sikerült és újrapróbáljuk; a számláló a kérés idejétől (`requested_at`) mér.
|
||||
- **Futó generálások:** a Modellek oldalon látszik minden készülő és sorban álló elemzés és válasz, a modelljével és azzal, amivel most indulna; újraindításkor a jelenlegi beállítással indul újra (`GET /v1/admin/jobs`, `POST /v1/admin/jobs/restart`). Minden kísérlet `started_at` értéke a „jegye”: az újraindítás törli, így a workerben még futó régi kísérlet nem írhatja felül az újat, és a worker 10 másodpercen belül eldobja.
|
||||
- **Újragenerálás:** admin minden elemzést egyenként újraírathat (`POST /v1/admin/analyses/{id}/regenerate`): ugyanaz a sor (az azonosító és a megosztások maradnak), a jelenlegi prompttal és modellel, egységlevonás nélkül.
|
||||
- **Újraszámolás:** számítási változás után a `backend recalculate` parancs minden mentett képletet, összevetést és előrejelzést újraszámol, majd minden elemzést újra sorba állít (egységlevonás nélkül).
|
||||
|
||||
@@ -49,11 +49,16 @@ pub struct AnalysisRow {
|
||||
pub credits_charged: i32,
|
||||
pub created_at: OffsetDateTime,
|
||||
pub completed_at: Option<OffsetDateTime>,
|
||||
/// Attempts so far; above 1 the page says it is being tried again.
|
||||
pub attempts: i32,
|
||||
pub next_attempt_at: OffsetDateTime,
|
||||
/// When the current text was asked for (set again by a regeneration).
|
||||
pub requested_at: OffsetDateTime,
|
||||
}
|
||||
|
||||
const ANALYSIS_COLUMNS: &str = "id, kind, chart_id, synastry_id, forecast_id, language, prompt_version, status, \
|
||||
content, error, \
|
||||
credits_charged, created_at, completed_at";
|
||||
credits_charged, created_at, completed_at, attempts, next_attempt_at, requested_at";
|
||||
|
||||
impl AnalysisRow {
|
||||
pub fn view(&self) -> Value {
|
||||
@@ -71,6 +76,9 @@ impl AnalysisRow {
|
||||
"credits_charged": self.credits_charged,
|
||||
"created_at": rfc3339(self.created_at),
|
||||
"completed_at": rfc3339_opt(self.completed_at),
|
||||
"attempts": self.attempts,
|
||||
"next_attempt_at": rfc3339(self.next_attempt_at),
|
||||
"requested_at": rfc3339(self.requested_at),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -391,7 +399,7 @@ pub async fn regenerate(
|
||||
"UPDATE analyses SET status = 'pending', content = NULL, error = NULL, model_used = NULL, \
|
||||
input_tokens = NULL, output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), \
|
||||
started_at = NULL, completed_at = NULL, model = $2, prompt_version = $3, cache_key = $4, \
|
||||
fallback_model = $5, on_fallback = false \
|
||||
fallback_model = $5, on_fallback = false, requested_at = now() \
|
||||
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {ANALYSIS_COLUMNS}"
|
||||
))
|
||||
.bind(id)
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
//! Admin: the error log of the analyses, answers and prompt test runs (every
|
||||
//! failed attempt, written by the worker), newest first.
|
||||
|
||||
use axum::Json;
|
||||
use axum::extract::{Query, State};
|
||||
use axum_extra::extract::WithRejection;
|
||||
use serde::Deserialize;
|
||||
use serde_json::{Value, json};
|
||||
use time::OffsetDateTime;
|
||||
use uuid::Uuid;
|
||||
|
||||
use super::St;
|
||||
use crate::auth::AdminUser;
|
||||
use crate::error::{ApiError, ApiResult};
|
||||
use crate::views::rfc3339;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct ErrorQuery {
|
||||
limit: Option<i64>,
|
||||
offset: Option<i64>,
|
||||
/// `analysis`, `question` or `preview`.
|
||||
source: Option<String>,
|
||||
/// `retry`, `fallback` or `failed`.
|
||||
outcome: Option<String>,
|
||||
}
|
||||
|
||||
type Row = (i64, OffsetDateTime, String, Uuid, String, String, i32, String, String, Option<i32>, Option<String>);
|
||||
|
||||
pub async fn list(
|
||||
State(state): St,
|
||||
_admin: AdminUser,
|
||||
WithRejection(Query(query), _): WithRejection<Query<ErrorQuery>, ApiError>,
|
||||
) -> ApiResult<Json<Value>> {
|
||||
let limit = query.limit.unwrap_or(100).clamp(1, 500);
|
||||
let offset = query.offset.unwrap_or(0).max(0);
|
||||
// Whose text it was: the requester of the analysis or the asker.
|
||||
let rows: Vec<Row> = sqlx::query_as(
|
||||
"SELECT e.id, e.created_at, e.source, e.item_id, e.kind, e.model, e.attempt, e.outcome, e.error, e.seconds, \
|
||||
u.email \
|
||||
FROM generation_errors e \
|
||||
LEFT JOIN analyses a ON e.source = 'analysis' AND a.id = e.item_id \
|
||||
LEFT JOIN questions q ON e.source = 'question' AND q.id = e.item_id \
|
||||
LEFT JOIN prompt_previews p ON e.source = 'preview' AND p.id = e.item_id \
|
||||
LEFT JOIN users u ON u.id = coalesce(a.requested_by, q.asked_by, p.created_by) \
|
||||
WHERE ($1::text IS NULL OR e.source = $1) AND ($2::text IS NULL OR e.outcome = $2) \
|
||||
ORDER BY e.id DESC LIMIT $3 OFFSET $4",
|
||||
)
|
||||
.bind(&query.source)
|
||||
.bind(&query.outcome)
|
||||
.bind(limit)
|
||||
.bind(offset)
|
||||
.fetch_all(&state.db)
|
||||
.await?;
|
||||
let (total, last_day): (i64, i64) = sqlx::query_as(
|
||||
"SELECT count(*), count(*) FILTER (WHERE created_at > now() - interval '1 day') FROM generation_errors \
|
||||
WHERE ($1::text IS NULL OR source = $1) AND ($2::text IS NULL OR outcome = $2)",
|
||||
)
|
||||
.bind(&query.source)
|
||||
.bind(&query.outcome)
|
||||
.fetch_one(&state.db)
|
||||
.await?;
|
||||
let errors: Vec<Value> = rows
|
||||
.into_iter()
|
||||
.map(|(id, at, source, item, kind, model, attempt, outcome, error, seconds, email)| {
|
||||
json!({
|
||||
"id": id, "created_at": rfc3339(at), "source": source, "item_id": item, "kind": kind, "model": model,
|
||||
"attempt": attempt, "outcome": outcome, "error": error, "seconds": seconds, "user": email,
|
||||
})
|
||||
})
|
||||
.collect();
|
||||
Ok(Json(json!({ "errors": errors, "total": total, "last_day": last_day, "limit": limit, "offset": offset })))
|
||||
}
|
||||
@@ -109,7 +109,8 @@ pub async fn restart(
|
||||
let (model, fallback, prompt_version) = super::questions::model_and_version(&state, &language).await?;
|
||||
sqlx::query(
|
||||
"UPDATE questions SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
||||
prompt_version = $4, attempts = 0, next_attempt_at = now(), started_at = NULL, error = NULL \
|
||||
prompt_version = $4, attempts = 0, next_attempt_at = now(), started_at = NULL, error = NULL, \
|
||||
requested_at = now() \
|
||||
WHERE id = $1 AND status IN ('pending', 'running')",
|
||||
)
|
||||
.bind(id)
|
||||
@@ -127,7 +128,7 @@ pub async fn restart(
|
||||
sqlx::query(
|
||||
"UPDATE analyses SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
||||
cache_key = $4, prompt_version = $5, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
||||
error = NULL \
|
||||
error = NULL, requested_at = now() \
|
||||
WHERE id = $1 AND status IN ('pending', 'running')",
|
||||
)
|
||||
.bind(id)
|
||||
|
||||
@@ -10,6 +10,7 @@ mod admin;
|
||||
mod analyses;
|
||||
mod auth;
|
||||
mod charts;
|
||||
mod errors;
|
||||
mod forecasts;
|
||||
mod invitations;
|
||||
mod jobs;
|
||||
@@ -117,6 +118,7 @@ pub fn router(state: Arc<AppState>) -> Router {
|
||||
.route("/v1/admin/prompts/{kind}/{language}", axum::routing::put(prompts::save))
|
||||
.route("/v1/admin/prompts/{kind}/{language}/preview", post(prompts::start_preview))
|
||||
.route("/v1/admin/previews/{id}", get(prompts::get_preview))
|
||||
.route("/v1/admin/errors", get(errors::list))
|
||||
.route("/v1/admin/jobs", get(jobs::list))
|
||||
.route("/v1/admin/jobs/restart", post(jobs::restart))
|
||||
.route("/v1/admin/models", get(models::list))
|
||||
|
||||
@@ -39,10 +39,13 @@ pub struct QuestionRow {
|
||||
pub credits_charged: i32,
|
||||
pub created_at: OffsetDateTime,
|
||||
pub completed_at: Option<OffsetDateTime>,
|
||||
pub attempts: i32,
|
||||
pub next_attempt_at: OffsetDateTime,
|
||||
pub requested_at: OffsetDateTime,
|
||||
}
|
||||
|
||||
const QUESTION_COLUMNS: &str =
|
||||
"id, analysis_id, question, status, answer, error, credits_charged, created_at, completed_at";
|
||||
const QUESTION_COLUMNS: &str = "id, analysis_id, question, status, answer, error, credits_charged, created_at, \
|
||||
completed_at, attempts, next_attempt_at, requested_at";
|
||||
|
||||
impl QuestionRow {
|
||||
pub fn view(&self) -> Value {
|
||||
@@ -56,6 +59,9 @@ impl QuestionRow {
|
||||
"credits_charged": self.credits_charged,
|
||||
"created_at": rfc3339(self.created_at),
|
||||
"completed_at": rfc3339_opt(self.completed_at),
|
||||
"attempts": self.attempts,
|
||||
"next_attempt_at": rfc3339(self.next_attempt_at),
|
||||
"requested_at": rfc3339(self.requested_at),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -191,7 +197,8 @@ pub async fn regenerate(
|
||||
let row: QuestionRow = sqlx::query_as(&format!(
|
||||
"UPDATE questions SET status = 'pending', answer = NULL, error = NULL, model_used = NULL, input_tokens = NULL, \
|
||||
output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
||||
completed_at = NULL, model = $2, prompt_version = $3, fallback_model = $4, on_fallback = false \
|
||||
completed_at = NULL, model = $2, prompt_version = $3, fallback_model = $4, on_fallback = false, \
|
||||
requested_at = now() \
|
||||
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {QUESTION_COLUMNS}"
|
||||
))
|
||||
.bind(id)
|
||||
|
||||
@@ -1695,3 +1695,44 @@ async fn admins_restart_open_generations_with_the_current_models(pool: PgPool) {
|
||||
sqlx::query("UPDATE analyses SET status = 'done', content = '{}'").execute(&pool).await.unwrap();
|
||||
assert_eq!(app.post("/v1/admin/jobs/restart", Some(&admin), json!({})).await.body["restarted"], 0);
|
||||
}
|
||||
|
||||
#[sqlx::test(migrator = "db::MIGRATOR")]
|
||||
async fn admins_read_the_error_log(pool: PgPool) {
|
||||
let app = TestApp::new(pool.clone()).await;
|
||||
let admin = app.register("admin@example.com", "Admin").await;
|
||||
let anna = app.register("anna@example.com", "Anna").await;
|
||||
let chart = app.chart(&anna, "Saját", true, "1990-03-15").await;
|
||||
let chart_id = chart["id"].as_str().unwrap();
|
||||
let analysis: uuid::Uuid = sqlx::query_scalar("SELECT id FROM analyses WHERE chart_id = $1::uuid")
|
||||
.bind(chart_id)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
for (attempt, outcome) in [(1, "retry"), (2, "failed")] {
|
||||
sqlx::query(
|
||||
"INSERT INTO generation_errors (source, item_id, kind, model, attempt, outcome, error, seconds) \
|
||||
VALUES ('analysis', $1, 'summary', 'cheap/model', $2, $3, 'answer cut off at the token limit', 301)",
|
||||
)
|
||||
.bind(analysis)
|
||||
.bind(attempt)
|
||||
.bind(outcome)
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
assert_eq!(app.get("/v1/admin/errors", Some(&anna)).await.status, StatusCode::FORBIDDEN);
|
||||
let r = app.get("/v1/admin/errors", Some(&admin)).await;
|
||||
assert_eq!((r.status, &r.body["total"], &r.body["last_day"]), (StatusCode::OK, &json!(2), &json!(2)), "{}", r.body);
|
||||
let first = &r.body["errors"][0];
|
||||
assert_eq!(
|
||||
(&first["outcome"], &first["attempt"], &first["user"], &first["seconds"]),
|
||||
(&json!("failed"), &json!(2), &json!("anna@example.com"), &json!(301))
|
||||
);
|
||||
let r = app.get("/v1/admin/errors?outcome=retry", Some(&admin)).await;
|
||||
assert_eq!(r.body["errors"].as_array().unwrap().len(), 1);
|
||||
assert_eq!(app.get("/v1/admin/errors?source=question", Some(&admin)).await.body["total"], 0);
|
||||
// users see the attempts of their analysis (the page says it is tried again)
|
||||
let r = app.get(&format!("/v1/analyses/{analysis}"), Some(&anna)).await;
|
||||
assert_eq!(r.body["attempts"], 0);
|
||||
assert!(r.body["next_attempt_at"].is_string());
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user