feat(hands,jobs): job-run watchdog looks a week back and alarms on jobs that stopped running

This commit is contained in:
hh committed 2026-09-22 20:15:15 +02:00
1 parent 426b4055f4
commit 91bd50a66d
6 files changed
+620 -71

No files matched your search

+32 -16
View File
@@ -277,7 +277,7 @@ that 231 times, and nothing reached a human: the outage surfaced ten days later,
by hand, out of curiosity. The record existed, the reader did not. This job is
the reader.
- `прогоны` (`:27` every hour) reads `job_runs` for the last 24 hours through
- `прогоны` (`:27` every hour) reads `job_runs` for the last 7 days through
the same connection the scheduler itself writes with
(`run.conversations.db.session()` and the `JobRunRecord` model) - no second
truth about where the journal lives.
@@ -290,8 +290,31 @@ the reader.
over the real outage it reads: «джоб `вайбграм` лежит 23 часа и дольше,
144 прогона подряд с 12.09 22:00 UTC: HubError: vibegram 530 bad_json». It repeats at most once a day per job, and
when the job recovers one short `отбой` line goes out and the counter resets.
The window is a week, not a day, so a daily job (`отметка`, `закрытие`,
`голосовой`) gets its three failures in a row in three days.
- **A job that stopped running is an alarm too, with its own text**: «джоб
`закрытие` не запускался 2 суток при периоде 24 часа». For every job with a
cron, the period comes from the registered cron (`run.scheduler.jobs`) via
the same `every_from_cron` the neighbour watchdog uses, and the last run is
taken from the whole journal, not the window. The threshold is **two
periods**: one missed fire happens honestly (a redeploy landed on the cron
minute, and the scheduler does not catch up), two in a row do not. It is
never finer than two of the watchdog's own periods (2 hours), because an
hourly watcher cannot tell a shorter gap from its own blindness between
runs. Time when the scheduler was not running non-critical jobs at all is
not counted as a miss: before the process started, and while the
watchdog's own hourly chain is broken (the gateway was down, or the budget
was throttled and non-critical crons were deferred without a record). Same
discipline as failures: once a day per job, and an `отбой` when a run shows
up.
- What silence does not cover, on purpose: jobs without a cron (webhooks,
events) are not measured at all; a job that has never appeared in the
journal waits silently for its first row; and a cron that does not reduce to
a regular period (`куратор`, `0 2,8,12,16,20 * * *`) is skipped and named in
the log as unguarded every run, because any number put in its place would
be made up. `куратор` still falls under the failure count.
- **An empty answer is a statement about the query, not about health.** Zero
rows in 24 hours (other than this job's own) means no job ran at all, and
rows in the last 24 hours (other than this job's own) means no job ran at all, and
that is its own alarm with its own text - the scheduler is what to look at,
not any one job. A failed query is no reading whatsoever: the run logs it and
stays silent, because a database that did not answer is not evidence of
@@ -303,22 +326,15 @@ the reader.
carrying the key can be named anything.
State between runs is `мета/бобер/.состояние/прогоны.json` (who was told and
when). A lost file means a lying job is reported again and an `отбой` for
when, separately for failures and for silence). A lost file means a lying job is reported again and an `отбой` for
something reported before the loss never comes - the evidence for it is gone.
Two blind spots, both deliberate:
- **The watchdog cannot report its own death.** Its own runs are excluded from
the count (`run.job.name`), because a job that is down will not file a report
about itself - an observer inside the object does not report the object's
death. If `прогоны` itself starts failing, nothing here will say so; that
needs an outside eye, like the watchdog ring above.
- **The window is 24 hours, so a job that runs less than three times a day can
never accumulate three consecutive failures inside it.** `отметка`,
`закрытие`, `голосовой` (daily) and `память` (weekly) are therefore
unwatched by this job in practice; `вайбграм`, `логи`, `куратор`, `досуг`,
`ротация` and `сторож` are inside it. Widening the counting window past the
query window is the fix, when it is worth it.
One blind spot, deliberate: **the watchdog cannot report its own death.** Its
own runs are excluded from both the failure count and the silence check
(`run.job.name`), because a job that is down will not file a report about
itself - an observer inside the object does not report the object's death. If
`прогоны` itself starts failing or stops running, nothing here will say so;
that needs an outside eye, like the watchdog ring above.
## Policy (`beaver_agent/policy.py`)