From 03f3c65a58457eb71c369a5e98378bcbb6f0d2ad Mon Sep 17 00:00:00 2001 From: Zwitschi Date: Fri, 6 Nov 2026 12:08:05 +0100 Subject: [PATCH] docs: inventory report artifacts and consumers --- REPORTS.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 REPORTS.md diff --git a/REPORTS.md b/REPORTS.md new file mode 100644 index 0000000..a96cf61 --- /dev/null +++ b/REPORTS.md @@ -0,0 +1,47 @@ +# Report Inventory + +This inventory records the current report surface before moving report reads +or writes into SQLite. Generated paths are rooted at `REPORT_DIR` (normally +`reports/`). The SQLite tables listed below already exist in schema v1 unless +noted otherwise. + +## Artifacts + +| Area | Producer and formats | Current role and consumers | Classification | +| --- | --- | --- | --- | +| `copyparty/` | `Analyzer.write_report()` creates timestamped `.txt`; `write_data()` creates per-IP CSV/JSON, denied-path CSV, flagged/banned IP TXT, `requests.csv`, `authed_users.json`, and `mikrotik-block.rsc`. | Reports API serves top-level text reports. `requests.csv` is imported by `app.db.backfill`; per-run data supports inspection and legacy workflows. Retention prunes generated files. | Request rows and per-scan IP stats are canonical in `requests` and `ip_stats`; rendered summaries and exports are derived. | +| `npm/` | Same analyzer outputs, plus user-agent and host summary CSV files. | Same consumers as copyparty. `requests.csv` is a legacy backfill input; retention prunes generated files. | Request rows and per-scan IP stats are canonical in `requests` and `ip_stats`; summaries and exports are derived. | +| `app/data/` | Consolidated scan writes `blocklist_ipv4.txt` and `blocklist_ipv6.txt`. | Blocklist UI/API reads these files; scan history reconstruction and firewall sync use them as current/compatibility inputs. Retention explicitly protects them. | Operational compatibility exports. Canonical blocked targets and events belong in `blocklist_entries` and `blocklist_events`; keep files until every consumer uses the DB and git-history contract is replaced. | +| `app/` timestamped JSON | `write_blocklist_report()` emits scan/firewall change reports when the router is reachable. | Human-readable audit trail containing run delta, cumulative targets, router changes, dry-run state, and errors; retained under report retention. | Derived audit artifact. Preserve structured fields and source scan association in the DB before considering file retirement. | +| `mikrotik/` | `MikrotikFirewall.save_state_report()` writes timestamped JSON snapshots and report data; latest state is also written to `data/mikrotik/*.json`. | Firewall UI/API displays current state and snapshot history. `app.db.backfill` currently imports only the latest `data/mikrotik` snapshot; runtime snapshot persistence uses `firewall_snapshots`, `firewall_address_list_entries`, and `firewall_rule_entries`. | Firewall rows and snapshots are canonical in the DB once persisted. Keep JSON exports during migration and retain any git-diff inputs required by deployment/audit. | + +## Consumers + +- `GET /api/reports/` lists top-level files in the selected report + directory; `GET /api/reports//` reads a file below it. +- The reports page consumes those endpoints. Scan status and dashboard helpers + select the newest source and MikroTik report files for display. +- `/api/blocklist` reads the consolidated IPv4/IPv6 text files. Scan history + reconstruction and firewall sync also consume those files alongside the + SQLite ledger and git history. +- RouterOS `.rsc` exports and firewall synchronization consume generated + targets; these are operational outputs, not disposable presentation files. +- `app.db.backfill` imports per-source `requests.csv`, blocklist history, and + the latest common MikroTik JSON snapshot. It does not currently import all + timestamped reports or all historical MikroTik snapshots. +- Report retention removes old generated artifacts after consolidated scans, + subject to configured age/file-count limits; the consolidated blocklist + files are protected. + +## Restructuring Implications + +- Keep `requests`, `ip_stats`, blocklist tables, and firewall snapshot tables + as the queryable source of truth for facts they already represent. Avoid + copying those facts into a generic report blob. +- A future report-artifact record should store metadata and structured fields + that have no canonical home yet. Rendered text/CSV/JSON should be generated + from those records or remain an explicit export. +- Preserve source paths, generated timestamps, scan IDs, and hashes during + import so DB records can be traced back to original files. +- Do not retire blocklist exports, RouterOS scripts, or source logs until + their consumers and audit requirements have explicit replacements.