diff --git a/REPORTS.md b/REPORTS.md index a96cf61..be36d17 100644 --- a/REPORTS.md +++ b/REPORTS.md @@ -45,3 +45,76 @@ noted otherwise. import so DB records can be traced back to original files. - Do not retire blocklist exports, RouterOS scripts, or source logs until their consumers and audit requirements have explicit replacements. + +## Database and API Design + +### Canonical Data + +Continue treating normalized domain tables as canonical for facts they +already represent: + +- `requests` for parsed request events. +- `ip_stats` for per-IP/per-scan aggregates. +- `scans` and `scans_sources` for run status, parameters, and source counts. +- `blocklist_entries`, `blocklist_reasons`, and `blocklist_events` for block + decisions and history. +- `firewall_snapshots`, `firewall_address_list_entries`, and + `firewall_rule_entries` for router state. + +Add a `report_artifacts` table only for report-specific summaries that cannot +be reconstructed from those canonical rows. Proposed columns: + +| Column | Purpose | +| --- | --- | +| `id` | Integer primary key. | +| `source` | `copyparty`, `npm`, `app`, or `mikrotik`. Keep as text because `log_sources` currently covers only request sources. | +| `scan_id` | Nullable link to the generating scan; legacy imports may not have a known scan. | +| `report_type` | Stable type such as `analyzer_summary`, `blocklist_change`, or `firewall_snapshot_summary`. | +| `generated_at` | Original generation timestamp when known; otherwise import time plus an explicit unknown/original-time flag in metadata. | +| `schema_version` | Version of the structured payload shape. | +| `payload_json` | Structured report-specific summary, not a duplicate of requests or raw logs. | +| `original_path` | Relative source path for provenance and legacy lookup. | +| `content_sha256` | Hash of imported source content for integrity and idempotent import. | +| `created_at` | Database import/creation timestamp. | + +Use a unique index on `(original_path, content_sha256)` for idempotent file +imports. Keep payload schemas type-specific and versioned. Do not copy +`requests.csv`, `ips.json`, firewall rows, or blocklists wholesale into +`payload_json`; query their canonical tables and generate exports from those +tables. If a rendered text report is retained as a database artifact, store +its report-specific structured inputs and render text on demand rather than +using opaque text as the only source of truth. + +### API Transition Contract + +- Add `GET /api/reports` with optional `source`, `type`, `since`, `until`, + `limit`, and opaque `cursor` filters. Return a stable list of artifact + metadata: `id`, `source`, `scan_id`, `type`, `generated_at`, and available + export formats. Enforce a bounded default and maximum page size. +- Add `GET /api/reports/` for the structured payload and + provenance metadata. Keep it separate from the existing + `GET /api/reports//` route to avoid changing that route's + meaning during migration. +- Add an explicit export route for supported report types and formats + (initially JSON and, where useful, CSV/TXT). Generate exports from canonical + tables and versioned artifact payloads; do not accept filesystem paths from + a database-backed export request. +- Preserve the current source/path routes as a read-only compatibility + fallback while files remain. The reports page should consume DB metadata + first, then offer legacy files when no imported artifact exists. Return + stable JSON errors for unknown IDs, unsupported formats, and unavailable + DB state. +- Keep ordering deterministic (`generated_at DESC`, then `id DESC`), validate + filter values, and use cursor pagination for large histories. Do not expose + absolute server paths in list responses. + +### Sensitive and Large Content + +Do not include raw request lines or full request histories in report payloads; +those belong in the dedicated log-storage design and require separate access +and retention controls. Keep high-volume CSV exports generated on demand. Treat +IP addresses, paths/query strings, user names, user agents, and firewall state +as sensitive operational data: the existing web server must remain +localhost-only, payloads should include only fields needed by the report, and +new endpoints must not expose raw log data by default. Backups and exported +files inherit the same access and retention requirements as the database.