docs(readme): source of truth is SQLite; add Deployment & updating section

- Blocklist source of truth is the SQLite ledger, not git history.
- data/mikrotik is a scratch mirror, not a git audit trail.
- Removed the 'commit blocklist files' advice.
- Added 'Deployment & updating (git pull)' section covering deploy.sh,
  runtime isolation, and python -m app.db --migrate (D008).
This commit is contained in:
zwitschi committed 2026-10-24 19:51:03 +02:00
1 parent 5d81f617f0
commit 1e80be746e
2 files changed
+51 -13

No files matched your search

+50 -12
View File
@@ -214,29 +214,36 @@ time and is never trimmed automatically.
it stops appearing in recent logs (for example after it leaves the NPM
~30-day log-retention window, or once WHOIS subnet consolidation groups it
into a larger range).
- The **source of truth** for "all blocked IPs" is the git history of the
consolidated blocklist files `reports/app/data/blocklist_ipv4.txt` and
`blocklist_ipv6.txt` (these files are git-tracked). After each scan the
updated blocklist files should be committed so the history keeps growing.
- The **source of truth** for "all blocked IPs" is the **SQLite blocklist
ledger** (`app.db` → `blocklist_entries` / `blocklist_events`), not git
history. Each scan syncs the desired blocklist into the ledger; the router's
cumulative state is derived from it. `app/utils/blocklist_history.py` is a
read-only fallback only.
- The sync only ever **adds** entries. The only automatic removals are router
entries that were never blocked by the system — i.e. absent from every git
version of the blocklist files (manual / rogue entries).
entries that were never blocked by the system — i.e. absent from the ledger
(manual / rogue entries).
- To explicitly unblock an IP, remove it manually via the web UI
(`/blocklist` → remove) or directly on the router; the automated sync will
not re-add entries that are not part of a scan.
> **Deployment note:** runtime data (blocklists, reports, the DB) is **never
> committed to git**. Keeping it untracked is what makes `git pull` updates
> conflict-free — see [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md)
> and the `deploy/` scripts.
Each consolidated scan with `--update-firewall` also writes a timestamped
change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list
of entries added / removed (complete-set delta), the full cumulative blocked
list, and the router sync changes.
list, and the router sync changes. It also persists a firewall snapshot into
the `firewall_snapshots` SQLite tables for long-term audit (F001).
### Report retention
Generated files under `reports/` are pruned automatically after each
consolidated scan: every subdirectory keeps at most
`REPORT_RETENTION_MAX_FILES` files (default 10), all newer than
`REPORT_RETENTION_DAYS` days (default 30). The blocklist source of truth
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) is always
`REPORT_RETENTION_DAYS` days (default 30). The blocklist files
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) are always
protected and never pruned. Tune via `REPORT_RETENTION_DAYS` /
`REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without
deleting.
@@ -268,6 +275,35 @@ from the UI — applying changes is an explicit action. The main JSON endpoints
`/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST
client.
## Deployment & updating (`git pull`)
The app runs scans continuously (web UI + cron + CLI), and all of it is
intended to be updated on the server with a plain `git pull`. To make that
**safe** — i.e. never conflict with a file a scan wrote since the last pull —
runtime data is strictly **never committed to git**:
- `app/data/*`, `app/reports/*` and the SQLite DB (`app/data/ids.db`) are
gitignored. The only tracked file under `data/` is the read-only
`app/data/paloalto/scanner-ips.txt` seed.
- The Cloudflare `ips-*.txt` whitelist seeds are **not** tracked: they are
runtime-refreshed downloads and are re-fetched if absent (built-in ranges are
the fallback).
- History that must survive — the cumulative blocklist and firewall audit —
lives in the SQLite store, not in git revisions.
Update a deployed instance with:
```sh
sudo deploy/deploy.sh
```
See [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md) for the full plan,
and `deploy/` for the scripts (`deploy.sh`, `restart-service.sh`,
`check_runtime_git.sh`, `ids.service` systemd template). The wrapper
fast-forwards only (`git pull --ff-only`), refuses to run on an unclean tree,
reinstalls deps if `requirements.txt` changed, applies DB migrations
(`python -m app.db --migrate`), and restarts the `ids` unit.
## Output
Each script writes to its own report directory. `app.py` also writes
@@ -318,9 +354,11 @@ reports/
```
The current router state (address lists + firewall rules) is also mirrored to
`data/mikrotik/` on every `--update-firewall` run. Those files are overwritten
each run and are **git-visible**, so `git diff` shows how the firewall rules
and address lists change over time.
`data/mikrotik/` on every `--update-firewall` run as a scratch pointer to the
latest snapshot. Those files are overwritten each run and are **not** tracked;
the long-term audit trail is the SQLite `firewall_snapshots` tables plus the
timestamped reports under `reports/mikrotik/`. `git pull` is therefore never
blocked by firewall-state files.
## Detection
+1 -1
View File
@@ -40,7 +40,7 @@ fail=0
# changes), so unborn commits don't cause false failures.
allowed="app/data/paloalto/scanner-ips.txt"
tracked=$(git ls-files app/data app/reports)
problem=$(printf '%s\n' "$tracked" | grep -v '^'"/\$allowed"'$' || true)
problem=$(printf '%s\n' "$tracked" | grep -xvF "$allowed" || true)
if [[ -n "$problem" ]]; then
echo "FAIL: unexpected tracked files under runtime dirs:" >&2
echo "$problem" >&2