diff --git a/README.md b/README.md index bfaee33..48ffed7 100644 --- a/README.md +++ b/README.md @@ -214,29 +214,36 @@ time and is never trimmed automatically. it stops appearing in recent logs (for example after it leaves the NPM ~30-day log-retention window, or once WHOIS subnet consolidation groups it into a larger range). -- The **source of truth** for "all blocked IPs" is the git history of the - consolidated blocklist files `reports/app/data/blocklist_ipv4.txt` and - `blocklist_ipv6.txt` (these files are git-tracked). After each scan the - updated blocklist files should be committed so the history keeps growing. +- The **source of truth** for "all blocked IPs" is the **SQLite blocklist + ledger** (`app.db` → `blocklist_entries` / `blocklist_events`), not git + history. Each scan syncs the desired blocklist into the ledger; the router's + cumulative state is derived from it. `app/utils/blocklist_history.py` is a + read-only fallback only. - The sync only ever **adds** entries. The only automatic removals are router - entries that were never blocked by the system — i.e. absent from every git - version of the blocklist files (manual / rogue entries). + entries that were never blocked by the system — i.e. absent from the ledger + (manual / rogue entries). - To explicitly unblock an IP, remove it manually via the web UI (`/blocklist` → remove) or directly on the router; the automated sync will not re-add entries that are not part of a scan. +> **Deployment note:** runtime data (blocklists, reports, the DB) is **never +> committed to git**. Keeping it untracked is what makes `git pull` updates +> conflict-free — see [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md) +> and the `deploy/` scripts. + Each consolidated scan with `--update-firewall` also writes a timestamped change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list of entries added / removed (complete-set delta), the full cumulative blocked -list, and the router sync changes. +list, and the router sync changes. It also persists a firewall snapshot into +the `firewall_snapshots` SQLite tables for long-term audit (F001). ### Report retention Generated files under `reports/` are pruned automatically after each consolidated scan: every subdirectory keeps at most `REPORT_RETENTION_MAX_FILES` files (default 10), all newer than -`REPORT_RETENTION_DAYS` days (default 30). The blocklist source of truth -(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) is always +`REPORT_RETENTION_DAYS` days (default 30). The blocklist files +(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) are always protected and never pruned. Tune via `REPORT_RETENTION_DAYS` / `REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without deleting. @@ -268,6 +275,35 @@ from the UI — applying changes is an explicit action. The main JSON endpoints `/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST client. +## Deployment & updating (`git pull`) + +The app runs scans continuously (web UI + cron + CLI), and all of it is +intended to be updated on the server with a plain `git pull`. To make that +**safe** — i.e. never conflict with a file a scan wrote since the last pull — +runtime data is strictly **never committed to git**: + +- `app/data/*`, `app/reports/*` and the SQLite DB (`app/data/ids.db`) are +gitignored. The only tracked file under `data/` is the read-only +`app/data/paloalto/scanner-ips.txt` seed. +- The Cloudflare `ips-*.txt` whitelist seeds are **not** tracked: they are +runtime-refreshed downloads and are re-fetched if absent (built-in ranges are +the fallback). +- History that must survive — the cumulative blocklist and firewall audit — +lives in the SQLite store, not in git revisions. + +Update a deployed instance with: + +```sh +sudo deploy/deploy.sh +``` + +See [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md) for the full plan, +and `deploy/` for the scripts (`deploy.sh`, `restart-service.sh`, +`check_runtime_git.sh`, `ids.service` systemd template). The wrapper +fast-forwards only (`git pull --ff-only`), refuses to run on an unclean tree, +reinstalls deps if `requirements.txt` changed, applies DB migrations +(`python -m app.db --migrate`), and restarts the `ids` unit. + ## Output Each script writes to its own report directory. `app.py` also writes @@ -318,9 +354,11 @@ reports/ ``` The current router state (address lists + firewall rules) is also mirrored to -`data/mikrotik/` on every `--update-firewall` run. Those files are overwritten -each run and are **git-visible**, so `git diff` shows how the firewall rules -and address lists change over time. +`data/mikrotik/` on every `--update-firewall` run as a scratch pointer to the +latest snapshot. Those files are overwritten each run and are **not** tracked; +the long-term audit trail is the SQLite `firewall_snapshots` tables plus the +timestamped reports under `reports/mikrotik/`. `git pull` is therefore never +blocked by firewall-state files. ## Detection diff --git a/deploy/check_runtime_git.sh b/deploy/check_runtime_git.sh index e1c951e..fdd80df 100644 --- a/deploy/check_runtime_git.sh +++ b/deploy/check_runtime_git.sh @@ -40,7 +40,7 @@ fail=0 # changes), so unborn commits don't cause false failures. allowed="app/data/paloalto/scanner-ips.txt" tracked=$(git ls-files app/data app/reports) -problem=$(printf '%s\n' "$tracked" | grep -v '^'"/\$allowed"'$' || true) +problem=$(printf '%s\n' "$tracked" | grep -xvF "$allowed" || true) if [[ -n "$problem" ]]; then echo "FAIL: unexpected tracked files under runtime dirs:" >&2 echo "$problem" >&2