docs(readme): source of truth is SQLite; add Deployment & updating section
- Blocklist source of truth is the SQLite ledger, not git history. - data/mikrotik is a scratch mirror, not a git audit trail. - Removed the 'commit blocklist files' advice. - Added 'Deployment & updating (git pull)' section covering deploy.sh, runtime isolation, and python -m app.db --migrate (D008).
This commit is contained in:
1 parent
5d81f617f0
commit
1e80be746e
2 files changed
+51
-13
No files matched your search
@@ -214,29 +214,36 @@ time and is never trimmed automatically.
|
|||||||
it stops appearing in recent logs (for example after it leaves the NPM
|
it stops appearing in recent logs (for example after it leaves the NPM
|
||||||
~30-day log-retention window, or once WHOIS subnet consolidation groups it
|
~30-day log-retention window, or once WHOIS subnet consolidation groups it
|
||||||
into a larger range).
|
into a larger range).
|
||||||
- The **source of truth** for "all blocked IPs" is the git history of the
|
- The **source of truth** for "all blocked IPs" is the **SQLite blocklist
|
||||||
consolidated blocklist files `reports/app/data/blocklist_ipv4.txt` and
|
ledger** (`app.db` → `blocklist_entries` / `blocklist_events`), not git
|
||||||
`blocklist_ipv6.txt` (these files are git-tracked). After each scan the
|
history. Each scan syncs the desired blocklist into the ledger; the router's
|
||||||
updated blocklist files should be committed so the history keeps growing.
|
cumulative state is derived from it. `app/utils/blocklist_history.py` is a
|
||||||
|
read-only fallback only.
|
||||||
- The sync only ever **adds** entries. The only automatic removals are router
|
- The sync only ever **adds** entries. The only automatic removals are router
|
||||||
entries that were never blocked by the system — i.e. absent from every git
|
entries that were never blocked by the system — i.e. absent from the ledger
|
||||||
version of the blocklist files (manual / rogue entries).
|
(manual / rogue entries).
|
||||||
- To explicitly unblock an IP, remove it manually via the web UI
|
- To explicitly unblock an IP, remove it manually via the web UI
|
||||||
(`/blocklist` → remove) or directly on the router; the automated sync will
|
(`/blocklist` → remove) or directly on the router; the automated sync will
|
||||||
not re-add entries that are not part of a scan.
|
not re-add entries that are not part of a scan.
|
||||||
|
|
||||||
|
> **Deployment note:** runtime data (blocklists, reports, the DB) is **never
|
||||||
|
> committed to git**. Keeping it untracked is what makes `git pull` updates
|
||||||
|
> conflict-free — see [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md)
|
||||||
|
> and the `deploy/` scripts.
|
||||||
|
|
||||||
Each consolidated scan with `--update-firewall` also writes a timestamped
|
Each consolidated scan with `--update-firewall` also writes a timestamped
|
||||||
change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list
|
change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list
|
||||||
of entries added / removed (complete-set delta), the full cumulative blocked
|
of entries added / removed (complete-set delta), the full cumulative blocked
|
||||||
list, and the router sync changes.
|
list, and the router sync changes. It also persists a firewall snapshot into
|
||||||
|
the `firewall_snapshots` SQLite tables for long-term audit (F001).
|
||||||
|
|
||||||
### Report retention
|
### Report retention
|
||||||
|
|
||||||
Generated files under `reports/` are pruned automatically after each
|
Generated files under `reports/` are pruned automatically after each
|
||||||
consolidated scan: every subdirectory keeps at most
|
consolidated scan: every subdirectory keeps at most
|
||||||
`REPORT_RETENTION_MAX_FILES` files (default 10), all newer than
|
`REPORT_RETENTION_MAX_FILES` files (default 10), all newer than
|
||||||
`REPORT_RETENTION_DAYS` days (default 30). The blocklist source of truth
|
`REPORT_RETENTION_DAYS` days (default 30). The blocklist files
|
||||||
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) is always
|
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) are always
|
||||||
protected and never pruned. Tune via `REPORT_RETENTION_DAYS` /
|
protected and never pruned. Tune via `REPORT_RETENTION_DAYS` /
|
||||||
`REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without
|
`REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without
|
||||||
deleting.
|
deleting.
|
||||||
@@ -268,6 +275,35 @@ from the UI — applying changes is an explicit action. The main JSON endpoints
|
|||||||
`/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST
|
`/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST
|
||||||
client.
|
client.
|
||||||
|
|
||||||
|
## Deployment & updating (`git pull`)
|
||||||
|
|
||||||
|
The app runs scans continuously (web UI + cron + CLI), and all of it is
|
||||||
|
intended to be updated on the server with a plain `git pull`. To make that
|
||||||
|
**safe** — i.e. never conflict with a file a scan wrote since the last pull —
|
||||||
|
runtime data is strictly **never committed to git**:
|
||||||
|
|
||||||
|
- `app/data/*`, `app/reports/*` and the SQLite DB (`app/data/ids.db`) are
|
||||||
|
gitignored. The only tracked file under `data/` is the read-only
|
||||||
|
`app/data/paloalto/scanner-ips.txt` seed.
|
||||||
|
- The Cloudflare `ips-*.txt` whitelist seeds are **not** tracked: they are
|
||||||
|
runtime-refreshed downloads and are re-fetched if absent (built-in ranges are
|
||||||
|
the fallback).
|
||||||
|
- History that must survive — the cumulative blocklist and firewall audit —
|
||||||
|
lives in the SQLite store, not in git revisions.
|
||||||
|
|
||||||
|
Update a deployed instance with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
sudo deploy/deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
See [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md) for the full plan,
|
||||||
|
and `deploy/` for the scripts (`deploy.sh`, `restart-service.sh`,
|
||||||
|
`check_runtime_git.sh`, `ids.service` systemd template). The wrapper
|
||||||
|
fast-forwards only (`git pull --ff-only`), refuses to run on an unclean tree,
|
||||||
|
reinstalls deps if `requirements.txt` changed, applies DB migrations
|
||||||
|
(`python -m app.db --migrate`), and restarts the `ids` unit.
|
||||||
|
|
||||||
## Output
|
## Output
|
||||||
|
|
||||||
Each script writes to its own report directory. `app.py` also writes
|
Each script writes to its own report directory. `app.py` also writes
|
||||||
@@ -318,9 +354,11 @@ reports/
|
|||||||
```
|
```
|
||||||
|
|
||||||
The current router state (address lists + firewall rules) is also mirrored to
|
The current router state (address lists + firewall rules) is also mirrored to
|
||||||
`data/mikrotik/` on every `--update-firewall` run. Those files are overwritten
|
`data/mikrotik/` on every `--update-firewall` run as a scratch pointer to the
|
||||||
each run and are **git-visible**, so `git diff` shows how the firewall rules
|
latest snapshot. Those files are overwritten each run and are **not** tracked;
|
||||||
and address lists change over time.
|
the long-term audit trail is the SQLite `firewall_snapshots` tables plus the
|
||||||
|
timestamped reports under `reports/mikrotik/`. `git pull` is therefore never
|
||||||
|
blocked by firewall-state files.
|
||||||
|
|
||||||
## Detection
|
## Detection
|
||||||
|
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ fail=0
|
|||||||
# changes), so unborn commits don't cause false failures.
|
# changes), so unborn commits don't cause false failures.
|
||||||
allowed="app/data/paloalto/scanner-ips.txt"
|
allowed="app/data/paloalto/scanner-ips.txt"
|
||||||
tracked=$(git ls-files app/data app/reports)
|
tracked=$(git ls-files app/data app/reports)
|
||||||
problem=$(printf '%s\n' "$tracked" | grep -v '^'"/\$allowed"'$' || true)
|
problem=$(printf '%s\n' "$tracked" | grep -xvF "$allowed" || true)
|
||||||
if [[ -n "$problem" ]]; then
|
if [[ -n "$problem" ]]; then
|
||||||
echo "FAIL: unexpected tracked files under runtime dirs:" >&2
|
echo "FAIL: unexpected tracked files under runtime dirs:" >&2
|
||||||
echo "$problem" >&2
|
echo "$problem" >&2
|
||||||
|
|||||||
Reference in new issue
Block a user