zwitschi/intrusion-detection
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
intrusion-detection/README.md
T
zwitschi 6051d2b548 test(db): repository + migration tests (T001); document analyze in README 
tests/test_repositories.py covers schema migrations (idempotent), scan
lifecycle, request insert/query/cross-source, grow-only blocklist ledger,
unblock, whitelist lookup/replace, and firewall snapshots. 10 tests total.
README usage documents the 'python app.py analyze' command.
2026-10-24 19:56:22 +02:00

462 lines
24 KiB
Markdown
Raw Permalink Blame History

# All You Can GET Intrusion Detection System
Local-development intrusion detection / log analysis for two log sources:
| Script | Source | Detection rule |
| --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `app.py` | unified entry point | runs both analyzers + optional firewall sync |
| `app/scan/` | consolidated scanner | runs both analyzers, merges blocklists, syncs firewall |
| `app/copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `app/npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `app/utils/` | shared utilities | constants, I/O helpers, IP utils, models, reporting, whitelists, WHOIS |
| `app/web/` | Flask web UI | app factory, routes, background scan runner |
| `app/analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write |
| `app/flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `app/mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | RouterOS SSH client | `MikrotikFirewall` — connect, sync blocklist, audit state |
Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik
RouterOS firewall script.
`app.py` runs both analyzers, merges their blocklists, and optionally
synchronizes the MikroTik RouterOS firewall via SSH.
> Status: **local dev**. Runs offline against log files on disk. Use
> `--update-firewall` to push changes to the router, or `--dry-run` to
> preview without applying.
## Architecture
```text
intrusion-detection/
├── app.py unified entry point
├── app/ application package
│ ├── __init__.py
│ ├── analyzer.py Analyzer pipeline base class
│ ├── bad_patterns.py malicious URL/file regex patterns
│ ├── copyparty.py copyparty log adapter (~150 lines)
│ ├── firewall.py MikroTik RouterOS SSH client
│ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
│ ├── mikrotik.py RouterOS .rsc script generation
│ ├── npm.py NPM proxy log adapter (~150 lines)
│ ├── web/ Flask web UI
│ │ ├── __init__.py app factory + helpers + serve()
│ │ ├── routes.py all HTTP route definitions
│ │ └── web_scan.py background scan runner + in-memory job store
│ ├── scan/ scan pipeline
│ │ ├── __init__.py consolidated scanner (run_scan, ScanParams, ScanResult)
│ │ └── scan_state.py incremental log scan state (load/save)
│ ├── db/ SQLite store (see .github/instructions/DATABASE.md)
│ │ ├── __init__.py connection factory + migrations runner
│ │ ├── schema.py DDL for all tables (PRAGMA user_version migrations)
│ │ └── repositories.py queries: logs, scans, blocklist, whitelist, firewall
│ ├── utils/ shared utilities
│ │ ├── constants.py all config constants (paths, env vars, thresholds)
│ │ ├── files.py read_lines, find_logs, write_csv/json/lines
│ │ ├── ip_utils.py IP parsing, whitelist lookups, sorting
│ │ ├── models.py Request, IPStats, aggregate()
│ │ ├── reporting.py Column, render_table, render_kv, timestamp_name
│ │ ├── utils.py shared CLI arg parser
│ │ ├── whitelists.py provider CIDR constants, local/remote load
│ │ ├── whois.py query_whois_subnet, consolidate_by_subnet
│ │ ├── blocklist_history.py git-backed cumulative blocklist
│ │ └── retention.py report file retention & pruning
│ ├── static/ style.css + app.js (dark-theme UI)
│ └── templates/ Jinja2 page templates (dashboard, scan, reports, …)
├── data/ whitelist source files (cached locally)
│ ├── init7/ (ranges built into app/utils/whitelists.py — no geofeed)
│ ├── cloudflare/
│ │ ├── ips-v4.txt
│ │ └── ips-v6.txt
│ ├── paloalto/
│ │ └── scanner-ips.txt
│ └── ids.db local SQLite store (log/scan/blocklist/whitelist history; auto-created)
├── reports/ generated reports (per-run + per-source)
│ ├── copyparty/
│ ├── npm/
│ └── mikrotik/ firewall audit snapshots
└── README.md
```
`app/copyparty.py` and `app/npm.py` are thin adapters. Each supplies a log
matcher, a line parser, a flagging rule, and column definitions.
Core responsibilities by module:
| Module | Owns |
| -------------------------------- | -------------------------------------------------------------------------------- |
| `app/utils/constants.py` | all config constants — paths, env vars, network/scan thresholds |
| `app/utils/models.py` | `Request`, `IPStats`, `aggregate()` |
| `app/utils/ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `app/utils/whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `app/bad_patterns.py` | malicious URL/file regex classification |
| `app/utils/whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | `MikrotikFirewall` — SSH connect, sync blocklist, audit state |
| `app/web/` | Flask app factory, JSON API routes, page rendering, background scan runner |
| `app/scan/` | consolidated scan pipeline, incremental scan state |
| `app/utils/files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `app/utils/reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
| `app/utils/blocklist_history.py` | git-backed cumulative blocklist reconstruction |
| `app/utils/retention.py` | report file retention & pruning |
Console output and saved reports are the **same string** (`Analyzer.text_report()`).
## Requirements
- Python 3.10+
- `paramiko` (MikroTik SSH), `python-dotenv` (`.env` loading), `flask` (web UI)
```powershell
pip install -r requirements.txt
```
## Configuration
All tunable settings live in `app/constants.py` and can be overridden per
deployment via environment variables. For local development, copy
`.env.example` to `.env` and fill in your values:
```powershell
copy .env.example .env
```
`app/constants.py` loads `.env` with `python-dotenv` at startup. Real system
environment variables always take precedence over `.env` values, so
production can set the same variables via the OS and skip `.env` entirely.
`.env` is git-ignored — do not commit secrets to it.
| Env var | Default | Used by |
| -------------------- | ------------------------ | --------------------- |
| `COPYPARTY_LOG_PATH` | `P:\copyparty` | copyparty scanner |
| `NPM_LOG_PATH` | `Y:\npm\logs` | npm scanner |
| `ROUTER_IP` | `192.168.88.1` | MikroTik connection |
| `ROUTER_USER` | `admin` | MikroTik connection |
| `ROUTER_PASSWORD` | _(set in .env)_ | MikroTik connection |
| `BLOCKLIST_NAME` | `Blocked` | MikroTik address list |
| `WHITELIST_NAMES` | `Cloudflare,Init7` | MikroTik whitelists |
| `WEB_HOST` | `127.0.0.1` | web UI bind host |
| `WEB_PORT` | `54666` | web UI port |
See `.env.example` for the complete list of supported variables (WHOIS
tuning, scan thresholds, whitelist source URLs, and more).
### Network context (2026-09-08)
Connectivity provider switched Openfactory → Init7 on 2026-09-08:
- WAN IPv4: `77.74.80.159/32` (init7 `77.74.80.0/24`, "Hybrid7-NET").
- IPv6: delegated `/48` `2001:1620:549f::/48` (init7 allocation `2001:1620::/30`).
- Upstream DNS: `77.109.128.2`, `213.144.129.20`.
The `Openfactory` whitelist (old AS58299 geofeed) is obsolete and must be
replaced with an `Init7` whitelist — see
[`.github/instructions/TODO.md`](.github/instructions/TODO.md) for the
switchover plan and the verified init7 CIDR ranges.
## Usage
```powershell
# web UI (default entry point)
python app.py
python app.py --web-port 54666
# consolidated CLI scan (scan subcommand)
python app.py scan
python app.py scan --quiet
python app.py scan --update-firewall --dry-run
python app.py scan --update-firewall
# detailed log-analysis report from the SQLite store (needs scans run first)
python app.py analyze # top IPs + cross-source + blocklist
python app.py analyze --ip 1.2.3.4 # drill into one IP's request history
python app.py analyze --json # machine-readable output
# individual scanners (still runnable directly)
python -m app.copyparty
python -m app.npm
```
All scripts accept CLI overrides:
| Flag | Default | Meaning |
| ------------------- | -------------------- | ---------------------------------- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
| `--update-firewall` | off | sync blocklist to MikroTik via SSH |
| `--dry-run` | off | preview firewall changes, no apply |
| `--web-host` | `127.0.0.1` | web UI bind host |
| `--web-port` | `54666` | web UI port |
`python app.py scan` accepts all flags from both scanners plus
`--copyparty-log-path`, `--copyparty-report-dir`, `--npm-log-path`,
`--npm-report-dir`, and `--report-dir` for the consolidated output.
Examples:
```powershell
python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python -m app.copyparty --top 100
python app.py scan --quiet --update-firewall --dry-run
```
## Blocklist behavior (cumulative / grow-only)
The MikroTik `Blocked` address list is **cumulative**: it only grows over
time and is never trimmed automatically.
- An IP (or subnet) that the system has ever blocked stays blocked — even if
it stops appearing in recent logs (for example after it leaves the NPM
~30-day log-retention window, or once WHOIS subnet consolidation groups it
into a larger range).
- The **source of truth** for "all blocked IPs" is the **SQLite blocklist
ledger** (`app.db` → `blocklist_entries` / `blocklist_events`), not git
history. Each scan syncs the desired blocklist into the ledger; the router's
cumulative state is derived from it. `app/utils/blocklist_history.py` is a
read-only fallback only.
- The sync only ever **adds** entries. The only automatic removals are router
entries that were never blocked by the system — i.e. absent from the ledger
(manual / rogue entries).
- To explicitly unblock an IP, remove it manually via the web UI
(`/blocklist` → remove) or directly on the router; the automated sync will
not re-add entries that are not part of a scan.
> **Deployment note:** runtime data (blocklists, reports, the DB) is **never
> committed to git**. Keeping it untracked is what makes `git pull` updates
> conflict-free — see [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md)
> and the `deploy/` scripts.
Each consolidated scan with `--update-firewall` also writes a timestamped
change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list
of entries added / removed (complete-set delta), the full cumulative blocked
list, and the router sync changes. It also persists a firewall snapshot into
the `firewall_snapshots` SQLite tables for long-term audit (F001).
### Report retention
Generated files under `reports/` are pruned automatically after each
consolidated scan: every subdirectory keeps at most
`REPORT_RETENTION_MAX_FILES` files (default 10), all newer than
`REPORT_RETENTION_DAYS` days (default 30). The blocklist files
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) are always
protected and never pruned. Tune via `REPORT_RETENTION_DAYS` /
`REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without
deleting.
## Web UI
`python app.py` starts a **localhost-only** Flask web UI (default
`http://127.0.0.1:54666`) for running scans and managing the MikroTik
blocklist / firewall. It is bound to `127.0.0.1` — do **not** expose
`WEB_HOST` beyond localhost.
Pages:
| Route | Purpose |
| ------------ | ----------------------------------------------------- |
| `/` | dashboard — status cards + latest scans |
| `/scan` | run copyparty / npm / consolidated scans (background) |
| `/reports` | browse reports per source |
| `/blocklist` | view the consolidated blocklist, add/remove entries |
| `/firewall` | view address lists + rules, preview/apply a sync |
| `/whitelist` | view/manage provider whitelist CIDRs |
| `/graph` | visualize the 0.0.0.0/8 address space (blocked/whitelisted pixels) |
| `/analysis` | detailed persisted-log analysis (top IPs, drill-down, cross-source) |
Scans run in a background thread (only one at a time) and report progress via
the JSON API, so the UI never blocks. Firewall sync is **dry-run by default**
from the UI — applying changes is an explicit action. The main JSON endpoints
(`/api/status`, `/api/scan/*`, `/api/reports/*`, `/api/blocklist`,
`/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST
client.
## Deployment & updating (`git pull`)
The app runs scans continuously (web UI + cron + CLI), and all of it is
intended to be updated on the server with a plain `git pull`. To make that
**safe** — i.e. never conflict with a file a scan wrote since the last pull —
runtime data is strictly **never committed to git**:
- `app/data/*`, `app/reports/*` and the SQLite DB (`app/data/ids.db`) are
gitignored. The only tracked file under `data/` is the read-only
`app/data/paloalto/scanner-ips.txt` seed.
- The Cloudflare `ips-*.txt` whitelist seeds are **not** tracked: they are
runtime-refreshed downloads and are re-fetched if absent (built-in ranges are
the fallback).
- History that must survive — the cumulative blocklist and firewall audit —
lives in the SQLite store, not in git revisions.
Update a deployed instance with:
```sh
sudo deploy/deploy.sh
```
See [`DEPLOYMENT.md`](.github/instructions/DEPLOYMENT.md) for the full plan,
and `deploy/` for the scripts (`deploy.sh`, `restart-service.sh`,
`check_runtime_git.sh`, `ids.service` systemd template). The wrapper
fast-forwards only (`git pull --ff-only`), refuses to run on an unclean tree,
reinstalls deps if `requirements.txt` changed, applies DB migrations
(`python -m app.db --migrate`), and restarts the `ids` unit.
## Output
Each script writes to its own report directory. `app.py` also writes
consolidated blocklists and firewall audit snapshots:
```text
reports/
├── copyparty/
│ ├── data/
│ │ ├── ips_ipv4.csv per-IP: total, denied, banned, users
│ │ ├── ips_ipv6.csv
│ │ ├── ips.json full per-IP detail w/ malicious hits
│ │ ├── denied_paths.csv denied paths, hit counts
│ │ ├── denied_ips.txt flagged IPv4 + metadata
│ │ ├── denied_ipv6.txt
│ │ ├── banned_ips.txt copyparty auto-banned IPv4
│ │ ├── banned_ipv6.txt
│ │ ├── blocked_ipv4.txt final block set (non-whitelisted)
│ │ ├── blocked_ipv6.txt
│ │ ├── authed_users.json authenticated user counts
│ │ └── requests.csv every parsed request
│ └── 2026-08-02-12-00-00.txt timestamped report
├── npm/
│ ├── data/
│ │ ├── ips_ipv4.csv per-IP: total, 4xx, 5xx, ratio, methods
│ │ ├── ips_ipv6.csv
│ │ ├── ips.json
│ │ ├── error_paths.csv
│ │ ├── scanner_ips.txt flagged IPv4 + metadata
│ │ ├── scanner_ipv6.txt
│ │ ├── blocked_ipv4.txt
│ │ ├── blocked_ipv6.txt
│ │ ├── ua_summary.csv user agent breakdown
│ │ ├── host_summary.csv virtual host breakdown
│ │ └── requests.csv
│ └── 2026-08-02-12-00-00.txt
├── app/
│ └── data/
│ ├── blocklist_ipv4.txt final consolidated IPv4 blocklist (CIDRs)
│ └── blocklist_ipv6.txt final consolidated IPv6 blocklist (CIDRs)
└── mikrotik/
├── data/
│ ├── address_list_ipv4.json
│ ├── address_list_ipv6.json
│ ├── firewall_ipv4.json
│ └── firewall_ipv6.json
└── 2026-08-04-18-30-28.json firewall audit snapshot
```
The current router state (address lists + firewall rules) is also mirrored to
`data/mikrotik/` on every `--update-firewall` run as a scratch pointer to the
latest snapshot. Those files are overwritten each run and are **not** tracked;
the long-term audit trail is the SQLite `firewall_snapshots` tables plus the
timestamped reports under `reports/mikrotik/`. `git pull` is therefore never
blocked by firewall-state files.
## Detection
### Three detection layers
Blocklist candidates come from three independent sources, merged at scan time:
1. **Error-rate flagging** (`npm.py`) — IPs with many 4xx/5xx responses and a
high error ratio:
$$\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5$$
2. **Denied-only flagging** (`copyparty.py`) — IPs that hit denied paths and
never authenticated:
$$\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}$$
3. **Malicious-pattern flagging** (both analyzers) — IPs whose request paths
match known probe/attack signatures (`.env`, `.git/config`, WordPress
probes, webshell filenames, etc.) at severity ≥ 2.
### Pattern categories (`bad_patterns.py`)
| Pattern | Severity | Matches |
| ------------------- | ------------ | ------------------------------------------------------------------- |
| `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` |
| `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` |
| `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` |
| `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
| `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` |
| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files |
| `well_known_probe` | 1 (moderate) | `.well-known/` paths |
Flagging metadata fields:
- `errors` / `denied` — hostile hit count
- `total` — all requests from that IP
- `ratio` — error ratio (npm only)
- `first` / `last` — first and last timestamp seen
- `malicious_hits` — count of malicious-pattern request paths
## Whitelisted ranges
Whitelists live in `whitelists.py` and `data/`. Loading priority per provider:
1. **Local files** in `data/<provider>/` (offline-safe, committed to repo)
2. **Remote download** (fetches latest, saves to local file on success)
3. **Built-in fallback** (hardcoded constants, always available)
Each layer overrides the previous. Known-good ranges are emitted as
`accept` rules placed **above** the `drop` rules in the MikroTik script.
### Subnet consolidation
Blocked IPs are grouped by WHOIS subnet (via `whois.radb.net`). If ≥ 3
malicious IPs share a subnet, the entire subnet is blocked instead of
individual `/32` or `/128` entries. Results are cached to
`reports/whois_cache.json`.
## Log format reference
### copyparty
```log
15:55:47.466 34.105.30.251 37140 LOG: GET //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140 LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140 CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144 BTW: banned for 86399 sec
16:00:34.385 hsrv LOG: ok bye
```
### Nginx Proxy Manager
```log
[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"
```
Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
`"path"`, `[Client IP]`, `[Length N]`, `[Gzip X]`, `[Sent-to IP]`, `"UA"`,
`"Referer"`.
## Extending
- **new whitelist provider** → add URLs, local file paths, and fallback CIDRs
to `app/whitelists.py`
- **new detection pattern** → add a `BadPattern` to `app/bad_patterns.py`
- **new detection rule** → add a `flag_*()` function to `app/flags.py`, return it
from the analyzer's `flag()` method
- **new log source** → subclass `app.analyzer.Analyzer`, implement `parse_line()` +
`flag()`, declare `ip_columns` / `request_columns`
- **new output file** → override `write_extra(data_dir)` and return the paths
written
- **new I/O format** → add writer to `app/files.py`
- **new report section** → add rendering logic to `app/reporting.py`
Reference in new issue View Git Blame Copy Permalink