Initial commit: intrusion detection system for copyparty and NPM proxy logs with MikroTik blocklist export

2026-08-03 16:34:45 +02:00
commit 15019de3d8
@@ -0,0 +1,246 @@
+# allucanget-ids
+
+Local-development intrusion detection / log analysis for two log sources:
+
+| Script | Source | Detection rule |
+| --- | --- | --- |
+| `copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
+| `npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
+| `scan.py` | both of the above | consolidated scan + unified MikroTik export |
+| `whitelists.py` | whitelist module | provider CIDR loading, refresh, validation |
+| `bad_patterns.py` | pattern detection | malicious URL/file regex classification |
+| `utils.py` | shared core (no CLI) | aggregation, reports, rsc, pipeline |
+
+Both per-source scripts parse logs, aggregate per-IP behaviour, split
+IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
+group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik
+RouterOS firewall script.
+
+`scan.py` runs both analyzers, merges their blocklists, and writes a single
+consolidated `.rsc`.
+
+> Status: **local dev**. Runs offline against log files on disk. It does not
+> talk to the router, does not run as a service, and blocks nothing by itself.
+> You review the generated `.rsc` and import it manually.
+
+## Architecture
+
+```
+allucanget-ids/
+├── scan.py              consolidated scanner entry point
+├── copyparty.py         copyparty log adapter (~150 lines)
+├── npm.py               NPM proxy log adapter (~150 lines)
+├── whitelists.py        whitelist constants, download, CIDR extraction
+├── bad_patterns.py      malicious URL/file regex patterns
+├── utils.py             shared core: aggregation, reporting, MikroTik export
+├── data/                whitelist source files (cached locally)
+│   ├── openfactory/
+│   │   └── geofeed.csv
+│   ├── cloudflare/
+│   │   ├── ips-v4.txt
+│   │   └── ips-v6.txt
+│   └── paloalto/
+│       └── scanner-ips.txt
+├── reports/             generated reports (per-run + per-source)
+│   ├── copyparty/
+│   ├── npm/
+│   └── mikrotik-block.rsc   consolidated .rsc (scan.py only)
+└── README.md
+```
+
+`copyparty.py` and `npm.py` are thin adapters. Each supplies a log matcher,
+a line parser, a flagging rule, and column definitions. Everything else lives in
+`utils.py`:
+
+- `Request` — normalised request record
+- `IPStats` + `aggregate()` — per-IP counters, malicious-hit tracking
+- `flag_scanners()` / `flag_denied_only()` / `flag_malicious_patterns()` — detection rules
+- `whitelist_of()` / `is_whitelisted()` — whitelist lookups (backed by `whitelists.py`)
+- `is_malicious()` / `classify_path()` — pattern classification (backed by `bad_patterns.py`)
+- `query_whois_subnet()` / `consolidate_by_subnet()` — WHOIS subnet grouping
+- IP version helpers, log discovery, transparent `.gz`, ANSI stripping
+- CSV/JSON/table writers, `mikrotik_script()` — RouterOS export
+- `Analyzer` — pipeline: discover → load → analyze → report → write
+
+Console output and saved reports are the **same string** (`Analyzer.text_report()`).
+
+## Requirements
+
+- Python 3.10+
+- stdlib only, no dependencies
+
+## Usage
+
+```powershell
+# individual scanners
+python copyparty.py
+python npm.py
+
+# consolidated (both sources, unified blocklist)
+python scan.py
+```
+
+All scripts accept CLI overrides:
+
+| Flag | Default | Meaning |
+| --- | --- | --- |
+| `--log-path` | see below | log directory |
+| `--report-dir` | `reports/<name>` | output directory |
+| `--top N` | `30` | rows in the top-IP table |
+| `--quiet` | off | write files, print only paths |
+| `--no-rsc` | off | skip MikroTik export |
+| `--min-errors N` | `5` (`npm.py`) | error count threshold |
+| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
+| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
+
+`scan.py` accepts all flags from both scanners plus `--copyparty-log-path`,
+`--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and
+`--report-dir` for the consolidated output.
+
+Default log paths (overridable by `.env`):
+
+| Script | Env var | Default |
+| --- | --- | --- |
+| `copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
+| `npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |
+
+Examples:
+
+```powershell
+python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
+python copyparty.py --top 100 --no-rsc
+python scan.py --quiet --no-rsc
+```
+
+## Output
+
+Each script writes to its own report directory. `scan.py` also writes a
+consolidated `.rsc`:
+
+```text
+reports/
+├── copyparty/
+│   ├── data/
+│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
+│   │   ├── ips_ipv6.csv
+│   │   ├── ips.json              full per-IP detail w/ malicious hits
+│   │   ├── denied_paths.csv      denied paths, hit counts
+│   │   ├── denied_ips.txt        flagged IPv4 + metadata
+│   │   ├── denied_ipv6.txt
+│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
+│   │   ├── banned_ipv6.txt
+│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
+│   │   ├── blocked_ipv6.txt
+│   │   ├── authed_users.json     authenticated user counts
+│   │   ├── requests.csv          every parsed request
+│   │   └── mikrotik-block.rsc
+│   └── 2026-08-02-12-00-00.txt   timestamped report
+├── npm/
+│   ├── data/
+│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
+│   │   ├── ips_ipv6.csv
+│   │   ├── ips.json
+│   │   ├── error_paths.csv
+│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
+│   │   ├── scanner_ipv6.txt
+│   │   ├── blocked_ipv4.txt
+│   │   ├── blocked_ipv6.txt
+│   │   ├── ua_summary.csv        user agent breakdown
+│   │   ├── host_summary.csv      virtual host breakdown
+│   │   ├── requests.csv
+│   │   └── mikrotik-block.rsc
+│   └── 2026-08-02-12-00-00.txt
+└── mikrotik-block.rsc             consolidated (scan.py only)
+```
+
+## Detection
+
+### Three detection layers
+
+Blocklist candidates come from three independent sources, merged at scan time:
+
+1. **Error-rate flagging** (`npm.py`) — IPs with many 4xx/5xx responses and a
+   high error ratio:
+
+   $$\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5$$
+
+2. **Denied-only flagging** (`copyparty.py`) — IPs that hit denied paths and
+   never authenticated:
+
+   $$\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}$$
+
+3. **Malicious-pattern flagging** (both analyzers) — IPs whose request paths
+   match known probe/attack signatures (`.env`, `.git/config`, WordPress
+   probes, webshell filenames, etc.) at severity ≥ 2.
+
+### Pattern categories (`bad_patterns.py`)
+
+| Pattern | Severity | Matches |
+| --- | --- | --- |
+| `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` |
+| `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` |
+| `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` |
+| `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
+| `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` |
+| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files |
+| `well_known_probe` | 1 (moderate) | `.well-known/` paths |
+
+Flagging metadata fields:
+
+- `errors` / `denied` — hostile hit count
+- `total` — all requests from that IP
+- `ratio` — error ratio (npm only)
+- `first` / `last` — first and last timestamp seen
+- `malicious_hits` — count of malicious-pattern request paths
+
+## Whitelisted ranges
+
+Whitelists live in `whitelists.py` and `data/`. Loading priority per provider:
+
+1. **Local files** in `data/<provider>/` (offline-safe, committed to repo)
+2. **Remote download** (fetches latest, saves to local file on success)
+3. **Built-in fallback** (hardcoded constants, always available)
+
+Each layer overrides the previous. Known-good ranges are emitted as
+`accept` rules placed **above** the `drop` rules in the MikroTik script.
+
+### Subnet consolidation
+
+Blocked IPs are grouped by WHOIS subnet (via `whois.radb.net`). If ≥ 3
+malicious IPs share a subnet, the entire subnet is blocked instead of
+individual `/32` or `/128` entries. Results are cached to
+`reports/whois_cache.json`.
+
+## Log format reference
+
+### copyparty
+
+```log
+15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
+15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
+15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
+15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
+16:00:34.385 hsrv                   LOG: ok bye
+```
+
+### Nginx Proxy Manager
+
+```log
+[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
+[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"
+```
+
+Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
+`"path"`, `[Client IP]`, `[Length N]`, `[Gzip X]`, `[Sent-to IP]`, `"UA"`,
+`"Referer"`.
+
+## Extending
+
+- **new whitelist provider** → add URLs, files, and fallback CIDRs to
+  `whitelists.py`
+- **new detection rule** → add a `BadPattern` to `bad_patterns.py` or a
+  `flag_*()` function to `utils.py`, return it from the analyzer's `flag()`
+- **new log source** → subclass `utils.Analyzer`, implement `parse_line()` +
+  `flag()`, declare `ip_columns` / `request_columns`
+- **new output file** → override `write_extra(data_dir)` and return the paths
+  written