# All You Can GET Intrusion Detection System Local-development intrusion detection / log analysis for two log sources: | Script | Source | Detection rule | | --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- | | `app.py` | unified entry point | runs both analyzers + optional firewall sync | | `app/scan.py` | consolidated scanner | runs both analyzers, merges blocklists, syncs firewall | | `app/copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans | | `app/npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) | | `app/utils.py` | re-export layer | thin barrel — imports from sub-modules below | | `app/models.py` | data model | `Request`, `IPStats`, `aggregate()` | | `app/ip_utils.py` | IP helpers | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` | | `app/flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` | | `app/analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write | | `app/whitelists.py` | whitelist module | provider CIDR constants, local/remote load, CIDR extraction | | `app/bad_patterns.py` | pattern detection | malicious URL/file regex classification | | `app/whois.py` | WHOIS query | `query_whois_subnet`, `consolidate_by_subnet` | | `app/mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` | | `app/firewall.py` | RouterOS SSH client | `MikrotikFirewall` — connect, sync blocklist, audit state | | `app/files.py` | I/O helpers | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc. | | `app/reporting.py` | text rendering | `Column`, `render_table`, `render_kv`, `timestamp_name` | Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script. `app.py` runs both analyzers, merges their blocklists, and optionally synchronizes the MikroTik RouterOS firewall via SSH. > Status: **local dev**. Runs offline against log files on disk. Use > `--update-firewall` to push changes to the router, or `--dry-run` to > preview without applying. ## Architecture ```text intrusion-detection/ ├── app.py unified entry point ├── app/ application package │ ├── __init__.py │ ├── constants.py all config constants (paths, env vars, thresholds) │ ├── scan.py consolidated scanner │ ├── copyparty.py copyparty log adapter (~150 lines) │ ├── npm.py NPM proxy log adapter (~150 lines) │ ├── firewall.py MikroTik RouterOS SSH client │ ├── utils.py thin re-export barrel (~90 lines) │ ├── models.py Request, IPStats, aggregate() │ ├── ip_utils.py IP parsing, whitelist lookups, sorting │ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns │ ├── analyzer.py Analyzer pipeline base class │ ├── whitelists.py provider CIDR constants, local/remote load │ ├── bad_patterns.py malicious URL/file regex patterns │ ├── whois.py query_whois_subnet, consolidate_by_subnet │ ├── mikrotik.py RouterOS .rsc script generation │ ├── files.py read_lines, find_logs, write_csv/json/lines │ ├── reporting.py Column, render_table, render_kv, timestamp_name │ ├── web.py Flask web UI (app factory, JSON API, pages) │ ├── web_scan.py background scan runner + in-memory job store │ ├── templates/ Jinja2 page templates (dashboard, scan, reports, …) │ └── static/ style.css + app.js (dark-theme UI) ├── data/ whitelist source files (cached locally) │ ├── openfactory/ │ │ └── geofeed.csv │ ├── cloudflare/ │ │ ├── ips-v4.txt │ │ └── ips-v6.txt │ └── paloalto/ │ └── scanner-ips.txt ├── reports/ generated reports (per-run + per-source) │ ├── copyparty/ │ ├── npm/ │ └── mikrotik/ firewall audit snapshots └── README.md ``` `app/copyparty.py` and `app/npm.py` are thin adapters. Each supplies a log matcher, a line parser, a flagging rule, and column definitions. Core responsibilities by module: | Module | Owns | | --------------------- | -------------------------------------------------------------------------------- | | `app/constants.py` | all config constants — paths, env vars, network/scan thresholds | | `app/models.py` | `Request`, `IPStats`, `aggregate()` | | `app/ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` | | `app/flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` | | `app/analyzer.py` | `Analyzer` — discover → load → analyze → report → write | | `app/whitelists.py` | provider CIDR constants, local-file / remote / built-in loading | | `app/bad_patterns.py` | malicious URL/file regex classification | | `app/whois.py` | `query_whois_subnet`, `consolidate_by_subnet` | | `app/mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` | | `app/firewall.py` | `MikrotikFirewall` — SSH connect, sync blocklist, audit state | | `app/web.py` | Flask app factory, JSON API routes, page rendering | | `app/web_scan.py` | background scan runner, in-memory job store, single-scan busy lock | | `app/files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` | | `app/reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` | Console output and saved reports are the **same string** (`Analyzer.text_report()`). ## Requirements - Python 3.10+ - `paramiko` (MikroTik SSH), `python-dotenv` (`.env` loading), `flask` (web UI) ```powershell pip install -r requirements.txt ``` ## Configuration All tunable settings live in `app/constants.py` and can be overridden per deployment via environment variables. For local development, copy `.env.example` to `.env` and fill in your values: ```powershell copy .env.example .env ``` `app/constants.py` loads `.env` with `python-dotenv` at startup. Real system environment variables always take precedence over `.env` values, so production can set the same variables via the OS and skip `.env` entirely. `.env` is git-ignored — do not commit secrets to it. | Env var | Default | Used by | | -------------------- | ------------------------ | --------------------- | | `COPYPARTY_LOG_PATH` | `P:\copyparty` | copyparty scanner | | `NPM_LOG_PATH` | `Y:\npm\logs` | npm scanner | | `ROUTER_IP` | `192.168.88.1` | MikroTik connection | | `ROUTER_USER` | `admin` | MikroTik connection | | `ROUTER_PASSWORD` | _(set in .env)_ | MikroTik connection | | `BLOCKLIST_NAME` | `Blocked` | MikroTik address list | | `WHITELIST_NAMES` | `Cloudflare,Openfactory` | MikroTik whitelists | | `WEB_HOST` | `127.0.0.1` | web UI bind host | | `WEB_PORT` | `54666` | web UI port | See `.env.example` for the complete list of supported variables (WHOIS tuning, scan thresholds, whitelist source URLs, and more). ## Usage ```powershell # web UI (default entry point) python app.py python app.py --web-port 54666 # consolidated CLI scan (scan subcommand) python app.py scan python app.py scan --quiet python app.py scan --update-firewall --dry-run python app.py scan --update-firewall # individual scanners (still runnable directly) python -m app.copyparty python -m app.npm ``` All scripts accept CLI overrides: | Flag | Default | Meaning | | ------------------- | -------------------- | ---------------------------------- | | `--log-path` | see below | log directory | | `--report-dir` | `reports/` | output directory | | `--top N` | `30` | rows in the top-IP table | | `--quiet` | off | write files, print only paths | | `--min-errors N` | `5` (`npm.py`) | error count threshold | | `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold | | `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold | | `--update-firewall` | off | sync blocklist to MikroTik via SSH | | `--dry-run` | off | preview firewall changes, no apply | | `--web-host` | `127.0.0.1` | web UI bind host | | `--web-port` | `54666` | web UI port | `python app.py scan` accepts all flags from both scanners plus `--copyparty-log-path`, `--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and `--report-dir` for the consolidated output. Examples: ```powershell python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8 python -m app.copyparty --top 100 python app.py scan --quiet --update-firewall --dry-run ``` ## Blocklist behavior (cumulative / grow-only) The MikroTik `Blocked` address list is **cumulative**: it only grows over time and is never trimmed automatically. - An IP (or subnet) that the system has ever blocked stays blocked — even if it stops appearing in recent logs (for example after it leaves the NPM ~30-day log-retention window, or once WHOIS subnet consolidation groups it into a larger range). - The **source of truth** for "all blocked IPs" is the git history of the consolidated blocklist files `reports/app/data/blocklist_ipv4.txt` and `blocklist_ipv6.txt` (these files are git-tracked). After each scan the updated blocklist files should be committed so the history keeps growing. - The sync only ever **adds** entries. The only automatic removals are router entries that were never blocked by the system — i.e. absent from every git version of the blocklist files (manual / rogue entries). - To explicitly unblock an IP, remove it manually via the web UI (`/blocklist` → remove) or directly on the router; the automated sync will not re-add entries that are not part of a scan. Each consolidated scan with `--update-firewall` also writes a timestamped change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list of entries added / removed (complete-set delta), the full cumulative blocked list, and the router sync changes. ## Web UI `python app.py` starts a **localhost-only** Flask web UI (default `http://127.0.0.1:54666`) for running scans and managing the MikroTik blocklist / firewall. It is bound to `127.0.0.1` — do **not** expose `WEB_HOST` beyond localhost. Pages: | Route | Purpose | | ------------ | ----------------------------------------------------- | | `/` | dashboard — status cards + latest scans | | `/scan` | run copyparty / npm / consolidated scans (background) | | `/reports` | browse reports per source | | `/blocklist` | view the consolidated blocklist, add/remove entries | | `/firewall` | view address lists + rules, preview/apply a sync | | `/whitelist` | view/manage provider whitelist CIDRs | Scans run in a background thread (only one at a time) and report progress via the JSON API, so the UI never blocks. Firewall sync is **dry-run by default** from the UI — applying changes is an explicit action. The main JSON endpoints (`/api/status`, `/api/scan/*`, `/api/reports/*`, `/api/blocklist`, `/api/firewall/*`, `/api/whitelist`) are also usable directly from a REST client. ## Output Each script writes to its own report directory. `app.py` also writes consolidated blocklists and firewall audit snapshots: ```text reports/ ├── copyparty/ │ ├── data/ │ │ ├── ips_ipv4.csv per-IP: total, denied, banned, users │ │ ├── ips_ipv6.csv │ │ ├── ips.json full per-IP detail w/ malicious hits │ │ ├── denied_paths.csv denied paths, hit counts │ │ ├── denied_ips.txt flagged IPv4 + metadata │ │ ├── denied_ipv6.txt │ │ ├── banned_ips.txt copyparty auto-banned IPv4 │ │ ├── banned_ipv6.txt │ │ ├── blocked_ipv4.txt final block set (non-whitelisted) │ │ ├── blocked_ipv6.txt │ │ ├── authed_users.json authenticated user counts │ │ └── requests.csv every parsed request │ └── 2026-08-02-12-00-00.txt timestamped report ├── npm/ │ ├── data/ │ │ ├── ips_ipv4.csv per-IP: total, 4xx, 5xx, ratio, methods │ │ ├── ips_ipv6.csv │ │ ├── ips.json │ │ ├── error_paths.csv │ │ ├── scanner_ips.txt flagged IPv4 + metadata │ │ ├── scanner_ipv6.txt │ │ ├── blocked_ipv4.txt │ │ ├── blocked_ipv6.txt │ │ ├── ua_summary.csv user agent breakdown │ │ ├── host_summary.csv virtual host breakdown │ │ └── requests.csv │ └── 2026-08-02-12-00-00.txt ├── app/ │ └── data/ │ ├── blocklist_ipv4.txt final consolidated IPv4 blocklist (CIDRs) │ └── blocklist_ipv6.txt final consolidated IPv6 blocklist (CIDRs) └── mikrotik/ ├── data/ │ ├── address_list_ipv4.json │ ├── address_list_ipv6.json │ ├── firewall_ipv4.json │ └── firewall_ipv6.json └── 2026-08-04-18-30-28.json firewall audit snapshot ``` The current router state (address lists + firewall rules) is also mirrored to `data/mikrotik/` on every `--update-firewall` run. Those files are overwritten each run and are **git-visible**, so `git diff` shows how the firewall rules and address lists change over time. ## Detection ### Three detection layers Blocklist candidates come from three independent sources, merged at scan time: 1. **Error-rate flagging** (`npm.py`) — IPs with many 4xx/5xx responses and a high error ratio: $$\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5$$ 2. **Denied-only flagging** (`copyparty.py`) — IPs that hit denied paths and never authenticated: $$\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}$$ 3. **Malicious-pattern flagging** (both analyzers) — IPs whose request paths match known probe/attack signatures (`.env`, `.git/config`, WordPress probes, webshell filenames, etc.) at severity ≥ 2. ### Pattern categories (`bad_patterns.py`) | Pattern | Severity | Matches | | ------------------- | ------------ | ------------------------------------------------------------------- | | `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` | | `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` | | `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` | | `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) | | `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` | | `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files | | `well_known_probe` | 1 (moderate) | `.well-known/` paths | Flagging metadata fields: - `errors` / `denied` — hostile hit count - `total` — all requests from that IP - `ratio` — error ratio (npm only) - `first` / `last` — first and last timestamp seen - `malicious_hits` — count of malicious-pattern request paths ## Whitelisted ranges Whitelists live in `whitelists.py` and `data/`. Loading priority per provider: 1. **Local files** in `data//` (offline-safe, committed to repo) 2. **Remote download** (fetches latest, saves to local file on success) 3. **Built-in fallback** (hardcoded constants, always available) Each layer overrides the previous. Known-good ranges are emitted as `accept` rules placed **above** the `drop` rules in the MikroTik script. ### Subnet consolidation Blocked IPs are grouped by WHOIS subnet (via `whois.radb.net`). If ≥ 3 malicious IPs share a subnet, the entire subnet is blocked instead of individual `/32` or `/128` entries. Results are cached to `reports/whois_cache.json`. ## Log format reference ### copyparty ```log 15:55:47.466 34.105.30.251 37140 LOG: GET //2020/wp-includes/wlwmanifest.xml @* 15:55:47.466 34.105.30.251 37140 LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml' 15:55:47.466 34.105.30.251 37140 CRIT: client banned: 403s 15:55:47.695 34.105.30.251 37144 BTW: banned for 86399 sec 16:00:34.385 hsrv LOG: ok bye ``` ### Nginx Proxy Manager ```log [02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-" [02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-" ``` Fields: `[datetime]`, upstream status, proxy status, method, scheme, host, `"path"`, `[Client IP]`, `[Length N]`, `[Gzip X]`, `[Sent-to IP]`, `"UA"`, `"Referer"`. ## Extending - **new whitelist provider** → add URLs, local file paths, and fallback CIDRs to `app/whitelists.py` - **new detection pattern** → add a `BadPattern` to `app/bad_patterns.py` - **new detection rule** → add a `flag_*()` function to `app/flags.py`, return it from the analyzer's `flag()` method - **new log source** → subclass `app.analyzer.Analyzer`, implement `parse_line()` + `flag()`, declare `ip_columns` / `request_columns` - **new output file** → override `write_extra(data_dir)` and return the paths written - **new I/O format** → add writer to `app/files.py` - **new report section** → add rendering logic to `app/reporting.py`