intrusion-detection/README.md

# All You Can GET Intrusion Detection System

Local-development intrusion detection / log analysis for two log sources:

| Script            | Source                                                          | Detection rule                                                             |
| ----------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `scan.py`         | both sources                                                    | consolidated scan + unified MikroTik export                                |
| `copyparty.py`    | [copyparty](https://github.com/9001/copyparty) server logs      | denied-only IPs + copyparty auto-bans                                      |
| `npm.py`          | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners)                                             |
| `utils.py`        | re-export layer                                                 | thin barrel — imports from sub-modules below                               |
| `models.py`       | data model                                                      | `Request`, `IPStats`, `aggregate()`                                        |
| `ip_utils.py`     | IP helpers                                                      | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `flags.py`        | detection rules                                                 | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `analyzer.py`     | pipeline                                                        | `Analyzer` — discover → load → analyze → report → write                    |
| `whitelists.py`   | whitelist module                                                | provider CIDR constants, local/remote load, CIDR extraction                |
| `bad_patterns.py` | pattern detection                                               | malicious URL/file regex classification                                    |
| `whois.py`        | WHOIS query                                                     | `query_whois_subnet`, `consolidate_by_subnet`                              |
| `mikrotik.py`     | RouterOS export                                                 | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions`                 |
| `files.py`        | I/O helpers                                                     | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc.                 |
| `reporting.py`    | text rendering                                                  | `Column`, `render_table`, `render_kv`, `timestamp_name`                    |

Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik
RouterOS firewall script.

`scan.py` runs both analyzers, merges their blocklists, and writes a single
consolidated `.rsc`.

> Status: **local dev**. Runs offline against log files on disk. It does not
> talk to the router, does not run as a service, and blocks nothing by itself.
> You review the generated `.rsc` and import it manually.

## Architecture

```text
intrusion-detection/
├── scan.py              consolidated scanner entry point
├── copyparty.py         copyparty log adapter (~150 lines)
├── npm.py               NPM proxy log adapter (~150 lines)
├── utils.py             thin re-export barrel (~90 lines)
├── models.py            Request, IPStats, aggregate()
├── ip_utils.py          IP parsing, whitelist lookups, sorting
├── flags.py             flag_scanners, flag_denied_only, flag_malicious_patterns
├── analyzer.py          Analyzer pipeline base class
├── whitelists.py        provider CIDR constants, local/remote load
├── bad_patterns.py      malicious URL/file regex patterns
├── whois.py             query_whois_subnet, consolidate_by_subnet
├── mikrotik.py          RouterOS .rsc script generation
├── files.py             read_lines, find_logs, write_csv/json/lines
├── reporting.py         Column, render_table, render_kv, timestamp_name
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik-block.rsc   consolidated .rsc (scan.py only)
└── README.md
```

`copyparty.py` and `npm.py` are thin adapters. Each supplies a log matcher,
a line parser, a flagging rule, and column definitions. `utils.py` is a barrel
module that re-exports everything from the sub-modules so callers only need
`import utils`.

Core responsibilities by module:

| Module            | Owns                                                                             |
| ----------------- | -------------------------------------------------------------------------------- |
| `models.py`       | `Request`, `IPStats`, `aggregate()`                                              |
| `ip_utils.py`     | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions`       |
| `flags.py`        | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total`       |
| `analyzer.py`     | `Analyzer` — discover → load → analyze → report → write                          |
| `whitelists.py`   | provider CIDR constants, local-file / remote / built-in loading                  |
| `bad_patterns.py` | malicious URL/file regex classification                                          |
| `whois.py`        | `query_whois_subnet`, `consolidate_by_subnet`                                    |
| `mikrotik.py`     | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions`                       |
| `files.py`        | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `reporting.py`    | `Column`, `render_table`, `render_kv`, `timestamp_name`                          |

Console output and saved reports are the **same string** (`Analyzer.text_report()`).

## Requirements

- Python 3.10+
- stdlib only, no dependencies

## Usage

```powershell
# individual scanners
python copyparty.py
python npm.py

# consolidated (both sources, unified blocklist)
python scan.py
```

All scripts accept CLI overrides:

| Flag              | Default              | Meaning                       |
| ----------------- | -------------------- | ----------------------------- |
| `--log-path`      | see below            | log directory                 |
| `--report-dir`    | `reports/<name>`     | output directory              |
| `--top N`         | `30`                 | rows in the top-IP table      |
| `--quiet`         | off                  | write files, print only paths |
| `--no-rsc`        | off                  | skip MikroTik export          |
| `--min-errors N`  | `5` (`npm.py`)       | error count threshold         |
| `--error-ratio F` | `0.5` (`npm.py`)     | error ratio threshold         |
| `--min-denied N`  | `1` (`copyparty.py`) | denied-hit threshold          |

`scan.py` accepts all flags from both scanners plus `--copyparty-log-path`,
`--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and
`--report-dir` for the consolidated output.

Default log paths (overridable by `.env`):

| Script         | Env var              | Default        |
| -------------- | -------------------- | -------------- |
| `copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
| `npm.py`       | `NPM_LOG_PATH`       | `Y:\npm\logs`  |

Examples:

```powershell
python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python copyparty.py --top 100 --no-rsc
python scan.py --quiet --no-rsc
```

## Output

Each script writes to its own report directory. `scan.py` also writes a
consolidated `.rsc`:

```text
reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   ├── requests.csv          every parsed request
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   ├── requests.csv
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt
└── mikrotik-block.rsc             consolidated (scan.py only)
```

## Detection

### Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

1. **Error-rate flagging** (`npm.py`) — IPs with many 4xx/5xx responses and a
   high error ratio:

   $$\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5$$

2. **Denied-only flagging** (`copyparty.py`) — IPs that hit denied paths and
   never authenticated:

   $$\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}$$

3. **Malicious-pattern flagging** (both analyzers) — IPs whose request paths
   match known probe/attack signatures (`.env`, `.git/config`, WordPress
   probes, webshell filenames, etc.) at severity ≥ 2.

### Pattern categories (`bad_patterns.py`)

| Pattern             | Severity     | Matches                                                             |
| ------------------- | ------------ | ------------------------------------------------------------------- |
| `env_file`          | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json`        |
| `vcs_probe`         | 3 (critical) | `.git/config`, `.svn/`                                              |
| `sensitive_file`    | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk`                          |
| `webshell_probe`    | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
| `wp_probe`          | 2 (high)     | `wp-admin`, `wp-login`, `wp-content`, `wp-includes`                 |
| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files                             |
| `well_known_probe`  | 1 (moderate) | `.well-known/` paths                                                |

Flagging metadata fields:

- `errors` / `denied` — hostile hit count
- `total` — all requests from that IP
- `ratio` — error ratio (npm only)
- `first` / `last` — first and last timestamp seen
- `malicious_hits` — count of malicious-pattern request paths

## Whitelisted ranges

Whitelists live in `whitelists.py` and `data/`. Loading priority per provider:

1. **Local files** in `data/<provider>/` (offline-safe, committed to repo)
2. **Remote download** (fetches latest, saves to local file on success)
3. **Built-in fallback** (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as
`accept` rules placed **above** the `drop` rules in the MikroTik script.

### Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via `whois.radb.net`). If ≥ 3
malicious IPs share a subnet, the entire subnet is blocked instead of
individual `/32` or `/128` entries. Results are cached to
`reports/whois_cache.json`.

## Log format reference

### copyparty

```log
15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye
```

### Nginx Proxy Manager

```log
[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"
```

Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
`"path"`, `[Client IP]`, `[Length N]`, `[Gzip X]`, `[Sent-to IP]`, `"UA"`,
`"Referer"`.

## Extending

- **new whitelist provider** → add URLs, local file paths, and fallback CIDRs
  to `whitelists.py`
- **new detection pattern** → add a `BadPattern` to `bad_patterns.py`
- **new detection rule** → add a `flag_*()` function to `flags.py`, return it
  from the analyzer's `flag()` method
- **new log source** → subclass `analyzer.Analyzer`, implement `parse_line()` +
  `flag()`, declare `ip_columns` / `request_columns`
- **new output file** → override `write_extra(data_dir)` and return the paths
  written
- **new I/O format** → add writer to `files.py`
- **new report section** → add rendering logic to `reporting.py`