intrusion-detection/README.md

# allucanget-ids

Local-development intrusion detection / log analysis for two log sources:

| Script | Source | Detection rule |
| --- | --- | --- |
| `copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `scan.py` | both of the above | consolidated scan + unified MikroTik export |
| `whitelists.py` | whitelist module | provider CIDR loading, refresh, validation |
| `bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `utils.py` | shared core (no CLI) | aggregation, reports, rsc, pipeline |

Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik
RouterOS firewall script.

`scan.py` runs both analyzers, merges their blocklists, and writes a single
consolidated `.rsc`.

> Status: **local dev**. Runs offline against log files on disk. It does not
> talk to the router, does not run as a service, and blocks nothing by itself.
> You review the generated `.rsc` and import it manually.

## Architecture

```
allucanget-ids/
├── scan.py              consolidated scanner entry point
├── copyparty.py         copyparty log adapter (~150 lines)
├── npm.py               NPM proxy log adapter (~150 lines)
├── whitelists.py        whitelist constants, download, CIDR extraction
├── bad_patterns.py      malicious URL/file regex patterns
├── utils.py             shared core: aggregation, reporting, MikroTik export
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik-block.rsc   consolidated .rsc (scan.py only)
└── README.md
```

`copyparty.py` and `npm.py` are thin adapters. Each supplies a log matcher,
a line parser, a flagging rule, and column definitions. Everything else lives in
`utils.py`:

- `Request` — normalised request record
- `IPStats` + `aggregate()` — per-IP counters, malicious-hit tracking
- `flag_scanners()` / `flag_denied_only()` / `flag_malicious_patterns()` — detection rules
- `whitelist_of()` / `is_whitelisted()` — whitelist lookups (backed by `whitelists.py`)
- `is_malicious()` / `classify_path()` — pattern classification (backed by `bad_patterns.py`)
- `query_whois_subnet()` / `consolidate_by_subnet()` — WHOIS subnet grouping
- IP version helpers, log discovery, transparent `.gz`, ANSI stripping
- CSV/JSON/table writers, `mikrotik_script()` — RouterOS export
- `Analyzer` — pipeline: discover → load → analyze → report → write

Console output and saved reports are the **same string** (`Analyzer.text_report()`).

## Requirements

- Python 3.10+
- stdlib only, no dependencies

## Usage

```powershell
# individual scanners
python copyparty.py
python npm.py

# consolidated (both sources, unified blocklist)
python scan.py
```

All scripts accept CLI overrides:

| Flag | Default | Meaning |
| --- | --- | --- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--no-rsc` | off | skip MikroTik export |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |

`scan.py` accepts all flags from both scanners plus `--copyparty-log-path`,
`--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and
`--report-dir` for the consolidated output.

Default log paths (overridable by `.env`):

| Script | Env var | Default |
| --- | --- | --- |
| `copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
| `npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |

Examples:

```powershell
python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python copyparty.py --top 100 --no-rsc
python scan.py --quiet --no-rsc
```

## Output

Each script writes to its own report directory. `scan.py` also writes a
consolidated `.rsc`:

```text
reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   ├── requests.csv          every parsed request
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   ├── requests.csv
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt
└── mikrotik-block.rsc             consolidated (scan.py only)
```

## Detection

### Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

1. **Error-rate flagging** (`npm.py`) — IPs with many 4xx/5xx responses and a
   high error ratio:

   $$\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5$$

2. **Denied-only flagging** (`copyparty.py`) — IPs that hit denied paths and
   never authenticated:

   $$\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}$$

3. **Malicious-pattern flagging** (both analyzers) — IPs whose request paths
   match known probe/attack signatures (`.env`, `.git/config`, WordPress
   probes, webshell filenames, etc.) at severity ≥ 2.

### Pattern categories (`bad_patterns.py`)

| Pattern | Severity | Matches |
| --- | --- | --- |
| `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` |
| `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` |
| `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` |
| `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
| `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` |
| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files |
| `well_known_probe` | 1 (moderate) | `.well-known/` paths |

Flagging metadata fields:

- `errors` / `denied` — hostile hit count
- `total` — all requests from that IP
- `ratio` — error ratio (npm only)
- `first` / `last` — first and last timestamp seen
- `malicious_hits` — count of malicious-pattern request paths

## Whitelisted ranges

Whitelists live in `whitelists.py` and `data/`. Loading priority per provider:

1. **Local files** in `data/<provider>/` (offline-safe, committed to repo)
2. **Remote download** (fetches latest, saves to local file on success)
3. **Built-in fallback** (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as
`accept` rules placed **above** the `drop` rules in the MikroTik script.

### Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via `whois.radb.net`). If ≥ 3
malicious IPs share a subnet, the entire subnet is blocked instead of
individual `/32` or `/128` entries. Results are cached to
`reports/whois_cache.json`.

## Log format reference

### copyparty

```log
15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye
```

### Nginx Proxy Manager

```log
[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"
```

Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
`"path"`, `[Client IP]`, `[Length N]`, `[Gzip X]`, `[Sent-to IP]`, `"UA"`,
`"Referer"`.

## Extending

- **new whitelist provider** → add URLs, files, and fallback CIDRs to
  `whitelists.py`
- **new detection rule** → add a `BadPattern` to `bad_patterns.py` or a
  `flag_*()` function to `utils.py`, return it from the analyzer's `flag()`
- **new log source** → subclass `utils.Analyzer`, implement `parse_line()` +
  `flag()`, declare `ip_columns` / `request_columns`
- **new output file** → override `write_extra(data_dir)` and return the paths
  written