Script	Source	Detection rule
`scan.py`	both sources	consolidated scan + unified MikroTik export
`copyparty.py`	copyparty server logs	denied-only IPs + copyparty auto-bans
`npm.py`	Nginx Proxy Manager proxy logs	high error-rate IPs (scanners)
`utils.py`	re-export layer	thin barrel — imports from sub-modules below
`models.py`	data model	`Request`, `IPStats`, `aggregate()`
`ip_utils.py`	IP helpers	`parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions`
`flags.py`	detection rules	`flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total`
`analyzer.py`	pipeline	`Analyzer` — discover → load → analyze → report → write
`whitelists.py`	whitelist module	provider CIDR constants, local/remote load, CIDR extraction
`bad_patterns.py`	pattern detection	malicious URL/file regex classification
`whois.py`	WHOIS query	`query_whois_subnet`, `consolidate_by_subnet`
`mikrotik.py`	RouterOS export	`mikrotik_script`, `write_mikrotik`, `split_cidr_versions`
`files.py`	I/O helpers	`read_lines`, `find_logs`, `write_csv`, `write_json`, etc.
`reporting.py`	text rendering	`Column`, `render_table`, `render_kv`, `timestamp_name`

Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script.

scan.py runs both analyzers, merges their blocklists, and writes a single consolidated .rsc.

Status: local dev. Runs offline against log files on disk. It does not talk to the router, does not run as a service, and blocks nothing by itself. You review the generated .rsc and import it manually.

Architecture

intrusion-detection/
├── scan.py              consolidated scanner entry point
├── copyparty.py         copyparty log adapter (~150 lines)
├── npm.py               NPM proxy log adapter (~150 lines)
├── utils.py             thin re-export barrel (~90 lines)
├── models.py            Request, IPStats, aggregate()
├── ip_utils.py          IP parsing, whitelist lookups, sorting
├── flags.py             flag_scanners, flag_denied_only, flag_malicious_patterns
├── analyzer.py          Analyzer pipeline base class
├── whitelists.py        provider CIDR constants, local/remote load
├── bad_patterns.py      malicious URL/file regex patterns
├── whois.py             query_whois_subnet, consolidate_by_subnet
├── mikrotik.py          RouterOS .rsc script generation
├── files.py             read_lines, find_logs, write_csv/json/lines
├── reporting.py         Column, render_table, render_kv, timestamp_name
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik-block.rsc   consolidated .rsc (scan.py only)
└── README.md

copyparty.py and npm.py are thin adapters. Each supplies a log matcher, a line parser, a flagging rule, and column definitions. utils.py is a barrel module that re-exports everything from the sub-modules so callers only need import utils.

Core responsibilities by module:

Module	Owns
`models.py`	`Request`, `IPStats`, `aggregate()`
`ip_utils.py`	`parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions`
`flags.py`	`flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total`
`analyzer.py`	`Analyzer` — discover → load → analyze → report → write
`whitelists.py`	provider CIDR constants, local-file / remote / built-in loading
`bad_patterns.py`	malicious URL/file regex classification
`whois.py`	`query_whois_subnet`, `consolidate_by_subnet`
`mikrotik.py`	`mikrotik_script`, `write_mikrotik`, `split_cidr_versions`
`files.py`	`read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines`
`reporting.py`	`Column`, `render_table`, `render_kv`, `timestamp_name`

Console output and saved reports are the same string (Analyzer.text_report()).

Requirements

Python 3.10+
stdlib only, no dependencies

Usage

# individual scanners
python copyparty.py
python npm.py

# consolidated (both sources, unified blocklist)
python scan.py

All scripts accept CLI overrides:

Flag	Default	Meaning
`--log-path`	see below	log directory
`--report-dir`	`reports/<name>`	output directory
`--top N`	`30`	rows in the top-IP table
`--quiet`	off	write files, print only paths
`--no-rsc`	off	skip MikroTik export
`--min-errors N`	`5` (`npm.py`)	error count threshold
`--error-ratio F`	`0.5` (`npm.py`)	error ratio threshold
`--min-denied N`	`1` (`copyparty.py`)	denied-hit threshold

scan.py accepts all flags from both scanners plus --copyparty-log-path, --copyparty-report-dir, --npm-log-path, --npm-report-dir, and --report-dir for the consolidated output.

Default log paths (overridable by .env):

Script	Env var	Default
`copyparty.py`	`COPYPARTY_LOG_PATH`	`P:\copyparty`
`npm.py`	`NPM_LOG_PATH`	`Y:\npm\logs`

Examples:

python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python copyparty.py --top 100 --no-rsc
python scan.py --quiet --no-rsc

Output

Each script writes to its own report directory. scan.py also writes a consolidated .rsc:

reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   ├── requests.csv          every parsed request
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   ├── requests.csv
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt
└── mikrotik-block.rsc             consolidated (scan.py only)

Detection

Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

Error-rate flagging (npm.py) — IPs with many 4xx/5xx responses and a high error ratio:
\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5
Denied-only flagging (copyparty.py) — IPs that hit denied paths and never authenticated:
\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}
Malicious-pattern flagging (both analyzers) — IPs whose request paths match known probe/attack signatures (.env, .git/config, WordPress probes, webshell filenames, etc.) at severity ≥ 2.

Pattern categories (`bad_patterns.py`)

Pattern	Severity	Matches
`env_file`	3 (critical)	`.env`, `.env.production`, `.aws/credentials`, `config.json`
`vcs_probe`	3 (critical)	`.git/config`, `.svn/`
`sensitive_file`	3 (critical)	`id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk`
`webshell_probe`	3 (critical)	Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.)
`wp_probe`	2 (high)	`wp-admin`, `wp-login`, `wp-content`, `wp-includes`
`generic_php_probe`	1 (moderate)	Single/double-char/numeric `.php` files
`well_known_probe`	1 (moderate)	`.well-known/` paths

Flagging metadata fields:

errors / denied — hostile hit count
total — all requests from that IP
ratio — error ratio (npm only)
first / last — first and last timestamp seen
malicious_hits — count of malicious-pattern request paths

Whitelisted ranges

Whitelists live in whitelists.py and data/. Loading priority per provider:

Local files in data/<provider>/ (offline-safe, committed to repo)
Remote download (fetches latest, saves to local file on success)
Built-in fallback (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as accept rules placed above the drop rules in the MikroTik script.

Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via whois.radb.net). If ≥ 3 malicious IPs share a subnet, the entire subnet is blocked instead of individual /32 or /128 entries. Results are cached to reports/whois_cache.json.

Log format reference

copyparty

15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye

Nginx Proxy Manager

[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"

Fields: [datetime], upstream status, proxy status, method, scheme, host, "path", [Client IP], [Length N], [Gzip X], [Sent-to IP], "UA", "Referer".

Extending

new whitelist provider → add URLs, local file paths, and fallback CIDRs to whitelists.py
new detection pattern → add a BadPattern to bad_patterns.py
new detection rule → add a flag_*() function to flags.py, return it from the analyzer's flag() method
new log source → subclass analyzer.Analyzer, implement parse_line() + flag(), declare ip_columns / request_columns
new output file → override write_extra(data_dir) and return the paths written
new I/O format → add writer to files.py
new report section → add rendering logic to reporting.py

README.md

All You Can GET Intrusion Detection System