Script	Source	Detection rule
`app.py`	unified entry point	runs both analyzers + optional firewall sync
`app/scan.py`	consolidated scanner	runs both analyzers, merges blocklists, syncs firewall
`app/copyparty.py`	copyparty server logs	denied-only IPs + copyparty auto-bans
`app/npm.py`	Nginx Proxy Manager proxy logs	high error-rate IPs (scanners)
`app/utils.py`	re-export layer	thin barrel — imports from sub-modules below
`app/models.py`	data model	`Request`, `IPStats`, `aggregate()`
`app/ip_utils.py`	IP helpers	`parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions`
`app/flags.py`	detection rules	`flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total`
`app/analyzer.py`	pipeline	`Analyzer` — discover → load → analyze → report → write
`app/whitelists.py`	whitelist module	provider CIDR constants, local/remote load, CIDR extraction
`app/bad_patterns.py`	pattern detection	malicious URL/file regex classification
`app/whois.py`	WHOIS query	`query_whois_subnet`, `consolidate_by_subnet`
`app/mikrotik.py`	RouterOS export	`mikrotik_script`, `write_mikrotik`, `split_cidr_versions`
`app/firewall.py`	RouterOS SSH client	`MikrotikFirewall` — connect, sync blocklist, audit state
`app/files.py`	I/O helpers	`read_lines`, `find_logs`, `write_csv`, `write_json`, etc.
`app/reporting.py`	text rendering	`Column`, `render_table`, `render_kv`, `timestamp_name`

Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script.

app.py runs both analyzers, merges their blocklists, and optionally synchronizes the MikroTik RouterOS firewall via SSH.

Status: local dev. Runs offline against log files on disk. Use --update-firewall to push changes to the router, or --dry-run to preview without applying.

Architecture

intrusion-detection/
├── app.py               unified entry point
├── app/                 application package
│   ├── __init__.py
│   ├── scan.py          consolidated scanner
│   ├── copyparty.py     copyparty log adapter (~150 lines)
│   ├── npm.py           NPM proxy log adapter (~150 lines)
│   ├── firewall.py      MikroTik RouterOS SSH client
│   ├── utils.py         thin re-export barrel (~90 lines)
│   ├── models.py        Request, IPStats, aggregate()
│   ├── ip_utils.py      IP parsing, whitelist lookups, sorting
│   ├── flags.py         flag_scanners, flag_denied_only, flag_malicious_patterns
│   ├── analyzer.py      Analyzer pipeline base class
│   ├── whitelists.py    provider CIDR constants, local/remote load
│   ├── bad_patterns.py  malicious URL/file regex patterns
│   ├── whois.py         query_whois_subnet, consolidate_by_subnet
│   ├── mikrotik.py      RouterOS .rsc script generation
│   ├── files.py         read_lines, find_logs, write_csv/json/lines
│   └── reporting.py     Column, render_table, render_kv, timestamp_name
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik/        firewall audit snapshots
└── README.md

app/copyparty.py and app/npm.py are thin adapters. Each supplies a log matcher, a line parser, a flagging rule, and column definitions.

Core responsibilities by module:

Module	Owns
`app/models.py`	`Request`, `IPStats`, `aggregate()`
`app/ip_utils.py`	`parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions`
`app/flags.py`	`flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total`
`app/analyzer.py`	`Analyzer` — discover → load → analyze → report → write
`app/whitelists.py`	provider CIDR constants, local-file / remote / built-in loading
`app/bad_patterns.py`	malicious URL/file regex classification
`app/whois.py`	`query_whois_subnet`, `consolidate_by_subnet`
`app/mikrotik.py`	`mikrotik_script`, `write_mikrotik`, `split_cidr_versions`
`app/firewall.py`	`MikrotikFirewall` — SSH connect, sync blocklist, audit state
`app/files.py`	`read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines`
`app/reporting.py`	`Column`, `render_table`, `render_kv`, `timestamp_name`

Console output and saved reports are the same string (Analyzer.text_report()).

Requirements

Python 3.10+
stdlib only, no dependencies

Usage

# unified entry point (both sources, optional firewall sync)
python app.py
python app.py --quiet
python app.py --update-firewall --dry-run
python app.py --update-firewall

# individual scanners (still runnable directly)
python -m app.copyparty
python -m app.npm

All scripts accept CLI overrides:

Flag	Default	Meaning
`--log-path`	see below	log directory
`--report-dir`	`reports/<name>`	output directory
`--top N`	`30`	rows in the top-IP table
`--quiet`	off	write files, print only paths
`--min-errors N`	`5` (`npm.py`)	error count threshold
`--error-ratio F`	`0.5` (`npm.py`)	error ratio threshold
`--min-denied N`	`1` (`copyparty.py`)	denied-hit threshold
`--update-firewall`	off	sync blocklist to MikroTik via SSH
`--dry-run`	off	preview firewall changes, no apply

app.py accepts all flags from both scanners plus --copyparty-log-path, --copyparty-report-dir, --npm-log-path, --npm-report-dir, and --report-dir for the consolidated output.

Default log paths (overridable by .env):

Script	Env var	Default
`app/copyparty.py`	`COPYPARTY_LOG_PATH`	`P:\copyparty`
`app/npm.py`	`NPM_LOG_PATH`	`Y:\npm\logs`
`app/firewall.py`	`ROUTER_IP`	`192.168.88.1`
	`ROUTER_USER`	`admin`
	`ROUTER_PASSWORD`	(set in .env)

Examples:

python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python -m app.copyparty --top 100
python app.py --quiet --update-firewall --dry-run

Output

Each script writes to its own report directory. app.py also writes firewall audit snapshots:

reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   └── requests.csv          every parsed request
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   └── requests.csv
│   └── 2026-08-02-12-00-00.txt
└── mikrotik/
    ├── data/
    │   ├── address_list_ipv4.json
    │   ├── address_list_ipv6.json
    │   ├── firewall_ipv4.json
    │   └── firewall_ipv6.json
    └── 2026-08-04-18-30-28.json   firewall audit snapshot

Detection

Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

Error-rate flagging (npm.py) — IPs with many 4xx/5xx responses and a high error ratio:
\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5
Denied-only flagging (copyparty.py) — IPs that hit denied paths and never authenticated:
\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}
Malicious-pattern flagging (both analyzers) — IPs whose request paths match known probe/attack signatures (.env, .git/config, WordPress probes, webshell filenames, etc.) at severity ≥ 2.

Pattern categories (`bad_patterns.py`)

Pattern	Severity	Matches
`env_file`	3 (critical)	`.env`, `.env.production`, `.aws/credentials`, `config.json`
`vcs_probe`	3 (critical)	`.git/config`, `.svn/`
`sensitive_file`	3 (critical)	`id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk`
`webshell_probe`	3 (critical)	Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.)
`wp_probe`	2 (high)	`wp-admin`, `wp-login`, `wp-content`, `wp-includes`
`generic_php_probe`	1 (moderate)	Single/double-char/numeric `.php` files
`well_known_probe`	1 (moderate)	`.well-known/` paths

Flagging metadata fields:

errors / denied — hostile hit count
total — all requests from that IP
ratio — error ratio (npm only)
first / last — first and last timestamp seen
malicious_hits — count of malicious-pattern request paths

Whitelisted ranges

Whitelists live in whitelists.py and data/. Loading priority per provider:

Local files in data/<provider>/ (offline-safe, committed to repo)
Remote download (fetches latest, saves to local file on success)
Built-in fallback (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as accept rules placed above the drop rules in the MikroTik script.

Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via whois.radb.net). If ≥ 3 malicious IPs share a subnet, the entire subnet is blocked instead of individual /32 or /128 entries. Results are cached to reports/whois_cache.json.

Log format reference

copyparty

15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye

Nginx Proxy Manager

[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"

Fields: [datetime], upstream status, proxy status, method, scheme, host, "path", [Client IP], [Length N], [Gzip X], [Sent-to IP], "UA", "Referer".

Extending

new whitelist provider → add URLs, local file paths, and fallback CIDRs to app/whitelists.py
new detection pattern → add a BadPattern to app/bad_patterns.py
new detection rule → add a flag_*() function to app/flags.py, return it from the analyzer's flag() method
new log source → subclass app.analyzer.Analyzer, implement parse_line() + flag(), declare ip_columns / request_columns
new output file → override write_extra(data_dir) and return the paths written
new I/O format → add writer to app/files.py
new report section → add rendering logic to app/reporting.py

README.md

All You Can GET Intrusion Detection System