intrusion-detection/README.md

allucanget-ids

Local-development intrusion detection / log analysis for two log sources:

Script	Source	Detection rule
copyparty.py	copyparty server logs	denied-only IPs + copyparty auto-bans
npm.py	Nginx Proxy Manager proxy logs	high error-rate IPs (scanners)
scan.py	both of the above	consolidated scan + unified MikroTik export
whitelists.py	whitelist module	provider CIDR loading, refresh, validation
bad_patterns.py	pattern detection	malicious URL/file regex classification
utils.py	shared core (no CLI)	aggregation, reports, rsc, pipeline

Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script.

scan.py runs both analyzers, merges their blocklists, and writes a single consolidated .rsc.

Status: local dev. Runs offline against log files on disk. It does not talk to the router, does not run as a service, and blocks nothing by itself. You review the generated .rsc and import it manually.

Architecture

allucanget-ids/
├── scan.py              consolidated scanner entry point
├── copyparty.py         copyparty log adapter (~150 lines)
├── npm.py               NPM proxy log adapter (~150 lines)
├── whitelists.py        whitelist constants, download, CIDR extraction
├── bad_patterns.py      malicious URL/file regex patterns
├── utils.py             shared core: aggregation, reporting, MikroTik export
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik-block.rsc   consolidated .rsc (scan.py only)
└── README.md

copyparty.py and npm.py are thin adapters. Each supplies a log matcher, a line parser, a flagging rule, and column definitions. Everything else lives in utils.py:

• Request — normalised request record
• IPStats + aggregate() — per-IP counters, malicious-hit tracking
• flag_scanners() / flag_denied_only() / flag_malicious_patterns() — detection rules
• whitelist_of() / is_whitelisted() — whitelist lookups (backed by whitelists.py)
• is_malicious() / classify_path() — pattern classification (backed by bad_patterns.py)
• query_whois_subnet() / consolidate_by_subnet() — WHOIS subnet grouping
• IP version helpers, log discovery, transparent .gz, ANSI stripping
• CSV/JSON/table writers, mikrotik_script() — RouterOS export
• Analyzer — pipeline: discover → load → analyze → report → write

Console output and saved reports are the same string (Analyzer.text_report()).

Requirements

• Python 3.10+
• stdlib only, no dependencies

Usage

# individual scanners
python copyparty.py
python npm.py

# consolidated (both sources, unified blocklist)
python scan.py

All scripts accept CLI overrides:

Flag	Default	Meaning
--log-path	see below	log directory
--report-dir	reports/<name>	output directory
--top N	30	rows in the top-IP table
--quiet	off	write files, print only paths
--no-rsc	off	skip MikroTik export
--min-errors N	5 (npm.py)	error count threshold
--error-ratio F	0.5 (npm.py)	error ratio threshold
--min-denied N	1 (copyparty.py)	denied-hit threshold

scan.py accepts all flags from both scanners plus --copyparty-log-path, --copyparty-report-dir, --npm-log-path, --npm-report-dir, and --report-dir for the consolidated output.

Default log paths (overridable by .env):

Script	Env var	Default
copyparty.py	COPYPARTY_LOG_PATH	P:\copyparty
npm.py	NPM_LOG_PATH	Y:\npm\logs

Examples:

python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python copyparty.py --top 100 --no-rsc
python scan.py --quiet --no-rsc

Output

Each script writes to its own report directory. scan.py also writes a consolidated .rsc:

reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   ├── requests.csv          every parsed request
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   ├── requests.csv
│   │   └── mikrotik-block.rsc
│   └── 2026-08-02-12-00-00.txt
└── mikrotik-block.rsc             consolidated (scan.py only)

Detection

Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

1. Error-rate flagging (npm.py) — IPs with many 4xx/5xx responses and a high error ratio:

   \text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5

2. Denied-only flagging (copyparty.py) — IPs that hit denied paths and never authenticated:

   \text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}

3. Malicious-pattern flagging (both analyzers) — IPs whose request paths match known probe/attack signatures (.env, .git/config, WordPress probes, webshell filenames, etc.) at severity ≥ 2.

Pattern categories (bad_patterns.py)

Pattern	Severity	Matches
env_file	3 (critical)	.env, .env.production, .aws/credentials, config.json
vcs_probe	3 (critical)	.git/config, .svn/
sensitive_file	3 (critical)	id_rsa, id_dsa, .pem, .key, .ppk
webshell_probe	3 (critical)	Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.)
wp_probe	2 (high)	wp-admin, wp-login, wp-content, wp-includes
generic_php_probe	1 (moderate)	Single/double-char/numeric .php files
well_known_probe	1 (moderate)	.well-known/ paths

Flagging metadata fields:

• errors / denied — hostile hit count
• total — all requests from that IP
• ratio — error ratio (npm only)
• first / last — first and last timestamp seen
• malicious_hits — count of malicious-pattern request paths

Whitelisted ranges

Whitelists live in whitelists.py and data/. Loading priority per provider:

1. Local files in data/<provider>/ (offline-safe, committed to repo)
2. Remote download (fetches latest, saves to local file on success)
3. Built-in fallback (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as accept rules placed above the drop rules in the MikroTik script.

Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via whois.radb.net). If ≥ 3 malicious IPs share a subnet, the entire subnet is blocked instead of individual /32 or /128 entries. Results are cached to reports/whois_cache.json.

Log format reference

copyparty

15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye

Nginx Proxy Manager

[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"

Fields: [datetime], upstream status, proxy status, method, scheme, host, "path", [Client IP], [Length N], [Gzip X], [Sent-to IP], "UA", "Referer".

Extending

• new whitelist provider → add URLs, files, and fallback CIDRs to whitelists.py
• new detection rule → add a BadPattern to bad_patterns.py or a flag_*() function to utils.py, return it from the analyzer's flag()
• new log source → subclass utils.Analyzer, implement parse_line() + flag(), declare ip_columns / request_columns
• new output file → override write_extra(data_dir) and return the paths written