intrusion-detection/README.md

All You Can GET Intrusion Detection System

Local-development intrusion detection / log analysis for two log sources:

Script	Source	Detection rule
app.py	unified entry point	runs both analyzers + optional firewall sync
app/scan.py	consolidated scanner	runs both analyzers, merges blocklists, syncs firewall
app/copyparty.py	copyparty server logs	denied-only IPs + copyparty auto-bans
app/npm.py	Nginx Proxy Manager proxy logs	high error-rate IPs (scanners)
app/utils.py	re-export layer	thin barrel — imports from sub-modules below
app/models.py	data model	Request, IPStats, aggregate()
app/ip_utils.py	IP helpers	parse_ip, whitelist_of, is_whitelisted, sort_ips, split_versions
app/flags.py	detection rules	flag_scanners, flag_denied_only, flag_malicious_patterns, by_total
app/analyzer.py	pipeline	Analyzer — discover → load → analyze → report → write
app/whitelists.py	whitelist module	provider CIDR constants, local/remote load, CIDR extraction
app/bad_patterns.py	pattern detection	malicious URL/file regex classification
app/whois.py	WHOIS query	query_whois_subnet, consolidate_by_subnet
app/mikrotik.py	RouterOS export	mikrotik_script, write_mikrotik, split_cidr_versions
app/firewall.py	RouterOS SSH client	MikrotikFirewall — connect, sync blocklist, audit state
app/files.py	I/O helpers	read_lines, find_logs, write_csv, write_json, etc.
app/reporting.py	text rendering	Column, render_table, render_kv, timestamp_name

Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script.

app.py runs both analyzers, merges their blocklists, and optionally synchronizes the MikroTik RouterOS firewall via SSH.

Status: local dev. Runs offline against log files on disk. Use --update-firewall to push changes to the router, or --dry-run to preview without applying.

Architecture

intrusion-detection/
├── app.py               unified entry point
├── app/                 application package
│   ├── __init__.py
│   ├── constants.py     all config constants (paths, env vars, thresholds)
│   ├── scan.py          consolidated scanner
│   ├── copyparty.py     copyparty log adapter (~150 lines)
│   ├── npm.py           NPM proxy log adapter (~150 lines)
│   ├── firewall.py      MikroTik RouterOS SSH client
│   ├── utils.py         thin re-export barrel (~90 lines)
│   ├── models.py        Request, IPStats, aggregate()
│   ├── ip_utils.py      IP parsing, whitelist lookups, sorting
│   ├── flags.py         flag_scanners, flag_denied_only, flag_malicious_patterns
│   ├── analyzer.py      Analyzer pipeline base class
│   ├── whitelists.py    provider CIDR constants, local/remote load
│   ├── bad_patterns.py  malicious URL/file regex patterns
│   ├── whois.py         query_whois_subnet, consolidate_by_subnet
│   ├── mikrotik.py      RouterOS .rsc script generation
│   ├── files.py         read_lines, find_logs, write_csv/json/lines
│   └── reporting.py     Column, render_table, render_kv, timestamp_name
├── data/                whitelist source files (cached locally)
│   ├── openfactory/
│   │   └── geofeed.csv
│   ├── cloudflare/
│   │   ├── ips-v4.txt
│   │   └── ips-v6.txt
│   └── paloalto/
│       └── scanner-ips.txt
├── reports/             generated reports (per-run + per-source)
│   ├── copyparty/
│   ├── npm/
│   └── mikrotik/        firewall audit snapshots
└── README.md

app/copyparty.py and app/npm.py are thin adapters. Each supplies a log matcher, a line parser, a flagging rule, and column definitions.

Core responsibilities by module:

Module	Owns
app/constants.py	all config constants — paths, env vars, network/scan thresholds
app/models.py	Request, IPStats, aggregate()
app/ip_utils.py	parse_ip, whitelist_of, is_whitelisted, sort_ips, split_versions
app/flags.py	flag_scanners, flag_denied_only, flag_malicious_patterns, by_total
app/analyzer.py	Analyzer — discover → load → analyze → report → write
app/whitelists.py	provider CIDR constants, local-file / remote / built-in loading
app/bad_patterns.py	malicious URL/file regex classification
app/whois.py	query_whois_subnet, consolidate_by_subnet
app/mikrotik.py	mikrotik_script, write_mikrotik, split_cidr_versions
app/firewall.py	MikrotikFirewall — SSH connect, sync blocklist, audit state
app/files.py	read_lines, find_logs, validate_dir, ensure_dirs, write_csv/json/lines
app/reporting.py	Column, render_table, render_kv, timestamp_name

Console output and saved reports are the same string (Analyzer.text_report()).

Requirements

• Python 3.10+
• paramiko (MikroTik SSH), python-dotenv (.env loading)

pip install -r requirements.txt

Configuration

All tunable settings live in app/constants.py and can be overridden per deployment via environment variables. For local development, copy .env.example to .env and fill in your values:

copy .env.example .env

app/constants.py loads .env with python-dotenv at startup. Real system environment variables always take precedence over .env values, so production can set the same variables via the OS and skip .env entirely. .env is git-ignored — do not commit secrets to it.

Env var	Default	Used by
COPYPARTY_LOG_PATH	P:\copyparty	copyparty scanner
NPM_LOG_PATH	Y:\npm\logs	npm scanner
ROUTER_IP	192.168.88.1	MikroTik connection
ROUTER_USER	admin	MikroTik connection
ROUTER_PASSWORD	(set in .env)	MikroTik connection
BLOCKLIST_NAME	Blocked	MikroTik address list
WHITELIST_NAMES	Cloudflare,Openfactory	MikroTik whitelists

See .env.example for the complete list of supported variables (WHOIS tuning, scan thresholds, whitelist source URLs, and more).

Usage

# unified entry point (both sources, optional firewall sync)
python app.py
python app.py --quiet
python app.py --update-firewall --dry-run
python app.py --update-firewall

# individual scanners (still runnable directly)
python -m app.copyparty
python -m app.npm

All scripts accept CLI overrides:

Flag	Default	Meaning
--log-path	see below	log directory
--report-dir	reports/<name>	output directory
--top N	30	rows in the top-IP table
--quiet	off	write files, print only paths
--min-errors N	5 (npm.py)	error count threshold
--error-ratio F	0.5 (npm.py)	error ratio threshold
--min-denied N	1 (copyparty.py)	denied-hit threshold
--update-firewall	off	sync blocklist to MikroTik via SSH
--dry-run	off	preview firewall changes, no apply

app.py accepts all flags from both scanners plus --copyparty-log-path, --copyparty-report-dir, --npm-log-path, --npm-report-dir, and --report-dir for the consolidated output.

Examples:

python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python -m app.copyparty --top 100
python app.py --quiet --update-firewall --dry-run

Output

Each script writes to its own report directory. app.py also writes consolidated blocklists and firewall audit snapshots:

reports/
├── copyparty/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, denied, banned, users
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json              full per-IP detail w/ malicious hits
│   │   ├── denied_paths.csv      denied paths, hit counts
│   │   ├── denied_ips.txt        flagged IPv4 + metadata
│   │   ├── denied_ipv6.txt
│   │   ├── banned_ips.txt        copyparty auto-banned IPv4
│   │   ├── banned_ipv6.txt
│   │   ├── blocked_ipv4.txt      final block set (non-whitelisted)
│   │   ├── blocked_ipv6.txt
│   │   ├── authed_users.json     authenticated user counts
│   │   └── requests.csv          every parsed request
│   └── 2026-08-02-12-00-00.txt   timestamped report
├── npm/
│   ├── data/
│   │   ├── ips_ipv4.csv          per-IP: total, 4xx, 5xx, ratio, methods
│   │   ├── ips_ipv6.csv
│   │   ├── ips.json
│   │   ├── error_paths.csv
│   │   ├── scanner_ips.txt       flagged IPv4 + metadata
│   │   ├── scanner_ipv6.txt
│   │   ├── blocked_ipv4.txt
│   │   ├── blocked_ipv6.txt
│   │   ├── ua_summary.csv        user agent breakdown
│   │   ├── host_summary.csv      virtual host breakdown
│   │   └── requests.csv
│   └── 2026-08-02-12-00-00.txt
├── app/
│   └── data/
│       ├── blocklist_ipv4.txt    final consolidated IPv4 blocklist (CIDRs)
│       └── blocklist_ipv6.txt    final consolidated IPv6 blocklist (CIDRs)
└── mikrotik/
    ├── data/
    │   ├── address_list_ipv4.json
    │   ├── address_list_ipv6.json
    │   ├── firewall_ipv4.json
    │   └── firewall_ipv6.json
    └── 2026-08-04-18-30-28.json   firewall audit snapshot

The current router state (address lists + firewall rules) is also mirrored to data/mikrotik/ on every --update-firewall run. Those files are overwritten each run and are git-visible, so git diff shows how the firewall rules and address lists change over time.

Detection

Three detection layers

Blocklist candidates come from three independent sources, merged at scan time:

1. Error-rate flagging (npm.py) — IPs with many 4xx/5xx responses and a high error ratio:

   \text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5

2. Denied-only flagging (copyparty.py) — IPs that hit denied paths and never authenticated:

   \text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist}

3. Malicious-pattern flagging (both analyzers) — IPs whose request paths match known probe/attack signatures (.env, .git/config, WordPress probes, webshell filenames, etc.) at severity ≥ 2.

Pattern categories (bad_patterns.py)

Pattern	Severity	Matches
env_file	3 (critical)	.env, .env.production, .aws/credentials, config.json
vcs_probe	3 (critical)	.git/config, .svn/
sensitive_file	3 (critical)	id_rsa, id_dsa, .pem, .key, .ppk
webshell_probe	3 (critical)	Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.)
wp_probe	2 (high)	wp-admin, wp-login, wp-content, wp-includes
generic_php_probe	1 (moderate)	Single/double-char/numeric .php files
well_known_probe	1 (moderate)	.well-known/ paths

Flagging metadata fields:

• errors / denied — hostile hit count
• total — all requests from that IP
• ratio — error ratio (npm only)
• first / last — first and last timestamp seen
• malicious_hits — count of malicious-pattern request paths

Whitelisted ranges

Whitelists live in whitelists.py and data/. Loading priority per provider:

1. Local files in data/<provider>/ (offline-safe, committed to repo)
2. Remote download (fetches latest, saves to local file on success)
3. Built-in fallback (hardcoded constants, always available)

Each layer overrides the previous. Known-good ranges are emitted as accept rules placed above the drop rules in the MikroTik script.

Subnet consolidation

Blocked IPs are grouped by WHOIS subnet (via whois.radb.net). If ≥ 3 malicious IPs share a subnet, the entire subnet is blocked instead of individual /32 or /128 entries. Results are cached to reports/whois_cache.json.

Log format reference

copyparty

15:55:47.466 34.105.30.251 37140    LOG: GET  //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140    LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140   CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144    BTW: banned for 86399 sec
16:00:34.385 hsrv                   LOG: ok bye

Nginx Proxy Manager

[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"

Fields: [datetime], upstream status, proxy status, method, scheme, host, "path", [Client IP], [Length N], [Gzip X], [Sent-to IP], "UA", "Referer".

Extending

• new whitelist provider → add URLs, local file paths, and fallback CIDRs to app/whitelists.py
• new detection pattern → add a BadPattern to app/bad_patterns.py
• new detection rule → add a flag_*() function to app/flags.py, return it from the analyzer's flag() method
• new log source → subclass app.analyzer.Analyzer, implement parse_line() + flag(), declare ip_columns / request_columns
• new output file → override write_extra(data_dir) and return the paths written
• new I/O format → add writer to app/files.py
• new report section → add rendering logic to app/reporting.py