All You Can GET Intrusion Detection System
Local-development intrusion detection / log analysis for two log sources:
| Script | Source | Detection rule |
|---|---|---|
app.py |
unified entry point | runs both analyzers + optional firewall sync |
app/scan/ |
consolidated scanner | runs both analyzers, merges blocklists, syncs firewall |
app/copyparty.py |
copyparty server logs | denied-only IPs + copyparty auto-bans |
app/npm.py |
Nginx Proxy Manager proxy logs | high error-rate IPs (scanners) |
app/utils/ |
shared utilities | constants, I/O helpers, IP utils, models, reporting, whitelists, WHOIS |
app/web/ |
Flask web UI | app factory, routes, background scan runner |
app/analyzer.py |
pipeline | Analyzer — discover → load → analyze → report → write |
app/flags.py |
detection rules | flag_scanners, flag_denied_only, flag_malicious_patterns, by_total |
app/bad_patterns.py |
pattern detection | malicious URL/file regex classification |
app/mikrotik.py |
RouterOS export | mikrotik_script, write_mikrotik, split_cidr_versions |
app/firewall.py |
RouterOS SSH client | MikrotikFirewall — connect, sync blocklist, audit state |
Both per-source scripts parse logs, aggregate per-IP behaviour, split IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns, group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik RouterOS firewall script.
app.py runs both analyzers, merges their blocklists, and optionally
synchronizes the MikroTik RouterOS firewall via SSH.
Status: local dev. Runs offline against log files on disk. Use
--update-firewallto push changes to the router, or--dry-runto preview without applying.
Architecture
intrusion-detection/
├── app.py unified entry point
├── app/ application package
│ ├── __init__.py
│ ├── analyzer.py Analyzer pipeline base class
│ ├── bad_patterns.py malicious URL/file regex patterns
│ ├── copyparty.py copyparty log adapter (~150 lines)
│ ├── firewall.py MikroTik RouterOS SSH client
│ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
│ ├── mikrotik.py RouterOS .rsc script generation
│ ├── npm.py NPM proxy log adapter (~150 lines)
│ ├── web/ Flask web UI
│ │ ├── __init__.py app factory + helpers + serve()
│ │ ├── routes.py all HTTP route definitions
│ │ └── web_scan.py background scan runner + in-memory job store
│ ├── scan/ scan pipeline
│ │ ├── __init__.py consolidated scanner (run_scan, ScanParams, ScanResult)
│ │ └── scan_state.py incremental log scan state (load/save)
│ ├── db/ SQLite store (see .github/instructions/DATABASE.md)
│ │ ├── __init__.py connection factory + migrations runner
│ │ ├── schema.py DDL for all tables (PRAGMA user_version migrations)
│ │ └── repositories.py queries: logs, scans, blocklist, whitelist, firewall
│ ├── utils/ shared utilities
│ │ ├── constants.py all config constants (paths, env vars, thresholds)
│ │ ├── files.py read_lines, find_logs, write_csv/json/lines
│ │ ├── ip_utils.py IP parsing, whitelist lookups, sorting
│ │ ├── models.py Request, IPStats, aggregate()
│ │ ├── reporting.py Column, render_table, render_kv, timestamp_name
│ │ ├── utils.py shared CLI arg parser
│ │ ├── whitelists.py provider CIDR constants, local/remote load
│ │ ├── whois.py query_whois_subnet, consolidate_by_subnet
│ │ ├── blocklist_history.py git-backed cumulative blocklist
│ │ └── retention.py report file retention & pruning
│ ├── static/ style.css + app.js (dark-theme UI)
│ └── templates/ Jinja2 page templates (dashboard, scan, reports, …)
├── data/ whitelist source files (cached locally)
│ ├── init7/ (ranges built into app/utils/whitelists.py — no geofeed)
│ ├── cloudflare/
│ │ ├── ips-v4.txt
│ │ └── ips-v6.txt
│ ├── paloalto/
│ │ └── scanner-ips.txt
│ └── ids.db local SQLite store (log/scan/blocklist/whitelist history; auto-created)
├── reports/ generated reports (per-run + per-source)
│ ├── copyparty/
│ ├── npm/
│ └── mikrotik/ firewall audit snapshots
└── README.md
app/copyparty.py and app/npm.py are thin adapters. Each supplies a log
matcher, a line parser, a flagging rule, and column definitions.
Core responsibilities by module:
| Module | Owns |
|---|---|
app/utils/constants.py |
all config constants — paths, env vars, network/scan thresholds |
app/utils/models.py |
Request, IPStats, aggregate() |
app/utils/ip_utils.py |
parse_ip, whitelist_of, is_whitelisted, sort_ips, split_versions |
app/flags.py |
flag_scanners, flag_denied_only, flag_malicious_patterns, by_total |
app/analyzer.py |
Analyzer — discover → load → analyze → report → write |
app/utils/whitelists.py |
provider CIDR constants, local-file / remote / built-in loading |
app/bad_patterns.py |
malicious URL/file regex classification |
app/utils/whois.py |
query_whois_subnet, consolidate_by_subnet |
app/mikrotik.py |
mikrotik_script, write_mikrotik, split_cidr_versions |
app/firewall.py |
MikrotikFirewall — SSH connect, sync blocklist, audit state |
app/web/ |
Flask app factory, JSON API routes, page rendering, background scan runner |
app/scan/ |
consolidated scan pipeline, incremental scan state |
app/utils/files.py |
read_lines, find_logs, validate_dir, ensure_dirs, write_csv/json/lines |
app/utils/reporting.py |
Column, render_table, render_kv, timestamp_name |
app/utils/blocklist_history.py |
git-backed cumulative blocklist reconstruction |
app/utils/retention.py |
report file retention & pruning |
Console output and saved reports are the same string (Analyzer.text_report()).
Requirements
- Python 3.10+
paramiko(MikroTik SSH),python-dotenv(.envloading),flask(web UI)
pip install -r requirements.txt
Configuration
All tunable settings live in app/constants.py and can be overridden per
deployment via environment variables. For local development, copy
.env.example to .env and fill in your values:
copy .env.example .env
app/constants.py loads .env with python-dotenv at startup. Real system
environment variables always take precedence over .env values, so
production can set the same variables via the OS and skip .env entirely.
.env is git-ignored — do not commit secrets to it.
| Env var | Default | Used by |
|---|---|---|
COPYPARTY_LOG_PATH |
P:\copyparty |
copyparty scanner |
NPM_LOG_PATH |
Y:\npm\logs |
npm scanner |
ROUTER_IP |
192.168.88.1 |
MikroTik connection |
ROUTER_USER |
admin |
MikroTik connection |
ROUTER_PASSWORD |
(set in .env) | MikroTik connection |
BLOCKLIST_NAME |
Blocked |
MikroTik address list |
WHITELIST_NAMES |
Cloudflare,Init7 |
MikroTik whitelists |
WEB_HOST |
127.0.0.1 |
web UI bind host |
WEB_PORT |
54666 |
web UI port |
See .env.example for the complete list of supported variables (WHOIS
tuning, scan thresholds, whitelist source URLs, and more).
Network context (2026-09-08)
Connectivity provider switched Openfactory → Init7 on 2026-09-08:
- WAN IPv4:
77.74.80.159/32(init777.74.80.0/24, "Hybrid7-NET"). - IPv6: delegated
/482001:1620:549f::/48(init7 allocation2001:1620::/30). - Upstream DNS:
77.109.128.2,213.144.129.20.
The Openfactory whitelist (old AS58299 geofeed) is obsolete and must be
replaced with an Init7 whitelist — see
.github/instructions/TODO.md for the
switchover plan and the verified init7 CIDR ranges.
Usage
# web UI (default entry point)
python app.py
python app.py --web-port 54666
# consolidated CLI scan (scan subcommand)
python app.py scan
python app.py scan --quiet
python app.py scan --update-firewall --dry-run
python app.py scan --update-firewall
# detailed log-analysis report from the SQLite store (needs scans run first)
python app.py analyze # top IPs + cross-source + blocklist
python app.py analyze --ip 1.2.3.4 # drill into one IP's request history
python app.py analyze --json # machine-readable output
# individual scanners (still runnable directly)
python -m app.copyparty
python -m app.npm
All scripts accept CLI overrides:
| Flag | Default | Meaning |
|---|---|---|
--log-path |
see below | log directory |
--report-dir |
reports/<name> |
output directory |
--top N |
30 |
rows in the top-IP table |
--quiet |
off | write files, print only paths |
--min-errors N |
5 (npm.py) |
error count threshold |
--error-ratio F |
0.5 (npm.py) |
error ratio threshold |
--min-denied N |
1 (copyparty.py) |
denied-hit threshold |
--update-firewall |
off | sync blocklist to MikroTik via SSH |
--dry-run |
off | preview firewall changes, no apply |
--web-host |
127.0.0.1 |
web UI bind host |
--web-port |
54666 |
web UI port |
python app.py scan accepts all flags from both scanners plus
--copyparty-log-path, --copyparty-report-dir, --npm-log-path,
--npm-report-dir, and --report-dir for the consolidated output.
Examples:
python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python -m app.copyparty --top 100
python app.py scan --quiet --update-firewall --dry-run
Blocklist behavior (cumulative / grow-only)
The MikroTik Blocked address list is cumulative: it only grows over
time and is never trimmed automatically.
- An IP (or subnet) that the system has ever blocked stays blocked — even if it stops appearing in recent logs (for example after it leaves the NPM ~30-day log-retention window, or once WHOIS subnet consolidation groups it into a larger range).
- The source of truth for "all blocked IPs" is the SQLite blocklist
ledger (
app.db→blocklist_entries/blocklist_events), not git history. Each scan syncs the desired blocklist into the ledger; the router's cumulative state is derived from it.app/utils/blocklist_history.pyis a read-only fallback only. - The sync only ever adds entries. The only automatic removals are router entries that were never blocked by the system — i.e. absent from the ledger (manual / rogue entries).
- To explicitly unblock an IP, remove it manually via the web UI
(
/blocklist→ remove) or directly on the router; the automated sync will not re-add entries that are not part of a scan.
Deployment note: runtime data (blocklists, reports, the DB) is never committed to git. Keeping it untracked is what makes
git pullupdates conflict-free — seeDEPLOYMENT.mdand thedeploy/scripts.
Each consolidated scan with --update-firewall also writes a timestamped
change report to reports/app/YYYY-MM-DD-hh-mm-ss.json: the number and list
of entries added / removed (complete-set delta), the full cumulative blocked
list, and the router sync changes. It also persists a firewall snapshot into
the firewall_snapshots SQLite tables for long-term audit (F001).
Report retention
Generated files under reports/ are pruned automatically after each
consolidated scan: every subdirectory keeps at most
REPORT_RETENTION_MAX_FILES files (default 10), all newer than
REPORT_RETENTION_DAYS days (default 30). The blocklist files
(reports/app/data/blocklist_ipv4.txt / blocklist_ipv6.txt) are always
protected and never pruned. Tune via REPORT_RETENTION_DAYS /
REPORT_RETENTION_MAX_FILES; --dry-run previews the cleanup without
deleting.
Web UI
python app.py starts a localhost-only Flask web UI (default
http://127.0.0.1:54666) for running scans and managing the MikroTik
blocklist / firewall. It is bound to 127.0.0.1 — do not expose
WEB_HOST beyond localhost.
Pages:
| Route | Purpose |
|---|---|
/ |
dashboard — status cards + latest scans |
/scan |
run copyparty / npm / consolidated scans (background) |
/reports |
browse reports per source |
/blocklist |
view the consolidated blocklist, add/remove entries |
/firewall |
view address lists + rules, preview/apply a sync |
/whitelist |
view/manage provider whitelist CIDRs |
/graph |
visualize the 0.0.0.0/8 address space (blocked/whitelisted pixels) |
/analysis |
detailed persisted-log analysis (top IPs, drill-down, cross-source) |
Scans run in a background thread (only one at a time) and report progress via
the JSON API, so the UI never blocks. Firewall sync is dry-run by default
from the UI — applying changes is an explicit action. The main JSON endpoints
(/api/status, /api/scan/*, /api/reports/*, /api/blocklist,
/api/firewall/*, /api/whitelist) are also usable directly from a REST
client.
Deployment & updating (git pull)
The app runs scans continuously (web UI + cron + CLI), and all of it is
intended to be updated on the server with a plain git pull. To make that
safe — i.e. never conflict with a file a scan wrote since the last pull —
runtime data is strictly never committed to git:
app/data/*,app/reports/*and the SQLite DB (app/data/ids.db) are gitignored. The only tracked file underdata/is the read-onlyapp/data/paloalto/scanner-ips.txtseed.- The Cloudflare
ips-*.txtwhitelist seeds are not tracked: they are runtime-refreshed downloads and are re-fetched if absent (built-in ranges are the fallback). - History that must survive — the cumulative blocklist and firewall audit — lives in the SQLite store, not in git revisions.
Update a deployed instance with:
sudo deploy/deploy.sh
See DEPLOYMENT.md for the full plan,
and deploy/ for the scripts (deploy.sh, restart-service.sh,
check_runtime_git.sh, ids.service systemd template). The wrapper
fast-forwards only (git pull --ff-only), refuses to run on an unclean tree,
reinstalls deps if requirements.txt changed, applies DB migrations
(python -m app.db --migrate), and restarts the ids unit.
Output
Each script writes to its own report directory. app.py also writes
consolidated blocklists and firewall audit snapshots:
reports/
├── copyparty/
│ ├── data/
│ │ ├── ips_ipv4.csv per-IP: total, denied, banned, users
│ │ ├── ips_ipv6.csv
│ │ ├── ips.json full per-IP detail w/ malicious hits
│ │ ├── denied_paths.csv denied paths, hit counts
│ │ ├── denied_ips.txt flagged IPv4 + metadata
│ │ ├── denied_ipv6.txt
│ │ ├── banned_ips.txt copyparty auto-banned IPv4
│ │ ├── banned_ipv6.txt
│ │ ├── blocked_ipv4.txt final block set (non-whitelisted)
│ │ ├── blocked_ipv6.txt
│ │ ├── authed_users.json authenticated user counts
│ │ └── requests.csv every parsed request
│ └── 2026-08-02-12-00-00.txt timestamped report
├── npm/
│ ├── data/
│ │ ├── ips_ipv4.csv per-IP: total, 4xx, 5xx, ratio, methods
│ │ ├── ips_ipv6.csv
│ │ ├── ips.json
│ │ ├── error_paths.csv
│ │ ├── scanner_ips.txt flagged IPv4 + metadata
│ │ ├── scanner_ipv6.txt
│ │ ├── blocked_ipv4.txt
│ │ ├── blocked_ipv6.txt
│ │ ├── ua_summary.csv user agent breakdown
│ │ ├── host_summary.csv virtual host breakdown
│ │ └── requests.csv
│ └── 2026-08-02-12-00-00.txt
├── app/
│ └── data/
│ ├── blocklist_ipv4.txt final consolidated IPv4 blocklist (CIDRs)
│ └── blocklist_ipv6.txt final consolidated IPv6 blocklist (CIDRs)
└── mikrotik/
├── data/
│ ├── address_list_ipv4.json
│ ├── address_list_ipv6.json
│ ├── firewall_ipv4.json
│ └── firewall_ipv6.json
└── 2026-08-04-18-30-28.json firewall audit snapshot
The current router state (address lists + firewall rules) is also mirrored to
data/mikrotik/ on every --update-firewall run as a scratch pointer to the
latest snapshot. Those files are overwritten each run and are not tracked;
the long-term audit trail is the SQLite firewall_snapshots tables plus the
timestamped reports under reports/mikrotik/. git pull is therefore never
blocked by firewall-state files.
Detection
Three detection layers
Blocklist candidates come from three independent sources, merged at scan time:
-
Error-rate flagging (
npm.py) — IPs with many 4xx/5xx responses and a high error ratio:\text{errors} \geq 5 \qquad \text{and} \qquad \frac{\text{errors}}{\text{total}} \geq 0.5 -
Denied-only flagging (
copyparty.py) — IPs that hit denied paths and never authenticated:\text{block} = \text{denied-only} \cup \text{banned} \setminus \text{whitelist} -
Malicious-pattern flagging (both analyzers) — IPs whose request paths match known probe/attack signatures (
.env,.git/config, WordPress probes, webshell filenames, etc.) at severity ≥ 2.
Pattern categories (bad_patterns.py)
| Pattern | Severity | Matches |
|---|---|---|
env_file |
3 (critical) | .env, .env.production, .aws/credentials, config.json |
vcs_probe |
3 (critical) | .git/config, .svn/ |
sensitive_file |
3 (critical) | id_rsa, id_dsa, .pem, .key, .ppk |
webshell_probe |
3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
wp_probe |
2 (high) | wp-admin, wp-login, wp-content, wp-includes |
generic_php_probe |
1 (moderate) | Single/double-char/numeric .php files |
well_known_probe |
1 (moderate) | .well-known/ paths |
Flagging metadata fields:
errors/denied— hostile hit counttotal— all requests from that IPratio— error ratio (npm only)first/last— first and last timestamp seenmalicious_hits— count of malicious-pattern request paths
Whitelisted ranges
Whitelists live in whitelists.py and data/. Loading priority per provider:
- Local files in
data/<provider>/(offline-safe, committed to repo) - Remote download (fetches latest, saves to local file on success)
- Built-in fallback (hardcoded constants, always available)
Each layer overrides the previous. Known-good ranges are emitted as
accept rules placed above the drop rules in the MikroTik script.
Subnet consolidation
Blocked IPs are grouped by WHOIS subnet (via whois.radb.net). If ≥ 3
malicious IPs share a subnet, the entire subnet is blocked instead of
individual /32 or /128 entries. Results are cached to
reports/whois_cache.json.
Log format reference
copyparty
15:55:47.466 34.105.30.251 37140 LOG: GET //2020/wp-includes/wlwmanifest.xml @*
15:55:47.466 34.105.30.251 37140 LOG: @* has no access to '/2020/wp-includes/wlwmanifest.xml'
15:55:47.466 34.105.30.251 37140 CRIT: client banned: 403s
15:55:47.695 34.105.30.251 37144 BTW: banned for 86399 sec
16:00:34.385 hsrv LOG: ok bye
Nginx Proxy Manager
[02/Aug/2026:05:01:31 +0200] - 401 401 - GET https example.tld "/wp-content/plugins/x.php" [Client 72.146.20.230] [Length 2127] [Gzip -] [Sent-to 192.168.88.12] "-" "-"
[02/Aug/2026:09:01:11 +0200] - 200 200 - GET https example.tld "/" [Client 2a00:6340:2008:ec80::1] [Length 1651] [Gzip 2.97] [Sent-to 192.168.88.12] "Mozilla/5.0 ..." "-"
Fields: [datetime], upstream status, proxy status, method, scheme, host,
"path", [Client IP], [Length N], [Gzip X], [Sent-to IP], "UA",
"Referer".
Extending
- new whitelist provider → add URLs, local file paths, and fallback CIDRs
to
app/whitelists.py - new detection pattern → add a
BadPatterntoapp/bad_patterns.py - new detection rule → add a
flag_*()function toapp/flags.py, return it from the analyzer'sflag()method - new log source → subclass
app.analyzer.Analyzer, implementparse_line()+flag(), declareip_columns/request_columns - new output file → override
write_extra(data_dir)and return the paths written - new I/O format → add writer to
app/files.py - new report section → add rendering logic to
app/reporting.py