zwitschi/intrusion-detection
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure sources into app package with unified entry point and update documentation

Browse Source
This commit is contained in:
zwitschi committed 2026-08-04 22:23:59 +02:00
1 parent 540154d426
commit f78faf8f8a
18 files changed
+167 -129
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+104 -88
View File
@@ -2,53 +2,59 @@
Local-development intrusion detection / log analysis for two log sources:
| Script | Source | Detection rule |
| ----------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `scan.py` | both sources | consolidated scan + unified MikroTik export |
| `copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `utils.py` | re-export layer | thin barrel — imports from sub-modules below |
| `models.py` | data model | `Request`, `IPStats`, `aggregate()` |
| `ip_utils.py` | IP helpers | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write |
| `whitelists.py` | whitelist module | provider CIDR constants, local/remote load, CIDR extraction |
| `bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `whois.py` | WHOIS query | `query_whois_subnet`, `consolidate_by_subnet` |
| `mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `files.py` | I/O helpers | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc. |
| `reporting.py` | text rendering | `Column`, `render_table`, `render_kv`, `timestamp_name` |
| Script | Source | Detection rule |
| --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `app.py` | unified entry point | runs both analyzers + optional firewall sync |
| `app/scan.py` | consolidated scanner | runs both analyzers, merges blocklists, syncs firewall |
| `app/copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `app/npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `app/utils.py` | re-export layer | thin barrel — imports from sub-modules below |
| `app/models.py` | data model | `Request`, `IPStats`, `aggregate()` |
| `app/ip_utils.py` | IP helpers | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write |
| `app/whitelists.py` | whitelist module | provider CIDR constants, local/remote load, CIDR extraction |
| `app/bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `app/whois.py` | WHOIS query | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | RouterOS SSH client | `MikrotikFirewall` — connect, sync blocklist, audit state |
| `app/files.py` | I/O helpers | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc. |
| `app/reporting.py` | text rendering | `Column`, `render_table`, `render_kv`, `timestamp_name` |
Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
group flagged IPs into WHOIS subnets, and emit a ready-to-import MikroTik
RouterOS firewall script.
`scan.py` runs both analyzers, merges their blocklists, and writes a single
consolidated `.rsc`.
`app.py` runs both analyzers, merges their blocklists, and optionally
synchronizes the MikroTik RouterOS firewall via SSH.
> Status: **local dev**. Runs offline against log files on disk. It does not
> talk to the router, does not run as a service, and blocks nothing by itself.
> You review the generated `.rsc` and import it manually.
> Status: **local dev**. Runs offline against log files on disk. Use
> `--update-firewall` to push changes to the router, or `--dry-run` to
> preview without applying.
## Architecture
```text
intrusion-detection/
├── scan.py consolidated scanner entry point
├── copyparty.py copyparty log adapter (~150 lines)
├── npm.py NPM proxy log adapter (~150 lines)
├── utils.py thin re-export barrel (~90 lines)
├── models.py Request, IPStats, aggregate()
├── ip_utils.py IP parsing, whitelist lookups, sorting
├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
├── analyzer.py Analyzer pipeline base class
├── whitelists.py provider CIDR constants, local/remote load
├── bad_patterns.py malicious URL/file regex patterns
├── whois.py query_whois_subnet, consolidate_by_subnet
├── mikrotik.py RouterOS .rsc script generation
├── files.py read_lines, find_logs, write_csv/json/lines
├── reporting.py Column, render_table, render_kv, timestamp_name
├── app.py unified entry point
├── app/ application package
│ ├── __init__.py
│ ├── scan.py consolidated scanner
│ ├── copyparty.py copyparty log adapter (~150 lines)
│ ├── npm.py NPM proxy log adapter (~150 lines)
│ ├── firewall.py MikroTik RouterOS SSH client
│ ├── utils.py thin re-export barrel (~90 lines)
│ ├── models.py Request, IPStats, aggregate()
│ ├── ip_utils.py IP parsing, whitelist lookups, sorting
│ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
│ ├── analyzer.py Analyzer pipeline base class
│ ├── whitelists.py provider CIDR constants, local/remote load
│ ├── bad_patterns.py malicious URL/file regex patterns
│ ├── whois.py query_whois_subnet, consolidate_by_subnet
│ ├── mikrotik.py RouterOS .rsc script generation
│ ├── files.py read_lines, find_logs, write_csv/json/lines
│ └── reporting.py Column, render_table, render_kv, timestamp_name
├── data/ whitelist source files (cached locally)
│ ├── openfactory/
│ │ └── geofeed.csv
@@ -60,29 +66,28 @@ intrusion-detection/
├── reports/ generated reports (per-run + per-source)
│ ├── copyparty/
│ ├── npm/
│ └── mikrotik-block.rsc consolidated .rsc (scan.py only)
│ └── mikrotik/ firewall audit snapshots
└── README.md
```
`copyparty.py` and `npm.py` are thin adapters. Each supplies a log matcher,
a line parser, a flagging rule, and column definitions. `utils.py` is a barrel
module that re-exports everything from the sub-modules so callers only need
`import utils`.
`app/copyparty.py` and `app/npm.py` are thin adapters. Each supplies a log
matcher, a line parser, a flagging rule, and column definitions.
Core responsibilities by module:
| Module | Owns |
| ----------------- | -------------------------------------------------------------------------------- |
| `models.py` | `Request`, `IPStats`, `aggregate()` |
| `ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `bad_patterns.py` | malicious URL/file regex classification |
| `whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
| Module | Owns |
| --------------------- | -------------------------------------------------------------------------------- |
| `app/models.py` | `Request`, `IPStats`, `aggregate()` |
| `app/ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `app/whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `app/bad_patterns.py` | malicious URL/file regex classification |
| `app/whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | `MikrotikFirewall` — SSH connect, sync blocklist, audit state |
| `app/files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `app/reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
Console output and saved reports are the **same string** (`Analyzer.text_report()`).
@@ -94,50 +99,57 @@ Console output and saved reports are the **same string** (`Analyzer.text_report(
## Usage
```powershell
# individual scanners
python copyparty.py
python npm.py
# unified entry point (both sources, optional firewall sync)
python app.py
python app.py --quiet
python app.py --update-firewall --dry-run
python app.py --update-firewall
# consolidated (both sources, unified blocklist)
python scan.py
# individual scanners (still runnable directly)
python -m app.copyparty
python -m app.npm
```
All scripts accept CLI overrides:
| Flag | Default | Meaning |
| ----------------- | -------------------- | ----------------------------- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--no-rsc` | off | skip MikroTik export |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
| Flag | Default | Meaning |
| ------------------- | -------------------- | ---------------------------------- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
| `--update-firewall` | off | sync blocklist to MikroTik via SSH |
| `--dry-run` | off | preview firewall changes, no apply |
`scan.py` accepts all flags from both scanners plus `--copyparty-log-path`,
`app.py` accepts all flags from both scanners plus `--copyparty-log-path`,
`--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and
`--report-dir` for the consolidated output.
Default log paths (overridable by `.env`):
| Script | Env var | Default |
| -------------- | -------------------- | -------------- |
| `copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
| `npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |
| Script | Env var | Default |
| ------------------ | -------------------- | --------------- |
| `app/copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
| `app/npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |
| `app/firewall.py` | `ROUTER_IP` | `192.168.88.1` |
| | `ROUTER_USER` | `admin` |
| | `ROUTER_PASSWORD` | _(set in .env)_ |
Examples:
```powershell
python npm.py --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python copyparty.py --top 100 --no-rsc
python scan.py --quiet --no-rsc
python -m app.npm --log-path D:\logs\npm --min-errors 10 --error-ratio 0.8
python -m app.copyparty --top 100
python app.py --quiet --update-firewall --dry-run
```
## Output
Each script writes to its own report directory. `scan.py` also writes a
consolidated `.rsc`:
Each script writes to its own report directory. `app.py` also writes
firewall audit snapshots:
```text
reports/
@@ -154,8 +166,7 @@ reports/
│ │ ├── blocked_ipv4.txt final block set (non-whitelisted)
│ │ ├── blocked_ipv6.txt
│ │ ├── authed_users.json authenticated user counts
│ │ ├── requests.csv every parsed request
│ │ └── mikrotik-block.rsc
│ │ └── requests.csv every parsed request
│ └── 2026-08-02-12-00-00.txt timestamped report
├── npm/
│ ├── data/
@@ -169,10 +180,15 @@ reports/
│ │ ├── blocked_ipv6.txt
│ │ ├── ua_summary.csv user agent breakdown
│ │ ├── host_summary.csv virtual host breakdown
│ │ ├── requests.csv
│ │ └── mikrotik-block.rsc
│ │ └── requests.csv
│ └── 2026-08-02-12-00-00.txt
└── mikrotik-block.rsc consolidated (scan.py only)
└── mikrotik/
├── data/
│ ├── address_list_ipv4.json
│ ├── address_list_ipv6.json
│ ├── firewall_ipv4.json
│ └── firewall_ipv6.json
└── 2026-08-04-18-30-28.json firewall audit snapshot
```
## Detection
@@ -259,13 +275,13 @@ Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
## Extending
- **new whitelist provider** → add URLs, local file paths, and fallback CIDRs
to `whitelists.py`
- **new detection pattern** → add a `BadPattern` to `bad_patterns.py`
- **new detection rule** → add a `flag_*()` function to `flags.py`, return it
to `app/whitelists.py`
- **new detection pattern** → add a `BadPattern` to `app/bad_patterns.py`
- **new detection rule** → add a `flag_*()` function to `app/flags.py`, return it
from the analyzer's `flag()` method
- **new log source** → subclass `analyzer.Analyzer`, implement `parse_line()` +
- **new log source** → subclass `app.analyzer.Analyzer`, implement `parse_line()` +
`flag()`, declare `ip_columns` / `request_columns`
- **new output file** → override `write_extra(data_dir)` and return the paths
written
- **new I/O format** → add writer to `files.py`
- **new report section** → add rendering logic to `reporting.py`
- **new I/O format** → add writer to `app/files.py`
- **new report section** → add rendering logic to `app/reporting.py`