zwitschi/intrusion-detection
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor monolith utils into domain modules and update docs

Browse Source
This commit is contained in:
zwitschi committed 2026-08-03 17:01:36 +02:00
1 parent 970e1b5eed
commit bbf07dec5a
11 files changed
+972 -54
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+76 -51
View File
@@ -1,15 +1,23 @@
# allucanget-ids
# All You Can GET Intrusion Detection System
Local-development intrusion detection / log analysis for two log sources:
| Script | Source | Detection rule |
| --- | --- | --- |
| `copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `scan.py` | both of the above | consolidated scan + unified MikroTik export |
| `whitelists.py` | whitelist module | provider CIDR loading, refresh, validation |
| `bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `utils.py` | shared core (no CLI) | aggregation, reports, rsc, pipeline |
| Script | Source | Detection rule |
| ----------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `scan.py` | both sources | consolidated scan + unified MikroTik export |
| `copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `utils.py` | re-export layer | thin barrel — imports from sub-modules below |
| `models.py` | data model | `Request`, `IPStats`, `aggregate()` |
| `ip_utils.py` | IP helpers | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write |
| `whitelists.py` | whitelist module | provider CIDR constants, local/remote load, CIDR extraction |
| `bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `whois.py` | WHOIS query | `query_whois_subnet`, `consolidate_by_subnet` |
| `mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `files.py` | I/O helpers | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc. |
| `reporting.py` | text rendering | `Column`, `render_table`, `render_kv`, `timestamp_name` |
Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
@@ -25,14 +33,22 @@ consolidated `.rsc`.
## Architecture
```
allucanget-ids/
```text
intrusion-detection/
├── scan.py consolidated scanner entry point
├── copyparty.py copyparty log adapter (~150 lines)
├── npm.py NPM proxy log adapter (~150 lines)
├── whitelists.py whitelist constants, download, CIDR extraction
├── utils.py thin re-export barrel (~90 lines)
├── models.py Request, IPStats, aggregate()
├── ip_utils.py IP parsing, whitelist lookups, sorting
├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
├── analyzer.py Analyzer pipeline base class
├── whitelists.py provider CIDR constants, local/remote load
├── bad_patterns.py malicious URL/file regex patterns
├── utils.py shared core: aggregation, reporting, MikroTik export
├── whois.py query_whois_subnet, consolidate_by_subnet
├── mikrotik.py RouterOS .rsc script generation
├── files.py read_lines, find_logs, write_csv/json/lines
├── reporting.py Column, render_table, render_kv, timestamp_name
├── data/ whitelist source files (cached locally)
│ ├── openfactory/
│ │ └── geofeed.csv
@@ -49,18 +65,24 @@ allucanget-ids/
```
`copyparty.py` and `npm.py` are thin adapters. Each supplies a log matcher,
a line parser, a flagging rule, and column definitions. Everything else lives in
`utils.py`:
a line parser, a flagging rule, and column definitions. `utils.py` is a barrel
module that re-exports everything from the sub-modules so callers only need
`import utils`.
- `Request` — normalised request record
- `IPStats` + `aggregate()` — per-IP counters, malicious-hit tracking
- `flag_scanners()` / `flag_denied_only()` / `flag_malicious_patterns()` — detection rules
- `whitelist_of()` / `is_whitelisted()` — whitelist lookups (backed by `whitelists.py`)
- `is_malicious()` / `classify_path()` — pattern classification (backed by `bad_patterns.py`)
- `query_whois_subnet()` / `consolidate_by_subnet()` — WHOIS subnet grouping
- IP version helpers, log discovery, transparent `.gz`, ANSI stripping
- CSV/JSON/table writers, `mikrotik_script()` — RouterOS export
- `Analyzer` — pipeline: discover → load → analyze → report → write
Core responsibilities by module:
| Module | Owns |
| ----------------- | -------------------------------------------------------------------------------- |
| `models.py` | `Request`, `IPStats`, `aggregate()` |
| `ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `bad_patterns.py` | malicious URL/file regex classification |
| `whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
Console output and saved reports are the **same string** (`Analyzer.text_report()`).
@@ -82,16 +104,16 @@ python scan.py
All scripts accept CLI overrides:
| Flag | Default | Meaning |
| --- | --- | --- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--no-rsc` | off | skip MikroTik export |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
| Flag | Default | Meaning |
| ----------------- | -------------------- | ----------------------------- |
| `--log-path` | see below | log directory |
| `--report-dir` | `reports/<name>` | output directory |
| `--top N` | `30` | rows in the top-IP table |
| `--quiet` | off | write files, print only paths |
| `--no-rsc` | off | skip MikroTik export |
| `--min-errors N` | `5` (`npm.py`) | error count threshold |
| `--error-ratio F` | `0.5` (`npm.py`) | error ratio threshold |
| `--min-denied N` | `1` (`copyparty.py`) | denied-hit threshold |
`scan.py` accepts all flags from both scanners plus `--copyparty-log-path`,
`--copyparty-report-dir`, `--npm-log-path`, `--npm-report-dir`, and
@@ -99,10 +121,10 @@ All scripts accept CLI overrides:
Default log paths (overridable by `.env`):
| Script | Env var | Default |
| --- | --- | --- |
| Script | Env var | Default |
| -------------- | -------------------- | -------------- |
| `copyparty.py` | `COPYPARTY_LOG_PATH` | `P:\copyparty` |
| `npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |
| `npm.py` | `NPM_LOG_PATH` | `Y:\npm\logs` |
Examples:
@@ -175,15 +197,15 @@ Blocklist candidates come from three independent sources, merged at scan time:
### Pattern categories (`bad_patterns.py`)
| Pattern | Severity | Matches |
| --- | --- | --- |
| `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` |
| `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` |
| `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` |
| `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
| `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` |
| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files |
| `well_known_probe` | 1 (moderate) | `.well-known/` paths |
| Pattern | Severity | Matches |
| ------------------- | ------------ | ------------------------------------------------------------------- |
| `env_file` | 3 (critical) | `.env`, `.env.production`, `.aws/credentials`, `config.json` |
| `vcs_probe` | 3 (critical) | `.git/config`, `.svn/` |
| `sensitive_file` | 3 (critical) | `id_rsa`, `id_dsa`, `.pem`, `.key`, `.ppk` |
| `webshell_probe` | 3 (critical) | Known webshell filenames (xstelth, adminfuns, wp_filemanager, etc.) |
| `wp_probe` | 2 (high) | `wp-admin`, `wp-login`, `wp-content`, `wp-includes` |
| `generic_php_probe` | 1 (moderate) | Single/double-char/numeric `.php` files |
| `well_known_probe` | 1 (moderate) | `.well-known/` paths |
Flagging metadata fields:
@@ -236,11 +258,14 @@ Fields: `[datetime]`, upstream status, proxy status, method, scheme, host,
## Extending
- **new whitelist provider** → add URLs, files, and fallback CIDRs to
`whitelists.py`
- **new detection rule** → add a `BadPattern` to `bad_patterns.py` or a
`flag_*()` function to `utils.py`, return it from the analyzer's `flag()`
- **new log source** → subclass `utils.Analyzer`, implement `parse_line()` +
- **new whitelist provider** → add URLs, local file paths, and fallback CIDRs
to `whitelists.py`
- **new detection pattern** → add a `BadPattern` to `bad_patterns.py`
- **new detection rule** → add a `flag_*()` function to `flags.py`, return it
from the analyzer's `flag()` method
- **new log source** → subclass `analyzer.Analyzer`, implement `parse_line()` +
`flag()`, declare `ip_columns` / `request_columns`
- **new output file** → override `write_extra(data_dir)` and return the paths
written
- **new I/O format** → add writer to `files.py`
- **new report section** → add rendering logic to `reporting.py`