zwitschi/intrusion-detection
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add WHOIS subnet querying and IP consolidation utility

Browse Source 
- Implemented a new utility for querying WHOIS information and consolidating IP addresses by subnet.
- Added caching mechanism for WHOIS responses to improve performance.
- Introduced functions for loading and saving the WHOIS cache, querying subnets, and consolidating IPs based on a threshold.

feat: create Flask web UI for intrusion detection system

- Developed a Flask web application to manage the intrusion detection system.
- Implemented routes for scanning, reports, blocklist management, and firewall configuration.
- Added helper functions for reading files and handling requests.

feat: define routes for web UI

- Registered routes for various web pages and API endpoints in the Flask application.
- Implemented functionality for scanning, reporting, blocklist management, and firewall state retrieval.

feat: implement background scan runner for web UI

- Created a background scan runner to handle scan jobs in a separate thread.
- Managed an in-memory job store for tracking scan jobs and their statuses.
- Added persistence for scan jobs to maintain state across application restarts.

chore: update start script for Flask application

- Enhanced the PowerShell start script to check for the existence of the virtual environment.
- Added comments and structured the script for better readability and usability.
This commit is contained in:
zwitschi committed 2026-08-08 14:33:51 +02:00
1 parent 41230ce04c
commit 9f8ab0f61c
102 files changed
+1102652 -268
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

README.md
+57 -42
View File
@@ -5,21 +5,16 @@ Local-development intrusion detection / log analysis for two log sources:
| Script | Source | Detection rule |
| --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------- |
| `app.py` | unified entry point | runs both analyzers + optional firewall sync |
| `app/scan.py` | consolidated scanner | runs both analyzers, merges blocklists, syncs firewall |
| `app/scan/` | consolidated scanner | runs both analyzers, merges blocklists, syncs firewall |
| `app/copyparty.py` | [copyparty](https://github.com/9001/copyparty) server logs | denied-only IPs + copyparty auto-bans |
| `app/npm.py` | [Nginx Proxy Manager](https://nginxproxymanager.com) proxy logs | high error-rate IPs (scanners) |
| `app/utils.py` | re-export layer | thin barrel — imports from sub-modules below |
| `app/models.py` | data model | `Request`, `IPStats`, `aggregate()` |
| `app/ip_utils.py` | IP helpers | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/utils/` | shared utilities | constants, I/O helpers, IP utils, models, reporting, whitelists, WHOIS |
| `app/web/` | Flask web UI | app factory, routes, background scan runner |
| `app/analyzer.py` | pipeline | `Analyzer` — discover → load → analyze → report → write |
| `app/whitelists.py` | whitelist module | provider CIDR constants, local/remote load, CIDR extraction |
| `app/flags.py` | detection rules | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/bad_patterns.py` | pattern detection | malicious URL/file regex classification |
| `app/whois.py` | WHOIS query | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | RouterOS export | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | RouterOS SSH client | `MikrotikFirewall` — connect, sync blocklist, audit state |
| `app/files.py` | I/O helpers | `read_lines`, `find_logs`, `write_csv`, `write_json`, etc. |
| `app/reporting.py` | text rendering | `Column`, `render_table`, `render_kv`, `timestamp_name` |
Both per-source scripts parse logs, aggregate per-IP behaviour, split
IPv4/IPv6, skip whitelisted provider ranges, detect malicious path patterns,
@@ -40,26 +35,33 @@ intrusion-detection/
├── app.py unified entry point
├── app/ application package
│ ├── __init__.py
│ ├── constants.py all config constants (paths, env vars, thresholds)
│ ├── scan.py consolidated scanner
│ ├── copyparty.py copyparty log adapter (~150 lines)
│ ├── npm.py NPM proxy log adapter (~150 lines)
│ ├── firewall.py MikroTik RouterOS SSH client
│ ├── utils.py thin re-export barrel (~90 lines)
│ ├── models.py Request, IPStats, aggregate()
│ ├── ip_utils.py IP parsing, whitelist lookups, sorting
│ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
│ ├── analyzer.py Analyzer pipeline base class
│ ├── whitelists.py provider CIDR constants, local/remote load
│ ├── bad_patterns.py malicious URL/file regex patterns
│ ├── whois.py query_whois_subnet, consolidate_by_subnet
│ ├── copyparty.py copyparty log adapter (~150 lines)
│ ├── firewall.py MikroTik RouterOS SSH client
│ ├── flags.py flag_scanners, flag_denied_only, flag_malicious_patterns
│ ├── mikrotik.py RouterOS .rsc script generation
│ ├── files.py read_lines, find_logs, write_csv/json/lines
│ ├── reporting.py Column, render_table, render_kv, timestamp_name
│ ├── web.py Flask web UI (app factory, JSON API, pages)
│ ├── web_scan.py background scan runner + in-memory job store
│ ├── templates/ Jinja2 page templates (dashboard, scan, reports, …)
│ └── static/ style.css + app.js (dark-theme UI)
│ ├── npm.py NPM proxy log adapter (~150 lines)
│ ├── web/ Flask web UI
│ │ ├── __init__.py app factory + helpers + serve()
│ │ ├── routes.py all HTTP route definitions
│ │ └── web_scan.py background scan runner + in-memory job store
│ ├── scan/ scan pipeline
│ │ ├── __init__.py consolidated scanner (run_scan, ScanParams, ScanResult)
│ │ └── scan_state.py incremental log scan state (load/save)
│ ├── utils/ shared utilities
│ │ ├── constants.py all config constants (paths, env vars, thresholds)
│ │ ├── files.py read_lines, find_logs, write_csv/json/lines
│ │ ├── ip_utils.py IP parsing, whitelist lookups, sorting
│ │ ├── models.py Request, IPStats, aggregate()
│ │ ├── reporting.py Column, render_table, render_kv, timestamp_name
│ │ ├── utils.py shared CLI arg parser
│ │ ├── whitelists.py provider CIDR constants, local/remote load
│ │ ├── whois.py query_whois_subnet, consolidate_by_subnet
│ │ ├── blocklist_history.py git-backed cumulative blocklist
│ │ └── retention.py report file retention & pruning
│ ├── static/ style.css + app.js (dark-theme UI)
│ └── templates/ Jinja2 page templates (dashboard, scan, reports, …)
├── data/ whitelist source files (cached locally)
│ ├── openfactory/
│ │ └── geofeed.csv
@@ -80,22 +82,24 @@ matcher, a line parser, a flagging rule, and column definitions.
Core responsibilities by module:
| Module | Owns |
| --------------------- | -------------------------------------------------------------------------------- |
| `app/constants.py` | all config constants — paths, env vars, network/scan thresholds |
| `app/models.py` | `Request`, `IPStats`, `aggregate()` |
| `app/ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `app/whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `app/bad_patterns.py` | malicious URL/file regex classification |
| `app/whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | `MikrotikFirewall` — SSH connect, sync blocklist, audit state |
| `app/web.py` | Flask app factory, JSON API routes, page rendering |
| `app/web_scan.py` | background scan runner, in-memory job store, single-scan busy lock |
| `app/files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `app/reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
| Module | Owns |
| -------------------------------- | -------------------------------------------------------------------------------- |
| `app/utils/constants.py` | all config constants — paths, env vars, network/scan thresholds |
| `app/utils/models.py` | `Request`, `IPStats`, `aggregate()` |
| `app/utils/ip_utils.py` | `parse_ip`, `whitelist_of`, `is_whitelisted`, `sort_ips`, `split_versions` |
| `app/flags.py` | `flag_scanners`, `flag_denied_only`, `flag_malicious_patterns`, `by_total` |
| `app/analyzer.py` | `Analyzer` — discover → load → analyze → report → write |
| `app/utils/whitelists.py` | provider CIDR constants, local-file / remote / built-in loading |
| `app/bad_patterns.py` | malicious URL/file regex classification |
| `app/utils/whois.py` | `query_whois_subnet`, `consolidate_by_subnet` |
| `app/mikrotik.py` | `mikrotik_script`, `write_mikrotik`, `split_cidr_versions` |
| `app/firewall.py` | `MikrotikFirewall` — SSH connect, sync blocklist, audit state |
| `app/web/` | Flask app factory, JSON API routes, page rendering, background scan runner |
| `app/scan/` | consolidated scan pipeline, incremental scan state |
| `app/utils/files.py` | `read_lines`, `find_logs`, `validate_dir`, `ensure_dirs`, `write_csv/json/lines` |
| `app/utils/reporting.py` | `Column`, `render_table`, `render_kv`, `timestamp_name` |
| `app/utils/blocklist_history.py` | git-backed cumulative blocklist reconstruction |
| `app/utils/retention.py` | report file retention & pruning |
Console output and saved reports are the **same string** (`Analyzer.text_report()`).
@@ -209,6 +213,17 @@ change report to `reports/app/YYYY-MM-DD-hh-mm-ss.json`: the number and list
of entries added / removed (complete-set delta), the full cumulative blocked
list, and the router sync changes.
### Report retention
Generated files under `reports/` are pruned automatically after each
consolidated scan: every subdirectory keeps at most
`REPORT_RETENTION_MAX_FILES` files (default 10), all newer than
`REPORT_RETENTION_DAYS` days (default 30). The blocklist source of truth
(`reports/app/data/blocklist_ipv4.txt` / `blocklist_ipv6.txt`) is always
protected and never pruned. Tune via `REPORT_RETENTION_DAYS` /
`REPORT_RETENTION_MAX_FILES`; `--dry-run` previews the cleanup without
deleting.
## Web UI
`python app.py` starts a **localhost-only** Flask web UI (default