- Add repo.record_blocklist(): grow-only, never auto-removes (mirrors router
cumulative model; preserves entries that leave the NPM log window).
- scan.py uses it on the scan path and derives the report's complete set from
the ledger (_db_complete_blocklist), falling back to git history only if the
DB is unavailable.
Reads the persisted SQLite store to print a detailed analysis: latest scans,
top IPs, cross-source IPs, active blocklist ledger, and a per-IP request
history drill-down (--ip). Supports --since, --top, --json. No scan is run.
save_state_report now also writes the address-list + firewall-rule snapshot into
the app.db firewall_snapshots tables, making the audit trail independent of git.
Removes the 'git-visible data/mikrotik' framing.
D006: idempotent --migrate, --info (schema version + row counts), and --path.
Used by deploy.sh to bring the DB up to date after a git pull without the web
server running.
The Cloudflare ips-v4/v6.txt and paloalto/.gitkeep were git-tracked. The
Cloudflare files are overwritten at runtime by every whitelist refresh, so on
a deployed server they become locally-modified tracked files that make 'git pull'
conflict. Keep the read-only paloalto/scanner-ips.txt seed tracked; ignore the
rest (they are re-downloaded if absent; built-in ranges are the fallback).
- Added schema definition and migration logic in app/db/schema.py.
- Created tables for log sources, requests, IP stats, scans, blocklist, whitelist, and firewall rules.
- Introduced functions to apply migrations and ensure database integrity.
feat: Enhance scan functionality with database integration
- Added _sync_blocklist_to_db function to synchronize blocklist with SQLite.
- Updated run_scan function to handle database connections and scan records.
- Implemented error handling for database operations during scans.
feat: Introduce log analysis page and API endpoints
- Created analysis.html template for displaying log analysis results.
- Added routes for fetching top IPs, IP request timelines, and cross-source IPs.
- Integrated analysis data into the dashboard and status endpoints.
fix: Ensure whitelists are synchronized with the database
- Implemented sync_whitelists_to_db function to persist in-memory whitelists.
- Updated constants.py to define the path for the SQLite database.
refactor: Modularize analysis helpers for better maintainability
- Created _analysis_helpers function to encapsulate SQLite query logic.
- Updated routes.py to utilize new analysis helper functions for API responses.
Delete historical Nginx Proxy Manager log analysis outputs from
app/reports/npm, including generated report snapshots and the
associated data artifacts (ips.json, requests.csv, error_paths.csv,
host_summary.csv, ua_summary.csv, scanner/blocked IP lists, and the
MikroTik block script). These files are regenerated on demand and no
longer need to be tracked.
- Implemented a new utility for querying WHOIS information and consolidating IP addresses by subnet.
- Added caching mechanism for WHOIS responses to improve performance.
- Introduced functions for loading and saving the WHOIS cache, querying subnets, and consolidating IPs based on a threshold.
feat: create Flask web UI for intrusion detection system
- Developed a Flask web application to manage the intrusion detection system.
- Implemented routes for scanning, reports, blocklist management, and firewall configuration.
- Added helper functions for reading files and handling requests.
feat: define routes for web UI
- Registered routes for various web pages and API endpoints in the Flask application.
- Implemented functionality for scanning, reporting, blocklist management, and firewall state retrieval.
feat: implement background scan runner for web UI
- Created a background scan runner to handle scan jobs in a separate thread.
- Managed an in-memory job store for tracking scan jobs and their statuses.
- Added persistence for scan jobs to maintain state across application restarts.
chore: update start script for Flask application
- Enhanced the PowerShell start script to check for the existence of the virtual environment.
- Added comments and structured the script for better readability and usability.
- Add `write_blocklist_files()` to emit final IPv4/IPv6 blocklists to `reports/app/data/`
- Add `APP_REPORT_DIR` / `APP_REPORT_DATA_DIR` constants for consolidated outputs
- Write firewall state snapshots to `data/mikrotik/` for git diff auditing
- Capture router state before scanning; fall back to scan-only if router is unreachable
- Use `split_cidr_versions` for CIDR-aware subnet splitting
- Update README with output layout and git-diffable state notes
zwitschi