calminer/changelog.md

Changelog

2025-11-12

• Fixed critical 500 error in reporting dashboard by correcting route reference in reporting.html template - changed 'reports.project_list_page' to 'projects.project_list_page' to resolve NoMatchFound error when accessing /ui/reporting.

• Completed navigation validation by inventorying all sidebar navigation links, identifying missing routes for simulations, reporting, settings, themes, and currencies, created new UI routes in routes/ui.py with proper authentication guards, built corresponding templates (simulations.html, reporting.html, settings.html, theme_settings.html, currencies.html), registered the UI router in main.py, updated sidebar navigation to use route names instead of hardcoded URLs, and enhanced navigation.js to use dynamic URL resolution for proper route handling.

• Fixed critical template rendering error in sidebar_nav.html where URL objects from request.url_for() were being used with string methods, causing TypeError. Added |string filters to convert URL objects to strings for proper template rendering.

• Integrated Plotly charting for interactive visualizations in reporting templates, added chart generation methods to ReportingService (_generate_npv_comparison_chart, _generate_distribution_histogram), updated project summary and scenario distribution contexts to include chart JSON data, enhanced templates with chart containers and JavaScript rendering, added chart-container CSS styling, and validated all reporting tests pass.

• Completed local run verification: started application with uvicorn main:app --reload without errors, verified authenticated routes (/login, /, /projects/ui, /projects) load correctly with seeded data, and summarized findings for deployment pipeline readiness.

• Fixed docker-compose.override.yml command array to remove duplicate "uvicorn" entry, enabling successful container startup with uvicorn reload in development mode.

• Completed deployment pipeline verification: built Docker image without errors, validated docker-compose configuration, deployed locally with docker-compose (app and postgres containers started successfully), and confirmed application startup logs showing database bootstrap and seeded data initialization.

• Completed documentation of current data models: updated calminer-docs/architecture/08_concepts/02_data_model.md with comprehensive SQLAlchemy model schemas, enumerations, Pydantic API schemas, and analysis of discrepancies between models and schemas.

• Switched models/performance_metric.py to reuse the shared declarative base from config.database, clearing the SQLAlchemy 2.0 declarative_base deprecation warning and verifying repository tests still pass.

• Replaced the Alembic migration workflow with the idempotent Pydantic-backed initializer (scripts/init_db.py), added a guarded reset utility (scripts/reset_db.py), removed migration artifacts/tooling (Alembic directory, config, Docker entrypoint), refreshed the container entrypoint to invoke uvicorn directly, and updated installation/architecture docs plus the README to direct developers to the new seeding/reset flow.

• Eliminated Bandit hardcoded-secret findings by replacing literal JWT tokens and passwords across auth/security tests with randomized helpers drawn from tests/utils/security.py, ensuring fixtures still assert expected behaviours.

• Centralized Bandit configuration in pyproject.toml, reran bandit -c pyproject.toml -r calminer tests, and verified the scan now reports zero issues.

• Updated .github/instructions/TODO.md and .github/instructions/DONE.md to reflect the completed security scan remediation workflow.

• Diagnosed admin bootstrap failure caused by legacy roles schema, added Alembic migration 20251112_00_add_roles_metadata_columns.py to backfill display_name, description, created_at, and updated_at, and verified the migration via full pytest run in the activated .venv.

• Resolved Ruff E402 warnings by moving module docstrings ahead of from __future__ import annotations across currency and pricing service modules, dropped the unused HTTPException import in monitoring/__init__.py, and confirmed a clean ruff check . run.

• Enhanced the deploy job in .gitea/workflows/cicache.yml to capture Kubernetes pod, deployment, and container logs into /logs/deployment/ for staging/production rollouts and publish them via a deployment-logs artifact, updating CI/CD documentation with retrieval instructions.

• Fixed CI dashboard template lookup failures by renaming templates/Dashboard.html to templates/dashboard.html and verifying tests/test_dashboard_route.py locally to ensure TemplateNotFound no longer occurs on case-sensitive filesystems.

• Implemented SQLite support as primary local database with environment-driven backend switching (CALMINER_USE_SQLITE=true), updated scripts/init_db.py for database-agnostic DDL generation (PostgreSQL enums vs SQLite CHECK constraints), tested compatibility with both backends, and verified application startup and seeded data initialization work seamlessly across SQLite and PostgreSQL.

2025-11-11

• Collapsed legacy Alembic revisions into alembic/versions/00_initial.py, removed superseded migration files, and verified the consolidated schema via SQLite upgrade and Postgres version stamping.
• Implemented base URL routing to redirect unauthenticated users to login and authenticated users to dashboard.
• Added comprehensive end-to-end tests for login flow, including redirects, session handling, and error messaging for invalid/inactive accounts.
• Updated header and footer templates to consistently use logo_big.png image instead of text logo, with appropriate CSS styling for sizing.
• Centralised ISO-4217 currency validation across scenarios, imports, and export filters (models/scenario.py, routes/scenarios.py, schemas/scenario.py, schemas/imports.py, services/export_query.py) so malformed codes are rejected consistently at every entry point.
• Updated scenario services and UI flows to surface friendly validation errors and added regression coverage for imports, exports, API creation, and lifecycle flows ensuring currencies are normalised end-to-end.
• Recorded the completed “Ensure currency is used consistently” work in .github/instructions/DONE.md and ran the full pytest suite (150 tests) to verify the refactor.
• Linked projects to their pricing settings by updating SQLAlchemy models, repositories, seeding utilities, and migrations, and added regression tests to cover the new association and default backfill.
• Bootstrapped database-stored pricing settings at application startup, aligned initial data seeding with the database-first metadata flow, and added tests covering pricing bootstrap creation, project assignment, and idempotency.
• Extended pricing configuration support to prefer persisted metadata via dependencies.get_pricing_metadata, added retrieval tests for project/default fallbacks, and refreshed docs (calminer-docs/specifications/price_calculation.md, pricing_settings_data_model.md) to describe the database-backed workflow and bootstrap behaviour.
• Added services/financial.py NPV, IRR, and payback helpers with robust cash-flow normalisation, convergence safeguards, and fractional period support, plus comprehensive pytest coverage exercising representative project scenarios and failure modes.
• Authored calminer-docs/specifications/financial_metrics.md capturing DCF assumptions, solver behaviours, and worked examples, and cross-linked the architecture concepts to the new reference for consistent navigation.
• Implemented services/simulation.py Monte Carlo engine with configurable distributions, summary aggregation, and reproducible RNG seeding, introduced regression tests in tests/test_simulation.py, and documented configuration/usage in calminer-docs/specifications/monte_carlo_simulation.md with architecture cross-links.
• Polished reporting HTML contexts by cleaning stray fragments in routes/reports.py, adding download action metadata for project and scenario pages, and generating scenario comparison download URLs with correctly serialised repeated scenario_ids parameters.
• Consolidated Alembic history into a single initial migration (20251111_00_initial_schema.py), removed superseded revision files, and ensured Alembic metadata still references the project metadata for clean bootstrap.
• Added scripts/run_migrations.py and a Docker entrypoint wrapper to run Alembic migrations before uvicorn starts, removed the fallback Base.metadata.create_all call, and updated calminer-docs/admin/installation.md so developers know how to apply migrations locally or via Docker.
• Configured pytest defaults to collect coverage (--cov) with an 80% fail-under gate, excluded entrypoint/reporting scaffolds from the calculation, updated contributor docs with the standard pytest command, and verified the suite now reports 83% coverage.
• Standardized color scheme and typography by moving alert styles to main.css, adding typography rules with CSS variables, updating auth templates for consistent button classes, and ensuring all templates use centralized color and spacing variables.
• Improved navigation flow by adding two big chevron buttons on top of the navigation sidebar to allow users to navigate to the previous and next page in the page navigation list, including JavaScript logic for determining current page and handling navigation.
• Established pytest-based unit and integration test suites with coverage thresholds, achieving 83% coverage across 181 tests, with configuration in pyproject.toml and documentation in CONTRIBUTING.md.
• Configured CI pipelines to run tests, linting, and security checks on each change, adding Bandit security scanning to the workflow and verifying execution on pushes and PRs to main/develop branches.
• Added deployment automation with Docker Compose for local development and Kubernetes manifests for production, ensuring environment parity and documenting processes in calminer-docs/admin/installation.md.
• Completed monitoring instrumentation by adding business metrics observation to project and scenario repository operations, and simulation performance tracking to Monte Carlo service with success/error status and duration metrics.
• Updated TODO list to reflect completed monitoring implementation tasks and validated changes with passing simulation tests.
• Implemented comprehensive performance monitoring for scalability (FR-006) with Prometheus metrics collection for HTTP requests, import/export operations, and general application metrics.
• Added database model for persistent metric storage with aggregation endpoints for KPIs like request latency, error rates, and throughput.
• Created FastAPI middleware for automatic request metric collection and background persistence to database.
• Extended monitoring router with performance metrics API endpoints and detailed health checks.
• Added Alembic migration for performance_metrics table and updated model imports.
• Completed concurrent interaction testing implementation, validating database transaction isolation under threading and establishing async testing framework for future concurrency enhancements.
• Implemented comprehensive deployment automation with Docker Compose configurations for development, staging, and production environments ensuring environment parity.
• Set up Kubernetes manifests with resource limits, health checks, and secrets management for production deployment.
• Configured CI/CD workflows for automated Docker image building, registry pushing, and Kubernetes deployment to staging/production environments.
• Documented deployment processes, environment configurations, and CI/CD workflows in project documentation.
• Validated deployment automation through Docker Compose configuration testing and CI/CD pipeline structure.

2025-11-10

• Added dedicated pytest coverage for guard dependencies, exercising success plus failure paths (missing session, inactive user, missing roles, project/scenario access errors) via tests/test_dependencies_guards.py.
• Added integration tests in tests/test_authorization_integration.py verifying anonymous 401 responses, role-based 403s, and authorized project manager flows across API and UI endpoints.
• Implemented environment-driven admin bootstrap settings, wired the bootstrap_admin helper into FastAPI startup, added pytest coverage for creation/idempotency/reset logic, and documented operational guidance in the RBAC plan and security concept.
• Retired the legacy authentication RBAC implementation plan document after migrating its guidance into live documentation and synchronized the contributor instructions to reflect the removal.
• Completed the Authentication & RBAC checklist by shipping the new models, migrations, repositories, guard dependencies, and integration tests.
• Documented the project/scenario import/export field mapping and file format guidelines in calminer-docs/requirements/FR-008.md, and introduced schemas/imports.py with Pydantic models that normalise incoming CSV/Excel rows for projects and scenarios.
• Added services/importers.py to load CSV/XLSX files into the new import schemas, pulled in openpyxl for Excel support, and covered the parsing behaviour with tests/test_import_parsing.py.
• Expanded the import ingestion workflow with staging previews, transactional persistence commits, FastAPI preview/commit endpoints under /imports, and new API tests (tests/test_import_ingestion.py, tests/test_import_api.py) ensuring end-to-end coverage.
• Added persistent audit logging via ImportExportLog, structured log emission, Prometheus metrics instrumentation, /metrics endpoint exposure, and updated operator/deployment documentation to guide monitoring setup.

2025-11-09

• Captured current implementation status, requirements coverage, missing features, and prioritized roadmap in calminer-docs/implementation_status.md to guide future development.
• Added core SQLAlchemy domain models, shared metadata descriptors, and Alembic migration setup (with initial schema snapshot) to establish the persistence layer foundation.
• Introduced repository and unit-of-work helpers for projects, scenarios, financial inputs, and simulation parameters to support service-layer operations.
• Added SQLite-backed pytest coverage for repository and unit-of-work behaviours to validate persistence interactions.
• Exposed project and scenario CRUD APIs with validated schemas and integrated them into the FastAPI application.
• Connected project and scenario routers to new Jinja2 list/detail/edit views with HTML forms and redirects.
• Implemented FR-009 client-side enhancements with responsive navigation toggle, mobile-first scenario tables, and shared asset loading across templates.
• Added scenario comparison validator, FastAPI comparison endpoint, and comprehensive unit tests to enforce FR-009 validation rules through API errors.
• Delivered a new dashboard experience with templates/dashboard.html, dedicated styling, and a FastAPI route supplying real project/scenario metrics via repository helpers.
• Extended repositories with count/recency utilities and added pytest coverage, including a dashboard rendering smoke test validating empty-state messaging.
• Brought project and scenario detail pages plus their forms in line with the dashboard visuals, adding metric cards, layout grids, and refreshed CTA styles.
• Reordered project route registration to prioritize static UI paths, eliminating 422 errors on /projects/ui and /projects/create, and added pytest smoke coverage for the navigation endpoints.
• Added end-to-end integration tests for project and scenario lifecycles, validating HTML redirects, template rendering, and API interactions, and updated ProjectRepository.get to deduplicate joined loads for detail views.
• Updated all Jinja2 template responses to the new Starlette signature to eliminate deprecation warnings while keeping request-aware context available to the templates.
• Introduced services/security.py to centralize Argon2 password hashing utilities and JWT creation/verification with typed payloads, and added pytest coverage for hashing, expiry, tampering, and token type mismatch scenarios.
• Added routes/auth.py with registration, login, and password reset flows, refreshed auth templates with error messaging, wired navigation links, and introduced end-to-end pytest coverage for the new forms and token flows.
• Implemented cookie-based authentication session middleware with automatic access token refresh, logout handling, navigation adjustments, and documentation/test updates capturing the new behaviour.
• Delivered idempotent seeding utilities with scripts/initial_data.py, entry-point runner scripts/00_initial_data.py, documentation updates, and pytest coverage to verify role/admin provisioning.
• Secured project and scenario routers with RBAC guard dependencies, enforced repository access checks via helper utilities, and aligned template routes with FastAPI dependency injection patterns.