zwitschi
5ecd2b8d19
feat: Refactor Dashboard template to extend base layout and improve structure feat: Enhance Parameter Input template with improved layout and feedback mechanisms feat: Update Scenario Form template to utilize base layout and improve user experience feat: Create base layout template for consistent styling across pages feat: Add Consumption, Costs, Equipment, Maintenance, Production, Reporting, and Simulations templates with placeholders for future functionality feat: Implement base header and footer partials for consistent navigation and footer across the application
140 lines
11 KiB
Markdown
140 lines
11 KiB
Markdown
# Architecture Documentation
|
||
|
||
## Overview
|
||
|
||
CalMiner is a FastAPI application that collects mining project inputs, persists scenario-specific records, and surfaces aggregated insights. The platform targets Monte Carlo driven planning, with deterministic CRUD features in place and simulation logic staged for future work.
|
||
|
||
Frontend components are server-rendered Jinja2 templates, with Chart.js powering the dashboard visualization.
|
||
|
||
The backend leverages SQLAlchemy for ORM mapping to a PostgreSQL database.
|
||
|
||
## System Components
|
||
|
||
- **FastAPI backend** (`main.py`, `routes/`): hosts REST endpoints for scenarios, parameters, costs, consumption, production, equipment, maintenance, simulations, and reporting. Each router encapsulates request/response schemas and DB access patterns, leveraging a shared dependency module (`routes/dependencies.get_db`) for SQLAlchemy session management.
|
||
- **Service layer** (`services/`): houses business logic. `services/reporting.py` produces statistical summaries, while `services/simulation.py` provides the Monte Carlo integration point.
|
||
- **Persistence** (`models/`, `config/database.py`): SQLAlchemy models map to PostgreSQL tables in schema `bricsium_platform`. Relationships connect scenarios to derived domain entities.
|
||
- **Presentation** (`templates/`, `components/`): server-rendered views extend a shared `base.html` layout, pull global styles from `static/css/main.css`, and surface data entry (scenario and parameter forms) alongside the Chart.js-powered dashboard.
|
||
- **Middleware** (`middleware/validation.py`): applies JSON validation before requests reach routers.
|
||
- **Testing** (`tests/unit/`): pytest suite covering route and service behavior.
|
||
|
||
## Runtime Flow
|
||
|
||
1. Users navigate to form templates or API clients to manage scenarios, parameters, and operational data.
|
||
2. FastAPI routers validate payloads with Pydantic models, then delegate to SQLAlchemy sessions for persistence.
|
||
3. Simulation runs (placeholder `services/simulation.py`) will consume stored parameters to emit iteration results via `/api/simulations/run`.
|
||
4. Reporting requests POST simulation outputs to `/api/reporting/summary`; the reporting service calculates aggregates (count, min/max, mean, median, percentiles, standard deviation, variance, and tail-risk metrics at the 95% confidence level).
|
||
5. `templates/Dashboard.html` fetches summaries, renders metric cards, and plots distribution charts with Chart.js for stakeholder review.
|
||
|
||
### Dashboard Flow Review — 2025-10-20
|
||
|
||
- The dashboard template depends on a future-facing HTML endpoint (e.g., `/dashboard`) that the current `routes/ui.py` router does not expose; wiring an explicit route is required before the page is reachable from the FastAPI app.
|
||
- Client-side logic calls `/api/reporting/summary` with raw simulation outputs and expects `result` fields, so any upstream changes to the reporting contract must maintain this schema.
|
||
- Initialization always loads the bundled sample data first, which is useful for demos but masks server errors—consider adding a failure banner when `/api/reporting/summary` is unavailable.
|
||
- No persistent storage backs the dashboard yet; users must paste or load JSON manually, aligning with the current MVP scope but highlighting an integration gap with the simulation results table.
|
||
|
||
### Reporting Pipeline and UI Integration
|
||
|
||
1. **Data Sources**
|
||
|
||
- Scenario-linked calculations (costs, consumption, production) produce raw figures stored in dedicated tables (`capex`, `opex`, `consumption`, `production_output`).
|
||
- Monte Carlo simulations (currently transient) generate arrays of `{ "result": float }` tuples that the dashboard or downstream tooling passes directly to reporting endpoints.
|
||
|
||
2. **API Contract**
|
||
|
||
- `POST /api/reporting/summary` accepts a JSON array of result objects and validates shape through `_validate_payload` in `routes/reporting.py`.
|
||
- On success it returns a structured payload (`ReportSummary`) containing count, mean, median, min/max, standard deviation, and percentile values, all as floats.
|
||
|
||
3. **Service Layer**
|
||
|
||
- `services/reporting.generate_report` converts the sanitized payload into descriptive statistics using Python’s standard library (`statistics` module) to avoid external dependencies.
|
||
- The service remains stateless; no database read/write occurs, which keeps summary calculations deterministic and idempotent.
|
||
- Extended KPIs (surfaced in the API and dashboard):
|
||
- `variance`: population variance computed as the square of the population standard deviation.
|
||
- `percentile_5` and `percentile_95`: lower and upper tail interpolated percentiles for sensitivity bounds.
|
||
- `value_at_risk_95`: 5th percentile threshold representing the minimum outcome within a 95% confidence band.
|
||
- `expected_shortfall_95`: mean of all outcomes at or below the `value_at_risk_95`, highlighting tail exposure.
|
||
|
||
4. **UI Consumption**
|
||
|
||
- `templates/Dashboard.html` posts the user-provided dataset to the summary endpoint, renders metric cards for each field, and charts the distribution using Chart.js.
|
||
- `SUMMARY_FIELDS` now includes variance, 5th/10th/90th/95th percentiles, and tail-risk metrics (VaR/Expected Shortfall at 95%); tooltip annotations surface the tail metrics alongside the percentile line chart.
|
||
- Error handling surfaces HTTP failures inline so users can address malformed JSON or backend availability issues without leaving the page.
|
||
|
||
5. **Future Integration Points**
|
||
- Once `/api/simulations/run` persists to `simulation_result`, the dashboard can fetch precalculated runs per scenario, removing the manual JSON step.
|
||
- Additional reporting endpoints (e.g., scenario comparisons) can reuse the same service layer, ensuring consistency across UI and API consumers.
|
||
|
||
## Data Model Highlights
|
||
|
||
- `scenario`: central entity describing a mining scenario; owns relationships to cost, consumption, production, equipment, and maintenance tables.
|
||
- `capex`, `opex`: monetary tracking linked to scenarios.
|
||
- `consumption`: resource usage entries parameterized by scenario and description.
|
||
- `parameter`: scenario inputs with base `value` and optional distribution linkage via `distribution_id`, `distribution_type`, and JSON `distribution_parameters` to support simulation sampling.
|
||
- `production_output`: production metrics per scenario.
|
||
- `equipment` and `maintenance`: equipment inventory and maintenance events with dates/costs.
|
||
- `simulation_result`: staging table for future Monte Carlo outputs (not yet populated by `run_simulation`).
|
||
|
||
Foreign keys secure referential integrity between domain tables and their scenarios, enabling per-scenario analytics.
|
||
|
||
## Integrations and Future Work
|
||
|
||
- **Monte Carlo engine**: `services/simulation.py` will incorporate stochastic sampling (e.g., NumPy, SciPy) to populate `simulation_result` and feed reporting.
|
||
- **Persistence of results**: `/api/simulations/run` currently returns in-memory results; next iteration should persist to `simulation_result` and reference scenarios.
|
||
- **Authentication**: not yet implemented; all endpoints are open.
|
||
- **Deployment**: documentation focuses on local development; containerization and CI/CD pipelines remain to be defined.
|
||
|
||
For extended diagrams and setup instructions reference:
|
||
|
||
- [docs/development_setup.md](development_setup.md) — environment provisioning and tooling.
|
||
- [docs/testing.md](testing.md) — pytest workflow and coverage expectations.
|
||
- [docs/mvp.md](mvp.md) — roadmap and milestone scope.
|
||
- [docs/implementation_plan.md](implementation_plan.md) — feature breakdown aligned with the TODO tracker.
|
||
- [docs/architecture_overview.md](architecture_overview.md) — supplementary module map and request flow diagram.
|
||
|
||
### UI Frontend-Backend Integration Requirements — 2025-10-20
|
||
|
||
#### Scenarios (`templates/ScenarioForm.html`)
|
||
|
||
- **Data**: `GET /api/scenarios/` to list existing scenarios for navigation and to hydrate dropdowns in downstream forms; optional aggregation of scenario counts for dashboard badges.
|
||
- **Actions**: `POST /api/scenarios/` to create new scenarios; future delete/update flows would reuse the same router once endpoints exist.
|
||
|
||
#### Parameters (`templates/ParameterInput.html`)
|
||
|
||
- **Data**: Scenario catalogue from `GET /api/scenarios/`; parameter inventory via `GET /api/parameters/` with client-side filtering by `scenario_id`; optional distribution catalogue from `models/distribution` when exposed.
|
||
- **Actions**: `POST /api/parameters/` to add parameters; extend UI to support editing or deleting parameters as routes arrive.
|
||
|
||
#### Costs (`templates/costs.html`)
|
||
|
||
- **Data**: CAPEX list `GET /api/costs/capex`; OPEX list `GET /api/costs/opex`; computed totals grouped by scenario for summary panels.
|
||
- **Actions**: `POST /api/costs/capex` and `POST /api/costs/opex` for new entries; planned future edits/deletes once routers expand.
|
||
|
||
#### Consumption (`templates/consumption.html`)
|
||
|
||
- **Data**: Consumption entries by scenario via `GET /api/consumption/`; scenario metadata for filtering and empty-state messaging.
|
||
- **Actions**: `POST /api/consumption/` to log consumption items; include optimistic UI refresh after persistence.
|
||
|
||
#### Production (`templates/production.html`)
|
||
|
||
- **Data**: Production records from `GET /api/production/`; scenario list for filter chips; optional aggregates (totals, averages) derived client-side.
|
||
- **Actions**: `POST /api/production/` to capture production outputs; notify users when data drives downstream reporting.
|
||
|
||
#### Equipment (`templates/equipment.html`)
|
||
|
||
- **Data**: Equipment roster from `GET /api/equipment/`; scenario context to scope inventory; maintenance counts per asset once joins are introduced.
|
||
- **Actions**: `POST /api/equipment/` to add equipment; integrate delete/edit when endpoints arrive.
|
||
|
||
#### Maintenance (`templates/maintenance.html`)
|
||
|
||
- **Data**: Maintenance schedule `GET /api/maintenance/` with pagination support (`skip`, `limit`); equipment list (`GET /api/equipment/`) to map IDs to names; scenario catalogue for filtering.
|
||
- **Actions**: CRUD operations through `POST /api/maintenance/`, `PUT /api/maintenance/{id}`, `DELETE /api/maintenance/{id}`; view detail via `GET /api/maintenance/{id}` for modal display.
|
||
|
||
#### Simulations (`templates/simulations.html`)
|
||
|
||
- **Data**: Scenario list `GET /api/scenarios/`; parameter sets `GET /api/parameters/` filtered client-side; persisted simulation results (future) via dedicated endpoint or by caching `/api/simulations/run` responses.
|
||
- **Actions**: `POST /api/simulations/run` to execute simulations; surface run configuration (iterations, seed) and feed response summary into reporting page.
|
||
|
||
#### Reporting (`templates/reporting.html` and `templates/Dashboard.html`)
|
||
|
||
- **Data**: Simulation outputs either from recent `/api/simulations/run` calls or a planned `GET` endpoint against `simulation_result`; summary metrics via `POST /api/reporting/summary`; scenario metadata for header context.
|
||
- **Actions**: Trigger summary refreshes by posting batched results; allow export/download actions once implemented; integrate future comparison requests across scenarios.
|