calminer/README.md

# CalMiner

A web application to plan mining projects and estimate costs, returns and profitability.

Focuses on ore mining operations and covering parameters such as capital and operational expenditures, resource consumption, production output, and Monte Carlo simulations for risk analysis.

The system is designed to help mining companies make informed decisions by simulating various scenarios and analyzing potential outcomes based on stochastic variables.

A range of features are implemented to support these functionalities.

## Features

- **Scenario Management**: Manage multiple mining scenarios with independent parameter sets and outputs.
- **Process Parameters**: Define and persist process inputs via FastAPI endpoints and template-driven forms.
- **Cost Tracking**: Capture capital (`capex`) and operational (`opex`) expenditures per scenario.
- **Consumption Tracking**: Record resource consumption (chemicals, fuel, water, scrap) tied to scenarios.
- **Production Output**: Store production metrics such as tonnage, recovery, and revenue drivers.
- **Equipment Management**: Register scenario-specific equipment inventories.
- **Maintenance Logging**: Log maintenance events against equipment with dates and costs.
- **Reporting Dashboard**: Surface aggregated statistics for simulation outputs with an interactive Chart.js dashboard.
- **Unified UI Shell**: Server-rendered templates extend a shared base layout with a persistent left sidebar linking scenarios, parameters, costs, consumption, production, equipment, maintenance, simulations, and reporting views.
- **Operations Overview Dashboard**: The root route (`/`) surfaces cross-scenario KPIs, charts, and maintenance reminders with a one-click refresh backed by aggregated loaders.
- **Theming Tokens**: Shared CSS variables in `static/css/main.css` centralize the UI color palette for consistent styling and rapid theme tweaks.
- **Modular Frontend Scripts**: Page-specific interactions now live in `static/js/` modules, keeping templates lean while enabling browser caching and reuse.
- **Monte Carlo Simulation (in progress)**: Services and routes are scaffolded for future stochastic analysis.

## Architecture

The architecture is documented in [docs/architecture.md](docs/architecture.md).

## Project Structure

The project is organized into several key directories:

- `models/`: Contains SQLAlchemy models representing database tables.
- `routes/`: Defines FastAPI routes for API endpoints; shared dependencies like `get_db` live in `routes/dependencies.py`.
- `services/`: Business logic and service layer.
- `components/`: Frontend components (to be defined).
- `config/`: Configuration files and settings.
- `middleware/`: Custom middleware for request/response processing.
- `tests/`: Unit and integration tests.
- `templates/`: Jinja2 HTML templates for server-side rendering.
- `docs/`: Documentation files.

Key files include:

- `main.py`: FastAPI application entry point.
- `.env`: Environment variables for configuration.
- `requirements.txt`: Python dependencies.

## Development

The development setup instructions are provided in [docs/development_setup.md](docs/development_setup.md).

To get started locally:

```powershell
# Clone the repository
git clone https://git.allucanget.biz/allucanget/calminer.git
cd calminer

# Create and activate a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

# Start the development server
uvicorn main:app --reload
```

## Usage Overview

- **API base URL**: `http://localhost:8000/api`
- **Key routes**:
  - `POST /api/scenarios/` create scenarios
  - `POST /api/parameters/` manage process parameters; payload supports optional `distribution_id` or inline `distribution_type`/`distribution_parameters` fields for simulation metadata
  - `POST /api/costs/capex` and `POST /api/costs/opex` capture project costs
  - `POST /api/consumption/` add consumption entries
  - `POST /api/production/` register production output
  - `POST /api/equipment/` create equipment records
  - `POST /api/maintenance/` log maintenance events
  - `POST /api/reporting/summary` aggregate simulation results, returning count, mean/median, min/max, standard deviation, variance, percentile bands (5/10/90/95), value-at-risk (95%) and expected shortfall (95%)
- **UI entries** (rendered via FastAPI templates, also reachable from the sidebar):
- `GET /` operations overview dashboard
  - `GET /ui/dashboard` legacy dashboard alias
  - `GET /ui/scenarios` scenario creation form
  - `GET /ui/parameters` parameter input form
  - `GET /ui/costs`, `/ui/consumption`, `/ui/production`, `/ui/equipment`, `/ui/maintenance`, `/ui/simulations`, `/ui/reporting` placeholder views aligned with future integrations

### Dashboard Preview

1. Start the FastAPI server and navigate to `/`.
2. Review the headline metrics, scenario snapshot table, and cost/activity charts sourced from the current database state.
3. Use the "Refresh Dashboard" button to pull freshly aggregated data via `/ui/dashboard/data` without reloading the page.
4. Populate scenarios, costs, production, consumption, simulations, and maintenance records to see charts and lists update.
5. The legacy `/ui/dashboard` route remains available but now serves the same consolidated overview.

## Testing

Testing guidelines and best practices are outlined in [docs/testing.md](docs/testing.md).

To execute the unit test suite:

```powershell
pytest
```

### End-to-End Tests

- Playwright-based E2E tests rely on a session-scoped `live_server` fixture that auto-starts the FastAPI app on `http://localhost:8001`, so no per-test `@pytest.mark.usefixtures("live_server")` annotations are required.
- The fixture now polls `[http://localhost:8001](http://localhost:8001)` until it responds (up to ~30s), ensuring the uvicorn subprocess is ready before Playwright starts navigation, then preloads `/` and waits for a `networkidle` state so sidebar navigation and global assets are ready for each test.
- Latest run (`pytest tests/e2e/` on 2025-10-21) passes end-to-end smoke and form coverage after aligning form selectors, titles, and the live server startup behaviour.

### Coverage Snapshot (2025-10-21)

- `pytest --cov=. --cov-report=term-missing` reports **91%** overall coverage.
- Recent additions pushed `routes/ui.py` and `services/simulation.py` to 100%; remaining gaps are concentrated in `config/database.py`, several `models/*.py` loaders, and `services/reporting.py` (95%).
- Playwright specs under `tests/e2e/` are excluded from the coverage run to keep browser automation optional; their files show as uncovered because they are not executed in the `pytest --cov` workflow.

## Database Objects

The database is composed of several tables that store different types of information.

- **CAPEX** — `capex`: Stores data on capital expenditures.
- **OPEX** — `opex`: Contains information on operational expenditures.
- **Chemical consumption** — `chemical_consumption`: Tracks the consumption of chemical reagents.
- **Fuel consumption** — `fuel_consumption`: Records the amount of fuel consumed.
- **Water consumption** — `water_consumption`: Monitors the use of water.
- **Scrap consumption** — `scrap_consumption`: Tracks the consumption of scrap materials.
- **Production output** — `production_output`: Stores data on production output, such as tons produced and recovery rates.
- **Equipment operation** — `equipment_operation`: Contains operational data for each piece of equipment.
- **Ore batch** — `ore_batch`: Stores information on ore batches, including their grade and other characteristics.
- **Exchange rate** — `exchange_rate`: Contains currency exchange rates.
- **Simulation result** — `simulation_result`: Stores the results of the Monte Carlo simulations.

### Currency normalization and migrations

The project now includes a referential `currency` table and associated migration and backfill tooling to normalize free-text currency fields into a canonical lookup.

- New model: `models/currency.py` implements the `currency` table (id, code, name, symbol, is_active).
- Database migration: `scripts/migrations/20251022_create_currency_table_and_fks.sql` creates the `currency` table, seeds common currencies, adds `currency_id` to `capex`/`opex`, backfills where possible, and adds foreign key constraints. The migration is written to be idempotent and safe on databases where parts of the work were already applied.
- Backfill utility: `scripts/backfill_currency.py` is an idempotent, developer-friendly script to preview and apply backfill updates. It supports `--dry-run` and `--create-missing` flags.
- API & models: `models/capex.py` and `models/opex.py` now reference `currency_id` and include a compatibility property accepting a legacy `currency_code` during creates/updates. Routes accept either `currency_id` or `currency_code`; see `routes/costs.py` and the new `routes/currencies.py` (GET /api/currencies/).
- Frontend: server templates and `static/js` now fetch `/api/currencies/` to populate currency selects for cost forms so users can pick a normalized currency instance.

How to run migrations and backfill (development)

1. Ensure `DATABASE_URL` is set in your PowerShell session to point at a development Postgres instance.

```powershell
$env:DATABASE_URL = 'postgresql://user:pass@host/db'
python scripts/run_migrations.py
python scripts/backfill_currency.py --dry-run
python scripts/backfill_currency.py --create-missing
```

Use `--dry-run` first to verify what will change. The migration and backfill scripts are designed to be safe and idempotent, but always back up production databases before applying schema changes.

## Static Parameters

These are values that are not expected to change frequently and are used for configuration purposes. Some examples include:

- **Currencies**: `currency_code`, `currency_name`.
- **Distribution types**: `distribution_name`.
- **Units**: `unit_name`, `unit_symbol`, `unit_system`, `conversion_to_base`.
- **Parameter categories**: `category_name`.
- **Material types**: `type_name`, `category`.
- **Chemical reagents**: `reagent_name`, `chemical_formula`.
- **Fuel**: `fuel_name`.
- **Water**: `water_type`.
- **Scrap material**: `scrap_name`.

## Variables

These are dynamic data points that are recorded over time and used in calculations and simulations. Some examples include:

- **CAPEX**: `amount`.
- **OPEX**: `amount`.
- **Chemical consumption**: `quantity`, `efficiency`, `waste_factor`.
- **Fuel consumption**: `quantity`.
- **Water consumption**: `quantity`.
- **Scrap consumption**: `quantity`.
- **Production output**: `tons_produced`, `recovery_rate`, `metal_content`, `metallurgical_loss`, `net_revenue`.
- **Equipment operation**: `hours_operated`, `downtime_hours`.
- **Ore batch**: `ore_grade`, `moisture`, `sulfur`, `chlorine`.
- **Exchange rate**: `rate`.
- **Parameter values**: `value`.
- **Simulation result**: NPV (`npv`), IRR (`irr`), EBITDA (`ebitda`), `net_revenue`.
- **Cementation parameters**: `temperature`, pH (`ph`), `reaction_time`, `copper_concentration`, `iron_surface_area`.
- **Precipitate product**: `density`, `melting_point`, `boiling_point`.