calminer/README.md

# CalMiner

A web application to plan mining projects and estimate costs, returns and profitability.

Focuses on ore mining operations and covering parameters such as capital and operational expenditures, resource consumption, production output, and Monte Carlo simulations for risk analysis.

The system is designed to help mining companies make informed decisions by simulating various scenarios and analyzing potential outcomes based on stochastic variables.

A range of features are implemented to support these functionalities.

## Features

- **Scenario Management**: Manage multiple mining scenarios with independent parameter sets and outputs.
- **Process Parameters**: Define and persist process inputs via FastAPI endpoints and template-driven forms.
- **Cost Tracking**: Capture capital (`capex`) and operational (`opex`) expenditures per scenario.
- **Consumption Tracking**: Record resource consumption (chemicals, fuel, water, scrap) tied to scenarios.
- **Production Output**: Store production metrics such as tonnage, recovery, and revenue drivers.
- **Equipment Management**: Register scenario-specific equipment inventories.
- **Maintenance Logging**: Log maintenance events against equipment with dates and costs.
- **Reporting Dashboard**: Surface aggregated statistics for simulation outputs with an interactive Chart.js dashboard.
- **Unified UI Shell**: Server-rendered templates extend a shared base layout with a persistent left sidebar linking scenarios, parameters, costs, consumption, production, equipment, maintenance, simulations, and reporting views.
- **Operations Overview Dashboard**: The root route (`/`) surfaces cross-scenario KPIs, charts, and maintenance reminders with a one-click refresh backed by aggregated loaders.
- **Theming Tokens**: Shared CSS variables in `static/css/main.css` centralize the UI color palette for consistent styling and rapid theme tweaks.
- **Modular Frontend Scripts**: Page-specific interactions now live in `static/js/` modules, keeping templates lean while enabling browser caching and reuse.
- **Monte Carlo Simulation (in progress)**: Services and routes are scaffolded for future stochastic analysis.

## Architecture

The architecture is documented in [docs/architecture.md](docs/architecture.md).

## Project Structure

The project is organized into several key directories:

- `models/`: Contains SQLAlchemy models representing database tables.
- `routes/`: Defines FastAPI routes for API endpoints; shared dependencies like `get_db` live in `routes/dependencies.py`.
- `services/`: Business logic and service layer.
- `components/`: Frontend components (to be defined).
- `config/`: Configuration files and settings.
- `middleware/`: Custom middleware for request/response processing.
- `tests/`: Unit and integration tests.
- `templates/`: Jinja2 HTML templates for server-side rendering.
- `docs/`: Documentation files.

Key files include:

- `main.py`: FastAPI application entry point.
- `.env`: Environment variables for configuration.
- `requirements.txt`: Python dependencies.

## Development

The development setup instructions are provided in [docs/development_setup.md](docs/development_setup.md).

To get started locally:

```powershell
# Clone the repository
git clone https://git.allucanget.biz/allucanget/calminer.git
cd calminer

# Create and activate a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

# Start the development server
uvicorn main:app --reload
```

## Usage Overview

- **API base URL**: `http://localhost:8000/api`
- **Key routes**:
  - `POST /api/scenarios/` create scenarios
  - `POST /api/parameters/` manage process parameters; payload supports optional `distribution_id` or inline `distribution_type`/`distribution_parameters` fields for simulation metadata
  - `POST /api/costs/capex` and `POST /api/costs/opex` capture project costs
  - `POST /api/consumption/` add consumption entries
  - `POST /api/production/` register production output
  - `POST /api/equipment/` create equipment records
  - `POST /api/maintenance/` log maintenance events
  - `POST /api/reporting/summary` aggregate simulation results, returning count, mean/median, min/max, standard deviation, variance, percentile bands (5/10/90/95), value-at-risk (95%) and expected shortfall (95%)
- **UI entries** (rendered via FastAPI templates, also reachable from the sidebar):
- `GET /` operations overview dashboard
  - `GET /ui/dashboard` legacy dashboard alias
  - `GET /ui/scenarios` scenario creation form
  - `GET /ui/parameters` parameter input form
  - `GET /ui/costs`, `/ui/consumption`, `/ui/production`, `/ui/equipment`, `/ui/maintenance`, `/ui/simulations`, `/ui/reporting` placeholder views aligned with future integrations

### Dashboard Preview

1. Start the FastAPI server and navigate to `/`.
2. Review the headline metrics, scenario snapshot table, and cost/activity charts sourced from the current database state.
3. Use the "Refresh Dashboard" button to pull freshly aggregated data via `/ui/dashboard/data` without reloading the page.
4. Populate scenarios, costs, production, consumption, simulations, and maintenance records to see charts and lists update.
5. The legacy `/ui/dashboard` route remains available but now serves the same consolidated overview.

## Testing

Testing guidelines and best practices are outlined in [docs/testing.md](docs/testing.md).

To execute the unit test suite:

```powershell
pytest
```

### End-to-End Tests

- Playwright-based E2E tests rely on a session-scoped `live_server` fixture that auto-starts the FastAPI app on `http://localhost:8001`, so no per-test `@pytest.mark.usefixtures("live_server")` annotations are required.
- The fixture now polls `[http://localhost:8001](http://localhost:8001)` until it responds (up to ~30s), ensuring the uvicorn subprocess is ready before Playwright starts navigation, then preloads `/` and waits for a `networkidle` state so sidebar navigation and global assets are ready for each test.
- Latest run (`pytest tests/e2e/` on 2025-10-21) passes end-to-end smoke and form coverage after aligning form selectors, titles, and the live server startup behaviour.

### Coverage Snapshot (2025-10-20)

- `pytest --cov=. --cov-report=term-missing` reports **95%** overall coverage across the project.
- Lower coverage hotspots to target next: `services/simulation.py` (79%), `middleware/validation.py` (78%), `routes/ui.py` (82%), and several API routers around lines 12-22 that create database sessions only.
- Deprecation cleanup migrated routes to Pydantic v2 patterns (`model_config = ConfigDict(...)`, `model_dump()`) and updated SQLAlchemy's `declarative_base`; reran `pytest` to confirm the suite passes without warnings.
- Coverage for route-heavy modules is primarily limited by error paths (e.g., bad request branches) that still need explicit tests.

## Database Objects

The database is composed of several tables that store different types of information.

- **CAPEX** — `capex`: Stores data on capital expenditures.
- **OPEX** — `opex`: Contains information on operational expenditures.
- **Chemical consumption** — `chemical_consumption`: Tracks the consumption of chemical reagents.
- **Fuel consumption** — `fuel_consumption`: Records the amount of fuel consumed.
- **Water consumption** — `water_consumption`: Monitors the use of water.
- **Scrap consumption** — `scrap_consumption`: Tracks the consumption of scrap materials.
- **Production output** — `production_output`: Stores data on production output, such as tons produced and recovery rates.
- **Equipment operation** — `equipment_operation`: Contains operational data for each piece of equipment.
- **Ore batch** — `ore_batch`: Stores information on ore batches, including their grade and other characteristics.
- **Exchange rate** — `exchange_rate`: Contains currency exchange rates.
- **Simulation result** — `simulation_result`: Stores the results of the Monte Carlo simulations.

## Static Parameters

These are values that are not expected to change frequently and are used for configuration purposes. Some examples include:

- **Currencies**: `currency_code`, `currency_name`.
- **Distribution types**: `distribution_name`.
- **Units**: `unit_name`, `unit_symbol`, `unit_system`, `conversion_to_base`.
- **Parameter categories**: `category_name`.
- **Material types**: `type_name`, `category`.
- **Chemical reagents**: `reagent_name`, `chemical_formula`.
- **Fuel**: `fuel_name`.
- **Water**: `water_type`.
- **Scrap material**: `scrap_name`.

## Variables

These are dynamic data points that are recorded over time and used in calculations and simulations. Some examples include:

- **CAPEX**: `amount`.
- **OPEX**: `amount`.
- **Chemical consumption**: `quantity`, `efficiency`, `waste_factor`.
- **Fuel consumption**: `quantity`.
- **Water consumption**: `quantity`.
- **Scrap consumption**: `quantity`.
- **Production output**: `tons_produced`, `recovery_rate`, `metal_content`, `metallurgical_loss`, `net_revenue`.
- **Equipment operation**: `hours_operated`, `downtime_hours`.
- **Ore batch**: `ore_grade`, `moisture`, `sulfur`, `chlorine`.
- **Exchange rate**: `rate`.
- **Parameter values**: `value`.
- **Simulation result**: NPV (`npv`), IRR (`irr`), EBITDA (`ebitda`), `net_revenue`.
- **Cementation parameters**: `temperature`, pH (`ph`), `reaction_time`, `copper_concentration`, `iron_surface_area`.
- **Precipitate product**: `density`, `melting_point`, `boiling_point`.