allucanget/calminer-docs
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Enhance documentation for data model and import/export processes

Browse Source 
- Updated data model documentation to clarify relationships between projects, scenarios, and profitability calculations.
- Introduced a new guide for data import/export templates, detailing CSV and Excel workflows for profitability, capex, and opex data.
- Created a comprehensive field inventory for data import/export, outlining input fields, derived outputs, and snapshot columns.
- Renamed "Initial Capex Planner" to "Capex Planner" and "Processing Opex Planner" to "Opex Planner" for consistency across user guides.
- Adjusted access paths and related resources in user guides to reflect the new naming conventions.
- Improved clarity and consistency in descriptions and instructions throughout the user documentation.
This commit is contained in:
zwitschi
2025-11-13 14:10:47 +01:00
parent fb6be6d84f
commit 07e68a553d
7 changed files with 583 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
api/README.md
View File

@@ -1,3 +1,22 @@
# API Documentation
<!-- TODO: Add API documentation -->
## Project & Scenario Endpoints
| Method | Path | Roles | Success | Common Errors | Notes |
| --- | --- | --- | --- | --- | --- |
| `GET` | `/projects` | viewer, analyst, project_manager, admin | 200 + `ProjectRead[]` | 401 unauthenticated, 403 insufficient role | Lists all projects visible to the caller. |
| `POST` | `/projects` | project_manager, admin | 201 + `ProjectRead` | 401, 403, 409 name conflict, 422 validation | Creates a project and seeds default pricing settings. |
| `GET` | `/projects/{project_id}` | viewer, analyst, project_manager, admin | 200 + `ProjectRead` | 401, 403, 404 missing project | Returns a single project by id. |
| `PUT` | `/projects/{project_id}` | project_manager, admin | 200 + `ProjectRead` | 401, 403, 404, 422 | Updates mutable fields (name, location, operation_type, description). |
| `DELETE` | `/projects/{project_id}` | project_manager, admin | 204 | 401, 403, 404 | Removes the project and cascading scenarios. |
| `GET` | `/projects/{project_id}/scenarios` | viewer, analyst, project_manager, admin | 200 + `ScenarioRead[]` | 401, 403, 404 | Lists scenarios that belong to the project. |
| `POST` | `/projects/{project_id}/scenarios` | project_manager, admin | 201 + `ScenarioRead` | 401, 403, 404 missing project, 409 duplicate name, 422 validation | Creates a scenario under the project. Currency defaults to pricing metadata when omitted. |
| `POST` | `/projects/{project_id}/scenarios/compare` | viewer, analyst, project_manager, admin | 200 + `ScenarioComparisonResponse` | 401, 403, 404, 422 mismatch/validation | Validates a list of scenario ids before comparison. |
| `GET` | `/scenarios/{scenario_id}` | viewer, analyst, project_manager, admin | 200 + `ScenarioRead` | 401, 403, 404 | Retrieves a single scenario. |
| `PUT` | `/scenarios/{scenario_id}` | project_manager, admin | 200 + `ScenarioRead` | 401, 403, 404, 422 | Updates scenario metadata (status, dates, currency, resource). |
| `DELETE` | `/scenarios/{scenario_id}` | project_manager, admin | 204 | 401, 403, 404 | Removes the scenario and dependent data. |
## OpenAPI Reference
- Interactive docs: `GET /docs`
- Raw schema (JSON): `GET /openapi.json`

30
architecture/08_concepts/02_data_model.md
View File

@@ -118,6 +118,18 @@ A specific configuration of assumptions for a project.
- `financial_inputs`: One-to-many with FinancialInput
- `simulation_parameters`: One-to-many with SimulationParameter
#### Projects → Scenarios → Profitability Calculations
Calminer organises feasibility data in a nested hierarchy. A project defines the overarching mining context and exposes a one-to-many `scenarios` collection. Each scenario captures a self-contained assumption set and anchors derived artefacts such as financial inputs, simulation parameters, and profitability snapshots. Profitability calculations execute at the scenario scope; when triggered, the workflow in `services/calculations.py` persists a `ScenarioProfitability` record and can optionally roll results up to project level by creating a `ProjectProfitability` snapshot. Consumers typically surface the most recent metrics via the `latest_profitability` helpers on both ORM models.
| Layer | ORM models | Pydantic schema(s) | Key relationships |
| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Project | `models.project.Project` | `schemas.project.ProjectRead` | `Project.scenarios`, `Project.profitability_snapshots`, `Project.latest_profitability` |
| Scenario | `models.scenario.Scenario` | `schemas.scenario.ScenarioRead` | `Scenario.project`, `Scenario.profitability_snapshots`, `Scenario.latest_profitability` |
| Profitability calculations | `models.profitability_snapshot.ProjectProfitability`, `models.profitability_snapshot.ScenarioProfitability` | `schemas.calculations.ProfitabilityCalculationRequest`, `schemas.calculations.ProfitabilityCalculationResult` | Persisted via `services.calculations.calculate_profitability`; aggregates scenario metrics into project snapshots |
Detailed CRUD endpoint behaviour for projects and scenarios is documented in `calminer-docs/api/README.md`.
#### FinancialInput
Line-item financial assumption attached to a scenario.
@@ -174,7 +186,7 @@ Project-level snapshot capturing aggregated initial capital expenditure metrics.
#### ScenarioCapexSnapshot
Scenario-level snapshot storing detailed initial capex results.
Scenario-level snapshot storing detailed capex results.
**Table:** `scenario_capex_snapshots`
@@ -200,11 +212,11 @@ Scenario-level snapshot storing detailed initial capex results.
- `scenario`: Many-to-one with Scenario
- `created_by`: Many-to-one with User (nullable)
#### ProjectProcessingOpexSnapshot
#### ProjectOpexSnapshot
Project-level snapshot persisting recurring processing opex metrics.
Project-level snapshot persisting recurring opex metrics.
**Table:** `project_processing_opex_snapshots`
**Table:** `project_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
@@ -214,7 +226,7 @@ Project-level snapshot persisting recurring processing opex metrics.
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual processing opex |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |
@@ -230,11 +242,11 @@ Project-level snapshot persisting recurring processing opex metrics.
- `project`: Many-to-one with Project
- `created_by`: Many-to-one with User (nullable)
#### ScenarioProcessingOpexSnapshot
#### ScenarioOpexSnapshot
Scenario-level snapshot persisting recurring processing opex metrics.
Scenario-level snapshot persisting recurring opex metrics.
**Table:** `scenario_processing_opex_snapshots`
**Table:** `scenario_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
@@ -244,7 +256,7 @@ Scenario-level snapshot persisting recurring processing opex metrics.
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual processing opex |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |

4
userguide/README.md
View File

@@ -4,5 +4,5 @@ CalMiner user-facing documentation is organized by feature area. Start with the
## Available Guides
- [Initial Capex Planner](initial_capex_planner.md) — capture upfront capital components, run calculations, and persist snapshots for projects and scenarios.
- [Processing Opex Planner](processing_opex_planner.md) — manage recurring operational costs, apply escalation/discount assumptions, and store calculation snapshots.
- [Capex Planner](initial_capex_planner.md) — capture upfront capital components, run calculations, and persist snapshots for projects and scenarios.
- [Opex Planner](opex_planner.md) — manage recurring operational costs, apply escalation/discount assumptions, and store calculation snapshots.

99
userguide/data_import_export.md Normal file
View File

@@ -0,0 +1,99 @@
# Data Import & Export Templates
This guide explains how to capture profitability, capex, and opex data for bulk upload or download using the standardized CSV and Excel templates. It builds on the field inventory documented in [data_import_export_field_inventory.md](data_import_export_field_inventory.md) and should be shared with anyone preparing data for the calculation engine.
## Template Package
The import/export toolkit contains two artifacts:
| File | Purpose |
| -------------------------- | ---------------------------------------------------------------------------------------------------- |
| `calminer_financials.csv` | Single-sheet CSV template for quick edits or integrations that generate structured rows dynamically. |
| `calminer_financials.xlsx` | Excel workbook with dedicated sheets, lookup lists, and validation rules for guided data capture. |
Always distribute the templates as a matched set so contributors can choose the format that best suits their workflow.
## CSV Workflow
### Structure Overview
The CSV template uses a `record_type` column to multiplex multiple logical tables inside one file. The exact column requirements, validation ranges, and record types are defined in the [Unified CSV Template Specification](data_import_export_field_inventory.md#unified-csv-template-specification). Key expectations:
- All rows include the scenario code (`scenario_code`), and optionally the project code, to keep capex and opex data aligned with profitability assumptions.
- Numeric cells must use `.` as the decimal separator and omit thousands separators, currency symbols, or formatting.
- Boolean values accept `true`/`false`, `yes`/`no`, or `1`/`0`; exporters should emit `true` or `false`.
### Authoring Steps
1. Populate at least one `profitability_input` row per scenario to establish the calculation context.
2. Add optional `profitability_impurity` rows where penalty modelling is required.
3. Capture capital costs via `capex_component` rows and configure options with a `capex_parameters` record.
4. Capture recurring costs via `processing_opex_component` rows. Add a `processing_opex_parameters` row to define escalation settings.
5. Validate the file against the rules listed in the specification prior to upload. Recommended tooling includes frictionless data checks or schema-enforced pipelines.
### Naming & Delivery
- Use the pattern `calminer_financials_<project-or-scenario-code>_<YYYYMMDD>.csv` when transferring files between teams.
- Provide a brief change log or summary per file so reviewers understand the scope of updates.
## Excel Workflow
### Workbook Layout
The Excel workbook contains structured sheets described in the [Excel Workbook Layout & Validation Rules](data_import_export_field_inventory.md#excel-workbook-layout--validation-rules) section. Highlights:
- The `Summary` sheet captures metadata (scenario, preparer, notes) that propagates into the downstream sheets via Excel Table references.
- Each data sheet (`Profitability_Input`, `Capex_Components`, `Processing_Opex`, etc.) is formatted as an Excel Table with frozen headers and data validation to enforce numeric ranges, lookup values, and boolean lists.
- The `Lookups` sheet holds named ranges for currencies, categories, and frequency values. Keep this sheet hidden to reduce accidental edits.
### Data Entry Checklist
- Confirm the `Summary` sheet lists every scenario included in the workbook before editing dependent sheets.
- Use the provided dropdowns for category, currency, frequency, and boolean fields to avoid invalid values.
- When copying data from other workbooks, paste values only to preserve validation rules.
- Turn on the totals row for component tables to quickly verify aggregate capex and opex amounts.
### Exporting From Excel
When exporting to CSV or ingesting into the API:
- Ensure the workbook is saved in `.xlsx` format; do not rename the sheets.
- If a CSV export is required, export each sheet separately and merge using the column headers defined in the CSV specification.
- Retain the original workbook as your source of truth; conversions to CSV should be treated as transient files for automation pipelines.
## Import Pipeline Expectations
The planned FastAPI import endpoints will perform the following checks:
- Validate record types, required columns, and numeric bounds according to the schemas described in the field inventory.
- Verify that referenced project/scenario codes exist and that foreign key relationships (components, parameters) target a single scenario within the file.
- Normalize casing for category slugs and currency codes to match the calculation services.
- Reject files that mix multiple projects unless intentionally enabled by configuration.
Until the endpoints are available, internal teams can use the specification as the contract for pre-validation scripts or interim ETL routines.
## Exporting Data
Export functionality will mirror the same structures:
- CSV exports will group rows by `record_type`, enabling direct round-tripping with minimal transformation.
- Excel exports will populate the same workbook sheets, preserving lookup lists and validation to support offline edits.
Document any local transformations or additional columns introduced during export so that downstream consumers can reconcile the data with the import templates.
## Change Management
- Increment the `import_version` value on the `Summary` sheet (and document the change here) whenever template-breaking updates occur.
- Update `data_import_export_field_inventory.md` first when adding or removing columns, then revise this guide to explain the user-facing workflow impact.
- Communicate updates to stakeholders and archive superseded templates to avoid drift.
## Stakeholder Review Checklist
Before finalising the templates for production use, circulate the following package to domain stakeholders (finance, engineering, data operations) and collect their feedback:
- `calminer_financials.csv` sample populated with representative profitability, capex, and opex data.
- `calminer_financials.xlsx` workbook showcasing all sheets, dropdowns, and validation rules.
- Links to documentation: this guide and the [field inventory](data_import_export_field_inventory.md).
- Summary of outstanding questions or assumptions (e.g., default currencies, additional categories).
Record the review session outcomes in the project tracker or meeting notes. Once sign-off is received, mark the stakeholder feedback step as complete in `.github/instructions/TODO.md` and proceed with implementation of import/export endpoints.

421
userguide/data_import_export_field_inventory.md Normal file
View File

@@ -0,0 +1,421 @@
# Data Import/Export Field Inventory
This inventory captures the current data fields involved in profitability, capex, and opex workflows. It consolidates inputs accepted by the calculation services, derived outputs that should be available for export, and persisted snapshot columns. The goal is to ground the upcoming CSV/Excel template design in authoritative field definitions.
## Profitability
### Calculation Inputs (`schemas/calculations.py::ProfitabilityCalculationRequest`)
| Field | Type | Constraints & Notes |
| -------------------------- | --------------------- | ---------------------------------------------------------- |
| `metal` | `str` | Required; trimmed lowercase; ore/metal identifier. |
| `ore_tonnage` | `PositiveFloat` | Required; tonnes processed (> 0). |
| `head_grade_pct` | `float` | Required; 0 < value ≤ 100. |
| `recovery_pct` | `float` | Required; 0 < value ≤ 100. |
| `payable_pct` | `float \| None` | Optional; 0 < value ≤ 100; overrides metadata default. |
| `reference_price` | `PositiveFloat` | Required; price per unit in base currency. |
| `treatment_charge` | `float` | ≥ 0. |
| `smelting_charge` | `float` | ≥ 0. |
| `moisture_pct` | `float` | ≥ 0 and ≤ 100. |
| `moisture_threshold_pct` | `float \| None` | Optional; ≥ 0 and ≤ 100. |
| `moisture_penalty_per_pct` | `float \| None` | Optional; penalty per excess moisture %. |
| `premiums` | `float` | Monetary premium adjustments (can be negative). |
| `fx_rate` | `PositiveFloat` | Multiplier to convert to scenario currency; defaults to 1. |
| `currency_code` | `str \| None` | Optional ISO 4217 code; uppercased. |
| `opex` | `float` | ≥ 0; feeds cost aggregation. |
| `sustaining_capex` | `float` | ≥ 0. |
| `capex` | `float` | ≥ 0. |
| `discount_rate` | `float \| None` | Optional; 0 ≤ value ≤ 100. |
| `periods` | `int` | Evaluation periods; 1 ≤ value ≤ 120. |
| `impurities` | `List[ImpurityInput]` | Optional; see table below. |
### Impurity Rows (`schemas/calculations.py::ImpurityInput`)
| Field | Type | Constraints & Notes |
| ----------- | --------------- | -------------------------------------------------- |
| `name` | `str` | Required; trimmed. |
| `value` | `float \| None` | Optional; ≥ 0; measured in ppm. |
| `threshold` | `float \| None` | Optional; ≥ 0. |
| `penalty` | `float \| None` | Optional; penalty factor applied beyond threshold. |
### Derived Outputs (selected)
| Structure | Fields |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PricingResult` (`services/pricing.py`) | `metal`, `ore_tonnage`, `head_grade_pct`, `recovery_pct`, `payable_metal_tonnes`, `reference_price`, `gross_revenue`, `moisture_penalty`, `impurity_penalty`, `treatment_smelt_charges`, `premiums`, `net_revenue`, `currency`. |
| `ProfitabilityCosts` | `opex_total`, `sustaining_capex_total`, `capex`. |
| `ProfitabilityMetrics` | `npv`, `irr`, `payback_period`, `margin`. |
| `CashFlowEntry` | `period`, `revenue`, `opex`, `sustaining_capex`, `net`. |
### Snapshot Columns (`models/profitability_snapshot.py`)
| Field | Type | Notes |
| ---------------------------- | ---------------- | ------------------------------------------------- |
| `project_id` / `scenario_id` | `int` | Foreign key; context. |
| `created_by_id` | `int \| None` | Author reference. |
| `calculation_source` | `str \| None` | Identifier for calculation run. |
| `calculated_at` | `datetime` | Server timestamp. |
| `currency_code` | `str \| None` | ISO code. |
| `npv` | `Numeric(18, 2)` | Net present value. |
| `irr_pct` | `Numeric(12, 6)` | Internal rate of return (%). |
| `payback_period_years` | `Numeric(12, 4)` | Years to payback. |
| `margin_pct` | `Numeric(12, 6)` | Profit margin %. |
| `revenue_total` | `Numeric(18, 2)` | Total revenue. |
| `opex_total` | `Numeric(18, 2)` | Linked opex cost. |
| `sustaining_capex_total` | `Numeric(18, 2)` | Sustaining capital spend. |
| `capex` | `Numeric(18, 2)` | Capex input. |
| `net_cash_flow_total` | `Numeric(18, 2)` | Sum of discounted cash flows. |
| `payload` | `JSON` | Serialized detail (cash flow arrays, parameters). |
| `created_at`, `updated_at` | `datetime` | Audit timestamps. |
## Capex
### Component Inputs (`schemas/calculations.py::CapexComponentInput`)
| Field | Type | Constraints & Notes |
| ------------ | ------------- | ----------------------------------------------- |
| `id` | `int \| None` | Optional persistent identifier; ≥ 1. |
| `name` | `str` | Required; trimmed. |
| `category` | `str` | Required; trimmed lowercase; category slug. |
| `amount` | `float` | Required; ≥ 0; nominal value. |
| `currency` | `str \| None` | Optional ISO 4217 code; uppercased. |
| `spend_year` | `int \| None` | Optional; 0 ≤ value ≤ 120; relative spend year. |
| `notes` | `str \| None` | Optional; max length 500. |
### Capex Parameters & Options
| Structure | Field | Type | Notes |
| ------------------------- | -------------------------- | --------------- | ------------------------------ |
| `CapexParameters` | `currency_code` | `str \| None` | Optional ISO code; uppercased. |
| | `contingency_pct` | `float \| None` | 0 ≤ value ≤ 100. |
| | `discount_rate_pct` | `float \| None` | 0 ≤ value ≤ 100. |
| | `evaluation_horizon_years` | `int \| None` | 1 ≤ value ≤ 100. |
| `CapexCalculationOptions` | `persist` | `bool` | Persist snapshot when true. |
### Derived Outputs (`CapexCalculationResult`)
| Segment | Fields |
| ------------------------------- | ---------------------------------------------------------------------------------------- |
| Totals (`CapexTotals`) | `overall`, `contingency_pct`, `contingency_amount`, `with_contingency`, `by_category[]`. |
| Timeline (`CapexTimelineEntry`) | `year`, `spend`, `cumulative`. |
| Echoed Inputs | Normalized `components`, resolved `parameters`, `options`, `currency`. |
### Snapshot Columns (`models/capex_snapshot.py`)
| Field | Type | Notes |
| ---------------------------- | ---------------- | -------------------------------------------- |
| `project_id` / `scenario_id` | `int` | Foreign key; context. |
| `created_by_id` | `int \| None` | Author reference. |
| `calculation_source` | `str \| None` | Origin tag. |
| `calculated_at` | `datetime` | Server timestamp. |
| `currency_code` | `str \| None` | ISO code. |
| `total_capex` | `Numeric(18, 2)` | Sum of component spend. |
| `contingency_pct` | `Numeric(12, 6)` | Applied contingency %. |
| `contingency_amount` | `Numeric(18, 2)` | Monetary contingency. |
| `total_with_contingency` | `Numeric(18, 2)` | All-in capex. |
| `component_count` | `int \| None` | Count of components. |
| `payload` | `JSON` | Serialized breakdown (components, timeline). |
| `created_at`, `updated_at` | `datetime` | Audit timestamps. |
## Opex
### Component Inputs (`schemas/calculations.py::OpexComponentInput`)
| Field | Type | Constraints & Notes |
| -------------- | ------------- | -------------------------------------------------- |
| `id` | `int \| None` | Optional persistent identifier; ≥ 1. |
| `name` | `str` | Required; trimmed. |
| `category` | `str` | Required; trimmed lowercase; category slug. |
| `unit_cost` | `float` | Required; ≥ 0; base currency. |
| `quantity` | `float` | Required; ≥ 0. |
| `frequency` | `str` | Required; trimmed lowercase; e.g. annual, monthly. |
| `currency` | `str \| None` | Optional ISO 4217 code; uppercased. |
| `period_start` | `int \| None` | Optional; ≥ 0; evaluation period index. |
| `period_end` | `int \| None` | Optional; ≥ 0; must be ≥ `period_start`. |
| `notes` | `str \| None` | Optional; max length 500. |
### Opex Parameters & Options
| Structure | Field | Type | Notes |
| ---------------- | -------------------------- | --------------- | ------------------------------ |
| `OpexParameters` | `currency_code` | `str \| None` | Optional ISO code; uppercased. |
| | `escalation_pct` | `float \| None` | 0 ≤ value ≤ 100. |
| | `discount_rate_pct` | `float \| None` | 0 ≤ value ≤ 100. |
| | `evaluation_horizon_years` | `int \| None` | 1 ≤ value ≤ 100. |
| | `apply_escalation` | `bool` | Toggle escalation usage. |
| `OpexOptions` | `persist` | `bool` | Persist snapshot when true. |
| | `snapshot_notes` | `str \| None` | Optional; max length 500. |
### Derived Outputs (`OpexCalculationResult`)
| Segment | Fields |
| ------------------------------ | ----------------------------------------------------------------------- |
| Totals (`OpexTotals`) | `overall_annual`, `escalated_total`, `escalation_pct`, `by_category[]`. |
| Timeline (`OpexTimelineEntry`) | `period`, `base_cost`, `escalated_cost`. |
| Metrics (`OpexMetrics`) | `annual_average`, `cost_per_ton`. |
| Echoed Inputs | Normalized `components`, `parameters`, `options`, resolved `currency`. |
### Snapshot Columns (`models/opex_snapshot.py`)
| Field | Type | Notes |
| ---------------------------- | ----------------- | ----------------------------------------------------- |
| `project_id` / `scenario_id` | `int` | Foreign key; context. |
| `created_by_id` | `int \| None` | Author reference. |
| `calculation_source` | `str \| None` | Origin tag. |
| `calculated_at` | `datetime` | Server timestamp. |
| `currency_code` | `str \| None` | ISO code. |
| `overall_annual` | `Numeric(18, 2)` | Annual recurring cost. |
| `escalated_total` | `Numeric(18, 2)` | Total cost over horizon with escalation. |
| `annual_average` | `Numeric(18, 2)` | Average annual spend. |
| `evaluation_horizon_years` | `Integer` | Horizon length. |
| `escalation_pct` | `Numeric(12, 6)` | Escalation %. |
| `apply_escalation` | `Boolean` | Flag used for calculation. |
| `component_count` | `Integer \| None` | Component count. |
| `payload` | `JSON` | Serialized breakdown (components, timeline, metrics). |
| `created_at`, `updated_at` | `datetime` | Audit timestamps. |
## Unified CSV Template Specification
### Overview
The CSV template consolidates profitability, capex, and opex data into a single file. Each row declares its purpose via a `record_type` flag so that parsers can route data to the appropriate service. The format is UTF-8 with a header row and uses commas as delimiters. Empty string cells are interpreted as `null`. Monetary values must not include currency symbols; decimals use a period.
### Shared Columns
| Column | Required | Applies To | Validation & Notes |
| --------------- | -------- | ---------------------------- | --------------------------------------------------------------------------------- |
| `record_type` | Yes | All rows | Lowercase slug describing the row (see record types below). |
| `project_code` | No | All rows | Optional external project identifier; trimmed; max length 64. |
| `scenario_code` | No | All rows | Optional external scenario identifier; trimmed; max length 64. |
| `sequence` | No | Component and impurity rows | Optional positive integer governing ordering in UI exports. |
| `notes` | No | Component and parameter rows | Free-form text ≤ 500 chars; trimmed; mirrors request `notes` fields when present. |
### Record Types
#### `profitability_input`
Single row per scenario capturing the fields required by `ProfitabilityCalculationRequest`.
| Column | Required | Validation Notes |
| -------------------------- | -------- | -------------------------------------- |
| `metal` | Yes | Lowercase slug; 1–32 chars. |
| `ore_tonnage` | Yes | Decimal > 0. |
| `head_grade_pct` | Yes | Decimal > 0 and ≤ 100. |
| `recovery_pct` | Yes | Decimal > 0 and ≤ 100. |
| `payable_pct` | No | Decimal > 0 and ≤ 100. |
| `reference_price` | Yes | Decimal > 0. |
| `treatment_charge` | No | Decimal ≥ 0. |
| `smelting_charge` | No | Decimal ≥ 0. |
| `moisture_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `moisture_threshold_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `moisture_penalty_per_pct` | No | Decimal; allow negative for credits. |
| `premiums` | No | Decimal; allow negative. |
| `fx_rate` | No | Decimal > 0; defaults to 1 when blank. |
| `currency_code` | No | ISO 4217 code; uppercase; length 3. |
| `opex` | No | Decimal ≥ 0. |
| `sustaining_capex` | No | Decimal ≥ 0. |
| `capex` | No | Decimal ≥ 0. |
| `discount_rate` | No | Decimal ≥ 0 and ≤ 100. |
| `periods` | No | Integer 1–120. |
#### `profitability_impurity`
Multiple rows permitted per scenario; maps to `ImpurityInput`.
| Column | Required | Validation Notes |
| ----------- | -------- | ------------------------------------- |
| `name` | Yes | Trimmed string; 1–64 chars. |
| `value` | No | Decimal ≥ 0. |
| `threshold` | No | Decimal ≥ 0. |
| `penalty` | No | Decimal; can be negative for credits. |
#### `capex_component`
One row per component feeding `CapexComponentInput`.
| Column | Required | Validation Notes |
| -------------- | -------- | ---------------------------------------------------- |
| `component_id` | No | Integer ≥ 1; references existing record for updates. |
| `name` | Yes | Trimmed string; 1–128 chars. |
| `category` | Yes | Lowercase slug; matches allowed UI categories. |
| `amount` | Yes | Decimal ≥ 0. |
| `currency` | No | ISO 4217 code; uppercase; length 3. |
| `spend_year` | No | Integer 0–120. |
#### `capex_parameters`
At most one row per scenario; populates `CapexParameters` and `CapexCalculationOptions`.
| Column | Required | Validation Notes |
| -------------------------- | -------- | ------------------------------------------------ |
| `currency_code` | No | ISO 4217 code; uppercase. |
| `contingency_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `discount_rate_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `evaluation_horizon_years` | No | Integer 1–100. |
| `persist_snapshot` | No | `true`/`false` case-insensitive; defaults false. |
#### `opex_component`
Maps to `OpexComponentInput`; multiple rows allowed.
| Column | Required | Validation Notes |
| -------------- | -------- | ------------------------------------------------------------------------- |
| `component_id` | No | Integer ≥ 1; references existing record for updates. |
| `name` | Yes | Trimmed string; 1–128 chars. |
| `category` | Yes | Lowercase slug; matches enum used in UI (e.g. energy, labor). |
| `unit_cost` | Yes | Decimal ≥ 0. |
| `quantity` | Yes | Decimal ≥ 0. |
| `frequency` | Yes | Lowercase slug; allowed values: `annual`, `monthly`, `quarterly`, `once`. |
| `currency` | No | ISO 4217 code; uppercase; length 3. |
| `period_start` | No | Integer ≥ 0; must be ≤ `period_end` when provided. |
| `period_end` | No | Integer ≥ 0; defaults to `period_start` when blank. |
#### `opex_parameters`
At most one row per scenario; maps to `OpexParameters` and options.
| Column | Required | Validation Notes |
| -------------------------- | -------- | ------------------------------- |
| `currency_code` | No | ISO 4217 code; uppercase. |
| `escalation_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `discount_rate_pct` | No | Decimal ≥ 0 and ≤ 100. |
| `evaluation_horizon_years` | No | Integer 1–100. |
| `apply_escalation` | No | `true`/`false`; defaults true. |
| `persist_snapshot` | No | `true`/`false`; defaults false. |
| `snapshot_notes` | No | Free-form text ≤ 500 chars. |
### Validation Rules Summary
- Parsers must group rows by `project_code` + `scenario_code`; missing codes fall back to payload metadata supplied during import.
- `record_type` values outside the table must raise a validation error.
- Component identifiers (`component_id`) are optional for inserts but required to overwrite existing records.
- Decimal columns should accept up to two fractional places for currency-aligned fields (`amount`, `overall`, etc.) and up to six for percentage columns.
- Boolean columns accept `true`, `false`, `1`, `0`, `yes`, `no` (case-insensitive); exporters should emit `true`/`false`.
## Excel Workbook Layout & Validation Rules
### Workbook Structure
| Sheet Name | Purpose |
| -------------------------- | ------------------------------------------------------------------------------------- |
| `Summary` | Capture project/scenario metadata and import scope details. |
| `Profitability_Input` | Tabular input for `ProfitabilityCalculationRequest`. |
| `Profitability_Impurities` | Optional impurity rows linked to a scenario. |
| `Capex_Components` | Component-level spend records for the capex planner. |
| `Capex_Parameters` | Global capex parameters/options for a scenario. |
| `Opex` | Component-level recurring cost records for opex. |
| `Opex_Parameters` | Global opex parameters/options for a scenario. |
| `Lookups` | Controlled vocabulary lists consumed by data validation (categories, booleans, etc.). |
All sheets except `Lookups` start with a frozen header row and are formatted as Excel Tables (e.g., `tbl_profitability`). Tables enforce consistent column names and simplify import parsing.
### `Summary` Sheet
| Column | Required | Validation & Notes |
| ---------------- | -------- | --------------------------------------------------------------------------------- |
| `import_version` | Yes | Static text `v1`; used to detect template drift. |
| `project_code` | No | Matches shared `project_code`; max 64 chars; trimmed. |
| `scenario_code` | Yes | Identifier tying all sheets together; max 64 chars; duplicates allowed for batch. |
| `prepared_by` | No | Free-form text ≤ 128 chars. |
| `prepared_on` | No | Excel date; data validation restricts to dates ≥ `TODAY()-365`. |
| `notes` | No | Free-form text ≤ 500 chars; carry-over to import metadata. |
### `Profitability_Input`
Columns mirror the CSV specification; data validation rules apply per cell:
- Numeric fields (`ore_tonnage`, `reference_price`, etc.) use decimal validation with explicit min/max aligned to service constraints.
- Percentage fields (`head_grade_pct`, `discount_rate`) use decimals with bounds (e.g., 0–100). Apply `Data Validation → Decimal` settings.
- `currency_code` validation references `Lookups!$A:$A` (ISO codes list).
- Table default rows include scenario code reference via structured formula: `=[@Scenario_Code]` autocompletes when set.
### `Profitability_Impurities`
| Column | Required | Validation & Notes |
| --------------- | -------- | ----------------------------------------------------- |
| `scenario_code` | Yes | Drop-down referencing `Summary!C:C`; ensures linkage. |
| `name` | Yes | Text ≤ 64 chars; duplicates allowed. |
| `value` | No | Decimal ≥ 0. |
| `threshold` | No | Decimal ≥ 0. |
| `penalty` | No | Decimal; allow negatives. |
### `Capex_Components`
| Column | Required | Validation & Notes |
| --------------- | -------- | ------------------------------------------------------ |
| `scenario_code` | Yes | Drop-down referencing `Summary!C:C`. |
| `component_id` | No | Whole number ≥ 1; optional when inserting new records. |
| `name` | Yes | Text ≤ 128 chars. |
| `category` | Yes | Drop-down referencing `Lookups!category_values`. |
| `amount` | Yes | Decimal ≥ 0; formatted as currency (no symbol). |
| `currency` | No | Drop-down referencing `Lookups!currency_codes`. |
| `spend_year` | No | Whole number 0–120. |
| `notes` | No | Text ≤ 500 chars. |
### `Capex_Parameters`
Single-row table per scenario with structured references:
| Column | Required | Validation & Notes |
| -------------------------- | -------- | ------------------------------------------------------------ |
| `scenario_code` | Yes | Drop-down referencing `Summary!C:C`. |
| `currency_code` | No | Drop-down referencing `Lookups!currency_codes`. |
| `contingency_pct` | No | Decimal 0–100 with two decimal places. |
| `discount_rate_pct` | No | Decimal 0–100. |
| `evaluation_horizon_years` | No | Whole number 1–100. |
| `persist_snapshot` | No | Drop-down referencing `Lookups!boolean_values` (True/False). |
| `notes` | No | Text ≤ 500 chars; maps to request options metadata. |
### `Opex`
| Column | Required | Validation & Notes |
| --------------- | -------- | -------------------------------------------------------------------------------------------------------------------------- |
| `scenario_code` | Yes | Drop-down referencing `Summary!C:C`. |
| `component_id` | No | Whole number ≥ 1; optional for inserts. |
| `name` | Yes | Text ≤ 128 chars. |
| `category` | Yes | Drop-down referencing `Lookups!opex_categories`. |
| `unit_cost` | Yes | Decimal ≥ 0. |
| `quantity` | Yes | Decimal ≥ 0. |
| `frequency` | Yes | Drop-down referencing `Lookups!frequency_values` (annual, monthly, quarterly, once). |
| `currency` | No | Drop-down referencing `Lookups!currency_codes`. |
| `period_start` | No | Whole number ≥ 0; additional rule ensures `period_end` ≥ `period_start` via custom formula `=IF(ISBLANK(H2),TRUE,H2<=I2)`. |
| `period_end` | No | Whole number ≥ 0. |
| `notes` | No | Text ≤ 500 chars. |
### `Opex_Parameters`
| Column | Required | Validation & Notes |
| -------------------------- | -------- | ----------------------------------------------- |
| `scenario_code` | Yes | Drop-down referencing `Summary!C:C`. |
| `currency_code` | No | Drop-down referencing `Lookups!currency_codes`. |
| `escalation_pct` | No | Decimal 0–100 with up to two decimals. |
| `discount_rate_pct` | No | Decimal 0–100. |
| `evaluation_horizon_years` | No | Whole number 1–100. |
| `apply_escalation` | No | Drop-down referencing `Lookups!boolean_values`. |
| `persist_snapshot` | No | Drop-down referencing `Lookups!boolean_values`. |
| `snapshot_notes` | No | Text ≤ 500 chars. |
### `Lookups` Sheet
Contains named ranges used by validation rules:
| Named Range | Column Contents |
| ------------------ | ---------------------------------------------- |
| `currency_codes` | ISO 4217 codes supported by the platform. |
| `category_values` | Allowed capex categories (e.g., engineering). |
| `opex_categories` | Allowed opex categories (e.g., energy, labor). |
| `frequency_values` | `annual`, `monthly`, `quarterly`, `once`. |
| `boolean_values` | `TRUE`, `FALSE`. |
The sheet is hidden by default to avoid accidental edits. Import logic should bundle the lookup dictionary alongside the workbook to verify user-supplied values.
### Additional Validation Guidance
- Protect header rows to prevent renaming; enable `Allow Users to Edit Ranges` for data sections only.
- Apply conditional formatting to highlight missing required fields (`ISBLANK`) and out-of-range values.
- Provide data validation error messages explaining expected ranges to reduce back-and-forth with users.
- Recommend enabling the Excel Table totals row for quick sanity checks (sum of amounts, counts of components).
## Next Use
The consolidated tables above provide the authoritative field inventory required to draft CSV column layouts and Excel worksheet structures. They also surface validation ranges and metadata that must be preserved during import/export.

10
userguide/initial_capex_planner.md
View File

@@ -1,11 +1,11 @@
# Initial Capex Planner
# Capex Planner
The Initial Capex Planner helps project teams capture upfront capital requirements, categorize spend, and produce a shareable breakdown that feeds downstream profitability analysis. The feature implements the acceptance criteria described in [FR-013](../requirements/FR-013.md) and persists calculation snapshots for both projects and scenarios when context is provided.
The Capex Planner helps project teams capture upfront capital requirements, categorize spend, and produce a shareable breakdown that feeds downstream profitability analysis. The feature implements the acceptance criteria described in [FR-013](../requirements/FR-013.md) and persists calculation snapshots for both projects and scenarios when context is provided.
## Access Paths
- **Workspace sidebar**: Navigate to _Workspace → Initial Capex Planner_.
- **Scenario detail page**: Use the _Initial Capex Planner_ button in the scenario header to open the planner pre-loaded with the selected project and scenario.
- **Workspace sidebar**: Navigate to _Workspace → Capex Planner_.
- **Scenario detail page**: Use the _Capex Planner_ button in the scenario header to open the planner pre-loaded with the selected project and scenario.
## Prerequisites
@@ -54,6 +54,6 @@ Refer to `tests/integration/test_capex_calculations.py` for example payloads and
## Related Resources
- [Initial Capex Planner template](../../calminer/templates/scenarios/capex.html)
- [Capex Planner template](../../calminer/templates/scenarios/capex.html)
- [Capex snapshot models](../../calminer/models/capex_snapshot.py)
- [FR-013 Requirement](../requirements/FR-013.md)

30
userguide/processing_opex_planner.md
View File

@@ -1,10 +1,10 @@
# Processing Opex Planner
# Opex Planner
The Processing Opex Planner captures recurring operational costs, applies escalation/discount assumptions, and persists calculation snapshots for projects and scenarios. It satisfies the Operational Expenditure tooling defined in [FR-014](../requirements/FR-014.md).
The Opex Planner captures recurring operational costs, applies escalation/discount assumptions, and persists calculation snapshots for projects and scenarios. It satisfies the Operational Expenditure tooling defined in [FR-014](../requirements/FR-014.md).
## Access Paths
- **Workspace sidebar**: Navigate to _Workspace → Processing Opex Planner_.
- **Workspace sidebar**: Navigate to _Workspace → Opex Planner_.
- **Scenario detail header**: Use the _Processing Opex Planner_ button to open the planner pre-loaded with the active project/scenario context.
## Prerequisites
@@ -19,7 +19,7 @@ The Processing Opex Planner captures recurring operational costs, applies escala
3. **Capture opex components** in the components table. Each row records the details listed in the _Component Fields_ table below. Rows support dynamic add/remove interactions and inline validation messages for missing data.
4. **Adjust global parameters** such as escalation percentage, discount rate, evaluation horizon, and whether escalation applies to the timeline. Parameter defaults derive from the active scenario, project, or environment configuration.
5. **Add optional snapshot notes** that persist alongside stored results and appear in the snapshot history UI (planned) and API responses.
6. **Run the calculation** with _Save & Calculate_. The POST handler validates input via `ProcessingOpexCalculationRequest`, calls `services.calculations.calculate_processing_opex`, and repopulates the form with normalized data. When validation fails, the page returns a 422 status with component-level alerts.
6. **Run the calculation** with _Save & Calculate_. The POST handler validates input via `OpexCalculationRequest`, calls `services.calculations.calculate_opex`, and repopulates the form with normalized data. When validation fails, the page returns a 422 status with component-level alerts.
7. **Review results** in the summary cards and detailed breakdowns: total annual opex, per-ton cost, category totals, escalated timeline table, and the normalized component listing. Alerts surface validation issues or component-level notices above the table.
### Component Fields
@@ -50,21 +50,21 @@ The Processing Opex Planner captures recurring operational costs, applies escala
## Persistence Behaviour
- When `project_id` or `scenario_id` is supplied and `options[persist]` evaluates true (default for HTML form), snapshots are stored via `ProjectProcessingOpexSnapshot` and `ScenarioProcessingOpexSnapshot` repositories before rendering the response.
- When `project_id` or `scenario_id` is supplied and `options[persist]` evaluates true (default for HTML form), snapshots are stored via `ProjectOpexSnapshot` and `ScenarioOpexSnapshot` repositories before rendering the response.
- Snapshot payloads capture normalized component entries, parameters, escalation settings, calculated totals, and optional notes, enabling historical comparison and downstream profitability inputs.
- JSON clients can disable persistence by sending `"options": {"persist": false}` or omit identifiers for ad hoc calculations.
## API Reference
| Route | Method | Description |
| ----------------------------------------------------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/calculations/processing-opex` (`calculations.processing_opex_form`) | GET | Renders the planner template with defaults and any latest snapshot context for the supplied project/scenario IDs. |
| `/calculations/processing-opex` (`calculations.processing_opex_submit`) | POST | Accepts form or JSON payload matching `ProcessingOpexCalculationRequest`, returns HTML or JSON results, and persists snapshots when context is present. |
| Route | Method | Description |
| ------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `/calculations/opex` (`calculations.opex_form`) | GET | Renders the planner template with defaults and any latest snapshot context for the supplied project/scenario IDs. |
| `/calculations/opex` (`calculations.opex_submit`) | POST | Accepts form or JSON payload matching `OpexCalculationRequest`, returns HTML or JSON results, and persists snapshots when context is present. |
Key schemas and services:
- `schemas.calculations.ProcessingOpexComponentInput`, `ProcessingOpexParameters`, `ProcessingOpexCalculationRequest`, `ProcessingOpexCalculationResult`
- `services.calculations.calculate_processing_opex`
- `schemas.calculations.OpexComponentInput`, `OpexParameters`, `OpexCalculationRequest`, `OpexCalculationResult`
- `services.calculations.calculate_opex`
- `_prepare_opex_context`, `_persist_opex_snapshots`, and related helpers in `routes/calculations.py`
### Example JSON Request
@@ -97,7 +97,7 @@ Key schemas and services:
}
```
The response payload mirrors `ProcessingOpexCalculationResult`, returning normalized components, aggregated totals, timeline series, and snapshot metadata when persistence is enabled.
The response payload mirrors `OpexCalculationResult`, returning normalized components, aggregated totals, timeline series, and snapshot metadata when persistence is enabled.
## Troubleshooting
@@ -107,8 +107,8 @@ The response payload mirrors `ProcessingOpexCalculationResult`, returning normal
## Related Resources
- [Processing Opex planner template](../../calminer/templates/scenarios/opex.html)
- [Opex planner template](../../calminer/templates/scenarios/opex.html)
- [Calculations route handlers](../../calminer/routes/calculations.py)
- [Opex schemas and results](../../calminer/schemas/calculations.py)
- [Service implementation and tests](../../calminer/services/calculations.py), [tests/services/test_calculations_processing_opex.py](../../calminer/tests/services/test_calculations_processing_opex.py)
- [Integration coverage](../../calminer/tests/integration/test_processing_opex_calculations.py)
- [Service implementation and tests](../../calminer/services/calculations.py), [tests/services/test_calculations_opex.py](../../calminer/tests/services/test_calculations_opex.py)
- [Integration coverage](../../calminer/tests/integration/test_opex_calculations.py)