diff --git a/architecture/08_concepts.md b/architecture/08_concepts.md index bfbf2aa..20f7e1a 100644 --- a/architecture/08_concepts.md +++ b/architecture/08_concepts.md @@ -71,11 +71,27 @@ Each tenant can have customized settings and preferences, managed through the Ma ## Data Model -The system employs a relational data model to manage and store information efficiently. The primary entities include Users, Projects, Datasets, and Results. Each entity is represented as a table in the database, with relationships defined through foreign keys. +The system employs a relational data model to manage and store information efficiently. Core entities include Users, Projects, Scenarios, Financial Inputs, Simulation Parameters, Pricing Settings, and supporting metadata tables. Relationships are enforced using foreign keys with cascading deletes where appropriate (e.g., scenarios cascade from projects, financial inputs cascade from scenarios). -A detailed [Data Model](08_concepts/02_data_model.md) documentation is available. +Domain-specific enumerations are mapped to PostgreSQL enum types and mirrored in the ORM layer via `models.enums`: -Discounted cash-flow metrics (NPV, IRR, Payback) referenced by the economic portion of the data model are described in the [Financial Metrics Specification](../specifications/financial_metrics.md), while stochastic extensions leverage the Monte Carlo engine documented in the [Monte Carlo Simulation Specification](../specifications/monte_carlo_simulation.md). +- `MiningOperationType` — categorises projects (open pit, underground, in-situ leach, etc.). +- `ScenarioStatus` — tracks lifecycle state (draft, active, archived). +- `FinancialCategory` and `CostBucket` — classify financial inputs for reporting granularity. +- `DistributionType` and `StochasticVariable` — describe probabilistic simulation parameters. +- `ResourceType` — annotates primary resource consumption for scenarios and stochastic parameters. + +These enums back the relevant model columns through named SQLAlchemy `Enum` definitions referencing the same PostgreSQL type names, ensuring parity between application logic and database constraints. + +Schema creation and baseline seed data are handled by the idempotent initializer (`scripts/init_db.py`). On every application startup the initializer guarantees that: + +1. Enum types exist exactly once. +2. Tables for authentication, pricing, and mining domain entities are present. +3. Default roles, administrator account, pricing settings, and representative demo projects/scenarios/financial inputs are available for immediate use. + +Developers can reset to a clean slate with `scripts/reset_db.py`, which safely drops managed tables and enum types in non-production environments before rerunning the initializer. + +A detailed [Data Model](08_concepts/02_data_model.md) documentation is available. Discounted cash-flow metrics (NPV, IRR, Payback) referenced by the economic portion of the data model are described in the [Financial Metrics Specification](../specifications/financial_metrics.md), while stochastic extensions leverage the Monte Carlo engine documented in the [Monte Carlo Simulation Specification](../specifications/monte_carlo_simulation.md). All data interactions are handled through the [Data Access Layer](05_building_block_view.md#data-access-layer), ensuring consistency and integrity across operations. diff --git a/architecture/08_concepts/02_data_model.md b/architecture/08_concepts/02_data_model.md index 75d7ab5..b55ff53 100644 --- a/architecture/08_concepts/02_data_model.md +++ b/architecture/08_concepts/02_data_model.md @@ -1,541 +1,626 @@ # Data Model -The data model for the system is designed to capture the essential entities and their relationships involved in mining projects. Each entity is represented as a table in the relational database, with attributes defining their properties and foreign keys establishing relationships between them. - -## Table of Contents - -- [Data Model](#data-model) - - [Table of Contents](#table-of-contents) - - [Relationships Diagram](#relationships-diagram) - - [User](#user) - - [User Diagram](#user-diagram) - - [User Roles](#user-roles) - - [System Administrator](#system-administrator) - - [Mandator Administrator](#mandator-administrator) - - [Project Manager](#project-manager) - - [Standard User](#standard-user) - - [Permissions](#permissions) - - [Mandator](#mandator) - - [Project](#project) - - [Location](#location) - - [Currency](#currency) - - [Unit](#unit) - - [Mining Technology](#mining-technology) - - [Product Model](#product-model) - - [Quality Metrics](#quality-metrics) - - [Financial Model](#financial-model) - - [Cost Model](#cost-model) - - [CAPEX](#capex) - - [OPEX](#opex) - - [Revenue Model](#revenue-model) - - [Revenue Streams](#revenue-streams) - - [Product Sales](#product-sales) - - [Investment Model](#investment-model) - - [Economic Model](#economic-model) - - [Discounted Cash Flow Metrics](#discounted-cash-flow-metrics) - - [Risk Model](#risk-model) - - [Parameter](#parameter) - - [Scenario](#scenario) - -## Relationships Diagram - -```mermaid -erDiagram - User ||--o{ UserRole : has - UserRole ||--o{ RolePermission : has - Permission ||--o{ RolePermission : assigned_to - - User ||--o{ Mandator : belongs_to - Mandator ||--o{ Project : has_many - Project ||--|| Location : located_at - Project ||--o{ MiningTechnology : uses - Project ||--o{ FinancialModel : has_many - Project ||--o{ Parameter : has_many - Project ||--o{ Scenario : has_many - - ProductModel ||--o{ QualityMetric : has - - FinancialModel ||--o{ CostModel : includes - FinancialModel ||--o{ RevenueModel : includes - FinancialModel ||--o{ InvestmentModel : includes - FinancialModel ||--o{ EconomicModel : includes - FinancialModel ||--o{ RiskModel : includes - MiningTechnology ||--o{ Parameter : has_many - MiningTechnology ||--o{ QualityMetric : has_many - CostModel ||--|| CAPEX : has - CostModel ||--|| OPEX : has - RevenueModel ||--o{ RevenueStream : has - RevenueModel ||--o{ ProductSale : has - - Currency ||--o{ CAPEX : used_in - Currency ||--o{ OPEX : used_in - Currency ||--o{ RevenueStream : used_in - Currency ||--o{ ProductSale : used_in - Unit ||--o{ ProductModel : used_in - Unit ||--o{ QualityMetric : used_in - Unit ||--o{ Parameter : used_in - - MiningTechnology ||--o{ Parameter : has_many - MiningTechnology ||--o{ QualityMetric : has_many - MiningTechnology ||--o{ MonteCarloSimulation : has_many -``` - -## User - -Users are individuals who have access to the Calminer system. - -Each User is associated with a single Mandator and can be assigned to multiple Projects under that Mandator. Users have specific roles and permissions that determine their level of access within the system. - -### User Diagram - -```mermaid -erDiagram - User { - string UserID PK - string Name - string Email - string PasswordHash - datetime CreatedAt - datetime UpdatedAt - datetime LastLogin - boolean IsActive - } -``` - -### User Roles - -Users can have different roles within the Calminer system, which define their permissions and access levels. - -```mermaid -erDiagram - User ||--o{ UserRole : has - UserRole { - string RoleID PK - string RoleName - string Description - } -``` - -#### System Administrator - -The System Administrator role is a special user role that has elevated privileges across the entire Calminer system. This role is responsible for managing system settings, configurations, managing Mandators, user accounts, and overall system maintenance. - -#### Mandator Administrator - -The Mandator Administrator role is a special user role assigned to individuals who manage a specific Mandator. This role has the authority to oversee Users and Projects within their Mandator, including user assignments and project configurations. - -#### Project Manager - -The Project Manager role is responsible for overseeing specific Projects within a Mandator. This role has permissions to manage project settings, assign Users to the Project, and monitor project progress. - -#### Standard User - -The Standard User role can participate in Projects assigned to them but has limited access to administrative functions. - -### Permissions - -Permissions are defined based on user roles, determining what actions a user can perform within the system. Permissions include: - -- View Projects -- Edit Projects -- Manage Users -- Configure System Settings -- Access Reports -- Run Simulations -- Manage Financial Models -- Export Data -- Import Data -- View Logs -- Manage Notifications -- Access API - -Permissions are assigned to roles, and users inherit permissions based on their assigned roles. For this purpose, a helper entity `RolePermission` is defined to map roles to their respective permissions. - -The Permissions entity (and RolePermission entity) is defined as follows: - -```mermaid -erDiagram - RolePermission { - string RoleID FK - string PermissionID FK - } - Permission { - string PermissionID PK - string PermissionName - string Description - } - UserRole ||--o{ RolePermission : has - Permission ||--o{ RolePermission : assigned_to -``` - -## Mandator - -The Mandator entity represents organizational units or clients using the system. A Mandator can have multiple Users and Projects associated with it. - -```mermaid -erDiagram - Mandator { - string MandatorID PK - string Name - string ContactInfo - datetime CreatedAt - datetime UpdatedAt - boolean IsActive - } -``` - -## Project - -The Project entity encapsulates the details of mining projects. Attributes include ProjectID, ProjectName, Description, LocationID (linking to the Location entity), CreationDate, and OwnerID (linking to the User entity). Projects can have multiple Datasets associated with them. - -```mermaid -erDiagram - Project { - string ProjectID PK - string ProjectName - string Description - string LocationID FK - datetime CreationDate - string OwnerID FK - string Status - datetime StartDate - } -``` - -## Location - -The Location entity represents the geographical context of the data being mined. Locations can be linked to multiple Projects. - -```mermaid -erDiagram - Location { - string LocationID PK - string Name - string Description - string Coordinates - string Region - string Country - } -``` - -## Currency - -The Currency entity defines the monetary units used in the system. Currency is used when dealing with any monetary values in Projects or Results. - -```mermaid -erDiagram - Currency { - string CurrencyID PK - string Code - string Name - string Symbol - datetime CreatedAt - datetime UpdatedAt - boolean IsActive - } -``` - -## Unit - -The Unit entity defines measurement units used in the system. Units are essential for standardizing data representation across mining projects, financial models, and results. - -```mermaid -erDiagram - Unit { - string UnitID PK - string Name - string Symbol - string Description - } -``` - -## Mining Technology - -The Mining Technology entity encompasses the various tools and techniques used in data mining projects. A Mining Project typically utilizes one specific Mining Technology. - -```mermaid -erDiagram - MiningTechnology { - string TechnologyID PK - string Name - string Description - } -``` - -## Product Model - -The Product Model entity captures the details of products extracted or processed in mining projects. It includes attributes such as ProductID, Name, Type and UnitID (linking to the Unit entity). A Product Model can have multiple Quality Metrics associated with it. - -```mermaid -erDiagram - ProductModel { - string ProductID PK - string Name - string Type - string UnitID FK - } - ProductModel ||--o{ QualityMetric : has - Unit ||--o{ ProductModel : used_in - Unit ||--o{ QualityMetric : used_in -``` - -### Quality Metrics - -Quality Metrics define the standards and specifications for products derived from mining projects. These metrics ensure that products meet industry and regulatory requirements. - -For example, Quality Metrics for a mineral product may include: - -- Purity Level -- Moisture Content -- Impurity Levels -- Grain Size Distribution -- Chemical Composition -- Physical Properties - -These metrics are essential for assessing the value and usability of the products in various applications and markets. - -```mermaid -erDiagram - QualityMetric { - string MetricID PK - string Name - string Description - float MinValue - float MaxValue - string UnitID FK - string AssociatedProductID FK - } - ProductModel ||--o{ QualityMetric : has - Unit ||--o{ QualityMetric : used_in -``` - -## Financial Model - -The Financial Model entity captures the economic aspects of mining projects. The Financial Model includes various sub-models and components that detail costs, revenues, investments, and risks associated with mining projects. Financial Models are the basis for Scenario analyses within Projects. - -```mermaid -erDiagram - FinancialModel { - string ModelID PK - string Name - string Description - string AssociatedProjectID FK - } -``` - -### Cost Model - -The Cost Model details the expenses associated with mining projects, including capital expenditures (CAPEX) and operational expenditures (OPEX). - -- CAPEX: Initial costs for equipment, infrastructure, and setup. -- OPEX: Ongoing costs for labor, maintenance, and operations. - -```mermaid -erDiagram - CostModel { - string ModelID PK - string Name - string Description - string AssociatedProjectID FK - } - CAPEX { - string CAPEXID PK - float EquipmentCost - float InfrastructureCost - float SetupCost - float ContingencyCost - float TotalCAPEX - string CurrencyID FK - } - OPEX { - string OPEXID PK - float LaborCost - float MaterialsCost - float EquipmentRentalCost - float UtilitiesCost - float MaintenanceCost - float TotalOPEX - string CurrencyID FK - } - FinancialModel ||--o{ CostModel : includes - CostModel ||--|| CAPEX : has - CostModel ||--|| OPEX : has - Currency ||--o{ CAPEX : used_in - Currency ||--o{ OPEX : used_in -``` - -#### CAPEX - -For a calculation of Capital Expenditures (CAPEX), the following attributes are included: - -- EquipmentCost -- InfrastructureCost -- SetupCost -- ContingencyCost -- TotalCAPEX -- CurrencyID (Foreign Key to Currency) - -#### OPEX - -For a calculation of Operational Expenditures (OPEX), the following attributes are included: - -- LaborCost -- MaterialsCost -- EquipmentRentalCost -- UtilitiesCost -- MaintenanceCost -- TotalOPEX -- CurrencyID (Foreign Key to Currency) - -### Revenue Model - -The Revenue Model outlines the income generated from mining projects. It includes various revenue streams and product sales. - -```mermaid -erDiagram - RevenueModel { - string ModelID PK - string Name - string Description - string AssociatedProjectID FK - } - FinancialModel ||--o{ RevenueModel : includes - RevenueStream { - string RevenueStreamID PK - string Name - string Description - float Amount - string CurrencyID FK - string AssociatedRevenueModelID FK - string Frequency - datetime StartDate - datetime EndDate - boolean IsRecurring - } - FinancialModel ||--o{ RevenueModel : includes - RevenueModel ||--o{ RevenueStream : has - Currency ||--o{ RevenueStream : used_in -``` - -#### Revenue Streams - -Revenue streams can include product sales, service fees, and other income sources related to the mining project. - -```mermaid -erDiagram - RevenueStream { - string RevenueStreamID PK - string Name - string Description - float Amount - string CurrencyID FK - string AssociatedRevenueModelID FK - string Frequency - datetime StartDate - datetime EndDate - boolean IsRecurring - } -``` - -#### Product Sales - -Mining product sales represent the primary revenue stream for most mining projects. Product sales revenue is calculated based on the quantity of product sold, price per unit, and any applicable adjustments for quality or other factors like penalties for impurities. Also see [Price Calculation](../../specifications/price_calculation.md) for more details. - -```mermaid -erDiagram - ProductSale { - string SaleID PK - string ProductID FK - float QuantitySold - float PricePerUnit - float TotalRevenue - string CurrencyID FK - datetime SaleDate - } - ProductModel ||--o{ ProductSale : has - Currency ||--o{ ProductSale : used_in -``` - -### Investment Model - -The Investment Model focuses on the funding and financial backing of mining projects. It includes details about investors, investment amounts, and dates. - -```mermaid -erDiagram - InvestmentModel { - string InvestmentID PK - string InvestorID FK - string ProjectID FK - float Amount - datetime InvestmentDate - } - FinancialModel ||--o{ InvestmentModel : includes -``` - -### Economic Model - -The Economic Model assesses the overall economic viability and impact of mining projects. Economic indicators such as Net Present Value (NPV), Internal Rate of Return (IRR), and Payback Period are calculated within this model. - -```mermaid -erDiagram - EconomicModel { - string ModelID PK - string Name - string Description - string AssociatedProjectID FK - } - FinancialModel ||--o{ EconomicModel : includes -``` - -#### Discounted Cash Flow Metrics - -CalMiner standardises the computation of NPV, IRR, and Payback Period through the shared helpers in `services/financial.py`. The detailed algorithms, assumptions (compounding frequency, period anchoring, residual handling), and regression coverage are documented in [Financial Metrics Specification](../../specifications/financial_metrics.md). Scenario evaluation services and downstream analytics should rely on these helpers to ensure consistency across API, UI, and reporting features. - -#### Monte Carlo Simulation - -Stochastic profitability analysis builds on the deterministic helpers by sampling cash-flow distributions defined per scenario. The configuration contracts, supported distributions, and result schema are described in [Monte Carlo Simulation Specification](../../specifications/monte_carlo_simulation.md). Scenario evaluation flows should call `services/simulation.py` to generate iteration summaries and percentile data for reporting and visualisation features. - -### Risk Model - -The Risk Model identifies and evaluates potential risks associated with mining projects. It includes risk factors, their probabilities, and potential impacts on project outcomes. - -```mermaid -erDiagram - RiskModel { - string ModelID PK - string Name - string Description - string AssociatedProjectID FK - } - FinancialModel ||--o{ RiskModel : includes -``` - -## Parameter - -Parameters are configurable inputs used within Financial Models to simulate different economic conditions and scenarios for mining projects. Each Parameter has a name, type, default value, and is linked to a specific Technology. - -```mermaid -erDiagram - Parameter { - string ParameterID PK - string Name - string Type - string DefaultValue - string Description - string AssociatedTechnologyID FK - } -``` - -## Scenario - -A Scenario represents a specific configuration or analysis case within a Project. Scenarios utilize various Financial Models and Parameters to simulate different outcomes for mining projects. - -```mermaid -erDiagram - Scenario { - string ScenarioID PK - string Name - string Description - string AssociatedProjectID FK - } -``` +This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization. + +## SQLAlchemy Models + +### User Management + +#### User + +Represents authenticated platform users with optional elevated privileges. + +**Table:** `users` + +| Attribute | Type | Description | +| ------------- | ------------ | ------------------------- | +| id | Integer (PK) | Primary key | +| email | String(255) | Unique email address | +| username | String(128) | Unique username | +| password_hash | String(255) | Argon2 hashed password | +| is_active | Boolean | Account activation status | +| is_superuser | Boolean | Superuser privileges | +| last_login_at | DateTime | Last login timestamp | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `role_assignments`: Many-to-many with Role via UserRole +- `roles`: Many-to-many with Role (viewonly) + +#### Role + +Role encapsulating a set of permissions. + +**Table:** `roles` + +| Attribute | Type | Description | +| ------------ | ------------ | --------------------- | +| id | Integer (PK) | Primary key | +| name | String(64) | Unique role name | +| display_name | String(128) | Human-readable name | +| description | Text | Role description | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `assignments`: One-to-many with UserRole +- `users`: Many-to-many with User (viewonly) + +#### UserRole + +Association between users and roles with assignment metadata. + +**Table:** `user_roles` + +| Attribute | Type | Description | +| ---------- | ----------------------- | -------------------- | +| user_id | Integer (FK → users.id) | User foreign key | +| role_id | Integer (FK → roles.id) | Role foreign key | +| granted_at | DateTime | Assignment timestamp | +| granted_by | Integer (FK → users.id) | Granting user | + +**Relationships:** + +- `user`: Many-to-one with User +- `role`: Many-to-one with Role +- `granted_by_user`: Many-to-one with User + +### Project Management + +#### Project + +Top-level mining project grouping multiple scenarios. + +**Table:** `projects` + +| Attribute | Type | Description | +| ------------------- | ---------------------------------- | --------------------- | +| id | Integer (PK) | Primary key | +| name | String(255) | Unique project name | +| location | String(255) | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | Text | Project description | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenarios`: One-to-many with Scenario +- `pricing_settings`: Many-to-one with PricingSettings + +#### Scenario + +A specific configuration of assumptions for a project. + +**Table:** `scenarios` + +| Attribute | Type | Description | +| ---------------- | -------------------------- | ------------------------- | +| id | Integer (PK) | Primary key | +| project_id | Integer (FK → projects.id) | Project foreign key | +| name | String(255) | Scenario name | +| description | Text | Scenario description | +| status | ScenarioStatus | Scenario lifecycle status | +| start_date | Date | Scenario start date | +| end_date | Date | Scenario end date | +| discount_rate | Numeric(5,2) | Discount rate percentage | +| currency | String(3) | ISO currency code | +| primary_resource | ResourceType | Primary resource type | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `project`: Many-to-one with Project +- `financial_inputs`: One-to-many with FinancialInput +- `simulation_parameters`: One-to-many with SimulationParameter + +#### FinancialInput + +Line-item financial assumption attached to a scenario. + +**Table:** `financial_inputs` + +| Attribute | Type | Description | +| -------------- | --------------------------- | -------------------------- | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | +| name | String(255) | Input name | +| category | FinancialCategory | Financial category | +| cost_bucket | CostBucket | Cost bucket classification | +| amount | Numeric(18,2) | Monetary amount | +| currency | String(3) | ISO currency code | +| effective_date | Date | Effective date | +| notes | Text | Additional notes | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario + +#### SimulationParameter + +Probability distribution settings for scenario simulations. + +**Table:** `simulation_parameters` + +| Attribute | Type | Description | +| ------------------ | --------------------------- | ------------------------ | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | +| name | String(255) | Parameter name | +| distribution | DistributionType | Distribution type | +| variable | StochasticVariable | Stochastic variable type | +| resource_type | ResourceType | Resource type | +| mean_value | Numeric(18,4) | Mean value | +| standard_deviation | Numeric(18,4) | Standard deviation | +| minimum_value | Numeric(18,4) | Minimum value | +| maximum_value | Numeric(18,4) | Maximum value | +| unit | String(32) | Unit of measurement | +| configuration | JSON | Additional configuration | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario + +### Pricing Configuration + +#### PricingSettings + +Persisted pricing defaults applied to scenario evaluations. + +**Table:** `pricing_settings` + +| Attribute | Type | Description | +| ------------------------ | ------------- | ----------------------------- | +| id | Integer (PK) | Primary key | +| name | String(128) | Unique settings name | +| slug | String(64) | Unique slug identifier | +| description | Text | Settings description | +| default_currency | String(3) | Default ISO currency code | +| default_payable_pct | Numeric(5,2) | Default payable percentage | +| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage | +| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent | +| metadata_payload | JSON | Additional metadata | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `metal_overrides`: One-to-many with PricingMetalSettings +- `impurity_overrides`: One-to-many with PricingImpuritySettings +- `projects`: One-to-many with Project + +#### PricingMetalSettings + +Contract-specific overrides for a particular metal. + +**Table:** `pricing_metal_settings` + +| Attribute | Type | Description | +| ------------------------ | ---------------------------------- | ---------------------------- | +| id | Integer (PK) | Primary key | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | +| metal_code | String(32) | Metal code | +| payable_pct | Numeric(5,2) | Payable percentage | +| moisture_threshold_pct | Numeric(5,2) | Moisture threshold | +| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty | +| data | JSON | Additional data | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `pricing_settings`: Many-to-one with PricingSettings + +#### PricingImpuritySettings + +Impurity penalty thresholds associated with pricing settings. + +**Table:** `pricing_impurity_settings` + +| Attribute | Type | Description | +| ------------------- | ---------------------------------- | ---------------------------- | +| id | Integer (PK) | Primary key | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | +| impurity_code | String(32) | Impurity code | +| threshold_ppm | Numeric(14,4) | Threshold in ppm | +| penalty_per_ppm | Numeric(14,4) | Penalty per ppm | +| notes | Text | Additional notes | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `pricing_settings`: Many-to-one with PricingSettings + +### Monitoring and Auditing + +#### PerformanceMetric + +Performance metrics for monitoring system health. + +**Table:** `performance_metrics` + +| Attribute | Type | Description | +| ---------------- | ------------ | --------------------- | +| id | Integer (PK) | Primary key | +| timestamp | DateTime | Metric timestamp | +| metric_name | String | Metric name | +| value | Float | Metric value | +| labels | String | JSON string of labels | +| endpoint | String | API endpoint | +| method | String | HTTP method | +| status_code | Integer | HTTP status code | +| duration_seconds | Float | Request duration | + +#### ImportExportLog + +Audit log for import and export operations. + +**Table:** `import_export_logs` + +| Attribute | Type | Description | +| ---------- | ----------------------- | ------------------------------------- | +| id | Integer (PK) | Primary key | +| action | String(32) | Action type (preview, commit, export) | +| dataset | String(32) | Dataset type (projects, scenarios) | +| status | String(16) | Operation status | +| filename | String(255) | File name | +| row_count | Integer | Number of rows | +| detail | Text | Additional details | +| user_id | Integer (FK → users.id) | User foreign key | +| created_at | DateTime | Creation timestamp | + +## Enumerations + +### MiningOperationType + +Supported mining operation categories. + +- `OPEN_PIT`: Open pit mining +- `UNDERGROUND`: Underground mining +- `IN_SITU_LEACH`: In-situ leaching +- `PLACER`: Placer mining +- `QUARRY`: Quarry operations +- `MOUNTAINTOP_REMOVAL`: Mountaintop removal +- `OTHER`: Other operations + +### ScenarioStatus + +Lifecycle states for project scenarios. + +- `DRAFT`: Draft status +- `ACTIVE`: Active status +- `ARCHIVED`: Archived status + +### FinancialCategory + +Enumeration of cost and revenue classifications. + +- `CAPITAL_EXPENDITURE`: Capital expenditures +- `OPERATING_EXPENDITURE`: Operating expenditures +- `REVENUE`: Revenue +- `CONTINGENCY`: Contingency +- `OTHER`: Other + +### DistributionType + +Supported stochastic distribution families for simulations. + +- `NORMAL`: Normal distribution +- `TRIANGULAR`: Triangular distribution +- `UNIFORM`: Uniform distribution +- `LOGNORMAL`: Lognormal distribution +- `CUSTOM`: Custom distribution + +### ResourceType + +Primary consumables and resources used in mining operations. + +- `DIESEL`: Diesel fuel +- `ELECTRICITY`: Electrical power +- `WATER`: Process water +- `EXPLOSIVES`: Blasting agents +- `REAGENTS`: Processing reagents +- `LABOR`: Direct labor +- `EQUIPMENT_HOURS`: Equipment operating hours +- `TAILINGS_CAPACITY`: Tailings storage + +### CostBucket + +Granular cost buckets aligned with project accounting. + +- `CAPITAL_INITIAL`: Initial capital +- `CAPITAL_SUSTAINING`: Sustaining capital +- `OPERATING_FIXED`: Fixed operating costs +- `OPERATING_VARIABLE`: Variable operating costs +- `MAINTENANCE`: Maintenance costs +- `RECLAMATION`: Reclamation costs +- `ROYALTIES`: Royalties +- `GENERAL_ADMIN`: General and administrative + +### StochasticVariable + +Domain variables that typically require probabilistic modelling. + +- `ORE_GRADE`: Ore grade variability +- `RECOVERY_RATE`: Metallurgical recovery +- `METAL_PRICE`: Commodity price +- `OPERATING_COST`: Operating cost per tonne +- `CAPITAL_COST`: Capital cost +- `DISCOUNT_RATE`: Discount rate +- `THROUGHPUT`: Plant throughput + +## Pydantic Schemas + +### Authentication Schemas (`schemas/auth.py`) + +#### RegistrationForm + +Form model for user registration. + +| Field | Type | Validation | +| ---------------- | ---- | ------------------------------- | +| username | str | 3-128 characters | +| email | str | Valid email format, 5-255 chars | +| password | str | 8-256 characters | +| confirm_password | str | Must match password | + +#### LoginForm + +Form model for user login. + +| Field | Type | Validation | +| -------- | ---- | ---------------- | +| username | str | 1-255 characters | +| password | str | 1-256 characters | + +#### PasswordResetRequestForm + +Form model for password reset request. + +| Field | Type | Validation | +| ----- | ---- | ------------------------------- | +| email | str | Valid email format, 5-255 chars | + +#### PasswordResetForm + +Form model for password reset. + +| Field | Type | Validation | +| ---------------- | ---- | ------------------- | +| token | str | Required | +| password | str | 8-256 characters | +| confirm_password | str | Must match password | + +### Project Schemas (`schemas/project.py`) + +#### ProjectBase + +Base schema for project data. + +| Field | Type | Description | +| -------------- | ------------------- | --------------------- | +| name | str | Project name | +| location | str \| None | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | str \| None | Project description | + +#### ProjectCreate + +Schema for creating projects (inherits ProjectBase). + +#### ProjectUpdate + +Schema for updating projects. + +| Field | Type | Description | +| -------------- | --------------------------- | --------------------- | +| name | str \| None | Project name | +| location | str \| None | Project location | +| operation_type | MiningOperationType \| None | Mining operation type | +| description | str \| None | Project description | + +#### ProjectRead + +Schema for reading projects (inherits ProjectBase). + +| Field | Type | Description | +| ---------- | -------- | ------------------ | +| id | int | Project ID | +| created_at | datetime | Creation timestamp | +| updated_at | datetime | Update timestamp | + +### Scenario Schemas (`schemas/scenario.py`) + +#### ScenarioBase + +Base schema for scenario data. + +| Field | Type | Description | +| ---------------- | -------------------- | -------------------------------- | +| name | str | Scenario name | +| description | str \| None | Scenario description | +| status | ScenarioStatus | Scenario status (default: DRAFT) | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | + +#### ScenarioCreate + +Schema for creating scenarios (inherits ScenarioBase). + +#### ScenarioUpdate + +Schema for updating scenarios. + +| Field | Type | Description | +| ---------------- | ---------------------- | -------------------- | +| name | str \| None | Scenario name | +| description | str \| None | Scenario description | +| status | ScenarioStatus \| None | Scenario status | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | + +#### ScenarioRead + +Schema for reading scenarios (inherits ScenarioBase). + +| Field | Type | Description | +| ---------- | -------- | ------------------ | +| id | int | Scenario ID | +| project_id | int | Project ID | +| created_at | datetime | Creation timestamp | +| updated_at | datetime | Update timestamp | + +#### ScenarioComparisonRequest + +Schema for scenario comparison requests. + +| Field | Type | Description | +| ------------ | --------- | --------------------------------------- | +| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) | + +#### ScenarioComparisonResponse + +Schema for scenario comparison responses. + +| Field | Type | Description | +| ---------- | ------------------ | --------------------- | +| project_id | int | Project ID | +| scenarios | list[ScenarioRead] | List of scenario data | + +### Import Schemas (`schemas/imports.py`) + +#### ProjectImportRow + +Schema for importing project data. + +| Field | Type | Description | +| -------------- | ------------------- | ----------------------- | +| name | str | Project name (required) | +| location | str \| None | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | str \| None | Project description | +| created_at | datetime \| None | Creation timestamp | +| updated_at | datetime \| None | Update timestamp | + +#### ScenarioImportRow + +Schema for importing scenario data. + +| Field | Type | Description | +| ---------------- | -------------------- | -------------------------------- | +| project_name | str | Project name (required) | +| name | str | Scenario name (required) | +| status | ScenarioStatus | Scenario status (default: DRAFT) | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | +| description | str \| None | Scenario description | +| created_at | datetime \| None | Creation timestamp | +| updated_at | datetime \| None | Update timestamp | + +### Export Schemas (`schemas/exports.py`) + +#### ExportFormat + +Enumeration for export formats. + +- `CSV`: CSV format +- `XLSX`: Excel format + +#### BaseExportRequest + +Base schema for export requests. + +| Field | Type | Description | +| ---------------- | ------------ | --------------------------------- | +| format | ExportFormat | Export format (default: CSV) | +| include_metadata | bool | Include metadata (default: False) | + +#### ProjectExportRequest + +Schema for project export requests (inherits BaseExportRequest). + +| Field | Type | Description | +| ------- | ---------------------------- | -------------- | +| filters | ProjectExportFilters \| None | Export filters | + +#### ScenarioExportRequest + +Schema for scenario export requests (inherits BaseExportRequest). + +| Field | Type | Description | +| ------- | ----------------------------- | -------------- | +| filters | ScenarioExportFilters \| None | Export filters | + +#### ExportTicket + +Schema for export tickets. + +| Field | Type | Description | +| -------- | -------------------------------- | ------------- | +| token | str | Export token | +| format | ExportFormat | Export format | +| resource | Literal["projects", "scenarios"] | Resource type | + +#### ExportResponse + +Schema for export responses. + +| Field | Type | Description | +| ------ | ------------ | ------------- | +| ticket | ExportTicket | Export ticket | + +## Discrepancies Between Data Models and Pydantic Schemas + +### Missing Pydantic Schemas + +The following SQLAlchemy models do not have corresponding Pydantic schemas: + +- `User` - Authentication handled through forms, not API schemas +- `Role` - Role management not exposed via API +- `UserRole` - User-role associations not exposed via API +- `FinancialInput` - Financial inputs managed through scenario context +- `SimulationParameter` - Simulation parameters managed through scenario context +- `PricingSettings` - Pricing settings managed through project context +- `PricingMetalSettings` - Pricing overrides managed through settings context +- `PricingImpuritySettings` - Pricing overrides managed through settings context +- `PerformanceMetric` - Metrics not exposed via API +- `ImportExportLog` - Audit logs not exposed via API + +### Schema Differences + +- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER` +- **Scenario schemas** include currency normalization validation not present in the model validator +- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing +- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document + +### Additional Validation + +Pydantic schemas include additional validation beyond SQLAlchemy model constraints: + +- Email format validation in auth forms +- Password confirmation matching +- Currency code normalization and validation +- Date range validation (end_date >= start_date) +- Required field validation for imports +- Enum value coercion with aliases for imports + +This documentation reflects the current implementation as of the latest development cycle.