# Data Model

This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization.

## SQLAlchemy Models

### User Management

#### User

Represents authenticated platform users with optional elevated privileges.

**Table:** `users`

| Attribute     | Type         | Description               |
| ------------- | ------------ | ------------------------- |
| id            | Integer (PK) | Primary key               |
| email         | String(255)  | Unique email address      |
| username      | String(128)  | Unique username           |
| password_hash | String(255)  | Argon2 hashed password    |
| is_active     | Boolean      | Account activation status |
| is_superuser  | Boolean      | Superuser privileges      |
| last_login_at | DateTime     | Last login timestamp      |
| created_at    | DateTime     | Creation timestamp        |
| updated_at    | DateTime     | Last update timestamp     |

**Relationships:**

- `role_assignments`: Many-to-many with Role via UserRole
- `roles`: Many-to-many with Role (viewonly)

#### Role

Role encapsulating a set of permissions.

**Table:** `roles`

| Attribute    | Type         | Description           |
| ------------ | ------------ | --------------------- |
| id           | Integer (PK) | Primary key           |
| name         | String(64)   | Unique role name      |
| display_name | String(128)  | Human-readable name   |
| description  | Text         | Role description      |
| created_at   | DateTime     | Creation timestamp    |
| updated_at   | DateTime     | Last update timestamp |

**Relationships:**

- `assignments`: One-to-many with UserRole
- `users`: Many-to-many with User (viewonly)

#### UserRole

Association between users and roles with assignment metadata.

**Table:** `user_roles`

| Attribute  | Type                    | Description          |
| ---------- | ----------------------- | -------------------- |
| user_id    | Integer (FK → users.id) | User foreign key     |
| role_id    | Integer (FK → roles.id) | Role foreign key     |
| granted_at | DateTime                | Assignment timestamp |
| granted_by | Integer (FK → users.id) | Granting user        |

**Relationships:**

- `user`: Many-to-one with User
- `role`: Many-to-one with Role
- `granted_by_user`: Many-to-one with User

### Project Management

#### Project

Top-level mining project grouping multiple scenarios.

**Table:** `projects`

| Attribute           | Type                               | Description           |
| ------------------- | ---------------------------------- | --------------------- |
| id                  | Integer (PK)                       | Primary key           |
| name                | String(255)                        | Unique project name   |
| location            | String(255)                        | Project location      |
| operation_type      | MiningOperationType                | Mining operation type |
| description         | Text                               | Project description   |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings      |
| created_at          | DateTime                           | Creation timestamp    |
| updated_at          | DateTime                           | Last update timestamp |

**Relationships:**

- `scenarios`: One-to-many with Scenario
- `pricing_settings`: Many-to-one with PricingSettings

#### Scenario

A specific configuration of assumptions for a project.

**Table:** `scenarios`

| Attribute        | Type                       | Description               |
| ---------------- | -------------------------- | ------------------------- |
| id               | Integer (PK)               | Primary key               |
| project_id       | Integer (FK → projects.id) | Project foreign key       |
| name             | String(255)                | Scenario name             |
| description      | Text                       | Scenario description      |
| status           | ScenarioStatus             | Scenario lifecycle status |
| start_date       | Date                       | Scenario start date       |
| end_date         | Date                       | Scenario end date         |
| discount_rate    | Numeric(5,2)               | Discount rate percentage  |
| currency         | String(3)                  | ISO currency code         |
| primary_resource | ResourceType               | Primary resource type     |
| created_at       | DateTime                   | Creation timestamp        |
| updated_at       | DateTime                   | Last update timestamp     |

**Relationships:**

- `project`: Many-to-one with Project
- `financial_inputs`: One-to-many with FinancialInput
- `simulation_parameters`: One-to-many with SimulationParameter

#### FinancialInput

Line-item financial assumption attached to a scenario.

**Table:** `financial_inputs`

| Attribute      | Type                        | Description                |
| -------------- | --------------------------- | -------------------------- |
| id             | Integer (PK)                | Primary key                |
| scenario_id    | Integer (FK → scenarios.id) | Scenario foreign key       |
| name           | String(255)                 | Input name                 |
| category       | FinancialCategory           | Financial category         |
| cost_bucket    | CostBucket                  | Cost bucket classification |
| amount         | Numeric(18,2)               | Monetary amount            |
| currency       | String(3)                   | ISO currency code          |
| effective_date | Date                        | Effective date             |
| notes          | Text                        | Additional notes           |
| created_at     | DateTime                    | Creation timestamp         |
| updated_at     | DateTime                    | Last update timestamp      |

**Relationships:**

- `scenario`: Many-to-one with Scenario

#### SimulationParameter

Probability distribution settings for scenario simulations.

**Table:** `simulation_parameters`

| Attribute          | Type                        | Description              |
| ------------------ | --------------------------- | ------------------------ |
| id                 | Integer (PK)                | Primary key              |
| scenario_id        | Integer (FK → scenarios.id) | Scenario foreign key     |
| name               | String(255)                 | Parameter name           |
| distribution       | DistributionType            | Distribution type        |
| variable           | StochasticVariable          | Stochastic variable type |
| resource_type      | ResourceType                | Resource type            |
| mean_value         | Numeric(18,4)               | Mean value               |
| standard_deviation | Numeric(18,4)               | Standard deviation       |
| minimum_value      | Numeric(18,4)               | Minimum value            |
| maximum_value      | Numeric(18,4)               | Maximum value            |
| unit               | String(32)                  | Unit of measurement      |
| configuration      | JSON                        | Additional configuration |
| created_at         | DateTime                    | Creation timestamp       |
| updated_at         | DateTime                    | Last update timestamp    |

**Relationships:**

- `scenario`: Many-to-one with Scenario

### Pricing Configuration

#### PricingSettings

Persisted pricing defaults applied to scenario evaluations.

**Table:** `pricing_settings`

| Attribute                | Type          | Description                   |
| ------------------------ | ------------- | ----------------------------- |
| id                       | Integer (PK)  | Primary key                   |
| name                     | String(128)   | Unique settings name          |
| slug                     | String(64)    | Unique slug identifier        |
| description              | Text          | Settings description          |
| default_currency         | String(3)     | Default ISO currency code     |
| default_payable_pct      | Numeric(5,2)  | Default payable percentage    |
| moisture_threshold_pct   | Numeric(5,2)  | Moisture threshold percentage |
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent  |
| metadata_payload         | JSON          | Additional metadata           |
| created_at               | DateTime      | Creation timestamp            |
| updated_at               | DateTime      | Last update timestamp         |

**Relationships:**

- `metal_overrides`: One-to-many with PricingMetalSettings
- `impurity_overrides`: One-to-many with PricingImpuritySettings
- `projects`: One-to-many with Project

#### PricingMetalSettings

Contract-specific overrides for a particular metal.

**Table:** `pricing_metal_settings`

| Attribute                | Type                               | Description                  |
| ------------------------ | ---------------------------------- | ---------------------------- |
| id                       | Integer (PK)                       | Primary key                  |
| pricing_settings_id      | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| metal_code               | String(32)                         | Metal code                   |
| payable_pct              | Numeric(5,2)                       | Payable percentage           |
| moisture_threshold_pct   | Numeric(5,2)                       | Moisture threshold           |
| moisture_penalty_per_pct | Numeric(14,4)                      | Moisture penalty             |
| data                     | JSON                               | Additional data              |
| created_at               | DateTime                           | Creation timestamp           |
| updated_at               | DateTime                           | Last update timestamp        |

**Relationships:**

- `pricing_settings`: Many-to-one with PricingSettings

#### PricingImpuritySettings

Impurity penalty thresholds associated with pricing settings.

**Table:** `pricing_impurity_settings`

| Attribute           | Type                               | Description                  |
| ------------------- | ---------------------------------- | ---------------------------- |
| id                  | Integer (PK)                       | Primary key                  |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| impurity_code       | String(32)                         | Impurity code                |
| threshold_ppm       | Numeric(14,4)                      | Threshold in ppm             |
| penalty_per_ppm     | Numeric(14,4)                      | Penalty per ppm              |
| notes               | Text                               | Additional notes             |
| created_at          | DateTime                           | Creation timestamp           |
| updated_at          | DateTime                           | Last update timestamp        |

**Relationships:**

- `pricing_settings`: Many-to-one with PricingSettings

### Monitoring and Auditing

#### PerformanceMetric

Performance metrics for monitoring system health.

**Table:** `performance_metrics`

| Attribute        | Type         | Description           |
| ---------------- | ------------ | --------------------- |
| id               | Integer (PK) | Primary key           |
| timestamp        | DateTime     | Metric timestamp      |
| metric_name      | String       | Metric name           |
| value            | Float        | Metric value          |
| labels           | String       | JSON string of labels |
| endpoint         | String       | API endpoint          |
| method           | String       | HTTP method           |
| status_code      | Integer      | HTTP status code      |
| duration_seconds | Float        | Request duration      |

#### ImportExportLog

Audit log for import and export operations.

**Table:** `import_export_logs`

| Attribute  | Type                    | Description                           |
| ---------- | ----------------------- | ------------------------------------- |
| id         | Integer (PK)            | Primary key                           |
| action     | String(32)              | Action type (preview, commit, export) |
| dataset    | String(32)              | Dataset type (projects, scenarios)    |
| status     | String(16)              | Operation status                      |
| filename   | String(255)             | File name                             |
| row_count  | Integer                 | Number of rows                        |
| detail     | Text                    | Additional details                    |
| user_id    | Integer (FK → users.id) | User foreign key                      |
| created_at | DateTime                | Creation timestamp                    |

## Enumerations

### MiningOperationType

Supported mining operation categories.

- `OPEN_PIT`: Open pit mining
- `UNDERGROUND`: Underground mining
- `IN_SITU_LEACH`: In-situ leaching
- `PLACER`: Placer mining
- `QUARRY`: Quarry operations
- `MOUNTAINTOP_REMOVAL`: Mountaintop removal
- `OTHER`: Other operations

### ScenarioStatus

Lifecycle states for project scenarios.

- `DRAFT`: Draft status
- `ACTIVE`: Active status
- `ARCHIVED`: Archived status

### FinancialCategory

Enumeration of cost and revenue classifications.

- `CAPITAL_EXPENDITURE`: Capital expenditures
- `OPERATING_EXPENDITURE`: Operating expenditures
- `REVENUE`: Revenue
- `CONTINGENCY`: Contingency
- `OTHER`: Other

### DistributionType

Supported stochastic distribution families for simulations.

- `NORMAL`: Normal distribution
- `TRIANGULAR`: Triangular distribution
- `UNIFORM`: Uniform distribution
- `LOGNORMAL`: Lognormal distribution
- `CUSTOM`: Custom distribution

### ResourceType

Primary consumables and resources used in mining operations.

- `DIESEL`: Diesel fuel
- `ELECTRICITY`: Electrical power
- `WATER`: Process water
- `EXPLOSIVES`: Blasting agents
- `REAGENTS`: Processing reagents
- `LABOR`: Direct labor
- `EQUIPMENT_HOURS`: Equipment operating hours
- `TAILINGS_CAPACITY`: Tailings storage

### CostBucket

Granular cost buckets aligned with project accounting.

- `CAPITAL_INITIAL`: Initial capital
- `CAPITAL_SUSTAINING`: Sustaining capital
- `OPERATING_FIXED`: Fixed operating costs
- `OPERATING_VARIABLE`: Variable operating costs
- `MAINTENANCE`: Maintenance costs
- `RECLAMATION`: Reclamation costs
- `ROYALTIES`: Royalties
- `GENERAL_ADMIN`: General and administrative

### StochasticVariable

Domain variables that typically require probabilistic modelling.

- `ORE_GRADE`: Ore grade variability
- `RECOVERY_RATE`: Metallurgical recovery
- `METAL_PRICE`: Commodity price
- `OPERATING_COST`: Operating cost per tonne
- `CAPITAL_COST`: Capital cost
- `DISCOUNT_RATE`: Discount rate
- `THROUGHPUT`: Plant throughput

## Pydantic Schemas

### Authentication Schemas (`schemas/auth.py`)

#### RegistrationForm

Form model for user registration.

| Field            | Type | Validation                      |
| ---------------- | ---- | ------------------------------- |
| username         | str  | 3-128 characters                |
| email            | str  | Valid email format, 5-255 chars |
| password         | str  | 8-256 characters                |
| confirm_password | str  | Must match password             |

#### LoginForm

Form model for user login.

| Field    | Type | Validation       |
| -------- | ---- | ---------------- |
| username | str  | 1-255 characters |
| password | str  | 1-256 characters |

#### PasswordResetRequestForm

Form model for password reset request.

| Field | Type | Validation                      |
| ----- | ---- | ------------------------------- |
| email | str  | Valid email format, 5-255 chars |

#### PasswordResetForm

Form model for password reset.

| Field            | Type | Validation          |
| ---------------- | ---- | ------------------- |
| token            | str  | Required            |
| password         | str  | 8-256 characters    |
| confirm_password | str  | Must match password |

### Project Schemas (`schemas/project.py`)

#### ProjectBase

Base schema for project data.

| Field          | Type                | Description           |
| -------------- | ------------------- | --------------------- |
| name           | str                 | Project name          |
| location       | str \| None         | Project location      |
| operation_type | MiningOperationType | Mining operation type |
| description    | str \| None         | Project description   |

#### ProjectCreate

Schema for creating projects (inherits ProjectBase).

#### ProjectUpdate

Schema for updating projects.

| Field          | Type                        | Description           |
| -------------- | --------------------------- | --------------------- |
| name           | str \| None                 | Project name          |
| location       | str \| None                 | Project location      |
| operation_type | MiningOperationType \| None | Mining operation type |
| description    | str \| None                 | Project description   |

#### ProjectRead

Schema for reading projects (inherits ProjectBase).

| Field      | Type     | Description        |
| ---------- | -------- | ------------------ |
| id         | int      | Project ID         |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp   |

### Scenario Schemas (`schemas/scenario.py`)

#### ScenarioBase

Base schema for scenario data.

| Field            | Type                 | Description                      |
| ---------------- | -------------------- | -------------------------------- |
| name             | str                  | Scenario name                    |
| description      | str \| None          | Scenario description             |
| status           | ScenarioStatus       | Scenario status (default: DRAFT) |
| start_date       | date \| None         | Start date                       |
| end_date         | date \| None         | End date                         |
| discount_rate    | float \| None        | Discount rate                    |
| currency         | str \| None          | ISO currency code                |
| primary_resource | ResourceType \| None | Primary resource                 |

#### ScenarioCreate

Schema for creating scenarios (inherits ScenarioBase).

#### ScenarioUpdate

Schema for updating scenarios.

| Field            | Type                   | Description          |
| ---------------- | ---------------------- | -------------------- |
| name             | str \| None            | Scenario name        |
| description      | str \| None            | Scenario description |
| status           | ScenarioStatus \| None | Scenario status      |
| start_date       | date \| None           | Start date           |
| end_date         | date \| None           | End date             |
| discount_rate    | float \| None          | Discount rate        |
| currency         | str \| None            | ISO currency code    |
| primary_resource | ResourceType \| None   | Primary resource     |

#### ScenarioRead

Schema for reading scenarios (inherits ScenarioBase).

| Field      | Type     | Description        |
| ---------- | -------- | ------------------ |
| id         | int      | Scenario ID        |
| project_id | int      | Project ID         |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp   |

#### ScenarioComparisonRequest

Schema for scenario comparison requests.

| Field        | Type      | Description                             |
| ------------ | --------- | --------------------------------------- |
| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) |

#### ScenarioComparisonResponse

Schema for scenario comparison responses.

| Field      | Type               | Description           |
| ---------- | ------------------ | --------------------- |
| project_id | int                | Project ID            |
| scenarios  | list[ScenarioRead] | List of scenario data |

### Import Schemas (`schemas/imports.py`)

#### ProjectImportRow

Schema for importing project data.

| Field          | Type                | Description             |
| -------------- | ------------------- | ----------------------- |
| name           | str                 | Project name (required) |
| location       | str \| None         | Project location        |
| operation_type | MiningOperationType | Mining operation type   |
| description    | str \| None         | Project description     |
| created_at     | datetime \| None    | Creation timestamp      |
| updated_at     | datetime \| None    | Update timestamp        |

#### ScenarioImportRow

Schema for importing scenario data.

| Field            | Type                 | Description                      |
| ---------------- | -------------------- | -------------------------------- |
| project_name     | str                  | Project name (required)          |
| name             | str                  | Scenario name (required)         |
| status           | ScenarioStatus       | Scenario status (default: DRAFT) |
| start_date       | date \| None         | Start date                       |
| end_date         | date \| None         | End date                         |
| discount_rate    | float \| None        | Discount rate                    |
| currency         | str \| None          | ISO currency code                |
| primary_resource | ResourceType \| None | Primary resource                 |
| description      | str \| None          | Scenario description             |
| created_at       | datetime \| None     | Creation timestamp               |
| updated_at       | datetime \| None     | Update timestamp                 |

### Export Schemas (`schemas/exports.py`)

#### ExportFormat

Enumeration for export formats.

- `CSV`: CSV format
- `XLSX`: Excel format

#### BaseExportRequest

Base schema for export requests.

| Field            | Type         | Description                       |
| ---------------- | ------------ | --------------------------------- |
| format           | ExportFormat | Export format (default: CSV)      |
| include_metadata | bool         | Include metadata (default: False) |

#### ProjectExportRequest

Schema for project export requests (inherits BaseExportRequest).

| Field   | Type                         | Description    |
| ------- | ---------------------------- | -------------- |
| filters | ProjectExportFilters \| None | Export filters |

#### ScenarioExportRequest

Schema for scenario export requests (inherits BaseExportRequest).

| Field   | Type                          | Description    |
| ------- | ----------------------------- | -------------- |
| filters | ScenarioExportFilters \| None | Export filters |

#### ExportTicket

Schema for export tickets.

| Field    | Type                             | Description   |
| -------- | -------------------------------- | ------------- |
| token    | str                              | Export token  |
| format   | ExportFormat                     | Export format |
| resource | Literal["projects", "scenarios"] | Resource type |

#### ExportResponse

Schema for export responses.

| Field  | Type         | Description   |
| ------ | ------------ | ------------- |
| ticket | ExportTicket | Export ticket |

## Discrepancies Between Data Models and Pydantic Schemas

### Missing Pydantic Schemas

The following SQLAlchemy models do not have corresponding Pydantic schemas:

- `User` - Authentication handled through forms, not API schemas
- `Role` - Role management not exposed via API
- `UserRole` - User-role associations not exposed via API
- `FinancialInput` - Financial inputs managed through scenario context
- `SimulationParameter` - Simulation parameters managed through scenario context
- `PricingSettings` - Pricing settings managed through project context
- `PricingMetalSettings` - Pricing overrides managed through settings context
- `PricingImpuritySettings` - Pricing overrides managed through settings context
- `PerformanceMetric` - Metrics not exposed via API
- `ImportExportLog` - Audit logs not exposed via API

### Schema Differences

- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER`
- **Scenario schemas** include currency normalization validation not present in the model validator
- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing
- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document

### Additional Validation

Pydantic schemas include additional validation beyond SQLAlchemy model constraints:

- Email format validation in auth forms
- Password confirmation matching
- Currency code normalization and validation
- Date range validation (end_date >= start_date)
- Required field validation for imports
- Enum value coercion with aliases for imports

This documentation reflects the current implementation as of the latest development cycle.