627 lines
25 KiB
Markdown
627 lines
25 KiB
Markdown
# Data Model
|
|
|
|
This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization.
|
|
|
|
## SQLAlchemy Models
|
|
|
|
### User Management
|
|
|
|
#### User
|
|
|
|
Represents authenticated platform users with optional elevated privileges.
|
|
|
|
**Table:** `users`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------- | ------------ | ------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| email | String(255) | Unique email address |
|
|
| username | String(128) | Unique username |
|
|
| password_hash | String(255) | Argon2 hashed password |
|
|
| is_active | Boolean | Account activation status |
|
|
| is_superuser | Boolean | Superuser privileges |
|
|
| last_login_at | DateTime | Last login timestamp |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `role_assignments`: Many-to-many with Role via UserRole
|
|
- `roles`: Many-to-many with Role (viewonly)
|
|
|
|
#### Role
|
|
|
|
Role encapsulating a set of permissions.
|
|
|
|
**Table:** `roles`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------ | ------------ | --------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| name | String(64) | Unique role name |
|
|
| display_name | String(128) | Human-readable name |
|
|
| description | Text | Role description |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `assignments`: One-to-many with UserRole
|
|
- `users`: Many-to-many with User (viewonly)
|
|
|
|
#### UserRole
|
|
|
|
Association between users and roles with assignment metadata.
|
|
|
|
**Table:** `user_roles`
|
|
|
|
| Attribute | Type | Description |
|
|
| ---------- | ----------------------- | -------------------- |
|
|
| user_id | Integer (FK → users.id) | User foreign key |
|
|
| role_id | Integer (FK → roles.id) | Role foreign key |
|
|
| granted_at | DateTime | Assignment timestamp |
|
|
| granted_by | Integer (FK → users.id) | Granting user |
|
|
|
|
**Relationships:**
|
|
|
|
- `user`: Many-to-one with User
|
|
- `role`: Many-to-one with Role
|
|
- `granted_by_user`: Many-to-one with User
|
|
|
|
### Project Management
|
|
|
|
#### Project
|
|
|
|
Top-level mining project grouping multiple scenarios.
|
|
|
|
**Table:** `projects`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------------- | ---------------------------------- | --------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| name | String(255) | Unique project name |
|
|
| location | String(255) | Project location |
|
|
| operation_type | MiningOperationType | Mining operation type |
|
|
| description | Text | Project description |
|
|
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `scenarios`: One-to-many with Scenario
|
|
- `pricing_settings`: Many-to-one with PricingSettings
|
|
|
|
#### Scenario
|
|
|
|
A specific configuration of assumptions for a project.
|
|
|
|
**Table:** `scenarios`
|
|
|
|
| Attribute | Type | Description |
|
|
| ---------------- | -------------------------- | ------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| project_id | Integer (FK → projects.id) | Project foreign key |
|
|
| name | String(255) | Scenario name |
|
|
| description | Text | Scenario description |
|
|
| status | ScenarioStatus | Scenario lifecycle status |
|
|
| start_date | Date | Scenario start date |
|
|
| end_date | Date | Scenario end date |
|
|
| discount_rate | Numeric(5,2) | Discount rate percentage |
|
|
| currency | String(3) | ISO currency code |
|
|
| primary_resource | ResourceType | Primary resource type |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `project`: Many-to-one with Project
|
|
- `financial_inputs`: One-to-many with FinancialInput
|
|
- `simulation_parameters`: One-to-many with SimulationParameter
|
|
|
|
#### FinancialInput
|
|
|
|
Line-item financial assumption attached to a scenario.
|
|
|
|
**Table:** `financial_inputs`
|
|
|
|
| Attribute | Type | Description |
|
|
| -------------- | --------------------------- | -------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
|
|
| name | String(255) | Input name |
|
|
| category | FinancialCategory | Financial category |
|
|
| cost_bucket | CostBucket | Cost bucket classification |
|
|
| amount | Numeric(18,2) | Monetary amount |
|
|
| currency | String(3) | ISO currency code |
|
|
| effective_date | Date | Effective date |
|
|
| notes | Text | Additional notes |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `scenario`: Many-to-one with Scenario
|
|
|
|
#### SimulationParameter
|
|
|
|
Probability distribution settings for scenario simulations.
|
|
|
|
**Table:** `simulation_parameters`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------------ | --------------------------- | ------------------------ |
|
|
| id | Integer (PK) | Primary key |
|
|
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
|
|
| name | String(255) | Parameter name |
|
|
| distribution | DistributionType | Distribution type |
|
|
| variable | StochasticVariable | Stochastic variable type |
|
|
| resource_type | ResourceType | Resource type |
|
|
| mean_value | Numeric(18,4) | Mean value |
|
|
| standard_deviation | Numeric(18,4) | Standard deviation |
|
|
| minimum_value | Numeric(18,4) | Minimum value |
|
|
| maximum_value | Numeric(18,4) | Maximum value |
|
|
| unit | String(32) | Unit of measurement |
|
|
| configuration | JSON | Additional configuration |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `scenario`: Many-to-one with Scenario
|
|
|
|
### Pricing Configuration
|
|
|
|
#### PricingSettings
|
|
|
|
Persisted pricing defaults applied to scenario evaluations.
|
|
|
|
**Table:** `pricing_settings`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------------------ | ------------- | ----------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| name | String(128) | Unique settings name |
|
|
| slug | String(64) | Unique slug identifier |
|
|
| description | Text | Settings description |
|
|
| default_currency | String(3) | Default ISO currency code |
|
|
| default_payable_pct | Numeric(5,2) | Default payable percentage |
|
|
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage |
|
|
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent |
|
|
| metadata_payload | JSON | Additional metadata |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `metal_overrides`: One-to-many with PricingMetalSettings
|
|
- `impurity_overrides`: One-to-many with PricingImpuritySettings
|
|
- `projects`: One-to-many with Project
|
|
|
|
#### PricingMetalSettings
|
|
|
|
Contract-specific overrides for a particular metal.
|
|
|
|
**Table:** `pricing_metal_settings`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------------------ | ---------------------------------- | ---------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
|
|
| metal_code | String(32) | Metal code |
|
|
| payable_pct | Numeric(5,2) | Payable percentage |
|
|
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold |
|
|
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty |
|
|
| data | JSON | Additional data |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `pricing_settings`: Many-to-one with PricingSettings
|
|
|
|
#### PricingImpuritySettings
|
|
|
|
Impurity penalty thresholds associated with pricing settings.
|
|
|
|
**Table:** `pricing_impurity_settings`
|
|
|
|
| Attribute | Type | Description |
|
|
| ------------------- | ---------------------------------- | ---------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
|
|
| impurity_code | String(32) | Impurity code |
|
|
| threshold_ppm | Numeric(14,4) | Threshold in ppm |
|
|
| penalty_per_ppm | Numeric(14,4) | Penalty per ppm |
|
|
| notes | Text | Additional notes |
|
|
| created_at | DateTime | Creation timestamp |
|
|
| updated_at | DateTime | Last update timestamp |
|
|
|
|
**Relationships:**
|
|
|
|
- `pricing_settings`: Many-to-one with PricingSettings
|
|
|
|
### Monitoring and Auditing
|
|
|
|
#### PerformanceMetric
|
|
|
|
Performance metrics for monitoring system health.
|
|
|
|
**Table:** `performance_metrics`
|
|
|
|
| Attribute | Type | Description |
|
|
| ---------------- | ------------ | --------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| timestamp | DateTime | Metric timestamp |
|
|
| metric_name | String | Metric name |
|
|
| value | Float | Metric value |
|
|
| labels | String | JSON string of labels |
|
|
| endpoint | String | API endpoint |
|
|
| method | String | HTTP method |
|
|
| status_code | Integer | HTTP status code |
|
|
| duration_seconds | Float | Request duration |
|
|
|
|
#### ImportExportLog
|
|
|
|
Audit log for import and export operations.
|
|
|
|
**Table:** `import_export_logs`
|
|
|
|
| Attribute | Type | Description |
|
|
| ---------- | ----------------------- | ------------------------------------- |
|
|
| id | Integer (PK) | Primary key |
|
|
| action | String(32) | Action type (preview, commit, export) |
|
|
| dataset | String(32) | Dataset type (projects, scenarios) |
|
|
| status | String(16) | Operation status |
|
|
| filename | String(255) | File name |
|
|
| row_count | Integer | Number of rows |
|
|
| detail | Text | Additional details |
|
|
| user_id | Integer (FK → users.id) | User foreign key |
|
|
| created_at | DateTime | Creation timestamp |
|
|
|
|
## Enumerations
|
|
|
|
### MiningOperationType
|
|
|
|
Supported mining operation categories.
|
|
|
|
- `OPEN_PIT`: Open pit mining
|
|
- `UNDERGROUND`: Underground mining
|
|
- `IN_SITU_LEACH`: In-situ leaching
|
|
- `PLACER`: Placer mining
|
|
- `QUARRY`: Quarry operations
|
|
- `MOUNTAINTOP_REMOVAL`: Mountaintop removal
|
|
- `OTHER`: Other operations
|
|
|
|
### ScenarioStatus
|
|
|
|
Lifecycle states for project scenarios.
|
|
|
|
- `DRAFT`: Draft status
|
|
- `ACTIVE`: Active status
|
|
- `ARCHIVED`: Archived status
|
|
|
|
### FinancialCategory
|
|
|
|
Enumeration of cost and revenue classifications.
|
|
|
|
- `CAPITAL_EXPENDITURE`: Capital expenditures
|
|
- `OPERATING_EXPENDITURE`: Operating expenditures
|
|
- `REVENUE`: Revenue
|
|
- `CONTINGENCY`: Contingency
|
|
- `OTHER`: Other
|
|
|
|
### DistributionType
|
|
|
|
Supported stochastic distribution families for simulations.
|
|
|
|
- `NORMAL`: Normal distribution
|
|
- `TRIANGULAR`: Triangular distribution
|
|
- `UNIFORM`: Uniform distribution
|
|
- `LOGNORMAL`: Lognormal distribution
|
|
- `CUSTOM`: Custom distribution
|
|
|
|
### ResourceType
|
|
|
|
Primary consumables and resources used in mining operations.
|
|
|
|
- `DIESEL`: Diesel fuel
|
|
- `ELECTRICITY`: Electrical power
|
|
- `WATER`: Process water
|
|
- `EXPLOSIVES`: Blasting agents
|
|
- `REAGENTS`: Processing reagents
|
|
- `LABOR`: Direct labor
|
|
- `EQUIPMENT_HOURS`: Equipment operating hours
|
|
- `TAILINGS_CAPACITY`: Tailings storage
|
|
|
|
### CostBucket
|
|
|
|
Granular cost buckets aligned with project accounting.
|
|
|
|
- `CAPITAL_INITIAL`: Initial capital
|
|
- `CAPITAL_SUSTAINING`: Sustaining capital
|
|
- `OPERATING_FIXED`: Fixed operating costs
|
|
- `OPERATING_VARIABLE`: Variable operating costs
|
|
- `MAINTENANCE`: Maintenance costs
|
|
- `RECLAMATION`: Reclamation costs
|
|
- `ROYALTIES`: Royalties
|
|
- `GENERAL_ADMIN`: General and administrative
|
|
|
|
### StochasticVariable
|
|
|
|
Domain variables that typically require probabilistic modelling.
|
|
|
|
- `ORE_GRADE`: Ore grade variability
|
|
- `RECOVERY_RATE`: Metallurgical recovery
|
|
- `METAL_PRICE`: Commodity price
|
|
- `OPERATING_COST`: Operating cost per tonne
|
|
- `CAPITAL_COST`: Capital cost
|
|
- `DISCOUNT_RATE`: Discount rate
|
|
- `THROUGHPUT`: Plant throughput
|
|
|
|
## Pydantic Schemas
|
|
|
|
### Authentication Schemas (`schemas/auth.py`)
|
|
|
|
#### RegistrationForm
|
|
|
|
Form model for user registration.
|
|
|
|
| Field | Type | Validation |
|
|
| ---------------- | ---- | ------------------------------- |
|
|
| username | str | 3-128 characters |
|
|
| email | str | Valid email format, 5-255 chars |
|
|
| password | str | 8-256 characters |
|
|
| confirm_password | str | Must match password |
|
|
|
|
#### LoginForm
|
|
|
|
Form model for user login.
|
|
|
|
| Field | Type | Validation |
|
|
| -------- | ---- | ---------------- |
|
|
| username | str | 1-255 characters |
|
|
| password | str | 1-256 characters |
|
|
|
|
#### PasswordResetRequestForm
|
|
|
|
Form model for password reset request.
|
|
|
|
| Field | Type | Validation |
|
|
| ----- | ---- | ------------------------------- |
|
|
| email | str | Valid email format, 5-255 chars |
|
|
|
|
#### PasswordResetForm
|
|
|
|
Form model for password reset.
|
|
|
|
| Field | Type | Validation |
|
|
| ---------------- | ---- | ------------------- |
|
|
| token | str | Required |
|
|
| password | str | 8-256 characters |
|
|
| confirm_password | str | Must match password |
|
|
|
|
### Project Schemas (`schemas/project.py`)
|
|
|
|
#### ProjectBase
|
|
|
|
Base schema for project data.
|
|
|
|
| Field | Type | Description |
|
|
| -------------- | ------------------- | --------------------- |
|
|
| name | str | Project name |
|
|
| location | str \| None | Project location |
|
|
| operation_type | MiningOperationType | Mining operation type |
|
|
| description | str \| None | Project description |
|
|
|
|
#### ProjectCreate
|
|
|
|
Schema for creating projects (inherits ProjectBase).
|
|
|
|
#### ProjectUpdate
|
|
|
|
Schema for updating projects.
|
|
|
|
| Field | Type | Description |
|
|
| -------------- | --------------------------- | --------------------- |
|
|
| name | str \| None | Project name |
|
|
| location | str \| None | Project location |
|
|
| operation_type | MiningOperationType \| None | Mining operation type |
|
|
| description | str \| None | Project description |
|
|
|
|
#### ProjectRead
|
|
|
|
Schema for reading projects (inherits ProjectBase).
|
|
|
|
| Field | Type | Description |
|
|
| ---------- | -------- | ------------------ |
|
|
| id | int | Project ID |
|
|
| created_at | datetime | Creation timestamp |
|
|
| updated_at | datetime | Update timestamp |
|
|
|
|
### Scenario Schemas (`schemas/scenario.py`)
|
|
|
|
#### ScenarioBase
|
|
|
|
Base schema for scenario data.
|
|
|
|
| Field | Type | Description |
|
|
| ---------------- | -------------------- | -------------------------------- |
|
|
| name | str | Scenario name |
|
|
| description | str \| None | Scenario description |
|
|
| status | ScenarioStatus | Scenario status (default: DRAFT) |
|
|
| start_date | date \| None | Start date |
|
|
| end_date | date \| None | End date |
|
|
| discount_rate | float \| None | Discount rate |
|
|
| currency | str \| None | ISO currency code |
|
|
| primary_resource | ResourceType \| None | Primary resource |
|
|
|
|
#### ScenarioCreate
|
|
|
|
Schema for creating scenarios (inherits ScenarioBase).
|
|
|
|
#### ScenarioUpdate
|
|
|
|
Schema for updating scenarios.
|
|
|
|
| Field | Type | Description |
|
|
| ---------------- | ---------------------- | -------------------- |
|
|
| name | str \| None | Scenario name |
|
|
| description | str \| None | Scenario description |
|
|
| status | ScenarioStatus \| None | Scenario status |
|
|
| start_date | date \| None | Start date |
|
|
| end_date | date \| None | End date |
|
|
| discount_rate | float \| None | Discount rate |
|
|
| currency | str \| None | ISO currency code |
|
|
| primary_resource | ResourceType \| None | Primary resource |
|
|
|
|
#### ScenarioRead
|
|
|
|
Schema for reading scenarios (inherits ScenarioBase).
|
|
|
|
| Field | Type | Description |
|
|
| ---------- | -------- | ------------------ |
|
|
| id | int | Scenario ID |
|
|
| project_id | int | Project ID |
|
|
| created_at | datetime | Creation timestamp |
|
|
| updated_at | datetime | Update timestamp |
|
|
|
|
#### ScenarioComparisonRequest
|
|
|
|
Schema for scenario comparison requests.
|
|
|
|
| Field | Type | Description |
|
|
| ------------ | --------- | --------------------------------------- |
|
|
| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) |
|
|
|
|
#### ScenarioComparisonResponse
|
|
|
|
Schema for scenario comparison responses.
|
|
|
|
| Field | Type | Description |
|
|
| ---------- | ------------------ | --------------------- |
|
|
| project_id | int | Project ID |
|
|
| scenarios | list[ScenarioRead] | List of scenario data |
|
|
|
|
### Import Schemas (`schemas/imports.py`)
|
|
|
|
#### ProjectImportRow
|
|
|
|
Schema for importing project data.
|
|
|
|
| Field | Type | Description |
|
|
| -------------- | ------------------- | ----------------------- |
|
|
| name | str | Project name (required) |
|
|
| location | str \| None | Project location |
|
|
| operation_type | MiningOperationType | Mining operation type |
|
|
| description | str \| None | Project description |
|
|
| created_at | datetime \| None | Creation timestamp |
|
|
| updated_at | datetime \| None | Update timestamp |
|
|
|
|
#### ScenarioImportRow
|
|
|
|
Schema for importing scenario data.
|
|
|
|
| Field | Type | Description |
|
|
| ---------------- | -------------------- | -------------------------------- |
|
|
| project_name | str | Project name (required) |
|
|
| name | str | Scenario name (required) |
|
|
| status | ScenarioStatus | Scenario status (default: DRAFT) |
|
|
| start_date | date \| None | Start date |
|
|
| end_date | date \| None | End date |
|
|
| discount_rate | float \| None | Discount rate |
|
|
| currency | str \| None | ISO currency code |
|
|
| primary_resource | ResourceType \| None | Primary resource |
|
|
| description | str \| None | Scenario description |
|
|
| created_at | datetime \| None | Creation timestamp |
|
|
| updated_at | datetime \| None | Update timestamp |
|
|
|
|
### Export Schemas (`schemas/exports.py`)
|
|
|
|
#### ExportFormat
|
|
|
|
Enumeration for export formats.
|
|
|
|
- `CSV`: CSV format
|
|
- `XLSX`: Excel format
|
|
|
|
#### BaseExportRequest
|
|
|
|
Base schema for export requests.
|
|
|
|
| Field | Type | Description |
|
|
| ---------------- | ------------ | --------------------------------- |
|
|
| format | ExportFormat | Export format (default: CSV) |
|
|
| include_metadata | bool | Include metadata (default: False) |
|
|
|
|
#### ProjectExportRequest
|
|
|
|
Schema for project export requests (inherits BaseExportRequest).
|
|
|
|
| Field | Type | Description |
|
|
| ------- | ---------------------------- | -------------- |
|
|
| filters | ProjectExportFilters \| None | Export filters |
|
|
|
|
#### ScenarioExportRequest
|
|
|
|
Schema for scenario export requests (inherits BaseExportRequest).
|
|
|
|
| Field | Type | Description |
|
|
| ------- | ----------------------------- | -------------- |
|
|
| filters | ScenarioExportFilters \| None | Export filters |
|
|
|
|
#### ExportTicket
|
|
|
|
Schema for export tickets.
|
|
|
|
| Field | Type | Description |
|
|
| -------- | -------------------------------- | ------------- |
|
|
| token | str | Export token |
|
|
| format | ExportFormat | Export format |
|
|
| resource | Literal["projects", "scenarios"] | Resource type |
|
|
|
|
#### ExportResponse
|
|
|
|
Schema for export responses.
|
|
|
|
| Field | Type | Description |
|
|
| ------ | ------------ | ------------- |
|
|
| ticket | ExportTicket | Export ticket |
|
|
|
|
## Discrepancies Between Data Models and Pydantic Schemas
|
|
|
|
### Missing Pydantic Schemas
|
|
|
|
The following SQLAlchemy models do not have corresponding Pydantic schemas:
|
|
|
|
- `User` - Authentication handled through forms, not API schemas
|
|
- `Role` - Role management not exposed via API
|
|
- `UserRole` - User-role associations not exposed via API
|
|
- `FinancialInput` - Financial inputs managed through scenario context
|
|
- `SimulationParameter` - Simulation parameters managed through scenario context
|
|
- `PricingSettings` - Pricing settings managed through project context
|
|
- `PricingMetalSettings` - Pricing overrides managed through settings context
|
|
- `PricingImpuritySettings` - Pricing overrides managed through settings context
|
|
- `PerformanceMetric` - Metrics not exposed via API
|
|
- `ImportExportLog` - Audit logs not exposed via API
|
|
|
|
### Schema Differences
|
|
|
|
- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER`
|
|
- **Scenario schemas** include currency normalization validation not present in the model validator
|
|
- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing
|
|
- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document
|
|
|
|
### Additional Validation
|
|
|
|
Pydantic schemas include additional validation beyond SQLAlchemy model constraints:
|
|
|
|
- Email format validation in auth forms
|
|
- Password confirmation matching
|
|
- Currency code normalization and validation
|
|
- Date range validation (end_date >= start_date)
|
|
- Required field validation for imports
|
|
- Enum value coercion with aliases for imports
|
|
|
|
This documentation reflects the current implementation as of the latest development cycle.
|