allucanget/calminer-docs
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add detailed SQLAlchemy models, navigation metadata, enumerations, Pydantic schemas, monitoring, and auditing documentation

Browse Source 
- Introduced SQLAlchemy models for user management, project management, financial inputs, and pricing configuration.
- Created navigation metadata tables for sidebar and top-level menus.
- Cataloged enumerations used across ORM models and Pydantic schemas.
- Documented Pydantic schemas for API request/response validation, including authentication, project, scenario, import, and export schemas.
- Added monitoring and auditing tables for performance metrics and import/export logs.
- Updated security documentation to reflect changes in data model references.
This commit is contained in:
zwitschi
2025-11-13 20:23:09 +01:00
parent 07e68a553d
commit 4dea0a9ae1
7 changed files with 842 additions and 749 deletions
Download Patch File Download Diff File Expand all files Collapse all files

764
architecture/08_concepts/02_data_model.md
View File

@@ -1,755 +1,23 @@
# Data Model
# Data Model Overview
This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization.
Calminer’s data model spans several distinct layers: persisted ORM entities, Pydantic schemas used by the API, navigation metadata, shared enumerations, and operational telemetry tables. To make the material easier to scan, the original monolithic document has been split into focused reference pages.
## SQLAlchemy Models
## Reference Structure
### User Management
- [SQLAlchemy Models](./02_data_model/01_sqlalchemy_models.md) — Domain entities that persist projects, scenarios, pricing configuration, snapshots, and supporting records.
- [Navigation Metadata](./02_data_model/02_navigation.md) — Sidebar and menu configuration tables plus seeding/runtime notes.
- [Enumerations](./02_data_model/03_enumerations.md) — Shared enum definitions used across ORM models and schemas.
- [Pydantic Schemas](./02_data_model/04_pydantic.md) — Request/response models, import/export payloads, and validation nuances.
- [Monitoring and Auditing](./02_data_model/05_monitoring.md) — Telemetry and audit tables supporting observability.
#### User
Each detailed page retains the original headings and tables, so existing anchors and references can migrate with minimal disruption.
Represents authenticated platform users with optional elevated privileges.
## How to Use This Overview
**Table:** `users`
- Start with the SQLAlchemy reference when you need to understand persistence concerns or relationships between core domain objects.
- Jump to the Pydantic schemas document when adjusting API payloads or validation logic.
- Consult the enumerations list before introducing new enum values to keep backend and frontend usage aligned.
- Review the navigation metadata page when seeding or modifying the application sidebar.
- Use the monitoring and auditing section to track telemetry fields that drive dashboards and compliance reporting.
| Attribute | Type | Description |
| ------------- | ------------ | ------------------------- |
| id | Integer (PK) | Primary key |
| email | String(255) | Unique email address |
| username | String(128) | Unique username |
| password_hash | String(255) | Argon2 hashed password |
| is_active | Boolean | Account activation status |
| is_superuser | Boolean | Superuser privileges |
| last_login_at | DateTime | Last login timestamp |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `role_assignments`: Many-to-many with Role via UserRole
#### Role
Defines user roles for role-based access control (RBAC).
**Table:** `roles`
| Attribute | Type | Description |
| ------------ | ------------ | --------------------- |
| id | Integer (PK) | Primary key |
| name | String(64) | Unique role name |
| display_name | String(128) | Human-readable name |
| description | Text | Role description |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `assignments`: One-to-many with UserRole
- `users`: Many-to-many with User (viewonly)
#### UserRole
Association between users and roles with assignment metadata.
**Table:** `user_roles`
| Attribute | Type | Description |
| ---------- | ----------------------- | -------------------- |
| user_id | Integer (FK → users.id) | User foreign key |
| role_id | Integer (FK → roles.id) | Role foreign key |
| granted_at | DateTime | Assignment timestamp |
| granted_by | Integer (FK → users.id) | Granting user |
**Relationships:**
- `user`: Many-to-one with User
- `role`: Many-to-one with Role
- `granted_by_user`: Many-to-one with User
### Project Management
#### Project
Top-level mining project grouping multiple scenarios.
**Table:** `projects`
| Attribute | Type | Description |
| ------------------- | ---------------------------------- | --------------------- |
| id | Integer (PK) | Primary key |
| name | String(255) | Unique project name |
| location | String(255) | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | Text | Project description |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenarios`: One-to-many with Scenario
- `pricing_settings`: Many-to-one with PricingSettings
#### Scenario
A specific configuration of assumptions for a project.
**Table:** `scenarios`
| Attribute | Type | Description |
| ---------------- | -------------------------- | ------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Project foreign key |
| name | String(255) | Scenario name |
| description | Text | Scenario description |
| status | ScenarioStatus | Scenario lifecycle status |
| start_date | Date | Scenario start date |
| end_date | Date | Scenario end date |
| discount_rate | Numeric(5,2) | Discount rate percentage |
| currency | String(3) | ISO currency code |
| primary_resource | ResourceType | Primary resource type |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `financial_inputs`: One-to-many with FinancialInput
- `simulation_parameters`: One-to-many with SimulationParameter
#### Projects → Scenarios → Profitability Calculations
Calminer organises feasibility data in a nested hierarchy. A project defines the overarching mining context and exposes a one-to-many `scenarios` collection. Each scenario captures a self-contained assumption set and anchors derived artefacts such as financial inputs, simulation parameters, and profitability snapshots. Profitability calculations execute at the scenario scope; when triggered, the workflow in `services/calculations.py` persists a `ScenarioProfitability` record and can optionally roll results up to project level by creating a `ProjectProfitability` snapshot. Consumers typically surface the most recent metrics via the `latest_profitability` helpers on both ORM models.
| Layer | ORM models | Pydantic schema(s) | Key relationships |
| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Project | `models.project.Project` | `schemas.project.ProjectRead` | `Project.scenarios`, `Project.profitability_snapshots`, `Project.latest_profitability` |
| Scenario | `models.scenario.Scenario` | `schemas.scenario.ScenarioRead` | `Scenario.project`, `Scenario.profitability_snapshots`, `Scenario.latest_profitability` |
| Profitability calculations | `models.profitability_snapshot.ProjectProfitability`, `models.profitability_snapshot.ScenarioProfitability` | `schemas.calculations.ProfitabilityCalculationRequest`, `schemas.calculations.ProfitabilityCalculationResult` | Persisted via `services.calculations.calculate_profitability`; aggregates scenario metrics into project snapshots |
Detailed CRUD endpoint behaviour for projects and scenarios is documented in `calminer-docs/api/README.md`.
#### FinancialInput
Line-item financial assumption attached to a scenario.
**Table:** `financial_inputs`
| Attribute | Type | Description |
| -------------- | --------------------------- | -------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
| name | String(255) | Input name |
| category | FinancialCategory | Financial category |
| cost_bucket | CostBucket | Cost bucket classification |
| amount | Numeric(18,2) | Monetary amount |
| currency | String(3) | ISO currency code |
| effective_date | Date | Effective date |
| notes | Text | Additional notes |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
### Project and Scenario Models
#### ProjectCapexSnapshot
Project-level snapshot capturing aggregated initial capital expenditure metrics.
**Table:** `project_capex_snapshots`
| Attribute | Type | Description |
| ---------------------- | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Associated project |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier (UI, API, etc.) |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency |
| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage |
| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount |
| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `created_by`: Many-to-one with User (nullable)
#### ScenarioCapexSnapshot
Scenario-level snapshot storing detailed capex results.
**Table:** `scenario_capex_snapshots`
| Attribute | Type | Description |
| ---------------------- | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Associated scenario |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency |
| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage |
| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount |
| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
- `created_by`: Many-to-one with User (nullable)
#### ProjectOpexSnapshot
Project-level snapshot persisting recurring opex metrics.
**Table:** `project_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Associated project |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |
| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied |
| apply_escalation | Boolean | Flag indicating whether escalation was applied |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `created_by`: Many-to-one with User (nullable)
#### ScenarioOpexSnapshot
Scenario-level snapshot persisting recurring opex metrics.
**Table:** `scenario_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Associated scenario |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |
| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied |
| apply_escalation | Boolean | Flag indicating whether escalation was applied |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
- `created_by`: Many-to-one with User (nullable)
#### SimulationParameter
Probability distribution settings for scenario simulations.
**Table:** `simulation_parameters`
| Attribute | Type | Description |
| ------------------ | --------------------------- | ------------------------ |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
| name | String(255) | Parameter name |
| distribution | DistributionType | Distribution type |
| variable | StochasticVariable | Stochastic variable type |
| resource_type | ResourceType | Resource type |
| mean_value | Numeric(18,4) | Mean value |
| standard_deviation | Numeric(18,4) | Standard deviation |
| minimum_value | Numeric(18,4) | Minimum value |
| maximum_value | Numeric(18,4) | Maximum value |
| unit | String(32) | Unit of measurement |
| configuration | JSON | Additional configuration |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
### Pricing Configuration
#### PricingSettings
Persisted pricing defaults applied to scenario evaluations.
**Table:** `pricing_settings`
| Attribute | Type | Description |
| ------------------------ | ------------- | ----------------------------- |
| id | Integer (PK) | Primary key |
| name | String(128) | Unique settings name |
| slug | String(64) | Unique slug identifier |
| description | Text | Settings description |
| default_currency | String(3) | Default ISO currency code |
| default_payable_pct | Numeric(5,2) | Default payable percentage |
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage |
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent |
| metadata_payload | JSON | Additional metadata |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `metal_overrides`: One-to-many with PricingMetalSettings
- `impurity_overrides`: One-to-many with PricingImpuritySettings
- `projects`: One-to-many with Project
#### PricingMetalSettings
Contract-specific overrides for a particular metal.
**Table:** `pricing_metal_settings`
| Attribute | Type | Description |
| ------------------------ | ---------------------------------- | ---------------------------- |
| id | Integer (PK) | Primary key |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| metal_code | String(32) | Metal code |
| payable_pct | Numeric(5,2) | Payable percentage |
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold |
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty |
| data | JSON | Additional data |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `pricing_settings`: Many-to-one with PricingSettings
#### PricingImpuritySettings
Impurity penalty thresholds associated with pricing settings.
**Table:** `pricing_impurity_settings`
| Attribute | Type | Description |
| ------------------- | ---------------------------------- | ---------------------------- |
| id | Integer (PK) | Primary key |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| impurity_code | String(32) | Impurity code |
| threshold_ppm | Numeric(14,4) | Threshold in ppm |
| penalty_per_ppm | Numeric(14,4) | Penalty per ppm |
| notes | Text | Additional notes |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `pricing_settings`: Many-to-one with PricingSettings
### Monitoring and Auditing
#### PerformanceMetric
Performance metrics for monitoring system health.
**Table:** `performance_metrics`
| Attribute | Type | Description |
| ---------------- | ------------ | --------------------- |
| id | Integer (PK) | Primary key |
| timestamp | DateTime | Metric timestamp |
| metric_name | String | Metric name |
| value | Float | Metric value |
| labels | String | JSON string of labels |
| endpoint | String | API endpoint |
| method | String | HTTP method |
| status_code | Integer | HTTP status code |
| duration_seconds | Float | Request duration |
#### ImportExportLog
Audit log for import and export operations.
**Table:** `import_export_logs`
| Attribute | Type | Description |
| ---------- | ----------------------- | ------------------------------------- |
| id | Integer (PK) | Primary key |
| action | String(32) | Action type (preview, commit, export) |
| dataset | String(32) | Dataset type (projects, scenarios) |
| status | String(16) | Operation status |
| filename | String(255) | File name |
| row_count | Integer | Number of rows |
| detail | Text | Additional details |
| user_id | Integer (FK → users.id) | User foreign key |
| created_at | DateTime | Creation timestamp |
## Enumerations
### MiningOperationType
Supported mining operation categories.
- `OPEN_PIT`: Open pit mining
- `UNDERGROUND`: Underground mining
- `IN_SITU_LEACH`: In-situ leaching
- `PLACER`: Placer mining
- `QUARRY`: Quarry operations
- `MOUNTAINTOP_REMOVAL`: Mountaintop removal
- `OTHER`: Other operations
### ScenarioStatus
Lifecycle states for project scenarios.
- `DRAFT`: Draft status
- `ACTIVE`: Active status
- `ARCHIVED`: Archived status
### FinancialCategory
Enumeration of cost and revenue classifications.
- `CAPITAL_EXPENDITURE`: Capital expenditures
- `OPERATING_EXPENDITURE`: Operating expenditures
- `REVENUE`: Revenue
- `CONTINGENCY`: Contingency
- `OTHER`: Other
### DistributionType
Supported stochastic distribution families for simulations.
- `NORMAL`: Normal distribution
- `TRIANGULAR`: Triangular distribution
- `UNIFORM`: Uniform distribution
- `LOGNORMAL`: Lognormal distribution
- `CUSTOM`: Custom distribution
### ResourceType
Primary consumables and resources used in mining operations.
- `DIESEL`: Diesel fuel
- `ELECTRICITY`: Electrical power
- `WATER`: Process water
- `EXPLOSIVES`: Blasting agents
- `REAGENTS`: Processing reagents
- `LABOR`: Direct labor
- `EQUIPMENT_HOURS`: Equipment operating hours
- `TAILINGS_CAPACITY`: Tailings storage
### CostBucket
Granular cost buckets aligned with project accounting.
- `CAPITAL_INITIAL`: Initial capital
- `CAPITAL_SUSTAINING`: Sustaining capital
- `OPERATING_FIXED`: Fixed operating costs
- `OPERATING_VARIABLE`: Variable operating costs
- `MAINTENANCE`: Maintenance costs
- `RECLAMATION`: Reclamation costs
- `ROYALTIES`: Royalties
- `GENERAL_ADMIN`: General and administrative
### StochasticVariable
Domain variables that typically require probabilistic modelling.
- `ORE_GRADE`: Ore grade variability
- `RECOVERY_RATE`: Metallurgical recovery
- `METAL_PRICE`: Commodity price
- `OPERATING_COST`: Operating cost per tonne
- `CAPITAL_COST`: Capital cost
- `DISCOUNT_RATE`: Discount rate
- `THROUGHPUT`: Plant throughput
## Pydantic Schemas
### Authentication Schemas (`schemas/auth.py`)
#### RegistrationForm
Form model for user registration.
| Field | Type | Validation |
| ---------------- | ---- | ------------------------------- |
| username | str | 3-128 characters |
| email | str | Valid email format, 5-255 chars |
| password | str | 8-256 characters |
| confirm_password | str | Must match password |
#### LoginForm
Form model for user login.
| Field | Type | Validation |
| -------- | ---- | ---------------- |
| username | str | 1-255 characters |
| password | str | 1-256 characters |
#### PasswordResetRequestForm
Form model for password reset request.
| Field | Type | Validation |
| ----- | ---- | ------------------------------- |
| email | str | Valid email format, 5-255 chars |
#### PasswordResetForm
Form model for password reset.
| Field | Type | Validation |
| ---------------- | ---- | ------------------- |
| token | str | Required |
| password | str | 8-256 characters |
| confirm_password | str | Must match password |
### Project Schemas (`schemas/project.py`)
#### ProjectBase
Base schema for project data.
| Field | Type | Description |
| -------------- | ------------------- | --------------------- |
| name | str | Project name |
| location | str \| None | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | str \| None | Project description |
#### ProjectCreate
Schema for creating projects (inherits ProjectBase).
#### ProjectUpdate
Schema for updating projects.
| Field | Type | Description |
| -------------- | --------------------------- | --------------------- |
| name | str \| None | Project name |
| location | str \| None | Project location |
| operation_type | MiningOperationType \| None | Mining operation type |
| description | str \| None | Project description |
#### ProjectRead
Schema for reading projects (inherits ProjectBase).
| Field | Type | Description |
| ---------- | -------- | ------------------ |
| id | int | Project ID |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp |
### Scenario Schemas (`schemas/scenario.py`)
#### ScenarioBase
Base schema for scenario data.
| Field | Type | Description |
| ---------------- | -------------------- | -------------------------------- |
| name | str | Scenario name |
| description | str \| None | Scenario description |
| status | ScenarioStatus | Scenario status (default: DRAFT) |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
#### ScenarioCreate
Schema for creating scenarios (inherits ScenarioBase).
#### ScenarioUpdate
Schema for updating scenarios.
| Field | Type | Description |
| ---------------- | ---------------------- | -------------------- |
| name | str \| None | Scenario name |
| description | str \| None | Scenario description |
| status | ScenarioStatus \| None | Scenario status |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
#### ScenarioRead
Schema for reading scenarios (inherits ScenarioBase).
| Field | Type | Description |
| ---------- | -------- | ------------------ |
| id | int | Scenario ID |
| project_id | int | Project ID |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp |
#### ScenarioComparisonRequest
Schema for scenario comparison requests.
| Field | Type | Description |
| ------------ | --------- | --------------------------------------- |
| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) |
#### ScenarioComparisonResponse
Schema for scenario comparison responses.
| Field | Type | Description |
| ---------- | ------------------ | --------------------- |
| project_id | int | Project ID |
| scenarios | list[ScenarioRead] | List of scenario data |
### Import Schemas (`schemas/imports.py`)
#### ProjectImportRow
Schema for importing project data.
| Field | Type | Description |
| -------------- | ------------------- | ----------------------- |
| name | str | Project name (required) |
| location | str \| None | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | str \| None | Project description |
| created_at | datetime \| None | Creation timestamp |
| updated_at | datetime \| None | Update timestamp |
#### ScenarioImportRow
Schema for importing scenario data.
| Field | Type | Description |
| ---------------- | -------------------- | -------------------------------- |
| project_name | str | Project name (required) |
| name | str | Scenario name (required) |
| status | ScenarioStatus | Scenario status (default: DRAFT) |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
| description | str \| None | Scenario description |
| created_at | datetime \| None | Creation timestamp |
| updated_at | datetime \| None | Update timestamp |
### Export Schemas (`schemas/exports.py`)
#### ExportFormat
Enumeration for export formats.
- `CSV`: CSV format
- `XLSX`: Excel format
#### BaseExportRequest
Base schema for export requests.
| Field | Type | Description |
| ---------------- | ------------ | --------------------------------- |
| format | ExportFormat | Export format (default: CSV) |
| include_metadata | bool | Include metadata (default: False) |
#### ProjectExportRequest
Schema for project export requests (inherits BaseExportRequest).
| Field | Type | Description |
| ------- | ---------------------------- | -------------- |
| filters | ProjectExportFilters \| None | Export filters |
#### ScenarioExportRequest
Schema for scenario export requests (inherits BaseExportRequest).
| Field | Type | Description |
| ------- | ----------------------------- | -------------- |
| filters | ScenarioExportFilters \| None | Export filters |
#### ExportTicket
Schema for export tickets.
| Field | Type | Description |
| -------- | -------------------------------- | ------------- |
| token | str | Export token |
| format | ExportFormat | Export format |
| resource | Literal["projects", "scenarios"] | Resource type |
#### ExportResponse
Schema for export responses.
| Field | Type | Description |
| ------ | ------------ | ------------- |
| ticket | ExportTicket | Export ticket |
## Discrepancies Between Data Models and Pydantic Schemas
### Missing Pydantic Schemas
The following SQLAlchemy models do not have corresponding Pydantic schemas:
- `User` - Authentication handled through forms, not API schemas
- `Role` - Role management not exposed via API
- `UserRole` - User-role associations not exposed via API
- `FinancialInput` - Financial inputs managed through scenario context
- `SimulationParameter` - Simulation parameters managed through scenario context
- `PricingSettings` - Pricing settings managed through project context
- `PricingMetalSettings` - Pricing overrides managed through settings context
- `PricingImpuritySettings` - Pricing overrides managed through settings context
- `PerformanceMetric` - Metrics not exposed via API
- `ImportExportLog` - Audit logs not exposed via API
### Schema Differences
- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER`
- **Scenario schemas** include currency normalization validation not present in the model validator
- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing
- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document
### Additional Validation
Pydantic schemas include additional validation beyond SQLAlchemy model constraints:
- Email format validation in auth forms
- Password confirmation matching
- Currency code normalization and validation
- Date range validation (end_date >= start_date)
- Required field validation for imports
- Enum value coercion with aliases for imports
This documentation reflects the current implementation as of the latest development cycle.
Cross-links between these documents mirror the previous inline references. Update any external links to point at the new files during your next documentation touchpoint.

369
architecture/08_concepts/02_data_model/01_sqlalchemy_models.md Normal file
View File

@@ -0,0 +1,369 @@
# SQLAlchemy Models
This reference describes the primary SQLAlchemy ORM models that underpin Calminer. It mirrors the original hierarchy from `02_data_model.md`, extracted so each domain area can evolve independently. See the [data model overview](../02_data_model.md) for context and navigation.
## User Management
### User
Represents authenticated platform users with optional elevated privileges.
**Table:** `users`
| Attribute | Type | Description |
| ------------- | ------------ | ------------------------- |
| id | Integer (PK) | Primary key |
| email | String(255) | Unique email address |
| username | String(128) | Unique username |
| password_hash | String(255) | Argon2 hashed password |
| is_active | Boolean | Account activation status |
| is_superuser | Boolean | Superuser privileges |
| last_login_at | DateTime | Last login timestamp |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `role_assignments`: Many-to-many with Role via UserRole
### Role
Defines user roles for role-based access control (RBAC).
**Table:** `roles`
| Attribute | Type | Description |
| ------------ | ------------ | --------------------- |
| id | Integer (PK) | Primary key |
| name | String(64) | Unique role name |
| display_name | String(128) | Human-readable name |
| description | Text | Role description |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `assignments`: One-to-many with UserRole
- `users`: Many-to-many with User (viewonly)
### UserRole
Association between users and roles with assignment metadata.
**Table:** `user_roles`
| Attribute | Type | Description |
| ---------- | ----------------------- | -------------------- |
| user_id | Integer (FK → users.id) | User foreign key |
| role_id | Integer (FK → roles.id) | Role foreign key |
| granted_at | DateTime | Assignment timestamp |
| granted_by | Integer (FK → users.id) | Granting user |
**Relationships:**
- `user`: Many-to-one with User
- `role`: Many-to-one with Role
- `granted_by_user`: Many-to-one with User
## Project Management
### Project
Top-level mining project grouping multiple scenarios.
**Table:** `projects`
| Attribute | Type | Description |
| ------------------- | ---------------------------------- | --------------------- |
| id | Integer (PK) | Primary key |
| name | String(255) | Unique project name |
| location | String(255) | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | Text | Project description |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenarios`: One-to-many with Scenario
- `pricing_settings`: Many-to-one with PricingSettings
### Scenario
A specific configuration of assumptions for a project.
**Table:** `scenarios`
| Attribute | Type | Description |
| ---------------- | -------------------------- | ------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Project foreign key |
| name | String(255) | Scenario name |
| description | Text | Scenario description |
| status | ScenarioStatus | Scenario lifecycle status |
| start_date | Date | Scenario start date |
| end_date | Date | Scenario end date |
| discount_rate | Numeric(5,2) | Discount rate percentage |
| currency | String(3) | ISO currency code |
| primary_resource | ResourceType | Primary resource type |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `financial_inputs`: One-to-many with FinancialInput
- `simulation_parameters`: One-to-many with SimulationParameter
### Projects → Scenarios → Profitability Calculations
Calminer organises feasibility data in a nested hierarchy. A project defines the overarching mining context and exposes a one-to-many `scenarios` collection. Each scenario captures a self-contained assumption set and anchors derived artefacts such as financial inputs, simulation parameters, and profitability snapshots. Profitability calculations execute at the scenario scope; when triggered, the workflow in `services/calculations.py` persists a `ScenarioProfitability` record and can optionally roll results up to project level by creating a `ProjectProfitability` snapshot. Consumers typically surface the most recent metrics via the `latest_profitability` helpers on both ORM models.
| Layer | ORM models | Pydantic schema(s) | Key relationships |
| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Project | `models.project.Project` | `schemas.project.ProjectRead` | `Project.scenarios`, `Project.profitability_snapshots`, `Project.latest_profitability` |
| Scenario | `models.scenario.Scenario` | `schemas.scenario.ScenarioRead` | `Scenario.project`, `Scenario.profitability_snapshots`, `Scenario.latest_profitability` |
| Profitability calculations | `models.profitability_snapshot.ProjectProfitability`, `models.profitability_snapshot.ScenarioProfitability` | `schemas.calculations.ProfitabilityCalculationRequest`, `schemas.calculations.ProfitabilityCalculationResult` | Persisted via `services.calculations.calculate_profitability`; aggregates scenario metrics into project snapshots |
Detailed CRUD endpoint behaviour for projects and scenarios is documented in `calminer-docs/api/README.md`.
### FinancialInput
Line-item financial assumption attached to a scenario.
**Table:** `financial_inputs`
| Attribute | Type | Description |
| -------------- | --------------------------- | -------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
| name | String(255) | Input name |
| category | FinancialCategory | Financial category |
| cost_bucket | CostBucket | Cost bucket classification |
| amount | Numeric(18,2) | Monetary amount |
| currency | String(3) | ISO currency code |
| effective_date | Date | Effective date |
| notes | Text | Additional notes |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
## Project and Scenario Snapshots
### ProjectCapexSnapshot
Project-level snapshot capturing aggregated initial capital expenditure metrics.
**Table:** `project_capex_snapshots`
| Attribute | Type | Description |
| ---------------------- | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Associated project |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier (UI, API, etc.) |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency |
| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage |
| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount |
| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `created_by`: Many-to-one with User (nullable)
### ScenarioCapexSnapshot
Scenario-level snapshot storing detailed capex results.
**Table:** `scenario_capex_snapshots`
| Attribute | Type | Description |
| ---------------------- | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Associated scenario |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency |
| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage |
| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount |
| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
- `created_by`: Many-to-one with User (nullable)
### ProjectOpexSnapshot
Project-level snapshot persisting recurring opex metrics.
**Table:** `project_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| project_id | Integer (FK → projects.id) | Associated project |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |
| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied |
| apply_escalation | Boolean | Flag indicating whether escalation was applied |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `project`: Many-to-one with Project
- `created_by`: Many-to-one with User (nullable)
### ScenarioOpexSnapshot
Scenario-level snapshot persisting recurring opex metrics.
**Table:** `scenario_opex_snapshots`
| Attribute | Type | Description |
| ------------------------ | --------------------------------- | ------------------------------------------------------- |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Associated scenario |
| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence |
| calculation_source | String(64), nullable | Originating workflow identifier |
| calculated_at | DateTime | Timestamp the calculation completed |
| currency_code | String(3), nullable | Currency for totals |
| overall_annual | Numeric(18,2), nullable | Total annual opex |
| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon |
| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon |
| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline |
| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied |
| apply_escalation | Boolean | Flag indicating whether escalation was applied |
| component_count | Integer, nullable | Number of normalized components captured |
| payload | JSON, nullable | Serialized component breakdown and calculation metadata |
| created_at | DateTime | Record creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
- `created_by`: Many-to-one with User (nullable)
### SimulationParameter
Probability distribution settings for scenario simulations.
**Table:** `simulation_parameters`
| Attribute | Type | Description |
| ------------------ | --------------------------- | ------------------------ |
| id | Integer (PK) | Primary key |
| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key |
| name | String(255) | Parameter name |
| distribution | DistributionType | Distribution type |
| variable | StochasticVariable | Stochastic variable type |
| resource_type | ResourceType | Resource type |
| mean_value | Numeric(18,4) | Mean value |
| standard_deviation | Numeric(18,4) | Standard deviation |
| minimum_value | Numeric(18,4) | Minimum value |
| maximum_value | Numeric(18,4) | Maximum value |
| unit | String(32) | Unit of measurement |
| configuration | JSON | Additional configuration |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `scenario`: Many-to-one with Scenario
## Pricing Configuration
### PricingSettings
Persisted pricing defaults applied to scenario evaluations.
**Table:** `pricing_settings`
| Attribute | Type | Description |
| ------------------------ | ------------- | ----------------------------- |
| id | Integer (PK) | Primary key |
| name | String(128) | Unique settings name |
| slug | String(64) | Unique slug identifier |
| description | Text | Settings description |
| default_currency | String(3) | Default ISO currency code |
| default_payable_pct | Numeric(5,2) | Default payable percentage |
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage |
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent |
| metadata_payload | JSON | Additional metadata |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `metal_overrides`: One-to-many with PricingMetalSettings
- `impurity_overrides`: One-to-many with PricingImpuritySettings
- `projects`: One-to-many with Project
### PricingMetalSettings
Contract-specific overrides for a particular metal.
**Table:** `pricing_metal_settings`
| Attribute | Type | Description |
| ------------------------ | ---------------------------------- | ---------------------------- |
| id | Integer (PK) | Primary key |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| metal_code | String(32) | Metal code |
| payable_pct | Numeric(5,2) | Payable percentage |
| moisture_threshold_pct | Numeric(5,2) | Moisture threshold |
| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty |
| data | JSON | Additional data |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `pricing_settings`: Many-to-one with PricingSettings
### PricingImpuritySettings
Impurity penalty thresholds associated with pricing settings.
**Table:** `pricing_impurity_settings`
| Attribute | Type | Description |
| ------------------- | ---------------------------------- | ---------------------------- |
| id | Integer (PK) | Primary key |
| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key |
| impurity_code | String(32) | Impurity code |
| threshold_ppm | Numeric(14,4) | Threshold in ppm |
| penalty_per_ppm | Numeric(14,4) | Penalty per ppm |
| notes | Text | Additional notes |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `pricing_settings`: Many-to-one with PricingSettings

65
architecture/08_concepts/02_data_model/02_navigation.md Normal file
View File

@@ -0,0 +1,65 @@
# Navigation Metadata
This page details the navigation metadata tables that drive the Calminer sidebar and top-level menus. It was split from `02_data_model.md` to isolate runtime navigation behaviour from the broader ORM catalogue.
## NavigationGroup
Represents a logical grouping of navigation links shown in the UI sidebar or header.
**Table:** `navigation_groups`
| Attribute | Type | Description |
| ------------ | ------------ | ----------------------------------------------- |
| id | Integer (PK) | Primary key |
| slug | String(64) | Unique slug identifier |
| title | String(128) | Display title |
| description | Text | Optional descriptive text |
| match_prefix | String(255) | URL prefix used to auto-expand in UI |
| weighting | Integer | Ordering weight |
| icon_name | String(64) | Optional icon identifier |
| feature_flag | String(64) | Optional feature flag for conditional rendering |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `links`: One-to-many with NavigationLink
## NavigationLink
Individual navigation entry that may have an associated feature flag or role restriction.
**Table:** `navigation_links`
| Attribute | Type | Description |
| ------------- | ----------------------------------- | ---------------------------------------------------- |
| id | Integer (PK) | Primary key |
| group_id | Integer (FK → navigation_groups.id) | Parent navigation group |
| parent_id | Integer (FK → navigation_links.id) | Optional parent link (for nested menus) |
| slug | String(64) | Unique slug identifier |
| title | String(128) | Display title |
| description | Text | Optional descriptive text |
| href | String(255) | Static fallback URL |
| match_prefix | String(255) | URL prefix for expansion |
| feature_flag | String(64) | Optional feature flag for conditional rendering |
| required_role | String(64) | Optional role required to view the link |
| weighting | Integer | Ordering weight within the parent scope |
| icon_name | String(64) | Optional icon identifier |
| metadata | JSON | Additional configuration (e.g. legacy route aliases) |
| created_at | DateTime | Creation timestamp |
| updated_at | DateTime | Last update timestamp |
**Relationships:**
- `group`: Many-to-one with NavigationGroup
- `parent`: Many-to-one self-reference
- `children`: One-to-many self-reference
## Seeding and Runtime Consumption
- **Seed script:** `scripts/init_db.py` provisions baseline groups and links, including nested scenario calculators and role-gated admin entries.
- **Service layer:** `services/navigation.py` maps ORM entities into response DTOs, resolves contextual URLs for projects and scenarios, and applies feature flag as well as role filters.
- **API exposure:** `routes/navigation.py` serves the `/navigation/sidebar` endpoint, combining seeded data with runtime context to produce navigation trees consumed by the frontend.
- **Testing:** `tests/services/test_navigation_service.py` covers URL resolution and child filtering logic, while `tests/integration/test_navigation_sidebar_calculations.py` verifies scenario calculator entries in the API payload.
Refer to the [navigation service documentation](../../../../api/README.md#navigation) for endpoint-level behaviour.

83
architecture/08_concepts/02_data_model/03_enumerations.md Normal file
View File

@@ -0,0 +1,83 @@
# Enumerations
This page catalogs the enumerations shared by Calminer ORM models and Pydantic schemas. The entries mirror the original content from `02_data_model.md` so enum updates can be maintained independently of the broader data model reference.
## MiningOperationType
Supported mining operation categories.
- `OPEN_PIT`: Open pit mining
- `UNDERGROUND`: Underground mining
- `IN_SITU_LEACH`: In-situ leaching
- `PLACER`: Placer mining
- `QUARRY`: Quarry operations
- `MOUNTAINTOP_REMOVAL`: Mountaintop removal
- `OTHER`: Other operations
## ScenarioStatus
Lifecycle states for project scenarios.
- `DRAFT`: Draft status
- `ACTIVE`: Active status
- `ARCHIVED`: Archived status
## FinancialCategory
Enumeration of cost and revenue classifications.
- `CAPITAL_EXPENDITURE`: Capital expenditures
- `OPERATING_EXPENDITURE`: Operating expenditures
- `REVENUE`: Revenue
- `CONTINGENCY`: Contingency
- `OTHER`: Other
## DistributionType
Supported stochastic distribution families for simulations.
- `NORMAL`: Normal distribution
- `TRIANGULAR`: Triangular distribution
- `UNIFORM`: Uniform distribution
- `LOGNORMAL`: Lognormal distribution
- `CUSTOM`: Custom distribution
## ResourceType
Primary consumables and resources used in mining operations.
- `DIESEL`: Diesel fuel
- `ELECTRICITY`: Electrical power
- `WATER`: Process water
- `EXPLOSIVES`: Blasting agents
- `REAGENTS`: Processing reagents
- `LABOR`: Direct labor
- `EQUIPMENT_HOURS`: Equipment operating hours
- `TAILINGS_CAPACITY`: Tailings storage
## CostBucket
Granular cost buckets aligned with project accounting.
- `CAPITAL_INITIAL`: Initial capital
- `CAPITAL_SUSTAINING`: Sustaining capital
- `OPERATING_FIXED`: Fixed operating costs
- `OPERATING_VARIABLE`: Variable operating costs
- `MAINTENANCE`: Maintenance costs
- `RECLAMATION`: Reclamation costs
- `ROYALTIES`: Royalties
- `GENERAL_ADMIN`: General and administrative
## StochasticVariable
Domain variables that typically require probabilistic modelling.
- `ORE_GRADE`: Ore grade variability
- `RECOVERY_RATE`: Metallurgical recovery
- `METAL_PRICE`: Commodity price
- `OPERATING_COST`: Operating cost per tonne
- `CAPITAL_COST`: Capital cost
- `DISCOUNT_RATE`: Discount rate
- `THROUGHPUT`: Plant throughput
Refer back to the [data model overview](../02_data_model.md) when choosing the appropriate enum for new features.

267
architecture/08_concepts/02_data_model/04_pydantic.md Normal file
View File

@@ -0,0 +1,267 @@
# Pydantic Schemas
This page documents the Pydantic models used for request/response validation across the Calminer API surface. It was extracted from `02_data_model.md` so API-facing contracts can evolve separately from the ORM layer.
## Authentication Schemas (`schemas/auth.py`)
### RegistrationForm
Form model for user registration.
| Field | Type | Validation |
| ---------------- | ---- | ------------------------------- |
| username | str | 3-128 characters |
| email | str | Valid email format, 5-255 chars |
| password | str | 8-256 characters |
| confirm_password | str | Must match password |
### LoginForm
Form model for user login.
| Field | Type | Validation |
| -------- | ---- | ---------------- |
| username | str | 1-255 characters |
| password | str | 1-256 characters |
### PasswordResetRequestForm
Form model for password reset request.
| Field | Type | Validation |
| ----- | ---- | ------------------------------- |
| email | str | Valid email format, 5-255 chars |
### PasswordResetForm
Form model for password reset.
| Field | Type | Validation |
| ---------------- | ---- | ------------------- |
| token | str | Required |
| password | str | 8-256 characters |
| confirm_password | str | Must match password |
## Project Schemas (`schemas/project.py`)
### ProjectBase
Base schema for project data.
| Field | Type | Description |
| -------------- | ------------------- | --------------------- |
| name | str | Project name |
| location | str \| None | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | str \| None | Project description |
### ProjectCreate
Schema for creating projects (inherits ProjectBase).
### ProjectUpdate
Schema for updating projects.
| Field | Type | Description |
| -------------- | --------------------------- | --------------------- |
| name | str \| None | Project name |
| location | str \| None | Project location |
| operation_type | MiningOperationType \| None | Mining operation type |
| description | str \| None | Project description |
### ProjectRead
Schema for reading projects (inherits ProjectBase).
| Field | Type | Description |
| ---------- | -------- | ------------------ |
| id | int | Project ID |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp |
## Scenario Schemas (`schemas/scenario.py`)
### ScenarioBase
Base schema for scenario data.
| Field | Type | Description |
| ---------------- | -------------------- | -------------------------------- |
| name | str | Scenario name |
| description | str \| None | Scenario description |
| status | ScenarioStatus | Scenario status (default: DRAFT) |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
### ScenarioCreate
Schema for creating scenarios (inherits ScenarioBase).
### ScenarioUpdate
Schema for updating scenarios.
| Field | Type | Description |
| ---------------- | ---------------------- | -------------------- |
| name | str \| None | Scenario name |
| description | str \| None | Scenario description |
| status | ScenarioStatus \| None | Scenario status |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
### ScenarioRead
Schema for reading scenarios (inherits ScenarioBase).
| Field | Type | Description |
| ---------- | -------- | ------------------ |
| id | int | Scenario ID |
| project_id | int | Project ID |
| created_at | datetime | Creation timestamp |
| updated_at | datetime | Update timestamp |
### ScenarioComparisonRequest
Schema for scenario comparison requests.
| Field | Type | Description |
| ------------ | --------- | --------------------------------------- |
| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) |
### ScenarioComparisonResponse
Schema for scenario comparison responses.
| Field | Type | Description |
| ---------- | ------------------ | --------------------- |
| project_id | int | Project ID |
| scenarios | list[ScenarioRead] | List of scenario data |
## Import Schemas (`schemas/imports.py`)
### ProjectImportRow
Schema for importing project data.
| Field | Type | Description |
| -------------- | ------------------- | ----------------------- |
| name | str | Project name (required) |
| location | str \| None | Project location |
| operation_type | MiningOperationType | Mining operation type |
| description | str \| None | Project description |
| created_at | datetime \| None | Creation timestamp |
| updated_at | datetime \| None | Update timestamp |
### ScenarioImportRow
Schema for importing scenario data.
| Field | Type | Description |
| ---------------- | -------------------- | -------------------------------- |
| project_name | str | Project name (required) |
| name | str | Scenario name (required) |
| status | ScenarioStatus | Scenario status (default: DRAFT) |
| start_date | date \| None | Start date |
| end_date | date \| None | End date |
| discount_rate | float \| None | Discount rate |
| currency | str \| None | ISO currency code |
| primary_resource | ResourceType \| None | Primary resource |
| description | str \| None | Scenario description |
| created_at | datetime \| None | Creation timestamp |
| updated_at | datetime \| None | Update timestamp |
## Export Schemas (`schemas/exports.py`)
### ExportFormat
Enumeration for export formats.
- `CSV`: CSV format
- `XLSX`: Excel format
### BaseExportRequest
Base schema for export requests.
| Field | Type | Description |
| ---------------- | ------------ | --------------------------------- |
| format | ExportFormat | Export format (default: CSV) |
| include_metadata | bool | Include metadata (default: False) |
### ProjectExportRequest
Schema for project export requests (inherits BaseExportRequest).
| Field | Type | Description |
| ------- | ---------------------------- | -------------- |
| filters | ProjectExportFilters \| None | Export filters |
### ScenarioExportRequest
Schema for scenario export requests (inherits BaseExportRequest).
| Field | Type | Description |
| ------- | ----------------------------- | -------------- |
| filters | ScenarioExportFilters \| None | Export filters |
### ExportTicket
Schema for export tickets.
| Field | Type | Description |
| -------- | -------------------------------- | ------------- |
| token | str | Export token |
| format | ExportFormat | Export format |
| resource | Literal["projects", "scenarios"] | Resource type |
### ExportResponse
Schema for export responses.
| Field | Type | Description |
| ------ | ------------ | ------------- |
| ticket | ExportTicket | Export ticket |
## Discrepancies Between Data Models and Pydantic Schemas
### Missing Pydantic Schemas
The following SQLAlchemy models do not have corresponding Pydantic schemas:
- `User` - Authentication handled through forms, not API schemas
- `Role` - Role management not exposed via API
- `UserRole` - User-role associations not exposed via API
- `FinancialInput` - Financial inputs managed through scenario context
- `SimulationParameter` - Simulation parameters managed through scenario context
- `PricingSettings` - Pricing settings managed through project context
- `PricingMetalSettings` - Pricing overrides managed through settings context
- `PricingImpuritySettings` - Pricing overrides managed through settings context
- `PerformanceMetric` - Metrics not exposed via API
- `ImportExportLog` - Audit logs not exposed via API
### Schema Differences
- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER`
- **Scenario schemas** include currency normalization validation not present in the model validator
- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing
- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document
### Additional Validation
Pydantic schemas include additional validation beyond SQLAlchemy model constraints:
- Email format validation in auth forms
- Password confirmation matching
- Currency code normalization and validation
- Date range validation (end_date >= start_date)
- Required field validation for imports
- Enum value coercion with aliases for imports
Refer back to the [data model overview](../02_data_model.md) when aligning ORM entities with these schemas.

41
architecture/08_concepts/02_data_model/05_monitoring.md Normal file
View File

@@ -0,0 +1,41 @@
# Monitoring and Auditing
This page documents the monitoring- and auditing-related tables that were previously embedded in `02_data_model.md`. Separating them keeps operational telemetry isolated from the core project/scenario data model reference.
## PerformanceMetric
Tracks API performance metrics used by the monitoring pipeline.
**Table:** `performance_metrics`
| Attribute | Type | Description |
| ---------------- | ------------ | --------------------- |
| id | Integer (PK) | Primary key |
| timestamp | DateTime | Metric timestamp |
| metric_name | String | Metric name |
| value | Float | Metric value |
| labels | String | JSON string of labels |
| endpoint | String | API endpoint |
| method | String | HTTP method |
| status_code | Integer | HTTP status code |
| duration_seconds | Float | Request duration |
## ImportExportLog
Audit log for import and export operations initiated by users.
**Table:** `import_export_logs`
| Attribute | Type | Description |
| ---------- | ----------------------- | ------------------------------------- |
| id | Integer (PK) | Primary key |
| action | String(32) | Action type (preview, commit, export) |
| dataset | String(32) | Dataset type (projects, scenarios) |
| status | String(16) | Operation status |
| filename | String(255) | File name |
| row_count | Integer | Number of rows |
| detail | Text | Additional details |
| user_id | Integer (FK → users.id) | User foreign key |
| created_at | DateTime | Creation timestamp |
These tables power telemetry dashboards and audit trails but are not exposed via public API endpoints.

2
architecture/08_concepts/03_security.md
View File

@@ -10,7 +10,7 @@ All sensitive data is encrypted at rest and in transit to prevent unauthorized a
Role-based access controls (RBAC) are implemented to restrict data access based on user roles and responsibilities.
Also see [Authentication and Authorization](../08_concepts.md#authentication-and-authorization) and the [Data Model](../08_concepts/02_data_model.md#user-roles) sections.
Also see [Authentication and Authorization](../08_concepts.md#authentication-and-authorization) and the [Data Model](../08_concepts/02_data_model/01_sqlalchemy_models.md#userrole) sections.
- Default administrative credentials are provided at deployment time through environment variables (`CALMINER_SEED_ADMIN_EMAIL`, `CALMINER_SEED_ADMIN_USERNAME`, `CALMINER_SEED_ADMIN_PASSWORD`, `CALMINER_SEED_ADMIN_ROLES`). These values are consumed by a shared bootstrap helper on application startup, ensuring mandatory roles and the administrator account exist before any user interaction.
- Operators can request a managed credential reset by setting `CALMINER_SEED_FORCE=true`. On the next startup the helper rotates the admin password and reapplies role assignments, so downstream environments must update stored secrets immediately after the reset.