diff --git a/architecture/08_concepts/02_data_model.md b/architecture/08_concepts/02_data_model.md index 0ed7a53..477986e 100644 --- a/architecture/08_concepts/02_data_model.md +++ b/architecture/08_concepts/02_data_model.md @@ -1,755 +1,23 @@ -# Data Model +# Data Model Overview -This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization. +Calminer’s data model spans several distinct layers: persisted ORM entities, Pydantic schemas used by the API, navigation metadata, shared enumerations, and operational telemetry tables. To make the material easier to scan, the original monolithic document has been split into focused reference pages. -## SQLAlchemy Models +## Reference Structure -### User Management +- [SQLAlchemy Models](./02_data_model/01_sqlalchemy_models.md) — Domain entities that persist projects, scenarios, pricing configuration, snapshots, and supporting records. +- [Navigation Metadata](./02_data_model/02_navigation.md) — Sidebar and menu configuration tables plus seeding/runtime notes. +- [Enumerations](./02_data_model/03_enumerations.md) — Shared enum definitions used across ORM models and schemas. +- [Pydantic Schemas](./02_data_model/04_pydantic.md) — Request/response models, import/export payloads, and validation nuances. +- [Monitoring and Auditing](./02_data_model/05_monitoring.md) — Telemetry and audit tables supporting observability. -#### User +Each detailed page retains the original headings and tables, so existing anchors and references can migrate with minimal disruption. -Represents authenticated platform users with optional elevated privileges. +## How to Use This Overview -**Table:** `users` +- Start with the SQLAlchemy reference when you need to understand persistence concerns or relationships between core domain objects. +- Jump to the Pydantic schemas document when adjusting API payloads or validation logic. +- Consult the enumerations list before introducing new enum values to keep backend and frontend usage aligned. +- Review the navigation metadata page when seeding or modifying the application sidebar. +- Use the monitoring and auditing section to track telemetry fields that drive dashboards and compliance reporting. -| Attribute | Type | Description | -| ------------- | ------------ | ------------------------- | -| id | Integer (PK) | Primary key | -| email | String(255) | Unique email address | -| username | String(128) | Unique username | -| password_hash | String(255) | Argon2 hashed password | -| is_active | Boolean | Account activation status | -| is_superuser | Boolean | Superuser privileges | -| last_login_at | DateTime | Last login timestamp | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `role_assignments`: Many-to-many with Role via UserRole - -#### Role - -Defines user roles for role-based access control (RBAC). - -**Table:** `roles` - -| Attribute | Type | Description | -| ------------ | ------------ | --------------------- | -| id | Integer (PK) | Primary key | -| name | String(64) | Unique role name | -| display_name | String(128) | Human-readable name | -| description | Text | Role description | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `assignments`: One-to-many with UserRole -- `users`: Many-to-many with User (viewonly) - -#### UserRole - -Association between users and roles with assignment metadata. - -**Table:** `user_roles` - -| Attribute | Type | Description | -| ---------- | ----------------------- | -------------------- | -| user_id | Integer (FK → users.id) | User foreign key | -| role_id | Integer (FK → roles.id) | Role foreign key | -| granted_at | DateTime | Assignment timestamp | -| granted_by | Integer (FK → users.id) | Granting user | - -**Relationships:** - -- `user`: Many-to-one with User -- `role`: Many-to-one with Role -- `granted_by_user`: Many-to-one with User - -### Project Management - -#### Project - -Top-level mining project grouping multiple scenarios. - -**Table:** `projects` - -| Attribute | Type | Description | -| ------------------- | ---------------------------------- | --------------------- | -| id | Integer (PK) | Primary key | -| name | String(255) | Unique project name | -| location | String(255) | Project location | -| operation_type | MiningOperationType | Mining operation type | -| description | Text | Project description | -| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `scenarios`: One-to-many with Scenario -- `pricing_settings`: Many-to-one with PricingSettings - -#### Scenario - -A specific configuration of assumptions for a project. - -**Table:** `scenarios` - -| Attribute | Type | Description | -| ---------------- | -------------------------- | ------------------------- | -| id | Integer (PK) | Primary key | -| project_id | Integer (FK → projects.id) | Project foreign key | -| name | String(255) | Scenario name | -| description | Text | Scenario description | -| status | ScenarioStatus | Scenario lifecycle status | -| start_date | Date | Scenario start date | -| end_date | Date | Scenario end date | -| discount_rate | Numeric(5,2) | Discount rate percentage | -| currency | String(3) | ISO currency code | -| primary_resource | ResourceType | Primary resource type | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `project`: Many-to-one with Project -- `financial_inputs`: One-to-many with FinancialInput -- `simulation_parameters`: One-to-many with SimulationParameter - -#### Projects → Scenarios → Profitability Calculations - -Calminer organises feasibility data in a nested hierarchy. A project defines the overarching mining context and exposes a one-to-many `scenarios` collection. Each scenario captures a self-contained assumption set and anchors derived artefacts such as financial inputs, simulation parameters, and profitability snapshots. Profitability calculations execute at the scenario scope; when triggered, the workflow in `services/calculations.py` persists a `ScenarioProfitability` record and can optionally roll results up to project level by creating a `ProjectProfitability` snapshot. Consumers typically surface the most recent metrics via the `latest_profitability` helpers on both ORM models. - -| Layer | ORM models | Pydantic schema(s) | Key relationships | -| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | -| Project | `models.project.Project` | `schemas.project.ProjectRead` | `Project.scenarios`, `Project.profitability_snapshots`, `Project.latest_profitability` | -| Scenario | `models.scenario.Scenario` | `schemas.scenario.ScenarioRead` | `Scenario.project`, `Scenario.profitability_snapshots`, `Scenario.latest_profitability` | -| Profitability calculations | `models.profitability_snapshot.ProjectProfitability`, `models.profitability_snapshot.ScenarioProfitability` | `schemas.calculations.ProfitabilityCalculationRequest`, `schemas.calculations.ProfitabilityCalculationResult` | Persisted via `services.calculations.calculate_profitability`; aggregates scenario metrics into project snapshots | - -Detailed CRUD endpoint behaviour for projects and scenarios is documented in `calminer-docs/api/README.md`. - -#### FinancialInput - -Line-item financial assumption attached to a scenario. - -**Table:** `financial_inputs` - -| Attribute | Type | Description | -| -------------- | --------------------------- | -------------------------- | -| id | Integer (PK) | Primary key | -| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | -| name | String(255) | Input name | -| category | FinancialCategory | Financial category | -| cost_bucket | CostBucket | Cost bucket classification | -| amount | Numeric(18,2) | Monetary amount | -| currency | String(3) | ISO currency code | -| effective_date | Date | Effective date | -| notes | Text | Additional notes | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `scenario`: Many-to-one with Scenario - -### Project and Scenario Models - -#### ProjectCapexSnapshot - -Project-level snapshot capturing aggregated initial capital expenditure metrics. - -**Table:** `project_capex_snapshots` - -| Attribute | Type | Description | -| ---------------------- | --------------------------------- | ------------------------------------------------------- | -| id | Integer (PK) | Primary key | -| project_id | Integer (FK → projects.id) | Associated project | -| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | -| calculation_source | String(64), nullable | Originating workflow identifier (UI, API, etc.) | -| calculated_at | DateTime | Timestamp the calculation completed | -| currency_code | String(3), nullable | Currency for totals | -| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency | -| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage | -| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount | -| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency | -| component_count | Integer, nullable | Number of normalized components captured | -| payload | JSON, nullable | Serialized component breakdown and calculation metadata | -| created_at | DateTime | Record creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `project`: Many-to-one with Project -- `created_by`: Many-to-one with User (nullable) - -#### ScenarioCapexSnapshot - -Scenario-level snapshot storing detailed capex results. - -**Table:** `scenario_capex_snapshots` - -| Attribute | Type | Description | -| ---------------------- | --------------------------------- | ------------------------------------------------------- | -| id | Integer (PK) | Primary key | -| scenario_id | Integer (FK → scenarios.id) | Associated scenario | -| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | -| calculation_source | String(64), nullable | Originating workflow identifier | -| calculated_at | DateTime | Timestamp the calculation completed | -| currency_code | String(3), nullable | Currency for totals | -| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency | -| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage | -| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount | -| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency | -| component_count | Integer, nullable | Number of normalized components captured | -| payload | JSON, nullable | Serialized component breakdown and calculation metadata | -| created_at | DateTime | Record creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `scenario`: Many-to-one with Scenario -- `created_by`: Many-to-one with User (nullable) - -#### ProjectOpexSnapshot - -Project-level snapshot persisting recurring opex metrics. - -**Table:** `project_opex_snapshots` - -| Attribute | Type | Description | -| ------------------------ | --------------------------------- | ------------------------------------------------------- | -| id | Integer (PK) | Primary key | -| project_id | Integer (FK → projects.id) | Associated project | -| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | -| calculation_source | String(64), nullable | Originating workflow identifier | -| calculated_at | DateTime | Timestamp the calculation completed | -| currency_code | String(3), nullable | Currency for totals | -| overall_annual | Numeric(18,2), nullable | Total annual opex | -| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon | -| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon | -| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline | -| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied | -| apply_escalation | Boolean | Flag indicating whether escalation was applied | -| component_count | Integer, nullable | Number of normalized components captured | -| payload | JSON, nullable | Serialized component breakdown and calculation metadata | -| created_at | DateTime | Record creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `project`: Many-to-one with Project -- `created_by`: Many-to-one with User (nullable) - -#### ScenarioOpexSnapshot - -Scenario-level snapshot persisting recurring opex metrics. - -**Table:** `scenario_opex_snapshots` - -| Attribute | Type | Description | -| ------------------------ | --------------------------------- | ------------------------------------------------------- | -| id | Integer (PK) | Primary key | -| scenario_id | Integer (FK → scenarios.id) | Associated scenario | -| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | -| calculation_source | String(64), nullable | Originating workflow identifier | -| calculated_at | DateTime | Timestamp the calculation completed | -| currency_code | String(3), nullable | Currency for totals | -| overall_annual | Numeric(18,2), nullable | Total annual opex | -| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon | -| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon | -| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline | -| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied | -| apply_escalation | Boolean | Flag indicating whether escalation was applied | -| component_count | Integer, nullable | Number of normalized components captured | -| payload | JSON, nullable | Serialized component breakdown and calculation metadata | -| created_at | DateTime | Record creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `scenario`: Many-to-one with Scenario -- `created_by`: Many-to-one with User (nullable) - -#### SimulationParameter - -Probability distribution settings for scenario simulations. - -**Table:** `simulation_parameters` - -| Attribute | Type | Description | -| ------------------ | --------------------------- | ------------------------ | -| id | Integer (PK) | Primary key | -| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | -| name | String(255) | Parameter name | -| distribution | DistributionType | Distribution type | -| variable | StochasticVariable | Stochastic variable type | -| resource_type | ResourceType | Resource type | -| mean_value | Numeric(18,4) | Mean value | -| standard_deviation | Numeric(18,4) | Standard deviation | -| minimum_value | Numeric(18,4) | Minimum value | -| maximum_value | Numeric(18,4) | Maximum value | -| unit | String(32) | Unit of measurement | -| configuration | JSON | Additional configuration | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `scenario`: Many-to-one with Scenario - -### Pricing Configuration - -#### PricingSettings - -Persisted pricing defaults applied to scenario evaluations. - -**Table:** `pricing_settings` - -| Attribute | Type | Description | -| ------------------------ | ------------- | ----------------------------- | -| id | Integer (PK) | Primary key | -| name | String(128) | Unique settings name | -| slug | String(64) | Unique slug identifier | -| description | Text | Settings description | -| default_currency | String(3) | Default ISO currency code | -| default_payable_pct | Numeric(5,2) | Default payable percentage | -| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage | -| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent | -| metadata_payload | JSON | Additional metadata | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `metal_overrides`: One-to-many with PricingMetalSettings -- `impurity_overrides`: One-to-many with PricingImpuritySettings -- `projects`: One-to-many with Project - -#### PricingMetalSettings - -Contract-specific overrides for a particular metal. - -**Table:** `pricing_metal_settings` - -| Attribute | Type | Description | -| ------------------------ | ---------------------------------- | ---------------------------- | -| id | Integer (PK) | Primary key | -| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | -| metal_code | String(32) | Metal code | -| payable_pct | Numeric(5,2) | Payable percentage | -| moisture_threshold_pct | Numeric(5,2) | Moisture threshold | -| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty | -| data | JSON | Additional data | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `pricing_settings`: Many-to-one with PricingSettings - -#### PricingImpuritySettings - -Impurity penalty thresholds associated with pricing settings. - -**Table:** `pricing_impurity_settings` - -| Attribute | Type | Description | -| ------------------- | ---------------------------------- | ---------------------------- | -| id | Integer (PK) | Primary key | -| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | -| impurity_code | String(32) | Impurity code | -| threshold_ppm | Numeric(14,4) | Threshold in ppm | -| penalty_per_ppm | Numeric(14,4) | Penalty per ppm | -| notes | Text | Additional notes | -| created_at | DateTime | Creation timestamp | -| updated_at | DateTime | Last update timestamp | - -**Relationships:** - -- `pricing_settings`: Many-to-one with PricingSettings - -### Monitoring and Auditing - -#### PerformanceMetric - -Performance metrics for monitoring system health. - -**Table:** `performance_metrics` - -| Attribute | Type | Description | -| ---------------- | ------------ | --------------------- | -| id | Integer (PK) | Primary key | -| timestamp | DateTime | Metric timestamp | -| metric_name | String | Metric name | -| value | Float | Metric value | -| labels | String | JSON string of labels | -| endpoint | String | API endpoint | -| method | String | HTTP method | -| status_code | Integer | HTTP status code | -| duration_seconds | Float | Request duration | - -#### ImportExportLog - -Audit log for import and export operations. - -**Table:** `import_export_logs` - -| Attribute | Type | Description | -| ---------- | ----------------------- | ------------------------------------- | -| id | Integer (PK) | Primary key | -| action | String(32) | Action type (preview, commit, export) | -| dataset | String(32) | Dataset type (projects, scenarios) | -| status | String(16) | Operation status | -| filename | String(255) | File name | -| row_count | Integer | Number of rows | -| detail | Text | Additional details | -| user_id | Integer (FK → users.id) | User foreign key | -| created_at | DateTime | Creation timestamp | - -## Enumerations - -### MiningOperationType - -Supported mining operation categories. - -- `OPEN_PIT`: Open pit mining -- `UNDERGROUND`: Underground mining -- `IN_SITU_LEACH`: In-situ leaching -- `PLACER`: Placer mining -- `QUARRY`: Quarry operations -- `MOUNTAINTOP_REMOVAL`: Mountaintop removal -- `OTHER`: Other operations - -### ScenarioStatus - -Lifecycle states for project scenarios. - -- `DRAFT`: Draft status -- `ACTIVE`: Active status -- `ARCHIVED`: Archived status - -### FinancialCategory - -Enumeration of cost and revenue classifications. - -- `CAPITAL_EXPENDITURE`: Capital expenditures -- `OPERATING_EXPENDITURE`: Operating expenditures -- `REVENUE`: Revenue -- `CONTINGENCY`: Contingency -- `OTHER`: Other - -### DistributionType - -Supported stochastic distribution families for simulations. - -- `NORMAL`: Normal distribution -- `TRIANGULAR`: Triangular distribution -- `UNIFORM`: Uniform distribution -- `LOGNORMAL`: Lognormal distribution -- `CUSTOM`: Custom distribution - -### ResourceType - -Primary consumables and resources used in mining operations. - -- `DIESEL`: Diesel fuel -- `ELECTRICITY`: Electrical power -- `WATER`: Process water -- `EXPLOSIVES`: Blasting agents -- `REAGENTS`: Processing reagents -- `LABOR`: Direct labor -- `EQUIPMENT_HOURS`: Equipment operating hours -- `TAILINGS_CAPACITY`: Tailings storage - -### CostBucket - -Granular cost buckets aligned with project accounting. - -- `CAPITAL_INITIAL`: Initial capital -- `CAPITAL_SUSTAINING`: Sustaining capital -- `OPERATING_FIXED`: Fixed operating costs -- `OPERATING_VARIABLE`: Variable operating costs -- `MAINTENANCE`: Maintenance costs -- `RECLAMATION`: Reclamation costs -- `ROYALTIES`: Royalties -- `GENERAL_ADMIN`: General and administrative - -### StochasticVariable - -Domain variables that typically require probabilistic modelling. - -- `ORE_GRADE`: Ore grade variability -- `RECOVERY_RATE`: Metallurgical recovery -- `METAL_PRICE`: Commodity price -- `OPERATING_COST`: Operating cost per tonne -- `CAPITAL_COST`: Capital cost -- `DISCOUNT_RATE`: Discount rate -- `THROUGHPUT`: Plant throughput - -## Pydantic Schemas - -### Authentication Schemas (`schemas/auth.py`) - -#### RegistrationForm - -Form model for user registration. - -| Field | Type | Validation | -| ---------------- | ---- | ------------------------------- | -| username | str | 3-128 characters | -| email | str | Valid email format, 5-255 chars | -| password | str | 8-256 characters | -| confirm_password | str | Must match password | - -#### LoginForm - -Form model for user login. - -| Field | Type | Validation | -| -------- | ---- | ---------------- | -| username | str | 1-255 characters | -| password | str | 1-256 characters | - -#### PasswordResetRequestForm - -Form model for password reset request. - -| Field | Type | Validation | -| ----- | ---- | ------------------------------- | -| email | str | Valid email format, 5-255 chars | - -#### PasswordResetForm - -Form model for password reset. - -| Field | Type | Validation | -| ---------------- | ---- | ------------------- | -| token | str | Required | -| password | str | 8-256 characters | -| confirm_password | str | Must match password | - -### Project Schemas (`schemas/project.py`) - -#### ProjectBase - -Base schema for project data. - -| Field | Type | Description | -| -------------- | ------------------- | --------------------- | -| name | str | Project name | -| location | str \| None | Project location | -| operation_type | MiningOperationType | Mining operation type | -| description | str \| None | Project description | - -#### ProjectCreate - -Schema for creating projects (inherits ProjectBase). - -#### ProjectUpdate - -Schema for updating projects. - -| Field | Type | Description | -| -------------- | --------------------------- | --------------------- | -| name | str \| None | Project name | -| location | str \| None | Project location | -| operation_type | MiningOperationType \| None | Mining operation type | -| description | str \| None | Project description | - -#### ProjectRead - -Schema for reading projects (inherits ProjectBase). - -| Field | Type | Description | -| ---------- | -------- | ------------------ | -| id | int | Project ID | -| created_at | datetime | Creation timestamp | -| updated_at | datetime | Update timestamp | - -### Scenario Schemas (`schemas/scenario.py`) - -#### ScenarioBase - -Base schema for scenario data. - -| Field | Type | Description | -| ---------------- | -------------------- | -------------------------------- | -| name | str | Scenario name | -| description | str \| None | Scenario description | -| status | ScenarioStatus | Scenario status (default: DRAFT) | -| start_date | date \| None | Start date | -| end_date | date \| None | End date | -| discount_rate | float \| None | Discount rate | -| currency | str \| None | ISO currency code | -| primary_resource | ResourceType \| None | Primary resource | - -#### ScenarioCreate - -Schema for creating scenarios (inherits ScenarioBase). - -#### ScenarioUpdate - -Schema for updating scenarios. - -| Field | Type | Description | -| ---------------- | ---------------------- | -------------------- | -| name | str \| None | Scenario name | -| description | str \| None | Scenario description | -| status | ScenarioStatus \| None | Scenario status | -| start_date | date \| None | Start date | -| end_date | date \| None | End date | -| discount_rate | float \| None | Discount rate | -| currency | str \| None | ISO currency code | -| primary_resource | ResourceType \| None | Primary resource | - -#### ScenarioRead - -Schema for reading scenarios (inherits ScenarioBase). - -| Field | Type | Description | -| ---------- | -------- | ------------------ | -| id | int | Scenario ID | -| project_id | int | Project ID | -| created_at | datetime | Creation timestamp | -| updated_at | datetime | Update timestamp | - -#### ScenarioComparisonRequest - -Schema for scenario comparison requests. - -| Field | Type | Description | -| ------------ | --------- | --------------------------------------- | -| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) | - -#### ScenarioComparisonResponse - -Schema for scenario comparison responses. - -| Field | Type | Description | -| ---------- | ------------------ | --------------------- | -| project_id | int | Project ID | -| scenarios | list[ScenarioRead] | List of scenario data | - -### Import Schemas (`schemas/imports.py`) - -#### ProjectImportRow - -Schema for importing project data. - -| Field | Type | Description | -| -------------- | ------------------- | ----------------------- | -| name | str | Project name (required) | -| location | str \| None | Project location | -| operation_type | MiningOperationType | Mining operation type | -| description | str \| None | Project description | -| created_at | datetime \| None | Creation timestamp | -| updated_at | datetime \| None | Update timestamp | - -#### ScenarioImportRow - -Schema for importing scenario data. - -| Field | Type | Description | -| ---------------- | -------------------- | -------------------------------- | -| project_name | str | Project name (required) | -| name | str | Scenario name (required) | -| status | ScenarioStatus | Scenario status (default: DRAFT) | -| start_date | date \| None | Start date | -| end_date | date \| None | End date | -| discount_rate | float \| None | Discount rate | -| currency | str \| None | ISO currency code | -| primary_resource | ResourceType \| None | Primary resource | -| description | str \| None | Scenario description | -| created_at | datetime \| None | Creation timestamp | -| updated_at | datetime \| None | Update timestamp | - -### Export Schemas (`schemas/exports.py`) - -#### ExportFormat - -Enumeration for export formats. - -- `CSV`: CSV format -- `XLSX`: Excel format - -#### BaseExportRequest - -Base schema for export requests. - -| Field | Type | Description | -| ---------------- | ------------ | --------------------------------- | -| format | ExportFormat | Export format (default: CSV) | -| include_metadata | bool | Include metadata (default: False) | - -#### ProjectExportRequest - -Schema for project export requests (inherits BaseExportRequest). - -| Field | Type | Description | -| ------- | ---------------------------- | -------------- | -| filters | ProjectExportFilters \| None | Export filters | - -#### ScenarioExportRequest - -Schema for scenario export requests (inherits BaseExportRequest). - -| Field | Type | Description | -| ------- | ----------------------------- | -------------- | -| filters | ScenarioExportFilters \| None | Export filters | - -#### ExportTicket - -Schema for export tickets. - -| Field | Type | Description | -| -------- | -------------------------------- | ------------- | -| token | str | Export token | -| format | ExportFormat | Export format | -| resource | Literal["projects", "scenarios"] | Resource type | - -#### ExportResponse - -Schema for export responses. - -| Field | Type | Description | -| ------ | ------------ | ------------- | -| ticket | ExportTicket | Export ticket | - -## Discrepancies Between Data Models and Pydantic Schemas - -### Missing Pydantic Schemas - -The following SQLAlchemy models do not have corresponding Pydantic schemas: - -- `User` - Authentication handled through forms, not API schemas -- `Role` - Role management not exposed via API -- `UserRole` - User-role associations not exposed via API -- `FinancialInput` - Financial inputs managed through scenario context -- `SimulationParameter` - Simulation parameters managed through scenario context -- `PricingSettings` - Pricing settings managed through project context -- `PricingMetalSettings` - Pricing overrides managed through settings context -- `PricingImpuritySettings` - Pricing overrides managed through settings context -- `PerformanceMetric` - Metrics not exposed via API -- `ImportExportLog` - Audit logs not exposed via API - -### Schema Differences - -- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER` -- **Scenario schemas** include currency normalization validation not present in the model validator -- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing -- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document - -### Additional Validation - -Pydantic schemas include additional validation beyond SQLAlchemy model constraints: - -- Email format validation in auth forms -- Password confirmation matching -- Currency code normalization and validation -- Date range validation (end_date >= start_date) -- Required field validation for imports -- Enum value coercion with aliases for imports - -This documentation reflects the current implementation as of the latest development cycle. +Cross-links between these documents mirror the previous inline references. Update any external links to point at the new files during your next documentation touchpoint. diff --git a/architecture/08_concepts/02_data_model/01_sqlalchemy_models.md b/architecture/08_concepts/02_data_model/01_sqlalchemy_models.md new file mode 100644 index 0000000..9125a68 --- /dev/null +++ b/architecture/08_concepts/02_data_model/01_sqlalchemy_models.md @@ -0,0 +1,369 @@ +# SQLAlchemy Models + +This reference describes the primary SQLAlchemy ORM models that underpin Calminer. It mirrors the original hierarchy from `02_data_model.md`, extracted so each domain area can evolve independently. See the [data model overview](../02_data_model.md) for context and navigation. + +## User Management + +### User + +Represents authenticated platform users with optional elevated privileges. + +**Table:** `users` + +| Attribute | Type | Description | +| ------------- | ------------ | ------------------------- | +| id | Integer (PK) | Primary key | +| email | String(255) | Unique email address | +| username | String(128) | Unique username | +| password_hash | String(255) | Argon2 hashed password | +| is_active | Boolean | Account activation status | +| is_superuser | Boolean | Superuser privileges | +| last_login_at | DateTime | Last login timestamp | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `role_assignments`: Many-to-many with Role via UserRole + +### Role + +Defines user roles for role-based access control (RBAC). + +**Table:** `roles` + +| Attribute | Type | Description | +| ------------ | ------------ | --------------------- | +| id | Integer (PK) | Primary key | +| name | String(64) | Unique role name | +| display_name | String(128) | Human-readable name | +| description | Text | Role description | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `assignments`: One-to-many with UserRole +- `users`: Many-to-many with User (viewonly) + +### UserRole + +Association between users and roles with assignment metadata. + +**Table:** `user_roles` + +| Attribute | Type | Description | +| ---------- | ----------------------- | -------------------- | +| user_id | Integer (FK → users.id) | User foreign key | +| role_id | Integer (FK → roles.id) | Role foreign key | +| granted_at | DateTime | Assignment timestamp | +| granted_by | Integer (FK → users.id) | Granting user | + +**Relationships:** + +- `user`: Many-to-one with User +- `role`: Many-to-one with Role +- `granted_by_user`: Many-to-one with User + +## Project Management + +### Project + +Top-level mining project grouping multiple scenarios. + +**Table:** `projects` + +| Attribute | Type | Description | +| ------------------- | ---------------------------------- | --------------------- | +| id | Integer (PK) | Primary key | +| name | String(255) | Unique project name | +| location | String(255) | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | Text | Project description | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenarios`: One-to-many with Scenario +- `pricing_settings`: Many-to-one with PricingSettings + +### Scenario + +A specific configuration of assumptions for a project. + +**Table:** `scenarios` + +| Attribute | Type | Description | +| ---------------- | -------------------------- | ------------------------- | +| id | Integer (PK) | Primary key | +| project_id | Integer (FK → projects.id) | Project foreign key | +| name | String(255) | Scenario name | +| description | Text | Scenario description | +| status | ScenarioStatus | Scenario lifecycle status | +| start_date | Date | Scenario start date | +| end_date | Date | Scenario end date | +| discount_rate | Numeric(5,2) | Discount rate percentage | +| currency | String(3) | ISO currency code | +| primary_resource | ResourceType | Primary resource type | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `project`: Many-to-one with Project +- `financial_inputs`: One-to-many with FinancialInput +- `simulation_parameters`: One-to-many with SimulationParameter + +### Projects → Scenarios → Profitability Calculations + +Calminer organises feasibility data in a nested hierarchy. A project defines the overarching mining context and exposes a one-to-many `scenarios` collection. Each scenario captures a self-contained assumption set and anchors derived artefacts such as financial inputs, simulation parameters, and profitability snapshots. Profitability calculations execute at the scenario scope; when triggered, the workflow in `services/calculations.py` persists a `ScenarioProfitability` record and can optionally roll results up to project level by creating a `ProjectProfitability` snapshot. Consumers typically surface the most recent metrics via the `latest_profitability` helpers on both ORM models. + +| Layer | ORM models | Pydantic schema(s) | Key relationships | +| -------------------------- | ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| Project | `models.project.Project` | `schemas.project.ProjectRead` | `Project.scenarios`, `Project.profitability_snapshots`, `Project.latest_profitability` | +| Scenario | `models.scenario.Scenario` | `schemas.scenario.ScenarioRead` | `Scenario.project`, `Scenario.profitability_snapshots`, `Scenario.latest_profitability` | +| Profitability calculations | `models.profitability_snapshot.ProjectProfitability`, `models.profitability_snapshot.ScenarioProfitability` | `schemas.calculations.ProfitabilityCalculationRequest`, `schemas.calculations.ProfitabilityCalculationResult` | Persisted via `services.calculations.calculate_profitability`; aggregates scenario metrics into project snapshots | + +Detailed CRUD endpoint behaviour for projects and scenarios is documented in `calminer-docs/api/README.md`. + +### FinancialInput + +Line-item financial assumption attached to a scenario. + +**Table:** `financial_inputs` + +| Attribute | Type | Description | +| -------------- | --------------------------- | -------------------------- | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | +| name | String(255) | Input name | +| category | FinancialCategory | Financial category | +| cost_bucket | CostBucket | Cost bucket classification | +| amount | Numeric(18,2) | Monetary amount | +| currency | String(3) | ISO currency code | +| effective_date | Date | Effective date | +| notes | Text | Additional notes | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario + +## Project and Scenario Snapshots + +### ProjectCapexSnapshot + +Project-level snapshot capturing aggregated initial capital expenditure metrics. + +**Table:** `project_capex_snapshots` + +| Attribute | Type | Description | +| ---------------------- | --------------------------------- | ------------------------------------------------------- | +| id | Integer (PK) | Primary key | +| project_id | Integer (FK → projects.id) | Associated project | +| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | +| calculation_source | String(64), nullable | Originating workflow identifier (UI, API, etc.) | +| calculated_at | DateTime | Timestamp the calculation completed | +| currency_code | String(3), nullable | Currency for totals | +| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency | +| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage | +| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount | +| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency | +| component_count | Integer, nullable | Number of normalized components captured | +| payload | JSON, nullable | Serialized component breakdown and calculation metadata | +| created_at | DateTime | Record creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `project`: Many-to-one with Project +- `created_by`: Many-to-one with User (nullable) + +### ScenarioCapexSnapshot + +Scenario-level snapshot storing detailed capex results. + +**Table:** `scenario_capex_snapshots` + +| Attribute | Type | Description | +| ---------------------- | --------------------------------- | ------------------------------------------------------- | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Associated scenario | +| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | +| calculation_source | String(64), nullable | Originating workflow identifier | +| calculated_at | DateTime | Timestamp the calculation completed | +| currency_code | String(3), nullable | Currency for totals | +| total_capex | Numeric(18,2), nullable | Aggregated capex before contingency | +| contingency_pct | Numeric(12,6), nullable | Applied contingency percentage | +| contingency_amount | Numeric(18,2), nullable | Monetary contingency amount | +| total_with_contingency | Numeric(18,2), nullable | Capex total after contingency | +| component_count | Integer, nullable | Number of normalized components captured | +| payload | JSON, nullable | Serialized component breakdown and calculation metadata | +| created_at | DateTime | Record creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario +- `created_by`: Many-to-one with User (nullable) + +### ProjectOpexSnapshot + +Project-level snapshot persisting recurring opex metrics. + +**Table:** `project_opex_snapshots` + +| Attribute | Type | Description | +| ------------------------ | --------------------------------- | ------------------------------------------------------- | +| id | Integer (PK) | Primary key | +| project_id | Integer (FK → projects.id) | Associated project | +| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | +| calculation_source | String(64), nullable | Originating workflow identifier | +| calculated_at | DateTime | Timestamp the calculation completed | +| currency_code | String(3), nullable | Currency for totals | +| overall_annual | Numeric(18,2), nullable | Total annual opex | +| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon | +| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon | +| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline | +| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied | +| apply_escalation | Boolean | Flag indicating whether escalation was applied | +| component_count | Integer, nullable | Number of normalized components captured | +| payload | JSON, nullable | Serialized component breakdown and calculation metadata | +| created_at | DateTime | Record creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `project`: Many-to-one with Project +- `created_by`: Many-to-one with User (nullable) + +### ScenarioOpexSnapshot + +Scenario-level snapshot persisting recurring opex metrics. + +**Table:** `scenario_opex_snapshots` + +| Attribute | Type | Description | +| ------------------------ | --------------------------------- | ------------------------------------------------------- | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Associated scenario | +| created_by_id | Integer (FK → users.id, nullable) | User that triggered persistence | +| calculation_source | String(64), nullable | Originating workflow identifier | +| calculated_at | DateTime | Timestamp the calculation completed | +| currency_code | String(3), nullable | Currency for totals | +| overall_annual | Numeric(18,2), nullable | Total annual opex | +| escalated_total | Numeric(18,2), nullable | Escalated cost across the evaluation horizon | +| annual_average | Numeric(18,2), nullable | Average annual cost over the horizon | +| evaluation_horizon_years | Integer, nullable | Number of years included in the timeline | +| escalation_pct | Numeric(12,6), nullable | Escalation percentage applied | +| apply_escalation | Boolean | Flag indicating whether escalation was applied | +| component_count | Integer, nullable | Number of normalized components captured | +| payload | JSON, nullable | Serialized component breakdown and calculation metadata | +| created_at | DateTime | Record creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario +- `created_by`: Many-to-one with User (nullable) + +### SimulationParameter + +Probability distribution settings for scenario simulations. + +**Table:** `simulation_parameters` + +| Attribute | Type | Description | +| ------------------ | --------------------------- | ------------------------ | +| id | Integer (PK) | Primary key | +| scenario_id | Integer (FK → scenarios.id) | Scenario foreign key | +| name | String(255) | Parameter name | +| distribution | DistributionType | Distribution type | +| variable | StochasticVariable | Stochastic variable type | +| resource_type | ResourceType | Resource type | +| mean_value | Numeric(18,4) | Mean value | +| standard_deviation | Numeric(18,4) | Standard deviation | +| minimum_value | Numeric(18,4) | Minimum value | +| maximum_value | Numeric(18,4) | Maximum value | +| unit | String(32) | Unit of measurement | +| configuration | JSON | Additional configuration | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `scenario`: Many-to-one with Scenario + +## Pricing Configuration + +### PricingSettings + +Persisted pricing defaults applied to scenario evaluations. + +**Table:** `pricing_settings` + +| Attribute | Type | Description | +| ------------------------ | ------------- | ----------------------------- | +| id | Integer (PK) | Primary key | +| name | String(128) | Unique settings name | +| slug | String(64) | Unique slug identifier | +| description | Text | Settings description | +| default_currency | String(3) | Default ISO currency code | +| default_payable_pct | Numeric(5,2) | Default payable percentage | +| moisture_threshold_pct | Numeric(5,2) | Moisture threshold percentage | +| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty per percent | +| metadata_payload | JSON | Additional metadata | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `metal_overrides`: One-to-many with PricingMetalSettings +- `impurity_overrides`: One-to-many with PricingImpuritySettings +- `projects`: One-to-many with Project + +### PricingMetalSettings + +Contract-specific overrides for a particular metal. + +**Table:** `pricing_metal_settings` + +| Attribute | Type | Description | +| ------------------------ | ---------------------------------- | ---------------------------- | +| id | Integer (PK) | Primary key | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | +| metal_code | String(32) | Metal code | +| payable_pct | Numeric(5,2) | Payable percentage | +| moisture_threshold_pct | Numeric(5,2) | Moisture threshold | +| moisture_penalty_per_pct | Numeric(14,4) | Moisture penalty | +| data | JSON | Additional data | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `pricing_settings`: Many-to-one with PricingSettings + +### PricingImpuritySettings + +Impurity penalty thresholds associated with pricing settings. + +**Table:** `pricing_impurity_settings` + +| Attribute | Type | Description | +| ------------------- | ---------------------------------- | ---------------------------- | +| id | Integer (PK) | Primary key | +| pricing_settings_id | Integer (FK → pricing_settings.id) | Pricing settings foreign key | +| impurity_code | String(32) | Impurity code | +| threshold_ppm | Numeric(14,4) | Threshold in ppm | +| penalty_per_ppm | Numeric(14,4) | Penalty per ppm | +| notes | Text | Additional notes | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `pricing_settings`: Many-to-one with PricingSettings diff --git a/architecture/08_concepts/02_data_model/02_navigation.md b/architecture/08_concepts/02_data_model/02_navigation.md new file mode 100644 index 0000000..86816c2 --- /dev/null +++ b/architecture/08_concepts/02_data_model/02_navigation.md @@ -0,0 +1,65 @@ +# Navigation Metadata + +This page details the navigation metadata tables that drive the Calminer sidebar and top-level menus. It was split from `02_data_model.md` to isolate runtime navigation behaviour from the broader ORM catalogue. + +## NavigationGroup + +Represents a logical grouping of navigation links shown in the UI sidebar or header. + +**Table:** `navigation_groups` + +| Attribute | Type | Description | +| ------------ | ------------ | ----------------------------------------------- | +| id | Integer (PK) | Primary key | +| slug | String(64) | Unique slug identifier | +| title | String(128) | Display title | +| description | Text | Optional descriptive text | +| match_prefix | String(255) | URL prefix used to auto-expand in UI | +| weighting | Integer | Ordering weight | +| icon_name | String(64) | Optional icon identifier | +| feature_flag | String(64) | Optional feature flag for conditional rendering | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `links`: One-to-many with NavigationLink + +## NavigationLink + +Individual navigation entry that may have an associated feature flag or role restriction. + +**Table:** `navigation_links` + +| Attribute | Type | Description | +| ------------- | ----------------------------------- | ---------------------------------------------------- | +| id | Integer (PK) | Primary key | +| group_id | Integer (FK → navigation_groups.id) | Parent navigation group | +| parent_id | Integer (FK → navigation_links.id) | Optional parent link (for nested menus) | +| slug | String(64) | Unique slug identifier | +| title | String(128) | Display title | +| description | Text | Optional descriptive text | +| href | String(255) | Static fallback URL | +| match_prefix | String(255) | URL prefix for expansion | +| feature_flag | String(64) | Optional feature flag for conditional rendering | +| required_role | String(64) | Optional role required to view the link | +| weighting | Integer | Ordering weight within the parent scope | +| icon_name | String(64) | Optional icon identifier | +| metadata | JSON | Additional configuration (e.g. legacy route aliases) | +| created_at | DateTime | Creation timestamp | +| updated_at | DateTime | Last update timestamp | + +**Relationships:** + +- `group`: Many-to-one with NavigationGroup +- `parent`: Many-to-one self-reference +- `children`: One-to-many self-reference + +## Seeding and Runtime Consumption + +- **Seed script:** `scripts/init_db.py` provisions baseline groups and links, including nested scenario calculators and role-gated admin entries. +- **Service layer:** `services/navigation.py` maps ORM entities into response DTOs, resolves contextual URLs for projects and scenarios, and applies feature flag as well as role filters. +- **API exposure:** `routes/navigation.py` serves the `/navigation/sidebar` endpoint, combining seeded data with runtime context to produce navigation trees consumed by the frontend. +- **Testing:** `tests/services/test_navigation_service.py` covers URL resolution and child filtering logic, while `tests/integration/test_navigation_sidebar_calculations.py` verifies scenario calculator entries in the API payload. + +Refer to the [navigation service documentation](../../../../api/README.md#navigation) for endpoint-level behaviour. diff --git a/architecture/08_concepts/02_data_model/03_enumerations.md b/architecture/08_concepts/02_data_model/03_enumerations.md new file mode 100644 index 0000000..1f4017b --- /dev/null +++ b/architecture/08_concepts/02_data_model/03_enumerations.md @@ -0,0 +1,83 @@ +# Enumerations + +This page catalogs the enumerations shared by Calminer ORM models and Pydantic schemas. The entries mirror the original content from `02_data_model.md` so enum updates can be maintained independently of the broader data model reference. + +## MiningOperationType + +Supported mining operation categories. + +- `OPEN_PIT`: Open pit mining +- `UNDERGROUND`: Underground mining +- `IN_SITU_LEACH`: In-situ leaching +- `PLACER`: Placer mining +- `QUARRY`: Quarry operations +- `MOUNTAINTOP_REMOVAL`: Mountaintop removal +- `OTHER`: Other operations + +## ScenarioStatus + +Lifecycle states for project scenarios. + +- `DRAFT`: Draft status +- `ACTIVE`: Active status +- `ARCHIVED`: Archived status + +## FinancialCategory + +Enumeration of cost and revenue classifications. + +- `CAPITAL_EXPENDITURE`: Capital expenditures +- `OPERATING_EXPENDITURE`: Operating expenditures +- `REVENUE`: Revenue +- `CONTINGENCY`: Contingency +- `OTHER`: Other + +## DistributionType + +Supported stochastic distribution families for simulations. + +- `NORMAL`: Normal distribution +- `TRIANGULAR`: Triangular distribution +- `UNIFORM`: Uniform distribution +- `LOGNORMAL`: Lognormal distribution +- `CUSTOM`: Custom distribution + +## ResourceType + +Primary consumables and resources used in mining operations. + +- `DIESEL`: Diesel fuel +- `ELECTRICITY`: Electrical power +- `WATER`: Process water +- `EXPLOSIVES`: Blasting agents +- `REAGENTS`: Processing reagents +- `LABOR`: Direct labor +- `EQUIPMENT_HOURS`: Equipment operating hours +- `TAILINGS_CAPACITY`: Tailings storage + +## CostBucket + +Granular cost buckets aligned with project accounting. + +- `CAPITAL_INITIAL`: Initial capital +- `CAPITAL_SUSTAINING`: Sustaining capital +- `OPERATING_FIXED`: Fixed operating costs +- `OPERATING_VARIABLE`: Variable operating costs +- `MAINTENANCE`: Maintenance costs +- `RECLAMATION`: Reclamation costs +- `ROYALTIES`: Royalties +- `GENERAL_ADMIN`: General and administrative + +## StochasticVariable + +Domain variables that typically require probabilistic modelling. + +- `ORE_GRADE`: Ore grade variability +- `RECOVERY_RATE`: Metallurgical recovery +- `METAL_PRICE`: Commodity price +- `OPERATING_COST`: Operating cost per tonne +- `CAPITAL_COST`: Capital cost +- `DISCOUNT_RATE`: Discount rate +- `THROUGHPUT`: Plant throughput + +Refer back to the [data model overview](../02_data_model.md) when choosing the appropriate enum for new features. diff --git a/architecture/08_concepts/02_data_model/04_pydantic.md b/architecture/08_concepts/02_data_model/04_pydantic.md new file mode 100644 index 0000000..1864fa8 --- /dev/null +++ b/architecture/08_concepts/02_data_model/04_pydantic.md @@ -0,0 +1,267 @@ +# Pydantic Schemas + +This page documents the Pydantic models used for request/response validation across the Calminer API surface. It was extracted from `02_data_model.md` so API-facing contracts can evolve separately from the ORM layer. + +## Authentication Schemas (`schemas/auth.py`) + +### RegistrationForm + +Form model for user registration. + +| Field | Type | Validation | +| ---------------- | ---- | ------------------------------- | +| username | str | 3-128 characters | +| email | str | Valid email format, 5-255 chars | +| password | str | 8-256 characters | +| confirm_password | str | Must match password | + +### LoginForm + +Form model for user login. + +| Field | Type | Validation | +| -------- | ---- | ---------------- | +| username | str | 1-255 characters | +| password | str | 1-256 characters | + +### PasswordResetRequestForm + +Form model for password reset request. + +| Field | Type | Validation | +| ----- | ---- | ------------------------------- | +| email | str | Valid email format, 5-255 chars | + +### PasswordResetForm + +Form model for password reset. + +| Field | Type | Validation | +| ---------------- | ---- | ------------------- | +| token | str | Required | +| password | str | 8-256 characters | +| confirm_password | str | Must match password | + +## Project Schemas (`schemas/project.py`) + +### ProjectBase + +Base schema for project data. + +| Field | Type | Description | +| -------------- | ------------------- | --------------------- | +| name | str | Project name | +| location | str \| None | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | str \| None | Project description | + +### ProjectCreate + +Schema for creating projects (inherits ProjectBase). + +### ProjectUpdate + +Schema for updating projects. + +| Field | Type | Description | +| -------------- | --------------------------- | --------------------- | +| name | str \| None | Project name | +| location | str \| None | Project location | +| operation_type | MiningOperationType \| None | Mining operation type | +| description | str \| None | Project description | + +### ProjectRead + +Schema for reading projects (inherits ProjectBase). + +| Field | Type | Description | +| ---------- | -------- | ------------------ | +| id | int | Project ID | +| created_at | datetime | Creation timestamp | +| updated_at | datetime | Update timestamp | + +## Scenario Schemas (`schemas/scenario.py`) + +### ScenarioBase + +Base schema for scenario data. + +| Field | Type | Description | +| ---------------- | -------------------- | -------------------------------- | +| name | str | Scenario name | +| description | str \| None | Scenario description | +| status | ScenarioStatus | Scenario status (default: DRAFT) | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | + +### ScenarioCreate + +Schema for creating scenarios (inherits ScenarioBase). + +### ScenarioUpdate + +Schema for updating scenarios. + +| Field | Type | Description | +| ---------------- | ---------------------- | -------------------- | +| name | str \| None | Scenario name | +| description | str \| None | Scenario description | +| status | ScenarioStatus \| None | Scenario status | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | + +### ScenarioRead + +Schema for reading scenarios (inherits ScenarioBase). + +| Field | Type | Description | +| ---------- | -------- | ------------------ | +| id | int | Scenario ID | +| project_id | int | Project ID | +| created_at | datetime | Creation timestamp | +| updated_at | datetime | Update timestamp | + +### ScenarioComparisonRequest + +Schema for scenario comparison requests. + +| Field | Type | Description | +| ------------ | --------- | --------------------------------------- | +| scenario_ids | list[int] | List of scenario IDs (minimum 2 unique) | + +### ScenarioComparisonResponse + +Schema for scenario comparison responses. + +| Field | Type | Description | +| ---------- | ------------------ | --------------------- | +| project_id | int | Project ID | +| scenarios | list[ScenarioRead] | List of scenario data | + +## Import Schemas (`schemas/imports.py`) + +### ProjectImportRow + +Schema for importing project data. + +| Field | Type | Description | +| -------------- | ------------------- | ----------------------- | +| name | str | Project name (required) | +| location | str \| None | Project location | +| operation_type | MiningOperationType | Mining operation type | +| description | str \| None | Project description | +| created_at | datetime \| None | Creation timestamp | +| updated_at | datetime \| None | Update timestamp | + +### ScenarioImportRow + +Schema for importing scenario data. + +| Field | Type | Description | +| ---------------- | -------------------- | -------------------------------- | +| project_name | str | Project name (required) | +| name | str | Scenario name (required) | +| status | ScenarioStatus | Scenario status (default: DRAFT) | +| start_date | date \| None | Start date | +| end_date | date \| None | End date | +| discount_rate | float \| None | Discount rate | +| currency | str \| None | ISO currency code | +| primary_resource | ResourceType \| None | Primary resource | +| description | str \| None | Scenario description | +| created_at | datetime \| None | Creation timestamp | +| updated_at | datetime \| None | Update timestamp | + +## Export Schemas (`schemas/exports.py`) + +### ExportFormat + +Enumeration for export formats. + +- `CSV`: CSV format +- `XLSX`: Excel format + +### BaseExportRequest + +Base schema for export requests. + +| Field | Type | Description | +| ---------------- | ------------ | --------------------------------- | +| format | ExportFormat | Export format (default: CSV) | +| include_metadata | bool | Include metadata (default: False) | + +### ProjectExportRequest + +Schema for project export requests (inherits BaseExportRequest). + +| Field | Type | Description | +| ------- | ---------------------------- | -------------- | +| filters | ProjectExportFilters \| None | Export filters | + +### ScenarioExportRequest + +Schema for scenario export requests (inherits BaseExportRequest). + +| Field | Type | Description | +| ------- | ----------------------------- | -------------- | +| filters | ScenarioExportFilters \| None | Export filters | + +### ExportTicket + +Schema for export tickets. + +| Field | Type | Description | +| -------- | -------------------------------- | ------------- | +| token | str | Export token | +| format | ExportFormat | Export format | +| resource | Literal["projects", "scenarios"] | Resource type | + +### ExportResponse + +Schema for export responses. + +| Field | Type | Description | +| ------ | ------------ | ------------- | +| ticket | ExportTicket | Export ticket | + +## Discrepancies Between Data Models and Pydantic Schemas + +### Missing Pydantic Schemas + +The following SQLAlchemy models do not have corresponding Pydantic schemas: + +- `User` - Authentication handled through forms, not API schemas +- `Role` - Role management not exposed via API +- `UserRole` - User-role associations not exposed via API +- `FinancialInput` - Financial inputs managed through scenario context +- `SimulationParameter` - Simulation parameters managed through scenario context +- `PricingSettings` - Pricing settings managed through project context +- `PricingMetalSettings` - Pricing overrides managed through settings context +- `PricingImpuritySettings` - Pricing overrides managed through settings context +- `PerformanceMetric` - Metrics not exposed via API +- `ImportExportLog` - Audit logs not exposed via API + +### Schema Differences + +- **Project schemas** include `operation_type` as required enum, while the model allows null but defaults to `OTHER` +- **Scenario schemas** include currency normalization validation not present in the model validator +- **Import schemas** include extensive validation and coercion logic for CSV/Excel data parsing +- **Export schemas** reference filter classes (`ProjectExportFilters`, `ScenarioExportFilters`) not defined in this document + +### Additional Validation + +Pydantic schemas include additional validation beyond SQLAlchemy model constraints: + +- Email format validation in auth forms +- Password confirmation matching +- Currency code normalization and validation +- Date range validation (end_date >= start_date) +- Required field validation for imports +- Enum value coercion with aliases for imports + +Refer back to the [data model overview](../02_data_model.md) when aligning ORM entities with these schemas. diff --git a/architecture/08_concepts/02_data_model/05_monitoring.md b/architecture/08_concepts/02_data_model/05_monitoring.md new file mode 100644 index 0000000..f5c2c29 --- /dev/null +++ b/architecture/08_concepts/02_data_model/05_monitoring.md @@ -0,0 +1,41 @@ +# Monitoring and Auditing + +This page documents the monitoring- and auditing-related tables that were previously embedded in `02_data_model.md`. Separating them keeps operational telemetry isolated from the core project/scenario data model reference. + +## PerformanceMetric + +Tracks API performance metrics used by the monitoring pipeline. + +**Table:** `performance_metrics` + +| Attribute | Type | Description | +| ---------------- | ------------ | --------------------- | +| id | Integer (PK) | Primary key | +| timestamp | DateTime | Metric timestamp | +| metric_name | String | Metric name | +| value | Float | Metric value | +| labels | String | JSON string of labels | +| endpoint | String | API endpoint | +| method | String | HTTP method | +| status_code | Integer | HTTP status code | +| duration_seconds | Float | Request duration | + +## ImportExportLog + +Audit log for import and export operations initiated by users. + +**Table:** `import_export_logs` + +| Attribute | Type | Description | +| ---------- | ----------------------- | ------------------------------------- | +| id | Integer (PK) | Primary key | +| action | String(32) | Action type (preview, commit, export) | +| dataset | String(32) | Dataset type (projects, scenarios) | +| status | String(16) | Operation status | +| filename | String(255) | File name | +| row_count | Integer | Number of rows | +| detail | Text | Additional details | +| user_id | Integer (FK → users.id) | User foreign key | +| created_at | DateTime | Creation timestamp | + +These tables power telemetry dashboards and audit trails but are not exposed via public API endpoints. diff --git a/architecture/08_concepts/03_security.md b/architecture/08_concepts/03_security.md index 1e0256d..61fff2c 100644 --- a/architecture/08_concepts/03_security.md +++ b/architecture/08_concepts/03_security.md @@ -10,7 +10,7 @@ All sensitive data is encrypted at rest and in transit to prevent unauthorized a Role-based access controls (RBAC) are implemented to restrict data access based on user roles and responsibilities. -Also see [Authentication and Authorization](../08_concepts.md#authentication-and-authorization) and the [Data Model](../08_concepts/02_data_model.md#user-roles) sections. +Also see [Authentication and Authorization](../08_concepts.md#authentication-and-authorization) and the [Data Model](../08_concepts/02_data_model/01_sqlalchemy_models.md#userrole) sections. - Default administrative credentials are provided at deployment time through environment variables (`CALMINER_SEED_ADMIN_EMAIL`, `CALMINER_SEED_ADMIN_USERNAME`, `CALMINER_SEED_ADMIN_PASSWORD`, `CALMINER_SEED_ADMIN_ROLES`). These values are consumed by a shared bootstrap helper on application startup, ensuring mandatory roles and the administrator account exist before any user interaction. - Operators can request a managed credential reset by setting `CALMINER_SEED_FORCE=true`. On the next startup the helper rotates the admin password and reapplies role assignments, so downstream environments must update stored secrets immediately after the reset.