calminer-docs/architecture/08_concepts/02_data_model.md

Data Model

This document describes the current data models implemented in the Calminer application, including SQLAlchemy ORM models and Pydantic schemas for API validation and serialization.

SQLAlchemy Models

User Management

User

Represents authenticated platform users with optional elevated privileges.

Table: users

Attribute	Type	Description
id	Integer (PK)	Primary key
email	String(255)	Unique email address
username	String(128)	Unique username
password_hash	String(255)	Argon2 hashed password
is_active	Boolean	Account activation status
is_superuser	Boolean	Superuser privileges
last_login_at	DateTime	Last login timestamp
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• role_assignments: Many-to-many with Role via UserRole
• roles: Many-to-many with Role (viewonly)

Role

Role encapsulating a set of permissions.

Table: roles

Attribute	Type	Description
id	Integer (PK)	Primary key
name	String(64)	Unique role name
display_name	String(128)	Human-readable name
description	Text	Role description
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• assignments: One-to-many with UserRole
• users: Many-to-many with User (viewonly)

UserRole

Association between users and roles with assignment metadata.

Table: user_roles

Attribute	Type	Description
user_id	Integer (FK → users.id)	User foreign key
role_id	Integer (FK → roles.id)	Role foreign key
granted_at	DateTime	Assignment timestamp
granted_by	Integer (FK → users.id)	Granting user

Relationships:

• user: Many-to-one with User
• role: Many-to-one with Role
• granted_by_user: Many-to-one with User

Project Management

Project

Top-level mining project grouping multiple scenarios.

Table: projects

Attribute	Type	Description
id	Integer (PK)	Primary key
name	String(255)	Unique project name
location	String(255)	Project location
operation_type	MiningOperationType	Mining operation type
description	Text	Project description
pricing_settings_id	Integer (FK → pricing_settings.id)	Pricing settings
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• scenarios: One-to-many with Scenario
• pricing_settings: Many-to-one with PricingSettings

Scenario

A specific configuration of assumptions for a project.

Table: scenarios

Attribute	Type	Description
id	Integer (PK)	Primary key
project_id	Integer (FK → projects.id)	Project foreign key
name	String(255)	Scenario name
description	Text	Scenario description
status	ScenarioStatus	Scenario lifecycle status
start_date	Date	Scenario start date
end_date	Date	Scenario end date
discount_rate	Numeric(5,2)	Discount rate percentage
currency	String(3)	ISO currency code
primary_resource	ResourceType	Primary resource type
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• project: Many-to-one with Project
• financial_inputs: One-to-many with FinancialInput
• simulation_parameters: One-to-many with SimulationParameter

FinancialInput

Line-item financial assumption attached to a scenario.

Table: financial_inputs

Attribute	Type	Description
id	Integer (PK)	Primary key
scenario_id	Integer (FK → scenarios.id)	Scenario foreign key
name	String(255)	Input name
category	FinancialCategory	Financial category
cost_bucket	CostBucket	Cost bucket classification
amount	Numeric(18,2)	Monetary amount
currency	String(3)	ISO currency code
effective_date	Date	Effective date
notes	Text	Additional notes
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• scenario: Many-to-one with Scenario

SimulationParameter

Probability distribution settings for scenario simulations.

Table: simulation_parameters

Attribute	Type	Description
id	Integer (PK)	Primary key
scenario_id	Integer (FK → scenarios.id)	Scenario foreign key
name	String(255)	Parameter name
distribution	DistributionType	Distribution type
variable	StochasticVariable	Stochastic variable type
resource_type	ResourceType	Resource type
mean_value	Numeric(18,4)	Mean value
standard_deviation	Numeric(18,4)	Standard deviation
minimum_value	Numeric(18,4)	Minimum value
maximum_value	Numeric(18,4)	Maximum value
unit	String(32)	Unit of measurement
configuration	JSON	Additional configuration
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• scenario: Many-to-one with Scenario

Pricing Configuration

PricingSettings

Persisted pricing defaults applied to scenario evaluations.

Table: pricing_settings

Attribute	Type	Description
id	Integer (PK)	Primary key
name	String(128)	Unique settings name
slug	String(64)	Unique slug identifier
description	Text	Settings description
default_currency	String(3)	Default ISO currency code
default_payable_pct	Numeric(5,2)	Default payable percentage
moisture_threshold_pct	Numeric(5,2)	Moisture threshold percentage
moisture_penalty_per_pct	Numeric(14,4)	Moisture penalty per percent
metadata_payload	JSON	Additional metadata
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• metal_overrides: One-to-many with PricingMetalSettings
• impurity_overrides: One-to-many with PricingImpuritySettings
• projects: One-to-many with Project

PricingMetalSettings

Contract-specific overrides for a particular metal.

Table: pricing_metal_settings

Attribute	Type	Description
id	Integer (PK)	Primary key
pricing_settings_id	Integer (FK → pricing_settings.id)	Pricing settings foreign key
metal_code	String(32)	Metal code
payable_pct	Numeric(5,2)	Payable percentage
moisture_threshold_pct	Numeric(5,2)	Moisture threshold
moisture_penalty_per_pct	Numeric(14,4)	Moisture penalty
data	JSON	Additional data
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• pricing_settings: Many-to-one with PricingSettings

PricingImpuritySettings

Impurity penalty thresholds associated with pricing settings.

Table: pricing_impurity_settings

Attribute	Type	Description
id	Integer (PK)	Primary key
pricing_settings_id	Integer (FK → pricing_settings.id)	Pricing settings foreign key
impurity_code	String(32)	Impurity code
threshold_ppm	Numeric(14,4)	Threshold in ppm
penalty_per_ppm	Numeric(14,4)	Penalty per ppm
notes	Text	Additional notes
created_at	DateTime	Creation timestamp
updated_at	DateTime	Last update timestamp

Relationships:

• pricing_settings: Many-to-one with PricingSettings

Monitoring and Auditing

PerformanceMetric

Performance metrics for monitoring system health.

Table: performance_metrics

Attribute	Type	Description
id	Integer (PK)	Primary key
timestamp	DateTime	Metric timestamp
metric_name	String	Metric name
value	Float	Metric value
labels	String	JSON string of labels
endpoint	String	API endpoint
method	String	HTTP method
status_code	Integer	HTTP status code
duration_seconds	Float	Request duration

ImportExportLog

Audit log for import and export operations.

Table: import_export_logs

Attribute	Type	Description
id	Integer (PK)	Primary key
action	String(32)	Action type (preview, commit, export)
dataset	String(32)	Dataset type (projects, scenarios)
status	String(16)	Operation status
filename	String(255)	File name
row_count	Integer	Number of rows
detail	Text	Additional details
user_id	Integer (FK → users.id)	User foreign key
created_at	DateTime	Creation timestamp

Enumerations

MiningOperationType

Supported mining operation categories.

• OPEN_PIT: Open pit mining
• UNDERGROUND: Underground mining
• IN_SITU_LEACH: In-situ leaching
• PLACER: Placer mining
• QUARRY: Quarry operations
• MOUNTAINTOP_REMOVAL: Mountaintop removal
• OTHER: Other operations

ScenarioStatus

Lifecycle states for project scenarios.

• DRAFT: Draft status
• ACTIVE: Active status
• ARCHIVED: Archived status

FinancialCategory

Enumeration of cost and revenue classifications.

• CAPITAL_EXPENDITURE: Capital expenditures
• OPERATING_EXPENDITURE: Operating expenditures
• REVENUE: Revenue
• CONTINGENCY: Contingency
• OTHER: Other

DistributionType

Supported stochastic distribution families for simulations.

• NORMAL: Normal distribution
• TRIANGULAR: Triangular distribution
• UNIFORM: Uniform distribution
• LOGNORMAL: Lognormal distribution
• CUSTOM: Custom distribution

ResourceType

Primary consumables and resources used in mining operations.

• DIESEL: Diesel fuel
• ELECTRICITY: Electrical power
• WATER: Process water
• EXPLOSIVES: Blasting agents
• REAGENTS: Processing reagents
• LABOR: Direct labor
• EQUIPMENT_HOURS: Equipment operating hours
• TAILINGS_CAPACITY: Tailings storage

CostBucket

Granular cost buckets aligned with project accounting.

• CAPITAL_INITIAL: Initial capital
• CAPITAL_SUSTAINING: Sustaining capital
• OPERATING_FIXED: Fixed operating costs
• OPERATING_VARIABLE: Variable operating costs
• MAINTENANCE: Maintenance costs
• RECLAMATION: Reclamation costs
• ROYALTIES: Royalties
• GENERAL_ADMIN: General and administrative

StochasticVariable

Domain variables that typically require probabilistic modelling.

• ORE_GRADE: Ore grade variability
• RECOVERY_RATE: Metallurgical recovery
• METAL_PRICE: Commodity price
• OPERATING_COST: Operating cost per tonne
• CAPITAL_COST: Capital cost
• DISCOUNT_RATE: Discount rate
• THROUGHPUT: Plant throughput

Pydantic Schemas

Authentication Schemas (schemas/auth.py)

RegistrationForm

Form model for user registration.

Field	Type	Validation
username	str	3-128 characters
email	str	Valid email format, 5-255 chars
password	str	8-256 characters
confirm_password	str	Must match password

LoginForm

Form model for user login.

Field	Type	Validation
username	str	1-255 characters
password	str	1-256 characters

PasswordResetRequestForm

Form model for password reset request.

Field	Type	Validation
email	str	Valid email format, 5-255 chars

PasswordResetForm

Form model for password reset.

Field	Type	Validation
token	str	Required
password	str	8-256 characters
confirm_password	str	Must match password

Project Schemas (schemas/project.py)

ProjectBase

Base schema for project data.

Field	Type	Description
name	str	Project name
location	str | None	Project location
operation_type	MiningOperationType	Mining operation type
description	str | None	Project description

ProjectCreate

Schema for creating projects (inherits ProjectBase).

ProjectUpdate

Schema for updating projects.

Field	Type	Description
name	str | None	Project name
location	str | None	Project location
operation_type	MiningOperationType | None	Mining operation type
description	str | None	Project description

ProjectRead

Schema for reading projects (inherits ProjectBase).

Field	Type	Description
id	int	Project ID
created_at	datetime	Creation timestamp
updated_at	datetime	Update timestamp

Scenario Schemas (schemas/scenario.py)

ScenarioBase

Base schema for scenario data.

Field	Type	Description
name	str	Scenario name
description	str | None	Scenario description
status	ScenarioStatus	Scenario status (default: DRAFT)
start_date	date | None	Start date
end_date	date | None	End date
discount_rate	float | None	Discount rate
currency	str | None	ISO currency code
primary_resource	ResourceType | None	Primary resource

ScenarioCreate

Schema for creating scenarios (inherits ScenarioBase).

ScenarioUpdate

Schema for updating scenarios.

Field	Type	Description
name	str | None	Scenario name
description	str | None	Scenario description
status	ScenarioStatus | None	Scenario status
start_date	date | None	Start date
end_date	date | None	End date
discount_rate	float | None	Discount rate
currency	str | None	ISO currency code
primary_resource	ResourceType | None	Primary resource

ScenarioRead

Schema for reading scenarios (inherits ScenarioBase).

Field	Type	Description
id	int	Scenario ID
project_id	int	Project ID
created_at	datetime	Creation timestamp
updated_at	datetime	Update timestamp

ScenarioComparisonRequest

Schema for scenario comparison requests.

Field	Type	Description
scenario_ids	list[int]	List of scenario IDs (minimum 2 unique)

ScenarioComparisonResponse

Schema for scenario comparison responses.

Field	Type	Description
project_id	int	Project ID
scenarios	list[ScenarioRead]	List of scenario data

Import Schemas (schemas/imports.py)

ProjectImportRow

Schema for importing project data.

Field	Type	Description
name	str	Project name (required)
location	str | None	Project location
operation_type	MiningOperationType	Mining operation type
description	str | None	Project description
created_at	datetime | None	Creation timestamp
updated_at	datetime | None	Update timestamp

ScenarioImportRow

Schema for importing scenario data.

Field	Type	Description
project_name	str	Project name (required)
name	str	Scenario name (required)
status	ScenarioStatus	Scenario status (default: DRAFT)
start_date	date | None	Start date
end_date	date | None	End date
discount_rate	float | None	Discount rate
currency	str | None	ISO currency code
primary_resource	ResourceType | None	Primary resource
description	str | None	Scenario description
created_at	datetime | None	Creation timestamp
updated_at	datetime | None	Update timestamp

Export Schemas (schemas/exports.py)

ExportFormat

Enumeration for export formats.

• CSV: CSV format
• XLSX: Excel format

BaseExportRequest

Base schema for export requests.

Field	Type	Description
format	ExportFormat	Export format (default: CSV)
include_metadata	bool	Include metadata (default: False)

ProjectExportRequest

Schema for project export requests (inherits BaseExportRequest).

Field	Type	Description
filters	ProjectExportFilters | None	Export filters

ScenarioExportRequest

Schema for scenario export requests (inherits BaseExportRequest).

Field	Type	Description
filters	ScenarioExportFilters | None	Export filters

ExportTicket

Schema for export tickets.

Field	Type	Description
token	str	Export token
format	ExportFormat	Export format
resource	Literal["projects", "scenarios"]	Resource type

ExportResponse

Schema for export responses.

Field	Type	Description
ticket	ExportTicket	Export ticket

Discrepancies Between Data Models and Pydantic Schemas

Missing Pydantic Schemas

The following SQLAlchemy models do not have corresponding Pydantic schemas:

• User - Authentication handled through forms, not API schemas
• Role - Role management not exposed via API
• UserRole - User-role associations not exposed via API
• FinancialInput - Financial inputs managed through scenario context
• SimulationParameter - Simulation parameters managed through scenario context
• PricingSettings - Pricing settings managed through project context
• PricingMetalSettings - Pricing overrides managed through settings context
• PricingImpuritySettings - Pricing overrides managed through settings context
• PerformanceMetric - Metrics not exposed via API
• ImportExportLog - Audit logs not exposed via API

Schema Differences

• Project schemas include operation_type as required enum, while the model allows null but defaults to OTHER
• Scenario schemas include currency normalization validation not present in the model validator
• Import schemas include extensive validation and coercion logic for CSV/Excel data parsing
• Export schemas reference filter classes (ProjectExportFilters, ScenarioExportFilters) not defined in this document

Additional Validation

Pydantic schemas include additional validation beyond SQLAlchemy model constraints:

• Email format validation in auth forms
• Password confirmation matching
• Currency code normalization and validation
• Date range validation (end_date >= start_date)
• Required field validation for imports
• Enum value coercion with aliases for imports

This documentation reflects the current implementation as of the latest development cycle.