245 lines
11 KiB
Markdown
245 lines
11 KiB
Markdown
# Cross-cutting Concepts
|
|
|
|
CalMiner aims to provide a comprehensive platform for mining project scenario analysis, enabling stakeholders to make informed decisions based on data-driven insights. This document outlines the key cross-cutting concepts that underpin the architecture of the CalMiner system.
|
|
|
|
## Table of Contents
|
|
|
|
- [Cross-cutting Concepts](#cross-cutting-concepts)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Multitenancy](#multitenancy)
|
|
- [Mandators](#mandators)
|
|
- [Data Isolation](#data-isolation)
|
|
- [Configuration Management](#configuration-management)
|
|
- [Data Model](#data-model)
|
|
- [Security](#security)
|
|
- [Authentication and Authorization](#authentication-and-authorization)
|
|
- [Authentication](#authentication)
|
|
- [Authorization](#authorization)
|
|
- [Session Management](#session-management)
|
|
- [Token Expiration](#token-expiration)
|
|
- [Multi-Factor Authentication (MFA)](#multi-factor-authentication-mfa)
|
|
- [Error Handling and Logging](#error-handling-and-logging)
|
|
- [Centralized Logging](#centralized-logging)
|
|
- [Error Categorization](#error-categorization)
|
|
- [Graceful Degradation](#graceful-degradation)
|
|
- [Alerting and Monitoring](#alerting-and-monitoring)
|
|
- [Log Retention Policies](#log-retention-policies)
|
|
- [Performance Optimization](#performance-optimization)
|
|
- [Caching](#caching)
|
|
- [Asynchronous Processing](#asynchronous-processing)
|
|
- [Database Indexing](#database-indexing)
|
|
- [Load Balancing](#load-balancing)
|
|
- [Scalability Strategies](#scalability-strategies)
|
|
- [Horizontal Scaling](#horizontal-scaling)
|
|
- [Vertical Scaling](#vertical-scaling)
|
|
- [Microservices Architecture](#microservices-architecture)
|
|
- [Load Testing](#load-testing)
|
|
- [User Interface Design Principles](#user-interface-design-principles)
|
|
- [Notification System](#notification-system)
|
|
- [Notification Channels](#notification-channels)
|
|
- [Configurable Preferences](#configurable-preferences)
|
|
- [Event Triggers](#event-triggers)
|
|
- [Delivery Reliability](#delivery-reliability)
|
|
- [Audit Trail](#audit-trail)
|
|
- [Data Integration](#data-integration)
|
|
- [API Integration](#api-integration)
|
|
- [Data Ingestion](#data-ingestion)
|
|
- [Data Transformation](#data-transformation)
|
|
- [Data Synchronization](#data-synchronization)
|
|
- [Error Handling](#error-handling)
|
|
- [Reporting and Analytics](#reporting-and-analytics)
|
|
|
|
## Multitenancy
|
|
|
|
The Calminer system is designed to support multitenancy, allowing multiple independent users or organizations (tenants) to share the same application instance while maintaining data isolation and security.
|
|
|
|
### Mandators
|
|
|
|
Each tenant is represented by a Mandator entity, which encapsulates all data and configurations specific to that tenant.
|
|
|
|
### Data Isolation
|
|
|
|
The architecture ensures that data belonging to different tenants is isolated, preventing unauthorized access between tenants.
|
|
|
|
### Configuration Management
|
|
|
|
Each tenant can have customized settings and preferences, managed through the Mandator entity.
|
|
|
|
## Data Model
|
|
|
|
The system employs a relational data model to manage and store information efficiently. Core entities include Users, Projects, Scenarios, Financial Inputs, Simulation Parameters, Pricing Settings, and supporting metadata tables. Relationships are enforced using foreign keys with cascading deletes where appropriate (e.g., scenarios cascade from projects, financial inputs cascade from scenarios).
|
|
|
|
Domain-specific enumerations are mapped to PostgreSQL enum types and mirrored in the ORM layer via `models.enums`:
|
|
|
|
- `MiningOperationType` — categorises projects (open pit, underground, in-situ leach, etc.).
|
|
- `ScenarioStatus` — tracks lifecycle state (draft, active, archived).
|
|
- `FinancialCategory` and `CostBucket` — classify financial inputs for reporting granularity.
|
|
- `DistributionType` and `StochasticVariable` — describe probabilistic simulation parameters.
|
|
- `ResourceType` — annotates primary resource consumption for scenarios and stochastic parameters.
|
|
|
|
These enums back the relevant model columns through named SQLAlchemy `Enum` definitions referencing the same PostgreSQL type names, ensuring parity between application logic and database constraints.
|
|
|
|
Schema creation and baseline seed data are handled by the idempotent initializer (`scripts/init_db.py`). On every application startup the initializer guarantees that:
|
|
|
|
1. Enum types exist exactly once.
|
|
2. Tables for authentication, pricing, and mining domain entities are present.
|
|
3. Default roles, administrator account, pricing settings, and representative demo projects/scenarios/financial inputs are available for immediate use.
|
|
|
|
Developers can reset to a clean slate with `scripts/reset_db.py`, which safely drops managed tables and enum types in non-production environments before rerunning the initializer.
|
|
|
|
A detailed [Data Model](08_concepts/02_data_model.md) documentation is available. Discounted cash-flow metrics (NPV, IRR, Payback) referenced by the economic portion of the data model are described in the [Financial Metrics Specification](../specifications/financial_metrics.md), while stochastic extensions leverage the Monte Carlo engine documented in the [Monte Carlo Simulation Specification](../specifications/monte_carlo_simulation.md).
|
|
|
|
All data interactions are handled through the [Data Access Layer](05_building_block_view.md#data-access-layer), ensuring consistency and integrity across operations.
|
|
|
|
## Security
|
|
|
|
The Calminer system incorporates robust security measures to protect sensitive data and ensure compliance with different industry and regulatory standards, such as GDPR or ISO/IEC 27001.
|
|
|
|
Within the system, security is enforced through multiple layers, including data encryption, access controls, audit logging, vulnerability management, and compliance monitoring.
|
|
|
|
A comprehensive [Security Concept](08_concepts/03_security.md) is being developed as well.
|
|
|
|
## Authentication and Authorization
|
|
|
|
The system utilizes a secure authentication mechanism to verify user identities and manage access to resources.
|
|
|
|
### Authentication
|
|
|
|
Users authenticate using secure methods, such as OAuth 2.0 or JWT tokens, to ensure the legitimacy of access requests.
|
|
|
|
### Authorization
|
|
|
|
Access to resources is controlled through role-based permissions, ensuring users can only access data and functionalities relevant to their roles.
|
|
|
|
### Session Management
|
|
|
|
User sessions are managed securely, with mechanisms in place to prevent session hijacking and ensure session integrity.
|
|
|
|
### Token Expiration
|
|
|
|
Access tokens have a limited lifespan and are refreshed periodically to enhance security.
|
|
|
|
### Multi-Factor Authentication (MFA)
|
|
|
|
The system supports MFA to provide an additional layer of security during user authentication.
|
|
|
|
## Error Handling and Logging
|
|
|
|
The system implements a comprehensive error handling and logging strategy to ensure reliability and facilitate troubleshooting.
|
|
|
|
### Centralized Logging
|
|
|
|
All application logs are centralized to a logging service, enabling efficient monitoring and analysis of system behavior.
|
|
|
|
### Error Categorization
|
|
|
|
Errors are categorized into different types (e.g., client errors, server errors) to facilitate targeted handling and resolution.
|
|
|
|
### Graceful Degradation
|
|
|
|
The system is designed to handle errors gracefully, providing meaningful feedback to users without exposing sensitive information.
|
|
|
|
### Alerting and Monitoring
|
|
|
|
The system includes mechanisms for alerting administrators about critical errors and monitoring system health.
|
|
|
|
### Log Retention Policies
|
|
|
|
Logs are retained for a specified period to comply with regulatory requirements and support forensic analysis.
|
|
|
|
## Performance Optimization
|
|
|
|
The system is designed with performance optimization strategies to ensure efficient operation and responsiveness.
|
|
|
|
### Caching
|
|
|
|
Frequently accessed data is cached to reduce database load and improve response times.
|
|
|
|
### Asynchronous Processing
|
|
|
|
Time-consuming operations are handled asynchronously to prevent blocking user interactions.
|
|
|
|
### Database Indexing
|
|
|
|
Appropriate indexing strategies are implemented to optimize query performance.
|
|
|
|
### Load Balancing
|
|
|
|
Traffic is distributed across multiple servers to ensure optimal resource utilization and minimize latency.
|
|
|
|
## Scalability Strategies
|
|
|
|
The system is architected to scale efficiently in response to varying workloads and user demands.
|
|
|
|
### Horizontal Scaling
|
|
|
|
The system can add more machines or instances to handle increased load, distributing traffic and processing across multiple nodes.
|
|
|
|
### Vertical Scaling
|
|
|
|
Individual components can be upgraded with more resources (CPU, memory) to improve performance without changing the overall architecture.
|
|
|
|
### Microservices Architecture
|
|
|
|
The system is designed as a collection of loosely coupled services, allowing for independent scaling of different components based on demand.
|
|
|
|
### Load Testing
|
|
|
|
Regular load testing is conducted to identify bottlenecks and ensure the system can handle expected traffic patterns.
|
|
|
|
## User Interface Design Principles
|
|
|
|
To ensure a consistent and user-friendly experience, the Calminer system adheres to established user interface design principles. A detailed [user interface (UI)](08_concepts/08_ui_design.md) design principles document is available for reference.
|
|
|
|
## Notification System
|
|
|
|
The system includes a notification mechanism to inform users about important events and updates.
|
|
|
|
### Notification Channels
|
|
|
|
The system supports multiple channels for notifications, including email, SMS, and in-app alerts.
|
|
|
|
### Configurable Preferences
|
|
|
|
Users can customize their notification preferences, selecting the types of notifications they wish to receive and the channels through which they are delivered.
|
|
|
|
### Event Triggers
|
|
|
|
Notifications are triggered by specific events within the system, such as project updates, data processing completions, or system alerts.
|
|
|
|
### Delivery Reliability
|
|
|
|
The notification system is designed to ensure reliable delivery of messages, with retry mechanisms in place for failed attempts.
|
|
|
|
### Audit Trail
|
|
|
|
A record of all sent notifications is maintained for auditing purposes and to track user engagement.
|
|
|
|
## Data Integration
|
|
|
|
The Calminer system is designed to integrate seamlessly with various external data sources and services.
|
|
|
|
### API Integration
|
|
|
|
The system can connect to external APIs to fetch or send data as needed.
|
|
|
|
### Data Ingestion
|
|
|
|
Mechanisms are in place to ingest data from various sources, including databases, file systems, and streaming platforms.
|
|
|
|
### Data Transformation
|
|
|
|
The system can transform data into the required format for processing and analysis.
|
|
|
|
### Data Synchronization
|
|
|
|
Regular synchronization processes ensure that data remains consistent across all integrated systems.
|
|
|
|
### Error Handling
|
|
|
|
Robust error handling mechanisms are implemented to manage integration failures and ensure data integrity.
|
|
|
|
## Reporting and Analytics
|
|
|
|
The Calminer system includes comprehensive reporting and analytics capabilities to support data-driven decision-making. Detailed [Reporting and Analytics](08_concepts/11_reporting_analytics.md) documentation is available.
|