12 KiB
Cross-cutting Concepts
CalMiner aims to provide a comprehensive platform for mining project scenario analysis, enabling stakeholders to make informed decisions based on data-driven insights. This document outlines the key cross-cutting concepts that underpin the architecture of the CalMiner system.
Table of Contents
- Cross-cutting Concepts
Multitenancy
The Calminer system is designed to support multitenancy, allowing multiple independent users or organizations (tenants) to share the same application instance while maintaining data isolation and security.
Mandators
Each tenant is represented by a Mandator entity, which encapsulates all data and configurations specific to that tenant.
Data Isolation
The architecture ensures that data belonging to different tenants is isolated, preventing unauthorized access between tenants.
Configuration Management
Each tenant can have customized settings and preferences, managed through the Mandator entity.
Data Model
The system employs a relational data model to manage and store information efficiently. Core entities include Users, Projects, Scenarios, Financial Inputs, Simulation Parameters, Pricing Settings, and supporting metadata tables. Relationships are enforced using foreign keys with cascading deletes where appropriate (e.g., scenarios cascade from projects, financial inputs cascade from scenarios).
Domain-specific enumerations are mapped to PostgreSQL enum types and mirrored in the ORM layer via models.enums:
MiningOperationType— categorises projects (open pit, underground, in-situ leach, etc.).ScenarioStatus— tracks lifecycle state (draft, active, archived).FinancialCategoryandCostBucket— classify financial inputs for reporting granularity.DistributionTypeandStochasticVariable— describe probabilistic simulation parameters.ResourceType— annotates primary resource consumption for scenarios and stochastic parameters.
These enums back the relevant model columns through named SQLAlchemy Enum definitions referencing the same PostgreSQL type names, ensuring parity between application logic and database constraints.
Schema creation and baseline seed data are handled by the idempotent initializer (scripts/init_db.py). On every application startup the initializer guarantees that:
- Enum types exist exactly once.
- Tables for authentication, pricing, and mining domain entities are present.
- Default roles, administrator account, pricing settings, and representative demo projects/scenarios/financial inputs are available for immediate use.
Developers can reset to a clean slate with scripts/reset_db.py, which safely drops managed tables and enum types in non-production environments before rerunning the initializer.
A detailed Data Model documentation is available. Discounted cash-flow metrics (NPV, IRR, Payback) referenced by the economic portion of the data model are described in the Financial Metrics Specification, while stochastic extensions leverage the Monte Carlo engine documented in the Monte Carlo Simulation Specification.
All data interactions are handled through the Data Access Layer, ensuring consistency and integrity across operations.
Security
The Calminer system incorporates robust security measures to protect sensitive data and ensure compliance with different industry and regulatory standards, such as GDPR or ISO/IEC 27001.
Within the system, security is enforced through multiple layers, including data encryption, access controls, audit logging, vulnerability management, and compliance monitoring.
A comprehensive Security Concept is being developed as well.
Authentication and Authorization
The system utilizes a secure authentication mechanism to verify user identities and manage access to resources.
Authentication
Users authenticate using secure methods, such as OAuth 2.0 or JWT tokens, to ensure the legitimacy of access requests.
Authorization
Access to resources is controlled through role-based permissions, ensuring users can only access data and functionalities relevant to their roles.
Session Management
User sessions are managed securely, with mechanisms in place to prevent session hijacking and ensure session integrity.
Token Expiration
Access tokens have a limited lifespan and are refreshed periodically to enhance security.
Multi-Factor Authentication (MFA)
The system supports MFA to provide an additional layer of security during user authentication.
Error Handling and Logging
The system implements a comprehensive error handling and logging strategy to ensure reliability and facilitate troubleshooting.
Centralized Logging
All application logs are centralized to a logging service, enabling efficient monitoring and analysis of system behavior.
Error Categorization
Errors are categorized into different types (e.g., client errors, server errors) to facilitate targeted handling and resolution.
Graceful Degradation
The system is designed to handle errors gracefully, providing meaningful feedback to users without exposing sensitive information.
Alerting and Monitoring
The system includes mechanisms for alerting administrators about critical errors and monitoring system health.
Log Retention Policies
Logs are retained for a specified period to comply with regulatory requirements and support forensic analysis.
Performance Optimization
The system is designed with performance optimization strategies to ensure efficient operation and responsiveness.
Caching
Frequently accessed data is cached to reduce database load and improve response times.
Asynchronous Processing
Time-consuming operations are handled asynchronously to prevent blocking user interactions.
Database Indexing
Appropriate indexing strategies are implemented to optimize query performance.
Load Balancing
Traffic is distributed across multiple servers to ensure optimal resource utilization and minimize latency.
Scalability Strategies
The system is architected to scale efficiently in response to varying workloads and user demands.
Horizontal Scaling
The system can add more machines or instances to handle increased load, distributing traffic and processing across multiple nodes.
Vertical Scaling
Individual components can be upgraded with more resources (CPU, memory) to improve performance without changing the overall architecture.
Microservices Architecture
The system is designed as a collection of loosely coupled services, allowing for independent scaling of different components based on demand.
Load Testing
Regular load testing is conducted to identify bottlenecks and ensure the system can handle expected traffic patterns.
User Interface Design Principles
To ensure a consistent and user-friendly experience, the Calminer system adheres to established user interface design principles. A detailed user interface (UI) design principles document is available for reference.
Notification System
The system includes a notification mechanism to inform users about important events and updates.
Notification Channels
The system supports multiple channels for notifications, including email, SMS, and in-app alerts.
Configurable Preferences
Users can customize their notification preferences, selecting the types of notifications they wish to receive and the channels through which they are delivered.
Event Triggers
Notifications are triggered by specific events within the system, such as project updates, data processing completions, or system alerts.
Delivery Reliability
The notification system is designed to ensure reliable delivery of messages, with retry mechanisms in place for failed attempts.
Audit Trail
A record of all sent notifications is maintained for auditing purposes and to track user engagement.
Data Integration
The Calminer system is designed to integrate seamlessly with various external data sources and services.
API Integration
The system can connect to external APIs to fetch or send data as needed.
Data Ingestion
Mechanisms are in place to ingest data from various sources, including databases, file systems, and streaming platforms.
Data Transformation
The system can transform data into the required format for processing and analysis.
Data Synchronization
Regular synchronization processes ensure that data remains consistent across all integrated systems.
Error Handling
Robust error handling mechanisms are implemented to manage integration failures and ensure data integrity.
Reporting and Analytics
The Calminer system includes comprehensive reporting and analytics capabilities to support data-driven decision-making.
Data Visualization
The system provides tools for visualizing data through charts, graphs, and dashboards, making it easier to identify trends and insights.
Custom Reports
Users can create custom reports based on specific criteria, allowing for tailored analysis of project performance and resource utilization.
Real-time Analytics
The system supports real-time data processing and analytics, enabling users to access up-to-date information and respond quickly to changing conditions.
Historical Analysis
The system maintains a history of key metrics and events, allowing for retrospective analysis and identification of long-term trends.