Refactor architecture documentation and enhance security features
Some checks failed
Run Tests / e2e tests (push) Failing after 1m20s
Run Tests / unit tests (push) Has been cancelled
Run Tests / lint tests (push) Has been cancelled

- Updated architecture constraints documentation to include detailed sections on technical, organizational, regulatory, environmental, and performance constraints.
- Created separate markdown files for each type of constraint for better organization and clarity.
- Revised the architecture scope section to provide a clearer overview of the system's key areas.
- Enhanced the solution strategy documentation with detailed explanations of the client-server architecture, technology choices, trade-offs, and future considerations.
- Added comprehensive descriptions of backend and frontend components, middleware, and utilities in the architecture documentation.
- Migrated UI, templates, and styling notes to a dedicated section for better structure.
- Updated requirements.txt to include missing dependencies.
- Refactored user authentication logic in the users.py and security.py files to improve code organization and maintainability, including the integration of OAuth2 password bearer token handling.
This commit is contained in:
2025-10-27 12:46:51 +01:00
parent 7f4cd33b65
commit ef4fb7dcf0
23 changed files with 271 additions and 400 deletions

View File

@@ -5,10 +5,11 @@ jobs:
tests:
name: ${{ matrix.target }} tests
runs-on: ubuntu-latest
container: ${{ matrix.target == 'e2e' && 'mcr.microsoft.com/playwright/python:v1.40.0-jammy' || '' }}
env:
DATABASE_DRIVER: postgresql
DATABASE_HOST: postgres
DATABASE_PORT: "5432"
DATABASE_PORT: '5432'
DATABASE_NAME: calminer_ci
DATABASE_USER: calminer
DATABASE_PASSWORD: secret
@@ -36,10 +37,17 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Cache pip dependencies
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt', 'requirements-test.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Prepare Python environment
uses: ./.gitea/actions/setup-python-env
with:
install-playwright: ${{ matrix.target == 'e2e' }}
install-playwright: ${{ matrix.target != 'e2e' }}
- name: Run tests
run: |
if [ "${{ matrix.target }}" = "unit" ]; then

View File

@@ -1,66 +1,18 @@
---
title: "02 — Architecture Constraints"
description: "Document imposed constraints: technical, organizational, regulatory, and environmental constraints that affect architecture decisions."
status: skeleton
title: '02 — Architecture Constraints'
description: 'Document imposed constraints: technical, organizational, regulatory, and environmental constraints that affect architecture decisions.'
status: draft
---
# 02 — Architecture Constraints
## Technical Constraints
## Constraints Overview
> e.g., choice of FastAPI, PostgreSQL, SQLAlchemy, Chart.js, Jinja2 templates.
The architecture of CalMiner is influenced by several technical constraints that shape its design and implementation:
1. **Framework Selection**: The choice of FastAPI as the web framework imposes constraints on how the application handles requests, routing, and middleware. FastAPI's asynchronous capabilities must be leveraged appropriately to ensure optimal performance.
2. **Database Technology**: The use of PostgreSQL as the primary database system dictates the data modeling, querying capabilities, and transaction management strategies. SQLAlchemy ORM is used for database interactions, which requires adherence to its conventions and limitations.
3. **Frontend Technologies**: The decision to use Jinja2 for server-side templating and Chart.js for data visualization influences the structure of the frontend code and the way dynamic content is rendered.
4. **Simulation Logic**: The Monte Carlo simulation logic must be designed to efficiently handle large datasets and perform computations within the constraints of the chosen programming language (Python) and its libraries.
## Organizational Constraints
> e.g., team skillsets, development workflows, CI/CD pipelines.
Restrictions arising from organizational factors include:
1. **Team Expertise**: The development teams familiarity with FastAPI, SQLAlchemy, and frontend technologies like Jinja2 and Chart.js influences the architecture choices to ensure maintainability and ease of development.
2. **Development Processes**: The adoption of Agile methodologies and CI/CD pipelines (using Gitea Actions) shapes the architecture to support continuous integration, automated testing, and deployment practices.
3. **Collaboration Tools**: The use of specific collaboration and version control tools (e.g., Gitea) affects how code is managed, reviewed, and integrated, impacting the overall architecture and development workflow.
4. **Documentation Standards**: The requirement for comprehensive documentation (as seen in the `docs/` folder) necessitates an architecture that is well-structured and easy to understand for both current and future team members.
5. **Knowledge Sharing**: The need for effective knowledge sharing and onboarding processes influences the architecture to ensure that it is accessible and understandable for new team members.
6. **Resource Availability**: The availability of hardware, software, and human resources within the organization can impose constraints on the architecture, affecting decisions related to scalability, performance, and feature implementation.
## Regulatory Constraints
> e.g., data privacy laws, industry standards.
Regulatory constraints that impact the architecture of CalMiner include:
1. **Data Privacy Compliance**: The architecture must ensure compliance with data privacy regulations such as GDPR or CCPA, which may dictate how user data is collected, stored, and processed.
2. **Industry Standards**: Adherence to industry-specific standards and best practices may influence the design of data models, security measures, and reporting functionalities.
3. **Auditability**: The system may need to incorporate logging and auditing features to meet regulatory requirements, affecting the architecture of data storage and access controls.
4. **Data Retention Policies**: Regulatory requirements regarding data retention and deletion may impose constraints on how long certain types of data can be stored, influencing database design and data lifecycle management.
5. **Security Standards**: Compliance with security standards (e.g., ISO/IEC 27001) may necessitate the implementation of specific security measures, such as encryption, access controls, and vulnerability management, which impact the overall architecture.
## Environmental Constraints
> e.g., deployment environments, cloud provider limitations.
Environmental constraints affecting the architecture include:
1. **Deployment Environments**: The architecture must accommodate various deployment environments (development, testing, production) with differing configurations and resource allocations.
2. **Cloud Provider Limitations**: If deployed on a specific cloud provider, the architecture may need to align with the provider's services, limitations, and best practices, such as using managed databases or specific container orchestration tools.
3. **Containerization**: The use of Docker for containerization imposes constraints on how the application is packaged, deployed, and scaled, influencing the architecture to ensure compatibility with container orchestration platforms.
4. **Scalability Requirements**: The architecture must be designed to scale efficiently based on anticipated load and usage patterns, considering the limitations of the chosen infrastructure.
## Performance Constraints
> e.g., response time requirements, scalability needs.
Current performance constraints include:
1. **Response Time Requirements**: The architecture must ensure that the system can respond to user requests within a specified time frame, which may impact design decisions related to caching, database queries, and API performance.
2. **Scalability Needs**: The system should be able to handle increased load and user traffic without significant degradation in performance, necessitating a scalable architecture that can grow with demand.
- [Technical Constraints](02_constraints/02_01_technical_constraints.md)
- [Organizational Constraints](02_constraints/02_02_organizational_constraints.md)
- [Regulatory Constraints](02_constraints/02_03_regulatory_constraints.md)
- [Environmental Constraints](02_constraints/02_04_environmental_constraints.md)
- [Performance Constraints](02_constraints/02_05_performance_constraints.md)
## Security Constraints

View File

@@ -0,0 +1,16 @@
---
title: '02 — Technical Constraints'
description: 'Technical constraints that affect architecture decisions.'
status: draft
---
# Technical Constraints
> e.g., choice of FastAPI, PostgreSQL, SQLAlchemy, Chart.js, Jinja2 templates.
The architecture of CalMiner is influenced by several technical constraints that shape its design and implementation:
1. **Framework Selection**: The choice of FastAPI as the web framework imposes constraints on how the application handles requests, routing, and middleware. FastAPI's asynchronous capabilities must be leveraged appropriately to ensure optimal performance.
2. **Database Technology**: The use of PostgreSQL as the primary database system dictates the data modeling, querying capabilities, and transaction management strategies. SQLAlchemy ORM is used for database interactions, which requires adherence to its conventions and limitations.
3. **Frontend Technologies**: The decision to use Jinja2 for server-side templating and Chart.js for data visualization influences the structure of the frontend code and the way dynamic content is rendered.
4. **Simulation Logic**: The Monte Carlo simulation logic must be designed to efficiently handle large datasets and perform computations within the constraints of the chosen programming language (Python) and its libraries.

View File

@@ -0,0 +1,18 @@
---
title: '02 — Organizational Constraints'
description: 'Organizational constraints that affect architecture decisions.'
status: draft
---
# Organizational Constraints
> e.g., team skillsets, development workflows, CI/CD pipelines.
Restrictions arising from organizational factors include:
1. **Team Expertise**: The development teams familiarity with FastAPI, SQLAlchemy, and frontend technologies like Jinja2 and Chart.js influences the architecture choices to ensure maintainability and ease of development.
2. **Development Processes**: The adoption of Agile methodologies and CI/CD pipelines (using Gitea Actions) shapes the architecture to support continuous integration, automated testing, and deployment practices.
3. **Collaboration Tools**: The use of specific collaboration and version control tools (e.g., Gitea) affects how code is managed, reviewed, and integrated, impacting the overall architecture and development workflow.
4. **Documentation Standards**: The requirement for comprehensive documentation (as seen in the `docs/` folder) necessitates an architecture that is well-structured and easy to understand for both current and future team members.
5. **Knowledge Sharing**: The need for effective knowledge sharing and onboarding processes influences the architecture to ensure that it is accessible and understandable for new team members.
6. **Resource Availability**: The availability of hardware, software, and human resources within the organization can impose constraints on the architecture, affecting decisions related to scalability, performance, and feature implementation.

View File

@@ -0,0 +1,17 @@
---
title: '02 — Regulatory Constraints'
description: 'Regulatory constraints that affect architecture decisions.'
status: draft
---
# Regulatory Constraints
> e.g., data privacy laws, industry standards.
Regulatory constraints that impact the architecture of CalMiner include:
1. **Data Privacy Compliance**: The architecture must ensure compliance with data privacy regulations such as GDPR or CCPA, which may dictate how user data is collected, stored, and processed.
2. **Industry Standards**: Adherence to industry-specific standards and best practices may influence the design of data models, security measures, and reporting functionalities.
3. **Auditability**: The system may need to incorporate logging and auditing features to meet regulatory requirements, affecting the architecture of data storage and access controls.
4. **Data Retention Policies**: Regulatory requirements regarding data retention and deletion may impose constraints on how long certain types of data can be stored, influencing database design and data lifecycle management.
5. **Security Standards**: Compliance with security standards (e.g., ISO/IEC 27001) may necessitate the implementation of specific security measures, such as encryption, access controls, and vulnerability management, which impact the overall architecture.

View File

@@ -0,0 +1,16 @@
---
title: '02 — Environmental Constraints'
description: 'Environmental constraints that affect architecture decisions.'
status: draft
---
# Environmental Constraints
> e.g., deployment environments, cloud provider limitations.
Environmental constraints affecting the architecture include:
1. **Deployment Environments**: The architecture must accommodate various deployment environments (development, testing, production) with differing configurations and resource allocations.
2. **Cloud Provider Limitations**: If deployed on a specific cloud provider, the architecture may need to align with the provider's services, limitations, and best practices, such as using managed databases or specific container orchestration tools.
3. **Containerization**: The use of Docker for containerization imposes constraints on how the application is packaged, deployed, and scaled, influencing the architecture to ensure compatibility with container orchestration platforms.
4. **Scalability Requirements**: The architecture must be designed to scale efficiently based on anticipated load and usage patterns, considering the limitations of the chosen infrastructure.

View File

@@ -0,0 +1,14 @@
---
title: '02 — Performance Constraints'
description: 'Performance constraints that affect architecture decisions.'
status: draft
---
# Performance Constraints
> e.g., response time requirements, scalability needs.
Current performance constraints include:
1. **Response Time Requirements**: The architecture must ensure that the system can respond to user requests within a specified time frame, which may impact design decisions related to caching, database queries, and API performance.
2. **Scalability Needs**: The system should be able to handle increased load and user traffic without significant degradation in performance, necessitating a scalable architecture that can grow with demand.

View File

@@ -18,24 +18,7 @@ The CalMiner system operates within the context of mining project management, pr
## Scope of the Architecture
The architecture encompasses the following key areas:
1. **Data Ingestion**: Mechanisms for collecting and processing data from various sources.
2. **Data Storage**: Solutions for storing and managing historical and real-time data.
3. **Simulation Engine**: Core algorithms and models for scenario analysis.
3.1. **Modeling Framework**: Tools for defining and managing simulation models.
3.2. **Parameter Management**: Systems for handling input parameters and configurations.
3.3. **Execution Engine**: Infrastructure for running simulations and processing results.
3.4. **Result Storage**: Systems for storing simulation outputs for analysis and reporting.
4. **Financial Reporting**: Tools for generating reports and visualizations based on simulation outcomes.
5. **Risk Assessment**: Frameworks for identifying and evaluating potential project risks.
6. **Profitability Analysis**: Modules for calculating and analyzing project profitability metrics.
7. **User Interface**: Design and implementation of the user-facing components of the system.
8. **Security and Compliance**: Measures to ensure data security and regulatory compliance.
9. **Scalability and Performance**: Strategies for ensuring the system can handle increasing data volumes and user loads.
10. **Integration Points**: Interfaces for integrating with external systems and services.
11. **Monitoring and Logging**: Systems for tracking system performance and user activity.
12. **Maintenance and Support**: Processes for ongoing system maintenance and user support.
See [Architecture Scope](03_scope/03_01_architecture_scope.md) for details.
## Diagram

View File

@@ -0,0 +1,26 @@
---
title: '03 — Architecture Scope'
description: 'Key areas encompassed by the architecture.'
status: draft
---
# Architecture Scope
The architecture encompasses the following key areas:
1. **Data Ingestion**: Mechanisms for collecting and processing data from various sources.
2. **Data Storage**: Solutions for storing and managing historical and real-time data.
3. **Simulation Engine**: Core algorithms and models for scenario analysis.
3.1. **Modeling Framework**: Tools for defining and managing simulation models.
3.2. **Parameter Management**: Systems for handling input parameters and configurations.
3.3. **Execution Engine**: Infrastructure for running simulations and processing results.
3.4. **Result Storage**: Systems for storing simulation outputs for analysis and reporting.
4. **Financial Reporting**: Tools for generating reports and visualizations based on simulation outcomes.
5. **Risk Assessment**: Frameworks for identifying and evaluating potential project risks.
6. **Profitability Analysis**: Modules for calculating and analyzing project profitability metrics.
7. **User Interface**: Design and implementation of the user-facing components of the system.
8. **Security and Compliance**: Measures to ensure data security and regulatory compliance.
9. **Scalability and Performance**: Strategies for ensuring the system can handle increasing data volumes and user loads.
10. **Integration Points**: Interfaces for integrating with external systems and services.
11. **Monitoring and Logging**: Systems for tracking system performance and user activity.
12. **Maintenance and Support**: Processes for ongoing system maintenance and user support.

View File

@@ -8,42 +8,9 @@ status: draft
This section outlines the high-level solution strategy for implementing the CalMiner system, focusing on major approaches, technology choices, and trade-offs.
## Client-Server Architecture
## Solution Strategy Overview
- **Backend**: FastAPI serves as the backend framework, providing RESTful APIs for data management, simulation execution, and reporting. It leverages SQLAlchemy for ORM-based database interactions with PostgreSQL.
- **Frontend**: Server-rendered Jinja2 templates deliver dynamic HTML views, enhanced with Chart.js for interactive data visualizations. This approach balances performance and simplicity, avoiding the complexity of a full SPA.
- **Middleware**: Custom middleware handles JSON validation to ensure data integrity before processing requests.
## Technology Choices
- **FastAPI**: Chosen for its high performance, ease of use, and modern features like async support and automatic OpenAPI documentation.
- **PostgreSQL**: Selected for its robustness, scalability, and support for complex queries, making it suitable for handling the diverse data needs of mining project management.
- **SQLAlchemy**: Provides a flexible and powerful ORM layer, facilitating database interactions while maintaining code readability and maintainability.
- **Chart.js**: Utilized for its simplicity and effectiveness in rendering interactive charts, enhancing the user experience on the dashboard.
- **Jinja2**: Enables server-side rendering of HTML templates, allowing for dynamic content generation while keeping the frontend lightweight.
- **Pydantic**: Used for data validation and serialization, ensuring that incoming request payloads conform to expected schemas.
- **Docker**: Employed for containerization, ensuring consistent deployment across different environments and simplifying dependency management.
- **Redis**: Used as an in-memory data store to cache frequently accessed data, improving application performance and reducing database load.
## Trade-offs
- **Server-Rendered vs. SPA**: Opted for server-rendered templates over a single-page application (SPA) to reduce complexity and improve initial load times, at the cost of some interactivity.
- **Synchronous vs. Asynchronous**: While FastAPI supports async operations, the initial implementation focuses on synchronous request handling for simplicity, with plans to introduce async features as needed.
- **Monolithic vs. Microservices**: The initial architecture follows a monolithic approach for ease of development and deployment, with the possibility of refactoring into microservices as the system scales.
- **In-Memory Caching**: Implementing Redis for caching introduces additional infrastructure complexity but significantly enhances performance for read-heavy operations.
- **Database Choice**: PostgreSQL was chosen over NoSQL alternatives due to the structured nature of the data and the need for complex querying capabilities, despite potential scalability challenges.
- **Technology Familiarity**: Selected technologies align with the team's existing skill set to minimize the learning curve and accelerate development, even if some alternatives may offer marginally better performance or features.
- **Extensibility vs. Simplicity**: The architecture is designed to be extensible for future features (e.g., Monte Carlo simulation engine) while maintaining simplicity in the initial implementation to ensure timely delivery of core functionalities.
## Future Considerations
- **Scalability**: As the user base grows, consider transitioning to a microservices architecture and implementing load balancing strategies.
- **Asynchronous Processing**: Introduce asynchronous task queues (e.g., Celery) for long-running simulations to improve responsiveness.
- **Enhanced Frontend**: Explore the possibility of integrating a frontend framework (e.g., React or Vue.js) for more dynamic user interactions in future iterations.
- **Advanced Analytics**: Plan for integrating advanced analytics and machine learning capabilities to enhance simulation accuracy and reporting insights.
- **Security Enhancements**: Implement robust authentication and authorization mechanisms to protect sensitive data and ensure compliance with industry standards.
- **Continuous Integration/Continuous Deployment (CI/CD)**: Establish CI/CD pipelines to automate testing, building, and deployment processes for faster and more reliable releases.
- **Monitoring and Logging**: Integrate monitoring tools (e.g., Prometheus, Grafana) and centralized logging solutions (e.g., ELK stack) to track application performance and troubleshoot issues effectively.
- **User Feedback Loop**: Implement mechanisms for collecting user feedback to inform future development priorities and improve user experience.
- **Documentation**: Maintain comprehensive documentation for both developers and end-users to facilitate onboarding and effective use of the system.
- **Testing Strategy**: Develop a robust testing strategy, including unit, integration, and end-to-end tests, to ensure code quality and reliability as the system evolves.
- [Client-Server Architecture](04_strategy/04_01_client_server_architecture.md)
- [Technology Choices](04_strategy/04_02_technology_choices.md)
- [Trade-offs](04_strategy/04_03_trade_offs.md)
- [Future Considerations](04_strategy/04_04_future_considerations.md)

View File

@@ -0,0 +1,10 @@
---
title: '04.01 — Client-Server Architecture'
description: 'Details on the client-server architecture of CalMiner.'
---
# 04.01 — Client-Server Architecture
- **Backend**: FastAPI serves as the backend framework, providing RESTful APIs for data management, simulation execution, and reporting. It leverages SQLAlchemy for ORM-based database interactions with PostgreSQL.
- **Frontend**: Server-rendered Jinja2 templates deliver dynamic HTML views, enhanced with Chart.js for interactive data visualizations. This approach balances performance and simplicity, avoiding the complexity of a full SPA.
- **Middleware**: Custom middleware handles JSON validation to ensure data integrity before processing requests.

View File

@@ -0,0 +1,15 @@
---
title: '04.02 — Technology Choices'
description: 'Detailed explanation of technology choices in CalMiner.'
---
# 04.02 — Technology Choices
- **FastAPI**: Chosen for its high performance, ease of use, and modern features like async support and automatic OpenAPI documentation.
- **PostgreSQL**: Selected for its robustness, scalability, and support for complex queries, making it suitable for handling the diverse data needs of mining project management.
- **SQLAlchemy**: Provides a flexible and powerful ORM layer, facilitating database interactions while maintaining code readability and maintainability.
- **Chart.js**: Utilized for its simplicity and effectiveness in rendering interactive charts, enhancing the user experience on the dashboard.
- **Jinja2**: Enables server-side rendering of HTML templates, allowing for dynamic content generation while keeping the frontend lightweight.
- **Pydantic**: Used for data validation and serialization, ensuring that incoming request payloads conform to expected schemas.
- **Docker**: Employed for containerization, ensuring consistent deployment across different environments and simplifying dependency management.
- **Redis**: Used as an in-memory data store to cache frequently accessed data, improving application performance and reducing database load.

View File

@@ -0,0 +1,14 @@
---
title: '04.03 — Trade-offs'
description: 'Discussion of trade-offs made in the CalMiner architecture.'
---
# 04.03 — Trade-offs
- **Server-Rendered vs. SPA**: Opted for server-rendered templates over a single-page application (SPA) to reduce complexity and improve initial load times, at the cost of some interactivity.
- **Synchronous vs. Asynchronous**: While FastAPI supports async operations, the initial implementation focuses on synchronous request handling for simplicity, with plans to introduce async features as needed.
- **Monolithic vs. Microservices**: The initial architecture follows a monolithic approach for ease of development and deployment, with the possibility of refactoring into microservices as the system scales.
- **In-Memory Caching**: Implementing Redis for caching introduces additional infrastructure complexity but significantly enhances performance for read-heavy operations.
- **Database Choice**: PostgreSQL was chosen over NoSQL alternatives due to the structured nature of the data and the need for complex querying capabilities, despite potential scalability challenges.
- **Technology Familiarity**: Selected technologies align with the team's existing skill set to minimize the learning curve and accelerate development, even if some alternatives may offer marginally better performance or features.
- **Extensibility vs. Simplicity**: The architecture is designed to be extensible for future features (e.g., Monte Carlo simulation engine) while maintaining simplicity in the initial implementation to ensure timely delivery of core functionalities.

View File

@@ -0,0 +1,17 @@
---
title: '04.04 — Future Considerations'
description: 'Future considerations for the CalMiner architecture.'
---
# 04.04 — Future Considerations
- **Scalability**: As the user base grows, consider transitioning to a microservices architecture and implementing load balancing strategies.
- **Asynchronous Processing**: Introduce asynchronous task queues (e.g., Celery) for long-running simulations to improve responsiveness.
- **Enhanced Frontend**: Explore the possibility of integrating a frontend framework (e.g., React or Vue.js) for more dynamic user interactions in future iterations.
- **Advanced Analytics**: Plan for integrating advanced analytics and machine learning capabilities to enhance simulation accuracy and reporting insights.
- **Security Enhancements**: Implement robust authentication and authorization mechanisms to protect sensitive data and ensure compliance with industry standards.
- **Continuous Integration/Continuous Deployment (CI/CD)**: Establish CI/CD pipelines to automate testing, building, and deployment processes for faster and more reliable releases.
- **Monitoring and Logging**: Integrate monitoring tools (e.g., Prometheus, Grafana) and centralized logging solutions (e.g., ELK stack) to track application performance and troubleshoot issues effectively.
- **User Feedback Loop**: Implement mechanisms for collecting user feedback to inform future development priorities and improve user experience.
- **Documentation**: Maintain comprehensive documentation for both developers and end-users to facilitate onboarding and effective use of the system.
- **Testing Strategy**: Develop a robust testing strategy, including unit, integration, and end-to-end tests, to ensure code quality and reliability as the system evolves.

View File

@@ -0,0 +1,13 @@
---
title: '05 — Architecture Overview'
description: "This overview complements architecture with a high-level map of CalMiner's module layout and request flow."
status: draft
---
This overview complements [architecture](README.md) with a high-level map of CalMiner's module layout and request flow.
Refer to the detailed architecture chapters in `docs/architecture/`:
- Module map & components: [Building Block View](../05_building_block_view.md)
- Request flow & runtime interactions: [Runtime View](../06_runtime_view.md)
- Simulation roadmap & strategy: [Solution Strategy](../04_solution_strategy.md)

View File

@@ -0,0 +1,13 @@
---
title: '05 — Backend Components'
description: 'Description of the backend components of the CalMiner application.'
status: draft
---
- **FastAPI application** (`main.py`): entry point that configures routers, middleware, and startup/shutdown events.
- **Routers** (`routes/`): modular route handlers for scenarios, parameters, costs, consumption, production, equipment, maintenance, simulations, and reporting. Each router defines RESTful endpoints, request/response schemas, and orchestrates service calls.
- leveraging a shared dependency module (`routes/dependencies.get_db`) for SQLAlchemy session management.
- **Models** (`models/`): SQLAlchemy ORM models representing database tables and relationships, encapsulating domain entities like Scenario, CapEx, OpEx, Consumption, ProductionOutput, Equipment, Maintenance, and SimulationResult.
- **Services** (`services/`): business logic layer that processes data, performs calculations, and interacts with models. Key services include reporting calculations and Monte Carlo simulation scaffolding.
- `services/settings.py`: manages application settings backed by the `application_setting` table, including CSS variable defaults, persistence, and environment-driven overrides that surface in both the API and UI.
- **Database** (`config/database.py`): sets up the SQLAlchemy engine and session management for PostgreSQL interactions.

View File

@@ -0,0 +1,11 @@
---
title: '05 — Frontend Components'
description: 'Description of the frontend components of the CalMiner application.'
status: draft
---
- **Templates** (`templates/`): Jinja2 templates for server-rendered HTML views, extending a shared base layout with a persistent sidebar for navigation.
- **Static Assets** (`static/`): CSS and JavaScript files for styling and interactivity. Shared CSS variables in `static/css/main.css` define the color palette, while page-specific JS modules in `static/js/` handle dynamic behaviors.
- **Reusable partials** (`templates/partials/components.html`): macro library that standardises select inputs, feedback/empty states, and table wrappers so pages remain consistent while keeping DOM hooks stable for existing JavaScript modules.
- `templates/settings.html`: Settings hub that renders theme controls and environment override tables using metadata provided by `routes/ui.py`.
- `static/js/settings.js`: applies client-side validation, form submission, and live CSS updates for theme changes, respecting environment-managed variables returned by the API.

View File

@@ -0,0 +1,8 @@
---
title: '05 — Middleware & Utilities'
description: 'Description of the middleware and utility components of the CalMiner application.'
status: draft
---
- **Middleware** (`middleware/validation.py`): applies JSON validation before requests reach routers.
- **Testing** (`tests/unit/`): pytest suite covering route and service behavior, including UI rendering checks and negative-path router validation tests to ensure consistent HTTP error semantics. Playwright end-to-end coverage is planned for core smoke flows (dashboard load, scenario inputs, reporting) and will attach in CI once scaffolding is completed.

View File

@@ -1,6 +1,6 @@
---
title: '05 — Building Block View'
description: 'Explain the static structure: modules, components, services and their relationships.'
title: "05 — Building Block View"
description: "Explain the static structure: modules, components, services and their relationships."
status: draft
---
@@ -8,257 +8,9 @@ status: draft
# 05 — Building Block View
## Architecture overview
## Building Block Overview
This overview complements [architecture](README.md) with a high-level map of CalMiner's module layout and request flow.
Refer to the detailed architecture chapters in `docs/architecture/`:
- Module map & components: [Building Block View](05_building_block_view.md)
- Request flow & runtime interactions: [Runtime View](06_runtime_view.md)
- Simulation roadmap & strategy: [Solution Strategy](04_solution_strategy.md)
## System Components
### Backend
- **FastAPI application** (`main.py`): entry point that configures routers, middleware, and startup/shutdown events.
- **Routers** (`routes/`): modular route handlers for scenarios, parameters, costs, consumption, production, equipment, maintenance, simulations, and reporting. Each router defines RESTful endpoints, request/response schemas, and orchestrates service calls.
- leveraging a shared dependency module (`routes/dependencies.get_db`) for SQLAlchemy session management.
- **Models** (`models/`): SQLAlchemy ORM models representing database tables and relationships, encapsulating domain entities like Scenario, CapEx, OpEx, Consumption, ProductionOutput, Equipment, Maintenance, and SimulationResult.
- **Services** (`services/`): business logic layer that processes data, performs calculations, and interacts with models. Key services include reporting calculations and Monte Carlo simulation scaffolding.
- `services/settings.py`: manages application settings backed by the `application_setting` table, including CSS variable defaults, persistence, and environment-driven overrides that surface in both the API and UI.
- **Database** (`config/database.py`): sets up the SQLAlchemy engine and session management for PostgreSQL interactions.
### Frontend
- **Templates** (`templates/`): Jinja2 templates for server-rendered HTML views, extending a shared base layout with a persistent sidebar for navigation.
- **Static Assets** (`static/`): CSS and JavaScript files for styling and interactivity. Shared CSS variables in `static/css/main.css` define the color palette, while page-specific JS modules in `static/js/` handle dynamic behaviors.
- **Reusable partials** (`templates/partials/components.html`): macro library that standardises select inputs, feedback/empty states, and table wrappers so pages remain consistent while keeping DOM hooks stable for existing JavaScript modules.
- `templates/settings.html`: Settings hub that renders theme controls and environment override tables using metadata provided by `routes/ui.py`.
- `static/js/settings.js`: applies client-side validation, form submission, and live CSS updates for theme changes, respecting environment-managed variables returned by the API.
### Middleware & Utilities
- **Middleware** (`middleware/validation.py`): applies JSON validation before requests reach routers.
- **Testing** (`tests/unit/`): pytest suite covering route and service behavior, including UI rendering checks and negative-path router validation tests to ensure consistent HTTP error semantics. Playwright end-to-end coverage is planned for core smoke flows (dashboard load, scenario inputs, reporting) and will attach in CI once scaffolding is completed.
### Level 1 Overview
```mermaid
graph LR
U["User (Browser)"]
subgraph FE[Frontend]
FE_TPL["Templates (Jinja2)"]
FE_STATIC["Static Assets (CSS/JS)"]
FE_PARTS["Reusable Partials"]
FE_SETTINGS["Settings View & JS"]
end
subgraph BE[Backend — FastAPI]
BE_APP["FastAPI App (main.py)"]
BE_ROUTES["Routers"]
BE_SERVICES["Services"]
BE_MODELS["Models (SQLAlchemy)"]
BE_DB["Database Layer"]
end
subgraph MW[Middleware & Utilities]
MW_VAL["JSON Validation Middleware"]
end
subgraph QA[Testing]
QA_UNIT["Unit Tests (pytest)"]
QA_E2E["E2E (Playwright, planned)"]
end
%% High-level flows
U -->|HTTP| BE_APP
U --> FE
FE --> BE_ROUTES
BE_APP --> BE_ROUTES
BE_ROUTES --> BE_SERVICES
BE_SERVICES --> BE_MODELS
BE_MODELS --> BE_DB
MW_VAL --> BE_APP
QA_UNIT --> BE_ROUTES
QA_UNIT --> BE_SERVICES
QA_UNIT --> FE
QA_UNIT --> MW_VAL
QA_E2E --> U
QA_E2E --> BE_APP
```
### Level 2 Overview
```mermaid
graph LR
%% Direction
%% LR = left-to-right for a wide architecture view
%% === Clients ===
U["User (Browser)"]
%% === Frontend ===
subgraph FE[Frontend]
TPL["Jinja2 Templates
(templates/)
• base layout + sidebar"]
PARTS["Reusable Partials
(templates/partials/components.html)
• inputs • empty states • table wrappers"]
STATIC["Static Assets
(static/)
• CSS: static/css/main.css (palette via CSS vars)
• JS: static/js/*.js (page modules)"]
SETPAGE["Settings View
(templates/settings.html)"]
SETJS["Settings Logic
(static/js/settings.js)
• validation • submit • live CSS updates"]
end
%% === Backend ===
subgraph BE[Backend — FastAPI]
MAIN["FastAPI App
(main.py)
• routers • middleware • startup/shutdown"]
subgraph ROUTES[Routers]
R_SCN["scenarios"]
R_PAR["parameters"]
R_CST["costs"]
R_CONS["consumption"]
R_PROD["production"]
R_EQP["equipment"]
R_MNT["maintenance"]
R_SIM["simulations"]
R_REP["reporting"]
R_UI["ui.py (metadata for UI)"]
DEP["dependencies.get_db
(shared SQLAlchemy session)"]
end
subgraph SRV[Services]
S_BLL["Business Logic Layer
• orchestrates models + calc"]
S_REP["Reporting Calculations"]
S_SIM["Monte Carlo
(simulation scaffolding)"]
S_SET["Settings Manager
(services/settings.py)
• defaults via CSS vars
• persistence in DB
• env overrides
• surfaces to API & UI"]
end
subgraph MOD[Models]
M_SCN["Scenario"]
M_CAP["CapEx"]
M_OPEX["OpEx"]
M_CONS["Consumption"]
M_PROD["ProductionOutput"]
M_EQP["Equipment"]
M_MNT["Maintenance"]
M_SIMR["SimulationResult"]
end
subgraph DB[Database Layer]
CFG["config/database.py
(SQLAlchemy engine & sessions)"]
PG[("PostgreSQL")]
APPSET["application_setting table"]
end
end
%% === Middleware & Utilities ===
subgraph MW[Middleware & Utilities]
VAL["JSON Validation Middleware
(middleware/validation.py)"]
end
subgraph TEST[Testing]
UNIT["pytest unit tests
(tests/unit/)
• routes • services • UI rendering
• negative-path validation"]
E2E["Playwright E2E (planned)
• dashboard • scenario inputs • reporting
• attach in CI"]
end
%% ===================== Edges / Flows =====================
%% User to Frontend/Backend
U -->|HTTP GET| MAIN
U --> TPL
TPL -->|server-rendered HTML| U
STATIC --> U
PARTS --> TPL
SETPAGE --> U
SETJS --> U
%% Frontend to Routers (AJAX/form submits)
SETJS -->|fetch/POST| R_UI
TPL -->|form submit / fetch| ROUTES
%% FastAPI app wiring and middleware
VAL --> MAIN
MAIN --> ROUTES
%% Routers to Services
ROUTES -->|calls| SRV
R_REP -->|calc| S_REP
R_SIM -->|run| S_SIM
R_UI -->|read/write settings meta| S_SET
%% Services to Models & DB
SRV --> MOD
MOD --> CFG
CFG --> PG
%% Settings manager persistence path
S_SET -->|persist/read| APPSET
APPSET --- PG
%% Shared DB session dependency
DEP -. provides .-> ROUTES
DEP -. session .-> SRV
%% Model entities mapping
S_BLL --> M_SCN & M_CAP & M_OPEX & M_CONS & M_PROD & M_EQP & M_MNT & M_SIMR
%% Testing coverage
UNIT --> ROUTES
UNIT --> SRV
UNIT --> TPL
UNIT --> VAL
E2E --> U
E2E --> MAIN
%% Legend
classDef store fill:#fff,stroke:#555,stroke-width:1px;
class PG store;
```
## Module Map (code)
- `scenario.py`: central scenario entity with relationships to cost, consumption, production, equipment, maintenance, and simulation results.
- `capex.py`, `opex.py`: financial expenditures tied to scenarios.
- `consumption.py`, `production_output.py`: operational data tables.
- `equipment.py`, `maintenance.py`: asset management models.
- `simulation_result.py`: stores Monte Carlo iteration outputs.
- `application_setting.py`: persists editable application configuration, currently focused on theme variables but designed to store future settings categories.
## Service Layer
- `reporting.py`: computes aggregates (count, min/max, mean, median, percentiles, standard deviation, variance, tail-risk metrics) from simulation results.
- `simulation.py`: scaffolds Monte Carlo simulation logic (currently in-memory; persistence planned).
- `currency.py`: handles currency normalization for cost tables.
- `utils.py`: shared helper functions (e.g., statistical calculations).
- `validation.py`: JSON schema validation middleware.
- `database.py`: SQLAlchemy engine and session setup.
- `dependencies.py`: FastAPI dependency injection for DB sessions.
- [Architecture Overview](05_blocks/05_01_architecture_overview.md)
- [Backend Components](05_blocks/05_02_backend_components.md)
- [Frontend Components](05_blocks/05_03_frontend_components.md)
- [Middleware & Utilities](05_blocks/05_04_middleware_utilities.md)

View File

@@ -1,7 +1,5 @@
# 13 — UI, templates and styling
Status: migrated
This chapter collects UI integration notes, reusable template components, styling audit points and per-page UI data/actions.
## Reusable Template Components

View File

@@ -8,3 +8,4 @@ jinja2
pandas
numpy
passlib
python-jose

View File

@@ -1,40 +1,15 @@
from fastapi import APIRouter, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer
from sqlalchemy.orm import Session
from config.database import get_db
from models.user import User
from services.security import get_password_hash, verify_password, create_access_token, SECRET_KEY, ALGORITHM
from services.security import get_password_hash, verify_password, create_access_token, SECRET_KEY, ALGORITHM, get_current_user, oauth2_scheme
from jose import jwt, JWTError
from schemas.user import UserCreate, UserInDB, UserLogin, UserUpdate, PasswordResetRequest, PasswordReset, Token
router = APIRouter(prefix="/users", tags=["users"])
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="users/login")
async def get_current_user(token: str = Depends(oauth2_scheme), db: Session = Depends(get_db)):
credentials_exception = HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
try:
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
username: str = payload.get("sub")
if username is None:
raise credentials_exception
if username is None:
raise credentials_exception
except JWTError:
raise credentials_exception
user = db.query(User).filter(User.username == username).first()
if user is None:
raise credentials_exception
return user
@router.post("/register", response_model=UserInDB, status_code=status.HTTP_201_CREATED)
async def register_user(user: UserCreate, db: Session = Depends(get_db)):
db_user = db.query(User).filter(User.username == user.username).first()

View File

@@ -1,8 +1,13 @@
from datetime import datetime, timedelta
from typing import Any, Union
from jose import jwt
from fastapi import HTTPException, status, Depends
from fastapi.security import OAuth2PasswordBearer
from jose import jwt, JWTError
from passlib.context import CryptContext
from sqlalchemy.orm import Session
from config.database import get_db
ACCESS_TOKEN_EXPIRE_MINUTES = 30
@@ -11,6 +16,8 @@ ALGORITHM = "HS256"
pwd_context = CryptContext(schemes=["pbkdf2_sha256"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="users/login")
def create_access_token(
subject: Union[str, Any], expires_delta: Union[timedelta, None] = None
@@ -30,3 +37,23 @@ def verify_password(plain_password: str, hashed_password: str) -> bool:
def get_password_hash(password: str) -> str:
return pwd_context.hash(password)
async def get_current_user(token: str = Depends(oauth2_scheme), db: Session = Depends(get_db)):
from models.user import User
credentials_exception = HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Could not validate credentials",
headers={"WWW-Authenticate": "Bearer"},
)
try:
payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])
username = payload.get("sub")
if username is None:
raise credentials_exception
except JWTError:
raise credentials_exception
user = db.query(User).filter(User.username == username).first()
if user is None:
raise credentials_exception
return user