allucanget/calminer
2
0
Fork 0
Code Pull Requests Actions Packages Releases Activity

Add UI and styling documentation; remove idempotency and logging audits
Some checks failed
CI / test (pull_request) Failing after 1m8s
Details

Browse Source 
- Introduced a new document outlining UI structure, reusable template components, CSS variable conventions, and per-page data/actions for the CalMiner application.
- Removed outdated idempotency audit and logging audit documents as they are no longer relevant.
- Updated quickstart guide to streamline developer setup instructions and link to relevant documentation.
- Created a roadmap document detailing scenario enhancements and data management strategies.
- Deleted the seed data plan document to consolidate information into the setup process.
- Refactored setup_database.py for improved logging and error handling during database setup and migration processes.
This commit is contained in:
zwitschi
2025-10-29 13:20:44 +01:00
parent 1f58de448c
commit 04d7f202b6
19 changed files with 609 additions and 752 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
docs/architecture/04_solution_strategy/04_01_implementation_plan_20251020.md
View File

@@ -1,110 +0,0 @@
# Implementation Plan 2025-10-20
This file contains the implementation plan (MVP features, steps, and estimates).
## Project Setup
1. Connect to PostgreSQL database with schema `calminer`.
1. Create and activate a virtual environment and install dependencies via `requirements.txt`.
1. Define database environment variables in `.env` (e.g., `DATABASE_DRIVER`, `DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_USER`, `DATABASE_PASSWORD`, `DATABASE_NAME`, optional `DATABASE_SCHEMA`).
1. Configure FastAPI entrypoint in `main.py` to include routers.
## Feature: Scenario Management
### Scenario Management — Steps
1. Create `models/scenario.py` for scenario CRUD.
1. Implement API endpoints in `routes/scenarios.py` (GET, POST, PUT, DELETE).
1. Write unit tests in `tests/unit/test_scenario.py`.
1. Build UI component `components/ScenarioForm.html`.
## Feature: Process Parameters
### Parameters — Steps
1. Create `models/parameters.py` for process parameters.
1. Implement Pydantic schemas in `routes/parameters.py`.
1. Add validation middleware in `middleware/validation.py`.
1. Write unit tests in `tests/unit/test_parameter.py`.
1. Build UI component `components/ParameterInput.html`.
## Feature: Stochastic Variables
### Stochastic Variables — Steps
1. Create `models/distribution.py` for variable distributions.
1. Implement API routes in `routes/distributions.py`.
1. Write Pydantic schemas and validations.
1. Write unit tests in `tests/unit/test_distribution.py`.
1. Build UI component `components/DistributionEditor.html`.
## Feature: Cost Tracking
### Cost Tracking — Steps
1. Create `models/capex.py` and `models/opex.py`.
1. Implement API routes in `routes/costs.py`.
1. Write Pydantic schemas for CAPEX/OPEX.
1. Write unit tests in `tests/unit/test_costs.py`.
1. Build UI component `components/CostForm.html`.
## Feature: Consumption Tracking
### Consumption Tracking — Steps
1. Create models for consumption: `chemical_consumption.py`, `fuel_consumption.py`, `water_consumption.py`, `scrap_consumption.py`.
1. Implement API routes in `routes/consumption.py`.
1. Write Pydantic schemas for consumption data.
1. Write unit tests in `tests/unit/test_consumption.py`.
1. Build UI component `components/ConsumptionDashboard.html`.
## Feature: Production Output
### Production Output — Steps
1. Create `models/production_output.py`.
1. Implement API routes in `routes/production.py`.
1. Write Pydantic schemas for production output.
1. Write unit tests in `tests/unit/test_production.py`.
1. Build UI component `components/ProductionChart.html`.
## Feature: Equipment Management
### Equipment Management — Steps
1. Create `models/equipment.py` for equipment data.
1. Implement API routes in `routes/equipment.py`.
1. Write Pydantic schemas for equipment.
1. Write unit tests in `tests/unit/test_equipment.py`.
1. Build UI component `components/EquipmentList.html`.
## Feature: Maintenance Logging
### Maintenance Logging — Steps
1. Create `models/maintenance.py` for maintenance events.
1. Implement API routes in `routes/maintenance.py`.
1. Write Pydantic schemas for maintenance logs.
1. Write unit tests in `tests/unit/test_maintenance.py`.
1. Build UI component `components/MaintenanceLog.html`.
## Feature: Monte Carlo Simulation Engine
### Monte Carlo Engine — Steps
1. Implement Monte Carlo logic in `services/simulation.py`.
1. Persist results in `models/simulation_result.py`.
1. Expose endpoint in `routes/simulations.py`.
1. Write integration tests in `tests/unit/test_simulation.py`.
1. Build UI component `components/SimulationRunner.html`.
## Feature: Reporting / Dashboard
### Reporting / Dashboard — Steps
1. Implement report calculations in `services/reporting.py`.
1. Add detailed and summary endpoints in `routes/reporting.py`.
1. Write unit tests in `tests/unit/test_reporting.py`.
1. Enhance UI in `components/Dashboard.html` with charts.
See [UI and Style](../13_ui_and_style.md) for the UI template audit, layout guidance, and next steps.

0
docs/architecture/05_frontend/05_03_theming.md → docs/architecture/05_blocks/05_03_theming.md
View File

17
docs/architecture/07_deployment/07_01_testing_ci.md
View File

@@ -21,10 +21,7 @@ CalMiner uses a combination of unit, integration, and end-to-end tests to ensure
### CI/CD
- Use Gitea Actions for CI/CD; workflows live under `.gitea/workflows/`.
- `test.yml` runs on every push, provisions a temporary Postgres 16 service, waits for readiness, executes the setup script in dry-run and live modes, then fans out into parallel matrix jobs for unit (`pytest tests/unit`) and end-to-end (`pytest tests/e2e`) suites. Playwright browsers install only for the E2E job.
- `build-and-push.yml` runs only after the **Run Tests** workflow finishes successfully (triggered via `workflow_run` on `main`). Once tests pass, it builds the Docker image with `docker/build-push-action@v2`, reuses cache-backed layers, and pushes to the Gitea registry.
- `deploy.yml` runs only after the build workflow reports success on `main`. It connects to the target host (via `appleboy/ssh-action`), pulls the Docker image tagged with the build commit SHA, and restarts the container with that exact image reference.
- Mandatory secrets: `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `REGISTRY_URL`, `SSH_HOST`, `SSH_USERNAME`, `SSH_PRIVATE_KEY`.
- `ci.yml` runs on push and pull requests to `main` and `develop` branches. It provisions a temporary PostgreSQL 15 service, sets up Python 3.11, installs dependencies from `requirements.txt` and `requirements-test.txt`, runs pytest with coverage on all tests, and builds the Docker image.
- Run tests on pull requests to shared branches; enforce coverage target ≥80% (pytest-cov).
### Running Tests
@@ -74,7 +71,7 @@ To run the Playwright tests:
```bash
pytest tests/e2e/
````
```
To run headed mode:
@@ -166,11 +163,11 @@ When adding new workflows, mirror this structure to ensure secrets, caching, and
- Usage sketch (in `test.yml`):
```yaml
- name: Prepare Python environment
uses: ./.gitea/actions/setup-python-env
with:
install-playwright: ${{ matrix.target == 'e2e' }}
db-dry-run: true
- name: Prepare Python environment
uses: ./.gitea/actions/setup-python-env
with:
install-playwright: ${{ matrix.target == 'e2e' }}
db-dry-run: true
```
- Benefits: centralizes proxy logic and dependency installs, reduces duplication across matrix jobs, and keeps future lint/type-check jobs lightweight by disabling database setup.

82
docs/architecture/07_deployment/07_02_database.md Normal file
View File

@@ -0,0 +1,82 @@
# Database Deployment
## Migrations & Baseline
A consolidated baseline migration (`scripts/migrations/000_base.sql`) captures all schema changes required for a fresh installation. The script is idempotent: it creates the `currency` and `measurement_unit` reference tables, provisions the `application_setting` store for configurable UI/system options, ensures consumption and production records expose unit metadata, and enforces the foreign keys used by CAPEX and OPEX.
Configure granular database settings in your PowerShell session before running migrations:
```powershell
$env:DATABASE_DRIVER = 'postgresql'
$env:DATABASE_HOST = 'localhost'
$env:DATABASE_PORT = '5432'
$env:DATABASE_USER = 'calminer'
$env:DATABASE_PASSWORD = 's3cret'
$env:DATABASE_NAME = 'calminer'
$env:DATABASE_SCHEMA = 'public'
python scripts/setup_database.py --run-migrations --seed-data --dry-run
python scripts/setup_database.py --run-migrations --seed-data
```
The dry-run invocation reports which steps would execute without making changes. The live run applies the baseline (if not already recorded in `schema_migrations`) and seeds the reference data relied upon by the UI and API.
> ℹ️ When `--seed-data` is supplied without `--run-migrations`, the bootstrap script automatically applies any pending SQL migrations first so the `application_setting` table (and future settings-backed features) are present before seeding.
>
> ℹ️ The application still accepts `DATABASE_URL` as a fallback if the granular variables are not set.
## Database bootstrap workflow
Provision or refresh a database instance with `scripts/setup_database.py`. Populate the required environment variables (an example lives at `config/setup_test.env.example`) and run:
```powershell
# Load test credentials (PowerShell)
Get-Content .\config\setup_test.env.example |
ForEach-Object {
if ($_ -and -not $_.StartsWith('#')) {
$name, $value = $_ -split '=', 2
Set-Item -Path Env:$name -Value $value
}
}
# Dry-run to inspect the planned actions
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data --dry-run -v
# Execute the full workflow
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data -v
```
Typical log output confirms:
- Admin and application connections succeed for the supplied credentials.
- Database and role creation are idempotent (`already present` when rerun).
- SQLAlchemy metadata either reports missing tables or `All tables already exist`.
- Migrations list pending files and finish with `Applied N migrations` (a new database reports `Applied 1 migrations` for `000_base.sql`).
After a successful run the target database contains all application tables plus `schema_migrations`, and that table records each applied migration file. New installations only record `000_base.sql`; upgraded environments retain historical entries alongside the baseline.
### Seeding reference data
`scripts/seed_data.py` provides targeted control over the baseline datasets when the full setup script is not required:
```powershell
python scripts/seed_data.py --currencies --units --dry-run
python scripts/seed_data.py --currencies --units
```
The seeder upserts the canonical currency catalog (`USD`, `EUR`, `CLP`, `RMB`, `GBP`, `CAD`, `AUD`) using ASCII-safe symbols (`USD$`, `EUR`, etc.) and the measurement units referenced by the UI (`tonnes`, `kilograms`, `pounds`, `liters`, `cubic_meters`, `kilowatt_hours`). The setup script invokes the same seeder when `--seed-data` is provided and verifies the expected rows afterward, warning if any are missing or inactive.
### Rollback guidance
`scripts/setup_database.py` now tracks compensating actions when it creates the database or application role. If a later step fails, the script replays those rollback actions (dropping the newly created database or role and revoking grants) before exiting. Dry runs never register rollback steps and remain read-only.
If the script reports that some rollback steps could not complete—for example because a connection cannot be established—rerun the script with `--dry-run` to confirm the desired end state and then apply the outstanding cleanup manually:
```powershell
python scripts/setup_database.py --ensure-database --ensure-role --dry-run -v
# Manual cleanup examples when automation cannot connect
psql -d postgres -c "DROP DATABASE IF EXISTS calminer"
psql -d postgres -c "DROP ROLE IF EXISTS calminer"
```
After a failure and rollback, rerun the full setup once the environment issues are resolved.

124
docs/architecture/13_ui_and_style.md
View File

@@ -1,124 +0,0 @@
# 13 — UI, templates and styling
This chapter collects UI integration notes, reusable template components, styling audit points and per-page UI data/actions.
## Reusable Template Components
To reduce duplication across form-centric pages, shared Jinja macros live in `templates/partials/components.html`.
- `select_field(...)`: renders labeled `<select>` controls with consistent placeholder handling and optional preselection. Existing JavaScript modules continue to target the generated IDs, so template calls must pass the same identifiers (`consumption-form-scenario`, etc.).
- `feedback(...)` and `empty_state(...)`: wrap status messages in standard classes (`feedback`, `empty-state`) with optional `hidden` toggles so scripts can control visibility without reimplementing markup.
- `table_container(...)`: provides a semantic wrapper and optional heading around tabular content; the `{% call %}` body supplies the `<thead>`, `<tbody>`, and `<tfoot>` elements while the macro applies the `table-container` class and manages hidden state.
Pages like `templates/consumption.html` and `templates/costs.html` already consume these helpers to keep markup aligned while preserving existing JavaScript selectors.
Import macros via:
```jinja
{% from "partials/components.html" import select_field, feedback, table_container with context %}
```
## Styling Audit Notes (2025-10-21)
- **Spacing**: Panels (`section.panel`) sometimes lack consistent vertical rhythm between headings, form grids, and tables. Extra top/bottom margin utilities would help align content.
- **Typography**: Headings rely on browser defaults; font-size scale is uneven between `<h2>` and `<h3>`. Define explicit scale tokens (e.g., `--font-size-lg`) for predictable sizing.
- **Forms**: `.form-grid` uses fixed column gaps that collapse on small screens; introduce responsive grid rules to stack gracefully below ~768px.
- **Tables**: `.table-container` wrappers need overflow handling for narrow viewports; consider `overflow-x: auto` with padding adjustments.
- **Feedback/Empty states**: Messages use default font weight and spacing; a utility class for margin/padding would ensure consistent separation from forms or tables.
## CSS Variable Naming Conventions
The project adheres to a clear and descriptive naming convention for CSS variables, primarily defined in `static/css/main.css`.
## Naming Structure
Variables are prefixed based on their category:
- `--color-`: For all color-related variables (e.g., `--color-primary`, `--color-background`, `--color-text-primary`).
- `--space-`: For spacing and layout-related variables (e.g., `--space-sm`, `--space-md`, `--space-lg`).
- `--font-size-`: For font size variables (e.g., `--font-size-base`, `--font-size-lg`).
- Other specific prefixes for components or properties (e.g., `--panel-radius`, `--table-radius`).
## Descriptive Names
Color names are chosen to be semantically meaningful rather than literal color values, allowing for easier theme changes. For example:
- `--color-primary`: Represents the main brand color.
- `--color-accent`: Represents an accent color used for highlights.
- `--color-text-primary`: The main text color.
- `--color-text-muted`: A lighter text color for less emphasis.
- `--color-surface`: The background color for UI elements like cards or panels.
- `--color-background`: The overall page background color.
This approach ensures that the CSS variables are intuitive, maintainable, and easily adaptable for future theme customizations.
## Per-page data & actions
Short reference of per-page APIs and primary actions used by templates and scripts.
- Scenarios (`templates/ScenarioForm.html`):
- Data: `GET /api/scenarios/`
- Actions: `POST /api/scenarios/`
- Parameters (`templates/ParameterInput.html`):
- Data: `GET /api/scenarios/`, `GET /api/parameters/`
- Actions: `POST /api/parameters/`
- Costs (`templates/costs.html`):
- Data: `GET /api/costs/capex`, `GET /api/costs/opex`
- Actions: `POST /api/costs/capex`, `POST /api/costs/opex`
- Consumption (`templates/consumption.html`):
- Data: `GET /api/consumption/`
- Actions: `POST /api/consumption/`
- Production (`templates/production.html`):
- Data: `GET /api/production/`
- Actions: `POST /api/production/`
- Equipment (`templates/equipment.html`):
- Data: `GET /api/equipment/`
- Actions: `POST /api/equipment/`
- Maintenance (`templates/maintenance.html`):
- Data: `GET /api/maintenance/` (pagination support)
- Actions: `POST /api/maintenance/`, `PUT /api/maintenance/{id}`, `DELETE /api/maintenance/{id}`
- Simulations (`templates/simulations.html`):
- Data: `GET /api/scenarios/`, `GET /api/parameters/`
- Actions: `POST /api/simulations/run`
- Reporting (`templates/reporting.html` and `templates/Dashboard.html`):
- Data: `POST /api/reporting/summary` (accepts arrays of `{ "result": float }` objects)
- Actions: Trigger summary refreshes and export/download actions.
## Navigation Structure
The application uses a sidebar navigation menu organized into the following top-level categories:
- **Dashboard**: Main overview page.
- **Overview**: Sub-menu for core scenario inputs.
- Parameters: Process parameters configuration.
- Costs: Capital and operating costs.
- Consumption: Resource consumption tracking.
- Production: Production output settings.
- Equipment: Equipment inventory (with Maintenance sub-item).
- **Simulations**: Monte Carlo simulation runs.
- **Analytics**: Reporting and analytics.
- **Settings**: Administrative settings (with Themes and Currency Management sub-items).
## UI Template Audit (2025-10-20)
- Existing HTML templates: `ScenarioForm.html`, `ParameterInput.html`, and `Dashboard.html` (reporting summary view).
- Coverage gaps remain for costs, consumption, production, equipment, maintenance, and simulation workflows—no dedicated templates yet.
- Shared layout primitives (navigation/header/footer) are absent; current pages duplicate boilerplate markup.
- Dashboard currently covers reporting metrics but should be wired to a central `/` route once the shared layout lands.
- Next steps: introduce a `base.html`, refactor existing templates to extend it, and scaffold placeholder pages for the remaining features.

77
docs/architecture/15_development_setup.md
View File

@@ -1,77 +0,0 @@
# 15 Development Setup Guide
This document outlines the local development environment and steps to get the project running.
## Prerequisites
- Python (version 3.10+)
- PostgreSQL (version 13+)
- Git
## Clone and Project Setup
````powershell
# Clone the repository
git clone https://git.allucanget.biz/allucanget/calminer.git
cd calminer
```python
## Virtual Environment
```powershell
# Create and activate a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```python
## Install Dependencies
```powershell
pip install -r requirements.txt
```python
## Database Setup
1. Create database user:
```sql
CREATE USER calminer_user WITH PASSWORD 'your_password';
````
1. Create database:
````sql
CREATE DATABASE calminer;
```python
## Environment Variables
1. Copy `.env.example` to `.env` at project root.
1. Edit `.env` to set database connection details:
```dotenv
DATABASE_DRIVER=postgresql
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USER=calminer_user
DATABASE_PASSWORD=your_password
DATABASE_NAME=calminer
DATABASE_SCHEMA=public
````
1. The application uses `python-dotenv` to load these variables. A legacy `DATABASE_URL` value is still accepted if the granular keys are omitted.
## Running the Application
````powershell
# Start the FastAPI server
uvicorn main:app --reload
```python
## Testing
```powershell
pytest
````
E2E tests use Playwright and a session-scoped `live_server` fixture that starts the app at `http://localhost:8001` for browser-driven tests.

26
docs/architecture/README.md
View File

@@ -1,6 +1,6 @@
---
title: "CalMiner Architecture Documentation"
description: "arc42-based architecture documentation for the CalMiner project"
title: 'CalMiner Architecture Documentation'
description: 'arc42-based architecture documentation for the CalMiner project'
---
# Architecture documentation (arc42 mapping)
@@ -11,16 +11,32 @@ This folder mirrors the arc42 chapter structure (adapted to Markdown).
- [01 Introduction and Goals](01_introduction_and_goals.md)
- [02 Architecture Constraints](02_architecture_constraints.md)
- [02_01 Technical Constraints](02_constraints/02_01_technical_constraints.md)
- [02_02 Organizational Constraints](02_constraints/02_02_organizational_constraints.md)
- [02_03 Regulatory Constraints](02_constraints/02_03_regulatory_constraints.md)
- [02_04 Environmental Constraints](02_constraints/02_04_environmental_constraints.md)
- [02_05 Performance Constraints](02_constraints/02_05_performance_constraints.md)
- [03 Context and Scope](03_context_and_scope.md)
- [03_01 Architecture Scope](03_scope/03_01_architecture_scope.md)
- [04 Solution Strategy](04_solution_strategy.md)
- [04_01 Client-Server Architecture](04_strategy/04_01_client_server_architecture.md)
- [04_02 Technology Choices](04_strategy/04_02_technology_choices.md)
- [04_03 Trade-offs](04_strategy/04_03_trade_offs.md)
- [04_04 Future Considerations](04_strategy/04_04_future_considerations.md)
- [05 Building Block View](05_building_block_view.md)
- [05_01 Architecture Overview](05_blocks/05_01_architecture_overview.md)
- [05_02 Backend Components](05_blocks/05_02_backend_components.md)
- [05_03 Frontend Components](05_blocks/05_03_frontend_components.md)
- [05_03 Theming](05_blocks/05_03_theming.md)
- [05_04 Middleware & Utilities](05_blocks/05_04_middleware_utilities.md)
- [06 Runtime View](06_runtime_view.md)
- [07 Deployment View](07_deployment_view.md)
- [Testing & CI](07_deployment/07_01_testing_ci.md.md)
- [07_01 Testing & CI](07_deployment/07_01_testing_ci.md.md)
- [07_02 Database](07_deployment/07_02_database.md)
- [08 Concepts](08_concepts.md)
- [08_01 Security](08_concepts/08_01_security.md)
- [08_02 Data Models](08_concepts/08_02_data_models.md)
- [09 Architecture Decisions](09_architecture_decisions.md)
- [10 Quality Requirements](10_quality_requirements.md)
- [11 Technical Risks](11_technical_risks.md)
- [12 Glossary](12_glossary.md)
- [13 UI and Style](13_ui_and_style.md)
- [15 Development Setup](15_development_setup.md)