Add UI and styling documentation; remove idempotency and logging audits

- Introduced a new document outlining UI structure, reusable template components, CSS variable conventions, and per-page data/actions for the CalMiner application. - Removed outdated idempotency audit and logging audit documents as they are no longer relevant. - Updated quickstart guide to streamline developer setup instructions and link to relevant documentation. - Created a roadmap document detailing scenario enhancements and data management strategies. - Deleted the seed data plan document to consolidate information into the setup process. - Refactored setup_database.py for improved logging and error handling during database setup and migration processes.
2025-10-29 13:20:44 +01:00
parent 1f58de448c
commit 04d7f202b6
19 changed files with 609 additions and 752 deletions
--- a/docs/developer/development_setup.md
+++ b/docs/developer/development_setup.md
@@ -0,0 +1,104 @@
+# Development Environment Setup
+
+This document outlines the local development environment and steps to get the project running.
+
+## Prerequisites
+
+- Python (version 3.11+)
+- PostgreSQL (version 13+)
+- Git
+- Docker and Docker Compose (optional, for containerized development)
+
+## Clone and Project Setup
+
+```powershell
+# Clone the repository
+git clone https://git.allucanget.biz/allucanget/calminer.git
+cd calminer
+```
+
+## Development with Docker Compose (Recommended)
+
+For a quick setup without installing PostgreSQL locally, use Docker Compose:
+
+```powershell
+# Start services
+docker-compose up
+
+# The app will be available at http://localhost:8000
+# Database is automatically set up
+```
+
+To run in background:
+
+```powershell
+docker-compose up -d
+```
+
+To stop:
+
+```powershell
+docker-compose down
+```
+
+## Manual Development Setup
+
+### Virtual Environment
+
+```powershell
+# Create and activate a virtual environment
+python -m venv .venv
+.\.venv\Scripts\Activate.ps1
+```
+
+### Install Dependencies
+
+```powershell
+pip install -r requirements.txt
+```
+
+### Database Setup
+
+1. Create database user:
+
+```sql
+CREATE USER calminer_user WITH PASSWORD 'your_password';
+```
+
+1. Create database:
+
+```sql
+CREATE DATABASE calminer;
+```
+
+### Environment Variables
+
+1. Copy `.env.example` to `.env` at project root.
+1. Edit `.env` to set database connection details:
+
+```dotenv
+DATABASE_DRIVER=postgresql
+DATABASE_HOST=localhost
+DATABASE_PORT=5432
+DATABASE_USER=calminer_user
+DATABASE_PASSWORD=your_password
+DATABASE_NAME=calminer
+DATABASE_SCHEMA=public
+```
+
+1. The application uses `python-dotenv` to load these variables. A legacy `DATABASE_URL` value is still accepted if the granular keys are omitted.
+
+### Running the Application
+
+```powershell
+# Start the FastAPI server
+uvicorn main:app --reload
+```
+
+## Testing
+
+```powershell
+pytest
+```
+
+E2E tests use Playwright and a session-scoped `live_server` fixture that starts the app at `http://localhost:8001` for browser-driven tests.
--- a/docs/developer/staging_environment_setup.md
+++ b/docs/developer/staging_environment_setup.md
@@ -0,0 +1,100 @@
+# Staging Environment Setup
+
+This guide outlines how to provision and validate the CalMiner staging database using `scripts/setup_database.py`. It complements the local and CI-focused instructions in `docs/quickstart.md`.
+
+## Prerequisites
+
+- Network access to the staging infrastructure (VPN or bastion, as required by ops).
+- Provisioned PostgreSQL instance with superuser or delegated admin credentials for maintenance.
+- Application credentials (role + password) dedicated to CalMiner staging.
+- The application repository checked out with Python dependencies installed (`pip install -r requirements.txt`).
+- Optional but recommended: a writable directory (for example `reports/`) to capture setup logs.
+
+> Replace the placeholder values in the examples below with the actual host, port, and credential details supplied by ops.
+
+## Environment Configuration
+
+Populate the following environment variables before invoking the setup script. Store them in a secure location such as `config/setup_staging.env` (excluded from source control) and load them with `dotenv` or your shell profile.
+
+| Variable                      | Description                                                                               |
+| ----------------------------- | ----------------------------------------------------------------------------------------- |
+| `DATABASE_HOST`               | Staging PostgreSQL hostname or IP (for example `staging-db.internal`).                    |
+| `DATABASE_PORT`               | Port exposed by the staging PostgreSQL service (default `5432`).                          |
+| `DATABASE_NAME`               | CalMiner staging database name (for example `calminer_staging`).                          |
+| `DATABASE_USER`               | Application role used by the FastAPI app (for example `calminer_app`).                    |
+| `DATABASE_PASSWORD`           | Password for the application role.                                                        |
+| `DATABASE_SCHEMA`             | Optional non-public schema; omit or set to `public` otherwise.                            |
+| `DATABASE_SUPERUSER`          | Administrative role with rights to create roles/databases (for example `calminer_admin`). |
+| `DATABASE_SUPERUSER_PASSWORD` | Password for the administrative role.                                                     |
+| `DATABASE_SUPERUSER_DB`       | Database to connect to for admin tasks (default `postgres`).                              |
+| `DATABASE_ADMIN_URL`          | Optional DSN that overrides the granular admin settings above.                            |
+
+You may also set `DATABASE_URL` for application runtime convenience, but the setup script only requires the values listed in the table.
+
+### Loading Variables (PowerShell example)
+
+```powershell
+$env:DATABASE_HOST = "staging-db.internal"
+$env:DATABASE_PORT = "5432"
+$env:DATABASE_NAME = "calminer_staging"
+$env:DATABASE_USER = "calminer_app"
+$env:DATABASE_PASSWORD = "<app-password>"
+$env:DATABASE_SUPERUSER = "calminer_admin"
+$env:DATABASE_SUPERUSER_PASSWORD = "<admin-password>"
+$env:DATABASE_SUPERUSER_DB = "postgres"
+```
+
+For bash shells, export the same variables using `export VARIABLE=value` or load them through `dotenv`.
+
+## Setup Workflow
+
+Run the setup script in three phases to validate idempotency and capture diagnostics:
+
+1. **Dry run (diagnostic):**
+
+   ```powershell
+   python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data --dry-run -v `
+     2>&1 | Tee-Object -FilePath reports/setup_staging_dry_run.log
+   ```
+
+   Confirm that the script reports planned actions without failures. If the application role is missing, a dry run will log skip messages until a live run creates the role.
+
+2. **Apply changes:**
+
+   ```powershell
+   python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data -v `
+     2>&1 | Tee-Object -FilePath reports/setup_staging_apply.log
+   ```
+
+   Verify the log for successful database creation, role grants, migration execution, and seed verification.
+
+3. **Post-apply dry run:**
+
+   ```powershell
+   python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data --dry-run -v `
+     2>&1 | Tee-Object -FilePath reports/setup_staging_post_apply.log
+   ```
+
+   This run should confirm that all schema objects, migrations, and seed data are already in place.
+
+## Validation Checklist
+
+- [ ] Confirm the staging application can connect using the application DSN (for example, run `pytest tests/e2e/test_smoke.py` against staging or trigger a smoke test workflow).
+- [ ] Inspect `schema_migrations` to ensure the baseline migration (`000_base.sql`) is recorded.
+- [ ] Spot-check seeded reference data (`currency`, `measurement_unit`) for correctness.
+- [ ] Capture and archive the three setup logs in a shared location for audit purposes.
+
+## Troubleshooting
+
+- If the dry run reports skipped actions because the application role does not exist, proceed with the live run; subsequent dry runs will validate as expected.
+- Connection errors usually stem from network restrictions or incorrect credentials. Validate reachability with `psql` or `pg_isready` using the same host/port and credentials.
+- For permission issues during migrations or seeding, confirm the admin role has rights on the target database and that the application role inherits the expected privileges.
+
+## Rollback Guidance
+
+- Database creation and role grants register rollback actions when not running in dry-run mode. If a later step fails, rerun the script without `--dry-run`; it will automatically revoke grants or drop newly created resources as part of the rollback routine.
+- For staged environments where manual intervention is required, coordinate with ops before dropping databases or roles.
+
+## Next Steps
+
+- Keep this document updated as staging infrastructure evolves (for example, when migrating to managed services or rotating credentials).
--- a/docs/developer/ui_and_style.md
+++ b/docs/developer/ui_and_style.md
@@ -0,0 +1,124 @@
+# UI, templates and styling
+
+This document outlines the UI structure, template components, CSS variable conventions, and per-page data/actions for the CalMiner application.
+
+## Reusable Template Components
+
+To reduce duplication across form-centric pages, shared Jinja macros live in `templates/partials/components.html`.
+
+- `select_field(...)`: renders labeled `<select>` controls with consistent placeholder handling and optional preselection. Existing JavaScript modules continue to target the generated IDs, so template calls must pass the same identifiers (`consumption-form-scenario`, etc.).
+- `feedback(...)` and `empty_state(...)`: wrap status messages in standard classes (`feedback`, `empty-state`) with optional `hidden` toggles so scripts can control visibility without reimplementing markup.
+- `table_container(...)`: provides a semantic wrapper and optional heading around tabular content; the `{% call %}` body supplies the `<thead>`, `<tbody>`, and `<tfoot>` elements while the macro applies the `table-container` class and manages hidden state.
+
+Pages like `templates/consumption.html` and `templates/costs.html` already consume these helpers to keep markup aligned while preserving existing JavaScript selectors.
+
+Import macros via:
+
+```jinja
+{% from "partials/components.html" import select_field, feedback, table_container with context %}
+```
+
+## Styling Audit Notes (2025-10-21)
+
+- **Spacing**: Panels (`section.panel`) sometimes lack consistent vertical rhythm between headings, form grids, and tables. Extra top/bottom margin utilities would help align content.
+- **Typography**: Headings rely on browser defaults; font-size scale is uneven between `<h2>` and `<h3>`. Define explicit scale tokens (e.g., `--font-size-lg`) for predictable sizing.
+- **Forms**: `.form-grid` uses fixed column gaps that collapse on small screens; introduce responsive grid rules to stack gracefully below ~768px.
+- **Tables**: `.table-container` wrappers need overflow handling for narrow viewports; consider `overflow-x: auto` with padding adjustments.
+- **Feedback/Empty states**: Messages use default font weight and spacing; a utility class for margin/padding would ensure consistent separation from forms or tables.
+
+## CSS Variable Naming Conventions
+
+The project adheres to a clear and descriptive naming convention for CSS variables, primarily defined in `static/css/main.css`.
+
+## Naming Structure
+
+Variables are prefixed based on their category:
+
+- `--color-`: For all color-related variables (e.g., `--color-primary`, `--color-background`, `--color-text-primary`).
+- `--space-`: For spacing and layout-related variables (e.g., `--space-sm`, `--space-md`, `--space-lg`).
+- `--font-size-`: For font size variables (e.g., `--font-size-base`, `--font-size-lg`).
+- Other specific prefixes for components or properties (e.g., `--panel-radius`, `--table-radius`).
+
+## Descriptive Names
+
+Color names are chosen to be semantically meaningful rather than literal color values, allowing for easier theme changes. For example:
+
+- `--color-primary`: Represents the main brand color.
+- `--color-accent`: Represents an accent color used for highlights.
+- `--color-text-primary`: The main text color.
+- `--color-text-muted`: A lighter text color for less emphasis.
+- `--color-surface`: The background color for UI elements like cards or panels.
+- `--color-background`: The overall page background color.
+
+This approach ensures that the CSS variables are intuitive, maintainable, and easily adaptable for future theme customizations.
+
+## Per-page data & actions
+
+Short reference of per-page APIs and primary actions used by templates and scripts.
+
+- Scenarios (`templates/ScenarioForm.html`):
+
+  - Data: `GET /api/scenarios/`
+  - Actions: `POST /api/scenarios/`
+
+- Parameters (`templates/ParameterInput.html`):
+
+  - Data: `GET /api/scenarios/`, `GET /api/parameters/`
+  - Actions: `POST /api/parameters/`
+
+- Costs (`templates/costs.html`):
+
+  - Data: `GET /api/costs/capex`, `GET /api/costs/opex`
+  - Actions: `POST /api/costs/capex`, `POST /api/costs/opex`
+
+- Consumption (`templates/consumption.html`):
+
+  - Data: `GET /api/consumption/`
+  - Actions: `POST /api/consumption/`
+
+- Production (`templates/production.html`):
+
+  - Data: `GET /api/production/`
+  - Actions: `POST /api/production/`
+
+- Equipment (`templates/equipment.html`):
+
+  - Data: `GET /api/equipment/`
+  - Actions: `POST /api/equipment/`
+
+- Maintenance (`templates/maintenance.html`):
+
+  - Data: `GET /api/maintenance/` (pagination support)
+  - Actions: `POST /api/maintenance/`, `PUT /api/maintenance/{id}`, `DELETE /api/maintenance/{id}`
+
+- Simulations (`templates/simulations.html`):
+
+  - Data: `GET /api/scenarios/`, `GET /api/parameters/`
+  - Actions: `POST /api/simulations/run`
+
+- Reporting (`templates/reporting.html` and `templates/Dashboard.html`):
+  - Data: `POST /api/reporting/summary` (accepts arrays of `{ "result": float }` objects)
+  - Actions: Trigger summary refreshes and export/download actions.
+
+## Navigation Structure
+
+The application uses a sidebar navigation menu organized into the following top-level categories:
+
+- **Dashboard**: Main overview page.
+- **Overview**: Sub-menu for core scenario inputs.
+  - Parameters: Process parameters configuration.
+  - Costs: Capital and operating costs.
+  - Consumption: Resource consumption tracking.
+  - Production: Production output settings.
+  - Equipment: Equipment inventory (with Maintenance sub-item).
+- **Simulations**: Monte Carlo simulation runs.
+- **Analytics**: Reporting and analytics.
+- **Settings**: Administrative settings (with Themes and Currency Management sub-items).
+
+## UI Template Audit (2025-10-20)
+
+- Existing HTML templates: `ScenarioForm.html`, `ParameterInput.html`, and `Dashboard.html` (reporting summary view).
+- Coverage gaps remain for costs, consumption, production, equipment, maintenance, and simulation workflows—no dedicated templates yet.
+- Shared layout primitives (navigation/header/footer) are absent; current pages duplicate boilerplate markup.
+- Dashboard currently covers reporting metrics but should be wired to a central `/` route once the shared layout lands.
+- Next steps: introduce a `base.html`, refactor existing templates to extend it, and scaffold placeholder pages for the remaining features.