allucanget/calminer
2
0
Fork 0
Code Pull Requests Actions Packages Releases Activity

Add comprehensive architecture documentation and related scripts

Browse Source 
- Introduced multiple architecture documentation files covering building block view, runtime view, deployment view, concepts, architecture decisions, quality requirements, technical risks, glossary, UI and styling, testing, CI, and development setup.
- Migrated existing content from `architecture_overview.md` and `implementation_plan.md` into structured documentation.
- Created scripts for checking broken links in documentation and formatting Markdown files for consistency.
- Updated quickstart guide to provide clearer setup instructions and usage overview.
- Removed outdated MVP features and testing strategy documents to streamline documentation.
This commit is contained in:
zwitschi
2025-10-21 15:39:17 +02:00
parent 2bae273d9e
commit 4b3a15ed15
26 changed files with 796 additions and 675 deletions
Download Patch File Download Diff File Expand all files Collapse all files

167
README.md
View File

@@ -24,169 +24,10 @@ A range of features are implemented to support these functionalities.
- **Modular Frontend Scripts**: Page-specific interactions now live in `static/js/` modules, keeping templates lean while enabling browser caching and reuse.
- **Monte Carlo Simulation (in progress)**: Services and routes are scaffolded for future stochastic analysis.
## Architecture
## Documentation & quickstart
The architecture is documented in [docs/architecture.md](docs/architecture.md).
This repository contains detailed developer and architecture documentation in the `docs/` folder. For a short quickstart, troubleshooting notes, migration/backfill instructions and the current implementation status, see `docs/quickstart.md`.
## Project Structure
Key architecture documents: see `docs/architecture/README.md` for the arc42-based architecture documentation.
The project is organized into several key directories:
- `models/`: Contains SQLAlchemy models representing database tables.
- `routes/`: Defines FastAPI routes for API endpoints; shared dependencies like `get_db` live in `routes/dependencies.py`.
- `services/`: Business logic and service layer.
- `components/`: Frontend components (to be defined).
- `config/`: Configuration files and settings.
- `middleware/`: Custom middleware for request/response processing.
- `tests/`: Unit and integration tests.
- `templates/`: Jinja2 HTML templates for server-side rendering.
- `docs/`: Documentation files.
Key files include:
- `main.py`: FastAPI application entry point.
- `.env`: Environment variables for configuration.
- `requirements.txt`: Python dependencies.
## Development
The development setup instructions are provided in [docs/development_setup.md](docs/development_setup.md).
To get started locally:
```powershell
# Clone the repository
git clone https://git.allucanget.biz/allucanget/calminer.git
cd calminer
# Create and activate a virtual environment
python -m venv .venv
.\.venv\Scripts\Activate.ps1
# Install dependencies
pip install -r requirements.txt
# Start the development server
uvicorn main:app --reload
```
## Usage Overview
- **API base URL**: `http://localhost:8000/api`
- **Key routes**:
- `POST /api/scenarios/` create scenarios
- `POST /api/parameters/` manage process parameters; payload supports optional `distribution_id` or inline `distribution_type`/`distribution_parameters` fields for simulation metadata
- `POST /api/costs/capex` and `POST /api/costs/opex` capture project costs
- `POST /api/consumption/` add consumption entries
- `POST /api/production/` register production output
- `POST /api/equipment/` create equipment records
- `POST /api/maintenance/` log maintenance events
- `POST /api/reporting/summary` aggregate simulation results, returning count, mean/median, min/max, standard deviation, variance, percentile bands (5/10/90/95), value-at-risk (95%) and expected shortfall (95%)
- **UI entries** (rendered via FastAPI templates, also reachable from the sidebar):
- `GET /` operations overview dashboard
- `GET /ui/dashboard` legacy dashboard alias
- `GET /ui/scenarios` scenario creation form
- `GET /ui/parameters` parameter input form
- `GET /ui/costs`, `/ui/consumption`, `/ui/production`, `/ui/equipment`, `/ui/maintenance`, `/ui/simulations`, `/ui/reporting` placeholder views aligned with future integrations
### Dashboard Preview
1. Start the FastAPI server and navigate to `/`.
2. Review the headline metrics, scenario snapshot table, and cost/activity charts sourced from the current database state.
3. Use the "Refresh Dashboard" button to pull freshly aggregated data via `/ui/dashboard/data` without reloading the page.
4. Populate scenarios, costs, production, consumption, simulations, and maintenance records to see charts and lists update.
5. The legacy `/ui/dashboard` route remains available but now serves the same consolidated overview.
## Testing
Testing guidelines and best practices are outlined in [docs/testing.md](docs/testing.md).
To execute the unit test suite:
```powershell
pytest
```
### End-to-End Tests
- Playwright-based E2E tests rely on a session-scoped `live_server` fixture that auto-starts the FastAPI app on `http://localhost:8001`, so no per-test `@pytest.mark.usefixtures("live_server")` annotations are required.
- The fixture now polls `[http://localhost:8001](http://localhost:8001)` until it responds (up to ~30s), ensuring the uvicorn subprocess is ready before Playwright starts navigation, then preloads `/` and waits for a `networkidle` state so sidebar navigation and global assets are ready for each test.
- Latest run (`pytest tests/e2e/` on 2025-10-21) passes end-to-end smoke and form coverage after aligning form selectors, titles, and the live server startup behaviour.
### Coverage Snapshot (2025-10-21)
- `pytest --cov=. --cov-report=term-missing` reports **91%** overall coverage.
- Recent additions pushed `routes/ui.py` and `services/simulation.py` to 100%; remaining gaps are concentrated in `config/database.py`, several `models/*.py` loaders, and `services/reporting.py` (95%).
- Playwright specs under `tests/e2e/` are excluded from the coverage run to keep browser automation optional; their files show as uncovered because they are not executed in the `pytest --cov` workflow.
## Database Objects
The database is composed of several tables that store different types of information.
- **CAPEX** — `capex`: Stores data on capital expenditures.
- **OPEX** — `opex`: Contains information on operational expenditures.
- **Chemical consumption** — `chemical_consumption`: Tracks the consumption of chemical reagents.
- **Fuel consumption** — `fuel_consumption`: Records the amount of fuel consumed.
- **Water consumption** — `water_consumption`: Monitors the use of water.
- **Scrap consumption** — `scrap_consumption`: Tracks the consumption of scrap materials.
- **Production output** — `production_output`: Stores data on production output, such as tons produced and recovery rates.
- **Equipment operation** — `equipment_operation`: Contains operational data for each piece of equipment.
- **Ore batch** — `ore_batch`: Stores information on ore batches, including their grade and other characteristics.
- **Exchange rate** — `exchange_rate`: Contains currency exchange rates.
- **Simulation result** — `simulation_result`: Stores the results of the Monte Carlo simulations.
### Currency normalization and migrations
The project now includes a referential `currency` table and associated migration and backfill tooling to normalize free-text currency fields into a canonical lookup.
- New model: `models/currency.py` implements the `currency` table (id, code, name, symbol, is_active).
- Database migration: `scripts/migrations/20251022_create_currency_table_and_fks.sql` creates the `currency` table, seeds common currencies, adds `currency_id` to `capex`/`opex`, backfills where possible, and adds foreign key constraints. The migration is written to be idempotent and safe on databases where parts of the work were already applied.
- Backfill utility: `scripts/backfill_currency.py` is an idempotent, developer-friendly script to preview and apply backfill updates. It supports `--dry-run` and `--create-missing` flags.
- API & models: `models/capex.py` and `models/opex.py` now reference `currency_id` and include a compatibility property accepting a legacy `currency_code` during creates/updates. Routes accept either `currency_id` or `currency_code`; see `routes/costs.py` and the new `routes/currencies.py` (GET /api/currencies/).
- Frontend: server templates and `static/js` now fetch `/api/currencies/` to populate currency selects for cost forms so users can pick a normalized currency instance.
How to run migrations and backfill (development)
1. Ensure `DATABASE_URL` is set in your PowerShell session to point at a development Postgres instance.
```powershell
$env:DATABASE_URL = 'postgresql://user:pass@host/db'
python scripts/run_migrations.py
python scripts/backfill_currency.py --dry-run
python scripts/backfill_currency.py --create-missing
```
Use `--dry-run` first to verify what will change. The migration and backfill scripts are designed to be safe and idempotent, but always back up production databases before applying schema changes.
## Static Parameters
These are values that are not expected to change frequently and are used for configuration purposes. Some examples include:
- **Currencies**: `currency_code`, `currency_name`.
- **Distribution types**: `distribution_name`.
- **Units**: `unit_name`, `unit_symbol`, `unit_system`, `conversion_to_base`.
- **Parameter categories**: `category_name`.
- **Material types**: `type_name`, `category`.
- **Chemical reagents**: `reagent_name`, `chemical_formula`.
- **Fuel**: `fuel_name`.
- **Water**: `water_type`.
- **Scrap material**: `scrap_name`.
## Variables
These are dynamic data points that are recorded over time and used in calculations and simulations. Some examples include:
- **CAPEX**: `amount`.
- **OPEX**: `amount`.
- **Chemical consumption**: `quantity`, `efficiency`, `waste_factor`.
- **Fuel consumption**: `quantity`.
- **Water consumption**: `quantity`.
- **Scrap consumption**: `quantity`.
- **Production output**: `tons_produced`, `recovery_rate`, `metal_content`, `metallurgical_loss`, `net_revenue`.
- **Equipment operation**: `hours_operated`, `downtime_hours`.
- **Ore batch**: `ore_grade`, `moisture`, `sulfur`, `chlorine`.
- **Exchange rate**: `rate`.
- **Parameter values**: `value`.
- **Simulation result**: NPV (`npv`), IRR (`irr`), EBITDA (`ebitda`), `net_revenue`.
- **Cementation parameters**: `temperature`, pH (`ph`), `reaction_time`, `copper_concentration`, `iron_surface_area`.
- **Precipitate product**: `density`, `melting_point`, `boiling_point`.
For contributors: the `routes/`, `models/` and `services/` folders contain the primary application code. Tests and E2E specs are in `tests/`.