zwitschi
04d7f202b6
Some checks failed
CI / test (pull_request) Failing after 1m8s
- Introduced a new document outlining UI structure, reusable template components, CSS variable conventions, and per-page data/actions for the CalMiner application. - Removed outdated idempotency audit and logging audit documents as they are no longer relevant. - Updated quickstart guide to streamline developer setup instructions and link to relevant documentation. - Created a roadmap document detailing scenario enhancements and data management strategies. - Deleted the seed data plan document to consolidate information into the setup process. - Refactored setup_database.py for improved logging and error handling during database setup and migration processes.
83 lines
4.6 KiB
Markdown
83 lines
4.6 KiB
Markdown
# Database Deployment
|
||
|
||
## Migrations & Baseline
|
||
|
||
A consolidated baseline migration (`scripts/migrations/000_base.sql`) captures all schema changes required for a fresh installation. The script is idempotent: it creates the `currency` and `measurement_unit` reference tables, provisions the `application_setting` store for configurable UI/system options, ensures consumption and production records expose unit metadata, and enforces the foreign keys used by CAPEX and OPEX.
|
||
|
||
Configure granular database settings in your PowerShell session before running migrations:
|
||
|
||
```powershell
|
||
$env:DATABASE_DRIVER = 'postgresql'
|
||
$env:DATABASE_HOST = 'localhost'
|
||
$env:DATABASE_PORT = '5432'
|
||
$env:DATABASE_USER = 'calminer'
|
||
$env:DATABASE_PASSWORD = 's3cret'
|
||
$env:DATABASE_NAME = 'calminer'
|
||
$env:DATABASE_SCHEMA = 'public'
|
||
python scripts/setup_database.py --run-migrations --seed-data --dry-run
|
||
python scripts/setup_database.py --run-migrations --seed-data
|
||
```
|
||
|
||
The dry-run invocation reports which steps would execute without making changes. The live run applies the baseline (if not already recorded in `schema_migrations`) and seeds the reference data relied upon by the UI and API.
|
||
|
||
> ℹ️ When `--seed-data` is supplied without `--run-migrations`, the bootstrap script automatically applies any pending SQL migrations first so the `application_setting` table (and future settings-backed features) are present before seeding.
|
||
>
|
||
> ℹ️ The application still accepts `DATABASE_URL` as a fallback if the granular variables are not set.
|
||
|
||
## Database bootstrap workflow
|
||
|
||
Provision or refresh a database instance with `scripts/setup_database.py`. Populate the required environment variables (an example lives at `config/setup_test.env.example`) and run:
|
||
|
||
```powershell
|
||
# Load test credentials (PowerShell)
|
||
Get-Content .\config\setup_test.env.example |
|
||
ForEach-Object {
|
||
if ($_ -and -not $_.StartsWith('#')) {
|
||
$name, $value = $_ -split '=', 2
|
||
Set-Item -Path Env:$name -Value $value
|
||
}
|
||
}
|
||
|
||
# Dry-run to inspect the planned actions
|
||
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data --dry-run -v
|
||
|
||
# Execute the full workflow
|
||
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data -v
|
||
```
|
||
|
||
Typical log output confirms:
|
||
|
||
- Admin and application connections succeed for the supplied credentials.
|
||
- Database and role creation are idempotent (`already present` when rerun).
|
||
- SQLAlchemy metadata either reports missing tables or `All tables already exist`.
|
||
- Migrations list pending files and finish with `Applied N migrations` (a new database reports `Applied 1 migrations` for `000_base.sql`).
|
||
|
||
After a successful run the target database contains all application tables plus `schema_migrations`, and that table records each applied migration file. New installations only record `000_base.sql`; upgraded environments retain historical entries alongside the baseline.
|
||
|
||
### Seeding reference data
|
||
|
||
`scripts/seed_data.py` provides targeted control over the baseline datasets when the full setup script is not required:
|
||
|
||
```powershell
|
||
python scripts/seed_data.py --currencies --units --dry-run
|
||
python scripts/seed_data.py --currencies --units
|
||
```
|
||
|
||
The seeder upserts the canonical currency catalog (`USD`, `EUR`, `CLP`, `RMB`, `GBP`, `CAD`, `AUD`) using ASCII-safe symbols (`USD$`, `EUR`, etc.) and the measurement units referenced by the UI (`tonnes`, `kilograms`, `pounds`, `liters`, `cubic_meters`, `kilowatt_hours`). The setup script invokes the same seeder when `--seed-data` is provided and verifies the expected rows afterward, warning if any are missing or inactive.
|
||
|
||
### Rollback guidance
|
||
|
||
`scripts/setup_database.py` now tracks compensating actions when it creates the database or application role. If a later step fails, the script replays those rollback actions (dropping the newly created database or role and revoking grants) before exiting. Dry runs never register rollback steps and remain read-only.
|
||
|
||
If the script reports that some rollback steps could not complete—for example because a connection cannot be established—rerun the script with `--dry-run` to confirm the desired end state and then apply the outstanding cleanup manually:
|
||
|
||
```powershell
|
||
python scripts/setup_database.py --ensure-database --ensure-role --dry-run -v
|
||
|
||
# Manual cleanup examples when automation cannot connect
|
||
psql -d postgres -c "DROP DATABASE IF EXISTS calminer"
|
||
psql -d postgres -c "DROP ROLE IF EXISTS calminer"
|
||
```
|
||
|
||
After a failure and rollback, rerun the full setup once the environment issues are resolved.
|