Add UI and styling documentation; remove idempotency and logging audits
Some checks failed
CI / test (pull_request) Failing after 1m8s
Some checks failed
CI / test (pull_request) Failing after 1m8s
- Introduced a new document outlining UI structure, reusable template components, CSS variable conventions, and per-page data/actions for the CalMiner application. - Removed outdated idempotency audit and logging audit documents as they are no longer relevant. - Updated quickstart guide to streamline developer setup instructions and link to relevant documentation. - Created a roadmap document detailing scenario enhancements and data management strategies. - Deleted the seed data plan document to consolidate information into the setup process. - Refactored setup_database.py for improved logging and error handling during database setup and migration processes.
This commit is contained in:
@@ -21,10 +21,7 @@ CalMiner uses a combination of unit, integration, and end-to-end tests to ensure
|
||||
### CI/CD
|
||||
|
||||
- Use Gitea Actions for CI/CD; workflows live under `.gitea/workflows/`.
|
||||
- `test.yml` runs on every push, provisions a temporary Postgres 16 service, waits for readiness, executes the setup script in dry-run and live modes, then fans out into parallel matrix jobs for unit (`pytest tests/unit`) and end-to-end (`pytest tests/e2e`) suites. Playwright browsers install only for the E2E job.
|
||||
- `build-and-push.yml` runs only after the **Run Tests** workflow finishes successfully (triggered via `workflow_run` on `main`). Once tests pass, it builds the Docker image with `docker/build-push-action@v2`, reuses cache-backed layers, and pushes to the Gitea registry.
|
||||
- `deploy.yml` runs only after the build workflow reports success on `main`. It connects to the target host (via `appleboy/ssh-action`), pulls the Docker image tagged with the build commit SHA, and restarts the container with that exact image reference.
|
||||
- Mandatory secrets: `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `REGISTRY_URL`, `SSH_HOST`, `SSH_USERNAME`, `SSH_PRIVATE_KEY`.
|
||||
- `ci.yml` runs on push and pull requests to `main` and `develop` branches. It provisions a temporary PostgreSQL 15 service, sets up Python 3.11, installs dependencies from `requirements.txt` and `requirements-test.txt`, runs pytest with coverage on all tests, and builds the Docker image.
|
||||
- Run tests on pull requests to shared branches; enforce coverage target ≥80% (pytest-cov).
|
||||
|
||||
### Running Tests
|
||||
@@ -74,7 +71,7 @@ To run the Playwright tests:
|
||||
|
||||
```bash
|
||||
pytest tests/e2e/
|
||||
````
|
||||
```
|
||||
|
||||
To run headed mode:
|
||||
|
||||
@@ -166,11 +163,11 @@ When adding new workflows, mirror this structure to ensure secrets, caching, and
|
||||
- Usage sketch (in `test.yml`):
|
||||
|
||||
```yaml
|
||||
- name: Prepare Python environment
|
||||
uses: ./.gitea/actions/setup-python-env
|
||||
with:
|
||||
install-playwright: ${{ matrix.target == 'e2e' }}
|
||||
db-dry-run: true
|
||||
- name: Prepare Python environment
|
||||
uses: ./.gitea/actions/setup-python-env
|
||||
with:
|
||||
install-playwright: ${{ matrix.target == 'e2e' }}
|
||||
db-dry-run: true
|
||||
```
|
||||
|
||||
- Benefits: centralizes proxy logic and dependency installs, reduces duplication across matrix jobs, and keeps future lint/type-check jobs lightweight by disabling database setup.
|
||||
|
||||
82
docs/architecture/07_deployment/07_02_database.md
Normal file
82
docs/architecture/07_deployment/07_02_database.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Database Deployment
|
||||
|
||||
## Migrations & Baseline
|
||||
|
||||
A consolidated baseline migration (`scripts/migrations/000_base.sql`) captures all schema changes required for a fresh installation. The script is idempotent: it creates the `currency` and `measurement_unit` reference tables, provisions the `application_setting` store for configurable UI/system options, ensures consumption and production records expose unit metadata, and enforces the foreign keys used by CAPEX and OPEX.
|
||||
|
||||
Configure granular database settings in your PowerShell session before running migrations:
|
||||
|
||||
```powershell
|
||||
$env:DATABASE_DRIVER = 'postgresql'
|
||||
$env:DATABASE_HOST = 'localhost'
|
||||
$env:DATABASE_PORT = '5432'
|
||||
$env:DATABASE_USER = 'calminer'
|
||||
$env:DATABASE_PASSWORD = 's3cret'
|
||||
$env:DATABASE_NAME = 'calminer'
|
||||
$env:DATABASE_SCHEMA = 'public'
|
||||
python scripts/setup_database.py --run-migrations --seed-data --dry-run
|
||||
python scripts/setup_database.py --run-migrations --seed-data
|
||||
```
|
||||
|
||||
The dry-run invocation reports which steps would execute without making changes. The live run applies the baseline (if not already recorded in `schema_migrations`) and seeds the reference data relied upon by the UI and API.
|
||||
|
||||
> ℹ️ When `--seed-data` is supplied without `--run-migrations`, the bootstrap script automatically applies any pending SQL migrations first so the `application_setting` table (and future settings-backed features) are present before seeding.
|
||||
>
|
||||
> ℹ️ The application still accepts `DATABASE_URL` as a fallback if the granular variables are not set.
|
||||
|
||||
## Database bootstrap workflow
|
||||
|
||||
Provision or refresh a database instance with `scripts/setup_database.py`. Populate the required environment variables (an example lives at `config/setup_test.env.example`) and run:
|
||||
|
||||
```powershell
|
||||
# Load test credentials (PowerShell)
|
||||
Get-Content .\config\setup_test.env.example |
|
||||
ForEach-Object {
|
||||
if ($_ -and -not $_.StartsWith('#')) {
|
||||
$name, $value = $_ -split '=', 2
|
||||
Set-Item -Path Env:$name -Value $value
|
||||
}
|
||||
}
|
||||
|
||||
# Dry-run to inspect the planned actions
|
||||
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data --dry-run -v
|
||||
|
||||
# Execute the full workflow
|
||||
python scripts/setup_database.py --ensure-database --ensure-role --ensure-schema --initialize-schema --run-migrations --seed-data -v
|
||||
```
|
||||
|
||||
Typical log output confirms:
|
||||
|
||||
- Admin and application connections succeed for the supplied credentials.
|
||||
- Database and role creation are idempotent (`already present` when rerun).
|
||||
- SQLAlchemy metadata either reports missing tables or `All tables already exist`.
|
||||
- Migrations list pending files and finish with `Applied N migrations` (a new database reports `Applied 1 migrations` for `000_base.sql`).
|
||||
|
||||
After a successful run the target database contains all application tables plus `schema_migrations`, and that table records each applied migration file. New installations only record `000_base.sql`; upgraded environments retain historical entries alongside the baseline.
|
||||
|
||||
### Seeding reference data
|
||||
|
||||
`scripts/seed_data.py` provides targeted control over the baseline datasets when the full setup script is not required:
|
||||
|
||||
```powershell
|
||||
python scripts/seed_data.py --currencies --units --dry-run
|
||||
python scripts/seed_data.py --currencies --units
|
||||
```
|
||||
|
||||
The seeder upserts the canonical currency catalog (`USD`, `EUR`, `CLP`, `RMB`, `GBP`, `CAD`, `AUD`) using ASCII-safe symbols (`USD$`, `EUR`, etc.) and the measurement units referenced by the UI (`tonnes`, `kilograms`, `pounds`, `liters`, `cubic_meters`, `kilowatt_hours`). The setup script invokes the same seeder when `--seed-data` is provided and verifies the expected rows afterward, warning if any are missing or inactive.
|
||||
|
||||
### Rollback guidance
|
||||
|
||||
`scripts/setup_database.py` now tracks compensating actions when it creates the database or application role. If a later step fails, the script replays those rollback actions (dropping the newly created database or role and revoking grants) before exiting. Dry runs never register rollback steps and remain read-only.
|
||||
|
||||
If the script reports that some rollback steps could not complete—for example because a connection cannot be established—rerun the script with `--dry-run` to confirm the desired end state and then apply the outstanding cleanup manually:
|
||||
|
||||
```powershell
|
||||
python scripts/setup_database.py --ensure-database --ensure-role --dry-run -v
|
||||
|
||||
# Manual cleanup examples when automation cannot connect
|
||||
psql -d postgres -c "DROP DATABASE IF EXISTS calminer"
|
||||
psql -d postgres -c "DROP ROLE IF EXISTS calminer"
|
||||
```
|
||||
|
||||
After a failure and rollback, rerun the full setup once the environment issues are resolved.
|
||||
Reference in New Issue
Block a user