feat: Update CI workflows for Docker image build and deployment, enhance test configurations, and add testing documentation

docs/architecture/07_deployment/07_01_testing_ci.md

@@ -0,0 +1,118 @@
+# 14 Testing, CI and Quality Assurance
+
+This chapter centralizes the project's testing strategy, CI configuration, and quality targets.
+
+## Overview
+
+CalMiner uses a combination of unit, integration, and end-to-end tests to ensure quality.
+
+### Frameworks
+
+- Backend: pytest for unit and integration tests.
+- Frontend: pytest with Playwright for E2E tests.
+- Database: pytest fixtures with psycopg2 for DB tests.
+
+### Test Types
+
+- Unit Tests: Test individual functions/modules.
+- Integration Tests: Test API endpoints and DB interactions.
+- E2E Tests: Playwright for full user flows.
+
+### CI/CD
+
+- Use Gitea Actions for CI/CD; workflows live under `.gitea/workflows/`.
+- `test.yml` runs on every push, provisions a temporary Postgres 16 service, waits for readiness, executes the setup script in dry-run and live modes, then fans out into parallel matrix jobs for unit (`pytest tests/unit`) and end-to-end (`pytest tests/e2e`) suites. Playwright browsers install only for the E2E job.
+- `build-and-push.yml` runs only after the **Run Tests** workflow finishes successfully (triggered via `workflow_run` on `main`). Once tests pass, it builds the Docker image with `docker/build-push-action@v2`, reuses cache-backed layers, and pushes to the Gitea registry.
+- `deploy.yml` runs only after the build workflow reports success on `main`. It connects to the target host (via `appleboy/ssh-action`), pulls the Docker image tagged with the build commit SHA, and restarts the container with that exact image reference.
+- Mandatory secrets: `REGISTRY_USERNAME`, `REGISTRY_PASSWORD`, `REGISTRY_URL`, `SSH_HOST`, `SSH_USERNAME`, `SSH_PRIVATE_KEY`.
+- Run tests on pull requests to shared branches; enforce coverage target ≥80% (pytest-cov).
+
+### Running Tests
+
+- Unit: `pytest tests/unit/`
+- E2E: `pytest tests/e2e/`
+- All: `pytest`
+
+### Test Directory Structure
+
+Organize tests under the `tests/` directory mirroring the application structure:
+
+````text
+tests/
+  unit/
+    test_<module>.py
+  e2e/
+    test_<flow>.py
+  fixtures/
+    conftest.py
+```python
+
+### Fixtures and Test Data
+
+- Define reusable fixtures in `tests/fixtures/conftest.py`.
+- Use temporary in-memory databases or isolated schemas for DB tests.
+- Load sample data via fixtures for consistent test environments.
+- Leverage the `seeded_ui_data` fixture in `tests/unit/conftest.py` to populate scenarios with related cost, maintenance, and simulation records for deterministic UI route checks.
+
+### E2E (Playwright) Tests
+
+The E2E test suite, located in `tests/e2e/`, uses Playwright to simulate user interactions in a live browser environment. These tests are designed to catch issues in the UI, frontend-backend integration, and overall application flow.
+
+#### Fixtures
+
+- `live_server`: A session-scoped fixture that launches the FastAPI application in a separate process, making it accessible to the browser.
+- `playwright_instance`, `browser`, `page`: Standard `pytest-playwright` fixtures for managing the Playwright instance, browser, and individual pages.
+
+#### Smoke Tests
+
+- UI Page Loading: `test_smoke.py` contains a parameterized test that systematically navigates to all UI routes to ensure they load without errors, have the correct title, and display a primary heading.
+- Form Submissions: Each major form in the application has a corresponding test file (e.g., `test_scenarios.py`, `test_costs.py`) that verifies: page loads, create item by filling the form, success message, and UI updates.
+
+### Running E2E Tests
+
+To run the Playwright tests:
+
+```bash
+pytest tests/e2e/
+````
+
+To run headed mode:
+
+```bash
+pytest tests/e2e/ --headed
+```
+
+### Mocking and Dependency Injection
+
+- Use `unittest.mock` to mock external dependencies.
+- Inject dependencies via function parameters or FastAPI's dependency overrides in tests.
+
+### Code Coverage
+
+- Install `pytest-cov` to generate coverage reports.
+- Run with coverage: `pytest --cov --cov-report=term` (use `--cov-report=html` when visualizing hotspots).
+- Target 95%+ overall coverage. Focus on historically low modules: `services/simulation.py`, `services/reporting.py`, `middleware/validation.py`, and `routes/ui.py`.
+- Latest snapshot (2025-10-21): `pytest --cov=. --cov-report=term-missing` returns **91%** overall coverage.
+
+### CI Integration
+
+`test.yml` encapsulates the steps below:
+
+- Check out the repository and set up Python 3.10.
+- Configure the runner's apt proxy (if available), install project dependencies (requirements + test extras), and download Playwright browsers.
+- Run `pytest` (extend with `--cov` flags when enforcing coverage).
+
+> The pip cache step is temporarily disabled in `test.yml` until the self-hosted cache service is exposed (see `docs/ci-cache-troubleshooting.md`).
+
+`build-and-push.yml` adds:
+
+- Registry login using repository secrets.
+- Docker image build/push with GHA cache storage (`cache-from/cache-to` set to `type=gha`).
+
+`deploy.yml` handles:
+
+- SSH into the deployment host.
+- Pull the tagged image from the registry.
+- Stop, remove, and relaunch the `calminer` container exposing port 8000.
+
+When adding new workflows, mirror this structure to ensure secrets, caching, and deployment steps remain aligned with the production environment.