Add integration tests for execution persistence, metrics, and opportunity writing

- Implemented integration tests for the execution writer to ensure trade orders and PnL are persisted correctly. - Created integration tests for the metrics calculator to summarize execution data accurately. - Added integration tests for the opportunity writer to verify event persistence. - Established PostgreSQL schema validation tests to ensure all expected tables, columns, and constraints exist. - Removed outdated unit tests that relied on DuckDB and replaced them with tests using PgStore.
2026-06-07 14:37:53 +02:00
parent 54feb2ecd4
commit 529ff967cc
44 changed files with 1955 additions and 1386 deletions
@@ -7,16 +7,16 @@ This guide provides two supported deployment paths for Arbitrade on Coolify:

 Reference docs:

- Coolify Applications: https://coolify.io/docs/applications
- Coolify Build Packs: https://coolify.io/docs/applications/build-packs
- Coolify Dockerfile Build Pack: https://coolify.io/docs/applications/build-packs/dockerfile
- Coolify Nixpacks Build Pack: https://coolify.io/docs/applications/build-packs/nixpacks
- Coolify CI/CD (Git providers): https://coolify.io/docs/applications/ci-cd
- Coolify Gitea integration: https://coolify.io/docs/applications/ci-cd/gitea/integration
- Coolify environment variables: https://coolify.io/docs/knowledge-base/environment-variables
- Coolify persistent storage: https://coolify.io/docs/knowledge-base/persistent-storage
- Coolify health checks: https://coolify.io/docs/knowledge-base/health-checks
- Coolify Docker registry credentials: https://coolify.io/docs/knowledge-base/docker/registry
+- [Coolify Applications](https://coolify.io/docs/applications)
+- [Coolify Build Packs](https://coolify.io/docs/applications/build-packs)
+- [Coolify Dockerfile Build Pack](https://coolify.io/docs/applications/build-packs/dockerfile)
+- [Coolify Nixpacks Build Pack](https://coolify.io/docs/applications/build-packs/nixpacks)
+- [Coolify CI/CD (Git providers)](https://coolify.io/docs/applications/ci-cd)
+- [Coolify Gitea integration](https://coolify.io/docs/applications/ci-cd/gitea/integration)
+- [Coolify environment variables](https://coolify.io/docs/knowledge-base/environment-variables)
+- [Coolify persistent storage](https://coolify.io/docs/knowledge-base/persistent-storage)
+- [Coolify health checks](https://coolify.io/docs/knowledge-base/health-checks)
+- [Coolify Docker registry credentials](https://coolify.io/docs/knowledge-base/docker/registry)

 ## Common Runtime Configuration

@@ -32,14 +32,14 @@ Use these values in both deployment modes.

 - Add a persistent volume
 - Mount path: `/app/data`
- Set DB path to: `DUCKDB_PATH=/app/data/arbitrade.duckdb`
+- Set PG connection: `PG_HOST=postgres`, `PG_PORT=5432`, `PG_DATABASE=arbitrade`, `PG_USER=arbitrade`, `PG_PASSWORD=arbitrade`

 ### Required environment variables

 - `APP_ENV=prod`
 - `APP_HOST=0.0.0.0`
 - `APP_PORT=9090`
- `DUCKDB_PATH=/app/data/arbitrade.duckdb`
+- `PG_DATABASE=arbitrade`
 - `LOG_LEVEL=INFO`
 - `LOG_JSON=true`
 - `KRAKEN_API_KEY=<set-in-coolify-secret>`
@@ -135,7 +135,7 @@ Update flow for new releases:
 - Ensure the deployed image/wheel includes package templates under `arbitrade/web/templates/*`.
 - If you build from source, do not remove packaged template files under `src/arbitrade/web/templates`.
 - DB resets after deploy:
- Confirm persistent mount exists at `/app/data` and `DUCKDB_PATH` points there.
+- Confirm PostgreSQL is reachable at `PG_HOST`.
 - Registry pull fails:
 - Re-check Docker registry credentials in Coolify.
 - App starts but unavailable externally:
@@ -8,7 +8,7 @@ Primary goals:

 - Detect and execute triangular opportunities on Kraken with fee/slippage-aware math.
 - Keep hot-path latency low with incremental order-book updates and event-driven scoring.
- Persist operational data in DuckDB for dev, test, and prod.
+- Persist operational data in PostgreSQL for all environments.
 - Provide operator controls, audit trail, and alerting through a server-rendered dashboard.
 - Support backtesting, parameter sweeps, and deferred experimental strategy work behind feature flags.

@@ -17,7 +17,7 @@ Primary goals:
 - Python 3.12+ runtime.
 - Native Kraken WebSocket on the hot path.
 - HTMX + Jinja2 UI, no SPA build step.
- DuckDB everywhere.
+- PostgreSQL everywhere.
 - Self-hosted Gitea Actions CI and Gitea registry.
 - Windows development support.
 - Secrets must stay out of the repository.
@@ -32,7 +32,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades

 - Kraken REST + WebSocket provide market data and execution.
 - FastAPI serves HTML fragments, JSON endpoints, and SSE streams.
- DuckDB stores trades, opportunities, snapshots, audit events, and runtime state.
+- PostgreSQL stores trades, opportunities, snapshots, audit events, and runtime state.
 - Coolify can deploy the published image using environment variables and persistent storage.

 ## 4. Solution Strategy
@@ -55,7 +55,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
 - `execution/` - multi-leg trade sequencing.
 - `backtesting/` - replay engine, parameter sweep, experiment scaffolds.
 - `strategy/` - experimental strategy modules such as stat-arb.
- `storage/` - DuckDB schema and repositories.
+- `storage/` - PostgreSQL schema and repositories.
 - `alerting/` - multi-channel notifications.
 - `runtime/` - startup recovery and graceful shutdown.

@@ -64,7 +64,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
 - `fastapi`, `uvicorn`, `jinja2`, `htmx`-driven templates.
 - `orjson` for low-alloc parsing.
 - `sortedcontainers` for book state.
- `duckdb` for persistence and analytics.
+- `asyncpg` for PostgreSQL persistence.
 - `pydantic` / `pydantic-settings` for typed configuration.
 - `cryptography` / keyring for secret handling.

@@ -77,7 +77,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
 3. Incremental detector scores impacted cycles.
 4. Risk manager validates the opportunity.
 5. Execution sequencer places legs if approved.
-6. Trades and snapshots persist to DuckDB.
+7. Trades and snapshots persist to PostgreSQL.
 7. Dashboard and alerts reflect state changes.

 ### 6.2 Dashboard Control Flow
@@ -112,7 +112,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades

 - Deploy from the published image.
 - Configure runtime via environment variables.
- Mount persistent storage at `/app/data` for DuckDB.
+- Connect to PostgreSQL at configured `PG_HOST`.

 ## 8. Cross-Cutting Concepts

@@ -126,7 +126,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
 ## 9. Architecture Decisions

 - Native Kraken WS instead of a generic exchange abstraction on the hot path.
- DuckDB as the single database engine.
+- PostgreSQL as the single database engine.
 - HTMX + Jinja2 instead of SPA frontend.
 - Backtesting reuses production detector/risk/execution logic.
 - Experimental stat-arb stays behind a feature flag.
@@ -152,5 +152,5 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
 - WS: WebSocket.
 - HTMX: HTML-over-the-wire UI library.
 - SSE: Server-Sent Events.
- DUCKDB: Embedded analytical database used for all environments.
+- PGSQL: PostgreSQL database used for all environments.
 - Stat arb: Statistical arbitrage, currently experimental and feature-flagged.
@@ -0,0 +1,88 @@
+# Database Layer: Schema & Repositories
+
+> **Database engine**: PostgreSQL 15+ on `192.168.88.35`
+> **Driver**: `asyncpg` (async connection pool)
+> **Store class**: `PgStore` in `src/arbitrade/storage/pg_store.py`
+
+## Connection Lifecycle
+
+```txt
+FastAPI lifespan (create_app)
+  └─ PgStore.start()         # creates asyncpg connection pool
+      └─ PgStore.migrate()   # reads schema_pg.sql, creates tables
+          └─ ... application runs ...
+  └─ PgStore.stop()          # closes the pool
+```
+
+All repository classes accept a `PgStore` instance and acquire connections
+via `async with self._store.pool.acquire() as conn:`.
+
+## Schema
+
+Defined in `src/arbitrade/storage/schema_pg.sql`. 15 tables:
+
+| Table                         | Purpose                    | PK              | Notes                                    |
+| ----------------------------- | -------------------------- | --------------- | ---------------------------------------- |
+| `schema_migrations`           | Version tracking           | `version`       | Single-row per version                   |
+| `config_sections`             | Config section metadata    | `id` (SERIAL)   | `name` UNIQUE                            |
+| `config_settings`             | Key-value config store     | `key` (VARCHAR) | JSON-serialized values                   |
+| `config_pairings`             | Currency pairs to monitor  | `id` (SERIAL)   | `(base_asset, quote_asset)` UNIQUE       |
+| `config_backtesting_defaults` | Default backtest params    | `id` (SERIAL)   | Singleton via `ORDER BY id DESC LIMIT 1` |
+| `opportunities`               | Detected arb opportunities | `id` (UUID)     |                                          |
+| `trades`                      | Executed trades            | `id` (UUID)     |                                          |
+| `orders`                      | Individual leg orders      | `id` (UUID)     |                                          |
+| `pnl_events`                  | P&L event stream           | `id` (UUID)     |                                          |
+| `portfolio_snapshots`         | Balance snapshots          | —               | Append-only                              |
+| `market_snapshots`            | Raw order-book snapshots   | —               | Append-only                              |
+| `audit_events`                | Audit trail                | `id` (UUID)     |                                          |
+| `runtime_state_snapshots`     | Runtime state history      | —               | Append-only                              |
+| `kraken_account_snapshots`    | Fee tier + account data    | —               | Append-only                              |
+| `backtest_jobs`               | Backtest job records       | `id` (UUID)     |                                          |
+
+JSON columns use `JSONB` for indexability. UUID primary keys use
+`gen_random_uuid()` (requires `pgcrypto` extension).
+
+## Repository Classes
+
+All in `src/arbitrade/storage/repositories.py`. Every method is `async def`.
+
+| Class                                 | Key Methods                                                | Used By                     |
+| ------------------------------------- | ---------------------------------------------------------- | --------------------------- |
+| `MarketSnapshotRepository`            | `insert()`                                                 | `AsyncMarketSnapshotWriter` |
+| `OpportunityRepository`               | `insert()`                                                 | `AsyncOpportunityWriter`    |
+| `TradeRepository`                     | `insert()`                                                 | `AsyncExecutionWriter`      |
+| `OrderRepository`                     | `insert()`                                                 | `AsyncExecutionWriter`      |
+| `PnLRepository`                       | `insert()`                                                 | `AsyncExecutionWriter`      |
+| `AuditRepository`                     | `insert()`, `list_recent()`                                | API routes, lifecycle       |
+| `RuntimeStateRepository`              | `insert()`, `latest()`                                     | Lifecycle, API              |
+| `ConfigSectionRepository`             | `create_section()`, `get_section()`, `list_sections()`     | Config service              |
+| `ConfigSettingRepository`             | Full CRUD + `get_latest_updated_at()`                      | Config service              |
+| `ConfigPairingRepository`             | Full CRUD + `upsert_pairing()`, `list_pairings()`          | Feeds, pairing sync         |
+| `ConfigBacktestingDefaultsRepository` | `create_defaults()`, `get_defaults()`, `update_defaults()` | Config service              |
+| `KrakenAccountSnapshotRepository`     | `insert_snapshot()`, `latest_snapshot()`                   | Fee sync loop               |
+| `BacktestJobRepository`               | Full CRUD                                                  | Backtesting UI + worker     |
+
+## Async Writers
+
+Three background writer tasks buffer high-frequency writes:
+
+- **`AsyncExecutionWriter`** — trades/orders/P&L queue
+- **`AsyncMarketSnapshotWriter`** — order-book snapshot queue
+- **`AsyncOpportunityWriter`** — opportunity event queue
+
+Each uses an `asyncio.Queue` and drains it in a background task with
+`await repo.insert(...)`.
+
+## Integration Tests
+
+`tests/integration/test_postgresql_schema.py` verifies:
+
+- Connection to PostgreSQL server
+- `pgcrypto` extension availability
+- All 15 tables exist after migration
+- Migration is idempotent
+- Correct columns per table
+- Primary keys and unique constraints
+- Tables start empty
+- Simple INSERT/SELECT round-trip
+- `ON CONFLICT ... DO UPDATE` on config_pairings