zwitschi
529ff967cc
CI / lint-test-build (push) Failing after 1m23s
- Implemented integration tests for the execution writer to ensure trade orders and PnL are persisted correctly. - Created integration tests for the metrics calculator to summarize execution data accurately. - Added integration tests for the opportunity writer to verify event persistence. - Established PostgreSQL schema validation tests to ensure all expected tables, columns, and constraints exist. - Removed outdated unit tests that relied on DuckDB and replaced them with tests using PgStore.
89 lines
5.7 KiB
Markdown
89 lines
5.7 KiB
Markdown
# Database Layer: Schema & Repositories
|
|
|
|
> **Database engine**: PostgreSQL 15+ on `192.168.88.35`
|
|
> **Driver**: `asyncpg` (async connection pool)
|
|
> **Store class**: `PgStore` in `src/arbitrade/storage/pg_store.py`
|
|
|
|
## Connection Lifecycle
|
|
|
|
```txt
|
|
FastAPI lifespan (create_app)
|
|
└─ PgStore.start() # creates asyncpg connection pool
|
|
└─ PgStore.migrate() # reads schema_pg.sql, creates tables
|
|
└─ ... application runs ...
|
|
└─ PgStore.stop() # closes the pool
|
|
```
|
|
|
|
All repository classes accept a `PgStore` instance and acquire connections
|
|
via `async with self._store.pool.acquire() as conn:`.
|
|
|
|
## Schema
|
|
|
|
Defined in `src/arbitrade/storage/schema_pg.sql`. 15 tables:
|
|
|
|
| Table | Purpose | PK | Notes |
|
|
| ----------------------------- | -------------------------- | --------------- | ---------------------------------------- |
|
|
| `schema_migrations` | Version tracking | `version` | Single-row per version |
|
|
| `config_sections` | Config section metadata | `id` (SERIAL) | `name` UNIQUE |
|
|
| `config_settings` | Key-value config store | `key` (VARCHAR) | JSON-serialized values |
|
|
| `config_pairings` | Currency pairs to monitor | `id` (SERIAL) | `(base_asset, quote_asset)` UNIQUE |
|
|
| `config_backtesting_defaults` | Default backtest params | `id` (SERIAL) | Singleton via `ORDER BY id DESC LIMIT 1` |
|
|
| `opportunities` | Detected arb opportunities | `id` (UUID) | |
|
|
| `trades` | Executed trades | `id` (UUID) | |
|
|
| `orders` | Individual leg orders | `id` (UUID) | |
|
|
| `pnl_events` | P&L event stream | `id` (UUID) | |
|
|
| `portfolio_snapshots` | Balance snapshots | — | Append-only |
|
|
| `market_snapshots` | Raw order-book snapshots | — | Append-only |
|
|
| `audit_events` | Audit trail | `id` (UUID) | |
|
|
| `runtime_state_snapshots` | Runtime state history | — | Append-only |
|
|
| `kraken_account_snapshots` | Fee tier + account data | — | Append-only |
|
|
| `backtest_jobs` | Backtest job records | `id` (UUID) | |
|
|
|
|
JSON columns use `JSONB` for indexability. UUID primary keys use
|
|
`gen_random_uuid()` (requires `pgcrypto` extension).
|
|
|
|
## Repository Classes
|
|
|
|
All in `src/arbitrade/storage/repositories.py`. Every method is `async def`.
|
|
|
|
| Class | Key Methods | Used By |
|
|
| ------------------------------------- | ---------------------------------------------------------- | --------------------------- |
|
|
| `MarketSnapshotRepository` | `insert()` | `AsyncMarketSnapshotWriter` |
|
|
| `OpportunityRepository` | `insert()` | `AsyncOpportunityWriter` |
|
|
| `TradeRepository` | `insert()` | `AsyncExecutionWriter` |
|
|
| `OrderRepository` | `insert()` | `AsyncExecutionWriter` |
|
|
| `PnLRepository` | `insert()` | `AsyncExecutionWriter` |
|
|
| `AuditRepository` | `insert()`, `list_recent()` | API routes, lifecycle |
|
|
| `RuntimeStateRepository` | `insert()`, `latest()` | Lifecycle, API |
|
|
| `ConfigSectionRepository` | `create_section()`, `get_section()`, `list_sections()` | Config service |
|
|
| `ConfigSettingRepository` | Full CRUD + `get_latest_updated_at()` | Config service |
|
|
| `ConfigPairingRepository` | Full CRUD + `upsert_pairing()`, `list_pairings()` | Feeds, pairing sync |
|
|
| `ConfigBacktestingDefaultsRepository` | `create_defaults()`, `get_defaults()`, `update_defaults()` | Config service |
|
|
| `KrakenAccountSnapshotRepository` | `insert_snapshot()`, `latest_snapshot()` | Fee sync loop |
|
|
| `BacktestJobRepository` | Full CRUD | Backtesting UI + worker |
|
|
|
|
## Async Writers
|
|
|
|
Three background writer tasks buffer high-frequency writes:
|
|
|
|
- **`AsyncExecutionWriter`** — trades/orders/P&L queue
|
|
- **`AsyncMarketSnapshotWriter`** — order-book snapshot queue
|
|
- **`AsyncOpportunityWriter`** — opportunity event queue
|
|
|
|
Each uses an `asyncio.Queue` and drains it in a background task with
|
|
`await repo.insert(...)`.
|
|
|
|
## Integration Tests
|
|
|
|
`tests/integration/test_postgresql_schema.py` verifies:
|
|
|
|
- Connection to PostgreSQL server
|
|
- `pgcrypto` extension availability
|
|
- All 15 tables exist after migration
|
|
- Migration is idempotent
|
|
- Correct columns per table
|
|
- Primary keys and unique constraints
|
|
- Tables start empty
|
|
- Simple INSERT/SELECT round-trip
|
|
- `ON CONFLICT ... DO UPDATE` on config_pairings
|