Add integration tests for execution persistence, metrics, and opportunity writing
CI / lint-test-build (push) Failing after 1m23s
CI / lint-test-build (push) Failing after 1m23s
- Implemented integration tests for the execution writer to ensure trade orders and PnL are persisted correctly. - Created integration tests for the metrics calculator to summarize execution data accurately. - Added integration tests for the opportunity writer to verify event persistence. - Established PostgreSQL schema validation tests to ensure all expected tables, columns, and constraints exist. - Removed outdated unit tests that relied on DuckDB and replaced them with tests using PgStore.
This commit is contained in:
@@ -38,6 +38,7 @@ dist/
|
|||||||
# Local database / runtime data
|
# Local database / runtime data
|
||||||
data/*.duckdb.wal
|
data/*.duckdb.wal
|
||||||
data/*.duckdb.tmp
|
data/*.duckdb.tmp
|
||||||
|
data/arbitrade.duckdb
|
||||||
logs/
|
logs/
|
||||||
ops/performance/latest_profile.json
|
ops/performance/latest_profile.json
|
||||||
|
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ Current stack:
|
|||||||
|
|
||||||
- Python 3.12+
|
- Python 3.12+
|
||||||
- FastAPI + HTMX/Jinja2
|
- FastAPI + HTMX/Jinja2
|
||||||
- DuckDB for dev/test/prod
|
- PostgreSQL for all environments (via asyncpg)
|
||||||
- Native Kraken WebSocket planned for market-data hot path
|
- Native Kraken WebSocket planned for market-data hot path
|
||||||
- Gitea Actions + Gitea container registry
|
- Gitea Actions + Gitea container registry
|
||||||
|
|
||||||
@@ -22,7 +22,7 @@ Bootstrap complete for foundation layer:
|
|||||||
- typed settings and env loading
|
- typed settings and env loading
|
||||||
- structured logging
|
- structured logging
|
||||||
- encrypted secret helpers
|
- encrypted secret helpers
|
||||||
- DuckDB connection + base schema
|
- PostgreSQL connection + full schema migration
|
||||||
- FastAPI app with health endpoint
|
- FastAPI app with health endpoint
|
||||||
- Gitea Actions CI scaffold
|
- Gitea Actions CI scaffold
|
||||||
- Docker / docker-compose scaffold
|
- Docker / docker-compose scaffold
|
||||||
@@ -152,7 +152,11 @@ APP_HOST=0.0.0.0
|
|||||||
APP_PORT=9090
|
APP_PORT=9090
|
||||||
LOG_LEVEL=INFO
|
LOG_LEVEL=INFO
|
||||||
LOG_JSON=true
|
LOG_JSON=true
|
||||||
DUCKDB_PATH=./data/arbitrade.duckdb
|
PG_HOST=192.168.88.35
|
||||||
|
PG_PORT=5432
|
||||||
|
PG_DATABASE=arbitrade
|
||||||
|
PG_USER=arbitrade
|
||||||
|
PG_PASSWORD=arbitrade
|
||||||
FERNET_KEY=
|
FERNET_KEY=
|
||||||
KRAKEN_API_KEY=
|
KRAKEN_API_KEY=
|
||||||
KRAKEN_API_SECRET=
|
KRAKEN_API_SECRET=
|
||||||
@@ -182,15 +186,19 @@ Health endpoints:
|
|||||||
|
|
||||||
## Database
|
## Database
|
||||||
|
|
||||||
DuckDB used everywhere: local dev, tests, production.
|
PostgreSQL used everywhere: local dev, tests, production.
|
||||||
|
|
||||||
Default database file:
|
Default connection:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
./data/arbitrade.duckdb
|
PG_HOST=192.168.88.35
|
||||||
|
PG_PORT=5432
|
||||||
|
PG_DATABASE=arbitrade
|
||||||
|
PG_USER=arbitrade
|
||||||
|
PG_PASSWORD=arbitrade
|
||||||
```
|
```
|
||||||
|
|
||||||
Schema bootstrap runs automatically on app startup.
|
Schema bootstrap runs automatically on app startup via `PgStore.migrate()`.
|
||||||
|
|
||||||
Current tables:
|
Current tables:
|
||||||
|
|
||||||
@@ -220,7 +228,7 @@ DELETE FROM audit_events
|
|||||||
WHERE occurred_at < NOW() - INTERVAL 30 DAY;
|
WHERE occurred_at < NOW() - INTERVAL 30 DAY;
|
||||||
```
|
```
|
||||||
|
|
||||||
- Back up archive files and the main DuckDB file together.
|
- Back up archive files and the PostgreSQL database together.
|
||||||
- For production, run archive + backup as scheduled maintenance (cron/task scheduler).
|
- For production, run archive + backup as scheduled maintenance (cron/task scheduler).
|
||||||
|
|
||||||
## Quality Checks
|
## Quality Checks
|
||||||
@@ -342,7 +350,7 @@ Add a persistent volume in Coolify:
|
|||||||
|
|
||||||
- Mount Path: `/app/data`
|
- Mount Path: `/app/data`
|
||||||
|
|
||||||
This preserves DuckDB and other runtime artifacts across restarts/redeploys.
|
This preserves PostgreSQL data and other runtime artifacts across restarts/redeploys.
|
||||||
|
|
||||||
### 5) Configure environment variables
|
### 5) Configure environment variables
|
||||||
|
|
||||||
@@ -351,7 +359,11 @@ Add runtime environment variables in Coolify (UI: Environment Variables):
|
|||||||
- `APP_ENV=prod`
|
- `APP_ENV=prod`
|
||||||
- `APP_HOST=0.0.0.0`
|
- `APP_HOST=0.0.0.0`
|
||||||
- `APP_PORT=9090`
|
- `APP_PORT=9090`
|
||||||
- `DUCKDB_PATH=/app/data/arbitrade.duckdb`
|
- `PG_HOST=postgres`
|
||||||
|
`PG_PORT=5432`
|
||||||
|
`PG_DATABASE=arbitrade`
|
||||||
|
`PG_USER=arbitrade`
|
||||||
|
`PG_PASSWORD=arbitrade`
|
||||||
- `LOG_LEVEL=INFO`
|
- `LOG_LEVEL=INFO`
|
||||||
- `LOG_JSON=true`
|
- `LOG_JSON=true`
|
||||||
- `KRAKEN_API_KEY=...`
|
- `KRAKEN_API_KEY=...`
|
||||||
|
|||||||
Binary file not shown.
+13
-13
@@ -7,16 +7,16 @@ This guide provides two supported deployment paths for Arbitrade on Coolify:
|
|||||||
|
|
||||||
Reference docs:
|
Reference docs:
|
||||||
|
|
||||||
- Coolify Applications: https://coolify.io/docs/applications
|
- [Coolify Applications](https://coolify.io/docs/applications)
|
||||||
- Coolify Build Packs: https://coolify.io/docs/applications/build-packs
|
- [Coolify Build Packs](https://coolify.io/docs/applications/build-packs)
|
||||||
- Coolify Dockerfile Build Pack: https://coolify.io/docs/applications/build-packs/dockerfile
|
- [Coolify Dockerfile Build Pack](https://coolify.io/docs/applications/build-packs/dockerfile)
|
||||||
- Coolify Nixpacks Build Pack: https://coolify.io/docs/applications/build-packs/nixpacks
|
- [Coolify Nixpacks Build Pack](https://coolify.io/docs/applications/build-packs/nixpacks)
|
||||||
- Coolify CI/CD (Git providers): https://coolify.io/docs/applications/ci-cd
|
- [Coolify CI/CD (Git providers)](https://coolify.io/docs/applications/ci-cd)
|
||||||
- Coolify Gitea integration: https://coolify.io/docs/applications/ci-cd/gitea/integration
|
- [Coolify Gitea integration](https://coolify.io/docs/applications/ci-cd/gitea/integration)
|
||||||
- Coolify environment variables: https://coolify.io/docs/knowledge-base/environment-variables
|
- [Coolify environment variables](https://coolify.io/docs/knowledge-base/environment-variables)
|
||||||
- Coolify persistent storage: https://coolify.io/docs/knowledge-base/persistent-storage
|
- [Coolify persistent storage](https://coolify.io/docs/knowledge-base/persistent-storage)
|
||||||
- Coolify health checks: https://coolify.io/docs/knowledge-base/health-checks
|
- [Coolify health checks](https://coolify.io/docs/knowledge-base/health-checks)
|
||||||
- Coolify Docker registry credentials: https://coolify.io/docs/knowledge-base/docker/registry
|
- [Coolify Docker registry credentials](https://coolify.io/docs/knowledge-base/docker/registry)
|
||||||
|
|
||||||
## Common Runtime Configuration
|
## Common Runtime Configuration
|
||||||
|
|
||||||
@@ -32,14 +32,14 @@ Use these values in both deployment modes.
|
|||||||
|
|
||||||
- Add a persistent volume
|
- Add a persistent volume
|
||||||
- Mount path: `/app/data`
|
- Mount path: `/app/data`
|
||||||
- Set DB path to: `DUCKDB_PATH=/app/data/arbitrade.duckdb`
|
- Set PG connection: `PG_HOST=postgres`, `PG_PORT=5432`, `PG_DATABASE=arbitrade`, `PG_USER=arbitrade`, `PG_PASSWORD=arbitrade`
|
||||||
|
|
||||||
### Required environment variables
|
### Required environment variables
|
||||||
|
|
||||||
- `APP_ENV=prod`
|
- `APP_ENV=prod`
|
||||||
- `APP_HOST=0.0.0.0`
|
- `APP_HOST=0.0.0.0`
|
||||||
- `APP_PORT=9090`
|
- `APP_PORT=9090`
|
||||||
- `DUCKDB_PATH=/app/data/arbitrade.duckdb`
|
- `PG_DATABASE=arbitrade`
|
||||||
- `LOG_LEVEL=INFO`
|
- `LOG_LEVEL=INFO`
|
||||||
- `LOG_JSON=true`
|
- `LOG_JSON=true`
|
||||||
- `KRAKEN_API_KEY=<set-in-coolify-secret>`
|
- `KRAKEN_API_KEY=<set-in-coolify-secret>`
|
||||||
@@ -135,7 +135,7 @@ Update flow for new releases:
|
|||||||
- Ensure the deployed image/wheel includes package templates under `arbitrade/web/templates/*`.
|
- Ensure the deployed image/wheel includes package templates under `arbitrade/web/templates/*`.
|
||||||
- If you build from source, do not remove packaged template files under `src/arbitrade/web/templates`.
|
- If you build from source, do not remove packaged template files under `src/arbitrade/web/templates`.
|
||||||
- DB resets after deploy:
|
- DB resets after deploy:
|
||||||
- Confirm persistent mount exists at `/app/data` and `DUCKDB_PATH` points there.
|
- Confirm PostgreSQL is reachable at `PG_HOST`.
|
||||||
- Registry pull fails:
|
- Registry pull fails:
|
||||||
- Re-check Docker registry credentials in Coolify.
|
- Re-check Docker registry credentials in Coolify.
|
||||||
- App starts but unavailable externally:
|
- App starts but unavailable externally:
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ Primary goals:
|
|||||||
|
|
||||||
- Detect and execute triangular opportunities on Kraken with fee/slippage-aware math.
|
- Detect and execute triangular opportunities on Kraken with fee/slippage-aware math.
|
||||||
- Keep hot-path latency low with incremental order-book updates and event-driven scoring.
|
- Keep hot-path latency low with incremental order-book updates and event-driven scoring.
|
||||||
- Persist operational data in DuckDB for dev, test, and prod.
|
- Persist operational data in PostgreSQL for all environments.
|
||||||
- Provide operator controls, audit trail, and alerting through a server-rendered dashboard.
|
- Provide operator controls, audit trail, and alerting through a server-rendered dashboard.
|
||||||
- Support backtesting, parameter sweeps, and deferred experimental strategy work behind feature flags.
|
- Support backtesting, parameter sweeps, and deferred experimental strategy work behind feature flags.
|
||||||
|
|
||||||
@@ -17,7 +17,7 @@ Primary goals:
|
|||||||
- Python 3.12+ runtime.
|
- Python 3.12+ runtime.
|
||||||
- Native Kraken WebSocket on the hot path.
|
- Native Kraken WebSocket on the hot path.
|
||||||
- HTMX + Jinja2 UI, no SPA build step.
|
- HTMX + Jinja2 UI, no SPA build step.
|
||||||
- DuckDB everywhere.
|
- PostgreSQL everywhere.
|
||||||
- Self-hosted Gitea Actions CI and Gitea registry.
|
- Self-hosted Gitea Actions CI and Gitea registry.
|
||||||
- Windows development support.
|
- Windows development support.
|
||||||
- Secrets must stay out of the repository.
|
- Secrets must stay out of the repository.
|
||||||
@@ -32,7 +32,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
|
|
||||||
- Kraken REST + WebSocket provide market data and execution.
|
- Kraken REST + WebSocket provide market data and execution.
|
||||||
- FastAPI serves HTML fragments, JSON endpoints, and SSE streams.
|
- FastAPI serves HTML fragments, JSON endpoints, and SSE streams.
|
||||||
- DuckDB stores trades, opportunities, snapshots, audit events, and runtime state.
|
- PostgreSQL stores trades, opportunities, snapshots, audit events, and runtime state.
|
||||||
- Coolify can deploy the published image using environment variables and persistent storage.
|
- Coolify can deploy the published image using environment variables and persistent storage.
|
||||||
|
|
||||||
## 4. Solution Strategy
|
## 4. Solution Strategy
|
||||||
@@ -55,7 +55,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
- `execution/` - multi-leg trade sequencing.
|
- `execution/` - multi-leg trade sequencing.
|
||||||
- `backtesting/` - replay engine, parameter sweep, experiment scaffolds.
|
- `backtesting/` - replay engine, parameter sweep, experiment scaffolds.
|
||||||
- `strategy/` - experimental strategy modules such as stat-arb.
|
- `strategy/` - experimental strategy modules such as stat-arb.
|
||||||
- `storage/` - DuckDB schema and repositories.
|
- `storage/` - PostgreSQL schema and repositories.
|
||||||
- `alerting/` - multi-channel notifications.
|
- `alerting/` - multi-channel notifications.
|
||||||
- `runtime/` - startup recovery and graceful shutdown.
|
- `runtime/` - startup recovery and graceful shutdown.
|
||||||
|
|
||||||
@@ -64,7 +64,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
- `fastapi`, `uvicorn`, `jinja2`, `htmx`-driven templates.
|
- `fastapi`, `uvicorn`, `jinja2`, `htmx`-driven templates.
|
||||||
- `orjson` for low-alloc parsing.
|
- `orjson` for low-alloc parsing.
|
||||||
- `sortedcontainers` for book state.
|
- `sortedcontainers` for book state.
|
||||||
- `duckdb` for persistence and analytics.
|
- `asyncpg` for PostgreSQL persistence.
|
||||||
- `pydantic` / `pydantic-settings` for typed configuration.
|
- `pydantic` / `pydantic-settings` for typed configuration.
|
||||||
- `cryptography` / keyring for secret handling.
|
- `cryptography` / keyring for secret handling.
|
||||||
|
|
||||||
@@ -77,7 +77,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
3. Incremental detector scores impacted cycles.
|
3. Incremental detector scores impacted cycles.
|
||||||
4. Risk manager validates the opportunity.
|
4. Risk manager validates the opportunity.
|
||||||
5. Execution sequencer places legs if approved.
|
5. Execution sequencer places legs if approved.
|
||||||
6. Trades and snapshots persist to DuckDB.
|
7. Trades and snapshots persist to PostgreSQL.
|
||||||
7. Dashboard and alerts reflect state changes.
|
7. Dashboard and alerts reflect state changes.
|
||||||
|
|
||||||
### 6.2 Dashboard Control Flow
|
### 6.2 Dashboard Control Flow
|
||||||
@@ -112,7 +112,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
|
|
||||||
- Deploy from the published image.
|
- Deploy from the published image.
|
||||||
- Configure runtime via environment variables.
|
- Configure runtime via environment variables.
|
||||||
- Mount persistent storage at `/app/data` for DuckDB.
|
- Connect to PostgreSQL at configured `PG_HOST`.
|
||||||
|
|
||||||
## 8. Cross-Cutting Concepts
|
## 8. Cross-Cutting Concepts
|
||||||
|
|
||||||
@@ -126,7 +126,7 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
## 9. Architecture Decisions
|
## 9. Architecture Decisions
|
||||||
|
|
||||||
- Native Kraken WS instead of a generic exchange abstraction on the hot path.
|
- Native Kraken WS instead of a generic exchange abstraction on the hot path.
|
||||||
- DuckDB as the single database engine.
|
- PostgreSQL as the single database engine.
|
||||||
- HTMX + Jinja2 instead of SPA frontend.
|
- HTMX + Jinja2 instead of SPA frontend.
|
||||||
- Backtesting reuses production detector/risk/execution logic.
|
- Backtesting reuses production detector/risk/execution logic.
|
||||||
- Experimental stat-arb stays behind a feature flag.
|
- Experimental stat-arb stays behind a feature flag.
|
||||||
@@ -152,5 +152,5 @@ The bot consumes Kraken market data, detects opportunities, and executes trades
|
|||||||
- WS: WebSocket.
|
- WS: WebSocket.
|
||||||
- HTMX: HTML-over-the-wire UI library.
|
- HTMX: HTML-over-the-wire UI library.
|
||||||
- SSE: Server-Sent Events.
|
- SSE: Server-Sent Events.
|
||||||
- DUCKDB: Embedded analytical database used for all environments.
|
- PGSQL: PostgreSQL database used for all environments.
|
||||||
- Stat arb: Statistical arbitrage, currently experimental and feature-flagged.
|
- Stat arb: Statistical arbitrage, currently experimental and feature-flagged.
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Database Layer: Schema & Repositories
|
||||||
|
|
||||||
|
> **Database engine**: PostgreSQL 15+ on `192.168.88.35`
|
||||||
|
> **Driver**: `asyncpg` (async connection pool)
|
||||||
|
> **Store class**: `PgStore` in `src/arbitrade/storage/pg_store.py`
|
||||||
|
|
||||||
|
## Connection Lifecycle
|
||||||
|
|
||||||
|
```txt
|
||||||
|
FastAPI lifespan (create_app)
|
||||||
|
└─ PgStore.start() # creates asyncpg connection pool
|
||||||
|
└─ PgStore.migrate() # reads schema_pg.sql, creates tables
|
||||||
|
└─ ... application runs ...
|
||||||
|
└─ PgStore.stop() # closes the pool
|
||||||
|
```
|
||||||
|
|
||||||
|
All repository classes accept a `PgStore` instance and acquire connections
|
||||||
|
via `async with self._store.pool.acquire() as conn:`.
|
||||||
|
|
||||||
|
## Schema
|
||||||
|
|
||||||
|
Defined in `src/arbitrade/storage/schema_pg.sql`. 15 tables:
|
||||||
|
|
||||||
|
| Table | Purpose | PK | Notes |
|
||||||
|
| ----------------------------- | -------------------------- | --------------- | ---------------------------------------- |
|
||||||
|
| `schema_migrations` | Version tracking | `version` | Single-row per version |
|
||||||
|
| `config_sections` | Config section metadata | `id` (SERIAL) | `name` UNIQUE |
|
||||||
|
| `config_settings` | Key-value config store | `key` (VARCHAR) | JSON-serialized values |
|
||||||
|
| `config_pairings` | Currency pairs to monitor | `id` (SERIAL) | `(base_asset, quote_asset)` UNIQUE |
|
||||||
|
| `config_backtesting_defaults` | Default backtest params | `id` (SERIAL) | Singleton via `ORDER BY id DESC LIMIT 1` |
|
||||||
|
| `opportunities` | Detected arb opportunities | `id` (UUID) | |
|
||||||
|
| `trades` | Executed trades | `id` (UUID) | |
|
||||||
|
| `orders` | Individual leg orders | `id` (UUID) | |
|
||||||
|
| `pnl_events` | P&L event stream | `id` (UUID) | |
|
||||||
|
| `portfolio_snapshots` | Balance snapshots | — | Append-only |
|
||||||
|
| `market_snapshots` | Raw order-book snapshots | — | Append-only |
|
||||||
|
| `audit_events` | Audit trail | `id` (UUID) | |
|
||||||
|
| `runtime_state_snapshots` | Runtime state history | — | Append-only |
|
||||||
|
| `kraken_account_snapshots` | Fee tier + account data | — | Append-only |
|
||||||
|
| `backtest_jobs` | Backtest job records | `id` (UUID) | |
|
||||||
|
|
||||||
|
JSON columns use `JSONB` for indexability. UUID primary keys use
|
||||||
|
`gen_random_uuid()` (requires `pgcrypto` extension).
|
||||||
|
|
||||||
|
## Repository Classes
|
||||||
|
|
||||||
|
All in `src/arbitrade/storage/repositories.py`. Every method is `async def`.
|
||||||
|
|
||||||
|
| Class | Key Methods | Used By |
|
||||||
|
| ------------------------------------- | ---------------------------------------------------------- | --------------------------- |
|
||||||
|
| `MarketSnapshotRepository` | `insert()` | `AsyncMarketSnapshotWriter` |
|
||||||
|
| `OpportunityRepository` | `insert()` | `AsyncOpportunityWriter` |
|
||||||
|
| `TradeRepository` | `insert()` | `AsyncExecutionWriter` |
|
||||||
|
| `OrderRepository` | `insert()` | `AsyncExecutionWriter` |
|
||||||
|
| `PnLRepository` | `insert()` | `AsyncExecutionWriter` |
|
||||||
|
| `AuditRepository` | `insert()`, `list_recent()` | API routes, lifecycle |
|
||||||
|
| `RuntimeStateRepository` | `insert()`, `latest()` | Lifecycle, API |
|
||||||
|
| `ConfigSectionRepository` | `create_section()`, `get_section()`, `list_sections()` | Config service |
|
||||||
|
| `ConfigSettingRepository` | Full CRUD + `get_latest_updated_at()` | Config service |
|
||||||
|
| `ConfigPairingRepository` | Full CRUD + `upsert_pairing()`, `list_pairings()` | Feeds, pairing sync |
|
||||||
|
| `ConfigBacktestingDefaultsRepository` | `create_defaults()`, `get_defaults()`, `update_defaults()` | Config service |
|
||||||
|
| `KrakenAccountSnapshotRepository` | `insert_snapshot()`, `latest_snapshot()` | Fee sync loop |
|
||||||
|
| `BacktestJobRepository` | Full CRUD | Backtesting UI + worker |
|
||||||
|
|
||||||
|
## Async Writers
|
||||||
|
|
||||||
|
Three background writer tasks buffer high-frequency writes:
|
||||||
|
|
||||||
|
- **`AsyncExecutionWriter`** — trades/orders/P&L queue
|
||||||
|
- **`AsyncMarketSnapshotWriter`** — order-book snapshot queue
|
||||||
|
- **`AsyncOpportunityWriter`** — opportunity event queue
|
||||||
|
|
||||||
|
Each uses an `asyncio.Queue` and drains it in a background task with
|
||||||
|
`await repo.insert(...)`.
|
||||||
|
|
||||||
|
## Integration Tests
|
||||||
|
|
||||||
|
`tests/integration/test_postgresql_schema.py` verifies:
|
||||||
|
|
||||||
|
- Connection to PostgreSQL server
|
||||||
|
- `pgcrypto` extension availability
|
||||||
|
- All 15 tables exist after migration
|
||||||
|
- Migration is idempotent
|
||||||
|
- Correct columns per table
|
||||||
|
- Primary keys and unique constraints
|
||||||
|
- Tables start empty
|
||||||
|
- Simple INSERT/SELECT round-trip
|
||||||
|
- `ON CONFLICT ... DO UPDATE` on config_pairings
|
||||||
@@ -39,7 +39,7 @@ Key end-to-end latency baselines from `latency_baseline.json`:
|
|||||||
|
|
||||||
## Optimization Note
|
## Optimization Note
|
||||||
|
|
||||||
`MetricsCalculator.compute()` was optimized to use DuckDB SQL aggregations and quantiles, reducing Python-side row scans.
|
`MetricsCalculator.compute()` uses PostgreSQL SQL aggregations and percentiles, reducing Python-side row scans.
|
||||||
|
|
||||||
Measured benchmark (`scripts/benchmark_metrics_compute.py`):
|
Measured benchmark (`scripts/benchmark_metrics_compute.py`):
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -54,7 +54,7 @@ pretty = true
|
|||||||
mypy_path = "src"
|
mypy_path = "src"
|
||||||
|
|
||||||
[[tool.mypy.overrides]]
|
[[tool.mypy.overrides]]
|
||||||
module = ["duckdb", "keyring", "sortedcontainers"]
|
module = ["asyncpg", "keyring", "sortedcontainers"]
|
||||||
ignore_missing_imports = true
|
ignore_missing_imports = true
|
||||||
|
|
||||||
[tool.pytest.ini_options]
|
[tool.pytest.ini_options]
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
# Unpinned dev dependencies (latest available)
|
# Unpinned dev dependencies (latest available)
|
||||||
|
asyncpg-stubs
|
||||||
black
|
black
|
||||||
mypy
|
mypy
|
||||||
pre-commit
|
pre-commit
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Unpinned runtime dependencies (latest available)
|
# Unpinned runtime dependencies (latest available)
|
||||||
|
asyncpg
|
||||||
cryptography
|
cryptography
|
||||||
duckdb
|
|
||||||
fastapi
|
fastapi
|
||||||
httptools
|
httptools
|
||||||
httpx
|
httpx
|
||||||
|
|||||||
+14
-10
@@ -25,15 +25,15 @@ from arbitrade.market_data.feed_builder import (
|
|||||||
)
|
)
|
||||||
from arbitrade.metrics import MetricsCalculator
|
from arbitrade.metrics import MetricsCalculator
|
||||||
from arbitrade.runtime.lifecycle import graceful_shutdown, restore_runtime_state
|
from arbitrade.runtime.lifecycle import graceful_shutdown, restore_runtime_state
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.market_snapshots import AsyncMarketSnapshotWriter
|
from arbitrade.storage.market_snapshots import AsyncMarketSnapshotWriter
|
||||||
from arbitrade.storage.opportunities import AsyncOpportunityWriter
|
from arbitrade.storage.opportunities import AsyncOpportunityWriter
|
||||||
from arbitrade.storage.repositories import AuditRepository, RuntimeStateRepository
|
from arbitrade.storage.repositories import AuditRepository, RuntimeStateRepository, MarketSnapshotRepository, OpportunityRepository
|
||||||
|
|
||||||
_LOG = structlog.get_logger(__name__)
|
_LOG = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
def _start_feed(app: FastAPI, *, kill_switch_only: bool = False) -> asyncio.Task[None] | None:
|
async def _start_feed(app: FastAPI, *, kill_switch_only: bool = False) -> asyncio.Task[None] | None:
|
||||||
"""Create and start a MarketDataFeed task from enabled pairings.
|
"""Create and start a MarketDataFeed task from enabled pairings.
|
||||||
|
|
||||||
If kill_switch_only=True, only create a kill-switch-bound stub (no detector/feed).
|
If kill_switch_only=True, only create a kill-switch-bound stub (no detector/feed).
|
||||||
@@ -45,14 +45,14 @@ def _start_feed(app: FastAPI, *, kill_switch_only: bool = False) -> asyncio.Task
|
|||||||
controls = app.state.dashboard_controls
|
controls = app.state.dashboard_controls
|
||||||
|
|
||||||
# Build detector from enabled pairings
|
# Build detector from enabled pairings
|
||||||
detector = build_detector_from_enabled_pairings(
|
detector = await build_detector_from_enabled_pairings(
|
||||||
db,
|
db,
|
||||||
fee_rate=0.0, # will be overridden by fee sync
|
fee_rate=0.0, # will be overridden by fee sync
|
||||||
max_depth_levels=controls.strategy_max_depth_levels,
|
max_depth_levels=controls.strategy_max_depth_levels,
|
||||||
min_profit_threshold=controls.strategy_profit_threshold,
|
min_profit_threshold=controls.strategy_profit_threshold,
|
||||||
)
|
)
|
||||||
|
|
||||||
symbols = get_enabled_pair_symbols(db)
|
symbols = await get_enabled_pair_symbols(db)
|
||||||
if not symbols and not kill_switch_only:
|
if not symbols and not kill_switch_only:
|
||||||
_LOG.warning("no_enabled_pair_symbols_feed_not_started")
|
_LOG.warning("no_enabled_pair_symbols_feed_not_started")
|
||||||
return None
|
return None
|
||||||
@@ -64,8 +64,8 @@ def _start_feed(app: FastAPI, *, kill_switch_only: bool = False) -> asyncio.Task
|
|||||||
|
|
||||||
ws_client.set_subscribed_symbols(symbols)
|
ws_client.set_subscribed_symbols(symbols)
|
||||||
|
|
||||||
snapshot_writer = AsyncMarketSnapshotWriter(db)
|
snapshot_writer = AsyncMarketSnapshotWriter(MarketSnapshotRepository(db))
|
||||||
opportunity_writer = AsyncOpportunityWriter(db)
|
opportunity_writer = AsyncOpportunityWriter(OpportunityRepository(db))
|
||||||
|
|
||||||
feed = MarketDataFeed(
|
feed = MarketDataFeed(
|
||||||
ws_client=ws_client,
|
ws_client=ws_client,
|
||||||
@@ -90,8 +90,8 @@ def _start_feed(app: FastAPI, *, kill_switch_only: bool = False) -> asyncio.Task
|
|||||||
def create_app(settings: Settings) -> FastAPI:
|
def create_app(settings: Settings) -> FastAPI:
|
||||||
configure_logging(settings.log_level, settings.log_json)
|
configure_logging(settings.log_level, settings.log_json)
|
||||||
|
|
||||||
db = DuckDBStore(settings)
|
db = PgStore(settings)
|
||||||
db.migrate()
|
|
||||||
kraken_client = KrakenRestClient(settings)
|
kraken_client = KrakenRestClient(settings)
|
||||||
fee_sync_stop_event = asyncio.Event()
|
fee_sync_stop_event = asyncio.Event()
|
||||||
pairing_sync_stop_event = asyncio.Event()
|
pairing_sync_stop_event = asyncio.Event()
|
||||||
@@ -101,6 +101,9 @@ def create_app(settings: Settings) -> FastAPI:
|
|||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
|
||||||
|
await app.state.store.start()
|
||||||
|
await app.state.store.migrate()
|
||||||
|
await app.state.configuration_service.load_database_settings()
|
||||||
await restore_runtime_state(app)
|
await restore_runtime_state(app)
|
||||||
fee_sync_task = asyncio.create_task(
|
fee_sync_task = asyncio.create_task(
|
||||||
run_fee_sync_loop(
|
run_fee_sync_loop(
|
||||||
@@ -123,7 +126,7 @@ def create_app(settings: Settings) -> FastAPI:
|
|||||||
name="backtest_worker",
|
name="backtest_worker",
|
||||||
)
|
)
|
||||||
# Start market data feed from enabled pairings
|
# Start market data feed from enabled pairings
|
||||||
_start_feed(app)
|
await _start_feed(app)
|
||||||
app.state.fee_sync_task = fee_sync_task
|
app.state.fee_sync_task = fee_sync_task
|
||||||
app.state.pairing_sync_task = pairing_sync_task
|
app.state.pairing_sync_task = pairing_sync_task
|
||||||
app.state.backtest_task = backtest_task
|
app.state.backtest_task = backtest_task
|
||||||
@@ -161,6 +164,7 @@ def create_app(settings: Settings) -> FastAPI:
|
|||||||
pass
|
pass
|
||||||
await kraken_client.close()
|
await kraken_client.close()
|
||||||
await graceful_shutdown(app)
|
await graceful_shutdown(app)
|
||||||
|
await app.state.store.stop()
|
||||||
|
|
||||||
app = FastAPI(title="arbitrade", version="0.1.0", lifespan=lifespan)
|
app = FastAPI(title="arbitrade", version="0.1.0", lifespan=lifespan)
|
||||||
app.state.settings = settings
|
app.state.settings = settings
|
||||||
|
|||||||
+153
-147
@@ -9,7 +9,6 @@ from pathlib import Path
|
|||||||
from typing import cast
|
from typing import cast
|
||||||
from urllib.parse import parse_qs
|
from urllib.parse import parse_qs
|
||||||
|
|
||||||
import duckdb
|
|
||||||
import orjson
|
import orjson
|
||||||
from fastapi import APIRouter, Depends, Request, Response
|
from fastapi import APIRouter, Depends, Request, Response
|
||||||
from fastapi.responses import HTMLResponse, JSONResponse, StreamingResponse
|
from fastapi.responses import HTMLResponse, JSONResponse, StreamingResponse
|
||||||
@@ -21,6 +20,7 @@ from arbitrade.api.control_state import DashboardControlState
|
|||||||
from arbitrade.config.pairing_sync import sync_pairings_from_kraken
|
from arbitrade.config.pairing_sync import sync_pairings_from_kraken
|
||||||
from arbitrade.config.service import ConfigPairing
|
from arbitrade.config.service import ConfigPairing
|
||||||
from arbitrade.detection.graph import CurrencyGraph, TriangularCycle
|
from arbitrade.detection.graph import CurrencyGraph, TriangularCycle
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import (
|
from arbitrade.storage.repositories import (
|
||||||
AuditRecord,
|
AuditRecord,
|
||||||
AuditRepository,
|
AuditRepository,
|
||||||
@@ -35,7 +35,8 @@ public_router = APIRouter()
|
|||||||
|
|
||||||
def _resolve_templates_directory() -> str:
|
def _resolve_templates_directory() -> str:
|
||||||
# Support source layout, Docker runtime (/app), and installed package data.
|
# Support source layout, Docker runtime (/app), and installed package data.
|
||||||
source_layout_path = Path(__file__).resolve().parents[3] / "web" / "templates"
|
source_layout_path = Path(
|
||||||
|
__file__).resolve().parents[3] / "web" / "templates"
|
||||||
if source_layout_path.is_dir():
|
if source_layout_path.is_dir():
|
||||||
return str(source_layout_path)
|
return str(source_layout_path)
|
||||||
|
|
||||||
@@ -44,7 +45,8 @@ def _resolve_templates_directory() -> str:
|
|||||||
return str(docker_runtime_path)
|
return str(docker_runtime_path)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
package_path = resources.files("arbitrade").joinpath("web", "templates")
|
package_path = resources.files(
|
||||||
|
"arbitrade").joinpath("web", "templates")
|
||||||
if package_path.is_dir():
|
if package_path.is_dir():
|
||||||
return str(package_path)
|
return str(package_path)
|
||||||
except (ModuleNotFoundError, AttributeError):
|
except (ModuleNotFoundError, AttributeError):
|
||||||
@@ -64,8 +66,8 @@ def _format_metric(value: float | None, *, precision: int = 2, suffix: str = "")
|
|||||||
return f"{value:.{precision}f}{suffix}"
|
return f"{value:.{precision}f}{suffix}"
|
||||||
|
|
||||||
|
|
||||||
def _dashboard_metrics(request: Request) -> dict[str, str]:
|
async def _dashboard_metrics(request: Request) -> dict[str, str]:
|
||||||
metrics = request.app.state.metrics.compute()
|
metrics = await request.app.state.metrics.compute()
|
||||||
return {
|
return {
|
||||||
"realized_pnl": _format_metric(metrics.realized_pnl_usd, precision=2, suffix=" USD"),
|
"realized_pnl": _format_metric(metrics.realized_pnl_usd, precision=2, suffix=" USD"),
|
||||||
"win_rate": _format_metric(
|
"win_rate": _format_metric(
|
||||||
@@ -91,57 +93,85 @@ def _dashboard_metrics(request: Request) -> dict[str, str]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _table_columns(conn: duckdb.DuckDBPyConnection, table_name: str) -> set[str]:
|
async def _dashboard_overview(request: Request) -> dict[str, object]:
|
||||||
rows = conn.execute(f"PRAGMA table_info('{table_name}')").fetchall()
|
store: PgStore = request.app.state.store
|
||||||
return {str(row[1]) for row in rows}
|
async with store.pool.acquire() as conn:
|
||||||
|
portfolio_row = await conn.fetchrow("""
|
||||||
|
|
||||||
def _dashboard_overview(request: Request) -> dict[str, object]:
|
|
||||||
store = request.app.state.store
|
|
||||||
with store.connect() as conn:
|
|
||||||
trade_columns = _table_columns(conn, "trades")
|
|
||||||
trade_ref_expr = "trade_ref" if "trade_ref" in trade_columns else "CAST(id AS VARCHAR)"
|
|
||||||
cycle_expr = "cycle" if "cycle" in trade_columns else "NULL"
|
|
||||||
if "finished_at" in trade_columns:
|
|
||||||
open_trade_filter = "finished_at IS NULL"
|
|
||||||
else:
|
|
||||||
open_trade_filter = "LOWER(status) NOT IN ('filled', 'closed', 'cancelled', 'canceled')"
|
|
||||||
|
|
||||||
portfolio_row = conn.execute("""
|
|
||||||
SELECT balances, total_value_usd
|
SELECT balances, total_value_usd
|
||||||
FROM portfolio_snapshots
|
FROM portfolio_snapshots
|
||||||
ORDER BY snapshot_at DESC
|
ORDER BY snapshot_at DESC
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
""").fetchone()
|
""")
|
||||||
open_trades = conn.execute(f"""
|
open_trades = await conn.fetch("""
|
||||||
SELECT {trade_ref_expr}, status, started_at, {cycle_expr}
|
SELECT trade_ref, status, started_at, cycle
|
||||||
FROM trades
|
FROM trades
|
||||||
WHERE {open_trade_filter}
|
WHERE finished_at IS NULL
|
||||||
ORDER BY started_at DESC
|
ORDER BY started_at DESC
|
||||||
LIMIT 5
|
LIMIT 5
|
||||||
""").fetchall()
|
""")
|
||||||
rpnl = conn.execute("""
|
rpnl = await conn.fetchrow("""
|
||||||
SELECT COALESCE(SUM(COALESCE(realized_pnl, 0)), 0)
|
SELECT COALESCE(SUM(COALESCE(realized_pnl, 0)), 0)
|
||||||
FROM trades
|
FROM trades
|
||||||
""").fetchone()
|
""")
|
||||||
latest_opportunities = conn.execute("""
|
latest_opportunities = await conn.fetch("""
|
||||||
SELECT cycle, net_pct, est_profit, detected_at
|
SELECT cycle, net_pct, est_profit, detected_at
|
||||||
FROM opportunities
|
FROM opportunities
|
||||||
ORDER BY detected_at DESC
|
ORDER BY detected_at DESC
|
||||||
LIMIT 5
|
LIMIT 5
|
||||||
""").fetchall()
|
""")
|
||||||
|
|
||||||
|
# Query equity from kraken_account_snapshots
|
||||||
|
equity_value = "—"
|
||||||
|
try:
|
||||||
|
equity_row = await conn.fetchrow("""
|
||||||
|
SELECT trade_balance_raw
|
||||||
|
FROM kraken_account_snapshots
|
||||||
|
ORDER BY snapshot_at DESC
|
||||||
|
LIMIT 1
|
||||||
|
""")
|
||||||
|
if equity_row is not None and equity_row["trade_balance_raw"] is not None:
|
||||||
|
tb_raw = equity_row["trade_balance_raw"]
|
||||||
|
if isinstance(tb_raw, str):
|
||||||
|
tb_raw = json.loads(tb_raw)
|
||||||
|
if isinstance(tb_raw, dict):
|
||||||
|
eb = tb_raw.get("eb")
|
||||||
|
equity_value = f"{float(eb):.2f} USD" if eb is not None else "—"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Query latest Kraken account snapshot for fee info
|
||||||
|
fee_tier = "—"
|
||||||
|
maker_fee = "—"
|
||||||
|
taker_fee = "—"
|
||||||
|
thirty_day_volume = "—"
|
||||||
|
try:
|
||||||
|
acct_row = await conn.fetchrow("""
|
||||||
|
SELECT fee_tier, maker_fee, taker_fee, thirty_day_volume
|
||||||
|
FROM kraken_account_snapshots
|
||||||
|
ORDER BY snapshot_at DESC
|
||||||
|
LIMIT 1
|
||||||
|
""")
|
||||||
|
if acct_row is not None:
|
||||||
|
fee_tier = str(
|
||||||
|
acct_row["fee_tier"]) if acct_row["fee_tier"] is not None else "—"
|
||||||
|
maker_fee = f"{float(acct_row['maker_fee']):.4%}" if acct_row["maker_fee"] is not None else "—"
|
||||||
|
taker_fee = f"{float(acct_row['taker_fee']):.4%}" if acct_row["taker_fee"] is not None else "—"
|
||||||
|
thirty_day_volume = f"{float(acct_row['thirty_day_volume']):.2f}" if acct_row[
|
||||||
|
"thirty_day_volume"] is not None else "—"
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
balances_value = "—"
|
balances_value = "—"
|
||||||
total_value = "—"
|
total_value = "—"
|
||||||
equity_value = "—"
|
|
||||||
if portfolio_row is not None:
|
if portfolio_row is not None:
|
||||||
balances_raw, total_value_raw = portfolio_row
|
balances_raw = portfolio_row["balances"]
|
||||||
|
total_value_raw = portfolio_row["total_value_usd"]
|
||||||
if isinstance(balances_raw, str) and balances_raw:
|
if isinstance(balances_raw, str) and balances_raw:
|
||||||
try:
|
try:
|
||||||
parsed = json.loads(balances_raw)
|
parsed = json.loads(balances_raw)
|
||||||
if isinstance(parsed, dict):
|
if isinstance(parsed, dict):
|
||||||
# Filter out zero balances, show non-zero as "AMT ASSET"
|
non_zero = {k: float(v)
|
||||||
non_zero = {k: float(v) for k, v in parsed.items() if float(v) > 0.0}
|
for k, v in parsed.items() if float(v) > 0.0}
|
||||||
if non_zero:
|
if non_zero:
|
||||||
balances_value = "<br>".join(
|
balances_value = "<br>".join(
|
||||||
f"{v:.6g} {k}" for k, v in sorted(non_zero.items())
|
f"{v:.6g} {k}" for k, v in sorted(non_zero.items())
|
||||||
@@ -157,63 +187,25 @@ def _dashboard_overview(request: Request) -> dict[str, object]:
|
|||||||
if total_value_raw is not None:
|
if total_value_raw is not None:
|
||||||
total_value = f"{float(total_value_raw):.2f} USD"
|
total_value = f"{float(total_value_raw):.2f} USD"
|
||||||
|
|
||||||
# Query equity from kraken_account_snapshots
|
|
||||||
try:
|
|
||||||
equity_row = conn.execute("""
|
|
||||||
SELECT trade_balance_raw
|
|
||||||
FROM kraken_account_snapshots
|
|
||||||
ORDER BY snapshot_at DESC
|
|
||||||
LIMIT 1
|
|
||||||
""").fetchone()
|
|
||||||
if equity_row is not None and equity_row[0] is not None:
|
|
||||||
tb_raw = equity_row[0]
|
|
||||||
if isinstance(tb_raw, str):
|
|
||||||
tb_raw = json.loads(tb_raw)
|
|
||||||
if isinstance(tb_raw, dict):
|
|
||||||
eb = tb_raw.get("eb")
|
|
||||||
equity_value = f"{float(eb):.2f} USD" if eb is not None else "—"
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
open_trade_rows = [
|
open_trade_rows = [
|
||||||
{
|
{
|
||||||
"trade_ref": str(row[0]),
|
"trade_ref": str(r["trade_ref"]),
|
||||||
"status": str(row[1]),
|
"status": str(r["status"]),
|
||||||
"started_at": row[2].isoformat() if isinstance(row[2], datetime) else "—",
|
"started_at": r["started_at"].isoformat() if isinstance(r["started_at"], datetime) else "—",
|
||||||
"cycle": str(row[3]) if row[3] is not None else "—",
|
"cycle": str(r["cycle"]) if r["cycle"] is not None else "—",
|
||||||
}
|
}
|
||||||
for row in open_trades
|
for r in open_trades
|
||||||
]
|
]
|
||||||
opportunity_rows = [
|
opportunity_rows = [
|
||||||
{
|
{
|
||||||
"cycle": str(row[0]),
|
"cycle": str(r["cycle"]),
|
||||||
"net_pct": f"{float(row[1]):.2f}%" if row[1] is not None else "—",
|
"net_pct": f"{float(r['net_pct']):.2f}%" if r["net_pct"] is not None else "—",
|
||||||
"est_profit": f"{float(row[2]):.2f} USD" if row[2] is not None else "—",
|
"est_profit": f"{float(r['est_profit']):.2f} USD" if r["est_profit"] is not None else "—",
|
||||||
"detected_at": row[3].isoformat() if isinstance(row[3], datetime) else "—",
|
"detected_at": r["detected_at"].isoformat() if isinstance(r["detected_at"], datetime) else "—",
|
||||||
}
|
}
|
||||||
for row in latest_opportunities
|
for r in latest_opportunities
|
||||||
]
|
]
|
||||||
|
|
||||||
# Query latest Kraken account snapshot for fee info
|
|
||||||
fee_tier = "—"
|
|
||||||
maker_fee = "—"
|
|
||||||
taker_fee = "—"
|
|
||||||
thirty_day_volume = "—"
|
|
||||||
try:
|
|
||||||
acct_row = conn.execute("""
|
|
||||||
SELECT fee_tier, maker_fee, taker_fee, thirty_day_volume
|
|
||||||
FROM kraken_account_snapshots
|
|
||||||
ORDER BY snapshot_at DESC
|
|
||||||
LIMIT 1
|
|
||||||
""").fetchone()
|
|
||||||
if acct_row is not None:
|
|
||||||
fee_tier = str(acct_row[0]) if acct_row[0] is not None else "—"
|
|
||||||
maker_fee = f"{float(acct_row[1]):.4%}" if acct_row[1] is not None else "—"
|
|
||||||
taker_fee = f"{float(acct_row[2]):.4%}" if acct_row[2] is not None else "—"
|
|
||||||
thirty_day_volume = f"{float(acct_row[3]):.2f}" if acct_row[3] is not None else "—"
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"status": "live",
|
"status": "live",
|
||||||
"generated_at": datetime.now(UTC).isoformat(),
|
"generated_at": datetime.now(UTC).isoformat(),
|
||||||
@@ -232,26 +224,28 @@ def _dashboard_overview(request: Request) -> dict[str, object]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _dashboard_charts(request: Request) -> dict[str, object]:
|
async def _dashboard_charts(request: Request) -> dict[str, object]:
|
||||||
store = request.app.state.store
|
store: PgStore = request.app.state.store
|
||||||
with store.connect() as conn:
|
async with store.pool.acquire() as conn:
|
||||||
opportunity_rows = conn.execute("""
|
rows = await conn.fetch("""
|
||||||
SELECT detected_at, cycle, net_pct, est_profit
|
SELECT detected_at, cycle, net_pct, est_profit
|
||||||
FROM opportunities
|
FROM opportunities
|
||||||
ORDER BY detected_at DESC
|
ORDER BY detected_at DESC
|
||||||
LIMIT 10
|
LIMIT 10
|
||||||
""").fetchall()
|
""")
|
||||||
|
|
||||||
cr = list(reversed(opportunity_rows))
|
cr = list(reversed(rows))
|
||||||
labels = []
|
labels = []
|
||||||
for index, row in enumerate(cr):
|
for index, row in enumerate(cr):
|
||||||
if isinstance(row[0], datetime):
|
if isinstance(row["detected_at"], datetime):
|
||||||
labels.append(row[0].isoformat())
|
labels.append(row["detected_at"].isoformat())
|
||||||
else:
|
else:
|
||||||
labels.append(f"opportunity-{index + 1}")
|
labels.append(f"opportunity-{index + 1}")
|
||||||
np = [float(row[2]) if row[2] is not None else 0.0 for row in cr]
|
np = [float(row["net_pct"]) if row["net_pct"]
|
||||||
ep = [float(row[3]) if row[3] is not None else 0.0 for row in cr]
|
is not None else 0.0 for row in cr]
|
||||||
cycles = [str(row[1]) for row in cr]
|
ep = [float(row["est_profit"]) if row["est_profit"]
|
||||||
|
is not None else 0.0 for row in cr]
|
||||||
|
cycles = [str(row["cycle"]) for row in cr]
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"labels": labels,
|
"labels": labels,
|
||||||
@@ -272,7 +266,7 @@ def _audit_repository(request: Request) -> AuditRepository | None:
|
|||||||
return cast(AuditRepository | None, repository)
|
return cast(AuditRepository | None, repository)
|
||||||
|
|
||||||
|
|
||||||
def _record_audit(
|
async def _record_audit(
|
||||||
request: Request,
|
request: Request,
|
||||||
*,
|
*,
|
||||||
actor: str,
|
actor: str,
|
||||||
@@ -288,7 +282,7 @@ def _record_audit(
|
|||||||
ret_pl = {str(key): payload[key] for key in payload}
|
ret_pl = {str(key): payload[key] for key in payload}
|
||||||
else:
|
else:
|
||||||
ret_pl = None
|
ret_pl = None
|
||||||
repository.insert(
|
await repository.insert(
|
||||||
AuditRecord(
|
AuditRecord(
|
||||||
occurred_at=datetime.now(UTC),
|
occurred_at=datetime.now(UTC),
|
||||||
actor=actor,
|
actor=actor,
|
||||||
@@ -300,7 +294,7 @@ def _record_audit(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _dashboard_audit(request: Request, *, limit: int = 15) -> dict[str, object]:
|
async def _dashboard_audit(request: Request, *, limit: int = 15) -> dict[str, object]:
|
||||||
repository = _audit_repository(request)
|
repository = _audit_repository(request)
|
||||||
if repository is None:
|
if repository is None:
|
||||||
return {
|
return {
|
||||||
@@ -308,7 +302,7 @@ def _dashboard_audit(request: Request, *, limit: int = 15) -> dict[str, object]:
|
|||||||
"generated_at": datetime.now(UTC).isoformat(),
|
"generated_at": datetime.now(UTC).isoformat(),
|
||||||
}
|
}
|
||||||
|
|
||||||
records = repository.list_recent(limit=limit)
|
records = await repository.list_recent(limit=limit)
|
||||||
entries: list[dict[str, str]] = []
|
entries: list[dict[str, str]] = []
|
||||||
for record in records:
|
for record in records:
|
||||||
payload_text = "—"
|
payload_text = "—"
|
||||||
@@ -355,7 +349,7 @@ def _alert_status_snapshot(request: Request) -> dict[str, object]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _dashboard_config_context(request: Request) -> dict[str, object]:
|
async def _dashboard_config_context(request: Request) -> dict[str, object]:
|
||||||
ctl = _dashboard_controls_state(request)
|
ctl = _dashboard_controls_state(request)
|
||||||
rs = request.app.state.settings
|
rs = request.app.state.settings
|
||||||
max_trade_capital_usd = (
|
max_trade_capital_usd = (
|
||||||
@@ -416,7 +410,8 @@ def _dashboard_config_context(request: Request) -> dict[str, object]:
|
|||||||
max_consecutive_failures_value = (
|
max_consecutive_failures_value = (
|
||||||
str(rs.max_consecutive_failures) if rs.max_consecutive_failures is not None else ""
|
str(rs.max_consecutive_failures) if rs.max_consecutive_failures is not None else ""
|
||||||
)
|
)
|
||||||
strategy_stat_arb_enabled = bool(getattr(rs, "strategy_enable_stat_arb_experiment", False))
|
strategy_stat_arb_enabled = bool(
|
||||||
|
getattr(rs, "strategy_enable_stat_arb_experiment", False))
|
||||||
|
|
||||||
return {
|
return {
|
||||||
# Runtime
|
# Runtime
|
||||||
@@ -537,7 +532,8 @@ def _dashboard_controls(request: Request) -> dict[str, object]:
|
|||||||
alerts_last_channel_results = [
|
alerts_last_channel_results = [
|
||||||
str(item) for item in cast(list[object], alert_status.get("last_channel_results", []))
|
str(item) for item in cast(list[object], alert_status.get("last_channel_results", []))
|
||||||
]
|
]
|
||||||
strategy_stat_arb_enabled = bool(getattr(rs, "strategy_enable_stat_arb_experiment", False))
|
strategy_stat_arb_enabled = bool(
|
||||||
|
getattr(rs, "strategy_enable_stat_arb_experiment", False))
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"execution_status": "running" if ctl.is_running else "stopped",
|
"execution_status": "running" if ctl.is_running else "stopped",
|
||||||
@@ -611,7 +607,7 @@ def _normalize_fee_profile(profile: str) -> str:
|
|||||||
return profile.strip().lower().replace("-", "_")
|
return profile.strip().lower().replace("-", "_")
|
||||||
|
|
||||||
|
|
||||||
def _fee_rate_for_profile(
|
async def _fee_rate_for_profile(
|
||||||
profile: str,
|
profile: str,
|
||||||
custom_fee_rate: float | None,
|
custom_fee_rate: float | None,
|
||||||
request: Request | None = None,
|
request: Request | None = None,
|
||||||
@@ -628,9 +624,9 @@ def _fee_rate_for_profile(
|
|||||||
if normalized == "api":
|
if normalized == "api":
|
||||||
if request is None:
|
if request is None:
|
||||||
raise ValueError("api fee profile requires request context")
|
raise ValueError("api fee profile requires request context")
|
||||||
store = request.app.state.store
|
store: PgStore = request.app.state.store
|
||||||
repo = KrakenAccountSnapshotRepository(store)
|
repo = KrakenAccountSnapshotRepository(store)
|
||||||
latest = repo.latest_snapshot()
|
latest = await repo.latest_snapshot()
|
||||||
if latest is not None and latest.maker_fee is not None:
|
if latest is not None and latest.maker_fee is not None:
|
||||||
return latest.maker_fee
|
return latest.maker_fee
|
||||||
# Fallback to standard if no snapshot yet
|
# Fallback to standard if no snapshot yet
|
||||||
@@ -700,11 +696,11 @@ def _build_cycles_from_events(
|
|||||||
return graph.index_cycles_by_pair(cycles), sorted(symbols)
|
return graph.index_cycles_by_pair(cycles), sorted(symbols)
|
||||||
|
|
||||||
|
|
||||||
def _recent_backtest_reports(request: Request) -> list[dict[str, object]]:
|
async def _recent_backtest_reports(request: Request) -> list[dict[str, object]]:
|
||||||
"""Fetch recent backtest jobs from DB."""
|
"""Fetch recent backtest jobs from DB."""
|
||||||
store = request.app.state.store
|
store: PgStore = request.app.state.store
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
jobs = repo.list_jobs(limit=20)
|
jobs = await repo.list_jobs(limit=20)
|
||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
"job_id": j.id or "",
|
"job_id": j.id or "",
|
||||||
@@ -719,7 +715,7 @@ def _recent_backtest_reports(request: Request) -> list[dict[str, object]]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
def _backtesting_panel_context(
|
async def _backtesting_panel_context(
|
||||||
request: Request,
|
request: Request,
|
||||||
*,
|
*,
|
||||||
status: str = "idle",
|
status: str = "idle",
|
||||||
@@ -742,7 +738,7 @@ def _backtesting_panel_context(
|
|||||||
if defaults is not None:
|
if defaults is not None:
|
||||||
default_values.update(defaults)
|
default_values.update(defaults)
|
||||||
|
|
||||||
reports = _recent_backtest_reports(request)
|
reports = await _recent_backtest_reports(request)
|
||||||
latest = latest_report or (reports[0] if reports else None)
|
latest = latest_report or (reports[0] if reports else None)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
@@ -814,10 +810,11 @@ async def dashboard_backtesting_page(request: Request) -> HTMLResponse:
|
|||||||
|
|
||||||
@router.get("/dashboard/fragment/backtesting", response_class=HTMLResponse)
|
@router.get("/dashboard/fragment/backtesting", response_class=HTMLResponse)
|
||||||
async def dashboard_backtesting_fragment(request: Request) -> HTMLResponse:
|
async def dashboard_backtesting_fragment(request: Request) -> HTMLResponse:
|
||||||
|
d_context = await _dashboard_config_context(request)
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/backtesting_panel.html",
|
name="partials/backtesting_panel.html",
|
||||||
context={"request": request, **_backtesting_panel_context(request)},
|
context={"request": request, **d_context},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -826,7 +823,7 @@ async def dashboard_backtesting_pairings_fragment(request: Request) -> HTMLRespo
|
|||||||
"""HTMX fragment: pairing checkboxes for backtest form."""
|
"""HTMX fragment: pairing checkboxes for backtest form."""
|
||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
repo = ConfigPairingRepository(store)
|
repo = ConfigPairingRepository(store)
|
||||||
pairings = repo.list_pairings()
|
pairings = await repo.list_pairings()
|
||||||
pairings.sort(key=lambda p: (p.base_asset, p.quote_asset))
|
pairings.sort(key=lambda p: (p.base_asset, p.quote_asset))
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
@@ -840,7 +837,7 @@ async def dashboard_metrics(request: Request) -> HTMLResponse:
|
|||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/metrics.html",
|
name="partials/metrics.html",
|
||||||
context={"request": request, **_dashboard_metrics(request)},
|
context={"request": request, **(await _dashboard_metrics(request))},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -849,7 +846,7 @@ async def dashboard_overview(request: Request) -> HTMLResponse:
|
|||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/overview.html",
|
name="partials/overview.html",
|
||||||
context={"request": request, **_dashboard_overview(request)},
|
context={"request": request, **(await _dashboard_overview(request))},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -867,7 +864,7 @@ async def dashboard_charts(request: Request) -> HTMLResponse:
|
|||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/charts.html",
|
name="partials/charts.html",
|
||||||
context={"request": request, **_dashboard_charts(request)},
|
context={"request": request, **(await _dashboard_charts(request))},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -879,7 +876,7 @@ async def dashboard_audit_page(request: Request) -> HTMLResponse:
|
|||||||
context={
|
context={
|
||||||
"title": "Arbitrade Audit Trail",
|
"title": "Arbitrade Audit Trail",
|
||||||
"request": request,
|
"request": request,
|
||||||
**_dashboard_audit(request),
|
**(await _dashboard_audit(request)),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -889,7 +886,7 @@ async def dashboard_audit_fragment(request: Request) -> HTMLResponse:
|
|||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/audit.html",
|
name="partials/audit.html",
|
||||||
context={"request": request, **_dashboard_audit(request)},
|
context={"request": request, **(await _dashboard_audit(request))},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -898,12 +895,13 @@ async def dashboard_audit(request: Request) -> HTMLResponse:
|
|||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/audit.html",
|
name="partials/audit.html",
|
||||||
context={"request": request, **_dashboard_audit(request)},
|
context={"request": request, **(await _dashboard_audit(request))},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/dashboard/config", response_class=HTMLResponse)
|
@router.get("/dashboard/config", response_class=HTMLResponse)
|
||||||
async def dashboard_config_page(request: Request) -> HTMLResponse:
|
async def dashboard_config_page(request: Request) -> HTMLResponse:
|
||||||
|
d_context = await _dashboard_config_context(request)
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="config.html",
|
name="config.html",
|
||||||
@@ -911,17 +909,18 @@ async def dashboard_config_page(request: Request) -> HTMLResponse:
|
|||||||
"title": "Arbitrade Configuration",
|
"title": "Arbitrade Configuration",
|
||||||
"request": request,
|
"request": request,
|
||||||
"config_endpoint": "/dashboard/control/config",
|
"config_endpoint": "/dashboard/control/config",
|
||||||
**_dashboard_config_context(request),
|
**d_context,
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/dashboard/fragment/config", response_class=HTMLResponse)
|
@router.get("/dashboard/fragment/config", response_class=HTMLResponse)
|
||||||
async def dashboard_config_fragment(request: Request) -> HTMLResponse:
|
async def dashboard_config_fragment(request: Request) -> HTMLResponse:
|
||||||
|
d_context = await _dashboard_config_context(request)
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/config.html",
|
name="partials/config.html",
|
||||||
context={"request": request, **_dashboard_config_context(request)},
|
context={"request": request, **d_context},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -932,7 +931,7 @@ async def dashboard_alert_status(request: Request) -> JSONResponse:
|
|||||||
|
|
||||||
@router.get("/dashboard/api/audit/recent", response_class=JSONResponse)
|
@router.get("/dashboard/api/audit/recent", response_class=JSONResponse)
|
||||||
async def dashboard_audit_recent(request: Request) -> JSONResponse:
|
async def dashboard_audit_recent(request: Request) -> JSONResponse:
|
||||||
return JSONResponse(_dashboard_audit(request, limit=25))
|
return JSONResponse(await _dashboard_audit(request, limit=25))
|
||||||
|
|
||||||
|
|
||||||
@router.get("/dashboard/api/backtesting/reports", response_class=JSONResponse)
|
@router.get("/dashboard/api/backtesting/reports", response_class=JSONResponse)
|
||||||
@@ -940,7 +939,7 @@ async def dashboard_backtesting_reports(request: Request) -> JSONResponse:
|
|||||||
return JSONResponse(
|
return JSONResponse(
|
||||||
{
|
{
|
||||||
"generated_at": datetime.now(UTC).isoformat(),
|
"generated_at": datetime.now(UTC).isoformat(),
|
||||||
"reports": _recent_backtest_reports(request),
|
"reports": await _recent_backtest_reports(request),
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -965,9 +964,10 @@ async def dashboard_backtesting_run(request: Request) -> HTMLResponse:
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
custom_fee_rate = (
|
custom_fee_rate = (
|
||||||
float(defaults["custom_fee_rate"]) if defaults["custom_fee_rate"].strip() else None
|
float(defaults["custom_fee_rate"]
|
||||||
|
) if defaults["custom_fee_rate"].strip() else None
|
||||||
)
|
)
|
||||||
fee_rate = _fee_rate_for_profile(defaults["fee_profile"], custom_fee_rate, request=request)
|
fee_rate = await _fee_rate_for_profile(defaults["fee_profile"], custom_fee_rate, request=request)
|
||||||
|
|
||||||
config_dict: dict[str, object] = {
|
config_dict: dict[str, object] = {
|
||||||
"source": defaults["source"],
|
"source": defaults["source"],
|
||||||
@@ -986,13 +986,13 @@ async def dashboard_backtesting_run(request: Request) -> HTMLResponse:
|
|||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
events_label = defaults["symbols"] if defaults["symbols"] else "DB-sourced"
|
events_label = defaults["symbols"] if defaults["symbols"] else "DB-sourced"
|
||||||
job = repo.create_job(events_label, config_dict)
|
job = await repo.create_job(events_label, config_dict)
|
||||||
msg_job = job.id[:8] if job.id else "unknown"
|
msg_job = job.id[:8] if job.id else "unknown"
|
||||||
|
|
||||||
queue = request.app.state.backtest_queue
|
queue = request.app.state.backtest_queue
|
||||||
await queue.put((job.id or "", config_dict))
|
await queue.put((job.id or "", config_dict))
|
||||||
|
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.backtesting.submit",
|
event_type="dashboard.backtesting.submit",
|
||||||
@@ -1000,14 +1000,14 @@ async def dashboard_backtesting_run(request: Request) -> HTMLResponse:
|
|||||||
payload={"job_id": job.id, "source": defaults["source"]},
|
payload={"job_id": job.id, "source": defaults["source"]},
|
||||||
)
|
)
|
||||||
|
|
||||||
context = _backtesting_panel_context(
|
context = await _backtesting_panel_context(
|
||||||
request,
|
request,
|
||||||
status="submitted",
|
status="submitted",
|
||||||
message=f"Job {msg_job}... queued. Refresh to see results.",
|
message=f"Job {msg_job}... queued. Refresh to see results.",
|
||||||
defaults=defaults,
|
defaults=defaults,
|
||||||
)
|
)
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
context = _backtesting_panel_context(
|
context = await _backtesting_panel_context(
|
||||||
request,
|
request,
|
||||||
status="failed",
|
status="failed",
|
||||||
message=str(exc),
|
message=str(exc),
|
||||||
@@ -1025,11 +1025,12 @@ async def dashboard_backtesting_run(request: Request) -> HTMLResponse:
|
|||||||
async def dashboard_backtesting_delete(request: Request, job_id: str) -> HTMLResponse:
|
async def dashboard_backtesting_delete(request: Request, job_id: str) -> HTMLResponse:
|
||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
repo.delete_job(job_id)
|
context = await _backtesting_panel_context(request)
|
||||||
|
await repo.delete_job(job_id)
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
name="partials/backtesting_panel.html",
|
name="partials/backtesting_panel.html",
|
||||||
context={"request": request, **_backtesting_panel_context(request)},
|
context={"request": request, **context},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -1037,7 +1038,7 @@ async def dashboard_backtesting_delete(request: Request, job_id: str) -> HTMLRes
|
|||||||
async def dashboard_backtesting_job_detail(request: Request, job_id: str) -> HTMLResponse:
|
async def dashboard_backtesting_job_detail(request: Request, job_id: str) -> HTMLResponse:
|
||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
job = repo.get_job(job_id)
|
job = await repo.get_job(job_id)
|
||||||
if job is None:
|
if job is None:
|
||||||
return HTMLResponse("<p>Job not found</p>", status_code=404)
|
return HTMLResponse("<p>Job not found</p>", status_code=404)
|
||||||
|
|
||||||
@@ -1069,7 +1070,7 @@ async def dashboard_backtesting_job_detail(request: Request, job_id: str) -> HTM
|
|||||||
async def dashboard_backtesting_export(request: Request, job_id: str) -> Response:
|
async def dashboard_backtesting_export(request: Request, job_id: str) -> Response:
|
||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
job = repo.get_job(job_id)
|
job = await repo.get_job(job_id)
|
||||||
if job is None:
|
if job is None:
|
||||||
return Response("Job not found", status_code=404)
|
return Response("Job not found", status_code=404)
|
||||||
|
|
||||||
@@ -1087,7 +1088,8 @@ async def dashboard_backtesting_export(request: Request, job_id: str) -> Respons
|
|||||||
return Response(
|
return Response(
|
||||||
content=orjson.dumps(payload).decode("utf-8"),
|
content=orjson.dumps(payload).decode("utf-8"),
|
||||||
media_type="application/x-jsonlines",
|
media_type="application/x-jsonlines",
|
||||||
headers={"Content-Disposition": f"attachment; filename=backtest_{job_id[:8]}.jsonl"},
|
headers={
|
||||||
|
"Content-Disposition": f"attachment; filename=backtest_{job_id[:8]}.jsonl"},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -1104,7 +1106,7 @@ async def dashboard_control_start(request: Request) -> HTMLResponse:
|
|||||||
title="Execution started",
|
title="Execution started",
|
||||||
message="Dashboard control started execution.",
|
message="Dashboard control started execution.",
|
||||||
)
|
)
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.control.start",
|
event_type="dashboard.control.start",
|
||||||
@@ -1131,7 +1133,7 @@ async def dashboard_control_stop(request: Request) -> HTMLResponse:
|
|||||||
title="Execution stopped",
|
title="Execution stopped",
|
||||||
message="Dashboard control stopped execution.",
|
message="Dashboard control stopped execution.",
|
||||||
)
|
)
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.control.stop",
|
event_type="dashboard.control.stop",
|
||||||
@@ -1162,7 +1164,7 @@ async def dashboard_control_kill_switch(request: Request) -> HTMLResponse:
|
|||||||
message="Kill switch triggered from dashboard control.",
|
message="Kill switch triggered from dashboard control.",
|
||||||
details={"reason": reason},
|
details={"reason": reason},
|
||||||
)
|
)
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.control.kill_switch",
|
event_type="dashboard.control.kill_switch",
|
||||||
@@ -1232,7 +1234,7 @@ async def dashboard_control_config(request: Request) -> HTMLResponse:
|
|||||||
"paper_trading_mode": "true" if rs.paper_trading_mode else "false",
|
"paper_trading_mode": "true" if rs.paper_trading_mode else "false",
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.control.config",
|
event_type="dashboard.control.config",
|
||||||
@@ -1258,11 +1260,12 @@ async def dashboard_control_config(request: Request) -> HTMLResponse:
|
|||||||
|
|
||||||
@router.get("/dashboard/stream/metrics")
|
@router.get("/dashboard/stream/metrics")
|
||||||
async def dashboard_metrics_stream(request: Request) -> StreamingResponse:
|
async def dashboard_metrics_stream(request: Request) -> StreamingResponse:
|
||||||
|
metrics = await _dashboard_metrics(request)
|
||||||
fragment = (
|
fragment = (
|
||||||
templates.get_template("partials/metrics.html")
|
templates.get_template("partials/metrics.html")
|
||||||
.render(
|
.render(
|
||||||
request=request,
|
request=request,
|
||||||
**_dashboard_metrics(request),
|
**metrics,
|
||||||
)
|
)
|
||||||
.strip()
|
.strip()
|
||||||
.replace("\n", "")
|
.replace("\n", "")
|
||||||
@@ -1277,9 +1280,10 @@ async def dashboard_metrics_stream(request: Request) -> StreamingResponse:
|
|||||||
|
|
||||||
@router.get("/dashboard/stream/overview")
|
@router.get("/dashboard/stream/overview")
|
||||||
async def dashboard_overview_stream(request: Request) -> StreamingResponse:
|
async def dashboard_overview_stream(request: Request) -> StreamingResponse:
|
||||||
|
overview = await _dashboard_overview(request)
|
||||||
fragment = (
|
fragment = (
|
||||||
templates.get_template("partials/overview.html")
|
templates.get_template("partials/overview.html")
|
||||||
.render(request=request, **_dashboard_overview(request))
|
.render(request=request, **overview)
|
||||||
.strip()
|
.strip()
|
||||||
.replace("\n", "")
|
.replace("\n", "")
|
||||||
)
|
)
|
||||||
@@ -1316,7 +1320,7 @@ async def dashboard_api_pairings(
|
|||||||
) -> JSONResponse:
|
) -> JSONResponse:
|
||||||
"""List pairings with optional filters."""
|
"""List pairings with optional filters."""
|
||||||
repo = _pairing_repo(request)
|
repo = _pairing_repo(request)
|
||||||
pairings = repo.list_pairings()
|
pairings = await repo.list_pairings()
|
||||||
|
|
||||||
# Apply filters
|
# Apply filters
|
||||||
if search:
|
if search:
|
||||||
@@ -1332,9 +1336,11 @@ async def dashboard_api_pairings(
|
|||||||
if source:
|
if source:
|
||||||
pairings = [p for p in pairings if p.source.lower() == source.lower()]
|
pairings = [p for p in pairings if p.source.lower() == source.lower()]
|
||||||
if base:
|
if base:
|
||||||
pairings = [p for p in pairings if p.base_asset.lower() == base.lower()]
|
pairings = [p for p in pairings if p.base_asset.lower() ==
|
||||||
|
base.lower()]
|
||||||
if quote:
|
if quote:
|
||||||
pairings = [p for p in pairings if p.quote_asset.lower() == quote.lower()]
|
pairings = [p for p in pairings if p.quote_asset.lower() ==
|
||||||
|
quote.lower()]
|
||||||
|
|
||||||
# Sort
|
# Sort
|
||||||
reverse = order.lower() == "desc"
|
reverse = order.lower() == "desc"
|
||||||
@@ -1370,7 +1376,7 @@ async def dashboard_pairings_fragment(
|
|||||||
) -> HTMLResponse:
|
) -> HTMLResponse:
|
||||||
"""HTMX fragment: pairing table for config page."""
|
"""HTMX fragment: pairing table for config page."""
|
||||||
repo = _pairing_repo(request)
|
repo = _pairing_repo(request)
|
||||||
pairings = repo.list_pairings()
|
pairings = await repo.list_pairings()
|
||||||
|
|
||||||
# Apply search filter
|
# Apply search filter
|
||||||
if search:
|
if search:
|
||||||
@@ -1406,7 +1412,7 @@ async def dashboard_api_pairings_toggle(request: Request) -> HTMLResponse:
|
|||||||
return HTMLResponse("Missing base_asset or quote_asset", status_code=400)
|
return HTMLResponse("Missing base_asset or quote_asset", status_code=400)
|
||||||
|
|
||||||
repo = _pairing_repo(request)
|
repo = _pairing_repo(request)
|
||||||
existing = repo.get_pairing(base_asset, quote_asset)
|
existing = await repo.get_pairing(base_asset, quote_asset)
|
||||||
if existing is None:
|
if existing is None:
|
||||||
return HTMLResponse("Pairing not found", status_code=404)
|
return HTMLResponse("Pairing not found", status_code=404)
|
||||||
|
|
||||||
@@ -1416,9 +1422,9 @@ async def dashboard_api_pairings_toggle(request: Request) -> HTMLResponse:
|
|||||||
enabled=not existing.enabled,
|
enabled=not existing.enabled,
|
||||||
source=existing.source,
|
source=existing.source,
|
||||||
)
|
)
|
||||||
repo.update_pairing(base_asset, quote_asset, toggled)
|
await repo.update_pairing(base_asset, quote_asset, toggled)
|
||||||
|
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.pairings.toggle",
|
event_type="dashboard.pairings.toggle",
|
||||||
@@ -1432,7 +1438,7 @@ async def dashboard_api_pairings_toggle(request: Request) -> HTMLResponse:
|
|||||||
|
|
||||||
# Return refreshed fragment
|
# Return refreshed fragment
|
||||||
pairings_repo = _pairing_repo(request)
|
pairings_repo = _pairing_repo(request)
|
||||||
pairings = pairings_repo.list_pairings()
|
pairings = await pairings_repo.list_pairings()
|
||||||
pairings.sort(key=lambda p: (p.base_asset, p.quote_asset))
|
pairings.sort(key=lambda p: (p.base_asset, p.quote_asset))
|
||||||
return templates.TemplateResponse(
|
return templates.TemplateResponse(
|
||||||
request=request,
|
request=request,
|
||||||
@@ -1448,7 +1454,7 @@ async def dashboard_api_pairings_sync(request: Request) -> JSONResponse:
|
|||||||
store = request.app.state.store
|
store = request.app.state.store
|
||||||
summary = await sync_pairings_from_kraken(kraken_client, store)
|
summary = await sync_pairings_from_kraken(kraken_client, store)
|
||||||
|
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
request,
|
request,
|
||||||
actor="dashboard_user",
|
actor="dashboard_user",
|
||||||
event_type="dashboard.pairings.sync",
|
event_type="dashboard.pairings.sync",
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ from arbitrade.execution.sequencer import TriangularExecutionSequencer
|
|||||||
from arbitrade.market_data.order_book import OrderBook
|
from arbitrade.market_data.order_book import OrderBook
|
||||||
from arbitrade.risk.pre_trade import PreTradeValidator
|
from arbitrade.risk.pre_trade import PreTradeValidator
|
||||||
from arbitrade.risk.trade_limits import TradeLimitsGuard
|
from arbitrade.risk.trade_limits import TradeLimitsGuard
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
@dataclass(slots=True)
|
||||||
@@ -185,8 +186,8 @@ def load_replay_events(path: Path) -> list[ReplayBookEvent]:
|
|||||||
return sorted(events, key=lambda event: event.occurred_at)
|
return sorted(events, key=lambda event: event.occurred_at)
|
||||||
|
|
||||||
|
|
||||||
def load_replay_events_from_db(
|
async def load_replay_events_from_db(
|
||||||
store: object,
|
store: PgStore,
|
||||||
*,
|
*,
|
||||||
symbols: list[str] | None = None,
|
symbols: list[str] | None = None,
|
||||||
start: datetime | None = None,
|
start: datetime | None = None,
|
||||||
@@ -197,32 +198,32 @@ def load_replay_events_from_db(
|
|||||||
Each market_snapshots row has snapshot_at, symbol, payload (raw Kraken WS).
|
Each market_snapshots row has snapshot_at, symbol, payload (raw Kraken WS).
|
||||||
Payload format: {channel, symbol, data: [{bids: [{price, qty}], asks: [{price, qty}]}]}
|
Payload format: {channel, symbol, data: [{bids: [{price, qty}], asks: [{price, qty}]}]}
|
||||||
"""
|
"""
|
||||||
with store.connect() as conn: # type: ignore
|
async with store.pool.acquire() as conn:
|
||||||
query = "SELECT snapshot_at, symbol, payload FROM market_snapshots WHERE 1=1"
|
query = "SELECT snapshot_at, symbol, payload FROM market_snapshots WHERE 1=1"
|
||||||
params: list[object] = []
|
params: list[object] = []
|
||||||
|
|
||||||
if symbols:
|
if symbols:
|
||||||
placeholders = ",".join("?" for _ in symbols)
|
placeholders = ",".join(f"${i+1}" for i in range(len(symbols)))
|
||||||
query += f" AND symbol IN ({placeholders})"
|
query += f" AND symbol IN ({placeholders})"
|
||||||
params.extend(symbols)
|
params.extend(symbols)
|
||||||
|
|
||||||
if start is not None:
|
if start is not None:
|
||||||
query += " AND snapshot_at >= ?"
|
|
||||||
params.append(start)
|
params.append(start)
|
||||||
|
query += f" AND snapshot_at >= ${len(params)}"
|
||||||
|
|
||||||
if end is not None:
|
if end is not None:
|
||||||
query += " AND snapshot_at <= ?"
|
|
||||||
params.append(end)
|
params.append(end)
|
||||||
|
query += f" AND snapshot_at <= ${len(params)}"
|
||||||
|
|
||||||
query += " ORDER BY snapshot_at ASC"
|
query += " ORDER BY snapshot_at ASC"
|
||||||
|
|
||||||
rows = conn.execute(query, params).fetchall()
|
rows = await conn.fetch(query, *params)
|
||||||
|
|
||||||
events: list[ReplayBookEvent] = []
|
events: list[ReplayBookEvent] = []
|
||||||
for row in rows:
|
for row in rows:
|
||||||
snapshot_at: datetime = row[0]
|
snapshot_at: datetime = row["snapshot_at"]
|
||||||
symbol: str = row[1]
|
symbol: str = row["symbol"]
|
||||||
payload_raw = row[2]
|
payload_raw = row["payload"]
|
||||||
|
|
||||||
if isinstance(payload_raw, str):
|
if isinstance(payload_raw, str):
|
||||||
payload = orjson.loads(payload_raw)
|
payload = orjson.loads(payload_raw)
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ from arbitrade.backtesting.replay import (
|
|||||||
load_replay_events_from_db,
|
load_replay_events_from_db,
|
||||||
)
|
)
|
||||||
from arbitrade.detection.graph import CurrencyGraph, TriangularCycle
|
from arbitrade.detection.graph import CurrencyGraph, TriangularCycle
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import BacktestJobRepository
|
from arbitrade.storage.repositories import BacktestJobRepository
|
||||||
|
|
||||||
_LOG = structlog.get_logger(__name__)
|
_LOG = structlog.get_logger(__name__)
|
||||||
@@ -50,11 +50,11 @@ def _parse_balances(raw: str) -> dict[str, float]:
|
|||||||
async def run_backtest_job(
|
async def run_backtest_job(
|
||||||
job_id: str,
|
job_id: str,
|
||||||
config_dict: dict[str, object] | None,
|
config_dict: dict[str, object] | None,
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Execute a single backtest job: load events from DB or file, run engine, store report."""
|
"""Execute a single backtest job: load events from DB or file, run engine, store report."""
|
||||||
repo = BacktestJobRepository(store)
|
repo = BacktestJobRepository(store)
|
||||||
repo.update_status(job_id, "running")
|
await repo.update_status(job_id, "running")
|
||||||
_LOG.info("backtest_job_started", job_id=job_id)
|
_LOG.info("backtest_job_started", job_id=job_id)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
@@ -79,7 +79,7 @@ async def run_backtest_job(
|
|||||||
elif isinstance(symbols_raw, list):
|
elif isinstance(symbols_raw, list):
|
||||||
symbols = [str(s).upper() for s in symbols_raw]
|
symbols = [str(s).upper() for s in symbols_raw]
|
||||||
|
|
||||||
events = load_replay_events_from_db(
|
events = await load_replay_events_from_db(
|
||||||
store,
|
store,
|
||||||
symbols=symbols,
|
symbols=symbols,
|
||||||
start=start_dt,
|
start=start_dt,
|
||||||
@@ -141,18 +141,18 @@ async def run_backtest_job(
|
|||||||
"finished_at": report.finished_at.isoformat(),
|
"finished_at": report.finished_at.isoformat(),
|
||||||
}
|
}
|
||||||
|
|
||||||
repo.store_report(job_id, report_dict)
|
await repo.store_report(job_id, report_dict)
|
||||||
repo.update_status(job_id, "completed")
|
await repo.update_status(job_id, "completed")
|
||||||
_LOG.info("backtest_job_completed", job_id=job_id, pnl=report.realized_pnl_usd)
|
_LOG.info("backtest_job_completed", job_id=job_id, pnl=report.realized_pnl_usd)
|
||||||
|
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
repo.update_status(job_id, "failed", error=str(exc))
|
await repo.update_status(job_id, "failed", error=str(exc))
|
||||||
_LOG.exception("backtest_job_failed", job_id=job_id, error=str(exc))
|
_LOG.exception("backtest_job_failed", job_id=job_id, error=str(exc))
|
||||||
|
|
||||||
|
|
||||||
async def backtest_worker(
|
async def backtest_worker(
|
||||||
queue: asyncio.Queue[tuple[str, dict[str, object] | None] | None],
|
queue: asyncio.Queue[tuple[str, dict[str, object] | None] | None],
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Worker coroutine: pull jobs from queue and execute them one at a time."""
|
"""Worker coroutine: pull jobs from queue and execute them one at a time."""
|
||||||
_LOG.info("backtest_worker_started")
|
_LOG.info("backtest_worker_started")
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import structlog
|
|||||||
from arbitrade.config.service import ConfigPairing
|
from arbitrade.config.service import ConfigPairing
|
||||||
from arbitrade.detection.graph import CurrencyGraph
|
from arbitrade.detection.graph import CurrencyGraph
|
||||||
from arbitrade.exchange.kraken_rest import KrakenRestClient
|
from arbitrade.exchange.kraken_rest import KrakenRestClient
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import ConfigPairingRepository
|
from arbitrade.storage.repositories import ConfigPairingRepository
|
||||||
|
|
||||||
_LOG = structlog.get_logger(__name__)
|
_LOG = structlog.get_logger(__name__)
|
||||||
@@ -17,7 +17,7 @@ _LOG = structlog.get_logger(__name__)
|
|||||||
|
|
||||||
async def sync_pairings_from_kraken(
|
async def sync_pairings_from_kraken(
|
||||||
kraken_client: KrakenRestClient,
|
kraken_client: KrakenRestClient,
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
) -> dict[str, int]:
|
) -> dict[str, int]:
|
||||||
"""Fetch all asset pairs from Kraken and upsert into config_pairings.
|
"""Fetch all asset pairs from Kraken and upsert into config_pairings.
|
||||||
|
|
||||||
@@ -37,7 +37,7 @@ async def sync_pairings_from_kraken(
|
|||||||
if symbol in seen_symbols:
|
if symbol in seen_symbols:
|
||||||
continue
|
continue
|
||||||
seen_symbols.add(symbol)
|
seen_symbols.add(symbol)
|
||||||
existing = repo.get_pairing(base, quote)
|
existing = await repo.get_pairing(base, quote)
|
||||||
pairing = ConfigPairing(
|
pairing = ConfigPairing(
|
||||||
base_asset=base,
|
base_asset=base,
|
||||||
quote_asset=quote,
|
quote_asset=quote,
|
||||||
@@ -45,7 +45,7 @@ async def sync_pairings_from_kraken(
|
|||||||
source="kraken",
|
source="kraken",
|
||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
repo.upsert_pairing(pairing)
|
await repo.upsert_pairing(pairing)
|
||||||
total += 1
|
total += 1
|
||||||
if existing:
|
if existing:
|
||||||
updated += 1
|
updated += 1
|
||||||
@@ -65,7 +65,7 @@ async def sync_pairings_from_kraken(
|
|||||||
|
|
||||||
async def run_pairing_sync_loop(
|
async def run_pairing_sync_loop(
|
||||||
kraken_client: KrakenRestClient,
|
kraken_client: KrakenRestClient,
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
stop_event: asyncio.Event,
|
stop_event: asyncio.Event,
|
||||||
interval_seconds: int = 86400,
|
interval_seconds: int = 86400,
|
||||||
) -> None:
|
) -> None:
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ import orjson
|
|||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
from arbitrade.config.settings import Settings
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
|
||||||
|
|
||||||
class ConfigSection(BaseModel):
|
class ConfigSection(BaseModel):
|
||||||
@@ -49,16 +49,15 @@ class ConfigBacktestingDefaults(BaseModel):
|
|||||||
class ConfigurationService:
|
class ConfigurationService:
|
||||||
"""Manages application configuration from environment and database sources."""
|
"""Manages application configuration from environment and database sources."""
|
||||||
|
|
||||||
def __init__(self, settings: Settings, store: DuckDBStore, audit_repo: Any) -> None:
|
def __init__(self, settings: Settings, store: PgStore, audit_repo: Any) -> None:
|
||||||
self._settings = settings
|
self._settings = settings
|
||||||
self._store = store
|
self._store = store
|
||||||
self._audit_repo = audit_repo
|
self._audit_repo = audit_repo
|
||||||
self._config_version = 0
|
self._config_version = 0
|
||||||
self._loaded_settings: dict[str, Any] = {}
|
self._loaded_settings: dict[str, Any] = {}
|
||||||
self._last_updated_at: datetime | None = None
|
self._last_updated_at: datetime | None = None
|
||||||
self._load_database_settings()
|
|
||||||
|
|
||||||
def _load_database_settings(self) -> None:
|
async def load_database_settings(self) -> None:
|
||||||
"""Load user settings from database and merge with defaults."""
|
"""Load user settings from database and merge with defaults."""
|
||||||
# Import here to avoid circular imports
|
# Import here to avoid circular imports
|
||||||
from arbitrade.storage.repositories import ConfigSettingRepository
|
from arbitrade.storage.repositories import ConfigSettingRepository
|
||||||
@@ -66,7 +65,7 @@ class ConfigurationService:
|
|||||||
setting_repo = ConfigSettingRepository(self._store)
|
setting_repo = ConfigSettingRepository(self._store)
|
||||||
|
|
||||||
# Load all settings from database
|
# Load all settings from database
|
||||||
db_settings = setting_repo.list_settings()
|
db_settings = await setting_repo.list_settings()
|
||||||
|
|
||||||
# Convert to dictionary for easy access
|
# Convert to dictionary for easy access
|
||||||
for setting in db_settings:
|
for setting in db_settings:
|
||||||
@@ -116,7 +115,7 @@ class ConfigurationService:
|
|||||||
"""Get the timestamp of the last configuration update."""
|
"""Get the timestamp of the last configuration update."""
|
||||||
return self._last_updated_at
|
return self._last_updated_at
|
||||||
|
|
||||||
def is_config_outdated(self) -> bool:
|
async def is_config_outdated(self) -> bool:
|
||||||
"""Check if configuration has been updated since last load."""
|
"""Check if configuration has been updated since last load."""
|
||||||
# Import here to avoid circular imports
|
# Import here to avoid circular imports
|
||||||
from arbitrade.storage.repositories import ConfigSettingRepository
|
from arbitrade.storage.repositories import ConfigSettingRepository
|
||||||
@@ -124,7 +123,7 @@ class ConfigurationService:
|
|||||||
setting_repo = ConfigSettingRepository(self._store)
|
setting_repo = ConfigSettingRepository(self._store)
|
||||||
|
|
||||||
# Get the latest update timestamp from database
|
# Get the latest update timestamp from database
|
||||||
latest_db_update = setting_repo.get_latest_updated_at()
|
latest_db_update = await setting_repo.get_latest_updated_at()
|
||||||
|
|
||||||
# Compare with our last loaded timestamp
|
# Compare with our last loaded timestamp
|
||||||
if latest_db_update and self._last_updated_at:
|
if latest_db_update and self._last_updated_at:
|
||||||
@@ -133,15 +132,15 @@ class ConfigurationService:
|
|||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def reload_if_changed(self) -> bool:
|
async def reload_if_changed(self) -> bool:
|
||||||
"""Reload configuration if it has been updated in the database."""
|
"""Reload configuration if it has been updated in the database."""
|
||||||
if self.is_config_outdated():
|
if await self.is_config_outdated():
|
||||||
self._load_database_settings()
|
await self.load_database_settings()
|
||||||
self._config_version += 1
|
self._config_version += 1
|
||||||
return True
|
return True
|
||||||
return False
|
return False
|
||||||
|
|
||||||
def set_setting(self, key: str, value: Any, updated_by: str | None = None) -> None:
|
async def set_setting(self, key: str, value: Any, updated_by: str | None = None) -> None:
|
||||||
"""Set a configuration setting value and persist to database."""
|
"""Set a configuration setting value and persist to database."""
|
||||||
# Import here to avoid circular imports
|
# Import here to avoid circular imports
|
||||||
from arbitrade.storage.repositories import ConfigSettingRepository
|
from arbitrade.storage.repositories import ConfigSettingRepository
|
||||||
@@ -183,13 +182,13 @@ class ConfigurationService:
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Check if setting exists
|
# Check if setting exists
|
||||||
existing_setting = setting_repo.get_setting(key)
|
existing_setting = await setting_repo.get_setting(key)
|
||||||
if existing_setting:
|
if existing_setting:
|
||||||
# Update existing setting
|
# Update existing setting
|
||||||
updated_setting = setting_repo.update_setting(key, setting)
|
updated_setting = await setting_repo.update_setting(key, setting)
|
||||||
else:
|
else:
|
||||||
# Create new setting
|
# Create new setting
|
||||||
updated_setting = setting_repo.create_setting(setting)
|
updated_setting = await setting_repo.create_setting(setting)
|
||||||
|
|
||||||
# Update in-memory cache
|
# Update in-memory cache
|
||||||
self._loaded_settings[key] = value
|
self._loaded_settings[key] = value
|
||||||
@@ -211,18 +210,18 @@ class ConfigurationService:
|
|||||||
|
|
||||||
return ConfigPairingRepository(self._store)
|
return ConfigPairingRepository(self._store)
|
||||||
|
|
||||||
def list_pairings(self) -> list[ConfigPairing]:
|
async def list_pairings(self) -> list[ConfigPairing]:
|
||||||
"""List all currency pairings."""
|
"""List all currency pairings."""
|
||||||
r = self._pairing_repo() # type: ignore[no-untyped-call]
|
r = self._pairing_repo() # type: ignore[no-untyped-call]
|
||||||
p = r.list_pairings()
|
p = await r.list_pairings()
|
||||||
return p # type: ignore[no-any-return]
|
return p # type: ignore[no-any-return]
|
||||||
|
|
||||||
def create_pairing(
|
async def create_pairing(
|
||||||
self, base_asset: str, quote_asset: str, source: str = "manual"
|
self, base_asset: str, quote_asset: str, source: str = "manual"
|
||||||
) -> ConfigPairing:
|
) -> ConfigPairing:
|
||||||
"""Create a new currency pairing."""
|
"""Create a new currency pairing."""
|
||||||
r = self._pairing_repo() # type: ignore[no-untyped-call]
|
r = self._pairing_repo() # type: ignore[no-untyped-call]
|
||||||
e = r.get_pairing(base_asset, quote_asset)
|
e = await r.get_pairing(base_asset, quote_asset)
|
||||||
if e:
|
if e:
|
||||||
return e # type: ignore[no-any-return]
|
return e # type: ignore[no-any-return]
|
||||||
pairing = ConfigPairing(
|
pairing = ConfigPairing(
|
||||||
|
|||||||
@@ -1,7 +1,6 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from pydantic import Field, field_validator, model_validator
|
from pydantic import Field, field_validator, model_validator
|
||||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||||
@@ -32,49 +31,78 @@ class Settings(BaseSettings):
|
|||||||
)
|
)
|
||||||
|
|
||||||
alerts_enabled: bool = Field(default=True, alias="ALERTS_ENABLED")
|
alerts_enabled: bool = Field(default=True, alias="ALERTS_ENABLED")
|
||||||
alert_min_severity: str = Field(default="warning", alias="ALERT_MIN_SEVERITY")
|
alert_min_severity: str = Field(
|
||||||
alert_dedup_seconds: float = Field(default=30.0, alias="ALERT_DEDUP_SECONDS")
|
default="warning", alias="ALERT_MIN_SEVERITY")
|
||||||
alert_on_trade_events: bool = Field(default=True, alias="ALERT_ON_TRADE_EVENTS")
|
alert_dedup_seconds: float = Field(
|
||||||
alert_on_error_events: bool = Field(default=True, alias="ALERT_ON_ERROR_EVENTS")
|
default=30.0, alias="ALERT_DEDUP_SECONDS")
|
||||||
alert_on_threshold_events: bool = Field(default=True, alias="ALERT_ON_THRESHOLD_EVENTS")
|
alert_on_trade_events: bool = Field(
|
||||||
alert_on_system_events: bool = Field(default=True, alias="ALERT_ON_SYSTEM_EVENTS")
|
default=True, alias="ALERT_ON_TRADE_EVENTS")
|
||||||
|
alert_on_error_events: bool = Field(
|
||||||
|
default=True, alias="ALERT_ON_ERROR_EVENTS")
|
||||||
|
alert_on_threshold_events: bool = Field(
|
||||||
|
default=True, alias="ALERT_ON_THRESHOLD_EVENTS")
|
||||||
|
alert_on_system_events: bool = Field(
|
||||||
|
default=True, alias="ALERT_ON_SYSTEM_EVENTS")
|
||||||
|
|
||||||
telegram_alerts_enabled: bool = Field(default=False, alias="TELEGRAM_ALERTS_ENABLED")
|
telegram_alerts_enabled: bool = Field(
|
||||||
telegram_bot_token: str | None = Field(default=None, alias="TELEGRAM_BOT_TOKEN")
|
default=False, alias="TELEGRAM_ALERTS_ENABLED")
|
||||||
telegram_chat_id: str | None = Field(default=None, alias="TELEGRAM_CHAT_ID")
|
telegram_bot_token: str | None = Field(
|
||||||
|
default=None, alias="TELEGRAM_BOT_TOKEN")
|
||||||
|
telegram_chat_id: str | None = Field(
|
||||||
|
default=None, alias="TELEGRAM_CHAT_ID")
|
||||||
|
|
||||||
discord_alerts_enabled: bool = Field(default=False, alias="DISCORD_ALERTS_ENABLED")
|
discord_alerts_enabled: bool = Field(
|
||||||
discord_webhook_url: str | None = Field(default=None, alias="DISCORD_WEBHOOK_URL")
|
default=False, alias="DISCORD_ALERTS_ENABLED")
|
||||||
|
discord_webhook_url: str | None = Field(
|
||||||
|
default=None, alias="DISCORD_WEBHOOK_URL")
|
||||||
|
|
||||||
email_alerts_enabled: bool = Field(default=False, alias="EMAIL_ALERTS_ENABLED")
|
email_alerts_enabled: bool = Field(
|
||||||
|
default=False, alias="EMAIL_ALERTS_ENABLED")
|
||||||
email_smtp_host: str | None = Field(default=None, alias="EMAIL_SMTP_HOST")
|
email_smtp_host: str | None = Field(default=None, alias="EMAIL_SMTP_HOST")
|
||||||
email_smtp_port: int = Field(default=587, alias="EMAIL_SMTP_PORT")
|
email_smtp_port: int = Field(default=587, alias="EMAIL_SMTP_PORT")
|
||||||
email_smtp_username: str | None = Field(default=None, alias="EMAIL_SMTP_USERNAME")
|
email_smtp_username: str | None = Field(
|
||||||
email_smtp_password: str | None = Field(default=None, alias="EMAIL_SMTP_PASSWORD")
|
default=None, alias="EMAIL_SMTP_USERNAME")
|
||||||
email_alert_from: str | None = Field(default=None, alias="EMAIL_ALERT_FROM")
|
email_smtp_password: str | None = Field(
|
||||||
|
default=None, alias="EMAIL_SMTP_PASSWORD")
|
||||||
|
email_alert_from: str | None = Field(
|
||||||
|
default=None, alias="EMAIL_ALERT_FROM")
|
||||||
email_alert_to: str | None = Field(default=None, alias="EMAIL_ALERT_TO")
|
email_alert_to: str | None = Field(default=None, alias="EMAIL_ALERT_TO")
|
||||||
email_smtp_use_tls: bool = Field(default=True, alias="EMAIL_SMTP_USE_TLS")
|
email_smtp_use_tls: bool = Field(default=True, alias="EMAIL_SMTP_USE_TLS")
|
||||||
|
|
||||||
duckdb_path: Path = Field(default=Path("./data/arbitrade.duckdb"), alias="DUCKDB_PATH")
|
# PostgreSQL connection settings
|
||||||
|
pg_host: str = Field(default="192.168.88.35", alias="PG_HOST")
|
||||||
|
pg_port: int = Field(default=5432, alias="PG_PORT")
|
||||||
|
pg_database: str = Field(default="arbitrade", alias="PG_DATABASE")
|
||||||
|
pg_user: str = Field(default="arbitrade", alias="PG_USER")
|
||||||
|
pg_password: str = Field(default="arbitrade", alias="PG_PASSWORD")
|
||||||
|
pg_min_connections: int = Field(default=2, alias="PG_MIN_CONNECTIONS")
|
||||||
|
pg_max_connections: int = Field(default=10, alias="PG_MAX_CONNECTIONS")
|
||||||
|
|
||||||
kraken_rest_url: str = Field(default="https://api.kraken.com", alias="KRAKEN_REST_URL")
|
kraken_rest_url: str = Field(
|
||||||
kraken_ws_url: str = Field(default="wss://ws.kraken.com/v2", alias="KRAKEN_WS_URL")
|
default="https://api.kraken.com", alias="KRAKEN_REST_URL")
|
||||||
|
kraken_ws_url: str = Field(
|
||||||
|
default="wss://ws.kraken.com/v2", alias="KRAKEN_WS_URL")
|
||||||
kraken_private_rate_limit_seconds: float = Field(
|
kraken_private_rate_limit_seconds: float = Field(
|
||||||
default=1.0, alias="KRAKEN_PRIVATE_RATE_LIMIT_SECONDS"
|
default=1.0, alias="KRAKEN_PRIVATE_RATE_LIMIT_SECONDS"
|
||||||
)
|
)
|
||||||
kraken_http_timeout_seconds: float = Field(default=10.0, alias="KRAKEN_HTTP_TIMEOUT_SECONDS")
|
kraken_http_timeout_seconds: float = Field(
|
||||||
kraken_retry_attempts: int = Field(default=3, alias="KRAKEN_RETRY_ATTEMPTS")
|
default=10.0, alias="KRAKEN_HTTP_TIMEOUT_SECONDS")
|
||||||
|
kraken_retry_attempts: int = Field(
|
||||||
|
default=3, alias="KRAKEN_RETRY_ATTEMPTS")
|
||||||
kraken_retry_base_delay_seconds: float = Field(
|
kraken_retry_base_delay_seconds: float = Field(
|
||||||
default=0.25, alias="KRAKEN_RETRY_BASE_DELAY_SECONDS"
|
default=0.25, alias="KRAKEN_RETRY_BASE_DELAY_SECONDS"
|
||||||
)
|
)
|
||||||
kraken_api_key: str | None = Field(default=None, alias="KRAKEN_API_KEY")
|
kraken_api_key: str | None = Field(default=None, alias="KRAKEN_API_KEY")
|
||||||
kraken_api_secret: str | None = Field(default=None, alias="KRAKEN_API_SECRET")
|
kraken_api_secret: str | None = Field(
|
||||||
|
default=None, alias="KRAKEN_API_SECRET")
|
||||||
kraken_api_key_permissions: str = Field(
|
kraken_api_key_permissions: str = Field(
|
||||||
default="query,trade",
|
default="query,trade",
|
||||||
alias="KRAKEN_API_KEY_PERMISSIONS",
|
alias="KRAKEN_API_KEY_PERMISSIONS",
|
||||||
)
|
)
|
||||||
ws_heartbeat_timeout_seconds: float = Field(default=20.0, alias="WS_HEARTBEAT_TIMEOUT_SECONDS")
|
ws_heartbeat_timeout_seconds: float = Field(
|
||||||
ws_max_staleness_seconds: float = Field(default=5.0, alias="WS_MAX_STALENESS_SECONDS")
|
default=20.0, alias="WS_HEARTBEAT_TIMEOUT_SECONDS")
|
||||||
|
ws_max_staleness_seconds: float = Field(
|
||||||
|
default=5.0, alias="WS_MAX_STALENESS_SECONDS")
|
||||||
strategy_enable_stat_arb_experiment: bool = Field(
|
strategy_enable_stat_arb_experiment: bool = Field(
|
||||||
default=False,
|
default=False,
|
||||||
alias="STRATEGY_ENABLE_STAT_ARB_EXPERIMENT",
|
alias="STRATEGY_ENABLE_STAT_ARB_EXPERIMENT",
|
||||||
@@ -97,20 +125,29 @@ class Settings(BaseSettings):
|
|||||||
)
|
)
|
||||||
paper_trading_mode: bool = Field(default=True, alias="PAPER_TRADING_MODE")
|
paper_trading_mode: bool = Field(default=True, alias="PAPER_TRADING_MODE")
|
||||||
trade_capital_usd: float = Field(default=100.0, alias="TRADE_CAPITAL_USD")
|
trade_capital_usd: float = Field(default=100.0, alias="TRADE_CAPITAL_USD")
|
||||||
max_trade_capital_usd: float = Field(default=100.0, alias="MAX_TRADE_CAPITAL_USD")
|
max_trade_capital_usd: float = Field(
|
||||||
max_concurrent_trades: int | None = Field(default=None, alias="MAX_CONCURRENT_TRADES")
|
default=100.0, alias="MAX_TRADE_CAPITAL_USD")
|
||||||
|
max_concurrent_trades: int | None = Field(
|
||||||
|
default=None, alias="MAX_CONCURRENT_TRADES")
|
||||||
max_exposure_per_asset_usd: float | None = Field(
|
max_exposure_per_asset_usd: float | None = Field(
|
||||||
default=None,
|
default=None,
|
||||||
alias="MAX_EXPOSURE_PER_ASSET_USD",
|
alias="MAX_EXPOSURE_PER_ASSET_USD",
|
||||||
)
|
)
|
||||||
quote_balance_asset: str = Field(default="USD", alias="QUOTE_BALANCE_ASSET")
|
quote_balance_asset: str = Field(
|
||||||
min_order_size_usd: float | None = Field(default=None, alias="MIN_ORDER_SIZE_USD")
|
default="USD", alias="QUOTE_BALANCE_ASSET")
|
||||||
|
min_order_size_usd: float | None = Field(
|
||||||
|
default=None, alias="MIN_ORDER_SIZE_USD")
|
||||||
kill_switch_active: bool = Field(default=False, alias="KILL_SWITCH_ACTIVE")
|
kill_switch_active: bool = Field(default=False, alias="KILL_SWITCH_ACTIVE")
|
||||||
daily_loss_limit_usd: float | None = Field(default=None, alias="DAILY_LOSS_LIMIT_USD")
|
daily_loss_limit_usd: float | None = Field(
|
||||||
cumulative_loss_limit_usd: float | None = Field(default=None, alias="CUMULATIVE_LOSS_LIMIT_USD")
|
default=None, alias="DAILY_LOSS_LIMIT_USD")
|
||||||
max_source_latency_ms: float | None = Field(default=None, alias="MAX_SOURCE_LATENCY_MS")
|
cumulative_loss_limit_usd: float | None = Field(
|
||||||
max_apply_latency_ms: float | None = Field(default=None, alias="MAX_APPLY_LATENCY_MS")
|
default=None, alias="CUMULATIVE_LOSS_LIMIT_USD")
|
||||||
max_consecutive_failures: int | None = Field(default=None, alias="MAX_CONSECUTIVE_FAILURES")
|
max_source_latency_ms: float | None = Field(
|
||||||
|
default=None, alias="MAX_SOURCE_LATENCY_MS")
|
||||||
|
max_apply_latency_ms: float | None = Field(
|
||||||
|
default=None, alias="MAX_APPLY_LATENCY_MS")
|
||||||
|
max_consecutive_failures: int | None = Field(
|
||||||
|
default=None, alias="MAX_CONSECUTIVE_FAILURES")
|
||||||
|
|
||||||
fernet_key: str | None = Field(default=None, alias="FERNET_KEY")
|
fernet_key: str | None = Field(default=None, alias="FERNET_KEY")
|
||||||
|
|
||||||
@@ -127,7 +164,8 @@ class Settings(BaseSettings):
|
|||||||
def _validate_log_level(cls, value: str) -> str:
|
def _validate_log_level(cls, value: str) -> str:
|
||||||
normalized = value.strip().upper()
|
normalized = value.strip().upper()
|
||||||
if normalized not in {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}:
|
if normalized not in {"DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"}:
|
||||||
raise ValueError("LOG_LEVEL must be one of: DEBUG, INFO, WARNING, ERROR, CRITICAL")
|
raise ValueError(
|
||||||
|
"LOG_LEVEL must be one of: DEBUG, INFO, WARNING, ERROR, CRITICAL")
|
||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
@field_validator("alert_min_severity")
|
@field_validator("alert_min_severity")
|
||||||
@@ -135,16 +173,19 @@ class Settings(BaseSettings):
|
|||||||
def _validate_alert_severity(cls, value: str) -> str:
|
def _validate_alert_severity(cls, value: str) -> str:
|
||||||
normalized = value.strip().lower()
|
normalized = value.strip().lower()
|
||||||
if normalized not in {"info", "warning", "error", "critical"}:
|
if normalized not in {"info", "warning", "error", "critical"}:
|
||||||
raise ValueError("ALERT_MIN_SEVERITY must be one of: info, warning, error, critical")
|
raise ValueError(
|
||||||
|
"ALERT_MIN_SEVERITY must be one of: info, warning, error, critical")
|
||||||
return normalized
|
return normalized
|
||||||
|
|
||||||
@model_validator(mode="after")
|
@model_validator(mode="after")
|
||||||
def _validate_security_constraints(self) -> Settings:
|
def _validate_security_constraints(self) -> Settings:
|
||||||
if bool(self.dashboard_auth_username) ^ bool(self.dashboard_auth_password):
|
if bool(self.dashboard_auth_username) ^ bool(self.dashboard_auth_password):
|
||||||
raise ValueError("dashboard auth requires both username and password")
|
raise ValueError(
|
||||||
|
"dashboard auth requires both username and password")
|
||||||
|
|
||||||
if bool(self.kraken_api_key) ^ bool(self.kraken_api_secret):
|
if bool(self.kraken_api_key) ^ bool(self.kraken_api_secret):
|
||||||
raise ValueError("Kraken API auth requires both API key and secret")
|
raise ValueError(
|
||||||
|
"Kraken API auth requires both API key and secret")
|
||||||
|
|
||||||
permissions = {
|
permissions = {
|
||||||
token.strip().lower()
|
token.strip().lower()
|
||||||
@@ -152,9 +193,11 @@ class Settings(BaseSettings):
|
|||||||
if token.strip()
|
if token.strip()
|
||||||
}
|
}
|
||||||
if permissions and ("query" not in permissions or "trade" not in permissions):
|
if permissions and ("query" not in permissions or "trade" not in permissions):
|
||||||
raise ValueError("KRAKEN_API_KEY_PERMISSIONS must include query and trade")
|
raise ValueError(
|
||||||
|
"KRAKEN_API_KEY_PERMISSIONS must include query and trade")
|
||||||
if "withdraw" in permissions or "withdrawals" in permissions:
|
if "withdraw" in permissions or "withdrawals" in permissions:
|
||||||
raise ValueError("KRAKEN_API_KEY_PERMISSIONS must not include withdrawal scope")
|
raise ValueError(
|
||||||
|
"KRAKEN_API_KEY_PERMISSIONS must not include withdrawal scope")
|
||||||
|
|
||||||
if self.alert_dedup_seconds < 0.0:
|
if self.alert_dedup_seconds < 0.0:
|
||||||
raise ValueError("ALERT_DEDUP_SECONDS must be >= 0")
|
raise ValueError("ALERT_DEDUP_SECONDS must be >= 0")
|
||||||
@@ -170,7 +213,8 @@ class Settings(BaseSettings):
|
|||||||
"STRATEGY_STAT_ARB_ENTRY_ZSCORE must be greater than STRATEGY_STAT_ARB_EXIT_ZSCORE"
|
"STRATEGY_STAT_ARB_ENTRY_ZSCORE must be greater than STRATEGY_STAT_ARB_EXIT_ZSCORE"
|
||||||
)
|
)
|
||||||
if self.strategy_stat_arb_max_holding_seconds <= 0.0:
|
if self.strategy_stat_arb_max_holding_seconds <= 0.0:
|
||||||
raise ValueError("STRATEGY_STAT_ARB_MAX_HOLDING_SECONDS must be > 0")
|
raise ValueError(
|
||||||
|
"STRATEGY_STAT_ARB_MAX_HOLDING_SECONDS must be > 0")
|
||||||
|
|
||||||
return self
|
return self
|
||||||
|
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ import orjson
|
|||||||
import structlog
|
import structlog
|
||||||
|
|
||||||
from arbitrade.exchange.kraken_rest import KrakenRestClient
|
from arbitrade.exchange.kraken_rest import KrakenRestClient
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import (
|
from arbitrade.storage.repositories import (
|
||||||
KrakenAccountSnapshot,
|
KrakenAccountSnapshot,
|
||||||
KrakenAccountSnapshotRepository,
|
KrakenAccountSnapshotRepository,
|
||||||
@@ -22,7 +22,7 @@ _FEE_REFRESH_INTERVAL_SECONDS = 86400 # 1 day
|
|||||||
|
|
||||||
async def fetch_and_store_account_snapshot(
|
async def fetch_and_store_account_snapshot(
|
||||||
client: KrakenRestClient,
|
client: KrakenRestClient,
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
) -> KrakenAccountSnapshot | None:
|
) -> KrakenAccountSnapshot | None:
|
||||||
"""Query TradeVolume + TradeBalance, persist as snapshot.
|
"""Query TradeVolume + TradeBalance, persist as snapshot.
|
||||||
|
|
||||||
@@ -42,9 +42,12 @@ async def fetch_and_store_account_snapshot(
|
|||||||
_LOG.exception("trade_balance_fetch_failed")
|
_LOG.exception("trade_balance_fetch_failed")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
fee_tier = volume_data.get("fee_tier") if isinstance(volume_data, dict) else None
|
fee_tier = volume_data.get("fee_tier") if isinstance(
|
||||||
fees_dict = volume_data.get("fees") if isinstance(volume_data, dict) else None
|
volume_data, dict) else None
|
||||||
fees_maker = volume_data.get("fees_maker") if isinstance(volume_data, dict) else None
|
fees_dict = volume_data.get("fees") if isinstance(
|
||||||
|
volume_data, dict) else None
|
||||||
|
fees_maker = volume_data.get("fees_maker") if isinstance(
|
||||||
|
volume_data, dict) else None
|
||||||
currency = volume_data.get("currency")
|
currency = volume_data.get("currency")
|
||||||
thirty_day_volume_str = volume_data.get("volume")
|
thirty_day_volume_str = volume_data.get("volume")
|
||||||
|
|
||||||
@@ -70,7 +73,8 @@ async def fetch_and_store_account_snapshot(
|
|||||||
if currency is not None:
|
if currency is not None:
|
||||||
fee_schedule["currency"] = currency
|
fee_schedule["currency"] = currency
|
||||||
|
|
||||||
thirty_day_volume = float(thirty_day_volume_str) if thirty_day_volume_str is not None else None
|
thirty_day_volume = float(
|
||||||
|
thirty_day_volume_str) if thirty_day_volume_str is not None else None
|
||||||
|
|
||||||
snapshot = KrakenAccountSnapshot(
|
snapshot = KrakenAccountSnapshot(
|
||||||
snapshot_at=datetime.now(UTC),
|
snapshot_at=datetime.now(UTC),
|
||||||
@@ -78,11 +82,12 @@ async def fetch_and_store_account_snapshot(
|
|||||||
maker_fee=maker_fee,
|
maker_fee=maker_fee,
|
||||||
taker_fee=taker_fee,
|
taker_fee=taker_fee,
|
||||||
thirty_day_volume=thirty_day_volume,
|
thirty_day_volume=thirty_day_volume,
|
||||||
trade_balance_raw=balance_data if isinstance(balance_data, dict) else None,
|
trade_balance_raw=balance_data if isinstance(
|
||||||
|
balance_data, dict) else None,
|
||||||
fee_schedule_raw=fee_schedule if fee_schedule else None,
|
fee_schedule_raw=fee_schedule if fee_schedule else None,
|
||||||
)
|
)
|
||||||
|
|
||||||
repo.insert_snapshot(snapshot)
|
await repo.insert_snapshot(snapshot)
|
||||||
_LOG.info(
|
_LOG.info(
|
||||||
"account_snapshot_stored",
|
"account_snapshot_stored",
|
||||||
fee_tier=fee_tier_str,
|
fee_tier=fee_tier_str,
|
||||||
@@ -97,15 +102,14 @@ async def fetch_and_store_account_snapshot(
|
|||||||
if isinstance(balance_data, dict):
|
if isinstance(balance_data, dict):
|
||||||
eb = balance_data.get("eb")
|
eb = balance_data.get("eb")
|
||||||
total_value = float(eb) if eb is not None else 0.0
|
total_value = float(eb) if eb is not None else 0.0
|
||||||
with store.connect() as conn:
|
async with store.pool.acquire() as conn:
|
||||||
conn.execute(
|
await conn.execute(
|
||||||
"INSERT INTO portfolio_snapshots"
|
"INSERT INTO portfolio_snapshots"
|
||||||
" (snapshot_at, balances, total_value_usd) VALUES (?, ?, ?)",
|
" (snapshot_at, balances, total_value_usd) VALUES ($1, $2, $3)",
|
||||||
(
|
|
||||||
datetime.now(UTC),
|
datetime.now(UTC),
|
||||||
orjson.dumps(wallet_balances).decode("utf-8") if wallet_balances else None,
|
orjson.dumps(wallet_balances).decode(
|
||||||
|
"utf-8") if wallet_balances else None,
|
||||||
total_value,
|
total_value,
|
||||||
),
|
|
||||||
)
|
)
|
||||||
_LOG.info("portfolio_snapshot_stored", total_value_usd=total_value)
|
_LOG.info("portfolio_snapshot_stored", total_value_usd=total_value)
|
||||||
except Exception:
|
except Exception:
|
||||||
@@ -116,14 +120,15 @@ async def fetch_and_store_account_snapshot(
|
|||||||
|
|
||||||
async def run_fee_sync_loop(
|
async def run_fee_sync_loop(
|
||||||
client: KrakenRestClient,
|
client: KrakenRestClient,
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
stop_event: asyncio.Event,
|
stop_event: asyncio.Event,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Periodic loop: fetch account snapshot every hour.
|
"""Periodic loop: fetch account snapshot every hour.
|
||||||
|
|
||||||
Runs until stop_event is set.
|
Runs until stop_event is set.
|
||||||
"""
|
"""
|
||||||
_LOG.info("fee_sync_loop_started", interval_s=_FEE_REFRESH_INTERVAL_SECONDS)
|
_LOG.info("fee_sync_loop_started",
|
||||||
|
interval_s=_FEE_REFRESH_INTERVAL_SECONDS)
|
||||||
|
|
||||||
while not stop_event.is_set():
|
while not stop_event.is_set():
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -6,14 +6,14 @@ import structlog
|
|||||||
|
|
||||||
from arbitrade.detection.engine import IncrementalCycleDetector
|
from arbitrade.detection.engine import IncrementalCycleDetector
|
||||||
from arbitrade.detection.graph import CurrencyGraph
|
from arbitrade.detection.graph import CurrencyGraph
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import ConfigPairingRepository
|
from arbitrade.storage.repositories import ConfigPairingRepository
|
||||||
|
|
||||||
_LOG = structlog.get_logger(__name__)
|
_LOG = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
def build_detector_from_enabled_pairings(
|
async def build_detector_from_enabled_pairings(
|
||||||
store: DuckDBStore,
|
store: PgStore,
|
||||||
*,
|
*,
|
||||||
fee_rate: float = 0.0,
|
fee_rate: float = 0.0,
|
||||||
max_depth_levels: int = 10,
|
max_depth_levels: int = 10,
|
||||||
@@ -24,7 +24,7 @@ def build_detector_from_enabled_pairings(
|
|||||||
Returns None if no enabled pairings exist.
|
Returns None if no enabled pairings exist.
|
||||||
"""
|
"""
|
||||||
repo = ConfigPairingRepository(store)
|
repo = ConfigPairingRepository(store)
|
||||||
pairings = repo.list_pairings(enabled_only=True)
|
pairings = await repo.list_pairings(enabled_only=True)
|
||||||
if not pairings:
|
if not pairings:
|
||||||
_LOG.warning("no_enabled_pairings_found_detector_not_created")
|
_LOG.warning("no_enabled_pairings_found_detector_not_created")
|
||||||
return None
|
return None
|
||||||
@@ -55,8 +55,8 @@ def build_detector_from_enabled_pairings(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def get_enabled_pair_symbols(store: DuckDBStore) -> list[str]:
|
async def get_enabled_pair_symbols(store: PgStore) -> list[str]:
|
||||||
"""Return list of enabled pair symbols (e.g. ['BTC/USD', 'ETH/BTC'])."""
|
"""Return list of enabled pair symbols (e.g. ['BTC/USD', 'ETH/BTC'])."""
|
||||||
repo = ConfigPairingRepository(store)
|
repo = ConfigPairingRepository(store)
|
||||||
pairings = repo.list_pairings(enabled_only=True)
|
pairings = await repo.list_pairings(enabled_only=True)
|
||||||
return [f"{p.base_asset}/{p.quote_asset}" for p in pairings if p.enabled]
|
return [f"{p.base_asset}/{p.quote_asset}" for p in pairings if p.enabled]
|
||||||
|
|||||||
+36
-34
@@ -3,7 +3,7 @@ from __future__ import annotations
|
|||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True, slots=True)
|
@dataclass(frozen=True, slots=True)
|
||||||
@@ -19,57 +19,55 @@ class PerformanceMetrics:
|
|||||||
|
|
||||||
|
|
||||||
class MetricsCalculator:
|
class MetricsCalculator:
|
||||||
def __init__(self, store: DuckDBStore) -> None:
|
def __init__(self, store: PgStore) -> None:
|
||||||
self._store = store
|
self._store = store
|
||||||
|
|
||||||
def compute(self) -> PerformanceMetrics:
|
async def compute(self) -> PerformanceMetrics:
|
||||||
with self._store.connect() as conn:
|
async with self._store.pool.acquire() as conn:
|
||||||
tm = conn.execute("""
|
tm = await conn.fetchrow("""
|
||||||
SELECT
|
SELECT
|
||||||
COALESCE(SUM(COALESCE(realized_pnl, 0)), 0) AS realized_pnl_usd,
|
COALESCE(SUM(COALESCE(realized_pnl, 0)), 0) AS realized_pnl_usd,
|
||||||
COUNT(*) AS total_trades,
|
COUNT(*) AS total_trades,
|
||||||
SUM(CASE WHEN realized_pnl > 0 THEN 1 ELSE 0 END) AS winning_trades,
|
SUM(CASE WHEN realized_pnl > 0 THEN 1 ELSE 0 END) AS winning_trades,
|
||||||
AVG(EPOCH(finished_at) - EPOCH(started_at)) AS avg_trade_duration_seconds,
|
AVG(EXTRACT(EPOCH FROM finished_at - started_at)) AS avg_trade_duration_seconds,
|
||||||
quantile_cont(
|
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM finished_at - started_at)) AS latency_p50_seconds,
|
||||||
EPOCH(finished_at) - EPOCH(started_at),
|
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM finished_at - started_at)) AS latency_p95_seconds,
|
||||||
0.50
|
PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM finished_at - started_at)) AS latency_p99_seconds
|
||||||
) AS latency_p50_seconds,
|
|
||||||
quantile_cont(
|
|
||||||
EPOCH(finished_at) - EPOCH(started_at),
|
|
||||||
0.95
|
|
||||||
) AS latency_p95_seconds,
|
|
||||||
quantile_cont(
|
|
||||||
EPOCH(finished_at) - EPOCH(started_at),
|
|
||||||
0.99
|
|
||||||
) AS latency_p99_seconds
|
|
||||||
FROM trades
|
FROM trades
|
||||||
WHERE finished_at IS NOT NULL
|
WHERE finished_at IS NOT NULL
|
||||||
""").fetchone()
|
""")
|
||||||
|
|
||||||
om = conn.execute("""
|
om = await conn.fetchrow("""
|
||||||
SELECT
|
SELECT
|
||||||
COUNT(*) AS opportunity_count,
|
COUNT(*) AS opportunity_count,
|
||||||
MIN(detected_at) AS first_detected_at,
|
MIN(detected_at) AS first_detected_at,
|
||||||
MAX(detected_at) AS last_detected_at
|
MAX(detected_at) AS last_detected_at
|
||||||
FROM opportunities
|
FROM opportunities
|
||||||
""").fetchone()
|
""")
|
||||||
|
|
||||||
fm = conn.execute("""
|
fm = await conn.fetchrow("""
|
||||||
SELECT AVG(filled_volume / volume) AS fill_rate
|
SELECT AVG(filled_volume / volume) AS fill_rate
|
||||||
FROM orders
|
FROM orders
|
||||||
WHERE volume > 0 AND filled_volume IS NOT NULL
|
WHERE volume > 0 AND filled_volume IS NOT NULL
|
||||||
""").fetchone()
|
""")
|
||||||
|
|
||||||
r_pnl_usd = float(tm[0]) if tm and tm[0] is not None else 0.0
|
r_pnl_usd = float(
|
||||||
tt = int(tm[1]) if tm and tm[1] is not None else 0
|
tm["realized_pnl_usd"]) if tm and tm["realized_pnl_usd"] is not None else 0.0
|
||||||
wt = int(tm[2]) if tm and tm[2] is not None else 0
|
tt = int(tm["total_trades"]
|
||||||
|
) if tm and tm["total_trades"] is not None else 0
|
||||||
|
wt = int(tm["winning_trades"]
|
||||||
|
) if tm and tm["winning_trades"] is not None else 0
|
||||||
wr = wt / tt if tt > 0 else None
|
wr = wt / tt if tt > 0 else None
|
||||||
|
|
||||||
atd = float(tm[3]) if tm and tm[3] is not None else None
|
atd = float(tm["avg_trade_duration_seconds"]
|
||||||
|
) if tm and tm["avg_trade_duration_seconds"] is not None else None
|
||||||
|
|
||||||
oc = int(om[0]) if om is not None and om[0] is not None else 0
|
oc = int(om["opportunity_count"]
|
||||||
fo = om[1] if om is not None and isinstance(om[1], datetime) else None
|
) if om is not None and om["opportunity_count"] is not None else 0
|
||||||
lo = om[2] if om is not None and isinstance(om[2], datetime) else None
|
fo = om["first_detected_at"] if om is not None and isinstance(
|
||||||
|
om["first_detected_at"], datetime) else None
|
||||||
|
lo = om["last_detected_at"] if om is not None and isinstance(
|
||||||
|
om["last_detected_at"], datetime) else None
|
||||||
|
|
||||||
opportunities_per_minute: float | None
|
opportunities_per_minute: float | None
|
||||||
if oc >= 2 and fo is not None and lo is not None:
|
if oc >= 2 and fo is not None and lo is not None:
|
||||||
@@ -82,11 +80,15 @@ class MetricsCalculator:
|
|||||||
else:
|
else:
|
||||||
opportunities_per_minute = None
|
opportunities_per_minute = None
|
||||||
|
|
||||||
fill_rate = float(fm[0]) if fm and fm[0] is not None else None
|
fill_rate = float(
|
||||||
|
fm["fill_rate"]) if fm and fm["fill_rate"] is not None else None
|
||||||
|
|
||||||
lp50 = float(tm[4]) if tm and tm[4] is not None else None
|
lp50 = float(tm["latency_p50_seconds"]
|
||||||
lp95 = float(tm[5]) if tm and tm[5] is not None else None
|
) if tm and tm["latency_p50_seconds"] is not None else None
|
||||||
lp99 = float(tm[6]) if tm and tm[6] is not None else None
|
lp95 = float(tm["latency_p95_seconds"]
|
||||||
|
) if tm and tm["latency_p95_seconds"] is not None else None
|
||||||
|
lp99 = float(tm["latency_p99_seconds"]
|
||||||
|
) if tm and tm["latency_p99_seconds"] is not None else None
|
||||||
|
|
||||||
return PerformanceMetrics(
|
return PerformanceMetrics(
|
||||||
realized_pnl_usd=r_pnl_usd,
|
realized_pnl_usd=r_pnl_usd,
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ from typing import Any, cast
|
|||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
|
|
||||||
from arbitrade.api.control_state import DashboardControlState
|
from arbitrade.api.control_state import DashboardControlState
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import (
|
from arbitrade.storage.repositories import (
|
||||||
AuditRecord,
|
AuditRecord,
|
||||||
AuditRepository,
|
AuditRepository,
|
||||||
@@ -29,8 +29,8 @@ def _controls(app: FastAPI) -> DashboardControlState:
|
|||||||
return cast(DashboardControlState, app.state.dashboard_controls)
|
return cast(DashboardControlState, app.state.dashboard_controls)
|
||||||
|
|
||||||
|
|
||||||
def _store(app: FastAPI) -> DuckDBStore:
|
def _store(app: FastAPI) -> PgStore:
|
||||||
return cast(DuckDBStore, app.state.store)
|
return cast(PgStore, app.state.store)
|
||||||
|
|
||||||
|
|
||||||
def _audit_repository(app: FastAPI) -> AuditRepository | None:
|
def _audit_repository(app: FastAPI) -> AuditRepository | None:
|
||||||
@@ -43,34 +43,34 @@ def _runtime_repository(app: FastAPI) -> RuntimeStateRepository | None:
|
|||||||
return repository if isinstance(repository, RuntimeStateRepository) else None
|
return repository if isinstance(repository, RuntimeStateRepository) else None
|
||||||
|
|
||||||
|
|
||||||
def _open_trade_count(store: DuckDBStore) -> int:
|
async def _open_trade_count(store: PgStore) -> int:
|
||||||
with store.connect() as conn:
|
async with store.pool.acquire() as conn:
|
||||||
row = conn.execute("""
|
row = await conn.fetchrow("""
|
||||||
SELECT COUNT(*)
|
SELECT COUNT(*)
|
||||||
FROM trades
|
FROM trades
|
||||||
WHERE finished_at IS NULL
|
WHERE finished_at IS NULL
|
||||||
""").fetchone()
|
""")
|
||||||
return int(row[0]) if row is not None else 0
|
return int(row[0]) if row is not None else 0
|
||||||
|
|
||||||
|
|
||||||
def _latest_balances(store: DuckDBStore) -> dict[str, Any] | None:
|
async def _latest_balances(store: PgStore) -> dict[str, Any] | None:
|
||||||
with store.connect() as conn:
|
async with store.pool.acquire() as conn:
|
||||||
row = conn.execute("""
|
row = await conn.fetchrow("""
|
||||||
SELECT balances
|
SELECT balances
|
||||||
FROM portfolio_snapshots
|
FROM portfolio_snapshots
|
||||||
ORDER BY snapshot_at DESC
|
ORDER BY snapshot_at DESC
|
||||||
LIMIT 1
|
LIMIT 1
|
||||||
""").fetchone()
|
""")
|
||||||
|
|
||||||
if row is None or row[0] is None:
|
if row is None or row["balances"] is None:
|
||||||
return None
|
return None
|
||||||
raw_balances = row[0]
|
raw_balances = row["balances"]
|
||||||
if isinstance(raw_balances, str):
|
if isinstance(raw_balances, str):
|
||||||
return {"raw": raw_balances}
|
return {"raw": raw_balances}
|
||||||
return {"raw": str(raw_balances)}
|
return {"raw": str(raw_balances)}
|
||||||
|
|
||||||
|
|
||||||
def _record_audit(
|
async def _record_audit(
|
||||||
app: FastAPI,
|
app: FastAPI,
|
||||||
*,
|
*,
|
||||||
event_type: str,
|
event_type: str,
|
||||||
@@ -80,7 +80,7 @@ def _record_audit(
|
|||||||
repository = _audit_repository(app)
|
repository = _audit_repository(app)
|
||||||
if repository is None:
|
if repository is None:
|
||||||
return
|
return
|
||||||
repository.insert(
|
await repository.insert(
|
||||||
AuditRecord(
|
AuditRecord(
|
||||||
occurred_at=datetime.now(UTC),
|
occurred_at=datetime.now(UTC),
|
||||||
actor="runtime",
|
actor="runtime",
|
||||||
@@ -106,7 +106,7 @@ async def _run_startup_reconciler(app: FastAPI) -> None:
|
|||||||
await result
|
await result
|
||||||
|
|
||||||
|
|
||||||
def persist_runtime_snapshot(app: FastAPI, *, note: str | None = None) -> RuntimeStateRecord | None:
|
async def persist_runtime_snapshot(app: FastAPI, *, note: str | None = None) -> RuntimeStateRecord | None:
|
||||||
repository = _runtime_repository(app)
|
repository = _runtime_repository(app)
|
||||||
if repository is None:
|
if repository is None:
|
||||||
return None
|
return None
|
||||||
@@ -118,11 +118,11 @@ def persist_runtime_snapshot(app: FastAPI, *, note: str | None = None) -> Runtim
|
|||||||
is_running=controls.is_running,
|
is_running=controls.is_running,
|
||||||
kill_switch_active=controls.kill_switch.is_active,
|
kill_switch_active=controls.kill_switch.is_active,
|
||||||
kill_switch_reason=controls.kill_switch.reason,
|
kill_switch_reason=controls.kill_switch.reason,
|
||||||
open_trade_count=_open_trade_count(store),
|
open_trade_count=await _open_trade_count(store),
|
||||||
last_known_balances=_latest_balances(store),
|
last_known_balances=await _latest_balances(store),
|
||||||
note=note,
|
note=note,
|
||||||
)
|
)
|
||||||
repository.insert(snapshot)
|
await repository.insert(snapshot)
|
||||||
return snapshot
|
return snapshot
|
||||||
|
|
||||||
|
|
||||||
@@ -134,7 +134,7 @@ async def restore_runtime_state(app: FastAPI) -> RuntimeRecoveryReport:
|
|||||||
restored_from_snapshot = False
|
restored_from_snapshot = False
|
||||||
snapshot_at: str | None = None
|
snapshot_at: str | None = None
|
||||||
|
|
||||||
latest = repo.latest() if repo is not None else None
|
latest = await repo.latest() if repo is not None else None
|
||||||
if latest is not None:
|
if latest is not None:
|
||||||
restored_from_snapshot = True
|
restored_from_snapshot = True
|
||||||
snapshot_at = latest.snapshot_at.isoformat()
|
snapshot_at = latest.snapshot_at.isoformat()
|
||||||
@@ -146,7 +146,7 @@ async def restore_runtime_state(app: FastAPI) -> RuntimeRecoveryReport:
|
|||||||
ctl.kill_switch.deactivate()
|
ctl.kill_switch.deactivate()
|
||||||
ctl.mark_updated()
|
ctl.mark_updated()
|
||||||
|
|
||||||
open_trades = _open_trade_count(store)
|
open_trades = await _open_trade_count(store)
|
||||||
restart_guard_active = False
|
restart_guard_active = False
|
||||||
if open_trades > 0:
|
if open_trades > 0:
|
||||||
ctl.is_running = False
|
ctl.is_running = False
|
||||||
@@ -163,7 +163,7 @@ async def restore_runtime_state(app: FastAPI) -> RuntimeRecoveryReport:
|
|||||||
)
|
)
|
||||||
app.state.recovery_report = report
|
app.state.recovery_report = report
|
||||||
|
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
app,
|
app,
|
||||||
event_type="runtime.startup_recovery",
|
event_type="runtime.startup_recovery",
|
||||||
decision="applied",
|
decision="applied",
|
||||||
@@ -212,7 +212,7 @@ async def graceful_shutdown(app: FastAPI) -> None:
|
|||||||
controls.is_running = False
|
controls.is_running = False
|
||||||
controls.mark_updated()
|
controls.mark_updated()
|
||||||
|
|
||||||
_record_audit(
|
await _record_audit(
|
||||||
app,
|
app,
|
||||||
event_type="runtime.shutdown",
|
event_type="runtime.shutdown",
|
||||||
decision="initiated",
|
decision="initiated",
|
||||||
@@ -220,4 +220,4 @@ async def graceful_shutdown(app: FastAPI) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
await drain_background_workers(app)
|
await drain_background_workers(app)
|
||||||
persist_runtime_snapshot(app, note="graceful_shutdown")
|
await persist_runtime_snapshot(app, note="graceful_shutdown")
|
||||||
|
|||||||
@@ -1,233 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from collections.abc import Iterator
|
|
||||||
from contextlib import contextmanager
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import duckdb
|
|
||||||
import structlog
|
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
|
||||||
|
|
||||||
_LOG = structlog.get_logger(__name__)
|
|
||||||
|
|
||||||
SCHEMA_SQL = """
|
|
||||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
|
||||||
version INTEGER PRIMARY KEY,
|
|
||||||
applied_at TIMESTAMP DEFAULT current_timestamp
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS config_sections (
|
|
||||||
id INTEGER PRIMARY KEY,
|
|
||||||
name VARCHAR UNIQUE NOT NULL,
|
|
||||||
description TEXT,
|
|
||||||
updated_at TIMESTAMP DEFAULT current_timestamp
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS config_settings (
|
|
||||||
key VARCHAR PRIMARY KEY,
|
|
||||||
section VARCHAR NOT NULL,
|
|
||||||
value_json TEXT NOT NULL,
|
|
||||||
value_type VARCHAR NOT NULL,
|
|
||||||
is_secret BOOLEAN DEFAULT FALSE,
|
|
||||||
is_runtime_reloadable BOOLEAN DEFAULT FALSE,
|
|
||||||
updated_at TIMESTAMP DEFAULT current_timestamp,
|
|
||||||
updated_by VARCHAR
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS config_pairings (
|
|
||||||
id INTEGER PRIMARY KEY,
|
|
||||||
base_asset VARCHAR NOT NULL,
|
|
||||||
quote_asset VARCHAR NOT NULL,
|
|
||||||
enabled BOOLEAN DEFAULT TRUE,
|
|
||||||
source VARCHAR NOT NULL,
|
|
||||||
created_at TIMESTAMP DEFAULT current_timestamp,
|
|
||||||
updated_at TIMESTAMP DEFAULT current_timestamp,
|
|
||||||
UNIQUE(base_asset, quote_asset)
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS config_backtesting_defaults (
|
|
||||||
id INTEGER PRIMARY KEY,
|
|
||||||
starting_balances JSON,
|
|
||||||
trade_capital DOUBLE,
|
|
||||||
min_profit_threshold DOUBLE,
|
|
||||||
slippage_bps INTEGER,
|
|
||||||
execution_latency_ms INTEGER,
|
|
||||||
fee_source VARCHAR DEFAULT 'api'
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS opportunities (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
detected_at TIMESTAMP NOT NULL,
|
|
||||||
cycle VARCHAR NOT NULL,
|
|
||||||
gross_pct DOUBLE,
|
|
||||||
net_pct DOUBLE,
|
|
||||||
est_profit DOUBLE,
|
|
||||||
executed BOOLEAN DEFAULT FALSE
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS trades (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
trade_ref VARCHAR NOT NULL,
|
|
||||||
started_at TIMESTAMP NOT NULL,
|
|
||||||
finished_at TIMESTAMP,
|
|
||||||
status VARCHAR NOT NULL,
|
|
||||||
realized_pnl DOUBLE,
|
|
||||||
estimated_pnl DOUBLE,
|
|
||||||
capital_used DOUBLE,
|
|
||||||
cycle VARCHAR,
|
|
||||||
leg_count INTEGER
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS orders (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
trade_ref VARCHAR NOT NULL,
|
|
||||||
order_ref VARCHAR NOT NULL,
|
|
||||||
leg_index INTEGER NOT NULL,
|
|
||||||
pair VARCHAR NOT NULL,
|
|
||||||
side VARCHAR NOT NULL,
|
|
||||||
volume DOUBLE NOT NULL,
|
|
||||||
user_ref INTEGER,
|
|
||||||
status VARCHAR,
|
|
||||||
filled_volume DOUBLE,
|
|
||||||
avg_price DOUBLE,
|
|
||||||
raw_response JSON,
|
|
||||||
recorded_at TIMESTAMP NOT NULL
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS pnl_events (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
trade_ref VARCHAR NOT NULL,
|
|
||||||
recorded_at TIMESTAMP NOT NULL,
|
|
||||||
kind VARCHAR NOT NULL,
|
|
||||||
pnl_usd DOUBLE NOT NULL,
|
|
||||||
source VARCHAR NOT NULL
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS portfolio_snapshots (
|
|
||||||
snapshot_at TIMESTAMP NOT NULL,
|
|
||||||
balances JSON,
|
|
||||||
total_value_usd DOUBLE
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS market_snapshots (
|
|
||||||
snapshot_at TIMESTAMP NOT NULL,
|
|
||||||
symbol VARCHAR NOT NULL,
|
|
||||||
source VARCHAR NOT NULL,
|
|
||||||
payload JSON NOT NULL,
|
|
||||||
latency_ms DOUBLE
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS audit_events (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
occurred_at TIMESTAMP NOT NULL,
|
|
||||||
actor VARCHAR NOT NULL,
|
|
||||||
event_type VARCHAR NOT NULL,
|
|
||||||
decision VARCHAR NOT NULL,
|
|
||||||
payload JSON,
|
|
||||||
correlation_id VARCHAR
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS runtime_state_snapshots (
|
|
||||||
snapshot_at TIMESTAMP NOT NULL,
|
|
||||||
is_running BOOLEAN NOT NULL,
|
|
||||||
kill_switch_active BOOLEAN NOT NULL,
|
|
||||||
kill_switch_reason VARCHAR,
|
|
||||||
open_trade_count INTEGER NOT NULL,
|
|
||||||
last_known_balances JSON,
|
|
||||||
note VARCHAR
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS kraken_account_snapshots (
|
|
||||||
snapshot_at TIMESTAMP NOT NULL,
|
|
||||||
fee_tier VARCHAR,
|
|
||||||
maker_fee DOUBLE,
|
|
||||||
taker_fee DOUBLE,
|
|
||||||
thirty_day_volume DOUBLE,
|
|
||||||
trade_balance_raw JSON,
|
|
||||||
fee_schedule_raw JSON
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS backtest_jobs (
|
|
||||||
id UUID DEFAULT uuid(),
|
|
||||||
status VARCHAR NOT NULL DEFAULT 'pending',
|
|
||||||
events_path VARCHAR NOT NULL,
|
|
||||||
config JSON,
|
|
||||||
report JSON,
|
|
||||||
error VARCHAR,
|
|
||||||
created_at TIMESTAMP DEFAULT current_timestamp,
|
|
||||||
started_at TIMESTAMP,
|
|
||||||
finished_at TIMESTAMP
|
|
||||||
);
|
|
||||||
"""
|
|
||||||
|
|
||||||
|
|
||||||
class DuckDBStore:
|
|
||||||
SCHEMA_VERSION = 5
|
|
||||||
|
|
||||||
def __init__(self, settings: Settings) -> None:
|
|
||||||
self._db_path = Path(settings.duckdb_path)
|
|
||||||
self._db_path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
self._use_memory_fallback = False
|
|
||||||
|
|
||||||
@contextmanager
|
|
||||||
def connect(self) -> Iterator[duckdb.DuckDBPyConnection]:
|
|
||||||
try:
|
|
||||||
conn = duckdb.connect(str(self._db_path))
|
|
||||||
except duckdb.IOException:
|
|
||||||
if not self._use_memory_fallback:
|
|
||||||
_LOG.warning(
|
|
||||||
"duckdb_path_unavailable_falling_back_to_memory", path=str(self._db_path)
|
|
||||||
)
|
|
||||||
self._use_memory_fallback = True
|
|
||||||
conn = duckdb.connect(":memory:")
|
|
||||||
try:
|
|
||||||
yield conn
|
|
||||||
finally:
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
def _get_table_columns(self, conn: duckdb.DuckDBPyConnection, table_name: str) -> set[str]:
|
|
||||||
try:
|
|
||||||
rows = conn.execute(f"PRAGMA table_info({table_name})").fetchall()
|
|
||||||
return {str(row[1]) for row in rows}
|
|
||||||
except Exception:
|
|
||||||
return set()
|
|
||||||
|
|
||||||
def _table_exists(self, conn: duckdb.DuckDBPyConnection, table_name: str) -> bool:
|
|
||||||
try:
|
|
||||||
result = conn.execute(
|
|
||||||
f"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='{table_name}'"
|
|
||||||
).fetchone()
|
|
||||||
count = result[0] if result else 0
|
|
||||||
return count > 0
|
|
||||||
except Exception:
|
|
||||||
return False
|
|
||||||
|
|
||||||
def _ensure_column(
|
|
||||||
self, conn: duckdb.DuckDBPyConnection, table_name: str, column_def: str
|
|
||||||
) -> None:
|
|
||||||
"""Add a column to a table if it doesn't already exist."""
|
|
||||||
existing = self._get_table_columns(conn, table_name)
|
|
||||||
col_name = column_def.split()[0]
|
|
||||||
if col_name not in existing:
|
|
||||||
conn.execute(f"ALTER TABLE {table_name} ADD COLUMN {column_def}")
|
|
||||||
|
|
||||||
def migrate(self) -> None:
|
|
||||||
with self.connect() as conn:
|
|
||||||
# Run CREATE TABLE IF NOT EXISTS for all tables
|
|
||||||
conn.execute(SCHEMA_SQL)
|
|
||||||
|
|
||||||
# Ensure schema_migrations table exists and get current version
|
|
||||||
if not self._table_exists(conn, "schema_migrations"):
|
|
||||||
conn.execute("""
|
|
||||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
|
||||||
version INTEGER PRIMARY KEY,
|
|
||||||
applied_at TIMESTAMP DEFAULT current_timestamp
|
|
||||||
)
|
|
||||||
""")
|
|
||||||
|
|
||||||
# Update version to current
|
|
||||||
conn.execute(
|
|
||||||
f"INSERT OR REPLACE INTO schema_migrations (version, applied_at) "
|
|
||||||
f"VALUES ({self.SCHEMA_VERSION}, current_timestamp)"
|
|
||||||
)
|
|
||||||
@@ -36,7 +36,8 @@ class AsyncExecutionWriter:
|
|||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
if self._task is None or self._task.done():
|
if self._task is None or self._task.done():
|
||||||
self._stop.clear()
|
self._stop.clear()
|
||||||
self._task = asyncio.create_task(self._run(), name="execution-writer")
|
self._task = asyncio.create_task(
|
||||||
|
self._run(), name="execution-writer")
|
||||||
|
|
||||||
async def stop(self) -> None:
|
async def stop(self) -> None:
|
||||||
self._stop.set()
|
self._stop.set()
|
||||||
@@ -55,11 +56,11 @@ class AsyncExecutionWriter:
|
|||||||
|
|
||||||
try:
|
try:
|
||||||
if isinstance(record, TradeRecord):
|
if isinstance(record, TradeRecord):
|
||||||
self._trade_repository.insert(record)
|
await self._trade_repository.insert(record)
|
||||||
elif isinstance(record, OrderRecord):
|
elif isinstance(record, OrderRecord):
|
||||||
self._order_repository.insert(record)
|
await self._order_repository.insert(record)
|
||||||
else:
|
else:
|
||||||
self._pnl_repository.insert(record)
|
await self._pnl_repository.insert(record)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_LOG.error("execution_write_failed", error=str(exc))
|
_LOG.error("execution_write_failed", error=str(exc))
|
||||||
finally:
|
finally:
|
||||||
|
|||||||
@@ -24,14 +24,16 @@ class MarketSnapshot:
|
|||||||
class AsyncMarketSnapshotWriter:
|
class AsyncMarketSnapshotWriter:
|
||||||
def __init__(self, repository: MarketSnapshotRepository, max_queue_size: int = 50_000) -> None:
|
def __init__(self, repository: MarketSnapshotRepository, max_queue_size: int = 50_000) -> None:
|
||||||
self._repository = repository
|
self._repository = repository
|
||||||
self._queue: asyncio.Queue[MarketSnapshot] = asyncio.Queue(maxsize=max_queue_size)
|
self._queue: asyncio.Queue[MarketSnapshot] = asyncio.Queue(
|
||||||
|
maxsize=max_queue_size)
|
||||||
self._task: asyncio.Task[None] | None = None
|
self._task: asyncio.Task[None] | None = None
|
||||||
self._stop = asyncio.Event()
|
self._stop = asyncio.Event()
|
||||||
|
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
if self._task is None or self._task.done():
|
if self._task is None or self._task.done():
|
||||||
self._stop.clear()
|
self._stop.clear()
|
||||||
self._task = asyncio.create_task(self._run(), name="market-snapshot-writer")
|
self._task = asyncio.create_task(
|
||||||
|
self._run(), name="market-snapshot-writer")
|
||||||
|
|
||||||
async def stop(self) -> None:
|
async def stop(self) -> None:
|
||||||
self._stop.set()
|
self._stop.set()
|
||||||
@@ -49,7 +51,7 @@ class AsyncMarketSnapshotWriter:
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
try:
|
try:
|
||||||
self._repository.insert(
|
await self._repository.insert(
|
||||||
MarketSnapshotRecord(
|
MarketSnapshotRecord(
|
||||||
snapshot_at=item.snapshot_at,
|
snapshot_at=item.snapshot_at,
|
||||||
symbol=item.symbol,
|
symbol=item.symbol,
|
||||||
@@ -59,6 +61,7 @@ class AsyncMarketSnapshotWriter:
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_LOG.error("market_snapshot_write_failed", error=str(exc), symbol=item.symbol)
|
_LOG.error("market_snapshot_write_failed",
|
||||||
|
error=str(exc), symbol=item.symbol)
|
||||||
finally:
|
finally:
|
||||||
self._queue.task_done()
|
self._queue.task_done()
|
||||||
|
|||||||
@@ -13,14 +13,16 @@ _LOG = structlog.get_logger(__name__)
|
|||||||
class AsyncOpportunityWriter:
|
class AsyncOpportunityWriter:
|
||||||
def __init__(self, repository: OpportunityRepository, max_queue_size: int = 50_000) -> None:
|
def __init__(self, repository: OpportunityRepository, max_queue_size: int = 50_000) -> None:
|
||||||
self._repository = repository
|
self._repository = repository
|
||||||
self._queue: asyncio.Queue[OpportunityEvent] = asyncio.Queue(maxsize=max_queue_size)
|
self._queue: asyncio.Queue[OpportunityEvent] = asyncio.Queue(
|
||||||
|
maxsize=max_queue_size)
|
||||||
self._task: asyncio.Task[None] | None = None
|
self._task: asyncio.Task[None] | None = None
|
||||||
self._stop = asyncio.Event()
|
self._stop = asyncio.Event()
|
||||||
|
|
||||||
async def start(self) -> None:
|
async def start(self) -> None:
|
||||||
if self._task is None or self._task.done():
|
if self._task is None or self._task.done():
|
||||||
self._stop.clear()
|
self._stop.clear()
|
||||||
self._task = asyncio.create_task(self._run(), name="opportunity-writer")
|
self._task = asyncio.create_task(
|
||||||
|
self._run(), name="opportunity-writer")
|
||||||
|
|
||||||
async def stop(self) -> None:
|
async def stop(self) -> None:
|
||||||
self._stop.set()
|
self._stop.set()
|
||||||
@@ -38,7 +40,7 @@ class AsyncOpportunityWriter:
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
try:
|
try:
|
||||||
self._repository.insert(
|
await self._repository.insert(
|
||||||
OpportunityRecord(
|
OpportunityRecord(
|
||||||
detected_at=event.detected_at,
|
detected_at=event.detected_at,
|
||||||
cycle=event.cycle,
|
cycle=event.cycle,
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
"""PostgreSQL store — async connection pool wrapper around asyncpg."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import asyncpg
|
||||||
|
import structlog
|
||||||
|
|
||||||
|
from arbitrade.config.settings import Settings
|
||||||
|
|
||||||
|
_LOG = structlog.get_logger(__name__)
|
||||||
|
|
||||||
|
SCHEMA_VERSION = 1
|
||||||
|
|
||||||
|
|
||||||
|
class PgStore:
|
||||||
|
"""Async PostgreSQL connection pool for the arbitrade bot.
|
||||||
|
|
||||||
|
Wraps an ``asyncpg.Pool`` with schema migration support.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, settings: Settings) -> None:
|
||||||
|
self._dsn: str | None = None
|
||||||
|
self._pool: asyncpg.Pool | None = None
|
||||||
|
self._settings = settings
|
||||||
|
|
||||||
|
# ── lifecycle ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async def start(self) -> None:
|
||||||
|
"""Create the connection pool."""
|
||||||
|
s = self._settings
|
||||||
|
self._pool = await asyncpg.create_pool(
|
||||||
|
host=s.pg_host,
|
||||||
|
port=s.pg_port,
|
||||||
|
database=s.pg_database,
|
||||||
|
user=s.pg_user,
|
||||||
|
password=s.pg_password,
|
||||||
|
min_size=s.pg_min_connections,
|
||||||
|
max_size=s.pg_max_connections,
|
||||||
|
)
|
||||||
|
_LOG.info(
|
||||||
|
"pg_pool_created",
|
||||||
|
host=s.pg_host,
|
||||||
|
database=s.pg_database,
|
||||||
|
min_size=s.pg_min_connections,
|
||||||
|
max_size=s.pg_max_connections,
|
||||||
|
)
|
||||||
|
|
||||||
|
async def stop(self) -> None:
|
||||||
|
"""Close the connection pool."""
|
||||||
|
if self._pool is not None:
|
||||||
|
await self._pool.close()
|
||||||
|
self._pool = None
|
||||||
|
_LOG.info("pg_pool_closed")
|
||||||
|
|
||||||
|
@property
|
||||||
|
def pool(self) -> asyncpg.Pool:
|
||||||
|
"""Return the underlying connection pool.
|
||||||
|
|
||||||
|
Raises ``RuntimeError`` if ``start()`` has not been called yet.
|
||||||
|
"""
|
||||||
|
if self._pool is None:
|
||||||
|
raise RuntimeError("PgStore not started — call start() first")
|
||||||
|
return self._pool
|
||||||
|
|
||||||
|
# ── schema migration ─────────────────────────────────────────
|
||||||
|
|
||||||
|
async def migrate(self) -> None:
|
||||||
|
"""Apply the PostgreSQL schema.
|
||||||
|
|
||||||
|
Reads ``schema_pg.sql`` from the same package directory and
|
||||||
|
executes it, then records the migration version.
|
||||||
|
"""
|
||||||
|
schema_path = Path(__file__).with_name("schema_pg.sql")
|
||||||
|
schema_sql = schema_path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
async with self.pool.acquire() as conn:
|
||||||
|
# Apply the full schema (CREATE TABLE IF NOT EXISTS …)
|
||||||
|
await conn.execute(schema_sql)
|
||||||
|
|
||||||
|
# Record the current schema version
|
||||||
|
await conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO schema_migrations (version, applied_at)
|
||||||
|
VALUES ($1, CURRENT_TIMESTAMP)
|
||||||
|
ON CONFLICT (version) DO UPDATE SET applied_at = CURRENT_TIMESTAMP
|
||||||
|
""",
|
||||||
|
SCHEMA_VERSION,
|
||||||
|
)
|
||||||
|
|
||||||
|
_LOG.info("pg_schema_migrated", version=SCHEMA_VERSION)
|
||||||
|
|
||||||
|
# ── helpers ──────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async def table_exists(self, table_name: str) -> bool:
|
||||||
|
"""Check if a table exists in the current schema."""
|
||||||
|
async with self.pool.acquire() as conn:
|
||||||
|
row = await conn.fetchrow(
|
||||||
|
"""
|
||||||
|
SELECT COUNT(*) AS cnt
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_schema = 'public' AND table_name = $1
|
||||||
|
""",
|
||||||
|
table_name,
|
||||||
|
)
|
||||||
|
return bool(row and row["cnt"] > 0)
|
||||||
|
|
||||||
|
async def get_table_columns(self, table_name: str) -> set[str]:
|
||||||
|
"""Return the set of column names for *table_name*."""
|
||||||
|
async with self.pool.acquire() as conn:
|
||||||
|
rows = await conn.fetch(
|
||||||
|
"""
|
||||||
|
SELECT column_name
|
||||||
|
FROM information_schema.columns
|
||||||
|
WHERE table_schema = 'public' AND table_name = $1
|
||||||
|
""",
|
||||||
|
table_name,
|
||||||
|
)
|
||||||
|
return {str(r["column_name"]) for r in rows}
|
||||||
|
|
||||||
|
async def ensure_column(self, table_name: str, column_def: str) -> None:
|
||||||
|
"""Add a column to *table_name* if it does not already exist.
|
||||||
|
|
||||||
|
``column_def`` should be something like ``"my_col VARCHAR"``.
|
||||||
|
"""
|
||||||
|
existing = await self.get_table_columns(table_name)
|
||||||
|
col_name = column_def.split()[0]
|
||||||
|
if col_name not in existing:
|
||||||
|
async with self.pool.acquire() as conn:
|
||||||
|
await conn.execute(
|
||||||
|
f"ALTER TABLE {table_name} ADD COLUMN {column_def}"
|
||||||
|
)
|
||||||
|
_LOG.info("pg_column_added", table=table_name, column=col_name)
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,167 @@
|
|||||||
|
-- PostgreSQL schema for arbitrade bot
|
||||||
|
-- Requires pgcrypto extension for gen_random_uuid()
|
||||||
|
|
||||||
|
CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||||
|
|
||||||
|
-- ========================================
|
||||||
|
-- Schema version tracking
|
||||||
|
-- ========================================
|
||||||
|
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||||
|
version INTEGER PRIMARY KEY,
|
||||||
|
applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ========================================
|
||||||
|
-- Configuration
|
||||||
|
-- ========================================
|
||||||
|
CREATE TABLE IF NOT EXISTS config_sections (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
name VARCHAR UNIQUE NOT NULL,
|
||||||
|
description TEXT,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS config_settings (
|
||||||
|
key VARCHAR PRIMARY KEY,
|
||||||
|
section VARCHAR NOT NULL,
|
||||||
|
value_json TEXT NOT NULL,
|
||||||
|
value_type VARCHAR NOT NULL,
|
||||||
|
is_secret BOOLEAN DEFAULT FALSE,
|
||||||
|
is_runtime_reloadable BOOLEAN DEFAULT FALSE,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_by VARCHAR
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS config_pairings (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
base_asset VARCHAR NOT NULL,
|
||||||
|
quote_asset VARCHAR NOT NULL,
|
||||||
|
enabled BOOLEAN DEFAULT TRUE,
|
||||||
|
source VARCHAR NOT NULL,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE(base_asset, quote_asset)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS config_backtesting_defaults (
|
||||||
|
id SERIAL PRIMARY KEY,
|
||||||
|
starting_balances JSONB,
|
||||||
|
trade_capital DOUBLE PRECISION,
|
||||||
|
min_profit_threshold DOUBLE PRECISION,
|
||||||
|
slippage_bps INTEGER,
|
||||||
|
execution_latency_ms INTEGER,
|
||||||
|
fee_source VARCHAR DEFAULT 'api'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ========================================
|
||||||
|
-- Detection & Execution
|
||||||
|
-- ========================================
|
||||||
|
CREATE TABLE IF NOT EXISTS opportunities (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
detected_at TIMESTAMP NOT NULL,
|
||||||
|
cycle VARCHAR NOT NULL,
|
||||||
|
gross_pct DOUBLE PRECISION,
|
||||||
|
net_pct DOUBLE PRECISION,
|
||||||
|
est_profit DOUBLE PRECISION,
|
||||||
|
executed BOOLEAN DEFAULT FALSE
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS trades (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
trade_ref VARCHAR NOT NULL,
|
||||||
|
started_at TIMESTAMP NOT NULL,
|
||||||
|
finished_at TIMESTAMP,
|
||||||
|
status VARCHAR NOT NULL,
|
||||||
|
realized_pnl DOUBLE PRECISION,
|
||||||
|
estimated_pnl DOUBLE PRECISION,
|
||||||
|
capital_used DOUBLE PRECISION,
|
||||||
|
cycle VARCHAR,
|
||||||
|
leg_count INTEGER
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS orders (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
trade_ref VARCHAR NOT NULL,
|
||||||
|
order_ref VARCHAR NOT NULL,
|
||||||
|
leg_index INTEGER NOT NULL,
|
||||||
|
pair VARCHAR NOT NULL,
|
||||||
|
side VARCHAR NOT NULL,
|
||||||
|
volume DOUBLE PRECISION NOT NULL,
|
||||||
|
user_ref INTEGER,
|
||||||
|
status VARCHAR,
|
||||||
|
filled_volume DOUBLE PRECISION,
|
||||||
|
avg_price DOUBLE PRECISION,
|
||||||
|
raw_response JSONB,
|
||||||
|
recorded_at TIMESTAMP NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS pnl_events (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
trade_ref VARCHAR NOT NULL,
|
||||||
|
recorded_at TIMESTAMP NOT NULL,
|
||||||
|
kind VARCHAR NOT NULL,
|
||||||
|
pnl_usd DOUBLE PRECISION NOT NULL,
|
||||||
|
source VARCHAR NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ========================================
|
||||||
|
-- Snapshots & Monitoring
|
||||||
|
-- ========================================
|
||||||
|
CREATE TABLE IF NOT EXISTS portfolio_snapshots (
|
||||||
|
snapshot_at TIMESTAMP NOT NULL,
|
||||||
|
balances JSONB,
|
||||||
|
total_value_usd DOUBLE PRECISION
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS market_snapshots (
|
||||||
|
snapshot_at TIMESTAMP NOT NULL,
|
||||||
|
symbol VARCHAR NOT NULL,
|
||||||
|
source VARCHAR NOT NULL,
|
||||||
|
payload JSONB NOT NULL,
|
||||||
|
latency_ms DOUBLE PRECISION
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS audit_events (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
occurred_at TIMESTAMP NOT NULL,
|
||||||
|
actor VARCHAR NOT NULL,
|
||||||
|
event_type VARCHAR NOT NULL,
|
||||||
|
decision VARCHAR NOT NULL,
|
||||||
|
payload JSONB,
|
||||||
|
correlation_id VARCHAR
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS runtime_state_snapshots (
|
||||||
|
snapshot_at TIMESTAMP NOT NULL,
|
||||||
|
is_running BOOLEAN NOT NULL,
|
||||||
|
kill_switch_active BOOLEAN NOT NULL,
|
||||||
|
kill_switch_reason VARCHAR,
|
||||||
|
open_trade_count INTEGER NOT NULL,
|
||||||
|
last_known_balances JSONB,
|
||||||
|
note VARCHAR
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS kraken_account_snapshots (
|
||||||
|
snapshot_at TIMESTAMP NOT NULL,
|
||||||
|
fee_tier VARCHAR,
|
||||||
|
maker_fee DOUBLE PRECISION,
|
||||||
|
taker_fee DOUBLE PRECISION,
|
||||||
|
thirty_day_volume DOUBLE PRECISION,
|
||||||
|
trade_balance_raw JSONB,
|
||||||
|
fee_schedule_raw JSONB
|
||||||
|
);
|
||||||
|
|
||||||
|
-- ========================================
|
||||||
|
-- Backtesting
|
||||||
|
-- ========================================
|
||||||
|
CREATE TABLE IF NOT EXISTS backtest_jobs (
|
||||||
|
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
|
||||||
|
status VARCHAR NOT NULL DEFAULT 'pending',
|
||||||
|
events_path VARCHAR NOT NULL,
|
||||||
|
config JSONB,
|
||||||
|
report JSONB,
|
||||||
|
error VARCHAR,
|
||||||
|
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
started_at TIMESTAMP,
|
||||||
|
finished_at TIMESTAMP
|
||||||
|
);
|
||||||
@@ -7,7 +7,7 @@
|
|||||||
<div class="chart-head">
|
<div class="chart-head">
|
||||||
<div>
|
<div>
|
||||||
<div class="label">Opportunity Trend</div>
|
<div class="label">Opportunity Trend</div>
|
||||||
<div class="meta">Recent opportunities from DuckDB. Updated {{ generated_at }}</div>
|
<div class="meta">Recent opportunities from PostgreSQL. Updated {{ generated_at }}</div>
|
||||||
</div>
|
</div>
|
||||||
<button type="button" class="button secondary" x-on:click="expanded = !expanded">
|
<button type="button" class="button secondary" x-on:click="expanded = !expanded">
|
||||||
<span x-text="expanded ? 'Hide chart' : 'Show chart'"></span>
|
<span x-text="expanded ? 'Hide chart' : 'Show chart'"></span>
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
"""End-to-end tests — require full app startup with PostgreSQL."""
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Integration tests for PostgreSQL schema and connectivity."""
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
"""pytest configuration for integration tests.
|
||||||
|
|
||||||
|
Integration tests require a live PostgreSQL server at the configured host.
|
||||||
|
They are skipped automatically if the server is unreachable.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def pytest_ignore_collect(path: str, config: pytest.Config) -> bool:
|
||||||
|
"""Skip integration tests unless --integration is passed."""
|
||||||
|
if "integration" in path and not config.getoption("--integration", False):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def pytest_addoption(parser: pytest.Parser) -> None:
|
||||||
|
parser.addoption(
|
||||||
|
"--integration",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
help="Run integration tests (requires PostgreSQL)",
|
||||||
|
)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pytest_asyncio
|
||||||
|
|
||||||
|
from arbitrade.config.settings import get_settings
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
from arbitrade.storage.repositories import AuditRecord, AuditRepository
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.integration
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _pg() -> AsyncIterator[PgStore]:
|
||||||
|
s = get_settings()
|
||||||
|
store = PgStore(s)
|
||||||
|
try:
|
||||||
|
await store.start()
|
||||||
|
await store.migrate()
|
||||||
|
yield store
|
||||||
|
finally:
|
||||||
|
await store.stop()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_audit_repository_inserts_and_lists_recent() -> None:
|
||||||
|
async with _pg() as store:
|
||||||
|
repository = AuditRepository(store)
|
||||||
|
|
||||||
|
await repository.insert(
|
||||||
|
AuditRecord(
|
||||||
|
occurred_at=datetime.now(UTC),
|
||||||
|
actor="dashboard_user",
|
||||||
|
event_type="dashboard.control.start",
|
||||||
|
decision="approved",
|
||||||
|
payload={"execution_status": "running"},
|
||||||
|
correlation_id="req-1",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
recent = await repository.list_recent(limit=5)
|
||||||
|
|
||||||
|
assert len(recent) == 1
|
||||||
|
assert recent[0].actor == "dashboard_user"
|
||||||
|
assert recent[0].event_type == "dashboard.control.start"
|
||||||
|
assert recent[0].decision == "approved"
|
||||||
|
assert recent[0].payload == {"execution_status": "running"}
|
||||||
|
assert recent[0].correlation_id == "req-1"
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from arbitrade.config.settings import get_settings
|
||||||
|
from arbitrade.detection.engine import OpportunityEvent
|
||||||
|
from arbitrade.execution.sequencer import TriangularExecutionSequencer
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
from arbitrade.storage.executions import AsyncExecutionWriter
|
||||||
|
from arbitrade.storage.repositories import OrderRepository, PnLRepository, TradeRepository
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.integration
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(slots=True)
|
||||||
|
class _FakeRestClient:
|
||||||
|
calls: int = 0
|
||||||
|
|
||||||
|
async def place_market_order(self, *, pair: str, side: str, volume: float) -> dict[str, object]:
|
||||||
|
self.calls += 1
|
||||||
|
return {"txid": [f"tx-{self.calls}"], "status": "submitted"}
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_event() -> OpportunityEvent:
|
||||||
|
return OpportunityEvent(
|
||||||
|
detected_at=datetime.now(UTC),
|
||||||
|
cycle="USD->BTC->ETH->USD",
|
||||||
|
updated_pair="BTC/USD",
|
||||||
|
gross_rate=1.04,
|
||||||
|
net_rate=1.03,
|
||||||
|
gross_pct=4.0,
|
||||||
|
net_pct=3.0,
|
||||||
|
est_profit=0.03,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _pg() -> AsyncIterator[PgStore]:
|
||||||
|
s = get_settings()
|
||||||
|
store = PgStore(s)
|
||||||
|
try:
|
||||||
|
await store.start()
|
||||||
|
await store.migrate()
|
||||||
|
yield store
|
||||||
|
finally:
|
||||||
|
await store.stop()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_execution_writer_persists_trade_order_and_pnl() -> None:
|
||||||
|
async with _pg() as store:
|
||||||
|
writer = AsyncExecutionWriter(
|
||||||
|
TradeRepository(store),
|
||||||
|
OrderRepository(store),
|
||||||
|
PnLRepository(store),
|
||||||
|
max_queue_size=10,
|
||||||
|
)
|
||||||
|
await writer.start()
|
||||||
|
|
||||||
|
client = _FakeRestClient()
|
||||||
|
sequencer = TriangularExecutionSequencer(
|
||||||
|
client,
|
||||||
|
available_pairs=["BTC/USD", "ETH/BTC", "ETH/USD"],
|
||||||
|
execution_writer=writer,
|
||||||
|
)
|
||||||
|
|
||||||
|
result = await sequencer.execute(_sample_event())
|
||||||
|
await writer.stop()
|
||||||
|
|
||||||
|
assert result.success
|
||||||
|
assert client.calls == 3
|
||||||
|
|
||||||
|
async with store.pool.acquire() as conn:
|
||||||
|
trades = await conn.fetch(
|
||||||
|
"SELECT trade_ref, status, estimated_pnl, capital_used, cycle, leg_count FROM trades"
|
||||||
|
)
|
||||||
|
orders = await conn.fetch(
|
||||||
|
"SELECT trade_ref, order_ref, leg_index, pair, side, volume, status "
|
||||||
|
"FROM orders ORDER BY leg_index"
|
||||||
|
)
|
||||||
|
pnls = await conn.fetch("SELECT trade_ref, kind, pnl_usd, source FROM pnl_events")
|
||||||
|
|
||||||
|
assert len(trades) == 1
|
||||||
|
assert trades[0]["status"] == "filled"
|
||||||
|
assert trades[0]["estimated_pnl"] == 0.03
|
||||||
|
assert trades[0]["capital_used"] == 1.0
|
||||||
|
assert trades[0]["cycle"] == "USD->BTC->ETH->USD"
|
||||||
|
assert trades[0]["leg_count"] == 3
|
||||||
|
|
||||||
|
assert len(orders) == 3
|
||||||
|
assert orders[0]["leg_index"] == 0
|
||||||
|
assert orders[1]["leg_index"] == 1
|
||||||
|
assert orders[2]["leg_index"] == 2
|
||||||
|
assert orders[0]["status"] == "submitted"
|
||||||
|
|
||||||
|
assert len(pnls) == 1
|
||||||
|
assert pnls[0]["kind"] == "estimated"
|
||||||
|
assert pnls[0]["pnl_usd"] == 0.03
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pytest_asyncio
|
||||||
|
|
||||||
|
from arbitrade.config.settings import get_settings
|
||||||
|
from arbitrade.metrics import MetricsCalculator
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.integration
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _pg() -> AsyncIterator[PgStore]:
|
||||||
|
s = get_settings()
|
||||||
|
store = PgStore(s)
|
||||||
|
try:
|
||||||
|
await store.start()
|
||||||
|
await store.migrate()
|
||||||
|
yield store
|
||||||
|
finally:
|
||||||
|
await store.stop()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_metrics_calculator_summarizes_execution_data() -> None:
|
||||||
|
async with _pg() as store:
|
||||||
|
started = datetime.now(UTC)
|
||||||
|
finished = started + timedelta(seconds=30)
|
||||||
|
started_two = started + timedelta(minutes=1)
|
||||||
|
finished_two = started_two + timedelta(seconds=90)
|
||||||
|
|
||||||
|
async with store.pool.acquire() as conn:
|
||||||
|
await conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO trades (
|
||||||
|
trade_ref, started_at, finished_at, status,
|
||||||
|
realized_pnl, estimated_pnl, capital_used, cycle, leg_count
|
||||||
|
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9),
|
||||||
|
($10, $11, $12, $13, $14, $15, $16, $17, $18)
|
||||||
|
""",
|
||||||
|
"trade-1", started, finished, "filled", 12.5, 10.0, 100.0, "USD->BTC->ETH->USD", 3,
|
||||||
|
"trade-2", started_two, finished_two, "filled", -
|
||||||
|
4.5, -2.0, 200.0, "USD->ETH->BTC->USD", 3,
|
||||||
|
)
|
||||||
|
await conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO opportunities (detected_at, cycle, gross_pct, net_pct, est_profit, executed)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6),
|
||||||
|
($7, $8, $9, $10, $11, $12),
|
||||||
|
($13, $14, $15, $16, $17, $18)
|
||||||
|
""",
|
||||||
|
started, "USD->BTC->ETH->USD", 4.0, 3.0, 0.03, True,
|
||||||
|
started_two, "USD->ETH->BTC->USD", 2.0, 1.0, 0.01, False,
|
||||||
|
started_two +
|
||||||
|
timedelta(
|
||||||
|
seconds=30), "USD->BTC->ETH->USD", 5.0, 4.0, 0.04, True,
|
||||||
|
)
|
||||||
|
await conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO orders (
|
||||||
|
trade_ref, order_ref, leg_index, pair, side, volume,
|
||||||
|
user_ref, status, filled_volume, avg_price, raw_response, recorded_at
|
||||||
|
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12),
|
||||||
|
($13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24)
|
||||||
|
""",
|
||||||
|
"trade-1", "order-1", 0, "BTC/USD", "buy", 2.0, 101, "closed", 2.0, 100.0, "{}", started,
|
||||||
|
"trade-2", "order-2", 0, "ETH/USD", "sell", 4.0, 202, "closed", 3.0, 200.0, "{}", started_two,
|
||||||
|
)
|
||||||
|
|
||||||
|
metrics = await MetricsCalculator(store).compute()
|
||||||
|
|
||||||
|
assert metrics.realized_pnl_usd == 8.0
|
||||||
|
assert metrics.win_rate == 0.5
|
||||||
|
assert metrics.avg_trade_duration_seconds == 60.0
|
||||||
|
assert metrics.opportunities_per_minute == 2.0
|
||||||
|
assert metrics.fill_rate == 0.875
|
||||||
|
assert metrics.latency_p50_seconds == 60.0
|
||||||
|
assert metrics.latency_p95_seconds == 87.0
|
||||||
|
assert metrics.latency_p99_seconds == pytest.approx(89.4)
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from arbitrade.config.settings import get_settings
|
||||||
|
from arbitrade.detection.engine import OpportunityEvent
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
from arbitrade.storage.opportunities import AsyncOpportunityWriter
|
||||||
|
from arbitrade.storage.repositories import OpportunityRepository
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.integration
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _pg() -> AsyncIterator[PgStore]:
|
||||||
|
s = get_settings()
|
||||||
|
store = PgStore(s)
|
||||||
|
try:
|
||||||
|
await store.start()
|
||||||
|
await store.migrate()
|
||||||
|
yield store
|
||||||
|
finally:
|
||||||
|
await store.stop()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_async_opportunity_writer_persists_events() -> None:
|
||||||
|
async with _pg() as store:
|
||||||
|
repository = OpportunityRepository(store)
|
||||||
|
writer = AsyncOpportunityWriter(repository, max_queue_size=10)
|
||||||
|
await writer.start()
|
||||||
|
|
||||||
|
event = OpportunityEvent(
|
||||||
|
detected_at=datetime.now(UTC),
|
||||||
|
cycle="USD->BTC->ETH->USD",
|
||||||
|
updated_pair="BTC/USD",
|
||||||
|
gross_rate=1.04,
|
||||||
|
net_rate=1.03,
|
||||||
|
gross_pct=4.0,
|
||||||
|
net_pct=3.0,
|
||||||
|
est_profit=0.03,
|
||||||
|
)
|
||||||
|
|
||||||
|
await writer.enqueue(event)
|
||||||
|
await writer.stop()
|
||||||
|
|
||||||
|
async with store.pool.acquire() as conn:
|
||||||
|
rows = await conn.fetch(
|
||||||
|
"SELECT cycle, gross_pct, net_pct, est_profit, executed FROM opportunities"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(rows) == 1
|
||||||
|
assert rows[0]["cycle"] == "USD->BTC->ETH->USD"
|
||||||
|
assert rows[0]["gross_pct"] == 4.0
|
||||||
|
assert rows[0]["net_pct"] == 3.0
|
||||||
|
assert rows[0]["est_profit"] == 0.03
|
||||||
|
assert rows[0]["executed"] is False
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
"""Integration tests: verify PostgreSQL schema and connection.
|
||||||
|
|
||||||
|
These tests connect to the PostgreSQL server at 192.168.88.35 and
|
||||||
|
validate that all expected tables, columns, and constraints exist.
|
||||||
|
They are skipped if the server is unreachable.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import AsyncIterator
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pytest_asyncio
|
||||||
|
|
||||||
|
from arbitrade.config.settings import Settings, get_settings
|
||||||
|
from arbitrade.storage.pg_store import PgStore
|
||||||
|
|
||||||
|
pytestmark = pytest.mark.integration
|
||||||
|
|
||||||
|
|
||||||
|
# ── expected schema ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
EXPECTED_TABLES: dict[str, list[str]] = {
|
||||||
|
"schema_migrations": ["version", "applied_at"],
|
||||||
|
"config_sections": ["id", "name", "description", "updated_at"],
|
||||||
|
"config_settings": [
|
||||||
|
"key", "section", "value_json", "value_type", "is_secret",
|
||||||
|
"is_runtime_reloadable", "updated_at", "updated_by",
|
||||||
|
],
|
||||||
|
"config_pairings": [
|
||||||
|
"id", "base_asset", "quote_asset", "enabled", "source",
|
||||||
|
"created_at", "updated_at",
|
||||||
|
],
|
||||||
|
"config_backtesting_defaults": [
|
||||||
|
"id", "starting_balances", "trade_capital", "min_profit_threshold",
|
||||||
|
"slippage_bps", "execution_latency_ms", "fee_source",
|
||||||
|
],
|
||||||
|
"opportunities": [
|
||||||
|
"id", "detected_at", "cycle", "gross_pct", "net_pct",
|
||||||
|
"est_profit", "executed",
|
||||||
|
],
|
||||||
|
"trades": [
|
||||||
|
"id", "trade_ref", "started_at", "finished_at", "status",
|
||||||
|
"realized_pnl", "estimated_pnl", "capital_used", "cycle", "leg_count",
|
||||||
|
],
|
||||||
|
"orders": [
|
||||||
|
"id", "trade_ref", "order_ref", "leg_index", "pair", "side",
|
||||||
|
"volume", "user_ref", "status", "filled_volume", "avg_price",
|
||||||
|
"raw_response", "recorded_at",
|
||||||
|
],
|
||||||
|
"pnl_events": [
|
||||||
|
"id", "trade_ref", "recorded_at", "kind", "pnl_usd", "source",
|
||||||
|
],
|
||||||
|
"portfolio_snapshots": ["snapshot_at", "balances", "total_value_usd"],
|
||||||
|
"market_snapshots": ["snapshot_at", "symbol", "source", "payload", "latency_ms"],
|
||||||
|
"audit_events": [
|
||||||
|
"id", "occurred_at", "actor", "event_type", "decision",
|
||||||
|
"payload", "correlation_id",
|
||||||
|
],
|
||||||
|
"runtime_state_snapshots": [
|
||||||
|
"snapshot_at", "is_running", "kill_switch_active", "kill_switch_reason",
|
||||||
|
"open_trade_count", "last_known_balances", "note",
|
||||||
|
],
|
||||||
|
"kraken_account_snapshots": [
|
||||||
|
"snapshot_at", "fee_tier", "maker_fee", "taker_fee",
|
||||||
|
"thirty_day_volume", "trade_balance_raw", "fee_schedule_raw",
|
||||||
|
],
|
||||||
|
"backtest_jobs": [
|
||||||
|
"id", "status", "events_path", "config", "report", "error",
|
||||||
|
"created_at", "started_at", "finished_at",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Tables that should have a primary key
|
||||||
|
TABLES_WITH_PRIMARY_KEY: dict[str, str | list[str]] = {
|
||||||
|
"schema_migrations": "version",
|
||||||
|
"config_sections": "id",
|
||||||
|
"config_settings": "key",
|
||||||
|
"config_pairings": "id",
|
||||||
|
"config_backtesting_defaults": "id",
|
||||||
|
"opportunities": "id",
|
||||||
|
"trades": "id",
|
||||||
|
"orders": "id",
|
||||||
|
"pnl_events": "id",
|
||||||
|
"audit_events": "id",
|
||||||
|
"backtest_jobs": "id",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Tables with a UNIQUE constraint beyond the primary key
|
||||||
|
TABLES_WITH_UNIQUE_CONSTRAINTS: dict[str, list[str]] = {
|
||||||
|
"config_sections": ["name"],
|
||||||
|
"config_pairings": ["base_asset, quote_asset"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ── fixtures ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def _pg_lifecycle() -> AsyncIterator[PgStore]:
|
||||||
|
"""Connect, yield store, then disconnect."""
|
||||||
|
settings = get_settings()
|
||||||
|
store = PgStore(settings)
|
||||||
|
try:
|
||||||
|
await store.start()
|
||||||
|
yield store
|
||||||
|
finally:
|
||||||
|
await store.stop()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest_asyncio.fixture(name="pg")
|
||||||
|
async def pg_fixture() -> AsyncIterator[PgStore]:
|
||||||
|
async with _pg_lifecycle() as store:
|
||||||
|
yield store
|
||||||
|
|
||||||
|
|
||||||
|
# ── helpers ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
async def _get_actual_tables(store: PgStore) -> dict[str, list[str]]:
|
||||||
|
"""Return {table_name: [column_name, ...]} for the public schema."""
|
||||||
|
actual: dict[str, list[str]] = {}
|
||||||
|
async with store.pool.acquire() as conn:
|
||||||
|
rows = await conn.fetch(
|
||||||
|
"SELECT table_name, column_name FROM information_schema.columns "
|
||||||
|
"WHERE table_schema = 'public' ORDER BY table_name, ordinal_position"
|
||||||
|
)
|
||||||
|
for row in rows:
|
||||||
|
tbl: str = row["table_name"]
|
||||||
|
col: str = row["column_name"]
|
||||||
|
actual.setdefault(tbl, []).append(col)
|
||||||
|
return actual
|
||||||
|
|
||||||
|
|
||||||
|
async def _table_row_count(store: PgStore, table: str) -> int:
|
||||||
|
async with store.pool.acquire() as conn:
|
||||||
|
row = await conn.fetchrow(f"SELECT COUNT(*) AS cnt FROM {table}")
|
||||||
|
return int(row["cnt"]) if row else 0
|
||||||
|
|
||||||
|
|
||||||
|
# ── tests ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_pg_connect(pg: PgStore) -> None:
|
||||||
|
"""Can connect to PostgreSQL and ping the server."""
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
val = await conn.fetchval("SELECT 1 AS val")
|
||||||
|
assert val == 1
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_pgcrypto_extension(pg: PgStore) -> None:
|
||||||
|
"""The pgcrypto extension is available (gen_random_uuid)."""
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
val = await conn.fetchval("SELECT gen_random_uuid()")
|
||||||
|
assert val is not None
|
||||||
|
# The result should be a UUID object
|
||||||
|
assert len(str(val)) == 36 # UUID string length
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_schema_migration_applies(pg: PgStore) -> None:
|
||||||
|
"""Migrate creates all expected tables."""
|
||||||
|
await pg.migrate()
|
||||||
|
actual = await _get_actual_tables(pg)
|
||||||
|
|
||||||
|
for table in EXPECTED_TABLES:
|
||||||
|
assert table in actual, (
|
||||||
|
f"Table '{table}' missing after migration. "
|
||||||
|
f"Found tables: {sorted(actual)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_migration_is_idempotent(pg: PgStore) -> None:
|
||||||
|
"""Running migrate twice does not raise."""
|
||||||
|
await pg.migrate()
|
||||||
|
await pg.migrate() # second call should be a no-op
|
||||||
|
actual = await _get_actual_tables(pg)
|
||||||
|
for table in EXPECTED_TABLES:
|
||||||
|
assert table in actual
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_table_columns(pg: PgStore) -> None:
|
||||||
|
"""Every expected table has the correct columns."""
|
||||||
|
await pg.migrate()
|
||||||
|
actual = await _get_actual_tables(pg)
|
||||||
|
|
||||||
|
for table, expected_cols in EXPECTED_TABLES.items():
|
||||||
|
actual_cols = actual.get(table, [])
|
||||||
|
for col in expected_cols:
|
||||||
|
assert col in actual_cols, (
|
||||||
|
f"Column '{col}' missing from table '{table}'. "
|
||||||
|
f"Actual columns: {actual_cols}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_primary_keys(pg: PgStore) -> None:
|
||||||
|
"""Tables that should have primary keys do."""
|
||||||
|
await pg.migrate()
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
for table, expected_pk in TABLES_WITH_PRIMARY_KEY.items():
|
||||||
|
rows = await conn.fetch(
|
||||||
|
"SELECT kcu.column_name FROM information_schema.table_constraints tc "
|
||||||
|
"JOIN information_schema.key_column_usage kcu "
|
||||||
|
"ON tc.constraint_name = kcu.constraint_name "
|
||||||
|
"WHERE tc.table_schema = 'public' AND tc.table_name = $1 "
|
||||||
|
"AND tc.constraint_type = 'PRIMARY KEY' "
|
||||||
|
"ORDER BY kcu.ordinal_position",
|
||||||
|
table,
|
||||||
|
)
|
||||||
|
pk_columns = [r["column_name"] for r in rows]
|
||||||
|
expected_list = [expected_pk] if isinstance(expected_pk, str) else expected_pk
|
||||||
|
for col in expected_list:
|
||||||
|
assert col in pk_columns, (
|
||||||
|
f"Table '{table}' should have PK column '{col}'. "
|
||||||
|
f"Actual PK columns: {pk_columns}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_unique_constraints(pg: PgStore) -> None:
|
||||||
|
"""Tables that should have UNIQUE constraints do."""
|
||||||
|
await pg.migrate()
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
for table, expected_ucs in TABLES_WITH_UNIQUE_CONSTRAINTS.items():
|
||||||
|
rows = await conn.fetch(
|
||||||
|
"SELECT kcu.column_name FROM information_schema.table_constraints tc "
|
||||||
|
"JOIN information_schema.key_column_usage kcu "
|
||||||
|
"ON tc.constraint_name = kcu.constraint_name "
|
||||||
|
"WHERE tc.table_schema = 'public' AND tc.table_name = $1 "
|
||||||
|
"AND tc.constraint_type = 'UNIQUE'",
|
||||||
|
table,
|
||||||
|
)
|
||||||
|
uc_columns = {r["column_name"] for r in rows}
|
||||||
|
for expected_cols in expected_ucs:
|
||||||
|
cols = [c.strip() for c in expected_cols.split(",")]
|
||||||
|
for col in cols:
|
||||||
|
assert col in uc_columns, (
|
||||||
|
f"Table '{table}' should have UNIQUE column '{col}'. "
|
||||||
|
f"Actual UNIQUE columns: {uc_columns}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_table_row_count_is_zero(pg: PgStore) -> None:
|
||||||
|
"""All tables start empty after migration."""
|
||||||
|
await pg.migrate()
|
||||||
|
for table in EXPECTED_TABLES:
|
||||||
|
count = await _table_row_count(pg, table)
|
||||||
|
assert count == 0, (
|
||||||
|
f"Table '{table}' should be empty after migration, "
|
||||||
|
f"but has {count} rows"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_schema_migration_version_recorded(pg: PgStore) -> None:
|
||||||
|
"""schema_migrations has the expected version after migrate."""
|
||||||
|
from arbitrade.storage.pg_store import SCHEMA_VERSION
|
||||||
|
|
||||||
|
await pg.migrate()
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
row = await conn.fetchrow(
|
||||||
|
"SELECT MAX(version) AS v FROM schema_migrations"
|
||||||
|
)
|
||||||
|
assert row is not None
|
||||||
|
assert row["v"] == SCHEMA_VERSION, (
|
||||||
|
f"Expected schema version {SCHEMA_VERSION}, "
|
||||||
|
f"got {row['v']}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_create_and_query_row(pg: PgStore) -> None:
|
||||||
|
"""Can INSERT a row and SELECT it back (round-trip for a simple table)."""
|
||||||
|
await pg.migrate()
|
||||||
|
async with pg.pool.acquire() as conn:
|
||||||
|
# ConfigSections round-trip
|
||||||
|
await conn.execute(
|
||||||
|
"INSERT INTO config_sections (name, description) VALUES ($1, $2)",
|
||||||
|
"test_section", "A test section for integration test",
|
||||||
|
)
|
||||||
|
row = await conn.fetchrow(
|
||||||
|
"SELECT name, description FROM config_sections WHERE name = $1",
|
||||||
|
"test_section",
|
||||||
|
)
|
||||||
|
assert row is not None
|
||||||
|
assert row["name"] == "test_section"
|
||||||
|
assert row["description"] == "A test section for integration test"
|
||||||
|
|
||||||
|
# Clean up
|
||||||
|
await conn.execute(
|
||||||
|
"DELETE FROM config_sections WHERE name = $1",
|
||||||
|
"test_section",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_config_pairings_upsert(pg: PgStore) -> None:
|
||||||
|
"""ON CONFLICT ... DO UPDATE works on config_pairings (unique constraint)."""
|
||||||
|
await pg.migrate()
|
||||||
|
from arbitrade.config.service import ConfigPairing
|
||||||
|
from arbitrade.storage.repositories import ConfigPairingRepository
|
||||||
|
|
||||||
|
repo = ConfigPairingRepository(pg)
|
||||||
|
|
||||||
|
# Insert
|
||||||
|
p1 = await repo.upsert_pairing(
|
||||||
|
ConfigPairing(base_asset="XBT", quote_asset="USD", enabled=True, source="kraken")
|
||||||
|
)
|
||||||
|
assert p1.id is not None
|
||||||
|
assert p1.base_asset == "XBT"
|
||||||
|
assert p1.enabled is True
|
||||||
|
|
||||||
|
# Upsert (update)
|
||||||
|
p2 = await repo.upsert_pairing(
|
||||||
|
ConfigPairing(base_asset="XBT", quote_asset="USD", enabled=False, source="manual")
|
||||||
|
)
|
||||||
|
assert p2.id == p1.id # same row
|
||||||
|
assert p2.enabled is False
|
||||||
|
assert p2.source == "manual"
|
||||||
|
|
||||||
|
# Clean up
|
||||||
|
deleted = await repo.delete_pairing("XBT", "USD")
|
||||||
|
assert deleted is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_audit_list_recent(pg: PgStore) -> None:
|
||||||
|
"""AuditRepository.list_recent returns records in desc order."""
|
||||||
|
await pg.migrate()
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from arbitrade.storage.repositories import AuditRecord, AuditRepository
|
||||||
|
|
||||||
|
repo = AuditRepository(pg)
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
|
||||||
|
# Insert a few records
|
||||||
|
for i in range(3):
|
||||||
|
await repo.insert(
|
||||||
|
AuditRecord(
|
||||||
|
occurred_at=now,
|
||||||
|
actor="test",
|
||||||
|
event_type="integration_test",
|
||||||
|
decision=f"decision_{i}",
|
||||||
|
payload={"index": i},
|
||||||
|
correlation_id=f"corr_{i}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
recent = await repo.list_recent(limit=5)
|
||||||
|
assert len(recent) >= 3
|
||||||
|
assert recent[0].decision in ("decision_2", "decision_1", "decision_0")
|
||||||
|
# Verify payload serialization worked
|
||||||
|
first = recent[0]
|
||||||
|
if first.payload:
|
||||||
|
assert "index" in first.payload
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import UTC, datetime
|
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
|
||||||
from arbitrade.storage.db import DuckDBStore
|
|
||||||
from arbitrade.storage.repositories import AuditRecord, AuditRepository
|
|
||||||
|
|
||||||
|
|
||||||
def test_audit_repository_inserts_and_lists_recent(tmp_path) -> None:
|
|
||||||
settings = Settings(_env_file=None, DUCKDB_PATH=tmp_path / "audit.duckdb")
|
|
||||||
store = DuckDBStore(settings)
|
|
||||||
store.migrate()
|
|
||||||
repository = AuditRepository(store)
|
|
||||||
|
|
||||||
repository.insert(
|
|
||||||
AuditRecord(
|
|
||||||
occurred_at=datetime.now(UTC),
|
|
||||||
actor="dashboard_user",
|
|
||||||
event_type="dashboard.control.start",
|
|
||||||
decision="approved",
|
|
||||||
payload={"execution_status": "running"},
|
|
||||||
correlation_id="req-1",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
recent = repository.list_recent(limit=5)
|
|
||||||
|
|
||||||
assert len(recent) == 1
|
|
||||||
assert recent[0].actor == "dashboard_user"
|
|
||||||
assert recent[0].event_type == "dashboard.control.start"
|
|
||||||
assert recent[0].decision == "approved"
|
|
||||||
assert recent[0].payload == {"execution_status": "running"}
|
|
||||||
assert recent[0].correlation_id == "req-1"
|
|
||||||
@@ -8,7 +8,7 @@ from arbitrade.config.service import (
|
|||||||
ConfigPairing,
|
ConfigPairing,
|
||||||
ConfigSetting,
|
ConfigSetting,
|
||||||
)
|
)
|
||||||
from arbitrade.storage.db import DuckDBStore
|
from arbitrade.storage.pg_store import PgStore
|
||||||
from arbitrade.storage.repositories import (
|
from arbitrade.storage.repositories import (
|
||||||
ConfigBacktestingDefaultsRepository,
|
ConfigBacktestingDefaultsRepository,
|
||||||
ConfigPairingRepository,
|
ConfigPairingRepository,
|
||||||
@@ -19,7 +19,7 @@ from arbitrade.storage.repositories import (
|
|||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def mock_store():
|
def mock_store():
|
||||||
"""Create a mock database store."""
|
"""Create a mock database store."""
|
||||||
store = Mock(spec=DuckDBStore)
|
store = Mock(spec=PgStore)
|
||||||
return store
|
return store
|
||||||
|
|
||||||
|
|
||||||
@@ -244,7 +244,8 @@ def test_config_pairing_repository_create_pairing(mock_store):
|
|||||||
]
|
]
|
||||||
|
|
||||||
# Create pairing
|
# Create pairing
|
||||||
pairing = ConfigPairing(base_asset="BTC", quote_asset="USD", enabled=True, source="Kraken")
|
pairing = ConfigPairing(
|
||||||
|
base_asset="BTC", quote_asset="USD", enabled=True, source="Kraken")
|
||||||
|
|
||||||
result = repo.create_pairing(pairing)
|
result = repo.create_pairing(pairing)
|
||||||
|
|
||||||
|
|||||||
@@ -1,89 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from datetime import UTC, datetime
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
|
||||||
from arbitrade.detection.engine import OpportunityEvent
|
|
||||||
from arbitrade.execution.sequencer import TriangularExecutionSequencer
|
|
||||||
from arbitrade.storage.db import DuckDBStore
|
|
||||||
from arbitrade.storage.executions import AsyncExecutionWriter
|
|
||||||
from arbitrade.storage.repositories import OrderRepository, PnLRepository, TradeRepository
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(slots=True)
|
|
||||||
class _FakeRestClient:
|
|
||||||
calls: int = 0
|
|
||||||
|
|
||||||
async def place_market_order(self, *, pair: str, side: str, volume: float) -> dict[str, object]:
|
|
||||||
self.calls += 1
|
|
||||||
return {"txid": [f"tx-{self.calls}"], "status": "submitted"}
|
|
||||||
|
|
||||||
|
|
||||||
def _sample_event() -> OpportunityEvent:
|
|
||||||
return OpportunityEvent(
|
|
||||||
detected_at=datetime.now(UTC),
|
|
||||||
cycle="USD->BTC->ETH->USD",
|
|
||||||
updated_pair="BTC/USD",
|
|
||||||
gross_rate=1.04,
|
|
||||||
net_rate=1.03,
|
|
||||||
gross_pct=4.0,
|
|
||||||
net_pct=3.0,
|
|
||||||
est_profit=0.03,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_execution_writer_persists_trade_order_and_pnl(tmp_path) -> None:
|
|
||||||
settings = Settings(_env_file=None, DUCKDB_PATH=tmp_path / "exec.duckdb")
|
|
||||||
store = DuckDBStore(settings)
|
|
||||||
store.migrate()
|
|
||||||
writer = AsyncExecutionWriter(
|
|
||||||
TradeRepository(store),
|
|
||||||
OrderRepository(store),
|
|
||||||
PnLRepository(store),
|
|
||||||
max_queue_size=10,
|
|
||||||
)
|
|
||||||
await writer.start()
|
|
||||||
|
|
||||||
client = _FakeRestClient()
|
|
||||||
sequencer = TriangularExecutionSequencer(
|
|
||||||
client,
|
|
||||||
available_pairs=["BTC/USD", "ETH/BTC", "ETH/USD"],
|
|
||||||
execution_writer=writer,
|
|
||||||
)
|
|
||||||
|
|
||||||
result = await sequencer.execute(_sample_event())
|
|
||||||
await writer.stop()
|
|
||||||
|
|
||||||
assert result.success
|
|
||||||
assert client.calls == 3
|
|
||||||
|
|
||||||
with store.connect() as conn:
|
|
||||||
trades = conn.execute(
|
|
||||||
"SELECT trade_ref, status, estimated_pnl, capital_used, cycle, leg_count FROM trades"
|
|
||||||
).fetchall()
|
|
||||||
orders = conn.execute(
|
|
||||||
"SELECT trade_ref, order_ref, leg_index, pair, side, volume, status "
|
|
||||||
"FROM orders ORDER BY leg_index"
|
|
||||||
).fetchall()
|
|
||||||
pnls = conn.execute("SELECT trade_ref, kind, pnl_usd, source FROM pnl_events").fetchall()
|
|
||||||
|
|
||||||
assert len(trades) == 1
|
|
||||||
assert trades[0][1] == "filled"
|
|
||||||
assert trades[0][2] == 0.03
|
|
||||||
assert trades[0][3] == 1.0
|
|
||||||
assert trades[0][4] == "USD->BTC->ETH->USD"
|
|
||||||
assert trades[0][5] == 3
|
|
||||||
|
|
||||||
assert len(orders) == 3
|
|
||||||
assert orders[0][2] == 0
|
|
||||||
assert orders[1][2] == 1
|
|
||||||
assert orders[2][2] == 2
|
|
||||||
assert orders[0][6] == "submitted"
|
|
||||||
|
|
||||||
assert len(pnls) == 1
|
|
||||||
assert pnls[0][1] == "estimated"
|
|
||||||
assert pnls[0][2] == 0.03
|
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import UTC, datetime, timedelta
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
|
||||||
from arbitrade.metrics import MetricsCalculator
|
|
||||||
from arbitrade.storage.db import DuckDBStore
|
|
||||||
|
|
||||||
|
|
||||||
def test_metrics_calculator_summarizes_execution_data(tmp_path) -> None:
|
|
||||||
settings = Settings(_env_file=None, DUCKDB_PATH=tmp_path / "metrics.duckdb")
|
|
||||||
store = DuckDBStore(settings)
|
|
||||||
store.migrate()
|
|
||||||
|
|
||||||
started = datetime.now(UTC)
|
|
||||||
finished = started + timedelta(seconds=30)
|
|
||||||
started_two = started + timedelta(minutes=1)
|
|
||||||
finished_two = started_two + timedelta(seconds=90)
|
|
||||||
|
|
||||||
with store.connect() as conn:
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO trades (
|
|
||||||
trade_ref,
|
|
||||||
started_at,
|
|
||||||
finished_at,
|
|
||||||
status,
|
|
||||||
realized_pnl,
|
|
||||||
estimated_pnl,
|
|
||||||
capital_used,
|
|
||||||
cycle,
|
|
||||||
leg_count
|
|
||||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?), (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
||||||
""",
|
|
||||||
[
|
|
||||||
"trade-1",
|
|
||||||
started,
|
|
||||||
finished,
|
|
||||||
"filled",
|
|
||||||
12.5,
|
|
||||||
10.0,
|
|
||||||
100.0,
|
|
||||||
"USD->BTC->ETH->USD",
|
|
||||||
3,
|
|
||||||
"trade-2",
|
|
||||||
started_two,
|
|
||||||
finished_two,
|
|
||||||
"filled",
|
|
||||||
-4.5,
|
|
||||||
-2.0,
|
|
||||||
200.0,
|
|
||||||
"USD->ETH->BTC->USD",
|
|
||||||
3,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO opportunities (
|
|
||||||
detected_at,
|
|
||||||
cycle,
|
|
||||||
gross_pct,
|
|
||||||
net_pct,
|
|
||||||
est_profit,
|
|
||||||
executed
|
|
||||||
) VALUES (?, ?, ?, ?, ?, ?), (?, ?, ?, ?, ?, ?), (?, ?, ?, ?, ?, ?)
|
|
||||||
""",
|
|
||||||
[
|
|
||||||
started,
|
|
||||||
"USD->BTC->ETH->USD",
|
|
||||||
4.0,
|
|
||||||
3.0,
|
|
||||||
0.03,
|
|
||||||
True,
|
|
||||||
started_two,
|
|
||||||
"USD->ETH->BTC->USD",
|
|
||||||
2.0,
|
|
||||||
1.0,
|
|
||||||
0.01,
|
|
||||||
False,
|
|
||||||
started_two + timedelta(seconds=30),
|
|
||||||
"USD->BTC->ETH->USD",
|
|
||||||
5.0,
|
|
||||||
4.0,
|
|
||||||
0.04,
|
|
||||||
True,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
conn.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO orders (
|
|
||||||
trade_ref,
|
|
||||||
order_ref,
|
|
||||||
leg_index,
|
|
||||||
pair,
|
|
||||||
side,
|
|
||||||
volume,
|
|
||||||
user_ref,
|
|
||||||
status,
|
|
||||||
filled_volume,
|
|
||||||
avg_price,
|
|
||||||
raw_response,
|
|
||||||
recorded_at
|
|
||||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?), (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
|
||||||
""",
|
|
||||||
[
|
|
||||||
"trade-1",
|
|
||||||
"order-1",
|
|
||||||
0,
|
|
||||||
"BTC/USD",
|
|
||||||
"buy",
|
|
||||||
2.0,
|
|
||||||
101,
|
|
||||||
"closed",
|
|
||||||
2.0,
|
|
||||||
100.0,
|
|
||||||
"{}",
|
|
||||||
started,
|
|
||||||
"trade-2",
|
|
||||||
"order-2",
|
|
||||||
0,
|
|
||||||
"ETH/USD",
|
|
||||||
"sell",
|
|
||||||
4.0,
|
|
||||||
202,
|
|
||||||
"closed",
|
|
||||||
3.0,
|
|
||||||
200.0,
|
|
||||||
"{}",
|
|
||||||
started_two,
|
|
||||||
],
|
|
||||||
)
|
|
||||||
|
|
||||||
metrics = MetricsCalculator(store).compute()
|
|
||||||
|
|
||||||
assert metrics.realized_pnl_usd == 8.0
|
|
||||||
assert metrics.win_rate == 0.5
|
|
||||||
assert metrics.avg_trade_duration_seconds == 60.0
|
|
||||||
assert metrics.opportunities_per_minute == 2.0
|
|
||||||
assert metrics.fill_rate == 0.875
|
|
||||||
assert metrics.latency_p50_seconds == 60.0
|
|
||||||
assert metrics.latency_p95_seconds == 87.0
|
|
||||||
assert metrics.latency_p99_seconds == pytest.approx(89.4)
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import UTC, datetime
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from arbitrade.config.settings import Settings
|
|
||||||
from arbitrade.detection.engine import OpportunityEvent
|
|
||||||
from arbitrade.storage.db import DuckDBStore
|
|
||||||
from arbitrade.storage.opportunities import AsyncOpportunityWriter
|
|
||||||
from arbitrade.storage.repositories import OpportunityRepository
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
|
||||||
async def test_async_opportunity_writer_persists_events(tmp_path) -> None:
|
|
||||||
settings = Settings(_env_file=None, DUCKDB_PATH=tmp_path / "test.duckdb")
|
|
||||||
store = DuckDBStore(settings)
|
|
||||||
store.migrate()
|
|
||||||
|
|
||||||
repository = OpportunityRepository(store)
|
|
||||||
writer = AsyncOpportunityWriter(repository, max_queue_size=10)
|
|
||||||
await writer.start()
|
|
||||||
|
|
||||||
event = OpportunityEvent(
|
|
||||||
detected_at=datetime.now(UTC),
|
|
||||||
cycle="USD->BTC->ETH->USD",
|
|
||||||
updated_pair="BTC/USD",
|
|
||||||
gross_rate=1.04,
|
|
||||||
net_rate=1.03,
|
|
||||||
gross_pct=4.0,
|
|
||||||
net_pct=3.0,
|
|
||||||
est_profit=0.03,
|
|
||||||
)
|
|
||||||
|
|
||||||
await writer.enqueue(event)
|
|
||||||
await writer.stop()
|
|
||||||
|
|
||||||
with store.connect() as conn:
|
|
||||||
rows = conn.execute(
|
|
||||||
"SELECT cycle, gross_pct, net_pct, est_profit, executed FROM opportunities"
|
|
||||||
).fetchall()
|
|
||||||
|
|
||||||
assert len(rows) == 1
|
|
||||||
assert rows[0][0] == "USD->BTC->ETH->USD"
|
|
||||||
assert rows[0][1] == 4.0
|
|
||||||
assert rows[0][2] == 3.0
|
|
||||||
assert rows[0][3] == 0.03
|
|
||||||
assert rows[0][4] is False
|
|
||||||
Reference in New Issue
Block a user