arbitrade/README.md

# Arbitrade

Low-latency cryptocurrency arbitrage bot scaffold for Kraken.

Current stack:

- Python 3.12+
- FastAPI + HTMX/Jinja2
- PostgreSQL for all environments (via asyncpg)
- Native Kraken WebSocket planned for market-data hot path
- Gitea Actions + Gitea container registry

Project plan lives in [PLAN.md](PLAN.md).
Task checklist lives in [.github/instructions/TODO.md](.github/instructions/TODO.md).
Coolify deployment runbooks live in [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md).

## Current Status

Bootstrap complete for foundation layer:

- repo initialized
- typed settings and env loading
- structured logging
- encrypted secret helpers
- PostgreSQL connection + full schema migration
- FastAPI app with health endpoint
- Gitea Actions CI scaffold
- Docker / docker-compose scaffold

Not implemented yet:

- Kraken REST client
- Kraken native WebSocket client
- arbitrage detection engine
- trade execution
- dashboard beyond health/bootstrap page

## Configuration Management

The arbitrage trading bot now includes a complete configuration management system that allows users to configure trading behavior, currency pairings, fees, and other application settings through a web interface. All user-configurable settings are persisted in the database while system variables remain in environment variables as per the settings split plan.

Key features include:

- Web-based configuration interface at `/dashboard/config/`
- Runtime hot-reloading of configuration changes
- Complete CRUD operations for all configuration entities
- Input validation and error handling
- Audit logging for all configuration changes
- Backtesting parameter configuration
- Fee configuration by pairing and market type

## Templates

Full page templates (`src/arbitrade/web/templates/`):

| Template           | Route                    | Purpose                                                 |
| ------------------ | ------------------------ | ------------------------------------------------------- |
| `base.html`        | — (root layout)          | Dark theme, `.shell` container, HTMX, CSS variables     |
| `dashboard.html`   | `/`, `/dashboard`        | Main dashboard: metrics, overview, controls, charts     |
| `config.html`      | `/dashboard/config`      | Full configuration: fees, runtime, alerts, Kraken, risk |
| `audit.html`       | `/dashboard/audit`       | Audit trail with auto-refresh via HTMX                  |
| `backtesting.html` | `/dashboard/backtesting` | Backtesting panel with replay/sweep forms               |
| `health.html`      | `/health`                | System health check                                     |

Dashboard partials (`src/arbitrade/web/templates/partials/`):

| Partial                  | In page          | Content                                                                                             |
| ------------------------ | ---------------- | --------------------------------------------------------------------------------------------------- |
| `metrics.html`           | Dashboard        | 6 KPI cards: P&L, win rate, avg duration, trade count, success %, profit factor                     |
| `overview.html`          | Dashboard        | Status, balances, fee tier, open trades list, opportunity feed                                      |
| `controls.html`          | Dashboard        | Runtime status, kill switch, config snapshot, alerting status, execution controls (Start/Stop/Kill) |
| `charts.html`            | Dashboard        | Opportunity trend chart (Chart.js, Alpine toggle)                                                   |
| `config.html`            | Config page      | Config form: Runtime, Alerts, Kraken, Risk, Strategy sections                                       |
| `config_fees.html`       | Config page      | Pair fee table + add/edit form                                                                      |
| `backtesting_panel.html` | Backtesting page | Run status, replay/sweep forms, recent runs                                                         |
| `audit.html`             | Audit page       | Audit trail table: time, actor, event, decision, payload                                            |

Legacy templates (`src/arbitrade/web/templates/dashboard/`):

- `config_settings.html`, `config_pairs.html`, `config_fees.html` — superseded by config page; retained for reference

## Prerequisites

- Python 3.12+
- `uv` for env/package management
- Git
- Docker Desktop or Docker Engine
- Gitea account on `git.allucanget.biz` for push/CI/registry access

Optional:

- PowerShell 7 on Windows

## Repository Setup

Clone repo:

```powershell
git clone https://git.allucanget.biz/allucanget/arbitrade.git
Set-Location arbitrade
```

If repo already exists locally, confirm remote:

```powershell
git remote -v
```

Expected origin:

```text
https://git.allucanget.biz/allucanget/arbitrade.git
```

## Local Development Setup

Create virtualenv with `uv`:

```powershell
uv venv
```

Activate env on Windows:

```powershell
.\.venv\Scripts\Activate.ps1
```

Install app + dev dependencies:

```powershell
uv pip install -e .[dev]
```

Dependency source of truth:

- Runtime dependencies live in `requirements/latest-runtime.in`.
- Dev dependencies live in `requirements/latest-dev.in`.
- `pyproject.toml` reads both files dynamically during package install.

Create local env file:

```powershell
Copy-Item .env.example .env
```

Minimum `.env` values:

```env
APP_ENV=dev
APP_HOST=0.0.0.0
APP_PORT=9090
LOG_LEVEL=INFO
LOG_JSON=true
PG_HOST=192.168.88.35
PG_PORT=5432
PG_DATABASE=arbitrade
PG_USER=arbitrade
PG_PASSWORD=arbitrade
FERNET_KEY=
KRAKEN_API_KEY=
KRAKEN_API_SECRET=
KRAKEN_API_KEY_PERMISSIONS=query,trade
```

Notes:

- Leave Kraken creds empty until Kraken integration lands.
- If Kraken creds are set, both key and secret are required.
- `KRAKEN_API_KEY_PERMISSIONS` must include `query,trade` and must not include withdrawal scope.
- `FERNET_KEY` optional. If empty, keyring-backed key generation used by secret helper.
- On Windows, app falls back to default `asyncio` loop. On non-Windows, `uvloop` installs automatically.

## Run App

Start app:

```powershell
python -m arbitrade.main
```

Health endpoints:

- HTML: `http://localhost:9090/`
- JSON: `http://localhost:9090/health`

## Database

PostgreSQL used everywhere: local dev, tests, production.

Default connection:

```text
PG_HOST=192.168.88.35
PG_PORT=5432
PG_DATABASE=arbitrade
PG_USER=arbitrade
PG_PASSWORD=arbitrade
```

Schema bootstrap runs automatically on app startup via `PgStore.migrate()`.

Current tables:

- `schema_migrations`
- `opportunities`
- `trades`
- `portfolio_snapshots`

Audit trail table:

- `audit_events` (append-only operational decision log)

Audit retention and compaction guidance:

- Keep at least 30 days of `audit_events` in active DB for incident triage.
- Archive older rows to a timestamped export file before deletion.
- Example monthly archive workflow:

```sql
COPY (
  SELECT *
  FROM audit_events
  WHERE occurred_at < NOW() - INTERVAL 30 DAY
) TO 'data/audit_events_archive_YYYYMM.parquet' (FORMAT PARQUET);

DELETE FROM audit_events
WHERE occurred_at < NOW() - INTERVAL 30 DAY;
```

- Back up archive files and the PostgreSQL database together.
- For production, run archive + backup as scheduled maintenance (cron/task scheduler).

## Quality Checks

Run tests:

```powershell
pytest -q
```

Run Ruff:

```powershell
ruff check .
```

Run Black check:

```powershell
black --check .
```

Run mypy:

```powershell
mypy src
```

Run dependency vulnerability audit:

```powershell
pip-audit -r requirements/latest-runtime.in
```

Run secret scan (worktree + git history):

```powershell
python scripts/security_scan.py
```

Generate latency profile baseline:

```powershell
python scripts/profile_latency.py --iterations 600 --output ops/performance/latency_baseline.json
```

Run latency regression guardrails:

```powershell
python scripts/check_latency_regression.py --baseline ops/performance/latency_baseline.json --thresholds ops/performance/latency_thresholds.json --iterations 600
```

Install pre-commit hooks:

```powershell
pre-commit install
```

Run hooks manually:

```powershell
pre-commit run --all-files
```

## Docker

Build locally:

```powershell
docker build -t arbitrade:local .
```

Container dependency install flow:

- Docker installs runtime dependencies from `requirements/latest-runtime.in`.
- Docker then installs the package with `--no-deps` so dependency resolution is driven by requirements files.

Run with compose:

```powershell
docker compose up --build
```

Compose mounts local `data/` folder into container at `/app/data`.

Important:

- [docker-compose.yml](docker-compose.yml) uses `git.allucanget.biz/allucanget/arbitrade:latest` as the default image reference.

## Coolify Deployment (Prebuilt Image)

Use this when deploying from the image published by CI instead of building from Git inside Coolify.

### 1) Create application in Coolify

- In Coolify, create a new `Application` using `Docker Image` / `Public Image` / `Private Registry Image`.
- Image: `git.allucanget.biz/allucanget/arbitrade:latest`
- Registry: `git.allucanget.biz`
- If registry auth is required, configure the same registry credentials in Coolify.

### 2) Configure build and start behavior

Set these in Coolify application settings:

- Build Command: leave empty.
- Install Command: leave empty.
- Start Command: leave empty unless you explicitly want to override the image default.
- Port: `9090` (coolify uses `8000` internally)

### 3) Configure health check and networking

- Health Check Path: `/health`
- Exposed Port: `9090`
- Use Coolify-generated domain or attach your own domain.

### 4) Configure persistent storage

Add a persistent volume in Coolify:

- Mount Path: `/app/data`

This preserves PostgreSQL data and other runtime artifacts across restarts/redeploys.

### 5) Configure environment variables

Add runtime environment variables in Coolify (UI: Environment Variables):

- `APP_ENV=prod`
- `APP_HOST=0.0.0.0`
- `APP_PORT=9090`
- `PG_HOST=postgres`
  `PG_PORT=5432`
  `PG_DATABASE=arbitrade`
  `PG_USER=arbitrade`
  `PG_PASSWORD=arbitrade`
- `LOG_LEVEL=INFO`
- `LOG_JSON=true`
- `KRAKEN_API_KEY=...`
- `KRAKEN_API_SECRET=...`
- `KRAKEN_API_KEY_PERMISSIONS=query,trade`

Recommended:

- Configure `FERNET_KEY` in Coolify secrets (do not commit it).
- Keep all exchange keys/secrets in Coolify secret variables only.

Coolify should own runtime configuration through environment variables. CI only publishes the image.

### 6) Deploy and verify

- Trigger deploy in Coolify after CI publishes `git.allucanget.biz/allucanget/arbitrade:latest`.
- Verify app boot logs show startup completed.
- Verify `GET /health` returns success on deployed URL.

## Gitea CI / Registry Setup

CI file:

- [.gitea/workflows/ci.yml](.gitea/workflows/ci.yml)

Required Gitea Actions secrets:

- `REGISTRY_USERNAME`
- `REGISTRY_TOKEN`

Example registry login:

```powershell
docker login git.allucanget.biz
```

Example pushed image tag shape:

```text
git.allucanget.biz/allucanget/arbitrade:latest
```

## Architecture Docs

Implementation detail moved into docs:

- [architecture overview](docs/architecture/README.md) - system context, building blocks, runtime, deployment, quality goals, risks.
- [current implementation snapshot](docs/architecture/current-implementation.md) - codebase state, active routes, backtesting, strategy flags, deployment flow.

For navigation from README, use the docs above instead of this file for deep architecture detail.