arbitrade/README.md

# Arbitrade

Low-latency cryptocurrency arbitrage bot scaffold for Kraken.

Current stack:

- Python 3.12+
- FastAPI + HTMX/Jinja2
- DuckDB for dev/test/prod
- Native Kraken WebSocket planned for market-data hot path
- Gitea Actions + Gitea container registry

Project plan lives in [PLAN.md](PLAN.md).
Task checklist lives in [.github/instructions/TODO.md](.github/instructions/TODO.md).

## Current Status

Bootstrap complete for foundation layer:

- repo initialized
- typed settings and env loading
- structured logging
- encrypted secret helpers
- DuckDB connection + base schema
- FastAPI app with health endpoint
- Gitea Actions CI scaffold
- Docker / docker-compose scaffold

Not implemented yet:

- Kraken REST client
- Kraken native WebSocket client
- arbitrage detection engine
- trade execution
- dashboard beyond health/bootstrap page

## Prerequisites

- Python 3.12+
- `uv` for env/package management
- Git
- Docker Desktop or Docker Engine
- Gitea account on `git.allucanget.biz` for push/CI/registry access

Optional:

- PowerShell 7 on Windows

## Repository Setup

Clone repo:

```powershell
git clone https://git.allucanget.biz/allucanget/arbitrade.git
Set-Location arbitrade
```

If repo already exists locally, confirm remote:

```powershell
git remote -v
```

Expected origin:

```text
https://git.allucanget.biz/allucanget/arbitrade.git
```

## Local Development Setup

Create virtualenv with `uv`:

```powershell
uv venv
```

Activate env on Windows:

```powershell
.\.venv\Scripts\Activate.ps1
```

Install app + dev dependencies:

```powershell
uv pip install -e .[dev]
```

Dependency source of truth:

- Runtime dependencies live in `requirements/latest-runtime.in`.
- Dev dependencies live in `requirements/latest-dev.in`.
- `pyproject.toml` reads both files dynamically during package install.

Create local env file:

```powershell
Copy-Item .env.example .env
```

Minimum `.env` values:

```env
APP_ENV=dev
APP_HOST=0.0.0.0
APP_PORT=9090
LOG_LEVEL=INFO
LOG_JSON=true
DUCKDB_PATH=./data/arbitrade.duckdb
FERNET_KEY=
KRAKEN_API_KEY=
KRAKEN_API_SECRET=
KRAKEN_API_KEY_PERMISSIONS=query,trade
```

Notes:

- Leave Kraken creds empty until Kraken integration lands.
- If Kraken creds are set, both key and secret are required.
- `KRAKEN_API_KEY_PERMISSIONS` must include `query,trade` and must not include withdrawal scope.
- `FERNET_KEY` optional. If empty, keyring-backed key generation used by secret helper.
- On Windows, app falls back to default `asyncio` loop. On non-Windows, `uvloop` installs automatically.

## Run App

Start app:

```powershell
python -m arbitrade.main
```

Health endpoints:

- HTML: `http://localhost:9090/`
- JSON: `http://localhost:9090/health`

## Database

DuckDB used everywhere: local dev, tests, production.

Default database file:

```text
./data/arbitrade.duckdb
```

Schema bootstrap runs automatically on app startup.

Current tables:

- `schema_migrations`
- `opportunities`
- `trades`
- `portfolio_snapshots`

Audit trail table:

- `audit_events` (append-only operational decision log)

Audit retention and compaction guidance:

- Keep at least 30 days of `audit_events` in active DB for incident triage.
- Archive older rows to a timestamped export file before deletion.
- Example monthly archive workflow:

```sql
COPY (
  SELECT *
  FROM audit_events
  WHERE occurred_at < NOW() - INTERVAL 30 DAY
) TO 'data/audit_events_archive_YYYYMM.parquet' (FORMAT PARQUET);

DELETE FROM audit_events
WHERE occurred_at < NOW() - INTERVAL 30 DAY;
```

- Back up archive files and the main DuckDB file together.
- For production, run archive + backup as scheduled maintenance (cron/task scheduler).

## Quality Checks

Run tests:

```powershell
pytest -q
```

Run Ruff:

```powershell
ruff check .
```

Run Black check:

```powershell
black --check .
```

Run mypy:

```powershell
mypy src
```

Run dependency vulnerability audit:

```powershell
pip-audit -r requirements/latest-runtime.in
```

Run secret scan (worktree + git history):

```powershell
python scripts/security_scan.py
```

Generate latency profile baseline:

```powershell
python scripts/profile_latency.py --iterations 600 --output ops/performance/latency_baseline.json
```

Run latency regression guardrails:

```powershell
python scripts/check_latency_regression.py --baseline ops/performance/latency_baseline.json --thresholds ops/performance/latency_thresholds.json --iterations 600
```

Install pre-commit hooks:

```powershell
pre-commit install
```

Run hooks manually:

```powershell
pre-commit run --all-files
```

## Docker

Build locally:

```powershell
docker build -t arbitrade:local .
```

Container dependency install flow:

- Docker installs runtime dependencies from `requirements/latest-runtime.in`.
- Docker then installs the package with `--no-deps` so dependency resolution is driven by requirements files.

Run with compose:

```powershell
docker compose up --build
```

Compose mounts local `data/` folder into container at `/app/data`.

Important:

- [docker-compose.yml](docker-compose.yml) uses `git.allucanget.biz/allucanget/arbitrade:latest` as the default image reference.

## Coolify Deployment (Prebuilt Image)

Use this when deploying from the image published by CI instead of building from Git inside Coolify.

### 1) Create application in Coolify

- In Coolify, create a new `Application` using `Docker Image` / `Public Image` / `Private Registry Image`.
- Image: `git.allucanget.biz/allucanget/arbitrade:latest`
- Registry: `git.allucanget.biz`
- If registry auth is required, configure the same registry credentials in Coolify.

### 2) Configure build and start behavior

Set these in Coolify application settings:

- Build Command: leave empty.
- Install Command: leave empty.
- Start Command: leave empty unless you explicitly want to override the image default.
- Port: `9090` (coolify uses `8000` internally)

### 3) Configure health check and networking

- Health Check Path: `/health`
- Exposed Port: `9090`
- Use Coolify-generated domain or attach your own domain.

### 4) Configure persistent storage

Add a persistent volume in Coolify:

- Mount Path: `/app/data`

This preserves DuckDB and other runtime artifacts across restarts/redeploys.

### 5) Configure environment variables

Add runtime environment variables in Coolify (UI: Environment Variables):

- `APP_ENV=prod`
- `APP_HOST=0.0.0.0`
- `APP_PORT=9090`
- `DUCKDB_PATH=/app/data/arbitrade.duckdb`
- `LOG_LEVEL=INFO`
- `LOG_JSON=true`
- `KRAKEN_API_KEY=...`
- `KRAKEN_API_SECRET=...`
- `KRAKEN_API_KEY_PERMISSIONS=query,trade`

Recommended:

- Configure `FERNET_KEY` in Coolify secrets (do not commit it).
- Keep all exchange keys/secrets in Coolify secret variables only.

Coolify should own runtime configuration through environment variables. CI only publishes the image.

### 6) Deploy and verify

- Trigger deploy in Coolify after CI publishes `git.allucanget.biz/allucanget/arbitrade:latest`.
- Verify app boot logs show startup completed.
- Verify `GET /health` returns success on deployed URL.

## Gitea CI / Registry Setup

CI file:

- [.gitea/workflows/ci.yml](.gitea/workflows/ci.yml)

Required Gitea Actions secrets:

- `REGISTRY_USERNAME`
- `REGISTRY_TOKEN`

Example registry login:

```powershell
docker login git.allucanget.biz
```

Example pushed image tag shape:

```text
git.allucanget.biz/allucanget/arbitrade:latest
```

## Architecture Docs

Implementation detail moved into arc42 docs:

- [arc42 overview](docs/architecture/arc42.md) - system context, building blocks, runtime, deployment, quality goals, risks.
- [current implementation snapshot](docs/architecture/current-implementation.md) - codebase state, active routes, backtesting, strategy flags, deployment flow.

For navigation from README, use the docs above instead of this file for deep architecture detail.