262 lines
10 KiB
Markdown
262 lines
10 KiB
Markdown
# Installation Guide
|
|
|
|
This document provides step-by-step instructions for installing Calminer on your system. Please follow the steps below to ensure a successful installation.
|
|
|
|
## Deployment Options
|
|
|
|
Calminer can be deployed using Docker containers, which simplifies the installation process and ensures consistency across different environments. The recommended deployment option is to use Docker Compose to manage the multi-container setup.
|
|
|
|
## Docker Installation
|
|
|
|
### Prerequisites
|
|
|
|
Before you begin, ensure that you have the following prerequisites installed on your system:
|
|
|
|
- Docker
|
|
- Docker Compose
|
|
- Git
|
|
|
|
### Steps
|
|
|
|
1. **Clone the Repository**
|
|
|
|
Open your terminal and run the following command to clone the Calminer repository:
|
|
|
|
```bash
|
|
git clone https://git.allucanget.biz/allucanget/calminer.git
|
|
cd calminer
|
|
```
|
|
|
|
2. **Environment Configuration**
|
|
|
|
Copy the appropriate environment file for your deployment:
|
|
|
|
```bash
|
|
# For development
|
|
cp .env.development .env
|
|
|
|
# For staging
|
|
cp .env.staging .env
|
|
|
|
# For production
|
|
cp .env.production .env
|
|
# Then edit .env with your actual database credentials
|
|
```
|
|
|
|
3. **Build and Start the Docker Containers**
|
|
|
|
Run the following command to build and start the Docker containers:
|
|
|
|
```bash
|
|
# For development (includes live reload and source mounting)
|
|
docker compose up --build
|
|
|
|
# For staging
|
|
docker compose -f docker-compose.yml -f docker-compose.staging.yml up --build
|
|
|
|
# For production
|
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build
|
|
```
|
|
|
|
This command will build the Docker images and start the containers as defined in the `docker-compose.yml` file.
|
|
|
|
4. **Access the Application**
|
|
|
|
Once the containers are up and running, you can access the Calminer application by navigating to `http://localhost:8003` in your web browser.
|
|
If you are running the application on a remote server, replace `localhost` with the server's IP address or domain name.
|
|
|
|
5. **Database Initialization & Seed Data**
|
|
|
|
On container startup the FastAPI application executes `scripts.init_db` during the first `startup` event. This idempotent initializer performs the following steps:
|
|
|
|
- ensures PostgreSQL enum types exist (no duplicates are created),
|
|
- creates the core tables required for authentication, pricing, projects, scenarios, financial inputs, and simulation parameters,
|
|
- seeds default roles, admin user, pricing settings, and demo projects/scenarios/financial inputs using `INSERT ... ON CONFLICT DO NOTHING` semantics.
|
|
|
|
No additional action is required when using Docker Compose—the `uvicorn` process runs the initializer automatically before the bootstrap routines.
|
|
|
|
For local development without Docker, run the initializer manually after exporting your environment variables:
|
|
|
|
```bash
|
|
# activate your virtualenv first
|
|
python -m scripts.init_db
|
|
```
|
|
|
|
Running the script multiple times is safe; it will reconcile records without duplicating data.
|
|
|
|
6. **Resetting the Schema (optional)**
|
|
|
|
To rebuild the schema from a clean slate in development, use the reset utility which drops the managed tables and enum types before rerunning `init_db`:
|
|
|
|
```bash
|
|
# activate your virtualenv first
|
|
CALMINER_ENV=development python -m scripts.reset_db
|
|
python -m scripts.init_db
|
|
```
|
|
|
|
The reset utility refuses to run when `CALMINER_ENV` indicates a production or staging environment, providing an extra safeguard against accidental destructive operations.
|
|
|
|
### Export Dependencies
|
|
|
|
Export and monitoring workflows require the following Python packages in addition to the core dependencies:
|
|
|
|
- `pandas`
|
|
- `openpyxl`
|
|
- `prometheus-client`
|
|
|
|
These libraries are already listed in `requirements.txt`. Ensure they are installed in your virtual environment if you are not using Docker.
|
|
|
|
### Environment Variables for Export Features
|
|
|
|
While exports reuse the existing database configuration, you may optionally set the following variables to adjust behavior:
|
|
|
|
- `CALMINER_EXPORT_MAX_ROWS` — override default pagination when generating exports (optional).
|
|
- `CALMINER_EXPORT_METADATA` — enable (`true`) or disable (`false`) the metadata sheet in Excel exports by default (UI form still allows per-request overrides).
|
|
|
|
Set these variables in your `.env` file or compose environment section before launching the stack.
|
|
|
|
## Docker Configuration
|
|
|
|
The `docker-compose.yml` file contains the configuration for the Calminer application, including the services, networks, and volumes. You can customize the configuration as needed to suit your deployment environment.
|
|
|
|
### Environment Variables
|
|
|
|
The application uses environment variables to configure various settings. You can set these variables in a `.env` file in the root directory of the project. Refer to the provided `.env.*` files for examples and default values.
|
|
|
|
Key variables relevant to import/export workflows:
|
|
|
|
| Variable | Development | Staging | Production | Description |
|
|
| ----------------------------- | ----------- | ------- | ---------- | ------------------------------------------------------------------------------- |
|
|
| `CALMINER_EXPORT_MAX_ROWS` | 1000 | 50000 | 100000 | Optional safety guard to limit the number of rows exported in a single request. |
|
|
| `CALMINER_EXPORT_METADATA` | `true` | `true` | `true` | Controls whether metadata sheets are generated by default during Excel exports. |
|
|
| `CALMINER_IMPORT_STAGING_TTL` | `300` | `600` | `3600` | Controls how long staged import tokens remain valid before expiration. |
|
|
| `CALMINER_IMPORT_MAX_ROWS` | `10000` | `50000` | `100000` | Optional guard to prevent excessively large import files. |
|
|
|
|
### Docker Environment Parity
|
|
|
|
The Docker Compose configurations ensure environment parity across development, staging, and production:
|
|
|
|
- **Development**: Uses `docker-compose.override.yml` with live code reloading, debug logging, and relaxed resource limits.
|
|
- **Staging**: Uses `docker-compose.staging.yml` with health checks, moderate resource limits, and staging-specific configurations.
|
|
- **Production**: Uses `docker-compose.prod.yml` with strict resource limits, production logging, and required external database configuration.
|
|
|
|
All environments use the same base `docker-compose.yml` and share common environment variables for consistency.
|
|
|
|
### Volumes
|
|
|
|
The application uses Docker volumes to persist data. The following volumes are defined in the `docker-compose.yml` file:
|
|
|
|
- `calminer_data`: Used to persist application data.
|
|
- `calminer_db_data`: Used to persist database data.
|
|
|
|
Ensure that these volumes are properly configured to avoid data loss during container restarts or removals.
|
|
|
|
## Kubernetes Deployment
|
|
|
|
For production deployments, Calminer can be deployed on a Kubernetes cluster using the provided manifests in the `k8s/` directory.
|
|
|
|
### K8s Prerequisites
|
|
|
|
- Kubernetes cluster (e.g., minikube for local testing, or cloud provider like GKE, EKS)
|
|
- kubectl configured to access the cluster
|
|
- Helm (optional, for advanced deployments)
|
|
|
|
### K8s Deployment Steps
|
|
|
|
1. **Clone the Repository and Build Image**
|
|
|
|
```bash
|
|
git clone https://git.allucanget.biz/allucanget/calminer.git
|
|
cd calminer
|
|
docker build -t registry.example.com/calminer:latest .
|
|
docker push registry.example.com/calminer:latest
|
|
```
|
|
|
|
2. **Update Manifests**
|
|
|
|
Edit the manifests in `k8s/` to match your environment:
|
|
|
|
- Update image registry in `deployment.yaml`
|
|
- Update host in `ingress.yaml`
|
|
- Update secrets in `secret.yaml` with base64 encoded values
|
|
|
|
3. **Deploy to Kubernetes**
|
|
|
|
```bash
|
|
kubectl apply -f k8s/
|
|
```
|
|
|
|
4. **Verify Deployment**
|
|
|
|
```bash
|
|
kubectl get pods
|
|
kubectl get services
|
|
kubectl get ingress
|
|
```
|
|
|
|
5. **Access the Application**
|
|
|
|
The application will be available at the ingress host (e.g., `https://calminer.example.com`).
|
|
|
|
### Environment Parity
|
|
|
|
The Kubernetes deployment uses the same environment variables as Docker Compose, ensuring consistency across environments. Secrets are managed via Kubernetes Secrets, and configurations via ConfigMaps.
|
|
|
|
### Scaling
|
|
|
|
The deployment is configured with 3 replicas for high availability. You can scale as needed:
|
|
|
|
```bash
|
|
kubectl scale deployment calminer-app --replicas=5
|
|
```
|
|
|
|
### Monitoring
|
|
|
|
Ensure your monitoring stack (e.g., Prometheus) scrapes the `/metrics` endpoint from the service.
|
|
|
|
## CI/CD Pipeline
|
|
|
|
Calminer uses Gitea Actions for continuous integration and deployment. The CI/CD pipeline is defined in `.gitea/workflows/cicache.yml` and includes the following stages:
|
|
|
|
### CI Stages
|
|
|
|
1. **Lint**: Runs Ruff for Python linting, Black for code formatting, and Bandit for security scanning.
|
|
2. **Test**: Executes the full pytest suite with coverage reporting (80% minimum), using a PostgreSQL service container.
|
|
3. **Build**: Builds Docker images using Buildx and pushes to the Gitea registry on the main branch.
|
|
|
|
### CD Stages
|
|
|
|
1. **Deploy**: Deploys to staging or production Kubernetes clusters based on commit messages containing `[deploy staging]` or `[deploy production]`.
|
|
|
|
### Required Secrets
|
|
|
|
The following secrets must be configured in your Gitea repository:
|
|
|
|
- `REGISTRY_URL`: Gitea registry URL
|
|
- `REGISTRY_USERNAME`: Registry username
|
|
- `REGISTRY_PASSWORD`: Registry password
|
|
- `STAGING_KUBE_CONFIG`: Base64-encoded kubeconfig for staging cluster
|
|
- `PROD_KUBE_CONFIG`: Base64-encoded kubeconfig for production cluster
|
|
|
|
### Deployment Triggers
|
|
|
|
- **Automatic**: Images are built and pushed on every push to `main` or `develop` branches.
|
|
- **Staging Deployment**: Include `[deploy staging]` in your commit message to trigger staging deployment.
|
|
- **Production Deployment**: Include `[deploy production]` in your commit message to trigger production deployment.
|
|
|
|
### Monitoring CI/CD
|
|
|
|
- View pipeline status in the Gitea Actions tab.
|
|
- Test artifacts (coverage, pytest reports) are uploaded for each run.
|
|
- Docker build logs are available for troubleshooting build failures.
|
|
- Deployment runs publish Kubernetes rollout diagnostics under the `deployment-logs` artifact (`/logs/deployment/`), which includes pod listings, deployment manifests, and recent container logs.
|
|
|
|
## Troubleshooting
|
|
|
|
If you encounter any issues during the installation or deployment process, refer to the following troubleshooting tips:
|
|
|
|
- Check the Docker and Docker Compose logs for error messages.
|
|
- Ensure that all prerequisites are installed and up to date.
|
|
- Verify that the ports used by the application are not blocked by firewalls or other applications.
|
|
- Consult the Calminer documentation and community forums for additional support.
|