calminer-docs/admin/installation.md

Installation Guide

This document provides step-by-step instructions for installing Calminer on your system. Please follow the steps below to ensure a successful installation.

Deployment Options

Calminer can be deployed using Docker containers, which simplifies the installation process and ensures consistency across different environments. The recommended deployment option is to use Docker Compose to manage the multi-container setup.

Docker Installation

Prerequisites

Before you begin, ensure that you have the following prerequisites installed on your system:

• Docker
• Docker Compose
• Git

Steps

1. Clone the Repository

   Open your terminal and run the following command to clone the Calminer repository:

   git clone https://git.allucanget.biz/allucanget/calminer.git
   cd calminer
   

2. Environment Configuration

   Copy the appropriate environment file for your deployment:

   # For development
   cp .env.development .env
   
   # For staging
   cp .env.staging .env
   
   # For production
   cp .env.production .env
   # Then edit .env with your actual database credentials
   

3. Build and Start the Docker Containers

   Run the following command to build and start the Docker containers:

   # For development (includes live reload and source mounting)
   docker compose up --build
   
   # For staging
   docker compose -f docker-compose.yml -f docker-compose.staging.yml up --build
   
   # For production
   docker compose -f docker-compose.yml -f docker-compose.prod.yml up --build
   

   This command will build the Docker images and start the containers as defined in the docker-compose.yml file.

4. Access the Application

   Once the containers are up and running, you can access the Calminer application by navigating to http://localhost:8003 in your web browser. If you are running the application on a remote server, replace localhost with the server's IP address or domain name.

5. Database Initialization & Seed Data

   On container startup the FastAPI application executes scripts.init_db during the first startup event. This idempotent initializer performs the following steps:

   • ensures PostgreSQL enum types exist (no duplicates are created),
   • creates the core tables required for authentication, pricing, projects, scenarios, financial inputs, and simulation parameters,
   • seeds default roles, admin user, pricing settings, and demo projects/scenarios/financial inputs using INSERT ... ON CONFLICT DO NOTHING semantics.

   No additional action is required when using Docker Compose—the uvicorn process runs the initializer automatically before the bootstrap routines.

   For local development without Docker, run the initializer manually after exporting your environment variables:

   # activate your virtualenv first
   python -m scripts.init_db
   

   Running the script multiple times is safe; it will reconcile records without duplicating data.

6. Resetting the Schema (optional)

   To rebuild the schema from a clean slate in development, use the reset utility which drops the managed tables and enum types before rerunning init_db:

   # activate your virtualenv first
   CALMINER_ENV=development python -m scripts.reset_db
   python -m scripts.init_db
   

   The reset utility refuses to run when CALMINER_ENV indicates a production or staging environment, providing an extra safeguard against accidental destructive operations.

Export Dependencies

Export and monitoring workflows require the following Python packages in addition to the core dependencies:

• pandas
• openpyxl
• prometheus-client

These libraries are already listed in requirements.txt. Ensure they are installed in your virtual environment if you are not using Docker.

Environment Variables for Export Features

While exports reuse the existing database configuration, you may optionally set the following variables to adjust behavior:

• CALMINER_EXPORT_MAX_ROWS — override default pagination when generating exports (optional).
• CALMINER_EXPORT_METADATA — enable (true) or disable (false) the metadata sheet in Excel exports by default (UI form still allows per-request overrides).

Set these variables in your .env file or compose environment section before launching the stack.

Docker Configuration

The docker-compose.yml file contains the configuration for the Calminer application, including the services, networks, and volumes. You can customize the configuration as needed to suit your deployment environment.

Environment Variables

The application uses environment variables to configure various settings. You can set these variables in a .env file in the root directory of the project. Refer to the provided .env.* files for examples and default values.

Key variables relevant to import/export workflows:

Variable	Development	Staging	Production	Description
CALMINER_EXPORT_MAX_ROWS	1000	50000	100000	Optional safety guard to limit the number of rows exported in a single request.
CALMINER_EXPORT_METADATA	true	true	true	Controls whether metadata sheets are generated by default during Excel exports.
CALMINER_IMPORT_STAGING_TTL	300	600	3600	Controls how long staged import tokens remain valid before expiration.
CALMINER_IMPORT_MAX_ROWS	10000	50000	100000	Optional guard to prevent excessively large import files.

Docker Environment Parity

The Docker Compose configurations ensure environment parity across development, staging, and production:

• Development: Uses docker-compose.override.yml with live code reloading, debug logging, and relaxed resource limits.
• Staging: Uses docker-compose.staging.yml with health checks, moderate resource limits, and staging-specific configurations.
• Production: Uses docker-compose.prod.yml with strict resource limits, production logging, and required external database configuration.

All environments use the same base docker-compose.yml and share common environment variables for consistency.

Volumes

The application uses Docker volumes to persist data. The following volumes are defined in the docker-compose.yml file:

• calminer_data: Used to persist application data.
• calminer_db_data: Used to persist database data.

Ensure that these volumes are properly configured to avoid data loss during container restarts or removals.

Kubernetes Deployment

For production deployments, Calminer can be deployed on a Kubernetes cluster using the provided manifests in the k8s/ directory.

K8s Prerequisites

• Kubernetes cluster (e.g., minikube for local testing, or cloud provider like GKE, EKS)
• kubectl configured to access the cluster
• Helm (optional, for advanced deployments)

K8s Deployment Steps

1. Clone the Repository and Build Image

   git clone https://git.allucanget.biz/allucanget/calminer.git
   cd calminer
   docker build -t registry.example.com/calminer:latest .
   docker push registry.example.com/calminer:latest
   

2. Update Manifests

   Edit the manifests in k8s/ to match your environment:

   • Update image registry in deployment.yaml
   • Update host in ingress.yaml
   • Update secrets in secret.yaml with base64 encoded values

3. Deploy to Kubernetes

   kubectl apply -f k8s/
   

4. Verify Deployment

   kubectl get pods
   kubectl get services
   kubectl get ingress
   

5. Access the Application

   The application will be available at the ingress host (e.g., https://calminer.example.com).

Environment Parity

The Kubernetes deployment uses the same environment variables as Docker Compose, ensuring consistency across environments. Secrets are managed via Kubernetes Secrets, and configurations via ConfigMaps.

Scaling

The deployment is configured with 3 replicas for high availability. You can scale as needed:

kubectl scale deployment calminer-app --replicas=5

Monitoring

Ensure your monitoring stack (e.g., Prometheus) scrapes the /metrics endpoint from the service.

CI/CD Pipeline

Calminer uses Gitea Actions for continuous integration and deployment. The CI/CD pipeline is defined in .gitea/workflows/cicache.yml and includes the following stages:

CI Stages

1. Lint: Runs Ruff for Python linting, Black for code formatting, and Bandit for security scanning.
2. Test: Executes the full pytest suite with coverage reporting (80% minimum), using a PostgreSQL service container.
3. Build: Builds Docker images using Buildx and pushes to the Gitea registry on the main branch.

CD Stages

1. Deploy: Deploys to staging or production Kubernetes clusters based on commit messages containing [deploy staging] or [deploy production].

Required Secrets

The following secrets must be configured in your Gitea repository:

• REGISTRY_URL: Gitea registry URL
• REGISTRY_USERNAME: Registry username
• REGISTRY_PASSWORD: Registry password
• STAGING_KUBE_CONFIG: Base64-encoded kubeconfig for staging cluster
• PROD_KUBE_CONFIG: Base64-encoded kubeconfig for production cluster

Deployment Triggers

• Automatic: Images are built and pushed on every push to main or develop branches.
• Staging Deployment: Include [deploy staging] in your commit message to trigger staging deployment.
• Production Deployment: Include [deploy production] in your commit message to trigger production deployment.

Monitoring CI/CD

• View pipeline status in the Gitea Actions tab.
• Test artifacts (coverage, pytest reports) are uploaded for each run.
• Docker build logs are available for troubleshooting build failures.
• Deployment runs publish Kubernetes rollout diagnostics under the deployment-logs artifact (/logs/deployment/), which includes pod listings, deployment manifests, and recent container logs.

Troubleshooting

If you encounter any issues during the installation or deployment process, refer to the following troubleshooting tips:

• Check the Docker and Docker Compose logs for error messages.
• Ensure that all prerequisites are installed and up to date.
• Verify that the ports used by the application are not blocked by firewalls or other applications.
• Consult the Calminer documentation and community forums for additional support.