calminer-docs/userguide/data_import_export.md

Data Import & Export Templates

This guide explains how to capture profitability, capex, and opex data for bulk upload or download using the standardized CSV and Excel templates. It builds on the field inventory documented in data_import_export_field_inventory.md and should be shared with anyone preparing data for the calculation engine.

Template Package

The import/export toolkit contains two artifacts:

File	Purpose
calminer_financials.csv	Single-sheet CSV template for quick edits or integrations that generate structured rows dynamically.
calminer_financials.xlsx	Excel workbook with dedicated sheets, lookup lists, and validation rules for guided data capture.

Always distribute the templates as a matched set so contributors can choose the format that best suits their workflow.

CSV Workflow

Structure Overview

The CSV template uses a record_type column to multiplex multiple logical tables inside one file. The exact column requirements, validation ranges, and record types are defined in the Unified CSV Template Specification. Key expectations:

• All rows include the scenario code (scenario_code), and optionally the project code, to keep capex and opex data aligned with profitability assumptions.
• Numeric cells must use . as the decimal separator and omit thousands separators, currency symbols, or formatting.
• Boolean values accept true/false, yes/no, or 1/0; exporters should emit true or false.

Authoring Steps

1. Populate at least one profitability_input row per scenario to establish the calculation context.
2. Add optional profitability_impurity rows where penalty modelling is required.
3. Capture capital costs via capex_component rows and configure options with a capex_parameters record.
4. Capture recurring costs via processing_opex_component rows. Add a processing_opex_parameters row to define escalation settings.
5. Validate the file against the rules listed in the specification prior to upload. Recommended tooling includes frictionless data checks or schema-enforced pipelines.

Naming & Delivery

• Use the pattern calminer_financials_<project-or-scenario-code>_<YYYYMMDD>.csv when transferring files between teams.
• Provide a brief change log or summary per file so reviewers understand the scope of updates.

Excel Workflow

Workbook Layout

The Excel workbook contains structured sheets described in the Excel Workbook Layout & Validation Rules section. Highlights:

• The Summary sheet captures metadata (scenario, preparer, notes) that propagates into the downstream sheets via Excel Table references.
• Each data sheet (Profitability_Input, Capex_Components, Processing_Opex, etc.) is formatted as an Excel Table with frozen headers and data validation to enforce numeric ranges, lookup values, and boolean lists.
• The Lookups sheet holds named ranges for currencies, categories, and frequency values. Keep this sheet hidden to reduce accidental edits.

Data Entry Checklist

• Confirm the Summary sheet lists every scenario included in the workbook before editing dependent sheets.
• Use the provided dropdowns for category, currency, frequency, and boolean fields to avoid invalid values.
• When copying data from other workbooks, paste values only to preserve validation rules.
• Turn on the totals row for component tables to quickly verify aggregate capex and opex amounts.

Exporting From Excel

When exporting to CSV or ingesting into the API:

• Ensure the workbook is saved in .xlsx format; do not rename the sheets.
• If a CSV export is required, export each sheet separately and merge using the column headers defined in the CSV specification.
• Retain the original workbook as your source of truth; conversions to CSV should be treated as transient files for automation pipelines.

Import Pipeline Expectations

The planned FastAPI import endpoints will perform the following checks:

• Validate record types, required columns, and numeric bounds according to the schemas described in the field inventory.
• Verify that referenced project/scenario codes exist and that foreign key relationships (components, parameters) target a single scenario within the file.
• Normalize casing for category slugs and currency codes to match the calculation services.
• Reject files that mix multiple projects unless intentionally enabled by configuration.

Until the endpoints are available, internal teams can use the specification as the contract for pre-validation scripts or interim ETL routines.

Exporting Data

Export functionality will mirror the same structures:

• CSV exports will group rows by record_type, enabling direct round-tripping with minimal transformation.
• Excel exports will populate the same workbook sheets, preserving lookup lists and validation to support offline edits.

Document any local transformations or additional columns introduced during export so that downstream consumers can reconcile the data with the import templates.

Change Management

• Increment the import_version value on the Summary sheet (and document the change here) whenever template-breaking updates occur.
• Update data_import_export_field_inventory.md first when adding or removing columns, then revise this guide to explain the user-facing workflow impact.
• Communicate updates to stakeholders and archive superseded templates to avoid drift.

Stakeholder Review Checklist

Before finalising the templates for production use, circulate the following package to domain stakeholders (finance, engineering, data operations) and collect their feedback:

• calminer_financials.csv sample populated with representative profitability, capex, and opex data.
• calminer_financials.xlsx workbook showcasing all sheets, dropdowns, and validation rules.
• Links to documentation: this guide and the field inventory.
• Summary of outstanding questions or assumptions (e.g., default currencies, additional categories).

Record the review session outcomes in the project tracker or meeting notes. Once sign-off is received, mark the stakeholder feedback step as complete in .github/instructions/TODO.md and proceed with implementation of import/export endpoints.