# Data Import & Export Templates

This guide explains how to capture profitability, capex, and opex data for bulk upload or download using the standardized CSV and Excel templates. It builds on the field inventory documented in [data_import_export_field_inventory.md](data_import_export_field_inventory.md) and should be shared with anyone preparing data for the calculation engine.

## Template Package

The import/export toolkit contains two artifacts:

| File                       | Purpose                                                                                              |
| -------------------------- | ---------------------------------------------------------------------------------------------------- |
| `calminer_financials.csv`  | Single-sheet CSV template for quick edits or integrations that generate structured rows dynamically. |
| `calminer_financials.xlsx` | Excel workbook with dedicated sheets, lookup lists, and validation rules for guided data capture.    |

Always distribute the templates as a matched set so contributors can choose the format that best suits their workflow.

## CSV Workflow

### Structure Overview

The CSV template uses a `record_type` column to multiplex multiple logical tables inside one file. The exact column requirements, validation ranges, and record types are defined in the [Unified CSV Template Specification](data_import_export_field_inventory.md#unified-csv-template-specification). Key expectations:

- All rows include the scenario code (`scenario_code`), and optionally the project code, to keep capex and opex data aligned with profitability assumptions.
- Numeric cells must use `.` as the decimal separator and omit thousands separators, currency symbols, or formatting.
- Boolean values accept `true`/`false`, `yes`/`no`, or `1`/`0`; exporters should emit `true` or `false`.

### Authoring Steps

1. Populate at least one `profitability_input` row per scenario to establish the calculation context.
2. Add optional `profitability_impurity` rows where penalty modelling is required.
3. Capture capital costs via `capex_component` rows and configure options with a `capex_parameters` record.
4. Capture recurring costs via `processing_opex_component` rows. Add a `processing_opex_parameters` row to define escalation settings.
5. Validate the file against the rules listed in the specification prior to upload. Recommended tooling includes frictionless data checks or schema-enforced pipelines.

### Naming & Delivery

- Use the pattern `calminer_financials_<project-or-scenario-code>_<YYYYMMDD>.csv` when transferring files between teams.
- Provide a brief change log or summary per file so reviewers understand the scope of updates.

## Excel Workflow

### Workbook Layout

The Excel workbook contains structured sheets described in the [Excel Workbook Layout & Validation Rules](data_import_export_field_inventory.md#excel-workbook-layout--validation-rules) section. Highlights:

- The `Summary` sheet captures metadata (scenario, preparer, notes) that propagates into the downstream sheets via Excel Table references.
- Each data sheet (`Profitability_Input`, `Capex_Components`, `Processing_Opex`, etc.) is formatted as an Excel Table with frozen headers and data validation to enforce numeric ranges, lookup values, and boolean lists.
- The `Lookups` sheet holds named ranges for currencies, categories, and frequency values. Keep this sheet hidden to reduce accidental edits.

### Data Entry Checklist

- Confirm the `Summary` sheet lists every scenario included in the workbook before editing dependent sheets.
- Use the provided dropdowns for category, currency, frequency, and boolean fields to avoid invalid values.
- When copying data from other workbooks, paste values only to preserve validation rules.
- Turn on the totals row for component tables to quickly verify aggregate capex and opex amounts.

### Exporting From Excel

When exporting to CSV or ingesting into the API:

- Ensure the workbook is saved in `.xlsx` format; do not rename the sheets.
- If a CSV export is required, export each sheet separately and merge using the column headers defined in the CSV specification.
- Retain the original workbook as your source of truth; conversions to CSV should be treated as transient files for automation pipelines.

## Import Pipeline Expectations

The planned FastAPI import endpoints will perform the following checks:

- Validate record types, required columns, and numeric bounds according to the schemas described in the field inventory.
- Verify that referenced project/scenario codes exist and that foreign key relationships (components, parameters) target a single scenario within the file.
- Normalize casing for category slugs and currency codes to match the calculation services.
- Reject files that mix multiple projects unless intentionally enabled by configuration.

Until the endpoints are available, internal teams can use the specification as the contract for pre-validation scripts or interim ETL routines.

## Exporting Data

Export functionality will mirror the same structures:

- CSV exports will group rows by `record_type`, enabling direct round-tripping with minimal transformation.
- Excel exports will populate the same workbook sheets, preserving lookup lists and validation to support offline edits.

Document any local transformations or additional columns introduced during export so that downstream consumers can reconcile the data with the import templates.

## Change Management

- Increment the `import_version` value on the `Summary` sheet (and document the change here) whenever template-breaking updates occur.
- Update `data_import_export_field_inventory.md` first when adding or removing columns, then revise this guide to explain the user-facing workflow impact.
- Communicate updates to stakeholders and archive superseded templates to avoid drift.

## Stakeholder Review Checklist

Before finalising the templates for production use, circulate the following package to domain stakeholders (finance, engineering, data operations) and collect their feedback:

- `calminer_financials.csv` sample populated with representative profitability, capex, and opex data.
- `calminer_financials.xlsx` workbook showcasing all sheets, dropdowns, and validation rules.
- Links to documentation: this guide and the [field inventory](data_import_export_field_inventory.md).
- Summary of outstanding questions or assumptions (e.g., default currencies, additional categories).

Record the review session outcomes in the project tracker or meeting notes. Once sign-off is received, mark the stakeholder feedback step as complete in `.github/instructions/TODO.md` and proceed with implementation of import/export endpoints.