S3 File Processing
Automated ingestion of lender CSV files from S3 into PostgreSQL RAW tables.
How It Works
Idempotent Processing
Every S3 file is tracked in RAW_S3_FILE_PROCESSING_LOG. Status transitions:
pending → processing → success | failedFiles with status=success are automatically skipped on subsequent runs.
Lender Registry
The S3_LENDER_CONFIGS dict in tasks/flows/s3_processing.py defines all lender configurations:
| Key | S3 Prefix | RAW Table |
|---|---|---|
arthmate | arthmate/ | RAW_ARTHMATE_NACH_STATUS_RERT |
dmi | dmi/ | RAW_DMI_NACH_STATUS_REPORT |
dmi_payment | dmi_payment/ | RAW_DMI_PAYMENT_REPORT |
gb | gb/ | RAW_GB_NACH_PAYMENT_REPORT |
ikf | ikf/ | RAW_IKF_NACH_STATUS_REPORT |
shivalik | shivalik/ | RAW_SHIVALIK_NACH_STATUS_REPORT |
shivalik_payment | shivalik_payment/ | RAW_SHIVALIK_PAYMENT_REPORT |
ugro | ugro/ | RAW_UGRO_NACH_PAYMENT_REPORT |
ugro_razorpay | ugro_razorpay/ | RAW_UGRO_RAZORPAY_NACH_STATUS_REPORT |
vivriti | vivriti/ | RAW_VIVRITI_PAYMENT_REPORT |
vivriti_nach_status | vivriti_nach_status/ | RAW_VIVRITI_PAYMENT_NACH_STATUS_REPORT |
Running the Flow
bash
export PREFECT_API_URL=http://127.0.0.1:4201/api
# All lenders
./venv/bin/python tasks/flows/s3_processing.py
# Specific lender
./venv/bin/python tasks/flows/s3_processing.py arthmate
./venv/bin/python tasks/flows/s3_processing.py ugro
./venv/bin/python tasks/flows/s3_processing.py dmi_paymentSource File Column
The source_file column is injected by utils/csv_processor.py into every CSV load. This column is critical for dbt's incremental filtering - dbt skips rows from files already present in the target table.
TIP
The source_file column must exist in the RAW table for dbt incremental models to work correctly.
Lender-Product Matrix
| Lender | Product ID | NACH CSV | Payment CSV | Razorpay | Cashfree |
|---|---|---|---|---|---|
| UGRO | 1 | Yes | - | Yes | Yes |
| DMI | 2 | Yes | Yes | - | - |
| ARTHMATE | 3 | Yes | - | - | - |
| SHIVALIK | 4 | Yes | Yes | - | - |
| VIVRITI | 5 | Yes | Yes | Yes | - |
| GB | 6 | Yes | - | - | Yes |
| IKF | 7 | Yes | - | - | - |