Running the Application

All four services must be running simultaneously. Use a separate terminal for each.

Services

1. Redis

redis-server

Required by RQ (job queue) and session management.

2. Prefect Server + Worker

bash start_prefect.sh

Starts the Prefect API server and worker on port 4201.

3. RQ Background Worker

./venv/bin/python worker.py

Processes webhook and Fineract jobs from the Redis queue.

4. Quart Web Server

./venv/bin/python app.py

Runs the async web server on port 5002.

Running Prefect Flows Manually

When running Prefect flows from a separate terminal, set the API URL first:

export PREFECT_API_URL=http://127.0.0.1:4201/api

S3 Processing

# All lenders
./venv/bin/python tasks/flows/s3_processing.py

# Specific lender
./venv/bin/python tasks/flows/s3_processing.py arthmate
./venv/bin/python tasks/flows/s3_processing.py ugro_razorpay

Repayment Pipeline

# Standard run (includes dbt refresh)
./venv/bin/python tasks/flows/webhook_repayment_pipeline.py

# Skip dbt (if you just ran it)
./venv/bin/python tasks/flows/webhook_repayment_pipeline.py --skip-dbt

# Dry run - inspect queue without posting
./venv/bin/python tasks/flows/webhook_repayment_pipeline.py --dry-run

Running dbt

All dbt commands must be run from the dbt/ directory:

cd dbt/

# Run all models
./venv/bin/dbt run

# Run specific model
./venv/bin/dbt run --select 'base_ugro_razorpay_api'

# Run tests
./venv/bin/dbt test

Or from the project root:

./venv/bin/dbt run --project-dir dbt/ --profiles-dir ~/.dbt/

Logging

Process	Log Location
Web server	logs/YYYY-MM-DD/app.log
RQ worker	logs/YYYY-MM-DD/worker.log
S3 upload	s3://turnofile/logs/YYYY-MM-DD/

After each S3 processing flow, logs are automatically uploaded to S3.