Payment Reconciliation Design

1. Overview

This document describes the payment reconciliation import functionality for PayFast and PayGate payment processors. The system imports transaction history from CSV exports to verify order payment status and calculate transaction fees.

2. Status

Implementation Status: Planned

Related Architecture: Financial Management Architecture

3. Purpose

3.1. Business Goals

  1. Payment Verification - Confirm orders are correctly marked as paid or unpaid

  2. Fee Tracking - Capture payment processor fees for cost analysis

  3. Audit Trail - Maintain record of all payment processor transactions

  4. Discrepancy Detection - Identify mismatches between orders and payments

3.2. Scope

  • PayFast transaction CSV import

  • PayGate transaction CSV import

  • Order matching by merchant reference

  • Fee calculation and apportionment

  • Duplicate transaction handling

Out of Scope:

  • Automatic payment status correction (flagging only)

  • Real-time webhook-based reconciliation

  • UI for reconciliation review

4. Database Schema

4.1. Entity Inheritance Model

The reconciliation entities use JPA JOINED inheritance strategy:

recon-entities

4.2. Table: recon (Base)

CREATE TABLE recon (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    type VARCHAR(2) NOT NULL,                    -- Discriminator: 'PF', 'PG'

    -- Transaction details
    date DATETIME,
    customer VARCHAR(50),
    currency VARCHAR(3),
    gross DECIMAL(19,2),
    fee DECIMAL(19,2),
    fee_tax DECIMAL(19,2),
    nett DECIMAL(19,2),

    -- Relationships
    order_id BIGINT,                             -- FK to sales_order
    order_system_id BIGINT,                      -- FK to registration_system

    CONSTRAINT fk_recon_order_system
        FOREIGN KEY (order_system_id) REFERENCES registration_system(id)
);

Note: order_id FK constraint is intentionally omitted due to legacy data containing PostIDs that don’t match current Order IDs. Matching is done at application level.

4.3. Table: recon_payfast

CREATE TABLE recon_payfast (
    id BIGINT PRIMARY KEY,                       -- FK to recon.id

    -- PayFast-specific fields
    payfast_payment_id BIGINT NOT NULL,          -- Unique PayFast transaction ID
    merchant_payment_id BIGINT,                  -- Merchant reference (for order matching)
    sign VARCHAR(6) NOT NULL,                    -- CREDIT or DEBIT
    party VARCHAR(50),
    name VARCHAR(50),
    description VARCHAR(100),
    funding_type VARCHAR(25),
    balance DECIMAL(19,2),
    account INT DEFAULT 2,
    custom1 VARCHAR(100),
    custom2 VARCHAR(100),
    custom3 VARCHAR(100),

    CONSTRAINT fk_recon_payfast_recon
        FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE,
    CONSTRAINT uk_payfast_payment_id
        UNIQUE (payfast_payment_id)
);

4.4. Table: recon_pay_gate

CREATE TABLE recon_pay_gate (
    id BIGINT PRIMARY KEY,                       -- FK to recon.id

    -- PayGate-specific fields
    transaction_id BIGINT,                       -- Unique PayGate transaction ID
    batch BIGINT,
    number VARCHAR(20),
    record_type VARCHAR(20),
    transaction_date DATETIME,
    auth_code VARCHAR(6),
    result_code BIGINT,
    result_description VARCHAR(50),
    transaction_source VARCHAR(25),
    original_currency VARCHAR(3),
    original_amount DECIMAL(19,2),
    comment VARCHAR(100),
    product_description VARCHAR(100),

    CONSTRAINT fk_recon_paygate_recon
        FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE,
    CONSTRAINT uk_paygate_transaction_id
        UNIQUE (transaction_id)
);

4.5. Enumerations

4.5.1. ReconType

public enum ReconType {
    PAYFAST("PF", "PayFast"),
    PAYGATE("PG", "PayGate");

    private final String code;   // 2-char discriminator
    private final String name;

    public static ReconType fromCode(String code) {
        // ... lookup by code
    }
}

4.5.2. ReconSign

public enum ReconSign {
    CREDIT("CREDIT"),
    DEBIT("DEBIT");

    private final String code;

    // Stored as varchar(6) in database
}

5. CSV Format Specifications

5.1. PayFast CSV Format

PayFast exports contain 25+ columns:

Column Description Entity Field

Date

Transaction date/time

Recon.date

Type

Transaction type (Settlement, etc.)

ReconPayfast.type (note: different from discriminator)

Sign

CREDIT or DEBIT

ReconPayfast.sign

Party

Customer party info

ReconPayfast.party

Name

Customer name

ReconPayfast.name

Description

Transaction description

ReconPayfast.description

Currency

Currency code (ZAR)

Recon.currency

Funding Type

Payment method

ReconPayfast.fundingType

Gross

Gross amount

Recon.gross

Fee

PayFast fee

Recon.fee

Net

Net amount after fee

Recon.nett

Balance

Running balance

ReconPayfast.balance

M Payment ID

Merchant payment ID (order reference)

ReconPayfast.merchantPaymentId

PF Payment ID

PayFast payment ID (unique)

ReconPayfast.payfastPaymentId

Custom_str1

Custom field 1

ReconPayfast.custom1

Custom_str2

Custom field 2

ReconPayfast.custom2

Custom_str3

Custom field 3

ReconPayfast.custom3

Date Format: yyyy-MM-dd HH:mm:ss

5.2. PayGate CSV Format

PayGate exports contain 18 columns:

Column Description Entity Field

Batch

Batch number

ReconPaygate.batch

Reference

Order reference (format: "12345-12345")

Used for order matching

Number

Card number (masked)

ReconPaygate.number

Customer Name

Customer name

Recon.customer

Amount

Transaction amount

Recon.gross

Type

Settlement, Auth OK, Auth Rejected

ReconPaygate.recordType

Transaction Date

Transaction date/time

ReconPaygate.transactionDate

Date Processed

Processing date

Recon.date

TransID

PayGate transaction ID (unique)

ReconPaygate.transactionId

Auth Code

Authorization code

ReconPaygate.authCode

Result Code

Result code (990018 = success)

ReconPaygate.resultCode

Result Desc

Result description

ReconPaygate.resultDescription

Transaction Source

Source (PayWeb v3)

ReconPaygate.transactionSource

Currency

Currency code

Recon.currency

Orig Amount

Original amount

ReconPaygate.originalAmount

Orig Currency

Original currency

ReconPaygate.originalCurrency

Comment

Comment

ReconPaygate.comment

Product Description

Product description

ReconPaygate.productDescription

Date Format: dd-MMM-yyyy HH:mm (e.g., "14-Jun-2019 23:04")

6. Import Logic

6.1. PayFast Import Service

@Service
public class PayfastReconImportService extends BaseReconImportService {

    public ReconImportResult importCsv(InputStream csvStream, Organisation org) {
        // 1. Parse CSV with SuperCSV
        // 2. For each row:
        //    - Check if payfastPaymentId already exists (skip if duplicate)
        //    - Create ReconPayfast entity
        //    - Set customer from party field
        //    - Match to Order via merchantPaymentId
        //    - Persist entity
        // 3. Return statistics (created, skipped, errors)
    }
}

Key Behaviors:

  • Header mapping uses case-insensitive matching with synonyms

  • Duplicate payfastPaymentId silently skipped (DataIntegrityViolationException caught)

  • Customer field populated from party or custom3

6.2. PayGate Import Service

@Service
public class PaygateReconImportService extends BaseReconImportService {

    private static final BigDecimal FIXED_FEE = new BigDecimal("2.00");
    private static final BigDecimal PERCENT_FEE = new BigDecimal("0.035");
    private static final BigDecimal VAT_RATE = new BigDecimal("0.15");

    public ReconImportResult importCsv(InputStream csvStream, Organisation org) {
        // 1. Parse CSV with SuperCSV
        // 2. For each row:
        //    - Check if transactionId already exists (skip if duplicate)
        //    - Create ReconPaygate entity
        //    - Calculate fees if resultCode == 990018
        //    - Match to Order via reference field
        //    - Persist entity
        // 3. Return statistics
    }

    private void calculateFees(ReconPaygate recon) {
        if (recon.getResultCode() == 990018L) {
            // Only calculate for successful transactions
            BigDecimal gross = recon.getGross();

            // Fee = R2.00 + 3.5% of gross
            BigDecimal fee = FIXED_FEE.add(gross.multiply(PERCENT_FEE));

            // VAT on fee = 15%
            BigDecimal feeTax = fee.multiply(VAT_RATE);

            // Net = gross - fee - feeTax
            BigDecimal nett = gross.subtract(fee).subtract(feeTax);

            recon.setFee(fee);
            recon.setFeeTax(feeTax);
            recon.setNett(nett);
        }
    }
}

Fee Calculation (PayGate only):

fee = R2.00 + (gross × 3.5%)
feeTax = fee × 15%
nett = gross - fee - feeTax

Note: PayFast includes fees in the CSV; PayGate requires calculation.

6.3. Order Matching Logic

protected Order findOrder(String reference) {
    if (reference == null || reference.isEmpty()) {
        return null;
    }

    // Handle reference formats:
    // - "12345" (simple ID)
    // - "12345-12345" (ID with suffix)
    String orderId = reference.contains("-")
        ? reference.substring(0, reference.indexOf("-"))
        : reference;

    try {
        Long id = Long.parseLong(orderId);
        return orderRepository.findById(id).orElse(null);
    } catch (NumberFormatException e) {
        return null;
    }
}

7. API Design

7.1. Endpoints

Method Endpoint Description

POST

/api/recon/payfast

Upload PayFast CSV file

POST

/api/recon/paygate

Upload PayGate CSV file

GET

/api/recon

List reconciliation records (paginated)

GET

/api/recon/{id}

Get single reconciliation record

7.2. Request/Response

7.2.1. Upload PayFast CSV

curl -X POST http://localhost:8080/api/recon/payfast \
  -H "Authorization: Bearer $TOKEN" \
  -F "[email protected]" \
  -F "organisationId=1"

Response:

{
  "created": 150,
  "skipped": 5,
  "errors": 0,
  "errorMessages": []
}

7.2.2. Upload PayGate CSV

curl -X POST http://localhost:8080/api/recon/paygate \
  -H "Authorization: Bearer $TOKEN" \
  -F "[email protected]" \
  -F "organisationId=1"

8. Migration Requirements

8.1. Discriminator Migration

Existing data uses long-form discriminator values:

-- Migrate discriminator values
UPDATE recon SET type = 'PF' WHERE type = 'Payfast';
UPDATE recon SET type = 'PG' WHERE type = 'Paygate';

-- Reduce column size
ALTER TABLE recon MODIFY type VARCHAR(2) NOT NULL;

8.2. Table Rename

-- Drop FK before rename
ALTER TABLE payfast_recon DROP FOREIGN KEY FK1md70kejy5u0ct02ue70lx5pb;

-- Rename table
RENAME TABLE payfast_recon TO recon_payfast;

-- Recreate FK
ALTER TABLE recon_payfast
    ADD CONSTRAINT fk_recon_payfast_recon
    FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE;

8.3. Add Missing FK

-- Add FK to recon_pay_gate (currently missing)
ALTER TABLE recon_pay_gate
    ADD CONSTRAINT fk_recon_paygate_recon
    FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE;

9. Testing Strategy

9.1. Unit Tests

  • CSV parsing with various formats

  • Fee calculation accuracy (PayGate)

  • Order reference parsing

  • Duplicate handling

9.2. Integration Tests

  • Full CSV upload to database

  • Unique constraint enforcement

  • Order linking

9.3. Test Data

  • Anonymized PayGate sample: recon/src/docs/PayGate/Transactions_*.csv

  • Synthetic PayFast test files (to be created)