Payment Reconciliation Design
1. Overview
This document describes the payment reconciliation import functionality for PayFast and PayGate payment processors. The system imports transaction history from CSV exports to verify order payment status and calculate transaction fees.
2. Status
Implementation Status: Planned
Related Architecture: Financial Management Architecture
3. Purpose
3.1. Business Goals
-
Payment Verification - Confirm orders are correctly marked as paid or unpaid
-
Fee Tracking - Capture payment processor fees for cost analysis
-
Audit Trail - Maintain record of all payment processor transactions
-
Discrepancy Detection - Identify mismatches between orders and payments
3.2. Scope
-
PayFast transaction CSV import
-
PayGate transaction CSV import
-
Order matching by merchant reference
-
Fee calculation and apportionment
-
Duplicate transaction handling
Out of Scope:
-
Automatic payment status correction (flagging only)
-
Real-time webhook-based reconciliation
-
UI for reconciliation review
4. Database Schema
4.2. Table: recon (Base)
CREATE TABLE recon (
id BIGINT AUTO_INCREMENT PRIMARY KEY,
type VARCHAR(2) NOT NULL, -- Discriminator: 'PF', 'PG'
-- Transaction details
date DATETIME,
customer VARCHAR(50),
currency VARCHAR(3),
gross DECIMAL(19,2),
fee DECIMAL(19,2),
fee_tax DECIMAL(19,2),
nett DECIMAL(19,2),
-- Relationships
order_id BIGINT, -- FK to sales_order
order_system_id BIGINT, -- FK to registration_system
CONSTRAINT fk_recon_order_system
FOREIGN KEY (order_system_id) REFERENCES registration_system(id)
);
Note: order_id FK constraint is intentionally omitted due to legacy data containing PostIDs that don’t match current Order IDs. Matching is done at application level.
4.3. Table: recon_payfast
CREATE TABLE recon_payfast (
id BIGINT PRIMARY KEY, -- FK to recon.id
-- PayFast-specific fields
payfast_payment_id BIGINT NOT NULL, -- Unique PayFast transaction ID
merchant_payment_id BIGINT, -- Merchant reference (for order matching)
sign VARCHAR(6) NOT NULL, -- CREDIT or DEBIT
party VARCHAR(50),
name VARCHAR(50),
description VARCHAR(100),
funding_type VARCHAR(25),
balance DECIMAL(19,2),
account INT DEFAULT 2,
custom1 VARCHAR(100),
custom2 VARCHAR(100),
custom3 VARCHAR(100),
CONSTRAINT fk_recon_payfast_recon
FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE,
CONSTRAINT uk_payfast_payment_id
UNIQUE (payfast_payment_id)
);
4.4. Table: recon_pay_gate
CREATE TABLE recon_pay_gate (
id BIGINT PRIMARY KEY, -- FK to recon.id
-- PayGate-specific fields
transaction_id BIGINT, -- Unique PayGate transaction ID
batch BIGINT,
number VARCHAR(20),
record_type VARCHAR(20),
transaction_date DATETIME,
auth_code VARCHAR(6),
result_code BIGINT,
result_description VARCHAR(50),
transaction_source VARCHAR(25),
original_currency VARCHAR(3),
original_amount DECIMAL(19,2),
comment VARCHAR(100),
product_description VARCHAR(100),
CONSTRAINT fk_recon_paygate_recon
FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE,
CONSTRAINT uk_paygate_transaction_id
UNIQUE (transaction_id)
);
5. CSV Format Specifications
5.1. PayFast CSV Format
PayFast exports contain 25+ columns:
| Column | Description | Entity Field |
|---|---|---|
Date |
Transaction date/time |
Recon.date |
Type |
Transaction type (Settlement, etc.) |
ReconPayfast.type (note: different from discriminator) |
Sign |
CREDIT or DEBIT |
ReconPayfast.sign |
Party |
Customer party info |
ReconPayfast.party |
Name |
Customer name |
ReconPayfast.name |
Description |
Transaction description |
ReconPayfast.description |
Currency |
Currency code (ZAR) |
Recon.currency |
Funding Type |
Payment method |
ReconPayfast.fundingType |
Gross |
Gross amount |
Recon.gross |
Fee |
PayFast fee |
Recon.fee |
Net |
Net amount after fee |
Recon.nett |
Balance |
Running balance |
ReconPayfast.balance |
M Payment ID |
Merchant payment ID (order reference) |
ReconPayfast.merchantPaymentId |
PF Payment ID |
PayFast payment ID (unique) |
ReconPayfast.payfastPaymentId |
Custom_str1 |
Custom field 1 |
ReconPayfast.custom1 |
Custom_str2 |
Custom field 2 |
ReconPayfast.custom2 |
Custom_str3 |
Custom field 3 |
ReconPayfast.custom3 |
Date Format: yyyy-MM-dd HH:mm:ss
5.2. PayGate CSV Format
PayGate exports contain 18 columns:
| Column | Description | Entity Field |
|---|---|---|
Batch |
Batch number |
ReconPaygate.batch |
Reference |
Order reference (format: "12345-12345") |
Used for order matching |
Number |
Card number (masked) |
ReconPaygate.number |
Customer Name |
Customer name |
Recon.customer |
Amount |
Transaction amount |
Recon.gross |
Type |
Settlement, Auth OK, Auth Rejected |
ReconPaygate.recordType |
Transaction Date |
Transaction date/time |
ReconPaygate.transactionDate |
Date Processed |
Processing date |
Recon.date |
TransID |
PayGate transaction ID (unique) |
ReconPaygate.transactionId |
Auth Code |
Authorization code |
ReconPaygate.authCode |
Result Code |
Result code (990018 = success) |
ReconPaygate.resultCode |
Result Desc |
Result description |
ReconPaygate.resultDescription |
Transaction Source |
Source (PayWeb v3) |
ReconPaygate.transactionSource |
Currency |
Currency code |
Recon.currency |
Orig Amount |
Original amount |
ReconPaygate.originalAmount |
Orig Currency |
Original currency |
ReconPaygate.originalCurrency |
Comment |
Comment |
ReconPaygate.comment |
Product Description |
Product description |
ReconPaygate.productDescription |
Date Format: dd-MMM-yyyy HH:mm (e.g., "14-Jun-2019 23:04")
6. Import Logic
6.1. PayFast Import Service
@Service
public class PayfastReconImportService extends BaseReconImportService {
public ReconImportResult importCsv(InputStream csvStream, Organisation org) {
// 1. Parse CSV with SuperCSV
// 2. For each row:
// - Check if payfastPaymentId already exists (skip if duplicate)
// - Create ReconPayfast entity
// - Set customer from party field
// - Match to Order via merchantPaymentId
// - Persist entity
// 3. Return statistics (created, skipped, errors)
}
}
Key Behaviors:
-
Header mapping uses case-insensitive matching with synonyms
-
Duplicate payfastPaymentId silently skipped (DataIntegrityViolationException caught)
-
Customer field populated from party or custom3
6.2. PayGate Import Service
@Service
public class PaygateReconImportService extends BaseReconImportService {
private static final BigDecimal FIXED_FEE = new BigDecimal("2.00");
private static final BigDecimal PERCENT_FEE = new BigDecimal("0.035");
private static final BigDecimal VAT_RATE = new BigDecimal("0.15");
public ReconImportResult importCsv(InputStream csvStream, Organisation org) {
// 1. Parse CSV with SuperCSV
// 2. For each row:
// - Check if transactionId already exists (skip if duplicate)
// - Create ReconPaygate entity
// - Calculate fees if resultCode == 990018
// - Match to Order via reference field
// - Persist entity
// 3. Return statistics
}
private void calculateFees(ReconPaygate recon) {
if (recon.getResultCode() == 990018L) {
// Only calculate for successful transactions
BigDecimal gross = recon.getGross();
// Fee = R2.00 + 3.5% of gross
BigDecimal fee = FIXED_FEE.add(gross.multiply(PERCENT_FEE));
// VAT on fee = 15%
BigDecimal feeTax = fee.multiply(VAT_RATE);
// Net = gross - fee - feeTax
BigDecimal nett = gross.subtract(fee).subtract(feeTax);
recon.setFee(fee);
recon.setFeeTax(feeTax);
recon.setNett(nett);
}
}
}
Fee Calculation (PayGate only):
fee = R2.00 + (gross × 3.5%) feeTax = fee × 15% nett = gross - fee - feeTax
Note: PayFast includes fees in the CSV; PayGate requires calculation.
6.3. Order Matching Logic
protected Order findOrder(String reference) {
if (reference == null || reference.isEmpty()) {
return null;
}
// Handle reference formats:
// - "12345" (simple ID)
// - "12345-12345" (ID with suffix)
String orderId = reference.contains("-")
? reference.substring(0, reference.indexOf("-"))
: reference;
try {
Long id = Long.parseLong(orderId);
return orderRepository.findById(id).orElse(null);
} catch (NumberFormatException e) {
return null;
}
}
7. API Design
7.1. Endpoints
| Method | Endpoint | Description |
|---|---|---|
POST |
|
Upload PayFast CSV file |
POST |
|
Upload PayGate CSV file |
GET |
|
List reconciliation records (paginated) |
GET |
|
Get single reconciliation record |
7.2. Request/Response
7.2.1. Upload PayFast CSV
curl -X POST http://localhost:8080/api/recon/payfast \
-H "Authorization: Bearer $TOKEN" \
-F "[email protected]" \
-F "organisationId=1"
Response:
{
"created": 150,
"skipped": 5,
"errors": 0,
"errorMessages": []
}
7.2.2. Upload PayGate CSV
curl -X POST http://localhost:8080/api/recon/paygate \
-H "Authorization: Bearer $TOKEN" \
-F "[email protected]" \
-F "organisationId=1"
8. Migration Requirements
8.1. Discriminator Migration
Existing data uses long-form discriminator values:
-- Migrate discriminator values
UPDATE recon SET type = 'PF' WHERE type = 'Payfast';
UPDATE recon SET type = 'PG' WHERE type = 'Paygate';
-- Reduce column size
ALTER TABLE recon MODIFY type VARCHAR(2) NOT NULL;
8.2. Table Rename
-- Drop FK before rename
ALTER TABLE payfast_recon DROP FOREIGN KEY FK1md70kejy5u0ct02ue70lx5pb;
-- Rename table
RENAME TABLE payfast_recon TO recon_payfast;
-- Recreate FK
ALTER TABLE recon_payfast
ADD CONSTRAINT fk_recon_payfast_recon
FOREIGN KEY (id) REFERENCES recon(id) ON DELETE CASCADE;
9. Testing Strategy
10. Related Documentation
-
Import Process Implementation (for async import patterns)