Data Synchronisation Architecture

1. Overview

This document describes the integration architecture for synchronising data between the Event and Membership System (EMS) and external platforms. The architecture supports both outbound-only and bi-directional synchronisation patterns while maintaining EMS as the authoritative data source.

2. Architecture Principles

2.1. Master Data Ownership

EMS is the single source of truth for all event, participant, and membership data. External systems receive data from EMS and may contribute changes, but EMS always has final authority.

2.2. Sync Direction Classification

Direction Pattern Use Case

Outbound-Only

EMS pushes data to remote; no changes flow back

Timing systems, marketing platforms

Bi-Directional

Data flows both ways; EMS wins on conflict

Registration platforms, customer support

2.3. Design Principles

  1. EMS Always Wins - When conflicts occur, EMS data takes precedence based on timestamp comparison

  2. Change Detection via Timestamps - The modifiedOn field on entities drives sync decisions

  3. Content Hash for Bounce-back - Prevents infinite sync loops when changes echo back

  4. Per-Entity Transactions - Single entity failures don’t block entire sync batches

  5. Supplementary Tables - Sync metadata stored separately from main entities

3. System Architecture

3.1. High-Level Architecture

sync-architecture

3.2. Component Interaction

sync-components

4. Sync Targets

4.1. Outbound-Only Targets

These systems receive data from EMS but do not send changes back:

Target Purpose Entities Synced

TagServer

In-house timing system for race events

Event, Person, Organisation, Race, StartGroup, StartGroupParticipant, EventParticipant, Tag

Mautic

Marketing automation platform

Person, Contact preferences

MailChimp

Email marketing platform

Person, List memberships

4.2. Bi-Directional Targets

These systems can create and modify data that syncs back to EMS:

Target Purpose Entities Synced

RunSignup

Third-party registration platform

Event, Race, EventParticipant, Person

ChatWoot

Customer support and engagement

Person, Conversation metadata

5. Data Model

5.1. Sync Record Concept

Each entity that participates in synchronisation has an associated Sync Record that tracks:

Field Purpose

Entity Reference

Foreign key to the main entity being synced

Sync Target

Identifies which remote system this record tracks

Entity Synced On

Timestamp when entity was last pushed TO remote

Remote Synced On

Timestamp when remote changes were last pulled (bi-directional only)

Remote ID

The entity’s identifier in the remote system

Data Hash

Content hash for change detection

5.2. Entity-Sync Record Relationship

sync-data-model
Each sync target has its own supplementary table, allowing different hash calculations and field mappings per target.

6. Synchronisation Flows

6.1. Outbound Sync Flow

The outbound sync process pushes changes from EMS to remote systems:

  1. Scheduler triggers sync at configured interval

  2. Query for changes: Find entities where modifiedOn > lastSyncedOn

  3. For each entity:

    • Calculate content hash

    • Compare with remote (if exists)

    • CREATE if new, UPDATE if changed, SKIP if unchanged

  4. Update sync record with current timestamp

  5. Handle deletions: Remove orphaned remote entities

6.2. Inbound Sync Flow (Bi-Directional)

The inbound sync process pulls changes from remote systems:

  1. Scheduler triggers sync at configured interval

  2. Query remote for entities changed since last sync

  3. For each remote change:

    • Check for bounce-back (our own change echoing back)

    • Check for conflict (EMS modified more recently)

    • Apply changes if no conflict (EMS wins otherwise)

  4. Update sync timestamps for both directions

6.3. Conflict Resolution

When both EMS and a remote system modify the same entity:

conflict-resolution

7. Scheduling

7.1. Sync Schedules

Sync Type Default Schedule Description

Outbound

Every 5 minutes

Push EMS changes to all enabled targets

Inbound

Every 10 minutes

Pull changes from bi-directional targets
Inbound sync only runs for bi-directional targets (RunSignup, ChatWoot).

7.2. Sync Order

For outbound sync, entities are processed in dependency order:

  1. Organisation (no dependencies)

  2. Person (no dependencies)

  3. Event (depends on Organisation)

  4. Race (depends on Event)

  5. StartGroup (depends on Race)

  6. EventParticipant (depends on Event, Person)

  7. StartGroupParticipant (depends on StartGroup, EventParticipant)

  8. Tag/BibTagMapping (depends on Person)

8. Error Handling

8.1. Per-Entity Isolation

Each entity is processed in its own transaction. If one entity fails to sync:

  • The failure is logged

  • Other entities continue processing

  • The failed entity will be retried on the next sync cycle

8.2. Retry Strategy

Failed sync operations are automatically retried:

  • Max attempts: 3 per sync cycle

  • Backoff: Exponential (1s, 2s, 4s)

  • Circuit breaker: Disabled after persistent failures

9. Security Considerations

9.1. Authentication

Each sync target uses its own authentication mechanism:

Target Authentication Method

TagServer

JWT token (obtained via login endpoint)

RunSignup

API key + secret

Mautic

OAuth2 or API key

MailChimp

API key

ChatWoot

API token

9.2. Data Protection

  • Credentials stored as environment variables or secrets

  • All API communication over HTTPS

  • Sensitive fields excluded from sync where not required

10. Related Documentation