Data Synchronisation

Introduction

Information for an event must reside in multiple locations. The primary location is always the Event & Membership Management System (EMS). It is the master for all data.

At some point the data must be shared with:

  • Registration platforms (e.g., RunSignup)

  • Timing systems (e.g., TagServer)

  • Supporting devices on event day

Information must flow in both directions, because data could be changed on any system. This documentation describes the design to track what has changed and when to sync data.

Sync Targets

The framework supports multiple sync targets, each with different characteristics:

Table 1. Sync Targets and Direction
Target Direction Purpose

TagServer

Outbound only

In-house timing system - receives event, race, participant, and tag data

Mautic

Outbound only

Marketing automation - receives contact and event data

MailChimp

Outbound only

Email marketing - receives contact and list data

RunSignup

Bi-directional

Registration platform - syncs participant registrations both ways

ChatWoot

Bi-directional

Customer support - syncs contact interactions both ways

Conflict Resolution

EMS always wins. When conflicts occur (both systems modified the same entity):

  • For outbound-only targets: No conflict possible - EMS data simply overwrites remote

  • For bi-directional targets: EMS data takes precedence, based on timestamp comparison

This simplifies the design significantly - EMS is the master system, and remote systems receive authoritative data from EMS.

Entity-Target Mapping

Different targets sync different entity subsets:

Target Entities

TagServer

Event, Person, Organisation, Race, StartGroup, StartGroupParticipant, EventParticipant, Tag

RunSignup

Event, Race, EventParticipant, Person (to be confirmed)

Mautic/MailChimp

Person, Contact preferences (to be confirmed)

ChatWoot

Person, Conversation history (to be confirmed)

Key Concepts

The synchronisation framework is built around four key concepts:

Entity to be Synced

A JPA entity which should be synchronised, in part or in full, with a remote system. At minimum the entity requires:

  • A primary key (typically Long)

  • A modifiedOn field (part of the Auditable pattern)

The modifiedOn field must be updated whenever the entity changes. This is critical for change detection.

Remote Entity

The corresponding record in the remote system. The remote entity:

  • Tracks a copy of the main entity’s data

  • Can also be updated, resulting in its state being synchronised back to the main entity

  • Must track its own modifiedOn field

  • Needs a notification or query mechanism to return records changed after the last sync

Sync Record

Since there can be many synchronisation processes attached to an entity (one per sync target), a SyncRecord maintains a ManyToOne JPA relationship with the entity. This is a unidirectional association from the SyncRecord side only.

Table 2. Sync Record Fields
Field Description

syncProcess

Identifies the process managing this sync (e.g., TAGSERVER, RUNSIGNUP)

entity

Foreign key reference to the main entity

entitySyncedOn

Timestamp when the entity was last synchronised TO the remote system

remoteSyncedOn

Timestamp when the remote system record was last synchronised back to the main entity

remoteId

Optional reference to the remote entity’s unique identifier

An indirect relationship exists between the SyncRecord and the RemoteEntity via a reference. The RemoteEntity will either reference the Entity ID (if it has provision to do so), or the SyncRecord stores a unique key of the RemoteEntity.

Sync Process

A controller for each synchronisation process manages the synchronisation between all entities under its control and the remote system. This includes:

  • Remote connection management

  • Data exchange protocol

  • Entity transformation (DTO mapping)

  • Error handling and retry logic

Change Detection

The two syncedOn properties allow the synchronisation process to identify when the associated entity was last synchronised in each direction.

Outbound Sync Detection
When entitySyncedOn < Entity.modifiedOn
  → Entity is due to be synchronised to the remote system
Inbound Sync Detection
When remoteSyncedOn < Remote.modifiedOn
  → Remote system record must be synchronised back to the Entity

Content Hash

To detect whether a change is a bounce-back (our own change echoing back) versus an actual remote change, the framework uses a content hash:

  1. Load the entity from DB

  2. Calculate the hash on that entity (from a method within the SyncProcess)

  3. Apply the changes from the RemoteEntity via a mapping function

  4. Calculate the hash again on the updated entity

  5. Compare the two hash values to determine change

If the hash differs, the content is copied over and persisted.

Synchronisation Flows

Outbound Sync - New Entity Records

When the sync process is first initiated, there will be no SyncRecords attached to the Entities for the given SyncProcess. A filter mechanism determines which entities should be synchronised.

Process:

  1. Apply filter to determine eligible entities

  2. Prepare and transmit data to remote system

  3. Create SyncRecord with:

    • Reference to Entity and RemoteEntity

    • entitySyncedOn set to the Entity’s modifiedOn value

This ensures that any subsequent change to the Entity can be detected later.

Outbound Sync - Updated Entity Record

When an Entity changes, the modifiedOn field changes. This allows a query to find entities where Entity.modifiedOn > SyncRecord.entitySyncedOn.

Detection Query Pattern
SELECT e.id as IDToSync
FROM entity e
LEFT JOIN sync_entity sr ON e.id = sr.entity_id
WHERE <FILTER ON e>
  AND (sr.id IS NULL OR e.modified_on > sr.entitySyncOn);

The Entity is transmitted to the remote system and entitySyncedOn is set to the Entity’s modifiedOn value.

Inbound Sync - Bounce-back Detection

Each time an outbound sync is triggered, it could result in the remote system subsequently indicating that a change has been made. It is important to determine if such events are actually due to a change in the remote system.

This is done by comparing the content hash of the Entity and RemoteEntity data elements:

  1. Retrieve the RemoteEntity

  2. Calculate hash on its data (excluding modifiedOn)

  3. Compare with corresponding fields in the Entity

  4. If hash differs, copy content and persist

  5. Set entitySyncedOn to the Entity’s modifiedOn value

  6. Set remoteSyncedOn to the RemoteEntity’s modifiedOn (or polling timestamp)

Inbound Sync - Polling for Remote Changes

For supporting remote systems, the sync process polls to determine any changes:

  1. Use the latest SyncRecord.remoteSyncedOn as a time reference

  2. Find all remote records changed after that time

  3. For each record, calculate and compare content hash

  4. If hash differs, apply changes to the local Entity

  5. Update both entitySyncedOn and remoteSyncedOn timestamps

Inbound Sync - Unchanged Fields

There could be instances where the remote entity was changed, but not on a field that has to be synced back. In this case, the process works the same as bounce-back detection - the hash comparison will show no difference, and only the remoteSyncedOn timestamp is updated.

Current Implementation

The current implementation targets the TagServer as the first sync target. See TagServer Integration for detailed implementation documentation.

Framework Architecture

Table 3. Core Classes
Class Purpose

BaseSyncService<E, S>

Abstract base class providing sync orchestration (admin-service)

BaseSyncEntity<E>

Abstract base class for sync record entities (database)

BaseSyncEntityRepository<T>

Base repository with common sync queries (database)

SyncController

Scheduler coordinating all sync services

Implementation Status

Feature Status Notes

Outbound sync (main → remote)

Implemented

Full CREATE, UPDATE, DELETE support

Inbound sync (remote → main)

Stub only

All syncToMain() methods log "not implemented yet"

Deletion handling

Implemented

Orphaned sync records cleaned up

Hash-based change detection

Implemented

Compares local vs remote hash

Per-entity transactions

Implemented

Prevents batch rollback on single failure

Scheduled sync

Implemented

Cron-based, default every minute

Testing Considerations

Ensuring the hash function works reliably is critical. If the same data is not correctly detected as being similar, it will result in an endless loop of updates.

Design comprehensive tests around:

  • Content hash checking for each entity type

  • All sync flows (outbound new, outbound update, inbound bounce-back, inbound change)

  • Edge cases (null fields, empty strings, timezone handling)

Target-Specific Documentation

Outbound-Only Targets

  • TagServer Integration - In-house timing system sync

  • MailChimp Integration - Future

  • Mautic Integration - Future

Bi-Directional Targets

  • RunSignup Integration - In design phase

  • ChatWoot Integration - Future