Sync Framework Design

Overview

This document describes the proposed improvements to the sync framework, designed to support:

  • Multiple sync targets (TagServer, RunSignup, Mautic, MailChimp, ChatWoot)

  • Both outbound-only and bi-directional sync patterns

  • EMS-wins conflict resolution

  • Extensibility for future targets

Architecture

Package Structure

za.co.idealogic.event.admin.service.sync/
│
├── core/                              # Framework core (target-agnostic)
│   ├── SyncDirection.java             # Enum: OUTBOUND, BIDIRECTIONAL
│   ├── SyncAction.java                # Enum: CREATE, UPDATE, DELETE, SKIP
│   ├── SyncResult.java                # Result object with status/errors
│   ├── SyncService.java               # Interface for all sync services
│   ├── AbstractSyncService.java       # Base implementation
│   ├── OutboundSyncService.java       # Base for outbound-only targets
│   ├── BidirectionalSyncService.java  # Base for bi-directional targets
│   └── SyncScheduler.java             # Generic scheduler
│
├── config/
│   ├── SyncProperties.java            # @ConfigurationProperties
│   └── SyncTargetConfig.java          # Per-target configuration
│
├── tagserver/                         # TagServer (outbound-only)
│   ├── TagServerClient.java           # Feign client interface
│   ├── TagServerConfig.java           # Target-specific config
│   ├── EventTagServerSyncService.java
│   ├── PersonTagServerSyncService.java
│   └── ...
│
├── runsignup/                         # RunSignup (bi-directional)
│   ├── RunSignupClient.java           # Feign client interface
│   ├── RunSignupConfig.java
│   ├── EventRunSignupSyncService.java
│   └── ...
│
├── mautic/                            # Mautic (outbound-only)
│   └── ...
│
└── chatwoot/                          # ChatWoot (bi-directional)
    └── ...

Class Hierarchy

                    ┌──────────────────────┐
                    │   SyncService<E,S>   │  Interface
                    │  + syncToRemote()    │
                    │  + syncToMain()      │
                    │  + getSyncDirection()│
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │ AbstractSyncService  │  Base implementation
                    │  - common logic      │
                    │  - hash comparison   │
                    │  - error handling    │
                    └──────────┬───────────┘
                               │
          ┌────────────────────┼────────────────────┐
          │                                         │
┌─────────▼──────────┐                 ┌────────────▼────────────┐
│ OutboundSyncService│                 │ BidirectionalSyncService│
│ - syncToRemote()   │                 │ - syncToRemote()        │
│ - syncToMain() ──► │ no-op           │ - syncToMain()          │
└─────────┬──────────┘                 │ - resolveConflict()     │
          │                            └────────────┬────────────┘
          │                                         │
┌─────────▼──────────┐                 ┌────────────▼────────────┐
│ TagServerSync      │                 │ RunSignupSyncService    │
│ MauticSync         │                 │ ChatWootSyncService     │
│ MailChimpSync      │                 └─────────────────────────┘
└────────────────────┘

Core Interface

public interface SyncService<E, S extends BaseSyncEntity<E>> {

    /**
     * Sync direction for this target
     */
    SyncDirection getSyncDirection();

    /**
     * Target identifier (TAGSERVER, RUNSIGNUP, etc.)
     */
    SyncTarget getSyncTarget();

    /**
     * Entity type name for logging
     */
    String getEntityName();

    /**
     * Sync local entities TO the remote system
     */
    SyncResult syncToRemote();

    /**
     * Sync remote changes BACK to local (bi-directional only)
     */
    SyncResult syncToMain();

    /**
     * Calculate content hash for change detection
     */
    int calculateHash(E entity);
}

public enum SyncDirection {
    OUTBOUND,       // EMS -> Remote only
    BIDIRECTIONAL   // EMS <-> Remote (EMS wins)
}

public enum SyncTarget {
    TAGSERVER,
    RUNSIGNUP,
    MAUTIC,
    MAILCHIMP,
    CHATWOOT
}

Outbound-Only Base Class

For targets like TagServer, Mautic, and MailChimp:

public abstract class OutboundSyncService<E, S extends BaseSyncEntity<E>>
        extends AbstractSyncService<E, S> {

    @Override
    public final SyncDirection getSyncDirection() {
        return SyncDirection.OUTBOUND;
    }

    @Override
    public final SyncResult syncToMain() {
        // No-op for outbound-only targets
        log.debug("{} is outbound-only, skipping syncToMain", getSyncTarget());
        return SyncResult.skipped("Outbound-only target");
    }

    // syncToRemote() inherited from AbstractSyncService
}

Bi-Directional Base Class

For targets like RunSignup and ChatWoot:

public abstract class BidirectionalSyncService<E, S extends BaseSyncEntity<E>>
        extends AbstractSyncService<E, S> {

    @Override
    public final SyncDirection getSyncDirection() {
        return SyncDirection.BIDIRECTIONAL;
    }

    @Override
    public SyncResult syncToMain() {
        log.info("Starting inbound sync from {} for {}", getSyncTarget(), getEntityName());

        Instant lastSync = syncRepository.findLastRemoteSyncTime();
        List<RemoteDTO> remoteChanges = fetchRemoteChanges(lastSync);

        int processed = 0, skipped = 0, conflicts = 0;

        for (RemoteDTO remote : remoteChanges) {
            S syncRecord = findOrCreateSyncRecord(remote);
            E localEntity = syncRecord.getMainEntity();

            // Bounce-back detection
            if (isBounceBack(localEntity, remote)) {
                updateRemoteSyncTimestamp(syncRecord, remote);
                skipped++;
                continue;
            }

            // Conflict detection: EMS wins based on timestamp
            if (hasConflict(localEntity, remote)) {
                log.info("Conflict detected for {} - EMS wins", getEntityId(localEntity));
                updateRemoteSyncTimestamp(syncRecord, remote);
                conflicts++;
                continue;
            }

            // Apply remote changes
            applyRemoteChanges(localEntity, remote);
            entityRepository.save(localEntity);
            updateSyncTimestamps(syncRecord, localEntity, remote);
            processed++;
        }

        return SyncResult.success(processed, skipped, conflicts);
    }

    /**
     * Check if local entity was modified after remote change (EMS wins)
     */
    protected boolean hasConflict(E localEntity, RemoteDTO remote) {
        Instant localModified = getModifiedOn(localEntity);
        Instant remoteModified = remote.getModifiedOn();

        // If local was modified after remote, EMS wins = conflict (skip remote change)
        return localModified != null && remoteModified != null
            && localModified.isAfter(remoteModified);
    }

    /**
     * Fetch changes from remote system since last sync
     */
    protected abstract List<RemoteDTO> fetchRemoteChanges(Instant since);

    /**
     * Apply remote changes to local entity
     */
    protected abstract void applyRemoteChanges(E entity, RemoteDTO remote);
}

Configuration

Application Properties

sync:
  enabled: true
  scheduler:
    outbound-cron: "0 */5 * * * *"    # Every 5 minutes
    inbound-cron: "0 */10 * * * *"    # Every 10 minutes

  # Outbound-only targets
  tagserver:
    enabled: true
    direction: OUTBOUND
    url: ${TAGSERVER_URL:http://localhost:12510}
    username: ${TAGSERVER_USERNAME}
    password: ${TAGSERVER_PASSWORD}
    timeout: 30s
    retry:
      max-attempts: 3
      initial-interval: 1s
      multiplier: 2

  mautic:
    enabled: false
    direction: OUTBOUND
    url: ${MAUTIC_URL}
    api-key: ${MAUTIC_API_KEY}

  mailchimp:
    enabled: false
    direction: OUTBOUND
    url: https://us1.api.mailchimp.com/3.0
    api-key: ${MAILCHIMP_API_KEY}

  # Bi-directional targets
  runsignup:
    enabled: false
    direction: BIDIRECTIONAL
    url: https://runsignup.com/rest
    api-key: ${RUNSIGNUP_API_KEY}
    api-secret: ${RUNSIGNUP_API_SECRET}

  chatwoot:
    enabled: false
    direction: BIDIRECTIONAL
    url: ${CHATWOOT_URL}
    api-key: ${CHATWOOT_API_KEY}

Configuration Properties Class

@ConfigurationProperties(prefix = "sync")
@Validated
public class SyncProperties {

    private boolean enabled = true;
    private SchedulerConfig scheduler = new SchedulerConfig();

    private TagServerConfig tagserver = new TagServerConfig();
    private RunSignupConfig runsignup = new RunSignupConfig();
    private MauticConfig mautic = new MauticConfig();
    private MailChimpConfig mailchimp = new MailChimpConfig();
    private ChatWootConfig chatwoot = new ChatWootConfig();

    @Data
    public static class SchedulerConfig {
        private String outboundCron = "0 */5 * * * *";
        private String inboundCron = "0 */10 * * * *";
    }

    @Data
    public static class TargetConfig {
        private boolean enabled = false;
        private SyncDirection direction;
        private String url;
        private Duration timeout = Duration.ofSeconds(30);
        private RetryConfig retry = new RetryConfig();
    }

    @Data
    public static class RetryConfig {
        private int maxAttempts = 3;
        private Duration initialInterval = Duration.ofSeconds(1);
        private double multiplier = 2.0;
    }
}

Feign Client Example

TagServer Client (Outbound)

@FeignClient(
    name = "tagserver",
    url = "${sync.tagserver.url}",
    configuration = TagServerFeignConfig.class
)
public interface TagServerClient {

    @PostMapping("/api/events")
    EventDTO createEvent(
        @RequestBody EventDTO event,
        @RequestHeader("X-Org-Id") String orgId
    );

    @PatchMapping("/api/events/{id}")
    EventDTO updateEvent(
        @PathVariable Long id,
        @RequestBody EventDTO event,
        @RequestHeader("X-Org-Id") String orgId
    );

    @GetMapping("/api/events/{id}")
    Optional<EventDTO> getEvent(
        @PathVariable Long id,
        @RequestHeader("X-Org-Id") String orgId
    );

    @DeleteMapping("/api/events/{id}")
    void deleteEvent(
        @PathVariable Long id,
        @RequestHeader("X-Org-Id") String orgId
    );
}

RunSignup Client (Bi-directional)

@FeignClient(
    name = "runsignup",
    url = "${sync.runsignup.url}",
    configuration = RunSignupFeignConfig.class
)
public interface RunSignupClient {

    // Outbound
    @PostMapping("/race/{raceId}/participant")
    ParticipantDTO createParticipant(
        @PathVariable Long raceId,
        @RequestBody ParticipantDTO participant
    );

    @PutMapping("/race/{raceId}/participant/{participantId}")
    ParticipantDTO updateParticipant(
        @PathVariable Long raceId,
        @PathVariable Long participantId,
        @RequestBody ParticipantDTO participant
    );

    // Inbound - fetch changes since timestamp
    @GetMapping("/race/{raceId}/participants")
    List<ParticipantDTO> getParticipants(
        @PathVariable Long raceId,
        @RequestParam("modified_after") Instant since
    );

    @GetMapping("/race/{raceId}/registrations")
    List<RegistrationDTO> getRegistrations(
        @PathVariable Long raceId,
        @RequestParam("modified_after") Instant since
    );
}

Scheduler

@Component
@ConditionalOnProperty(name = "sync.enabled", havingValue = "true")
public class SyncScheduler {

    private final List<SyncService<?, ?>> syncServices;
    private final SyncProperties properties;

    @Scheduled(cron = "${sync.scheduler.outbound-cron}")
    public void runOutboundSync() {
        log.info("Starting scheduled outbound sync");

        for (SyncService<?, ?> service : syncServices) {
            if (isEnabled(service)) {
                try {
                    SyncResult result = service.syncToRemote();
                    log.info("{} {} outbound: {}",
                        service.getSyncTarget(),
                        service.getEntityName(),
                        result);
                } catch (Exception e) {
                    log.error("Outbound sync failed for {} {}",
                        service.getSyncTarget(),
                        service.getEntityName(), e);
                }
            }
        }
    }

    @Scheduled(cron = "${sync.scheduler.inbound-cron}")
    public void runInboundSync() {
        log.info("Starting scheduled inbound sync");

        for (SyncService<?, ?> service : syncServices) {
            if (isEnabled(service) &&
                service.getSyncDirection() == SyncDirection.BIDIRECTIONAL) {
                try {
                    SyncResult result = service.syncToMain();
                    log.info("{} {} inbound: {}",
                        service.getSyncTarget(),
                        service.getEntityName(),
                        result);
                } catch (Exception e) {
                    log.error("Inbound sync failed for {} {}",
                        service.getSyncTarget(),
                        service.getEntityName(), e);
                }
            }
        }
    }
}

Database Schema Addition

Add remote_id field to all sync entities:

ALTER TABLE event_sync_tag_server
    ADD COLUMN remote_id VARCHAR(255);

ALTER TABLE person_sync_tag_server
    ADD COLUMN remote_id VARCHAR(255);

-- ... repeat for all sync tables

Updated BaseSyncEntity:

@MappedSuperclass
public abstract class BaseSyncEntity<E> {

    private Instant lastModifiedMain;
    private Instant lastModifiedRemote;

    @Column(name = "remote_id")
    private String remoteId;  // NEW: Remote system's entity ID

    // ... existing methods ...

    public String getRemoteId() { return remoteId; }
    public void setRemoteId(String remoteId) { this.remoteId = remoteId; }
}

Migration Path

Phase 1: Quick Wins

  1. Externalize configuration (application.yml)

  2. Add remoteId field to sync entities

  3. Convert to Feign clients (TagServer first)

Phase 2: Framework Refactoring

  1. Create SyncDirection enum and base classes

  2. Refactor TagServer services to extend OutboundSyncService

  3. Update scheduler for separate outbound/inbound schedules

Phase 3: RunSignup Integration

  1. Implement RunSignupClient Feign interface

  2. Create BidirectionalSyncService implementations

  3. Implement inbound sync with conflict resolution

Phase 4: Additional Targets

  1. Mautic integration (outbound)

  2. MailChimp integration (outbound)

  3. ChatWoot integration (bi-directional)

Summary

This design provides:

  • Clear separation between outbound-only and bi-directional targets

  • EMS-wins conflict resolution built into the bi-directional base class

  • Extensibility - new targets follow established patterns

  • Configuration-driven - enable/disable targets via properties

  • Testability - Feign clients can be easily mocked