Hazelcast Configuration

1. Overview

Hazelcast runs embedded inside every EMS Spring Boot service that needs one of:

  • HTTP session replication (portal gateways — so sticky-session failover does not log users out)

  • Hibernate second-level cache (admin-service)

  • Distributed application caches with explicit TTL/eviction (admin-service’s userOrgAccess / userPersonAccess security-dimension caches)

Every service auto-discovers its peers. There is no external Hazelcast cluster to manage. The cluster forms itself based on environment:

Environment Discovery mechanism

Kubernetes (kubernetes profile)

Hazelcast Kubernetes plugin queries the service DNS for peer pods

Dev with local JHipster ServiceRegistry (dev profile + jhipsterRegistry)

TCP-IP member list calculated as localhost + port offset server.port + 5701

Prod with ServiceRegistry (legacy)

TCP-IP member list using hostnames and static port 5701

Standalone (dev profile, no registry)

Single-node; no cluster formed

This page covers the cluster model, the EMS-custom map configurations, the DevTools compatibility workaround, and how new services (specifically admin-portal) should configure their own Hazelcast instance.

See also: Portal Pattern, Session-Held JWT & JSESSIONID.

2. Cluster Topology

Each service runs an embedded Hazelcast member (not a client). Every member is a peer with equal participation; there is no primary. Data is partitioned across members with configurable backup count (default: 1 — each partition has one replica on another member).

hazelcast-topology

Important: service type owns its cluster. admin-service pods cluster with other admin-service pods; they do not join registration-portal’s cluster. Cluster name is set per service via HazelcastProperties.clusterName → different value per service ensures isolation.

Cross-service caching (e.g. admin-service exposing a cache to portals) happens via admin-service’s REST API, not via a shared Hazelcast cluster. Keep services decoupled.

3. Configuration Class

Both admin-service and registration-portal ship a CacheConfiguration class (~318 lines in admin-service, lighter in registration-portal). Structure:

@Configuration
@EnableCaching
public class CacheConfiguration {
    @Bean
    public Config hazelcastConfig(...) {
        Config config = new Config();
        config.setClusterName(hazelcastProperties.getClusterName());
        configureNetworking(config);             // discovery + port
        configureMapConfigs(config);             // per-map TTL, eviction, backup
        configureSerialization(config);          // JSON + Java
        return config;
    }

    @Bean
    public HazelcastInstance hazelcastInstance(Config config) {
        return Hazelcast.newHazelcastInstance(config);
    }

    @Bean
    public CacheManager cacheManager(HazelcastInstance hz) {
        return new HazelcastCacheManager(hz);
    }
}

Reference: [admin-service/src/main/java/…​/config/CacheConfiguration.java](admin-service/src/main/java/za/co/idealogic/event/admin/config/CacheConfiguration.java).

3.1. Discovery: the four-tier ladder

if (isKubernetesProfile()) {
    // Kubernetes plugin: hazelcast-kubernetes
    JoinConfig join = networkConfig.getJoin();
    join.getKubernetesConfig()
        .setEnabled(true)
        .setProperty("service-name", hazelcastProperties.getServiceName());
    // Peers resolve via DNS; port 5701 by default
} else if (isDevProfileWithRegistry()) {
    // Localhost + port offset — multiple pods on the same host, differentiated by port
    int hzPort = serverPort + 5701;
    TcpIpConfig tcp = join.getTcpIpConfig().setEnabled(true);
    tcp.addMember("127.0.0.1:" + hzPort);
} else if (isProdProfileWithRegistry()) {
    // Static hostname list — pre-K8s style
    TcpIpConfig tcp = join.getTcpIpConfig().setEnabled(true);
    for (String host : hazelcastProperties.getPeers()) {
        tcp.addMember(host + ":5701");
    }
} else {
    // Standalone: no multicast, no TCP-IP, no discovery
    join.getMulticastConfig().setEnabled(false);
    join.getTcpIpConfig().setEnabled(false);
}

The Kubernetes branch is the one that matters for production. The service-name property names the Kubernetes Service that fronts the service’s pods; Hazelcast queries its endpoints and adds each as a peer.

3.2. Per-map configuration

EMS-custom (not in stock JHipster) — admin-service defines two security-dimension caches:

MapConfig userOrgAccess = new MapConfig()
    .setName("userOrgAccess")
    .setBackupCount(1)
    .setTimeToLiveSeconds(300)
    .setEvictionConfig(new EvictionConfig()
        .setEvictionPolicy(EvictionPolicy.LRU)
        .setMaxSizePolicy(MaxSizePolicy.USED_HEAP_PERCENTAGE)
        .setSize(10));
config.addMapConfig(userOrgAccess);
// Similar for userPersonAccess

These are read-through caches in front of SecurityDimensionService.getUserAccessiblePersonIds() etc. — queries that appear in the critical path of most authenticated requests. TTL is 300s so stale-but-correct data is acceptable; LRU + heap-percentage eviction prevents unbounded growth.

Stock JHipster caches (user, authority, etc.) are still generated but have default TTLs; review per-cache if hit ratios surprise you.

3.3. Serialisation

Default JSON for cross-version compatibility. Explicit Java serialisation classes registered for TenantContext and similar small value objects where deserialisation cost dominates. Not a performance-critical path; defaults suffice.

4. Session Replication for Portals

Portal gateways enable session replication via Spring Session + Hazelcast:

spring:
  session:
    store-type: hazelcast
    hazelcast:
      map-name: spring:session:sessions
      flush-mode: immediate

Sticky ingress keeps steady-state traffic on one pod; Hazelcast replication covers pod rotations. In a three-pod portal deployment, losing any one pod does not log anyone out.

For admin-portal this is mandatory — session holds the admin-service JWT (see Session-Held JWT & JSESSIONID), and losing the session means the user has to re-auth.

For registration-portal it is similarly enabled; anonymous sessions survive pod restarts.

Tuning:

  • flush-mode: immediate — ensures the session is replicated before the response goes out. Safer than on_save for our use case (re-read of session attributes from a different pod on the very next request).

  • map-name — keep default unless running multiple Spring Session stores in the same cluster.

  • TTL — inherit from server.servlet.session.timeout (typical: 30 minutes of inactivity).

5. DevTools Compatibility Workaround

admin-service ships a System property set in main() to disable Spring DevTools restart:

public static void main(String[] args) {
    System.setProperty("spring.devtools.restart.enabled", "false");
    // ...
}

Reason: DevTools uses a separate classloader for the restart context. Hazelcast holds references across the classloader boundary, and a restart triggers a cascade of `ClassCastException`s and connection leaks.

Cost: no DevTools auto-reload in admin-service dev. Benefit: no flaky dev startups. Accept the trade-off.

registration-portal does not ship this workaround — its Hazelcast footprint is lighter. If admin-portal’s Hazelcast usage grows (e.g. adds security-dimension caches or Hibernate L2), bring the workaround along.

6. admin-portal Configuration

Modelled on registration-portal’s lighter configuration:

  • Same four-tier discovery

  • Cluster name: ems-admin-portal-<env> (distinct from registration-portal’s cluster)

  • Session replication enabled (critical — holds the admin-service JWT + current tenant)

  • Hibernate L2 cache: not needed (admin-portal has no domain entities of its own)

  • Map configs: only those Spring Session uses; no EMS-custom maps at launch

If future admin-portal work adds its own cached computed state (unlikely given the gateway scope rule), add map configs then.

7. Dev and Test

Local dev:

  • dev profile, no ServiceRegistry → standalone mode. One pod, no cluster. Hazelcast runs on 5701 (or server.port + 5701 if ServiceRegistry is on).

  • ./mvnw -Pdev spring-boot:run — starts with the right discovery branch.

  • Sessions do not replicate (single node). Fine for local testing.

Tests:

  • Unit tests use NoOpCache via @MockBean CacheManager or an @Profile("!prod") CacheConfiguration override.

  • Integration tests boot a real Hazelcast in standalone mode; cluster formation is skipped.

8. Monitoring

Hazelcast emits metrics via the Spring Boot Actuator’s micrometer integration:

  • hazelcast.map.entries (count per cache)

  • hazelcast.map.hits, hazelcast.map.misses (hit ratio calculable)

  • hazelcast.partition.is-migrating (true during rebalance — spike on pod rotation)

Watch hit ratios for the security-dimension caches specifically; a ratio below ~70% suggests TTL is too aggressive or cache size too small.

Cluster membership changes log at INFO:

  • Members [N] { …​ } after join/leave

  • Member <host:port> removed during rotation

Operators should alert on persistent split-brain indicators (two partitions with differing member lists for > 30s).

9. Reference

File Role

admin-service/src/main/java/…​/config/CacheConfiguration.java

Full Hazelcast wiring with four-tier discovery + EMS-custom map configs

registration-portal/src/main/java/…​/config/CacheConfiguration.java

Lighter variant — no domain caches, session-replication oriented

admin-service/src/main/java/…​/config/HazelcastProperties.java

@ConfigurationProperties for cluster name, peer list, K8s service name

admin-service/src/main/resources/config/application-kubernetes.yml

Kubernetes profile — Hazelcast service-name + K8s config client

admin-service/AdminServiceApp.java (main)

DevTools workaround

10. Related Documentation

11. Change History

Date Change

2026-04-24

Initial draft. Grounded in config-scan findings (userOrgAccess / userPersonAccess security caches, four-tier discovery, DevTools workaround).