Microservice Pattern — EMS Service Bootstrap Baseline

1. Overview

Every EMS micro-service — backend (admin-service), portal (registration-portal, admin-portal, …​), MCP adapter (mcp-server, …​) — shares this bootstrap baseline. The baseline is a deliberate, copyable shape: a Maven module, a Jib-built container image, a per-service Helm chart, ArgoCD Applications per environment, a fixed set of GitHub Actions workflows, an OpenTelemetry javaagent baked into the image, and the same probe/profile/security conventions across the fleet.

Three specialisations build on top of this baseline:

  • Portal Pattern — services that serve a browser SPA: gateway + Angular SPA, OIDC client, session-held JWT, tenant resolution.

  • MCP Pattern — services that expose admin-service via the Model Context Protocol to LLM-driven clients: OAuth resource server, MCP transport, tool catalogue, audit log.

  • Java Library Pattern — Maven JAR artefacts published to GitHub Packages and consumed as dependencies (parent-pom, event-database, wordpress-database, spreadsheet-importer). Re-uses the Maven module + CI/CD halves of this baseline; drops Jib, Helm, ArgoCD, OTel, ports and probes.

If you are bootstrapping a new EMS service or library, read this page first, then read whichever specialisation matches the persona. Add ADRs only for points where you genuinely deviate.

2. When to Use This Pattern

Use this pattern for any new service intended to:

  • Run in the EMS Kubernetes cluster (idl-xnl-jhb1-rc01).

  • Be deployed via the existing ArgoCD app-of-apps in idl-xnl-jhb-rc01.

  • Be released through the EMS GitFlow pipeline.

  • Carry the same observability and security posture as the rest of the fleet.

If any of those four bullets do not apply, the service belongs outside this pattern (e.g. an external Lambda, a third-party SaaS we self-host elsewhere, a desktop tool).

3. What’s In Scope (Common Across All Services)

Concern Treatment

Maven module structure

Single jar artefact, parented to za.co.idealogic:event:1.3.1. CI-friendly ${revision} versioning. <repositories> block referencing github-christhonie so a clean checkout can resolve the parent.

Spring Boot platform

Spring Boot 3.4.x via JHipster 8.11.0 dependency platform inherited from the parent POM. No service runs on a different platform without a hard reason.

Container image

Jib build from eclipse-temurin:17-jre-focal (Java 17 runtime). No Dockerfile, no Docker daemon. Image pushed to docker.io/christhonie/<service-name>.

OpenTelemetry agent

opentelemetry-javaagent jar baked into every image at /javaagent/opentelemetry-javaagent.jar via a Maven dependency:copy execution + Jib <extraDirectories>. JVM -javaagent flag set unconditionally.

Helm chart

Per-service chart under src/main/helm/. Standard schema (config.profiles, config.existingsecret, image., ingress., resources.*, replicaCount, podAnnotations).

ArgoCD manifests

One Application manifest per (service × environment) under ~/dev/idl-xnl-jhb-rc01/argocd/<service-name>-{dev,stage,prod}.yml. Auto-discovered by the cluster’s cluster-bootstrap.yml app-of-apps.

CI/CD

Five GitHub Actions wrapper workflows in the service repo’s .github/workflows/, each delegating to a reusable workflow in christhonie/event/.github/workflows/. See § CI/CD below.

Health probes

/livez (kubelet liveness) and /readyz (kubelet readiness) — small Spring @RestController returning 200/OK. Distinct from /actuator/health/** which is heavier and authenticated.

Container security

runAsNonRoot: true, runAsUser: 1000, allowPrivilegeEscalation: false, capabilities.drop: [ALL]. Image pull secret christhonie-docker in every namespace.

Logging

Logback with MDC fields traceId, spanId, plus per-service contextual fields (actor, tool, actorOrg, …​). OTel logback appender ships logs to the collector with trace correlation when the otlp profile is active.

Spring profile model

Conventional set: dev / prod (mutually exclusive), otlp (observability), api-docs (springdoc), tls (when needed). Combinations like prod,api-docs,otlp are normal.

Test conventions

Smoke @SpringBootTest context-load test in every module. Integration tests named *IT.java run via maven-failsafe-plugin. Spotless + checkstyle. ArchUnit optional, retained where existing services have it.

Application bootstrap

SpringApplication Bootstrap startup banner with profile, hostname, port, DB URL (when applicable). Reject incompatible profile combinations (dev + prod) at startup.

4. What’s Out of Scope (Specialised per Service)

Concern Where it lives

Auth model (OIDC client vs OAuth resource server vs API-key only)

Portal Pattern / MCP Pattern / per-service decision

Database, JPA, Liquibase

admin-service has these; portals proxy; MCP adapters don’t have them. Per-service.

Frontend / SPA

Portal Pattern only

Reverse-proxy / BFF endpoints

Portal Pattern only

Hazelcast (session replication, distributed caches)

admin-service + portals only; MCP adapters are stateless

Tool catalogue / MCP transport

MCP Pattern only

Tenant resolution model

Portal Pattern (server-side, in session) vs MCP Pattern (per-tool-call parameter)

5. Maven Module Structure

5.1. POM essentials

<parent>
    <groupId>za.co.idealogic</groupId>
    <artifactId>event</artifactId>
    <version>1.3.1</version>
</parent>

<artifactId>my-service</artifactId>
<version>${revision}</version>
<packaging>jar</packaging>

<properties>
    <revision>0.1.0-SNAPSHOT</revision>
    <java.version>17</java.version>
    <start-class>za.co.idealogic.event.myservice.MyServiceApp</start-class>
    <!-- OTel agent locations — same conventions across the fleet -->
    <agent-extraction-root>${project.build.directory}/jib-agents</agent-extraction-root>
    <agent-install-location>/javaagent</agent-install-location>
    <opentelemetry-javaagent-filename>opentelemetry-javaagent.jar</opentelemetry-javaagent-filename>
    <opentelemetry-javaagent.version>2.10.0</opentelemetry-javaagent.version>
</properties>

<repositories>
    <repository>
        <id>github-christhonie</id>
        <url>https://maven.pkg.github.com/christhonie/event</url>
        <releases><enabled>true</enabled><checksumPolicy>fail</checksumPolicy></releases>
        <snapshots><enabled>false</enabled></snapshots>
    </repository>
</repositories>

The ${revision} flow gives CI-friendly versioning: GitFlow release-start rewrites it to 0.1.0, release-finish bumps it to 0.1.1-SNAPSHOT on develop and tags 0.1.0 on main.

5.2. Port assignments

The EMS port range is reserved at 12500. Each service gets a stable port:

Port Service Notes

12504

admin-service

Backend REST API

12505

registration-portal

Public registration portal

12506

admin-portal

Staff admin portal (greenfield, in flight)

12507

mcp-server

First MCP adapter

12508+

(next service)

Reserve at module bootstrap; record here

6. CI/CD

Every service repo has the same five wrapper workflows under .github/workflows/, each delegating to a reusable workflow in christhonie/event/.github/workflows/ (the parent-pom repo). Pipeline shape, reusable workflow inventory, required secrets, repository settings, and YAML templates are documented in GitHub Actions CI/CD. Read that page for any depth.

Summary at the microservice-pattern level:

Wrapper Trigger and purpose

push-dev.yml

Push to develop → test → package → publish → docker → helm → bump dev ArgoCD manifest

push-main.yml

Push to main (only via merged release PR) → package (prod) → docker → helm → bump stage ArgoCD manifest → GitFlow release-finish

push-release.yml

Push to release/** → test only (RC verification)

pr-non-main.yml

PR to non-main branch → test (backend + frontend filter for portals)

manual-release-start.yml

workflow_dispatchmvn gitflow:release-start → opens release PR back to main

Three GitHub-side bring-up requirements are easy to miss; they are covered in GitHub Actions CI/CD § Required Repository Settings:

  • Default workflow permissions must be set to write (otherwise reusable workflows fail at startup with no log output).

  • Reusable workflow access on christhonie/event must allow user-owned repos.

  • service-name input to argocd-update.yml is the chart name (= image name = ArgoCD manifest filename prefix), not the GitHub repo name.

Three repository secrets are required: EVENT_PACKAGE_REPO_TOKEN, DOCKER_PAT, ARGOCD_REPO_TOKEN.

7. Image Build (Jib)

Jib builds the image without a Dockerfile or Docker daemon. The OTel javaagent is always baked in (the agent is always present; whether it sends data depends on the Spring otlp profile + endpoint config).

7.1. Pattern

<!-- 1. Always download the OTel agent into the build output -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy-javaagent</id>
            <phase>package</phase>
            <goals><goal>copy</goal></goals>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>io.opentelemetry.javaagent</groupId>
                        <artifactId>opentelemetry-javaagent</artifactId>
                        <version>${opentelemetry-javaagent.version}</version>
                        <outputDirectory>${agent-extraction-root}</outputDirectory>
                        <destFileName>${opentelemetry-javaagent-filename}</destFileName>
                    </artifactItem>
                </artifactItems>
            </configuration>
        </execution>
    </executions>
</plugin>

<!-- 2. Jib copies it into the image at /javaagent/opentelemetry-javaagent.jar -->
<plugin>
    <groupId>com.google.cloud.tools</groupId>
    <artifactId>jib-maven-plugin</artifactId>
    <configuration>
        <to>
            <image>docker.io/christhonie/<service-name>:${project.version}</image>
        </to>
        <extraDirectories>
            <paths>
                <path>
                    <from>${agent-extraction-root}</from>
                    <into>${agent-install-location}</into>
                </path>
            </paths>
        </extraDirectories>
        <container>
            <ports><port>12507</port></ports>
            <environment>
                <OTEL_SERVICE_NAME>${project.artifactId}</OTEL_SERVICE_NAME>
            </environment>
            <jvmFlags>
                <jvmFlag>-Djava.security.egd=file:/dev/./urandom</jvmFlag>
                <jvmFlag>-XX:+UseContainerSupport</jvmFlag>
                <jvmFlag>-XX:MaxRAMPercentage=75.0</jvmFlag>
                <jvmFlag>-javaagent:${agent-install-location}/${opentelemetry-javaagent-filename}</jvmFlag>
                <jvmFlag>-Dotel.logs.exporter=otlp</jvmFlag>
                <jvmFlag>-Dotel.traces.exporter=otlp</jvmFlag>
                <jvmFlag>-Dotel.metrics.exporter=otlp</jvmFlag>
            </jvmFlags>
            <mainClass>${start-class}</mainClass>
        </container>
    </configuration>
</plugin>

Anti-pattern (do not do this): downloading the agent into src/main/jib/ with <extraDirectories><paths>src/main/docker/jib</paths></extraDirectories>. The path mismatch silently drops the agent from the image; the JVM emits a warning at startup and OTel emits nothing. Source-tree pollution is also wrong — agent jars belong in target/.

See Jib Docker Image Build for layering, registry auth, and tagging detail.

8. Helm Chart

Per-service chart under src/main/helm/. Schema is shared across services so operators see the same keys regardless of which service they are configuring.

8.1. Layout

<service>/src/main/helm/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── serviceaccount.yaml
    ├── _helpers.tpl
    └── NOTES.txt

8.2. Standard values keys

config:
    profiles: "prod,otlp"             # comma-separated Spring profiles
    existingsecret: ""                 # K8s Secret name; chart reads, never creates in prod

image:
    repository: docker.io/christhonie/<service-name>
    pullPolicy: IfNotPresent           # Always for dev (mutable SNAPSHOT tags)

imagePullSecrets:
    - name: christhonie-docker

ingress:
    enabled: false
    className: nginx
    annotations: {}
    hosts: []
    tls: []

resources:
    requests: { cpu: 100m, memory: 256Mi }
    limits:   { memory: 512Mi }

replicaCount: 1

podAnnotations: {}                     # populated by the workflow with build-sha; see § Rollover trigger

podSecurityContext:
    fsGroup: 1000
securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: false
    capabilities:
        drop: [ALL]

8.3. Rollover trigger (the podAnnotations propagation pattern)

The chart’s templates/deployment.yaml must propagate .Values.podAnnotations onto the pod template:

template:
  metadata:
    annotations:
      config/secret-version: {{ .Values.config.existingsecret | default "none" | quote }}
      {{- with .Values.podAnnotations }}
      {{- toYaml . | nindent 8 }}
      {{- end }}

Without this, the argocd-update.yml reusable’s yq -i …​ podAnnotations.app.kubernetes.io/build-sha=<sha> has no effect on the pod spec. SNAPSHOT image rebuilds (which carry the same chart version) won’t trigger pod rollover. With this in place, every commit’s SHA differs, the rendered pod template differs, and Helm/ArgoCD rolls the deployment.

Reference implementation: registration-portal/src/main/helm/templates/deployment.yaml.

8.4. Ingress

Two schema variants exist in the EMS fleet:

  • Single-host flat schema (admin-service): hostname, pathType, tls: true. Older.

  • Multi-host array schema (mcp-server): hosts: [{host, paths}], tls: [{secretName, hosts}]. Newer; matches mainline Helm chart conventions.

New services should use the multi-host array schema. Legacy services can keep their existing schema until they are touched for unrelated reasons.

See Helm Chart Structure for full template detail.

9. ArgoCD Manifests

One Application manifest per (service × environment) under ~/dev/idl-xnl-jhb-rc01/argocd/<service-name>-<env>.yml. The cluster’s cluster-bootstrap.yml (an Application that watches the argocd/ directory) auto-creates child Applications for each manifest.

Standard shape:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <service-name>-<env>
  namespace: argocd
spec:
  project: default
  destination:
    name: idl-xnl-jhb1-rc01
    namespace: event-<env>
  source:
    repoURL: registry-1.docker.io
    chart: christhonie/<service-name>
    targetRevision: <version>-RELEASE       # or -SNAPSHOT-RELEASE for dev
    helm:
      releaseName: <env>-<service-name>
      valuesObject: { ... per-env overrides ... }
  syncPolicy:
    automated: { prune: true, selfHeal: true, allowEmpty: false }
    syncOptions: [CreateNamespace=true, ServerSideApply=true]
  revisionHistoryLimit: 3

Promotion convention:

  • dev → auto-bumped by push-dev.yml on every commit to develop.

  • stage → auto-bumped by push-main.yml on every merge to main (i.e. every release).

  • prod → manual targetRevision edit after stage validation. Approval gate by convention, not by automation.

See ArgoCD Deployment Patterns for promotion procedures, secret management, and troubleshooting.

10. OpenTelemetry

Agent is always baked into the image (see § Image Build above). Spring-side instrumentation is profile-gated under otlp.

10.1. otlp profile dependencies (in pom.xml)

<profile>
    <id>otlp</id>
    <properties>
        <profile.otlp>,otlp</profile.otlp>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom</artifactId>
                <version>2.10.0</version>
                <type>pom</type><scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-instrumentation-annotations</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry.instrumentation</groupId>
            <artifactId>opentelemetry-logback-appender-1.0</artifactId>
            <version>2.10.0-alpha</version>
        </dependency>
    </dependencies>
</profile>

10.2. application-otlp.yml

management:
  metrics:
    export:
      otlp:
        enabled: true

otel:
  java:
    global-autoconfigure:
      enabled: true
  exporter:
    otlp:
      endpoint: 'http://opentelemetry-collector.observability.svc.cluster.local:4317'
    jaeger:
      enabled: false
    zipkin:
      enabled: false
  springboot:
    resource:
      enable: true
  resource:
    attributes:
      'service.version': '${project.version}'
      'deployment.environment': production
  instrumentation:
    annotations:
      enabled: true
    logback-appender.enabled: true
    spring-web.enabled: false
    spring-webmvc.enabled: false
    spring-webflux.enabled: false

The Spring Web auto-instrumentation is disabled because the javaagent already instruments these at bytecode level — leaving both on produces duplicate spans.

See OpenTelemetry Configuration for collector topology, sampling, custom metrics, and per-service variations.

11. Profiles

Profile Active when

dev

Local development, dev cluster

prod

Stage, prod

otlp

All deployed environments (dev/stage/prod) — observability

api-docs

All environments (springdoc enabled to serve OpenAPI spec at /v3/api-docs)

tls

Only when terminating TLS in-app (rare; ingress usually handles it)

kubernetes

Some legacy services; not required for new services

dev + prod are mutually exclusive — services should reject the combination at startup (@PostConstruct check).

12. Health Probes

Two endpoints, both permitAll() in the security filter chain:

  • GET /livez — kubelet liveness. Returns 200 OK if the JVM is responsive.

  • GET /readyz — kubelet readiness. Returns 200 OK once the service is ready to accept traffic.

Distinct from /actuator/health/** (heavier, authenticated, used for diagnostic). Helm chart probe stanza:

readinessProbe:
  httpGet: { path: /readyz, port: http }
  initialDelaySeconds: 20
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /livez, port: http }
  initialDelaySeconds: 60
  periodSeconds: 30
  timeoutSeconds: 3
  failureThreshold: 5

13. Logging

Logback config under src/main/resources/logback-spring.xml. Per-line pattern includes MDC fields:

%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%thread] %X{traceId:-} %X{actor:-} %logger{36} - %msg%n

Per-service contextual MDC fields (e.g. mcp-server adds actor and tool) are populated by an aspect or filter. The OTel logback appender automatically emits traceId and spanId when the otlp profile is active, correlating logs to traces in the backend.

13.1. Structured-log conventions (audit / business events)

Use a dedicated logger name (e.g. audit.tool for MCP, audit.access for backend) and emit key=value or JSON for fields. Field names align with admin-service’s LoggingAspect for cross-service grep:

log.info("event=tool.invocation actor={} tool={} params={} latencyMs={} outcome={}",
    sub, toolName, args, latency, outcome);

Elastic EDOT picks up the structured fields directly when OTel logback appender is active.

14. Bootstrap Checklist

Step-by-step from "git init" to "deployed in dev":

  1. GitHub repo created (e.g. christhonie/<service-name>). .gitignore generated.

  2. Required secrets added to the new repo: EVENT_PACKAGE_REPO_TOKEN, DOCKER_PAT, ARGOCD_REPO_TOKEN.

  3. Repository workflow permissions flipped to Read and write.

  4. Reusable workflow access on christhonie/event confirmed to allow this repo (it’s allowed by default if the existing rule is "user-owned").

  5. Maven module scaffolded:

    1. POM with parent za.co.idealogic:event:1.3.1, ${revision} versioning, repositories block, OTel agent properties, Jib + helm-maven-plugin in pluginManagement.

    2. Sources: <package>/<Service>App.java main class.

    3. Resources: application.yml + application-{dev,prod,otlp}.yml, logback-spring.xml, banner.txt.

    4. Test: smoke @SpringBootTest context-load test under src/test/java.

  6. Helm chart under src/main/helm/ — copy from a sibling service and adapt the chart name + port.

  7. .github/workflows/ populated with the five wrappers (push-dev, push-main, push-release, pr-non-main, manual-release-start). Set service-name: to the chart name.

  8. Local verification: mvn clean test (smoke test loads), mvn package jib:dockerBuild (image builds), mvn helm:init helm:lint helm:template helm:package (chart packages).

  9. First push to develop triggers push-dev.yml. Watch via gh run watch.

  10. ArgoCD manifest at idl-xnl-jhb-rc01/argocd/<service-name>-dev.yml modelled on a sibling service. Commit + push.

  11. Pre-create dev Secret in event-dev namespace with whatever the chart’s existingsecret references (API keys, OAuth client secrets, etc.).

  12. Confirm christhonie-docker image-pull secret exists in event-dev.

  13. Watch the rolloutkubectl get pod -n event-dev -l app.kubernetes.io/name=<service-name> -w. First rollout typically completes within 2 minutes of the push-dev workflow finishing.

  14. Smoke-check via the dev URL: curl -fsS https://<service>-dev.idealogic.co.za/livez and any service-specific endpoints.

For staging + prod, follow the GitFlow release flow (see ArgoCD Deployment Patterns § Promotion).

15. Reference Implementations

Service Persona Notes

admin-service

Backend

Heaviest implementation. Liquibase + JPA + Hazelcast + every cross-cutting concern. The deployment-shell reference for backends.

registration-portal

Portal

The reference for portal pattern. Spring Cloud Gateway MVC, session JWT, tenant resolution. Portal Pattern specialises this.

admin-portal (planned)

Portal

Greenfield staff portal. Currently bootstrapping. Same baseline + portal pattern.

mcp-server

MCP adapter

The reference for MCP pattern. Spring AI MCP server + OAuth resource server + per-org API keys. MCP Pattern specialises this.

16. Specialisations

  • Portal Pattern — gateway + Angular SPA topology, OIDC client, session JWT, tenant resolution, BFF endpoints, Hazelcast session replication.

  • MCP Pattern — OAuth resource server, MCP transport (Spring AI starter), @Tool-annotated methods, audit aspect, per-organisation API key map, rate limiting.

  • Java Library Pattern — Maven JAR libraries (parent-pom, event-database, wordpress-database, spreadsheet-importer) published to GitHub Packages. Re-uses the Maven module + CI/CD halves of this baseline; drops the runtime container baseline (Jib, Helm, ArgoCD, OTel, ports, probes).

When no specialisation fits, write a new sibling pattern doc and link it here.

17. Further Reading

Architecture:

Build, deployment, operations:

18. Change History

Date Change

2026-04-27

Initial draft. Extracts the deployment baseline shared by Portal Pattern and MCP Pattern. Captures the GitHub Actions wrapper convention, OTel agent baking, podAnnotations rollover trigger, port assignments, bootstrap checklist.

2026-04-29

Add Java Library Pattern as a third specialisation — re-uses the Maven module + CI/CD halves, drops the runtime container baseline. Triggered by getting spreadsheet-importer onto CI/CD.