ArgoCD Deployment Patterns

1. Overview

EMS services are deployed to Kubernetes via ArgoCD. One ArgoCD Application manifest per (service × environment) declares which chart version to pull, which values to apply, and which namespace to deploy into. ArgoCD watches the manifest repository and reconciles the live cluster state against the declared state.

Manifests live outside this repository — in ~/dev/idl-xnl-jhb-rc01/argocd/ (locally; repository name idl-xnl-jhb-rc01). That repository is the source of truth for what runs where. This page documents the patterns so new services can slot in.

See also: Helm Chart Structure, Jib Docker Build.

2. Manifest Layout

idl-xnl-jhb-rc01/argocd/
├── event-admin-service-dev.yml
├── event-admin-service-stage.yml
├── event-admin-service-prod.yml
├── registration-portal-dev.yml
├── registration-portal-stage.yml
├── registration-portal-prod.yml
├── membership-ui-dev.yml
├── ...
├── admin-portal-dev.yml           # future (WS7)
├── admin-portal-stage.yml
├── admin-portal-prod.yml
├── opentelemetry-collector.yml
├── mysql-idealogic-prod.yml
├── external-dns.yml
├── certificates.yml               # cert-manager issuers
└── ...

Convention: one file per ArgoCD Application, named <service>-<env>.yml. Infrastructure (OTel collector, ingress certs, MySQL operator) has its own files at the top level.

3. Application Manifest Structure

Reference: idl-xnl-jhb-rc01/argocd/event-admin-service-prod.yml.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: event-admin-service-prod
  namespace: argocd
spec:
  project: default
  destination:
    name: idl-xnl-jhb1-rc01
    namespace: event-prod
  source:
    repoURL: registry-1.docker.io
    chart: christhonie/event-admin-service
    targetRevision: 2.3.31-RELEASE
    helm:
      releaseName: prod-event-admin-service
      valuesObject:
        config:
          profiles: "prod,kubernetes"
          existingsecret: event-admin-service
          db:
            url: mysql://idealogic-prod.mysql.svc.cluster.local:6446/wpca_prod?useUnicode=true&characterEncoding=utf8
            username: event-membership-prod
          security:
            oauth2:
              enabled: false
              issuer: https://...
          mail:
            host: myriadevents-co-za.mail.protection.outlook.com
            port: 25
            username: [email protected]
            # password: via existingsecret, NOT inline
          liquibase:
            contexts: "prod"
          otel:
            enabled: true
            url: "http://opentelemetry-collector.observability.svc.cluster.local:4318"
        image:
          pullPolicy: IfNotPresent
        imagePullSecrets:
          - name: christhonie-docker
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Key fields:

Field Purpose

metadata.name

<service>-<env> — matches filename

spec.project

default for EMS services; separate projects for infra if RBAC needs it

spec.destination.name

ArgoCD cluster secret name (currently all one cluster: idl-xnl-jhb1-rc01)

spec.destination.namespace

event-dev / event-stage / event-prod / observability / mysql etc.

spec.source.repoURL

registry-1.docker.io for OCI-hosted charts (our pattern); Git URL for Git-hosted charts

spec.source.chart

christhonie/event-admin-service — chart name within the OCI registry

spec.source.targetRevision

Pinned chart version — bump this to promote a release

spec.source.helm.releaseName

<env>-<chartname> — yields service names like prod-event-admin-service

spec.source.helm.valuesObject

Environment-specific values; merged over chart defaults

spec.syncPolicy.automated

prune: true removes resources that disappear from the manifest; selfHeal: true reconciles manual drifts

4. Environments

Env Namespace Purpose

dev

event-dev

Continuous deployment from develop branch builds (-SNAPSHOT images). Auto-upgrade; no guardrails. Shared data.

stage

event-stage

Pre-release testing. Manual promotion of specific release versions. Prod-cloned data (see design-journal/2026-03/prod-to-qa-data-clone.adoc).

prod

event-prod

Live traffic. Manual promotion, approval gate, pinned versions only.

Release flow:

  1. Feature branch merged to develop — CI builds X.Y.Z-SNAPSHOT, dev ArgoCD picks up automatically.

  2. Release branch cut — CI builds X.Y.Z-RELEASE.

  3. Stage ArgoCD manifest updated (commit to idl-xnl-jhb-rc01) to point at X.Y.Z-RELEASE — ArgoCD reconciles within seconds.

  4. UAT in stage.

  5. Prod ArgoCD manifest updated to X.Y.Z-RELEASE — ArgoCD reconciles.

No latest tag ever referenced in stage or prod manifests.

5. Secret Management

Every prod manifest uses existingsecret: <name>. The secret is created out-of-band with kubectl create secret or via SealedSecrets. Never put secrets in the ArgoCD manifest — it is committed to git.

Known secret shapes:

5.1. event-admin-service secret (prod)

Keys:

  • jwtencryptionkey — admin-service signing key

  • dbpassword — MySQL password

  • mailpassword — SMTP password

  • apikey — admin-service’s own copy of the portal-to-admin-service API key

  • externalauthsecretkey — base64 secret for external-auth endpoints

  • jasperreportspassword — credentials for the Jasper Reports integration

5.2. event-admin-portal secret (future prod)

Keys:

  • apikey — portal’s X-API-KEY for calling admin-service

  • oidcclientsecret — OIDC client secret

  • any portal-specific credentials

Create the secret when onboarding the environment:

kubectl create secret generic event-admin-portal \
  -n event-prod \
  --from-literal=apikey=<value> \
  --from-literal=oidcclientsecret=<value>

Rotation is a runbook activity — see API-Key Injection § Rotation Procedure.

5.3. Image pull secret

christhonie-docker must exist in every namespace that pulls EMS images:

kubectl create secret docker-registry christhonie-docker \
  -n event-prod \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=christhonie \
  --docker-password=<token>

6. Promotion Pattern

Concrete steps to promote admin-service from stage to prod:

  1. Verify stage has the new version running cleanly — kubectl get deploy -n event-stage, smoke-test the URL.

  2. In idl-xnl-jhb-rc01, edit argocd/event-admin-service-prod.yml:

    spec:
  source:
    targetRevision: 2.3.32-RELEASE   # was 2.3.31-RELEASE

  3. Commit + push. ArgoCD webhook or poll picks up the change within seconds.

  4. Watch rollout:

    argocd app get event-admin-service-prod --refresh
kubectl rollout status deploy/prod-event-admin-service -n event-prod

  5. If rollout fails — rollback is reverting the commit (argocd auto-reconciles back to the previous version).

ArgoCD’s automated.selfHeal: true means manual kubectl edit on a live resource is reverted on the next sync. Always make changes via the manifest, not the cluster.

7. Infrastructure Applications

Non-service ArgoCD apps in the same repo:

App Purpose

opentelemetry-collector.yml

OTel Collector deployment — receives OTLP from all services, exports to the observability backend. See OpenTelemetry Configuration.

mysql-idealogic-prod.yml

Oracle MySQL Operator + cluster. Managed by the operator; we provide the InnoDBCluster CR and backup schedule.

mysql-operator.yml

The operator itself

external-dns.yml

External-DNS configuration for ingress record management

certificates.yml

cert-manager ClusterIssuers (Let’s Encrypt prod + staging)

redis-prod.yml

Redis for session sharing (fallback to Hazelcast; keep for future needs)

memcache-prod.yml

Memcached — legacy, under review

greenmail-dev.yml

GreenMail deployment in the dev cluster for SMTP testing

storage.yml

StorageClass + PV setups

ssh-bastion.yml

Bastion host for MySQL / in-cluster debugging — see memory/reference_mysql_claude_user.md

sonarqube.yml, chatwoot-prod.yml, wordpress-wpca-prod.yml, etc.

Adjacent systems; not EMS core but deployed by the same mechanism

8. admin-portal ArgoCD Onboarding

WS7 tasks:

  1. Create admin-portal-dev.yml / admin-portal-stage.yml / admin-portal-prod.yml in idl-xnl-jhb-rc01/argocd/. Copy from registration-portal-<env>.yml as the shape closest to admin-portal.

  2. Adjust:

    • chart: christhonie/event-admin-portal

    • targetRevision: <first-published-version>

    • releaseName: dev-event-admin-portal / stage-event-admin-portal / prod-event-admin-portal

    • Namespace follows the convention (event-dev / event-stage / event-prod)

    • config.profiles: "dev,kubernetes,api-docs" for dev, "prod,kubernetes,api-docs,otlp" for prod

    • config.services.eventadminservice: in-cluster URL

    • config.security.oauth2: populated with the staff IdP

    • Ingress host: admin-dev.event.idealogic.co.za, admin-stage.event.idealogic.co.za, admin.event.idealogic.co.za

  3. Create the event-admin-portal secret in each namespace with apikey + oidcclientsecret.

  4. Verify christhonie-docker image-pull secret exists in each namespace.

  5. Push the manifests; ArgoCD reconciles.

9. Operational Notes

  • Sync hooks: none in current use. ArgoCD’s pre/post sync hooks (e.g. database migration jobs) are not needed because Liquibase runs on app startup.

  • Sync wave: ordering is not currently enforced via argocd.argoproj.io/sync-wave. All resources come up simultaneously.

  • Health checks: default Kubernetes health (Deployment Ready) is used. Custom health for Helm-managed resources relies on the chart’s own probes (/livez, /readyz).

If a chart upgrade introduces a CRD or new resource type, ArgoCD may need a restart to recognise it — rare; the standard Kubernetes types cover every current resource.

10. Troubleshooting

10.1. ArgoCD app stays OutOfSync after chart bump

  1. argocd app get <name> --refresh — force a re-read from the chart repo.

  2. If still stuck, inspect the rendered manifest: argocd app manifests <name> — validate against kubectl apply --dry-run=server -f -.

  3. Check the chart was actually published: helm pull oci://registry-1.docker.io/christhonie/event-admin-service --version <ver> locally.

10.2. Secret reference missing

kubectl get events -n <namespace> shows CreateContainerConfigError with the secret name. Either the secret doesn’t exist (create it) or the namespace is wrong (verify destination.namespace).

10.3. Automated sync loop

If you see ArgoCD reverting a manual change, that’s selfHeal: true working as designed. Make the change in the manifest, commit, push.

To pause reconciliation temporarily:

argocd app set <name> --sync-policy none
# do thing
argocd app set <name> --sync-policy auto

11. Reference

File Role

~/dev/idl-xnl-jhb-rc01/argocd/event-admin-service-prod.yml

Full prod Application example

~/dev/idl-xnl-jhb-rc01/argocd/registration-portal-prod.yml

Portal-shaped Application example

~/dev/idl-xnl-jhb-rc01/argocd/opentelemetry-collector.yml

Infrastructure Application example

admin-service/src/main/helm/

The chart that ArgoCD templates

12. Related Documentation

13. Change History

Date Change

2026-04-24

Initial draft. Grounded in ~/dev/idl-xnl-jhb-rc01/argocd/ prod manifests (event-admin-service-prod, registration-portal-prod).