# GitOps with ArgoCD — A Practical Guide to Declarative Deployments

## The Deployment Problem Nobody Talks About

Your CI pipeline builds the image. Then what? Someone runs `kubectl apply`. Or a Helm command buried in a deploy step. Or a script that SSHes into a bastion host. Three months later, nobody knows what's actually running in production — because the cluster state drifted from what's in Git.

GitOps fixes this by making Git the single source of truth for your cluster state. [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) is the most widely adopted GitOps operator for Kubernetes, and for good reason: it's declarative, it auto-syncs, and it shows you exactly where reality diverges from intent.

This guide covers how to set it up properly — not the "hello world" version, but the version that survives real workloads.

## What GitOps Actually Means

GitOps is a deployment pattern with four principles, [formalized by the OpenGitOps project](https://opengitops.dev/):

1. **Declarative** — the entire desired system state is described declaratively (YAML, Helm charts, Kustomize overlays)
2. **Versioned and immutable** — the desired state is stored in Git, giving you audit trail and rollback for free
3. **Pulled automatically** — an agent (ArgoCD) continuously reconciles cluster state to match Git
4. **Continuously reconciled** — drift is detected and corrected without manual intervention

The key shift: your CI pipeline no longer touches the cluster. It builds and pushes an image, then updates a manifest in Git. ArgoCD handles the rest.

## Installing ArgoCD

The recommended installation uses the official Helm chart. For production, use the HA manifest:

```bash
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml
```

Or via Helm for more control:

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --set server.extraArgs={--insecure} \
  --set configs.params."server\.insecure"=true
```

The `--insecure` flag disables TLS on the ArgoCD server itself — you'll terminate TLS at your ingress controller, which is the standard pattern. Don't expose ArgoCD without TLS termination somewhere in the chain.

Get the initial admin password:

```bash
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d
```

Change it immediately. Better yet, [configure SSO via OIDC](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#sso) and disable the admin account entirely.

## Repository Structure That Scales

The most common mistake: putting application manifests in the same repo as the application code. This creates circular dependencies — a code change triggers CI, which updates manifests, which triggers CI again.

Use a dedicated **config repository**:

```
infra-gitops/
├── apps/
│   ├── api/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   ├── kustomization.yaml
│   │       │   └── replicas-patch.yaml
│   │       └── production/
│   │           ├── kustomization.yaml
│   │           └── replicas-patch.yaml
│   └── frontend/
│       ├── base/
│       └── overlays/
├── platform/
│   ├── cert-manager/
│   ├── ingress-nginx/
│   └── monitoring/
└── argocd/
    ├── projects.yaml
    └── applicationsets.yaml
```

Key decisions:

- **Kustomize over Helm for app manifests** — Helm is great for third-party charts. For your own services, Kustomize overlays are simpler to review in PRs and easier to diff.
- **Separate `apps/` from `platform/`** — platform components (cert-manager, ingress, monitoring) have different change cadences and approval requirements.
- **One directory per environment overlay** — makes promotion explicit and auditable.

## Application Manifests

An ArgoCD `Application` resource tells the controller what to deploy and where:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-staging
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/infra-gitops.git
    targetRevision: main
    path: apps/api/overlays/staging
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
```

The critical settings:

- **`automated.prune: true`** — removes resources that no longer exist in Git. Without this, deleted manifests leave orphaned resources in the cluster.
- **`automated.selfHeal: true`** — reverts manual `kubectl` changes. This is the whole point of GitOps — if someone patches something by hand, ArgoCD corrects it.
- **`retry`** — transient failures happen (API server overload, webhook timeouts). Exponential backoff prevents cascading retries.

## ApplicationSets for Multi-Environment

Managing individual `Application` resources per service per environment doesn't scale. [ApplicationSets](https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/) generate Applications from templates:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/your-org/infra-gitops.git
        revision: main
        directories:
          - path: apps/*/overlays/*
  template:
    metadata:
      name: '{{ index .path.segments 1 }}-{{ index .path.segments 3 }}'
    spec:
      project: default
      source:
        repoURL: https://github.com/your-org/infra-gitops.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ index .path.segments 1 }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
```

This auto-discovers every `apps/<service>/overlays/<env>` directory and creates an Application for it. The `goTemplate: true` flag enables Go template syntax, which is the recommended mode for new ApplicationSets. Add a new service? Create the directory structure, push, done.

## Secrets Management

The one thing you **cannot** put in Git as-is: secrets. Three viable approaches:

### Sealed Secrets

[Bitnami Sealed Secrets](https://sealed-secrets.netlify.app/) encrypts secrets client-side with a cluster-specific public key. Only the controller in the cluster can decrypt them.

```bash
kubeseal --format yaml < secret.yaml > sealed-secret.yaml
```

The sealed secret is safe to commit. Simple, but key rotation requires re-encrypting all secrets.

### External Secrets Operator

[External Secrets Operator](https://external-secrets.io/) syncs secrets from external stores (AWS Secrets Manager, Vault, GCP Secret Manager) into Kubernetes secrets:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: api-secrets
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: /production/api/database-url
```

This is the recommended approach for production. Secrets live in a dedicated secrets manager with its own access policies, audit logging, and rotation. The Kubernetes secret is derived, not source-of-truth.

### SOPS

[Mozilla SOPS](https://github.com/getsops/sops) encrypts specific values in YAML files. ArgoCD has [native SOPS support via a plugin](https://github.com/argoproj-labs/argocd-vault-plugin). Good middle ground if you don't want to run an external secrets manager.

## Environment Promotion

The promotion flow in GitOps:

1. CI builds image `api:sha-abc123` and pushes to registry
2. CI opens a PR against the config repo updating the staging overlay's image tag
3. ArgoCD syncs staging automatically
4. After validation, a second PR (or manual merge) updates the production overlay
5. ArgoCD syncs production

Automate step 2 with [image updater](https://argocd-image-updater.readthedocs.io/en/stable/) or a simple CI job:

```yaml
# In your app repo's CI pipeline (separate step after cloning infra-gitops)
- name: Update staging manifest
  run: |
    kustomize edit set image api=$IMAGE_TAG
    git commit -am "chore: update api to $IMAGE_TAG"
    git push
  working-directory: infra-gitops/apps/api/overlays/staging
```

For production promotion, require a PR with approval. This gives you:

- **Audit trail** — who approved the production deploy and when
- **Rollback** — `git revert` the merge commit
- **Diff review** — the PR shows exactly what changed between environments

## Sync Windows and Waves

Not everything should sync immediately. ArgoCD supports [sync windows](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) to restrict when syncs happen:

```yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
spec:
  syncWindows:
    - kind: allow
      schedule: '0 8-17 * * 1-5'
      duration: 9h
      applications: ['*']
      timeZone: Europe/Berlin
```

This restricts production syncs to business hours on weekdays — when someone is around to respond if things break.

For ordering dependencies (database migration before app deployment), use [sync waves](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/):

```yaml
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"  # Runs before wave 0
```

Negative waves run first. Use wave -1 for migrations, wave 0 for the app, wave 1 for post-deploy checks.

## Monitoring and Alerts

ArgoCD exposes Prometheus metrics on `:8082/metrics`. The essential alerts:

- **App sync failed** — `argocd_app_info{sync_status="OutOfSync"}` for more than 10 minutes
- **App health degraded** — `argocd_app_info{health_status!="Healthy"}` for more than 5 minutes
- **Sync operation errors** — `argocd_app_sync_total{phase="Error"}` rate increase

ArgoCD also supports [notifications](https://argo-cd.readthedocs.io/en/stable/operator-manual/notifications/) via Slack, Teams, webhooks, or email. Configure at minimum: sync failed and health degraded notifications for production apps.

## Common Pitfalls

**Putting secrets in Git unencrypted.** Sounds obvious, but it happens. Use pre-commit hooks with [gitleaks](https://github.com/gitleaks/gitleaks) or [detect-secrets](https://github.com/Yelp/detect-secrets) to catch this before it lands.

**Not enabling pruning.** Without `prune: true`, deleting a manifest from Git leaves the resource running. You end up with ghost services that nobody maintains.

**Ignoring resource hooks.** ArgoCD respects [resource hooks](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/) for pre-sync and post-sync jobs. Use `PreSync` hooks for database migrations instead of init containers — they're visible in the ArgoCD UI and have proper error handling.

**Syncing everything to one namespace.** Use the destination namespace in your Application to enforce namespace boundaries. Combine with ArgoCD projects to restrict which namespaces a team can deploy to.

## The Bottom Line

GitOps with ArgoCD removes the "what's running in production?" guessing game. Every deployment is a Git commit. Every rollback is a revert. Every change is reviewable.

The setup investment is front-loaded — repository structure, ApplicationSets, secrets management, sync policies. Once it's running, deployments become boring. And boring deployments are exactly what you want.

Start with one service in staging. Get the feedback loop right. Then expand to production and add services incrementally. Don't try to migrate everything at once.