Autoscaling Azure DevOps Agents with Container Apps and KEDA

I wanted my self-hosted Azure DevOps agents to disappear when there was no work. Keeping VMs running for an empty queue made little sense, but moving back to Microsoft-hosted agents would have meant giving up the custom tools and private network access I needed.

I ended up using Azure Container Apps Jobs with KEDA. The pool scales from zero to ten agents, with one warm standby during active periods and an offline placeholder to stop Azure DevOps rejecting jobs when the pool is empty.

The Problem with Traditional Self-Hosted Agents

Self-hosted Azure DevOps agents often run on VMs or containers that are always on, so the infrastructure is billed whether pipelines are running or not.

Microsoft-hosted agents solve the idle cost problem but introduce others:

• Cold start delays of 30-60 seconds per job
• Limited customisation - can’t install proprietary tools
• No network access to private resources without complex workarounds

My requirements were scale-to-zero when idle, custom tooling and quick pickup when several jobs arrived together.

Container Apps Jobs with KEDA

Azure Container Apps has a feature called Jobs. Unlike regular Container Apps, which run continuously, Jobs spin up on demand, execute and terminate. That matches the lifecycle of a one-shot pipeline agent.

KEDA (Kubernetes Event-Driven Autoscaling) is built into Container Apps and includes a scaler specifically for Azure Pipelines. It monitors your agent pool’s queue and triggers new containers when jobs are waiting.

How It Works

Pipeline queues job
        │
        ▼
   KEDA polls Azure DevOps API (every 10-30s)
        │
        ▼
   Queue length > 0 detected
        │
        ▼
   Container App Job triggered
        │
        ▼
   Agent container starts (~30-60s)
        │
        ▼
   Agent registers to pool
        │
        ▼
   Agent picks up job and runs it
        │
        ▼
   Job completes → Agent unregisters → Container exits
        │
        ▼
   Queue empty → No new containers started

KEDA handles the scaling decision. The container registers as an agent, runs one pipeline job, unregisters and exits.

Infrastructure as Code

I defined the infrastructure in Terraform.

Container Apps Environment

First, we need an environment to host our jobs:

resource "azurerm_container_app_environment" "agents" {
  name                       = "cae-azp-agents"
  location                   = azurerm_resource_group.agents.location
  resource_group_name        = azurerm_resource_group.agents.name
  log_analytics_workspace_id = azurerm_log_analytics_workspace.agents.id
}

The Agent Job with KEDA Scaling

The Container App Job carries the scaling configuration:

resource "azurerm_container_app_job" "agents" {
  name                         = "azp-agent-job"
  container_app_environment_id = azurerm_container_app_environment.agents.id
  
  replica_timeout_in_seconds = 1800  # 30 min max job duration

  event_trigger_config {
    scale {
      min_executions              = 0   # Scale to zero!
      max_executions              = 10  # Cap concurrent agents
      polling_interval_in_seconds = 10  # Check queue frequently

      rules {
        name             = "azure-pipelines"
        custom_rule_type = "azure-pipelines"

        metadata = {
          poolName                   = "p80-uksouth"
          targetPipelinesQueueLength = "1"  # 1 agent per job
        }
      }
    }
  }
  # ... container template with agent image
}

The targetPipelinesQueueLength = "1" means KEDA will spawn one agent per queued job. If 5 jobs are waiting, 5 containers spin up (up to max_executions).

The N+1 Warm Standby Pattern

Pure event-driven scaling still has a cold start. When the queue goes from empty to active, the container has to start and register before it can take work, which takes 30-60 seconds here.

For bursts of queued work, I added an N+1 standby agent. It stays ready to take the next job while the one-shot agent is busy.

How N+1 Works

When a job is queued, both agents spin up:

Job Queued
    │
    ├──► Job Agent (Container App Job)
    │       • Runs the pipeline job
    │       • Exits when done (--once mode)
    │
    └──► Standby Agent (Container App)
            • Registers and waits
            • Stays running (persistent mode)
            • Picks up NEXT job instantly
            • Scales to 0 after 5 min idle

The standby uses a regular Container App (not a Job) with the same KEDA scale rule. It scales up when the queue has jobs and scales down after 5 minutes of inactivity.

resource "azurerm_container_app" "standby" {
  name          = "azp-agent-standby"
  revision_mode = "Single"

  template {
    min_replicas = 0
    max_replicas = 1

    custom_scale_rule {
      name             = "azure-pipelines-standby"
      custom_rule_type = "azure-pipelines"
      metadata = {
        poolName                   = "p80-uksouth"
        targetPipelinesQueueLength = "1"
      }
    }
  }
}

The difference is in the agent’s run mode. Job agents run with --once and exit after completing a single job. The standby runs persistently, handling multiple jobs while it’s up.

The Placeholder Agent Trick

Azure DevOps has an awkward edge case here: if an agent pool is completely empty, with no online or offline agents, jobs targeting it fail immediately rather than waiting.

I worked around that with a placeholder agent. It is registered once by a manual Container App Job and remains offline in the pool:

resource "azurerm_container_app_job" "placeholder" {
  name = "azp-agent-placeholder"
  
  manual_trigger_config {
    parallelism = 1
  }
  
  template {
    container {
      env {
        name  = "AZP_PLACEHOLDER"
        value = "1"
      }
    }
  }
}

When AZP_PLACEHOLDER=1, the start script registers the agent and exits without cleaning up. The offline record is enough for Azure DevOps to queue jobs normally.

The Agent Docker Image

The agent runs Ubuntu 24.04 with the tools we need:

• .NET 10 SDK
• Node.js 20
• Azure CLI
• Terraform
• PowerShell

The start.sh script handles three modes:

1. One-shot mode (default): Run one job and exit (--once)
2. Persistent mode (AZP_RUN_ONCE=false): Keep running for the standby
3. Placeholder mode (AZP_PLACEHOLDER=1): Register and exit without cleanup

Cost Analysis

Here’s what this architecture costs for a small team:

Scenario	Monthly Cost
Idle (no pipelines)	$0
100 hours of pipeline execution	~$5-10
500 hours of pipeline execution	~$25-50

Compare this to running even a single B2s VM 24/7: ~$30/month. And that’s without autoscaling or redundancy.

The N+1 standby adds ~$0.05-0.10/hour during active periods, but saves 30-60 seconds on every subsequent job in a burst.

Deployment Pipeline

The infrastructure deploys via Azure DevOps (using Microsoft-hosted agents for bootstrapping):

1. Terraform Apply - Creates Container Apps environment, ACR, jobs
2. Build Agent Image - Builds Dockerfile in ACR
3. Register Placeholder - Runs the placeholder job once

After the first deployment, the pool is ready. Queue a job and watch KEDA spin up agents automatically.

Things that caught me out

The KEDA polling interval matters. The default 30 seconds means a job can wait that long before an agent even starts. I reduced it to 10 seconds.

The placeholder is not optional for a new empty pool. Without at least one agent record, even an offline one, Azure DevOps rejects the job.

The cleanup trap needs an exception. Agents normally unregister on exit. The placeholder must skip that cleanup or it disappears as soon as its container stops.

Jobs and regular Container Apps have different roles. The one-shot agents use Jobs; the persistent standby uses a regular Container App.

The result

The result is a pool that costs nothing when idle, starts one agent per queued job up to the configured limit, and keeps a warm spare around while work is active. The placeholder is a slightly odd part of the design, but it makes scale-to-zero behave properly with Azure DevOps.

The full Terraform code is available in my devops repository.