Docker Agent Setup: Environment Variables Guide

Environment variables make configuring Docker containers easier and more secure. Here’s why they matter and how to use them for setting up the Datadog Docker Agent:

• Why use them?
  • Change container settings without modifying code or rebuilding images.
  • Keep sensitive data (like API keys) secure.
  • Easily switch between development, testing, and production setups.
• What you need:
  • Docker installed and running.
  • Datadog API key for authentication.
  • Proper permissions to manage Docker and system files.
• Key environment variables:
  • DD_API_KEY: Required for authentication.
  • DD_SITE: Specifies your Datadog site (e.g., datadoghq.com).
  • Optional: DD_TAGS, DD_LOG_LEVEL, DD_LOGS_ENABLED, and more for custom configurations.
• Setup methods:
  • Use the Docker CLI with the -e flag for quick setups.
  • Define variables in a Dockerfile for consistency.
  • Use Docker Compose for multi-container environments.
• Best practices:
  • Secure sensitive data using Docker Secrets.
  • Test configurations with docker inspect and docker exec.
  • Adjust logging levels for troubleshooting.

Environment variables offer flexibility and security for managing the Datadog Docker Agent. Ready to dive in? Follow the guide to get started.

TNS Tutorials: Datadog - Monitoring Docker Containers on AWS

Setup Requirements

Getting started with the Datadog Docker Agent requires the right tools and permissions to ensure everything runs smoothly.

Required Software and Access

Before diving in, make sure you have these essential components in place:

Core Requirements:

• Docker runtime: Either Docker Community Edition or Enterprise Edition will work.
• Docker daemon access: This is crucial for managing containers.
• Datadog API key: You'll need a valid key for authentication.
• Cgroup for memory management: Ensure this is enabled on your host system.

If you're planning to monitor Java applications using JMX, you'll need a Docker image that supports JMX.

Once the software is installed, double-check that your permissions align with these requirements.

Required Permissions

Proper permissions are essential for smooth operation. Here's what to check:

System-Level Access:

• Access to the Docker daemon socket.
• The ability to manage environment variables.
• Permissions to access necessary system files.
• Authorization to mount host directories when required.

Avoid using overly broad commands like sudo chmod 666 /var/run/docker.sock. Instead, focus on setting up targeted access controls to maintain security.

Special Considerations:

• Linux: Port 9001 must be accessible. If you're using SELinux, you may need to adjust its settings or run in privileged mode.
• Windows: Ensure Hyper-V is enabled to support container isolation.
• Both platforms: Verify that all required tools are included in the container's PATH.

Main Environment Variables

To set up the Datadog Docker Agent, you'll need to configure some key environment variables. These settings are essential for connecting your Agent to Datadog and customizing its behavior.

Required Variables

At a minimum, you must define two critical environment variables:

• DD_API_KEY: This is your unique Datadog API key, used for authentication.
• DD_SITE: Indicates your Datadog site location (e.g., datadoghq.com for the US or datadoghq.eu for the EU).

These variables are essential for enabling communication between the Docker Agent and Datadog's backend services. Here's an example of a basic setup:

docker run -e DD_API_KEY=<YOUR_API_KEY> -e DD_SITE="datadoghq.com" datadog/agent:latest

While these variables are the foundation, you can further fine-tune the Agent's behavior with custom configurations.

Custom Variables

Custom environment variables allow you to tailor the Agent to your specific needs. Here are some examples:

These options provide flexibility, letting you adapt the Agent to your environment.

Network and Log Settings

For more advanced configurations, you can define variables to manage network and logging behaviors.

Log Collection Settings:

• DD_LOGS_ENABLED: Turns on log collection.
• DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL: Automatically collects logs from all containers.

Network Configuration:

• DD_AGENT_HOST: Specifies the host running the Datadog Agent.
• DD_APM_NON_LOCAL_TRAFFIC: Allows APM traffic from external sources.

If you're using Alpine-based images, note that configuration files are stored in /opt/datadog-agent/agent/conf.d/ instead of /etc/dd-agent/conf.d/.

Dynamic Configuration with Environment Variables

One of the Agent's strengths is its ability to dynamically configure settings using environment variables. For example, when setting up database monitoring, you can use placeholders like %%env_VARIABLE_NAME%% to inject values dynamically:

com.datadoghq.ad.checks='{"postgres": {
    "init_config": {},
    "instances": [{
      "dbm": true,
      "host": "%%env_POSTGRES_HOST%%",
      "port": 5432,
      "username": "datadog",
      "password": "%%env_POSTGRES_PASSWORD%%"
    }]
  }}'

This approach ensures flexibility in your monitoring setup, making it easier to adapt configurations as needed.

Setting Up Environment Variables

You can configure environment variables using three main approaches. These methods ensure that the required and custom variables are applied effectively.

Docker CLI Setup

With Docker CLI, the -e flag allows you to set environment variables directly:

docker run -d --name dd-agent \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e DD_SITE="datadoghq.com" \
  -e DD_HOSTNAME="prod-web-01" \
  -e DD_TAGS="env:production,team:backend" \
  -e DD_LOG_LEVEL="INFO" \
  datadog/agent:latest

Note: If you're using Amazon Linux versions earlier than 2, replace the volume flag with: -v /cgroup/:/host/sys/fs/cgroup:ro.

For configurations that need to persist, consider using the Dockerfile method described below.

Dockerfile Setup

To make environment variables persistent, define them directly in your Dockerfile:

FROM datadog/agent:latest

ENV DD_SITE="datadoghq.com"
ENV DD_LOG_LEVEL="INFO"

ARG DD_API_KEY
ENV DD_API_KEY=$DD_API_KEY

ENV DD_DOGSTATSD_NON_LOCAL_TRAFFIC="true"
ENV DD_APM_ENABLED="true"

Then, build your image using the following command:

docker build --build-arg DD_API_KEY=<YOUR_API_KEY> -t custom-datadog-agent .

This method is especially useful for creating consistent configurations across different containers. For multi-container setups, check out the Docker Compose option.

Docker Compose Setup

Docker Compose simplifies managing environment variables in multi-container environments. Here's an example configuration:

version: '3'
services:
  datadog:
    image: datadog/agent:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
    environment:
      - DD_API_KEY=${DD_API_KEY}
      - DD_SITE=datadoghq.com
      - DD_APM_ENABLED=true
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true
      - DD_HOSTNAME=compose-host
      - DD_TAGS=env:staging,service:api
    restart: unless-stopped

  app:
    build: .
    depends_on:
      - datadog

To securely store your API key, use a .env file:

DD_API_KEY=<YOUR_API_KEY>

This configuration ensures that metrics are collected from all containers within your Docker Compose environment.

Advanced Settings

These advanced settings help you fine-tune your Datadog Docker Agent, building on the basics to improve performance as your infrastructure grows.

Autodiscovery Setup

Autodiscovery simplifies container monitoring by automatically detecting and configuring checks. To enable it, use the following command:

docker run -d --name dd-agent \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
  -e DD_API_KEY=<YOUR_API_KEY> \
  -e SD_BACKEND=docker \
  -e DD_COLLECT_LABELS_AS_TAGS='com.docker.label.foo,com.docker.label.bar' \
  datadog/agent:latest

For more precise detection, you can define additional key-value variables:

Once autodiscovery is set up, make sure to secure sensitive variables to complete your advanced configuration.

Secure Variable Management

Protect sensitive information, like API keys, by using Docker Secrets. This method stores and transmits sensitive data securely, mounting it as files in the /run/secrets/ directory. Here's an example configuration:

version: '3.8'
services:
  datadog:
    image: datadog/agent:latest
    secrets:
      - dd_api_key
    environment:
      - DD_API_KEY_FILE=/run/secrets/dd_api_key

secrets:
  dd_api_key:
    external: true

By managing your secrets securely, you can ensure your setup is both safe and efficient.

Configuration for Growth

As your infrastructure scales, it's important to optimize your monitoring setup. Use variables like DD_TAGS, DD_LOG_LEVEL, and NON_LOCAL_TRAFFIC to manage resources effectively and streamline data collection:

-e DD_TAGS="env:production,team:backend,region:us-east"
-e DD_LOG_LEVEL="INFO"
-e NON_LOCAL_TRAFFIC=true

These settings help you categorize metrics and handle increasing container counts without sacrificing performance. For autodiscovery templates, centralizing configurations in a key-value store can simplify maintenance and improve scalability.

For more detailed advice on scaling your infrastructure with Datadog, check out Scaling with Datadog for SMBs.

Testing and Fixes

Make sure your Docker Agent environment variables are correctly configured to ensure proper monitoring and functionality.

Checking Variables

You can verify your environment variables using the following commands:

# List all environment variables
docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' dd-agent

# Check a specific variable
docker exec dd-agent sh -c 'echo $DD_API_KEY'

# View all variables within the container
docker exec dd-agent /bin/sh -c 'printenv'

For sensitive variables like API keys, you can extract just the value with this command:

docker exec dd-agent /bin/sh -c 'printenv' | grep '^DD_API_KEY=' | cut -d= -f2-

These steps help confirm that your environment variables are configured correctly before diving into troubleshooting.

Common Problems and Solutions

Below are some common configuration issues and how to resolve them:

If these fixes don’t resolve your issue, you may need to analyze logs for more details.

Log Analysis

Adjusting the logging level can help you pinpoint issues more effectively. For example, you can increase the logging detail level by running:

docker run -d \
  -e DD_LOG_LEVEL="DEBUG" \
  -e DD_API_KEY=<YOUR_API_KEY> \
  datadog/agent:latest

"The syntax is simple. Use %%env_<VARIABLE_NAME>%% - this way the datadog-agent knows where to get the environment variables from." - Niklas Metje

When troubleshooting, begin with INFO level logging, which provides a good balance of detail and performance. Only switch to DEBUG mode if you need more in-depth diagnostics. This approach ensures you gather the necessary insights without impacting system performance.

Summary

Setup Steps Review

Here’s a quick rundown of the key configuration, security, and scalability practices we’ve covered so far:

Critical Configuration Steps:

• Set DD_API_KEY for authentication.
• Define DD_HOSTNAME to identify the agent.
• Enable autodiscovery for streamlined service detection.
• Adjust logging levels to assist with troubleshooting.

Security Essentials:

• Avoid embedding sensitive data directly in Dockerfiles.
• Use Docker secrets or secure environment variables for storing credentials.
• Always run containers as non-root users.
• Apply the --security-opt=no-new-privileges flag to enhance container security.

These steps lay the groundwork for building a robust and scalable monitoring setup.

Moving Forward

To ensure efficient monitoring, keep these best practices in mind:

Configuration Management Tips:

Here’s an example of how to implement these practices in a docker run command:

docker run -d \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /proc/:/host/proc/:ro \
  -v /sys/fs/cgroup/:/host/sys/cgroup:ro \
  -e DD_API_KEY={your_api_key} \
  -e SD_BACKEND=docker \
  datadog/agent:latest

"The syntax is simple. Use %%env_<VARIABLE_NAME>%% - this way the datadog-agent knows where to get the environment variables from." – Niklas Metje

FAQs

How can I securely manage sensitive environment variables when setting up the Datadog Docker Agent?

To keep sensitive environment variables safe when setting up the Datadog Docker Agent, here are some essential practices to follow:

• Utilize Docker Secrets: Instead of passing sensitive data directly as environment variables, use Docker Secrets. This approach keeps your credentials out of Docker images and ensures tighter access control.
• Avoid Embedding Credentials: Never hardcode sensitive information into your Dockerfiles or environment variables. Instead, rely on secret management tools like HashiCorp Vault or AWS Secrets Manager to securely store and retrieve credentials when needed.
• Use Environment Files: Create .env files specific to each environment (e.g., development or production) to control variable exposure. Make sure these files are excluded from version control and accessible only to authorized users.

Following these practices helps minimize the risk of exposing sensitive data in your Docker Agent setup.

What are the best practices for managing environment variables in a Docker Compose multi-container setup?

To manage environment variables effectively in a Docker Compose multi-container setup, here are some practical tips:

• Use a .env file: Place your environment variables in a .env file located in the same directory as your docker-compose.yml. This approach centralizes configurations, simplifies management, and makes sharing easier, all while keeping sensitive data out of version control.
• Avoid embedding sensitive data: Never hardcode sensitive information directly in your docker-compose.yml file. Instead, use Docker secrets or external configuration tools to securely manage sensitive data.
• Adopt environment-specific files: Maintain separate .env files for development, testing, and production environments. This lets you adjust configurations for different stages without altering the main docker-compose.yml.

These steps help ensure your multi-container applications remain secure, well-organized, and efficient.

Why aren’t my environment variables working with the Datadog Docker Agent?

If the Datadog Docker Agent isn't recognizing your environment variables, the first step is to verify that they're properly set up in your Docker configuration. These variables can be defined either in your Docker Compose file or directly in the docker run command. Remember, if a variable is specified in both the global configuration file (datadog.yaml) and as an environment variable, the environment variable will override the global setting.

Once you've confirmed the setup, check the container logs for any warnings or errors related to these variables. Look for messages indicating missing or incorrectly configured variables, and double-check their spelling and formatting. Additionally, ensure the Datadog Agent has the necessary permissions to access these variables, particularly in environments like CI/CD pipelines. If the problem continues, diving into the specific Docker container logs can often reveal the underlying issue.

Related posts

• Getting Started with Datadog on Engine Yard: Manual Setup Guide
• How to Monitor Fly.io Apps with Datadog Using Sidecar or API Integration
• Datadog on DigitalOcean: Monitoring Droplets, DOKS, and More
• Linux vs. Windows: Datadog Agent Installation