5 Tips To Optimize Datadog Data Retention

Want to cut Datadog costs while keeping critical data? Start by fine-tuning your data retention strategy. Here's a quick guide to help you save money, improve system performance, and stay compliant - all without losing essential insights.

Key Takeaways:

• Filter Logs: Exclude unnecessary logs (e.g., debug logs, health checks) to reduce storage costs by up to 58%.
• Custom Retention Rules: Use tiered retention periods (e.g., 30 days for hot data, 180+ days for archives) to balance cost and accessibility.
• Tag-Based Storage: Organize data with tags (e.g., priority:high) for tailored retention policies.
• Reduce Alert Noise: Consolidate alerts and adjust thresholds to avoid excess data storage.
• Adjust Data Rollups: Use lower resolutions for older data to save space while retaining key insights.

Quick Comparison: Data Retention Strategies

These tips make it simple to manage Datadog data efficiently. Start implementing them today to cut storage costs and streamline your monitoring setup.

Datadog Cloud Cost Management: Actionable Cost Data

1. Set Up Log Filters

Managing log filters effectively can help keep storage costs under control. For example, Datadog charges $0.10 per GB-month, so filtering unnecessary logs is crucial.

Start by identifying logs that don't add much value:

• Debug or verbose logs in production
• Repeated health check logs
• Infrastructure noise that isn't actionable
• Duplicate error logs caused by retry mechanisms

Here’s how to configure filters in Datadog:

1. Go to Logs → Processing → Exclusion Filters.
2. Use specific query syntax to create a filter (e.g., http.url_details.path:"/health" AND status:info).
3. Set the exclusion percentage (start with 100% for logs that aren’t critical).
4. Add a clear, descriptive name to your rule for easy tracking.

For example, a Fortune 500 retailer applied 12 exclusion rules to 23 million daily logs and reduced their costs by 58%.

To refine filtering further, pair service tags with exclusion rules. For instance:

env:staging AND (service:payment-api OR team:frontend)

This approach gives you detailed control while keeping essential logs visible.

"Over-filtering can create monitoring blind spots - maintain 10-15% sampling of excluded log types." - Datadog's engineering team

Tiered Filtering Strategy

A tiered approach ensures you balance cost savings with necessary visibility:

For instance, CloudPosse used Lambda Forwarder filtering with EXCLUDE_AT_MATCH='health check', reducing daily indexed logs by 41%. This saved $12,500 per month while still storing all logs in S3 archives.

2. Create Custom Retention Rules

Custom retention rules can help manage costs and ensure compliance. Datadog’s standard retention periods - 15 days for logs and 15 months for metrics - might not match your business requirements.

Implementing a Tiered Retention Strategy

A tiered retention strategy organizes data by importance and usage patterns, balancing cost and accessibility:

For example, in January 2025, a healthcare provider cut storage costs by 35% while staying HIPAA-compliant by archiving patient logs after 90 days.

Once you’ve defined your strategy, you can configure these rules directly in Datadog.

Setting Up Custom Rules

Follow these steps to configure retention rules in Datadog:

1. Go to Organization Settings → Retention Filters.
2. Click New Retention Filter.
3. Enter a filtering query.
4. Set your desired retention period.
5. Add a Cost Center tag for tracking.

"The 90-Day Baseline approach works best for SMBs - start with 90-day retention for core metrics, then adjust based on actual usage patterns and compliance needs." - Datadog's SMB Success Team

Data Type Considerations

Retention strategies vary depending on the type of data you’re handling:

For instance, a fintech startup saved $1,200 per month by retaining transaction logs for 90 days while limiting debug logs to 7 days.

These guidelines can help you fine-tune your retention rules and automate processes effectively.

Automation and Monitoring

Use Datadog’s tools to monitor and automate your retention rules:

• Track storage usage with datadog.estimated_usage.logs.ingested_bytes and review costs in the Billing dashboard.
• Verify rule changes in Audit Explorer using @evt.name.

One media company used Datadog’s API to automate retention rules, reducing storage costs by 22% by sunsetting campaign data after 30 days.

Keep in mind, retention changes only apply to new data.

3. Use Tags to Control Data Storage

Tags are a practical way to manage your data storage at a detailed level. By categorizing data with tags like department, project, or priority, you can apply specific retention policies that suit your needs.

Tag-Based Storage Management

A structured tagging system helps you manage data retention effectively. Here's how tags can be used:

Implementing Tag-Based Retention Rules

To set up and manage retention rules using tags, follow these steps:

1. Define Your Tagging Strategy

Create a clear and consistent naming convention for your tags. This ensures they remain organized and easy to use.

2. Assign Tags

Use the Datadog API or UI to apply tags to your resources. Proper tagging ensures resources are categorized correctly.

3. Set Retention Filters

Create filters based on tag queries to enforce retention policies that match your storage needs.

4. Monitor and Adjust

Use the Datadog Usage dashboard to track how well your tags are performing and adjust as necessary to improve retention management.

Best Practices for Tag Management

• Use lowercase letters and hyphens for consistency.
• Keep tags short but descriptive to avoid confusion.
• Maintain a central repository to document your tagging conventions.
• Conduct regular audits to check for outdated or incorrect tags.
• Remove tags that are no longer in use to keep the system clean.

4. Reduce Alert Noise

Improve your data retention efforts by managing alert notifications more effectively. Too many alerts can overwhelm your system, inflate data storage, and make it harder to focus on real issues.

Set Clear Alert Thresholds

Define specific, multi-level alert thresholds with time-based evaluations. This helps you identify genuine problems while avoiding unnecessary alerts. By narrowing your focus to actionable incidents, you can cut down on excess data and prevent alert fatigue.

Combine Similar Alerts

Group related alerts into a single notification to reduce redundancy. This keeps your monitoring efficient and ensures you're focusing on the overall health of your systems without collecting unnecessary data.

Adjust Alert Timing Dynamically

Tailor alert timing to your application's activity. Use tighter evaluation windows during peak usage and more relaxed ones during quieter periods. This approach reduces noise while maintaining effective monitoring and supporting efficient data retention.

5. Adjust Data Rollup Settings

Fine-tune your data rollup settings to maintain access to historical data while cutting down on storage costs. Aggregating metrics over time helps retain key insights without sacrificing system performance.

Configure Resolution Tiers

Set up your rollup settings based on the following tiers:

Use Smart Aggregation Techniques

Select aggregation methods that retain the most useful information:

• Average: Best for metrics that remain steady over time.
• Max: Helps capture spikes or unusual activity.
• Sum: Works well for metrics that accumulate, like totals.

Optimize Storage Allocation

Distribute your storage resources wisely. Allocate more space to recent, high-resolution data, and compress older data into lower-resolution formats. This approach minimizes storage needs while keeping the most critical data accessible. Regularly review these settings to ensure you're striking the right balance between cost and performance.

Monitor Rollup Effectiveness

Keep an eye on how your adjustments are working. Track storage usage, system performance, query speed, and data availability. This ongoing review will help you maintain efficiency and control costs over time.

Conclusion

The strategies outlined - like log filtering and adjusting data rollups - help you simplify your Datadog setup. By fine-tuning data retention, you can cut costs while keeping critical system insights intact. Using log filters, custom retention rules, tag-based storage, reducing alert noise, and tweaking data rollups keeps data flow and storage manageable without losing important details.

Make it a habit to revisit these settings as your business grows to strike the right balance between data access and resource efficiency. This focused approach lets you maintain strong performance without overspending. For more tips, check out Scaling with Datadog for SMBs to improve your Datadog setup.

FAQs

How can I filter logs in Datadog without losing important information?

To filter logs effectively in Datadog without missing critical details, start by identifying the specific data you need to retain for monitoring and troubleshooting. Focus on creating filters that exclude irrelevant or redundant logs while preserving essential information. Be cautious with overly aggressive filters, as they can unintentionally remove valuable insights.

Use Datadog's tag-based filtering to narrow down logs by attributes like service, environment, or status. Regularly review and adjust your filters to ensure they align with your current monitoring needs. Testing your filters in a staging environment can also help confirm that important logs are not being excluded before applying changes to production.

What are the best practices for creating a tiered data retention plan in Datadog to balance cost and accessibility?

To create an effective tiered data retention plan in Datadog, start by identifying the types of data your organization collects and classifying them based on their importance and usage. Critical data that requires frequent access should have a longer retention period, while less critical data can be stored for shorter durations to save costs.

Consider leveraging Datadog's retention settings to customize how long specific metrics, logs, or traces are stored. Regularly review and adjust these settings to align with your business needs and compliance requirements. This approach ensures you maintain accessibility to essential data while optimizing storage costs and system performance.

How can using tag-based storage help improve data retention and management in Datadog?

Tag-based storage in Datadog allows you to organize and filter your data more effectively by assigning meaningful tags to your metrics, logs, and traces. This approach helps you focus on the most relevant data for your business needs, improving data retention policies and reducing unnecessary storage costs.

By categorizing data with tags, you can set custom retention periods for specific data types or sources, ensuring critical information is retained longer while less critical data is archived or deleted sooner. This not only optimizes your storage usage but also enhances system performance and simplifies data analysis.

Related posts

• How to Optimize Datadog for Your Business
• Avoiding Common Datadog Mistakes for SMBs
• A Beginner's Guide to Using Datadog Effectively
• Datadog Anomaly Detection: Cost-Saving Tips