5 Steps to Enforce Data Retention in Datadog

Managing data retention in Datadog is crucial for compliance, cost control, and efficient data management. Datadog’s default retention settings often don’t align with industry regulations like HIPAA or PCI-DSS, making custom policies essential. This guide outlines five steps to help you enforce data retention in Datadog effectively:

1. Review Default Settings: Understand Datadog’s retention periods for metrics (15 months), logs (15 days), RUM data (30 days), and APM traces (15 months). These defaults may fall short of regulatory needs.
2. Match Data to Compliance Needs: Align data types (metrics, logs, traces) with specific regulations such as HIPAA (6 years for logs) or IRS guidelines (7 years for financial records).
3. Set Custom Retention Policies: Use tiered storage (hot, warm, cold) and filters to balance accessibility and costs, while meeting compliance requirements.
4. Automate Management: Leverage Datadog workflows to handle archiving, tagging, and filtering, paired with alerts for storage thresholds and workflow errors.
5. Review and Update Regularly: Conduct audits to ensure policies remain aligned with changing business needs and regulations.

What is a Data Retention Policy?

Step 1: Review Datadog's Default Data Retention Settings

Before diving into custom configurations, it’s crucial to understand Datadog’s default retention settings for each data type. These defaults vary and may not always align with your compliance needs or business goals.

Here’s a quick overview of the retention periods:

• Metrics: Stored for 15 months, offering over a year of historical performance data.
• Logs: Retained for just 15 days, which may be too short for compliance requirements like HIPAA or PCI-DSS.
• RUM (Real User Monitoring) Data: Kept for 30 days.
• APM Traces: Retained for 15 months, similar to metrics.

Compliance Considerations

For industries with strict compliance standards, such as healthcare or finance, the default settings for logs may fall short. For example:

• HIPAA: Requires audit logs to be retained for six years.
• PCI-DSS: Mandates at least one year of log retention for most data types.

To check your log retention settings, head to Organization Settings > Preferences > "Out of Contract Retention Period for Log Indexes" in your Datadog account. This section allows you to review and adjust your log retention configuration to ensure it aligns with your specific needs.

Cost Implications of Log Retention

Log retention costs can add up quickly. For example, Datadog charges $0.10 per indexed log per GB and $1.27 per million ingested events. If you’re ingesting 100 GB of logs daily, the 15-day retention period would cost about $150 per month. If your compliance requirements demand longer retention, you’ll need to factor in these additional expenses.

It’s also worth noting that once logs expire in Datadog, they’re permanently deleted unless you’ve configured log forwarding to external storage (like Amazon S3). Without this setup, you risk compliance gaps, so planning ahead is critical.

Optimizing Retention and Costs

Once you’ve reviewed the defaults, the next step is to optimize your settings. For metrics, the 15-month retention period typically works well for most small to medium-sized businesses, providing enough historical data for trend analysis and capacity planning. However, you can reduce costs by shortening retention for non-essential metrics, like temporary debugging data.

Logs, on the other hand, often require longer retention. If you need to keep logs beyond 15 days, configure log forwarding to an external archive solution like Amazon S3. This ensures you can store the data for compliance purposes while keeping Datadog storage costs in check. Archived logs can be rehydrated into Datadog for analysis later, though this process may take time and incur additional charges.

To manage costs effectively, use tagging to apply specific retention policies to different data types. For example, critical system health metrics might warrant the full 15-month retention, while less critical metrics or logs can be set to expire sooner. Tailoring retention policies to your needs is a key step in balancing compliance and cost-efficiency.

Step 2: Match Data Types to Your Compliance Requirements

Once you've reviewed the default settings, the next step is to align each data type with the compliance standards that apply to your organization. This process ensures your data retention policies meet both regulatory obligations and business goals. By doing this, you'll identify which data types in Datadog need custom retention policies.

Identify Your Key Data Types

Datadog primarily handles three main types of data: metrics, logs, and traces.

• Metrics measure system performance and help analyze trends. They provide numerical insights like CPU usage, memory consumption, request rates, and error rates, offering a high-level view of system health to guide operational decisions.
• Logs capture system events, errors, and transactions. Since logs often contain sensitive or regulated information, they may need longer retention periods. Datadog's sensitive data scanning tools can also assist with meeting requirements like PCI compliance.
• Traces monitor request flows across systems, helping pinpoint bottlenecks and latency issues. While traces are generally less sensitive than logs, retention settings should still reflect your compliance and operational needs.

Align Data With Regulatory and Operational Standards

After identifying your critical data types, the next step is to align them with the relevant regulations and business requirements. Key standards to consider include HIPAA, PCI-DSS, GDPR, and IRS guidelines, each specifying retention periods for different types of data.

For example:

• In the healthcare and financial sectors, regulations often require extended log retention. HIPAA mandates six years for audit logs, while PCI compliance governs financial service logs.
• IRS guidelines specify a seven-year retention period for financial records, which applies to transaction logs.

In December 2024, Datacate recommended specific retention timelines for small retail businesses, such as three years for customer history and seven years for invoices.

To streamline this process, consider grouping your data into categories based on retention needs, such as short-term, medium-term, and long-term. A documentation table can also help map Datadog data types to their respective retention periods, making it easier to configure custom policies later.

Step 3: Set Up Custom Data Retention Policies in Datadog

Once you've mapped out your compliance requirements, it's time to configure custom retention policies that align with both regulatory needs and cost targets. By setting up specific retention tiers and filters, you can fine-tune your approach to data storage.

Configure Custom Retention for Logs, Traces, and Metrics

Custom retention policies rely on a tiered system to strike the right balance between accessibility and cost. Typically, retention is divided into three tiers:

• Hot storage (0–30 days): Immediate access for recent data.
• Warm storage (31–180 days): Accessible but less frequently needed data.
• Cold storage (181+ days): Archival data for long-term storage.

For example, one organization reduced its storage costs by 35% while meeting HIPAA requirements by archiving patient logs after 90 days and retaining transaction logs for the same period. Debug logs, on the other hand, were limited to just seven days, saving significant resources.

Another crucial factor is data resolution. By keeping high-detail data for the first week and transitioning to summarized views for older data, you can maintain analysis capabilities while optimizing storage.

Once these tiers are defined, refine your strategy with targeted filters and tags to further optimize retention.

Apply Filters and Attributes

Filters and tagging are essential tools for customizing retention policies. Tags help categorize data, allowing for tailored retention policies by department, project, or priority. Meanwhile, filter queries can exclude unnecessary logs, significantly cutting down storage costs.

Automated filters can be particularly effective. For instance:

• Exclude debug logs completely after three days.
• Apply sampling to health-check logs after seven days.
• Retain business-critical logs for the full 15-month default period without filtering.

Tag-based organization adds another layer of control. Use tags such as dept:finance or dept:hr to apply retention policies specific to different departments. Priority tags like priority:high or priority:low can help you focus on the importance of data, while project tags like project:alpha or project:beta allow for project-specific retention strategies.

A Fortune 500 retailer demonstrated the power of this approach by implementing 12 exclusion rules across their 23 million daily logs. By identifying patterns in non-essential data and creating targeted filters, they reduced storage costs by 58% while preserving critical information.

Here’s an example of how filtering can look in practice:

The secret to effective filtering is understanding your data patterns and business needs. Start by excluding obviously unnecessary data, then refine your filters based on real-world usage and compliance demands. This ensures you're only paying to store data that serves a meaningful purpose.

Step 4: Automate Retention Policy Management

Once you've established your custom retention policies, automation can take the heavy lifting off your plate as your data continues to grow. Managing retention policies manually gets harder as data volumes increase, but automation ensures policies are consistently enforced while reducing the workload on your team. Tools like Datadog's automation features can help you stay compliant and save time. Let’s look at how automating workflows can make this process even smoother.

Use Datadog Workflows to Simplify Automation

Datadog Workflows are a game-changer for automating tasks that would otherwise demand constant manual effort. These workflows can handle tasks like archiving logs older than 30 days to cost-effective storage, filtering out unnecessary logs (like debug or health check logs) before they’re indexed, and tagging essential logs for longer retention periods while applying shorter retention times to routine data.

Start by identifying the retention tasks that eat up the most time. For instance, automated log filtering can eliminate nonessential data, helping you cut costs. Similarly, custom retention workflows can create tiered storage plans, which often lead to cost savings of 35% or more - all while staying compliant with your policies.

Tag-based workflows add another layer of efficiency. By automatically applying tags based on predefined criteria, you can better control your storage. For example, logs from your payment processing system can be tagged with identifiers like priority:high and retention:extended, ensuring they’re treated according to their importance.

Begin with simple filtering rules to remove obviously unnecessary data. As you become more familiar with the system, you can implement advanced tagging and retention strategies to optimize your processes further.

Set Up Alerts to Track Retention Policies

Automation works best when paired with proactive alerts to keep you on track. Well-designed alerts help you avoid policy violations and stay ahead of potential issues without overwhelming your team with unnecessary notifications. Key alerts to consider include warnings for nearing retention limits, errors in workflows, and unexpected spikes in log volume or storage use.

Here are a few examples of useful alerts:

• Threshold-based alerts: These notify you when storage usage approaches a predefined limit, giving you time to adjust policies and avoid unnecessary costs. Set these thresholds based on your monthly storage capacity to maintain a buffer for unexpected data surges.
• Workflow failure alerts: These alerts flag issues like archiving errors, allowing you to address problems quickly and ensure compliance.
• Volume spike alerts: These help you detect unusual patterns in data, such as a sudden increase in daily log volume, which could indicate system issues or a need to tweak your retention policies.

To avoid notification fatigue, group related alerts and assign appropriate severity levels. This way, critical issues get immediate attention while routine alerts remain manageable.

Start by setting up essential alerts, like those for storage thresholds and workflow failures. Over time, as your automation becomes more advanced, you can expand your monitoring to include more detailed and sophisticated alerts. This gradual approach ensures you catch important issues without getting bogged down by unnecessary notifications.

Step 5: Review and Update Retention Policies

Keeping retention policies up to date is essential as your business grows, regulations shift, and data volumes change. Regular reviews ensure your policies stay aligned with compliance requirements, help manage storage costs effectively, and prevent unnecessary data retention or the loss of critical information. Routine audits are a key part of this process, offering insights into necessary adjustments as your needs evolve.

Run Regular Policy Audits

Make it a habit to conduct regular reviews - quarterly audits are a good benchmark - to identify and address potential issues early. During these audits, assess data volumes, storage expenses, and compliance requirements. For instance, if storage costs suddenly spike, investigate the cause. You might discover that debug logs, which typically should be deleted after three days, are being retained longer than necessary. Similarly, business-critical data with a 15-month retention period might need updating to meet new compliance standards.

Take a closer look at your filtering rules to confirm they’re still effective. Properly configured filtering can lead to substantial savings, so periodic reviews are worth the effort. Additionally, revisit your tagging strategy. Ensure tags like dept:finance or priority:high accurately reflect your current organizational structure and priorities, which is vital for maintaining effective data retention practices.

Adjust Policies as Your Business Grows

Insights from these audits should guide you in refining your retention policies to match your company’s growth. As your business expands, your retention strategy needs to scale too. For example, a tiered storage approach can be adjusted to handle increasing data volumes:

Another area to consider is data resolution. High-resolution data, such as 10-second intervals, is valuable for the first week when detailed analysis is critical. After that, you can transition to medium resolution (e.g., 5-minute intervals) for days 8–30 and lower resolution (e.g., 1-hour intervals) for days 31–90. This approach optimizes storage without sacrificing important insights.

Before rolling out any changes, test them on a smaller scale to minimize risks. This careful testing phase ensures you don’t accidentally lose essential data while fine-tuning your retention strategy.

Conclusion

By following these five steps, you can create a scalable data retention strategy for your SMB within Datadog. Reviewing defaults, aligning data with compliance requirements, customizing policies, automating workflows, and conducting regular audits establish a solid foundation that grows with your business. This approach not only helps meet compliance standards but also boosts operational efficiency.

Streamlined retention practices cut costs and make operations smoother. Teams can quickly find the data they need without wading through irrelevant information, while automation frees up time for strategic tasks. Using tiered retention policies to optimize storage costs offers immediate savings compared to keeping all data uniformly.

Filtering out unnecessary logs is another way to save costs right away, while tiered retention ensures critical data is always accessible when needed. Custom indexes and strategic tagging, as discussed earlier, provide precise control over what data is stored and for how long, tailored to your business needs and compliance goals.

Automated workflows and targeted alerts simplify compliance management, reducing manual effort and helping you avoid last-minute issues. This proactive approach keeps your retention strategy ahead of potential challenges.

As your SMB grows, the retention framework you’ve built evolves alongside it. Regular reviews ensure your policies stay aligned with increasing data volumes, changing regulations, and shifting business priorities. This keeps costs under control and prevents compliance gaps.

With this adaptable framework, your data retention strategy remains a dynamic tool that supports both current compliance needs and future business growth.

FAQs

How can I make sure my data retention policies in Datadog meet industry standards like HIPAA or PCI-DSS?

To align your data retention policies in Datadog with regulations like HIPAA or PCI-DSS, start by adjusting retention settings to meet the specific standards. For instance, PCI-DSS typically mandates keeping audit logs for a minimum of one year. Beyond that, enforce access controls to limit who can view sensitive data and apply data redaction policies to protect confidential information.

It's also important to regularly audit your configurations and keep an eye on stored logs. Doing so not only helps you stay compliant but also reduces potential risks to your organization.

How can I manage data retention in Datadog to control costs while staying compliant?

To manage data retention effectively in Datadog, begin by setting custom retention periods tailored to the importance and regulatory requirements of your data. Implement retention filters to minimize the storage of unnecessary information, and create multiple log indexes to address varied retention needs. This approach helps maintain compliance without overwhelming your storage capacity.

You can also optimize data collection by turning off ingestion for irrelevant logs or traces and using tags to monitor and organize your data. These methods not only help control costs but also ensure your organization stays compliant while maintaining efficient storage practices.

How can I simplify data retention management in Datadog to save time and stay compliant?

To make managing data retention in Datadog easier and less time-consuming, you can rely on customizable retention policies and dynamic retention filters. These features let you automatically control how long various types of data are stored, based on their importance. This not only helps you stay compliant but also keeps storage costs in check.

Dynamic retention filters are particularly helpful for automating the process of removing unnecessary data. By applying predefined rules, you can ensure your data management remains efficient and compliant without needing constant attention. This allows SMBs to concentrate on growing their business while Datadog takes care of the complexities of data retention.

Related posts

• Datadog Anomaly Detection: Cost-Saving Tips
• 5 Tips To Optimize Datadog Data Retention
• How to Archive Logs in Datadog
• How To Optimize Datadog Log Retention Settings