How To Optimize Datadog Log Retention Settings

Want to cut costs while managing your Datadog logs effectively? Here's how:

• Prioritize critical logs like errors and security breaches for longer retention.
• Use tiered storage options: Indexed Logs for fast access and Flex Storage for cost-effective long-term storage.
• Set retention periods based on compliance needs and operational priorities.
• Filter and sample logs to reduce volume - exclude low-value logs and convert routine data into metrics.
• Monitor costs with Datadog's Usage Metrics and set daily quotas to avoid unexpected expenses.

Quick Tip: Datadog's log retention costs range from $1.06 per GB (3 days) to $2.50 per GB (30 days). Use Flex Storage at $0.05 per million events for affordable long-term storage.

L05.1 - Cut Your Logging Costs with Datadog’s Logging Without Limits 🚀

Datadog Log Retention Basics

Understanding how retention settings and system components work together is essential for managing costs while maintaining the visibility you need.

Standard Retention Settings and Pricing

Datadog employs a tiered retention system that directly affects monitoring expenses. Retention periods typically range from 7 to 60 days, with costs varying depending on the duration you select.

The platform offers two main storage options: Indexed Logs and Flex Storage. Indexed Logs are designed for fast searches and in-depth analysis, making them ideal for critical errors. On the other hand, Flex Storage provides a more affordable option for long-term retention, suitable for routine logs.

Now, let’s break down the key components that power Datadog’s log management system.

Log Management Components

A solid understanding of these components helps you route logs effectively and manage retention costs:

• Log Collection and Processing
  Datadog supports log collection from over 800 built-in data sources. Once collected, logs are processed through pipelines that parse, enrich, and route them efficiently.
• Index Management
  Indexes act as searchable repositories for logs. You can create custom indexes tailored to specific needs, such as log severity, application components, environments, or team ownership.
• Archive Configuration
  Datadog’s Security and Cloud SIEM feature supports retention for over 15 months. This extended retention is useful for historical investigations, compliance requirements, and cost-effective long-term storage.

Additionally, the platform integrates with automated workflows, offering more than 300 actions aligned with MITRE ATT&CK framework standards. By configuring these components, you can filter and route logs based on their importance - keeping critical events indexed for quick access while storing routine logs more economically.

Setting Up Log Retention for SMBs

Selecting Critical Logs

To manage logs effectively, start by identifying the ones that are essential to your operations. Classify logs based on their importance and decide whether they need to be readily accessible or can be archived for later use. Here's a quick guide:

Research shows that 95% of businesses struggle with managing unstructured data effectively. By categorizing logs systematically, you can ensure critical data is always accessible while keeping costs under control.

Once you've classified your logs, configure log indexes to strike the right balance between quick access and cost efficiency.

Index-Based Retention Setup

With your prioritized log list in hand, you can fine-tune index configurations to match your retention goals. Datadog provides up to 100 indexes by default, giving you plenty of flexibility to manage your data.

• Organize indexes based on business value
  Datadog's analysis reveals that the top 10 indexes typically account for 80% of logs and 90% of costs. Focus on these key indexes to optimize storage and access.
• Set appropriate retention periods
  Align your retention periods with compliance rules and operational needs. If you're handling massive data - like over 10 billion events per month - Flex Logs can offer a cost-effective solution for long-term storage.
• Establish daily quotas
  To avoid unexpected expenses, set daily quotas and configure warning thresholds.

Creating Retention Filters

Strategic filtering is essential for getting the most out of your log retention policies. Here’s how to optimize:

Sampling Strategies

• Exclude over 90% of low-value logs to reduce volume without losing critical insights.
• Convert routine status logs into metrics for long-term trend monitoring.
• Use trace-based sampling for applications that require detailed log correlation.

Access Control
Limit who can modify index settings - restrict these changes to senior admins. Datadog's Audit Trail feature is a great way to keep track of any updates to retention policies or exclusion filters.

Additionally, edge log aggregation can help cut down log volume by removing duplicates, aggregating data, and applying sampling rules. This approach keeps your logs lean and focused on what truly matters.

Archive Management and Log Recovery

Archive Configuration

When planning log archives, it's crucial to strike a balance between cost and accessibility. AWS S3 buckets are a reliable solution for storing your log data over the long term.

Here’s how to set up archiving:

• Create a Dedicated S3 Bucket
  Set up an AWS S3 bucket specifically for your log archives. Take advantage of S3's storage tiers to manage costs effectively:
  • Use Standard for logs you access frequently.
  • Opt for Standard-Infrequent Access for less commonly accessed logs.
  • Choose Glacier for long-term storage needs.
    Be sure to review the S3 pricing for your region to plan your costs.
• Configure Archive Settings
  In your Datadog account, enable automatic archiving to compress and store logs in JSON format. This ensures efficient storage and retrieval.

Log Recovery Process

Once your archives are set up, you can recover logs using Datadog's rehydration feature. This tool allows you to bring archived logs back to the Log Explorer for detailed analysis.

Optimizing Historical Views
To make the most of historical log recovery:

• Define specific time ranges to reduce the amount of data being processed.
• Use targeted queries to focus on the logs you need.
• Set appropriate retention periods for the recovered logs.

Datadog documentation explains that you can create a historical view by selecting the relevant archive, specifying the time frame for the logs, and choosing how long the recovered logs will remain available.

Best Practices for Efficient Recovery
Organizing your archives based on usage patterns can save time and effort:

• Keep security-related logs in dedicated archives for quick access.
• Store compliance logs separately with longer retention periods.
• Create service-specific archives to streamline troubleshooting.

When investigating an issue, precise search parameters can be a game-changer. For example, to troubleshoot payment errors, you could scan a four-day archive for logs containing the phrase "payment rejected" from your web-store service.

Managing Rehydration Costs
To keep log recovery costs under control:

• Analyze smaller time windows to reduce the volume of data.
• Use detailed queries to rehydrate only what’s necessary.
• Schedule rehydration during off-peak hours to optimize resource usage.
• Regularly monitor usage metrics to track and manage costs effectively.

Managing Retention Settings

Keeping a handle on log retention settings is key to balancing costs while ensuring you retain the data you need. Without active management, log volumes can balloon by 200–300% each year, according to recent studies. Let’s dive into how usage metrics can help you stay on top of costs.

Cost Tracking with Usage Metrics

Datadog's Usage Metrics offers a clear breakdown of your log management expenses. Here’s how the pricing works: ingesting logs costs $0.10 per GB, and retention costs range from $1.06 per GB for 3 days to $2.50 per GB for 30 days.

To keep costs in check:

• Use the Usage Attribution Dashboard to pinpoint high-volume log sources.
• Track daily ingestion rates across your services.
• Regularly review retention periods for each index.

With these insights, you can set up quota alerts to ensure your budget stays intact.

Quota Alert Setup

Quota alerts are a proactive way to manage your log usage. Here’s how to set them up:

1. Configure Daily Quotas
   Assign quotas and warning thresholds to specific indexes.
2. Monitor at the Service Level
   Keep an eye on usage metrics by service and source to quickly spot trends.
3. Set Up Notifications
   Create alert systems for different levels of severity:
   • Critical alerts: Triggered when quotas are exceeded.
   • Warning alerts: Activated when thresholds are close.
   • Daily summaries: Provide a snapshot of usage trends.

Regular Policy Updates

Companies that frequently review their log management policies often uncover savings of 15–25% each quarter.

Monthly Review Checklist:

• Examine log trends for changes in volume or usage.
• Reassess retention periods for all indexes.
• Update exclusion filters and sampling rules.
• Evaluate archive usage for older logs.

Cost-Saving Strategies:

• Move older logs to cheaper storage options.
• Apply sampling to reduce high-volume, low-priority logs.
• Adjust exclusion filters to reflect current needs.

For example, Holland & Barrett managed to save over £60,000 (around $75,000) on their Datadog costs through consistent reviews.

Conclusion

The strategies outlined above provide SMBs with practical ways to manage Datadog log retention while balancing costs and data accessibility. By fine-tuning retention settings, businesses can maintain critical monitoring capabilities without overspending.

Using a tiered storage approach strikes a balance between cost-efficiency and performance. Real-world examples have shown that thoughtful log retention planning can lead to noticeable cost savings while ensuring operational needs are met.

For most organizations, a minimum one-year retention period is necessary to stay compliant with regulations. This can be achieved through:

• Classifying data and defining retention periods: Focus on what truly needs long-term storage.
• Applying intelligent filtering and optimizing storage: Store only the logs that add value.
• Automating retention processes: Reduce manual work while ensuring consistency.
• Regularly reviewing and updating policies: Adapt to changing needs and compliance requirements.

FAQs

How do I identify which logs to retain longer in Datadog?

When deciding which logs to keep longer in Datadog, prioritize those that are essential for monitoring performance, diagnosing issues, or maintaining security. Common examples include error logs, authentication logs, and logs that record major system events or unusual activities.

Datadog's Log Explorer can help you spot trends and identify which logs are most often used during investigations or monitoring tasks. By setting up retention filters, you can focus on preserving high-priority logs while discarding less important ones. This approach ensures you maintain the right balance between visibility and storage costs.

What’s the cost difference between using Indexed Logs and Flex Storage for log retention in Datadog?

When it comes to managing logs in Datadog, Indexed Logs can get pricey, as the cost is tied to the number of log events indexed. On average, it runs about $1.27 per million events. If you're looking for a more affordable solution, Flex Storage is a solid option for long-term log retention, priced at just $0.05 per million events per month. This makes it particularly appealing for small to mid-sized businesses aiming to keep costs under control while preserving logs for longer periods.

What sets Flex Storage apart is its approach to separating storage from compute. This setup lets businesses strike a balance between staying within budget and maintaining access to essential data, ensuring they don’t compromise on visibility while managing expenses.

How can I use Datadog's quota alerts to control log retention costs effectively?

To keep your log retention costs in check with Datadog, consider setting up quota alerts. These alerts can notify you when your log index usage hits about 80% of its daily limit. This proactive measure gives you enough time to act before exceeding your quota and racking up additional charges.

Another handy tool is retention filters, which let you automatically discard low-value logs. This ensures you’re only holding onto the data that’s crucial for your business operations. Make it a habit to regularly review your log usage and tweak your quotas or retention policies as needed. The goal? Retain the logs that offer actionable insights while cutting down on unnecessary storage costs.

Related posts

• How to Optimize Datadog for Your Business
• Avoiding Common Datadog Mistakes for SMBs
• Datadog Anomaly Detection: Cost-Saving Tips
• 5 Tips To Optimize Datadog Data Retention