How to Archive Logs in Datadog

Want to archive logs in Datadog efficiently? Here's how:

1. Set Up Storage: Use an AWS S3 bucket for storing archived logs. Choose storage tiers like Standard, Glacier, or Deep Archive to balance cost and retrieval time.
2. Secure Your Data: Encrypt archives with AWS KMS and control access using strict IAM permissions.
3. Optimize Costs: Enable compression (e.g., GZIP) and set lifecycle policies to move older logs to cheaper storage tiers.
4. Archive Configuration: In Datadog, define rules for filtering logs, retention periods, and storage settings.
5. Retrieve Logs: Use Datadog’s Log Rehydration™ to restore specific logs for analysis when needed.

Key Tip: Regularly monitor storage usage, test recovery processes, and update access controls for compliance and cost efficiency.

For detailed steps, configurations, and best practices, keep reading!

Keynote: Log Rehydration and Logs to Metrics

Getting Started with Log Archives

Setting up log archiving in Datadog involves a few key steps to ensure your logs are securely stored and accessible when needed.

Key Components You’ll Need

Before diving in, make sure you’ve got the following covered:

Datadog Account Permissions
Your account must include the logs_write_archive permission. This ensures only the right people can configure and manage log archives.

Cloud Storage Setup
You’ll need an AWS S3 bucket as the designated storage location for your archived logs.

Access Permissions
Set up the necessary IAM roles and permissions to enable secure communication between Datadog and your AWS S3 bucket.

Once these are in place, you can connect Datadog to your AWS S3 storage to establish a secure data flow.

Connecting Datadog to AWS S3

The integration between Datadog and AWS S3 is the foundation of your log archiving system. For detailed guidance, check out the Datadog Log Archives documentation.

Here’s a quick overview of the connection process:

• Access Configuration: Log in to the Datadog dashboard to begin setup.
• Choose Storage: Select AWS S3 as your archive storage destination.
• Verify Permissions: Double-check that the necessary IAM roles are in place.
• Test the Connection: Run a validation test to confirm the integration works as expected.

Setting Up Your First Archive

Once your storage and permissions are ready, it’s time to configure your first archive in Datadog. This process ensures your logs are retained according to your organization’s requirements.

Follow these steps to set it up:

• Navigate to Archive Settings
  In the Datadog dashboard, go to Logs > Configuration, then open the Archives tab. Click New Archive to start the setup.
• Define Archive Rules
  Customize your archive by specifying:
  • Filtering criteria for the logs you want to archive
  • Retention periods to determine how long logs are stored
  • The storage class for your AWS S3 bucket
  • Compression settings to optimize storage space

Working with Archived Logs

Once you've archived your logs in AWS S3, you can leverage Datadog's Log Rehydration™ to retrieve them whenever needed while keeping storage costs in check. Whether you need to restore logs for deeper analysis or search through them directly, Datadog provides tools to make the process efficient.

Restore Archived Logs

With Log Rehydration™, you can bring back archived logs into your Datadog account for further analysis. Here's how:

• Navigate to Rehydration: Go to the Logs Configuration page and select the Rehydrate from Archives tab. Then click on New Historical View to start the process.
• Set Up Your Historical View: Define the parameters for the logs you want to rehydrate, including:
  • The source archive
  • The specific time range
  • Relevant search terms or attributes
  • The retention period for the rehydrated logs
• Be Precise: To optimize the process, narrow down your search using specific terms, time ranges, and attributes. This ensures you retrieve only the logs you truly need.

"With Datadog's Log Rehydration™ - part of our Logging without Limits™ feature set - you can efficiently archive all of your logs and retrieve the exact ones you need, when you need them."

Find Specific Logs

If you don't need to restore logs but just want to locate specific entries, Datadog allows you to search archived logs directly. This can save time and resources when you're looking for particular details.

Use Targeted Queries

To pinpoint the logs you need, you can create search queries using:

• Service names
• Error codes
• Custom attributes
• Exact phrases enclosed in double quotes

For example, if you're troubleshooting payment issues, you might search your "Prod Archive" for error logs from the web-store service that include the phrase "payment rejected" within a specific four-day window.

Tips for Better Searches

• Focus on smaller time ranges to narrow down results.
• Combine multiple attributes to refine your query.
• Use exact match phrases for precision.
• Consider creating separate archives for logs you access frequently.

These tools and techniques make it easier to manage and retrieve archived logs, ensuring you can access the data you need when you need it.

Reduce Storage Expenses

Cut down on storage costs while keeping essential log access intact. The key lies in fine-tuning your storage settings to strike the right balance between performance and affordability.

Choose S3 Storage Types

Selecting the right S3 storage tier based on how often logs are accessed can lead to significant savings. For instance, S3 Intelligent-Tiering automatically adjusts log storage based on usage patterns, offering potential savings of 40%, 68%, or even 95%.

• Start with S3 Standard: Use this tier for logs that are accessed frequently. It provides 99.99% availability and millisecond-level access times, making it ideal for recent logs.
• Set Lifecycle Policies: Transition logs to lower-cost tiers over time. For example:
  • Logs older than 30 days: Move to Standard-Infrequent Access.
  • Logs over 90 days: Shift to Glacier Instant Retrieval.
  • Logs exceeding 180 days: Archive in Glacier Deep Archive.

To illustrate, S3 Glacier Flexible Retrieval offers cost-effective options for accessing archived logs, with retrieval times ranging from minutes to hours and free bulk retrievals. Once your logs are sorted into the right tiers, compress them to save even more.

Save Space with Compression

Compression is a simple yet powerful way to reduce storage needs and cut costs. By configuring log forwarders and organizing archives efficiently, you can maintain easy access to your data without breaking the bank.

• Enable GZIP Compression: When setting up log forwarders like Fluentd, Fluent Bit, or the Stackdriver Logging Agent, enable GZIP compression. This reduces both network bandwidth and storage requirements.
• Streamline Archive Organization:
  • Group similar logs and consolidate smaller files to improve compression efficiency.
  • Be mindful of file sizes: S3 Glacier storage classes add approximately 40 KB of metadata per archived object.
  • For high-volume logs, apply sampling techniques to retain meaningful data while reducing overall storage.

Security and Compliance Steps

Protect your archived logs while ensuring you meet compliance requirements.

Set Up Data Encryption

Keeping your archived logs secure is essential for maintaining data integrity and adhering to compliance standards. Start by encrypting your log archives using AWS Key Management Service (KMS). You can choose from these server-side encryption options:

• AWS-managed Keys (SSE-S3)
  This is the default option, fully managed by AWS. It offers basic encryption but gives you limited control over key management.
• KMS-managed Keys (SSE-KMS)
  A great choice for most organizations, this option provides automatic key rotation (about every 365 days), detailed audit trails, and fine-grained IAM access controls.
• Customer-managed Keys (CMK)
  Ideal for organizations needing full control. You can set custom key rotation schedules, manage the key lifecycle, and retain complete ownership of your keys.

Once encryption is set up, strengthen your security framework by enforcing strict access controls and tracking usage.

Control Access and Track Usage

Limiting access and keeping a close eye on usage are critical for compliance.

Key IAM Permissions:

To monitor access, use tools like:

• S3 server access logging to maintain a record of requests.
• CloudTrail to track API calls and user activity.
• Automated alerts to flag unusual patterns.
• Approval workflows for restoring logs securely.

Integrate these monitoring tools with your compliance systems to generate reports that align with regulatory requirements.

Protect Sensitive Data:

• Use Datadog's data scrubbing features to redact sensitive information and create custom rules for organization-specific patterns.
• Regularly scan archived logs to confirm proper handling of sensitive data.
• Keep detailed documentation of your procedures to demonstrate compliance.

For disaster recovery, enable cross-region replication and turn on versioning for critical log buckets. Make sure to test your recovery procedures every quarter to ensure they work as expected.

Summary

Setting up efficient and secure log archiving in Datadog requires thoughtful planning and attention to detail. Here's a quick guide to ensure success:

Storage Setup and Configuration

• Make sure your dedicated AWS S3 bucket is up and running.
• Set IAM permissions to strictly control who can access your archives.
• Enable compression to cut down on storage costs.
• Choose the right storage classes for long-term data retention.

Security and Compliance

• Use AWS KMS encryption to protect your data.
• Enforce strict IAM access controls to limit unauthorized access.
• Keep audit logging active to track any changes or access attempts.
• Regularly test archive recovery to ensure your data is always accessible when needed.

This approach aligns with Datadog's focus on secure and efficient monitoring practices.

Best Practices for Maintenance

• Take advantage of Datadog's rehydration feature to analyze older data when needed.
• Use targeted queries to quickly pinpoint specific logs.
• Organize your data with tags and metadata for easier searching.
• Regularly monitor storage costs and usage patterns to stay on top of expenses.

For more detailed guidance, check out the earlier sections covering configuration and maintenance tips.

FAQs

How can I securely archive logs in Datadog while staying compliant with industry standards?

To keep your logs secure and compliant with industry regulations, start by enabling log archiving in your Datadog account settings. With Datadog, you can set up archiving to trusted cloud storage platforms like Amazon S3 or Google Cloud Storage, providing a safe and scalable solution for your log data.

When configuring storage, make sure it aligns with regulations such as GDPR, HIPAA, or SOC 2. Protect sensitive data by using encryption for both data in transit and at rest. Also, implement strict access controls to limit who can view or modify your archived logs.

By taking these precautions, you can safeguard your logs while using Datadog's advanced log management tools to support your business requirements.

What are the best ways to manage storage costs when archiving logs in Datadog?

To keep storage costs under control while archiving logs in Datadog, here are some practical tips to consider:

• Define log retention policies: Assign different retention periods based on the importance of each log type. For instance, you might keep critical logs for a longer duration while archiving less essential ones sooner.
• Exclude irrelevant logs: Use exclusion filters to prevent storing logs that don’t serve your monitoring or compliance needs. This helps cut down on the volume of data being archived.
• Leverage cost-efficient storage: Datadog supports archiving to external storage solutions like Amazon S3 or Google Cloud Storage. Pick an option that fits your budget and scales with your needs.

By applying these approaches, you can streamline your log archiving process, save on storage costs, and still ensure access to the data that matters most.

What is Datadog's Log Rehydration™ feature, and how can I use it effectively?

Datadog's Log Rehydration™ makes it simple to access archived logs on-demand, giving you the flexibility to analyze historical data whenever necessary. This feature is especially helpful for tasks like troubleshooting issues, conducting compliance audits, or extracting insights from past events - all without the need to store every log in active storage.

Here’s how it works: First, configure your log archiving to save logs in a supported storage solution, like Amazon S3. When specific logs are required, you can "rehydrate" them by querying the archived data and temporarily bringing it back into Datadog for analysis. This method strikes a balance between cost efficiency and maintaining access to critical historical information.

Related posts

• How to Optimize Datadog for Your Business
• How to Set Up Log Management in Datadog
• 5 Tips To Optimize Datadog Data Retention