The idea of paying money to store logs nobody is looking at may seem like a waste.
Well, that is until you need those logs.
At that point, you see how valuable log retention is, especially if there’s a security or compliance issue, or post-mortem review of a major outage. When you prioritize log retention, you can look back to investigate an incident or provide data for an audit — especially when you centralize log and metric data in one platform.
A log doesn’t need to be a lengthy, complicated text file. Logs are temporal data events and can come from anywhere—client update events, network flow data, web server logs, you name it. All digital actions create logs. However, even if all logs are collected, not all are stored.
Here is what you need to know about log retention, the importance of a log retention policy, best practices, and more.
Log retention refers to the archiving of event logs, particularly those related to security, concerning the duration for which you store these log entries. These entries typically refer to all cybersecurity, allowing companies to hold information on security-related activities. By keeping records, IT teams can gain critical insight into the activities happening on their network.
Log retention refers to the policies and practices involved in storing log data for a specified period of time and where they are to be archived. This period, known as the retention period, dictates how long log files and related data are kept before being deleted or archived. Log retention is crucial for various troubleshooting, operational, security, compliance, and analytical purposes.
For example, when did the error codes begin occurring and on what systems? What activities were performed on the system and when? What user account or software was involved in the activity, and what was the outcome?
What are log types?
Log retention is all about where and for how long organizations keep log files. These files are created by different systems, parts of the architecture, and apps across different environments. Each type of log serves a different purpose:
System logs: Generated by the operating system and are often used for troubleshooting any hardware or software issues.
Application logs: These are specific to software applications and are useful for debugging specific application issues or monitoring the activity of an application.
Security logs: Record all security-related events, including logins, access to certain resources, and administrative tasks. They are critical for incident response and compliance.
Audit logs: These track any changes and access to data which is essential for regulatory compliance.
What is a log retention period?
A log retention period is the amount of time you keep logs. For example, you may keep audit logs and firewall logs for two months. However, if your organization must follow strict laws and regulations, you may keep the most critical logs anywhere between six months and seven years. This timeframe is the log retention period.
Retaining logs for extended periods is optimal for security and compliance measures. However, this can become costly. That is why you need to create log retention policies that outline set log retention periods.
Think about a typical breach discovery timeline to justify a log retention period. The average timeline to discover an incident is 100 to 200 days depending on the source. In that sense, keeping logs for a year is sensible.
How long should you store log data?
The answer to this question is it depends.
While some companies store log data for a year or less, some organizations must comply with regulatory mandates. In this case, you may need to retain a security log for several years. Every organization operates in varying business environments. While one organization may retain logs for six months, another may keep logs for 18+ months.
The key is knowing the requirements your organizations need to comply with, based on the nature of your business.
Most organizations find that a minimum of one year meets most regulatory requirements. However, you must be mindful of your organization’s industry standards, regulations, and laws, as well as your company’s cybersecurity concerns.
Compliance: What regulatory reasons are there to keep log messages and for how long? GDPR, PCI, NISPOM, and corporate policies all come into play here. For example, should you keep specific logs containing personally identifiable information for shorter periods? Should some of the fields of these logs be encrypted?
Operational needs: Are there use cases where logs would be beneficial after the fact, such as concerning a third-party copyright claim or other potential legality issues?
Costs: Logs take up space, and space costs money. There comes a time when storing certain logs for longer outweighs the benefits they bring.
Managing costs and storage solutions
Effectively managing costs and storage for log retention is crucial for balancing compliance, security, and budget. Start with a thorough cost analysis, comparing on-premises and cloud storage options. Cloud storage, such as AWS or Azure, offers scalability and pay-as-you-go models, which can be more cost-effective as data needs grow.
Implement tiered storage solutions to optimize costs: store frequently accessed logs in high-performance (and higher-cost) storage, and archive less frequently accessed logs in cheaper, slower storage. Use data compression and deduplication to minimize storage needs and costs. Automated archiving solutions can move older logs to more cost-effective storage tiers as they age.
Monitoring tools like storage analytics and cost management tools from cloud providers can help track usage and costs, providing insights to optimize storage strategies. This approach ensures compliance and maintains data integrity while keeping storage costs manageable.
By carefully evaluating storage options and using cost-saving strategies, organizations can efficiently manage log retention expenses while meeting their security and compliance needs.
What is a log retention policy?
A log retention policy is a formalized set of guidelines that dictate how long log data is stored, managed, and eventually disposed of within an organization. This policy ensures that log data is retained for the appropriate duration to meet regulatory, operational, and security requirements while managing storage costs and maintaining data integrity. Retention policies can be internal to only your business or could be an industry mandated policy or combination of these two. Log retention policies can also be basic guidelines for a business to follow without strict enforcement or can be heavily enforced with specific audit requirements. Many companies do not have a log retention policy and not all logs are equal in value or business purpose.
When defining a retention policy, these are some of the considerations and decisions to be made to define the policy:
What type of log data will you store?
What is the classification and business purpose of each log type?
How long will you store each log type?
What will be the default retention if a new log type or undefined log type becomes available?
How frequently do we anticipate searching for this data?
Based on frequency of search and length of retention, where will this log data be archived and stored?
What is the cost for each of our storage/archive types?
What is the anticipated volume of each of our log types?
Are there any regulatory, audit or compliance rules or guidelines for my industry that apply?
Is data confidentiality a concern and which log types would it apply to?
Online vs. offline data, (frozen vs thawed vs hot)
Here is a simple example of a log retention policy from the University at Buffalo. In this policy log file retention times and log types are specified in the Retention Guidelines for Log Files. The default log retention time is 30 days unless the log file type falls under one of three categories, such as an IDM audit or UNIX auth/login. In these cases, the retention time is six months. Instructions are outlined in the policy concerning how log files are handled. Depending on your organization and infrastructure, your policy may be more complex.
You may also create several retention policies for each of your systems, which are then prioritized based on their importance. Anything concerning security should be part of an extended log retention policy, such as data related to user authentication or credit card authorization. Accounting and tax compliance software may also require longer retention policies since corporate auditors and government authorities can request retroactive analysis.
How often an application is used is also something to consider. If there is an application you do not often use, such as once a month, you’ll want to consider an extended retention policy just in case developers need to verify the source of an issue. A lower frequency of use would require developers to go back further to debug effectively.
Classifying business records
Within your log retention policy, you must classify logs based on varying categories and also weigh this against the volume of data and method in which the data will be ingested and stored. These include the following.
‘Must keep’ logs: These are the files and events you are legally required to store. This category should also prioritize logs that may relate to potential legal issues. Store these logs based on the length of time required by regulations and laws relevant to your industry or until they are no longer relevant for potential legal disputes.
‘Want to keep’ logs: These logs will hold value to your business but are not required to be stored by law. Your policy should address an appropriate log retention period for these records. You should also state how often these logs are reviewed to determine whether they still hold any importance.
‘Destroy’ logs: Just as you would shred any hard copy documents that are no longer needed, it’s important to delete logs that are either obsolete or logs that are not being used by your company yet act as a potential security threat.
Although the most important logs for your organization will vary depending on the scope of your business and industry as a whole, most organizations prioritize the following log types.
User IDs and credentials
Data and time of access to critical assets (e.g., activation and deactivation of firewalls, important security events, etc.)
Failed and successful login attempts
Changes made to system configurations
Details of incidents
Log retention best practices
The key to better log retention is to follow best practices and implement proactive strategies from day one. When it comes to your company’s log management and retention, you always want to remain proactive, not reactive. If something happens and you do not have the systems in place, your efforts moving forward may be too little, too late, or too costly.
If you log too many events, the most critical logs can become challenging to find. Log the most critical events so that essential information isn’t lost or overlooked. These events include account lockouts, login failures, and file access. Again, log management can be challenging when organizing a large IT environment, which is even more reason to follow best practices. Successfully maintaining critical logs is not just a sound security posture—taking the following steps will improve peace of mind and ongoing business operations if something happens. You need to protect your overall IT infrastructure while gaining valuable insight and having access to the right logs at the right time.
Archive data centrally when possible
If something happens, your security logs act as the evidence you need to investigate. Without these logs, you cannot conduct an analysis. It is recommended that you archive your logs centrally to ensure their integrity and compliance. Based on the business requirements, log type, and anticipated search needs you may not always be able to affordably store all log types into a central repository but proper storage cost analysis can help these decisions.
When archiving these logs, you’ll want to ensure they are in a secure location, preferably encrypted and with access controls and audit logging in place. Other techniques, such as hashing, are also optimal to enhance security.
Alert on log data in real time before archiving
You should also monitor your logs, implementing a tool to identify issues and send real time alerts based on known issues, error codes, threats, indicators of compromise, anomaly detection, or changes in volume
Depending on your chosen archival methods, log types and ingest methods, you should closely watch the amounts of data and use it to predict normal growth patterns and expected volume. That way, you can scale efficiently by adding more bandwidth to the network or any licensing needs to avoid data overages. The last thing you want is to lose data because of insufficient bandwidth, storage space or lack of licensing, especially when you think you’re remaining compliant.
Create a log retention policy
Security logs should be retained for longer periods compared to application log data. If you experience a breach or attack, your logs are your evidence. However, it’s not beneficial to hold logs forever, which is where a log retention policy comes into play, as discussed earlier. Creating a log retention policy sets critical guidelines for your team to follow.
When creating your policy, focus on both log security and log retention. For example, are you storing clients’ data or internal access keys for API? These examples are the type of data you want to encrypt.
Log retention guidelines are critical, but so is log monitoring. You can achieve this by leveraging a tool or platform that simplifies this process. Do not underestimate the value of log analysis. Tools that offer AI and machine learning abilities sift through logs to determine a baseline and detect anomalies.
The best way to ensure the logging process isn’t overly complicated is to invest in a platform that makes this step seamless. Also, the more humans you involve in the process, the more likely errors will result. A unified platform will allow you to take advantage of greater automation. When you can easily access all the information you need to investigate an issue across your technology ecosystem, this will significantly speed up the process. Learn more about LM Logs here.
Wrapping up
Log retention is a vital component of any organization’s operations. Consistent monitoring and analysis of logs are essential for enhancing operational efficiency and ensuring data security. Investing in a unified logging platform can streamline operations and minimize human errors. With all necessary information readily accessible, investigating issues and anomalies across an organization’s technology ecosystem becomes quicker and more efficient.
Ultimately, implementing effective log retention practices is crucial for organizations aiming to enhance their operational processes and maintain secure data management. Contact LogicMonitor today to learn more about how we simplify log analysis and improve operations.utomation. When you can easily access all the information you need to investigate an issue across your technology ecosystem, this will significantly speed up the process.
By Patrick Sites | AKA "The Logfather"
Product Architect of Logs, LogicMonitor
Subject matter expert in the Log Monitoring space with 25+ years experience spanning Product Management, Presales Sales Engineering and Post-Sales PS/Support Roles.
Disclaimer: The views expressed on this blog are those of the author and do not necessarily reflect the views of LogicMonitor or its affiliates.
Blogs
Check out our latest resources
See only what you need, right when you need it. Immediate actionable alerts with our dynamic topology and out-of-the-box AIOps capabilities.
