Data is not the new oil, contrary to the popular assertion. Unlike oil, data is not finite, exclusive, depletable or limited to specific use. "Good data" can be synthesised, repurposed and reused. Its favourable attributes make it a major asset of great value which, however, comes with increased regulatory attention, storage and governance costs and legal risk.
As Australia's Privacy Commissioner announces the updated guidance about the retention, destruction and de-identification of personal information for not-for-profits, even resource-constrained organisations are reminded that "Indefinite retention of all personal information is unlikely to satisfy an entity's APP obligations". Moreover, it can result in exacerbated "risk or impact of data breaches" and adverse regulatory action.
Data governance including a data retention policy is key. With data sharing for innovation being promoted at international and governmental levels, good data governance will help organisations unlock the value of data and explore opportunities across the data-driven economy.
Apart from significantly reducing the cost of storage and data management and limiting exposure to legal risk, there are environmental benefits too - less data helps reduce emissions and landfill (as server hard drives are often physically destroyed upon deletion for data security reasons).
Getting data retention right will eventually have a direct positive impact on the organisation's balance sheet.
Data governance is key to forming a data retention policy
Data governance is a set of rules about all activities around the data lifecycle including the collection, storage, use and disposal of enterprise data. Data retention is a key aspect of it, and its concerns cut across the organisation, touching upon operational needs, information security, legal compliance and data privacy.
Data governance is often pushed aside by spending priorities, such as revenue generating activities. Meanwhile, simply keeping all data is easier than having to manage it, even if it means seeing the cost of data redundancy rise and ignoring the intrinsic value of data.
Understanding the value of enterprise data is important. Apart from its critical use for various business purposes, emerging analytics and AI data marketplaces may offer compensation for datasets suitable for analytics and the development of AI.
From a simplified point of view, some necessary data governance steps in designing a data retention policy will relate to:
If data is collected for targeted lawful purposes, properly classified, we know where to find it, it is reasonably accurate to achieve our purpose, reasonably utilised, repurposed and reformatted for secondary purposes in a timely manner, tracked throughout its lifecycle and erased across all assets and devices when no longer needed, the organisation is succeeding in formulating and executing a data retention policy. This may translate into the recognition of new capital assets, identifying new savings and financial gains.
Setting a data retention period is hard
Data retention strategies are essential in maximising value and reducing the risk of data. They strive on the "less is more" principle. The benefits of good data retention lie in purging useless, unnecessary or otherwise "bad data", and typically there is plenty of it around.
Often organisations view the risk of not having data as greater than the risk of having data. However, the law does not reward data hoarding. Often storage of data gives rise to risk such as data breaches, unauthorised access to data in breach of policies, interference with the privacy of staff or customers, increased storage costs and the natural staff turnaround results in loss of organisational familiarity about stored files further diminishing their value.
Caution often drives organisations to set long retention periods inspired by the prescribed statutory limitation periods, e.g. 7 years from the cause of action arising, during which one can bring a legal claim. However, the age of a document may have little bearing on its evidential value. Besides, a document can be as helpful as it can be damaging in proving the organisation's case. Adopting such a blanket rule for all documents is difficult to reconcile with any logic or legal requirement. Nevertheless, it is a popular starting point; albeit not one to be recommended
Data retention strategies must often find a common ground between varying corporate interests, such as:
In practice, setting retention periods is a complex task. Expect plenty of internal battles before being able to formulate a data retention policy. Starting with achievable short-term goals and longer time periods which are gradually tightened and granularised might work best, as long as this is based on sound justifications.
Various other laws prescribe rules about data retention. The Freedom of Information Act 1982 (Cth) prescribes certain publication of government data by federal agencies and ministers; the Corporations Act 2001 (Cth) requires organisations to keep accounting records, registers and meeting minutes; Fair Work Act 2009 (Cth) requires organisations to keep certain employee records; the Workplace Surveillance Act 2005 (NSW) requires organisations to take reasonable safeguards including data erasure to protect surveillance records; the Telecommunications (Interception and Access) Act 1979 establishes a data retention scheme for telecommunications service providers, etc.
Data erasure is a technological process
Being one of the hardest decisions in data governance for which no one wishes to take responsibility, data erasure presents a number of challenges; the obvious one being whether deleting data in the near future will present a cost saving and risk mitigation, or instead result in a loss of opportunity or inability to defend against claims.
From a practical point of view, data indexing across all domains may be difficult considering data storage architecture such as data lakes, warehouses and siloed data marts, virtual or physical storage facilities, on-prem or remote, centralised or decentralised, long-term or instant access storage, etc. However, without such indexing, data erasure cannot be implemented.
This is exacerbated by the unreliability of conventional erasure techniques. A "DIY" approach might not result in the desired risk mitigation. Third party enterprise data erasure software and drive eraser tools conforming to standards such as NIST 800-88, IEEE 2883-2022, DoD 5220.22-M/ECE and possibly certified under the Australian Information Security Evaluation Program (AISEP), integrated with major data governance and data storage providers, will be required to wipe all accessible data storage blank. Getting erasure wrong might mean that the data removal will not result in the expected risk mitigation.
Conclusion
Data retention is hard. However, without data governance and a data retention policy, significant insights, monetisation, risk mitigation, cost-saving opportunities and environmental benefits could be lost.
Besides, organisations have a legal obligation to engage with data governance, and failing to do so could result in an infringement of cyber, data privacy and confidentiality obligations and give rise to reputational, regulatory and litigation risk.