Advances in technology have made it possible to collect nearly infinite amounts of data. However, leveraging such data is a continuing challenge. Especially challenging is the potential loss of valuable information due to data that cannot be accessed to conduct analysis or develop insights. However, thanks to the development of big data, this previously inaccessible “dark data” can be converted from analog to digital form for analysis and subsequent migration to the cloud. This digitized data also plays a potentially valuable role in facilitating nationwide property data search for home buyers, realtors, lenders, and anyone involved with real estate.
Shedding the Light on Dark Data
Gartner originally coined the term dark data, along with its definition as "the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes.” In plain English, dark data refers to intelligence or information that is dormant. It is collected and stored, but has not been analyzed or even effectively categorized.
Also known as “dusty data,” dark data is often as extensive and voluminous as big data. However, unlike big data, dark data is often viewed as having no intrinsic value. In some cases, dark data is the by-product of a data collection process. That is, dark data is information that is acquired in the process of collecting data on a different topic, which may or may not be related.
However, dark data may be outdated versions of previously viable data. Information such as old financial records or notes from previous presentations may or may not have been replaced with updated records. Likewise, raw data, which is also frequently viewed as dark data, is often retained after the original analysis for which it was gathered has been conducted.
Hazards of Dark Data
Dark data holds intrinsic hazards beyond the costs associated with its retention. Specifically, dark data is subject to security dangers and compliance issues. Dark data hazards apply to data stored in house, in the cloud, or in a hybrid storage strategy. In-house storage is convenient, but vulnerable to loss due to fire, natural disaster or theft, while cloud storage limits the immediate availability of data. Hybrid storage shares both the advantages along with redundancy against loss. However, hybrid storage can leave data vulnerable on multiple fronts.
Whether stored in-house or in the cloud, many companies make the mistake of placing a low priority on the storage and security of dark data. As a result, such data is often at risk of unauthorized access. Similarly, dark data may be stored in a form that is non-compliant with important regulations such as HIPAA or GDPR — with violations leaving companies subject to fines and penalties, not to mention potentially catastrophic data breaches.
The risks associated with dark data can be mitigated by including dark data in a company’s periodic evaluation and audit process of data to determine whether it should be retained or discarded. Of course, dark data that appears to have minimal value may become essential information at a later date. Therefore, wholesale discarding of dark data is ill advised. Instead, strong encryption of digital data and secure storage that provides protection against both theft and catastrophic loss is needed.
The Silver Lining of the Dark Data Cloud
In acknowledging the risks associated with dark data, it is important to avoid overlooking its potential opportunities and benefits. Modern technology and advances allow a lot of dark data to be synthesized and analyzed to become active intelligence.
Data that is retained in its “dark” state has potential value as well. For example, historic data often provides valuable context to contemporary data in observing change over time. Examples of dark data that are potentially useful in their original form include customer call records, sales figures and website visitor information.
Dark data also has potential value for document reporting services. As long as such data is securely maintained, it can be promoted to potential and existing clients along with current data as a means of providing historical insights into relevant categories such as home value trends or default rates in specific neighborhoods.
Leveraging Dark Data as a Business Strategy
Dark data has both pros and cons. Drawbacks associated with dark data are often more apparent than benefits, and are legitimately serious. Lack of security and noncompliance associated with dark data can leave companies vulnerable to cyber attacks, while non-compliance issues can lead to legal and regulatory headaches.
However, dark data is often simply data that has not been properly categorized or analyzed. Similarly, it is frequently valuable in its present form. Whether digitized or left raw, dark data presents a tremendous opportunity for savvy businesses to leverage. Check out the DataTree blog for the latest news and information on this new, important tech trend.