Removing Common Files in E-Discovery Processing: De-NISTing Explained

Reducing the number of documents to review during an e-discovery project is a high priority for both attorneys and their clients. One commonly used technique is to remove files from a document set that are known to belong to certain software programs. This process is called “Known File Filtering” and is often referred to as “de-NISTing” since it uses a list of file hashes created by the National Institute of Standards and Technology (NIST).

The “NIST” list is actually a database called the National Software Reference Library (NSRL). This list contains information about software, including “hash” values which uniquely identify the data within a file, regardless of its name, date of creation or location. If two files contain identical data they will also have identical hash values.

The NSRL database represents a collection of categorized file information for software of all kinds. It organizes programs into groups, such as word processing software, system files, gaming programs, etc. This is different than several earlier
file collections of common computer file information (like HashKeeper), the NSRL does not make a distinction between “good” and “bad” files and does not contain lists of contraband data, such as child pornography.

One of the key features of the NSRL is that anyone can submit software for review and inclusion in the list which has helped keep the list up to date. In fact, many e-discovery and digital forensics software companies have included the NSRL in their products to assist with culling out irrelevant data in the early stages of investigation.

The NSRL currently contains approximately 53,000,000 file entries and a new list is released every month to address software updates and newly available programs. Using a complete list and the most current version is a very important step to properly reduce the amount of data that must be reviewed, since some software is not automatically updated and does not contain the complete list.

De-NISTing is a very helpful part of ESI processing but is not a “silver bullet” for reducing e-discovery document sets. There are certain files or programs in the list that may actually be relevant depending on the scope of a particular case. For example, remote access software programs have legitimate IT functions but can also be utilized for nefarious purposes. In situations where misuse of this type of program is suspected, it is critical to communicate this information to an e-discovery or digital forensics vendor to ensure that key information is not inadvertently excluded. Likewise, commercially available data wiping software (such as Evidence Eliminator or Disk Redactor) is certainly present in the NSRL but may be a pivotal part of an investigation involving data deletion.

Since the NSRL is simply a list of categorized programs that are known, and makes no distinction between those that are “good” and “bad”, it should not be used without careful thought. Consideration should be given to the particulars of each
case and whether key information could reside on programs listed on the NSRL.

—
Article is a public service announcement from Avansic.

그랬군. Known file filtering 을 De-NISTing이라고 부르다니...
De-NISTing sounds so COOOOOOOL~~~.