Reviewer QA: 6 Steps to Better Document Review

When discussing ediscovery cost control, the focus tends to be on pricing and speed. What is not mentioned as often is the quality of review. Savings associated with lower vendor fees and quicker review times are important indicators of a well-managed review process. However, taking into account other pertinent factors, particularly accuracy and quality, can make for a more sustainable approach. Performing rigorous QA on review work is a safeguard against costly mistakes that could quickly escalate, as well as a way to prepare for the vagaries inherent in any given matter.

To facilitate this crucial component of case management, Everlaw has an integrated set of tools that case leads can use to perform quality assurance on review work. These accuracy analytics provide information on rating conflicts, as well as user-level predictions of error rates. Below is a step-by-step guide on how to take advantage of error rate predictions:

 

Step 1

Make sure your review team is well-trained on the central issues and legal questions grounding the case. Mark Lyon has some tips on how to successfully train a review team.

Step 2

Have reviewers rate their first batch of documents. For better results, ensure that each reviewer’s batch is a large enough, or representative enough, set of documents relative to the size and scope of the full document set.

Step 3

Review the ratings for a minimal number of documents per reviewer. If your rating on a document differs from that of the reviewer’s, an error is registered for that reviewer. One way to determine the number of documents you need to review is to establish a desired confidence level given available time and resources. Reviewing more documents will raise the confidence level of the error rate predictions our system generates.

Step 4

On the analytics page, you can see the predicted error rate (as a percentage) for each user, as well as a confidence bar indicating how confident the system is in its predictions. Hovering over the confidence bar will provide concrete next steps on how to improve the statistical confidence to 95%, if desired. The buttons under the “next steps” column provide you quick access to the documents you need to either verify (if the statistical confidence is below 95%) or review (if the statistical confidence is above 95% confidence).

Step 5

From there, you can review the specific documents on which errors have been registered, looking for general patterns of mistakes or conceptual misunderstandings. This allows you to evaluate both reviewer-level and team performance.

Step 6

Using this information, you can speak to each reviewer about performance, provide additional training, or create action steps for improving review quality. Based on the strengths and weaknesses uncovered during the reviewer QA process, you can also tailor future assignments to match each reviewer’s capabilities.

Everlaw’s reviewer accuracy tools provide actionable information that you can use for better case management, higher quality review work, and, ultimately, improved cost management. Of course, we’re always looking to improve the review QA process; if you have any suggestions, we’d love to hear from you!

http://blog.everlaw.com/2015/09/28/how-to-reviewer-qa/