GenAI Law Cases Analysis

Executive Summary

Problem Statement

The Client, ACME Law operates in the high-stakes litigation funding market, a domain characterized by high financial risk and complex, information-dense cases. Their core business challenge was a systemic lack of data-driven decision-making.

Key information required for risk assessment and portfolio management was locked in vast volumes of unstructured and semi-structured legal data, including case histories, documents (PDFs, DOCX, emails), and memos. This created several critical business problems:

• Manual and Subjective Risk Assessment:
  Financial decisions and case assessments were heavily reliant on manual review, making the process slow, labor-intensive, and subject to individual bias.
• Inability to Forecast Outcomes:
  Without a structured data foundation, ACME Law could not reliably predict key case metrics, such as probable outcomes, durations, or financial returns.
• Lack of Portfolio-Level Insight:
  Data was fragmented across individual cases, making it impossible to identify systemic risks, emerging trends, or performance bottlenecks across their entire investment portfolio.
• Opaque and Inefficient Processes:
  The “messy” nature of the data meant that expert legal judgment was spent on low-value data retrieval and organization rather than high-value strategic analysis.

Solution

The primary objective was to develop an (gen)AI risk and portfolio intelligence tool. A platform that could ingest high-volume, unstructured data and transform it into a structured, analysis-ready asset.

The solution needed to:

1. Automate risk assessments.
2. Predict case outcomes with a high degree of accuracy.
3. Generate actionable, portfolio-level insights.
4. Ensure Trust by delivering “explainable AI” (XAI), where predictive models were transparent and auditable, empowering rather than replacing human expertise.

The engagement began with a focused Phase 1 (MVP) to rapidly validate the use case and deliver the foundational capabilities for this scalable, market-ready solution.

Outcome

After 6 weeks of iterative design and development, the MVP was delivered and accepted by the client.

The following features and capabilities were developed:

1. Python package to extract data from the legacy CRM system using REST and SOAP APIs.
2. ETL pipeline in Databricks to sync and deduplicate the CRM data to Unity catalog.
3. Data warehouse in Unity catalog using the medallion architecture.
4. Data cleaning and preprocessing pipeline to standardize the data.
   • used various document parsers to extract the data from the various document formats and convert to markdown.
   • Leveraged NLP and simple heuristics to clean free text data and reduce its volume by 25%.
5. Feature engineering pipeline to create new features from the silver layer using DBX native LLM functions:
   • named entity extraction
   • document classification (e.g. case type, case status, case outcome, etc.)
   • document summarization
   • date and time extraction from the documents’ free text
6. An interactive dashboard build with Streamlit and deployed on Databricks Apps for the client to use.

---

The challenge

1. Extracting the data from the source

All of the data had to be sourced from a CRM like tool that is designed for the legal industry. It held structured and unstructured data for all cases.

Huge parts of the “structured data” was stored in a custom key-value “schema” which varied between different types of cases (e.g. personal injury, fraud, etc.). As an added bonus, there was no guarantee it was regularly updated or maintained (but that was a problem for the post processing pipeline).

The unstructured data was stored in a variety of formats DOC, DOCX, several types of PDF, emails, text messages, etc. Each document had a fairly structured metadata. But with the exception of the timestamps and UID fields, it was not to be trusted. For example, there were two different fields that defined the document MIME type, 95% of the time one of them was accurate, but that was not good enough. So in the cleaning pipeline we had to design a clever (lazy) system that would try to infer the correct MIME type (more on that later).

However the biggest challenge was not the data itself, but the dinosaur CRM and its SOAP and REST APIs. We were actually very happy it had an API to begin with. So we tasked the best LLM of the day to read through 100s of pages and generate a Python client (one for each API). However the combination of inconsistent API design, incorrect documentation and LLM hallucinations made it impossible to use without manual rewrite. Through trial and error we quickly managed to identify the correct sequence of endpoints and payloads to use. Certain data we could get only through REST, and others only through SOAP.

Luckily there were a few endpoints that allowed to do an almost full “table” dump for a single case. Unfortunately that meant that we had to come up with a deduplication strategy. Unity catalog was able to help us with that by using MERGE INTO and unique keys.

Another and much bigger challenge was the data volume and lack of rate limiting on the server side. The CRM was a self-hosted solution and every time we tried to pull more than 1 case at a time (100s of flat files per case), the server would hang and break the connection. This also created a big problem for ACME Law, as this was their production environment and they could not afford to have the server down.

To mitigate this problem we tested several strategies and ended up using them all, in parallel:

1. Single threaded requests. By not DOS-ing the server, we thought we could avoid crashing it. Alas that was not enough. It was crashing due to some unknown OOM problem.
2. Client (our side) rate limiting. We implemented a simple rate limiter that would implement a max number of requests per second. This was implement in a very clever way by monkey patching the retry library to add a global variable. We then worked with the client to find a rate that would NOT crash the server and not move at a glacial pace.
3. Only pulling files that are not in the Unity catalog. Our EDA showed that documents (flat files) are never updated, only inserted or deleted. This meant that we can request only “new” documents and skip the ones that are already in the catalog. The ETL pipeline and database had to be carefully crafted to have the concept of state for each document (new, failed, parsed, etc.)

2. Document processing with genAI

Databricks has native LLM functions that not only wrap around an LLM API, but also provide a lot of helpful functions to process the data. Summarization, classification, entity extraction, etc. are all supported out of the box. We did not run proper tests, but anecdotal evidence showed that the results were accurate about 90% of the time. We never saw an instance of hallucinations, but we never looked too hard for it either, we deferred that problem to the future, where we could collect feedback from the domain experts. Most of the inaccuracy had to do with missing or mismatched facts. e.g. the defendant lawyer was classified as the defendant. For this stage of the project, we deemed it to be good enough.

Email chains

To no one’s surprise it became obvious that GPT was trained on emails, as we had no problem extracting sentiment and summarizing them. The main issue we had was low entropy of the data.

As you can imagine most of the emails in this domain are very long chains between two parties. And with every reply you get a copy of all previous emails in the chain. This created 2 problems:

• Earlier messages repeat more frequently than later ones, which might bias the LLM (speculative).
• A torrent of signatures and footers with phone number, addresses and GDPR compliance notes that are completely irrelevant.

A quick analysis showed that the real substance of those email chains was less than 20% of the characters. And anecdotal evidence pointed to that large fluff as the culprit of the occasional mislabelled entities. Rather than trying to fix the problem by throwing MORE LLM at it, I decided to try some “old school” NLP techniques.

We used a combination of n-grams and simple heuristics (e.g. a signature always starts in a new line and is at the end of the email) to identify strings that repeat most frequently. They would be stored in a designated table and updated on a regular basis (for MVP, this was manually). During the cleaning process we would use regex for fast search and replace.

A few other heuristics were also implemented - removing phone numbers, multiple whitespaces and newlines, etc. Using genAI to write the regex patterns and multiple unit tests (including test data) was a breeze and saved us a lot of time. As expected this did increase the processing time, but it we saved a lot more in waiting for the LLM response (and not the mention the cost on pointless tokens).

Word files

In 2007 Microsoft introduced the Open XML format for Word documents - aka DOCX. It has a complex history and a fascinating technical details that could be a subject of its own article. There are a lot of libraries and tools in the Python ecosystem that can read, write and process it in various ways. In this case we were interested in converting it to a markdown format.

But as fate would have it, 18 years later, some are still using the old DOC format. Very few libraries can support it, and they are just Python wrappers around the native C++ libraries. Since we wanted to preserve the formatting (headers, bullets, tables, etc.) simple plain text conversion was not desirable. Instead we converted the DOC to DOCX with a CLI tool and then the rest was the same. Due to the complexity of this ancient format, the process would sometimes fail with cryptic errors. In those cases we would extract as plain text and skip the markdown conversion.

Another interesting challenge was identifying when we were handling DOC or a DOCX file. As mentioned previously, we did have a MIME type available in a few fields, but it was not always accurate. The file extension was also misleading.

So what do you do when you can’t trust your metadata? We just assume its a DOCX (highest probability) and if we get an exception, we assume its a DOC, convert to DOCX and try again. No need to read file headers and search for hexadecimal signatures. Asking for forgiveness rather than permission, is the Pythonic way.

Tables in PDFs

This problem deserves a brief mention. There are many tools and libraries that can convert PDF to markdown. Not all of them are created equal, especially when it comes to tables.

My experience has showed me that there is no one library to rule them all. There are some nice LLM based OCR models that have emerged recently ( OLMo or LlamaParse or Mistral OCR ) but for an MVP something like PyMuPDF is more than enough.