Partial automation of legal analysis classification

Nathanial Yzer worked on partially automating legal analysis classification during his undergraduate thesis. Legal analysis provides a method for structuring legal constructs and creating a common language. Essentially, a logical model is created from which legal scenarios can be derived. These scenarios are classified using the JRM classification scheme and can then be verbalized.

This allows for better communication between experts, but is time-consuming. Partial automation leaves less manual work, allowing experts to focus on other parts of their craft.

Nathaniel used the legal reference model JRM 2.0 for this purpose, and further explored what methods might be appropriate to automate analysis classification. In doing so, he found that this was a natural language processing (NLP) problem. Therefore, he tested a variety of suitable machine learning models, including logistic regression, support vector machines, decision tree, random forest model and a transformer-based model. These are all well-known models within the world of machine learning. The performance of these models was evaluated based on common text classification techniques: precision, recall, F1 score, accuracy, macro average and weighted average.

After collecting all the scores for each model, Nathaniel found that the logistic regression model performed best with the chosen evaluation methods.

Although his research showed that this model worked best, there was still room for improvement. Indeed, not all legal elements were classified equally well. In addition, there were possibilities beyond these five models, which certainly still had potential.

So after the completion of Nathaniel’s research in 2023, PNA Group still saw opportunities and continued the research by diving deeper into the models and combining them with a large language model (LLM).

One of the biggest challenges of the study was the limited amount of available data, a problem that PNA Group continues to address by continuously training the model as new data becomes available.