Being able to quickly and accurately partition the legal space into coherent, useful categories is crucial to conducting legal research. In service of this goal, various lists and taxonomies of legal topics have been created. Recently, Ravel reworked our own legal topics system to provide ourselves the flexibility to more easily expand the topic taxonomy and to provide a clearer, more immediately useful service for users.
Document Classification is a common task that usually relies on at least some amount of human-sourced information to assign labels to documents. In our case, we have a provided set of labels corresponding to legal topics that we must assign to opinions. We allow potentially many labels per document, making this an instance of multi-class, multi-label classification.
For our particular use case, we use a Binary Relevance approach, where a classifier for each individual label/topic is trained independently and subsequently evaluated on texts. This setup allows for multi-labeled predictions while still using straight-forward linear models. While this is a fairly standard approach, we make use of an interesting annotation framework to maximize the effectiveness of our annotators’ time and reduce the time needed to roll out new topics into our system.
Prior to this work, the topic model in use at Ravel was based on a fixed Latent Dirichlet Allocation (LDA) model, summarized in Fig. 1. After selecting an arbitary value for the number of LDA clusters, the trained model was manually examined and human-readable labels were simply assigned to the various clusters as appropriate. New documents were assigned an LDA cluster and simply inherited that cluster’s topic label.
Because many of the clusters represented no particular coherent topic, only a subset of opinions received a label under this scheme, and each opinion could only receive a maximum of one – both of which are not ideal properties for our purposes. Furthermore, to add new topics to this model it would need to be entirely retrained and each new cluster would need to be reanalyzed – there would be no assurances that existing clusters would survive the retraining. This particular fact is the most problematic for a system that will eventually expand to several hundred topics in the end.
Typical document classification systems involve getting top level labels for exemplar documents from annotators and allowing a featurization routine to extract useful features from those documents that are then used to make predictions on new, unseen data. In the original Ravel topic system, there was a single feature (namely the LDA cluster that the document fell into), and a hard-coded mapping to labels based on those features. Because there were no top level labels on the documents, the actual learning potential of the model was minimal; the model itself was unsupervised.
As a pilot study for a new model (shown in Fig. 2), our annotation team provided several hundred top-level labels for documents in our corpus. Then, rather than using LDA clusters as features, a standard lexical featurization was conducted, where the features for a document became the counts of the words used in that doucment. The intuition being that if documents we label as “Intellectual Property” frequently contain references to things like ‘copyright’ or ‘trademark’, we can learn to associate those words with the topic of “Intellectual Property”.
While this process is a well-used one, the particular details of our situation complicate it. For one thing, the taxonomy of topics into which we are classifying is rather large and will fluctuate and expand over time. As such, we place a high value on our ability to quickly and reliably create new topics. However, adding a new label under standard document classification practices involves actually obtaining a set of documents that should have that new label – the more the better, but a few dozen as a reasonable minimum. This is not an easy task if, for example, the topic is highly specific; we may not have access to a large enough set of relevant documents to support the new topic.
Instead of requiring that we assemble a new set of documents for every new topic that we wish to support, our solution allows our annotation team to simply describe the nature of those documents. This is done by having the annotators associate important phrases from documents with topic labels rather than providing top-level labels for the entire document itself, shown in Fig. 3. While the existence of a labeled document set is helpful in that it provides a straightforward source for those important phrases, this process can actually be run entirely independent of the corpus – annotators can opt to supply their own key phrases in lieu of actually having to obtain a corresponding set of documents. In our case, a smaller number of whole documents were labeled after the fact in order to provide an evaluation set.
Structuring the model in this way provides a few benefits. First, these new annotations directly provide the feature weightings that all topic models need to function rather than relying on the learning algorithm to both select important phrases and then learn the weightings. Second, the annotation task is now very low-intensity and can be done quickly and cheaply (the interface used by the annotators is shown in Fig. 4). Finally, building the model around key terms and phrases relates the entire system to the well-known idea of boolean search; in some ways, each topic is a sort of archived search result, where our annotators have provided a tuned search query that returns a particular set of topically relevant documents.
The most obvious changes made by this work are that a much higher percentage of the Ravel corpus are now assigned topics, and most labeled documents now receive multiple topics. This means that finding interesting cases at the intersection of topics is easier than ever. For instance, browsing case law by sets of topics like “Intellectual Property” combined with “Employment Law” is now possible.
To see one example of the new topics system in action, check out the new Law Firms Analytics dashboards (on the left) to see firm by firm breakdowns of commonly litigated topics.
Starting today, Ravel’s Judge Analytics now covers magistrate and bankruptcy judges. We are proud to be the first to offer data analytics about these judges, providing unique new insights into how and why they’ve ruled.
In just the past six months, we have expanded our Judge Analytics to cover all federal judges, all state appellate judges, and now magistrate and bankruptcy judges. This extensive and deep data analytics coverage of both federal and state judges is unparalleled. Our rapid expansion is made possible by our extraordinary team of engineers, data scientists, and lawyers, and the best-in-class data processing engine and algorithms at the core of our application.
The analytics and insights we offer for these judges is more robust than ever. Powered by natural language processing, machine learning, and data science, Ravel’s Judge Analytics enable lawyers to research judges in dramatically faster, more effective ways than conventional tools allow. For example, lawyers can instantly sort through an individual judge’s decisions to find those that deal only with certain types of motions or topics or have a particular outcome. At the same time, Ravel’s pattern-spotting technology identifies the language a judge has used in the past, as well as the other judges they find influential and the cases and courts they consider most important.
With these tools, lawyers are drafting better arguments, making better client pitches, avoiding nasty surprises, and saving hours of research time. Contact us to learn more about Judge Analytics and how it is being used by many of the country’s leading lawyers and firms.
As a company founded by lawyers, we developed Judge Analytics to answer questions we and our peers had in practice: What factors do judges consider in making a ruling? How do they rule on particular motion types, and why? What language influences judges and what cases do they consider the most persuasive? In short, we created Judge Analytics to deliver those insights with hard data.
The terrific response that we’ve received since launching Judge Analytics (and our latest enhancements) has proven that we are not alone in seeking objective, data-driven information for legal strategy and research. We’re honored that the American Association of Law Libraries (AALL) has recognized us with its New Product of the Year Award for 2016. We understand that the awards committee undertook a rigorous process to evaluate our platform and that this honor is not bestowed every year.
Our team worked extremely hard to imagine and create Judge Analytics, but we also received deep support from our customers and users who worked side by side with our engineers and designers to provide feedback and insights along the way. Special thanks in particular to Jean O’Grady and DLA Piper; Rachelle Rennagel and White & Case; Cooley; Patterson Belknap; Bartlit Beck; and Simpson Thacher. We’re excited to continue building tools that can transform how lawyers understand the law and prepare for litigation. Thanks for your support!
In our digital age, data is an essential currency. Government at all levels, from local municipalities to the White House, has started to recognize the opportunities that come when open, machine-readable data is the default for government information.
Still stuck in an analog age, however, is the judicial branch. Legal materials largely remain locked behind expensive paywalls or archived in books gathering dust. Our collaboration with Harvard Law School is changing that, and we’re taking the next step in making the law open and accessible to all.
Starting today, for the first time ever, the comprehensive, authoritative collection of New York case law is now digitized and available to anyone with an internet connection. Everyone can search and read all of New York’s cases for free, including milestone cases like Palsgraf v. Long Island Railroad, which greatly influenced the development of American common law on negligence and torts. Using our visualizations, anyone can explore a case map to identify key cases and trace the evolution of legal topics, taking the guesswork out of research.
Some people may wonder how making all this information freely available fits with our mission as a for-profit company. How does Ravel make money, they ask? The answer is simple: we do not believe in charging for access to legal information, and our subscription-based services are built on top of this data to help people work efficiently, thoroughly, and with data-driven intelligence. So for the public, this is an extraordinary new resource. For legal professionals considering or already using Ravel, this is an enrichment that adds even more depth to our paid, advanced services like Judge Analytics and Case Analytics.
New York State’s legal history stretches back to the founding of our country, and the Ravel team is immensely proud that we’re able to make this important material available to our country and the world. In addition to providing advanced search capabilities to New York’s court decisions, Ravel has started to add New York state judges to our Judge Analytics platform, for professionals who subscribe to our analytical tools.
This is just the beginning of what can happen when open data is combined with the power of data analytics. Now that we’ve digitized the case law of California and New York, we’ll be moving even faster in bringing the remaining states’ online. Stay tuned for updates.
Since launching Judge Analytics last year, we’ve been inspired to see lawyers use the platform to craft litigation strategy and conduct novel research. Partners have used it to forecast the outcome of motions, and associates have used it to identify the most persuasive ways to make an argument. As a senior partner at an Am Law 10 firm told us, it “provides reliable, objective data about the way the judge thinks that you can’t get from any other platform.”
We’ve been hard at work expanding our analytics, and today we’re introducing a powerful update to Judge Analytics. This new version retains all the features you already enjoy, and adds more content and analysis to help you make the best possible argument. Here’s what’s new:
- More content: In addition to all federal judges, Judge Analytics now includes appellate judges from New York, California, Florida, Illinois, and Delaware. We’ll be rolling out more state coverage in the months to come, so stay tuned.
- More functionality: We’ve created the ability to dive even deeper into a judge’s history, exploring how they have ruled on motions and specific topics, with detail about what they grant and deny. This adds granularity to tools that identify the specific language and rules that a judge favors and finds persuasive. It’s also easier than ever to share research and create custom reports.
- More insights at your fingertips: We took a fresh look at Judge Analytics and created a clean, new design, along with a major speed increase.
The core features of Judge Analytics remain the same. Powered by natural language processing, data science, and machine learning, our all-in-one dashboard identifies the rules, cases, and specific language that a judge commonly cites. It also displays the cases a judge has authored and an analysis of other judges they are influenced by and other jurisdictions they consider most persuasive. With deeper insight about judges, litigators can make more informed strategic decisions about everything from how to frame arguments to whether to file a particular motion — decisions that can make or break a case.
We’re excited to build products that empower attorneys to make data-driven decisions, and we look forward to hearing your feedback on Judge Analytics’ new features.