January is a wonderful time to visit Palo Alto, especially if you are a graduate student in Hyde Park, Chicago. I was always on the lookout for an excuse to be in California during my days at Chicago or Boston. There were many highlights of these trips and one of them was meeting an eccentric and incredibly sharp college mate who was a graduate student working with Vijay Pande.
Our research interests had significant overlap and we would talk a lot about work which at the time was mostly protein structure and dynamics. However, my brilliant friend had the audacity to envision simulating entire cells and even combinations of cells. But here’s the thing: we didn’t have the tools or even the language to meaningfully translate any of those conversations to a plan.
Every time we spoke, it was exhilarating but also discouraging. It was like having a key to the door that didn’t exist yet.
My friend went on to pursue a career with Goldman Sachs after his PhD and almost a quarter century later, I find myself revisiting those discussions, specially when I see most recent papers coming out in the space of computational biology. It feels like finally we have the language and tools now to execute on bold audacious ideas. Deep Learning models, in particular, have opened up a world of possibilities that were once only theoretical.
I believe we are entering a three-to-five-year window where the integration of multimodal data will dramatically accelerate drug discovery and development.
From target identification to clinical trials, we’re witnessing the dawn of an end-to-end, AI-powered journey that could redefine biopharma R&D. In the last 18 months alone, we’ve seen a surge of companies scrambling to build foundation models for the different stages of the R&D pipeline. This is valuable but also suddenly a very crowded space.
At Elucidata, we’re taking a very contrarian approach.
Instead of building one foundation model for a part of the drug discovery value chain, we’re developing a robust playbook—a set of three to five key "knobs" that we can tweak to out perform any foundation model and make it more relevant to your specific biology, therapeutic modality, or question of interest.
This approach allows us to serve an unmet need to find the right model(s) and make them work for your specific science!We have created a Multimodal AI platform, designed to integrate diverse data sources—EHR, imaging, molecular data — to optimize model performance and improve efficiency throughout the R&D pipeline.
We’ve narrowed down our focus to three core areas, or axes, that determine the performance of a foundation model.
The first is data for pre-training models. Many of the models that have been published so far are built on training data that’s not harmonized or uniformly processed. This leads to models to not being able to parse real signal (biological variance) from artifacts (variance due to different processing pipelines).At Elucidata, our Harmonization Engine plays a critical role here. By consistently processing large datasets, for example single-cell RNA sequencing data, we’re able to build models that outperform those built on noisy, author-processed data.This consistency allows us to create AI-ready data that can be used across various R&D stages, from target identification to clinical trials.
Our Results:
When we trained our El-scGPT model using harmonized cells, we didn’t just match state-of-the-art performance—we surpassed it, all while using 7x fewer cells. In an ideal world you would want more data that is clean as well but in practice clean data is of higher value than more data. You can find the preliminary results here.
The second axis is feature representation. What is the best way to describe a cell for a deep learning model? Is it sufficient to rely on the top 1,200 highly variable genes? Or is there a better way to represent a cell’s phenotype? While models like GPT and AlphaFold have made great strides in language and protein structure prediction, we still have much to learn about how to represent cells or more complex biological systems.
By improving data feature representation, we’re working towards creating better models for predicting biological processes and clinical phenotypes, particularly in areas like cancer and autoimmune diseases.
The third axis is architecture. Over the past 15 months, we’ve seen many models rely on transformers, which are great for sequential data. But there hasn’t been a systematic comparison of different architectures, like CNNs or GNNs, to see if they might perform better for certain tasks or data modalities.
When we evaluated whether fine-tuning Geneformer with highly curated metadata would improve performance across multiple tasks, we found the results to be positive.
We believe there’s a lot of room to innovate in fine-tuning strategies, especially when it comes to foundation models for single-cell RNA sequencing data—a particular area of interest for us given our work in spatial biology. This focus has already led us to sign MOU with National Cancer Institute (NCI) to build foundation models with multi modal data, using imaging and gene expression data to predict clinical outcomes.
It is a noisy world out there. 10s of papers on foundation model every week, 100s of conferences, your news feed, AI companies doing massive funding rounds. It can be overwhelming to make sense of all of it and see what can actually work for your team. Elucidata is committed to lead the way in realizing the promise of this technology to bring better care for patients. As we continue to define our approach to leveraging foundation models for drug discovery, I’m reminded of the early conversations I had with my friend on sunny January afternoons at Stanford.
Today it feels like there are early signs of a key that just might open the door.
We’re at a turning point where the tools and technologies and equally importantly a language to describe the problems and solutions, we once lacked are finally available to us.
Can’t wait to see what is on the other side of the door!
.svg)
%201.svg){kind=icon}