Small Data, Big Value

Small Data, Big Value

Author: Mason Dykstra, Ph.D., VP Energy Solutions 

Enthought welcomes Mason as VP, Energy Solutions, whose background from Anadarko, Statoil, and as a professor at the Colorado School of Mines qualifies him to make the case for ensuring ‘Small Data’ is equally part of the Fourth Industrial Revolution. The first in a Small Data series.

The origin of the term Big Data will likely never be agreed. However, in the world of science and computing, the case can be made that the term originated in Silicon Graphics in the 1990’s, whose work in video, for surveillance and Hollywood special effects, had it facing orders of magnitude more data than ever before. Recent advances in scientific computing technology and techniques, and massive generation of data, in particular by consumers and from social media, have put the term Big Data at center stage.  

However, in many scientific fields, Big Data does not exist. It’s all about getting the most from ‘Small Data’, ensuring scientific challenges with minimal data also benefit from the ‘Fourth Industrial Revolution’. In many natural sciences and engineering disciplines large volumes of data can be hard or very expensive to generate. The reality is these datasets are often limited in size, poorly curated, and bespoke to particular problems. So, either the fields lacking in Big Data will be left out of the ‘Revolution’, or we need to work on ways of unleashing the power of Small Data.

Scientists are particularly adept at teasing meaning out of Small Data and drawing important conclusions with limited datasets. The future will be a collaboration between humans and machines, but clearly we don’t only want to solve the problems that have Big Data behind them. In cases where datasets are relatively small, or important pieces of information are missing, how can we develop this type of ‘intelligence’ in machines? 

We need to engineer applications that can approach problems the way a scientist would. Scientists typically hypothesize as they go, which is to say they don’t wait until they have enough data to draw conclusions, but they actually generate, evolve and discard hypotheses along the way. While gathering data we are already engaging in problem-solving. 

For example, when a geologist is creating a map of the geologic layers and faults under the Earth, they continually make educated guesses about what some of the map features will look like before they have gathered all the data. Not only does this give the geologist something early on paper (ok, on screen), but actually it provides a basis for hypothesis testing, and can help steer the succeeding data-gathering step. Think of this as akin to coming into a new town for the first time – even though you might never have been to that particular town before, all towns share certain traits and tend to have similarities which we can use to imagine the parts we haven’t yet seen. This kind of intuitive thinking and rule-of-thumb-based guessing, although critical for many sciences, has not been the realm of computers. Yet.

So the real question is can we capture the essential parts of that rule-making process and combine it with ‘machine reasoning’ to develop Small Data approaches that are akin to the way a scientist would approach a problem? But much faster and more consistent? This is one of the major challenges for many scientists today, whether they recognize it yet or not.

One thing we do know, paraphrasing Antonio di Leva in The Lancet; ‘Machines will not replace scientists, but scientists using AI will soon replace those not using it.’

About the Author

Mason Dykstra, Ph.D., VP Energy Solutions  at Enthought, holds a PhD from the University of California Santa Barbara, an MS from the University of Colorado Boulder, and a BS from Northern Arizona University, all in the Geosciences. Mason has worked in Oil and Gas exploration, development, and production for over twenty years, split between oil industry-focused applied research at Colorado School of Mines and the University of California, Santa Barbara; and within companies including Anadarko Petroleum Corporation and Statoil (Equinor).

Share this article:

Related Content

Making the Most of Small Data in Scientific R&D

March 9, 2023|Life Sciences, Materials Science, Transformation Making the Most of Small Data in Scientific R&D For many traditional innovation-driven organizations, scientific data is generated…

Read More

ChatGPT on Software Engineering

Recently, I’ve been working on a new course offering in Enthought Academy titled Software Engineering for Scientists and Engineers course. I’ve focused on distilling the…

Read More

What’s in a __name__?

if __name__ == “__main__”: When I was new to Python, I ran into a mysterious block of code that looked something like: def main():  …

Read More

3 Trends for Scientists To Watch in 2023

As a company that delivers Digital Transformation for Science, part of our job at Enthought is to understand the trends that will affect how our…

Read More

Accelerating Science: the Classical Mechanics Perspective

When thinking about enhancing R&D processes, Newton’s second law of motion provides the perfect framework. Classical mechanics teaches us that putting a body into motion…

Read More

Announcing Enthought Academy

Dear Students and Friends of Enthought,  I am pleased to announce Enthought Academy—the culmination of over twenty years of teaching Scientific Python. Since our founding…

Read More

Extracting Target Labels from Deep Learning Classification Models

In the blog post Configuring a Neural Network Output Layer we highlighted how to correctly set up an output layer for deep learning models. Here,…

Read More

True DX in the Pharma R&D Lab Defined by Enthought

Enthought’s team in Japan exhibited at the Pharma IT & Digital Health Expo 2022 life sciences conference in Tokyo, to meet with pharmaceutical industry leaders…

Read More

Life Sciences Labs Optimize with New Digital Technologies and Upskilling

Labs are resetting the trajectory for drug development: reducing timelines from years to months; decreasing costs from billions to millions; and gaining an advantage by…

Read More

Webinar Q&A: Accelerating Product Reformulation with Machine Learning

In our recent C&EN Webinar: Accelerating Consumer Products Reformulation with Machine Learning, we demonstrated how to leverage digital tools and technology to bring new products…

Read More