Some flavor of machine learning with
applications in geophysics

Léonard Seydoux
ISTerre, Univeristé Grenoble Alpes

leonard.seydoux@univ-grenoble-alpes.fr

Frontiers in Geosciences

Main targets

• Flyover machine learning basics and main applications
• Decrypt a machine-learning based geoscience paper
• Implement machine-learning solutions when needed

We will also answer any questions you may have

References

The deep learning book

• History of machine learning
• Introduction to linear algebra
• Machine-learning basics
• Advanced deep learning
• Full online content at www.deeplearningbook.org

scikit-learn
a machine learning library

• Open source python library
• Algorithms linked with courses
• Greybox-style

Lots of examples there!

3blue1brown.com

• Animated math
• Ultra pedagogic
• Some videos about deep nets

I strongly recommand

What is machine learning?

Machine Learning – high-level definition

Machine learning is the research field where are studied algorithms that
improve the performance $P$ at solving a task $T$ from experience $E$

A well-defined learning problem is given by $P$, $T$ and $E$

Other possible definitions that also work

• Machine learning = solving inverse problems
• Machine learning = statistical model testing
• Machine learning = statistical model inference

Machine learning is useful, when

• human expertise does not exist (navigation on Mars)
• human expertise does not formalize easily (face recognition)
• too many variables are to take into account (medical decision)
• the data is too big (data compression)
• performance improvement is needed (computational speed)

No need to use machine learning if classical strategies work

Famous examples – Image classification

• Task: recognize and locate objects in image
• Performance: classification/location error
• Experience: labeled images

Various applications in medicine, anomaly detection, etc.

Famous examples – Image captioning

• Task: generate caption from image
• Performance: caption quality
• Experience: already captionned images

Example of application : autonomious cars

Famous examples – Image captionning

https://vimeo.com/146492001

Famous examples – Data compression

Data compression with autoencoder

Other applications ?

Famous examples – Hand-written digits recognition

• Task: recognize digits from pixels
• Performance: recognition success
• Experience: labeled hand-written digits

How would you proceed without learning?

Main types of machine learning

SUPERVISED LEARNING
Find the mapping $f(x) = y$ given training data
samples $x$ and labels $y$. (e.g. regression, classification, ...)

UNSUPERVISED LEARNING
Given training data $x$, infer any existing internal structure.
(clustering, anomaly detection, blind source separtion ...)

Supervised machine learning

Solve $y = f(x)$, please.

Useful definitions

Common supervized tasks

• Regression: both $x$ and $y$ are continuous
• Classification: $x$ is continuous, $y$ is descrete

Supervised task – Regression

Given data samples $x \in \mathbb{X}$ and derised corresponding
labels $y \in \mathbb{Y}$, find a mapping function $f : \mathbb{X} \rightarrow \mathbb{Y}$
between $x$ and $y$ so that $f(x) = y$ is true most of the time.

→ Illustration with linear regression

Linear Regression

Which coefficients $a, b \in \mathbb{R}$ best explain the relationship between $x$ and $y \in \mathbb{R}$ so that

$y = ax + b$ ?

We need to optimize $a$ and $b$. Here, optimize = learn

Linear Regression

Brute-force learning
– a modern way for
"testing plenty of solutions"

How to select the best model?

Linear Regression

$\tilde{y}_i$ is the label predicted by the model for a given data sample $x_i$

$\tilde{y}_i = ax_i + b$

With the true label $y_i$, we can define the prediction loss:

$\epsilon_P = \sum_{i=1}^{N} \left( \tilde{y}_i - y_i\right)^2$

Possible loss – least squares

Linear Regression

The task becomes finding
$\theta^* = (a^*, b^*) \in \mathbb{R}$ so that

$\displaystyle \theta^* = \argmin_{a, b}\sum_i \left( \tilde{y}_i - y_i \right)$

Brute-force can get a little crazy
Any better idea?

Parameter search with gradient descent

How do we infer $\displaystyle \theta^* = \argmin_{a, b}\sum_i \left( \tilde{y}_i - y_i \right)$ efficiently?

1. Define an initial model $\theta_0 = (a_0, b_0)$
2. Observe $\delta \epsilon$ for $\theta_1 = (a_0 + \delta a, b_0 + \delta b)$
3. Update the model accordingly $\theta_1 \rightarrow \theta_0$
4. Iterate until the loss does not evolve anymore

The perturbation amplitude is called the learning rate
An iteration is called a training epoch

Parameter search with gradient descent

Every data scientist need to cross-validate the learning rate

Different gradient descend techniques

Need to select the right optimizer (watch out the loss, intuition about physics,...)

Overfitting

Many parameters may be used for complex regressions.

The green model error $\rightarrow$ 0, yet the model does not generalize.

How do we prevent overfitting?

Prevent overfitting with
training & testing data

The parameters are learned from the training data with the training loss


The algorithm must also minimize the testing loss
(= generalization error)

We stop at the tradeoff

Prevent overfitting with
regularization penalty

Adding a penalty to the loss function for preventing noise contamination

$\displaystyle \min_\theta \sum L (f_\theta(x_i), y_i) + \color{red}{R(f_\theta)}$

How strong? Which penalty?

Which regression algorithm?

• Different loss functions
  e.g.: least square, entropy, likelihood, ...
• Different regularization
  e.g.: L1, L2, random dropout, ...
• Different models
  e.g.: linear, nearest neightbors, ...

As many definitions as regression algorithms

Another supervised algorithm – Support-vector machine

Works for regression or classification

Finding the right hyperplane

In two dimensions, a hyperplane is a straight line.
What if te data cannot be split by a line?

Representation matters! Here comes the kernel trick

kernel trick

Increase the data dimension with non-linear feature combinations, and find the hyperplane "there" in the high-dimensional space.

Here, we have the Radial Basis Function (RBF) kernel $\phi(x, y) = (x, y, e^{-x^2 + y^2})$

Back in 2D, the kernel projection is non-linear

Example of Support vector machine with different kernels

Supervised machine learning strategies

• Representation matters
• Kernel tricks are common strategies to improve performances
• As many regression/classification algorithms as combinations of losses, models and optimizers

Machine learning for signal analysis

How do we deal with waveforms?

Naive attempt – Waveform as a set of feature

A $N$-point waveform can be seen as a
single data sample with $N$ features

Naive attempt – Waveform as a set of feature

These two waveforms are shifted a tiny little bit in the time domain.
This makes a huge change in the feature comparison...

A simple distance measurement lead to a 32% correlation between the waveforms
We should change the cost function or extract more relevant features

Manual feature extraction for classification of volcanic events

Manual feature extraction for classification of volcanic events

What if we used the spectrogram directly?
What would be the drawback?

Manual feature extraction for time-to-failure earthquake prediction

Task – predict the time-to-failure from the "seismic" data

Manual feature extraction for time-to-failure earthquake prediction

Random forest regressor

Random forest = forest of decision trees

While decision trees tend to overfit, random forests are powerful

Manual feature extraction for time-to-failure earthquake prediction

Here we go! Remember, that's in a laboratory!

Manual feature extraction for time-to-failure earthquake prediction

Here are the feature that are of greatest help to perform the task

Deep neural networks

Learning abstract features from data

Which features should matter?

An artificial neuron
$y = \varphi(\Sigma\omega x + b)$

Where $x \in \mathbb{R}^m$ is an input data sample, $w \in \mathbb{R}^{m}$ are the weights and $b \in \mathbb{R}$ the bias, and the function $\varphi$ is the non-linear activation. The output is a new feature $y \in \mathbb{R}$.

The idea is to transform the input data into another feature space

Activations allow for non-linear regression

Fully-connected neural network

The depth correspond the the number of hidden layers

Fully-connected neural network

HUGE number of parameters to invert !...
Requires BIG datasets

Let's try out

Convolutional neural networks

Instead of using weights and bias, one can use convolutional filters

Let's try out with convolutional filters

Different convolution arithmetics

ConvNetQuake – a CNN for earthquake detection and location

Black dots are the test data...

ConvNetQuake – a CNN for earthquake detection and location

Results show that we can (roughly) learn the
location of an earthquake from a single station!

PhaseNet – a seismic phase detector

Task: learn to recognize P and S picking from manual picks

PhaseNet – a seismic phase detector

P accuracy: 93%
S accuracy: 85%

PhaseNet – a seismic phase detector

P accuracy: 93%
S accuracy: 85%

PhaseNet – a seismic phase detector

P accuracy: 93%
S accuracy: 85%

Unsupervised learning

Exploring the data with implicit models

Unsupervised-learning common tasks

• Clustering
• Blind source separation
• Data compression
• De-noising

Clustering – most common definitions

Depending on the definition used, the techniques and results may vary.
Clustering is an exploratory task, every result make sense

Most simple (yet, powerful) example – K-means

Find $K$ cluster centroids $\mu_k$ that minimize
the within-cluster variance. In order words:

$\displaystyle\argmin_s \sum_{k=1}^K \sum_{\mathbf{x} \in C_k} \| \mathbf{x} - \mathbf{\mu}_i\| ^ 2$

We have the data and a task to solve

Most simple (yet, powerful) example – K-means

1. Initialize with (random) centroids
2. Assign each data samples to the closest centroid
3. Update the centroid with the within-cluster data
4. Iterate until minimum is reached

Most simple (yet, powerful) example – K-means

1. Initialize with (random) centroids
2. Assign each data samples to the closest centroid
3. Update the centroid with the within-cluster data
4. Iterate until minimum is reached

The final decision boundaries are called Voronoid cells

How many clusters?

Silhouette = distance of $C_i$- to $C_j$-samples. When ~ 0, it indicates bad splitting.

How many clusters?

Clustering splitting occurs when silhouettes below average silhouette score

How many clusters?

$K$-means is adapted for linearly separable and balanced problems

Is your data linearly separable?

What algorithm is the best in each case?

Is your data linearly separable?

Should we change the features or the algorithm? We will try out tomorrow

Clustering of volcanic tremors at Kamchatka

We will do better than this!

Unsupervised learning
Principal component analysis

Let consider $\mathbf{x} \in \mathbb{R}^{F \times N}$, find a decomposition of $\mathbf{x}$ so that

The $\mathbf{u}_i$ are the (orthogonal)
principal components.

Principal component analysis for compression

The image looks good after 64 principal component (instead of 128)

Is your data linearly separable?

If not, you can use the kernel PCA!

Another decomposition: Independant component analysis

Same as PCA without the constrain on orthogonality

Independant component analysis for blind-source separation

PCA fails to separate the different sources in that case → ICA does better

Example of clustering – Human-induced seismic sources

Example of clustering – Human-induced seismic sources

Feature extraction from seismograms

Example of clustering – Human-induced seismic sources

The clusters naturally highlight the amount of injected water

Deep unsupervised learning

Learn how to encode the data from iteself

Deep autoencoders

Task: learn to reconstrut the data ($\tilde{X}$) from itself ($X$) with constrains

an autoencoder is an deep-net extent of principal component analysis

Autoencoder in action
Seismic data compression

Learn to encode a 512-point seismogram into a 32-point with best reconstuction quality

Autoencoder in action II
Data quality assessment

Rejecting the seismograms that are hard to decode with low errow.

The end !

Any ?

Aknowledgements

• These slides have been made with Marp Next, a framework for slide formatting with Markdown.
• The content of these slides were inspired from the scikit-learn toolkit, a wonderful open source python library.
• Also, the structure of the course was largely based on the one of the Deep learning book.
• Online videos from 3Blue1Brown where extremely inspiring for pedagogic explanations, and some of them used in this course.