.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/gaussian_process/plot_gpr_prior_posterior.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:here  to download the full example code or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_gaussian_process_plot_gpr_prior_posterior.py: ========================================================================== Illustration of prior and posterior Gaussian process for different kernels ========================================================================== This example illustrates the prior and posterior of a :class:~sklearn.gaussian_process.GaussianProcessRegressor with different kernels. Mean, standard deviation, and 5 samples are shown for both prior and posterior distributions. Here, we only give some illustration. To know more about kernels' formulation, refer to the :ref:User Guide . .. GENERATED FROM PYTHON SOURCE LINES 15-20 .. code-block:: default # Authors: Jan Hendrik Metzen # Guillaume Lemaitre # License: BSD 3 clause .. GENERATED FROM PYTHON SOURCE LINES 21-33 Helper function --------------- Before presenting each individual kernel available for Gaussian processes, we will define an helper function allowing us plotting samples drawn from the Gaussian process. This function will take a :class:~sklearn.gaussian_process.GaussianProcessRegressor model and will drawn sample from the Gaussian process. If the model was not fit, the samples are drawn from the prior distribution while after model fitting, the samples are drawn from the posterior distribution. .. GENERATED FROM PYTHON SOURCE LINES 33-82 .. code-block:: default import matplotlib.pyplot as plt import numpy as np def plot_gpr_samples(gpr_model, n_samples, ax): """Plot samples drawn from the Gaussian process model. If the Gaussian process model is not trained then the drawn samples are drawn from the prior distribution. Otherwise, the samples are drawn from the posterior distribution. Be aware that a sample here corresponds to a function. Parameters ---------- gpr_model : GaussianProcessRegressor A :class:~sklearn.gaussian_process.GaussianProcessRegressor model. n_samples : int The number of samples to draw from the Gaussian process distribution. ax : matplotlib axis The matplotlib axis where to plot the samples. """ x = np.linspace(0, 5, 100) X = x.reshape(-1, 1) y_mean, y_std = gpr_model.predict(X, return_std=True) y_samples = gpr_model.sample_y(X, n_samples) for idx, single_prior in enumerate(y_samples.T): ax.plot( x, single_prior, linestyle="--", alpha=0.7, label=f"Sampled function #{idx + 1}", ) ax.plot(x, y_mean, color="black", label="Mean") ax.fill_between( x, y_mean - y_std, y_mean + y_std, alpha=0.1, color="black", label=r"$\pm$ 1 std. dev.", ) ax.set_xlabel("x") ax.set_ylabel("y") ax.set_ylim([-3, 3]) .. GENERATED FROM PYTHON SOURCE LINES 83-86 Dataset and Gaussian process generation --------------------------------------- We will create a training dataset that we will use in the different sections. .. GENERATED FROM PYTHON SOURCE LINES 86-91 .. code-block:: default rng = np.random.RandomState(4) X_train = rng.uniform(0, 5, 10).reshape(-1, 1) y_train = np.sin((X_train[:, 0] - 2.5) ** 2) n_samples = 5 .. GENERATED FROM PYTHON SOURCE LINES 92-100 Kernel cookbook --------------- In this section, we illustrate some samples drawn from the prior and posterior distributions of the Gaussian process with different kernels. Radial Basis Function kernel ............................ .. GENERATED FROM PYTHON SOURCE LINES 100-122 .. code-block:: default from sklearn.gaussian_process import GaussianProcessRegressor from sklearn.gaussian_process.kernels import RBF kernel = 1.0 * RBF(length_scale=1.0, length_scale_bounds=(1e-1, 10.0)) gpr = GaussianProcessRegressor(kernel=kernel, random_state=0) fig, axs = plt.subplots(nrows=2, sharex=True, sharey=True, figsize=(10, 8)) # plot prior plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[0]) axs[0].set_title("Samples from prior distribution") # plot posterior gpr.fit(X_train, y_train) plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[1]) axs[1].scatter(X_train[:, 0], y_train, color="red", zorder=10, label="Observations") axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left") axs[1].set_title("Samples from posterior distribution") fig.suptitle("Radial Basis Function kernel", fontsize=18) plt.tight_layout() .. image-sg:: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_001.png :alt: Radial Basis Function kernel, Samples from prior distribution, Samples from posterior distribution :srcset: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 123-129 .. code-block:: default print(f"Kernel parameters before fit:\n{kernel})") print( f"Kernel parameters after fit: \n{gpr.kernel_} \n" f"Log-likelihood: {gpr.log_marginal_likelihood(gpr.kernel_.theta):.3f}" ) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Kernel parameters before fit: 1**2 * RBF(length_scale=1)) Kernel parameters after fit: 0.594**2 * RBF(length_scale=0.279) Log-likelihood: -0.067 .. GENERATED FROM PYTHON SOURCE LINES 130-132 Rational Quadradtic kernel .......................... .. GENERATED FROM PYTHON SOURCE LINES 132-153 .. code-block:: default from sklearn.gaussian_process.kernels import RationalQuadratic kernel = 1.0 * RationalQuadratic(length_scale=1.0, alpha=0.1, alpha_bounds=(1e-5, 1e15)) gpr = GaussianProcessRegressor(kernel=kernel, random_state=0) fig, axs = plt.subplots(nrows=2, sharex=True, sharey=True, figsize=(10, 8)) # plot prior plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[0]) axs[0].set_title("Samples from prior distribution") # plot posterior gpr.fit(X_train, y_train) plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[1]) axs[1].scatter(X_train[:, 0], y_train, color="red", zorder=10, label="Observations") axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left") axs[1].set_title("Samples from posterior distribution") fig.suptitle("Rational Quadratic kernel", fontsize=18) plt.tight_layout() .. image-sg:: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_002.png :alt: Rational Quadratic kernel, Samples from prior distribution, Samples from posterior distribution :srcset: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out Out: .. code-block:: none /home/runner/work/scikit-learn/scikit-learn/sklearn/gaussian_process/_gpr.py:446: UserWarning: Predicted variances smaller than 0. Setting those variances to 0. warnings.warn( /home/runner/work/scikit-learn/scikit-learn/sklearn/gaussian_process/_gpr.py:493: RuntimeWarning: covariance is not positive-semidefinite. y_samples = rng.multivariate_normal(y_mean, y_cov, n_samples).T .. GENERATED FROM PYTHON SOURCE LINES 154-160 .. code-block:: default print(f"Kernel parameters before fit:\n{kernel})") print( f"Kernel parameters after fit: \n{gpr.kernel_} \n" f"Log-likelihood: {gpr.log_marginal_likelihood(gpr.kernel_.theta):.3f}" ) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Kernel parameters before fit: 1**2 * RationalQuadratic(alpha=0.1, length_scale=1)) Kernel parameters after fit: 0.594**2 * RationalQuadratic(alpha=8.66e+09, length_scale=0.279) Log-likelihood: -0.054 .. GENERATED FROM PYTHON SOURCE LINES 161-163 Exp-Sine-Squared kernel ............... .. GENERATED FROM PYTHON SOURCE LINES 163-189 .. code-block:: default from sklearn.gaussian_process.kernels import ExpSineSquared kernel = 1.0 * ExpSineSquared( length_scale=1.0, periodicity=3.0, length_scale_bounds=(0.1, 10.0), periodicity_bounds=(1.0, 10.0), ) gpr = GaussianProcessRegressor(kernel=kernel, random_state=0) fig, axs = plt.subplots(nrows=2, sharex=True, sharey=True, figsize=(10, 8)) # plot prior plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[0]) axs[0].set_title("Samples from prior distribution") # plot posterior gpr.fit(X_train, y_train) plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[1]) axs[1].scatter(X_train[:, 0], y_train, color="red", zorder=10, label="Observations") axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left") axs[1].set_title("Samples from posterior distribution") fig.suptitle("Exp-Sine-Squared kernel", fontsize=18) plt.tight_layout() .. image-sg:: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_003.png :alt: Exp-Sine-Squared kernel, Samples from prior distribution, Samples from posterior distribution :srcset: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 190-196 .. code-block:: default print(f"Kernel parameters before fit:\n{kernel})") print( f"Kernel parameters after fit: \n{gpr.kernel_} \n" f"Log-likelihood: {gpr.log_marginal_likelihood(gpr.kernel_.theta):.3f}" ) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Kernel parameters before fit: 1**2 * ExpSineSquared(length_scale=1, periodicity=3)) Kernel parameters after fit: 0.799**2 * ExpSineSquared(length_scale=0.791, periodicity=2.87) Log-likelihood: 3.394 .. GENERATED FROM PYTHON SOURCE LINES 197-199 Dot-product kernel .................. .. GENERATED FROM PYTHON SOURCE LINES 199-222 .. code-block:: default from sklearn.gaussian_process.kernels import ConstantKernel, DotProduct kernel = ConstantKernel(0.1, (0.01, 10.0)) * ( DotProduct(sigma_0=1.0, sigma_0_bounds=(0.1, 10.0)) ** 2 ) gpr = GaussianProcessRegressor(kernel=kernel, random_state=0) fig, axs = plt.subplots(nrows=2, sharex=True, sharey=True, figsize=(10, 8)) # plot prior plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[0]) axs[0].set_title("Samples from prior distribution") # plot posterior gpr.fit(X_train, y_train) plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[1]) axs[1].scatter(X_train[:, 0], y_train, color="red", zorder=10, label="Observations") axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left") axs[1].set_title("Samples from posterior distribution") fig.suptitle("Dot-product kernel", fontsize=18) plt.tight_layout() .. image-sg:: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_004.png :alt: Dot-product kernel, Samples from prior distribution, Samples from posterior distribution :srcset: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_004.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out Out: .. code-block:: none /home/runner/work/scikit-learn/scikit-learn/sklearn/gaussian_process/_gpr.py:630: ConvergenceWarning: lbfgs failed to converge (status=2): ABNORMAL_TERMINATION_IN_LNSRCH. Increase the number of iterations (max_iter) or scale the data as shown in: https://scikit-learn.org/stable/modules/preprocessing.html _check_optimize_result("lbfgs", opt_res) /home/runner/work/scikit-learn/scikit-learn/sklearn/gaussian_process/_gpr.py:446: UserWarning: Predicted variances smaller than 0. Setting those variances to 0. warnings.warn( .. GENERATED FROM PYTHON SOURCE LINES 223-229 .. code-block:: default print(f"Kernel parameters before fit:\n{kernel})") print( f"Kernel parameters after fit: \n{gpr.kernel_} \n" f"Log-likelihood: {gpr.log_marginal_likelihood(gpr.kernel_.theta):.3f}" ) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Kernel parameters before fit: 0.316**2 * DotProduct(sigma_0=1) ** 2) Kernel parameters after fit: 3**2 * DotProduct(sigma_0=7.8) ** 2 Log-likelihood: -7173415029.706 .. GENERATED FROM PYTHON SOURCE LINES 230-232 Matérn kernel .............. .. GENERATED FROM PYTHON SOURCE LINES 232-253 .. code-block:: default from sklearn.gaussian_process.kernels import Matern kernel = 1.0 * Matern(length_scale=1.0, length_scale_bounds=(1e-1, 10.0), nu=1.5) gpr = GaussianProcessRegressor(kernel=kernel, random_state=0) fig, axs = plt.subplots(nrows=2, sharex=True, sharey=True, figsize=(10, 8)) # plot prior plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[0]) axs[0].set_title("Samples from prior distribution") # plot posterior gpr.fit(X_train, y_train) plot_gpr_samples(gpr, n_samples=n_samples, ax=axs[1]) axs[1].scatter(X_train[:, 0], y_train, color="red", zorder=10, label="Observations") axs[1].legend(bbox_to_anchor=(1.05, 1.5), loc="upper left") axs[1].set_title("Samples from posterior distribution") fig.suptitle("Matérn kernel", fontsize=18) plt.tight_layout() .. image-sg:: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_005.png :alt: Matérn kernel, Samples from prior distribution, Samples from posterior distribution :srcset: /auto_examples/gaussian_process/images/sphx_glr_plot_gpr_prior_posterior_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 254-259 .. code-block:: default print(f"Kernel parameters before fit:\n{kernel})") print( f"Kernel parameters after fit: \n{gpr.kernel_} \n" f"Log-likelihood: {gpr.log_marginal_likelihood(gpr.kernel_.theta):.3f}" ) .. rst-class:: sphx-glr-script-out Out: .. code-block:: none Kernel parameters before fit: 1**2 * Matern(length_scale=1, nu=1.5)) Kernel parameters after fit: 0.609**2 * Matern(length_scale=0.484, nu=1.5) Log-likelihood: -1.185 .. rst-class:: sphx-glr-timing **Total running time of the script:** ( 0 minutes 1.049 seconds) .. _sphx_glr_download_auto_examples_gaussian_process_plot_gpr_prior_posterior.py: .. only :: html .. container:: sphx-glr-footer :class: sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/main?urlpath=lab/tree/notebooks/auto_examples/gaussian_process/plot_gpr_prior_posterior.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-python :download:Download Python source code: plot_gpr_prior_posterior.py  .. container:: sphx-glr-download sphx-glr-download-jupyter :download:Download Jupyter notebook: plot_gpr_prior_posterior.ipynb  .. only:: html .. rst-class:: sphx-glr-signature Gallery generated by Sphinx-Gallery _