.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/decomposition/plot_faces_decomposition.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. or to run this example in your browser via JupyterLite or Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_decomposition_plot_faces_decomposition.py: ============================ Faces dataset decompositions ============================ This example applies to :ref:`olivetti_faces_dataset` different unsupervised matrix decomposition (dimension reduction) methods from the module :mod:`sklearn.decomposition` (see the documentation chapter :ref:`decompositions`). - Authors: Vlad Niculae, Alexandre Gramfort - License: BSD 3 clause .. GENERATED FROM PYTHON SOURCE LINES 15-19 .. code-block:: Python # Authors: The scikit-learn developers # SPDX-License-Identifier: BSD-3-Clause .. GENERATED FROM PYTHON SOURCE LINES 20-24 Dataset preparation ------------------- Loading and preprocessing the Olivetti faces dataset. .. GENERATED FROM PYTHON SOURCE LINES 24-49 .. code-block:: Python import logging import matplotlib.pyplot as plt from numpy.random import RandomState from sklearn import cluster, decomposition from sklearn.datasets import fetch_olivetti_faces rng = RandomState(0) # Display progress logs on stdout logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s") faces, _ = fetch_olivetti_faces(return_X_y=True, shuffle=True, random_state=rng) n_samples, n_features = faces.shape # Global centering (focus on one feature, centering all samples) faces_centered = faces - faces.mean(axis=0) # Local centering (focus on one sample, centering all features) faces_centered -= faces_centered.mean(axis=1).reshape(n_samples, -1) print("Dataset consists of %d faces" % n_samples) .. rst-class:: sphx-glr-script-out .. code-block:: none Dataset consists of 400 faces .. GENERATED FROM PYTHON SOURCE LINES 50-51 Define a base function to plot the gallery of faces. .. GENERATED FROM PYTHON SOURCE LINES 51-83 .. code-block:: Python n_row, n_col = 2, 3 n_components = n_row * n_col image_shape = (64, 64) def plot_gallery(title, images, n_col=n_col, n_row=n_row, cmap=plt.cm.gray): fig, axs = plt.subplots( nrows=n_row, ncols=n_col, figsize=(2.0 * n_col, 2.3 * n_row), facecolor="white", constrained_layout=True, ) fig.set_constrained_layout_pads(w_pad=0.01, h_pad=0.02, hspace=0, wspace=0) fig.set_edgecolor("black") fig.suptitle(title, size=16) for ax, vec in zip(axs.flat, images): vmax = max(vec.max(), -vec.min()) im = ax.imshow( vec.reshape(image_shape), cmap=cmap, interpolation="nearest", vmin=-vmax, vmax=vmax, ) ax.axis("off") fig.colorbar(im, ax=axs, orientation="horizontal", shrink=0.99, aspect=40, pad=0.01) plt.show() .. GENERATED FROM PYTHON SOURCE LINES 84-86 Let's take a look at our data. Gray color indicates negative values, white indicates positive values. .. GENERATED FROM PYTHON SOURCE LINES 86-89 .. code-block:: Python plot_gallery("Faces from dataset", faces_centered[:n_components]) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_001.png :alt: Faces from dataset :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 90-99 Decomposition ------------- Initialise different estimators for decomposition and fit each of them on all images and plot some results. Each estimator extracts 6 components as vectors :math:`h \in \mathbb{R}^{4096}`. We just displayed these vectors in human-friendly visualisation as 64x64 pixel images. Read more in the :ref:`User Guide `. .. GENERATED FROM PYTHON SOURCE LINES 101-112 Eigenfaces - PCA using randomized SVD ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Linear dimensionality reduction using Singular Value Decomposition (SVD) of the data to project it to a lower dimensional space. .. note:: The Eigenfaces estimator, via the :py:mod:`sklearn.decomposition.PCA`, also provides a scalar `noise_variance_` (the mean of pixelwise variance) that cannot be displayed as an image. .. GENERATED FROM PYTHON SOURCE LINES 114-122 .. code-block:: Python pca_estimator = decomposition.PCA( n_components=n_components, svd_solver="randomized", whiten=True ) pca_estimator.fit(faces_centered) plot_gallery( "Eigenfaces - PCA using randomized SVD", pca_estimator.components_[:n_components] ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_002.png :alt: Eigenfaces - PCA using randomized SVD :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_002.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 123-127 Non-negative components - NMF ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Estimate non-negative original data as production of two non-negative matrices. .. GENERATED FROM PYTHON SOURCE LINES 129-133 .. code-block:: Python nmf_estimator = decomposition.NMF(n_components=n_components, tol=5e-3) nmf_estimator.fit(faces) # original non- negative dataset plot_gallery("Non-negative components - NMF", nmf_estimator.components_[:n_components]) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_003.png :alt: Non-negative components - NMF :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_003.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 134-138 Independent components - FastICA ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Independent component analysis separates a multivariate vectors into additive subcomponents that are maximally independent. .. GENERATED FROM PYTHON SOURCE LINES 140-148 .. code-block:: Python ica_estimator = decomposition.FastICA( n_components=n_components, max_iter=400, whiten="arbitrary-variance", tol=15e-5 ) ica_estimator.fit(faces_centered) plot_gallery( "Independent components - FastICA", ica_estimator.components_[:n_components] ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_004.png :alt: Independent components - FastICA :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_004.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 149-156 Sparse components - MiniBatchSparsePCA ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Mini-batch sparse PCA (:class:`~sklearn.decomposition.MiniBatchSparsePCA`) extracts the set of sparse components that best reconstruct the data. This variant is faster but less accurate than the similar :class:`~sklearn.decomposition.SparsePCA`. .. GENERATED FROM PYTHON SOURCE LINES 158-167 .. code-block:: Python batch_pca_estimator = decomposition.MiniBatchSparsePCA( n_components=n_components, alpha=0.1, max_iter=100, batch_size=3, random_state=rng ) batch_pca_estimator.fit(faces_centered) plot_gallery( "Sparse components - MiniBatchSparsePCA", batch_pca_estimator.components_[:n_components], ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_005.png :alt: Sparse components - MiniBatchSparsePCA :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_005.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 168-174 Dictionary learning ^^^^^^^^^^^^^^^^^^^ By default, :class:`~sklearn.decomposition.MiniBatchDictionaryLearning` divides the data into mini-batches and optimizes in an online manner by cycling over the mini-batches for the specified number of iterations. .. GENERATED FROM PYTHON SOURCE LINES 176-182 .. code-block:: Python batch_dict_estimator = decomposition.MiniBatchDictionaryLearning( n_components=n_components, alpha=0.1, max_iter=50, batch_size=3, random_state=rng ) batch_dict_estimator.fit(faces_centered) plot_gallery("Dictionary learning", batch_dict_estimator.components_[:n_components]) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_006.png :alt: Dictionary learning :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_006.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 183-191 Cluster centers - MiniBatchKMeans ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :class:`sklearn.cluster.MiniBatchKMeans` is computationally efficient and implements on-line learning with a :meth:`~sklearn.cluster.MiniBatchKMeans.partial_fit` method. That is why it could be beneficial to enhance some time-consuming algorithms with :class:`~sklearn.cluster.MiniBatchKMeans`. .. GENERATED FROM PYTHON SOURCE LINES 193-207 .. code-block:: Python kmeans_estimator = cluster.MiniBatchKMeans( n_clusters=n_components, tol=1e-3, batch_size=20, max_iter=50, random_state=rng, ) kmeans_estimator.fit(faces_centered) plot_gallery( "Cluster centers - MiniBatchKMeans", kmeans_estimator.cluster_centers_[:n_components], ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_007.png :alt: Cluster centers - MiniBatchKMeans :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_007.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 208-215 Factor Analysis components - FA ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :class:`~sklearn.decomposition.FactorAnalysis` is similar to :class:`~sklearn.decomposition.PCA` but has the advantage of modelling the variance in every direction of the input space independently (heteroscedastic noise). Read more in the :ref:`User Guide `. .. GENERATED FROM PYTHON SOURCE LINES 217-237 .. code-block:: Python fa_estimator = decomposition.FactorAnalysis(n_components=n_components, max_iter=20) fa_estimator.fit(faces_centered) plot_gallery("Factor Analysis (FA)", fa_estimator.components_[:n_components]) # --- Pixelwise variance plt.figure(figsize=(3.2, 3.6), facecolor="white", tight_layout=True) vec = fa_estimator.noise_variance_ vmax = max(vec.max(), -vec.min()) plt.imshow( vec.reshape(image_shape), cmap=plt.cm.gray, interpolation="nearest", vmin=-vmax, vmax=vmax, ) plt.axis("off") plt.title("Pixelwise variance from \n Factor Analysis (FA)", size=16, wrap=True) plt.colorbar(orientation="horizontal", shrink=0.8, pad=0.03) plt.show() .. rst-class:: sphx-glr-horizontal * .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_008.png :alt: Factor Analysis (FA) :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_008.png :class: sphx-glr-multi-img * .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_009.png :alt: Pixelwise variance from Factor Analysis (FA) :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_009.png :class: sphx-glr-multi-img .. GENERATED FROM PYTHON SOURCE LINES 238-251 Decomposition: Dictionary learning ---------------------------------- In the further section, let's consider :ref:`DictionaryLearning` more precisely. Dictionary learning is a problem that amounts to finding a sparse representation of the input data as a combination of simple elements. These simple elements form a dictionary. It is possible to constrain the dictionary and/or coding coefficients to be positive to match constraints that may be present in the data. :class:`~sklearn.decomposition.MiniBatchDictionaryLearning` implements a faster, but less accurate version of the dictionary learning algorithm that is better suited for large datasets. Read more in the :ref:`User Guide `. .. GENERATED FROM PYTHON SOURCE LINES 253-256 Plot the same samples from our dataset but with another colormap. Red indicates negative values, blue indicates positive values, and white represents zeros. .. GENERATED FROM PYTHON SOURCE LINES 256-259 .. code-block:: Python plot_gallery("Faces from dataset", faces_centered[:n_components], cmap=plt.cm.RdBu) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_010.png :alt: Faces from dataset :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_010.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 260-270 Similar to the previous examples, we change parameters and train :class:`~sklearn.decomposition.MiniBatchDictionaryLearning` estimator on all images. Generally, the dictionary learning and sparse encoding decompose input data into the dictionary and the coding coefficients matrices. :math:`X \approx UV`, where :math:`X = [x_1, . . . , x_n]`, :math:`X \in \mathbb{R}^{m×n}`, dictionary :math:`U \in \mathbb{R}^{m×k}`, coding coefficients :math:`V \in \mathbb{R}^{k×n}`. Also below are the results when the dictionary and coding coefficients are positively constrained. .. GENERATED FROM PYTHON SOURCE LINES 272-276 Dictionary learning - positive dictionary ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In the following section we enforce positivity when finding the dictionary. .. GENERATED FROM PYTHON SOURCE LINES 278-293 .. code-block:: Python dict_pos_dict_estimator = decomposition.MiniBatchDictionaryLearning( n_components=n_components, alpha=0.1, max_iter=50, batch_size=3, random_state=rng, positive_dict=True, ) dict_pos_dict_estimator.fit(faces_centered) plot_gallery( "Dictionary learning - positive dictionary", dict_pos_dict_estimator.components_[:n_components], cmap=plt.cm.RdBu, ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_011.png :alt: Dictionary learning - positive dictionary :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_011.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 294-298 Dictionary learning - positive code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Below we constrain the coding coefficients as a positive matrix. .. GENERATED FROM PYTHON SOURCE LINES 300-316 .. code-block:: Python dict_pos_code_estimator = decomposition.MiniBatchDictionaryLearning( n_components=n_components, alpha=0.1, max_iter=50, batch_size=3, fit_algorithm="cd", random_state=rng, positive_code=True, ) dict_pos_code_estimator.fit(faces_centered) plot_gallery( "Dictionary learning - positive code", dict_pos_code_estimator.components_[:n_components], cmap=plt.cm.RdBu, ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_012.png :alt: Dictionary learning - positive code :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_012.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 317-322 Dictionary learning - positive dictionary & code ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Also below are the results if the dictionary values and coding coefficients are positively constrained. .. GENERATED FROM PYTHON SOURCE LINES 324-340 .. code-block:: Python dict_pos_estimator = decomposition.MiniBatchDictionaryLearning( n_components=n_components, alpha=0.1, max_iter=50, batch_size=3, fit_algorithm="cd", random_state=rng, positive_dict=True, positive_code=True, ) dict_pos_estimator.fit(faces_centered) plot_gallery( "Dictionary learning - positive dictionary & code", dict_pos_estimator.components_[:n_components], cmap=plt.cm.RdBu, ) .. image-sg:: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_013.png :alt: Dictionary learning - positive dictionary & code :srcset: /auto_examples/decomposition/images/sphx_glr_plot_faces_decomposition_013.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 7.801 seconds) .. _sphx_glr_download_auto_examples_decomposition_plot_faces_decomposition.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/1.6.X?urlpath=lab/tree/notebooks/auto_examples/decomposition/plot_faces_decomposition.ipynb :alt: Launch binder :width: 150 px .. container:: lite-badge .. image:: images/jupyterlite_badge_logo.svg :target: ../../lite/lab/index.html?path=auto_examples/decomposition/plot_faces_decomposition.ipynb :alt: Launch JupyterLite :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_faces_decomposition.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_faces_decomposition.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_faces_decomposition.zip ` .. include:: plot_faces_decomposition.recommendations .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_