.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/linear_model/plot_lasso_lars_ic.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_auto_examples_linear_model_plot_lasso_lars_ic.py>`
        to download the full example code or to run this example in your browser via Binder

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_linear_model_plot_lasso_lars_ic.py:


==============================================
Lasso model selection via information criteria
==============================================

This example reproduces the example of Fig. 2 of [ZHT2007]_. A
:class:`~sklearn.linear_model.LassoLarsIC` estimator is fit on a
diabetes dataset and the AIC and the BIC criteria are used to select
the best model.

.. note::
    It is important to note that the optimization to find `alpha` with
    :class:`~sklearn.linear_model.LassoLarsIC` relies on the AIC or BIC
    criteria that are computed in-sample, thus on the training set directly.
    This approach differs from the cross-validation procedure. For a comparison
    of the two approaches, you can refer to the following example:
    :ref:`sphx_glr_auto_examples_linear_model_plot_lasso_model_selection.py`.

.. topic:: References

    .. [ZHT2007] :arxiv:`Zou, Hui, Trevor Hastie, and Robert Tibshirani.
       "On the degrees of freedom of the lasso."
       The Annals of Statistics 35.5 (2007): 2173-2192.
       <0712.0881>`

.. GENERATED FROM PYTHON SOURCE LINES 26-31

.. code-block:: default


    # Author: Alexandre Gramfort
    #         Guillaume Lemaitre
    # License: BSD 3 clause








.. GENERATED FROM PYTHON SOURCE LINES 32-36

.. code-block:: default

    import sklearn

    sklearn.set_config(display="diagram")








.. GENERATED FROM PYTHON SOURCE LINES 37-38

We will use the diabetes dataset.

.. GENERATED FROM PYTHON SOURCE LINES 38-44

.. code-block:: default

    from sklearn.datasets import load_diabetes

    X, y = load_diabetes(return_X_y=True, as_frame=True)
    n_samples = X.shape[0]
    X.head()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>age</th>
          <th>sex</th>
          <th>bmi</th>
          <th>bp</th>
          <th>s1</th>
          <th>s2</th>
          <th>s3</th>
          <th>s4</th>
          <th>s5</th>
          <th>s6</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>0.038076</td>
          <td>0.050680</td>
          <td>0.061696</td>
          <td>0.021872</td>
          <td>-0.044223</td>
          <td>-0.034821</td>
          <td>-0.043401</td>
          <td>-0.002592</td>
          <td>0.019908</td>
          <td>-0.017646</td>
        </tr>
        <tr>
          <th>1</th>
          <td>-0.001882</td>
          <td>-0.044642</td>
          <td>-0.051474</td>
          <td>-0.026328</td>
          <td>-0.008449</td>
          <td>-0.019163</td>
          <td>0.074412</td>
          <td>-0.039493</td>
          <td>-0.068330</td>
          <td>-0.092204</td>
        </tr>
        <tr>
          <th>2</th>
          <td>0.085299</td>
          <td>0.050680</td>
          <td>0.044451</td>
          <td>-0.005671</td>
          <td>-0.045599</td>
          <td>-0.034194</td>
          <td>-0.032356</td>
          <td>-0.002592</td>
          <td>0.002864</td>
          <td>-0.025930</td>
        </tr>
        <tr>
          <th>3</th>
          <td>-0.089063</td>
          <td>-0.044642</td>
          <td>-0.011595</td>
          <td>-0.036656</td>
          <td>0.012191</td>
          <td>0.024991</td>
          <td>-0.036038</td>
          <td>0.034309</td>
          <td>0.022692</td>
          <td>-0.009362</td>
        </tr>
        <tr>
          <th>4</th>
          <td>0.005383</td>
          <td>-0.044642</td>
          <td>-0.036385</td>
          <td>0.021872</td>
          <td>0.003935</td>
          <td>0.015596</td>
          <td>0.008142</td>
          <td>-0.002592</td>
          <td>-0.031991</td>
          <td>-0.046641</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 45-53

Scikit-learn provides an estimator called
:class:`~sklearn.linear_model.LinearLarsIC` that uses either Akaike's
information criterion (AIC) or the Bayesian information criterion (BIC) to
select the best model. Before fitting
this model, we will scale the dataset.

In the following, we are going to fit two models to compare the values
reported by AIC and BIC.

.. GENERATED FROM PYTHON SOURCE LINES 53-62

.. code-block:: default

    from sklearn.preprocessing import StandardScaler
    from sklearn.linear_model import LassoLarsIC
    from sklearn.pipeline import make_pipeline

    lasso_lars_ic = make_pipeline(
        StandardScaler(), LassoLarsIC(criterion="aic", normalize=False)
    ).fit(X, y)









.. GENERATED FROM PYTHON SOURCE LINES 63-68

To be in line with the defintion in [ZHT2007]_, we need to rescale the
AIC and the BIC. Indeed, Zou et al. are ignoring some constant terms
compared to the original definition of AIC derived from the maximum
log-likelihood of a linear model. You can refer to
:ref:`mathematical detail section for the User Guide <lasso_lars_ic>`.

.. GENERATED FROM PYTHON SOURCE LINES 68-73

.. code-block:: default

    def zou_et_al_criterion_rescaling(criterion, n_samples, noise_variance):
        """Rescale the information criterion to follow the definition of Zou et al."""
        return criterion - n_samples * np.log(2 * np.pi * noise_variance) - n_samples









.. GENERATED FROM PYTHON SOURCE LINES 74-86

.. code-block:: default

    import numpy as np

    aic_criterion = zou_et_al_criterion_rescaling(
        lasso_lars_ic[-1].criterion_,
        n_samples,
        lasso_lars_ic[-1].noise_variance_,
    )

    index_alpha_path_aic = np.flatnonzero(
        lasso_lars_ic[-1].alphas_ == lasso_lars_ic[-1].alpha_
    )[0]








.. GENERATED FROM PYTHON SOURCE LINES 87-99

.. code-block:: default

    lasso_lars_ic.set_params(lassolarsic__criterion="bic").fit(X, y)

    bic_criterion = zou_et_al_criterion_rescaling(
        lasso_lars_ic[-1].criterion_,
        n_samples,
        lasso_lars_ic[-1].noise_variance_,
    )

    index_alpha_path_bic = np.flatnonzero(
        lasso_lars_ic[-1].alphas_ == lasso_lars_ic[-1].alpha_
    )[0]








.. GENERATED FROM PYTHON SOURCE LINES 100-103

Now that we collected the AIC and BIC, we can as well check that the minima
of both criteria happen at the same alpha. Then, we can simplify the
following plot.

.. GENERATED FROM PYTHON SOURCE LINES 103-105

.. code-block:: default

    index_alpha_path_aic == index_alpha_path_bic





.. rst-class:: sphx-glr-script-out

 Out:

 .. code-block:: none


    True



.. GENERATED FROM PYTHON SOURCE LINES 106-108

Finally, we can plot the AIC and BIC criterion and the subsequent selected
regularization parameter.

.. GENERATED FROM PYTHON SOURCE LINES 108-124

.. code-block:: default

    import matplotlib.pyplot as plt

    plt.plot(aic_criterion, color="tab:blue", marker="o", label="AIC criterion")
    plt.plot(bic_criterion, color="tab:orange", marker="o", label="BIC criterion")
    plt.vlines(
        index_alpha_path_bic,
        aic_criterion.min(),
        aic_criterion.max(),
        color="black",
        linestyle="--",
        label="Selected alpha",
    )
    plt.legend()
    plt.ylabel("Information criterion")
    plt.xlabel("Lasso model sequence")
    _ = plt.title("Lasso model selection via AIC and BIC")



.. image-sg:: /auto_examples/linear_model/images/sphx_glr_plot_lasso_lars_ic_001.png
   :alt: Lasso model selection via AIC and BIC
   :srcset: /auto_examples/linear_model/images/sphx_glr_plot_lasso_lars_ic_001.png
   :class: sphx-glr-single-img






.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 0 minutes  0.089 seconds)


.. _sphx_glr_download_auto_examples_linear_model_plot_lasso_lars_ic.py:


.. only :: html

 .. container:: sphx-glr-footer
    :class: sphx-glr-footer-example


  .. container:: binder-badge

    .. image:: images/binder_badge_logo.svg
      :target: https://mybinder.org/v2/gh/scikit-learn/scikit-learn/1.0.X?urlpath=lab/tree/notebooks/auto_examples/linear_model/plot_lasso_lars_ic.ipynb
      :alt: Launch binder
      :width: 150 px


  .. container:: sphx-glr-download sphx-glr-download-python

     :download:`Download Python source code: plot_lasso_lars_ic.py <plot_lasso_lars_ic.py>`



  .. container:: sphx-glr-download sphx-glr-download-jupyter

     :download:`Download Jupyter notebook: plot_lasso_lars_ic.ipynb <plot_lasso_lars_ic.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_