.. _minimal_reproducer: ============================================== Crafting a minimal reproducer for scikit-learn ============================================== Whether submitting a bug report, designing a suite of tests, or simply posting a question in the discussions, being able to craft minimal, reproducible examples (or minimal, workable examples) is the key to communicating effectively and efficiently with the community. There are very good guidelines on the internet such as `this StackOverflow document `_ or `this blogpost by Matthew Rocklin `_ on crafting Minimal Complete Verifiable Examples (referred below as MCVE). Our goal is not to be repetitive with those references but rather to provide a step-by-step guide on how to narrow down a bug until you have reached the shortest possible code to reproduce it. The first step before submitting a bug report to scikit-learn is to read the `Issue template `_. It is already quite informative about the information you will be asked to provide. .. _good_practices: Good practices ============== In this section we will focus on the **Steps/Code to Reproduce** section of the `Issue template `_. We will start with a snippet of code that already provides a failing example but that has room for readability improvement. We then craft a MCVE from it. **Example** .. code-block:: python # I am currently working in a ML project and when I tried to fit a # GradientBoostingRegressor instance to my_data.csv I get a UserWarning: # "X has feature names, but DecisionTreeRegressor was fitted without # feature names". You can get a copy of my dataset from # https://example.com/my_data.csv and verify my features do have # names. The problem seems to arise during fit when I pass an integer # to the n_iter_no_change parameter. df = pd.read_csv('my_data.csv') X = df[["feature_name"]] # my features do have names y = df["target"] # We set random_state=42 for the train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.33, random_state=42 ) scaler = StandardScaler(with_mean=False) X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) # An instance with default n_iter_no_change raises no error nor warnings gbdt = GradientBoostingRegressor(random_state=0) gbdt.fit(X_train, y_train) default_score = gbdt.score(X_test, y_test) # the bug appears when I change the value for n_iter_no_change gbdt = GradientBoostingRegressor(random_state=0, n_iter_no_change=5) gbdt.fit(X_train, y_train) other_score = gbdt.score(X_test, y_test) other_score = gbdt.score(X_test, y_test) Provide a failing code example with minimal comments ---------------------------------------------------- Writing instructions to reproduce the problem in English is often ambiguous. Better make sure that all the necessary details to reproduce the problem are illustrated in the Python code snippet to avoid any ambiguity. Besides, by this point you already provided a concise description in the **Describe the bug** section of the `Issue template `_. The following code, while **still not minimal**, is already **much better** because it can be copy-pasted in a Python terminal to reproduce the problem in one step. In particular: - it contains **all necessary imports statements**; - it can fetch the public dataset without having to manually download a file and put it in the expected location on the disk. **Improved example** .. code-block:: python import pandas as pd df = pd.read_csv("https://example.com/my_data.csv") X = df[["feature_name"]] y = df["target"] from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.33, random_state=42 ) from sklearn.preprocessing import StandardScaler scaler = StandardScaler(with_mean=False) X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test) from sklearn.ensemble import GradientBoostingRegressor gbdt = GradientBoostingRegressor(random_state=0) gbdt.fit(X_train, y_train) # no warning default_score = gbdt.score(X_test, y_test) gbdt = GradientBoostingRegressor(random_state=0, n_iter_no_change=5) gbdt.fit(X_train, y_train) # raises warning other_score = gbdt.score(X_test, y_test) other_score = gbdt.score(X_test, y_test) Boil down your script to something as small as possible ------------------------------------------------------- You have to ask yourself which lines of code are relevant and which are not for reproducing the bug. Deleting unnecessary lines of code or simplifying the function calls by omitting unrelated non-default options will help you and other contributors narrow down the cause of the bug. In particular, for this specific example: - the warning has nothing to do with the `train_test_split` since it already appears in the training step, before we use the test set. - similarly, the lines that compute the scores on the test set are not necessary; - the bug can be reproduced for any value of `random_state` so leave it to its default; - the bug can be reproduced without preprocessing the data with the `StandardScaler`. **Improved example** .. code-block:: python import pandas as pd df = pd.read_csv("https://example.com/my_data.csv") X = df[["feature_name"]] y = df["target"] from sklearn.ensemble import GradientBoostingRegressor gbdt = GradientBoostingRegressor() gbdt.fit(X, y) # no warning gbdt = GradientBoostingRegressor(n_iter_no_change=5) gbdt.fit(X, y) # raises warning **DO NOT** report your data unless it is extremely necessary ------------------------------------------------------------ The idea is to make the code as self-contained as possible. For doing so, you can use a :ref:`synth_data`. It can be generated using numpy, pandas or the :mod:`sklearn.datasets` module. Most of the times the bug is not related to a particular structure of your data. Even if it is, try to find an available dataset that has similar characteristics to yours and that reproduces the problem. In this particular case, we are interested in data that has labeled feature names. **Improved example** .. code-block:: python import pandas as pd from sklearn.ensemble import GradientBoostingRegressor df = pd.DataFrame( { "feature_name": [-12.32, 1.43, 30.01, 22.17], "target": [72, 55, 32, 43], } ) X = df[["feature_name"]] y = df["target"] gbdt = GradientBoostingRegressor() gbdt.fit(X, y) # no warning gbdt = GradientBoostingRegressor(n_iter_no_change=5) gbdt.fit(X, y) # raises warning As already mentioned, the key to communication is the readability of the code and good formatting can really be a plus. Notice that in the previous snippet we: - try to limit all lines to a maximum of 79 characters to avoid horizontal scrollbars in the code snippets blocks rendered on the GitHub issue; - use blank lines to separate groups of related functions; - place all the imports in their own group at the beginning. The simplification steps presented in this guide can be implemented in a different order than the progression we have shown here. The important points are: - a minimal reproducer should be runnable by a simple copy-and-paste in a python terminal; - it should be simplified as much as possible by removing any code steps that are not strictly needed to reproducing the original problem; - it should ideally only rely on a minimal dataset generated on-the-fly by running the code instead of relying on external data, if possible. Use markdown formatting ----------------------- To format code or text into its own distinct block, use triple backticks. `Markdown `_ supports an optional language identifier to enable syntax highlighting in your fenced code block. For example:: ```python from sklearn.datasets import make_blobs n_samples = 100 n_components = 3 X, y = make_blobs(n_samples=n_samples, centers=n_components) ``` will render a python formatted snippet as follows .. code-block:: python from sklearn.datasets import make_blobs n_samples = 100 n_components = 3 X, y = make_blobs(n_samples=n_samples, centers=n_components) It is not necessary to create several blocks of code when submitting a bug report. Remember other reviewers are going to copy-paste your code and having a single cell will make their task easier. In the section named **Actual results** of the `Issue template `_ you are asked to provide the error message including the full traceback of the exception. In this case, use the `python-traceback` qualifier. For example:: ```python-traceback --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in 4 vectorizer = CountVectorizer(input=docs, analyzer='word') 5 lda_features = vectorizer.fit_transform(docs) ----> 6 lda_model = LatentDirichletAllocation( 7 n_topics=10, 8 learning_method='online', TypeError: __init__() got an unexpected keyword argument 'n_topics' ``` yields the following when rendered: .. code-block:: python --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in 4 vectorizer = CountVectorizer(input=docs, analyzer='word') 5 lda_features = vectorizer.fit_transform(docs) ----> 6 lda_model = LatentDirichletAllocation( 7 n_topics=10, 8 learning_method='online', TypeError: __init__() got an unexpected keyword argument 'n_topics' .. _synth_data: Synthetic dataset ================= Before choosing a particular synthetic dataset, first you have to identify the type of problem you are solving: Is it a classification, a regression, a clustering, etc? Once that you narrowed down the type of problem, you need to provide a synthetic dataset accordingly. Most of the times you only need a minimalistic dataset. Here is a non-exhaustive list of tools that may help you. NumPy ----- NumPy tools such as `numpy.random.randn `_ and `numpy.random.randint `_ can be used to create dummy numeric data. - regression Regressions take continuous numeric data as features and target. .. code-block:: python import numpy as np rng = np.random.RandomState(0) n_samples, n_features = 5, 5 X = rng.randn(n_samples, n_features) y = rng.randn(n_samples) A similar snippet can be used as synthetic data when testing scaling tools such as :class:`sklearn.preprocessing.StandardScaler`. - classification If the bug is not raised during when encoding a categorical variable, you can feed numeric data to a classifier. Just remember to ensure that the target is indeed an integer. .. code-block:: python import numpy as np rng = np.random.RandomState(0) n_samples, n_features = 5, 5 X = rng.randn(n_samples, n_features) y = rng.randint(0, 2, n_samples) # binary target with values in {0, 1} If the bug only happens with non-numeric class labels, you might want to generate a random target with `numpy.random.choice `_. .. code-block:: python import numpy as np rng = np.random.RandomState(0) n_samples, n_features = 50, 5 X = rng.randn(n_samples, n_features) y = np.random.choice( ["male", "female", "other"], size=n_samples, p=[0.49, 0.49, 0.02] ) Pandas ------ Some scikit-learn objects expect pandas dataframes as input. In this case you can transform numpy arrays into pandas objects using `pandas.DataFrame `_, or `pandas.Series `_. .. code-block:: python import numpy as np import pandas as pd rng = np.random.RandomState(0) n_samples, n_features = 5, 5 X = pd.DataFrame( { "continuous_feature": rng.randn(n_samples), "positive_feature": rng.uniform(low=0.0, high=100.0, size=n_samples), "categorical_feature": rng.choice(["a", "b", "c"], size=n_samples), } ) y = pd.Series(rng.randn(n_samples)) In addition, scikit-learn includes various :ref:`sample_generators` that can be used to build artificial datasets of controlled size and complexity. `make_regression` ----------------- As hinted by the name, :class:`sklearn.datasets.make_regression` produces regression targets with noise as an optionally-sparse random linear combination of random features. .. code-block:: python from sklearn.datasets import make_regression X, y = make_regression(n_samples=1000, n_features=20) `make_classification` --------------------- :class:`sklearn.datasets.make_classification` creates multiclass datasets with multiple Gaussian clusters per class. Noise can be introduced by means of correlated, redundant or uninformative features. .. code-block:: python from sklearn.datasets import make_classification X, y = make_classification( n_features=2, n_redundant=0, n_informative=2, n_clusters_per_class=1 ) `make_blobs` ------------ Similarly to `make_classification`, :class:`sklearn.datasets.make_blobs` creates multiclass datasets using normally-distributed clusters of points. It provides greater control regarding the centers and standard deviations of each cluster, and therefore it is useful to demonstrate clustering. .. code-block:: python from sklearn.datasets import make_blobs X, y = make_blobs(n_samples=10, centers=3, n_features=2) Dataset loading utilities ------------------------- You can use the :ref:`datasets` to load and fetch several popular reference datasets. This option is useful when the bug relates to the particular structure of the data, e.g. dealing with missing values or image recognition. .. code-block:: python from sklearn.datasets import load_breast_cancer X, y = load_breast_cancer(return_X_y=True)