Path: blob/master/section-3-structured-data-projects/end-to-end-bluebook-bulldozer-price-regression-v2.ipynb
874 views
Predicting the Sale Price of Bulldozers using Machine Learning 🚜
In this notebook, we're going to go through an example machine learning project to use the characteristics of bulldozers and their past sales prices to predict the sale price of future bulldozers based on their characteristics.
Inputs: Bulldozer characteristics such as make year, base model, model series, state of sale (e.g. which US state was it sold in), drive system and more.
Outputs: Bulldozer sale price (in USD).
Since we're trying to predict a number, this kind of problem is known as a regression problem.
And since we're going to predicting results with a time component (predicting future sales based on past sales), this is also known as a time series or forecasting problem.
The data and evaluation metric we'll be using (root mean square log error or RMSLE) is from the Kaggle Bluebook for Bulldozers competition.
The techniques used in here have been inspired and adapted from the fast.ai machine learning course.
Overview
Since we already have a dataset, we'll approach the problem with the following machine learning modelling framework.
| 
| |:--😐 | 6 Step Machine Learning Modelling Framework (read more) |
To work through these topics, we'll use pandas, Matplotlib and NumPy for data analysis, as well as, Scikit-Learn for machine learning and modelling tasks.
| | |:--😐 | Tools that can be used for each step of the machine learning modelling process. |
We'll work through each step and by the end of the notebook, we'll have a trained machine learning model which predicts the sale price of a bulldozer given different characteristics about it.
6 Step Machine Learning Framework
1. Problem Definition
For this dataset, the problem we're trying to solve, or better, the question we're trying to answer is,
How well can we predict the future sale price of a bulldozer, given its characteristics previous examples of how much similar bulldozers have been sold for?
2. Data
Looking at the dataset from Kaggle we see that it contains historical sales data of bulldozers. Including things like, model type, size, sale date and more.
There are 3 datasets:
Train.csv - Historical bulldozer sales examples up to 2011 (close to 400,000 examples with 50+ different attributes, including
SalePricewhich is the target variable).Valid.csv - Historical bulldozer sales examples from January 1 2012 to April 30 2012 (close to 12,000 examples with the same attributes as Train.csv).
Test.csv - Historical bulldozer sales examples from May 1 2012 to November 2012 (close to 12,000 examples but missing the
SalePriceattribute, as this is what we'll be trying to predict).
Note: You can download the dataset
bluebook-for-bulldozersdataset directly from Kaggle. Alternatively, you can also download it directly from the course GitHub.
3. Evaluation
For this problem, Kaggle has set the evaluation metric to being root mean squared log error (RMSLE). As with many regression evaluations, the goal will be to get this value as low as possible (a low error value means our model's predictions are close to what the real values are).
To see how well our model is doing, we'll calculate the RMSLE and then compare our results to others on the Kaggle leaderboard.
4. Features
Features are different parts and attributes of the data.
During this step, you'll want to start finding out what you can about the data.
One of the most common ways to do this is to create a data dictionary.
For this dataset, Kaggle provides a data dictionary which contains information about what each attribute of the dataset means.
For example:
| Variable Name | Description | Variable Type |
|---|---|---|
| SalesID | unique identifier of a particular sale of a machine at auction | Independent variable |
| MachineID | identifier for a particular machine; machines may have multiple sales | Independent variable |
| ModelID | identifier for a unique machine model (i.e. fiModelDesc) | Independent variable |
| datasource | source of the sale record; some sources are more diligent about reporting attributes of the machine than others. Note that a particular datasource may report on multiple auctioneerIDs. | Independent variable |
| auctioneerID | identifier of a particular auctioneer, i.e. company that sold the machine at auction. Not the same as datasource. | Independent variable |
| YearMade | year of manufacturer of the Machine | Independent variable |
| MachineHoursCurrentMeter | current usage of the machine in hours at time of sale (saledate); null or 0 means no hours have been reported for that sale | Independent variable |
| UsageBand | value (low, medium, high) calculated comparing this particular Machine-Sale hours to average usage for the fiBaseModel; e.g. 'Low' means this machine has fewer hours given its lifespan relative to the average of fiBaseModel. | Independent variable |
| Saledate | time of sale | Independent variable |
| fiModelDesc | Description of a unique machine model (see ModelID); concatenation of fiBaseModel & fiSecondaryDesc & fiModelSeries & fiModelDescriptor | Independent variable |
| State | US State in which sale occurred | Independent variable |
| Drive_System | machine configuration; typically describes whether 2 or 4 wheel drive | Independent variable |
| Enclosure | machine configuration - does the machine have an enclosed cab or not | Independent variable |
| Forks | machine configuration - attachment used for lifting | Independent variable |
| Pad_Type | machine configuration - type of treads a crawler machine uses | Independent variable |
| Ride_Control | machine configuration - optional feature on loaders to make the ride smoother | Independent variable |
| Transmission | machine configuration - describes type of transmission; typically automatic or manual | Independent variable |
| ... | ... | ... |
| SalePrice | cost of sale in USD | Target/dependent variable |
You can download the full version of this file directly from the Kaggle competition page (Kaggle account required) or view it on Google Sheets.
With all of this being known, let's get started!
First, we'll import the dataset and start exploring.
1. Importing the data and preparing it for modelling
First thing is first, let's get the libraries we need imported and the data we'll need for the project.
We'll start by importing pandas, NumPy and matplotlib.
Now we've got our tools for data analysis ready, we can import the data and start to explore it.
For this project, I've downloaded the data from Kaggle and stored it on the course GitHub under the file path ../data/bluebook-for-bulldozers.
We can write some code to check if the files are available locally (on our computer) and if not, we can download them.
Note: If you're running this notebook on Google Colab, the code below will enable you to download the dataset programmatically. Just beware that each time Google Colab shuts down, the data will have to be redownloaded. There's also an example Google Colab notebook showing how to download the data programmatically.
Dataset downloaded!
Let's check what files are available.
You can explore each of these files individually or read about them on the Kaggle Competition page.
For now, the main file we're interested in is TrainAndValid.csv (this is also a combination of Train.csv and Valid.csv), this is a combination of the training and validation datasets.
The training data (
Train.csv) contains sale data from 1989 up to the end of 2011.The validation data (
Valid.csv) contains sale data from January 1, 2012 - April 30, 2012.The test data (
Test.csv) contains sale data from May 1, 2012 - November 2012.
We'll use the training data to train our model to predict the sale price of bulldozers, we'll then validate its performance on the validation data to see if our model can be improved in any way. Finally, we'll evaluate our best model on the test dataset.
But more on this later on.
Let's import the TrainAndValid.csv file and turn it into a pandas DataFrame.
Wonderful! We've got our DataFrame ready to explore.
You might see a warning appear in the form:
DtypeWarning: Columns (13,39,40,41) have mixed types. Specify dtype option on import or set low_memory=False. df = pd.read_csv("../data/bluebook-for-bulldozers/TrainAndValid.csv")
This is just saying that some of our columns have multiple/mixed data types. For example, a column may contain strings but also contain integers. This is okay for now and can be addressed later on if necessary.
How about we get some information about our DataFrame?
Woah! Over 400,000 entries!
That's a much larger dataset than what we've worked with before.
One thing you might have noticed is that the saledate column value is being treated as a Python object (it's okay if you didn't notice, these things take practice).
When the Dtype is object, it's saying that it's a string.
However, when we look at it...
We can see that these object's are in the form of dates.
Since we're working on a time series problem (a machine learning problem with a time component), it's probably worth it to turn these strings into Python datetime objects.
Before we do, let's try visualize our saledate column against our SalePrice column.
To do so, we can create a scatter plot.
And to prevent our plot from being too big, how about we visualize the first 1000 values?
Hmm... looks like the x-axis is quite crowded.
Maybe we can fix this by turning the saledate column into datetime format.
Good news is that is looks like our SalePrice column is already in float64 format so we can view its distribution directly from the DataFrame using a histogram plot.
1.1 Parsing dates
When working with time series data, it's a good idea to make sure any date data is the format of a datetime object (a Python data type which encodes specific information about dates).
We can tell pandas which columns to read in as dates by setting the parse_dates parameter in pd.read_csv.
Once we've imported our CSV with the saledate column parsed, we can view information about our DataFrame again with df.info().
Nice!
Looks like our saledate column is now of type datetime64[ns], a NumPy-specific datetime format with high precision.
Since pandas works well with NumPy, we can keep it in this format.
How about we view a few samples from our SaleDate column again?
Beautiful! That's looking much better already.
We'll see how having our dates in this format is really helpful later on.
For now, how about we visualize our saledate column against our SalePrice column again?
1.2 Sorting our DataFrame by saledate
Now we've formatted our saledate column to be NumPy datetime64[ns] objects, we can use built-in pandas methods such as sort_values to sort our DataFrame by date.
And considering this is a time series problem, sorting our DataFrame by date has the added benefit of making sure our data is sequential.
In other words, we want to use examples from the past (example sale prices from previous dates) to try and predict future bulldozer sale prices.
Let's use the pandas.DataFrame.sort_values method to sort our DataFrame by saledate in ascending order.
Nice!
Looks like our older samples are now coming first and the newer samples are towards the end of the DataFrame.
1.3 Adding extra features to our DataFrame
One way to potentially increase the predictive power of our data is to enhance it with more features.
This practice is known as feature engineering, taking existing features and using them to create more/different features.
There is no set in stone way to do feature engineering and often it takes quite a bit of practice/exploration/experimentation to figure out what might work and what won't.
For now, we'll use our saledate column to add extra features such as:
Year of sale
Month of sale
Day of sale
Day of week sale (e.g. Monday = 1, Tuesday = 2)
Day of year sale (e.g. January 1st = 1, January 2nd = 2)
Since we're going to be manipulating the data, we'll make a copy of the original DataFrame and perform our changes there.
This will keep the original DataFrame in tact if we need it again.
Because we imported the data using read_csv() and we asked pandas to parse the dates using parase_dates=["saledate"], we can now access the different datetime attributes of the saledate column.
Let's use these attributes to add a series of different feature columns to our dataset.
After we've added these extra columns, we can remove the original saledate column as its information will be dispersed across these new columns.
We could add more of these style of columns, such as, whether it was the start or end of a quarter (the sale being at the end of a quarter may bye influenced by things such as quarterly budgets) but these will do for now.
Challenge: See what other datetime attributes you can add to
df_tmpusing a similar technique to what we've used above. Hint: check the bottom of thepandas.DatetimeIndexdocs.
How about we view some of our newly created columns?
Cool!
Now we've broken our saledate column into columns/features, we can perform further exploratory analysis such as visualizing the SalePrice against the saleMonth.
How about we view the first 10,000 samples (we could also randomly select 10,000 samples too) to see if reveals anything about which month has the highest sales?
Hmm... doesn't look like there's too much conclusive evidence here about which month has the highest sales value.
How about we plot the median sale price of each month?
We can do so by grouping on the saleMonth column with pandas.DataFrame.groupby and then getting the median of the SalePrice column.
Ohhh it looks like the median sale prices of January and February (months 1 and 2) are quite a bit higher than the other months of the year.
Could this be because of New Year budget spending?
Perhaps... but this would take a bit more investigation.
In the meantime, there are many other values we could look further into.
1.4 Inspect values of other columns
When first exploring a new problem, it's often a good idea to become as familiar with the data as you can.
Of course, with a dataset that has over 400,000 samples, it's unlikely you'll ever get through every sample.
But that's where the power of data analysis and machine learning can help.
We can use pandas to aggregate thousands of samples into smaller more managable pieces.
And as we'll see later on, we can use machine learning models to model the data and then later inspect which features the model thought were most important.
How about we see which states sell the most bulldozers?
Woah! Looks like Flordia sells a fair few bulldozers.
How about we go even further and group our samples by state and then find the median SalePrice per state?
We also compare this to the median SalePrice for all samples.
Now that's a nice looking figure!
Interestingly Florida has the most sales and the median sale price is above the overall median of all other states.
And if you had a bulldozer and were chasing the highest sale price, the data would reveal that perhaps selling in South Dakota would be your best bet.
Perhaps bulldozers are in higher demand in South Dakota because of a building or mining boom?
Answering this would require a bit more research.
But what we're doing here is slowly building up a mental model of our data.
So that if we saw an example in the future, we could compare its values to the ones we've already seen.
2. Model driven data exploration
We've performed a small Exploratory Data Analysis (EDA) as well as enriched it with some datetime attributes, now let's try to model it.
Why model so early?
Well, we know the evaluation metric (root mean squared log error or RMSLE) we're heading towards.
We could spend more time doing EDA, finding more out about the data ourselves but what we'll do instead is use a machine learning model to help us do EDA whilst simultaneously working towards the best evaluation metric we can get.
Remember, one of the biggest goals of starting any new machine learning project is reducing the time between experiments.
Following the Scikit-Learn machine learning map and taking into account the fact we've got over 100,000 examples, we find a sklearn.linear_model.SGDRegressor or a sklearn.ensemble.RandomForestRegressor model might be a good candidate.
Since we're worked with the Random Forest algorithm before (on the heart disease classification problem), let's try it out on our regression problem.
Note: We're trying just one model here for now. But you can try many other kinds of models from the Scikit-Learn library, they mostly work with a similar API. There are even libraries such as
LazyPredictwhich will try many models simultaneously and return a table with the results.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/c4/qj4gdk190td18bqvjjh0p3p00000gn/T/ipykernel_20423/2824176890.py in ?()
1 # This won't work since we've got missing numbers and categories
2 from sklearn.ensemble import RandomForestRegressor
3
4 model = RandomForestRegressor(n_jobs=-1)
----> 5 model.fit(X=df_tmp.drop("SalePrice", axis=1), # use all columns except SalePrice as X input
6 y=df_tmp.SalePrice) # use SalePrice column as y input
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(estimator, *args, **kwargs)
1469 skip_parameter_validation=(
1470 prefer_skip_nested_validation or global_skip_validation
1471 )
1472 ):
-> 1473 return fit_method(estimator, *args, **kwargs)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X, y, sample_weight)
359 # Validate or convert input data
360 if issparse(y):
361 raise ValueError("sparse multilabel-indicator for y is not supported.")
362
--> 363 X, y = self._validate_data(
364 X,
365 y,
366 multi_output=True,
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
646 if "estimator" not in check_y_params:
647 check_y_params = {**default_check_params, **check_y_params}
648 y = check_array(y, input_name="y", **check_y_params)
649 else:
--> 650 X, y = check_X_y(X, y, **check_params)
651 out = X, y
652
653 if not no_val_X and check_params.get("ensure_2d", True):
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
1297 raise ValueError(
1298 f"{estimator_name} requires y to be passed, but the target y is None"
1299 )
1300
-> 1301 X = check_array(
1302 X,
1303 accept_sparse=accept_sparse,
1304 accept_large_sparse=accept_large_sparse,
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
1009 )
1010 array = xp.astype(array, dtype, copy=False)
1011 else:
1012 array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1013 except ComplexWarning as complex_warning:
1014 raise ValueError(
1015 "Complex data not supported\n{}\n".format(array)
1016 ) from complex_warning
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/_array_api.py in ?(array, dtype, order, copy, xp, device)
747 # Use NumPy API to support order
748 if copy is True:
749 array = numpy.array(array, order=order, dtype=dtype)
750 else:
--> 751 array = numpy.asarray(array, order=order, dtype=dtype)
752
753 # At this point array is a NumPy ndarray. We convert it to an array
754 # container that is consistent with the input's namespace.
~/miniforge3/envs/ai/lib/python3.11/site-packages/pandas/core/generic.py in ?(self, dtype, copy)
2149 def __array__(
2150 self, dtype: npt.DTypeLike | None = None, copy: bool_t | None = None
2151 ) -> np.ndarray:
2152 values = self._values
-> 2153 arr = np.asarray(values, dtype=dtype)
2154 if (
2155 astype_is_view(values.dtype, arr.dtype)
2156 and using_copy_on_write()
ValueError: could not convert string to float: 'Low'
Oh no!
When we try to fit our model to the data, we get a value error similar to:
ValueError: could not convert string to float: 'Low'
The problem here is that some of the features of our data are in string format and machine learning models love numbers.
Not to mention some of our samples have missing values.
And typically, machine learning models require all data to be in numerical format as well as all missing values to be filled.
Let's start to fix this by inspecting the different datatypes in our DataFrame.
We can do so using the pandas.DataFrame.info() method, this will give us the different datatypes as well as how many non-null (a null value is generally a missing value) in our df_tmp DataFrame.
Note: There are some ML models such as
sklearn.ensemble.HistGradientBoostingRegressor, CatBoost and XGBoost which can handle missing values, however, I'll leave exploring each of these as extra-curriculum/extensions.
Ok, it seems as though we've got a fair few different datatypes.
There are int64 types such as MachineID.
There are float64 types such as SalePrice.
And there are object (the object dtype can hold any Python object, including strings) types such as UseageBand.
Resource: You can see a list of all the pandas dtypes in the pandas user guide.
How about we find out how many missing values are in each column?
We can do so using pandas.DataFrame.isna() (isna stands for 'is null or NaN') which will return a boolean True/False if a value is missing (True if missing, False if not).
Let's start by checking the missing values in the head of our DataFrame.
Alright it seems as though we've got some missing values in the MachineHoursCurrentMeter as well as the UsageBand and a few other columns.
But so far we've only viewed the first few rows.
It'll be very time consuming to go through each row one by one so how about we get the total missing values per column?
We can do so by calling .isna() on the whole DataFrame and then chaining it together with .sum().
Doing so will give us the total True/False values in a given column (when summing, True = 1, False = 0).
Woah! It looks like our DataFrame has quite a few missing values.
Not to worry, we can work on fixing this later on.
How about we start by tring to turn all of our data in numbers?
2.1 Inspecting the datatypes in our DataFrame
One way to help turn all of our data into numbers is to convert the columns with the object datatype into a category datatype using pandas.CategoricalDtype.
Note: There are many different ways to convert values into numbers. And often the best way will be specific to the value you're trying to convert. The method we're going to use, converting all objects (that are mostly strings) to categories is one of the faster methods as it makes a quick assumption that each unique value is its own number.
We can check the datatype of an individual column using the .dtype attribute and we can get its full name using .dtype.name.
Beautiful!
Now we've got a way to check a column's datatype individually.
There's also another group of methods to check a column's datatype directly.
For example, using pd.api.types.is_object_dtype(arr_or_dtype) we can get a boolean response as to whether the input is an object or not.
Note: There are many more of these checks you can perform for other datatypes such as strings under a similar name space
pd.api.types.is_XYZ_dtype. See the pandas documentation for more.
Let's see how it works on our df_tmp["UsageBand"] column.
We can also check whether a column is a string with pd.api.types.is_string_dtype(arr_or_dtype).
Nice!
We can even loop through the items (columns and their labels) in our DataFrame using pandas.DataFrame.items() (in Python dictionary terms, calling .items() on a DataFrame will treat the column names as the keys and the column values as the values) and print out samples of columns which have the string datatype.
As an extra check, passing the sample to pd.api.types.infer_dtype() will return the datatype of the sample.
This will be a good way to keep exploring our data.
Hmm... it seems that there are many more columns in the df_tmp with the object type that didn't display when checking for the string datatype (we know there are many object datatype columns in our DataFrame from using df_tmp.info()).
How about we try the same as above, except this time instead of pd.api.types.is_string_dtype, we use pd.api.types.is_object_dtype?
Let's try it.
Wonderful, looks like we've got sample outputs from all of the columns with the object datatype.
It also looks like that many of random samples are missing values.
2.2 Converting strings to categories with pandas
In pandas, one way to convert object/string values to numerical values is to convert them to categories or more specifically, the pd.CategoricalDtype datatype.
This datatype keeps the underlying data the same (e.g. doesn't change the string) but enables easy conversion to a numeric code using .cat.codes.
For example, the column state might have the values 'Alabama', 'Alaska', 'Arizona'... and these could be mapped to numeric values 1, 2, 3... respectively.
To see this in action, let's first convert the object datatype columns to "category" datatype.
We can do so by looping through the .items() of our DataFrame and reassigning each object datatype column using pandas.Series.astype(dtype="category").
Wonderful!
Now let's check if it worked by calling .info() on our DataFrame.
It looks like it worked!
All of the object datatype columns now have the category datatype.
We can inspect this on a single column using pandas.Series.dtype.
Excellent, notice how the column is now of type pd.CategoricalDtype.
We can also access these categories using pandas.Series.cat.categories.
Finally, we can get the category codes (the numeric values representing the category) using pandas.Series.cat.codes.
This gives us a numeric representation of our object/string datatype columns.
Epic!
All of our data is categorical and thus we can now turn the categories into numbers, however it's still missing values, not to worry though, we'll get to these shortly.
2.3 Saving our preprocessed data (part 1)
Before we start doing any further preprocessing steps on our DataFrame, how about we save our current DataFrame to file so we could import it again later if necessary.
Saving and updating your dataset as you go is common practice in machine learning problems. As your problem changes and evolves, the dataset you're working with will likely change too.
Making checkpoints of your dataset is similar to making checkpoints of your code.
Now we've saved our preprocessed data to file, we can re-import it and make sure it's in the same format.
Excellent, looking at the tale end (the far right side) our processed DataFrame has the columns we added to it (the extra data features) but it's still missing values.
But if we check df_tmp.info()...
Hmm... what happened here?
Notice that all of the category datatype columns are back to the object datatype.
This is strange since we already converted the object datatype columns to category.
Well then why did they change back?
This happens because of the limitations of the CSV (.csv) file format, it doesn't preserve data types, rather it stores all the values as strings.
So when we read in a CSV, pandas defaults to interpreting strings as object datatypes.
Not to worry though, we can easily convert them to the category datatype as we did before.
Note: If you'd like to retain the datatypes when saving your data, you can use file formats such as
parquet(Apache Parquet) andfeather. These filetypes have several advantages over CSV in terms of processing speeds and storage size. However, data stored in these formats is not human-readable so you won't be able to open the files and inspect them without specific tools. For more on different file formats in pandas, see the IO tools documentation page.
Now if we wanted to preserve the datatypes of our data, we can save to parquet or feather format.
Let's try using parquet format.
To do so, we can use the pandas.DataFrame.to_parquet() method.
Files in the parquet format typically have the file extension of .parquet.
Wonderful! Now let's try importing our DataFrame from the parquet format and check it using df_tmp.info().
Nice! Looks like using the parquet format preserved all of our datatypes.
For more on the parquet and feather formats, be sure to check out the pandas IO (input/output) documentation.
2.4 Finding and filling missing values
Let's remind ourselves of the missing values by getting the top 20 columns with the most missing values.
We do so by summing the results of pandas.DataFrame.isna() and then using sort_values(ascending=False) to showcase the rows with the most missing.
Ok, it seems like there are a fair few columns with missing values and there are several datatypes across these columns (numerical, categorical).
How about we break the problem down and work on filling each datatype separately?
2.5 Filling missing numerical values
There's no set way to fill missing values in your dataset.
And unless you're filling the missing samples with newly discovered actual data, every way you fill your dataset's missing values will introduce some sort of noise or bias.
We'll start by filling the missing numerical values in ourdataet.
To do this, we'll first find the numeric datatype columns.
We can do by looping through the columns in our DataFrame and calling pd.api.types.is_numeric_dtype(arr_or_dtype) on them.
Beautiful! Looks like we've got a mixture of int64 and float64 numerical datatypes.
Now how about we find out which numeric columns are missing values?
We can do so by using pandas.isnull(obj).sum() to detect and sum the missing values in a given array-like object (in our case, the data in a target column).
Let's loop through our DataFrame columns, find the numeric datatypes and check if they have any missing values.
Okay, it looks like our auctioneerID and MachineHoursCurrentMeter columns have missing numeric values.
Let's have a look at how we might handle these.
2.6 Discussing possible ways to handle missing values
As previously discussed, there are many ways to fill missing values.
For missing numeric values, some potential options are:
| Method | Pros | Cons |
|---|---|---|
| Fill with mean of column | - Easy to calculate/implement - Retains overall data distribution | - Averages out variation - Affected by outliers (e.g. if one value is much higher/lower than others) |
| Fill with median of column | - Easy to calculate/implement - Robust to outliers - Preserves center of data | - Ignores data distribution shape |
| Fill with mode of column | - Easy to calculate/implement - More useful for categorical-like data | - May not make sense for continuous/numerical data |
| Fill with 0 (or another constant) | - Simple to implement - Useful in certain contexts like counts | - Introduces bias (e.g. if 0 was a value that meant something) - Skews data (e.g. if many missing values, replacing all with 0 makes it look like that's the most common value) |
| Forward/Backward fill (use previous/future values to fill future/previous values) | - Maintains temporal continuity (for time series) | - Assumes data is continuous, which may not be valid |
| Use a calculation from other columns | - Takes existing information and reinterprets it | - Can result in unlikely outputs if calculations are not continuous |
| Interpolate (e.g. like dragging a cell in Excel/Google Sheets) | - Captures trends - Suitable for ordered data | - Can introduce errors - May assume linearity (data continues in a straight line) |
| Drop missing values | - Ensures complete data (only use samples with all information) - Useful for small datasets | - Can result in data loss (e.g. if many missing values are scattered across columns, data size can be dramatically reduced) - Reduces dataset size |
Which method you choose will be dataset and problem dependant and will likely require several phases of experimentation to see what works and what doesn't.
For now, we'll fill our missing numeric values with the median value of the target column.
We'll also add a binary column (0 or 1) with rows reflecting whether or not a value was missing.
For example, MachineHoursCurrentMeter_is_missing will be a column with rows which have a value of 0 if that row's MachineHoursCurrentMeter column was not missing and 1 if it was.
Why add a binary column indicating whether the data was missing or not?
We can easily fill all of the missing numeric values in our dataset with the median.
However, a numeric value may be missing for a reason.
Adding a binary column which indicates whether the value was missing or not helps to retain this information. It also means we can inspect these rows later on.
Missing numeric values filled!
How about we check again whether or not the numeric columns have missing values?
Woohoo! Numeric missing values filled!
And thanks to our binary _is_missing columns, we can even check how many were missing.
2.7 Filling missing categorical values with pandas
Now we've filled the numeric values, we'll do the same with the categorical values whilst ensuring that they are all numerical too.
Let's first investigate the columns which aren't numeric (we've already worked with these).
Okay, we've got plenty of category type columns.
Let's now write some code to fill the missing categorical values as well as ensure they are numerical (non-string).
To do so, we'll:
Create a blank column to category dictionary, we'll use this to store categorical value names (e.g. their string name) as well as their categorical code. We'll end with a dictionary of dictionaries in the form
{"column_name": {category_code: "category_value"...}...}.Loop through the items in the DataFrame.
Check if the column is numeric or not.
Add a binary column in the form
ORIGINAL_COLUMN_NAME_is_missingwith a0or1value for if the row had a missing value.Ensure the column values are in the
pd.Categoricaldatatype and get their category codes withpd.Series.cat.codes(we'll add1to these values since pandas defaults to assigning-1toNaNvalues, we'll use0instead).Turn the column categories and column category codes from 5 into a dictionary with Python's
dict(zip(category_names, category_codes))and save this to the blank dictionary from 1 with the target column name as key.Set the target column value to the numerical category values from 5.
Phew!
That's a fair few steps but nothing we can't handle.
Let's do it!
Ho ho! No errors!
Let's check out a few random samples of our DataFrame.
Beautiful! Looks like our data is all in numerical form.
How about we investigate an item from our column_to_category_dict?
This will show the mapping from numerical value to category (most likely a string) value.
Note: Categorical values do not necessarily have order. They are strictly a mapping from number to value. In this case, our categorical values are mapped in numerical order. If you feel that the order of a value may influence a model in a negative way (e.g.
1 -> Highis lower than3 -> Mediumbut should be higher), you may want to look into ordering the values in a particular way or using a different numerical encoding technique such as one-hot encoding.
And we can do the same for the state column values.
Beautiful!
How about we check to see all of the missing values have been filled?
2.8 Saving our preprocessed data (part 2)
One more step before we train new model!
Let's save our work so far so we could re-import our preprocessed dataset if we wanted to.
We'll save it to the parquet format again, this time with a suffix to show we've filled the missing values.
And to make sure it worked, we can re-import it.
Does it have any missing values?
Checkpoint reached!
We've turned all of our data into numbers as well as filled the missing values, time to try fitting a model to it again.
2.9 Fitting a machine learning model to our preprocessed data
Now all of our data is numeric and there are no missing values, we should be able to fit a machine learning model to it!
Let's reinstantiate our trusty sklearn.ensemble.RandomForestRegressor model.
Since our dataset has a substantial amount of rows (~400k+), let's first make sure the model will work on a smaller sample of 1000 or so.
Note: It's common practice on machine learning problems to see if your experiments will work on smaller scale problems (e.g. smaller amounts of data) before scaling them up to the full dataset. This practice enables you to try many different kinds of experiments with faster runtimes. The benefit of this is that you can figure out what doesn't work before spending more time on what does.
Our X values (features) will be every column except the SalePrice column.
And our y values (labels) will be the entirety of the SalePrice column.
We'll time how long our smaller experiment takes using the magic function %%time and placing it at the top of the notebook cell.
Note: You can find out more about the
%%timemagic command by typing%%time?(note the question mark on the end) in a notebook cell.
Woah! It looks like things worked!
And quite quick too (since we're only using a relatively small number of rows).
How about we score our model?
We can do so using the built-in method score().
By default, sklearn.ensemble.RandomForestRegressor uses coefficient of determination ( or R-squared) as the evaluation metric (higher is better, with a score of 1.0 being perfect).
Wow, it looks like our model got a pretty good score on only 1000 samples (the best possible score it could achieve would've been 1.0).
How about we try our model on the whole dataset?
Ok, that took a little bit longer than fitting on 1000 samples (but that's too be expected, as many more calculations had to be made).
There's a reason we used n_jobs=-1 too.
If we stuck with the default of n_jobs=None (the same as n_jobs=1), it would've taken much longer.
| Configuration (MacBook Pro M1 Pro, 10 Cores) | CPU Times (User) | CPU Times (Sys) | CPU Times (Total) | Wall Time |
|---|---|---|---|---|
n_jobs=-1 (all cores) | 9min 14s | 3.85s | 9min 18s | 1min 15s |
n_jobs=None (default) | 7min 14s | 1.75s | 7min 16s | 7min 25s |
And as we've discussed many times, one of the main goals when starting a machine learning project is to reduce your time between experiments.
How about we score the model trained on all of the data?
An even better score!
Oh wait...
Oh no...
I think we've got an error... (you might've noticed it already)
Why might this metric be unreliable?
Hint: Compare the data we trained on versus the data we evaluated on.
2.10 A big (but fixable) mistake
One of the hard things about bugs in machine learning projects is that they are often silent.
For example, our model seems to have fit the data with no issues and then evaluated with a good score.
So what's wrong?
It seems we've stumbled across one of the most common bugs in machine learning and that's data leakage (data from the training set leaking into the validation/testing sets).
We've evaluated our model on the same data it was trained on.
This isn't the model's fault either.
It's our fault.
Right back at the start we imported a file called TrainAndValid.csv, this file contains both the training and validation data.
And while we preprocessed it to make sure there were no missing values and the samples were all numeric, we never split the data into separate training and validation splits.
The right workflow would've been to train the model on the training split and then evaluate it on the unseen and separate validation split.
Our evaluation scores above are quite good but they can't necessarily be trusted to be replicated on unseen data (data in the real world) because they've been obtained by evaluating the model on data its already seen during training.
This would be the equivalent of a final exam at university containing all of the same questions as the practice exam without any changes, you may get a good grade, but does that good grade translate to the real world?
Not to worry, we can fix this!
How?
We can import the training and validation datasets separately via Train.csv and Valid.csv respectively.
Or we could import TrainAndValid.csv and perform the appropriate splits according the original Kaggle competition page (training data includes all samples prior to 2012 and validation data includes samples from January 1 2012 to April 30 2012).
In both methods, we'll have to perform similar preprocessing steps we've done so far.
Except because the validation data is supposed to remain as unseen data, we'll only use information from the training set to preprocess the validation set (and not mix the two).
We'll work on this in the subsequent sections.
The takeaway?
Always (if possible) create appropriate data splits at the start of a project.
Because it's one thing to train a machine learning model but if you can't evaluate it properly (on unseen data), how can you know how it'll perform (or may perform) in the real world on new and unseen data?
3. Splitting data into the right train/validation sets
The bad news is, we evaluated our model on the same data we trained it on.
The good news is, we get to practice importing and preprocessing our data again.
This time we'll make sure we've got separate training and validation splits.
Previously, we used pandas to ensure our data was all numeric and had no missing values.
And we can still use pandas for things such as creating/altering date-related columns.
But using pandas for all of our data preprocessing can be an issue with larger scale datasets or when new data is introduced.
How about this time we add Scikit-Learn to the mix and make a reproducible pipeline for our data preprocessing needs?
Note: Scikit-Learn has a fantastic guide on data transformations and in particular data preprocessing. I'd highly recommend spending an hour or so reading through this documentation, even if it doesn't make a lot of sense to begin with. Rest assured, with practice and experimentation you'll start to get the hang of it.
According to the Kaggle data page, the train, validation and test sets are split according to dates.
This makes sense since we're working on a time series problem (using past sale prices to try and predict future sale prices).
Knowing this, randomly splitting our data into train, validation and test sets using something like sklearn.model_selection.train_test_split() wouldn't work as this would mix samples from different dates in an unintended way.
Instead, we split our data into training, validation and test sets using the date each sample occured.
In our case:
Training data (
Train.csv) = all samples up until 2011.Validation data (
Valid.csv) = all samples form January 1, 2012 - April 30, 2012.Testing data (
Test.csv) = all samples from May 1, 2012 - November 2012.
Previously we imported TrainAndValid.csv which is a combination of Train.csv and Valid.csv in one file.
We could split this based on the saledate column.
However, we could also import the Train.csv and Valid.csv files separately (we'll import Test.csv later on when we've trained a model).
We'll also import ValidSolution.csv which contains the SalePrice of Valid.csv and make sure we match the columns based on the SalesID key.
Note: For more on making good training, validation and test sets, check out the post How (and why) to create a good validation set by Rachel Thomas as well as The importance of a test set by Daniel Bourke.
Nice!
We've now got separate training and validation datasets imported.
In a previous section, we created a function to decompose the saledate column into multiple features such as saleYear, saleMonth, saleDay and more.
Let's now replicate that function here and apply it to our train_df and valid_df.
Wonderful, now let's make sure it worked by inspecting the last 5 columns of train_df.
Perfect! How about we try and fit a model?
3.1 Trying to fit a model on our training data
I'm a big fan of trying to fit a model on your dataset as early as possible.
If it works, you'll have to inspect and check its results.
And if it doesn't work, you'll get some insights into what you may have to do to your dataset to prepare it.
Let's turn our DataFrames into features (X) by dropping the SalePrice column (this is the value we're trying to predict) and labels (y) by extracting the SalePrice column.
Then we'll create a model using sklearn.ensemble.RandomForestRegressor and finally we'll try to fit it to only the training data.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/c4/qj4gdk190td18bqvjjh0p3p00000gn/T/ipykernel_20423/150598518.py in ?()
9 # Create a model
10 model = RandomForestRegressor(n_jobs=-1)
11
12 # Fit a model to the training data only
---> 13 model.fit(X=X_train,
14 y=y_train)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(estimator, *args, **kwargs)
1469 skip_parameter_validation=(
1470 prefer_skip_nested_validation or global_skip_validation
1471 )
1472 ):
-> 1473 return fit_method(estimator, *args, **kwargs)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X, y, sample_weight)
359 # Validate or convert input data
360 if issparse(y):
361 raise ValueError("sparse multilabel-indicator for y is not supported.")
362
--> 363 X, y = self._validate_data(
364 X,
365 y,
366 multi_output=True,
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
646 if "estimator" not in check_y_params:
647 check_y_params = {**default_check_params, **check_y_params}
648 y = check_array(y, input_name="y", **check_y_params)
649 else:
--> 650 X, y = check_X_y(X, y, **check_params)
651 out = X, y
652
653 if not no_val_X and check_params.get("ensure_2d", True):
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
1297 raise ValueError(
1298 f"{estimator_name} requires y to be passed, but the target y is None"
1299 )
1300
-> 1301 X = check_array(
1302 X,
1303 accept_sparse=accept_sparse,
1304 accept_large_sparse=accept_large_sparse,
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
1009 )
1010 array = xp.astype(array, dtype, copy=False)
1011 else:
1012 array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1013 except ComplexWarning as complex_warning:
1014 raise ValueError(
1015 "Complex data not supported\n{}\n".format(array)
1016 ) from complex_warning
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/_array_api.py in ?(array, dtype, order, copy, xp, device)
747 # Use NumPy API to support order
748 if copy is True:
749 array = numpy.array(array, order=order, dtype=dtype)
750 else:
--> 751 array = numpy.asarray(array, order=order, dtype=dtype)
752
753 # At this point array is a NumPy ndarray. We convert it to an array
754 # container that is consistent with the input's namespace.
~/miniforge3/envs/ai/lib/python3.11/site-packages/pandas/core/generic.py in ?(self, dtype, copy)
2149 def __array__(
2150 self, dtype: npt.DTypeLike | None = None, copy: bool_t | None = None
2151 ) -> np.ndarray:
2152 values = self._values
-> 2153 arr = np.asarray(values, dtype=dtype)
2154 if (
2155 astype_is_view(values.dtype, arr.dtype)
2156 and using_copy_on_write()
ValueError: could not convert string to float: 'Medium'
Oh no!
We run into the error:
ValueError: could not convert string to float: 'Medium'
Hmm...
Where have we seen this error before?
It looks like since we re-imported our training dataset (from Train.csv) its no longer all numerical (hence the ValueError above).
Not to worry, we can fix this!
3.2 Encoding categorical features as numbers using Scikit-Learn
We've preprocessed our data previously with pandas.
And while this is a viable approach, how about we practice using another method?
This time we'll use Scikit-Learn's built-in preprocessing methods.
Why?
Because it's good exposure to different techniques.
And Scikit-Learn has many built-in helpful and well tested methods for preparing data.
You can also string together many of these methods and create a reusable pipeline (you can think of this pipeline as plumbing for data).
To preprocess our data with Scikit-Learn, we'll first define the numerical and categorical features of our dataset.
Nice!
We define our different feature types so we can use different preprocessing methods on each type.
Scikit-Learn has many built-in methods for preprocessing data under the sklearn.preprocessing module.
And I'd encourage you to spend some time reading the preprocessing data section of the Scikit-Learn user guide for more details.
For now, let's focus on turning our categorical features into numbers (from object/string datatype to numeric datatype).
The practice of turning non-numerical features into numerical features is often referred to as encoding.
There are several encoders available for different use cases.
| Encoder | Description | Use case | For use on |
|---|---|---|---|
LabelEncoder | Encode target labels with values between 0 and n_classes-1. | Useful for turning classification target values into numeric representations. | Target labels. |
OneHotEncoder | Encode categorical features as a one-hot numeric array. | Turns every positive class of a unique category into a 1 and every negative class into a 0. | Categorical variables/features. |
OrdinalEncoder | Encode categorical features as an integer array. | Turn unique categorical values into a range of integers, for example, 0 maps to 'cat', 1 maps to 'dog', etc. | Categorical variables/features. |
TargetEncoder | Encode regression and classification targets into a shrunk estimate of the average target values for observations of the category. | Useful for converting targets into a certain range of values. | Target variables. |
For our case, we're going to start with OrdinalEncoder.
When transforming/encoding values with Scikit-Learn, the steps as follows:
Instantiate an encoder, for example,
sklearn.preprocessing.OrdinalEncoder.Use the
sklearn.preprocessing.OrdinalEncoder.fitmethod on the training data (this helps the encoder learn a mapping of categorical to numeric values).Use the
sklearn.preprocessing.OrdinalEncoder.transformmethod on the training data to apply the learned mapping from categorical to numeric values.Note: The
sklearn.preprocessing.OrdinalEncoder.fit_transformmethod combines steps 1 & 2 into a single method.
Apply the learned mapping to subsequent datasets such as validation and test splits using
sklearn.preprocessing.OrdinalEncoder.transformonly.
Notice how the fit and fit_transform methods were reserved for the training dataset only.
This is because in practice the validation and testing datasets are meant to be unseen, meaning only information from the training dataset should be used to preprocess the validation/test datasets.
In short:
Instantiate an encoder such as
sklearn.preprocessing.OrdinalEncoder.Fit the encoder to and transform the training dataset categorical variables/features with
sklearn.preprocessing.OrdinalEncoder.fit_transform.Transform categorical variables/features from subsequent datasets such as the validation and test datasets with the learned encoding from step 2 using
sklearn.preprocessing.OridinalEncoder.transform.Note: Notice the use of the
transformmethod on validation/test datasets rather thanfit_transform.
Let's do it!
We'll use the OrdinalEncoder class to fill any missing values with np.nan (NaN).
We'll also make sure to only use the OrdinalEncoder on the categorical features of our DataFrame.
Finally, the OrdinalEncoder expects all input variables to be of the same type (e.g. either numeric only or string only) so we'll make sure all the input variables are strings only using pandas.DataFrame.astype(str).
Wonderful!
Let's see if it worked.
First, we'll check out the original X_train DataFrame.
And how about X_train_preprocessed?
Beautiful!
Notice all of the non-numerical values in X_train have been converted to numerical values in X_train_preprocessed.
Now how about missing values?
Let's see the top 10 columns with the highest number of missing values from X_train.
Ok, plenty of missing values.
How about X_train_preprocessed?
Perfect! No missing values as well!
Now, what if we wanted to retrieve the original categorical values?
We can do using the OrdinalEncoder.categories_ attribute.
This will return the categories of each feature found during fit (or during fit_transform), the categories will be in the order of the features seen (same order as the columns of the DataFrame).
Since these come in the order of the features seen, we can create a mapping of these using the categorical column names of our DataFrame.
We can also reverse our OrdinalEncoder values with the inverse_transform() method.
This is helpful for reversing a preprocessing step or viewing the original data again if necessary.
Nice!
Now how about we try fitting a model again?
3.3 Fitting a model to our preprocessed training data
We've used Scikit-Learn to convert the categorical data in our training and validation sets into numbers.
But we haven't yet done anything with missing numerical values.
As it turns out, we can still try and fit a model.
Why?
Because there are several estimators/models in Scikit-Learn that can handle missing (NaN) values.
And our trusty sklearn.ensemble.RandomForestRegressor is one of them!
Let's try it out on our X_train_preprocessed DataFrame.
Note: For a list of all Scikit-Learn estimators that can handle
NaNvalues, check out the Scikit-Learn imputation of missing values user guide.
It worked!
Now you might be thinking, "well if we could fit a model on a dataset with missing values, why did we bother filling them before?"
And that's a great question.
The main reason is to practice, practice, practice.
While there are some models which can handle missing values, others can't.
So it's good to have experience with both of these scenarios.
Let's see how our model scores on the validation set, data our model has never seen.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<timed eval> in ?()
1 'Could not get source, probably due dynamically evaluated source code.'
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(self, X, y, sample_weight)
844 """
845
846 from .metrics import r2_score
847
--> 848 y_pred = self.predict(X)
849 return r2_score(y, y_pred, sample_weight=sample_weight)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X)
1059 The predicted values.
1060 """
1061 check_is_fitted(self)
1062 # Check data
-> 1063 X = self._validate_X_predict(X)
1064
1065 # Assign chunk of trees to jobs
1066 n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X)
637 force_all_finite = "allow-nan"
638 else:
639 force_all_finite = True
640
--> 641 X = self._validate_data(
642 X,
643 dtype=DTYPE,
644 accept_sparse="csr",
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
629 out = y
630 else:
631 out = X, y
632 elif not no_val_X and no_val_y:
--> 633 out = check_array(X, input_name="X", **check_params)
634 elif no_val_X and not no_val_y:
635 out = _check_y(y, **check_params)
636 else:
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
1009 )
1010 array = xp.astype(array, dtype, copy=False)
1011 else:
1012 array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1013 except ComplexWarning as complex_warning:
1014 raise ValueError(
1015 "Complex data not supported\n{}\n".format(array)
1016 ) from complex_warning
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/_array_api.py in ?(array, dtype, order, copy, xp, device)
747 # Use NumPy API to support order
748 if copy is True:
749 array = numpy.array(array, order=order, dtype=dtype)
750 else:
--> 751 array = numpy.asarray(array, order=order, dtype=dtype)
752
753 # At this point array is a NumPy ndarray. We convert it to an array
754 # container that is consistent with the input's namespace.
~/miniforge3/envs/ai/lib/python3.11/site-packages/pandas/core/generic.py in ?(self, dtype, copy)
2149 def __array__(
2150 self, dtype: npt.DTypeLike | None = None, copy: bool_t | None = None
2151 ) -> np.ndarray:
2152 values = self._values
-> 2153 arr = np.asarray(values, dtype=dtype)
2154 if (
2155 astype_is_view(values.dtype, arr.dtype)
2156 and using_copy_on_write()
ValueError: could not convert string to float: 'Low'
Oops!
Looks like we get an error:
ValueError: could not convert string to float: 'Low'
This is because we tried to evaluate our model on the original X_valid dataset which still contains strings rather than X_valid_preprocessed which contains all numerical values.
As we've discussed before, in machine learning problems, it's important to evaluate your models on data in the same format as they were trained on.
Knowing this, let's evaluate our model on our preprocessed validation dataset.
Excellent!
Now you might be wondering why this score ( or R-squared by default) is lower than the previous score of ~0.9875.
That's because this score is based on a model that has only seen the training data and is being evaluated on an unseen dataset (training on Train.csv, evaluating on Valid.csv).
Our previous score was from a model that had all of the evaluation samples in the training data (training and evaluating on TrainAndValid.csv).
So in practice, we would consider the most recent score as a much more reliable metric of how well our model might perform on future unseen data.
Just for fun, let's see how our model scores on the training dataset.
As expected our model performs better on the training set than the validation set.
It also scores much closer to the previous score of ~0.9875 we obtained when training and scoring on TrainAndValid.csv combined.
Note: It is common to see a model perform slightly worse on a validation/testing dataset than on a training set. This is because the model has seen all of the examples in the training set, where as, if done correctly, the validation and test sets are keep separate during training. So you would expect a model to do better on problems that it has seen before versus problems it hasn't. If you find your model scoring much higher on unseen data versus seen data (e.g. higher scores on the test set compared to the training set), you might want to inspect your data to make sure there isn't any leakage from the validation/test set into the training set.
4. Building an evaluation function
Evaluating a machine learning model is just as important as training one.
And so because of this, let's create an evaluation function to make evaluation faster and reproducible.
According to Kaggle for the Bluebook for Bulldozers competition, the evaluation function they use is root mean squared log error (RMSLE).
Where:
is the predicted value,
is the actual value,
is the number of observations.
Contrast this with mean absolute error (MAE), another common regression metric.
With RMSLE, the relative error is more meaningful than the absolute error. You care more about ratios than absolute errors. For example, being off by $100 on a $1000 prediction (10% error) is more significant than being off by $100 on a $10,000 prediction (1% error). RMSLE is sensitive to large percentage errors.
Where as with MAE, is more about exact differences, a $100 prediction error is weighted the same regardless of the actual value.
In each of case, a lower value (closer to 0) is better.
For any problem, it's important to define the evaluation metric you're going to try and improve on.
In our case, let's create a function that calculates multiple evaluation metrics.
Namely, we'll use:
MAE (mean absolute error) via
sklearn.metrics.mean_absolute_error- lower is better.RMSLE (root mean squared log error) via
sklearn.metrics.root_mean_squared_log_error- lower is better.(R-squared or coefficient of determination) via the
scoremethod - higher is better.
For MAE and RMSLE we'll be comparing the model's predictions to the truth labels.
We can get an array of predicted values from our model using model.predict(X=features_to_predict_on).
Now that's a nice looking function!
How about we test it out?
Beautiful!
Now we can reuse this in the future for evaluating other models.
5. Tuning our model's hyperparameters
Hyperparameters are the settings we can change on our model.
And tuning hyperparameters on a given model can often alter its performance on a given dataset.
Ideally, changing hyperparameters would lead to better results.
However, it's often hard to know what hyperparameter changes would improve a model ahead of time.
So what we can do is run several experiments across various different hyperparameter settings and record which lead to the best results.
5.1 Making our modelling experiments faster (to speed up hyperparameter tuning)
Because of the size of our dataset (~400,000 rows), retraining an entire model (about 1-1.5 minutes on my MacBook Pro M1 Pro) for each new set of hyperparameters would take far too long to continuing experimenting as fast as we want to.
So what we'll do is take a sample of the training set and tune the hyperparameters on that before training a larger model.
Note: If you're experiments are taking longer than 10-seconds (or far longer than what you can interact with), you should be trying to speed things up. You can speed experiments up by sampling less data, using a faster computer or using a smaller model.
We can take a artificial sample of the training set by altering the number of samples seen by each n_estimator (an n_estimator is a decision tree a random forest will create during training, more trees generally leads to better performance but sacrifices compute time) in sklearn.ensemble.RandomForestRegressor using the max_samples parameter.
For example, setting max_samples to 10,000 means every n_estimator (default 100) in our RandomForestRegressor will only see 10,000 random samples from our DataFrame instead of the entire ~400,000.
In other words, we'll be looking at 40x less samples which means we should get faster computation speeds but we should also expect our results to worsen (because the model has less samples to learn patterns from).
Let's see if reducing the number samples speeds up our modelling time.
Nice! That worked much faster than training on the whole dataset.
Let's evaluate our model with our show_scores function.
Excellent! Even though our new model saw far less data than the previous model, it still looks to be performing quite well.
With this faster model, we can start to run a series of different hyperparameter experiments.
5.2 Hyperparameter tuning with RandomizedSearchCV
The goal of hyperparameter tuning is to values for our model's settings which lead to better results.
We could sit there and do this by hand, adjusting parameters on sklearn.ensemble.RandomForestRegressor such as n_estimators, max_depth, min_samples_split and more.
However, this would quite tedious.
Instead, we can define a dictionary of hyperparametmer settings in the form {"hyperparamter_name": [values_to_test]} and then use sklearn.model_selection.RandomizedSearchCV (randomly search for best combination of hyperparameters) or sklearn.model_selection.GridSearchCV (exhaustively search for best combination of hyperparameters) to go through all of these settings for us on a given model and dataset and then record which perform best.
A general workflow is to start with a large number and wide range of potential settings and use RandomizedSearchCV to search across these randomly for a limited number of iterations (e.g. n_iter=10).
And then take the best results and narrow the search space down before exhaustively search for the best hyperparameters with GridSearchCV.
Let's start trying to find better hyperparameters by:
Define a dictionary of hyperparameter values for our
RandomForestRegressormodel. We'll keepmax_samples=10000so our experiments run faster.Setup an instance of
RandomizedSearchCVto explore the parameter values defined in step 1. We can adjust how many sets of hyperparameters our model tries using then_iterparameter as well as how many times our model performs cross-validation using thecvparameter. For example, settingn_iter=20andcv=3means there will be 3 cross-validation folds for each of the 20 different combinations of hyperparameters, a total of 60 (3*20) experiments.Fit the instance of
RandomizedSearchCVto the data. This will automatically go through the defined number of iterations and record the results for each. The best model gets loaded at the end.
Note: You can read more about the tuning of hyperparameters of an esimator/model in the Scikit-Learn user guide.
Phew! That's quite a bit of testing!
Good news for us is that we can check the best hyperparameters with the best_params_ attribute.
And we can evaluate this model with our show_scores function.
5.3 Training a model with the best hyperparameters
Like all good machine learning cooking shows, I prepared a model earlier.
I tried 100 different combinations of hyperparameters (setting n_iter=100 in RandomizedSearchCV) and found the best results came from the settings below.
n_estimators=90max_depth=Nonemin_samples_leaf=1min_samples_split=5max_features=0.5n_jobs=-1max_samples=None
Note: This search (
n_iter=100) took ~2-hours on my MacBook Pro M1 Pro. So it's kind of a set and come back later experiment. That's one of the things you'll have to get used to as a machine learning engineer, figuring out what to do whilst your model trains. I like to go for long walks or to the gym (rule of thumb: while my model trains, I train).
We'll instantiate a new model with these discovered hyperparameters and reset the max_samples back to its original value.
And of course, we can evaluate our ideal_model with our show_scores function.
Woohoo!
With these new hyperparameters as well as using all the samples, we can see an improvement to our models performance.
One thing to keep in mind is that a larger model isn't always the best for a given problem even if it performs better.
For example, you may require a model that performs inference (makes predictions) very fast with a slight tradeoff to performance.
One way to a faster model is by altering some of the hyperparameters to create a smaller overall model.
Particularly by lowering n_estimators since each increase in n_estimators is basically building another small model.
Let's half our n_estimators value and see how it goes.
Nice! The faster model fits to the training data in about half the time of the full model.
Now how does it go on performance?
Woah! Looks like our faster model evaluates (performs inference/makes predictions) in about half the time too.
And only for a small tradeoff in validation RMSLE performance.
5.4 Comparing our model's scores
We've built four models so far with varying amounts of data and hyperparameters.
Let's compile the results into a DataFrame and then make a plot to compare them.
Now we've got our model result data in a DataFrame, let's turn it into a bar plot comparing the validation RMSLE of each model.
By the looks of the plot, our ideal_model is indeed the ideal model, slightly edging out fast_model in terms of validation RMSLE.
6. Saving our best model to file
Since we've confirmed our best model as our ideal_model object, we can save it to file so we can load it in later and use it without having to retrain it.
Note: For more on model saving options with Scikit-Learn, see the documentation on model persistence.
To save our model we can use the joblib.dump method.
And to load our model we can use the joblib.load method.
We can make sure our model saving and loading worked by evaluating our best_model with show_scores.
And to confirm our ideal_model and best_model results are very close (if not the exact same), we can compare them with:
The equality operator
==.np.icloseand setting the absolute tolerance (atol) to1e-4.
Note: When saving and loading a model, it is often the case to have very slightly different values at the extremes. For example, the pre-saved model may have an RMSLE of
0.24654150224930685where as the loaded model may have an RMSLE of0.24654150224930684where in this case the values are off by0.00000000000000001(a very small number). This is due to the precision of computing and the way computers store values, where numbers are exact but can be represented up to a certain amount of precision. This is why we generally compare results with many decimals usingnp.iscloserather than the==operator.
7. Making predictions on test data
Now we've got a trained model saved and loaded, it's time to make predictions on the test data.
Our model is trained on data prior to 2011, however, the test data is from May 1 2012 to November 2012.
So what we're doing is trying to use the patterns our model has learned from the training data to predict the sale price of a bulldozer with characteristics it's never seen before but are assumed to be similar to that of those in the training data.
Let's load in the test data from Test.csv, we'll make sure to parse the dates of the saledate column.
You might notice that the test_df is missing the SalePrice column.
That's because that's the variable we're trying to predict based on all of the other variables.
We can make predictions with our best_model using the predict method.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[127], line 2
1 # Let's see how the model goes predicting on the test data
----> 2 test_preds = best_model.predict(X=test_df)
, in ForestRegressor.predict(self, X)
1061 check_is_fitted(self)
1062 # Check data
-> 1063 X = self._validate_X_predict(X)
1065 # Assign chunk of trees to jobs
1066 n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)
, in BaseForest._validate_X_predict(self, X)
638 else:
639 force_all_finite = True
--> 641 X = self._validate_data(
642 X,
643 dtype=DTYPE,
644 accept_sparse="csr",
645 reset=False,
646 force_all_finite=force_all_finite,
647 )
648 if issparse(X) and (X.indices.dtype != np.intc or X.indptr.dtype != np.intc):
649 raise ValueError("No support for np.int64 index based sparse matrices")
, in BaseEstimator._validate_data(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
537 def _validate_data(
538 self,
539 X="no_validation",
(...)
544 **check_params,
545 ):
546 """Validate input data and set or check the `n_features_in_` attribute.
547
548 Parameters
(...)
606 validated.
607 """
--> 608 self._check_feature_names(X, reset=reset)
610 if y is None and self._get_tags()["requires_y"]:
611 raise ValueError(
612 f"This {self.__class__.__name__} estimator "
613 "requires y to be passed, but the target y is None."
614 )
, in BaseEstimator._check_feature_names(self, X, reset)
530 if not missing_names and not unexpected_names:
531 message += (
532 "Feature names must be in the same order as they were in fit.\n"
533 )
--> 535 raise ValueError(message)
ValueError: The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- saledate
Feature names seen at fit time, yet now missing:
- saleDay
- saleDayofweek
- saleDayofyear
- saleMonth
- saleYear
Oh no!
We get an error:
ValueError: The feature names should match those that were passed during fit. Feature names unseen at fit time:
saledate Feature names seen at fit time, yet now missing:
saleDay
saleDayofweek
saleDayofyear
saleMonth
saleYear
Ahhh... the test data isn't in the same format of our other data, so we have to fix it.
7.1 Preprocessing the test data (to be in the same format as the training data)
Our model has been trained on data preprocessed in a certain way.
This means in order to make predictions on the test data, we need to take the same steps we used to preprocess the training data to preprocess the test data.
Remember, whatever you do to preprocess the training data, you have to do to the test data.
Let's recreate the steps we used for preprocessing the training data except this time we'll do it on the test data.
First, we'll add the extra date features to breakdown the saledate column.
Date features added!
Now can we make predictions with our model on the test data?
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/var/folders/c4/qj4gdk190td18bqvjjh0p3p00000gn/T/ipykernel_20423/2042912174.py in ?()
1 # Try to predict with model
----> 2 test_preds = best_model.predict(test_df)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X)
1059 The predicted values.
1060 """
1061 check_is_fitted(self)
1062 # Check data
-> 1063 X = self._validate_X_predict(X)
1064
1065 # Assign chunk of trees to jobs
1066 n_jobs, _, _ = _partition_estimators(self.n_estimators, self.n_jobs)
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/ensemble/_forest.py in ?(self, X)
637 force_all_finite = "allow-nan"
638 else:
639 force_all_finite = True
640
--> 641 X = self._validate_data(
642 X,
643 dtype=DTYPE,
644 accept_sparse="csr",
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/base.py in ?(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
629 out = y
630 else:
631 out = X, y
632 elif not no_val_X and no_val_y:
--> 633 out = check_array(X, input_name="X", **check_params)
634 elif no_val_X and not no_val_y:
635 out = _check_y(y, **check_params)
636 else:
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/validation.py in ?(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
1009 )
1010 array = xp.astype(array, dtype, copy=False)
1011 else:
1012 array = _asarray_with_order(array, order=order, dtype=dtype, xp=xp)
-> 1013 except ComplexWarning as complex_warning:
1014 raise ValueError(
1015 "Complex data not supported\n{}\n".format(array)
1016 ) from complex_warning
~/miniforge3/envs/ai/lib/python3.11/site-packages/sklearn/utils/_array_api.py in ?(array, dtype, order, copy, xp, device)
747 # Use NumPy API to support order
748 if copy is True:
749 array = numpy.array(array, order=order, dtype=dtype)
750 else:
--> 751 array = numpy.asarray(array, order=order, dtype=dtype)
752
753 # At this point array is a NumPy ndarray. We convert it to an array
754 # container that is consistent with the input's namespace.
~/miniforge3/envs/ai/lib/python3.11/site-packages/pandas/core/generic.py in ?(self, dtype, copy)
2149 def __array__(
2150 self, dtype: npt.DTypeLike | None = None, copy: bool_t | None = None
2151 ) -> np.ndarray:
2152 values = self._values
-> 2153 arr = np.asarray(values, dtype=dtype)
2154 if (
2155 astype_is_view(values.dtype, arr.dtype)
2156 and using_copy_on_write()
ValueError: could not convert string to float: 'Low'
Another error...
ValueError: could not convert string to float: 'Low'
We can fix this by running our ordinal_encoder (that we used to preprocess the training data) on the categorical features in our test DataFrame.
Ok, date features created and categorical features turned into numbers, can we make predictions on the test data now?
Holy smokes! It worked!
Let's check out our test_preds.
Wonderful, looks like we're getting the price predictions of a given bulldozer.
How many predictions are there?
Perfect, looks like theres one prediction per sample in the test DataFrame.
Now how would we submit our predictions to Kaggle?
Well, when looking at the Kaggle submission requirements, we see that if we wanted to make a submission, the data is required to be in a certain format.
Namely, a DataFrame containing the SalesID and the predicted SalePrice of the bulldozer.
Let's make it.
Excellent! We've got a SalePrice prediction for every SalesID in the test DataFrame.
Let's save this to CSV so we could upload it or share it with someone else if we had to.
8. Making a prediction on a custom sample
We've made predictions on the test dataset which contains sale data from May to November 2012.
But how does our model go on a more recent bulldozer sale?
If we were to find an advertisement on a bulldozer sale, could we use our model on the information in the advertisement to predict the sale price?
In other words, how could we use our model on a single custom sample?
It's one thing to predict on data that has already been formatted but it's another thing to be able to predict a on a completely new and unseen sample.
Note: For predicting on a custom sample, the same rules apply as making predictions on the test dataset. The data you make predictions on should be in the same format that your model was trained on. For example, it should have all the same features and the numerical encodings should be in the same ballpark (e.g. preprocessed by the
ordinal_encoderwe fit to the training set). It's likely that samples you collect from the wild may not be as well formatted as samples in a pre-existing dataset. So it's the job of the machine learning engineer to be able to format/preprocess new samples in the same way a model was trained on.
If we're going to make a prediction on a custom sample, it'll need to be in the same format as our other datasets.
So let's remind ourselves of the columns/features in our test dataset.
Wonderful, so if we're going to make a prediction on a custom sample, we'll need to fill out these details as much as we can.
Let's try and make a prediction on the example test sample.
Nice!
We get an output array containing a predicted SalePrice.
Let's now try it on a custom sample.
Again, like all good machine learning cooking shows, I've searched the internet for "bulldozer sales in America" and found a sale from 6th July 2024 (I'm writing these materials in mid 2024 so if it's many years in the future and the link doesn't work, check out the screenshot below).
| 
| |:--😐 | Screenshot of a bulldozer sale advertisement. I took information from this advertisement to create our own custom sample for testing our machine learning model on data from the wild. Source. |
I went through the advertisement online and collected as much detail as I could and formatted the dictionary below with all of the related fields.
It may not be perfect but data in the real world is rarely perfect.
For values I couldn't find or were inconspicuous, I filled them with np.nan (or NaN).
Some values such as SalesID were unobtainable because they were part of the original collected dataset, for these I've also used np.nan.
Also notice how I've already created the extra date features saleYear, saleMonth, saleDay and more by manually breaking down the listed sale date of 6 July 2024.
Now we've got a single custom sample in the form of a dictionary, we can turn it into a DataFrame.
And of course, we can preprocess the categoricial features using our ordinal_encoder (we use the same instance of OrdinalEncoder as we trained on the training dataset).
Custom sample preprocessed, let's make a prediction!
Now how close was this to the actual sale price (listed on the advertisement) of $72,600?
Woah!
We get a quite high MAE value, however, it looks like our model's RMSLE performance on the custom sample was even better than the best_model on the validation dataset.
Not too bad for a model trained on sales data over 12 years older than our custom sample's sale date.
Note: In practice, to make this process easier, rather than manually typing out all of the feature values by hand, you might want to create an application capable of ingesting these values in a nice user interface. To create such machine learning applications, I'd practice by checking out Streamlit or Gradio.
9. Finding the most important predictive features
Since we've built a model which is able to make predictions, the people you share these predictions with (or yourself) might be curious of what parts of the data led to these predictions.
This is where feature importance comes in.
Feature importance seeks to figure out which different attributes of the data were most important when it comes to predicting the target variable.
In our case, after our model learned the patterns in the data, which bulldozer sale attributes were most important for predicting its overall sale price?
We can do this for our sklearn.ensemble.RandomForestRegressor instance using the feature_importances_ attribute.
Let's check it out.
Woah, looks like we get one value per feature in our dataset.
We can inspect these further by turning them into a DataFrame.
We'll sort it descending order so we can see which feature our model is assigning the highest value.
Hmmm... looks like YearMade may be contributing the most value in the model's eyes.
How about we turn our DataFrame into a plot to compare values?
Ok, looks like the top 4 features contributing to our model's predictions are YearMade, ProductSize, Enclosure and saleYear.
Referring to the original data dictionary, do these values make sense to be contributing the most to the model?
YearMade- Year of manufacture of the machine.ProductSize- Size of the bulldozer.Enclosure- Type of bulldozer enclosure (e.g. OROPS = Open Rollover Protective Structures, EROPS = Enclosed Rollover Protective Structures).saleYear- The year the bulldozer was sold (this is one of our engineered features fromsaledate).
Now I've never sold a bulldozer but reading about each of these values seems to make sense that they would contribute significantly to the sale price.
I know when I've bought cars in the past, the year that is was made was an important part of my decision.
And it also makes sense that ProductSize be an important feature when deciding on the price of a bulldozer.
Let's check out the unique values for ProductSize and Enclosure.
My guess is that a bulldozer with a ProductSize of 'Mini' would sell for less than a bulldozer with a size of 'Large'.
We could investigate this further in an extension to model driven data exploratory analysis or we could take this information to a colleague or client to discuss further.
Either way, we've now got a machine learning model capable of predicting the sale price of bulldozers given their features/attributes!
That's a huuuuuuge effort!
And you should be very proud of yourself for making it this far.
Summary
We've covered a lot of ground.
But there are some main takeaways to go over.
Every machine learning problem is different - Since machine learning is such a widespread technology, it can be used for a multitude of different problems. In saying this, there will often be many different ways to approach a problem. In this example, we've focused on predicting a number, which is a regression problem. And since our data had a time component, it could also be considered a time series problem.
The machine learner's motto: Experiment, experiment, experiment! - Since there are many different ways to approach machine learning problems, one of the best habits you can develop is an experimental mindset. That means not being afraid to try new things over and over. Because the more things you try, the quicker you can figure what doesn't work and the quicker you can start to move towards what does.
Always keep the test set separate - If you can't evaluate your model on unseen data, how would you know how it will perform in the real world on future unseen data? Of course, using a test set isn't a perfect replica of the real world but if it's done right, it can give you a good idea. Because evaluating a model is just as important as training a model.
If you've trained a model on a data in a certain format, you'll have to make predictions in the same format - Any preprocessing you do to the training dataset, you'll have to do to the validation, test and custom data. Any computed values should happen on the training set only and then be used to update any subsequent datasets.
Exercises
Fill the missing values in the numeric columns with the median using Scikit-Learn and see if that helps our best model's performance (hint: see
sklearn.impute.SimpleImputerfor more).Try putting multiple steps together (e.g. preprocessing -> modelling) with Scikit-Learn's
sklearn.pipeline.Pipelinefeatures.Try using another regression model/estimator on our preprocessed dataset and see how it goes. See the Scikit-Learn machine learning map for potential model options.
Try replacing the
sklearn.preprocessing.OrdinalEncoderwe used for the categorical variables withsklearn.preprocessing.OneHotEncoder(you may even want to do this within a pipeline) with thesklearn.ensemble.RandomForestRegressormodel and see how it performs. Which is better for our specific dataset?
Extra-curriculum
The following resources are suggested extra reading and activities to add backing to the materials we've covered in this project.
Reading documentation and knowing where to find information is one of the best skills you can develop as an engineer.
Read the pandas IO tools documentation page for an idea of all the possible ways to get data in and out of pandas.
See all of the available datatypes in the pandas user guide (knowing what type your data is in can help prevent a lot of future errors).
Read the Scikit-Learn dataset transformations and data preprocessing guide for an overview of all the different ways you can preprocess and transform data.
For more on saving and loading model objects with Scikit-Learn, see the documentation on model persistence.
For more on the importance of creating good validation and test sets, I'd recommend reading How (and why) to create a good validation set by Rachel Thomas as well as The importance of a test set by Daniel Bourke.
We've covered a handful of models in the Scikit-Learn library, however, there are some other ML models which are worth exploring such as CatBoost and XGBoost. Both of these models can handle missing values and are often touted as some of the most performant ML models on the market. A good extension would be to try get one of them working on our bulldozer data.
Bonus: You can also see a list of models in Scikit-Learn which can handle missing/NaN values.
Example Exercise Solutions
The following are examples of how to solve the above exercises.
1. Fill the missing values in the numeric columns with the median using Scikit-Learn and see if that helps our best model's performance
Looks like filling the missing numeric values made our ideal_model_2 perform slightly worse than our original ideal_model.
ideal_model_2 had a validation RMSLE of 0.24697812443315573 where as ideal_model had a validation RMSLE of 0.24654150224930685.
2. Try putting multiple steps together (e.g. preprocessing -> modelling) with Scikit-Learn's sklearn.pipeline.Pipeline
3. Try using another regression model/estimator on our preprocessed dataset and see how it goes
Going to use sklearn.linear_model.HistGradientBoostingRegressor.
4. Try replacing the sklearn.preprocessing.OrdinalEncoder we used for the categorical variables with sklearn.preprocessing.OneHotEncoder
Note: This may take quite a long time depending on your machine. For example, on my MacBook Pro M1 Pro it took ~10 minutes with
n_estimators=10(9x lower than what we used for ourbest_model). This is because usingsklearn.preprocessing.OneHotEncoderadds many more features to our dataset (each feature gets turned into an array of 0's and 1's for each unique value). And the more features, the longer it takes to compute and find patterns between them.