python pandas replacing null values using fillna method

Table of Contents

1. Replace all null values
2. Replace null values in a particular column
3. Use of the ffill method
4. Use of the bfill method
5. Replacing limited number of null values
6. Video Tutorial

1. Replace all null values

In order to start the practical, import Pandas library and read the data file (Stock_data.csv) and assign it in to the stock_data data frame as shown in figure 1.

The fillna() method can be used to replace the null values by a given value. In order to replace the all null values which are present in the data frame use fillna() method as shown below. Assume that the null values should be replaced by 1.

stock_data.fillna(1)

Let’s print out only the first 6 rows using head(6) as shown in figure 2. Observe that the null values are replaced by 1.

2. Replace null values in a particular column

In order to replace the null values in a particular column, the relevant column names should be passed into the fillna() method as a dictionary as shown below,

stock_data.fillna({‘Open’:1})

Let’s assume we need the Open column’s null values to be replaced by 1. Specify the value which is needed to replace where there is NaN by using a column as shown above. Let’s print the first 6 rows. As you can see in figure 3, the null values are replaced by 1 in the Open column only.

Let’s look at another example. This time assumes that the Open column’s null values should be replaced by 1 and the High column’s null values should be replaced by 0. To the same dictionary, we need to pass the High column value as shown below.

stock_data.fillna({‘Open’:1 , ‘High’:0})

Execute the code. Observe the data in figure 4. The Null values contained in the Open column were replaced by 1 and High column was replaced by 0.

3. Use of ffill method

The ffill means forward fill. It can be used to replace or fill the null values in a data set by the forward value (above cell value). This can be specified under the method parameter inside the fillna() function.

stock_data.fillna(method= ‘ffill’)

Execute the command. Observe figure 5and compare the tables at the top and bottom. It can be noted that the null values are replaced by the value which is above that null value. Also, note that the first cell of the High column is a Null value but it isn’t replaced. That is because there is no value above that record.

4. Use of bfill method

The bfill means backward fill. It can be used to replace or fill the null values in a data set by the backward value (below cell value). This can be specified under method parameter inside the fillna() function as the same way before.

stock_data.fillna(method= ‘bfill’)

Execute the command. Observe figure 6. It can be noted that the null values are replaced by the value which is below that null value. Also note that the first cell of the High column is a Null value but, it is now replaced by the below value. And the 2^nd cell of the Open column has become NaN as the below value is NaN.

5. Replacing limited number of null values

An additional parameter can be used inside the fillna() method in order to limit the number of null values that needed to be replaced. This can be done using the limit parameter.

First, let’s use ffill method for forward filling and then put the limit as 1. This means only the first null value should be replaced by the above value.

stock_data.fillna(method= ‘ffill’, limit = 1)

Execute the code. Take figure 7 as the reference and compare it with figure 8. Note that only the first null value in each column is replaced by the value above that null value. Other null values remain the same.

Video Tutorial