python pandas eliminates null values and whitespace and Nan data replacement methods

When collecting data manually, it is often possible to mix the null value with the space at the beginning of the 1, and you will not notice the space added to the originally empty cell. This causes problems for data handlers because null values and Spaces represent no data, and the Series method in ES1en, notnull(), will include data with Spaces, so we won't get the data we want completely. Here's a simple way to handle this.

Method 1:

Since we assume that both null and space represent no data, we can get Boolean arrays in both cases first.

Here, our dataset of type DataFrame is df and there is one variable VIN, so the Boolean array that takes null values and Spaces is NONE_VIN. And then through that Boolean array, we get the data that we want

NONE_VIN = (df["VIN"].isnull()) | (df["VIN"].apply(lambda x: str(x).isspace()))
df_null = df[NONE_VIN]
df_not_null = df[~NONE_VIN]

Method 2:

Each value in the variable VIN is modified directly using the.apply method of Series. If it is found to be a space, it returns Nan, otherwise it returns the original value.

df["VIN"]=df["VIN"].apply(lambda x: np.NaN if str(x).isspace() else x)
df_null = df[df["VIN"].isnull()]
df_not_null = df[df["VIN"].notnull()]

Replace NaN in dataframe with the desired value

import pandas as pd
df1 = pd.DataFrame([{'col1':'a', 'col2':1}, {'col1':'b', 'col2':2}])
df2 = pd.DataFrame([{'col1':'a', 'col3':11}, {'col1':'c', 'col3':33}])

data = pd.merge(left=df1, right=df2, how='left', left_on='col1', right_on='col1')
print data
#  will NaN Replace with None
print data.where(data.notnull(), None)

Output results:

 col1 col2 col3
0 a  1 11
1 b  2 NaN
 col1 col2 col3
0 a  1 11
1 b  2 None

Conclusion:

The idea behind method 1 is to simply determine if it is a space and include the space in the selection. The idea for method 2 is to first convert the space to NaN and then normally use.isnull () or.notnull () to get the data we want.