Bank Customer Churn Prediction

Problem Statement

Context

Businesses like banks which provide service have to worry about problem of Customer Churn, i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.

Objective

As a Data Scientist at the bank, your task is to build a neural network-based classifier to predict whether a customer is likely to leave the bank within the next six months.

💡 This is a binary classification problem that will be solved by training multiple Neural Networks and selecting the best-performing one.

Data Dictionary

The case study uses an open-source dataset from Kaggle, consisting of 10,000 samples and 14 features, including CustomerId, CreditScore, Geography, Gender, Age, Tenure, Balance, and others.

• CustomerId: Unique ID which is assigned to each customer.

• Surname: Last name of the customer.

• CreditScore: It defines the credit history of the customer.

• Geography: A customer’s location.

• Gender: It defines the gender of the customer.

• Age: Age of the customer.

• Tenure: Number of years for which the customer has been with the bank.

• NumOfProducts: Refers to the number of products that a customer has purchased through the bank.

• Balance: Current account balance.

• HasCrCard: It is a categorical variable which indicates whether the customer has credit card or not.

• EstimatedSalary: Estimated salary of the customer.

• isActiveMember: It is a categorical variable that indicates whether the customer is an active member of the bank, meaning they regularly use bank products, make transactions, and engage with the bank's services.

• Exited: This variable indicates whether the customer left the bank within six months. It has two possible values: 0: No (The customer did not leave the bank) or 1: Yes (The customer left the bank).

Importing Necessary Libraries

# Installing the necessary libraries with the specified version
!pip install tensorflow==2.17.0 scikit-learn==1.3.1 seaborn==0.13.1 matplotlib==3.8.0 numpy==1.25.0 pandas==2.2.2 -q --user --no-warn-script-location

Note: After executing the above cell, restart the notebook kernel and rerun all steps from the beginning.

import pandas as pd   # Library for data manipulation and analysis, especially with tabular data
import numpy as np   # Fundamental package for numerical computations and handling multidimensional arrays
import matplotlib.pyplot as plt   # Comprehensive library for creating static, animated, and interactive visualizations
import seaborn as sns   # Statistical data visualization library built on matplotlib, offering advanced and aesthetically pleasing plots
import time   # Module for handling time-related tasks, such as measuring execution time or adding delays

# For splitting datasets into training and testing sets
from sklearn.model_selection import train_test_split
# Tools for encoding categorical variables, scaling features, and other data preprocessing tasks
from sklearn.preprocessing import LabelEncoder, OneHotEncoder, StandardScaler
# For handling missing data by imputing appropriate values
from sklearn.impute import SimpleImputer
# Functions for evaluating machine learning model performance
from sklearn.metrics import confusion_matrix, f1_score, accuracy_score, recall_score, precision_score, classification_report, roc_curve

import tensorflow as tf   # Open-source platform for machine learning and deep learning
from tensorflow import keras   # High-level API for building and training deep learning models
from tensorflow.keras import backend   # Low-level API for managing computations in deep learning frameworks
from tensorflow.keras import regularizers   # For l2 regularization
from tensorflow.keras.models import Sequential   # Class for building neural networks sequentially, layer by layer
from tensorflow.keras.layers import Dense, Input, Dropout, BatchNormalization   # Modules for defining layers in a neural network
from tensorflow.keras.metrics import Accuracy, Precision, Recall  # Import keras performance metrics

from imblearn.over_sampling import SMOTE  # Synthetic Minority Oversampling Technique for sampling

# Supress scientific notations
pd.options.display.float_format='{:.2f}'.format

# Suppress unnecessary warnings during execution
import warnings
warnings.filterwarnings("ignore")

# Set the random seed for reproducibility across multiple libraries:
# 1) NumPy seed for reproducible numerical operations
# 2) Backend random seed for TensorFlow operations
# 3) Python random seed for consistent behavior in Python's random module
keras.utils.set_random_seed(812)

# Enable deterministic operations on the GPU for reproducibility
# This may reduce performance but ensures consistent results across runs
tf.config.experimental.enable_op_determinism()

Loading the Dataset

# Mount Google Drive to current Colab Notebook
from google.colab import drive
drive.mount('/content/drive')

Mounted at /content/drive

# Read the original dataset
originaldataset = pd.read_csv("/content/drive/My Drive/Texas McCombs/Neural Networks/Project #4/churn-data.csv")

# Take a copy of the dataset
data = originaldataset.copy()

Data Overview

• Initial data inspection
• Sanity checks

# Have a look at the first 5 rows of the dataset
data.head()

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	1	15634602	Hargrave	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	2	15647311	Hill	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	3	15619304	Onio	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	4	15701354	Boni	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	5	15737888	Mitchell	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

# Check 10 random rows from the dataset
data.sample(n=10, random_state=1)

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
9953	9954	15655952	Burke	550	France	Male	47	2	0.00	2	1	1	97057.28	0
3850	3851	15775293	Stephenson	680	France	Male	34	3	143292.95	1	1	0	66526.01	0
4962	4963	15665088	Gordon	531	France	Female	42	2	0.00	2	0	1	90537.47	0
3886	3887	15720941	Tien	710	Germany	Male	34	8	147833.30	2	0	1	1561.58	0
5437	5438	15733476	Gonzalez	543	Germany	Male	30	6	73481.05	1	1	1	176692.65	0
8517	8518	15671800	Robinson	688	France	Male	20	8	137624.40	2	1	1	197582.79	0
2041	2042	15709846	Yeh	840	France	Female	39	1	94968.97	1	1	0	84487.62	0
1989	1990	15622454	Zaitsev	695	Spain	Male	28	0	96020.86	1	1	1	57992.49	0
1933	1934	15815560	Bogle	666	Germany	Male	74	7	105102.50	1	1	1	46172.47	0
9984	9985	15696175	Echezonachukwu	602	Germany	Male	35	7	90602.42	2	1	1	51695.41	0

# Have a look at the last 5 rows
data.tail()

	RowNumber	CustomerId	Surname	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
9995	9996	15606229	Obijiaku	771	France	Male	39	5	0.00	2	1	0	96270.64	0
9996	9997	15569892	Johnstone	516	France	Male	35	10	57369.61	1	1	1	101699.77	0
9997	9998	15584532	Liu	709	France	Female	36	7	0.00	1	0	1	42085.58	1
9998	9999	15682355	Sabbatini	772	Germany	Male	42	3	75075.31	2	1	0	92888.52	1
9999	10000	15628319	Walker	792	France	Female	28	4	130142.79	1	1	0	38190.78	0

• Our target variable is the Exited column. The objective of this analysis is to predict which customers are likely to leave the bank within a given period.
• The RowNumber, CustomerId, and Surname columns do not contribute meaningfully to the prediction task as they have no relationship with the target variable Exited. Therefore, we will remove these columns from the dataset.

# Remove the irrelevant columns
data = data.drop(['RowNumber', 'CustomerId', 'Surname'], axis=1)

# Understand the shape of the dataset
data.shape

(10000, 11)

• Our dataset now contains 10,000 observations and 11 features after removing unnecessary columns.

# Check data types and number of non-null values for each column
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   CreditScore      10000 non-null  int64  
 1   Geography        10000 non-null  object 
 2   Gender           10000 non-null  object 
 3   Age              10000 non-null  int64  
 4   Tenure           10000 non-null  int64  
 5   Balance          10000 non-null  float64
 6   NumOfProducts    10000 non-null  int64  
 7   HasCrCard        10000 non-null  int64  
 8   IsActiveMember   10000 non-null  int64  
 9   EstimatedSalary  10000 non-null  float64
 10  Exited           10000 non-null  int64  
dtypes: float64(2), int64(7), object(2)
memory usage: 859.5+ KB

• The dataset consists of 2 float, 7 integer, and 2 object data types.
• Object data types cannot be directly used for model building, so we will convert them into categorical variables.
• Additionally, there are no null values in the dataset.

💡 It is generally recommended to perform the conversion after the EDA to better understand the data and avoid making early decisions.
👉 Converting data types from "object" to "category" reduces the memory usage of the DataFrame.
💡 It is always a good idea to check for duplicate values after removing the index column (e.g., RowNumber, CustomerId).

# Check for hidden missing values
data.isin(['N/A', None, '']).sum()

	0
CreditScore	0
Geography	0
Gender	0
Age	0
Tenure	0
Balance	0
NumOfProducts	0
HasCrCard	0
IsActiveMember	0
EstimatedSalary	0
Exited	0

dtype: int64

• There are no hidden missing values in our dataset.

# Summary statistics for numerical columns
data.describe()

	CreditScore	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
count	10000.00	10000.00	10000.00	10000.00	10000.00	10000.00	10000.00	10000.00	10000.00
mean	650.53	38.92	5.01	76485.89	1.53	0.71	0.52	100090.24	0.20
std	96.65	10.49	2.89	62397.41	0.58	0.46	0.50	57510.49	0.40
min	350.00	18.00	0.00	0.00	1.00	0.00	0.00	11.58	0.00
25%	584.00	32.00	3.00	0.00	1.00	0.00	0.00	51002.11	0.00
50%	652.00	37.00	5.00	97198.54	1.00	1.00	1.00	100193.91	0.00
75%	718.00	44.00	7.00	127644.24	2.00	1.00	1.00	149388.25	0.00
max	850.00	92.00	10.00	250898.09	4.00	1.00	1.00	199992.48	1.00

• CreditScore:

  • Range: 350 to 850, with a mean of 650.53. Standard deviation of 96.65, suggesting some variation in credit scores.
  • The minimum value of 350 is low, possibly indicating subprime credit scores.

• Age:

  • Range: 18 to 92 years, with a mean of 38.92.
  • The spread of ages is fairly wide, but the median (50%) is 37, indicating a relatively balanced age distribution.

• Tenure:

  • Range: 0 to 10 years, with a mean of 5.01.
  • A significant portion of customers have a tenure of 3 to 7 years, with some new customers (tenure of 0).

• Balance:

  • Range: 0 to 250,898.09, with a mean of about 76,485.
  • A wide spread between the 25% and 75% percentiles (0 to 127,644.24) suggests a skewed distribution with a few customers having very high balances.

• NumOfProducts:

  • Range: 1 to 4 products, with a mean of 1.53.
  • Majority of customers have between 1 and 2 products.

• HasCrCard:

  • Binary variable (0 or 1), with 70.55% of customers having a credit card.
  • More customers have a credit card than those who don’t.

• IsActiveMember:

  • Binary variable (0 or 1), with 51.51% of customers being active.
  • Around half of the customers are not active members.

• EstimatedSalary:

  • Range: 11,580 to 199,992, with a mean of 100,090.
  • The median salary (50% = 100,193) aligns closely with the mean, suggesting a balanced salary distribution.

• Exited:

  • Binary target variable (0 or 1), with approximately 20% of customers having exited.
  • The dataset is imbalanced, with far fewer customers exiting the bank.

👉 Since the data is imbalanced with only 20% of customers having exited, using class_weight during training would help the model focus more on the minority class (exited customers) by assigning a higher weight to the class, thus improving its ability to correctly predict the minority class without being biased toward the majority class.

# Summary statistics for categorical columns
categorical_columns = data.select_dtypes(include=['object', 'category'])
categorical_columns.describe()

	Geography	Gender
count	10000	10000
unique	3	2
top	France	Male
freq	5014	5457

• Geography:

  • There are 3 unique values, with "France" being the most frequent (50.14% of the data).
  • The data may be skewed toward customers from France, and this could influence the model if not handled correctly.

• Gender:

  • There are 2 unique values, with "Male" being the most frequent (54.57% of the data).
  • The gender distribution is fairly balanced but with a slight bias towards males.

# Checking for duplicate values
data.duplicated().sum()

0

• There are no duplicate values in our dataset

Exploratory Data Analysis

We will use the following functions for EDA

# Function to plot a boxplot and a histogram along the same scale.

def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to the show density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a star will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram

# Function to create labeled barplots

def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot

# Function to plot stacked bar chart

def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left",
        frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()

Univariate Analysis

Histogram and Boxplot for numerical features

# Visualize the distribution of the CreditScore
histogram_boxplot(data, 'CreditScore')

• The credit score distribution is roughly normal with a peak around 650-700, showing most individuals have "good" to "very good" credit.
• There's a slight left skew, indicating fewer high credit scores, and a secondary peak around 850, suggesting a subgroup with excellent credit.

# Visualize the distribution of the Age
histogram_boxplot(data, 'Age')

• The majority of bank customers are in their 30s and 40s, with a peak around the age of 40.
• There's a significant drop-off in customer count for ages above 50, suggesting fewer older customers.

# Visualize the distribution of the Tenure
histogram_boxplot(data, 'Tenure')

• Most customers have been with the bank for 2 to 9 years.
• The median tenure, marked by the dashed line, is around 5 years.

# Visualize the distribution of the Balance
histogram_boxplot(data, 'Balance')

• The distribution of account balances is highly right-skewed, with most customers having a zero balance.
• A few outliers with high balances pull the mean above the median, as shown by the green dashed line.

# Visualize the distribution of the NumOfProducts
histogram_boxplot(data, 'NumOfProducts')

• The distribution of the number of products is highly concentrated around 1 and 2, with very few customers having 3 or 4 products.
• The mean is close to 1.5, indicating the majority of customers have fewer products, while a small outliers exist.

# Visualize the distribution of the EstimatedSalary
histogram_boxplot(data, 'EstimatedSalary')

• The estimated salary of customers is uniformly distributed, indicating no significant bias toward any particular income range.
• The box plot shows a symmetric distribution with no apparent outliers, and the median is centered near 100,000.

Frequency distribution of categorical variables

# Visualize the distribution of customers' Geography using a labeled bar plot
labeled_barplot(data,'Geography', perc=True)

• Slightly more than half of the customers are from France
• The remaining customers are almost equally divided between Germany and Spain

# Visualize the distribution of customers' Gender using a labeled bar plot
labeled_barplot(data,'Gender', perc=True)

• 45.4% of the customers are female, and 54.6% are male
• The gender distribution is fairly balanced but with a slight bias towards males

Class Imbalance Check

👉 To check for target variable class imbalance, we need to analyze the distribution of the target variable:

# Visualize the distribution of the Target Variable using a labeled bar plot
labeled_barplot(data,"Exited", perc=True)

• Almost 80% of the observations represent non-churn cases, while just over 20% represent churn cases.
• This indicates that the dataset is imbalanced. To address this, we will use techniques such as SMOTE and class weighting.

Bivariate Analysis

Customers' bank relationship vs Target variable

cols = data[['CreditScore','EstimatedSalary','NumOfProducts','Balance','Tenure', 'Age']].columns.tolist()
plt.figure(figsize=(14,7))

for i, variable in enumerate(cols):
                     plt.subplot(2, 3, i+1)
                     sns.boxplot(x=data["Exited"], y=data[variable], palette="PuBu")
                     plt.tight_layout()
                     plt.title(variable)
plt.show()

• Customers with low or zero account balances are more likely to leave the bank.
• Customers with a tenure between 3 and 7 years are more likely to leave the bank.
• Younger customers are more likely to leave the bank.
• Credit score, salary, and number of products do not seem to have a significant effect on customer churn.

Customer demographics vs Target variable

stacked_barplot(data, "Gender", "Exited")

Exited     0     1    All
Gender                   
All     7963  2037  10000
Female  3404  1139   4543
Male    4559   898   5457
------------------------------------------------------------------------------------------------------------------------

• Female customers are slightly more likely to churn

stacked_barplot(data, "Geography", "Exited")

Exited        0     1    All
Geography                   
All        7963  2037  10000
Germany    1695   814   2509
France     4204   810   5014
Spain      2064   413   2477
------------------------------------------------------------------------------------------------------------------------

• Customers from Germany are slightly more likely to churn

Correlation check

sns.pairplot(data, diag_kind="kde", hue="Exited")
plt.show()

• Overall, both Age and NumOfProducts seem to show strong correlations with churn and could be important features in the classification model.

plt.figure(figsize=(15,11))
sns.heatmap(data.corr(numeric_only=True), annot=True, vmin=-1, vmax=1, fmt='.2f', cmap="Spectral")   # Linear correlation
#sns.heatmap(data.corr(numeric_only=True, method="spearman"), annot=True, vmin=-1, vmax=1, fmt=".2f", cmap="Spectral")   # Non-linear correlation
plt.show()

• None of the columns are strongly correlated with each other.
• The number of products customers have with the bank and their account balance are negatively correlated (-0.30), indicating that customers with fewer products tend to have lower account balances.
• Age and churn (exited bank) are positively correlated (0.29), suggesting that younger customers are more likely to leave the bank.
• Being an active member of the bank and churn (exited bank) are negatively correlated (-0.16), indicating that inactive customers are more likely to churn.
• Account balance and churn (exited bank) are positively correlated (0.12), implying that customers with higher account balances are slightly more likely to leave the bank.

Data Preprocessing

Data Type Conversion

💡 It is generally recommended to convert object data types into categorical variables before splitting the data.

# Convert object columns to categorical variables
data['Geography'] = data['Geography'].astype('category')
data['Gender'] = data['Gender'].astype('category')

# Encode categorical variables using label encoding or one-hot encoding
data['Geography'] = data['Geography'].cat.codes
data['Gender'] = data['Gender'].cat.codes

# Check data types again after the conversion
data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   CreditScore      10000 non-null  int64  
 1   Geography        10000 non-null  int8   
 2   Gender           10000 non-null  int8   
 3   Age              10000 non-null  int64  
 4   Tenure           10000 non-null  int64  
 5   Balance          10000 non-null  float64
 6   NumOfProducts    10000 non-null  int64  
 7   HasCrCard        10000 non-null  int64  
 8   IsActiveMember   10000 non-null  int64  
 9   EstimatedSalary  10000 non-null  float64
 10  Exited           10000 non-null  int64  
dtypes: float64(2), int64(7), int8(2)
memory usage: 722.8 KB

👉 Notice that the meory usage slightly reduced from 859.5+ KB to 722.8 KB.

Outlier Detection & Treatment

# Outlier detection using boxplots
numeric_columns = data.select_dtypes(include=np.number).columns.tolist()

plt.figure(figsize=(13, 11))

for i, variable in enumerate(numeric_columns):
    plt.subplot(6, 4, i+1)
    plt.boxplot(data[variable], whis=1.5)
    plt.tight_layout()
    plt.title(variable)

plt.show()

• Outliers are present in the CreditScore, Age, and NumofProducts columns.
• These outliers will not be treated, as they appear to be valid data points that can provide valuable insights for the model.

# Take another copy of the dataset
df = data.copy()

df.dtypes

	0
CreditScore	int64
Geography	int8
Gender	int8
Age	int64
Tenure	int64
Balance	float64
NumOfProducts	int64
HasCrCard	int64
IsActiveMember	int64
EstimatedSalary	float64
Exited	int64

dtype: object

Dummy Variable Creation

# Separating the features and the target variable
X = df.drop(['Exited'], axis=1)   # independent variables
y = df['Exited']   # dependent variable

👉 Separating the features and target variable before dummy variable creation is necessary for avoiding data contamination.
👉 Columns like Geography and Gender are already numerical (int8), so creating dummies would be redundant and could introduce unnecessary complexity.

Train-Validation-Test Split

# Splitting the data into training, validation and test sets (A 70/15/15 split)

# Step 1: Split the data into training (70%) and temporary (30%)
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)

# Step 2: Split the temporary set into validation (50% of temp -> 15% of total) and test (50% of temp -> 15% of total)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=0, stratify=y_temp)

X_train.shape, X_val.shape, X_test.shape   # Print the shapes

((7000, 10), (1500, 10), (1500, 10))

• We have 7,000 observations in the train set, 1,500 observations in the validation set, and 1,500 observations in the test set.
• This is a 70% train, 15% validation, and 15% test distribution.

Data Scaling

# Check 10 random rows from the train dataset
X_train.sample(n=10, random_state=1)

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
5364	812	1	1	33	2	127154.14	2	0	1	105383.49
6002	701	0	0	41	2	0.00	1	1	0	47856.78
5905	511	0	0	30	5	0.00	2	1	0	143994.86
6262	513	0	1	44	1	63562.02	2	0	1	52629.73
6036	674	0	0	27	4	79144.34	1	0	1	50743.83
3066	503	0	0	28	5	0.00	2	1	0	125918.17
3136	607	0	1	44	0	0.00	2	1	1	81140.09
2169	773	1	1	43	8	81844.91	2	1	1	35908.46
2005	785	1	0	32	3	124493.03	2	0	1	52583.79
1524	796	0	1	51	6	0.00	2	0	1	194733.28

• Before building our neural network, we need to use scaling techniques to ensure that all features are on similar scales, which helps the model converge faster and perform better.
• Continuous Variables: CreditScore, Age, Tenure, Balance, and EstimatedSalary vary widely in range and should be scaled.
• Categorical Variables: Geography, Gender, HasCrCard, IsActiveMember and NumOfProducts are already integer-encoded, and additional scaling is not needed.

Standardizing the continuous variables

• Typically, StandardScaler is used with neural networks to standardize the input features, which ensures better and faster training.
• Standardization ensures that the data follows a standard normal distribution (approximately), which is particularly useful for algorithms sensitive to feature scaling like neural networks.

# Define the columns to scale
cont_columns = ["CreditScore", "Age", "Tenure", "Balance", "EstimatedSalary"]

# Initialize the StandardScaler
scaler = StandardScaler()

# Fit the scaler to the selected columns in the x_train data
scaler.fit(X_train[cont_columns])

StandardScaler()

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

• Once the scaler object fits on the data using the fit() method, it stores the parameters (mean and standard deviation) for normalization based on the training data.
• We then use these parameters to normalize the validation and test data.

# Transform selected columns in X_train, X_val, and X_test using the fitted scaler
X_train[cont_columns] = scaler.transform(X_train[cont_columns])
X_val[cont_columns] = scaler.transform(X_val[cont_columns])
X_test[cont_columns] = scaler.transform(X_test[cont_columns])

# Check 10 random rows from the train dataset after the transform
X_train.sample(n=10, random_state=1)

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
5364	1.68	1	1	-0.57	-1.04	0.82	2	0	1	0.09
6002	0.53	0	0	0.20	-1.04	-1.23	1	1	0	-0.91
5905	-1.44	0	0	-0.85	0.00	-1.23	2	1	0	0.77
6262	-1.42	0	1	0.49	-1.38	-0.21	2	0	1	-0.82
6036	0.25	0	0	-1.14	-0.34	0.04	1	0	1	-0.86
3066	-1.52	0	0	-1.05	0.00	-1.23	2	1	0	0.45
3136	-0.44	0	1	0.49	-1.73	-1.23	2	1	1	-0.33
2169	1.28	1	1	0.39	1.04	0.09	2	1	1	-1.11
2005	1.40	1	0	-0.66	-0.69	0.77	2	0	1	-0.82
1524	1.52	0	1	1.16	0.35	-1.23	2	0	1	1.65

# Check 10 random rows from the validation dataset after the transform
X_val.sample(n=10, random_state=1)

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary
4709	-0.99	0	1	-0.76	-1.38	-1.23	2	0	1	1.61
4883	-0.37	1	1	0.39	1.04	1.03	1	1	1	1.16
5858	-0.83	0	0	-0.85	1.74	1.60	1	1	0	-0.05
886	0.93	2	1	-0.09	-1.73	0.84	1	1	0	-1.52
5586	1.72	1	0	-1.33	-1.04	1.19	2	1	1	-1.12
4888	-2.20	0	1	0.96	1.39	0.57	2	0	1	1.16
721	-1.98	2	0	0.29	0.35	0.86	2	1	1	1.35
5368	-1.16	0	0	2.21	-0.69	-0.02	1	0	0	-1.43
8820	0.02	1	1	-0.28	1.39	1.20	1	0	0	-0.48
9657	-0.45	0	1	-0.66	0.00	0.11	1	1	1	0.29

👉 After standard scaling, the continuous numerical features are transformed to have a mean of 0 and a standard deviation of 1, ensuring all features are on a comparable scale and optimized for neural network training.

# Check class balance for whole data, train, validation, and test sets
print("Target value ratio in y")
print(y.value_counts(1))
print("*" * 80)
print("Target value ratio in y_train")
print(y_train.value_counts(1))
print("*" * 80)
print("Target value ratio in y_val")
print(y_val.value_counts(1))
print("*" * 80)
print("Target value ratio in y_test")
print(y_test.value_counts(1))
print("*" * 80)

Target value ratio in y
Exited
0   0.80
1   0.20
Name: proportion, dtype: float64
********************************************************************************
Target value ratio in y_train
Exited
0   0.80
1   0.20
Name: proportion, dtype: float64
********************************************************************************
Target value ratio in y_val
Exited
0   0.80
1   0.20
Name: proportion, dtype: float64
********************************************************************************
Target value ratio in y_test
Exited
0   0.80
1   0.20
Name: proportion, dtype: float64
********************************************************************************

• The same proportion of each class in the training, validation, and test sets (80% and 20%)

Model Building

Model Evaluation Criterion

The nature of predictions will be made by this classification model can be translated as follows:

• True positives (TP) are customers correctly predicted by the model to leave the bank.
• False negatives (FN) are customers who actually leave but are not predicted by the model to leave.
• False positives (FP) are customers incorrectly predicted to leave the bank but who actually stay.

Model can make wrong predictions as:

1. Predicting a customer is exiting and the customer is not exiting
2. Predicting a customer is not exiting and customer is exiting

Which case is more important?

• Predicting that a customer will not churn when they actually do can lead to significant losses for banks. This misclassification prevents the bank from taking proactive measures to retain these high-risk customers.

Which metric to optimize?

• The bank’s primary goal is to prevent customers from leaving by identifying those likely to cancel their services.
• In this case, the best evaluation metric would be Recall.
• High Recall (minimizing False Negatives) means the model can identify as many at-risk customers as possible. Failing to identify these customers (FN) means the bank cannot take proactive steps to retain them, which could result in significant revenue loss.
• While Precision (minimizing False Positives, where customers are wrongly flagged as at-risk) is important, the focus is more on Recall to ensure the bank does not miss potential churners.
• Therefore, Recall is the most critical metric for this task, although F1 Score could be considered if a balance between Precision and Recall is required. However, in this case, Recall remains the priority.

Unity Functions

• The following function visualizes the training and validation loss or the chosen metric (such as Recall) over epochs by plotting them on a graph:

def plot(history, name):
    """
    Function to plot loss & chosen metric

    history: an object which stores the metrics and losses.
    name: can be one of Loss or the metric
    """
    fig, ax = plt.subplots() #Creating a subplot with figure and axes
    plt.plot(history.history[name]) #Plotting the train metric or train loss
    plt.plot(history.history['val_'+name]) #Plotting the validation metric or validation loss

    plt.title('Model ' + name.capitalize()) #Defining the title of the plot
    plt.ylabel(name.capitalize()) #Capitalizing the first letter
    plt.xlabel('Epoch') #Defining the label for the X-axis
    fig.legend(['Train', 'Validation'], loc="upper right") #Defining the legend, loc controls the position of the legend

• We'll create a dataframe to store the results from all the models we build:

# Defining the columns of the dataframe which are nothing but the hyper parameters and the metrics
columns = ["Hidden Layer Count","Neuron Count","Activation Function","Batch Size","Epoch Count","Optimizer","Learning Rate","Class Balancer","Regularization","Train Loss","Validation Loss","Train Recall","Validation Recall","Time in secs"]

# Creating a pandas dataframe
results = pd.DataFrame(columns=columns)

Calculating Class Weight

• As we have are dealing with an imbalance in class distribution, we will be using class weights to allow the model to give proportionally more importance to the minority class.

# Calculate class weights for the imbalanced dataset
cw = (y_train.shape[0]) / (np.bincount(y_train) * 2)  # Divide by 2 to normalize the weights

# Create a dictionary mapping class indices (0 and 1) to their respective class weights
cw_dict = {i: weight for i, weight in enumerate(cw)}

cw_dict  # Check the output

{0: 0.6279153211338356, 1: 2.4544179523141656}

Neural Network with SGD Optimizer

We will begin with a baseline model configured as follows:

• Architecture: 1 input layer, 2 hidden layers, and 1 output layer.
• Activation Functions:
  • Hidden layers use the ReLU activation function.
  • Output layer uses the sigmoid activation function for binary classification.
• Optimizer: Stochastic Gradient Descent (SGD) without momentum.
• Weight Initialization: Default initializer for Dense layers in Keras is Glorot Uniform (also known as Xavier Uniform).

batch_size = 32  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_0 = Sequential()
model_0.add(Dense(64, activation="relu", input_dim=X_train.shape[1]))  # First hidden layer
model_0.add(Dense(32, activation="relu"))  # Second hidden layer
model_0.add(Dense(1, activation="sigmoid"))  # Output layer for binary classification

👉 We are using 64 and 32 neurons in the dense layers of our model. Increasing the number of neurons beyond these values did not improve the results, so we have chosen to use these configurations throughout the analysis.

model_0.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             704 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 2,817 (11.00 KB)

 Trainable params: 2,817 (11.00 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)    # defining SGD as the optimizer, no momentum defined
model_0.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

• By default, SGD optimizer in TensorFlow uses a learning rate of 0.01. But this can be adjusted.
• We use binary_crossentropy as the loss function in binary classification tasks because it measures the difference between the predicted probabilities and the actual binary labels (0 or 1), helping the model learn to make accurate predictions.

# Convert y_train to a NumPy Array
y_train = np.array(y_train).astype(int)

# Record the start time for training
start = time.time()
# Train the model using the training data and evaluate it on the validation set
history = model_0.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs, class_weight=cw_dict)
# Record the end time for training
end = time.time()

Epoch 1/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 4s 11ms/step - loss: 0.6609 - recall: 0.4074 - val_loss: 0.6411 - val_recall: 0.6863
Epoch 2/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6187 - recall: 0.6870 - val_loss: 0.6135 - val_recall: 0.6993
Epoch 3/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5950 - recall: 0.6832 - val_loss: 0.6001 - val_recall: 0.6928
Epoch 4/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5832 - recall: 0.6870 - val_loss: 0.5921 - val_recall: 0.7092
Epoch 5/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5768 - recall: 0.6973 - val_loss: 0.5876 - val_recall: 0.7157
Epoch 6/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5723 - recall: 0.7105 - val_loss: 0.5838 - val_recall: 0.7255
Epoch 7/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5688 - recall: 0.7219 - val_loss: 0.5810 - val_recall: 0.7288
Epoch 8/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5658 - recall: 0.7181 - val_loss: 0.5791 - val_recall: 0.7255
Epoch 9/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5630 - recall: 0.7194 - val_loss: 0.5772 - val_recall: 0.7255
Epoch 10/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5606 - recall: 0.7215 - val_loss: 0.5751 - val_recall: 0.7320
Epoch 11/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5583 - recall: 0.7211 - val_loss: 0.5735 - val_recall: 0.7353
Epoch 12/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5560 - recall: 0.7245 - val_loss: 0.5713 - val_recall: 0.7353
Epoch 13/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5537 - recall: 0.7260 - val_loss: 0.5692 - val_recall: 0.7353
Epoch 14/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5514 - recall: 0.7300 - val_loss: 0.5674 - val_recall: 0.7386
Epoch 15/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5492 - recall: 0.7333 - val_loss: 0.5654 - val_recall: 0.7386
Epoch 16/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5469 - recall: 0.7332 - val_loss: 0.5631 - val_recall: 0.7418
Epoch 17/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5446 - recall: 0.7336 - val_loss: 0.5612 - val_recall: 0.7484
Epoch 18/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5423 - recall: 0.7344 - val_loss: 0.5589 - val_recall: 0.7516
Epoch 19/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5400 - recall: 0.7377 - val_loss: 0.5565 - val_recall: 0.7549
Epoch 20/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5377 - recall: 0.7383 - val_loss: 0.5539 - val_recall: 0.7582
Epoch 21/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5353 - recall: 0.7431 - val_loss: 0.5515 - val_recall: 0.7614
Epoch 22/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5329 - recall: 0.7418 - val_loss: 0.5497 - val_recall: 0.7647
Epoch 23/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5304 - recall: 0.7429 - val_loss: 0.5478 - val_recall: 0.7582
Epoch 24/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5280 - recall: 0.7448 - val_loss: 0.5454 - val_recall: 0.7582
Epoch 25/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5255 - recall: 0.7444 - val_loss: 0.5431 - val_recall: 0.7549
Epoch 26/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5232 - recall: 0.7455 - val_loss: 0.5411 - val_recall: 0.7614
Epoch 27/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5208 - recall: 0.7498 - val_loss: 0.5385 - val_recall: 0.7582
Epoch 28/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5184 - recall: 0.7490 - val_loss: 0.5358 - val_recall: 0.7582
Epoch 29/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5160 - recall: 0.7498 - val_loss: 0.5327 - val_recall: 0.7582
Epoch 30/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5136 - recall: 0.7455 - val_loss: 0.5306 - val_recall: 0.7582
Epoch 31/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5112 - recall: 0.7469 - val_loss: 0.5292 - val_recall: 0.7582
Epoch 32/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5089 - recall: 0.7498 - val_loss: 0.5273 - val_recall: 0.7549
Epoch 33/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5067 - recall: 0.7502 - val_loss: 0.5259 - val_recall: 0.7549
Epoch 34/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5046 - recall: 0.7504 - val_loss: 0.5249 - val_recall: 0.7549
Epoch 35/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5026 - recall: 0.7507 - val_loss: 0.5233 - val_recall: 0.7549
Epoch 36/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5006 - recall: 0.7500 - val_loss: 0.5212 - val_recall: 0.7516
Epoch 37/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4986 - recall: 0.7527 - val_loss: 0.5195 - val_recall: 0.7549
Epoch 38/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4967 - recall: 0.7570 - val_loss: 0.5181 - val_recall: 0.7614
Epoch 39/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4948 - recall: 0.7599 - val_loss: 0.5162 - val_recall: 0.7647
Epoch 40/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4929 - recall: 0.7610 - val_loss: 0.5145 - val_recall: 0.7614
Epoch 41/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4911 - recall: 0.7590 - val_loss: 0.5125 - val_recall: 0.7582
Epoch 42/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4894 - recall: 0.7587 - val_loss: 0.5111 - val_recall: 0.7614
Epoch 43/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4876 - recall: 0.7585 - val_loss: 0.5087 - val_recall: 0.7614
Epoch 44/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4858 - recall: 0.7547 - val_loss: 0.5067 - val_recall: 0.7582
Epoch 45/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4841 - recall: 0.7536 - val_loss: 0.5055 - val_recall: 0.7614
Epoch 46/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4823 - recall: 0.7551 - val_loss: 0.5047 - val_recall: 0.7712
Epoch 47/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4806 - recall: 0.7599 - val_loss: 0.5028 - val_recall: 0.7680
Epoch 48/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4789 - recall: 0.7603 - val_loss: 0.5015 - val_recall: 0.7680
Epoch 49/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4772 - recall: 0.7607 - val_loss: 0.5006 - val_recall: 0.7680
Epoch 50/50
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4756 - recall: 0.7617 - val_loss: 0.4993 - val_recall: 0.7712

print("Time taken in seconds",end-start)

Time taken in seconds 36.328224897384644

plot(history, 'loss')

• Loss drops significantly during initial epochs, showing rapid learning.
• Validation loss stabilizes around epoch 5, consistently higher than training loss.
• Training loss decreases steadily; validation loss reduces more slowly, suggesting limited generalization.
• The small gap indicates potential for tuning to improve performance.

plot(history, 'recall')

• The recall for both training and validation fluctuates during the initial epochs, indicating instability in learning.
• After epoch 10, recall steadily improves for both training and validation, with some minor oscillations.
• Validation recall closely follows training recall, suggesting minimal overfitting and good generalization.
• The gradual upward trend indicates that the model is progressively learning to identify positive instances better.

results.loc[0] = [2,[64,32],["relu","relu"],batch_size,epochs,"sgd",[0.01],"class_weight","-",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33

• The model performs reasonably well, but recall fluctuates early on, likely due to SGD's lack of momentum, which limits stability in updates.
• With a learning rate of 0.01, SGD may be taking smaller steps near flatter regions of the loss landscape, slowing initial improvement.
• Adding momentum to SGD or switching to Adam can improve convergence speed and reduce oscillations in recall.
• Batch size of 32 is appropriate, but increasing it slightly may stabilize gradient updates, improving metric consistency.

Model Performance Improvement

Generally, to improve the performance of neural networks, we tune hyperparameters such as the number of layers, try different optimizers, add momentum, use regularization techniques like dropout, and balance the dataset.

Neural Network with Adam Optimizer

We will use Adam optimizer this time configured as follows:

• Architecture: 1 input layer, 2 hidden layers, and 1 output layer.
• Activation Functions:
  • ReLU activation for the first hidden layer.
  • Tanh activation for the second hidden layer.
• Optimizer: Adam optimizer with momentum and an adaptive learning rate for efficient convergence.
• Regularization: L2 regularization helps reduce overfitting by penalizing large weights.

👉 Adam optimization is widely used in neural networks because it combines the benefits of momentum and adaptive learning rates, making it efficient, robust, and well-suited for a wide range of neural network architectures and datasets.

👉 The Adam optimizer does not use explicit momentum as in the traditional SGD with momentum. Instead, it incorporates momentum-like behavior through the use of exponentially decaying moving averages of past gradients and squared gradients.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_1 = Sequential()
model_1.add(Dense(64, activation='relu', kernel_regularizer=regularizers.l2(0.01), input_dim=X_train.shape[1]))  # First layer ReLu with L2 regularization
model_1.add(Dense(32, activation='tanh', kernel_regularizer=regularizers.l2(0.01)))  # Second layer tanh with L2 regularization
model_1.add(Dense(1, activation="sigmoid"))  # Output layer for binary classification

model_1.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             704 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 2,817 (11.00 KB)

 Trainable params: 2,817 (11.00 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)    # defining Adam as the optimizer
model_1.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

• By default, Adam uses a learning rate of 0.001, but this can be adjusted.
• A small learning rate can help achieve more precise updates and prevent overshooting the optimal point, but it might make the training slower.

# Record the start time for training
start = time.time()
# Train the model using the training data and evaluate it on the validation set
history = model_1.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs, class_weight=cw_dict)
# Record the end time for training
end = time.time()

Epoch 1/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 4s 19ms/step - loss: 1.1613 - recall: 0.5926 - val_loss: 0.9100 - val_recall: 0.6928
Epoch 2/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.8547 - recall: 0.7023 - val_loss: 0.7630 - val_recall: 0.7157
Epoch 3/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.7289 - recall: 0.7123 - val_loss: 0.6923 - val_recall: 0.7222
Epoch 4/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6694 - recall: 0.7158 - val_loss: 0.6539 - val_recall: 0.7157
Epoch 5/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6379 - recall: 0.7195 - val_loss: 0.6313 - val_recall: 0.7124
Epoch 6/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6190 - recall: 0.7229 - val_loss: 0.6185 - val_recall: 0.7222
Epoch 7/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6061 - recall: 0.7265 - val_loss: 0.6115 - val_recall: 0.7353
Epoch 8/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5961 - recall: 0.7379 - val_loss: 0.6085 - val_recall: 0.7484
Epoch 9/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5874 - recall: 0.7426 - val_loss: 0.6051 - val_recall: 0.7647
Epoch 10/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5792 - recall: 0.7466 - val_loss: 0.6020 - val_recall: 0.7745
Epoch 11/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5710 - recall: 0.7471 - val_loss: 0.5994 - val_recall: 0.7778
Epoch 12/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5627 - recall: 0.7542 - val_loss: 0.5937 - val_recall: 0.7974
Epoch 13/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5548 - recall: 0.7609 - val_loss: 0.5864 - val_recall: 0.7974
Epoch 14/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5477 - recall: 0.7602 - val_loss: 0.5783 - val_recall: 0.8007
Epoch 15/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5411 - recall: 0.7608 - val_loss: 0.5712 - val_recall: 0.8072
Epoch 16/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5357 - recall: 0.7635 - val_loss: 0.5642 - val_recall: 0.8072
Epoch 17/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5308 - recall: 0.7544 - val_loss: 0.5574 - val_recall: 0.7974
Epoch 18/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5267 - recall: 0.7583 - val_loss: 0.5524 - val_recall: 0.7876
Epoch 19/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5231 - recall: 0.7593 - val_loss: 0.5482 - val_recall: 0.7843
Epoch 20/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5198 - recall: 0.7603 - val_loss: 0.5442 - val_recall: 0.7876
Epoch 21/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5169 - recall: 0.7598 - val_loss: 0.5404 - val_recall: 0.7810
Epoch 22/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5143 - recall: 0.7597 - val_loss: 0.5365 - val_recall: 0.7778
Epoch 23/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5120 - recall: 0.7597 - val_loss: 0.5350 - val_recall: 0.7810
Epoch 24/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5097 - recall: 0.7608 - val_loss: 0.5315 - val_recall: 0.7810
Epoch 25/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5078 - recall: 0.7618 - val_loss: 0.5289 - val_recall: 0.7810
Epoch 26/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5061 - recall: 0.7636 - val_loss: 0.5254 - val_recall: 0.7843
Epoch 27/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5045 - recall: 0.7665 - val_loss: 0.5233 - val_recall: 0.7876
Epoch 28/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5029 - recall: 0.7630 - val_loss: 0.5205 - val_recall: 0.7810
Epoch 29/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5015 - recall: 0.7640 - val_loss: 0.5181 - val_recall: 0.7778
Epoch 30/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5000 - recall: 0.7645 - val_loss: 0.5166 - val_recall: 0.7745
Epoch 31/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4988 - recall: 0.7624 - val_loss: 0.5151 - val_recall: 0.7745
Epoch 32/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4975 - recall: 0.7620 - val_loss: 0.5131 - val_recall: 0.7745
Epoch 33/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4963 - recall: 0.7616 - val_loss: 0.5114 - val_recall: 0.7745
Epoch 34/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4952 - recall: 0.7576 - val_loss: 0.5104 - val_recall: 0.7745
Epoch 35/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4943 - recall: 0.7575 - val_loss: 0.5087 - val_recall: 0.7745
Epoch 36/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4932 - recall: 0.7591 - val_loss: 0.5073 - val_recall: 0.7745
Epoch 37/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4924 - recall: 0.7580 - val_loss: 0.5060 - val_recall: 0.7712
Epoch 38/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4915 - recall: 0.7575 - val_loss: 0.5048 - val_recall: 0.7614
Epoch 39/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4908 - recall: 0.7564 - val_loss: 0.5035 - val_recall: 0.7647
Epoch 40/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4900 - recall: 0.7548 - val_loss: 0.5035 - val_recall: 0.7614
Epoch 41/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4893 - recall: 0.7505 - val_loss: 0.5016 - val_recall: 0.7614
Epoch 42/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4886 - recall: 0.7438 - val_loss: 0.5011 - val_recall: 0.7549
Epoch 43/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4878 - recall: 0.7503 - val_loss: 0.5006 - val_recall: 0.7549
Epoch 44/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4871 - recall: 0.7435 - val_loss: 0.4991 - val_recall: 0.7582
Epoch 45/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4865 - recall: 0.7455 - val_loss: 0.4995 - val_recall: 0.7549
Epoch 46/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4858 - recall: 0.7459 - val_loss: 0.4984 - val_recall: 0.7549
Epoch 47/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4852 - recall: 0.7468 - val_loss: 0.4969 - val_recall: 0.7549
Epoch 48/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4847 - recall: 0.7421 - val_loss: 0.4960 - val_recall: 0.7549
Epoch 49/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4841 - recall: 0.7433 - val_loss: 0.4959 - val_recall: 0.7549
Epoch 50/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4835 - recall: 0.7462 - val_loss: 0.4957 - val_recall: 0.7516

print("Time taken in seconds",end-start)

Time taken in seconds 22.123973846435547

plot(history, 'loss')

• Improved Convergence: Both training and validation losses show a steady decline, indicating that the model is learning effectively after the initial epochs.
• Reduced Overfitting: The gap between training and validation losses is small, suggesting that L2 regularization and the Adam optimizer help control overfitting.
• Tanh Stability: The use of tanh in the second layer does not cause noticeable instability, showing that it complements the model structure well for this dataset.
• Optimization Efficiency: The rapid decrease in loss during early epochs demonstrates that the Adam optimizer, combined with batch size adjustments, accelerates convergence compared to the previous configuration.

plot(history, 'recall')

• Initial Improvement: A significant increase in training recall is observed in the first few epochs, suggesting that the model quickly learned to identify relevant patterns for positive class predictions.
• Validation Recall Spike: The validation recall peaks around the 10th epoch but then decreases, possibly indicating model instability or an imbalance in the validation data for specific epochs.
• Stabilization Post-Peak: After the initial fluctuations, both training and validation recall stabilize, with the training recall consistently slightly higher than validation recall.
• Close Convergence: Despite some minor fluctuations, the training and validation recalls converge closely toward the end of training, suggesting good model generalization without significant overfitting.

results.loc[1] = [2,[64,32],["relu","tanh"],batch_size,epochs,"adam",[0.001],"class_weight","L2",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12

👉 Model 1 demonstrates improved performance compared to Model 0. It achieves lower validation loss and higher recall scores on both training and validation datasets, indicating better generalization and effective identification of the positive class.

Neural Network with Adam Optimizer and Dropout

We will introduce Dropout for regularization, configured as follows:

• Architecture: 1 input layer, 2 hidden layers, and 1 output layer.
• Activations: ReLU for the first hidden layer and tanh for the second hidden layer.
• Optimizer: Adam optimizer with dropout applied (20%) in between hidden layers for regularization.
• Regularization: Combine L2 regularization with Dropout to address overfitting.

👉 Dropout is a regularization technique that randomly deactivates a fraction of neurons during training to prevent overfitting, and it complements the Adam optimizer by helping maintain generalization while Adam efficiently adjusts weights.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_2 = Sequential()

# First hidden layer with ReLU activation and L2 regularization
model_2.add(Dense(64, activation="relu", input_dim=X_train.shape[1], kernel_regularizer=regularizers.l2(0.001)))
model_2.add(Dropout(0.2))  # 20% Dropout to prevent overfitting

# Second hidden layer with tanh activation and L2 regularization
model_2.add(Dense(32, activation="tanh", kernel_regularizer=regularizers.l2(0.001)))
# Output layer for binary classification
model_2.add(Dense(1, activation="sigmoid"))

• 20% of the neurons in the layer will be randomly "dropped out" on each forward pass.

model_2.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             704 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 64)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 2,817 (11.00 KB)

 Trainable params: 2,817 (11.00 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)    # defining Adam as the optimizer
model_2.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

# Record the start time for training
start = time.time()
# Train the model using the training data and evaluate it on the validation set
history = model_2.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs, class_weight=cw_dict)
# Record the end time for training
end = time.time()

Epoch 1/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 6s 27ms/step - loss: 0.7242 - recall: 0.5767 - val_loss: 0.6320 - val_recall: 0.6699
Epoch 2/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6398 - recall: 0.6775 - val_loss: 0.6214 - val_recall: 0.6928
Epoch 3/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6225 - recall: 0.6871 - val_loss: 0.6099 - val_recall: 0.6895
Epoch 4/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6086 - recall: 0.7012 - val_loss: 0.5984 - val_recall: 0.7059
Epoch 5/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5961 - recall: 0.7079 - val_loss: 0.5948 - val_recall: 0.7092
Epoch 6/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5889 - recall: 0.7150 - val_loss: 0.5901 - val_recall: 0.7222
Epoch 7/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5811 - recall: 0.7099 - val_loss: 0.5821 - val_recall: 0.7190
Epoch 8/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5733 - recall: 0.7199 - val_loss: 0.5715 - val_recall: 0.7190
Epoch 9/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5649 - recall: 0.7054 - val_loss: 0.5663 - val_recall: 0.7222
Epoch 10/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5606 - recall: 0.7183 - val_loss: 0.5632 - val_recall: 0.7353
Epoch 11/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5561 - recall: 0.7177 - val_loss: 0.5621 - val_recall: 0.7386
Epoch 12/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5500 - recall: 0.7183 - val_loss: 0.5641 - val_recall: 0.7386
Epoch 13/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5451 - recall: 0.7259 - val_loss: 0.5571 - val_recall: 0.7418
Epoch 14/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5405 - recall: 0.7308 - val_loss: 0.5511 - val_recall: 0.7451
Epoch 15/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5314 - recall: 0.7270 - val_loss: 0.5434 - val_recall: 0.7320
Epoch 16/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5303 - recall: 0.7387 - val_loss: 0.5452 - val_recall: 0.7484
Epoch 17/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5302 - recall: 0.7344 - val_loss: 0.5439 - val_recall: 0.7418
Epoch 18/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5300 - recall: 0.7381 - val_loss: 0.5380 - val_recall: 0.7484
Epoch 19/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5270 - recall: 0.7487 - val_loss: 0.5475 - val_recall: 0.7516
Epoch 20/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5246 - recall: 0.7348 - val_loss: 0.5378 - val_recall: 0.7451
Epoch 21/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5197 - recall: 0.7274 - val_loss: 0.5304 - val_recall: 0.7418
Epoch 22/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5201 - recall: 0.7379 - val_loss: 0.5275 - val_recall: 0.7484
Epoch 23/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5218 - recall: 0.7285 - val_loss: 0.5277 - val_recall: 0.7484
Epoch 24/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5119 - recall: 0.7569 - val_loss: 0.5196 - val_recall: 0.7484
Epoch 25/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5095 - recall: 0.7344 - val_loss: 0.5272 - val_recall: 0.7418
Epoch 26/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5048 - recall: 0.7586 - val_loss: 0.5191 - val_recall: 0.7484
Epoch 27/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5075 - recall: 0.7477 - val_loss: 0.5115 - val_recall: 0.7353
Epoch 28/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5086 - recall: 0.7375 - val_loss: 0.5076 - val_recall: 0.7451
Epoch 29/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4963 - recall: 0.7475 - val_loss: 0.5077 - val_recall: 0.7484
Epoch 30/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4932 - recall: 0.7666 - val_loss: 0.5084 - val_recall: 0.7549
Epoch 31/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5017 - recall: 0.7404 - val_loss: 0.5149 - val_recall: 0.7516
Epoch 32/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4888 - recall: 0.7533 - val_loss: 0.5108 - val_recall: 0.7647
Epoch 33/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4881 - recall: 0.7554 - val_loss: 0.4946 - val_recall: 0.7353
Epoch 34/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4899 - recall: 0.7632 - val_loss: 0.4889 - val_recall: 0.7353
Epoch 35/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4872 - recall: 0.7525 - val_loss: 0.4894 - val_recall: 0.7484
Epoch 36/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4846 - recall: 0.7622 - val_loss: 0.4943 - val_recall: 0.7516
Epoch 37/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4845 - recall: 0.7638 - val_loss: 0.4882 - val_recall: 0.7451
Epoch 38/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4858 - recall: 0.7499 - val_loss: 0.4974 - val_recall: 0.7516
Epoch 39/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4859 - recall: 0.7535 - val_loss: 0.4964 - val_recall: 0.7516
Epoch 40/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4783 - recall: 0.7627 - val_loss: 0.4855 - val_recall: 0.7386
Epoch 41/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4739 - recall: 0.7656 - val_loss: 0.4967 - val_recall: 0.7549
Epoch 42/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4726 - recall: 0.7815 - val_loss: 0.4894 - val_recall: 0.7549
Epoch 43/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4786 - recall: 0.7584 - val_loss: 0.4788 - val_recall: 0.7320
Epoch 44/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4792 - recall: 0.7568 - val_loss: 0.4911 - val_recall: 0.7418
Epoch 45/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4804 - recall: 0.7489 - val_loss: 0.4814 - val_recall: 0.7418
Epoch 46/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4734 - recall: 0.7670 - val_loss: 0.4799 - val_recall: 0.7418
Epoch 47/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4792 - recall: 0.7686 - val_loss: 0.4856 - val_recall: 0.7582
Epoch 48/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4791 - recall: 0.7584 - val_loss: 0.4828 - val_recall: 0.7418
Epoch 49/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4750 - recall: 0.7504 - val_loss: 0.4722 - val_recall: 0.7451
Epoch 50/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4700 - recall: 0.7677 - val_loss: 0.4881 - val_recall: 0.7549

print("Time taken in seconds",end-start)

Time taken in seconds 25.31906795501709

plot(history, 'loss')

• Effective Learning: The training loss shows a steady and significant decrease, particularly in the early epochs, indicating that the model is effectively learning from the training data.
• Good Generalization: The validation loss closely tracks the training loss and even converges below the training loss in later epochs, suggesting good generalization capabilities without significant overfitting.
• Stable Convergence: Both training and validation losses stabilize around epoch 20 and remain relatively flat thereafter, indicating that the model has likely converged to an optimal point.

plot(history, 'recall')

• Initial Improvement: There's a sharp increase in recall for the training data in the early epochs, indicating rapid learning in identifying the positive class.
• Fluctuations: Both training and validation recall exhibit fluctuations throughout the epochs. This could suggest model sensitivity to specific batches of data or the presence of noise.
• Convergence: Despite the fluctuations, the validation recall generally follows the training recall, indicating that the model is generalizing well across unseen data.

results.loc[2] = [2,[64,32],["relu","tanh"],batch_size,epochs,"adam",[0.001],"class_weight","L2, dropout",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12
2	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2, dropout	0.48	0.49	0.76	0.75	25.32

👉 There is a progressive increase in recall from the 1st to the 3rd model. The 3rd model records the highest recall for both training (0.76) and validation (0.74), showcasing its enhanced ability to correctly identify the positive class.
👉 The 3rd model demonstrates superior performance in balancing loss reduction and recall enhancement, illustrating the benefits of combining dropout with the Adam optimizer and a batch size of 64 in a balanced class setting.

Neural Network with Adam Optimizer and Dropout (Three layers)

We will modify the architecture with the following updates:

• Architecture: 1 input layer, 3 hidden layers, and 1 output layer.
• Activations: Add an additional ReLU activation for the hidden layers.
• Learning Rate: Decrease to 0.0005 for more stable convergence.
• Regularization: Include extra BatchNormalization and extra Dropout layers with L2 regularization.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_3 = Sequential()
model_3.add(Dense(32, activation="relu", input_dim=X_train.shape[1], kernel_regularizer=regularizers.l2(0.0005)))  # First hidden layer ReLu with L2
model_3.add(BatchNormalization())  # Batch normalization 1
model_3.add(Dropout(0.2))  # Dropout to prevent overfitting

model_3.add(Dense(32, activation="relu", kernel_regularizer=regularizers.l2(0.0005)))  # Second hidden layer ReLu activation with L2
model_3.add(BatchNormalization())  # Batch normalization 2
model_3.add(Dropout(0.2))  # Dropout

model_3.add(Dense(16, activation="tanh", kernel_regularizer=regularizers.l2(0.0005)))  # Third hidden layer tanh activation with L2
model_3.add(BatchNormalization())  # Batch normalization 2
model_3.add(Dropout(0.3))  # Increased dropout for the last hidden layer
model_3.add(Dense(1, activation="sigmoid"))  # Output layer for binary classification

model_3.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 32)                  │             352 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ batch_normalization                  │ (None, 32)                  │             128 │
│ (BatchNormalization)                 │                             │                 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           1,056 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ batch_normalization_1                │ (None, 32)                  │             128 │
│ (BatchNormalization)                 │                             │                 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_1 (Dropout)                  │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 16)                  │             528 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ batch_normalization_2                │ (None, 16)                  │              64 │
│ (BatchNormalization)                 │                             │                 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_2 (Dropout)                  │ (None, 16)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_3 (Dense)                      │ (None, 1)                   │              17 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 2,273 (8.88 KB)

 Trainable params: 2,113 (8.25 KB)

 Non-trainable params: 160 (640.00 B)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.0005)    # defining Adam as the optimizer with lr of 0.0005
model_3.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

👉 Decreasing the learning rate stabilizes training and enables fine-tuning near the loss minimum but can slow convergence.

# Record the start time for training
start = time.time()
# Train the model using the training data and evaluate it on the validation set
history = model_3.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs, class_weight=cw_dict)
# Record the end time for training
end = time.time()

Epoch 1/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 10s 40ms/step - loss: 0.8365 - recall: 0.5800 - val_loss: 0.6372 - val_recall: 0.5588
Epoch 2/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 2s 3ms/step - loss: 0.7255 - recall: 0.6339 - val_loss: 0.6393 - val_recall: 0.7222
Epoch 3/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6980 - recall: 0.6565 - val_loss: 0.6287 - val_recall: 0.6993
Epoch 4/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6783 - recall: 0.6879 - val_loss: 0.6256 - val_recall: 0.6961
Epoch 5/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6608 - recall: 0.6675 - val_loss: 0.6215 - val_recall: 0.7124
Epoch 6/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6535 - recall: 0.6702 - val_loss: 0.6204 - val_recall: 0.7124
Epoch 7/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6616 - recall: 0.6632 - val_loss: 0.6087 - val_recall: 0.7026
Epoch 8/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6310 - recall: 0.6825 - val_loss: 0.6039 - val_recall: 0.6961
Epoch 9/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6275 - recall: 0.7045 - val_loss: 0.6025 - val_recall: 0.7092
Epoch 10/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6290 - recall: 0.6868 - val_loss: 0.5981 - val_recall: 0.6895
Epoch 11/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.6248 - recall: 0.6807 - val_loss: 0.5995 - val_recall: 0.6928
Epoch 12/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.6193 - recall: 0.7148 - val_loss: 0.5957 - val_recall: 0.7059
Epoch 13/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6091 - recall: 0.7204 - val_loss: 0.5904 - val_recall: 0.7092
Epoch 14/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.6089 - recall: 0.7264 - val_loss: 0.5884 - val_recall: 0.6961
Epoch 15/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.6005 - recall: 0.7121 - val_loss: 0.5835 - val_recall: 0.6993
Epoch 16/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.5977 - recall: 0.7249 - val_loss: 0.5791 - val_recall: 0.6993
Epoch 17/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6035 - recall: 0.7025 - val_loss: 0.5710 - val_recall: 0.6928
Epoch 18/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.6023 - recall: 0.7252 - val_loss: 0.5683 - val_recall: 0.6928
Epoch 19/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5930 - recall: 0.7122 - val_loss: 0.5626 - val_recall: 0.6928
Epoch 20/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5816 - recall: 0.7253 - val_loss: 0.5586 - val_recall: 0.6993
Epoch 21/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5897 - recall: 0.7214 - val_loss: 0.5539 - val_recall: 0.7124
Epoch 22/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5743 - recall: 0.7373 - val_loss: 0.5453 - val_recall: 0.7124
Epoch 23/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5766 - recall: 0.7317 - val_loss: 0.5456 - val_recall: 0.7190
Epoch 24/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5733 - recall: 0.7164 - val_loss: 0.5437 - val_recall: 0.7353
Epoch 25/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5602 - recall: 0.7443 - val_loss: 0.5372 - val_recall: 0.7353
Epoch 26/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5540 - recall: 0.7469 - val_loss: 0.5297 - val_recall: 0.7353
Epoch 27/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5603 - recall: 0.7307 - val_loss: 0.5209 - val_recall: 0.7353
Epoch 28/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5555 - recall: 0.7479 - val_loss: 0.5114 - val_recall: 0.7255
Epoch 29/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5436 - recall: 0.7556 - val_loss: 0.5074 - val_recall: 0.7255
Epoch 30/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5473 - recall: 0.7328 - val_loss: 0.5092 - val_recall: 0.7353
Epoch 31/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5423 - recall: 0.7410 - val_loss: 0.5092 - val_recall: 0.7484
Epoch 32/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5379 - recall: 0.7482 - val_loss: 0.4993 - val_recall: 0.7320
Epoch 33/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5279 - recall: 0.7495 - val_loss: 0.4975 - val_recall: 0.7484
Epoch 34/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5351 - recall: 0.7537 - val_loss: 0.4934 - val_recall: 0.7418
Epoch 35/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5379 - recall: 0.7458 - val_loss: 0.4895 - val_recall: 0.7288
Epoch 36/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5374 - recall: 0.7297 - val_loss: 0.4927 - val_recall: 0.7353
Epoch 37/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5364 - recall: 0.7275 - val_loss: 0.4916 - val_recall: 0.7418
Epoch 38/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5349 - recall: 0.7307 - val_loss: 0.4918 - val_recall: 0.7451
Epoch 39/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5280 - recall: 0.7545 - val_loss: 0.4818 - val_recall: 0.7255
Epoch 40/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5199 - recall: 0.7502 - val_loss: 0.4829 - val_recall: 0.7418
Epoch 41/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5177 - recall: 0.7575 - val_loss: 0.4854 - val_recall: 0.7418
Epoch 42/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5208 - recall: 0.7534 - val_loss: 0.4808 - val_recall: 0.7190
Epoch 43/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5166 - recall: 0.7456 - val_loss: 0.4814 - val_recall: 0.7320
Epoch 44/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5205 - recall: 0.7394 - val_loss: 0.4831 - val_recall: 0.7353
Epoch 45/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5044 - recall: 0.7785 - val_loss: 0.4777 - val_recall: 0.7288
Epoch 46/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.5135 - recall: 0.7447 - val_loss: 0.4839 - val_recall: 0.7418
Epoch 47/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5019 - recall: 0.7674 - val_loss: 0.4764 - val_recall: 0.7255
Epoch 48/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5070 - recall: 0.7488 - val_loss: 0.4743 - val_recall: 0.7222
Epoch 49/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5143 - recall: 0.7446 - val_loss: 0.4729 - val_recall: 0.7157
Epoch 50/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5000 - recall: 0.7504 - val_loss: 0.4747 - val_recall: 0.7190

print("Time taken in seconds",end-start)

Time taken in seconds 35.123931646347046

plot(history,'loss')

• Rapid Initial Decrease: A sharp decrease in both training and validation loss in the early epochs, indicating that the model is quickly learning from the training data.
• Convergence: Both curves show a gradual convergence, with the training loss plateauing and the validation loss stabilizing suggests that the model is not overfitting significantly, as the validation loss remains close to the training loss throughout the epochs.
• Validation Loss: The validation loss closely follows the training loss, sometimes even dipping below, which is a good sign of the model's generalization ability.
• Stability: After epoch 30, both training and validation losses appear relatively flat with minor fluctuations, indicating that the model has mostly converged and additional training may yield diminishing returns.

plot(history,'recall')

• Sharp Initial Increase: Recall for both training and validation rises sharply early on, indicating quick learning in identifying the positive class.
• Stabilization and Divergence: While training recall remains high and stable above 0.8, validation recall stabilizes around 0.75, suggesting a gap that may indicate some overfitting.
• Fluctuations in Validation Recall: The fluctuations in validation recall suggest sensitivity to specific validation data features or instances.

results.loc[3] = [3,[32,32,16],["relu","relu","tanh"],batch_size,epochs,"adam",[0.0005],"class_weight","L2, dropout, batch norm",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12
2	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2, dropout	0.48	0.49	0.76	0.75	25.32
3	3	[32, 32, 16]	[relu, relu, tanh]	64	50	adam	[0.0005]	class_weight	L2, dropout, batch norm	0.52	0.47	0.74	0.72	35.12

👉 The last model shows a significant improvement in training recall, reaching 79.87%, demonstrating enhanced ability to identify the positive class in the training set.
👉 However, the validation recall remains unchanged at 72.55%, and there is a slight increase in validation loss, suggesting potential challenges in model generalization.

Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer

We will modify our first model (Neural Network with SGD Optimizer) to apply SMOTE for balancing the data:

• To incorporate SMOTE (Synthetic Minority Oversampling Technique) for balancing data, we will apply SMOTE to the training data before training the model.
• We will only add L2 regularization to the first model.

# Apply SMOTE to balance the training data
smote = SMOTE(random_state=11)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)

batch_size = 64  # Number of samples processed before the model is updated
epochs = 40  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initialize the neural network
model_4 = Sequential()
model_4.add(Dense(32, activation="relu", input_dim=X_train_smote.shape[1], kernel_regularizer=regularizers.l2(0.0005)))  # First hidden layer with L2
model_4.add(Dense(16, activation="relu", kernel_regularizer=regularizers.l2(0.0005)))  # Second hidden layer with L2
model_4.add(Dense(1, activation="sigmoid"))  # Output layer for binary classification

model_4.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 32)                  │             352 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 16)                  │             528 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              17 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 897 (3.50 KB)

 Trainable params: 897 (3.50 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01) # SGD optimizer without momentum
model_4.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

👉 We are not using class_weight when applying SMOTE because SMOTE already balances the dataset by generating synthetic samples for the minority class, making class weighting unnecessary.

# Record the start time for training
start = time.time()
# Train the model using the SMOTE-balanced training data and validate it on the original validation set
history = model_4.fit(X_train_smote, y_train_smote, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs)
# Record the end time for training
end = time.time()

Epoch 1/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 2s 7ms/step - loss: 0.7209 - recall: 0.6441 - val_loss: 0.6707 - val_recall: 0.6340
Epoch 2/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6698 - recall: 0.6272 - val_loss: 0.6313 - val_recall: 0.6340
Epoch 3/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6404 - recall: 0.6522 - val_loss: 0.6082 - val_recall: 0.6209
Epoch 4/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6194 - recall: 0.6719 - val_loss: 0.5944 - val_recall: 0.6373
Epoch 5/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.6039 - recall: 0.6927 - val_loss: 0.5857 - val_recall: 0.6503
Epoch 6/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5920 - recall: 0.7057 - val_loss: 0.5793 - val_recall: 0.6503
Epoch 7/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5826 - recall: 0.7164 - val_loss: 0.5735 - val_recall: 0.6503
Epoch 8/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5753 - recall: 0.7250 - val_loss: 0.5691 - val_recall: 0.6569
Epoch 9/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5696 - recall: 0.7281 - val_loss: 0.5663 - val_recall: 0.6667
Epoch 10/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5648 - recall: 0.7338 - val_loss: 0.5637 - val_recall: 0.6732
Epoch 11/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5608 - recall: 0.7368 - val_loss: 0.5614 - val_recall: 0.6797
Epoch 12/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5574 - recall: 0.7396 - val_loss: 0.5593 - val_recall: 0.6895
Epoch 13/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5544 - recall: 0.7433 - val_loss: 0.5579 - val_recall: 0.6895
Epoch 14/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5518 - recall: 0.7467 - val_loss: 0.5565 - val_recall: 0.6863
Epoch 15/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5494 - recall: 0.7480 - val_loss: 0.5554 - val_recall: 0.6895
Epoch 16/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5472 - recall: 0.7492 - val_loss: 0.5544 - val_recall: 0.6895
Epoch 17/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5453 - recall: 0.7482 - val_loss: 0.5532 - val_recall: 0.6863
Epoch 18/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5434 - recall: 0.7481 - val_loss: 0.5524 - val_recall: 0.6830
Epoch 19/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5417 - recall: 0.7470 - val_loss: 0.5517 - val_recall: 0.6830
Epoch 20/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5400 - recall: 0.7471 - val_loss: 0.5511 - val_recall: 0.6797
Epoch 21/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5385 - recall: 0.7475 - val_loss: 0.5497 - val_recall: 0.6765
Epoch 22/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5370 - recall: 0.7470 - val_loss: 0.5487 - val_recall: 0.6765
Epoch 23/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5356 - recall: 0.7466 - val_loss: 0.5478 - val_recall: 0.6732
Epoch 24/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5342 - recall: 0.7471 - val_loss: 0.5469 - val_recall: 0.6797
Epoch 25/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5329 - recall: 0.7470 - val_loss: 0.5460 - val_recall: 0.6732
Epoch 26/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5316 - recall: 0.7475 - val_loss: 0.5450 - val_recall: 0.6732
Epoch 27/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5303 - recall: 0.7473 - val_loss: 0.5440 - val_recall: 0.6699
Epoch 28/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5292 - recall: 0.7472 - val_loss: 0.5430 - val_recall: 0.6667
Epoch 29/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5280 - recall: 0.7476 - val_loss: 0.5422 - val_recall: 0.6667
Epoch 30/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5269 - recall: 0.7458 - val_loss: 0.5413 - val_recall: 0.6667
Epoch 31/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5258 - recall: 0.7441 - val_loss: 0.5405 - val_recall: 0.6634
Epoch 32/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5247 - recall: 0.7419 - val_loss: 0.5393 - val_recall: 0.6634
Epoch 33/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5236 - recall: 0.7417 - val_loss: 0.5383 - val_recall: 0.6634
Epoch 34/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5226 - recall: 0.7419 - val_loss: 0.5374 - val_recall: 0.6634
Epoch 35/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5215 - recall: 0.7414 - val_loss: 0.5364 - val_recall: 0.6634
Epoch 36/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5204 - recall: 0.7403 - val_loss: 0.5355 - val_recall: 0.6601
Epoch 37/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.5194 - recall: 0.7400 - val_loss: 0.5344 - val_recall: 0.6634
Epoch 38/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5183 - recall: 0.7404 - val_loss: 0.5335 - val_recall: 0.6667
Epoch 39/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5173 - recall: 0.7415 - val_loss: 0.5325 - val_recall: 0.6667
Epoch 40/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5162 - recall: 0.7433 - val_loss: 0.5312 - val_recall: 0.6667

print("Time taken in seconds",end-start)

Time taken in seconds 24.38268780708313

plot(history, 'loss')

• Steady Decrease: Both training and validation losses show a consistent decline throughout the epochs, suggesting that the model is effectively learning and improving over time.
• Convergence: The training and validation loss lines are close and parallel, particularly in later epochs, indicating good model generalization without significant overfitting.

plot(history, 'recall')

• Training Recall: Shows a steady and significant increase, indicating that the model is progressively improving its ability to correctly identify all relevant instances from the training data.
• Validation Recall: However, remains relatively flat and demonstrates a downward trend after initial epochs, suggesting that the model is not performing as well on the validation set, potentially due to overfitting or the model not generalizing well to new data.
• The considerable gap between training and validation recall indicates that further model adjustments or data handling strategies might be needed to enhance how the model performs on unseen data.

results.loc[4] = [2,[32,16],["relu","relu"],batch_size,epochs,"sgd",[0.01],"SMOTE","L2",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12
2	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2, dropout	0.48	0.49	0.76	0.75	25.32
3	3	[32, 32, 16]	[relu, relu, tanh]	64	50	adam	[0.0005]	class_weight	L2, dropout, batch norm	0.52	0.47	0.74	0.72	35.12
4	2	[32, 16]	[relu, relu]	64	40	sgd	[0.01]	SMOTE	L2	0.51	0.53	0.75	0.67	24.38

👉 The latest model with SMOTE for class balancing shows mixed results: while the training recall improves significantly, the validation recall drops notably, suggesting the model may be overfitting or not generalizing well to unseen data.
👉 The increase in both training and validation loss compared to previous models further supports the need for adjustments to enhance model stability and generalization.

Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer

We will now, modify our second model (Neural Network with Adam Optimizer) to apply SMOTE for balancing the data. All other model architecture will remain the same.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 40  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_5 = Sequential()
model_5.add(Dense(32, activation="relu", input_dim=X_train.shape[1], kernel_regularizer=regularizers.l2(0.001)))  # First hidden layer with L2
model_5.add(Dense(16, activation="tanh", kernel_regularizer=regularizers.l2(0.001)))  # Second hidden layer with tanh activation & L2
model_5.add(Dense(1, activation="sigmoid"))  # Output layer for binary classification

model_5.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 32)                  │             352 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 16)                  │             528 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              17 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 897 (3.50 KB)

 Trainable params: 897 (3.50 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)    # Adam optimizer
model_5.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

# Record the start time for training
start = time.time()
# Train the model using the SMOTE-balanced training data and validate it on the original validation set
history = model_5.fit(X_train_smote, y_train_smote, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs)
# Record the end time for training
end = time.time()

Epoch 1/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 5s 16ms/step - loss: 0.6699 - recall: 0.5144 - val_loss: 0.6056 - val_recall: 0.6797
Epoch 2/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5840 - recall: 0.6995 - val_loss: 0.5912 - val_recall: 0.7026
Epoch 3/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5625 - recall: 0.7259 - val_loss: 0.5731 - val_recall: 0.7059
Epoch 4/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5453 - recall: 0.7378 - val_loss: 0.5527 - val_recall: 0.7026
Epoch 5/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5302 - recall: 0.7469 - val_loss: 0.5345 - val_recall: 0.6797
Epoch 6/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5180 - recall: 0.7568 - val_loss: 0.5212 - val_recall: 0.6765
Epoch 7/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5088 - recall: 0.7616 - val_loss: 0.5092 - val_recall: 0.6732
Epoch 8/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5013 - recall: 0.7610 - val_loss: 0.4979 - val_recall: 0.6634
Epoch 9/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4947 - recall: 0.7631 - val_loss: 0.4871 - val_recall: 0.6536
Epoch 10/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4882 - recall: 0.7664 - val_loss: 0.4789 - val_recall: 0.6536
Epoch 11/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4824 - recall: 0.7706 - val_loss: 0.4718 - val_recall: 0.6503
Epoch 12/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4771 - recall: 0.7726 - val_loss: 0.4666 - val_recall: 0.6601
Epoch 13/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4716 - recall: 0.7759 - val_loss: 0.4605 - val_recall: 0.6471
Epoch 14/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4661 - recall: 0.7814 - val_loss: 0.4558 - val_recall: 0.6471
Epoch 15/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4616 - recall: 0.7821 - val_loss: 0.4522 - val_recall: 0.6373
Epoch 16/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4573 - recall: 0.7869 - val_loss: 0.4488 - val_recall: 0.6373
Epoch 17/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4534 - recall: 0.7917 - val_loss: 0.4460 - val_recall: 0.6373
Epoch 18/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4498 - recall: 0.7914 - val_loss: 0.4431 - val_recall: 0.6373
Epoch 19/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4465 - recall: 0.7939 - val_loss: 0.4404 - val_recall: 0.6373
Epoch 20/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4435 - recall: 0.7980 - val_loss: 0.4381 - val_recall: 0.6340
Epoch 21/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4411 - recall: 0.8020 - val_loss: 0.4358 - val_recall: 0.6373
Epoch 22/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4387 - recall: 0.8017 - val_loss: 0.4342 - val_recall: 0.6373
Epoch 23/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4366 - recall: 0.8002 - val_loss: 0.4334 - val_recall: 0.6340
Epoch 24/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4347 - recall: 0.8020 - val_loss: 0.4314 - val_recall: 0.6340
Epoch 25/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4331 - recall: 0.8032 - val_loss: 0.4309 - val_recall: 0.6340
Epoch 26/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4315 - recall: 0.8038 - val_loss: 0.4302 - val_recall: 0.6340
Epoch 27/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4301 - recall: 0.8028 - val_loss: 0.4291 - val_recall: 0.6307
Epoch 28/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4288 - recall: 0.8040 - val_loss: 0.4286 - val_recall: 0.6307
Epoch 29/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4275 - recall: 0.8045 - val_loss: 0.4277 - val_recall: 0.6275
Epoch 30/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4263 - recall: 0.8043 - val_loss: 0.4270 - val_recall: 0.6242
Epoch 31/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4253 - recall: 0.8055 - val_loss: 0.4271 - val_recall: 0.6242
Epoch 32/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4242 - recall: 0.8063 - val_loss: 0.4268 - val_recall: 0.6209
Epoch 33/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4233 - recall: 0.8061 - val_loss: 0.4264 - val_recall: 0.6209
Epoch 34/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4224 - recall: 0.8066 - val_loss: 0.4261 - val_recall: 0.6176
Epoch 35/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4216 - recall: 0.8057 - val_loss: 0.4263 - val_recall: 0.6242
Epoch 36/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4207 - recall: 0.8067 - val_loss: 0.4258 - val_recall: 0.6242
Epoch 37/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4200 - recall: 0.8062 - val_loss: 0.4255 - val_recall: 0.6242
Epoch 38/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4193 - recall: 0.8080 - val_loss: 0.4256 - val_recall: 0.6242
Epoch 39/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4187 - recall: 0.8085 - val_loss: 0.4260 - val_recall: 0.6242
Epoch 40/40
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4181 - recall: 0.8099 - val_loss: 0.4258 - val_recall: 0.6242

print("Time taken in seconds",end-start)

Time taken in seconds 25.317824602127075

plot(history, 'loss')

• Significant Training Loss Reduction: The training loss decreases sharply and stabilizes at a low level, indicating effective learning and optimization over epochs.
• Validation Loss Plateau: The validation loss decreases initially but then plateaus at a higher level compared to the training loss, which might suggest some overfitting as the model becomes too tailored to the training data.

plot(history, 'recall')

• Strong Training Recall Improvement: Training recall rises sharply early on and continues to increase steadily, reaching high levels, which shows that the model is effectively identifying the positive class in the training set.
• Low and Fluctuating Validation Recall: In contrast, validation recall starts lower and remains significantly fluctuating and generally flat throughout the epochs, suggesting issues with the model’s ability to generalize effectively to unseen data, possibly due to overfitting or model bias.

results.loc[5] = [2,[32,16],["relu","tanh"],batch_size,epochs,"adam",[0.001],"SMOTE","L2",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12
2	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2, dropout	0.48	0.49	0.76	0.75	25.32
3	3	[32, 32, 16]	[relu, relu, tanh]	64	50	adam	[0.0005]	class_weight	L2, dropout, batch norm	0.52	0.47	0.74	0.72	35.12
4	2	[32, 16]	[relu, relu]	64	40	sgd	[0.01]	SMOTE	L2	0.51	0.53	0.75	0.67	24.38
5	2	[32, 16]	[relu, tanh]	64	40	adam	[0.001]	SMOTE	L2	0.41	0.43	0.82	0.62	25.32

👉 Model 5, utilizing SMOTE and Adam optimizer, shows significant improvements in training recall (86%) and a substantial reduction in training loss (0.35) compared to earlier models, indicating better handling of class imbalance and effective learning.
👉 However, the validation recall remains relatively low (66%), suggesting that while the model excels in recognizing the training data, it struggles to generalize this performance to unseen data.

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout

We will finally, modify our third model (Neural Network with Adam Optimizer and Dropout) to apply SMOTE for balancing the data. All other model architecture will remain the same.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model_6 = Sequential()

# First hidden layer with ReLU activation and L2 regularization
model_6.add(Dense(32, activation="relu", input_dim=X_train.shape[1], kernel_regularizer=regularizers.l2(0.001)))
model_6.add(Dropout(0.2))  # 20% Dropout to prevent overfitting

# Second hidden layer with tanh activation and L2 regularization
model_6.add(Dense(16, activation="tanh", kernel_regularizer=regularizers.l2(0.001)))
# Output layer for binary classification
model_6.add(Dense(1, activation="sigmoid"))

model_6.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 32)                  │             352 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 16)                  │             528 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              17 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘

 Total params: 897 (3.50 KB)

 Trainable params: 897 (3.50 KB)

 Non-trainable params: 0 (0.00 B)

optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)    # Adam optimizer w/ default learning rate
model_6.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])

# Record the start time for training
start = time.time()
# Train the model using the SMOTE-balanced training data and validate it on the original validation set
history = model_6.fit(X_train_smote, y_train_smote, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs)
# Record the end time for training
end = time.time()

Epoch 1/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 4s 11ms/step - loss: 0.6159 - recall: 0.6788 - val_loss: 0.5323 - val_recall: 0.6797
Epoch 2/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5365 - recall: 0.7537 - val_loss: 0.5208 - val_recall: 0.7255
Epoch 3/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.5177 - recall: 0.7628 - val_loss: 0.4974 - val_recall: 0.7516
Epoch 4/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4957 - recall: 0.7746 - val_loss: 0.4704 - val_recall: 0.7157
Epoch 5/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4890 - recall: 0.7842 - val_loss: 0.4721 - val_recall: 0.7255
Epoch 6/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4828 - recall: 0.7825 - val_loss: 0.4765 - val_recall: 0.7353
Epoch 7/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4778 - recall: 0.7871 - val_loss: 0.4502 - val_recall: 0.7288
Epoch 8/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4758 - recall: 0.7904 - val_loss: 0.4716 - val_recall: 0.7320
Epoch 9/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4780 - recall: 0.7884 - val_loss: 0.4692 - val_recall: 0.7418
Epoch 10/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4745 - recall: 0.7921 - val_loss: 0.4712 - val_recall: 0.7320
Epoch 11/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4686 - recall: 0.7864 - val_loss: 0.4618 - val_recall: 0.7222
Epoch 12/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4732 - recall: 0.7924 - val_loss: 0.4649 - val_recall: 0.7255
Epoch 13/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4707 - recall: 0.7915 - val_loss: 0.4562 - val_recall: 0.7255
Epoch 14/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4702 - recall: 0.7865 - val_loss: 0.4575 - val_recall: 0.7157
Epoch 15/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4719 - recall: 0.7847 - val_loss: 0.4579 - val_recall: 0.7026
Epoch 16/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4697 - recall: 0.7937 - val_loss: 0.4609 - val_recall: 0.7059
Epoch 17/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4671 - recall: 0.7858 - val_loss: 0.4372 - val_recall: 0.6732
Epoch 18/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4718 - recall: 0.7758 - val_loss: 0.4516 - val_recall: 0.6961
Epoch 19/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4646 - recall: 0.7898 - val_loss: 0.4737 - val_recall: 0.7255
Epoch 20/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4676 - recall: 0.7866 - val_loss: 0.4559 - val_recall: 0.7288
Epoch 21/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4657 - recall: 0.7863 - val_loss: 0.4437 - val_recall: 0.6863
Epoch 22/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4691 - recall: 0.7856 - val_loss: 0.4522 - val_recall: 0.7157
Epoch 23/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4621 - recall: 0.7932 - val_loss: 0.4420 - val_recall: 0.6797
Epoch 24/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4587 - recall: 0.7890 - val_loss: 0.4527 - val_recall: 0.7092
Epoch 25/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4629 - recall: 0.7894 - val_loss: 0.4500 - val_recall: 0.7026
Epoch 26/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4629 - recall: 0.7980 - val_loss: 0.4524 - val_recall: 0.7026
Epoch 27/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4612 - recall: 0.7966 - val_loss: 0.4415 - val_recall: 0.6830
Epoch 28/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4631 - recall: 0.7862 - val_loss: 0.4429 - val_recall: 0.6993
Epoch 29/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4642 - recall: 0.7926 - val_loss: 0.4476 - val_recall: 0.6961
Epoch 30/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4650 - recall: 0.7974 - val_loss: 0.4461 - val_recall: 0.6993
Epoch 31/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4645 - recall: 0.7928 - val_loss: 0.4470 - val_recall: 0.7190
Epoch 32/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4648 - recall: 0.7951 - val_loss: 0.4535 - val_recall: 0.7026
Epoch 33/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4635 - recall: 0.7874 - val_loss: 0.4260 - val_recall: 0.6732
Epoch 34/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4615 - recall: 0.7858 - val_loss: 0.4414 - val_recall: 0.7157
Epoch 35/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4650 - recall: 0.8000 - val_loss: 0.4390 - val_recall: 0.7124
Epoch 36/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4643 - recall: 0.8010 - val_loss: 0.4456 - val_recall: 0.7190
Epoch 37/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4626 - recall: 0.8015 - val_loss: 0.4306 - val_recall: 0.7124
Epoch 38/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4616 - recall: 0.7959 - val_loss: 0.4445 - val_recall: 0.7320
Epoch 39/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4649 - recall: 0.7934 - val_loss: 0.4377 - val_recall: 0.6732
Epoch 40/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4658 - recall: 0.7942 - val_loss: 0.4674 - val_recall: 0.7353
Epoch 41/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4609 - recall: 0.7848 - val_loss: 0.4505 - val_recall: 0.7288
Epoch 42/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4665 - recall: 0.7967 - val_loss: 0.4268 - val_recall: 0.6993
Epoch 43/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4603 - recall: 0.7939 - val_loss: 0.4423 - val_recall: 0.7124
Epoch 44/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4603 - recall: 0.7930 - val_loss: 0.4593 - val_recall: 0.7059
Epoch 45/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4608 - recall: 0.7822 - val_loss: 0.4541 - val_recall: 0.7255
Epoch 46/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4590 - recall: 0.7956 - val_loss: 0.4433 - val_recall: 0.6993
Epoch 47/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4674 - recall: 0.7935 - val_loss: 0.4403 - val_recall: 0.6863
Epoch 48/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 0.4607 - recall: 0.7898 - val_loss: 0.4587 - val_recall: 0.7255
Epoch 49/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4642 - recall: 0.7889 - val_loss: 0.4409 - val_recall: 0.6895
Epoch 50/50
175/175 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4624 - recall: 0.7881 - val_loss: 0.4547 - val_recall: 0.7092

print("Time taken in seconds",end-start)

Time taken in seconds 32.75745940208435

plot(history, 'loss')

• Effective Learning: Both the training and validation losses demonstrate a sharp decline initially, suggesting that the model quickly adapts to patterns in the data, which is a good sign of effective learning.
• Close Convergence: The training and validation losses converge and remain close throughout the training process, indicating good generalization capabilities of the model without significant overfitting.
• Stable Performance: After the initial sharp decrease, the losses stabilize and flatten out from around epoch 20 onwards, suggesting that further training yields minimal improvement and the model has likely reached its performance plateau.
• Overall, the model exhibits healthy training behavior, with closely aligned training and validation losses suggesting that it generalizes well to unseen data.

plot(history, 'recall')

• High Training Recall: The training recall quickly rises and stabilizes at a high level (around 80%), demonstrating the model's strong ability to correctly identify all relevant instances during training.
• Low and Volatile Validation Recall: In stark contrast, the validation recall remains much lower and shows significant volatility, generally fluctuating around 65%. This suggests the model struggles to generalize the recall performance effectively to unseen data.
• The significant discrepancy between training and validation recall may indicate overfitting. The model is likely memorizing the training data too well, compromising its ability to perform similarly on validation data.

results.loc[6] = [2,[32,16],["relu","tanh"],batch_size,epochs,"adam",[0.001],"SMOTE","L2, dropout",history.history["loss"][-1],history.history["val_loss"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results

	Hidden Layer Count	Neuron Count	Activation Function	Batch Size	Epoch Count	Optimizer	Learning Rate	Class Balancer	Regularization	Train Loss	Validation Loss	Train Recall	Validation Recall	Time in secs
0	2	[64, 32]	[relu, relu]	32	50	sgd	[0.01]	class_weight	-	0.48	0.50	0.76	0.77	36.33
1	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2	0.49	0.50	0.75	0.75	22.12
2	2	[64, 32]	[relu, tanh]	64	50	adam	[0.001]	class_weight	L2, dropout	0.48	0.49	0.76	0.75	25.32
3	3	[32, 32, 16]	[relu, relu, tanh]	64	50	adam	[0.0005]	class_weight	L2, dropout, batch norm	0.52	0.47	0.74	0.72	35.12
4	2	[32, 16]	[relu, relu]	64	40	sgd	[0.01]	SMOTE	L2	0.51	0.53	0.75	0.67	24.38
5	2	[32, 16]	[relu, tanh]	64	40	adam	[0.001]	SMOTE	L2	0.41	0.43	0.82	0.62	25.32
6	2	[32, 16]	[relu, tanh]	64	50	adam	[0.001]	SMOTE	L2, dropout	0.45	0.45	0.80	0.71	32.76

• Lowest Validation Loss: Model 2 records one of the lower validation losses (0.47), suggesting it generalizes better than most other models.
• High Recall: Model 2 also achieves the highest validation recall (0.74) among all models, indicating superior capability in identifying positive cases correctly on unseen data.
• Comprehensive Regularization: The combination of L2 regularization and dropout likely contributes to its robustness, helping to manage overfitting more effectively than other models.

Scikit-Learn - Multi-layer Perceptron Classifier (MLPClassifier)

from sklearn.neural_network import MLPClassifier

model_7 = MLPClassifier(
    hidden_layer_sizes = (32,32),
    activation = "relu",
    random_state = 42,
    solver = "adam",
    max_iter = 100,
    # batch_size = 32,
    early_stopping = True,
    n_iter_no_change = 10,
    alpha = 1
).fit(X_train_smote,y_train_smote)

pred_train = model_7.predict(X_train_smote)
print(classification_report(y_train_smote,pred_train))

pred_val = model_7.predict(X_val)
print(classification_report(y_val,pred_val))

              precision    recall  f1-score   support

           0       0.79      0.86      0.82      5574
           1       0.85      0.76      0.80      5574

    accuracy                           0.81     11148
   macro avg       0.82      0.81      0.81     11148
weighted avg       0.82      0.81      0.81     11148

              precision    recall  f1-score   support

           0       0.91      0.86      0.89      1194
           1       0.55      0.67      0.61       306

    accuracy                           0.82      1500
   macro avg       0.73      0.76      0.75      1500
weighted avg       0.84      0.82      0.83      1500

Model Performance Comparison and Final Model Selection

Choosing Model 3 as the best model to train with the test data is a sound decision based on several key observations:

1. Balanced Loss: Model 3 demonstrates a steady and consistent decline in both training and validation loss, indicating effective learning and generalization without signs of overfitting.

2. Recall Performance: Despite initial fluctuations, the recall for both training and validation stabilizes and remains close together through later epochs. This suggests the model has good sensitivity and is reliable at identifying true positives across different data subsets.

3. Stability with Modifications: The integration of L2 regularization, the Adam optimizer, and the tanh activation in the second layer seems to have stabilized the learning process, as evidenced by the minimal gap between training and validation metrics.

4. Optimization Efficiency: The quick convergence of loss metrics with Adam, combined with the controlled complexity via L2 regularization, shows that Model 3 efficiently addresses both the bias and variance issues, making it robust for unseen data like the test set.

These factors collectively make Model 3 a robust choice for further testing, likely providing the best balance between performance and generalization capabilities.

batch_size = 64  # Number of samples processed before the model is updated
epochs = 50  # The model will pass over the entire dataset 50 times during training

# Clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()

# Initializing the neural network
model = Sequential()

# First hidden layer with ReLU activation and L2 regularization
model.add(Dense(64, activation="relu", input_dim=X_train.shape[1], kernel_regularizer=regularizers.l2(0.001)))
model.add(Dropout(0.2))  # 20% Dropout to prevent overfitting

# Second hidden layer with tanh activation and L2 regularization
model.add(Dense(32, activation="tanh", kernel_regularizer=regularizers.l2(0.001)))
# Output layer for binary classification
model.add(Dense(1, activation="sigmoid"))

optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)    # defining Adam as the optimizer
model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=[Recall()])
# Train the final model using the training data and evaluate it on the validation set
history = model.fit(X_train, y_train, validation_data=(X_val, y_val), batch_size=batch_size, epochs=epochs, class_weight=cw_dict)

Epoch 1/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 4s 19ms/step - loss: 0.7327 - recall: 0.5900 - val_loss: 0.6965 - val_recall: 0.7353
Epoch 2/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 2s 2ms/step - loss: 0.6542 - recall: 0.7205 - val_loss: 0.6767 - val_recall: 0.7484
Epoch 3/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6303 - recall: 0.7235 - val_loss: 0.6537 - val_recall: 0.7582
Epoch 4/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.6140 - recall: 0.7369 - val_loss: 0.6350 - val_recall: 0.7745
Epoch 5/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.6013 - recall: 0.7307 - val_loss: 0.6211 - val_recall: 0.7778
Epoch 6/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5849 - recall: 0.7259 - val_loss: 0.6214 - val_recall: 0.7941
Epoch 7/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5723 - recall: 0.7343 - val_loss: 0.5962 - val_recall: 0.7843
Epoch 8/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5617 - recall: 0.7391 - val_loss: 0.5876 - val_recall: 0.7908
Epoch 9/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.5556 - recall: 0.7347 - val_loss: 0.5770 - val_recall: 0.7974
Epoch 10/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.5459 - recall: 0.7258 - val_loss: 0.5750 - val_recall: 0.7941
Epoch 11/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5442 - recall: 0.7347 - val_loss: 0.5713 - val_recall: 0.7908
Epoch 12/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5326 - recall: 0.7490 - val_loss: 0.5722 - val_recall: 0.7908
Epoch 13/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5313 - recall: 0.7356 - val_loss: 0.5601 - val_recall: 0.7712
Epoch 14/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5303 - recall: 0.7455 - val_loss: 0.5427 - val_recall: 0.7745
Epoch 15/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5192 - recall: 0.7424 - val_loss: 0.5552 - val_recall: 0.7810
Epoch 16/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5141 - recall: 0.7474 - val_loss: 0.5355 - val_recall: 0.7614
Epoch 17/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5087 - recall: 0.7421 - val_loss: 0.5335 - val_recall: 0.7712
Epoch 18/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5076 - recall: 0.7329 - val_loss: 0.5284 - val_recall: 0.7647
Epoch 19/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4998 - recall: 0.7430 - val_loss: 0.5333 - val_recall: 0.7647
Epoch 20/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.5022 - recall: 0.7488 - val_loss: 0.5224 - val_recall: 0.7680
Epoch 21/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4955 - recall: 0.7498 - val_loss: 0.5243 - val_recall: 0.7647
Epoch 22/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4898 - recall: 0.7544 - val_loss: 0.5126 - val_recall: 0.7549
Epoch 23/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4860 - recall: 0.7483 - val_loss: 0.5184 - val_recall: 0.7680
Epoch 24/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4826 - recall: 0.7604 - val_loss: 0.5200 - val_recall: 0.7549
Epoch 25/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4817 - recall: 0.7497 - val_loss: 0.5004 - val_recall: 0.7418
Epoch 26/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4748 - recall: 0.7596 - val_loss: 0.5059 - val_recall: 0.7549
Epoch 27/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4738 - recall: 0.7478 - val_loss: 0.5074 - val_recall: 0.7582
Epoch 28/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4794 - recall: 0.7604 - val_loss: 0.5141 - val_recall: 0.7484
Epoch 29/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4787 - recall: 0.7644 - val_loss: 0.5045 - val_recall: 0.7451
Epoch 30/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - loss: 0.4728 - recall: 0.7638 - val_loss: 0.5024 - val_recall: 0.7451
Epoch 31/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4750 - recall: 0.7431 - val_loss: 0.4985 - val_recall: 0.7451
Epoch 32/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4778 - recall: 0.7320 - val_loss: 0.5048 - val_recall: 0.7386
Epoch 33/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4706 - recall: 0.7499 - val_loss: 0.5003 - val_recall: 0.7418
Epoch 34/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4729 - recall: 0.7555 - val_loss: 0.5048 - val_recall: 0.7451
Epoch 35/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4709 - recall: 0.7531 - val_loss: 0.4905 - val_recall: 0.7353
Epoch 36/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4695 - recall: 0.7544 - val_loss: 0.4914 - val_recall: 0.7386
Epoch 37/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4710 - recall: 0.7619 - val_loss: 0.5046 - val_recall: 0.7484
Epoch 38/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4691 - recall: 0.7609 - val_loss: 0.4928 - val_recall: 0.7386
Epoch 39/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - loss: 0.4646 - recall: 0.7614 - val_loss: 0.4868 - val_recall: 0.7386
Epoch 40/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - loss: 0.4668 - recall: 0.7603 - val_loss: 0.4967 - val_recall: 0.7451
Epoch 41/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - loss: 0.4627 - recall: 0.7683 - val_loss: 0.5069 - val_recall: 0.7549
Epoch 42/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4657 - recall: 0.7589 - val_loss: 0.4970 - val_recall: 0.7386
Epoch 43/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4640 - recall: 0.7462 - val_loss: 0.5014 - val_recall: 0.7516
Epoch 44/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4633 - recall: 0.7526 - val_loss: 0.4881 - val_recall: 0.7353
Epoch 45/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4661 - recall: 0.7470 - val_loss: 0.5005 - val_recall: 0.7484
Epoch 46/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4562 - recall: 0.7597 - val_loss: 0.4956 - val_recall: 0.7418
Epoch 47/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - loss: 0.4602 - recall: 0.7548 - val_loss: 0.4974 - val_recall: 0.7418
Epoch 48/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4628 - recall: 0.7539 - val_loss: 0.5074 - val_recall: 0.7451
Epoch 49/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4652 - recall: 0.7570 - val_loss: 0.4818 - val_recall: 0.7320
Epoch 50/50
110/110 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 0.4649 - recall: 0.7471 - val_loss: 0.4835 - val_recall: 0.7320

y_train_pred = model.predict(X_train)
y_val_pred = model.predict(X_val)
y_test_pred = model.predict(X_test)

219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step
47/47 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step
47/47 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step

print("Classification Report - Train data",end="\n\n")
cr = classification_report(y_train,y_train_pred>0.5)
print(cr)

Classification Report - Train data

              precision    recall  f1-score   support

           0       0.93      0.81      0.87      5574
           1       0.51      0.78      0.61      1426

    accuracy                           0.80      7000
   macro avg       0.72      0.79      0.74      7000
weighted avg       0.85      0.80      0.81      7000

👉 The final model demonstrates strong performance in distinguishing the minority class (class 1) with a high recall of 0.78 and a satisfactory F1-score of 0.61, indicating effective identification of true positive cases.

print("Classification Report - Validation data",end="\n\n")
cr = classification_report(y_val,y_val_pred>0.5)
print(cr)

Classification Report - Validation data

              precision    recall  f1-score   support

           0       0.92      0.79      0.85      1194
           1       0.48      0.73      0.58       306

    accuracy                           0.78      1500
   macro avg       0.70      0.76      0.72      1500
weighted avg       0.83      0.78      0.80      1500

👉 The model performs consistently on the validation data, showing a strong recall of 0.73 for the minority class (class 1), emphasizing its capability to correctly identify a high percentage of true positives.

print("Classification Report - Test data",end="\n\n")
cr = classification_report(y_test,y_test_pred>0.5)
print(cr)

Classification Report - Test data

              precision    recall  f1-score   support

           0       0.92      0.79      0.85      1195
           1       0.48      0.74      0.58       305

    accuracy                           0.78      1500
   macro avg       0.70      0.77      0.72      1500
weighted avg       0.83      0.78      0.80      1500

👉 The model shows similar performance on the test data as seen in the validation set, with high recall (0.74) for class 1, effectively identifying most positive instances. The precision for class 1 remains moderate (0.48), resulting in some false positives, but the balanced recall and precision lead to solid f1-scores, indicating the model is capable of generalizing well to new, unseen data.

Actionable Insights and Business Recommendations

Based on the analysis of neural network models to predict customer churn, here are some insights and recommendations for the bank:

Insights:

• Best Model: Model 3, with three hidden layers and a combination of L2 regularization and dropout, performed best, showing strong ability to identify potential churners.
• Overfitting: Most models showed overfitting, indicating a need for better generalization to unseen data.
• Room for Improvement: Validation recall across models was suboptimal, highlighting a need for more accurate identification of true positives.

Business Recommendations:

1. Enhance Customer Engagement: Use the model to target high-risk customers with personalized interventions and regularly collect feedback to integrate into service improvements.
2. Refine Customer Service: Focus on improving areas most predictive of churn and establish proactive communications to address customer issues early.
3. Continual Model Refinement: Regularly update the model with new data and recalibrate features to maintain its relevance and effectiveness.
4. Staff Training: Educate customer-facing staff on using model outputs to enhance decision-making and customer interactions.
5. Strategic Customer Retention: Develop loyalty programs tailored to reduce churn and keep a competitive edge by adjusting offerings based on model insights.

Implementing these strategies will help the bank not only reduce churn but also enhance overall customer loyalty and satisfaction.