Audio Digit Recognition¶
Context¶
In the past decades, significant advances have been achieved in the area of audio recognition and a lot of research is going on globally to recognize audio data or speech using Deep Learning. The most common use case in this field is converting audio to spectrograms and vice versa.
Audio in its raw form is usually a wave and to capture that using a data structure, we need to have a huge array of amplitudes even for a very short audio clip. Although it depends on the sampling rate of the sound wave, this structured data conversion for any audio wave is very voluminous even for low sampling rates. So it becomes a problem to store and computationally very expensive to do even simple calculations on such data.
One of the best economical alternatives to this is using spectrograms. Spectrograms are created by doing Fourier or Short Time Fourier Transforms on sound waves. There are various kinds of spectrograms but the ones we will be using are called MFCC spectrograms. To put it in simple terms, a spectrogram is a way to visually encapsulate audio data. It is a graph on a 2-D plane where the X-axis represents time and the Y-axis represents Mel Coefficients. But since it is continuous on a 2-D plane, we can treat this as an image.
Objective¶
The objective here is to build an Artificial Neural Network that can look at Mel or MFCC spectrograms of audio files and classify them into 10 classes. The audio files are recordings of different speakers uttering a particular digit and the corresponding class to be predicted is the digit itself.
Dataset¶
The dataset we will use is the Audio MNIST dataset, which contains audio files in .wav format, organized into 10 different folders. The Audio MNIST dataset is analogous to the classic MNIST dataset but represents data in the form of audio rather than images. It includes recordings of various speakers pronouncing digits from 0 to 9. Each folder corresponds to a particular speaker and contains audio files of that speaker saying the digits.
Packages Used¶
Librosa: Librosa is a Python package that helps in dealing with audio data. librosa.display visualizes and displays the audio data using Matplotlib. Similarly, there exists a collection of submodules under librosa that provides various other functionalities.IPython.display: Display is a public API to display the tools available in Ipython. In this analysis, we will create an audio object to display the digits in the MNIST audio data.tqdm: tqdm is a Python package that allows us to add a progress bar to our application. This package will help us in iterating over the audio data.
Installing Librosa¶
In [ ]:
!pip install librosa
Requirement already satisfied: librosa in /usr/local/lib/python3.10/dist-packages (0.10.2.post1) Requirement already satisfied: audioread>=2.1.9 in /usr/local/lib/python3.10/dist-packages (from librosa) (3.0.1) Requirement already satisfied: numpy!=1.22.0,!=1.22.1,!=1.22.2,>=1.20.3 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.26.4) Requirement already satisfied: scipy>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.13.1) Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.5.2) Requirement already satisfied: joblib>=0.14 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.4.2) Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.10/dist-packages (from librosa) (4.4.2) Requirement already satisfied: numba>=0.51.0 in /usr/local/lib/python3.10/dist-packages (from librosa) (0.60.0) Requirement already satisfied: soundfile>=0.12.1 in /usr/local/lib/python3.10/dist-packages (from librosa) (0.12.1) Requirement already satisfied: pooch>=1.1 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.8.2) Requirement already satisfied: soxr>=0.3.2 in /usr/local/lib/python3.10/dist-packages (from librosa) (0.5.0.post1) Requirement already satisfied: typing-extensions>=4.1.1 in /usr/local/lib/python3.10/dist-packages (from librosa) (4.12.2) Requirement already satisfied: lazy-loader>=0.1 in /usr/local/lib/python3.10/dist-packages (from librosa) (0.4) Requirement already satisfied: msgpack>=1.0 in /usr/local/lib/python3.10/dist-packages (from librosa) (1.1.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lazy-loader>=0.1->librosa) (24.2) Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba>=0.51.0->librosa) (0.43.0) Requirement already satisfied: platformdirs>=2.5.0 in /usr/local/lib/python3.10/dist-packages (from pooch>=1.1->librosa) (4.3.6) Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from pooch>=1.1->librosa) (2.32.3) Requirement already satisfied: threadpoolctl>=3.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.20.0->librosa) (3.5.0) Requirement already satisfied: cffi>=1.0 in /usr/local/lib/python3.10/dist-packages (from soundfile>=0.12.1->librosa) (1.17.1) Requirement already satisfied: pycparser in /usr/local/lib/python3.10/dist-packages (from cffi>=1.0->soundfile>=0.12.1->librosa) (2.22) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->pooch>=1.1->librosa) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->pooch>=1.1->librosa) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->pooch>=1.1->librosa) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->pooch>=1.1->librosa) (2024.8.30)
Importing the necessary libraries¶
In [ ]:
# For Audio Preprocessing
import librosa
import librosa.display as dsp
from IPython.display import Audio
# For Data Preprocessing
import pandas as pd
import numpy as np
import os
# For Data Visualization
import matplotlib.pyplot as plt
import seaborn as sns
from tqdm import tqdm
# For unzipping the data if necessary
import zipfile
import os
In [ ]:
# Use the specified theme in Seaborn
sns.set_style("darkgrid")
Mounting the Drive and Unzipping the file¶
In [ ]:
# Mount Google Drive to current Colab notebook
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [ ]:
path = '/content/drive/My Drive/Texas McCombs/Neural Networks/Case Studies/CaseStudy - Audio MNIST Digit Recognition/audio-MNIST-data'
# If the data is provided as a zip file, extract the files from the zip file first
# with zipfile.ZipFile(path, 'r') as zip_ref:
# zip_ref.extractall()
Let's read and check some of the audio samples¶
The below function called "get_audio" takes a digit as an argument and plots the audio wave and returns the audio for a given digit.
Let's understand how the get_audio() function works:
.wav: We will load the .wav files using the librosa package.dsp.waveshow(): It visualizes the waveform in the time domain. This method creates a plot that alternates between a raw samples-based view of the signal and an amplitude-envelope view of the signal. The "sr" parameter is the sampling rate, i.e., samples per second.Audio(): From the IPython package, we can create an audio object.
In [ ]:
def get_audio(digit = 0):
# Audio Sample Directory
sample = np.random.randint(1, 10)
# Index of Audio
index = np.random.randint(1, 5)
# Audio file location
if sample < 10:
file = f"{path}/0{sample}/{digit}_0{sample}_{index}.wav"
else:
file = f"{path}/{sample}/{digit}_{sample}_{index}.wav"
# Get Audio from the location
# Audio will be automatically resampled to the given rate (default sr = 22050)
data, sample_rate = librosa.load(file)
# Plot the audio wave
dsp.waveshow(data, sr = sample_rate)
plt.show()
# Show the audio widget
return Audio(data = data, rate = sample_rate)
In [ ]:
# Show the audio and plot of digit 0
get_audio(0)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 1
get_audio(1)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 2
get_audio(2)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 3
get_audio(3)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 4
get_audio(4)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 5
get_audio(5)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 6
get_audio(6)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 7
get_audio(7)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 8
get_audio(8)
Out[ ]:
In [ ]:
# Show the audio and plot of digit 9
get_audio(9)
Out[ ]:
Observations:
- The X-axis represents time and Y-axis represents the amplitude of the vibrations.
- The intuition behind the Fourier Transform is that any wave can be broken down or deconstructed as a sum of many composite sine waves. Since these are composed of sine waves, they are symmetric around the time axis, i.e, they extend equally above and below the time axis at any particular time.
- From the various audio plots ranging from 0 to 9, we can observe the amplitude at a given point in time. For example, when we say "Zero", the "Z" sound has low amplitude and the "ero" sound has higher amplitude. Similarly, the remaining digits can be interpreted by looking at the visualizations.
Visualizing the spectrogram of the audio data¶
What is a spectrogram?¶
A spectrogram is a visual way of representing the signal strength or “loudness” of a signal over time at various frequencies or time steps present in a particular waveform. A spectrogram gives a detailed view of audio. It represents amplitude, frequency, and time in a single plot. Since spectrograms are continuous plots, they can be interpreted as an image.
Different spectrograms have different attributes on their axes and they are usually different to interpret. In a Research and Development scenario, we make use of a vocoder, which is an encoder that converts spectrograms back to audio using parameters learned by machine learning. One great vocoder is the WaveNet vocoder which is used in almost all Text to Speech architectures.
Here, we will be using MFCC spectrograms, which are also called Mel spectrograms.
In [ ]:
# A function which returns audio file for a mentioned digit
def get_audio_raw(digit = 0):
# Audio sample directory
sample = np.random.randint(1, 10)
# Index of audio
index = np.random.randint(1, 5)
# Modified file location
if sample < 10:
file = f"{path}/0{sample}/{digit}_0{sample}_{index}.wav"
else:
file = f"{path}/{sample}/{digit}_{sample}_{index}.wav"
# Get audio from the location
data, sample_rate = librosa.load(file)
# Return audio
return data, sample_rate
Extracting features from the audio file¶
Mel-frequency cepstral coefficients (MFCCs) Feature Extraction
MFCCs are usually the final features used in many machine learning models trained on audio data. They are usually a set of mel coefficients defined for each time step through which the raw audio data can be encoded. For example, if we have an audio sample extending for 30 time steps, and we are defining each time step by 40 Mel Coefficients, our entire sample can be represented by 40 * 30 Mel Coefficients. And, if we want to create a Mel Spectrogram out of it, our spectrogram will resemble a 2-D array of 40 horizontal rows and 30 vertical columns.
In this step, we will first extract the Mel Coefficents for each audio file and add them to our dataset.
extract_features: Returns the MFCC extracted features for an audio file.process_and_create_dataset: Iterate through the audio of each digit, extract the features using the extract_features() function, and append the data into a DataFrame.
Creating a function that extracts the data from audio files
In [ ]:
# Will take an audio file as input and return extracted features using MEL_FREQUENCY CEPSTRAL COEFFICIENT as the output
def extract_features(file):
# Load audio and its sample rate
audio, sample_rate = librosa.load(file)
# Extract features using mel-frequency coefficient
extracted_features = librosa.feature.mfcc(y = audio,
sr = sample_rate,
n_mfcc = 40)
# Scale the extracted features
extracted_features = np.mean(extracted_features.T, axis = 0)
# Return the extracted features
return extracted_features
def preprocess_and_create_dataset():
# Path of the folder where the audio files are present
root_folder_path = path
# Empty List to create dataset
dataset = []
# Iterating through folders where each folder has the audio of each digit
for folder in tqdm(range(1, 11)):
if folder < 10:
# Path of the folder
folder = os.path.join(root_folder_path, "0" + str(folder))
else:
folder = os.path.join(root_folder_path, str(folder))
# Iterate through each file of the present folder
for file in tqdm(os.listdir(folder)):
# Path of the file
abs_file_path = os.path.join(folder, file)
# Pass path of file to the extracted_features() function to create features
extracted_features = extract_features(abs_file_path)
# Class of the audio, i.e., the digit it represents
class_label = file[0]
# Append a list where the feature represents a column and class of the digit represents another column
dataset.append([extracted_features, class_label])
# After iterating through all the folders, convert the list to a DataFrame
return pd.DataFrame(dataset, columns = ['features', 'class'])
Now, let's create the dataset using the defined function
In [ ]:
# Create the dataset by calling the function
dataset = preprocess_and_create_dataset()
0%| | 0/10 [00:00<?, ?it/s] 0%| | 0/500 [00:00<?, ?it/s] 0%| | 2/500 [00:00<00:25, 19.63it/s] 1%| | 4/500 [00:00<00:35, 13.79it/s] 1%| | 6/500 [00:00<00:55, 8.94it/s] 2%|▏ | 8/500 [00:06<10:07, 1.23s/it] 2%|▏ | 11/500 [00:06<05:36, 1.45it/s] 3%|▎ | 13/500 [00:06<04:03, 2.00it/s] 3%|▎ | 16/500 [00:06<02:33, 3.15it/s] 4%|▍ | 20/500 [00:07<01:33, 5.12it/s] 5%|▍ | 23/500 [00:07<01:12, 6.56it/s] 5%|▌ | 26/500 [00:07<00:57, 8.27it/s] 6%|▌ | 29/500 [00:07<00:44, 10.62it/s] 6%|▋ | 32/500 [00:07<00:35, 13.10it/s] 7%|▋ | 36/500 [00:07<00:27, 17.08it/s] 8%|▊ | 39/500 [00:07<00:24, 18.91it/s] 8%|▊ | 42/500 [00:07<00:22, 20.43it/s] 9%|▉ | 45/500 [00:08<00:23, 19.23it/s] 10%|▉ | 48/500 [00:08<00:22, 20.27it/s] 10%|█ | 51/500 [00:08<00:21, 21.29it/s] 11%|█ | 54/500 [00:08<00:20, 21.75it/s] 12%|█▏ | 58/500 [00:08<00:17, 24.66it/s] 12%|█▏ | 62/500 [00:08<00:16, 27.29it/s] 13%|█▎ | 65/500 [00:08<00:15, 27.94it/s] 14%|█▎ | 68/500 [00:08<00:15, 28.34it/s] 14%|█▍ | 71/500 [00:09<00:17, 24.90it/s] 15%|█▍ | 74/500 [00:09<00:17, 24.00it/s] 15%|█▌ | 77/500 [00:09<00:19, 21.88it/s] 16%|█▌ | 80/500 [00:09<00:19, 21.66it/s] 17%|█▋ | 84/500 [00:09<00:16, 25.63it/s] 18%|█▊ | 88/500 [00:09<00:14, 27.56it/s] 18%|█▊ | 91/500 [00:09<00:15, 26.01it/s] 19%|█▉ | 94/500 [00:09<00:16, 24.97it/s] 19%|█▉ | 97/500 [00:10<00:15, 26.01it/s] 20%|██ | 100/500 [00:10<00:16, 24.86it/s] 21%|██ | 103/500 [00:10<00:15, 25.74it/s] 21%|██ | 106/500 [00:10<00:15, 24.89it/s] 22%|██▏ | 109/500 [00:10<00:15, 25.75it/s] 23%|██▎ | 113/500 [00:10<00:13, 27.80it/s] 23%|██▎ | 116/500 [00:10<00:15, 25.20it/s] 24%|██▍ | 119/500 [00:10<00:15, 24.89it/s] 24%|██▍ | 122/500 [00:11<00:15, 24.53it/s] 25%|██▌ | 125/500 [00:11<00:15, 24.20it/s] 26%|██▌ | 128/500 [00:11<00:15, 23.29it/s] 26%|██▌ | 131/500 [00:11<00:16, 22.99it/s] 27%|██▋ | 134/500 [00:11<00:15, 23.49it/s] 28%|██▊ | 138/500 [00:11<00:14, 25.85it/s] 28%|██▊ | 141/500 [00:11<00:13, 25.86it/s] 29%|██▉ | 144/500 [00:11<00:13, 26.12it/s] 29%|██▉ | 147/500 [00:12<00:13, 25.30it/s] 30%|███ | 150/500 [00:12<00:14, 23.78it/s] 31%|███ | 154/500 [00:12<00:13, 25.43it/s] 31%|███▏ | 157/500 [00:12<00:12, 26.45it/s] 32%|███▏ | 160/500 [00:12<00:13, 25.08it/s] 33%|███▎ | 163/500 [00:12<00:13, 25.16it/s] 33%|███▎ | 166/500 [00:12<00:14, 23.51it/s] 34%|███▍ | 169/500 [00:13<00:16, 20.00it/s] 34%|███▍ | 172/500 [00:13<00:17, 18.57it/s] 35%|███▍ | 174/500 [00:13<00:19, 17.12it/s] 35%|███▌ | 176/500 [00:13<00:21, 15.36it/s] 36%|███▌ | 179/500 [00:13<00:19, 16.77it/s] 36%|███▌ | 181/500 [00:13<00:18, 17.40it/s] 37%|███▋ | 183/500 [00:13<00:19, 16.35it/s] 37%|███▋ | 185/500 [00:14<00:20, 15.04it/s] 38%|███▊ | 188/500 [00:14<00:19, 16.37it/s] 38%|███▊ | 190/500 [00:14<00:19, 15.65it/s] 38%|███▊ | 192/500 [00:14<00:20, 15.35it/s] 39%|███▉ | 194/500 [00:14<00:18, 16.19it/s] 39%|███▉ | 196/500 [00:14<00:21, 14.23it/s] 40%|███▉ | 198/500 [00:15<00:21, 14.35it/s] 40%|████ | 200/500 [00:15<00:19, 15.12it/s] 41%|████ | 204/500 [00:15<00:14, 19.80it/s] 42%|████▏ | 208/500 [00:15<00:12, 22.67it/s] 42%|████▏ | 212/500 [00:15<00:11, 25.67it/s] 43%|████▎ | 215/500 [00:15<00:11, 25.90it/s] 44%|████▎ | 218/500 [00:15<00:11, 25.49it/s] 44%|████▍ | 221/500 [00:15<00:10, 26.22it/s] 45%|████▍ | 224/500 [00:15<00:10, 26.14it/s] 46%|████▌ | 228/500 [00:16<00:09, 28.07it/s] 46%|████▌ | 231/500 [00:16<00:09, 28.31it/s] 47%|████▋ | 235/500 [00:16<00:08, 31.03it/s] 48%|████▊ | 239/500 [00:16<00:07, 32.72it/s] 49%|████▊ | 243/500 [00:16<00:08, 30.53it/s] 49%|████▉ | 247/500 [00:16<00:08, 30.96it/s] 50%|█████ | 251/500 [00:16<00:07, 31.65it/s] 51%|█████ | 255/500 [00:16<00:07, 31.59it/s] 52%|█████▏ | 259/500 [00:17<00:07, 32.92it/s] 53%|█████▎ | 263/500 [00:17<00:07, 29.92it/s] 53%|█████▎ | 267/500 [00:17<00:07, 29.24it/s] 54%|█████▍ | 271/500 [00:17<00:07, 30.07it/s] 55%|█████▌ | 275/500 [00:17<00:07, 31.11it/s] 56%|█████▌ | 279/500 [00:17<00:06, 32.14it/s] 57%|█████▋ | 283/500 [00:17<00:06, 32.03it/s] 57%|█████▋ | 287/500 [00:17<00:06, 30.58it/s] 58%|█████▊ | 291/500 [00:18<00:07, 29.03it/s] 59%|█████▉ | 295/500 [00:18<00:06, 30.83it/s] 60%|█████▉ | 299/500 [00:18<00:06, 32.20it/s] 61%|██████ | 303/500 [00:18<00:05, 33.16it/s] 61%|██████▏ | 307/500 [00:18<00:05, 32.94it/s] 62%|██████▏ | 311/500 [00:18<00:06, 30.97it/s] 63%|██████▎ | 315/500 [00:18<00:06, 28.48it/s] 64%|██████▎ | 318/500 [00:19<00:06, 27.40it/s] 64%|██████▍ | 321/500 [00:19<00:06, 25.86it/s] 65%|██████▍ | 324/500 [00:19<00:07, 24.27it/s] 65%|██████▌ | 327/500 [00:19<00:07, 24.59it/s] 66%|██████▌ | 330/500 [00:19<00:06, 24.85it/s] 67%|██████▋ | 333/500 [00:19<00:06, 25.35it/s] 67%|██████▋ | 337/500 [00:19<00:05, 27.56it/s] 68%|██████▊ | 341/500 [00:19<00:05, 28.91it/s] 69%|██████▉ | 344/500 [00:20<00:05, 28.78it/s] 69%|██████▉ | 347/500 [00:20<00:05, 28.60it/s] 70%|███████ | 350/500 [00:20<00:05, 25.77it/s] 71%|███████ | 353/500 [00:20<00:05, 24.64it/s] 71%|███████ | 356/500 [00:20<00:05, 25.86it/s] 72%|███████▏ | 359/500 [00:20<00:05, 25.24it/s] 73%|███████▎ | 363/500 [00:20<00:05, 27.31it/s] 73%|███████▎ | 367/500 [00:20<00:04, 28.68it/s] 74%|███████▍ | 370/500 [00:20<00:04, 27.82it/s] 75%|███████▍ | 373/500 [00:21<00:04, 25.41it/s] 75%|███████▌ | 376/500 [00:21<00:04, 25.60it/s] 76%|███████▌ | 379/500 [00:21<00:04, 26.08it/s] 76%|███████▋ | 382/500 [00:21<00:04, 26.20it/s] 77%|███████▋ | 385/500 [00:21<00:04, 26.19it/s] 78%|███████▊ | 388/500 [00:21<00:04, 26.61it/s] 78%|███████▊ | 392/500 [00:21<00:03, 28.22it/s] 79%|███████▉ | 395/500 [00:21<00:03, 27.93it/s] 80%|███████▉ | 398/500 [00:22<00:03, 27.04it/s] 80%|████████ | 401/500 [00:22<00:03, 27.24it/s] 81%|████████ | 404/500 [00:22<00:03, 27.25it/s] 81%|████████▏ | 407/500 [00:22<00:03, 27.18it/s] 82%|████████▏ | 410/500 [00:22<00:03, 27.04it/s] 83%|████████▎ | 415/500 [00:22<00:02, 32.68it/s] 84%|████████▍ | 419/500 [00:22<00:02, 34.50it/s] 85%|████████▌ | 426/500 [00:22<00:01, 42.83it/s] 86%|████████▋ | 432/500 [00:22<00:01, 46.39it/s] 87%|████████▋ | 437/500 [00:23<00:01, 46.32it/s] 89%|████████▊ | 443/500 [00:23<00:01, 49.16it/s] 90%|█████████ | 450/500 [00:23<00:00, 53.24it/s] 91%|█████████▏| 457/500 [00:23<00:00, 55.92it/s] 93%|█████████▎| 463/500 [00:23<00:00, 51.23it/s] 94%|█████████▍| 470/500 [00:23<00:00, 54.45it/s] 95%|█████████▌| 476/500 [00:23<00:00, 55.33it/s] 97%|█████████▋| 483/500 [00:23<00:00, 57.55it/s] 98%|█████████▊| 489/500 [00:23<00:00, 54.69it/s] 100%|██████████| 500/500 [00:24<00:00, 20.69it/s] 10%|█ | 1/10 [00:24<03:37, 24.22s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 6/500 [00:00<00:12, 38.60it/s] 2%|▏ | 10/500 [00:02<02:27, 3.33it/s] 3%|▎ | 16/500 [00:02<01:15, 6.41it/s] 5%|▍ | 23/500 [00:02<00:43, 10.90it/s] 6%|▌ | 28/500 [00:02<00:33, 14.25it/s] 7%|▋ | 34/500 [00:02<00:24, 19.36it/s] 8%|▊ | 40/500 [00:03<00:18, 24.71it/s] 9%|▉ | 46/500 [00:03<00:15, 30.12it/s] 10%|█ | 52/500 [00:03<00:12, 34.98it/s] 12%|█▏ | 58/500 [00:03<00:11, 38.72it/s] 13%|█▎ | 64/500 [00:03<00:10, 42.86it/s] 14%|█▍ | 70/500 [00:03<00:09, 46.33it/s] 15%|█▌ | 76/500 [00:03<00:09, 46.57it/s] 16%|█▋ | 82/500 [00:03<00:09, 44.90it/s] 18%|█▊ | 88/500 [00:04<00:08, 47.82it/s] 19%|█▉ | 94/500 [00:04<00:08, 49.83it/s] 20%|██ | 100/500 [00:04<00:08, 49.02it/s] 21%|██ | 106/500 [00:04<00:07, 50.57it/s] 22%|██▏ | 112/500 [00:04<00:08, 47.84it/s] 24%|██▎ | 118/500 [00:04<00:07, 50.64it/s] 25%|██▍ | 124/500 [00:04<00:07, 53.05it/s] 26%|██▌ | 130/500 [00:04<00:07, 48.15it/s] 27%|██▋ | 136/500 [00:04<00:07, 50.66it/s] 28%|██▊ | 142/500 [00:05<00:07, 50.64it/s] 30%|██▉ | 148/500 [00:05<00:07, 49.50it/s] 31%|███ | 154/500 [00:05<00:06, 51.33it/s] 32%|███▏ | 160/500 [00:05<00:06, 51.94it/s] 33%|███▎ | 166/500 [00:05<00:06, 50.03it/s] 34%|███▍ | 172/500 [00:05<00:06, 48.99it/s] 36%|███▌ | 178/500 [00:05<00:06, 50.41it/s] 37%|███▋ | 184/500 [00:05<00:06, 49.09it/s] 38%|███▊ | 189/500 [00:06<00:06, 48.37it/s] 39%|███▉ | 195/500 [00:06<00:06, 49.90it/s] 40%|████ | 201/500 [00:06<00:05, 52.05it/s] 41%|████▏ | 207/500 [00:06<00:05, 53.80it/s] 43%|████▎ | 213/500 [00:06<00:05, 55.35it/s] 44%|████▍ | 219/500 [00:06<00:05, 51.28it/s] 45%|████▌ | 226/500 [00:06<00:05, 54.32it/s] 46%|████▋ | 232/500 [00:06<00:04, 54.73it/s] 48%|████▊ | 238/500 [00:06<00:04, 54.68it/s] 49%|████▉ | 244/500 [00:07<00:04, 52.35it/s] 50%|█████ | 250/500 [00:07<00:04, 53.03it/s] 51%|█████ | 256/500 [00:07<00:04, 54.67it/s] 53%|█████▎ | 263/500 [00:07<00:04, 57.11it/s] 54%|█████▍ | 269/500 [00:07<00:04, 57.35it/s] 55%|█████▌ | 275/500 [00:07<00:04, 53.13it/s] 56%|█████▌ | 281/500 [00:07<00:04, 54.55it/s] 57%|█████▋ | 287/500 [00:07<00:04, 52.97it/s] 59%|█████▊ | 293/500 [00:07<00:03, 53.80it/s] 60%|██████ | 300/500 [00:08<00:03, 56.82it/s] 61%|██████ | 306/500 [00:08<00:03, 56.64it/s] 62%|██████▏ | 312/500 [00:08<00:03, 53.84it/s] 64%|██████▎ | 318/500 [00:08<00:04, 43.56it/s] 65%|██████▍ | 323/500 [00:08<00:04, 36.76it/s] 66%|██████▌ | 328/500 [00:08<00:05, 33.81it/s] 66%|██████▋ | 332/500 [00:08<00:05, 32.65it/s] 67%|██████▋ | 336/500 [00:09<00:04, 32.91it/s] 68%|██████▊ | 340/500 [00:09<00:04, 33.01it/s] 69%|██████▉ | 344/500 [00:09<00:04, 32.25it/s] 70%|██████▉ | 348/500 [00:09<00:04, 31.65it/s] 70%|███████ | 352/500 [00:09<00:04, 30.94it/s] 71%|███████ | 356/500 [00:09<00:04, 30.41it/s] 72%|███████▏ | 360/500 [00:09<00:04, 28.47it/s] 73%|███████▎ | 363/500 [00:10<00:05, 27.40it/s] 73%|███████▎ | 367/500 [00:10<00:04, 27.12it/s] 74%|███████▍ | 370/500 [00:10<00:04, 27.19it/s] 75%|███████▍ | 373/500 [00:10<00:04, 26.11it/s] 75%|███████▌ | 376/500 [00:10<00:05, 24.51it/s] 76%|███████▌ | 379/500 [00:10<00:05, 23.24it/s] 76%|███████▋ | 382/500 [00:10<00:05, 23.27it/s] 77%|███████▋ | 385/500 [00:10<00:04, 23.72it/s] 78%|███████▊ | 389/500 [00:11<00:04, 26.08it/s] 78%|███████▊ | 392/500 [00:11<00:04, 26.23it/s] 79%|███████▉ | 396/500 [00:11<00:03, 27.15it/s] 80%|███████▉ | 399/500 [00:11<00:04, 25.20it/s] 80%|████████ | 402/500 [00:11<00:04, 24.38it/s] 81%|████████ | 405/500 [00:11<00:03, 24.36it/s] 82%|████████▏ | 408/500 [00:11<00:03, 23.99it/s] 82%|████████▏ | 411/500 [00:11<00:03, 25.06it/s] 83%|████████▎ | 414/500 [00:12<00:03, 26.18it/s] 84%|████████▎ | 418/500 [00:12<00:02, 28.47it/s] 84%|████████▍ | 422/500 [00:12<00:02, 29.93it/s] 85%|████████▌ | 426/500 [00:12<00:02, 31.54it/s] 86%|████████▌ | 430/500 [00:12<00:02, 30.68it/s] 87%|████████▋ | 434/500 [00:12<00:02, 28.08it/s] 87%|████████▋ | 437/500 [00:12<00:02, 26.24it/s] 88%|████████▊ | 440/500 [00:12<00:02, 26.52it/s] 89%|████████▉ | 444/500 [00:13<00:02, 27.37it/s] 89%|████████▉ | 447/500 [00:13<00:01, 27.21it/s] 90%|█████████ | 451/500 [00:13<00:01, 29.68it/s] 91%|█████████ | 455/500 [00:13<00:01, 28.34it/s] 92%|█████████▏| 458/500 [00:13<00:01, 27.19it/s] 92%|█████████▏| 462/500 [00:13<00:01, 28.22it/s] 93%|█████████▎| 465/500 [00:13<00:01, 26.12it/s] 94%|█████████▎| 468/500 [00:14<00:01, 26.13it/s] 94%|█████████▍| 472/500 [00:14<00:00, 28.33it/s] 95%|█████████▌| 476/500 [00:14<00:00, 29.76it/s] 96%|█████████▌| 480/500 [00:14<00:00, 30.47it/s] 97%|█████████▋| 484/500 [00:14<00:00, 30.64it/s] 98%|█████████▊| 488/500 [00:14<00:00, 31.34it/s] 98%|█████████▊| 492/500 [00:14<00:00, 27.50it/s] 99%|█████████▉| 495/500 [00:14<00:00, 27.92it/s] 100%|██████████| 500/500 [00:15<00:00, 33.14it/s] 20%|██ | 2/10 [00:39<02:30, 18.87s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 3/500 [00:00<00:16, 29.36it/s] 1%| | 6/500 [00:00<00:22, 22.19it/s] 2%|▏ | 9/500 [00:02<03:09, 2.59it/s] 3%|▎ | 15/500 [00:02<01:25, 5.65it/s] 4%|▍ | 21/500 [00:02<00:50, 9.43it/s] 5%|▌ | 27/500 [00:02<00:33, 14.03it/s] 7%|▋ | 33/500 [00:03<00:24, 19.27it/s] 8%|▊ | 39/500 [00:03<00:19, 23.13it/s] 9%|▉ | 45/500 [00:03<00:15, 28.65it/s] 10%|█ | 51/500 [00:03<00:13, 34.01it/s] 11%|█▏ | 57/500 [00:03<00:12, 36.85it/s] 12%|█▏ | 62/500 [00:03<00:11, 37.88it/s] 14%|█▎ | 68/500 [00:03<00:10, 42.49it/s] 15%|█▍ | 74/500 [00:03<00:09, 45.12it/s] 16%|█▌ | 80/500 [00:03<00:08, 48.58it/s] 17%|█▋ | 86/500 [00:04<00:08, 49.02it/s] 18%|█▊ | 92/500 [00:04<00:08, 46.01it/s] 20%|█▉ | 98/500 [00:04<00:08, 48.35it/s] 21%|██ | 104/500 [00:04<00:07, 51.11it/s] 22%|██▏ | 110/500 [00:04<00:07, 50.31it/s] 23%|██▎ | 116/500 [00:04<00:08, 45.94it/s] 24%|██▍ | 122/500 [00:04<00:07, 48.87it/s] 26%|██▌ | 128/500 [00:04<00:07, 47.45it/s] 27%|██▋ | 134/500 [00:05<00:07, 50.02it/s] 28%|██▊ | 140/500 [00:05<00:06, 51.43it/s] 29%|██▉ | 146/500 [00:05<00:07, 48.24it/s] 30%|███ | 152/500 [00:05<00:06, 51.04it/s] 32%|███▏ | 158/500 [00:05<00:06, 51.83it/s] 33%|███▎ | 164/500 [00:05<00:06, 50.82it/s] 34%|███▍ | 170/500 [00:05<00:06, 52.69it/s] 35%|███▌ | 176/500 [00:05<00:06, 53.64it/s] 37%|███▋ | 183/500 [00:06<00:05, 56.94it/s] 38%|███▊ | 190/500 [00:06<00:05, 58.59it/s] 39%|███▉ | 196/500 [00:06<00:05, 58.47it/s] 40%|████ | 202/500 [00:06<00:05, 54.81it/s] 42%|████▏ | 208/500 [00:06<00:05, 55.36it/s] 43%|████▎ | 214/500 [00:06<00:05, 55.02it/s] 44%|████▍ | 220/500 [00:06<00:05, 52.93it/s] 45%|████▌ | 226/500 [00:06<00:05, 53.22it/s] 46%|████▋ | 232/500 [00:06<00:04, 54.37it/s] 48%|████▊ | 238/500 [00:07<00:04, 55.60it/s] 49%|████▉ | 244/500 [00:07<00:04, 55.97it/s] 50%|█████ | 250/500 [00:07<00:04, 54.29it/s] 51%|█████ | 256/500 [00:07<00:04, 54.05it/s] 52%|█████▏ | 262/500 [00:07<00:04, 53.61it/s] 54%|█████▎ | 268/500 [00:07<00:04, 51.66it/s] 55%|█████▍ | 274/500 [00:07<00:04, 51.72it/s] 56%|█████▌ | 280/500 [00:07<00:04, 52.95it/s] 57%|█████▋ | 286/500 [00:07<00:03, 54.02it/s] 58%|█████▊ | 292/500 [00:08<00:03, 55.20it/s] 60%|█████▉ | 298/500 [00:08<00:03, 56.03it/s] 61%|██████ | 304/500 [00:08<00:03, 56.63it/s] 62%|██████▏ | 310/500 [00:08<00:03, 55.91it/s] 63%|██████▎ | 316/500 [00:08<00:03, 53.62it/s] 64%|██████▍ | 322/500 [00:08<00:03, 53.55it/s] 66%|██████▌ | 328/500 [00:08<00:03, 52.16it/s] 67%|██████▋ | 334/500 [00:08<00:03, 52.38it/s] 68%|██████▊ | 340/500 [00:08<00:03, 51.05it/s] 69%|██████▉ | 346/500 [00:09<00:02, 53.33it/s] 70%|███████ | 352/500 [00:09<00:02, 54.80it/s] 72%|███████▏ | 358/500 [00:09<00:02, 52.00it/s] 73%|███████▎ | 364/500 [00:09<00:02, 53.63it/s] 74%|███████▍ | 370/500 [00:09<00:02, 51.22it/s] 75%|███████▌ | 376/500 [00:09<00:02, 49.47it/s] 76%|███████▋ | 382/500 [00:09<00:02, 48.37it/s] 78%|███████▊ | 388/500 [00:09<00:02, 50.27it/s] 79%|███████▉ | 394/500 [00:09<00:02, 51.95it/s] 80%|████████ | 400/500 [00:10<00:01, 52.66it/s] 81%|████████ | 406/500 [00:10<00:01, 50.14it/s] 82%|████████▏ | 412/500 [00:10<00:01, 51.92it/s] 84%|████████▎ | 418/500 [00:10<00:01, 50.68it/s] 85%|████████▍ | 424/500 [00:10<00:01, 41.31it/s] 86%|████████▌ | 429/500 [00:10<00:01, 41.46it/s] 87%|████████▋ | 434/500 [00:10<00:01, 39.27it/s] 88%|████████▊ | 439/500 [00:11<00:01, 33.55it/s] 89%|████████▊ | 443/500 [00:11<00:01, 32.70it/s] 89%|████████▉ | 447/500 [00:11<00:01, 30.93it/s] 90%|█████████ | 451/500 [00:11<00:01, 30.08it/s] 91%|█████████ | 455/500 [00:11<00:01, 28.07it/s] 92%|█████████▏| 458/500 [00:11<00:01, 28.33it/s] 92%|█████████▏| 461/500 [00:11<00:01, 28.18it/s] 93%|█████████▎| 464/500 [00:12<00:01, 28.17it/s] 94%|█████████▎| 468/500 [00:12<00:01, 29.21it/s] 94%|█████████▍| 471/500 [00:12<00:01, 28.68it/s] 95%|█████████▍| 474/500 [00:12<00:00, 28.37it/s] 95%|█████████▌| 477/500 [00:12<00:00, 28.06it/s] 96%|█████████▌| 480/500 [00:12<00:00, 28.25it/s] 97%|█████████▋| 483/500 [00:12<00:00, 27.29it/s] 97%|█████████▋| 486/500 [00:12<00:00, 27.18it/s] 98%|█████████▊| 489/500 [00:12<00:00, 26.88it/s] 98%|█████████▊| 492/500 [00:13<00:00, 26.25it/s] 99%|█████████▉| 495/500 [00:13<00:00, 26.44it/s] 100%|██████████| 500/500 [00:13<00:00, 37.42it/s] 30%|███ | 3/10 [00:52<01:54, 16.38s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 4/500 [00:00<00:14, 34.21it/s] 2%|▏ | 8/500 [00:02<03:26, 2.38it/s] 3%|▎ | 14/500 [00:02<01:34, 5.12it/s] 4%|▍ | 19/500 [00:03<01:00, 7.96it/s] 5%|▌ | 25/500 [00:03<00:38, 12.18it/s] 6%|▌ | 31/500 [00:03<00:27, 17.08it/s] 7%|▋ | 37/500 [00:03<00:20, 22.34it/s] 9%|▊ | 43/500 [00:03<00:16, 27.60it/s] 10%|▉ | 49/500 [00:03<00:13, 32.45it/s] 11%|█ | 55/500 [00:03<00:12, 36.24it/s] 12%|█▏ | 61/500 [00:03<00:10, 40.36it/s] 13%|█▎ | 67/500 [00:04<00:09, 43.67it/s] 15%|█▍ | 73/500 [00:04<00:09, 47.13it/s] 16%|█▌ | 79/500 [00:04<00:08, 46.80it/s] 17%|█▋ | 85/500 [00:04<00:09, 43.12it/s] 18%|█▊ | 91/500 [00:04<00:08, 45.97it/s] 19%|█▉ | 96/500 [00:04<00:08, 45.29it/s] 20%|██ | 102/500 [00:04<00:08, 47.75it/s] 22%|██▏ | 108/500 [00:04<00:08, 48.87it/s] 23%|██▎ | 114/500 [00:04<00:07, 50.70it/s] 24%|██▍ | 120/500 [00:05<00:07, 51.70it/s] 25%|██▌ | 126/500 [00:05<00:06, 53.76it/s] 26%|██▋ | 132/500 [00:05<00:07, 47.61it/s] 27%|██▋ | 137/500 [00:05<00:07, 47.53it/s] 29%|██▊ | 143/500 [00:05<00:07, 49.89it/s] 30%|██▉ | 149/500 [00:05<00:07, 50.06it/s] 31%|███ | 155/500 [00:05<00:07, 47.91it/s] 32%|███▏ | 161/500 [00:05<00:06, 50.94it/s] 34%|███▎ | 168/500 [00:06<00:06, 54.25it/s] 35%|███▌ | 175/500 [00:06<00:05, 56.68it/s] 36%|███▌ | 181/500 [00:06<00:06, 49.78it/s] 37%|███▋ | 187/500 [00:06<00:06, 52.10it/s] 39%|███▉ | 194/500 [00:06<00:05, 54.87it/s] 40%|████ | 201/500 [00:06<00:05, 57.11it/s] 41%|████▏ | 207/500 [00:06<00:05, 56.45it/s] 43%|████▎ | 213/500 [00:06<00:05, 54.65it/s] 44%|████▍ | 219/500 [00:06<00:05, 55.30it/s] 45%|████▌ | 225/500 [00:07<00:05, 53.28it/s] 46%|████▌ | 231/500 [00:07<00:04, 54.32it/s] 47%|████▋ | 237/500 [00:07<00:05, 47.01it/s] 49%|████▉ | 244/500 [00:07<00:04, 51.55it/s] 50%|█████ | 250/500 [00:07<00:04, 52.63it/s] 51%|█████▏ | 257/500 [00:07<00:04, 55.50it/s] 53%|█████▎ | 263/500 [00:07<00:04, 55.41it/s] 54%|█████▍ | 269/500 [00:07<00:04, 53.31it/s] 55%|█████▌ | 276/500 [00:08<00:03, 56.32it/s] 57%|█████▋ | 283/500 [00:08<00:03, 57.69it/s] 58%|█████▊ | 289/500 [00:08<00:04, 50.05it/s] 59%|█████▉ | 295/500 [00:08<00:04, 46.62it/s] 60%|██████ | 300/500 [00:08<00:04, 47.33it/s] 61%|██████ | 306/500 [00:08<00:03, 49.88it/s] 62%|██████▏ | 312/500 [00:08<00:03, 51.26it/s] 64%|██████▎ | 318/500 [00:08<00:03, 48.08it/s] 65%|██████▌ | 325/500 [00:09<00:03, 52.10it/s] 66%|██████▋ | 332/500 [00:09<00:03, 54.86it/s] 68%|██████▊ | 338/500 [00:09<00:02, 54.23it/s] 69%|██████▉ | 344/500 [00:09<00:02, 53.45it/s] 70%|███████ | 351/500 [00:09<00:02, 56.28it/s] 72%|███████▏ | 358/500 [00:09<00:02, 58.03it/s] 73%|███████▎ | 364/500 [00:09<00:02, 53.71it/s] 74%|███████▍ | 370/500 [00:09<00:02, 52.24it/s] 75%|███████▌ | 376/500 [00:09<00:02, 50.09it/s] 76%|███████▋ | 382/500 [00:10<00:02, 52.12it/s] 78%|███████▊ | 388/500 [00:10<00:02, 52.42it/s] 79%|███████▉ | 394/500 [00:10<00:02, 48.96it/s] 80%|████████ | 400/500 [00:10<00:01, 51.42it/s] 81%|████████ | 406/500 [00:10<00:01, 51.89it/s] 82%|████████▏ | 412/500 [00:10<00:01, 53.93it/s] 84%|████████▍ | 419/500 [00:10<00:01, 57.03it/s] 85%|████████▌ | 425/500 [00:10<00:01, 56.73it/s] 86%|████████▌ | 431/500 [00:11<00:01, 49.45it/s] 87%|████████▋ | 437/500 [00:11<00:01, 51.85it/s] 89%|████████▊ | 443/500 [00:11<00:01, 51.99it/s] 90%|████████▉ | 449/500 [00:11<00:00, 52.55it/s] 91%|█████████ | 455/500 [00:11<00:00, 53.86it/s] 92%|█████████▏| 462/500 [00:11<00:00, 56.76it/s] 94%|█████████▎| 468/500 [00:11<00:00, 56.83it/s] 95%|█████████▌| 475/500 [00:11<00:00, 58.29it/s] 96%|█████████▌| 481/500 [00:11<00:00, 57.60it/s] 97%|█████████▋| 487/500 [00:12<00:00, 55.41it/s] 99%|█████████▊| 493/500 [00:12<00:00, 55.87it/s] 100%|██████████| 500/500 [00:12<00:00, 40.69it/s] 40%|████ | 4/10 [01:05<01:28, 14.78s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 6/500 [00:00<00:08, 57.91it/s] 2%|▏ | 12/500 [00:02<02:01, 4.00it/s] 3%|▎ | 16/500 [00:02<01:22, 5.88it/s] 4%|▍ | 20/500 [00:02<00:59, 8.12it/s] 5%|▍ | 24/500 [00:02<00:44, 10.62it/s] 6%|▌ | 28/500 [00:03<00:35, 13.48it/s] 6%|▋ | 32/500 [00:03<00:28, 16.66it/s] 7%|▋ | 36/500 [00:03<00:23, 19.76it/s] 8%|▊ | 40/500 [00:03<00:21, 21.25it/s] 9%|▉ | 44/500 [00:03<00:20, 22.31it/s] 9%|▉ | 47/500 [00:03<00:19, 23.54it/s] 10%|█ | 50/500 [00:03<00:19, 23.57it/s] 11%|█ | 54/500 [00:03<00:16, 26.27it/s] 11%|█▏ | 57/500 [00:04<00:16, 26.53it/s] 12%|█▏ | 60/500 [00:04<00:16, 26.88it/s] 13%|█▎ | 63/500 [00:04<00:16, 26.07it/s] 13%|█▎ | 67/500 [00:04<00:15, 28.34it/s] 14%|█▍ | 70/500 [00:04<00:16, 25.96it/s] 15%|█▍ | 73/500 [00:04<00:17, 24.78it/s] 15%|█▌ | 76/500 [00:04<00:16, 25.00it/s] 16%|█▌ | 79/500 [00:04<00:17, 24.72it/s] 17%|█▋ | 83/500 [00:05<00:15, 27.49it/s] 17%|█▋ | 87/500 [00:05<00:13, 30.26it/s] 18%|█▊ | 91/500 [00:05<00:13, 30.77it/s] 19%|█▉ | 95/500 [00:05<00:12, 32.18it/s] 20%|█▉ | 99/500 [00:05<00:12, 32.44it/s] 21%|██ | 103/500 [00:05<00:11, 33.16it/s] 21%|██▏ | 107/500 [00:05<00:11, 33.79it/s] 22%|██▏ | 111/500 [00:05<00:11, 33.37it/s] 23%|██▎ | 115/500 [00:06<00:12, 30.56it/s] 24%|██▍ | 119/500 [00:06<00:12, 29.62it/s] 25%|██▍ | 123/500 [00:06<00:12, 31.13it/s] 25%|██▌ | 127/500 [00:06<00:11, 31.16it/s] 26%|██▌ | 131/500 [00:06<00:11, 31.64it/s] 27%|██▋ | 135/500 [00:06<00:10, 33.34it/s] 28%|██▊ | 139/500 [00:06<00:10, 33.58it/s] 29%|██▊ | 143/500 [00:06<00:10, 32.51it/s] 29%|██▉ | 147/500 [00:07<00:11, 31.26it/s] 30%|███ | 151/500 [00:07<00:10, 31.76it/s] 31%|███▏ | 157/500 [00:07<00:08, 38.58it/s] 32%|███▏ | 161/500 [00:07<00:08, 38.01it/s] 33%|███▎ | 166/500 [00:07<00:08, 41.06it/s] 34%|███▍ | 172/500 [00:07<00:07, 44.52it/s] 36%|███▌ | 178/500 [00:07<00:06, 48.05it/s] 37%|███▋ | 184/500 [00:07<00:06, 48.39it/s] 38%|███▊ | 189/500 [00:07<00:07, 43.14it/s] 39%|███▉ | 195/500 [00:08<00:06, 46.82it/s] 40%|████ | 200/500 [00:08<00:06, 47.36it/s] 41%|████ | 206/500 [00:08<00:06, 48.90it/s] 42%|████▏ | 211/500 [00:08<00:06, 47.55it/s] 43%|████▎ | 217/500 [00:08<00:05, 49.12it/s] 45%|████▍ | 223/500 [00:08<00:05, 51.63it/s] 46%|████▌ | 230/500 [00:08<00:04, 54.54it/s] 47%|████▋ | 236/500 [00:08<00:04, 55.97it/s] 48%|████▊ | 242/500 [00:08<00:04, 52.81it/s] 50%|████▉ | 249/500 [00:09<00:04, 55.46it/s] 51%|█████ | 255/500 [00:09<00:04, 53.71it/s] 52%|█████▏ | 262/500 [00:09<00:04, 56.19it/s] 54%|█████▎ | 268/500 [00:09<00:04, 53.99it/s] 55%|█████▌ | 275/500 [00:09<00:03, 56.28it/s] 56%|█████▋ | 282/500 [00:09<00:03, 57.88it/s] 58%|█████▊ | 288/500 [00:09<00:03, 58.42it/s] 59%|█████▉ | 294/500 [00:09<00:03, 57.63it/s] 60%|██████ | 300/500 [00:09<00:03, 57.01it/s] 61%|██████ | 306/500 [00:10<00:03, 56.10it/s] 62%|██████▏ | 312/500 [00:10<00:03, 54.44it/s] 64%|██████▎ | 318/500 [00:10<00:03, 54.68it/s] 65%|██████▍ | 324/500 [00:10<00:03, 53.12it/s] 66%|██████▌ | 330/500 [00:10<00:03, 52.06it/s] 67%|██████▋ | 336/500 [00:10<00:03, 53.38it/s] 69%|██████▊ | 343/500 [00:10<00:02, 56.20it/s] 70%|██████▉ | 349/500 [00:10<00:02, 55.24it/s] 71%|███████ | 355/500 [00:11<00:02, 52.07it/s] 72%|███████▏ | 361/500 [00:11<00:02, 53.86it/s] 73%|███████▎ | 367/500 [00:11<00:02, 54.20it/s] 75%|███████▍ | 373/500 [00:11<00:02, 54.46it/s] 76%|███████▌ | 379/500 [00:11<00:02, 55.34it/s] 77%|███████▋ | 385/500 [00:11<00:02, 50.17it/s] 78%|███████▊ | 391/500 [00:11<00:02, 49.38it/s] 79%|███████▉ | 397/500 [00:11<00:02, 50.61it/s] 81%|████████ | 403/500 [00:11<00:02, 47.32it/s] 82%|████████▏ | 409/500 [00:12<00:01, 49.00it/s] 83%|████████▎ | 415/500 [00:12<00:01, 50.95it/s] 84%|████████▍ | 422/500 [00:12<00:01, 54.07it/s] 86%|████████▌ | 429/500 [00:12<00:01, 56.13it/s] 87%|████████▋ | 435/500 [00:12<00:01, 56.65it/s] 88%|████████▊ | 441/500 [00:12<00:01, 53.99it/s] 90%|████████▉ | 448/500 [00:12<00:00, 57.01it/s] 91%|█████████ | 454/500 [00:12<00:00, 57.31it/s] 92%|█████████▏| 460/500 [00:13<00:00, 50.97it/s] 93%|█████████▎| 466/500 [00:13<00:00, 53.19it/s] 94%|█████████▍| 472/500 [00:13<00:00, 54.26it/s] 96%|█████████▌| 478/500 [00:13<00:00, 54.88it/s] 97%|█████████▋| 484/500 [00:13<00:00, 56.21it/s] 98%|█████████▊| 490/500 [00:13<00:00, 55.13it/s] 100%|██████████| 500/500 [00:13<00:00, 36.37it/s] 50%|█████ | 5/10 [01:18<01:12, 14.43s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 5/500 [00:00<00:11, 41.40it/s] 2%|▏ | 10/500 [00:02<02:11, 3.73it/s] 3%|▎ | 15/500 [00:02<01:15, 6.45it/s] 4%|▍ | 21/500 [00:02<00:45, 10.58it/s] 5%|▌ | 27/500 [00:02<00:30, 15.35it/s] 7%|▋ | 33/500 [00:02<00:22, 20.59it/s] 8%|▊ | 38/500 [00:02<00:18, 24.83it/s] 9%|▊ | 43/500 [00:02<00:16, 28.09it/s] 10%|▉ | 48/500 [00:03<00:14, 30.49it/s] 11%|█ | 53/500 [00:03<00:13, 32.87it/s] 12%|█▏ | 60/500 [00:03<00:11, 39.84it/s] 13%|█▎ | 65/500 [00:03<00:11, 38.43it/s] 14%|█▍ | 70/500 [00:03<00:10, 40.19it/s] 15%|█▌ | 75/500 [00:03<00:11, 37.82it/s] 16%|█▌ | 80/500 [00:03<00:11, 36.80it/s] 17%|█▋ | 84/500 [00:04<00:12, 34.21it/s] 18%|█▊ | 88/500 [00:04<00:12, 31.79it/s] 18%|█▊ | 92/500 [00:04<00:12, 33.65it/s] 19%|█▉ | 96/500 [00:04<00:12, 32.39it/s] 20%|██ | 100/500 [00:04<00:12, 31.69it/s] 21%|██ | 104/500 [00:04<00:13, 29.96it/s] 22%|██▏ | 108/500 [00:04<00:12, 30.79it/s] 22%|██▏ | 112/500 [00:04<00:12, 30.69it/s] 23%|██▎ | 116/500 [00:05<00:12, 30.22it/s] 24%|██▍ | 120/500 [00:05<00:12, 29.36it/s] 25%|██▍ | 123/500 [00:05<00:13, 28.82it/s] 25%|██▌ | 127/500 [00:05<00:11, 31.33it/s] 26%|██▌ | 131/500 [00:05<00:11, 31.30it/s] 27%|██▋ | 135/500 [00:05<00:11, 32.66it/s] 28%|██▊ | 139/500 [00:05<00:10, 33.14it/s] 29%|██▊ | 143/500 [00:05<00:12, 28.58it/s] 29%|██▉ | 146/500 [00:06<00:13, 26.47it/s] 30%|██▉ | 149/500 [00:06<00:13, 25.66it/s] 30%|███ | 152/500 [00:06<00:13, 25.58it/s] 31%|███ | 155/500 [00:06<00:13, 25.68it/s] 32%|███▏ | 159/500 [00:06<00:12, 28.02it/s] 33%|███▎ | 163/500 [00:06<00:11, 29.10it/s] 33%|███▎ | 166/500 [00:06<00:11, 28.67it/s] 34%|███▍ | 169/500 [00:06<00:11, 28.56it/s] 35%|███▍ | 173/500 [00:07<00:10, 30.93it/s] 35%|███▌ | 177/500 [00:07<00:10, 31.34it/s] 36%|███▌ | 181/500 [00:07<00:09, 32.07it/s] 37%|███▋ | 185/500 [00:07<00:09, 32.93it/s] 38%|███▊ | 189/500 [00:07<00:09, 32.89it/s] 39%|███▊ | 193/500 [00:07<00:09, 33.39it/s] 39%|███▉ | 197/500 [00:07<00:08, 33.67it/s] 40%|████ | 201/500 [00:07<00:08, 34.25it/s] 41%|████ | 205/500 [00:08<00:09, 32.61it/s] 42%|████▏ | 209/500 [00:08<00:09, 31.04it/s] 43%|████▎ | 213/500 [00:08<00:10, 27.38it/s] 43%|████▎ | 216/500 [00:08<00:10, 26.11it/s] 44%|████▍ | 219/500 [00:08<00:10, 26.27it/s] 44%|████▍ | 222/500 [00:08<00:10, 26.67it/s] 45%|████▌ | 225/500 [00:08<00:10, 26.57it/s] 46%|████▌ | 228/500 [00:08<00:09, 27.34it/s] 46%|████▌ | 231/500 [00:09<00:10, 26.65it/s] 47%|████▋ | 234/500 [00:09<00:10, 25.46it/s] 47%|████▋ | 237/500 [00:09<00:09, 26.64it/s] 48%|████▊ | 240/500 [00:09<00:09, 26.12it/s] 49%|████▉ | 244/500 [00:09<00:08, 28.73it/s] 49%|████▉ | 247/500 [00:09<00:09, 27.52it/s] 50%|█████ | 250/500 [00:09<00:09, 27.02it/s] 51%|█████ | 253/500 [00:09<00:09, 25.62it/s] 51%|█████ | 256/500 [00:10<00:10, 23.38it/s] 52%|█████▏ | 259/500 [00:10<00:10, 23.10it/s] 52%|█████▏ | 262/500 [00:10<00:10, 23.29it/s] 53%|█████▎ | 265/500 [00:10<00:09, 24.81it/s] 54%|█████▎ | 268/500 [00:10<00:09, 24.00it/s] 54%|█████▍ | 271/500 [00:10<00:09, 24.76it/s] 55%|█████▌ | 275/500 [00:10<00:08, 26.82it/s] 56%|█████▌ | 279/500 [00:10<00:07, 29.58it/s] 57%|█████▋ | 284/500 [00:10<00:06, 34.73it/s] 58%|█████▊ | 289/500 [00:11<00:05, 36.36it/s] 59%|█████▉ | 295/500 [00:11<00:04, 41.53it/s] 60%|██████ | 301/500 [00:11<00:04, 45.46it/s] 61%|██████▏ | 307/500 [00:11<00:03, 48.49it/s] 63%|██████▎ | 313/500 [00:11<00:03, 49.40it/s] 64%|██████▍ | 319/500 [00:11<00:03, 51.13it/s] 65%|██████▌ | 325/500 [00:11<00:03, 48.57it/s] 66%|██████▌ | 331/500 [00:11<00:03, 50.93it/s] 67%|██████▋ | 337/500 [00:11<00:03, 52.65it/s] 69%|██████▊ | 343/500 [00:12<00:03, 46.86it/s] 70%|██████▉ | 349/500 [00:12<00:03, 49.43it/s] 71%|███████ | 355/500 [00:12<00:02, 49.23it/s] 72%|███████▏ | 361/500 [00:12<00:02, 51.51it/s] 73%|███████▎ | 367/500 [00:12<00:02, 53.14it/s] 75%|███████▍ | 373/500 [00:12<00:02, 53.41it/s] 76%|███████▌ | 379/500 [00:12<00:02, 50.58it/s] 77%|███████▋ | 385/500 [00:12<00:02, 50.23it/s] 78%|███████▊ | 391/500 [00:13<00:02, 48.87it/s] 79%|███████▉ | 396/500 [00:13<00:02, 44.94it/s] 80%|████████ | 402/500 [00:13<00:02, 48.09it/s] 82%|████████▏ | 408/500 [00:13<00:01, 49.74it/s] 83%|████████▎ | 414/500 [00:13<00:01, 50.46it/s] 84%|████████▍ | 420/500 [00:13<00:01, 48.44it/s] 85%|████████▌ | 426/500 [00:13<00:01, 48.83it/s] 86%|████████▋ | 432/500 [00:13<00:01, 50.39it/s] 88%|████████▊ | 438/500 [00:14<00:01, 52.34it/s] 89%|████████▉ | 444/500 [00:14<00:01, 48.89it/s] 90%|█████████ | 450/500 [00:14<00:00, 51.45it/s] 91%|█████████ | 456/500 [00:14<00:00, 53.28it/s] 92%|█████████▏| 462/500 [00:14<00:00, 54.26it/s] 94%|█████████▎| 468/500 [00:14<00:00, 55.26it/s] 95%|█████████▍| 474/500 [00:14<00:00, 55.50it/s] 96%|█████████▌| 481/500 [00:14<00:00, 57.23it/s] 97%|█████████▋| 487/500 [00:14<00:00, 49.61it/s] 99%|█████████▊| 493/500 [00:15<00:00, 47.23it/s] 100%|██████████| 500/500 [00:15<00:00, 32.76it/s] 60%|██████ | 6/10 [01:34<00:58, 14.73s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 6/500 [00:00<00:08, 57.07it/s] 2%|▏ | 12/500 [00:02<01:37, 4.99it/s] 4%|▎ | 18/500 [00:02<00:56, 8.58it/s] 5%|▍ | 24/500 [00:02<00:36, 12.89it/s] 6%|▌ | 29/500 [00:02<00:28, 16.62it/s] 7%|▋ | 35/500 [00:02<00:20, 22.17it/s] 8%|▊ | 41/500 [00:02<00:16, 27.63it/s] 9%|▉ | 47/500 [00:02<00:14, 32.29it/s] 11%|█ | 53/500 [00:02<00:12, 35.34it/s] 12%|█▏ | 60/500 [00:02<00:10, 41.91it/s] 13%|█▎ | 67/500 [00:03<00:09, 47.30it/s] 15%|█▍ | 73/500 [00:03<00:08, 49.96it/s] 16%|█▌ | 79/500 [00:03<00:08, 51.27it/s] 17%|█▋ | 85/500 [00:03<00:07, 52.33it/s] 18%|█▊ | 91/500 [00:03<00:07, 52.84it/s] 19%|█▉ | 97/500 [00:03<00:07, 54.71it/s] 21%|██ | 103/500 [00:03<00:08, 47.68it/s] 22%|██▏ | 109/500 [00:03<00:07, 49.07it/s] 23%|██▎ | 115/500 [00:04<00:07, 51.50it/s] 24%|██▍ | 121/500 [00:04<00:07, 52.54it/s] 25%|██▌ | 127/500 [00:04<00:07, 53.08it/s] 27%|██▋ | 133/500 [00:04<00:06, 53.45it/s] 28%|██▊ | 139/500 [00:04<00:06, 52.69it/s] 29%|██▉ | 145/500 [00:04<00:06, 54.64it/s] 30%|███ | 151/500 [00:04<00:06, 54.13it/s] 31%|███▏ | 157/500 [00:04<00:07, 48.84it/s] 33%|███▎ | 163/500 [00:04<00:07, 47.75it/s] 34%|███▍ | 169/500 [00:05<00:06, 50.58it/s] 35%|███▌ | 175/500 [00:05<00:06, 50.28it/s] 36%|███▌ | 181/500 [00:05<00:06, 52.54it/s] 37%|███▋ | 187/500 [00:05<00:05, 52.56it/s] 39%|███▊ | 193/500 [00:05<00:05, 54.21it/s] 40%|███▉ | 199/500 [00:05<00:06, 46.05it/s] 41%|████ | 204/500 [00:05<00:07, 40.79it/s] 42%|████▏ | 209/500 [00:05<00:07, 37.06it/s] 43%|████▎ | 213/500 [00:06<00:08, 35.51it/s] 43%|████▎ | 217/500 [00:06<00:08, 32.38it/s] 44%|████▍ | 221/500 [00:06<00:08, 31.45it/s] 45%|████▌ | 225/500 [00:06<00:09, 29.15it/s] 46%|████▌ | 228/500 [00:06<00:10, 26.93it/s] 46%|████▌ | 231/500 [00:06<00:10, 26.43it/s] 47%|████▋ | 234/500 [00:06<00:10, 26.39it/s] 47%|████▋ | 237/500 [00:07<00:09, 26.61it/s] 48%|████▊ | 240/500 [00:07<00:10, 25.81it/s] 49%|████▉ | 244/500 [00:07<00:09, 27.43it/s] 49%|████▉ | 247/500 [00:07<00:09, 27.61it/s] 50%|█████ | 250/500 [00:07<00:09, 26.73it/s] 51%|█████ | 253/500 [00:07<00:09, 25.77it/s] 51%|█████ | 256/500 [00:07<00:09, 24.61it/s] 52%|█████▏ | 259/500 [00:07<00:10, 23.62it/s] 52%|█████▏ | 262/500 [00:08<00:09, 23.84it/s] 53%|█████▎ | 265/500 [00:08<00:09, 25.38it/s] 54%|█████▍ | 271/500 [00:08<00:06, 32.74it/s] 55%|█████▌ | 275/500 [00:08<00:07, 31.27it/s] 56%|█████▌ | 279/500 [00:08<00:06, 32.29it/s] 57%|█████▋ | 283/500 [00:08<00:06, 32.02it/s] 57%|█████▋ | 287/500 [00:08<00:06, 32.26it/s] 58%|█████▊ | 291/500 [00:08<00:06, 31.44it/s] 59%|█████▉ | 295/500 [00:09<00:06, 31.99it/s] 60%|█████▉ | 299/500 [00:09<00:06, 32.76it/s] 61%|██████ | 303/500 [00:09<00:06, 31.12it/s] 61%|██████▏ | 307/500 [00:09<00:06, 31.10it/s] 62%|██████▏ | 311/500 [00:09<00:05, 32.17it/s] 63%|██████▎ | 315/500 [00:09<00:05, 31.98it/s] 64%|██████▍ | 319/500 [00:09<00:07, 23.89it/s] 64%|██████▍ | 322/500 [00:10<00:09, 19.65it/s] 65%|██████▌ | 325/500 [00:10<00:10, 16.47it/s] 65%|██████▌ | 327/500 [00:10<00:10, 16.32it/s] 66%|██████▌ | 329/500 [00:10<00:11, 14.99it/s] 66%|██████▌ | 331/500 [00:10<00:11, 14.44it/s] 67%|██████▋ | 333/500 [00:11<00:10, 15.24it/s] 67%|██████▋ | 336/500 [00:11<00:08, 18.25it/s] 68%|██████▊ | 339/500 [00:11<00:07, 20.34it/s] 68%|██████▊ | 342/500 [00:11<00:07, 22.26it/s] 69%|██████▉ | 346/500 [00:11<00:05, 25.88it/s] 70%|███████ | 350/500 [00:11<00:05, 28.40it/s] 71%|███████ | 354/500 [00:11<00:04, 30.00it/s] 72%|███████▏ | 358/500 [00:11<00:04, 30.93it/s] 72%|███████▏ | 362/500 [00:11<00:04, 31.28it/s] 73%|███████▎ | 366/500 [00:12<00:04, 32.38it/s] 74%|███████▍ | 370/500 [00:12<00:04, 30.88it/s] 75%|███████▍ | 374/500 [00:12<00:04, 30.86it/s] 76%|███████▌ | 378/500 [00:12<00:04, 28.96it/s] 76%|███████▌ | 381/500 [00:12<00:04, 25.57it/s] 77%|███████▋ | 384/500 [00:12<00:04, 23.91it/s] 77%|███████▋ | 387/500 [00:12<00:04, 24.97it/s] 78%|███████▊ | 392/500 [00:13<00:03, 30.51it/s] 80%|███████▉ | 399/500 [00:13<00:02, 39.09it/s] 81%|████████ | 406/500 [00:13<00:02, 45.74it/s] 82%|████████▏ | 412/500 [00:13<00:01, 49.49it/s] 84%|████████▎ | 418/500 [00:13<00:01, 51.91it/s] 85%|████████▍ | 424/500 [00:13<00:01, 49.50it/s] 86%|████████▌ | 430/500 [00:13<00:01, 45.59it/s] 87%|████████▋ | 435/500 [00:13<00:01, 45.61it/s] 88%|████████▊ | 440/500 [00:13<00:01, 46.58it/s] 89%|████████▉ | 446/500 [00:14<00:01, 49.04it/s] 90%|█████████ | 451/500 [00:14<00:01, 47.73it/s] 91%|█████████ | 456/500 [00:14<00:01, 43.90it/s] 92%|█████████▏| 461/500 [00:14<00:00, 41.37it/s] 93%|█████████▎| 466/500 [00:14<00:00, 39.53it/s] 94%|█████████▍| 471/500 [00:14<00:00, 41.55it/s] 95%|█████████▌| 476/500 [00:14<00:00, 41.30it/s] 96%|█████████▋| 482/500 [00:14<00:00, 43.96it/s] 97%|█████████▋| 487/500 [00:15<00:00, 43.62it/s] 99%|█████████▊| 493/500 [00:15<00:00, 47.16it/s] 100%|██████████| 500/500 [00:15<00:00, 32.72it/s] 70%|███████ | 7/10 [01:49<00:44, 14.93s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 6/500 [00:00<00:10, 45.94it/s] 2%|▏ | 11/500 [00:02<02:16, 3.59it/s] 3%|▎ | 17/500 [00:02<01:13, 6.55it/s] 4%|▍ | 22/500 [00:02<00:50, 9.53it/s] 5%|▌ | 27/500 [00:02<00:35, 13.19it/s] 6%|▋ | 32/500 [00:03<00:26, 17.39it/s] 8%|▊ | 38/500 [00:03<00:20, 23.04it/s] 9%|▉ | 44/500 [00:03<00:15, 28.69it/s] 10%|█ | 50/500 [00:03<00:14, 30.92it/s] 11%|█ | 55/500 [00:03<00:14, 30.89it/s] 12%|█▏ | 60/500 [00:03<00:13, 31.91it/s] 13%|█▎ | 66/500 [00:03<00:11, 36.90it/s] 14%|█▍ | 72/500 [00:03<00:10, 41.70it/s] 16%|█▌ | 78/500 [00:04<00:09, 45.29it/s] 17%|█▋ | 84/500 [00:04<00:08, 48.19it/s] 18%|█▊ | 90/500 [00:04<00:08, 49.48it/s] 19%|█▉ | 96/500 [00:04<00:08, 49.88it/s] 20%|██ | 102/500 [00:04<00:08, 47.82it/s] 21%|██▏ | 107/500 [00:04<00:09, 42.79it/s] 22%|██▏ | 112/500 [00:04<00:08, 43.72it/s] 23%|██▎ | 117/500 [00:04<00:08, 44.88it/s] 25%|██▍ | 123/500 [00:04<00:07, 47.66it/s] 26%|██▌ | 128/500 [00:05<00:08, 44.60it/s] 27%|██▋ | 134/500 [00:05<00:07, 47.71it/s] 28%|██▊ | 140/500 [00:05<00:07, 50.73it/s] 29%|██▉ | 146/500 [00:05<00:06, 52.35it/s] 30%|███ | 152/500 [00:05<00:07, 44.65it/s] 31%|███▏ | 157/500 [00:05<00:07, 45.45it/s] 32%|███▏ | 162/500 [00:05<00:07, 45.46it/s] 34%|███▎ | 168/500 [00:05<00:06, 48.55it/s] 35%|███▍ | 174/500 [00:06<00:06, 49.77it/s] 36%|███▌ | 180/500 [00:06<00:06, 49.54it/s] 37%|███▋ | 186/500 [00:06<00:06, 49.48it/s] 38%|███▊ | 192/500 [00:06<00:05, 51.66it/s] 40%|███▉ | 198/500 [00:06<00:07, 42.00it/s] 41%|████ | 203/500 [00:06<00:07, 40.99it/s] 42%|████▏ | 208/500 [00:06<00:06, 42.68it/s] 43%|████▎ | 214/500 [00:06<00:06, 45.97it/s] 44%|████▍ | 220/500 [00:07<00:05, 49.25it/s] 45%|████▌ | 226/500 [00:07<00:05, 50.78it/s] 47%|████▋ | 233/500 [00:07<00:04, 53.97it/s] 48%|████▊ | 239/500 [00:07<00:04, 55.01it/s] 49%|████▉ | 245/500 [00:07<00:05, 46.74it/s] 50%|█████ | 250/500 [00:07<00:05, 42.40it/s] 51%|█████ | 255/500 [00:07<00:05, 42.73it/s] 52%|█████▏ | 260/500 [00:07<00:05, 42.66it/s] 53%|█████▎ | 265/500 [00:08<00:05, 40.00it/s] 54%|█████▍ | 270/500 [00:08<00:06, 35.02it/s] 55%|█████▍ | 274/500 [00:08<00:06, 33.31it/s] 56%|█████▌ | 278/500 [00:08<00:07, 30.87it/s] 56%|█████▋ | 282/500 [00:08<00:07, 29.01it/s] 57%|█████▋ | 286/500 [00:08<00:07, 29.99it/s] 58%|█████▊ | 290/500 [00:08<00:06, 30.67it/s] 59%|█████▉ | 295/500 [00:09<00:05, 34.30it/s] 60%|█████▉ | 299/500 [00:09<00:06, 33.15it/s] 61%|██████ | 303/500 [00:09<00:06, 32.33it/s] 61%|██████▏ | 307/500 [00:09<00:06, 28.85it/s] 62%|██████▏ | 310/500 [00:09<00:06, 27.69it/s] 63%|██████▎ | 313/500 [00:09<00:07, 25.98it/s] 63%|██████▎ | 316/500 [00:09<00:07, 25.89it/s] 64%|██████▍ | 319/500 [00:09<00:07, 25.65it/s] 64%|██████▍ | 322/500 [00:10<00:06, 26.42it/s] 65%|██████▌ | 325/500 [00:10<00:06, 26.04it/s] 66%|██████▌ | 328/500 [00:10<00:06, 26.11it/s] 66%|██████▋ | 332/500 [00:10<00:05, 28.93it/s] 67%|██████▋ | 335/500 [00:10<00:06, 27.09it/s] 68%|██████▊ | 338/500 [00:10<00:06, 24.07it/s] 68%|██████▊ | 341/500 [00:10<00:06, 24.71it/s] 69%|██████▉ | 345/500 [00:10<00:05, 26.10it/s] 70%|██████▉ | 349/500 [00:11<00:05, 27.90it/s] 70%|███████ | 352/500 [00:11<00:05, 28.21it/s] 71%|███████ | 355/500 [00:11<00:05, 27.86it/s] 72%|███████▏ | 359/500 [00:11<00:04, 29.27it/s] 73%|███████▎ | 363/500 [00:11<00:04, 30.18it/s] 73%|███████▎ | 367/500 [00:11<00:04, 29.55it/s] 74%|███████▍ | 370/500 [00:11<00:04, 29.13it/s] 75%|███████▍ | 373/500 [00:11<00:04, 27.57it/s] 75%|███████▌ | 376/500 [00:12<00:04, 27.81it/s] 76%|███████▌ | 380/500 [00:12<00:04, 29.13it/s] 77%|███████▋ | 384/500 [00:12<00:03, 30.25it/s] 78%|███████▊ | 388/500 [00:12<00:03, 28.31it/s] 78%|███████▊ | 391/500 [00:12<00:04, 26.93it/s] 79%|███████▉ | 394/500 [00:12<00:04, 25.87it/s] 79%|███████▉ | 397/500 [00:12<00:04, 25.61it/s] 80%|████████ | 401/500 [00:12<00:03, 27.55it/s] 81%|████████ | 405/500 [00:13<00:03, 28.83it/s] 82%|████████▏ | 409/500 [00:13<00:03, 28.88it/s] 82%|████████▏ | 412/500 [00:13<00:03, 28.33it/s] 83%|████████▎ | 415/500 [00:13<00:03, 28.13it/s] 84%|████████▎ | 418/500 [00:13<00:03, 27.16it/s] 84%|████████▍ | 421/500 [00:13<00:02, 26.45it/s] 85%|████████▍ | 424/500 [00:13<00:02, 26.56it/s] 85%|████████▌ | 427/500 [00:13<00:02, 24.88it/s] 86%|████████▌ | 430/500 [00:14<00:02, 26.03it/s] 87%|████████▋ | 433/500 [00:14<00:02, 26.33it/s] 87%|████████▋ | 437/500 [00:14<00:02, 28.90it/s] 88%|████████▊ | 440/500 [00:14<00:02, 28.56it/s] 89%|████████▊ | 443/500 [00:14<00:01, 28.90it/s] 89%|████████▉ | 446/500 [00:14<00:01, 27.23it/s] 90%|████████▉ | 449/500 [00:14<00:01, 26.69it/s] 90%|█████████ | 452/500 [00:14<00:01, 24.95it/s] 91%|█████████ | 456/500 [00:14<00:01, 26.91it/s] 92%|█████████▏| 459/500 [00:15<00:01, 26.98it/s] 92%|█████████▏| 462/500 [00:15<00:01, 26.69it/s] 93%|█████████▎| 465/500 [00:15<00:01, 25.76it/s] 94%|█████████▎| 468/500 [00:15<00:01, 25.32it/s] 94%|█████████▍| 471/500 [00:15<00:01, 25.83it/s] 95%|█████████▌| 475/500 [00:15<00:00, 25.57it/s] 96%|█████████▌| 478/500 [00:15<00:00, 23.61it/s] 96%|█████████▌| 481/500 [00:16<00:00, 22.12it/s] 97%|█████████▋| 484/500 [00:16<00:00, 23.50it/s] 98%|█████████▊| 489/500 [00:16<00:00, 28.79it/s] 99%|█████████▊| 493/500 [00:16<00:00, 31.20it/s] 100%|██████████| 500/500 [00:16<00:00, 30.11it/s] 80%|████████ | 8/10 [02:06<00:30, 15.48s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 6/500 [00:00<00:11, 41.31it/s] 2%|▏ | 11/500 [00:02<01:59, 4.09it/s] 3%|▎ | 15/500 [00:02<01:18, 6.21it/s] 4%|▍ | 19/500 [00:02<00:54, 8.83it/s] 5%|▍ | 24/500 [00:02<00:36, 12.87it/s] 6%|▌ | 29/500 [00:02<00:27, 17.29it/s] 7%|▋ | 33/500 [00:02<00:23, 20.08it/s] 7%|▋ | 37/500 [00:02<00:20, 22.91it/s] 9%|▊ | 43/500 [00:03<00:15, 29.38it/s] 10%|▉ | 49/500 [00:03<00:12, 35.26it/s] 11%|█ | 54/500 [00:03<00:11, 37.91it/s] 12%|█▏ | 59/500 [00:03<00:10, 40.47it/s] 13%|█▎ | 64/500 [00:03<00:10, 42.00it/s] 14%|█▍ | 69/500 [00:03<00:10, 42.98it/s] 15%|█▍ | 74/500 [00:03<00:09, 43.20it/s] 16%|█▌ | 79/500 [00:03<00:11, 37.22it/s] 17%|█▋ | 84/500 [00:04<00:11, 36.46it/s] 18%|█▊ | 88/500 [00:04<00:11, 34.78it/s] 18%|█▊ | 92/500 [00:04<00:11, 36.00it/s] 19%|█▉ | 97/500 [00:04<00:10, 38.92it/s] 21%|██ | 103/500 [00:04<00:09, 43.51it/s] 22%|██▏ | 109/500 [00:04<00:08, 46.75it/s] 23%|██▎ | 114/500 [00:04<00:08, 45.27it/s] 24%|██▍ | 119/500 [00:04<00:09, 41.48it/s] 25%|██▍ | 124/500 [00:05<00:09, 39.44it/s] 26%|██▌ | 130/500 [00:05<00:08, 44.24it/s] 27%|██▋ | 136/500 [00:05<00:07, 47.56it/s] 28%|██▊ | 142/500 [00:05<00:07, 49.55it/s] 30%|██▉ | 148/500 [00:05<00:07, 48.90it/s] 31%|███ | 153/500 [00:05<00:07, 48.61it/s] 32%|███▏ | 158/500 [00:05<00:07, 43.94it/s] 33%|███▎ | 163/500 [00:05<00:07, 43.98it/s] 34%|███▎ | 168/500 [00:05<00:07, 42.99it/s] 35%|███▍ | 173/500 [00:06<00:08, 40.51it/s] 36%|███▌ | 178/500 [00:06<00:07, 42.53it/s] 37%|███▋ | 184/500 [00:06<00:06, 46.35it/s] 38%|███▊ | 189/500 [00:06<00:06, 46.33it/s] 39%|███▉ | 194/500 [00:06<00:06, 45.10it/s] 40%|███▉ | 199/500 [00:06<00:06, 45.36it/s] 41%|████ | 204/500 [00:06<00:06, 45.84it/s] 42%|████▏ | 209/500 [00:06<00:06, 43.60it/s] 43%|████▎ | 214/500 [00:06<00:06, 41.53it/s] 44%|████▍ | 219/500 [00:07<00:06, 40.88it/s] 45%|████▌ | 225/500 [00:07<00:06, 44.64it/s] 46%|████▌ | 230/500 [00:07<00:06, 41.76it/s] 47%|████▋ | 235/500 [00:07<00:06, 42.88it/s] 48%|████▊ | 241/500 [00:07<00:05, 46.16it/s] 49%|████▉ | 246/500 [00:07<00:05, 45.93it/s] 50%|█████ | 251/500 [00:07<00:05, 42.85it/s] 51%|█████ | 256/500 [00:07<00:05, 41.44it/s] 52%|█████▏ | 261/500 [00:08<00:05, 43.39it/s] 53%|█████▎ | 266/500 [00:08<00:05, 41.79it/s] 54%|█████▍ | 272/500 [00:08<00:04, 46.01it/s] 56%|█████▌ | 278/500 [00:08<00:04, 48.25it/s] 57%|█████▋ | 284/500 [00:08<00:04, 50.39it/s] 58%|█████▊ | 290/500 [00:08<00:04, 51.85it/s] 59%|█████▉ | 296/500 [00:08<00:04, 49.69it/s] 60%|██████ | 302/500 [00:09<00:05, 37.87it/s] 61%|██████▏ | 307/500 [00:09<00:04, 38.71it/s] 62%|██████▏ | 312/500 [00:09<00:04, 39.80it/s] 63%|██████▎ | 317/500 [00:09<00:04, 41.31it/s] 64%|██████▍ | 322/500 [00:09<00:04, 42.69it/s] 65%|██████▌ | 327/500 [00:09<00:04, 40.76it/s] 66%|██████▋ | 332/500 [00:09<00:04, 38.75it/s] 67%|██████▋ | 336/500 [00:09<00:04, 37.01it/s] 68%|██████▊ | 340/500 [00:09<00:04, 35.40it/s] 69%|██████▉ | 344/500 [00:10<00:04, 34.17it/s] 70%|██████▉ | 348/500 [00:10<00:04, 32.96it/s] 70%|███████ | 352/500 [00:10<00:04, 33.03it/s] 71%|███████ | 356/500 [00:10<00:04, 31.19it/s] 72%|███████▏ | 360/500 [00:10<00:04, 31.73it/s] 73%|███████▎ | 364/500 [00:10<00:04, 31.69it/s] 74%|███████▎ | 368/500 [00:10<00:04, 30.79it/s] 74%|███████▍ | 372/500 [00:11<00:04, 28.58it/s] 75%|███████▌ | 375/500 [00:11<00:04, 28.71it/s] 76%|███████▌ | 379/500 [00:11<00:04, 29.89it/s] 77%|███████▋ | 383/500 [00:11<00:03, 29.69it/s] 77%|███████▋ | 387/500 [00:11<00:03, 30.57it/s] 78%|███████▊ | 391/500 [00:11<00:03, 31.37it/s] 79%|███████▉ | 395/500 [00:11<00:03, 31.09it/s] 80%|███████▉ | 399/500 [00:11<00:03, 28.30it/s] 80%|████████ | 402/500 [00:12<00:03, 27.21it/s] 81%|████████ | 405/500 [00:12<00:03, 26.78it/s] 82%|████████▏ | 408/500 [00:12<00:03, 27.21it/s] 82%|████████▏ | 411/500 [00:12<00:03, 27.51it/s] 83%|████████▎ | 415/500 [00:12<00:02, 28.81it/s] 84%|████████▎ | 418/500 [00:12<00:02, 28.19it/s] 84%|████████▍ | 421/500 [00:12<00:02, 26.89it/s] 85%|████████▍ | 424/500 [00:12<00:02, 26.34it/s] 85%|████████▌ | 427/500 [00:13<00:02, 25.35it/s] 86%|████████▌ | 430/500 [00:13<00:02, 24.81it/s] 87%|████████▋ | 433/500 [00:13<00:02, 24.55it/s] 87%|████████▋ | 437/500 [00:13<00:02, 27.35it/s] 88%|████████▊ | 442/500 [00:13<00:01, 30.63it/s] 89%|████████▉ | 446/500 [00:13<00:01, 32.65it/s] 90%|█████████ | 450/500 [00:13<00:01, 30.55it/s] 91%|█████████ | 454/500 [00:13<00:01, 28.84it/s] 92%|█████████▏| 458/500 [00:14<00:01, 29.70it/s] 92%|█████████▏| 462/500 [00:14<00:01, 28.39it/s] 93%|█████████▎| 465/500 [00:14<00:01, 27.80it/s] 94%|█████████▎| 468/500 [00:14<00:01, 27.83it/s] 94%|█████████▍| 471/500 [00:14<00:01, 25.98it/s] 95%|█████████▍| 474/500 [00:14<00:00, 26.90it/s] 96%|█████████▌| 478/500 [00:14<00:00, 28.37it/s] 96%|█████████▌| 481/500 [00:14<00:00, 27.07it/s] 97%|█████████▋| 484/500 [00:15<00:00, 26.60it/s] 98%|█████████▊| 488/500 [00:15<00:00, 29.53it/s] 98%|█████████▊| 492/500 [00:15<00:00, 31.41it/s] 99%|█████████▉| 496/500 [00:15<00:00, 32.93it/s] 100%|██████████| 500/500 [00:15<00:00, 32.19it/s] 90%|█████████ | 9/10 [02:21<00:15, 15.51s/it] 0%| | 0/500 [00:00<?, ?it/s] 1%| | 4/500 [00:00<00:14, 34.53it/s] 2%|▏ | 8/500 [00:02<03:29, 2.35it/s] 2%|▏ | 12/500 [00:03<01:58, 4.10it/s] 3%|▎ | 16/500 [00:03<01:16, 6.35it/s] 4%|▍ | 20/500 [00:03<00:52, 9.07it/s] 5%|▍ | 24/500 [00:03<00:39, 12.04it/s] 6%|▌ | 29/500 [00:03<00:28, 16.70it/s] 7%|▋ | 34/500 [00:03<00:21, 21.66it/s] 8%|▊ | 40/500 [00:03<00:16, 27.83it/s] 9%|▉ | 45/500 [00:03<00:15, 29.97it/s] 10%|█ | 50/500 [00:03<00:13, 32.29it/s] 11%|█ | 55/500 [00:04<00:12, 35.26it/s] 12%|█▏ | 60/500 [00:04<00:12, 34.58it/s] 13%|█▎ | 64/500 [00:04<00:12, 34.55it/s] 14%|█▎ | 68/500 [00:04<00:12, 34.13it/s] 14%|█▍ | 72/500 [00:04<00:12, 35.10it/s] 16%|█▌ | 78/500 [00:04<00:10, 40.67it/s] 17%|█▋ | 84/500 [00:04<00:09, 44.53it/s] 18%|█▊ | 90/500 [00:04<00:08, 47.85it/s] 19%|█▉ | 96/500 [00:05<00:08, 49.35it/s] 20%|██ | 102/500 [00:05<00:07, 52.02it/s] 22%|██▏ | 108/500 [00:05<00:07, 52.18it/s] 23%|██▎ | 114/500 [00:05<00:08, 46.32it/s] 24%|██▍ | 119/500 [00:05<00:08, 45.19it/s] 25%|██▍ | 124/500 [00:05<00:08, 45.17it/s] 26%|██▌ | 129/500 [00:05<00:08, 46.06it/s] 27%|██▋ | 135/500 [00:05<00:07, 49.35it/s] 28%|██▊ | 141/500 [00:05<00:07, 48.99it/s] 29%|██▉ | 146/500 [00:06<00:07, 48.84it/s] 30%|███ | 151/500 [00:06<00:07, 48.72it/s] 31%|███ | 156/500 [00:06<00:07, 43.72it/s] 32%|███▏ | 161/500 [00:06<00:08, 41.80it/s] 33%|███▎ | 166/500 [00:06<00:08, 40.26it/s] 34%|███▍ | 172/500 [00:06<00:07, 43.87it/s] 35%|███▌ | 177/500 [00:06<00:07, 44.68it/s] 36%|███▋ | 182/500 [00:06<00:06, 45.88it/s] 37%|███▋ | 187/500 [00:07<00:07, 44.43it/s] 39%|███▊ | 193/500 [00:07<00:06, 47.22it/s] 40%|███▉ | 198/500 [00:07<00:07, 42.30it/s] 41%|████ | 203/500 [00:07<00:07, 41.96it/s] 42%|████▏ | 208/500 [00:07<00:06, 43.98it/s] 43%|████▎ | 213/500 [00:07<00:06, 42.06it/s] 44%|████▎ | 218/500 [00:07<00:06, 43.98it/s] 45%|████▍ | 223/500 [00:07<00:06, 44.40it/s] 46%|████▌ | 228/500 [00:07<00:06, 44.54it/s] 47%|████▋ | 234/500 [00:08<00:05, 48.14it/s] 48%|████▊ | 240/500 [00:08<00:05, 51.19it/s] 49%|████▉ | 246/500 [00:08<00:06, 41.65it/s] 50%|█████ | 251/500 [00:08<00:05, 43.14it/s] 51%|█████ | 256/500 [00:08<00:05, 43.90it/s] 52%|█████▏ | 261/500 [00:08<00:05, 42.34it/s] 53%|█████▎ | 266/500 [00:08<00:05, 43.50it/s] 54%|█████▍ | 271/500 [00:08<00:05, 43.61it/s] 55%|█████▌ | 276/500 [00:09<00:04, 45.26it/s] 56%|█████▋ | 282/500 [00:09<00:04, 47.86it/s] 57%|█████▋ | 287/500 [00:09<00:04, 46.01it/s] 58%|█████▊ | 292/500 [00:09<00:04, 45.75it/s] 59%|█████▉ | 297/500 [00:09<00:04, 45.75it/s] 61%|██████ | 303/500 [00:09<00:04, 48.15it/s] 62%|██████▏ | 308/500 [00:09<00:04, 44.72it/s] 63%|██████▎ | 313/500 [00:09<00:04, 46.06it/s] 64%|██████▍ | 319/500 [00:09<00:03, 48.25it/s] 65%|██████▍ | 324/500 [00:10<00:03, 45.77it/s] 66%|██████▌ | 330/500 [00:10<00:03, 48.09it/s] 67%|██████▋ | 335/500 [00:10<00:03, 45.34it/s] 68%|██████▊ | 340/500 [00:10<00:03, 43.89it/s] 69%|██████▉ | 345/500 [00:10<00:03, 45.40it/s] 70%|███████ | 350/500 [00:10<00:03, 45.92it/s] 71%|███████ | 355/500 [00:10<00:03, 42.56it/s] 72%|███████▏ | 360/500 [00:10<00:03, 43.26it/s] 73%|███████▎ | 365/500 [00:10<00:03, 44.98it/s] 74%|███████▍ | 371/500 [00:11<00:02, 48.12it/s] 75%|███████▌ | 376/500 [00:11<00:02, 47.45it/s] 76%|███████▌ | 381/500 [00:11<00:02, 46.16it/s] 77%|███████▋ | 386/500 [00:11<00:02, 43.62it/s] 78%|███████▊ | 391/500 [00:11<00:02, 40.99it/s] 79%|███████▉ | 396/500 [00:11<00:02, 41.59it/s] 80%|████████ | 401/500 [00:11<00:02, 41.64it/s] 81%|████████▏ | 407/500 [00:11<00:02, 44.74it/s] 82%|████████▏ | 412/500 [00:12<00:02, 38.87it/s] 83%|████████▎ | 417/500 [00:12<00:02, 32.20it/s] 84%|████████▍ | 421/500 [00:12<00:02, 29.76it/s] 85%|████████▌ | 425/500 [00:12<00:02, 27.98it/s] 86%|████████▌ | 428/500 [00:12<00:02, 28.02it/s] 86%|████████▌ | 431/500 [00:12<00:02, 28.35it/s] 87%|████████▋ | 435/500 [00:12<00:02, 30.41it/s] 88%|████████▊ | 439/500 [00:13<00:01, 31.52it/s] 89%|████████▊ | 443/500 [00:13<00:01, 31.50it/s] 89%|████████▉ | 447/500 [00:13<00:01, 29.15it/s] 90%|█████████ | 450/500 [00:13<00:01, 26.80it/s] 91%|█████████ | 453/500 [00:13<00:01, 25.96it/s] 91%|█████████ | 456/500 [00:13<00:01, 25.53it/s] 92%|█████████▏| 459/500 [00:13<00:01, 26.08it/s] 92%|█████████▏| 462/500 [00:14<00:01, 23.91it/s] 93%|█████████▎| 465/500 [00:14<00:01, 24.32it/s] 94%|█████████▎| 468/500 [00:14<00:01, 23.94it/s] 94%|█████████▍| 471/500 [00:14<00:01, 23.71it/s] 95%|█████████▍| 474/500 [00:14<00:01, 24.75it/s] 96%|█████████▌| 478/500 [00:14<00:00, 27.01it/s] 96%|█████████▌| 481/500 [00:14<00:00, 25.42it/s] 97%|█████████▋| 484/500 [00:14<00:00, 25.85it/s] 97%|█████████▋| 487/500 [00:14<00:00, 25.90it/s] 98%|█████████▊| 490/500 [00:15<00:00, 26.67it/s] 99%|█████████▊| 493/500 [00:15<00:00, 27.18it/s] 99%|█████████▉| 496/500 [00:15<00:00, 26.30it/s] 100%|██████████| 500/500 [00:15<00:00, 32.25it/s] 100%|██████████| 10/10 [02:37<00:00, 15.73s/it]
View first 5 rows of the data
In [24]:
# View the head of the DataFrame
dataset.head()
Out[24]:
| features | class | |
|---|---|---|
| 0 | [-618.43463, 92.98388, 22.846985, 21.094965, 2... | 0 |
| 1 | [-600.72955, 100.824356, 3.30687, 20.44151, 27... | 0 |
| 2 | [-626.0543, 94.19148, 25.224388, 33.466988, 28... | 0 |
| 3 | [-589.7548, 86.972855, 20.047338, 26.504921, 1... | 0 |
| 4 | [-591.3263, 110.81089, 2.8627186, 20.751932, 2... | 0 |
In [25]:
# Storing the class as int
dataset['class'] = [int(x) for x in dataset['class']]
In [26]:
# Check the frequency of classes in the dataset
dataset['class'].value_counts()
Out[26]:
| count | |
|---|---|
| class | |
| 0 | 500 |
| 1 | 500 |
| 2 | 500 |
| 3 | 500 |
| 4 | 500 |
| 5 | 500 |
| 6 | 500 |
| 8 | 500 |
| 9 | 500 |
| 7 | 500 |
Visualizing the Mel Frequency Cepstral Coefficients Using a Spectrogram¶
draw_spectrograms: From the Mel Coefficients we are extracting for a particular audio, this function is creating the 2-D graph of those coefficients with the X-axis representing time and the Y-axis shows the corresponding Mel coefficients in that point of time.
In [27]:
# A function which returns MFCC
def draw_spectrograms(audio_data, sample_rate):
# Extract features
extracted_features = librosa.feature.mfcc(y = audio_data,
sr = sample_rate,
n_mfcc = 40)
# Return features without scaling
return extracted_features
The very first MFCC coefficient (0th coefficient) does not provide information about the overall shape of the spectrum. It simply communicates a constant offset or the addition of a constant value to the full spectrum. As a result, when performing classification, many practitioners will disregard the initial MFCC. In the images, you can see those represented by blue pixels.
We can plot the MFCCs, but it's difficult to tell what kind of signal is hiding behind such representation.
In [28]:
# Creating subplots
fig, ax = plt.subplots(5, 2, figsize = (15, 30))
# Initializing row and column variables for subplots
row = 0
column = 0
for digit in range(10):
# Get the audio of different classes (0-9)
audio_data, sample_rate = get_audio_raw(digit)
# Extract their MFCC
mfcc = draw_spectrograms(audio_data, sample_rate)
print(f"Shape of MFCC of audio digit {digit} ---> ", mfcc.shape)
# Display the plots and its title
ax[row,column].set_title(f"MFCC of audio class {digit} across time")
librosa.display.specshow(mfcc, sr = 22050, ax = ax[row, column])
# Set X-labels and Y-labels
ax[row,column].set_xlabel("Time")
ax[row,column].set_ylabel("MFCC Coefficients")
# Conditions for positioning of the plots
if column == 1:
column = 0
row += 1
else:
column+=1
plt.tight_layout(pad = 3)
plt.show()
Shape of MFCC of audio digit 0 ---> (40, 21) Shape of MFCC of audio digit 1 ---> (40, 24) Shape of MFCC of audio digit 2 ---> (40, 21) Shape of MFCC of audio digit 3 ---> (40, 23) Shape of MFCC of audio digit 4 ---> (40, 29) Shape of MFCC of audio digit 5 ---> (40, 22) Shape of MFCC of audio digit 6 ---> (40, 26) Shape of MFCC of audio digit 7 ---> (40, 28) Shape of MFCC of audio digit 8 ---> (40, 22) Shape of MFCC of audio digit 9 ---> (40, 24)
Visual Inspection of MFCC Spectrograms:
On inspecting them visually, we can see that there are a lot of deviations from the spectrograms of one audio to another. There are a lot of tiny rectangles and bars whose positions are unique to each audio. So, the Artificial Neural Network should be able to perform decently in identifying these audios.
Perform Train-Test Split¶
- Split the data into train and test sets
In [29]:
# Import train_test_split function
from sklearn.model_selection import train_test_split
X = np.array(dataset['features'].to_list())
Y = np.array(dataset['class'].to_list())
# Create train set and test set
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, train_size = 0.75, shuffle = True, random_state = 8)
In [30]:
# Checking the shape of the data
X_train.shape
Out[30]:
(3750, 40)
Modelling¶
Create an artificial neural network to recognize the digit.
About the libraries:
Keras: Keras is an open-source deep-learning library in Python. Keras is popular because the API is clean and simple, allowing standard deep learning models to be defined, fit, and evaluated in just a few lines of code.Sklearn: Sklearn has simple and efficient tools for predictive data analysis. It's accessible to everybody, and reusable in various contexts.
Import necessary libraries for building the model¶
In [31]:
# To create an ANN model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
# To create a checkpoint and save the best model
from tensorflow.keras.callbacks import ModelCheckpoint
# To load the model
from tensorflow.keras.models import load_model
# To evaluate the model
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.preprocessing import LabelBinarizer
Model Creation¶
Why are we using ANN's?¶
When we are converting audios to their corresponding spectrograms, we will have similar spectrograms for similar audios irrespective of who the speaker is, and what is their pitch and timber like. So, local spatiality is never going to be a problem. Having convolutional layers on top of our fully connected layers is just adding to our computational redundancy.
We will use a Sequential model with multiple connected hidden layers, and an output layer that returns a single, continuous value:
- A Sequential model is a linear stack of layers. Sequential models can be created by giving a list of layer instances.
- A dense layer of neurons is a simple layer of neurons in which each neuron receives input from all of the neurons in the previous layer.
- The most popular function employed for hidden layers is the rectified linear activation function, or ReLU activation function. It's popular because it's easy to use and effective in getting around the limitations of other popular activation functions like Sigmoid and Tanh.
In [32]:
# Crete a Sequential Object
model = Sequential()
# Add first layer with 100 neurons to the sequental object
model.add(Dense(100, input_shape = (40, ), activation = 'relu'))
# Add second layer with 100 neurons to the sequental object
model.add(Dense(100, activation = 'relu'))
# Add third layer with 100 neurons to the sequental object
model.add(Dense(100, activation = 'relu'))
# Output layer with 10 neurons as it has 10 classes
model.add(Dense(10, activation = 'softmax'))
/usr/local/lib/python3.10/dist-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [33]:
# Print Summary of the model
model.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩ │ dense (Dense) │ (None, 100) │ 4,100 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_1 (Dense) │ (None, 100) │ 10,100 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_2 (Dense) │ (None, 100) │ 10,100 │ ├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤ │ dense_3 (Dense) │ (None, 10) │ 1,010 │ └──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
Total params: 25,310 (98.87 KB)
Trainable params: 25,310 (98.87 KB)
Non-trainable params: 0 (0.00 B)
In [35]:
# Compile the model using accuracy metric and adam optimizer
model.compile(loss = 'sparse_categorical_crossentropy',
metrics = ['accuracy'],
optimizer = 'adam')
Model Checkpoint & Training¶
In [36]:
# Set the number of epochs for training
num_epochs = 100
# Set the batch size for training
batch_size = 32
# Fit the model
model.fit(X_train, Y_train, validation_data = (X_test, Y_test), epochs = num_epochs, batch_size = batch_size, verbose = 1)
Epoch 1/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 2s 7ms/step - accuracy: 0.3061 - loss: 10.4127 - val_accuracy: 0.5856 - val_loss: 1.1168 Epoch 2/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.7413 - loss: 0.7128 - val_accuracy: 0.7592 - val_loss: 0.6596 Epoch 3/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8666 - loss: 0.3825 - val_accuracy: 0.8584 - val_loss: 0.3549 Epoch 4/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8858 - loss: 0.3003 - val_accuracy: 0.9112 - val_loss: 0.2214 Epoch 5/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8904 - loss: 0.3275 - val_accuracy: 0.9248 - val_loss: 0.2158 Epoch 6/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9449 - loss: 0.1450 - val_accuracy: 0.9664 - val_loss: 0.1092 Epoch 7/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9412 - loss: 0.1448 - val_accuracy: 0.9640 - val_loss: 0.1042 Epoch 8/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9156 - loss: 0.2653 - val_accuracy: 0.9752 - val_loss: 0.0767 Epoch 9/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9698 - loss: 0.0852 - val_accuracy: 0.9496 - val_loss: 0.1415 Epoch 10/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9639 - loss: 0.1028 - val_accuracy: 0.9696 - val_loss: 0.0992 Epoch 11/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9679 - loss: 0.0854 - val_accuracy: 0.9800 - val_loss: 0.0732 Epoch 12/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9677 - loss: 0.0888 - val_accuracy: 0.9128 - val_loss: 0.2249 Epoch 13/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9548 - loss: 0.1295 - val_accuracy: 0.8896 - val_loss: 0.3591 Epoch 14/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9672 - loss: 0.0957 - val_accuracy: 0.9480 - val_loss: 0.1781 Epoch 15/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9544 - loss: 0.1346 - val_accuracy: 0.9744 - val_loss: 0.0712 Epoch 16/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9736 - loss: 0.0705 - val_accuracy: 0.9736 - val_loss: 0.0631 Epoch 17/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9878 - loss: 0.0326 - val_accuracy: 0.9544 - val_loss: 0.1310 Epoch 18/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9867 - loss: 0.0409 - val_accuracy: 0.9784 - val_loss: 0.0760 Epoch 19/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9573 - loss: 0.1130 - val_accuracy: 0.9744 - val_loss: 0.0784 Epoch 20/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9798 - loss: 0.0538 - val_accuracy: 0.9384 - val_loss: 0.2808 Epoch 21/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9668 - loss: 0.1089 - val_accuracy: 0.9720 - val_loss: 0.0924 Epoch 22/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9808 - loss: 0.0584 - val_accuracy: 0.9528 - val_loss: 0.1539 Epoch 23/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9552 - loss: 0.1253 - val_accuracy: 0.9832 - val_loss: 0.0497 Epoch 24/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9845 - loss: 0.0375 - val_accuracy: 0.9776 - val_loss: 0.0704 Epoch 25/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9784 - loss: 0.0699 - val_accuracy: 0.9720 - val_loss: 0.0941 Epoch 26/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9797 - loss: 0.0578 - val_accuracy: 0.9752 - val_loss: 0.0714 Epoch 27/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9737 - loss: 0.0620 - val_accuracy: 0.9664 - val_loss: 0.0936 Epoch 28/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9797 - loss: 0.0501 - val_accuracy: 0.9656 - val_loss: 0.0966 Epoch 29/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9729 - loss: 0.0758 - val_accuracy: 0.9824 - val_loss: 0.0634 Epoch 30/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9871 - loss: 0.0351 - val_accuracy: 0.9808 - val_loss: 0.0689 Epoch 31/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9851 - loss: 0.0322 - val_accuracy: 0.9632 - val_loss: 0.1220 Epoch 32/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9759 - loss: 0.0691 - val_accuracy: 0.9832 - val_loss: 0.0587 Epoch 33/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9929 - loss: 0.0184 - val_accuracy: 0.9840 - val_loss: 0.0497 Epoch 34/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9868 - loss: 0.0467 - val_accuracy: 0.9888 - val_loss: 0.0399 Epoch 35/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9929 - loss: 0.0204 - val_accuracy: 0.9856 - val_loss: 0.0422 Epoch 36/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9974 - loss: 0.0132 - val_accuracy: 0.9880 - val_loss: 0.0408 Epoch 37/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9877 - loss: 0.0337 - val_accuracy: 0.9672 - val_loss: 0.1211 Epoch 38/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9771 - loss: 0.0663 - val_accuracy: 0.9848 - val_loss: 0.0518 Epoch 39/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9912 - loss: 0.0230 - val_accuracy: 0.9848 - val_loss: 0.0608 Epoch 40/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.9942 - loss: 0.0182 - val_accuracy: 0.9744 - val_loss: 0.0879 Epoch 41/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9835 - loss: 0.0450 - val_accuracy: 0.9744 - val_loss: 0.1010 Epoch 42/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9827 - loss: 0.0426 - val_accuracy: 0.9784 - val_loss: 0.0835 Epoch 43/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9894 - loss: 0.0304 - val_accuracy: 0.9768 - val_loss: 0.0749 Epoch 44/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9786 - loss: 0.0619 - val_accuracy: 0.9656 - val_loss: 0.1142 Epoch 45/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9897 - loss: 0.0273 - val_accuracy: 0.9832 - val_loss: 0.0644 Epoch 46/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9933 - loss: 0.0199 - val_accuracy: 0.9840 - val_loss: 0.0548 Epoch 47/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9771 - loss: 0.0645 - val_accuracy: 0.9856 - val_loss: 0.0404 Epoch 48/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9978 - loss: 0.0073 - val_accuracy: 0.9896 - val_loss: 0.0330 Epoch 49/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9973 - loss: 0.0074 - val_accuracy: 0.9872 - val_loss: 0.0551 Epoch 50/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9899 - loss: 0.0308 - val_accuracy: 0.9824 - val_loss: 0.0800 Epoch 51/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9962 - loss: 0.0158 - val_accuracy: 0.9824 - val_loss: 0.0698 Epoch 52/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9951 - loss: 0.0174 - val_accuracy: 0.9856 - val_loss: 0.0505 Epoch 53/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9899 - loss: 0.0318 - val_accuracy: 0.9880 - val_loss: 0.0433 Epoch 54/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9953 - loss: 0.0087 - val_accuracy: 0.9856 - val_loss: 0.0470 Epoch 55/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9949 - loss: 0.0157 - val_accuracy: 0.9800 - val_loss: 0.0670 Epoch 56/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9511 - loss: 0.1953 - val_accuracy: 0.9512 - val_loss: 0.1829 Epoch 57/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9784 - loss: 0.0722 - val_accuracy: 0.9832 - val_loss: 0.0520 Epoch 58/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9938 - loss: 0.0198 - val_accuracy: 0.9856 - val_loss: 0.0660 Epoch 59/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9839 - loss: 0.0418 - val_accuracy: 0.9848 - val_loss: 0.0607 Epoch 60/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9940 - loss: 0.0114 - val_accuracy: 0.9856 - val_loss: 0.0621 Epoch 61/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9991 - loss: 0.0040 - val_accuracy: 0.9872 - val_loss: 0.0436 Epoch 62/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9991 - loss: 0.0041 - val_accuracy: 0.9856 - val_loss: 0.0762 Epoch 63/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9969 - loss: 0.0110 - val_accuracy: 0.9880 - val_loss: 0.0456 Epoch 64/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.9991 - loss: 0.0023 - val_accuracy: 0.9896 - val_loss: 0.0402 Epoch 65/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 8.8939e-04 - val_accuracy: 0.9904 - val_loss: 0.0356 Epoch 66/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9951 - loss: 0.0188 - val_accuracy: 0.9880 - val_loss: 0.0633 Epoch 67/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9981 - loss: 0.0050 - val_accuracy: 0.9904 - val_loss: 0.0393 Epoch 68/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9983 - loss: 0.0030 - val_accuracy: 0.9888 - val_loss: 0.0362 Epoch 69/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 3.7418e-04 - val_accuracy: 0.9856 - val_loss: 0.0546 Epoch 70/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 8.7515e-04 - val_accuracy: 0.9912 - val_loss: 0.0354 Epoch 71/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 6.6085e-04 - val_accuracy: 0.9864 - val_loss: 0.0452 Epoch 72/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 0.0014 - val_accuracy: 0.9896 - val_loss: 0.0380 Epoch 73/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 1.0000 - loss: 2.3551e-04 - val_accuracy: 0.9904 - val_loss: 0.0383 Epoch 74/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.9997 - loss: 8.9116e-04 - val_accuracy: 0.9720 - val_loss: 0.1012 Epoch 75/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9795 - loss: 0.0881 - val_accuracy: 0.9488 - val_loss: 0.1836 Epoch 76/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.9848 - loss: 0.0530 - val_accuracy: 0.9840 - val_loss: 0.0570 Epoch 77/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.9803 - loss: 0.0678 - val_accuracy: 0.9880 - val_loss: 0.0413 Epoch 78/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9959 - loss: 0.0142 - val_accuracy: 0.9904 - val_loss: 0.0405 Epoch 79/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 0.0017 - val_accuracy: 0.9888 - val_loss: 0.0542 Epoch 80/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.9999 - loss: 0.0011 - val_accuracy: 0.9912 - val_loss: 0.0347 Epoch 81/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 0.0010 - val_accuracy: 0.9912 - val_loss: 0.0383 Epoch 82/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 3.4966e-04 - val_accuracy: 0.9912 - val_loss: 0.0392 Epoch 83/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 3.5039e-04 - val_accuracy: 0.9904 - val_loss: 0.0390 Epoch 84/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 2.0895e-04 - val_accuracy: 0.9928 - val_loss: 0.0376 Epoch 85/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 4.1444e-04 - val_accuracy: 0.9904 - val_loss: 0.0377 Epoch 86/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 1.9898e-04 - val_accuracy: 0.9904 - val_loss: 0.0380 Epoch 87/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 1.7281e-04 - val_accuracy: 0.9904 - val_loss: 0.0382 Epoch 88/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 1.5216e-04 - val_accuracy: 0.9912 - val_loss: 0.0389 Epoch 89/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 2.2941e-04 - val_accuracy: 0.9904 - val_loss: 0.0380 Epoch 90/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 1.1161e-04 - val_accuracy: 0.9912 - val_loss: 0.0392 Epoch 91/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 1.4453e-04 - val_accuracy: 0.9912 - val_loss: 0.0399 Epoch 92/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 2.9565e-04 - val_accuracy: 0.9920 - val_loss: 0.0395 Epoch 93/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 2.1551e-04 - val_accuracy: 0.9904 - val_loss: 0.0389 Epoch 94/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 1.1799e-04 - val_accuracy: 0.9904 - val_loss: 0.0389 Epoch 95/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 1.0000 - loss: 7.9486e-05 - val_accuracy: 0.9912 - val_loss: 0.0397 Epoch 96/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 1.0000 - loss: 1.5066e-04 - val_accuracy: 0.9912 - val_loss: 0.0404 Epoch 97/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 1.0000 - loss: 1.1335e-04 - val_accuracy: 0.9904 - val_loss: 0.0380 Epoch 98/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 1.0000 - loss: 6.4013e-05 - val_accuracy: 0.9912 - val_loss: 0.0382 Epoch 99/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 1.0000 - loss: 7.2552e-05 - val_accuracy: 0.9904 - val_loss: 0.0397 Epoch 100/100 118/118 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 1.0000 - loss: 8.3802e-05 - val_accuracy: 0.9912 - val_loss: 0.0399
Out[36]:
<keras.src.callbacks.history.History at 0x7eb23de22350>
Model Evaluation¶
In [38]:
# Make predictions on the test set
Y_pred = model.predict(X_test)
Y_pred = [np.argmax(i) for i in Y_pred]
40/40 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
In [43]:
# Set style as dark
sns.set_style("dark")
# Set figure size
plt.figure(figsize = (11, 9))
# Plot the title
plt.title("CONFUSION MATRIX FOR MNIST AUDIO PREDICTION")
# Confusion matrix
cm = confusion_matrix([int(x) for x in Y_test], Y_pred)
# Plot the confusion matrix as heatmap
sns.heatmap(cm, annot=True, cmap="Pastel1", fmt='g', cbar=False)
# Set X-label and Y-label
plt.xlabel("ACTUAL VALUES")
plt.ylabel("PREDICTED VALUES")
# Show the plot
plt.show()
# Print the metrics
print(classification_report(Y_test, Y_pred))
precision recall f1-score support
0 0.99 0.96 0.98 114
1 0.98 0.99 0.99 122
2 1.00 0.98 0.99 130
3 0.97 0.99 0.98 127
4 1.00 1.00 1.00 120
5 0.99 0.99 0.99 135
6 1.00 1.00 1.00 123
7 0.99 1.00 0.99 135
8 1.00 1.00 1.00 127
9 0.99 0.99 0.99 117
accuracy 0.99 1250
macro avg 0.99 0.99 0.99 1250
weighted avg 0.99 0.99 0.99 1250
Observations:
- From the confusion matrix, we can observe that most of the observations are correctly identified by the model.
- In very few cases, the model is not able to identify the correct digit. For example, 9 observations are 0 but the model has predicted them as 2.
- The model has given a great performance with 99% recall, precision and F1-score.
Conclusion
- Digit Recognition, being an extremely small subset of the general category of Speech Recognition problems, is simple enough that our Artificial Neural Network is able to achieve almost 100% accuracy.
- Speech Recognition is a very active area of research, and before the advent of Deep Learning, statistical machine learning methods did not do well on the relatively straight forward problem statement of converting audio recordings of human speech into text.
- In the industry, Artificial Neural Networks being the simplest type of Deep Learning architecture find very limited use and are mostly utilized only as a modular component of more advanced Deep Learning architectures.
- However ANNs show their worth on tabular data sets which are highly numerical and high in volume - beyond a certain threshold in terms of number of instances, ANNs are able to learn fine patterns from the data in a way that Ensemble techniques or Support Vector Methods struggle to do so, and this is largely due to the hierarchical arrangement of the neurons in the Neural Network.
- A digit speech recognition model such as the above may find use in certain applications for voice activated security applications with digit codes, automated telephone operators and the like.