# Loading Data¶

In [1]:

    import pandas as pd
    import numpy as np

    df = pd.DataFrame(pd.read_csv('doc_reports-1.csv'))
    df

Out[1]:

| id | user_id | result | visual_authenticity_result | image_integrity_result | face_detection_result | image_quality_result | created_at | supported_document_result | conclusive_document_quality_result | colour_picture_result | data_validation_result | data_consistency_result | data_comparison_result | attempt_id | police_record_result | compromised_document_result | properties | sub_result
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 0.0 | ab23fae164e34af0a1ad1423ce9fd9f0 | consider | consider | clear | clear | clear | 2017-06-20T23:12:57Z | clear | NaN | NaN | clear | clear | NaN | 050a0596de424fab83c433eaa18b3f8d | clear | NaN | {'gender': 'Male', 'nationality': 'IRL', 'docu... | caution
1 | 1.0 | 15a84e8951254011b47412fa4e8f65b8 | clear | clear | clear | clear | clear | 2017-06-20T23:16:04Z | clear | NaN | NaN | clear | NaN | NaN | f69c1e5f45a64e50a26740b9bfb978b7 | clear | NaN | {'gender': 'Female', 'document_type': 'driving... | clear
2 | 2.0 | ffb82fda52b041e4b9af9cb4ef298c85 | clear | clear | clear | clear | clear | 2017-06-20T17:59:49Z | clear | NaN | NaN | clear | clear | NaN | f9f84f3055714d8e8f7419dc984d1769 | clear | NaN | {'gender': 'Male', 'nationality': 'ITA', 'docu... | clear
3 | 3.0 | bd4a8b3e3601427e88aa1d9eab9f4290 | clear | clear | clear | clear | clear | 2017-06-20T17:59:38Z | clear | NaN | NaN | clear | clear | NaN | 10a54a1ecf794404be959e030f11fef6 | clear | NaN | {'gender': 'Male', 'issuing_date': '2007-08', ... | clear
4 | 4.0 | f52ad1c7e69543a9940c3e7f8ed28a39 | clear | clear | clear | clear | clear | 2017-06-20T18:08:09Z | clear | NaN | NaN | clear | clear | NaN | 1f320d1d07de493292b7e0d5ebfb1cb9 | clear | NaN | {'gender': 'Male', 'nationality': 'POL', 'docu... | clear
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ...
16378 | 17511.0 | 7049eba7d145462282bd9f19a834e43a | clear | clear | clear | clear | clear | 2017-10-11T21:54:34Z | clear | clear | clear | clear | clear | NaN | 4cd2b6b70f354904852804f69d4e40c3 | clear | clear | {'gender': 'Male', 'nationality': 'GBR', 'docu... | clear
16379 | 17512.0 | 41c6441d28c646168e948c3b3764ddbf | clear | clear | clear | clear | clear | 2017-10-11T21:55:26Z | clear | clear | clear | clear | NaN | NaN | 709462b908fe4017a67ac049036a7553 | clear | clear | {'issuing_date': '2017-06-17', 'document_type'... | clear
16380 | 17513.0 | 77ecd7d832314de2b9eafd607ec80e0d | clear | clear | clear | clear | clear | 2017-10-11T21:55:12Z | clear | clear | clear | clear | clear | NaN | 999dcabb1cdb4deabcce50518555d72d | clear | clear | {'gender': 'Female', 'nationality': 'GBR', 'do... | clear
16381 | 17514.0 | d4af182a07e7428f973a722147baa0be | clear | clear | clear | clear | clear | 2017-10-11T21:56:45Z | clear | clear | clear | clear | clear | NaN | 956a86269cb548a7a681f1a2ab360ae9 | clear | clear | {'gender': 'Male', 'nationality': 'GBR', 'docu... | clear
16382 | 17515.0 | 4e11eeeaeac5415ebf73db9f64fcc05c | clear | clear | clear | clear | clear | 2017-10-11T21:56:19Z | clear | clear | clear | clear | clear | NaN | cd5b5d5cf3ab4701a947e43e534f86f3 | clear | clear | {'gender': 'Male', 'nationality': 'PRT', 'docu... | clear

16383 rows × 19 columns

# Feature Extraction¶

#### Extracting features from 'properties' column¶

In [2]:

    prop_list = list(pd.Series(df['properties']))

#### Extraxting data from dictionaries stored as a string in 'feature' column¶

In [3]:

    cols = ['gender', 'nationality', 'document_type', 'date_of_expiry', 'issuing_country']
    gen = []
    nat = []
    doc = []
    dat = []
    isc = []
    del_rows = []

    for i in range(len(prop_list)):

        if len(prop_list[i]) > 2:

            d = eval(prop_list[i])

            try:
                gen.append(d['gender'])
            except KeyError:
                gen.append(np.nan)
            try:
                nat.append(d['nationality'])
            except KeyError:
                nat.append(np.nan)
            try:
                doc.append(d['document_type'])
            except KeyError:
                doc.append(np.nan)
            try:
                dat.append(d['date_of_expiry'])
            except KeyError:
                dat.append(np.nan)
            try:
                isc.append(d['issuing_country'])
            except KeyError:
                isc.append(np.nan)

        else:
            del_rows.append(i)
            pass

    data = [gen, nat, doc, dat, isc]
    data = np.transpose(data)

#### Creating a new DataFrame for 'properties' data features so that we can merge it later with our old DataFrame and also dropping rows from our old DataFrame which we dropped in the new DataFrame.¶

In [4]:

    new_df = pd.DataFrame(data, columns=cols)
    df.drop(labels=del_rows, axis=0, inplace=True)

#### Adding 'id' column in our new DataFrame so that we can merge this it with our old DataFrame with reference to 'id' values¶

In [5]:

    new_df['id'] = df['id'].values

#### Re-arranging Columns of DataFrame so that 'id' will always be first column in our merged DataFrame and result collumn must be last¶

In [6]:

    new_df = new_df.reindex(columns=['id', 'gender', 'nationality', 'document_type', 'date_of_expiry', 'issuing_country'])

In [7]:

    cols = []
    for val in df.columns:
        if val != 'result':
            cols.append(val)
    cols.append('result')

    df = df.reindex(columns=cols)

#### Merging both DataFrame¶

In [8]:

    c_df = pd.merge(new_df, df, on='id')

# Data Cleaning¶

#### Dropping the fetures which will not help in data analysis like IDs and dates¶

In [9]:

    c_df.drop(['user_id', 'created_at', 'attempt_id', 'properties', 'date_of_expiry'], axis=1, inplace=True)

#### Checking for null values in the DataFrame¶

In [10]:

    c_df.isnull().sum()

Out[10]:

    id                                        0
    gender                                 3061
    nationality                            6904
    document_type                             1
    issuing_country                           0
    visual_authenticity_result                2
    image_integrity_result                    0
    face_detection_result                     3
    image_quality_result                      0
    supported_document_result                 0
    conclusive_document_quality_result      885
    colour_picture_result                   882
    data_validation_result                  698
    data_consistency_result                5246
    data_comparison_result                14389
    police_record_result                    517
    compromised_document_result            1672
    sub_result                                0
    result                                    0
    dtype: int64

#### Dropping 'data_comparison_result', 'data_consistency_result' and 'nationality as they all contains null values mostly and are not helping in deriving results¶

In [11]:

    c_df.drop(['data_comparison_result', 'data_consistency_result', 'nationality'], axis=1, inplace=True)

#### Dropping rows which contains null values in any of the remaining feature columns¶

In [12]:

    c_df.dropna(inplace=True)

#### Confirming that the DataFrame does not contain any null values in any feature column¶

In [13]:

    c_df.isnull().sum()

Out[13]:

    id                                    0
    gender                                0
    document_type                         0
    issuing_country                       0
    visual_authenticity_result            0
    image_integrity_result                0
    face_detection_result                 0
    image_quality_result                  0
    supported_document_result             0
    conclusive_document_quality_result    0
    colour_picture_result                 0
    data_validation_result                0
    police_record_result                  0
    compromised_document_result           0
    sub_result                            0
    result                                0
    dtype: int64

#### Resetting indexes and observing the clean data¶

In [14]:

    c_df.reset_index(inplace=True)
    c_df.drop(['index'], axis=1, inplace=True)
    c_df

Out[14]:

| id | gender | document_type | issuing_country | visual_authenticity_result | image_integrity_result | face_detection_result | image_quality_result | supported_document_result | conclusive_document_quality_result | colour_picture_result | data_validation_result | police_record_result | compromised_document_result | sub_result | result
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 718.0 | Male | passport | GBR | clear | consider | clear | clear | clear | consider | clear | consider | clear | clear | caution | consider
1 | 720.0 | Female | driving_licence | GBR | clear | consider | clear | clear | clear | consider | clear | clear | clear | clear | caution | consider
2 | 723.0 | Male | driving_licence | GBR | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear
3 | 725.0 | Female | passport | NZL | clear | consider | clear | clear | clear | consider | clear | clear | clear | clear | caution | consider
4 | 729.0 | Male | national_identity_card | LTU | clear | consider | clear | clear | clear | consider | clear | clear | clear | clear | caution | consider
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ...
9820 | 17510.0 | Male | passport | GBR | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear
9821 | 17511.0 | Male | passport | GBR | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear
9822 | 17513.0 | Female | passport | GBR | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear
9823 | 17514.0 | Male | passport | GBR | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear
9824 | 17515.0 | Male | national_identity_card | PRT | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear | clear

9825 rows × 16 columns

#### 60% Data preserved after data cleaning and feature extraction¶

# Data Encoding¶

In [15]:

    from sklearn import preprocessing

    ---------------------------------------------------------------------------
    ModuleNotFoundError                       Traceback (most recent call last)
    Input In [15], in <module>
    ----> 1 from sklearn import preprocessing

    ModuleNotFoundError: No module named 'sklearn'

In [16]:

    encoder = preprocessing.OrdinalEncoder( dtype=int)

    c_df.iloc[:, 1:] = encoder.fit_transform(c_df.iloc[:, 1:])
    c_df

Out[16]:

| id | gender | document_type | issuing_country | visual_authenticity_result | image_integrity_result | face_detection_result | image_quality_result | supported_document_result | conclusive_document_quality_result | colour_picture_result | data_validation_result | police_record_result | compromised_document_result | sub_result | result
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 718.0 | 1 | 2 | 35 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1
1 | 720.0 | 0 | 0 | 35 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1
2 | 723.0 | 1 | 0 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
3 | 725.0 | 0 | 2 | 69 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1
4 | 729.0 | 1 | 1 | 57 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ...
9820 | 17510.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
9821 | 17511.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
9822 | 17513.0 | 0 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
9823 | 17514.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0
9824 | 17515.0 | 1 | 1 | 74 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0

9825 rows × 16 columns

#### As we can see the 'sub_result' is encoded reversely we should flip it to make it in a matching pattern with other columns of data-set as 'consider' = 1 & 'clear' = 0.¶

In [17]:

    c_df['sub_result'].replace({0: 1, 1: 0}, inplace=True)
    c_df

Out[17]:

| id | gender | document_type | issuing_country | visual_authenticity_result | image_integrity_result | face_detection_result | image_quality_result | supported_document_result | conclusive_document_quality_result | colour_picture_result | data_validation_result | police_record_result | compromised_document_result | sub_result | result
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 718.0 | 1 | 2 | 35 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1
1 | 720.0 | 0 | 0 | 35 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1
2 | 723.0 | 1 | 0 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
3 | 725.0 | 0 | 2 | 69 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1
4 | 729.0 | 1 | 1 | 57 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ...
9820 | 17510.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
9821 | 17511.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
9822 | 17513.0 | 0 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
9823 | 17514.0 | 1 | 2 | 35 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
9824 | 17515.0 | 1 | 1 | 74 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0

9825 rows × 16 columns

# Feature Selection¶

#### Visualizing correlation between data features¶

In [18]:

    cor = c_df.corr()
    cor.style.background_gradient(cmap='coolwarm', axis=None)

Out[18]:

| id | gender | document_type | issuing_country | visual_authenticity_result | image_integrity_result | face_detection_result | image_quality_result | supported_document_result | conclusive_document_quality_result | colour_picture_result | data_validation_result | police_record_result | compromised_document_result | sub_result | result
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
id | 1.000000 | 0.030311 | -0.064577 | -0.076511 | -0.018751 | 0.154250 | 0.036549 | nan | nan | 0.156256 | -0.023275 | -0.072004 | nan | 0.021647 | 0.101027 | 0.127531
gender | 0.030311 | 1.000000 | -0.023036 | -0.015933 | 0.038617 | -0.020222 | 0.011487 | nan | nan | -0.020701 | 0.010530 | 0.010910 | nan | 0.013055 | 0.003406 | -0.006583
document_type | -0.064577 | -0.023036 | 1.000000 | 0.088797 | 0.040532 | 0.061854 | -0.012433 | nan | nan | 0.061130 | 0.017443 | -0.024664 | nan | -0.029587 | 0.065538 | 0.061017
issuing_country | -0.076511 | -0.015933 | 0.088797 | 1.000000 | -0.004406 | -0.018619 | 0.010115 | nan | nan | -0.018232 | -0.002777 | -0.036764 | nan | -0.009010 | -0.032152 | -0.027826
visual_authenticity_result | -0.018751 | 0.038617 | 0.040532 | -0.004406 | 1.000000 | -0.005935 | 0.419638 | nan | nan | -0.020149 | 0.227302 | 0.011478 | nan | -0.002454 | 0.455263 | 0.252002
image_integrity_result | 0.154250 | -0.020222 | 0.061854 | -0.018619 | -0.005935 | 1.000000 | -0.013605 | nan | nan | 0.997563 | 0.061537 | 0.008584 | nan | -0.009065 | 0.827921 | 0.930830
face_detection_result | 0.036549 | 0.011487 | -0.012433 | 0.010115 | 0.419638 | -0.013605 | 1.000000 | nan | nan | -0.013506 | -0.001881 | -0.006525 | nan | -0.001030 | 0.092099 | 0.105749
image_quality_result | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan
supported_document_result | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan
conclusive_document_quality_result | 0.156256 | -0.020701 | 0.061130 | -0.018232 | -0.020149 | 0.997563 | -0.013506 | nan | nan | 1.000000 | -0.000876 | 0.006537 | nan | -0.009043 | 0.822829 | 0.928561
colour_picture_result | -0.023275 | 0.010530 | 0.017443 | -0.002777 | 0.227302 | 0.061537 | -0.001881 | nan | nan | -0.000876 | 1.000000 | 0.025646 | nan | -0.000558 | 0.090083 | 0.057281
data_validation_result | -0.072004 | 0.010910 | -0.024664 | -0.036764 | 0.011478 | 0.008584 | -0.006525 | nan | nan | 0.006537 | 0.025646 | 1.000000 | nan | -0.001935 | 0.227737 | 0.198704
police_record_result | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan | nan
compromised_document_result | 0.021647 | 0.013055 | -0.029587 | -0.009010 | -0.002454 | -0.009065 | -0.001030 | nan | nan | -0.009043 | -0.000558 | -0.001935 | nan | 1.000000 | 0.063996 | 0.031363
sub_result | 0.101027 | 0.003406 | 0.065538 | -0.032152 | 0.455263 | 0.827921 | 0.092099 | nan | nan | 0.822829 | 0.090083 | 0.227737 | nan | 0.063996 | 1.000000 | 0.961849
result | 0.127531 | -0.006583 | 0.061017 | -0.027826 | 0.252002 | 0.930830 | 0.105749 | nan | nan | 0.928561 | 0.057281 | 0.198704 | nan | 0.031363 | 0.961849 | 1.000000

#### We can see from above correlation values 'image_integrity_result', 'conclusive_document_quality_result' and 'sub_result' are showing strong positive relationship with our predictor column 'result' , hence we can simply drop other features of data-set. Also note that our 'sub_result' is having a relationship with 'visual_authenticity_result' to some extinct so we keep that feature as well.¶

In [19]:

    final_data = c_df[['image_integrity_result', 'conclusive_document_quality_result', 'visual_authenticity_result', 'sub_result', 'result']]
    final_data

Out[19]:

| image_integrity_result | conclusive_document_quality_result | visual_authenticity_result | sub_result | result
--- | --- | --- | --- | --- | ---
0 | 1 | 1 | 0 | 1 | 1
1 | 1 | 1 | 0 | 1 | 1
2 | 0 | 0 | 0 | 0 | 0
3 | 1 | 1 | 0 | 1 | 1
4 | 1 | 1 | 0 | 1 | 1
... | ... | ... | ... | ... | ...
9820 | 0 | 0 | 0 | 0 | 0
9821 | 0 | 0 | 0 | 0 | 0
9822 | 0 | 0 | 0 | 0 | 0
9823 | 0 | 0 | 0 | 0 | 0
9824 | 0 | 0 | 0 | 0 | 0

9825 rows × 5 columns

# Results¶

#### From feature selection we can clearly see that the final result is highly depending on 'sub_result' and the 'sub_result' is depending on the following selected data features:¶

    1) Image Integrity Result
    2) Conclusive Document Quality Result
    3) Visual Authenticity Result

#### And the results are independent from the other features such as:¶

    1) Gender
    2) Country
    3) Document Type

##### Conclusions:¶

- The pass rate can be decline if any of the selected data features results in giving output as 'consider' more frequently.
- 'Image integrity' can be effected if the image may not be in a readable format.
- The 'document quality' can be effected if the scanned documents have missing segments or have incomplete details.
- 'Visual authenticity' can be effected if images are blurred or not still.

# Visualize¶

In [20]:

    import seaborn as sns
    import matplotlib.pyplot as plt

In [21]:

    sns.set(rc = {'figure.figsize':(15,8)})
    sns.countplot(x="variable", hue="value", data=pd.melt(final_data))
    plt.legend(labels=["c l e a r", "c o n s i d e r", "s u s p e c t e d"])
    plt.show()

In [ ]: