zimmersandro/cds-104
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Tag 4

Browse Source
This commit is contained in:
git-sandro 2026-06-04 13:33:39 +02:00
parent 5acf4bf1ac
commit 8b2dc788c1
10 changed files with 2478 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

427
notebooks/Data_Quality.ipynb Normal file
View File

@ -0,0 +1,427 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Data Quality\n",
"## Authors: Dr. Yves Staudt\n",
"In diesem Kurs beschreiben wir die Qualität des Datensatzes. Dabei gehen wir nur auf spezifische Punkte ein. Wir beschreiben objektiv die Datengrösse und die Anzahl fehlende Werte. Eine kleine subjektive Bewertung wird auch vorgenommen. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Packages "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Load Data"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"df = pd.read_csv(\"../../../Data/hotel_bookings/hotel_bookings.csv\")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>hotel</th>\n",
" <th>is_canceled</th>\n",
" <th>lead_time</th>\n",
" <th>arrival_date_year</th>\n",
" <th>arrival_date_month</th>\n",
" <th>arrival_date_week_number</th>\n",
" <th>arrival_date_day_of_month</th>\n",
" <th>stays_in_weekend_nights</th>\n",
" <th>stays_in_week_nights</th>\n",
" <th>adults</th>\n",
" <th>...</th>\n",
" <th>deposit_type</th>\n",
" <th>agent</th>\n",
" <th>company</th>\n",
" <th>days_in_waiting_list</th>\n",
" <th>customer_type</th>\n",
" <th>adr</th>\n",
" <th>required_car_parking_spaces</th>\n",
" <th>total_of_special_requests</th>\n",
" <th>reservation_status</th>\n",
" <th>reservation_status_date</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>Resort Hotel</td>\n",
" <td>0</td>\n",
" <td>342</td>\n",
" <td>2015</td>\n",
" <td>July</td>\n",
" <td>27</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>...</td>\n",
" <td>No Deposit</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>Transient</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>Check-Out</td>\n",
" <td>2015-07-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>Resort Hotel</td>\n",
" <td>0</td>\n",
" <td>737</td>\n",
" <td>2015</td>\n",
" <td>July</td>\n",
" <td>27</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>...</td>\n",
" <td>No Deposit</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>Transient</td>\n",
" <td>0.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>Check-Out</td>\n",
" <td>2015-07-01</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>Resort Hotel</td>\n",
" <td>0</td>\n",
" <td>7</td>\n",
" <td>2015</td>\n",
" <td>July</td>\n",
" <td>27</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>No Deposit</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>Transient</td>\n",
" <td>75.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>Check-Out</td>\n",
" <td>2015-07-02</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>Resort Hotel</td>\n",
" <td>0</td>\n",
" <td>13</td>\n",
" <td>2015</td>\n",
" <td>July</td>\n",
" <td>27</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>1</td>\n",
" <td>...</td>\n",
" <td>No Deposit</td>\n",
" <td>304.0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>Transient</td>\n",
" <td>75.0</td>\n",
" <td>0</td>\n",
" <td>0</td>\n",
" <td>Check-Out</td>\n",
" <td>2015-07-02</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>Resort Hotel</td>\n",
" <td>0</td>\n",
" <td>14</td>\n",
" <td>2015</td>\n",
" <td>July</td>\n",
" <td>27</td>\n",
" <td>1</td>\n",
" <td>0</td>\n",
" <td>2</td>\n",
" <td>2</td>\n",
" <td>...</td>\n",
" <td>No Deposit</td>\n",
" <td>240.0</td>\n",
" <td>NaN</td>\n",
" <td>0</td>\n",
" <td>Transient</td>\n",
" <td>98.0</td>\n",
" <td>0</td>\n",
" <td>1</td>\n",
" <td>Check-Out</td>\n",
" <td>2015-07-03</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>5 rows × 32 columns</p>\n",
"</div>"
],
"text/plain": [
" hotel is_canceled lead_time arrival_date_year arrival_date_month \\\n",
"0 Resort Hotel 0 342 2015 July \n",
"1 Resort Hotel 0 737 2015 July \n",
"2 Resort Hotel 0 7 2015 July \n",
"3 Resort Hotel 0 13 2015 July \n",
"4 Resort Hotel 0 14 2015 July \n",
"\n",
" arrival_date_week_number arrival_date_day_of_month \\\n",
"0 27 1 \n",
"1 27 1 \n",
"2 27 1 \n",
"3 27 1 \n",
"4 27 1 \n",
"\n",
" stays_in_weekend_nights stays_in_week_nights adults ... deposit_type \\\n",
"0 0 0 2 ... No Deposit \n",
"1 0 0 2 ... No Deposit \n",
"2 0 1 1 ... No Deposit \n",
"3 0 1 1 ... No Deposit \n",
"4 0 2 2 ... No Deposit \n",
"\n",
" agent company days_in_waiting_list customer_type adr \\\n",
"0 NaN NaN 0 Transient 0.0 \n",
"1 NaN NaN 0 Transient 0.0 \n",
"2 NaN NaN 0 Transient 75.0 \n",
"3 304.0 NaN 0 Transient 75.0 \n",
"4 240.0 NaN 0 Transient 98.0 \n",
"\n",
" required_car_parking_spaces total_of_special_requests reservation_status \\\n",
"0 0 0 Check-Out \n",
"1 0 0 Check-Out \n",
"2 0 0 Check-Out \n",
"3 0 0 Check-Out \n",
"4 0 1 Check-Out \n",
"\n",
" reservation_status_date \n",
"0 2015-07-01 \n",
"1 2015-07-01 \n",
"2 2015-07-02 \n",
"3 2015-07-02 \n",
"4 2015-07-03 \n",
"\n",
"[5 rows x 32 columns]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Grösse des Datensatzes"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(119390, 32)"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Der Datensatz stellt mit 119'390 Beobachtungen und 32 Variablen einen guten Datensatz mit deren Grössen für einen Ansatz mit Machine Learning Modellen. Es gibt keine Regeln die sagen ab, wann ein Datensatz zu klein ist. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Anzahl fehlende Werte"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Die Methode df.isnull() liefert uns den gleichen Datensatz mit boolschen Werten, welche wiedergeben ob die einzelnen Einträge einen Wert besitzen (False bzw. 0) oder NaN sind (True bzw. 1). Da die boolschen Werte von Python bei Berechnungen als 1 für True und 0 für False dargestellt werden können, wird das arithmetische Mittel .mean() verwendet, um den Anteil fehlender Einträge für jede Variable zu errechnen."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"company 94.306893\n",
"agent 13.686238\n",
"country 0.408744\n",
"children 0.003350\n",
"dtype: float64\n"
]
}
],
"source": [
"# Berechnung des Anteiles fehlender Einträge \n",
"missing_values = df.isnull().mean() * 100 # # Errechnung der Anteile fehlender Werte für jede Spalte bzw. Variable\n",
"missing_values = missing_values[missing_values > 0] # Filtern der Spalten ohne fehlende Einträge\n",
"\n",
"# Sortieren der Spalten nach höchsten Anteilen von fehlenden Werten\n",
"missing_values = missing_values.sort_values(ascending=False)\n",
"\n",
"# Ausgabe des finalen Dataframes\n",
"print(missing_values)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Die geringe Anzahl fehlende Werte der Spalten `country` und `children` können ohne weiteres entfernt oder gefüllt werden. Hingegen die Variablen `company` und `agent` zeigen eine hohe Anzahl fehlender Werte auf. Deren fehlende Einträge sollten daher nicht gefüllt werden. Die betroffenen Datenpunkte zeigen eine Spezifikation auf, in diesem Fall hat das Fehlen einer Information vermutlich eine reale Bedeutung. In diesem Fall bedeutet ein fehlender Wert in der Variable `company` und `agent`, dass die betroffenen Buchungen nicht von einem Agenten oder einem Buchungsbüro bzw. einer Firma durchgeführt worden ist."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Wenn ich die Variable `company` als Referenz nehme, deren Wert zu 94 % fehlt, dann bleiben mir nur 6 % der vollständigen Beobachtungen. Das deutet objektiv darauf hin, dass der Datensatz von schlechter Qualität ist.\n",
"\n",
"Betrachte ich jedoch eine alternative objektive Messung – den Anteil vollständiger Variablen – erhalte ich einen Wert von 87,5 %, was keineswegs schlecht ist.\n",
"Der Anteil vollständiger Variablen wird wie folgt berechnet: \n",
"1 - (Anzahl der Variablen mit fehlenden Werten / Gesamtanzahl der Variablen)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Die Prozentanzahl vollständige Variablen: 0.875\n"
]
}
],
"source": [
"print(f\"Die Prozentanzahl vollständige Variablen: {1 - 4/32}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Subjektive Datenanalyse\n",
"Wenn man sich mit dem Datensatz und die gegeben Literaturen anschaut, merkt man, dass der Datensatz einige Schwächen aufzeigt. Der Datensatz stellt eine relatistische Problematik dar, welche für solche Zwecke die Gegebenheiten der Praxis wieder gibt. Aus diesem Grund stellt dieser Datensatz eine gute Ausgangslage für die zur analysierenden Fragestellung. Somit schätze ich die subjektive Gegebenheit mit den gegebenen Varialben als gut ein."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Referenzen \n",
"- Nuno Antonio, Ana Almeida, and Luis Nunes, Predicting Hotel Bookings Cancellation With a Machine Learning Classification Model, 16th IEEE Internation Conference on Machine Learning and Applications, 2017a\n",
"- Nuno Antonio, Ana Almeida, and Luis Nunes, Predicting hotel booking cancellations to decrease uncertainty and increase revenue, Tourism and Management Studies, Volume 13, Issue 2, 2017b\n",
"- Nuno Antonio, Ana Almeida, and Luis Nunes, Big Data in Hotel Revenue Management: Exploring Cancellation Drivers to Gain Insights Into Booking Cancellation Behavior, Cornell Hospitality Quarterly, Volume 60 (4), 2019a\n",
"- Nuno Antonio, Ana Almeida, and Luis Nunes, Hotel booking demand datasets, Data in Brief, Volume 22, 2019b"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "ml2",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

2049
notebooks/Imputing_Data.ipynb Normal file
View File

File diff suppressed because it is too large Load Diff

3
requirements.txt
View File

@ -1 +1,2 @@
psycopg2-binary == 2.9.11
psycopg2-binary == 2.9.11
pandas == 3.0.3

BIN
unterlagen/M01_Data_driven_Services_Products.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/M02_Introduction_to_ML.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/M03_Data_Thinking_Canvas.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/M04_Data_Strategy_Quality_Tidy_Data.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/aufgaben/M01_Data_driven_Services_Products_Aufgabenblatt.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/aufgaben/ML_Kurs_Aufgabenblatt_Data_Quality.pdf Normal file
View File

Binary file not shown.

BIN
unterlagen/aufgaben/ML_Kurs_Aufgabenblatt_Data_Thinking.pdf Normal file
View File

Binary file not shown.