{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {"display_name": "Python 3", "language": "python", "name": "python3"},
  "language_info": {"name": "python", "version": "3.10.0"}
 },
 "cells": [

  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 02 · Task 3 — Independent EDA and Cleaning · SOLUTIONS\n",
    "\n",
    "**Dataset:** `dataset_D_git_classroom_activity.csv`\n",
    "\n",
    "---"
   ]
  },

  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 — Load and Inspect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import sweetviz as sv\n",
    "import dtale\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('dataset_D_git_classroom_activity.csv')\n",
    "\n",
    "print(f'Shape: {df.shape}')\n",
    "print('\\nColumn types:')\n",
    "print(df.dtypes)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **What to note:**\n",
    "> - `coverage_percent` should be numeric but is `object` — formatting problem in raw values\n",
    "> - `is_weekend`, `label_is_high_quality`, `exam_period` should be boolean but are `object`\n",
    "> - `commit_message_length` is `float64` rather than `int` — a sign that missing values forced a float type\n",
    "\n",
    "---\n",
    "\n",
    "## Part 2 — Automated Profiling with SweetViz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report = sv.analyze(df)\n",
    "report.show_html('sweetviz_git_raw.html', open_browser=False)\n",
    "print('Report saved. Open sweetviz_git_raw.html in your browser.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Triage checklist — answers\n",
    "\n",
    "| Question | Finding |\n",
    "|---|---|\n",
    "| Which columns have missing values? Which has the most? | `pr_merge_time_hours` (71.7%), `commit_message_length` (7%), `build_duration_s` (2.1%), `time_to_ci_minutes` (2%) |\n",
    "| Which columns should be boolean? | `is_weekend`, `label_is_high_quality`, `exam_period` |\n",
    "| Which columns should be numeric? | `coverage_percent` — shown as TEXT due to comma decimal separators |\n",
    "| `event_type` distinct count | ~42 — should be 7; case/whitespace variants |\n",
    "| What is unusual about `ci_status`? | Besides case/whitespace variants, `FAILED` and `FAILURE` are synonyms that need merging |\n",
    "| Suspicious numeric ranges | `lines_added` max 5000, `time_to_ci_minutes` max 1578, `pr_merge_time_hours` has negative values |\n",
    "\n",
    "---\n",
    "\n",
    "## Part 3 — Navigate and Inspect with D-Tale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = dtale.show(df, host='127.0.0.1', subprocess=False, open_browser=False)\n",
    "print('D-Tale running at:', d._url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Part 4 — Clean with Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean = df.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.1 — Boolean columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect\n",
    "print(sorted(df_clean['is_weekend'].dropna().unique().tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Boolean keys included so the mapping is safe to re-run\n",
    "bool_map = {\n",
    "    'True': True,  'true': True,  '1': True,  'Yes': True,  True: True,\n",
    "    'False': False, 'false': False, '0': False, 'No': False, False: False\n",
    "}\n",
    "\n",
    "for col in ['is_weekend', 'label_is_high_quality', 'exam_period']:\n",
    "    df_clean[col] = df_clean[col].map(bool_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify\n",
    "for col in ['is_weekend', 'label_is_high_quality', 'exam_period']:\n",
    "    print(f\"{col}: {df_clean[col].value_counts().to_dict()} | nulls: {df_clean[col].isna().sum()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.2 — `is_bot_user`: case and whitespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect\n",
    "print(df_clean['is_bot_user'].value_counts().to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean['is_bot_user'] = df_clean['is_bot_user'].str.strip().str.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify\n",
    "print(df_clean['is_bot_user'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.3 — Categorical columns: case and whitespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect\n",
    "print(f'dominant_language unique before: {df_clean[\"dominant_language\"].nunique()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Strip and lowercase for columns with pure case/whitespace variance\n",
    "for col in ['dominant_language', 'editor', 'event_type']:\n",
    "    df_clean[col] = df_clean[col].str.strip().str.lower()\n",
    "\n",
    "# os: strip and lowercase, then merge win → windows\n",
    "df_clean['os'] = (\n",
    "    df_clean['os']\n",
    "    .str.strip()\n",
    "    .str.lower()\n",
    "    .replace({'win': 'windows'})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify\n",
    "for col in ['dominant_language', 'editor', 'os', 'event_type']:\n",
    "    print(f\"{col} ({df_clean[col].nunique()} unique): {sorted(df_clean[col].dropna().unique().tolist())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.4 — `ci_status`: case, whitespace, and synonym merging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect\n",
    "print(df_clean['ci_status'].value_counts().to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1: strip and lowercase\n",
    "# Step 2: merge 'failure' into 'failed'\n",
    "# Rationale: both indicate the CI pipeline did not complete successfully.\n",
    "# 'failed' is the more common and explicit term in CI tooling (GitHub Actions, Jenkins).\n",
    "df_clean['ci_status'] = (\n",
    "    df_clean['ci_status']\n",
    "    .str.strip()\n",
    "    .str.lower()\n",
    "    .replace({'failure': 'failed'})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify\n",
    "print(df_clean['ci_status'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Decision:** `failure` → `failed`. Both mean the CI pipeline did not complete successfully. `failed` is the canonical term used by major CI tools (GitHub Actions, Jenkins, GitLab CI) and is more explicit.\n",
    "\n",
    "---\n",
    "\n",
    "### 4.5 — `coverage_percent`: comma decimal separator, type conversion, and outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect\n",
    "print(f'dtype: {df_clean[\"coverage_percent\"].dtype}')\n",
    "comma_rows = df_clean['coverage_percent'].astype(str).str.contains(',', na=False)\n",
    "print(f'Rows with comma: {comma_rows.sum()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix: replace comma, convert to float\n",
    "df_clean['coverage_percent'] = (\n",
    "    df_clean['coverage_percent']\n",
    "    .astype(str)\n",
    "    .str.replace(',', '.', regex=False)\n",
    "    .replace('nan', float('nan'))\n",
    "    .astype(float)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verify — also check for values outside the valid 0-100 range\n",
    "print(f'dtype: {df_clean[\"coverage_percent\"].dtype}')\n",
    "print(df_clean['coverage_percent'].describe().round(2))\n",
    "print(f'\\nValues < 0:   {(df_clean[\"coverage_percent\"] < 0).sum()} rows')\n",
    "print(f'Values > 100: {(df_clean[\"coverage_percent\"] > 100).sum()} rows')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# coverage_percent must be in [0, 100] — values outside this range are logging errors\n",
    "invalid_cov = (df_clean['coverage_percent'] < 0) | (df_clean['coverage_percent'] > 100)\n",
    "df_clean.loc[invalid_cov, 'coverage_percent'] = float('nan')\n",
    "print(f'Invalid coverage values set to NaN: {invalid_cov.sum()}')\n",
    "print(f'Range after: {df_clean[\"coverage_percent\"].min():.1f} – {df_clean[\"coverage_percent\"].max():.1f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.6 — Missing values: decisions and strategy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect missing counts\n",
    "missing = df_clean.isnull().sum()\n",
    "pct = (missing / len(df_clean) * 100).round(1)\n",
    "pd.DataFrame({'missing': missing, '%': pct})[missing > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Investigate pr_merge_time_hours — which event types have non-null values?\n",
    "print(df_clean.loc[df_clean['pr_merge_time_hours'].notna(), 'event_type'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Finding:** `pr_merge_time_hours` is only non-null for `pr_merged` and `pr_opened` events — exactly the rows where a merge time is meaningful. This is **structural missingness (MNAR — Missing Not At Random)**, not a data quality problem. Imputing or dropping these rows would destroy valid analytical signal. Keep as NaN.\n",
    "\n",
    "| Column | Decision | Rationale |\n",
    "|---|---|---|\n",
    "| `pr_merge_time_hours` | Keep NaN | Structural: only meaningful for PR events |\n",
    "| `commit_message_length` | Keep NaN | Unclear cause — may be bot commits or merge commits without messages |\n",
    "| `build_duration_s` | Keep NaN | Sporadic; likely CI jobs that did not reach the build phase |\n",
    "| `time_to_ci_minutes` | Keep NaN | Sporadic; likely events that did not trigger CI |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All four columns: leave as NaN — no action needed\n",
    "# (Documented above)\n",
    "print('Missing value strategy: all four columns kept as NaN.')\n",
    "print('No rows dropped.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.7 — Outliers and impossible values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A — Negative pr_merge_time_hours\n",
    "neg_mask = df_clean['pr_merge_time_hours'] < 0\n",
    "print(f'Negative pr_merge_time_hours: {neg_mask.sum()}')\n",
    "print(df_clean.loc[neg_mask, ['event_type', 'pr_merge_time_hours']].head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix A\n",
    "df_clean.loc[neg_mask, 'pr_merge_time_hours'] = float('nan')\n",
    "print(f'Negative values set to NaN: {neg_mask.sum()}')\n",
    "print(f'New min: {df_clean[\"pr_merge_time_hours\"].min():.2f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B — tests_failed > tests_run (cross-column logical check)\n",
    "impossible_mask = df_clean['tests_failed'] > df_clean['tests_run']\n",
    "print(f'Rows where tests_failed > tests_run: {impossible_mask.sum()}')\n",
    "print(df_clean.loc[impossible_mask, ['tests_run', 'tests_failed']].describe().round(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix B — set tests_failed to NaN for impossible rows\n",
    "# We do not touch tests_run — it may be correct; tests_failed is the unreliable value\n",
    "df_clean.loc[impossible_mask, 'tests_failed'] = float('nan')\n",
    "print(f'tests_failed set to NaN: {impossible_mask.sum()}')\n",
    "# Verify: no remaining impossible rows\n",
    "remaining = df_clean['tests_failed'] > df_clean['tests_run']\n",
    "print(f'Remaining impossible rows: {remaining.sum()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# C — lines_added and lines_deleted outliers\n",
    "print('lines_added distribution:')\n",
    "print(df_clean['lines_added'].describe().round(1))\n",
    "print(f'\\nRows > 1000 lines added: {(df_clean[\"lines_added\"] > 1000).sum()}')\n",
    "print(df_clean.loc[df_clean['lines_added'] > 1000,\n",
    "                   ['event_type', 'lines_added', 'lines_deleted', 'dominant_language']].head(8).to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Decision: commits adding or deleting > 1000 lines are flagged as outliers.\n",
    "# While large commits can be legitimate (adding a framework, vendoring dependencies),\n",
    "# values of 5000 lines are extreme for a classroom context and likely logging errors.\n",
    "# We set them to NaN rather than dropping — other columns in these rows remain valid.\n",
    "threshold = 1000\n",
    "large_add = df_clean['lines_added'] > threshold\n",
    "large_del = df_clean['lines_deleted'] > threshold\n",
    "\n",
    "df_clean.loc[large_add, 'lines_added'] = float('nan')\n",
    "df_clean.loc[large_del, 'lines_deleted'] = float('nan')\n",
    "\n",
    "print(f'lines_added outliers set to NaN: {large_add.sum()}')\n",
    "print(f'lines_deleted outliers set to NaN: {large_del.sum()}')\n",
    "print(f'\\nlines_added max after: {df_clean[\"lines_added\"].max()}')\n",
    "print(f'lines_deleted max after: {df_clean[\"lines_deleted\"].max()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### 4.8 — `timestamp`: mixed datetime formats *(optional)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First pass — handles ISO 8601 formats\n",
    "df_clean['timestamp'] = pd.to_datetime(df_clean['timestamp'], utc=True, errors='coerce')\n",
    "print(f'timestamp dtype: {df_clean[\"timestamp\"].dtype}')\n",
    "print(f'Unparsed (NaT) after first pass: {df_clean[\"timestamp\"].isna().sum()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The remaining NaTs are DD/MM/YYYY format rows — apply a second pass\n",
    "# using the systematic try_formats approach from Task 2\n",
    "\n",
    "def try_formats(series, formats):\n",
    "    result = pd.Series(pd.NaT, index=series.index, dtype='datetime64[ns, UTC]')\n",
    "    remaining = series.copy()\n",
    "    for fmt in formats:\n",
    "        parsed = pd.to_datetime(remaining, format=fmt, errors='coerce', utc=True)\n",
    "        resolved_idx = parsed.index[parsed.notna()]\n",
    "        result.loc[resolved_idx] = parsed.loc[resolved_idx]\n",
    "        remaining = remaining.drop(index=resolved_idx)\n",
    "    return result\n",
    "\n",
    "candidate_formats = [\n",
    "    '%d/%m/%Y %H:%M',\n",
    "    '%m/%d/%Y %H:%M',\n",
    "    '%d/%m/%Y',\n",
    "    '%m/%d/%Y',\n",
    "]\n",
    "\n",
    "unparsed_idx = df_clean.index[df_clean['timestamp'].isna()]\n",
    "raw_unparsed = df.loc[unparsed_idx, 'timestamp']\n",
    "resolved = try_formats(raw_unparsed, candidate_formats)\n",
    "df_clean.loc[unparsed_idx, 'timestamp'] = resolved\n",
    "\n",
    "print(f'Resolved in second pass: {resolved.notna().sum()}')\n",
    "print(f'Still NaT (truly ambiguous): {df_clean[\"timestamp\"].isna().sum()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Part 5 — Verify with D-Tale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d.kill()\n",
    "d_clean = dtale.show(df_clean, host='127.0.0.1', subprocess=False, open_browser=False)\n",
    "print('D-Tale (cleaned) running at:', d_clean._url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Part 6 — Before vs After with SweetViz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Exclude timestamp — SweetViz cannot compare string vs datetime64\n",
    "exclude = ['timestamp']\n",
    "compare = sv.compare(\n",
    "    [df.drop(columns=exclude), 'Raw'],\n",
    "    [df_clean.drop(columns=exclude).reset_index(drop=True), 'Cleaned']\n",
    ")\n",
    "compare.show_html('sweetviz_git_comparison.html', open_browser=False)\n",
    "print('Comparison report saved.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Part 7 — Save"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_clean.to_csv('dataset_D_git_classroom_activity_clean.csv', index=False)\n",
    "print(f'Saved: {len(df_clean)} rows, {len(df_clean.columns)} columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Reflection — Suggested Answers\n",
    "\n",
    "**1. `pr_merge_time_hours` missing 71.7% — is this a data quality problem?**  \n",
    "No. The missingness is structural: only `pr_merged` and `pr_opened` events have a merge time by definition. Every other event type (commit, push, CI run, etc.) has no merge time to record. This is MNAR — Missing Not At Random — and the pattern itself carries meaning. Imputing or dropping these rows would be wrong.\n",
    "\n",
    "**2. What does the cross-column check reveal that single-column inspection misses?**  \n",
    "Single-column inspection of `tests_failed` shows values ranging from 0 to 245 — nothing obviously wrong. Single-column inspection of `tests_run` also looks normal. Only by comparing the two together does the logical impossibility appear: you cannot fail more tests than you ran. This is a category of data quality issue that automated profiling tools like SweetViz do not detect.\n",
    "\n",
    "**3. What knowledge beyond the data was needed for `ci_status`?**  \n",
    "Domain knowledge about CI systems: that `failed` and `failure` refer to the same pipeline outcome, and that `failed` is the conventional term in tools like GitHub Actions and Jenkins. Without this knowledge, a purely statistical analysis would treat them as two separate categories and silently undercount CI failures.\n",
    "\n",
    "**4. What was the same as Task 2? What was new?**  \n",
    "Same: boolean encoding chaos, case/whitespace inconsistency, comma decimal separator, structural missingness reasoning, negative value treatment, D-Tale navigation, SweetViz before/after.  \n",
    "New: synonym merging (`ci_status`), cross-column logical consistency check (`tests_failed > tests_run`), out-of-range numeric check (`coverage_percent` outside 0–100).  \n",
    "Takeaway: the core cleaning patterns transfer across domains. What changes is the domain knowledge needed to make the decisions — which canonical form to use, what physical constraints apply to each variable, what constitutes a structurally justified missing value."
   ]
  }

 ]
}
