{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a1d9b1a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JinmiaoChenLab/DISCOtoolkit_py/blob/main/docs/download_data.ipynb)\n",
    "# Download Data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cfdcaaf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "We recommend the user visit our [DISCO](https://disco.bii.a-star.edu.sg/v1/) website and filter the sample based on the metadata provided on the website.\n",
    "\n",
    "In this tutorial, we will provide a quick guideline for downloading the data using discotoolkit.\n",
    "Steps:\n",
    "- Visit the website to find a sample or cell type of interest.\n",
    "- Utilize the `dt.filter_disco_metadata` function to filter the data from the database.\n",
    "- Employ the `dt.download_disco_data` function to download the data in the `h5ad` extension based on the filtered sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ea0bd44c-44a6-4761-af8b-115838fb99df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for google colab\n",
    "# pip install discotoolkit\n",
    "# first import the installed package\n",
    "import discotoolkit as dt\n",
    "import os\n",
    "import scanpy as sc\n",
    "import anndata as ad\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# adding ignore warning to clean the code\n",
    "import warnings\n",
    "\n",
    "# Ignore all warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e2870f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# setting params for the visualisation\n",
    "# verbosity: errors (0), warnings (1), info (2), hints (3)\n",
    "sc.settings.verbosity = 1\n",
    "sc.settings.set_figure_params(\n",
    "    dpi=300, frameon=False, figsize=(3, 3), facecolor='white')\n",
    "\n",
    "# Set the plotting backend to 'umap-learn' for interactive plot\n",
    "sc.settings.plotting_backend = 'umap-learn'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004c0c10",
   "metadata": {},
   "outputs": [],
   "source": [
    "Before filtering, see which metadata fields DISCO has, and which values one of them takes. Any of these fields' values can be used in `dt.Filter`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "01acf74f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sample_id', 'project_id', 'sample_type', 'tissue', 'anatomical_site', 'disease', 'platform', 'age_group', 'cell_sorting', 'disease_subtype', 'treatment', 'time_point', 'subject_id', 'age', 'gender', 'race', 'infection', 'batch', 'disease_stage', 'genotype', 'rna_source', 'other_metadata', 'disease_grade', 'cell_number', 'median_umi', 'rds_md5', 'rds_size', 'source_cell_line', 'source_tissue', 'source_disease', 'source_cell_type', 'induced_cell_tissue', 'collect_time']\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"10x'3\", \"10x3'\", \"10x5'\", '10xATAC+RNA', 'BD Rhapsody', 'CEL-Seq2', \"Crohn's disease\", 'Drop-Seq', 'Drop-seq', 'FocuSCOPE', 'GEXSCOPE', 'ICELL8', 'InDrop', 'MARS-Seq', 'Microwell-seq', 'Organoid', 'QUARTZ-SEQ2', 'SORTseq', 'STRT-Seq', 'Seq-Well', 'Seq-Well/Drop-seq', 'Split-Seq', 'SureCell', 'inDrop', 'nan', 'sc-FTDseq', 'unknown']\n"
     ]
    }
   ],
   "source": [
    "# every metadata field of the DISCO samples\n",
    "print(dt.list_all_columns())\n",
    "\n",
    "# the values one field takes, e.g. the sequencing platforms\n",
    "print(sorted(map(str, dt.list_metadata_item(\"platform\"))))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "638e2e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "For this example, we wanted to download LYVE1 macrophage from lung tissue as our cell type of interest. We subset only the highly confident cell type prediction based on our manual annotation and the minimum 300 cells per sample. Notice that we specify `include_cell_type_children` as true to include all sub-cell types if applicable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4ae4b2e6-40b9-4f85-ac23-71352bf9ccd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filtering the sample based on the metadata\n",
    "filter = dt.Filter(\n",
    "    sample_id = None,\n",
    "    project_id = None,\n",
    "    tissue = \"lung\",\n",
    "    platform = [\"10x3'\"],\n",
    "    sample_type = [\"control\"],\n",
    "    cell_type = \"LYVE1 macrophage\",\n",
    "    cell_type_confidence = \"high\",\n",
    "    include_cell_type_children = True,\n",
    "    min_cell_per_sample = 300\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bd237d4f-5741-4240-b2a4-98524bad983d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:Filtering sample\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:Retrieving ontology from DISCO database\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:13 samples and 7074 cells were found\n"
     ]
    }
   ],
   "source": [
    "# apply the filter of interest\n",
    "metadata = dt.filter_disco_metadata(filter)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c37d2131",
   "metadata": {},
   "outputs": [],
   "source": [
    "After applying the filter to the database, now we can download the dataset into the `disco_data` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "318f7ff1-5658-46ae-8803-fb534d3dfa54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download data to the local directory\n",
    "dir_path = \"disco_data\"\n",
    "download_log = dt.download_disco_data(metadata, output_dir = dir_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "82347faa",
   "metadata": {},
   "outputs": [],
   "source": [
    "We can check to see if the files have been download successfully using `os` library with function `listdir()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "04f3eb72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['GSM5068929.h5ad',\n",
       " 'GSM5068936.h5ad',\n",
       " 'GSM5068924.h5ad',\n",
       " 'GSM7017456.h5ad',\n",
       " 'GSM5068932.h5ad',\n",
       " 'GSM5068933.h5ad',\n",
       " 'GSM4827185.h5ad',\n",
       " 'GSM3891621.h5ad',\n",
       " 'GSM5068934.h5ad',\n",
       " 'GSM3891625.h5ad',\n",
       " 'GSM5068927.h5ad',\n",
       " 'GSM5068928.h5ad',\n",
       " 'GSM4827184.h5ad']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double checking to see if the files have been download successf\n",
    "os.listdir(dir_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "38cbbee3",
   "metadata": {},
   "outputs": [],
   "source": [
    "Since the data has been successfully downloaded, we can then use the `scanpy` library to visualize the UMAP of the downloaded data. This can be achieved by concatenating the files in the downloaded directory into one combined `anndata`. The combined `anndata` is useful for downstream analysis or integration, enabling further study of the scRNA-seq."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "35a379ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AnnData object with n_obs × n_vars = 7074 × 33538\n",
      "    obs: 'cell_type', 'sample_id', 'project_id'\n",
      "    layers: None (.X)\n"
     ]
    }
   ],
   "source": [
    "directory = 'disco_data'\n",
    "adata_list = []\n",
    "\n",
    "# Read and store datasets\n",
    "for filename in os.listdir(directory):\n",
    "    if filename.endswith('.h5ad'):\n",
    "        filepath = os.path.join(directory, filename)\n",
    "        adata = sc.read(filepath)\n",
    "\n",
    "        # Rename columns with periods in `.obs` attribute\n",
    "        for col in adata.obs.columns:\n",
    "            new_col = re.sub(r'\\.', '_', col)\n",
    "            adata.obs.rename(columns={col: new_col}, inplace=True)\n",
    "\n",
    "        # Rename columns with periods in `.var` attribute\n",
    "        for col in adata.var.columns:\n",
    "            new_col = re.sub(r'\\.', '_', col)\n",
    "            adata.var.rename(columns={col: new_col}, inplace=True)\n",
    "\n",
    "        del adata.raw\n",
    "\n",
    "        adata_list.append(adata)\n",
    "        \n",
    "# Concatenate the datasets\n",
    "combined_adata = ad.concat(adata_list)\n",
    "\n",
    "# making the obs name unique\n",
    "combined_adata.obs_names_make_unique()\n",
    "\n",
    "# Print the concatenated dataset\n",
    "print(combined_adata)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c42936f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "Great! We have successfully filtered and downloaded the cell type-specific data from the DISCO database."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "disco",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}