{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "011263bd",
   "metadata": {},
   "source": [
    "# Downloading Public Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14bfa755",
   "metadata": {},
   "source": [
    "## What you will learn in this tutorial:\n",
    "\n",
    "* how to download and extract one of the available public datasets\n",
    "* how to customize the default directory structure"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c31a9917",
   "metadata": {},
   "source": [
    "## Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12af367f",
   "metadata": {},
   "source": [
    "We import `pymovements` as the alias `pm` for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bef4ae0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymovements as pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9096f56b",
   "metadata": {},
   "source": [
    "pymovements provides a library of publicly available datasets.\n",
    "\n",
    "You can browse through the available dataset definitions here:\n",
    "[Datasets](https://pymovements.readthedocs.io/en/latest/reference/pymovements.datasets.html#module-pymovements.datasets)\n",
    "\n",
    "For this tutorial we will limit ourselves to the `ToyDataset` due to its minimal space requirements.\n",
    "\n",
    "Other datasets can be downloaded by simply replacing `ToyDataset` with one of the other available datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dfd61b9",
   "metadata": {},
   "source": [
    " ## Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c891cb45",
   "metadata": {},
   "source": [
    "First we initialize our public dataset by specifying its name and the root data directory.\n",
    "\n",
    "Our dataset will then be placed in a directory with the name of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "375b5f97",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')\n",
    "\n",
    "dataset.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6d791e5",
   "metadata": {},
   "source": [
    "If you only want to specify a root directory which contains all your datasets, you can pass a `DatasetPaths` instance.\n",
    "\n",
    "The directory of your dataset will have the same name as in the dataset definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1cdaaa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths = pm.DatasetPaths(root='data/')\n",
    "dataset = pm.Dataset('ToyDataset', path=dataset_paths)\n",
    "\n",
    "dataset.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe8e3189",
   "metadata": {},
   "source": [
    "Can also specify an alternative dataset directory for your downloaded dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31d6980d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths_alt = pm.DatasetPaths(root='data/', dataset='my_dataset')\n",
    "dataset_alt = pm.Dataset('ToyDataset', path=dataset_paths_alt)\n",
    "\n",
    "dataset_alt.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f644b5b",
   "metadata": {},
   "source": [
    "## Downloading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f4c2dca",
   "metadata": {},
   "source": [
    "The dataset will then be downloaded by calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80b2e7c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5884f578",
   "metadata": {},
   "source": [
    "As we see from the download message, the dataset resource has been downloaded to a downloads directory.\n",
    "\n",
    "You can get the path to this directory from the `Datset.paths.downloads` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77990896",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.paths.downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a4486e2",
   "metadata": {},
   "source": [
    "You can also specify a custom directory name during initialization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c2054cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths_3 = pm.DatasetPaths(root='data/', downloads='new_downloads')\n",
    "dataset_3 = pm.Dataset('ToyDataset', path=dataset_paths_3)\n",
    "\n",
    "dataset_3.paths.downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8310fc82",
   "metadata": {},
   "source": [
    "By default, all archives are recursively extracted to `Dataset.paths.raw`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "782311cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.paths.raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f93ef98",
   "metadata": {},
   "source": [
    "If you want to remove the downloaded archives after extraction to save some space, you can set `remove_finished` to `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dda16489",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.extract(remove_finished=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f2e1a8",
   "metadata": {},
   "source": [
    "This is also available for the `PublicDataset.download()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0afffa82",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.download(remove_finished=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5226af41",
   "metadata": {},
   "source": [
    "## Loading into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f07d791",
   "metadata": {},
   "source": [
    "The `PublicDataset` class is a subset of the `Dataset` class and thus inherits all its functionality.\n",
    "\n",
    "Hende, we can load the data into our working memory by using the common `load()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b38485fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71eedf37",
   "metadata": {},
   "source": [
    "Let's verify that we have correctly scanned the dataset files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7e7d82",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.fileinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8d46dd",
   "metadata": {},
   "source": [
    "Wonderful, all of our data has been downloaded and loaded in successfully!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "252493f7",
   "metadata": {},
   "source": [
    "## What you have learned in this tutorial:\n",
    "\n",
    "* how to initialize a public dataset\n",
    "* how to download and extract dataset resources\n",
    "* how to customize the default directory structure\n",
    "* how to load the dataset into your working memory"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}