{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Downloading Public Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "## What you will learn in this tutorial:\n",
    "\n",
    "* how to download and extract one of the available public datasets\n",
    "* how to customize the default directory structure"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3",
   "metadata": {},
   "source": [
    "We import `pymovements` as the alias `pm` for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymovements as pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {},
   "source": [
    "pymovements provides a library of publicly available datasets.\n",
    "\n",
    "You can browse through the available dataset definitions here:\n",
    "[Datasets](https://pymovements.readthedocs.io/en/latest/reference/pymovements.datasets.html#module-pymovements.datasets)\n",
    "\n",
    "For this tutorial we will limit ourselves to the `ToyDataset` due to its minimal space requirements.\n",
    "\n",
    "Other datasets can be downloaded by simply replacing `ToyDataset` with one of the other available datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6",
   "metadata": {},
   "source": [
    " ## Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "First we initialize our public dataset by specifying its name and the root data directory.\n",
    "\n",
    "Our dataset will then be placed in a directory with the name of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')\n",
    "\n",
    "dataset.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "If you only want to specify a root directory which contains all your datasets, you can pass a `DatasetPaths` instance.\n",
    "\n",
    "The directory of your dataset will have the same name as in the dataset definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths = pm.DatasetPaths(root='data/')\n",
    "dataset = pm.Dataset('ToyDataset', path=dataset_paths)\n",
    "\n",
    "dataset.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "Can also specify an alternative dataset directory for your downloaded dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths_alt = pm.DatasetPaths(root='data/', dataset='my_dataset')\n",
    "dataset_alt = pm.Dataset('ToyDataset', path=dataset_paths_alt)\n",
    "\n",
    "dataset_alt.path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "## Downloading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "The dataset will then be downloaded by calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16",
   "metadata": {},
   "source": [
    "As we see from the download message, the dataset resource has been downloaded to a downloads directory.\n",
    "\n",
    "You can get the path to this directory from the `Datset.paths.downloads` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.paths.downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {},
   "source": [
    "You can also specify a custom directory name during initialization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths_3 = pm.DatasetPaths(root='data/', downloads='new_downloads')\n",
    "dataset_3 = pm.Dataset('ToyDataset', path=dataset_paths_3)\n",
    "\n",
    "dataset_3.paths.downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {},
   "source": [
    "By default, all archives are recursively extracted to `Dataset.paths.raw`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.paths.raw"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "If you want to remove the downloaded archives after extraction to save some space, you can set `remove_finished` to `True`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.extract(remove_finished=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24",
   "metadata": {},
   "source": [
    "This is also available for the `PublicDataset.download()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.download(remove_finished=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26",
   "metadata": {},
   "source": [
    "## Loading into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27",
   "metadata": {},
   "source": [
    "The `PublicDataset` class is a subset of the `Dataset` class and thus inherits all its functionality.\n",
    "\n",
    "Hende, we can load the data into our working memory by using the common `load()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29",
   "metadata": {},
   "source": [
    "Let's verify that we have correctly scanned the dataset files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.fileinfo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31",
   "metadata": {},
   "source": [
    "Wonderful, all of our data has been downloaded and loaded in successfully!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32",
   "metadata": {},
   "source": [
    "## What you have learned in this tutorial:\n",
    "\n",
    "* how to initialize a public dataset\n",
    "* how to download and extract dataset resources\n",
    "* how to customize the default directory structure\n",
    "* how to load the dataset into your working memory"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}