{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "955f8324",
   "metadata": {},
   "source": [
    "# Working with Local Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41dfe059",
   "metadata": {},
   "source": [
    "In this tutorial, we will show how to use your own local dataset with the Dataset class. The Dataset class can help you to manage and process your eyetracking data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15ff8a0c",
   "metadata": {},
   "source": [
    "## Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a01fa671",
   "metadata": {},
   "source": [
    "We import `pymovements` as the alias `pm` for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25f67fb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymovements as pm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ac248d",
   "metadata": {},
   "source": [
    "For demonstration purposes, we will use the raw data provided by the Toy dataset, a sample dataset that comes with *pymovements*.\n",
    "\n",
    "We will download the resources of this dataset the directory to simulate a local dataset for you.\n",
    "All downloaded archive files are automatically extracted and then removed.\n",
    "The directory of the dataset will be `data/my_dataset`.\n",
    "\n",
    "After that we won't use the python class anymore and delete the object\n",
    "(the files on your system will stay in place).\n",
    "Don't worry if you're confused about these lines as they are not relevant to your use case.\n",
    "\n",
    "Just keep in mind that we now have some files with gaze data in the directory `data/my_dataset`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1d8e6e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "toy_dataset = pm.Dataset('ToyDataset', path='data/my_dataset')\n",
    "toy_dataset.download(remove_finished=True)\n",
    "\n",
    "del toy_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405ebd53",
   "metadata": {},
   "source": [
    "## Define your Experiment\n",
    "\n",
    "To use the Dataset class, we first need to create an Experiment instance. This class represents the properties of the experiment, such as the screen dimensions and sampling rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a89a38a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = pm.gaze.Experiment(\n",
    "    screen_width_px=1280,\n",
    "    screen_height_px=1024,\n",
    "    screen_width_cm=38,\n",
    "    screen_height_cm=30.2,\n",
    "    distance_cm=68,\n",
    "    origin='lower left',\n",
    "    sampling_rate=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddbcf20e",
   "metadata": {},
   "source": [
    "## Parameters for File Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd6ab2fe",
   "metadata": {},
   "source": [
    "We also define a `filename_format` which is a pattern expression used to match and extract values from filenames of data files in the dataset. For example, `r'trial_{text_id:d}_{page_id:d}.csv'` will match filenames that follow the pattern `trial_{text_id}_{page_id}.csv` and extract the values of `text_id` and `page_id` for each file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a7d236",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_format = r'trial_{text_id:d}_{page_id:d}.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d38ae134",
   "metadata": {},
   "source": [
    "Both values of `text_id` and `page_id` are numeric. We can use a map to define the casting of these values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24a626f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename_format_dtypes = {\n",
    "    'text_id': int,\n",
    "    'page_id': int,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e32b202",
   "metadata": {},
   "source": [
    "We can also adjust how the CSV files are read."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e039de9b",
   "metadata": {},
   "source": [
    "The `column_map` dictionary maps the original column names in the CSV files to the desired column names. Here the original column names are 'timestamp', **'x'**, and **'y'**, and the desired column names are **'time'**, **'x_pix'**, and **'y_pix'**, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19d897a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "column_map = {\n",
    "    'timestamp': 'time',\n",
    "    'x': 'x_pix',\n",
    "    'y': 'y_pix',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1f91fdd",
   "metadata": {},
   "source": [
    "Here, we specify that the separator in the CSV files is a tab (**'\\t'**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14b40a9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_read_kwargs = {\n",
    "    'separator': '\\t',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f1c04a7",
   "metadata": {},
   "source": [
    "## Define and load the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b869d6c5",
   "metadata": {},
   "source": [
    "Next we use all these definitions and create a `DatasetDefinition` by passing in the root directory, Experiment instance, and other optional parameters such as the filename regular expression and custom CSV reading parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a9e7ac4",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_definition = pm.DatasetDefinition(\n",
    "    name='my_dataset',\n",
    "    experiment=experiment,\n",
    "    filename_format=filename_format,\n",
    "    filename_format_dtypes=filename_format_dtypes,\n",
    "    column_map=column_map,\n",
    "    custom_read_kwargs=custom_read_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "190ce9f6",
   "metadata": {},
   "source": [
    "Finally we create a `Dataset` instance by using the `DatasetDefinition` and specifying the directory path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22cb63cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pm.Dataset(\n",
    "    definition=dataset_definition,\n",
    "    path='data/my_dataset/',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73774d9c",
   "metadata": {},
   "source": [
    "If we have a root data directory which holds all your local datasets we can further need to define the paths of the dataset.\n",
    "\n",
    "The `dataset`, `raw`, `preprocessed`, and `events` parameters define the names of the directories for the dataset, raw data, preprocessed data, and events data, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ebeb02a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_paths = pm.DatasetPaths(\n",
    "    root='data/',\n",
    "    raw='raw',\n",
    "    preprocessed='preprocessed',\n",
    "    events='events',\n",
    ")\n",
    "\n",
    "dataset = pm.Dataset(\n",
    "    definition=dataset_definition,\n",
    "    path=dataset_paths,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ce1117c-b90c-44d9-8701-f12c01d47409",
   "metadata": {},
   "source": [
    "Now let's load the dataset into memory. Here we select a subset including the first page of texts with ID 1 and 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bab0b02-6040-4927-8926-b5be44dc0a75",
   "metadata": {},
   "outputs": [],
   "source": [
    "subset = {\n",
    "    'text_id': [1, 2],\n",
    "    'page_id': 1,\n",
    "}\n",
    "\n",
    "dataset.load(subset=subset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b805c88-20b5-4e5b-8532-030bbd30d051",
   "metadata": {},
   "source": [
    "## Use the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d00d7e6c",
   "metadata": {},
   "source": [
    "Once we have created the Dataset instance, we can use its methods to preprocess and analyze data in our local dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e09dfb0-fe97-4c55-bfec-665833d0fc37",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.gaze[0].frame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f1eb66d",
   "metadata": {},
   "source": [
    "Here we use the `pix2deg` method to convert the pixel coordinates to degrees of visual angle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08024785",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.pix2deg()\n",
    "\n",
    "dataset.gaze[0].frame"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b43cf3af",
   "metadata": {},
   "source": [
    "We can use the `pos2vel` method to calculate the velocity of the gaze position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c9ed003-45ca-40a7-8a1d-d397babaf31c",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.pos2vel(method='savitzky_golay', window_length=7, polyorder=2)\n",
    "\n",
    "dataset.gaze[0].frame"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}