Working with Local Dataset#
In this tutorial, we will show how to use your own local dataset with the Dataset class. The Dataset class can help you to manage and process your eyetracking data.
For demonstration purposes, we will use the raw data provided by the Toy dataset, a sample dataset that comes with pymovements.
[1]:
import pymovements as pm
toy_dataset = pm.datasets.ToyDataset(
root='data/',
download=True,
extract=True,
remove_finished=True,
)
/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/v0.7.0/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Downloading http://github.com/aeye-lab/pymovements-toy-dataset/zipball/6cb5d663317bf418cec0c9abe1dde5085a8a8ebd/ to data/ToyDataset/downloads/pymovements-toy-dataset.zip
pymovements-toy-dataset.zip: 100%|██████████| 3.06M/3.06M [00:00<00:00, 24.7MB/s]
Define your Experiment#
To use the Dataset class, we first need to create an Experiment instance. This class represents the properties of the experiment, such as the screen dimensions and sampling rate.
[2]:
experiment = pm.gaze.Experiment(
screen_width_px=1280,
screen_height_px=1024,
screen_width_cm=38,
screen_height_cm=30.2,
distance_cm=68,
origin='lower left',
sampling_rate=1000,
)
Parameters for File Parsing#
We also define a filename_regex which is a regular expression used to match and extract values from filenames of data files in the dataset. For example, r'trial_(?P<text_id>\d+)_(?P<page_id>\d+).csv' will match filenames that follow the pattern trial_{text_id}_{page_id}.csv and extract the values of text_id and page_id for each file.
[3]:
filename_regex = r'trial_(?P<text_id>\d+)_(?P<page_id>\d+).csv'
Both values of text_id and page_id are numeric. We can use a map to define the casting of these values.
[4]:
filename_regex_dtypes = {
'text_id': int,
'page_id': int,
}
We can also adjust how the CSV files are read.
The column_map dictionary maps the original column names in the CSV files to the desired column names. Here the original column names are ‘timestamp’, ‘x’, and ‘y’, and the desired column names are ‘time’, ‘x_right_pix’, and ‘y_right_pix’, respectively.
[5]:
column_map = {
'timestamp': 'time',
'x': 'x_right_pix',
'y': 'y_right_pix',
}
Here, we specify that the separator in the CSV files is a tab (‘:nbsphinx-math:`t’`), and we provide the list of original column names and desired column names as the ‘columns’ and ‘new_columns’ parameters, respectively.
[6]:
read_csv_kwargs = {
'sep': '\t',
'columns': list(column_map.keys()),
'new_columns': list(column_map.values()),
}
Define and load the Dataset#
Finaly we create a Dataset instance by passing in the root directory, Experiment instance, and other optional parameters such as the filename regular expression and custom CSV reading parameters. The dataset_dirname, raw_dirname, preprocessed_dirname, and events_dirname parameters define the names of the directories for the dataset, raw data, preprocessed data, and events data, respectively.
[7]:
# Define the path to the dataset directory
dataset_dir = './data/ToyDataset/'
# Set up the Dataset object
dataset = pm.datasets.Dataset(
root=dataset_dir,
experiment=experiment,
filename_regex=filename_regex,
filename_regex_dtypes=filename_regex_dtypes,
custom_read_kwargs=read_csv_kwargs,
dataset_dirname='.',
raw_dirname='raw',
preprocessed_dirname='preprocessed',
events_dirname='events',
)
Now we can load the dataset. Here we select a subset including the first page of texts with ID 1 and 2.
[8]:
subset = {
'text_id': [1, 2],
'page_id': 1,
}
dataset.load(subset=subset)
100%|██████████| 2/2 [00:00<00:00, 184.06it/s]
Use the Dataset#
Once we have created the Dataset instance, we can use its methods to preprocess and analyze data in our local dataset.
[9]:
dataset.gaze[0].frame
[9]:
| text_id | page_id | time | x_right_pix | y_right_pix |
|---|---|---|---|---|
| i64 | i64 | f64 | f64 | f64 |
| 1 | 1 | 2.415266e6 | 176.8 | 140.2 |
| 1 | 1 | 2.415267e6 | 176.7 | 139.8 |
| 1 | 1 | 2.415268e6 | 176.7 | 139.3 |
| 1 | 1 | 2.415269e6 | 176.6 | 139.3 |
| 1 | 1 | 2.41527e6 | 176.7 | 139.3 |
| 1 | 1 | 2.415271e6 | 176.8 | 139.5 |
| 1 | 1 | 2.415272e6 | 177.3 | 139.8 |
| 1 | 1 | 2.415273e6 | 177.8 | 140.0 |
| 1 | 1 | 2.415274e6 | 178.3 | 140.0 |
| 1 | 1 | 2.415275e6 | 178.3 | 139.9 |
| 1 | 1 | 2.415276e6 | 178.0 | 140.2 |
| 1 | 1 | 2.415277e6 | 177.7 | 140.4 |
| ... | ... | ... | ... | ... |
| 1 | 1 | 2.438308e6 | 649.1 | 633.7 |
| 1 | 1 | 2.438309e6 | 648.8 | 633.9 |
| 1 | 1 | 2.43831e6 | 649.1 | 634.1 |
| 1 | 1 | 2.438311e6 | 649.6 | 634.2 |
| 1 | 1 | 2.438312e6 | 650.1 | 634.1 |
| 1 | 1 | 2.438313e6 | 650.0 | 634.0 |
| 1 | 1 | 2.438314e6 | 649.9 | 633.9 |
| 1 | 1 | 2.438315e6 | 649.9 | 633.9 |
| 1 | 1 | 2.438316e6 | 650.1 | 633.7 |
| 1 | 1 | 2.438317e6 | 650.2 | 633.5 |
| 1 | 1 | 2.438318e6 | 650.0 | 633.2 |
| 1 | 1 | 2.438319e6 | 649.7 | 633.1 |
Here we use the pix2deg method to convert the pixel coordinates to degrees of visual angle.
[10]:
dataset.pix2deg()
dataset.gaze[0].frame
100%|██████████| 2/2 [00:00<00:00, 453.88it/s]
[10]:
| text_id | page_id | time | x_right_pix | y_right_pix | y_right_pos | x_right_pos |
|---|---|---|---|---|---|---|
| i64 | i64 | f64 | f64 | f64 | f64 | f64 |
| 1 | 1 | 2.415266e6 | 176.8 | 140.2 | -12.297242 | -8.259494 |
| 1 | 1 | 2.415267e6 | 176.7 | 139.8 | -12.306793 | -8.261927 |
| 1 | 1 | 2.415268e6 | 176.7 | 139.3 | -12.318732 | -8.261927 |
| 1 | 1 | 2.415269e6 | 176.6 | 139.3 | -12.318732 | -8.264361 |
| 1 | 1 | 2.41527e6 | 176.7 | 139.3 | -12.318732 | -8.261927 |
| 1 | 1 | 2.415271e6 | 176.8 | 139.5 | -12.313957 | -8.259494 |
| 1 | 1 | 2.415272e6 | 177.3 | 139.8 | -12.306793 | -8.247325 |
| 1 | 1 | 2.415273e6 | 177.8 | 140.0 | -12.302018 | -8.235155 |
| 1 | 1 | 2.415274e6 | 178.3 | 140.0 | -12.302018 | -8.222985 |
| 1 | 1 | 2.415275e6 | 178.3 | 139.9 | -12.304406 | -8.222985 |
| 1 | 1 | 2.415276e6 | 178.0 | 140.2 | -12.297242 | -8.230287 |
| 1 | 1 | 2.415277e6 | 177.7 | 140.4 | -12.292466 | -8.237589 |
| ... | ... | ... | ... | ... | ... | ... |
| 1 | 1 | 2.438308e6 | 649.1 | 633.7 | -0.145082 | 3.415265 |
| 1 | 1 | 2.438309e6 | 648.8 | 633.9 | -0.140079 | 3.407836 |
| 1 | 1 | 2.43831e6 | 649.1 | 634.1 | -0.135077 | 3.415265 |
| 1 | 1 | 2.438311e6 | 649.6 | 634.2 | -0.132575 | 3.427645 |
| 1 | 1 | 2.438312e6 | 650.1 | 634.1 | -0.135077 | 3.440025 |
| 1 | 1 | 2.438313e6 | 650.0 | 634.0 | -0.137578 | 3.437549 |
| 1 | 1 | 2.438314e6 | 649.9 | 633.9 | -0.140079 | 3.435073 |
| 1 | 1 | 2.438315e6 | 649.9 | 633.9 | -0.140079 | 3.435073 |
| 1 | 1 | 2.438316e6 | 650.1 | 633.7 | -0.145082 | 3.440025 |
| 1 | 1 | 2.438317e6 | 650.2 | 633.5 | -0.150085 | 3.442501 |
| 1 | 1 | 2.438318e6 | 650.0 | 633.2 | -0.157589 | 3.437549 |
| 1 | 1 | 2.438319e6 | 649.7 | 633.1 | -0.160091 | 3.430121 |
We can use the pos2vel method to calculate the velocity of the gaze position.
[11]:
dataset.pos2vel(method='savitzky_golay', window_length=7, polyorder=2)
dataset.gaze[0].frame
100%|██████████| 2/2 [00:00<00:00, 293.35it/s]
[11]:
| text_id | page_id | time | x_right_pix | y_right_pix | y_right_pos | x_right_pos | x_right_vel | y_right_vel |
|---|---|---|---|---|---|---|---|---|
| i64 | i64 | f64 | f64 | f64 | f64 | f64 | f64 | f64 |
| 1 | 1 | 2.415266e6 | 176.8 | 140.2 | -12.297242 | -8.259494 | -5.302027 | -13.473668 |
| 1 | 1 | 2.415267e6 | 176.7 | 139.8 | -12.306793 | -8.261927 | -3.04214 | -9.49412 |
| 1 | 1 | 2.415268e6 | 176.7 | 139.3 | -12.318732 | -8.261927 | -0.782253 | -5.514573 |
| 1 | 1 | 2.415269e6 | 176.6 | 139.3 | -12.318732 | -8.264361 | 1.477633 | -1.535025 |
| 1 | 1 | 2.41527e6 | 176.7 | 139.3 | -12.318732 | -8.261927 | 4.085296 | 1.53496 |
| 1 | 1 | 2.415271e6 | 176.8 | 139.5 | -12.313957 | -8.259494 | 6.780025 | 3.411015 |
| 1 | 1 | 2.415272e6 | 177.3 | 139.8 | -12.306793 | -8.247325 | 8.083958 | 3.15519 |
| 1 | 1 | 2.415273e6 | 177.8 | 140.0 | -12.302018 | -8.235155 | 6.867045 | 3.155252 |
| 1 | 1 | 2.415274e6 | 178.3 | 140.0 | -12.302018 | -8.222985 | 3.998521 | 2.899534 |
| 1 | 1 | 2.415275e6 | 178.3 | 139.9 | -12.304406 | -8.222985 | 0.347668 | 3.411407 |
| 1 | 1 | 2.415276e6 | 178.0 | 140.2 | -12.297242 | -8.230287 | -1.738596 | 4.093787 |
| 1 | 1 | 2.415277e6 | 177.7 | 140.4 | -12.292466 | -8.237589 | -1.999405 | 5.287927 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1 | 1 | 2.438308e6 | 649.1 | 633.7 | -0.145082 | 3.415265 | -0.265312 | 0.714688 |
| 1 | 1 | 2.438309e6 | 648.8 | 633.9 | -0.140079 | 3.407836 | 2.210777 | 2.590745 |
| 1 | 1 | 2.43831e6 | 649.1 | 634.1 | -0.135077 | 3.415265 | 4.06786 | 2.590745 |
| 1 | 1 | 2.438311e6 | 649.6 | 634.2 | -0.132575 | 3.427645 | 5.129065 | 0.714688 |
| 1 | 1 | 2.438312e6 | 650.1 | 634.1 | -0.135077 | 3.440025 | 4.686916 | -0.536016 |
| 1 | 1 | 2.438313e6 | 650.0 | 634.0 | -0.137578 | 3.437549 | 3.006674 | -1.786721 |
| 1 | 1 | 2.438314e6 | 649.9 | 633.9 | -0.140079 | 3.435073 | 1.503314 | -2.680081 |
| 1 | 1 | 2.438315e6 | 649.9 | 633.9 | -0.140079 | 3.435073 | 0.265288 | -3.484104 |
| 1 | 1 | 2.438316e6 | 650.1 | 633.7 | -0.145082 | 3.440025 | -0.353725 | -4.020119 |
| 1 | 1 | 2.438317e6 | 650.2 | 633.5 | -0.150085 | 3.442501 | -1.650699 | -4.913478 |
| 1 | 1 | 2.438318e6 | 650.0 | 633.2 | -0.157589 | 3.437549 | -2.947673 | -5.806836 |
| 1 | 1 | 2.438319e6 | 649.7 | 633.1 | -0.160091 | 3.430121 | -4.244647 | -6.700195 |