Saving and Loading Event Data#
What you will learn in this tutorial:#
how to save your event data
how to load your event data
Preparations#
We import pymovements as the alias pm for convenience.
[1]:
import pymovements as pm
/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/stable/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Let’s start by downloading our ToyDataset and loading in its data:
[2]:
dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
dataset.download()
dataset.load()
Using already downloaded and verified file: data/ToyDataset/downloads/pymovements-toy-dataset.zip
Extracting pymovements-toy-dataset.zip to data/ToyDataset/raw
100%|██████████| 20/20 [00:00<00:00, 20.84it/s]
[2]:
<pymovements.dataset.dataset.Dataset at 0x7f3ffaf21e80>
Now let’s do some preprocessing:
[3]:
dataset.pix2deg()
dataset.pos2vel()
dataset.gaze[0].frame.head()
100%|██████████| 20/20 [00:01<00:00, 10.50it/s]
100%|██████████| 20/20 [00:01<00:00, 19.52it/s]
[3]:
shape: (5, 8)
| time | stimuli_x | stimuli_y | text_id | page_id | pixel | position | velocity |
|---|---|---|---|---|---|---|---|
| f32 | f32 | f32 | i64 | i64 | list[f32] | list[f32] | list[f32] |
| 1.988145e6 | -1.0 | -1.0 | 0 | 1 | [206.800003, 152.399994] | [-10.697598, -8.8524] | [null, null] |
| 1.988146e6 | -1.0 | -1.0 | 0 | 1 | [206.899994, 152.100006] | [-10.695184, -8.859678] | [null, null] |
| 1.988147e6 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.800003] | [-10.692768, -8.866957] | [1.610438, -5.256017] |
| 1.988148e6 | -1.0 | -1.0 | 0 | 1 | [207.100006, 151.699997] | [-10.690351, -8.869382] | [0.40261, -4.447301] |
| 1.988149e6 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.5] | [-10.692768, -8.874233] | [0.402451, -3.234386] |
Detect some events:
[4]:
dataset.detect_events('ivt')
dataset.events[0].frame.head()
20it [00:00, 37.67it/s]
[4]:
shape: (5, 6)
| text_id | page_id | name | onset | offset | duration |
|---|---|---|---|---|---|
| i64 | i64 | str | i64 | i64 | i64 |
| 0 | 1 | "fixation" | 1988147 | 1988322 | 175 |
| 0 | 1 | "fixation" | 1988351 | 1988546 | 195 |
| 0 | 1 | "fixation" | 1988592 | 1988736 | 144 |
| 0 | 1 | "fixation" | 1988788 | 1989013 | 225 |
| 0 | 1 | "fixation" | 1989060 | 1989170 | 110 |
[5]:
dataset.detect_events('microsaccades')
dataset.events[0].frame.head()
20it [00:00, 25.18it/s]
[5]:
shape: (5, 6)
| text_id | page_id | name | onset | offset | duration |
|---|---|---|---|---|---|
| i64 | i64 | str | i64 | i64 | i64 |
| 0 | 1 | "fixation" | 1988147 | 1988322 | 175 |
| 0 | 1 | "fixation" | 1988351 | 1988546 | 195 |
| 0 | 1 | "fixation" | 1988592 | 1988736 | 144 |
| 0 | 1 | "fixation" | 1988788 | 1989013 | 225 |
| 0 | 1 | "fixation" | 1989060 | 1989170 | 110 |
We now have event dataframes available with our detected event data.
Saving#
Saving your event data is as simple as:
[6]:
dataset.save_events()
100%|██████████| 20/20 [00:00<00:00, 2030.80it/s]
[6]:
<pymovements.dataset.dataset.Dataset at 0x7f3ffaf21e80>
All of the event data is saved into this directory:
[7]:
dataset.paths.events
[7]:
PosixPath('data/ToyDataset/events')
Let’s confirm it by printing all the new files in this directory:
[8]:
print(list(dataset.paths.events.glob('*/*/*')))
[PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.feather'), PosixPath('data/ToyDataset/events/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.feather')]
All of the files have been saved into the Dataset.paths.events as feather files.
If we want to save the data into an alternative directory and also use a different file format like csv we can use the following:
[9]:
dataset.save_events(events_dirname='events_csv', extension='csv')
100%|██████████| 20/20 [00:00<00:00, 895.32it/s]
[9]:
<pymovements.dataset.dataset.Dataset at 0x7f3ffaf21e80>
Let’s confirm again by printing all the new files in this alternative directory:
[10]:
alternative_dirpath = dataset.path / 'events_csv'
print(list(alternative_dirpath.glob('*/*/*')))
[PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.csv'), PosixPath('data/ToyDataset/events_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.csv')]
Loading#
Now let’s imagine that this preprocessing and saving was done in another file and we only want to load the preprocessed data.
We simulate this by initializing a new dataset object. We don’t need to download any additional data.
[11]:
preprocessed_dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
The preprocessed data can now simply be loaded by setting preprocessed to True:
[12]:
preprocessed_dataset.load(events=True)
dataset.events[0].frame.head()
100%|██████████| 20/20 [00:00<00:00, 20.60it/s]
100%|██████████| 20/20 [00:00<00:00, 842.99it/s]
[12]:
shape: (5, 6)
| text_id | page_id | name | onset | offset | duration |
|---|---|---|---|---|---|
| i64 | i64 | str | i64 | i64 | i64 |
| 0 | 1 | "fixation" | 1988147 | 1988322 | 175 |
| 0 | 1 | "fixation" | 1988351 | 1988546 | 195 |
| 0 | 1 | "fixation" | 1988592 | 1988736 | 144 |
| 0 | 1 | "fixation" | 1988788 | 1989013 | 225 |
| 0 | 1 | "fixation" | 1989060 | 1989170 | 110 |
By default, the events directory and the feather extension will be chosen.
In case of alternative directory names or other file formats you can use the following:
[13]:
preprocessed_dataset.load(
events=True,
events_dirname='events_csv',
extension='csv',
)
dataset.events[0].frame.head()
100%|██████████| 20/20 [00:00<00:00, 20.67it/s]
100%|██████████| 20/20 [00:00<00:00, 631.73it/s]
[13]:
shape: (5, 6)
| text_id | page_id | name | onset | offset | duration |
|---|---|---|---|---|---|
| i64 | i64 | str | i64 | i64 | i64 |
| 0 | 1 | "fixation" | 1988147 | 1988322 | 175 |
| 0 | 1 | "fixation" | 1988351 | 1988546 | 195 |
| 0 | 1 | "fixation" | 1988592 | 1988736 | 144 |
| 0 | 1 | "fixation" | 1988788 | 1989013 | 225 |
| 0 | 1 | "fixation" | 1989060 | 1989170 | 110 |
What you have learned in this tutorial:#
saving your preprocesed data using
Dataset.save_preprocessed()load your preprocesed data using
Dataset.load(preprocessed=True)using custom directory names by specifying
preprocessed_dirnameusing other file formats than the default
featherformat by specifyingextension