Saving and Loading Preprocessed Data#
What you will learn in this tutorial:#
how to save your preprocessed data
how to load your preprocessed data
Preparations#
We import pymovements
as the alias pm
for convenience.
[1]:
import pymovements as pm
/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/latest/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Let’s start by downloading our ToyDataset
and loading in its data:
[2]:
dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
dataset.download()
dataset.load()
Using already downloaded and verified file: data/ToyDataset/downloads/pymovements-toy-dataset.zip
Extracting pymovements-toy-dataset.zip to data/ToyDataset/raw
100%|██████████| 20/20 [00:00<00:00, 35.42it/s]
[2]:
<pymovements.dataset.dataset.Dataset at 0x7efe0486ce80>
Now let’s load in the data and do some preprocessing:
[3]:
dataset.pix2deg()
dataset.pos2vel()
dataset.gaze[0].frame.head()
100%|██████████| 20/20 [00:00<00:00, 21.14it/s]
100%|██████████| 20/20 [00:00<00:00, 37.95it/s]
[3]:
time | stimuli_x | stimuli_y | text_id | page_id | pixel | position | velocity |
---|---|---|---|---|---|---|---|
f32 | f32 | f32 | i64 | i64 | list[f32] | list[f32] | list[f32] |
1.988145e6 | -1.0 | -1.0 | 0 | 1 | [206.800003, 152.399994] | [-10.697598, -8.8524] | [null, null] |
1.988146e6 | -1.0 | -1.0 | 0 | 1 | [206.899994, 152.100006] | [-10.695184, -8.859678] | [null, null] |
1.988147e6 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.800003] | [-10.692768, -8.866957] | [1.610438, -5.256017] |
1.988148e6 | -1.0 | -1.0 | 0 | 1 | [207.100006, 151.699997] | [-10.690351, -8.869382] | [0.40261, -4.447301] |
1.988149e6 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.5] | [-10.692768, -8.874233] | [0.402451, -3.234386] |
We have now added some additional columns for degrees in visual angle and velocity.
Saving#
Saving your preprocessed data is as simple as:
[4]:
dataset.save_preprocessed()
100%|██████████| 20/20 [00:00<00:00, 236.35it/s]
[4]:
<pymovements.dataset.dataset.Dataset at 0x7efe0486ce80>
All of the preprocessed data is saved into this directory:
[5]:
dataset.paths.preprocessed
[5]:
PosixPath('data/ToyDataset/preprocessed')
Let’s confirm it by printing all the new files in this directory:
[6]:
print(list(dataset.paths.preprocessed.glob('*/*/*')))
[PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.feather')]
All of the files have been saved into the Dataset.paths.preprocessed
as feather
files.
If we want to save the data into an alternative directory and also use a different file format like csv
we can use the following:
[7]:
dataset.save_preprocessed(preprocessed_dirname='preprocessed_csv', extension='csv')
100%|██████████| 20/20 [00:00<00:00, 39.01it/s]
[7]:
<pymovements.dataset.dataset.Dataset at 0x7efe0486ce80>
Let’s confirm again by printing all the new files in this alternative directory:
[8]:
alternative_dirpath = dataset.path / 'preprocessed_csv'
print(list(alternative_dirpath.glob('*/*/*')))
[PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.csv')]
Loading#
Now let’s imagine that this preprocessing and saving was done in another file and we only want to load the preprocessed data.
We simulate this by initializing a new dataset object. We don’t need to download any additional data.
[9]:
events_dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
The preprocessed data can now simply be loaded by setting preprocessed
to True
:
[10]:
events_dataset.load(preprocessed=True)
events_dataset.gaze[0].frame.head()
100%|██████████| 20/20 [00:00<00:00, 320.18it/s]
[10]:
time | stimuli_x | stimuli_y | pixel | position | velocity | text_id | page_id |
---|---|---|---|---|---|---|---|
f32 | f32 | f32 | list[f32] | list[f32] | list[f32] | i64 | i64 |
1.988145e6 | -1.0 | -1.0 | [206.800003, 152.399994] | [-10.697598, -8.8524] | [null, null] | 0 | 1 |
1.988146e6 | -1.0 | -1.0 | [206.899994, 152.100006] | [-10.695184, -8.859678] | [null, null] | 0 | 1 |
1.988147e6 | -1.0 | -1.0 | [207.0, 151.800003] | [-10.692768, -8.866957] | [1.610438, -5.256017] | 0 | 1 |
1.988148e6 | -1.0 | -1.0 | [207.100006, 151.699997] | [-10.690351, -8.869382] | [0.40261, -4.447301] | 0 | 1 |
1.988149e6 | -1.0 | -1.0 | [207.0, 151.5] | [-10.692768, -8.874233] | [0.402451, -3.234386] | 0 | 1 |
By default, the preprocessed
directory and the feather
extension will be chosen.
In case of alternative directory names or other file formats you can use the following:
[11]:
events_dataset.load(
preprocessed=True,
preprocessed_dirname='preprocessed_csv',
extension='csv',
)
events_dataset.gaze[0].frame.head()
100%|██████████| 20/20 [00:01<00:00, 15.99it/s]
[11]:
time | stimuli_x | stimuli_y | text_id | page_id | pixel | position | velocity |
---|---|---|---|---|---|---|---|
i64 | f64 | f64 | i64 | i64 | list[f64] | list[f64] | list[f64] |
1988145 | -1.0 | -1.0 | 0 | 1 | [206.8, 152.4] | [-10.697598, -8.8524] | [null, null] |
1988146 | -1.0 | -1.0 | 0 | 1 | [206.9, 152.1] | [-10.695184, -8.859678] | [null, null] |
1988147 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.8] | [-10.692768, -8.866957] | [1.6104381, -5.256017] |
1988148 | -1.0 | -1.0 | 0 | 1 | [207.1, 151.7] | [-10.690352, -8.869382] | [0.40261, -4.447301] |
1988149 | -1.0 | -1.0 | 0 | 1 | [207.0, 151.5] | [-10.692768, -8.874233] | [0.402451, -3.234386] |
What you have learned in this tutorial:#
saving your preprocesed data using
Dataset.save_preprocessed()
load your preprocesed data using
Dataset.load(preprocessed=True)
using custom directory names by specifying
preprocessed_dirname
using other file formats than the default
feather
format by specifyingextension