Saving and Loading Preprocessed Data

What you will learn in this tutorial:

• how to save your preprocessed data

• how to load your preprocessed data

Preparations

We import pymovements as the alias pm for convenience.

import pymovements as pm

/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/stable/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Let’s start by downloading our ToyDataset and loading in its data:

dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
dataset.download()
dataset.load()

Using already downloaded and verified file: data/ToyDataset/downloads/pymovements-toy-dataset.zip
Extracting pymovements-toy-dataset.zip to data/ToyDataset/raw

100%|██████████| 20/20 [00:00<00:00, 20.67it/s]

<pymovements.dataset.dataset.Dataset at 0x7f28d9aa9fd0>

Now let’s load in the data and do some preprocessing:

dataset.pix2deg()
dataset.pos2vel()

dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:01<00:00, 10.49it/s]
100%|██████████| 20/20 [00:01<00:00, 19.60it/s]

shape: (5, 8)

time	stimuli_x	stimuli_y	text_id	page_id	pixel	position	velocity
f32	f32	f32	i64	i64	list[f32]	list[f32]	list[f32]
1.988145e6	-1.0	-1.0	0	1	[206.800003, 152.399994]	[-10.697598, -8.8524]	[null, null]
1.988146e6	-1.0	-1.0	0	1	[206.899994, 152.100006]	[-10.695184, -8.859678]	[null, null]
1.988147e6	-1.0	-1.0	0	1	[207.0, 151.800003]	[-10.692768, -8.866957]	[1.610438, -5.256017]
1.988148e6	-1.0	-1.0	0	1	[207.100006, 151.699997]	[-10.690351, -8.869382]	[0.40261, -4.447301]
1.988149e6	-1.0	-1.0	0	1	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402451, -3.234386]

We have now added some additional columns for degrees in visual angle and velocity.

Saving

Saving your preprocessed data is as simple as:

dataset.save_preprocessed()

100%|██████████| 20/20 [00:00<00:00, 131.50it/s]

<pymovements.dataset.dataset.Dataset at 0x7f28d9aa9fd0>

All of the preprocessed data is saved into this directory:

dataset.paths.preprocessed

PosixPath('data/ToyDataset/preprocessed')

Let’s confirm it by printing all the new files in this directory:

print(list(dataset.paths.preprocessed.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.feather')]

All of the files have been saved into the Dataset.paths.preprocessed as feather files.

If we want to save the data into an alternative directory and also use a different file format like csv we can use the following:

dataset.save_preprocessed(preprocessed_dirname='preprocessed_csv', extension='csv')

100%|██████████| 20/20 [00:00<00:00, 20.27it/s]

<pymovements.dataset.dataset.Dataset at 0x7f28d9aa9fd0>

Let’s confirm again by printing all the new files in this alternative directory:

alternative_dirpath = dataset.path / 'preprocessed_csv'
print(list(alternative_dirpath.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.csv')]

Loading

Now let’s imagine that this preprocessing and saving was done in another file and we only want to load the preprocessed data.

We simulate this by initializing a new dataset object. We don’t need to download any additional data.

events_dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')

The preprocessed data can now simply be loaded by setting preprocessed to True:

events_dataset.load(preprocessed=True)

events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:00<00:00, 158.22it/s]

shape: (5, 8)

time	stimuli_x	stimuli_y	pixel	position	velocity	text_id	page_id
f32	f32	f32	list[f32]	list[f32]	list[f32]	i64	i64
1.988145e6	-1.0	-1.0	[206.800003, 152.399994]	[-10.697598, -8.8524]	[null, null]	0	1
1.988146e6	-1.0	-1.0	[206.899994, 152.100006]	[-10.695184, -8.859678]	[null, null]	0	1
1.988147e6	-1.0	-1.0	[207.0, 151.800003]	[-10.692768, -8.866957]	[1.610438, -5.256017]	0	1
1.988148e6	-1.0	-1.0	[207.100006, 151.699997]	[-10.690351, -8.869382]	[0.40261, -4.447301]	0	1
1.988149e6	-1.0	-1.0	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402451, -3.234386]	0	1

By default, the preprocessed directory and the feather extension will be chosen.

In case of alternative directory names or other file formats you can use the following:

events_dataset.load(
    preprocessed=True,
    preprocessed_dirname='preprocessed_csv',
    extension='csv',
)
events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:02<00:00,  8.08it/s]

shape: (5, 8)

time	stimuli_x	stimuli_y	text_id	page_id	pixel	position	velocity
i64	f64	f64	i64	i64	list[f64]	list[f64]	list[f64]
1988145	-1.0	-1.0	0	1	[206.8, 152.4]	[-10.697598, -8.8524]	[null, null]
1988146	-1.0	-1.0	0	1	[206.9, 152.1]	[-10.695184, -8.859678]	[null, null]
1988147	-1.0	-1.0	0	1	[207.0, 151.8]	[-10.692768, -8.866957]	[1.6104381, -5.256017]
1988148	-1.0	-1.0	0	1	[207.1, 151.7]	[-10.690352, -8.869382]	[0.40261, -4.447301]
1988149	-1.0	-1.0	0	1	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402451, -3.234386]

What you have learned in this tutorial:

• saving your preprocesed data using Dataset.save_preprocessed()

• load your preprocesed data using Dataset.load(preprocessed=True)

• using custom directory names by specifying preprocessed_dirname

• using other file formats than the default feather format by specifying extension