Saving and Loading Preprocessed Data#

What you will learn in this tutorial:#

how to save your preprocessed data
how to load your preprocessed data

Preparations#

We import pymovements as the alias pm for convenience.

[1]:

import pymovements as pm

/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/v0.17.2/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Let’s start by downloading our ToyDataset and loading in its data:

[2]:

dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
dataset.download()
dataset.load()

Using already downloaded and verified file: data/ToyDataset/downloads/pymovements-toy-dataset.zip
Extracting pymovements-toy-dataset.zip to data/ToyDataset/raw

100%|██████████| 20/20 [00:00<00:00, 42.12it/s]

[2]:

<pymovements.dataset.dataset.Dataset at 0x7f13df141a90>

Now let’s load in the data and do some preprocessing:

[3]:

dataset.pix2deg()
dataset.pos2vel()

dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:01<00:00, 19.43it/s]
100%|██████████| 20/20 [00:00<00:00, 34.60it/s]

[3]:

shape: (5, 8)

text_id	page_id	time	stimuli_x	stimuli_y	pixel	position	velocity
i64	i64	f32	f32	f32	list[f32]	list[f32]	list[f32]
0	1	1.988145e6	-1.0	-1.0	[206.800003, 152.399994]	[-10.697598, -8.8524]	[null, null]
0	1	1.988146e6	-1.0	-1.0	[206.899994, 152.100006]	[-10.695184, -8.859678]	[null, null]
0	1	1.988147e6	-1.0	-1.0	[207.0, 151.800003]	[-10.692768, -8.866957]	[1.610438, -5.256017]
0	1	1.988148e6	-1.0	-1.0	[207.100006, 151.699997]	[-10.690351, -8.869382]	[0.40261, -4.447301]
0	1	1.988149e6	-1.0	-1.0	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402451, -3.234386]

We have now added some additional columns for degrees in visual angle and velocity.

Saving#

Saving your preprocessed data is as simple as:

[4]:

dataset.save_preprocessed()

100%|██████████| 20/20 [00:00<00:00, 315.95it/s]

[4]:

<pymovements.dataset.dataset.Dataset at 0x7f13df141a90>

All of the preprocessed data is saved into this directory:

[5]:

dataset.paths.preprocessed

[5]:

PosixPath('data/ToyDataset/preprocessed')

Let’s confirm it by printing all the new files in this directory:

[6]:

print(list(dataset.paths.preprocessed.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.feather')]

All of the files have been saved into the Dataset.paths.preprocessed as feather files.

If we want to save the data into an alternative directory and also use a different file format like csv we can use the following:

[7]:

dataset.save_preprocessed(preprocessed_dirname='preprocessed_csv', extension='csv')

100%|██████████| 20/20 [00:00<00:00, 38.64it/s]

[7]:

<pymovements.dataset.dataset.Dataset at 0x7f13df141a90>

Let’s confirm again by printing all the new files in this alternative directory:

[8]:

alternative_dirpath = dataset.path / 'preprocessed_csv'
print(list(alternative_dirpath.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.csv')]

Loading#

Now let’s imagine that this preprocessing and saving was done in another file and we only want to load the preprocessed data.

We simulate this by initializing a new dataset object. We don’t need to download any additional data.

[9]:

events_dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')

The preprocessed data can now simply be loaded by setting preprocessed to True:

[10]:

events_dataset.load(preprocessed=True)

events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:00<00:00, 181.30it/s]

[10]:

shape: (5, 8)

text_id	page_id	time	stimuli_x	stimuli_y	pixel	position	velocity
i64	i64	f32	f32	f32	list[f32]	list[f32]	list[f32]
0	1	1.988145e6	-1.0	-1.0	[206.800003, 152.399994]	[-10.697598, -8.8524]	[null, null]
0	1	1.988146e6	-1.0	-1.0	[206.899994, 152.100006]	[-10.695184, -8.859678]	[null, null]
0	1	1.988147e6	-1.0	-1.0	[207.0, 151.800003]	[-10.692768, -8.866957]	[1.610438, -5.256017]
0	1	1.988148e6	-1.0	-1.0	[207.100006, 151.699997]	[-10.690351, -8.869382]	[0.40261, -4.447301]
0	1	1.988149e6	-1.0	-1.0	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402451, -3.234386]

By default, the preprocessed directory and the feather extension will be chosen.

In case of alternative directory names or other file formats you can use the following:

[11]:

events_dataset.load(
    preprocessed=True,
    preprocessed_dirname='preprocessed_csv',
    extension='csv',
)
events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:00<00:00, 90.15it/s]

[11]:

shape: (5, 11)

text_id	page_id	time	stimuli_x	stimuli_y	pixel_x	pixel_y	position_x	position_y	velocity_x	velocity_y
i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
0	1	1.988145e6	-1.0	-1.0	206.8	152.4	-10.697598	-8.8524	null	null
0	1	1.988146e6	-1.0	-1.0	206.9	152.1	-10.695184	-8.859678	null	null
0	1	1.988147e6	-1.0	-1.0	207.0	151.8	-10.692768	-8.866957	1.6104381	-5.256017
0	1	1.988148e6	-1.0	-1.0	207.1	151.7	-10.690352	-8.869382	0.40261	-4.447301
0	1	1.988149e6	-1.0	-1.0	207.0	151.5	-10.692768	-8.874233	0.402451	-3.234386

What you have learned in this tutorial:#

saving your preprocesed data using Dataset.save_preprocessed()
load your preprocesed data using Dataset.load(preprocessed=True)
using custom directory names by specifying preprocessed_dirname
using other file formats than the default feather format by specifying extension