Saving and Loading Preprocessed Data#

What you will learn in this tutorial:#

how to save your preprocessed data
how to load your preprocessed data

Preparations#

We import pymovements as the alias pm for convenience.

[1]:

import pymovements as pm

/home/docs/checkouts/readthedocs.org/user_builds/pymovements/envs/v0.16.0/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Let’s start by downloading our ToyDataset and loading in its data:

[2]:

dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')
dataset.download()
dataset.load()

Using already downloaded and verified file: data/ToyDataset/downloads/pymovements-toy-dataset.zip
Extracting pymovements-toy-dataset.zip to data/ToyDataset/raw

100%|██████████| 20/20 [00:00<00:00, 46.87it/s]

[2]:

<pymovements.dataset.dataset.Dataset at 0x7fa77e436f70>

Now let’s load in the data and do some preprocessing:

[3]:

dataset.pix2deg()
dataset.pos2vel()

dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:00<00:00, 23.05it/s]
100%|██████████| 20/20 [00:00<00:00, 49.20it/s]

[3]:

shape: (5, 8)

text_id	page_id	time	stimuli_x	stimuli_y	pixel	position	velocity
i64	i64	f64	f64	f64	list[f64]	list[f64]	list[f64]
0	1	1.988145e6	-1.0	-1.0	[206.8, 152.4]	[-10.697598, -8.852399]	[null, null]
0	1	1.988146e6	-1.0	-1.0	[206.9, 152.1]	[-10.695183, -8.859678]	[null, null]
0	1	1.988147e6	-1.0	-1.0	[207.0, 151.8]	[-10.692768, -8.866956]	[1.610194, -5.256267]
0	1	1.988148e6	-1.0	-1.0	[207.1, 151.7]	[-10.690352, -8.869381]	[0.402548, -4.447465]
0	1	1.988149e6	-1.0	-1.0	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402561, -3.234462]

We have now added some additional columns for degrees in visual angle and velocity.

Saving#

Saving your preprocessed data is as simple as:

[4]:

dataset.save_preprocessed()

100%|██████████| 20/20 [00:00<00:00, 332.47it/s]

[4]:

<pymovements.dataset.dataset.Dataset at 0x7fa77e436f70>

All of the preprocessed data is saved into this directory:

[5]:

dataset.paths.preprocessed

[5]:

PosixPath('data/ToyDataset/preprocessed')

Let’s confirm it by printing all the new files in this directory:

[6]:

print(list(dataset.paths.preprocessed.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.feather'), PosixPath('data/ToyDataset/preprocessed/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.feather')]

All of the files have been saved into the Dataset.paths.preprocessed as feather files.

If we want to save the data into an alternative directory and also use a different file format like csv we can use the following:

[7]:

dataset.save_preprocessed(preprocessed_dirname='preprocessed_csv', extension='csv')

100%|██████████| 20/20 [00:00<00:00, 42.95it/s]

[7]:

<pymovements.dataset.dataset.Dataset at 0x7fa77e436f70>

Let’s confirm again by printing all the new files in this alternative directory:

[8]:

alternative_dirpath = dataset.path / 'preprocessed_csv'
print(list(alternative_dirpath.glob('*/*/*')))

[PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_5.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_2.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_0_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_3.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_1_1.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_2_4.csv'), PosixPath('data/ToyDataset/preprocessed_csv/aeye-lab-pymovements-toy-dataset-6cb5d66/data/trial_3_1.csv')]

Loading#

Now let’s imagine that this preprocessing and saving was done in another file and we only want to load the preprocessed data.

We simulate this by initializing a new dataset object. We don’t need to download any additional data.

[9]:

events_dataset = pm.Dataset('ToyDataset', path='data/ToyDataset')

The preprocessed data can now simply be loaded by setting preprocessed to True:

[10]:

events_dataset.load(preprocessed=True)

events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:00<00:00, 1470.47it/s]

[10]:

shape: (5, 8)

text_id	page_id	time	stimuli_x	stimuli_y	pixel	position	velocity
i64	i64	f64	f64	f64	list[f64]	list[f64]	list[f64]
0	1	1.988145e6	-1.0	-1.0	[206.8, 152.4]	[-10.697598, -8.852399]	[null, null]
0	1	1.988146e6	-1.0	-1.0	[206.9, 152.1]	[-10.695183, -8.859678]	[null, null]
0	1	1.988147e6	-1.0	-1.0	[207.0, 151.8]	[-10.692768, -8.866956]	[1.610194, -5.256267]
0	1	1.988148e6	-1.0	-1.0	[207.1, 151.7]	[-10.690352, -8.869381]	[0.402548, -4.447465]
0	1	1.988149e6	-1.0	-1.0	[207.0, 151.5]	[-10.692768, -8.874233]	[0.402561, -3.234462]

By default, the preprocessed directory and the feather extension will be chosen.

In case of alternative directory names or other file formats you can use the following:

[11]:

events_dataset.load(
    preprocessed=True,
    preprocessed_dirname='preprocessed_csv',
    extension='csv',
)
events_dataset.gaze[0].frame.head()

100%|██████████| 20/20 [00:01<00:00, 17.29it/s]

[11]:

shape: (5, 8)

text_id	page_id	time	stimuli_x	stimuli_y	pixel	position	velocity
i64	i64	f64	f64	f64	list[f64]	list[f64]	list[f64]
0	1	1.988145e6	-1.0	-1.0	[152.4, 206.8]	[-8.852399, -10.697598]	[null, null]
0	1	1.988146e6	-1.0	-1.0	[152.1, 206.9]	[-8.859678, -10.695183]	[null, null]
0	1	1.988147e6	-1.0	-1.0	[151.8, 207.0]	[-8.866956, -10.692768]	[-5.256267, 1.610194]
0	1	1.988148e6	-1.0	-1.0	[151.7, 207.1]	[-8.869381, -10.690352]	[-4.447465, 0.402548]
0	1	1.988149e6	-1.0	-1.0	[151.5, 207.0]	[-8.874233, -10.692768]	[-3.234462, 0.402561]

What you have learned in this tutorial:#

saving your preprocesed data using Dataset.save_preprocessed()
load your preprocesed data using Dataset.load(preprocessed=True)
using custom directory names by specifying preprocessed_dirname
using other file formats than the default feather format by specifying extension