Dataset Classes

class multimodal_compare.models.datasets.BaseDataset(pth, testpth, mod_type)

Bases: object

Abstract dataset class shared for all datasets

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess(output_data)

Postprocesses the output data according to modality type

Returns:

postprocessed data

Return type:

list

_postprocess_all2img(data)

Converts any kind of data to images to save traversal visualizations

Parameters:

data (torch.tensor) – input data

Returns:

processed images

Return type:

torch.tensor

_preprocess()

Preprocesses the loaded data according to modality type

Returns:

preprocessed data

Return type:

list

_preprocess_images(dimensions)

General function for loading images and preparing them as torch tensors

Parameters:

dimensions (list) – feature_dim for the image modality

Returns:

preprocessed data

Return type:

torch.tensor

_preprocess_text_onehot()

General function for loading text strings and preparing them as torch one-hot encodings

Returns:

torch with text encodings and masks

Return type:

torch.tensor

current_datatype()

Returns whther the current path to data points to test data or train data

eval_statistics_fn()

(optional) Returns a dataset-specific function that runs systematic evaluation

get_data()

Returns processed data

Returns:

processed data

Return type:

list

get_data_raw()

Loads raw data from path

Returns:

loaded raw data

Return type:

list

get_labels(split='train')

Returns labels for the given split: train or test

get_processed_recons(recons_raw)

Returns the postprocessed data that came from the decoders

Parameters:

recons_raw (torch.tensor) – tensor with output reconstructions

Returns:

postprocessed data as returned by the specific _postprocess function

Return type:

list

get_test_data()

Returns processed test data if available

Returns:

processed data

Return type:

list

labels()

Returns labels for the whole dataset

save_traversals(recons, path, num_dims)

Makes a grid of traversals and saves as image

Parameters:

• recons (torch.tensor) – data to save

• path (str) – path to save the traversal to

• num_dims (int) – number of latent dimensions

class multimodal_compare.models.datasets.CDSPRITESPLUS(pth, testpth, mod_type)

Bases: BaseDataset

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_images(data)

_postprocess_text(data)

_preprocess_images()

General function for loading images and preparing them as torch tensors

Parameters:

dimensions (list) – feature_dim for the image modality

Returns:

preprocessed data

Return type:

torch.tensor

_preprocess_text()

eval_statistics_fn()

(optional) Returns a dataset-specific function that runs systematic evaluation

feature_dims = {'image': [64, 64, 3], 'text': [45, 27, 1]}

labels()

Extract text labels based on the dataset level :return: list of labels as strings :rtype: list

save_recons(data, recons, path, mod_names)

set_vis_image_shape()

class multimodal_compare.models.datasets.CELEBA(pth, testpth, mod_type)

Bases: BaseDataset

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_all2img(data)

Converts any kind of data to images to save traversal visualizations

Parameters:

data (torch.tensor) – input data

Returns:

processed images

Return type:

torch.tensor

_postprocess_atts(data)

_postprocess_images(data)

_preprocess_atts()

_preprocess_images()

General function for loading images and preparing them as torch tensors

Parameters:

dimensions (list) – feature_dim for the image modality

Returns:

preprocessed data

Return type:

torch.tensor

feature_dims = {'atts': [4, 2], 'image': [64, 64, 3]}

save_recons(data, recons, path, mod_names)

save_traversals(recons, path, num_dims)

Makes a grid of traversals and saves as image

Parameters:

• recons (torch.tensor) – data to save

• path (str) – path to save the traversal to

• num_dims (int) – number of latent dimensions

class multimodal_compare.models.datasets.CUB(pth, testpth, mod_type)

Bases: BaseDataset

Dataset class for our processed version of Caltech-UCSD birds dataset. We use the original images and text represented as sequences of one-hot-encodings for each character (incl. spaces)

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_text(data)

_preprocess_images()

General function for loading images and preparing them as torch tensors

Parameters:

dimensions (list) – feature_dim for the image modality

Returns:

preprocessed data

Return type:

torch.tensor

_preprocess_text()

_preprocess_text_onehot()

General function for loading text strings and preparing them as torch one-hot encodings

Returns:

torch with text encodings and masks

Return type:

torch.tensor

feature_dims = {'image': [64, 64, 3], 'text': [246, 27, 1]}

labels()

No labels for T-SNAE available

save_recons(data, recons, path, mod_names)

class multimodal_compare.models.datasets.FASHIONMNIST(pth, testpth, mod_type)

Bases: BaseDataset

Dataset class for the FashionMNIST dataset

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_image(data)

_postprocess_label(data)

_process_image()

_process_label()

feature_dims = {'image': [28, 28, 1], 'label': [10]}

get_data_raw()

Loads raw data from path

Returns:

loaded raw data

Return type:

list

labels()

Returns labels for the whole dataset

save_recons(data, recons, path, mod_names)

class multimodal_compare.models.datasets.MNIST_SVHN(pth, testpth, mod_type)

Bases: BaseDataset

Dataset class for the MNIST-SVHN bimodal dataset (can be also used for unimodal training)

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_all2img(data)

Converts any kind of data to images to save traversal visualizations

Parameters:

data (torch.tensor) – input data

Returns:

processed images

Return type:

torch.tensor

_postprocess_mnist(data)

_postprocess_svhn(data)

_process_mnist()

_process_svhn()

check_indices_present()

feature_dims = {'mnist': [28, 28, 1], 'svhn': [32, 32, 3]}

labels()

Returns labels for the whole dataset

save_recons(data, recons, path, mod_names)

class multimodal_compare.models.datasets.POLYMNIST(pth, testpth, mod_type)

Bases: BaseDataset

Dataset class for the POLYMNIST dataset

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_mnist(data)

_process_mnist()

feature_dims = {'m0': [28, 28, 3], 'm1': [28, 28, 3], 'm2': [28, 28, 3], 'm3': [28, 28, 3], 'm4': [28, 28, 3]}

save_recons(data, recons, path, mod_names)

save_traversals(recons, path, num_dims)

Makes a grid of traversals and saves as image

Parameters:

• recons (torch.tensor) – data to save

• path (str) – path to save the traversal to

• num_dims (int) – number of latent dimensions

class multimodal_compare.models.datasets.SPRITES(pth, testpth, mod_type)

Bases: BaseDataset

_mod_specific_loaders()

Assigns the preprocessing function based on the mod_type

_mod_specific_savers()

Assigns the postprocessing function based on the mod_type

_postprocess_actions(data)

_postprocess_all2img(data)

Converts any kind of data to images to save traversal visualizations

Parameters:

data (torch.tensor) – input data

Returns:

processed images

Return type:

torch.tensor

_postprocess_attributes(data)

_postprocess_frames(data)

eval_statistics_fn()

(optional) Returns a dataset-specific function that runs systematic evaluation

feature_dims = {'actions': [9], 'attributes': [4, 6], 'frames': [8, 64, 64, 3]}

get_actions()

get_attributes()

get_frames()

iter_over_inputs(outs, data, mod_names, f=0)

labels()

Returns labels for the whole dataset

make_masks(shape)

save_recons(data, recons, path, mod_names)

save_traversals(recons, path)

Makes a grid of traversals and saves as animated gif image

Parameters:

• recons (torch.tensor) – data to save

• path (str) – path to save the traversal to