tonic.prototype.datasets
#
Subpackages#
Submodules#
Package Contents#
Classes#
`Gen1 Automotive Detection Dataset <https://www.prophesee.ai/2020/01/24/prophesee- |
|
`Gen4 Automotive Detection <https://www.prophesee.ai/2020/11/24/automotive-megapixel-event- |
|
`Gen4 Automotive Detection <https://www.prophesee.ai/2020/11/24/automotive-megapixel-event- |
|
Iterable-style DataPipe. |
|
- class tonic.prototype.datasets.NCARS(root: os.PathLike, train: Optional[bool] = True, skip_sha256_check: Optional[bool] = True)[source]#
Bases:
tonic.prototype.datasets.utils._dataset.Dataset
This datasets needs ‘expelliarmus’ installed on the system. Events have “txyp” ordering.
@article{Sironi_2018_CVPR, author = {Sironi, Amos and Brambilla, Manuele and Bourdis, Nicolas and Lagorce, Xavier and Benosman, Ryad}, title = {HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification}, booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2018} }
- Parameters:
root (string) – Location to decompressed archive.
train (bool) – If True, uses training subset, otherwise testing subset.
skip_sha256_check (Optional[bool]) –
- sensor_size#
- class tonic.prototype.datasets.NMNIST(root: os.PathLike, train: Optional[bool] = True, first_saccade_only: Optional[bool] = False, keep_compressed: Optional[bool] = False)[source]#
Bases:
tonic.prototype.datasets.utils._dataset.Dataset
Events have (xytp) ordering.
@article{orchard2015converting, title={Converting static image datasets to spiking neuromorphic datasets using saccades}, author={Orchard, Garrick and Jayawant, Ajinkya and Cohen, Gregory K and Thakor, Nitish}, journal={Frontiers in neuroscience}, volume={9}, pages={437}, year={2015}, publisher={Frontiers} }
- Parameters:
root (string) – Location to save files to on disk.
train (bool) – If True, uses training subset, otherwise testing subset.
first_saccade_only (bool) – If True, only work with events of the first of three saccades. Results in about a third of the events overall.
keep_compressed (Optional[bool]) –
- sensor_size = (34, 34, 2)#
- class tonic.prototype.datasets.Gen1AutomotiveDetection(root: os.PathLike, split: str = 'train', shuffle: bool = False)[source]#
Bases:
AutomotiveDetectionBaseClass
Gen1 Automotive Detection Dataset
To download the data, you’ll need to agree to Prophesee’s Terms and Conditions.
Then, the steps to acquire the data can be as follows:
Download the torrent file for the dataset:
wget https://dataset.prophesee.ai/index.php/s/uE0QGLaFAEQnPwy/download\?path\=%2F\&files\=ATIS%20Automotive%20Detection%20Dataset.torrent -O Gen1Prophesee.torrent
Download the data using peer-to-peer connections. On Linux this can be done using aria2c on the command line:
aria2c Gen1Prophesee.torrent
This will download several 7z archives for training and testing. We’ll need to unpack them manually by looping over the 7z files and feeding them to 7z:
sudo apt-get install p7zip-full for i in *.7z; do 7z x $i; done
- Parameters:
root (string) – Location to decompressed archive.
split (str) – Can be ‘train’ (default), ‘valid’ or ‘test’.
shuffle (bool) – If True, the dataset will be shuffled randomly.
- sensor_size#
- class_map#
- class tonic.prototype.datasets.Gen4Automotive(root: os.PathLike, split: str = 'train', shuffle: bool = False)[source]#
Bases:
AutomotiveDetectionBaseClass
This datasets needs ‘expelliarmus’ installed on the system. Events have “txyp” ordering.:
@article{de2020large, title={A large scale event-based detection dataset for automotive}, author={De Tournemire, Pierre and Nitti, Davide and Perot, Etienne and Migliore, Davide and Sironi, Amos}, journal={arXiv preprint arXiv:2001.08499}, year={2020} }
To download the data, you’ll need to agree to Prophesee’s Terms and Conditions.
Then, the steps to acquire the data can be as follows:
Download the torrent file for the dataset:
wget https://dataset.prophesee.ai/index.php/s/8HY0Bv4mOU4RzBm/download?path=%2F&files=Large_Automotive_Detection_Dataset.torrent -O Gen4Prophesee.torrent
Download the data using peer-to-peer connections. On Linux this can be done using aria2c on the command line:
aria2c Gen4Prophesee.torrent
This will download several 7z archives for training, validation and testing. We’ll need to unpack them manually by looping over the 7z files and feeding them to 7z:
sudo apt-get install p7zip-full for i in *.7z; do 7z x $i; done
- Parameters:
root (string) – Location to decompressed archive.
split (str) – Can be ‘train’ (default), ‘valid’ or ‘test’.
shuffle (bool) – If True, the dataset will be shuffled randomly.
- sensor_size#
- class_map#
- class tonic.prototype.datasets.Gen4AutomotiveDetectionMini(root: os.PathLike, split: str = 'train', shuffle: bool = False, skip_sha256_check: Optional[bool] = True)[source]#
Bases:
AutomotiveDetectionBaseClass
This datasets needs ‘expelliarmus’ installed on the system. Events have “txyp” ordering.:
@article{de2020large, title={A large scale event-based detection dataset for automotive}, author={De Tournemire, Pierre and Nitti, Davide and Perot, Etienne and Migliore, Davide and Sironi, Amos}, journal={arXiv preprint arXiv:2001.08499}, year={2020} }
Note
The hosting server is very flaky and often interrupts the download before it is completed. If you end up with anything smaller than 23GB on disk, delete and try again.
- Parameters:
root (string) – Location to decompressed archive.
split (str) – Can be ‘train’ (default), ‘valid’ or ‘test’.
shuffle (bool) – If True, the dataset will be shuffled randomly.
skip_sha256_check (Optional[bool]) –
- sensor_size#
- class_map#
- class tonic.prototype.datasets.Gen4Downsampled(root: os.PathLike, split: str = 'train', skip_sha256_check: Optional[bool] = True, shuffle=False)[source]#
Bases:
tonic.prototype.datasets.utils._dataset.Dataset
Iterable-style DataPipe.
All DataPipes that represent an iterable of data samples should subclass this. This style of DataPipes is particularly useful when data come from a stream, or when the number of samples is too large to fit them all in memory.
IterDataPipe
is lazily initialized and its elements are computed only whennext()
is called on the iterator of anIterDataPipe
.All subclasses should overwrite
__iter__()
, which would return an iterator of samples in this DataPipe. Calling__iter__
of anIterDataPipe
automatically invokes its methodreset()
, which by default performs no operation. When writing a customIterDataPipe
, users should overridereset()
if necessary. The common usages include resetting buffers, pointers, and various state variables within the customIterDataPipe
.Note
Only one iterator can be valid for each
IterDataPipe
at a time, and the creation a second iterator will invalidate the first one. This constraint is necessary because someIterDataPipe
have internal buffers, whose states can become invalid if there are multiple iterators. The code example below presents details on how this constraint looks in practice. If you have any feedback related to this constraint, please see GitHub IterDataPipe Single Iterator Issue.These DataPipes can be invoked in two ways, using the class constructor or applying their functional form onto an existing
IterDataPipe
(recommended, available to most but not all DataPipes). You can chain multiple IterDataPipe together to form a pipeline that will perform multiple operations in succession.Note
When a subclass is used with
DataLoader
, each item in the DataPipe will be yielded from theDataLoader
iterator. Whennum_workers > 0
, each worker process will have a different copy of the DataPipe object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers.get_worker_info()
, when called in a worker process, returns information about the worker. It can be used in either the dataset’s__iter__()
method or theDataLoader
‘sworker_init_fn
option to modify each copy’s behavior.Examples
- General Usage:
>>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper, Mapper >>> dp = IterableWrapper(range(10)) >>> map_dp_1 = Mapper(dp, lambda x: x + 1) # Using class constructor >>> map_dp_2 = dp.map(lambda x: x + 1) # Using functional form (recommended) >>> list(map_dp_1) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> list(map_dp_2) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] >>> filter_dp = map_dp_1.filter(lambda x: x % 2 == 0) >>> list(filter_dp) [2, 4, 6, 8, 10]
- Single Iterator Constraint Example:
>>> from torchdata.datapipes.iter import IterableWrapper, Mapper >>> source_dp = IterableWrapper(range(10)) >>> it1 = iter(source_dp) >>> list(it1) [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> it1 = iter(source_dp) >>> it2 = iter(source_dp) # The creation of a new iterator invalidates `it1` >>> next(it2) 0 >>> next(it1) # Further usage of `it1` will raise a `RunTimeError`
- Parameters:
root (os.PathLike) –
split (str) –
skip_sha256_check (Optional[bool]) –
- class tonic.prototype.datasets.STMNIST(root: os.PathLike, keep_compressed: Optional[bool] = False, skip_sha256_check: Optional[bool] = True, shuffle: bool = False)[source]#
Bases:
tonic.prototype.datasets.utils._dataset.Dataset
Neuromorphic Spiking Tactile MNIST (ST-MNIST) dataset, which comprises handwritten digits obtained by human participants writing on a neuromorphic tactile sensor array. The original paper can be found at https://arxiv.org/abs/2005.04319. Data is provided with the MAT format. Download of the compressed dataset has to be done by the user by accessing https://scho larbank.nus.edu.sg/bitstream/10635/168106/2/STMNIST%20dataset%20NUS%20Tee%20Research%20Group.zi p, where a form has to be completed. Then, the path to the ZIP archive has to be provided to the STMNIST constructor root argument.
Events have (xytp) ordering. :param root: Parent folder of ‘STMNIST/STMNIST dataset NUS Tee Research Group.zip’. The STMNIST folder is related to the Tonic class name and is needed currently. :type root: string :param shuffle: Whether to shuffle the dataset. More efficient if done based on file paths. :type shuffle: bool
- Returns:
Torchdata data pipe that yields a tuple of events (or transformed events) and target.
- Return type:
dp (IterDataPipe[Sample])
- Parameters:
root (os.PathLike) –
keep_compressed (Optional[bool]) –
skip_sha256_check (Optional[bool]) –
shuffle (bool) –
- sensor_size#