This repository contains code for VPRTempo, a spiking neural network that uses temporally encoding to perform visual place recognition tasks. The network is based off of BLiTNet and adapted to the VPRSNN framework.
VPRTempo is built on a torch.nn framework and employs custom learning rules based on the temporal codes of spikes in order to train layer weights.
In this repository, we provide two networks:
VPRTempo: Our base network architecture to perform visual place recognition (fp32)VPRTempoQuant: A modified base network with Quantization Aware Training (QAT) enabled (int8)
To use VPRTempo, please follow the instructions below for installation and usage.
- Provided support for MPS Apple Silicon 🍏
- Minor bug fixes in evaluation metrics 🐛
- New auto-downloader for pre-trained models and Nordland image subsets for easier trialling 📡
This repository is licensed under the MIT License
If you use our code, please cite our IEEE ICRA paper:
@inproceedings{hines2024vprtempo,
title={VPRTempo: A Fast Temporally Encoded Spiking Neural Network for Visual Place Recognition},
author={Adam D. Hines and Peter G. Stratton and Michael Milford and Tobias Fischer},
year={2024},
pages={10200-10207},
booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)}
}
VPRTempo uses PyTorch with the capability for CUDA acceleration. Please use one of the following options below to install the required dependencies, and if desired follow the instructions to install CUDA for your hardware and operating system.
Download the Github repository.
git clone https://github.com/QVPR/VPRTempo.git
cd ~/VPRTempoOnce downloaded, please install the required dependencies to run the network through one of the following options:
Dependencies for VPRTempo can downloaded from our PyPi package. Please ensure python --version is >=3.6 and <3.13.
pip install vprtempoIf you wish to enable CUDA, please follow the instructions on the PyTorch - Get Started page to install the required software versions for your hardware and operating system.
Dependencies can be installed either through our provided requirements.txt files. Please ensure python --version is >=3.6 and <3.13.
pip install -r requirements.txtAs above, if you wish to install CUDA please visit PyTorch - Get Started.
❗ Recommended: Use Mambaforge instead of conda.
Requirements for VPRTempo may be installed using our conda-forge package.
# Linux/OS X
conda create -n vprtempo -c conda-forge vprtempo
# Linux CUDA enabled
conda create -n vprtempo -c conda-forge -c pytorch -c nvidia vprtempo pytorch-cuda cudatoolkit
# Windows
conda create -n vprtempo -c pytorch python pytorch torchvision torchaudio cpuonly prettytable tqdm numpy pandas matplotlib requests
# Windows CUDA enabled
conda create -n vprtempo -c pytorch -c nvidia python torchvision torchaudio pytorch-cuda=11.7 cudatoolkit prettytable tqdm numpy pandas matplotlib requests
VPRTempo was developed to be simple to train and test a variety of datasets. Please see the information below about running a test with the Nordland and Oxford RobotCar datasets and how to organize custom datasets.
Please note that while we trained 3,300 places for Nordland and 450 for OxfordRobot car we only evaluated 2,700 and 360 places, respectively, ignoring the first 20% (see Sect.4B Datasets).
VPRTempo was developed and tested using the Nordland traversal dataset. To download the full dataset, please visit this repository.
To simplify first usage, we have set the defaults in main.py to train and test on a small subset of Nordland data with pre-trained models, automatically downloaded on first usage (see Pre-trained models, below).
For convenience, all data should be organised in the ./dataset folder in the following way in order to train the network on multiple traversals of the same location.
--dataset
|--summer
|--spring
|--fall
|--winter
To replicate the results in our paper, please run the following.
# Train the Nordland model
python main.py --train_new_model --database_places 3300 --database_dirs spring,fall --skip 0 --max_module 1100 --dataset nordland --dims 28,28 --patches 7 --filter 7
# Test the Nordland model
python main.py --database_places 3300 --database_dirs spring,fall --skip 7999 --dataset nordland --dims 28,28 --patches 7 --filter 7 --query_dir summer --query_places 2700 --sim_mat --max_module 1100In order to train and test on Oxford RobotCar, you will need to register an account to get access to download the dataset before proceeding. We use 3 traverses (sun 2015-08-12-15-04-18, dusk 2014-11-21-16-07-03, and rain 2015-10-29-12-18-17) recorded from the stero_left camera, which can be downloaded using the RobotCarDataset-Scraper in the following way:
# Copy the orc_lists.txt from this repo into the RobotCarDataset-Scraper repo
python scrape_mrgdatashare.py --choice_sensors stereo_left --choice_runs_file orc_list.txt --downloads_dir ~/VPRTempo/vprtempo/dataset/orc --datasets_file datasets.csv --username USERNAME --password PASSWORDNext, use our helper script process_orc.py to demosaic and denoise the downloaded images. You'll need to download the robotcar-dataset-sdk repository and place the process_orc.py file into the python directory of the repository. Modify the base_path variable of process_orc.py to the location of your downloaded images.
# Navigate to python directory, ensure process_orc.py and orc.csv are in this directory
cd ~/robotcar-dataset-sdk/python
# Run the demosaic and denoise
python process_orc.pyTo replicate the results in our paper, please run the following.
# Train the ORC model
python main.py --train_new_model --database_places 450 --database_dirs sun,rain --skip 0 --max_module 450 --dataset orc --dims 28,28 --patches 7 --filter 7
# Test the ORC model
python main.py --database_places 450 --database_dirs sun,rain --skip 630 --dataset orc --dims 28,28 --patches 7 --filter 7 --query_dir dusk --query_places 360 --sim_mat --max_module 450To define your own custom dataset to use with VPRTempo, you will need to follow the conventions for PyTorch Datasets & Dataloaders. We have included a convenient script ./vprtempo/src/create_data_csv.py which will generate a .csv file that can be used to load custom datasets for training and inferencing. Simply modify the dataset_name variable to the folder containing your images.
To train a new model with a custom dataset, you can do the following.
# Train new model - requires .csv file generated by create_data_csv.py
python main.py --train_new_model --dataset <your custom database name> --database_dirs <your custom database name>
# Test new model
python main.py --database_dirs <your custom database name> --dataset <your custom query name> --query_dir <your custom query name>If image names are equivelant between database and query directories, you can simply use the one .csv file for both as in the example of Nordland and Oxford RobotCar.
Running VPRTempo and VPRTempoQuant is handlded by main.py, which can be operated either through the command terminal or directly running the script. See below for more details.
- Training and testing data is organized as above (see Datasets on how to set up the Nordland dataset)
- The VPRTempo dependencies have been installed and/or the conda environment has been activated
We provide two pretrained models, for VPRTempo and VPRTempoQuant, that have learned a 500 place sequence from two Nordland traversals (Spring & Fall) which can be used to inference with Summer or Winter. To get the pre-trained models and Nordland images, simply run either inference network (below) which will automatically download and sort the models and images into the VPRTempo folder.
The main.py script handles running the inference network, there are two options:
python main.py
To run the quantized network, parse the --quantize argument. (Please note, MPS is not currently supported with PyTorch QAT)
python main.py --quantize
If you do not wish to use the pretrained models or you would like to train your own, we can parse the --train_new_model flag to main.py. Note, if a pretrained model already exists you will be prompted if you would like to retrain it.
# For VPRTempo
python main.py --train_new_model
# For VPRTempoQuant
python main.py --train_new_model --quantize
We provide a series of Jupyter Notebook tutorials that go through the basic operations and logic for VPRTempo and VPRTempoQuant.
If you encounter problems whilst running the code or if you have a suggestion for a feature or improvement, please report it as an issue.