As an introduction into Luigi, I am following this tutorial with some modifications, e.g. installation using conda.
The problems and solutions described in the examples below have led to the development of sciluigi,
kevinzakka
/ data_loader.py
Last active
August 6, 2025 12:37
Train, Validation and Test Split for torchvision Datasets
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Create train, valid, test iterators for CIFAR-10 [1]. | |
| Easily extended to MNIST, CIFAR-100 and Imagenet. | |
| [1]: https://discuss.pytorch.org/t/feedback-on-pytorch-for-kaggle-competitions/2252/4 | |
| """ | |
| import torch | |
| import numpy as np |
tomsing1
/ luigi_first_steps.md
Last active
April 11, 2025 18:01
First steps with the Luigi workflow manager
mohanraj-r
/ scp-speed-test.sh
Last active
August 10, 2024 13:59
[speed test] Test ssh connection speed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # scp-speed-test.sh | |
| # Author: Alec Jacobson alecjacobsonATgmailDOTcom | |
| # http://www.alecjacobson.com/weblog/?p=635 | |
| # | |
| # Test ssh connection speed by uploading and then downloading a 10000kB test | |
| # file (optionally user-specified size) | |
| # | |
| # Usage: | |
| # ./scp-speed-test.sh user@hostname [test file size in kBs] |
goulu
/ points_on_sphere.py
Created
April 3, 2014 06:42
Generate N evenly distributed points on the unit sphere
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def points_on_sphere(N): | |
| """ Generate N evenly distributed points on the unit sphere centered at | |
| the origin. Uses the 'Golden Spiral'. | |
| Code by Chris Colbert from the numpy-discussion list. | |
| """ | |
| import numpy as np | |
| phi = (1 + np.sqrt(5)) / 2 # the golden ratio | |
| long_incr = 2*np.pi / phi # how much to increment the longitude | |
| dz = 2.0 / float(N) # a unit sphere has diameter 2 |