Skip to content

Instantly share code, notes, and snippets.

@spartonia

spartonia/Blocks.ipynb

Created September 4, 2015 12:39
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save spartonia/03e059cd3b7d64ad1355 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save spartonia/03e059cd3b7d64ad1355 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Blocks.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"# Blocks & Fuel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Fuel"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Simplified overview of Fuel"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"<img src=\"https://fuel.readthedocs.org/en/latest/_images/graphviz-55b97fd87a48956a5fb72f98894ccccead1cde38.png\"/>"
],
"text/plain": [
"<IPython.core.display.Image object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from IPython.display import Image, display\n",
"img = Image(url='https://fuel.readthedocs.org/en/latest/_images/graphviz-55b97fd87a48956a5fb72f98894ccccead1cde38.png')\n",
"display(img) \n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Dataset \n",
"* an interface to the data we are trying to access "
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [],
"source": [
"from fuel.datasets import MNIST\n",
"mnist = MNIST(which_set=('train'))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Data sream/Data stream wrapper\n",
"* uses the interface of a dataset to e.g. iterate over the data"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Iteration scheme\n",
"* describes how we should proceed to iterate over the data"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Request iterator \n",
"* represents a single epoch of requests, as determined by the iteration scheme"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Data iterator\n",
"* uses a request iterator and returns data at each step "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from fuel.streams import DataStream # Data Stream\n",
"from fuel.transformers import Flatten # DS Wrapper (another DS)\n",
"from fuel.schemes import ShuffledScheme # Iteration Scheme \n",
"stream = Flatten( \n",
" DataStream.default_stream(\n",
" mnist,\n",
" iteration_scheme=ShuffledScheme(\n",
" mnist.num_examples, \n",
" 512\n",
" ),\n",
" ),\n",
" which_sources=('features',)\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Blocks \n",
"* Helps to build and manage neural network models using Theano.\n",
"* developed in parallel with Fuel, a dataset processing framework. "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Features "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Uses 'bricks' to build models, which are parameterized theano operations "
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from blocks.bricks import Tanh, Sigmoid, MLP\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Pattern matching to select variables and bricks in large models by role "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from blocks.graph import ComputationGraph\n",
"cg = ComputationGraph(cost)\n",
"from blocks.filter import VariableFilter\n",
"from blocks.roles import WEIGHT\n",
"print(VariableFilter(roles=[WEIGHT])(cg.variables))"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Algorithms to optimize your model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from blocks.algorithms import GradientDescent, Scale\n",
"algorithm=GradientDescent(\n",
" cost=cost,\n",
" parameters=[a],\n",
" step_rule=Scale(learning_rate=0.1)\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Saving and resuming of training"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from blocks.extensions.saveload import Checkpoint, LoadFromDump "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Monitoring and analyzing values during training progress"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from blocks.extensions.monitoring import TrainingDataMonitoring"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Application of graph transformations, such as dropout "
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from blocks.graph import ComputationGraph, apply_dropout"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"dropout_y_hat = apply_dropout(\n",
" cg, \n",
" list_of_dropped_out_variables, \n",
" include_prob=0.5, \n",
" scale=2.0\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Bricks "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* Blocks uses 'bricks' to build models\n",
"* Bricks are parameterized theano operations\n",
"* Defined by a set of **attributes** and a set of **parameters**\n",
" * **attributes**: specifying the attributes that define the Block e.g., the number of input and output units \n",
" * **parameters**: the parameters of the brick object that will vary during learning (e.g., the weights and the biases)."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from blocks.initialization import IsotropicGaussian, Constant \n",
"from blocks.bricks import Linear\n",
"linear = Linear(\n",
" input_dim=10,\n",
" output_dim=5, # <-- attributes; fixed \n",
" weights_init=IsotropicGaussian(),\n",
" biases_init=Constant(0.01) # <-- parameters; will vary during learnig \n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Brick life-cycle:"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
" * **configure**: set attributes and parameters"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"linear = Linear(\n",
" input_dim=10,\n",
" output_dim=5, \n",
" weights_init=IsotropicGaussian(),\n",
" biases_init=Constant(0.01) \n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* **Allocation(optional)**: allocate the Theano shared variables for the parameters"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* **Allpication**: linking the inputs and the outputs of the brick through its parameters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"y = linear.apply(x)"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* **Initialization**: set the numeric values to parameter\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"linear.parameters\n",
"linear.parameters[1].get_value() "
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"[W, b]\n",
"array([ nan, nan, nan, nan, nan]) "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
" * After Initialization:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [],
"source": [
"linear.initialize()\n",
"linear.parameters[1].get_value() "
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"array([ 0.01, 0.01, 0.01, 0.01, 0.01])"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Nested Bricks "
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* NNs have hierarchical structure \n",
" * Simple MLP: linear -> non-linear "
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"['linear_0', 'sigmoid_0', 'linear_1', 'sigmoid_1']"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# - Bricks can have children \n",
"# - parent configures the children \n",
"from blocks.bricks import MLP, Sigmoid\n",
"mlp = MLP(\n",
" activations=[\n",
" Sigmoid(name='sigmoid_0'),\n",
" Sigmoid(name='sigmoid_1')\n",
" ], \n",
" dims=[16, 8, 4],\n",
" weights_init=IsotropicGaussian(), biases_init=Constant(0.01)\n",
")\n",
"\n",
"[child.name for child in mlp.children]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"* MLP brick automatically constructed two child bricks to perform the linear transformations\n",
"* Parent will initialize children"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"mlp.initialize()\n",
"mlp.children[1].parameters[0].get_value() "
]
},
{
"cell_type": "raw",
"metadata": {},
"source": [
"array([[-0.38312393, -1.7718271 , 0.78074479, -0.74750996],\n",
" ...\n",
" [ 1.32390416, -0.56375355, -0.24268186, -2.06008577]])"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"# mnist classifiacation "
]
},
{
"cell_type": "raw",
"metadata": {
"collapsed": true
},
"source": [
"http://blocks.readthedocs.org/en/latest/tutorial.html"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment