Skip to content

Instantly share code, notes, and snippets.

@viniciusmss

viniciusmss/Exploration.ipynb

Last active February 9, 2020 15:02
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save viniciusmss/8664939e32384ee7fa8900f3adffd1dc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save viniciusmss/8664939e32384ee7fa8900f3adffd1dc to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
Exploration.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Genetic Matching for Robustness and Model Independence"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"__Tasks__\n",
"\n",
"[ ] Get the nsw dataset\n",
"\n",
"[ ] # SECOND OPTION: Maximize gamma associated with losing statistical significance instead of the max pvalue of gamma = 1.5"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prelude\n",
"\n",
"Let's get a taste of the libraries we will be using."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:34.964151Z",
"start_time": "2020-02-09T14:59:34.576Z"
}
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Loading required package: Matching\n",
"Loading required package: MASS\n",
"## \n",
"## Matching (Version 4.9-3, Build Date: 2018-05-03)\n",
"## See http://sekhon.berkeley.edu/matching for additional documentation.\n",
"## Please cite software as:\n",
"## Jasjeet S. Sekhon. 2011. ``Multivariate and Propensity Score Matching\n",
"## Software with Automated Balance Optimization: The Matching package for R.''\n",
"## Journal of Statistical Software, 42(7): 1-52. \n",
"##\n",
"\n",
"## rgenoud (Version 5.8-2.0, Build Date: 2018-04-03)\n",
"## See http://sekhon.berkeley.edu/rgenoud for additional documentation.\n",
"## Please cite software as:\n",
"## Walter Mebane, Jr. and Jasjeet S. Sekhon. 2011.\n",
"## ``Genetic Optimization Using Derivatives: The rgenoud package for R.''\n",
"## Journal of Statistical Software, 42(11): 1-26. \n",
"##\n",
"\n"
]
}
],
"source": [
"# Load imports\n",
"library(rbounds)\n",
"library(rgenoud)\n",
"\n",
"# Load data\n",
"data(lalonde)\n",
"attach(lalonde)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:35.007036Z",
"start_time": "2020-02-09T14:59:34.579Z"
}
},
"outputs": [],
"source": [
"# Create data objects\n",
"Y <- re78\n",
"Tr <- treat\n",
"X <- cbind(age, educ, black, hisp, married, nodegr, re74, re75, u74, u75)\n",
"BalanceMat <- cbind(age, I(age^2), educ, I(educ^2), black,\n",
" hisp, married, nodegr, re74 , I(re74^2), re75, I(re75^2),\n",
" u74, u75, I(re74*re75), I(age*nodegr), I(educ*re74), I(educ*75))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:45.456108Z",
"start_time": "2020-02-09T14:59:34.584Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Sun Feb 09 22:59:35 2020\n",
"Domains:\n",
" 0.000000e+00 <= X1 <= 1.000000e+03 \n",
" 0.000000e+00 <= X2 <= 1.000000e+03 \n",
" 0.000000e+00 <= X3 <= 1.000000e+03 \n",
" 0.000000e+00 <= X4 <= 1.000000e+03 \n",
" 0.000000e+00 <= X5 <= 1.000000e+03 \n",
" 0.000000e+00 <= X6 <= 1.000000e+03 \n",
" 0.000000e+00 <= X7 <= 1.000000e+03 \n",
" 0.000000e+00 <= X8 <= 1.000000e+03 \n",
" 0.000000e+00 <= X9 <= 1.000000e+03 \n",
" 0.000000e+00 <= X10 <= 1.000000e+03 \n",
"\n",
"Data Type: Floating Point\n",
"Operators (code number, name, population) \n",
"\t(1) Cloning........................... \t7\n",
"\t(2) Uniform Mutation.................. \t6\n",
"\t(3) Boundary Mutation................. \t6\n",
"\t(4) Non-Uniform Mutation.............. \t6\n",
"\t(5) Polytope Crossover................ \t6\n",
"\t(6) Simple Crossover.................. \t6\n",
"\t(7) Whole Non-Uniform Mutation........ \t6\n",
"\t(8) Heuristic Crossover............... \t6\n",
"\t(9) Local-Minimum Crossover........... \t0\n",
"\n",
"SOFT Maximum Number of Generations: 100\n",
"Maximum Nonchanging Generations: 4\n",
"Population size : 50\n",
"Convergence Tolerance: 1.000000e-03\n",
"\n",
"Not Using the BFGS Derivative Based Optimizer on the Best Individual Each Generation.\n",
"Not Checking Gradients before Stopping.\n",
"Using Out of Bounds Individuals.\n",
"\n",
"Maximization Problem.\n",
"\n",
"\n",
"Generation#\t Solution Values (lexical)\n",
"\n",
" 0 \t1.291598e-02 1.291598e-02 5.790511e-02 5.790511e-02 5.797446e-02 7.850709e-02 8.243333e-02 8.243333e-02 8.952089e-02 9.118517e-02 1.567422e-01 1.567422e-01 2.570747e-01 2.753568e-01 2.801446e-01 2.839948e-01 3.173158e-01 3.173158e-01 3.567475e-01 4.933736e-01 4.933736e-01 4.933736e-01 4.933736e-01 5.640577e-01 5.640577e-01 6.620291e-01 6.620291e-01 6.742277e-01 7.509896e-01 7.509896e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 9.999999e-01 \n",
"\n",
" 1 \t1.291598e-02 1.291598e-02 5.790511e-02 5.790511e-02 6.032554e-02 7.681657e-02 8.243333e-02 8.243333e-02 9.745311e-02 9.904345e-02 1.567422e-01 1.567422e-01 1.900074e-01 2.583489e-01 2.749694e-01 2.875271e-01 3.173158e-01 3.173158e-01 3.574803e-01 4.793119e-01 4.793119e-01 4.933736e-01 4.933736e-01 4.933736e-01 4.933736e-01 5.640577e-01 5.640577e-01 6.753476e-01 7.509896e-01 7.509896e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 9.999999e-01 \n",
"\n",
" 2 \t2.992821e-02 2.992821e-02 5.690230e-02 5.690230e-02 7.900165e-02 7.925902e-02 1.204723e-01 1.330314e-01 1.361999e-01 1.456090e-01 1.567422e-01 1.567422e-01 2.544068e-01 2.566608e-01 2.566608e-01 2.704556e-01 2.766902e-01 3.173158e-01 3.173158e-01 5.751819e-01 5.773083e-01 5.877717e-01 5.877717e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.509896e-01 7.509896e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 3 \t3.396340e-02 3.396340e-02 5.690230e-02 5.690230e-02 7.798296e-02 8.430245e-02 1.204723e-01 1.256852e-01 1.261811e-01 1.455481e-01 1.567422e-01 1.567422e-01 2.566608e-01 2.566608e-01 2.596667e-01 2.704556e-01 2.766213e-01 3.173158e-01 3.173158e-01 5.751809e-01 5.913615e-01 5.913615e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.509896e-01 7.509896e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 4 \t3.396340e-02 3.396340e-02 9.102583e-02 1.238769e-01 1.271575e-01 1.345788e-01 1.567422e-01 1.567422e-01 1.792463e-01 1.792463e-01 2.098974e-01 2.142486e-01 3.090959e-01 3.173158e-01 3.173158e-01 3.308752e-01 3.538026e-01 6.550725e-01 6.550725e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.910316e-01 7.638865e-01 7.638865e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999999e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 6 \t3.814020e-02 3.814020e-02 4.462685e-02 4.462685e-02 5.790511e-02 5.790511e-02 8.403891e-02 9.187654e-02 9.314475e-02 1.034323e-01 1.138309e-01 1.415297e-01 1.567422e-01 1.567422e-01 2.724452e-01 2.854497e-01 3.173158e-01 3.173158e-01 3.173158e-01 3.173158e-01 4.085644e-01 5.781929e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.509896e-01 7.509896e-01 8.404996e-01 8.404996e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 \n",
"\n",
" 7 \t3.814020e-02 3.814020e-02 9.214648e-02 1.016995e-01 1.016995e-01 1.143044e-01 1.418078e-01 1.567359e-01 1.635396e-01 1.745811e-01 1.792463e-01 1.792463e-01 2.724455e-01 3.173158e-01 3.173158e-01 3.586566e-01 3.998497e-01 4.144313e-01 4.144313e-01 5.782228e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.627265e-01 7.627265e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 8 \t3.814020e-02 3.814020e-02 9.552639e-02 1.016995e-01 1.016995e-01 1.143044e-01 1.418078e-01 1.423816e-01 1.428271e-01 1.745811e-01 1.792463e-01 1.792463e-01 2.724455e-01 3.173158e-01 3.173158e-01 3.586566e-01 3.998497e-01 4.144313e-01 4.144313e-01 5.782228e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.627265e-01 7.627265e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 10 \t3.814020e-02 3.814020e-02 9.570633e-02 1.016995e-01 1.016995e-01 1.143044e-01 1.291365e-01 1.312865e-01 1.418078e-01 1.745811e-01 1.792463e-01 1.792463e-01 2.724455e-01 3.173158e-01 3.173158e-01 3.998497e-01 4.058229e-01 4.144313e-01 4.144313e-01 5.782228e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 8.312170e-01 8.312170e-01 8.396947e-01 8.396947e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 14 \t3.814020e-02 3.814020e-02 9.590518e-02 1.016995e-01 1.016995e-01 1.143044e-01 1.418078e-01 1.421055e-01 1.425976e-01 1.745811e-01 1.792463e-01 1.792463e-01 2.724455e-01 3.173158e-01 3.173158e-01 3.586566e-01 3.998497e-01 4.144313e-01 4.144313e-01 5.782228e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.627265e-01 7.627265e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.999750e-01 1.000000e+00 1.000000e+00 \n",
"\n",
" 17 \t3.814020e-02 3.814020e-02 1.016995e-01 1.016995e-01 1.035572e-01 1.040383e-01 1.102962e-01 1.367048e-01 1.792463e-01 1.792463e-01 2.850664e-01 3.086501e-01 3.173158e-01 3.173158e-01 3.294224e-01 3.524231e-01 3.880707e-01 6.510175e-01 6.510175e-01 6.550725e-01 6.550725e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.269639e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 1.000000e+00 1.000000e+00 1.000000e+00 \n",
"\n",
"'wait.generations' limit reached.\n",
"No significant improvement in 4 generations.\n",
"\n",
"Solution Lexical Fitness Value:\n",
"3.814020e-02 3.814020e-02 1.016995e-01 1.016995e-01 1.035572e-01 1.040383e-01 1.102962e-01 1.367048e-01 1.792463e-01 1.792463e-01 2.850664e-01 3.086501e-01 3.173158e-01 3.173158e-01 3.294224e-01 3.524231e-01 3.880707e-01 6.510175e-01 6.510175e-01 6.550725e-01 6.550725e-01 6.645532e-01 6.645532e-01 6.645532e-01 6.645532e-01 7.269639e-01 8.312170e-01 8.312170e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 9.993801e-01 1.000000e+00 1.000000e+00 1.000000e+00 \n",
"\n",
"Parameters at the Solution:\n",
"\n",
" X[ 1] :\t4.288913e+02\n",
" X[ 2] :\t9.424650e+01\n",
" X[ 3] :\t7.621631e+02\n",
" X[ 4] :\t1.591589e+02\n",
" X[ 5] :\t8.722614e+02\n",
" X[ 6] :\t1.610621e-01\n",
" X[ 7] :\t6.336749e+02\n",
" X[ 8] :\t7.989082e+02\n",
" X[ 9] :\t1.191244e+02\n",
" X[10] :\t1.535799e+02\n",
"\n",
"Solution Found Generation 17\n",
"Number of Generations Run 22\n",
"\n",
"Sun Feb 09 22:59:45 2020\n",
"Total run time : 0 hours 0 minutes and 10 seconds\n"
]
}
],
"source": [
"set.seed(12345)\n",
"# Genetic matching \n",
"gen1 <- GenMatch(Tr=Tr, X=X, BalanceMat=BalanceMat, pop.size=50,\n",
" data.type.int=FALSE, print=1, replace=FALSE)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:45.504008Z",
"start_time": "2020-02-09T14:59:34.587Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Estimate... 1789.4 \n",
"SE......... 705.77 \n",
"T-stat..... 2.5354 \n",
"p.val...... 0.011233 \n",
"\n",
"Original number of observations.............. 445 \n",
"Original number of treated obs............... 185 \n",
"Matched number of observations............... 185 \n",
"Matched number of observations (unweighted). 185 \n",
"\n"
]
}
],
"source": [
"# Matching\n",
"mgen1 <- Match(Y=Y, Tr=Tr, X=X, Weight.matrix=gen1, replace=FALSE)\n",
"summary(mgen1)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:00:44.849221Z",
"start_time": "2020-02-09T15:00:42.706Z"
},
"scrolled": true
},
"outputs": [],
"source": [
"# Balance\n",
"mbgen1 <- MatchBalance(Tr ~ age + I(age^2)+ educ + I(educ^2) + black +\n",
" hisp + married + nodegr + re74 + I(re74^2) + re75 + I(re75^2) +\n",
" u74 + u75 + I(re74*re75) + I(age*nodegr) + I(educ*re74) + I(educ*75),\n",
" data=lalonde, match.out = mgen1, print.level=0)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:47.864697Z",
"start_time": "2020-02-09T14:59:34.594Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1] 0.0381402\n",
"[1] 0.0381402\n"
]
}
],
"source": [
"# Note that the smallest pvalue in genmatch and balancematch are the same\n",
"print(gen1$value[1])\n",
"print(mbgen1$AMsmallest.p.value)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:51.331438Z",
"start_time": "2020-02-09T14:59:34.600Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<dl>\n",
"\t<dt>$Gamma</dt>\n",
"\t\t<dd>1.5</dd>\n",
"\t<dt>$GammaInc</dt>\n",
"\t\t<dd>0.1</dd>\n",
"\t<dt>$pval</dt>\n",
"\t\t<dd>0.0103</dd>\n",
"\t<dt>$msg</dt>\n",
"\t\t<dd>'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\n'</dd>\n",
"\t<dt>$bounds</dt>\n",
"\t\t<dd><table>\n",
"<thead><tr><th scope=col>Gamma</th><th scope=col>Lower bound</th><th scope=col>Upper bound</th></tr></thead>\n",
"<tbody>\n",
"\t<tr><td>1.0 </td><td>0.0103</td><td>0.0103</td></tr>\n",
"\t<tr><td>1.1 </td><td>0.0021</td><td>0.0376</td></tr>\n",
"\t<tr><td>1.2 </td><td>0.0004</td><td>0.0979</td></tr>\n",
"\t<tr><td>1.3 </td><td>0.0001</td><td>0.1981</td></tr>\n",
"\t<tr><td>1.4 </td><td>0.0000</td><td>0.3309</td></tr>\n",
"\t<tr><td>1.5 </td><td>0.0000</td><td>0.4778</td></tr>\n",
"</tbody>\n",
"</table>\n",
"</dd>\n",
"\t<dt>$note</dt>\n",
"\t\t<dd>'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'</dd>\n",
"</dl>\n"
],
"text/latex": [
"\\begin{description}\n",
"\\item[\\$Gamma] 1.5\n",
"\\item[\\$GammaInc] 0.1\n",
"\\item[\\$pval] 0.0103\n",
"\\item[\\$msg] 'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\textbackslash{}n'\n",
"\\item[\\$bounds] \\begin{tabular}{r|lll}\n",
" Gamma & Lower bound & Upper bound\\\\\n",
"\\hline\n",
"\t 1.0 & 0.0103 & 0.0103\\\\\n",
"\t 1.1 & 0.0021 & 0.0376\\\\\n",
"\t 1.2 & 0.0004 & 0.0979\\\\\n",
"\t 1.3 & 0.0001 & 0.1981\\\\\n",
"\t 1.4 & 0.0000 & 0.3309\\\\\n",
"\t 1.5 & 0.0000 & 0.4778\\\\\n",
"\\end{tabular}\n",
"\n",
"\\item[\\$note] 'Note: Gamma is Odds of Differential Assignment To\\textbackslash{}n Treatment Due to Unobserved Factors \\textbackslash{}n'\n",
"\\end{description}\n"
],
"text/markdown": [
"$Gamma\n",
": 1.5\n",
"$GammaInc\n",
": 0.1\n",
"$pval\n",
": 0.0103\n",
"$msg\n",
": 'Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \\n'\n",
"$bounds\n",
": \n",
"Gamma | Lower bound | Upper bound | \n",
"|---|---|---|---|---|---|\n",
"| 1.0 | 0.0103 | 0.0103 | \n",
"| 1.1 | 0.0021 | 0.0376 | \n",
"| 1.2 | 0.0004 | 0.0979 | \n",
"| 1.3 | 0.0001 | 0.1981 | \n",
"| 1.4 | 0.0000 | 0.3309 | \n",
"| 1.5 | 0.0000 | 0.4778 | \n",
"\n",
"\n",
"\n",
"$note\n",
": 'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'\n",
"\n",
"\n"
],
"text/plain": [
"\n",
" Rosenbaum Sensitivity Test for Wilcoxon Signed Rank P-Value \n",
" \n",
"Unconfounded estimate .... 0.0103 \n",
"\n",
" Gamma Lower bound Upper bound\n",
" 1.0 0.0103 0.0103\n",
" 1.1 0.0021 0.0376\n",
" 1.2 0.0004 0.0979\n",
" 1.3 0.0001 0.1981\n",
" 1.4 0.0000 0.3309\n",
" 1.5 0.0000 0.4778\n",
"\n",
" Note: Gamma is Odds of Differential Assignment To\n",
" Treatment Due to Unobserved Factors \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<ol class=list-inline>\n",
"\t<li>0.9897</li>\n",
"\t<li>0.9624</li>\n",
"\t<li>0.9021</li>\n",
"\t<li>0.8019</li>\n",
"\t<li>0.6691</li>\n",
"\t<li>0.5222</li>\n",
"</ol>\n"
],
"text/latex": [
"\\begin{enumerate*}\n",
"\\item 0.9897\n",
"\\item 0.9624\n",
"\\item 0.9021\n",
"\\item 0.8019\n",
"\\item 0.6691\n",
"\\item 0.5222\n",
"\\end{enumerate*}\n"
],
"text/markdown": [
"1. 0.9897\n",
"2. 0.9624\n",
"3. 0.9021\n",
"4. 0.8019\n",
"5. 0.6691\n",
"6. 0.5222\n",
"\n",
"\n"
],
"text/plain": [
"[1] 0.9897 0.9624 0.9021 0.8019 0.6691 0.5222"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<dl>\n",
"\t<dt>$Gamma</dt>\n",
"\t\t<dd>1.5</dd>\n",
"\t<dt>$GammaInc</dt>\n",
"\t\t<dd>0.1</dd>\n",
"\t<dt>$pval</dt>\n",
"\t\t<dd>1584.2950913997</dd>\n",
"\t<dt>$msg</dt>\n",
"\t\t<dd>'Rosenbaum Sensitivity Test for Hodges-Lehmann Point Estimate \\n'</dd>\n",
"\t<dt>$bounds</dt>\n",
"\t\t<dd><table>\n",
"<thead><tr><th scope=col>Gamma</th><th scope=col>Lower bound</th><th scope=col>Upper bound</th></tr></thead>\n",
"<tbody>\n",
"\t<tr><td>1.0 </td><td>1584.300</td><td>1584.3 </td></tr>\n",
"\t<tr><td>1.1 </td><td> 970.200</td><td>1717.9 </td></tr>\n",
"\t<tr><td>1.2 </td><td> 622.300</td><td>2016.4 </td></tr>\n",
"\t<tr><td>1.3 </td><td> 332.300</td><td>2276.1 </td></tr>\n",
"\t<tr><td>1.4 </td><td> 88.195</td><td>2575.9 </td></tr>\n",
"\t<tr><td>1.5 </td><td> -91.005</td><td>2850.4 </td></tr>\n",
"</tbody>\n",
"</table>\n",
"</dd>\n",
"\t<dt>$note</dt>\n",
"\t\t<dd>'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'</dd>\n",
"</dl>\n"
],
"text/latex": [
"\\begin{description}\n",
"\\item[\\$Gamma] 1.5\n",
"\\item[\\$GammaInc] 0.1\n",
"\\item[\\$pval] 1584.2950913997\n",
"\\item[\\$msg] 'Rosenbaum Sensitivity Test for Hodges-Lehmann Point Estimate \\textbackslash{}n'\n",
"\\item[\\$bounds] \\begin{tabular}{r|lll}\n",
" Gamma & Lower bound & Upper bound\\\\\n",
"\\hline\n",
"\t 1.0 & 1584.300 & 1584.3 \\\\\n",
"\t 1.1 & 970.200 & 1717.9 \\\\\n",
"\t 1.2 & 622.300 & 2016.4 \\\\\n",
"\t 1.3 & 332.300 & 2276.1 \\\\\n",
"\t 1.4 & 88.195 & 2575.9 \\\\\n",
"\t 1.5 & -91.005 & 2850.4 \\\\\n",
"\\end{tabular}\n",
"\n",
"\\item[\\$note] 'Note: Gamma is Odds of Differential Assignment To\\textbackslash{}n Treatment Due to Unobserved Factors \\textbackslash{}n'\n",
"\\end{description}\n"
],
"text/markdown": [
"$Gamma\n",
": 1.5\n",
"$GammaInc\n",
": 0.1\n",
"$pval\n",
": 1584.2950913997\n",
"$msg\n",
": 'Rosenbaum Sensitivity Test for Hodges-Lehmann Point Estimate \\n'\n",
"$bounds\n",
": \n",
"Gamma | Lower bound | Upper bound | \n",
"|---|---|---|---|---|---|\n",
"| 1.0 | 1584.300 | 1584.3 | \n",
"| 1.1 | 970.200 | 1717.9 | \n",
"| 1.2 | 622.300 | 2016.4 | \n",
"| 1.3 | 332.300 | 2276.1 | \n",
"| 1.4 | 88.195 | 2575.9 | \n",
"| 1.5 | -91.005 | 2850.4 | \n",
"\n",
"\n",
"\n",
"$note\n",
": 'Note: Gamma is Odds of Differential Assignment To\\n Treatment Due to Unobserved Factors \\n'\n",
"\n",
"\n"
],
"text/plain": [
"\n",
" Rosenbaum Sensitivity Test for Hodges-Lehmann Point Estimate \n",
" \n",
"Unconfounded estimate .... 1584.295 \n",
"\n",
" Gamma Lower bound Upper bound\n",
" 1.0 1584.300 1584.3\n",
" 1.1 970.200 1717.9\n",
" 1.2 622.300 2016.4\n",
" 1.3 332.300 2276.1\n",
" 1.4 88.195 2575.9\n",
" 1.5 -91.005 2850.4\n",
"\n",
" Note: Gamma is Odds of Differential Assignment To\n",
" Treatment Due to Unobserved Factors \n",
" "
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Sensitivity Analysis\n",
"p <- psens(mgen1, Gamma=1.5, GammaInc=.1)\n",
"p\n",
"1 - p$bounds$`Upper bound`\n",
"hlsens(mgen1, Gamma=1.5, GammaInc=.1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Match vs. GenMatch Investigation "
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T14:59:51.371302Z",
"start_time": "2020-02-09T14:59:34.605Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1] \"39 / 185 are different matches\"\n"
]
}
],
"source": [
"# Are matches from the genetic matching object the same and the ones from the match object?\n",
"m = sum(gen1$matches[, 2] != mgen1$index.control)\n",
"n = nrow(gen1$matches)\n",
"print(sprintf(\"%d / %d are different matches\", m, n))"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:00:06.732658Z",
"start_time": "2020-02-09T14:59:34.608Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1] \"0 / 272 are different matches\"\n",
"[1] \"yes :)\"\n"
]
}
],
"source": [
"# Is it because of the stochastic behavior of ties = FALSE?\n",
"\n",
"gen2 <- GenMatch(Tr=Tr, X=X, BalanceMat=BalanceMat, pop.size=50,\n",
" data.type.int=FALSE, print=0)\n",
"\n",
"mgen2 <- Match(Y=Y, Tr=Tr, X=X, Weight.matrix=gen2)\n",
"\n",
"mbgen2 <- MatchBalance(Tr ~ age + I(age^2)+ educ + I(educ^2) + black +\n",
" hisp + married + nodegr + re74 + I(re74^2) + re75 + I(re75^2) +\n",
" u74 + u75 + I(re74*re75) + I(age*nodegr) + I(educ*re74) + I(educ*75),\n",
" data=lalonde, match.out = mgen2, print=0)\n",
"\n",
"stopifnot(gen1$value[1] == mbgen1$AMsmallest.p.value)\n",
"\n",
"m = sum(gen2$matches[, 2] != mgen2$index.control)\n",
"n = nrow(gen2$matches)\n",
"print(sprintf(\"%d / %d are different matches\", m, n))\n",
"\n",
"print(\"yes :)\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Thoughts\n",
"\n",
"There is an issue moving forward. Since the fit function is computed over the matches generated by `GenMatch`, with `ties=FALSE` we cannot guarantee that the loss of the `Match` output will be the same as the `GenMatch` output. The cause seems to be that an equal distance is the matching space (i.e., a tie among controls) does not entail that the ties controls result in the same level of robustness. In other words, a tie in the \"matching space\" is not equivalent to a tie in the \"robustness space\".\n",
"\n",
"Note that the only reason we do not want to allow ties is because the `psens` function does not account for it in its calculation of Rosenbaum's $\\Gamma$. A possible way to resolve the lack of stability in `GenMatch + psens` is to allow ties, but break them deterministically instead of stochastically inside the fitness function, and then do the same with the output of `Match`. For example, since the tied controls are ordered numerically, one could pick the first control only."
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:01:05.155727Z",
"start_time": "2020-02-09T15:01:05.119Z"
},
"code_folding": []
},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<tbody>\n",
"\t<tr><td>6 </td><td>188</td><td>0.5</td></tr>\n",
"\t<tr><td>6 </td><td>244</td><td>0.5</td></tr>\n",
"</tbody>\n",
"</table>\n"
],
"text/latex": [
"\\begin{tabular}{lll}\n",
"\t 6 & 188 & 0.5\\\\\n",
"\t 6 & 244 & 0.5\\\\\n",
"\\end{tabular}\n"
],
"text/markdown": [
"\n",
"| 6 | 188 | 0.5 | \n",
"| 6 | 244 | 0.5 | \n",
"\n",
"\n"
],
"text/plain": [
" [,1] [,2] [,3]\n",
"[1,] 6 188 0.5 \n",
"[2,] 6 244 0.5 "
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"'Instead of using 2 controls, we could deterministically pick the first one'"
],
"text/latex": [
"'Instead of using 2 controls, we could deterministically pick the first one'"
],
"text/markdown": [
"'Instead of using 2 controls, we could deterministically pick the first one'"
],
"text/plain": [
"[1] \"Instead of using 2 controls, we could deterministically pick the first one\""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Example\n",
"first.tie.trt.idx <- which(gen2$matches[,3] < 1)[1]\n",
"first.tie.rows.idx <- which(gen2$matches[,1] == first.tie.trt.idx)\n",
"gen2$matches[first.tie.rows.idx, ]\n",
"sprintf(\"Instead of using %d controls, we could deterministically pick the first one\", \n",
" length(first.tie.rows.idx))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Robustness Optimization with `rbounds`"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:01:08.323739Z",
"start_time": "2020-02-09T15:01:06.656Z"
}
},
"outputs": [],
"source": [
"# Sensitivity optim function\n",
"robust.fitfunc <- function(matches, BM) {\n",
"\n",
" # my.fitfunc requires that the last column of the BM matrix contain the outcomes\n",
" # hence, it is VERY important not to run standard pval balance genetic matching\n",
" # with the same BM!\n",
" \n",
" # NOTE: psens does NOT acount for 1:many matching!\n",
" index.treated <- matches[,1]\n",
" index.control <- matches[,2]\n",
" \n",
" # Get treated and control unit outcomes\n",
" trt <- BM[, ncol(BM)][index.treated]\n",
" ctrl <- BM[, ncol(BM)][index.control]\n",
" \n",
" # Compute sensitivity\n",
" psens.out <- psens(trt, ctrl, Gamma=1.5, GammaInc=.1)\n",
" \n",
" # We want to minimize the largest p-value. Thus,\n",
" pvals <- psens.out$bounds$`Upper bound`\n",
" \n",
" return(sort(pvals, decreasing=TRUE))\n",
"}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Fitness Function Demonstration"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:01:11.435408Z",
"start_time": "2020-02-09T15:01:07.330Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<ol class=list-inline>\n",
"\t<li>0.4778</li>\n",
"\t<li>0.3309</li>\n",
"\t<li>0.1981</li>\n",
"\t<li>0.0979</li>\n",
"\t<li>0.0376</li>\n",
"\t<li>0.0103</li>\n",
"</ol>\n"
],
"text/latex": [
"\\begin{enumerate*}\n",
"\\item 0.4778\n",
"\\item 0.3309\n",
"\\item 0.1981\n",
"\\item 0.0979\n",
"\\item 0.0376\n",
"\\item 0.0103\n",
"\\end{enumerate*}\n"
],
"text/markdown": [
"1. 0.4778\n",
"2. 0.3309\n",
"3. 0.1981\n",
"4. 0.0979\n",
"5. 0.0376\n",
"6. 0.0103\n",
"\n",
"\n"
],
"text/plain": [
"[1] 0.4778 0.3309 0.1981 0.0979 0.0376 0.0103"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<ol class=list-inline>\n",
"\t<li>0.3194</li>\n",
"\t<li>0.1986</li>\n",
"\t<li>0.1052</li>\n",
"\t<li>0.0452</li>\n",
"\t<li>0.0148</li>\n",
"\t<li>0.0034</li>\n",
"</ol>\n"
],
"text/latex": [
"\\begin{enumerate*}\n",
"\\item 0.3194\n",
"\\item 0.1986\n",
"\\item 0.1052\n",
"\\item 0.0452\n",
"\\item 0.0148\n",
"\\item 0.0034\n",
"\\end{enumerate*}\n"
],
"text/markdown": [
"1. 0.3194\n",
"2. 0.1986\n",
"3. 0.1052\n",
"4. 0.0452\n",
"5. 0.0148\n",
"6. 0.0034\n",
"\n",
"\n"
],
"text/plain": [
"[1] 0.3194 0.1986 0.1052 0.0452 0.0148 0.0034"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Demonstration ties=FALSE\n",
"BM <- cbind(BalanceMat, re78)\n",
"df <- data.frame(trt = mgen1$index.treated, ctrl = mgen1$index.control)\n",
"robust.fitfunc(df, BM) # With matches from Match()\n",
"robust.fitfunc(gen1$matches, BM) # With matches from GenMatch()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:01:14.595539Z",
"start_time": "2020-02-09T15:01:07.520Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<ol class=list-inline>\n",
"\t<li>0.7011</li>\n",
"\t<li>0.5261</li>\n",
"\t<li>0.3333</li>\n",
"\t<li>0.1667</li>\n",
"\t<li>0.0603</li>\n",
"\t<li>0.014</li>\n",
"</ol>\n"
],
"text/latex": [
"\\begin{enumerate*}\n",
"\\item 0.7011\n",
"\\item 0.5261\n",
"\\item 0.3333\n",
"\\item 0.1667\n",
"\\item 0.0603\n",
"\\item 0.014\n",
"\\end{enumerate*}\n"
],
"text/markdown": [
"1. 0.7011\n",
"2. 0.5261\n",
"3. 0.3333\n",
"4. 0.1667\n",
"5. 0.0603\n",
"6. 0.014\n",
"\n",
"\n"
],
"text/plain": [
"[1] 0.7011 0.5261 0.3333 0.1667 0.0603 0.0140"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<ol class=list-inline>\n",
"\t<li>0.7011</li>\n",
"\t<li>0.5261</li>\n",
"\t<li>0.3333</li>\n",
"\t<li>0.1667</li>\n",
"\t<li>0.0603</li>\n",
"\t<li>0.014</li>\n",
"</ol>\n"
],
"text/latex": [
"\\begin{enumerate*}\n",
"\\item 0.7011\n",
"\\item 0.5261\n",
"\\item 0.3333\n",
"\\item 0.1667\n",
"\\item 0.0603\n",
"\\item 0.014\n",
"\\end{enumerate*}\n"
],
"text/markdown": [
"1. 0.7011\n",
"2. 0.5261\n",
"3. 0.3333\n",
"4. 0.1667\n",
"5. 0.0603\n",
"6. 0.014\n",
"\n",
"\n"
],
"text/plain": [
"[1] 0.7011 0.5261 0.3333 0.1667 0.0603 0.0140"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# with ties=TRUE the results are the same, but the pvals are probably incorrect\n",
"BM <- cbind(BalanceMat, re78)\n",
"df <- data.frame(trt = mgen2$index.treated, ctrl = mgen2$index.control)\n",
"robust.fitfunc(df, BM) # With matches from Match()\n",
"robust.fitfunc(gen2$matches, BM) # With matches from GenMatch()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Genetic Matching \n",
"Before resolving the issue of ties, let us check whether genetic matching works in this simple case."
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:02:01.517882Z",
"start_time": "2020-02-09T15:01:07.882Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"Sun Feb 09 23:01:18 2020\n",
"Domains:\n",
" 0.000000e+00 <= X1 <= 1.000000e+03 \n",
" 0.000000e+00 <= X2 <= 1.000000e+03 \n",
" 0.000000e+00 <= X3 <= 1.000000e+03 \n",
" 0.000000e+00 <= X4 <= 1.000000e+03 \n",
" 0.000000e+00 <= X5 <= 1.000000e+03 \n",
" 0.000000e+00 <= X6 <= 1.000000e+03 \n",
" 0.000000e+00 <= X7 <= 1.000000e+03 \n",
" 0.000000e+00 <= X8 <= 1.000000e+03 \n",
" 0.000000e+00 <= X9 <= 1.000000e+03 \n",
" 0.000000e+00 <= X10 <= 1.000000e+03 \n",
"\n",
"Data Type: Floating Point\n",
"Operators (code number, name, population) \n",
"\t(1) Cloning........................... \t65\n",
"\t(2) Uniform Mutation.................. \t62\n",
"\t(3) Boundary Mutation................. \t62\n",
"\t(4) Non-Uniform Mutation.............. \t62\n",
"\t(5) Polytope Crossover................ \t62\n",
"\t(6) Simple Crossover.................. \t62\n",
"\t(7) Whole Non-Uniform Mutation........ \t62\n",
"\t(8) Heuristic Crossover............... \t62\n",
"\t(9) Local-Minimum Crossover........... \t0\n",
"\n",
"SOFT Maximum Number of Generations: 100\n",
"Maximum Nonchanging Generations: 20\n",
"Population size : 500\n",
"Convergence Tolerance: 1.000000e-03\n",
"\n",
"Not Using the BFGS Derivative Based Optimizer on the Best Individual Each Generation.\n",
"Not Checking Gradients before Stopping.\n",
"Using Out of Bounds Individuals.\n",
"\n",
"Minimization Problem.\n",
"\n",
"\n",
"Generation#\t Solution Values (lexical)\n",
"\n",
" 0 \t5.610000e-02 2.520000e-02 9.300000e-03 2.700000e-03 6.000000e-04 1.000000e-04 \n",
"\n",
" 1 \t3.780000e-02 1.610000e-02 5.600000e-03 1.500000e-03 3.000000e-04 0.000000e+00 \n",
"\n",
" 7 \t3.510000e-02 1.470000e-02 5.000000e-03 1.300000e-03 2.000000e-04 0.000000e+00 \n",
"\n",
" 8 \t3.280000e-02 1.350000e-02 4.500000e-03 1.200000e-03 2.000000e-04 0.000000e+00 \n",
"\n",
" 11 \t2.670000e-02 1.080000e-02 3.500000e-03 9.000000e-04 2.000000e-04 0.000000e+00 \n",
"\n",
" 12 \t2.130000e-02 8.400000e-03 2.700000e-03 7.000000e-04 1.000000e-04 0.000000e+00 \n",
"\n",
" 13 \t1.600000e-02 6.100000e-03 1.900000e-03 4.000000e-04 1.000000e-04 0.000000e+00 \n",
"\n",
" 18 \t1.120000e-02 4.000000e-03 1.200000e-03 3.000000e-04 0.000000e+00 0.000000e+00 \n",
"\n",
"'wait.generations' limit reached.\n",
"No significant improvement in 20 generations.\n",
"\n",
"Solution Lexical Fitness Value:\n",
"1.120000e-02 4.000000e-03 1.200000e-03 3.000000e-04 0.000000e+00 0.000000e+00 \n",
"\n",
"Parameters at the Solution:\n",
"\n",
" X[ 1] :\t9.981413e+02\n",
" X[ 2] :\t6.870264e+02\n",
" X[ 3] :\t9.340755e+02\n",
" X[ 4] :\t2.365987e+02\n",
" X[ 5] :\t9.109633e+02\n",
" X[ 6] :\t7.569893e+02\n",
" X[ 7] :\t8.208707e+02\n",
" X[ 8] :\t3.793979e+01\n",
" X[ 9] :\t5.976252e+02\n",
" X[10] :\t7.132821e-01\n",
"\n",
"Solution Found Generation 18\n",
"Number of Generations Run 39\n",
"\n",
"Sun Feb 09 23:02:01 2020\n",
"Total run time : 0 hours 0 minutes and 43 seconds\n",
"\n",
"Estimate... 2469.8 \n",
"SE......... 689.26 \n",
"T-stat..... 3.5832 \n",
"p.val...... 0.00033942 \n",
"\n",
"Original number of observations.............. 445 \n",
"Original number of treated obs............... 185 \n",
"Matched number of observations............... 185 \n",
"Matched number of observations (unweighted). 185 \n",
"\n"
]
}
],
"source": [
"# Genetic Optimization\n",
"genout3 <- GenMatch(Tr=Tr, X=X, BalanceMat=Y, pop.size=500,\n",
" print=1, ties=FALSE, wait.generations = 20, fit.func = robust.fitfunc)\n",
"\n",
"mout <- Match(Y=Y, Tr=treat, X=X, ties=FALSE, Weight.matrix=genout3)\n",
"summary(mout)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:02:04.823046Z",
"start_time": "2020-02-09T15:01:08.030Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<table>\n",
"<tbody>\n",
"\t<tr><td>0e+00 </td><td>0.0000</td><td>0.0003</td><td>0.0012</td><td>0.0040</td><td>0.0112</td></tr>\n",
"\t<tr><td>2e-04 </td><td>0.0014</td><td>0.0059</td><td>0.0186</td><td>0.0463</td><td>0.0951</td></tr>\n",
"</tbody>\n",
"</table>\n"
],
"text/latex": [
"\\begin{tabular}{llllll}\n",
"\t 0e+00 & 0.0000 & 0.0003 & 0.0012 & 0.0040 & 0.0112\\\\\n",
"\t 2e-04 & 0.0014 & 0.0059 & 0.0186 & 0.0463 & 0.0951\\\\\n",
"\\end{tabular}\n"
],
"text/markdown": [
"\n",
"| 0e+00 | 0.0000 | 0.0003 | 0.0012 | 0.0040 | 0.0112 | \n",
"| 2e-04 | 0.0014 | 0.0059 | 0.0186 | 0.0463 | 0.0951 | \n",
"\n",
"\n"
],
"text/plain": [
" [,1] [,2] [,3] [,4] [,5] [,6] \n",
"[1,] 0e+00 0.0000 0.0003 0.0012 0.0040 0.0112\n",
"[2,] 2e-04 0.0014 0.0059 0.0186 0.0463 0.0951"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Notice that the pvalues from inside genetic matching \n",
"# and from the match object are very different\n",
"rbind(sort(genout3$value),\n",
" psens(mout, Gamma=1.5, GammaInc=.1)$bounds[, 3])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Robustness Optimization with `sensitivitymv` "
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"ExecuteTime": {
"end_time": "2020-02-09T15:02:08.136701Z",
"start_time": "2020-02-09T15:01:08.406Z"
}
},
"outputs": [],
"source": [
"library(sensitivitymv)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "R [conda env:renv]",
"language": "R",
"name": "conda-env-renv-r"
},
"language_info": {
"codemirror_mode": "r",
"file_extension": ".r",
"mimetype": "text/x-r-source",
"name": "R",
"pygments_lexer": "r",
"version": "3.5.1"
},
"toc": {
"base_numbering": 1,
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": false,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment