{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eb02b7cc",
   "metadata": {},
   "source": [
    "# Post-estimation Overview - Poisson\n",
    "\n",
    "This notebook provides an overview of post-estimation results that are available in several models, illustrated for the Poisson Model.\n",
    "\n",
    "see also https://github.com/statsmodels/statsmodels/issues/7707\n",
    "\n",
    "Traditionally the results classes for the models provided Wald inference and prediction. Several models now have additional methods for postestimation results, for inference, prediction and specification or diagnostic tests.\n",
    "\n",
    "The following is based on the current pattern for maximum likelihood models outside tsa, mainly for the discrete models. Other models still follow to some extend a different API pattern. Linear models like OLS and WLS have their special implementation, for example OLS influence. GLM also still has some features that are model specific.\n",
    "\n",
    "The main post-estimation features are\n",
    "\n",
    "- Inference - Wald tests [section](#Inference---Wald)\n",
    "- Inference - score tests [section](#Inference---score_test)\n",
    "- `get_prediction` prediction with inferential statistics [section](#Prediction)\n",
    "- `get_distribution` distribution class based on estimated parameters [section](#Distribution)\n",
    "- `get_diagnostic` diagnostic and specification tests, measures and plots [section](#Diagnostic)\n",
    "- `get_influence` outlier and influence diagnostics [section](#Outliers-and-Influence)\n",
    "\n",
    "**Warning** Recently added features are not stable.  \n",
    "The main features have been unit tested and verified against other statistical packages. However, not every option is fully tested. The API, options, defaults and return types may still change as more features are added. \n",
    "(The current emphasis is on adding features and not on finding a convenient and futureproof interface.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3707c29c",
   "metadata": {},
   "source": [
    "## A simulated example \n",
    "\n",
    "For the illustration we simulate data for the Poisson regression, that is correctly specified and has a relatively large sample. One regressor is categorical with two levels, The second regressor is uniformly distributed on the unit interval.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f9f42c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import stats\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from statsmodels.discrete.discrete_model import Poisson\n",
    "from statsmodels.discrete.diagnostic import PoissonDiagnostic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a85c587",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(983154356)\n",
    "\n",
    "nr = 10\n",
    "n_groups = 2\n",
    "labels = np.arange(n_groups)\n",
    "x = np.repeat(labels, np.array([40, 60]) * nr)\n",
    "nobs = x.shape[0]\n",
    "exog = (x[:, None] == labels).astype(np.float64)\n",
    "xc = np.random.rand(len(x))\n",
    "exog = np.column_stack((exog, xc))\n",
    "# reparameterize to explicit constant\n",
    "# exog[:, 1] = 1\n",
    "beta = np.array([0.2, 0.3, 0.5], np.float64)\n",
    "\n",
    "linpred = exog @ beta\n",
    "mean = np.exp(linpred)\n",
    "y = np.random.poisson(mean)\n",
    "len(y), y.mean(), (y == 0).mean()\n",
    "\n",
    "res = Poisson(y, exog).fit(disp=0)\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35c50372",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "bd5e9115",
   "metadata": {},
   "source": [
    "## Inference - Wald\n",
    "\n",
    "Wald tests and other inferential statistics like confidence intervals based on Wald test have been a feature of the models since the beginning. Wald inference is based on the Hessian or expected information matrix evaluted at the estimated parameters.  \n",
    "The covariance matrix of the parameter is optionally of the sandwich form which is robust to unspecified heteroscedasticity or serial or cluster correlation (`cov_type` option for `fit`).\n",
    "\n",
    "The currently available methods, aside from the statistics in the parmeter table, are\n",
    "\n",
    "- t_test\n",
    "- wald_test\n",
    "- t_test_pairwise\n",
    "- wald_test_terms\n",
    "\n",
    "`f_test` is available as legacy method. It is the same as `wald_test` with keyword option `use_f=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fb15ec2",
   "metadata": {},
   "outputs": [],
   "source": [
    "res.t_test(\"x1=x2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a71b50a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "res.wald_test(\"x1=x2, x3\", scalar=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9d1cfbe",
   "metadata": {},
   "source": [
    "## Inference - score_test\n",
    "\n",
    "new in statsmodels 0.14 for most discrete models and for GLM.\n",
    "\n",
    "Score or lagrange multiplier (LM) tests are based on the model estimated under the null hypothesis. A common example are variable addition tests for which we estimate the model parameters under null restrictions but evaluate the score and hessian under for the full model to test whether an additional variable is statistically significant.\n",
    "\n",
    "\n",
    "**Note:** Similar to the Wald tests, the score test implemented in the discrete models and GLM also has the option to use a heteroscedasticity or correlation robust covariance type.  \n",
    "It currently uses the same implementation and defaults for the robust covariance matrix as in the Wald tests. In some cases the small sample corrections included in the `cov_type` for Wald tests will not be appropriate for score tests. In many cases Wald tests overjects but score tests can underreject. Using the Wald small sample corrections for score tests might leads then to more conservative p-values.  \n",
    "(The defaults for small sample corrections might change in future. There is currently only little general information available about small sample corrections for heteroscedasticity and correlation robust score tests. Other statistical packages only implement it for a few special cases.)\n",
    "\n",
    "We can use the variable addition score_test for specification testing. In the following example we test whether there is some misspecified nonlinearity in the model by adding quadratic or polynomial tersm.\n",
    "\n",
    "In our example we can expect that these specification tests do not reject the null hypotheses because the model is correctly specified and the sample size is large,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "898bb259",
   "metadata": {},
   "outputs": [],
   "source": [
    "res.score_test(exog_extra=xc**2)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a32fca97",
   "metadata": {},
   "source": [
    "A reset test is a test for correct specification of the link function. The standard form of the test adds polynomial terms of the linear predictor as extra regressors and test for their significance. \n",
    "\n",
    "Here we use the variable addition score test for the reset test with powers 2 and 3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c7fd63e",
   "metadata": {},
   "outputs": [],
   "source": [
    "linpred = res.predict(which=\"linear\")\n",
    "res.score_test(exog_extra=linpred[:,None]**[2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c36ec60",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "The model and results classes have `predict` method which only returns the predicted values. The `get_prediction` method adds inferential statistics for the prediction, standard errors, pvalues and confidence intervals.\n",
    "\n",
    "\n",
    "For the following example, we create new sets of explanatory variables that is split by the categorical level and over a uniform grid of the continuous variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e782b14",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 11\n",
    "exc = np.linspace(0, 1, n)\n",
    "ex1 = np.column_stack((np.ones(n), np.zeros(n), exc))\n",
    "ex2 = np.column_stack((np.zeros(n), np.ones(n), exc))\n",
    "\n",
    "m1 = res.get_prediction(ex1)\n",
    "m2 = res.get_prediction(ex2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7297efb",
   "metadata": {},
   "source": [
    "The available methods and attributes of the prediction results class are"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eea3c225",
   "metadata": {},
   "outputs": [],
   "source": [
    "[i for i in dir(m1) if not i.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2a25e09",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(exc, np.column_stack([m1.predicted, m2.predicted]))\n",
    "ci = m1.conf_int()\n",
    "plt.fill_between(exc, ci[:, 0], ci[:, 1], color='b', alpha=.1)\n",
    "ci = m2.conf_int()\n",
    "plt.fill_between(exc, ci[:, 0], ci[:, 1], color='r', alpha=.1)\n",
    "# to add observed points:\n",
    "# y1 = y[x == 0]\n",
    "# plt.plot(xc[x == 0], y1, \".\", color=\"b\", alpha=.3)\n",
    "# y2 = y[x == 1]\n",
    "# plt.plot(xc[x == 1], y2, \".\", color=\"r\", alpha=.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1beaeb7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1e25469",
   "metadata": {},
   "source": [
    "One of the available statistics that we can predict, specified by the \"which\" keyword, is the expected frequencies or probabilities of the predictive distribution. This shows us what the predicted probability of obsering count = 1, 2, 3, ... is for a given set of explanatory variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f61baf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_max = 5\n",
    "f1 = res.get_prediction(ex1, which=\"prob\", y_values=np.arange(y_max + 1))\n",
    "f2 = res.get_prediction(ex2, which=\"prob\", y_values=np.arange(y_max + 1))\n",
    "f1.predicted.mean(0), f2.predicted.mean(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bcc96fc7",
   "metadata": {},
   "source": [
    "We can also get the confidence intervals for the predicted probabilities.\n",
    "However, if we want the confidence interval for the average predicted probabilities, then we need to aggregate inside the predict function. The relevant keyword is \"average\" which computes the average of the predictions over the observations given by the `exog` array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c514f3af",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1 = res.get_prediction(ex1, which=\"prob\", y_values=np.arange(y_max + 1), average=True)\n",
    "f2 = res.get_prediction(ex2, which=\"prob\", y_values=np.arange(y_max + 1), average=True)\n",
    "f1.predicted, f2.predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35071145",
   "metadata": {},
   "outputs": [],
   "source": [
    "f1.conf_int()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f24e88e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "f2.conf_int()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c801d6",
   "metadata": {},
   "source": [
    "To get more information about the predict methods and the available options, see  \n",
    "`help(res.get_prediction)`  \n",
    "`help(res.model.predict)`  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01cf4bed",
   "metadata": {},
   "source": [
    "## Distribution\n",
    "\n",
    "For given parameters we can create an instance of a scipy or scipy-compatible distribution class. This provides us with access to any of the methods in the distribution, pmf/pdf, cdf, stats.\n",
    "\n",
    "The `get_distribution` method of the results class uses the provided array of explanatory variables and the estimated parameters to specify a vectorized distribution. The `get_prediction` method of the model can be used for user specified parameters `params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "796db909",
   "metadata": {},
   "outputs": [],
   "source": [
    "distr = res.get_distribution()\n",
    "distr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2ce91ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "distr.pmf(0)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c833de45",
   "metadata": {},
   "source": [
    "The mean of the conditional distribution is the same as the predicted mean from the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7581e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "distr.mean()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "628c0593",
   "metadata": {},
   "outputs": [],
   "source": [
    "res.predict()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f504d66",
   "metadata": {},
   "source": [
    "We can also obtain the distribution for a new set of explanatory variables. Explanatory variables can be provided in the same way as for the predict method.\n",
    "\n",
    "We use again the grid of explanatory variables from the prediction section. As example for its usage we can compute the probability that a count (strictly) larger than 5 will be observed conditional on the values of the explanatory variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83cbe355",
   "metadata": {},
   "outputs": [],
   "source": [
    "distr1 = res.get_distribution(ex1)\n",
    "distr2 = res.get_distribution(ex2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cfe5e90",
   "metadata": {},
   "outputs": [],
   "source": [
    "distr1.sf(5), distr2.sf(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01db4a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(exc, np.column_stack([distr1.sf(5), distr2.sf(5)]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f16e953",
   "metadata": {},
   "source": [
    "We can also use the distribution to find an upper confidence limit on a new observation. The following plot and table show the upper limit counts for given explanatory variables. The probability of observing this count or less is at least 0.99. \n",
    "\n",
    "Note, this takes parameters as fixed and does not take parameter uncertainty into account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e0d28d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(exc, np.column_stack([distr1.ppf(0.99), distr2.ppf(0.99)]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f10b68be",
   "metadata": {},
   "outputs": [],
   "source": [
    "[distr1.ppf(0.99), distr2.ppf(0.99)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1364c5bd",
   "metadata": {},
   "source": [
    "## Diagnostic\n",
    "\n",
    "Poisson is the first model that has a diagnostic class that can be obtained from the results using `get_diagnostic`. Other count models have a generic count diagnostic class that has currently only a limited number of methods.\n",
    "\n",
    "The Poisson model in our example is correctly specified. Additionally we have a large sample size. So, in this case none of the diagnostic tests reject the null hypothesis of correct specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35dec411",
   "metadata": {},
   "outputs": [],
   "source": [
    "dia = res.get_diagnostic()\n",
    "[i for i in dir(dia) if not i.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "042fa854",
   "metadata": {},
   "outputs": [],
   "source": [
    "dia.plot_probs();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60d5a69e",
   "metadata": {},
   "source": [
    "**test for excess dispersion**\n",
    "\n",
    "There are several dispersion tests available for Poisson. Currently all of them are returned.\n",
    "The DispersionResults class has a summary_frame method. The returned dataframe provides an overview of the results that is  easier to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a66859b",
   "metadata": {},
   "outputs": [],
   "source": [
    "td = dia.test_dispersion()\n",
    "td"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "747edf4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = td.summary_frame()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdd50bc1",
   "metadata": {},
   "source": [
    "**test for zero-inflation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4394c8f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dia.test_poisson_zeroinflation()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae2da82e",
   "metadata": {},
   "source": [
    "chisquare test for zero-inflation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed583f69",
   "metadata": {},
   "outputs": [],
   "source": [
    "dia.test_chisquare_prob(bin_edges=np.arange(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdb54ac4",
   "metadata": {},
   "source": [
    "**goodness of fit test for predicted frequencies**\n",
    "\n",
    "This is a chisquare test that takes into account that parameters are estimated.\n",
    "Counts larger than the largest bin edge will be added to the last bin, so that the sum over bins is one.\n",
    "\n",
    "For example using 5 bins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b32625bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt = dia.test_chisquare_prob(bin_edges=np.arange(6))\n",
    "dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73bdbcc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dt.diff1.mean(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6335b247",
   "metadata": {},
   "outputs": [],
   "source": [
    "vars(dia)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fb99cca",
   "metadata": {},
   "source": [
    "## Outliers and Influence\n",
    "\n",
    "Statsmodels provides a general MLEInfluence class for nonlinear models (models with nonlinear expected mean) that for the discrete models and other maximum likelihood based models such as the Beta regression model.\n",
    "The provided measures are based on general definitions, for example generalized leverage instead of the diagonal of the hat matrix in linear models.\n",
    "\n",
    "The results method `get_influence` returns and instance of the MLEInfluence class which has various methods for outlier and influence measures. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd36e5bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "infl = res.get_influence()\n",
    "[i for i in dir(infl) if not i.startswith(\"_\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "431088a9",
   "metadata": {},
   "source": [
    "The influence class has two plot methods. However, the plots are too crowded in this case because of the large sample size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1c2c0e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "infl.plot_influence();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55c97fef",
   "metadata": {},
   "outputs": [],
   "source": [
    "infl.plot_index(y_var=\"resid_studentized\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51ad5c18",
   "metadata": {},
   "source": [
    "A `summary_frame` shows the main influence and outlier measures for each observations.\n",
    "\n",
    "We have 1000 observations in our example which is too many to easily display. We can sort the summary dataframe by one of the columns and list the observations with the largest outlier or influence measure. In the example below, we sort by Cook's distance and by `standard_resid` which is the Pearson residual in the generic case.\n",
    "\n",
    "Because we simulated a \"nice\" model, there are no observations with large influence or that are large outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecba7d2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_infl = infl.summary_frame()\n",
    "df_infl.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2981abb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_infl.sort_values(\"cooks_d\", ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adce9aae",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_infl.sort_values(\"standard_resid\", ascending=False)[:10]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
