U mdV@sddlmZddlmZmZmZddlmZddlm Z ddl Z dddZ ddd Z dd d Zd d ZddZdddZdddZifddZGdddZGdddeZdS))RegularizedResults)_calc_nodewise_row_calc_nodewise_weight_calc_approx_inv_cov)LikelihoodModelResults)OLSNcCs|dkrtd|jf|jS)aestimates the regularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized Returns ------- An array of the parameters for the regularized fit NzD_est_regularized_naive currently requires that fit_kwds not be None.) ValueErrorfit_regularizedparamsmodpnum partitionsfit_kwdsr`/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/statsmodels/base/distributed_estimation.py_est_regularized_naiveKsrcCs|dkrtd|jf|jS)aestimates the unregularized fitted parameters. Parameters ---------- mod : statsmodels model class instance The model for the current partition. pnum : scalar Index of current partition partitions : scalar Total number of partitions fit_kwds : dict-like or None Keyword arguments to be given to fit Returns ------- An array of the parameters for the fit NzF_est_unregularized_naive currently requires that fit_kwds not be None.)rfitr r rrr_est_unregularized_naiveesrcCsNt|d}t|}t|}|D] }||7}q"||}d|t||k<|S)a joins the results from each run of _est__naive and returns the mean estimate of the coefficients Parameters ---------- params_l : list A list of arrays of coefficients. threshold : scalar The threshold at which the coefficients will be cut. r)lennpzerosabs)Zparams_l thresholdpr params_mnr rrr _join_naives   rcCs*|jt|f| }||d|7}|S)acalculates the log-likelihood gradient for the debiasing Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. alpha : scalar or array_like The penalty weight. If a scalar, the same penalty weight applies to all variables in the model. If a vector, it must have the same length as `params`, and contains a penalty weight for each coefficient. L1_wt : scalar The fraction of the penalty given to the L1 penalty term. Must be between 0 and 1 (inclusive). If 0, the fit is a ridge fit, if 1 it is a lasso fit. score_kwds : dict-like or None Keyword arguments for the score function. Returns ------- An array-like object of the same dimension as params Notes ----- In general: gradient l_k(params) where k corresponds to the index of the partition For OLS: X^T(y - X^T params) )Zscorerasarray)r r alphaL1_wt score_kwdsgradrrr _calc_grads&r#cCs0t|jt|f|}|dddf|jS)acalculates the weighted design matrix necessary to generate the approximate inverse covariance matrix Parameters ---------- mod : statsmodels model class instance The model for the current partition. params : array_like The estimated coefficients for the current partition. hess_kwds : dict-like or None Keyword arguments for the hessian function. Returns ------- An array-like object, updated design matrix, same dimension as mod.exog N)rsqrtZhessian_factorrexog)r r hess_kwdsZrhessrrr_calc_wdesign_matsr'cCs|dkr in|}|dkrin|}|dkr2tdn|d}d|krL|d}nd}|jj\}} ttd| |} |jf|j} t|| ||||} t || |} g}g}t || t |d| | D]2}t | ||}| |t| |||}| |q| | ||fS)aestimates the regularized fitted parameters, is the default estimation_method for class DistributedModel. Parameters ---------- mod : statsmodels model class instance The model for the current partition. mnum : scalar Index of current partition. partitions : scalar Total number of partitions. fit_kwds : dict-like or None Keyword arguments to be given to fit_regularized score_kwds : dict-like or None Keyword arguments for the score function. hess_kwds : dict-like or None Keyword arguments for the Hessian function. Returns ------- A tuple of parameters for regularized fit An array-like object of the fitted parameters, params An array-like object for the gradient A list of array like objects for nodewise_row A list of array like objects for nodewise_weight NzG_est_regularized_debiased currently requires that fit_kwds not be None.rr rg?)rr%shapeintrceilr r r#r'rangeminrappendr)r Zmnumrrr!r&rr ZnobsrZp_partr r"Zwexognodewise_row_lnodewise_weight_lidxZ nodewise_rowZnodewise_weightrrr_est_regularized_debiaseds.        r1c Cst|dd}t|}t|}t|}g}g}|D]8}||d7}||d7}||d||dq8t|}t|}||}|d|9}t||} || |} d| t| |k<| S)ajoins the results from each run of _est_regularized_debiased and returns the debiased estimate of the coefficients Parameters ---------- results_l : list A list of tuples each one containing the params, grad, nodewise_row and nodewise_weight values for each partition. threshold : scalar The threshold at which the coefficients will be cut. rrg)rrrextendarrayrdotr) results_lrrrrZgrad_mnr.r/rZapprox_inv_covZdebiased_paramsrrr_join_debiaseds&         r9c CsF|j}|||j||f|}|j|||jfd|i|j}|S)ahandles the model fitting for each machine. NOTE: this is primarily handled outside of DistributedModel because joblib cannot handle class methods. Parameters ---------- self : DistributedModel class instance An instance of DistributedModel. pnum : scalar index of current partition. endog : array_like endogenous data for current partition. exog : array_like exogenous data for current partition. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_e : dict-like Additional init_kwds to add for each partition. Returns ------- estimation_method result. For the default, _est_regularized_debiased, a tuple. r) init_kwdscopyupdate model_classestimation_methodrestimation_kwds) selfr endogr%r init_kwds_eZtemp_init_kwdsmodelresultsrrr_helper_fit_partitionHs  rEc@s8eZdZdZd ddZd ddZddd Zdd d ZdS)DistributedModela Distributed model class Parameters ---------- partitions : scalar The number of partitions that the data will be split into. model_class : statsmodels model class The model class which will be used for estimation. If None this defaults to OLS. init_kwds : dict-like or None Keywords needed for initializing the model, in addition to endog and exog. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS estimation_method : function or None The method that performs the estimation for each partition. If None this defaults to _est_regularized_debiased. estimation_kwds : dict-like or None Keywords to be passed to estimation_method. join_method : function or None The method used to recombine the results from each partition. If None this defaults to _join_debiased. join_kwds : dict-like or None Keywords to be passed to join_method. results_class : results class or None The class of results that should be returned. If None this defaults to RegularizedResults. results_kwds : dict-like or None Keywords to be passed to results class. Attributes ---------- partitions : scalar See Parameters. model_class : statsmodels model class See Parameters. init_kwds : dict-like See Parameters. init_kwds_generator : generator or None See Parameters. estimation_method : function See Parameters. estimation_kwds : dict-like See Parameters. join_method : function See Parameters. join_kwds : dict-like See Parameters. results_class : results class See Parameters. results_kwds : dict-like See Parameters. Notes ----- Examples -------- Nc Cs||_|dkrt|_n||_|dkr,i|_n||_|dkrBt|_n||_|dkrXi|_n||_|dkrnt|_n||_|dkri|_ n||_ |dkrt |_ n||_ | dkri|_ n| |_ dSN) rrr=r:r1r>r?r9 join_method join_kwdsr results_class results_kwds) r@rr=r:r>r?rHrIrJrKrrr__init__s2zDistributedModel.__init__ sequentialc Cs|dkr i}|dkr$||||}n&|dkr>|||||}n td||j|f|j}|jdgdgf|j}|j||f|jS)aePerforms the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like or None Keywords needed for the model fitting. parallel_method : str type of distributed estimation to be used, currently "sequential", "joblib" and "dask" are supported. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. NrMZjoblibz.parallel_method: %s is currently not supportedr) fit_sequential fit_joblibrrHrIr=r:rJrK) r@data_generatorrZparallel_methodparallel_backendinit_kwds_generatorr7r Zres_modrrrrs"zDistributedModel.fitc Csg}|dkr>t|D]&\}\}}t|||||}||qn@tt||} | D],\}\\}}} t|||||| }||qP|S)a*Sequentially performs the distributed estimation using the corresponding DistributedModel Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. N) enumeraterEr-zip) r@rPrrRr7r rAr%rDtup_genrBrrrrNs"   zDistributedModel.fit_sequentialc sddlm}|tj\}}|dkrN|dkrN|fddt|D}n|dk r|dkr|$|fddt|D}W5QRXn|dkr|dk rtt||} |fdd| D}nL|dk r|dk rtt||} | |fdd| D}W5QRX|S) aPerforms the distributed estimation in parallel using joblib Parameters ---------- data_generator : generator A generator that produces a sequence of tuples where the first element in the tuple corresponds to an endog array and the element corresponds to an exog array. fit_kwds : dict-like Keywords needed for the model fitting. parallel_backend : None or joblib parallel_backend object used to allow support for more complicated backends, ex: dask.distributed init_kwds_generator : generator or None Additional keyword generator that produces model init_kwds that may vary based on data partition. The current usecase is for WLS and GLS Returns ------- join_method result. For the default, _join_debiased, it returns a p length array. r) parallel_funcNc3s&|]\}\}}|||VqdSrGr.0r rAr%frr@rr cs z.DistributedModel.fit_joblib..c3s&|]\}\}}|||VqdSrGrrWrYrrr[is c3s,|]$\}\\}}}||||VqdSrGrrXr rAr%r:rYrrr[osc3s,|]$\}\\}}}||||VqdSrGrr\rYrrr[vs)Zstatsmodels.tools.parallelrVrErrSrT) r@rPrrQrRrVparZn_jobsr7rUrrYrrODs.   zDistributedModel.fit_joblib)NNNNNNNN)NrMNN)N)N)__name__ __module__ __qualname____doc__rLrrNrOrrrrrFms$? / : 0rFcs(eZdZdZfddZddZZS)DistributedResultsaT Class to contain model results Parameters ---------- model : class instance Class instance for model used for distributed data, this particular instance uses fake data and is really only to allow use of methods like predict. params : ndarray Parameter estimates from the fit model. cstt|||dSrG)superrbrL)r@rCr  __class__rrrLszDistributedResults.__init__cOs|jj|j|f||S)aCalls self.model.predict for the provided exog. See Results.predict. Parameters ---------- exog : array_like NOT optional The values for which we want to predict, unlike standard predict this is NOT optional since the data in self.model is fake. *args : Some models can take additional arguments. See the predict method of the model for the details. **kwargs : Some models can take additional keywords arguments. See the predict method of the model for the details. Returns ------- prediction : ndarray, pandas.Series or pandas.DataFrame See self.model.predict )rCpredictr )r@r%argskwargsrrrrfszDistributedResults.predict)r^r_r`rarLrf __classcell__rrrdrrb}s rb)N)N)r)NNN)r)Zstatsmodels.base.elastic_netrZ(statsmodels.stats.regularized_covariancerrrZstatsmodels.base.modelrZ#statsmodels.regression.linear_modelrnumpyrrrrr#r'r1r9rErFrbrrrrs(   E   + A . %