U md8y@sdZddlmZddlmZmZddlmZmZm Z ddl Z ddl m Z mZmZddlmZmZmZmZmZddlmZdd lmZdd lmZmZmZmZdd lm Z m!Z!m"Z"erdd l#m$Z$d ddddZ%dd dddZ&ddddddZ'ddddgZ(d d!d"d#d$d%d&d'd(d)d*d+d,d-gZ)d.d/d.d0d1d2Z*d.d d3d4d5d6Z+dd9d.d:d;d3d.dd d?d@Z,dd9d/d:d.d3d.ddd d|d}d~Z?d dd dddZ@dS)z$ Routines for filling missing data. ) annotations)partialwraps) TYPE_CHECKINGAnycastN)NaTalgoslib) ArrayLikeAxisAxisIntFnpt)import_optional_dependency)infer_dtype_from) is_array_likeis_numeric_v_string_likeis_object_dtypeneeds_i8_conversion)is_valid_na_for_dtypeisnana_value_for_dtype)Indexznpt.NDArray[np.bool_]int)masklengthcCs8t|r4t||kr,tdt|d|||}|S)zJ Validate the size of the values passed to ExtensionArray.fillna. z'Length of 'value' does not match. Got (z ) expected )rlen ValueError)valuerrr L/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/pandas/core/missing.pycheck_value_size1s r"r )arrreturnc Cst|\}}tj||d}d}t|r4d}t|}t|}||}tj|jtd}|D]b}t||rjqZ|rtj|jtj d} |||k| |<n"||k} t | tj s| j tdd} || O}qZ| r|t|O}|S)a  Return a masking array of same size/shape as arr with entries equaling any member of values_to_mask set to True Parameters ---------- arr : ArrayLike values_to_mask: list, tuple, or scalar Returns ------- np.ndarray[bool] )dtypeFT)r%Zna_value)rnparrayrrZzerosshapeboolrZbool_ isinstancendarrayZto_numpyany) r#Zvalues_to_maskr%Z potential_naZarr_maskZna_maskZnonnarxZnew_maskr r r! mask_missing@s,       r.Fz str | Noner))method allow_nearestcCsv|dkr dSt|tr8|}|dkr,d}n |dkr8d}ddg}d}|rV|dd}||krrtd |d ||S) N)NZasfreqZffillpadZbfillbackfillzpad (ffill) or backfill (bfill)nearestz(pad (ffill), backfill (bfill) or nearestzInvalid fill method. Expecting z. Got )r*strlowerappendr)r/r0Z valid_methodsZ expectingr r r!clean_fill_methodys   r7lineartimeindexvaluesr3zeroslinear quadraticcubic barycentrickroghspline polynomialfrom_derivativespiecewise_polynomialpchipakima cubicspliner4r)r/r:r$cKsh|d}|dkr"|dkr"tdtt}||krHtd|d|d|dkrd|jsdt|d|S) Norder)rBrCz7You must specify the order of the spline or polynomial.zmethod must be one of z. Got 'z ' instead.)rArErFz4 interpolation requires that the index be monotonic.)getr NP_METHODS SP_METHODSZis_monotonic_increasing)r/r:kwargsrIvalidr r r!clean_interp_methods rOz int | None)howis_validr$cCs|dks tt|dkrdS|jdkr2|jdd}|dkrL|dd}n&|dkrrt|d|ddd }||}|sdS|S) aG Retrieves the index of the first valid value. Parameters ---------- values : ndarray or ExtensionArray how : {'first', 'last'} Use this parameter to change between the first or last valid index. is_valid: np.ndarray Mask to find na_values. Returns ------- int or None )firstlastrNaxisrRrS)AssertionErrorrndimr,Zargmax)r;rPrQZidxposZ chk_notnar r r!find_valid_indexs    r[r1forward np.ndarrayr z Index | Nonez Any | NoneNone) datar/rWr:limitlimit_direction limit_area fill_valuecoercedowncastr$c Ksz t|} Wntk r$d} YnX| dk rR|dk r>tdt|| |||dn,|dk s^ttf||||||||d| dS)z Wrapper to dispatch to either interpolate_2d or _interpolate_2d_with_fill. Notes ----- Alters 'data' in-place. Nz&Cannot pass both fill_value and method)r/rWr`rb)r_r:rWr/r`rarbrc)r7rinterpolate_2drY_interpolate_2d_with_fill) r_r/rWr:r`rarbrcrdrerMmr r r!interpolate_array_2ds6    ri) r_r:rWr/r`rarbrcr$c  st|ft|jr(t|jdddkrFt|jsBtdddddg} | krvtd | d d d k rd dg} | krtd| ddtjd dt |dddfdd } t | ||d S)z Column-wise application of _interpolate_1d. Notes ----- Alters 'data' in-place. The signature does differ from _interpolate_1d because it only includes what is needed for Block.interpolate. F)compatr9zStime-weighted interpolation only works on Series or DataFrames with a DatetimeIndexr;r\backwardZbothz*Invalid limit_direction: expecting one of z, got 'z'.Ninsideoutsidez%Invalid limit_area: expecting one of z, got .)Znobsr`r]r^)yvaluesr$c s$tf|dddS)NF)indicesror/r`rarbrc bounds_error)_interpolate_1d)rorcrprMr`rbrar/r r!funcRs z'_interpolate_2d_with_fill..func) rOrr%rrrr5r Zvalidate_limit_index_to_interp_indicesr&apply_along_axis) r_r:rWr/r`rarbrcrMZvalid_limit_directionsZvalid_limit_areasrtr rsr!rgs4     rg)r:r/r$cCs`|j}t|jr|d}|dkr4|}ttj|}n(t|}|dkr\|jtjkr\t |}|S)zE Convert Index to ndarray of indices to pass to NumPy/SciPy. i8r8)r;r:) _valuesrr%viewrr&r+asarrayZobject_r Zmaybe_convert_objects)r:r/ZxarrZindsr r r!rujs     ru) rpror/r`rarbrcrqrIr$c Kst|} | } | sdS| r&dStt| } t|d| d} | dkrNd} tt| }t|d| d}|dkrxt|}ttd|t| }|dkr|tt | |dB}n.|dkr|tt | d|B}ntt | ||}|d kr|||BO}n|d kr| ||}||O}t |}t |j }|r0| d }|tkrpt|| }t|| || ||| ||| <n.t|| || || f||||d | || <|rtj||<n tj||<dS) a Logic for the 1-d interpolation. The input indices and yvalues will each be 1-d arrays of the same length. Bounds_error is currently hardcoded to False since non-scipy ones don't take it as an argument. Notes ----- Fills 'yvalues' in-place. NrRrPrQrrSrUr\rkrlrmrw)r/rcrqrI)rr,allsetr&Z flatnonzeror[ranger _interp_limitsortedrr%ryrKZargsortZinterp_interpolate_scipy_wrapperrrnan)rpror/r`rarbrcrqrIrMinvalidrNZall_nansZfirst_valid_indexZ start_nansZlast_valid_indexZend_nansZ preserve_nansZmid_nansZis_datetimelikeZindexerr r r!rrsh           rr)rqcKsx|d}td|dddlm} t|}| j| jttd} t|ddrb|j d | d }}|d krv| j | d <n"|d krt | d <n|d krt | d <d dddddg} || kr|dkr|}| j|||||d} | |} n|dkr&t|s|dkrtd|| j||fd|i|} | |} nN|jjs8|}|jjsJ|}|jjs\|}| |}||||f|} | S)z Passed off to scipy.interpolate.interp1d. method is scipy's kind. Returns an array interpolated at new_x. Add any new methods to the list in _clean_interp_method. z interpolation requires SciPy.scipy)extrar interpolate)r@rArDrEZ _is_all_datesFrwrFrGrHr3r<r=r>r?rC)kindrcrqrBz;order needs to be specified and greater than 0; got order: k)rrrr&rzZbarycentric_interpolateZkrogh_interpolate_from_derivativesgetattrrxZastypeZpchip_interpolate_akima_interpolate_cubicspline_interpolateZinterp1drrZUnivariateSplineflagsZ writeablecopy)r-yZnew_xr/rcrqrIrMrrZ alt_methodsZinterp1d_methodsZterpZnew_yr r r!rsf             rzint | list[int] | None)der extrapolatec Cs4ddlm}|jj}|||dd||d}||S)a Convenience function for interpolate.BPoly.from_derivatives. Construct a piecewise polynomial in the Bernstein basis, compatible with the specified values and derivatives at breakpoints. Parameters ---------- xi : array-like sorted 1D array of x-coordinates yi : array-like or list of array-likes yi[i][j] is the j-th derivative known at xi[i] order: None or int or array-like of ints. Default: None. Specifies the degree of local polynomials. If not None, some derivatives are ignored. der : int or list How many derivatives to extract; None for all potentially nonzero derivatives (that is a number equal to the number of points), or a list of derivatives to extract. This number includes the function value as 0th derivative. extrapolate : bool, optional Whether to extrapolate to ouf-of-bounds points based on first and last intervals, or to return NaNs. Default: True. See Also -------- scipy.interpolate.BPoly.from_derivatives Returns ------- y : scalar or array-like The result, of length R or length M or M by R. rrrXrU)Zordersr)rrZBPolyrDreshape) xiyir-rIrrrr/rhr r r!r9s$ r)rrWcCs(ddlm}|j|||d}|||dS)a[ Convenience function for akima interpolation. xi and yi are arrays of values used to approximate some function f, with ``yi = f(xi)``. See `Akima1DInterpolator` for details. Parameters ---------- xi : array-like A sorted list of x-coordinates, of length N. yi : array-like A 1-D array of real values. `yi`'s length along the interpolation axis must be equal to the length of `xi`. If N-D array, use axis parameter to select correct axis. x : scalar or array-like Of length M. der : int, optional How many derivatives to extract; None for all potentially nonzero derivatives (that is a number equal to the number of points), or a list of derivatives to extract. This number includes the function value as 0th derivative. axis : int, optional Axis in the yi array corresponding to the x-coordinate values. See Also -------- scipy.interpolate.Akima1DInterpolator Returns ------- y : scalar or array-like The result, of length R or length M or M by R, rrrV)nu)rrZAkima1DInterpolator)rrr-rrWrPr r r!rfs$ r not-a-knotzstr | tuple[Any, Any])rWbc_typecCs(ddlm}|j|||||d}||S)aq Convenience function for cubic spline data interpolator. See `scipy.interpolate.CubicSpline` for details. Parameters ---------- xi : array-like, shape (n,) 1-d array containing values of the independent variable. Values must be real, finite and in strictly increasing order. yi : array-like Array containing values of the dependent variable. It can have arbitrary number of dimensions, but the length along ``axis`` (see below) must match the length of ``x``. Values must be finite. x : scalar or array-like, shape (m,) axis : int, optional Axis along which `y` is assumed to be varying. Meaning that for ``x[i]`` the corresponding values are ``np.take(y, i, axis=axis)``. Default is 0. bc_type : string or 2-tuple, optional Boundary condition type. Two additional equations, given by the boundary conditions, are required to determine all coefficients of polynomials on each segment [2]_. If `bc_type` is a string, then the specified condition will be applied at both ends of a spline. Available conditions are: * 'not-a-knot' (default): The first and second segment at a curve end are the same polynomial. It is a good default when there is no information on boundary conditions. * 'periodic': The interpolated functions is assumed to be periodic of period ``x[-1] - x[0]``. The first and last value of `y` must be identical: ``y[0] == y[-1]``. This boundary condition will result in ``y'[0] == y'[-1]`` and ``y''[0] == y''[-1]``. * 'clamped': The first derivative at curves ends are zero. Assuming a 1D `y`, ``bc_type=((1, 0.0), (1, 0.0))`` is the same condition. * 'natural': The second derivative at curve ends are zero. Assuming a 1D `y`, ``bc_type=((2, 0.0), (2, 0.0))`` is the same condition. If `bc_type` is a 2-tuple, the first and the second value will be applied at the curve start and end respectively. The tuple values can be one of the previously mentioned strings (except 'periodic') or a tuple `(order, deriv_values)` allowing to specify arbitrary derivatives at curve ends: * `order`: the derivative order, 1 or 2. * `deriv_value`: array-like containing derivative values, shape must be the same as `y`, excluding ``axis`` dimension. For example, if `y` is 1D, then `deriv_value` must be a scalar. If `y` is 3D with the shape (n0, n1, n2) and axis=2, then `deriv_value` must be 2D and have the shape (n0, n1). extrapolate : {bool, 'periodic', None}, optional If bool, determines whether to extrapolate to out-of-bounds points based on first and last intervals, or to return NaNs. If 'periodic', periodic extrapolation is used. If None (default), ``extrapolate`` is set to 'periodic' for ``bc_type='periodic'`` and to True otherwise. See Also -------- scipy.interpolate.CubicHermiteSpline Returns ------- y : scalar or array-like The result, of shape (m,) References ---------- .. [1] `Cubic Spline Interpolation `_ on Wikiversity. .. [2] Carl de Boor, "A Practical Guide to Splines", Springer-Verlag, 1978. rr)rWrr)rrZ CubicSpline)rrr-rWrrrrr r r!rsM r)r;r/r`rbr$cCst|}|}|st|d|d}|dkr0d}t|d|d}|dkrNt|}t|||d|dkrvd|||d <n$|d krd|d|<||d d<tj||<dS) a Apply interpolation and limit_area logic to values along a to-be-specified axis. Parameters ---------- values: np.ndarray Input array. method: str Interpolation method. Could be "bfill" or "pad" limit: int, optional Index limit on interpolation. limit_area: str Limit area for interpolation. Can be "inside" or "outside" Notes ----- Modifies values in-place. rRr{NrrS)r/r`rlFrUrm)rr|r[rrfr&r)r;r/r`rbrrQrRrSr r r!_interpolate_with_limit_areas&rr )r;r/rWr`rbr$cCs|dk r&ttt|||d||dS|dkr6ddndd}|jdkrl|dkrXtd|td |j}t |}||}|d krt ||d n t ||d dS) a Perform an actual interpolation of values, values will be make 2-d if needed fills inplace, returns the result. Parameters ---------- values: np.ndarray Input array. method: str, default "pad" Interpolation method. Could be "bfill" or "pad" axis: 0 or 1 Interpolation axis limit: int, optional Index limit on interpolation. limit_area: str, optional Limit area for interpolation. Can be "inside" or "outside" Notes ----- Modifies values in-place. N)r/r`rbrcSs|SNr r-r r r!Iz interpolate_2d..cSs|jSr)Trr r r!rIrrUz0cannot interpolate on a ndim == 1 with axis != 0)rUr1r`) r&rvrrrZrYrtupler(r7_pad_2d _backfill_2d)r;r/rWr`rbZtransfZtvaluesr r r!rfs.   rfznpt.NDArray[np.bool_] | None)rr$cCs |dkrt|}|tj}|Sr)rryr&Zuint8r;rr r r! _fillna_prep]s rr)rtr$cs tdfdd }tt|S)z> Wrapper to handle datetime64 and timedelta64 dtypes. NcsPt|jrB|dkrt|}|d||d\}}||j|fS|||dS)Nrw)r`r)rr%rry)r;r`rresultrtr r!new_funcns  z&_datetimelike_compat..new_func)NN)rrr)rtrr rr!_datetimelike_compatis rz(tuple[np.ndarray, npt.NDArray[np.bool_]])r;r`rr$cCs"t||}tj|||d||fSNr)rr Z pad_inplacer;r`rr r r!_pad_1d}s rcCs"t||}tj|||d||fSr)rr Zbackfill_inplacerr r r! _backfill_1ds rrcCs0t||}t|jr(tj|||dn||fSr)rr&r|r(r Zpad_2d_inplacerr r r!rs  r)rcCs0t||}t|jr(tj|||dn||fSr)rr&r|r(r Zbackfill_2d_inplacerr r r!rs  rr1r2rU)rZcCs&t|}|dkrt|Sttd|S)NrUr)r7 _fill_methodsrr)r/rZr r r! get_fill_funcsr)r$cCs t|ddS)NT)r0)r7)r/r r r!clean_reindex_fill_methodsr)rcst|t}t}fdd}|dk rN|dkrDtt|d}n |||}|dk r|dkrb|St||ddd|}tdt|}|dkr|S||@S)ak Get indexers of values that won't be filled because they exceed the limits. Parameters ---------- invalid : np.ndarray[bool] fw_limit : int or None forward limit to index bw_limit : int or None backward limit to index Returns ------- set of indexers Notes ----- This is equivalent to the more readable, but slower .. code-block:: python def _interp_limit(invalid, fw_limit, bw_limit): for x in np.where(invalid)[0]: if invalid[max(0, x - fw_limit):x + bw_limit + 1].all(): yield x cs`t|}t||dd}tt|d|tt|d|ddkdB}|S)NrUr)min_rolling_windowr|r}r&whereZcumsum)rr`ZwindowedidxNr r!inners  "z_interp_limit..innerNrrXrU)rr}r&rlistrz)rZfw_limitZbw_limitZf_idxZb_idxrZ b_idx_invr rr!rs   r)awindowr$cCsJ|jdd|jd|d|f}|j|jdf}tjjj|||dS)z [True, True, False, True, False], 2 -> [ [True, True], [True, False], [False, True], [True, False], ] NrXrU)r(strides)r(rr&r Z stride_tricksZ as_strided)rrr(rr r r!rs $r)F) r1rNNr\NNFN)r8Nr\NN)r8Nr\NNFN)NFN)NrF)rr)rrN)r1rNN)N)NN)NN)NN)NN)rU)A__doc__ __future__r functoolsrrtypingrrrnumpyr&Z pandas._libsrr r Zpandas._typingr r r rrZpandas.compat._optionalrZpandas.core.dtypes.castrZpandas.core.dtypes.commonrrrrZpandas.core.dtypes.missingrrrZpandasrr"r.r7rKrLrOr[rirgrurrrrrrrrfrrrrrrrrrrrr r r r!s    9 +$9 R"qN-/V1H     ?