U mdV@sddlZddlmZddlZddlZddlmZ ddl m Z ddl m ZddlmZmZddlmZmZddlmZd d lmZd d lmZd d lmZd!e eeeeeeeeeeej d ddZ!ddej"dddddfe eeeeeeeeeeeeeedej d ddZ#dddej"dddddddddf e eeeeeeeeeeeeeeeedeeeeeeej ddd Z$dS)"N)Optional)AnnData)logging)settings Verbosity)sanitize_anndatacheck_nonnegative_integers)Literal) _get_mean_var)materialize_as_ndarray) filter_genesT333333?F) adatalayer n_top_genes batch_key check_valuesspansubsetinplacereturnc%Cszddlm}Wntk r,tdYnXtj|jd} |dk rN|j|n|j} |rlt| slt dt t | \| d<| d<|dkrt tj|jdtd } n |j|j} g} t| D]} | | | k}t |\}}|dk}tj| jd tjd }t||}t||}||||d d }||jj||<td |}|tj}|jd}t|}|||}t|rt|}|j ||j!k}||j!||j |<t"|#d j$dd}t"|j$dd}n>t%||j}t&|||k|t'|j$dd}|j$dd}d |d t'||t'||d ||}| (|)d dqtj*| dd} tj+tj+| d dd d} | tj,} tj$| |ktdd}!tj-| | |k<tj./| }"tj.j0|"dd1tj-}#|!| d<|#| d<tj2| dd| d<| ddgj3ddgddgddj4}$d| d<d| j5|$dt|df<|sT|rddi|j6d<t78d| dj|j9d<| dj|j9d<| dj|j9d<| dj|j9d<| djjddd|j9d<|dk r| dj|j9d<|r|:| djn|dkr| j;dgd d} | SdS)a& See `highly_variable_genes`. For further implementation details see https://www.overleaf.com/read/ckptrbgzzzpg Returns ------- Depending on `inplace` returns calculated metrics (:class:`~pd.DataFrame`) or updates `.var` with the following fields: highly_variable : bool boolean indicator of highly-variable genes. **means** means per gene. **variances** variance per gene. **variances_norm** normalized variance per gene, averaged in the case of multiple batches. highly_variable_rank : float Rank of the gene according to normalized variance, median rank in the case of multiple batches. highly_variable_nbatches : int If batch_key is given, this denotes in how many batches genes are detected as HVG. r)loesszAPlease install skmisc package via `pip install --user scikit-misc)indexNzI`flavor='seurat_v3'` expects raw count data, but non-integers were found.meansZ variances)Zdtyper r)rZdegree Zaxishighly_variable_nbatchesZhighly_variable_rankZvariances_normTFlast) ascending na_positionhighly_variableflavor seurat_v3hvgzadded 'highly_variable', boolean vector (adata.var) 'highly_variable_rank', float vector (adata.var) 'means', float vector (adata.var) 'variances', float vector (adata.var) 'variances_norm', float vector (adata.var)float64copy) `adata.n_var`, returning all genes.zb`n_top_genes` > number of normalized dispersions, returning all genes with normalized dispersions.zthe z; top genes correspond to a normalized dispersion cutoff of r$)/r/r0Zuns_keysrQr4logexpm1r r rKrir,r-cutgroupbyrNZstdZisnullwherer9tolistlenrRdebugZ statsmodelsrmZr_infZ percentileZarangerMr1catch_warnings simplefilterapplyZmad ValueErrorisnansortZn_varsinfosizer2r3Z nan_to_num logical_andreduce)rrrdrerfrgrrhr%r0rNrTZ dispersionrVZ disp_groupedZ disp_mean_binZ disp_std_binZone_gene_per_binZ gen_indicesrmZdisp_median_binZ disp_mad_bindispersion_normZ disp_cut_off gene_subsetr]r]r^#_highly_variable_genes_single_batchs            *           r)rbrcr&)rrrrdrerfrgrrhr%rrrrrc Cs|dk r,tdd||||fDs,tdtd}t|tsHtd| dkrht|||| | || | dS| dkrt||||||||| d }nZt||j | j j }g}|j }|D]}||j | |k}t jtjt|d d d d }W5QRX|dd|f}t||||||||| d }tjtt|t|jf|jd}|dt|d<|||d<|j j|d<tj||gdd}tt|d t|d f}|j t!|}|"|qtj|d d}|dt#|d<|$d%t&tj'tj'tj'tj(d}|j)t&dddd|dt|k|d<|dk r|j*ddgd dddt|j+d }d|d|<|t|d<|j |j ddf}nV|j |j }|j,j}d |t-|<tj./|j0|k|j0|k|j,|k|j,|kf}||d<tjd|d| s| rd| i|j1d<t2d |dj|j3d<|d!j|j3d!<|d"j|j3d"<|djjd#d d$|j3d<| dk r|dj|j3d<|dj|j3d<| r|4|djn|SdS)%uJ Annotate highly variable genes [Satija15]_ [Zheng17]_ [Stuart19]_. Expects logarithmized data, except when `flavor='seurat_v3'`, in which count data is expected. Depending on `flavor`, this reproduces the R-implementations of Seurat [Satija15]_, Cell Ranger [Zheng17]_, and Seurat v3 [Stuart19]_. For the dispersion-based methods ([Satija15]_ and [Zheng17]_), the normalized dispersion is obtained by scaling with the mean and standard deviation of the dispersions for genes falling into a given bin for mean expression of genes. This means that for each bin of mean expression, highly variable genes are selected. For [Stuart19]_, a normalized variance for each gene is computed. First, the data are standardized (i.e., z-score normalization per feature) with a regularized standard deviation. Next, the normalized variance is computed as the variance of each gene after the transformation. Genes are ranked by the normalized variance. See also `scanpy.experimental.pp._highly_variable_genes` for additional flavours (e.g. Pearson residuals). Parameters ---------- adata The annotated data matrix of shape `n_obs` × `n_vars`. Rows correspond to cells and columns to genes. layer If provided, use `adata.layers[layer]` for expression values instead of `adata.X`. n_top_genes Number of highly-variable genes to keep. Mandatory if `flavor='seurat_v3'`. min_mean If `n_top_genes` unequals `None`, this and all other cutoffs for the means and the normalized dispersions are ignored. Ignored if `flavor='seurat_v3'`. max_mean If `n_top_genes` unequals `None`, this and all other cutoffs for the means and the normalized dispersions are ignored. Ignored if `flavor='seurat_v3'`. min_disp If `n_top_genes` unequals `None`, this and all other cutoffs for the means and the normalized dispersions are ignored. Ignored if `flavor='seurat_v3'`. max_disp If `n_top_genes` unequals `None`, this and all other cutoffs for the means and the normalized dispersions are ignored. Ignored if `flavor='seurat_v3'`. span The fraction of the data (cells) used when estimating the variance in the loess model fit if `flavor='seurat_v3'`. n_bins Number of bins for binning the mean gene expression. Normalization is done with respect to each bin. If just a single gene falls into a bin, the normalized dispersion is artificially set to 1. You'll be informed about this if you set `settings.verbosity = 4`. flavor Choose the flavor for identifying highly variable genes. For the dispersion based methods in their default workflows, Seurat passes the cutoffs whereas Cell Ranger passes `n_top_genes`. subset Inplace subset to highly-variable genes if `True` otherwise merely indicate highly variable genes. inplace Whether to place calculated metrics in `.var` or return them. batch_key If specified, highly-variable genes are selected within each batch separately and merged. This simple process avoids the selection of batch-specific genes and acts as a lightweight batch correction method. For all flavors, genes are first sorted by how many batches they are a HVG. For dispersion-based flavors ties are broken by normalized dispersion. If `flavor = 'seurat_v3'`, ties are broken by the median (across batches) rank based on within-batch normalized variance. check_values Check if counts in selected layer are integers. A Warning is returned if set to True. Only used if `flavor='seurat_v3'`. Returns ------- Depending on `inplace` returns calculated metrics (:class:`~pandas.DataFrame`) or updates `.var` with the following fields highly_variable : bool boolean indicator of highly-variable genes **means** means per gene **dispersions** For dispersion-based flavors, dispersions per gene **dispersions_norm** For dispersion-based flavors, normalized dispersions per gene **variances** For `flavor='seurat_v3'`, variance per gene **variances_norm** For `flavor='seurat_v3'`, normalized variance per gene, averaged in the case of multiple batches highly_variable_rank : float For `flavor='seurat_v3'`, rank of the gene according to normalized variance, median rank in the case of multiple batches highly_variable_nbatches : int If batch_key is given, this denotes in how many batches genes are detected as HVG highly_variable_intersection : bool If batch_key is given, this denotes the genes that are highly variable in all batches Notes ----- This function replaces :func:`~scanpy.pp.filter_genes_dispersion`. Ncss|]}|dkVqdS)Nr]).0mr]r]r^ sz(highly_variable_genes..z3If you pass `n_top_genes`, all cutoffs are ignored.z extracting highly variable geneszv`pp.highly_variable_genes` expects an `AnnData` argument, pass `inplace=False` if you want to return a `pd.DataFrame`.r&)rrrrrrr)rrdrerfrgrrhr%r F)Z min_cellsrr)columnsr$ZgeneT)Z ignore_indexr)rrkrlr$r )r$)rrZhighly_variable_intersectionrlr!)r"r#rz finished)timer%r'zadded 'highly_variable', boolean vector (adata.var) 'means', float vector (adata.var) 'dispersions', float vector (adata.var) 'dispersions_norm', float vector (adata.var)rrkrJr))5allrRr isinstancerr}r_rrr8cat categoriesr.r verbosityoverridererrorrr,r-r4r5rFrwrr?boolr9concatrHrurPrIrGr7rtZaggdictZnanmeanZnansumrenamerOr6rlr~rrrrQrSrTrU)rrrrdrerfrgrrhr%rrrrstartrVZbatchesZ gene_listbatchZ adata_subsetZfiltr'Z missing_hvgZidxsZhigh_varrrr]r]r^highly_variable_genes'sw         $            r)NrNTrFT)%r1typingrnumpyr4Zpandasr,Z scipy.sparsesparser@ZanndatarrrRZ _settingsrr_utilsrr Z_compatr r Z _distributedr _simplerstrr7rfloatr-r_ryrrr]r]r]r^s         " x