U md'2@s<ddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddl&m'Z'm(Z(ddlm)Z)m*Z*ddlm+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3e e"e%gZ4d d!d"Z5d#d$Z6d%d&Z7d'd(Z8d)d*Z9ej:;d+d,d gd-d.Zd3d4Z?d5d6Z@d7d8ZAd9d:ZBd;d<ZCd=d>ZDd?d@ZEdAdBZFdCdDZGej:;dEeHddFddGgddFdGgfeHddFddGgdfdHdIdHdJgdHdIdJgfdHdIdHdJgdfgdKdLZIej:;dEeHddGddGgddFdGgfeHdHdMdHdMgdHdIdMgfgdNdOZJej:;dEeHddFdGdGgdfdHdIdJdJgdfddFdGdGgddFdGgfdHdIdJdJgdHdIdJgfgdPdQZKej:;dRdSdTdUgdVdWZLdXdYZMej:;dZd[eHddFdGdGgddGdgfd[eHdHdIdJdJgdHdHdIgfd\eHddGddGgdfd]eHdHdIdJdJgdHdJdIgfd^eHddFdGdGgddFgfd^eHdHdIdJdJgdHdIgfd_eHddFdGdGgddFdGd`gfd_eHdHdIdJdJgdHdIdJdMgfdaeHdHdIdJdbgdHdIdJgfdaeHdHdIdJdMgdHdIdJgfdaeHddFdGd`gddFdGgfg ej:;dcdddegdfdgZNej:;dhdidjdddkfdldjdedkfdmddgdnfdodddpdqfdrdcdsifdtifgdudvZOdwdxZPej:;dye4dzd{ZQej:;dye4d|d}ZRej:;dye4d~dZSddZTddZUddZVddZWddZXddZYej:;ddddFgddpdFgdgdgfdddFgdddpgdgdgfdddFgdpddFgdgdgfdddFgddpdgdgdgfddFdgddpdFgdpgdgfddFdgdddpgdpgdgfddFdgdpddFgdpgdgfddFdgddpdgdpgdgfddFdFgddpdFgdgdgfddFdFgdddpgdgdgfddFdFgdpddFgdgdgfddFdFgddpdgdgdgfdFddgddpdFgdFdFdpgddFdFgfdFddgdddpgdFdFdpgddFdFgfdFddgdpddFgdFdFdpgddFdFgfdFddgddpdgdFdFdpgddFdFgfdFddFgddpdFgdFdFdgddpdpgfdFddFgdddpgdFdFdgddpdpgfdFddFgdpddFgdFdFdgddpdpgfdFddFgddpdgdFdFdgddpdpgfgddZZej:;ddFdgdpdpgdFgdgfddFgdpdpgdFgdgfdddFgddpdpgdpgdgfddFdgddpdpgdpgdgfddFdFgddpdpgdgdgfdFddgddpdpgdFgdgfdFddFgddpdpgdFgdgfdFdFdgddpdpgdFgdgfgddZ[ddZ\ej:;ddddpddFgddZ]ej:;dddddddFgdddddFdFgddddFdFdFgdddFdFdFdFgddFdFdFdFdFggddZ^ej:;dddFgddpdFgdfddFdFgddpgdfdddgddpdFgdfdFdFdFgddpdFgdfdddgdddgdfgddZ_ddZ`ddZaddZbddZcddZdddZeddZfdddZgej:;deaeeedebfej:;de!effddZhddZiej:;ddej:;dd¡ej:;dejdFddńZkddDŽZlddɄZmdd˄Znej:;ddFddFgddpdpgfdFddFgddpdpggfdFddFggddpdpgfgdd΄ZoddЄZpdd҄ZqddԄZrddքZsdd؄ZtddڄZudd܄ZvddބZwddZxej:;dd,d gddZyddZzddZ{ddZ|ej:;dddFdGd`gdFdfddFdGd`gdGdpfddFdGd`gd`dfgddZ}ej:;deHdddFdFgdFdFfeHddFddFgdFdpfeHddFddFgdGdFfeHddddgdFdFfeHddddgdFdpfeHddddgdGdFfgddZ~ej:;deHddFdFdGgdddFdGd`gfeHddFdFdFgdpddFdGd`gfeHdFdFdFdFgdpddFdGd`gfeHdHdbdbdHgddHdIdMdbgfgej:;dd,d gddZddZej:;dddFdGd`gdFdfddFdGd`gdGdpfddFdGd`gd`dFfgddZej:;dddFdGd`gdfddFdGd`gdfgddZej:;ddddFdGgdddgdddgdddgddpdggddfddFdGd`gdddgdddgdddgddpdggddfdJdJdHdIgdddgdddgdddgddpdggdHdIdJdJgdfdJdJdHdIgdddgdddgdddgddpdggdHdJdIgdfdddFdGgdddgdddgdddgddpdggddFdGd`gdfdddFdGgdddgdddgdddgddpdggddFd`gdfddFgdpddgdddggddfgd d Zd d ZdS(N) csr_matrix)stats)datasets)svm)softmax)make_multilabel_classification)_sparse_random_matrix) check_arraycheck_consistent_length)check_random_state)assert_allcloseassert_almost_equal)assert_array_equal)assert_array_almost_equal)accuracy_scoreauc)average_precision_score)coverage_error) det_curve)%label_ranking_average_precision_score)precision_recall_curve)label_ranking_loss) roc_auc_score) roc_curve)_ndcg_sample_scores_dcg_sample_scores) ndcg_score dcg_score)top_k_accuracy_score)UndefinedMetricWarning)train_test_split)LogisticRegression)label_binarizeFc Cs|dkrt}|j}|j}|r:||dk||dk}}|j\}}t|}td}||||||}}t |d}tj d}tj || |d|f}tjdddd} | |d||d|||d} |r| ddd f} | ||d} ||d} | | | fS) zMake some classification predictions on a toy dataset using a SVC If binary is True restrict to a binary classification problem instead of a multiclass classification problem N%rZlinearT)ZkernelZ probability random_state)rZ load_irisdatatargetshapenparanger shuffleintrandom RandomStateZc_ZrandnrZSVCfit predict_probaZpredict) ZdatasetbinaryXy n_samples n_featuresprngZhalfclfy_scorey_predy_truer@[/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/sklearn/metrics/tests/test_ranking.pymake_prediction4s*     * rBcCsdt|d}|||k}|||k}|dd|dd}t|dk}|tt|t|S)zKAlternative implementation to check for correctness of `roc_auc_score`.r)r)r-uniquereshapesumfloatlen)r?r= pos_labelposnegZ diff_matrixZ n_correctr@r@rA_aucds   rLc Cst|d}t||k}t|ddd}||}||}d}tt|D]P}|||krPd}td|dD]}|||krr|d7}qr||d}||7}qP||S)a>Alternative implementation to check for correctness of `average_precision_score`. Note that this implementation fails on some edge cases. For example, for constant predictions e.g. [0.5, 0.5, 0.5], y_true = [1, 0, 0] returns an average precision of 0.33... but y_true = [0, 0, 1] returns 1.0. r)NrCr?)r-rDrFZargsortrangerH) r?r=rIZn_posorderscoreiprecjr@r@rA_average_precisionss      rTcCsdt||\}}}tt|}tt|}d}tdt|D]$}|||||||d7}q:|S)aoA second alternative implementation of average precision that closely follows the Wikipedia article's definition (see References). This should give identical results as `average_precision_score` for all inputs. References ---------- .. [1] `Wikipedia entry for the Average precision `_ rr))rlistreversedrNrH)r?r= precisionZrecall thresholdZaverage_precisionrQr@r@rA_average_precision_slows   "rYc Cs^dd}||||\}}t||}d}|}d||||} ||} dd|| | | S)zcAlternative implementation to check for correctness of `roc_auc_score` with `max_fpr` set. c Sst||\}}}|||k}t||}|||k}t||k}|d} || ||g} || ||g} t|t|| | }||fS)Nr))rr-appendargmaxZinterp) r? y_predictmax_fprfprtpr_new_fprnew_tprZidx_outZidx_inZx_interpZy_interpr@r@rA _partial_rocs   z,_partial_roc_auc_score.._partial_rocr?r)r) r?r\r]rcrarbZ partial_aucZfpr1Zfpr2Zmin_areaZmax_arear@r@rA_partial_roc_auc_scores  redropTc Csztdd\}}}t||}t|||d\}}}t||}t||ddt|t|||j|jksft|j|jksvtdS)NTr5Zdrop_intermediater%decimal) rBrLrrrrrr,AssertionError) rfr?r`r=Z expected_aucr^r_ thresholdsroc_aucr@r@rAtest_roc_curves  rncCstjd}tdgddgd}|jddd}t||dd\}}}|ddksXt|d dksht|j|jksxt|j|jkstdS) Nr2r)dsizeTrhrC)r-r1r2arrayrandintrrkr,)r;r?r>r^r_Zthrr@r@rAtest_roc_curve_end_pointss rvc Cstdd\}}}t||\}}}g}|D]2}t||k|@}t|} |d|| q(t||dd|j|jkszt|j|jkstdS)NTrgrMr%ri)rBrr-rFrZrr,rk) r?r`r=r^r_rlZ tpr_correctttpr:r@r@rAtest_roc_returns_consistencys ryc Cs4tdd\}}}ttt||W5QRXdS)NFrg)rBpytestraises ValueErrorr)r?r`r=r@r@rAtest_roc_curve_multis r}cCs`tdd\}}}t||d\}}}t||}t|ddd|j|jksLt|j|jks\tdS)NTrgrd?r%ri)rBrrrr,rk)r?r`r=r^r_rlrmr@r@rAtest_roc_curve_confidences  rcCstdd\}}}t|j}t||\}}}t||}t|ddd|j|jksTt|j|jksdtt|j}t||\}}}t||}t|ddd|j|jkst|j|jkstt||\}}}t||}t|ddd|j|jkst|j|jkstdS)NTrgrdr%rig(\?) rBr-onesr,rrrrkzeros)r?predr=Z trivial_predr^r_rlrmr@r@rAtest_roc_curve_hards$     rc Cs ddddddddddg }ddddddddddg }d}tjt|dt||\}}}W5QRXt|tt|tj|j |j kst |j |j kst d}tjt|d tdd|D|\}}}W5QRXt|tt|tj|j |j kst |j |j kst dS)Nr)rINo negative samples in y_true, false positive value should be meaninglessmatchHNo positive samples in y_true, true positive value should be meaninglesscSsg|] }d|qS)r)r@).0xr@r@rA 1sz,test_roc_curve_one_label..) rzwarnsr!rrr-fullrHnanr,rk)r?r>expected_messager^r_rlr@r@rAtest_roc_curve_one_labels $rc Csddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|ddgt|ddgt|dddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|ddgt|ddgt|dddg}ddg}d}tjt|d t||\}}}W5QRXttt||W5QRXt|dddgt|t j t j t j gddg}ddg}d }tjt|d t||\}}}W5QRXttt||W5QRXt|t j t j t j gt|dddgt ddgddgg}t ddgddgg}ttt||d d W5QRXttt||d d W5QRXtt||dd dtt||dd dt ddgddgg}t ddgddgg}ttt||d d W5QRXttt||d d W5QRXtt||dd dtt||dd dt ddgddgg}t ddgddgg}tt||d d dtt||d d dtt||dd dtt||dd dt ddgddgg}t ddgddgg}tt||d d dtt||d d dtt||dd dtt||dd ddS)Nrr)rMrd??rrrmacroaverageweightedsamplesmicro) rrrrrzrr!r{r|r-rrt)r?r=r_r^r`rmrr@r@rAtest_roc_curve_toydata8s                rc Csddddddg}ddddddg}t||d d \}}}t|d dddgdddddddddddddg }dd dddd dddd dddg }t||d d \}}}t|d dddddgdS)Nrr)r皙?rd333333?ffffff?rMTrh@皙?皙?r~)rr)r?r=r_r^rlr@r@rA test_roc_curve_drop_intermediatesrcCstdddddg}dddddg}tdd }t|||d \}}}t|dkdksVtt|dkdksptdS) Nrr)rr333333?皙?rdr sample_weight)r-repeatrdiffrFrk)r?r=rr^r_r`r@r@rA!test_roc_curve_fpr_tpr_increasings  rcCsddg}ddg}tt||dddg}ddg}tt||ddddg}dddg}tt||dddg}ddg}tt||ddddg}dddg}tt||ddSNrr)rd)rr)rr7r@r@rAtest_aucs    rc CstttdddgddgW5QRXtttdgdgW5QRXdddd g}d d d d g}dt|}tjtt|dt||W5QRXdS)NrrdrMrrr%r)rprz+x is neither increasing nor decreasing : {}r) rzr{r|rformatr-rtreescape)rr7 error_messager@r@rAtest_auc_errorss    rzy_true, labelsr)r%abcc Cs^tdddgdddgdddgdd dgg}td dd gdddg}tdd dgdddg}||d }td d dgdddg}tddd gdddg}||d }td dgdd g} tdd gddg} | | d } ||| d } tt|||d d| ||| g} dddg}tj| |d}tt|||d dd|d}tjt|dt|||d ddW5QRXdS)Nrrrrffffff?rd333333?rrr)r%rpovolabels multi_classr)weightsrrrrz6average=None is not implemented for multi_class='ovo'.r)r-rtrrrrzr{NotImplementedError)r?ry_scoresscore_01score_10Zaverage_score_01Zscore_02Zscore_20Zaverage_score_02Zscore_12Zscore_21Zaverage_score_12Zovo_unweighted_scoreZ pair_scoresZ prevalenceZovo_weighted_scorerr@r@rA#test_multiclass_ovo_roc_auc_toydatasB "     rdcCstdddgdddgdddgdddgg}tdd dd gddddg}td dd dgddddg}||d }tt|||d d |tt|||d d d|dS)Nrrrrrg?g?r)rr%rrrrr-rtrr)r?rrrrZ ovo_scorer@r@rA*test_multiclass_ovo_roc_auc_toydata_binarys( " rcCstdddgdddgdddgdddgg}tdd d d g|ddd f}td dd d g|dddf}td d ddg|ddd f}tt||d |dd |||g|||d }tt||d |d||d|d|d}tt||d |dd |dS)NrMrrrdrrrr)rr%ovr)rrrg@)rrrrr)r?rrZout_0Zout_1Zout_2Zresult_unweightedZresult_weightedr@r@rA#test_multiclass_ovr_roc_auc_toydata=s4 "rzmulti_class, average)rr)rr)rrcCstddddg}ddddgddddgddddgdddd gg}tt||||d dddddgddddgddddgddddgg}t||||d dkstd td }t||||d td kstdS)Nrpr)r%rrrMrg?rrrr)rrrd)r-rtrrrkrrzapprox)rrr?Z y_perfectZ y_imperfectZy_chancer@r@rA0test_perfect_imperfect_chance_multiclass_roc_auchs8         rc s|tjjdddgdd}tfdd|D}t|dd d gd }t||\}}}t||}t ||d d d}|t |kst dS)NrrMrd)rsr(cs"g|]}tjjd|dqS)r))nr:r()rZ multinomialrvsr[)rZy_pred_iseedr@rArsz3test_micro_averaged_ovr_roc_auc..rr)r%)classesrrr) rZ dirichletrr-asarrayr$rZravelrrrzrrk) Zglobal_random_seedr>r?Zy_onehotr^r_r`Zroc_auc_by_handZ roc_auc_autor@rrAtest_micro_averaged_ovr_roc_aucs  rzmsg, y_true, labelsz!Parameter 'labels' must be uniquezKNumber of classes in y_true not equal to the number of columns in 'y_score'z"Parameter 'labels' must be orderedzMNumber of given labels, 2, not equal to the number of columns in 'y_score', 3zMNumber of given labels, 4, not equal to the number of columns in 'y_score', 3rpz2'y_true' contains labels not in parameter 'labels'errrc CsXtdddgdddgdddgdd dgg}tjt|d t||||d W5QRXdS) Nrrrrrrdrrrrr)r-rtrzr{r|r)msgr?rrrr@r@rA*test_roc_auc_score_multiclass_labels_errors ?"rz msg, kwargszLaverage must be one of \('macro', 'weighted', None\) for multiclass problemsr)rrzUaverage must be one of \('micro', 'macro', 'weighted', None\) for multiclass problemszksample_weight is not supported for multiclass one-vs-one ROC AUC, 'sample_weight' must be None in this case)rrz|Partial AUC computation not available in multiclass setting, 'max_fpr' must be set to `None`, received `max_fpr=0.5` insteadrd)rr]zbmulti_class='ovp' is not supported for multiclass ROC AUC, multi_class must be in \('ovo', 'ovr'\)Zovpz'multi_class must be in \('ovo', 'ovr'\)c CsXtd}|dd}t|}|jdddd}tjt|dt||f|W5QRXdS)Nrprrrr)r randrrurzr{r|r)rkwargsr;r=Zy_probr?r@r@rA#test_roc_auc_score_multiclass_errors . rc Csttd}|d}tjddd}d}tjt|dt||W5QRXtjddd}tjt|dt||W5QRXtj dddd}tjt|dt||W5QRXt j dd td}|d}tjddd}tjt|dt||W5QRXtjddd}tjt|dt||W5QRXtj dddd}tjt|dt||W5QRXW5QRXdS) Nr r0ZdtypezROC AUC score is not definedrrCT)record) r rr-rrzr{r|rrrwarningscatch_warnings)r;r>r?err_msgr@r@rAtest_auc_score_non_binary_class#s0  r curve_funcc CsNtd}|jdddd}|d}d}tjt|d|||W5QRXdS)Nrrrprrrz"multiclass format is not supportedr)r rurrzr{r|)rr;r?r>rr@r@rA&test_binary_clf_curve_multiclass_errorCs  rc Csd}tjt|d"|tjddgddddgW5QRXtjt|d"|tjddgtdddgW5QRXd }tjt|d"|tjd d gd dddgW5QRXddd dg}|ddddg|}|ddddg|}t||D]\}}tj||qdS)Nzy_true takes value in {'a', 'b'} and pos_label is not specified: either make y_true take value in {0, 1} or {-1, 1} or pass pos_label explicitly.rrrzZ int_curveZ float_curveZint_curve_partZfloat_curve_partr@r@rA(test_binary_clf_curve_implicit_pos_labelMs&&& rcCsdddddg}dddddg}dddddg}||||d}||dd |dd |dd d}t||D]\}}t||qhdS) Nrr)rrrrrdrrC)rr )rr?r=rZresult_1Zresult_2Zarr_1Zarr_2r@r@rA(test_binary_clf_curve_zero_sample_weightqs&rc Cs4tdd\}}}t||t|dd|dd\}}}|ddksJt|d|ddksftd|t|dk<|}t||t||ddddg}dddd g}t||\}}}t |t d d d ddgt |t dd d d d gt |t dddd g|j |j kst|j |j dks0tdS) NTrgr)rrMrCr%rprrdgQUU?r) rB_test_precision_recall_curverrkmeanr-wherecopyrrrtrs) r?r`r=r:rrwZ y_true_copyrZpredict_probasr@r@rAtest_precision_recall_curve~s"      rcCst||\}}}t||}t|ddt|t||tt|||dd|j|jksZt|j|jdksntt|t |\}}}|j|jkst|j|jdkstdS)Ngrh|?rpr%rir)) rrYrrrrTrsrkr- zeros_like)r?r=r:rrlZprecision_recall_aucr@r@rArs"  rc Cstjddddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|ddgt|ddgt|dddg}ddg}t||\}}}t||}t|dddgt|dddgt|dddg}ddg}t||\}}}t||}t|ddgt|ddgt|dddg}dd g}tjtd d t||\}}}W5QRXtjtd d t||}W5QRXt |dddgt |dddgt |dddg}dd g}t||\}}}tt||dt|dddgt|dddgt ddgddgg}t ddgddgg}tjtd d t t||d d dW5QRXtjtd d t t||dd dW5QRXt t||dd dt t||dd dt ddgddgg}t ddgddgg}tjtd d t t||d d dW5QRXtjtd d t t||dd dW5QRXt t||dd d t t||dd dt ddgddgg}t ddgddgg}tt||d d dtt||dd dtt||dd dtt||dd dt ddgddgg}t ddgddgg}tjtd d t t||d d dW5QRXt t||dd dtjtd d t t||dd dW5QRXtjtd d t t||dd dW5QRXt ddgddgg}t ddgddgg}t t||d d dt t||dd dt t||dd dt t||dd dt ddgddgg}t ddgddgg}tt||d d dtt||dd dtt||dd dtt||dd dW5QRXtjdd`t ddgddgg}t ddgddgg}tjtd d t t||dd dW5QRXW5QRXdS)Nraise)allrr)rdrMrrrz!No positive class found in y_truerrrrrrignore) r-Zerrstaterrrrrzr UserWarningr rt)r?r=r:rr`Zauc_prcr@r@rA#test_precision_recall_curve_toydatasD                                rcCs<tjdtd}d|ddd<td}t||dks8tdS)Nrqrr)rr)r-rr0rrrkr?r=r@r@rA&test_average_precision_constant_valuesOs rc Cstddg}tddg}d}tjt|dt||ddW5QRXtddgddgddgddgg}tddgddgd d gd d gg}d }tjt|dt||ddW5QRXdS) Nrr)z>pos_label=2 is not a valid label. It should be one of \[0, 1\]rr%rIr~rrrznParameter pos_label is fixed to 1 for multilabel-indicator y_true. Do not set pos_label or set pos_label to 1.)r-rtrzr{r|rr?r>rr@r@rA-test_average_precision_score_pos_label_errors]s""rc Cstdd\}}}t||}t|d|}t|d|}t||d}||ksPt||ks\t||kshtt||}t|d|}t|d|} t||d} ||kst|| kst|| kstdS)NTrgrqgư>r)rBrrkr) r?r`r=rmZroc_auc_scaled_upZroc_auc_scaled_downZroc_auc_shiftedZpr_aucZpr_auc_scaled_upZpr_auc_scaled_downZpr_auc_shiftedr@r@rAtest_score_scale_invarianceps       rz(y_true,y_score,expected_fpr,expected_fnrrrrcCs(t||\}}}t||t||dSNrr r?r=Z expected_fprZ expected_fnrr^fnrr`r@r@rAtest_det_curve_toydatas rcCs(t||\}}}t||t||dSrrrr@r@rAtest_det_curve_tie_handlings rc Cs>ttdddgdddgtddddddgddddddgdSr)r rr@r@r@rAtest_det_curve_sanity_checks rr=cCsNtddddddgtd|d\}}}t|dgt|dgt||gdS)Nrr)rr)rr-rr )r=r^rrXr@r@rAtest_det_curve_constant_scoress   rr?cCs.t||d\}}}t|dgt|dgdS)Nrrr)r?r^rr`r@r@rAtest_det_curve_perfect_scoress  rzy_true, y_pred, err_msgzinconsistent numbers of samplesz Only one class present in y_truecancer not cancerrrrzpos_label is not specifiedc Cs(tjt|dt||W5QRXdS)Nr)rzr{r|rrr@r@rAtest_det_curve_bad_inputsr c Csdgddgd}tddddd dd dd d g }d |}t||dd\}}}t||dd\}}}|dtdksxt|dtdkstt||dddt||ddddS)Nrrprrrrrrrrdrr~r)rrrC)r-rtrrzrrkr ) r?Zy_pred_pos_not_cancerZy_pred_pos_cancerZfpr_pos_cancerZfnr_pos_cancerZ th_pos_cancerZfpr_pos_not_cancerZfnr_pos_not_cancerZth_pos_not_cancerr@r@rAtest_det_curve_pos_labels"  r cCsPt|ddggddggdt|ddggddggdt|ddggddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|ddggddggdt|ddggddggdt|ddggddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggdt|dddggdddggd t|ddddggddddggddS) Nrr)rrrdgUUUUUU?g?g?UUUUUU?r  lrap_scorer@r@rAcheck_lrap_toysj                      rcCstd}tddD]}|jd|fd}t|}td|f}|||dksPt|||dksbttd|f}|||dkst|||dkstqt|dgdgdgdggdgdgdgdggddS)Nrr%rr)rrrMrd) r rNuniformr-rrrkrr)r r(n_labelsr=Z y_score_tiesr?r@r@rA!check_zero_or_all_relevant_labelsKs (rc Cstt|dddgdddgW5QRXtt.|dddgdddgd ddgd ddggW5QRXtt.|dddgdddgd ddgd ddggW5QRXtt |ddgddggddgW5QRXtt"|ddgddggddggW5QRXtt$|ddgddggdgdggW5QRXtt"|ddggddgddggW5QRXtt$|dgdggddgddggW5QRXtt$|ddgddggdgdggW5QRXdS) Nrr)rrrr%rrrr)rzr{r|r r@r@rAcheck_lrap_error_raisedbs*   2  $ & ( & ( rc CsztddD]j}td|f}td|D]L}t||D]:}td|f}d|d|||f<t|||||q6q&q dS)Nr%rr)r)rNr-rrr)r rr= n_relevantrJr?r@r@rAcheck_lrap_only_ties|src stddD]}|t|d|fd}td|f}d|d<d|d<t|||d|ddtd|D]`t|D]Ntd|f}d|df<t|||tfddtDq~qnq dS) Nr%rr))rr)rrCrc3s&|]}|d|dVqdS)r)Nr@rrrrJr@rA sz>check_lrap_without_tie_and_increasing_score..)rNr-r.rErrrF)r rr=r?r@rrA+check_lrap_without_tie_and_increasing_scores rc s t||t|}t|}|j\}}t|f}t|D]}tj||dd\}}|j}||tj|dd } | || d} | jdks| j|krd||<q8d||<| D]4t fdd| D} ||| 7<q||| j<q8| S) z8Simple implementation of label ranking average precisionT)Zreturn_inverser))Z minlengthrrc3s|]}|kVqdSrr@rlabelZrankr@rArsz_my_lrap..) r r r,r-emptyrNrDrsZbincountZcumsumZnonzerorFr) r?r=r8rrPrQZ unique_rankZinv_rankZn_ranksZ corr_rankZrelevantZn_ranked_abover@rrA_my_lraps*    rrrc Cstdd|||d\}}t|jd|jd|d}t|drB|}t||}t||}t||t|}|j ||fd}t||}t||}t||dS)Nr)F)r9Zallow_unlabeledr( n_classesr8r)Z n_componentsr9r(toarrayrr) rrr,hasattrrrrrr r) r rr8r(r`r?r=Z score_lrapZ score_my_lrapr@r@rA%check_alternative_lrap_implementations,       r checkfunccCs ||dSrr@)r!r"r@r@rAtest_label_ranking_avps r#cCs ttdSr)rrr@r@r@rAtest_lrap_error_raisedsr$r8)r)r%rrr)r%rrr(cCstt|||dSr)r r)r8rr(r@r@rA$test_alternative_lrap_implementations r%cCstjddddgddddgddddggtd}tddddgddddgddddgg}tdd d g}td d d g}tt|||d t||t|dS) Nr)rrrrrrrdrrMrr)r-rtboolrrrF)r?r=Zsamplewise_lrapsrr@r@rA&test_lrap_sample_weighting_zero_labelss, r'cCs$ttddggddggdttddggddggdttddggddggdttddggddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddgdddggdd d gdddggd ttdddgdddgdddggdd d gdddgdddggd ttdddgdddgdddggdd d gdddgdddggd dS)Nrr)rrr%rdrpr$@rrg@rrr@r@r@rAtest_coverage_errors\                        (r+cCstttddggddggdttddggddggdttddggddggdttddggddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggddS)Nrrdr)r%rrpr*r@r@r@rAtest_coverage_tie_handlingTs       r,zy_true, y_scorec Cs(tjtddt||W5QRXdS)Nz'Expected 2D array, got 1D array insteadr)rzr{r|rrr@r@rAtest_coverage_1d_error_messageds r-c Csttddggddggdttddggddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttddggddggdttddggddggdttddggddggdttddggddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddgdddggddd gddd ggdttdddgdddgdddggdd d gddd gdd dggdttdddgdddgdddggdd d gd dd gdd dggdttttdddgdddggdd d gd dd ggddS) Nrr)rrrdrMrr(r)rprr%)rrrr-rtr@r@r@rAtest_label_ranking_lossssR          (r.c Cs4tt tddgddggddgW5QRXtt"tddgddggddggW5QRXtt$tddgddggdgdggW5QRXtt"tddggddgddggW5QRXtt$tdgdggddgddggW5QRXtt$tddgddggdgdggW5QRXdS)Nrr))rzr{r|rr@r@r@rA$test_ranking_appropriate_input_shapes $ & ( & ( r/cCsttddggddggdttddggddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggdttdddggdddggddS)Nr)rrdr)rrr@r@r@rAtest_ranking_loss_ties_handlings     r0cCsHtddd\}}| d}t||tjdd\}}t||dSNrr)r(rr))r%rqr)r_test_dcg_score_forr-r1r2 random_sampler`r?r=r@r@rAtest_dcg_scores   r5cCstt|jdd}t||}t||}||ks>tt||dd|ksXt|j|jdfksnt|j|jdfkst|tt |dddddf|j ddkstdS)Nr)r%rkrrCZaxis) r-log2r.r,rrrkrzrsortrF)r?r=ZdiscountidealrPr@r@rAr2s  r2c CsNttdg}t|j}t||}t||dd}dttdd}|t| | gkslt |t||dddddf gkst d|dd df<t||}t||dd}|t||dddddf gkst |t|dd |dd df |dd |ddd f gksJt dS) NrT ignore_tiesr)r%rrCrrp) r-rr.rr,rr9rzrrFrrk)r?r=ZdcgZdcg_ignore_tiesZ discountsr@r@rA test_dcg_tiess"   . .""r>c Cs>tdd}t||dddtt||dddks:tdS)N )r%rrpT)r7r=)r-r.rErrzrrk)rr@r@rAtest_ndcg_ignore_ties_with_ksr@c Csftdddddgg}tdddd d gg}d }tjt|d t||td ksXtW5QRXdS)Ng{Gzg(\gGz޿g(\?gQ?gQ?gףp= ?rgQ?gHzG?zndcg_score should not be used on negative y_true values. ndcg_score will raise a ValueError on negative y_true values starting from version 1.4.rg&†x@)r-rtrzr FutureWarningrrrk)r?r=rr@r@rAtest_ndcg_negative_ndarray_warns rBcCstddd}|tjdjdd|jd}t||}t||dd }|t |ksZt |t d kslt |d 7}t||t d kst dS) NFrrr皙ɿrrrTr<rMr) r-r.rEr1r2rr,rrzrrk)r?r=ZndcgZ ndcg_no_tiesr@r@rAtest_ndcg_invariants rEr=c Cs"dtddd}ttdddd}|tjdjdd |jd }t|||d t dt td dksxt t|||d t dt td dkst t |||d t d t td dkst t||d|dt dttd dkst t|||d t d t td dks:t t|||d t dt td dksnt dtd}dt td d}t|||d t |tdkst t |||d t tdkst t|||d t |kst t|||d t dkst dS)NrprrrrC)rr)rrDrrrr<r%r)r)Zlog_baser=)rr rM)r-eyeZtiler.r1r2rr,rrzrr9rkrlog10rrrrrF)r=r?r=Z y_score_noisyZexpected_dcg_scorer@r@rAtest_ndcg_toy_exampless|      rIcCsHtddd\}}| d}t||tjdd\}}t||dSr1)r_test_ndcg_score_forr-r1r2r3r4r@r@rAtest_ndcg_score.s   rKcCst||}t||}||ks$t|dkjdd}||tt|ksXt||tt|ksxt||tt |||t |||kst||tt|kst|j |j dfkst|j |j dfkstdS)Nrr)r8) rrrkrzrr-rrFrrr,)r?r=r;rPZall_zeror@r@rArJ6s  $    rJc CsJtddddg}t||dddks(tt||dddks>tttt||dds\tW5QRXttt||ddstW5QRXttt||ddstW5QRXtddddg}t||dd}t||}||kstt||d dd ksttd d \}}}td ddD]"}t t|||dt |||q"dS)Nrr))r]gMbP?gg?rg{Gz?rrdTrgg-C6?r) r-rtrrkrzr{r|rBZlinspacerre)r?rZroc_auc_with_max_fpr_oneZunconstrained_roc_aucr>r`r]r@r@rAtest_partial_roc_auc_scoreFs(       rLzy_true, k, true_scorec CsVtddddgddddgddddgddddgg}t|||d}|t|ksRtdS)Nrrrrr6r-rtr rzrrk)r?r7 true_scorer=rPr@r@rAtest_top_k_accuracy_score`s     rOzy_score, k, true_scorerCrcCsddddg}|dkr(|dkr(dnd}|dkrD||ktjn|}t|||d}t||}||krzt|ksnt dS)Nrr)rdr6) minmaxZastyper-Zint64r rrzrrk)r=r7rNr?rXr>rPZ score_accr@r@rA test_top_k_accuracy_score_binaryus   rRzy_true, true_score, labelslabels_as_ndarrayc Csf|rt|}tddddgddddgddddgddddgg}t||d|d}|t|ksbtdS)z,Test when labels and y_score are multiclass.rrrrr%r7rN)r-rrtr rzrrk)r?rNrrSr=rPr@r@rA0test_top_k_accuracy_score_multiclass_with_labelss      rUcstjddddd\tdd\}}}}tdd||t||f||fD]<\fddtddD}tt |dksTt qTdS) Nrrr)rr8Z n_informativer()r(cs g|]}t|dqS)r6)r r4)rr7r6r<r7r@rArsz8test_top_k_accuracy_score_increasing..r%) rZmake_classificationr"r#r3rrNr-rrrk)ZX_trainZX_testZy_trainZy_testZscoresr@rVrA$test_top_k_accuracy_score_increasings   rWc CsRtddddgddddgddddgddddgg}t|||dt|ksNtdS)Nrrrr)rpr6rM)r?r7rNr=r@r@rAtest_top_k_accuracy_score_tiess     rXz y_true, krc Csntddddgddddgddddgddddgg}d}tjt|dt|||d}W5QRX|dksjtdS) Nrrrrzu'k' \(\d+\) greater than or equal to 'n_classes' \(\d+\) will result in a perfect score and is therefore meaningless.rr6r))r-rtrzrr!r rk)r?r7r=rrPr@r@rA!test_top_k_accuracy_score_warnings     rYzy_true, y_score, labels, msgg= ףp=?rrz9y type must be 'binary' or 'multiclass', got 'continuous'zZNumber of classes in 'y_true' \(4\) not equal to the number of classes in 'y_score' \(3\).z"Parameter 'labels' must be unique.z#Parameter 'labels' must be ordered.zSNumber of given labels \(4\) not equal to the number of classes in 'y_score' \(3\).z3'y_true' contains labels not in parameter 'labels'.z}`y_true` is binary while y_score is 2d with 3 classes. If `y_true` does not contain all the labels, `labels` must be providedc Cs.tjt|dt||d|dW5QRXdS)Nrr%rT)rzr{r|r )r?r=rrr@r@rAtest_top_k_accuracy_score_errorsQrZcCsRtdddgdddgg}tdddgdddgg}t||}|tdksNtdS)Nr)rrdr~rr )rr-rtrrzrrk)r?r=resultr@r@rAOtest_label_ranking_avg_precision_score_should_allow_csr_matrix_for_y_true_input?s r\)NF)rrr)rrznumpyr-rZ scipy.sparserZscipyrZsklearnrrZsklearn.utils.extmathrZsklearn.datasetsrZsklearn.random_projectionrZsklearn.utils.validationr r r Zsklearn.utils._testingr rrrZsklearn.metricsrrrrrrrrrrZsklearn.metrics._rankingrrrrr Zsklearn.exceptionsr!Zsklearn.model_selectionr"Zsklearn.linear_modelr#Zsklearn.preprocessingr$Z CURVE_FUNCSrBrLrTrYremarkZ parametrizernrvryr}rrrrrrrrrtrrrrrrrrrrrrrrrrrrrrrrr r rrrrrrr r#r$rNr%r'r+r,r-r.r/r0r5r2r>r@rBrErIrKrJrLrOrRrUrWrXrYrZr\r@r@r@rAs                            0   h  2  " "  =     *    #  $""""""""     ;'     7 3     &                                       P