U mdG@sdZddlZddlZddlmZddlmZddlmZddl Z ddl Z ddl mZddlmZdd lmZmZmZeeed d d Zeee jd d dZddZdddZdddZdddZdS)z9Implementation of ARFF parsers: via LIAC-ARFF and pandas.N) OrderedDict) Generator)List)_arff)ArffSparseDataType)_chunk_generatorcheck_pandas_supportget_chunk_n_rows) arff_datainclude_columnsreturncCstttf}ddt|D}t|d|d|dD]@\}}}||kr:|d||d||d||q:|S)aObtains several columns from sparse ARFF representation. Additionally, the column indices are re-labelled, given the columns that are not included. (e.g., when including [1, 2, 3], the columns will be relabelled to [0, 1, 2]). Parameters ---------- arff_data : tuple A tuple of three lists of equal size; first list indicating the value, second the x coordinate and the third the y coordinate. include_columns : list A list of columns to include. Returns ------- arff_data_new : tuple Subset of arff data with only the include columns indicated by the include_columns argument. cSsi|]\}}||qSr.0Z array_idxZ column_idxrrV/home/sam/Atlas/atlas_env/lib/python3.8/site-packages/sklearn/datasets/_arff_parser.py -sz)_split_sparse_columns..rr)list enumeratezipappend)r r Z arff_data_newreindexed_columnsvalrow_idxcol_idxrrr_split_sparse_columnss"rc Cs~t|dd}|t|f}ddt|D}tj|tjd}t|d|d|dD]"\}}}||krV|||||f<qV|S)NrcSsi|]\}}||qSrrrrrrr?sz)_sparse_data_to_array..dtyperr)maxlenrnpemptyfloat64r) r r num_obsZy_shaperyrrrrrr_sparse_data_to_array8s "r&cCsD||}t|dkr||}nt|dkr8||d}nd}||fS)a Post process a dataframe to select the desired columns in `X` and `y`. Parameters ---------- frame : dataframe The dataframe to split into `X` and `y`. feature_names : list of str The list of feature names to populate `X`. target_names : list of str The list of target names to populate `y`. Returns ------- X : dataframe The dataframe containing the features. y : {series, dataframe} or None The series or dataframe containing the target. rrrN)r )frameZ feature_namesZ target_namesXr%rrr_post_process_frameJs   r)c"shdd}||}|dkrtjntj}|dk } tj||| d} ||fdd| dD|dkrtd } t| d} t| } t| d }| j |g| d }|j d d  }t |}fdd| D}||g}t | d |D]}|| j || d |q| j|d d}~~i}|jD]P}|d}|dkrFd||<n&|dkr^d||<n|j|||<q||}t|||\}n| d }fdd|D}fdd|D}t|tr@|dkrtd|ddkrd}n|d|d}tjtj|d|d}|j|}|dd|f}|dd|fnt|trt||}t |dd}|t!|f} t"j#j$|d|d|dff| tj%d }|&}t'||ntd!t(|fd"d#|D}!|!sn._io_to_generatorsparsepandas) return_typeencode_nominalcs(i|] \}}t|tr|kr||qSr) isinstancer)rnamecatcolumns_to_selectrrrs  z%_liac_arff_parser.. attributeszfetch_openml with as_frame=Truedata)columnsT)deepcsg|]}|kr|qSrrrcolr6rr sz%_liac_arff_parser..)Z ignore_index data_typeintegerInt64nominalcategorycsg|]}t|dqSindexintrcol_nameopenml_columns_inforrr>scsg|]}t|dqSrDrFrHrJrrr>sNz6shape must be provided when arr['data'] is a Generatorrrr#)rcountr)shaperz-Unexpected type for data obtained from arff: csh|] }|kqSrrrH) categoriesrr sz$_liac_arff_parser..c sJg|]B\}}ttj|dddd||dfjtddqS)OrNrF)copy)r!ZtakeZasarraypopastyperG)rirI)rOr%rrr>s  zAMix of nominal and non-nominal targets is not currently supported)rL).rZCOOZ DENSE_GENloadr rrkeysnextZ DataFrameZ memory_usagesumr rrconcatr:lowerdtypesrTr)r3r ValueErrorr!Zfromiter itertoolschain from_iterableZreshapetuplerrr spr/Z coo_matrixr#Ztocsrr&typeallZhstackranyrN)"r,output_arrays_typerKfeature_names_to_selecttarget_names_to_selectrNr.streamr1r2Zarff_containerpdZ columns_infoZ columns_names first_rowZfirst_dfZ row_bytes chunksizecolumns_to_keepdfsr9r'r\r4 column_dtyper(r Zfeature_indices_to_selectZtarget_indices_to_selectrMZ arff_data_Xr$ZX_shapeZis_classificationr)rOr7rKr%r_liac_arff_parserjs7                               rpc sddl|D]}|ddr q*q i|D]:}||d}|dkrXd|<q2|dkr2d |<q2fd d t|D} dd d gdddd| d} | |pi}j|f|} zdd|D| _Wn0tk r} zj d| W5d} ~ XYnX||fdd| jD} | | } t dfdd}fdd| j D}|D]}| |j|| |<qRt| ||\}}|dkr||| dfS||}}fdd | j D}||d|fS)a^ARFF parser using `pandas.read_csv`. This parser uses the metadata fetched directly from OpenML and skips the metadata headers of ARFF file itself. The data is loaded as a CSV file. Parameters ---------- gzip_file : GzipFile instance The GZip compressed file with the ARFF formatted payload. output_arrays_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities are: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected to build `X`. target_names_to_select : list of str A list of the target names to be selected to build `y`. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. rNr*z@datar?r@rArBrCcs"i|]\}}|kr||qSrr)rrr4)r\rrrusz'_pandas_arff_parser..F?%"T\)headerZ index_colZ na_valuescomment quotecharskipinitialspace escapecharrcSsg|]}|qSrr)rr4rrrr>sz'_pandas_arff_parser..zwThe number of columns provided by OpenML does not match the number of columns inferred by pandas when reading the file.csg|]}|kr|qSrrr<r6rrr>sz^'(?P.*)'$cs"t|}|dkr|S|dS)Ncontents)researchgroup)Z input_stringmatch)single_quote_patternrrstrip_single_quotess z0_pandas_arff_parser..strip_single_quotescs"g|]\}}jj|r|qSr)apitypesis_categorical_dtyperr4rrjrrr>sr0cs*i|]"\}}jj|r||jqSr)rrrrOtolistrrrrrs)r0r+r[ startswithrZread_csvr:r]errorsZ ParserErrorr{compiler\itemsr5Zrename_categoriesr)Zto_numpy)r,rfrKrgrhread_csv_kwargsr-r4roZdtypes_positionalZdefault_read_csv_kwargsr'excrmrZcategorical_columnsr=r(r%rOr)r7r\rjrr_pandas_arff_parser+sf8             rcCsH|dkrt||||||S|dkr4t||||||Std|ddS)a6Load a compressed ARFF file using a given parser. Parameters ---------- gzip_file : GzipFile instance The file compressed to be read. parser : {"pandas", "liac-arff"} The parser used to parse the ARFF file. "pandas" is recommended but only supports loading dense datasets. output_type : {"numpy", "sparse", "pandas"} The type of the arrays that will be returned. The possibilities ara: - `"numpy"`: both `X` and `y` will be NumPy arrays; - `"sparse"`: `X` will be sparse matrix and `y` will be a NumPy array; - `"pandas"`: `X` will be a pandas DataFrame and `y` will be either a pandas Series or DataFrame. openml_columns_info : dict The information provided by OpenML regarding the columns of the ARFF file. feature_names_to_select : list of str A list of the feature names to be selected. target_names_to_select : list of str A list of the target names to be selected. read_csv_kwargs : dict, default=None Keyword arguments to pass to `pandas.read_csv`. It allows to overwrite the default options. Returns ------- X : {ndarray, sparse matrix, dataframe} The data matrix. y : {ndarray, dataframe, series} The target. frame : dataframe or None A dataframe containing both `X` and `y`. `None` if `output_array_type != "pandas"`. categories : list of str or None The names of the features that are categorical. `None` if `output_array_type == "pandas"`. z liac-arffr0zUnknown parser: 'z%'. Should be 'liac-arff' or 'pandas'.N)rprr])r,parser output_typerKrgrhrNrrrrload_arff_from_gzip_files*;  r)N)N)NN)__doc__r^r{ collectionsrcollections.abcrtypingrnumpyr!ZscipyrbZ externalsrZexternals._arffrutilsrr r rZndarrayr&r)rprrrrrrs8      $ & H