U mdg@sddlZddlZddlZddlZddlZddddgZedZedZ edZ ed Z ed Z ed Z ed Zed ZedZedZedZGdddeZGdddeZGdddZGdddeZGdddeZGdddeZGdddeZGdddeZdd Zd!d"Zd#d$Zd>d%d&Zd'd(Z d)d*Z!d+d,Z"d-d.Z#d/d0Z$Gd1ddZ%d2dZ&d3d4Z'd5d6Z(d7d8Z)d9d:Z*d;e*_+e,dgsz,NominalAttribute.__init__..)superrvaluesrrZstring_maxr)rrr* __class__r r rcs zNominalAttribute.__init__cCs4t|}|r(t|d\}}t|StddS)aGiven a string containing a nominal type, returns a tuple of the possible values. A nominal type is defined as something framed between braces ({}). Parameters ---------- atrv : str Nominal type definition Returns ------- poss_vals : tuple possible values Examples -------- >>> get_nom_val("{floup, bouga, fl, ratata}") ('floup', 'bouga', 'fl', 'ratata') z(This does not look like a nominal stringN) r_nominalmatchsplit_data_linegrouptuple ValueError)atrvmattrs_r r r _get_nom_valis  zNominalAttribute._get_nom_valcCs(|ddkr ||}|||SdSdS)z Parse the attribute line if it knows how. Returns the parsed attribute, or None. For nominal attributes, the attribute string would be like '{, , }'. r{N)r9)rrrr*r r r rs   z NominalAttribute.parse_attributecCs8||jkr|S|dkr|Stdt|t|jfdS)r?z%s value not in %sN)r*r4strrr r r rs  zNominalAttribute.parse_datacCsN|jd}tt|jdD]}||j|d7}q||jd7}|d7}|S)Nz,{r.r})rrr$r*rmsgr'r r r r!s  zNominalAttribute.__str__) rrr rr staticmethodr9r"rrr! __classcell__r r r,r r#_s    r#cs8eZdZfddZeddZddZddZZS) NumericAttributecst|d|_tj|_dS)Nnumeric)r)rrrZfloat_rrr,r r rs zNumericAttribute.__init__cCsX|}|dtddksH|dtddksH|dtddkrP||SdSdS)z Parse the attribute line if it knows how. Returns the parsed attribute, or None. For numeric attributes, the attribute string would be like 'numeric' or 'int' or 'real'. NrDintreallowerstripr$rr r r rs z NumericAttribute.parse_attributecCsd|krtjSt|SdS)a Parse a value of this type. Parameters ---------- data_str : str string to convert Returns ------- f : float where float can be nan Examples -------- >>> atr = NumericAttribute('atr') >>> atr.parse_data('1') 1.0 >>> atr.parse_data('1\n') 1.0 >>> atr.parse_data('?\n') nan r;N)rnanfloatrr r r rszNumericAttribute.parse_datacCs<|jd|jd}t|t|t|t||fSNg?r.sizerZnanminZnanmaxmeanstd)rdatanbfacr r r _basic_statss  zNumericAttribute._basic_stats) rrr rr"rrrSrBr r r,r rCs   rCcs(eZdZfddZeddZZS)StringAttributecst|d|_dS)Nstring)r)rrrr,r r rs zStringAttribute.__init__cCs0|}|dtddkr(||SdSdS)z Parse the attribute line if it knows how. Returns the parsed attribute, or None. For string attributes, the attribute string would be like 'string'. NrUrGrr r r rs zStringAttribute.parse_attribute)rrr rr"rrBr r r,r rTs rTcsHeZdZfddZeddZeddZddZfd d Z Z S) DateAttributecs8t|||_||_d|_||_td|j|_dS)Ndater) r)r date_format datetime_unitrrr datetime64r)rrrXrYr,r r rs  zDateAttribute.__init__cCst|}|r|d}d}d|kr:|dd}d}n|dd}d}d|krb|dd}d }d |krz|d d }d }d |kr|d d}d}d|kr|dd}d}d|kr|dd}d}d|ksd|krtd|dkrtd||fStddS)Nr.Zyyyyz%YYyyz%yZMMz%mMddz%dDZHHz%Hhmmz%Mr6ssz%SszZz6Date type attributes with time zone not supported, yetz"Invalid or unsupported date formatzInvalid or no date format)r_dater0r2rIreplacer4)r5r6patternrYr r r _get_date_format s<        zDateAttribute._get_date_formatcCsB|}|dtddkr:||\}}||||SdSdS) Parse the attribute line if it knows how. Returns the parsed attribute, or None. For date attributes, the attribute string would be like 'date '. NrW)rHrIr$ri)rrrattr_string_lowerrXrYr r r r0s  zDateAttribute.parse_attributecCsT|dd}|dkr*td|jStj||j}t|d|jSdS)r'"r;ZNaTzdatetime64[%s]N)rIrrZrYdatetimestrptimerXZastype)rrZdate_strdtr r r rBs zDateAttribute.parse_datacstd|jS)Nr)r)r!rXr r,r r r!NszDateAttribute.__str__) rrr rrArir"rrr!rBr r r,r rVs  &  rVcs<eZdZfddZeddZddZfddZZS) RelationalAttributecs*t|d|_tj|_g|_d|_dS)N relational)r)rrrrr attributesdialectrr,r r rTs  zRelationalAttribute.__init__cCs0|}|dtddkr(||SdSdS)rjNrrrG)rrrrkr r r r[s z#RelationalAttribute.parse_attributecs|tttj}|d}g}|dD]4}t|j\_| t fdd|Dq.t |ddjDS)Nzunicode-escape cs g|]}j||qSr )rsrr%rowrr r xsz2RelationalAttribute.parse_data..cSsg|]}|j|jfqSr rrr&ar r r rx{s) listrr$rsencodedecodesplitr1rtappendr3rarray)rrelemsZescaped_stringZ row_tuplesrawr rvr rlszRelationalAttribute.parse_datacs$tdddd|jDS)Nz css|]}t|VqdSr )r<rzr r r r(sz.RelationalAttribute.__str__..)r)r!joinrsr r,r r r!}s zRelationalAttribute.__str__) rrr rr"rrr!rBr r r,r rqRs   rqcCsDtttttf}|D] }|||}|dk r|Sqtd|dS)Nzunknown attribute %s)r#rCrVrTrqrr)rrZ attr_classesrattrr r r to_attributes  rcCs8ttdd}|dkr4td}|jdkt_tj}|S)zL Checks if the bug https://bugs.python.org/issue30157 is unpatched. has_bugNz3, 'a'rl)getattrcsv_sniffer_has_bug_last_fieldcsvSniffersniff quotecharr)rrtr r r rs   rcCstrd}dd|dfD]*}t|tjtjB}||}|rqBq||krNdS|j}t|dksdt|d}|dd} || } |d d} || } |d d} t || } td t | | d tj} t | |}| |_ | |kr| |_ ||_| |_dS) zT Workaround for the bug https://bugs.python.org/issue30157 if is unpatched. zG(?P[^\w\n"\'])(?P ?)(?P["\']).*?(?P=quote)(?:$|\n)zI(?P[^\w\n"\'])(?P ?)(?P["\']).*?(?P=quote)(?P=delim)zG(?:^|\n)(?P["\']).*?(?P=quote)(?P[^\w\n"\'])(?P ?)z-(?:^|\n)(?P["\']).*?(?P=quote)(?:$|\n)Nr.rquotedelimspacez]((%(delim)s)|^)\W*%(quote)s[^%(delim)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s\W*((%(delim)s)|$))rr)rrecompileDOTALL MULTILINEfindall groupindexr$AssertionErrorboolescapesearchr delimiter doublequoteskipinitialspace) sniff_linert delimitersZ right_regexrestrregexpmatchesrr6nrrr dq_regexprr r r %workaround_csv_sniffer_bug_last_fieldsD     rcsd}tttdjdddkr6dd}tfdd|Ds`|d7}|dkrtj ||d}t |||d t t g|}||fS) Nz, r=ruc3s|]}|kVqdSr r )r&dliner r r(sz"split_data_line..r)r)rrtr) rfield_size_limitrEctypesZc_ulongvaluerIanyrrrnextreader)rrtrrrwr rr r1s   r1cCs|}t|}|rj|d}t|r@t|\}}t|}qvt|r`t|\}}t|}qvt dn t d|t ||}| dkrt |||}||fS)akParse a raw string in header (e.g., starts by @attribute). Given a raw string attribute, try to get the name and type of the attribute. Constraints: * The first line must start with @attribute (case insensitive, and space like characters before @attribute are allowed) * Works also if the attribute is spread on multilines. * Works if empty lines or comments are in between Parameters ---------- attribute : str the attribute string. Returns ------- name : str name of the attribute value : str value of the attribute next : str next line to be parsed Examples -------- If attribute is a string defined in python as r"floupi real", will return floupi as name, and real as value. >>> iterable = iter([0] * 10) # dummy iterator >>> tokenize_attribute(iterable, r"@attribute floupi real") ('floupi', 'real', 0) If attribute is r"'floupi 2' real", will return 'floupi 2' as name, and real as value. >>> tokenize_attribute(iterable, r" @attribute 'floupi 2' real ") ('floupi 2', 'real', 0) r.zmulti line not supported yetzFirst line unparsable: %srr) rI r_attributer0r2 r_comattrvaltokenize_single_commar r_wcomattrvaltokenize_single_wcommar4rrHread_relational_attribute)iterable attributeZsattrZmattrr5rtypeZ next_itemr r r tokenize_attributes )             rc Csrt|}|r^z |d}|d}Wqjtk rZ}ztd|W5d}~XYqjXn td|||fSNr.rz Error while tokenizing attributez Error while tokenizing single %s)rr0r2rI IndexErrorr4valr6rrer r r r:s  rc Csrt|}|r^z |d}|d}Wqjtk rZ}ztd|W5d}~XYqjXn td|||fSr)rr0r2rIrr4rr r r rIs  rcCsztd|jd}||snt|}|rdt|}|rVt||\}}|j|qlt d|qt |}qt |}|S)z4Read the nested attributes of a relational attributez^@[Ee][Nn][Dd]\s*z\s*$Error parsing line %s) rrrr0 r_headerlinerrrsrr4r)ofileZrelational_attributer'Zr_end_relationalr6isattrrr r r rXs     rcCst|}t|rt|}qd}g}t|st|}|rt|}|rdt||\}}||qt|}|r~| d}n t d|t|}q$t|}q$||fS)z&Read the header of the iterable ofile.Nr.r) r r_commentr0 r_datametarrrr r_relationr2r4)rr'Zrelationrsr6rrZisrelr r r read_headerns&           rc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS)raSmall container to keep useful information on a ARFF dataset. Knows about attributes names and types. Examples -------- :: data, meta = loadarff('iris.arff') # This will print the attributes names of the iris.arff dataset for i in meta: print(i) # This works too meta.names() # Getting attribute type types = meta.types() Methods ------- names types Notes ----- Also maintains the list of attributes in order, i.e., doing for i in meta, where meta is an instance of MetaData, will return the different attribute names in the order they were defined. cCs||_dd|D|_dS)NcSsi|] }|j|qSr )rrzr r r sz%MetaData.__init__..)r _attributes)rrelrr r r rszMetaData.__init__cCsfd}|d|j7}|jD]H}|d||j|jf7}|j|jrX|dt|j|j7}|d7}q|S)Nz Dataset: %s z %s's type is %sz , range is %sru)rrrrr<r?r r r __repr__s   zMetaData.__repr__cCs t|jSr )iterrr r r r __iter__szMetaData.__iter__cCs|j|}|j|jfSr )rrr)rkeyrr r r __getitem__s zMetaData.__getitem__cCs t|jS)zReturn the list of attribute names. Returns ------- attrnames : list of str The attribute names. )r|rr r r r namesszMetaData.namescsfddjD}|S)zReturn the list of attribute types. Returns ------- attr_types : list of str The attribute types. csg|]}j|jqSr )rr)r&rr r r rxsz"MetaData.types..)r)rZ attr_typesr r r typess zMetaData.typesN) rrr __doc__rrrrrrr r r r rs  cCs>t|dr|}n t|d}z t|WS||k r8|XdS)a Read an arff file. The data is returned as a record array, which can be accessed much like a dictionary of NumPy arrays. For example, if one of the attributes is called 'pressure', then its first 10 data points can be accessed from the ``data`` record array like so: ``data['pressure'][0:10]`` Parameters ---------- f : file-like or str File-like object to read from, or filename to open. Returns ------- data : record array The data of the arff file, accessible by attribute names. meta : `MetaData` Contains information about the arff file such as name and type of attributes, the relation (name of the dataset), etc. Raises ------ ParseArffError This is raised if the given file is not ARFF-formatted. NotImplementedError The ARFF file has an attribute which is not supported yet. Notes ----- This function should be able to read most arff files. Not implemented functionality include: * date type attributes * string type attributes It can read files with numeric and nominal attributes. It cannot read files with sparse data ({} in the file). However, this function can read files with missing data (? in the file), representing the data points as NaNs. Examples -------- >>> from scipy.io import arff >>> from io import StringIO >>> content = """ ... @relation foo ... @attribute width numeric ... @attribute height numeric ... @attribute color {red,green,blue,yellow,black} ... @data ... 5.0,3.25,blue ... 4.5,3.75,green ... 3.0,4.00,red ... """ >>> f = StringIO(content) >>> data, meta = arff.loadarff(f) >>> data array([(5.0, 3.25, 'blue'), (4.5, 3.75, 'green'), (3.0, 4.0, 'red')], dtype=[('width', '>> meta Dataset: foo width's type is numeric height's type is numeric color's type is nominal, range is ('red', 'green', 'blue', 'yellow', 'black') readrtN)hasattropenclose _loadarff)frr r r rsF   c szt|\}Wn8tk rH}zdt|}t||W5d}~XYnXd}D]}t|trRd}qRt|}|r|tdtd fdd }t ||}t |dd D}||fS) Nz'Error while parsing header, error was: FTz*String attributes not supported yet, sorryrc3s\tt}d}|D]B}t|st|r.qt||\}tfdd|DVqdS)Ncsg|]}||qSr )rr%)rrwr r rxasz0_loadarff..generator..)r|rrr0r_emptyr1r3)Zrow_iterrrrtrrni)rwr generatorGs z_loadarff..generatorcSsg|]}|j|jfqSr ryrzr r r rxesz_loadarff..)r) rr4r<r isinstancerTrNotImplementedErrorr$r|rr) rrrr@hasstrr{metarrQr rr r(s"    rcCs<|jd|jd}t|t|t|t||fSrLrM)rQrRr r r basic_statslsrcCsX|j}|dks|dks|dkrHt|\}}}}td||||||fn tt|dS)NrDrFintegerz%s,%s,%f,%f,%f,%f)rrprintr<)rtprQrminr+rOrPr r r print_attributeqs rcCsFt|\}}tt|jt|j|D]}t|||||q(dSr )rrr$rrNr)filenamerQrr'r r r test_wekazs   rF__main__r.)N)0rrnnumpyrrr__all__rZr_metarrrrrrr/rfrrOSErrorrrr r#rCrTrVrqrrrr1rrrrrrrrrrrZ__test__rsysargvrr r r r sX            K=S3 2 #DJQD