U md@4@sdZddlmZmZddlmZddlZddlm Z ddl m Z dddZ d d Zdd d ZddZddZddZdddZdddZdddZdddZdddZddd Zdd#d$Zdd&d'Zed(kr|ed)ed*d+d+d+d+d+gZd,d-d.d/d0gZeD]ZeeeqeD]Zeeeqeeeeeed1d2d3d4d5d6d+d7d8d9d:d;de #d?e$dd!d@Z%e &e%ee%e !d=e "dAe #d?e$dd!dBZ%e &e%eeee%d!e%e'dddCgdDdDdDgdEdDdFggZ(e()dZ*e()d!Z+ee*Z,ee+Z-ee(Z.ee*e+e(Z/ee+e*e(Z0edej1e2e*e+Z3edej1e2e+e*Z4ee*e+e(Z5edGee,e-e.e/e0e3e4e5edHe'dIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\dYd]d^d_d`dadbdcdddedfdgdhdidjdkdld^dmdndodpdqdrdsdtdudvdwdxg2ZeeZ6eedyedzed{ed|ed}ed~e*edd!e*dZ7ede7ee7d!e7gZ8ede8ede8e7e9de/fedede7e,d!e9dfedede'dddgdCdCdCgdFdCd} || | |kr|||| <q|| } |d7}|||| <q|S)z Discretize `X` Parameters ---------- bins : int, optional Number of bins. Default is floor(sqrt(N)) method : str "ef" is equal-frequency binning "ew" is equal-width binning Examples -------- Nrewrr) lenr floorsqrtceilrZrankdatarminZfastsortZzerosrange) rmethodZnbinsZnobsZdiscretewidthZsvecZivecZbinnumbaseirrr discretize[s(  r&cCst|t|S)z There is a one-to-one transformation of the entropy value from a log base b to a log base a : H_{b}(X)=log_{b}(a)[H_{a}(X)] Returns ------- log_{b}(a) )r r)rbrrr logbasechanges r(cCsttjd|S)z$ Converts from nats to bits r(r errrr natstobitssr,cCstdtj|S)z$ Converts from bits to nats r)r*rrrr bitstonatssr-r)cCsht|}t|dkr&t|dks.tdtt|t| }|dkr`td||S|SdS)aC This is Shannon's entropy Parameters ---------- logbase, int or np.e The base of the log px : 1d or 2d array_like Can be a discrete probability distribution, a 2d joint distribution, or a sequence of probabilities. Returns ----- For log base 2 (bits) given a discrete distribution H(p) = sum(px * log2(1/px) = -sum(pk*log2(px)) = E[log2(1/p(X))] For log base 2 (bits) given a joint distribution H(px,py) = -sum_{k,j}*w_{kj}log2(w_{kj}) Notes ----- shannonentropy(0) is defined as 0 rr&px does not define proper distributionr)N)r r r ValueErrorr nan_to_numlog2r()pxlogbaseentropyrrrshannonentropys r5cCs\t|}t|dkr&t|dks.td|dkrLtd| t|St| SdS)z Shannon's information Parameters ---------- px : float or array_like `px` is a discrete probability distribution Returns ------- For logbase = 2 np.log2(px) rrr.r)N)r r rr/r(r1)r2r3rrr shannoninfos  r6c Cst|rt|std|dk r0t|s0td|dkrDt||}t|tt||}|dkrn|Std||SdS)a Return the conditional entropy of X given Y. Parameters ---------- px : array_like py : array_like pxpy : array_like, optional If pxpy is None, the distributions are assumed to be independent and conendtropy(px,py) = shannonentropy(px) logbase : int or np.e Returns ------- sum_{kj}log(q_{j}/w_{kj} where q_{j} = Y[j] and w_kj = X[k,j] 1px or py is not a proper probability distributionN&pxpy is not a proper joint distribtionr))rr/r outerrr0r1r()r2pypxpyr3condentrrr condentropys r=cCs`t|rt|std|dk r0t|s0td|dkrDt||}t||dt||||dS)aC Returns the mutual information between X and Y. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like The joint probability distribution of random variables X and Y. Note that if X and Y are independent then the mutual information is zero. logbase : int or np.e, optional Default is 2 (bits) Returns ------- shannonentropy(px) - condentropy(px,py,pxpy) r7Nr8r3)rr/r r9r5r=r2r:r;r3rrr mutualinfos r@cCs`t|rt|std|dk r0t|s0td|dkrDt||}t||||dt||dS)aa An information theoretic correlation measure. Reflects linear and nonlinear correlation between two random variables X and Y, characterized by the discrete probability distributions px and py respectively. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) Returns ------- mutualinfo(px,py,pxpy,logbase=logbase)/shannonentropy(py,logbase=logbase) Notes ----- This is also equivalent to corrent(px,py,pxpy) = 1 - condent(px,py,pxpy)/shannonentropy(py) r7Nr8r>)rr/r r9r@r5r?rrrcorrents rAcCsdt|rt|std|dk r0t|s0td|dkrDt||}t||||dt||||dS)ak An information theoretic covariance measure. Reflects linear and nonlinear correlation between two random variables X and Y, characterized by the discrete probability distributions px and py respectively. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) Returns ------- condent(px,py,pxpy,logbase=logbase) + condent(py,px,pxpy, logbase=logbase) Notes ----- This is also equivalent to covent(px,py,pxpy) = condent(px,py,pxpy) + condent(py,px,pxpy) r7Nr8r>)rr/r r9r<r?rrrcovent=s rBrRcCst|stdt|}|dkrBt|}|dkr>td||S|Sdt|ks\|tjkrnt t | S||}t | }|dkrdd||Sdd|td||SdS)as Renyi's generalized entropy Parameters ---------- px : array_like Discrete probability distribution of random variable X. Note that px is assumed to be a proper probability distribution. logbase : int or np.e, optional Default is 2 (bits) alpha : float or inf The order of the entropy. The default is 1, which in the limit is just Shannon's entropy. 2 is Renyi (Collision) entropy. If the string "inf" or numpy.inf is specified the min-entropy is returned. measure : str, optional The type of entropy measure desired. 'R' returns Renyi entropy measure. 'T' returns the Tsallis entropy measure. Returns ------- 1/(1-alpha)*log(sum(px**alpha)) In the limit as alpha -> 1, Shannon's entropy is returned. In the limit as alpha -> inf, min-entropy is returned. z+px is not a proper probability distributionrr)infN) rr/floatr5r(strlowerr rDrrr)r2alphar3measureZgenentrrr renyientropyksrJTcCsdS)a Generalized cross-entropy measures. Parameters ---------- px : array_like Discrete probability distribution of random variable X py : array_like Discrete probability distribution of random variable Y pxpy : 2d array_like, optional Joint probability distribution of X and Y. If pxpy is None, X and Y are assumed to be independent. logbase : int or np.e, optional Default is 2 (bits) measure : str, optional The measure is the type of generalized cross-entropy desired. 'T' is the cross-entropy version of the Tsallis measure. 'CR' is Cressie-Read measure. Nr)r2r:r;rHr3rIrrrgencrossentropysrL__main__zQFrom Golan (2008) "Information and Entropy Econometrics -- A Review and Synthesisz Table 3.1g?gS㥛?g;On?g'1Z?gK?gMbp?gh㈵>g-C6?gMbP?g{Gz?g?g333333?g?g333333?gffffff?g?g?g?oZ InformationZ ProbabilityiZEntropyegUUUUUU?gqq?gqq?gUUUUUU?z Table 3.3zdiscretize functionsg3333335@g@F@g?@g3@gLD@gYC@g333333&@g/@gfffff?@g9@g3333334@gffffff,@g8@g5@g&@g2@gL0@g3333336@g333333@g;@ǧA@g-@g1@g333333<@gffffff0@g0@gG@g#@g2@g @@g:@g0@g333333@gffffff5@g4@gL=@g @g6@g)@gfffff:@g9@gfffff6@gffffff&@g333334@g333333:@g"@g%@g333333/@z0Example in section 3.6 of Golan, using table 3.3z'Bounding errors using Fano's inequalityz"H(P_{e}) + P_{e}log(K-1) >= H(X|Y)zor, a weaker inequalityzP_{e} >= [H(X|Y) - 1]/log(K)z P(x) = %sz?X = 3 has the highest probability, so this is the estimate Xhatz1The probability of error Pe is 1 - p(X=3) = %0.4gzH(Pe) = %0.4g and K=3z-H(Pe) + Pe*log(K-1) = %0.4g >= H(X|Y) = %0.4gzor using the weaker inequalityz'Pe = %0.4g >= [H(X) - 1]/log(K) = %0.4gz>Consider now, table 3.5, where there is additional informationz.The conditional probabilities of P(X|Y=y) are gg?z2The probability of error given this information iszPe = [H(X|Y) -1]/log(K) = %0.4gz+such that more information lowers the errorgV-?gV-?gw/?g(\?g+?g%C?gzG?gPn?)N)rN)r))r))Nr))r))r))r))rr)rC)rr)rK)=__doc__Zstatsmodels.compat.pythonrrZscipyrnumpyr Z matplotlibrZpltZ scipy.specialrr rr&r(r,r-r5r6r=r@rArBrJrL__name__printrYr%pZsubplotZylabelZxlabelZlinspacexZplotarraywrr2r:ZH_XZH_YZH_XYZ H_XgivenYZ H_YgivenXr+r4ZD_YXZD_XYZI_XYZdiscXpeZH_per1Zw2ZmeanZ markovchainrrrrs"    $ (  #   ( . 3          "         "6