entropy Editorial New Developments in Statistical Information Theory Based on Entropy and Divergence Measures Leandro Pardo Department of Satistics and Operation Research, Faculty of Mathematics, Universidad Complutense de Madrid, 28040 Madrid, Spain; lpardo@mat.ucm.es Received: 28 March 2019; Accepted: 9 April 2019; Published: 11 April 2019 In the last decades the interest in statistical methods based on information measures and particularly in pseudodistances or divergences has grown substantially. Minimization of a suitable pseudodistance or divergence measure gives estimators (minimum pseudodistance estimators or minimum divergence estimators) that have nice robustness properties in relation to the classical maximum likelihood estimators with a not signiﬁcant lost of efﬁciency. For more details we refer the monographs of Basu et al. [1] and Pardo [2]. Parametric test statistics based on the minimum divergence estimators have also given interesting results in relation to the robustness in comparison with the classical likelihood ratio test, Wald test statistic and Rao’s score statistic. Worthy of special mention are the Waldtype test statistics obtained as an extension of the classical Wald test statistic. These test statistics are based on minimum divergence estimators instead of the maximum likelihood estimators and have been considered in many different statistical problems: Censoring, see Ghosh et al. [3], equality of means in normal and lognormal models, see Basu et al. [4,5], logistic regression models, see Basu et al. [6], polytomous logistic regression models, see Castilla et al. [7], composite likelihood methods, see Martín et al. [8], etc. This Special Issue focuses on original and new research based on minimum divergence estimators, divergence statistics as well as parametric tests based on pseudodistances or divergences, from a theoretical and applied point of view, in different statistical problems with special emphasis on efﬁciency and robustness. It comprises 15 selected papers that address novel issues, as well as speciﬁc topics illustrating the importance of the divergence measures or pseudodistances in statistics. In the following, the manuscripts are presented in alphabetical order. The paper, “A Generalized Relative (α, β)Entropy Geometric properties and Applications to Robust Statistical Inference”, by A. Ghosh and A. Basu [9], proposes an alternative information theoretic formulation of the logarithmic super divergence (LSD), Magie et al. [10], as a two parametric generalization of the relative α−entropy, which they refer as the general (α, β)entropy. The paper explores its relation with various other entropies and divergences, which also generates a twoparameter extension of Renyi entropy measure as a byproduct. The paper is primarily focused on the geometric properties of the relative (α, β)entropy or the LSD measures: Continuity and convexity in both the arguments along with an extended Pythagorean relation under a powertransformation of the domain space. They also derived a set of sufﬁcient conditions under which the forward and the reverse projections of the relative (α, β)entropy exist and are unique. Finally, they brieﬂy discuss the potential applications of the relative (α, β)entropy or the LSD measures in statistical inference, in particular, for robust parameter estimation and hypothesis testing. The results in the reverse projection of the relative (α, β)entropy establish, for the ﬁrst time, the existence and uniqueness of the minimum LSD estimators. Numerical illustrations are also provided for the problem of estimating the binomial parameter. In the work “Asymptotic Properties for methods Combining the Minimum Hellinger Distance Estimate and the Bayesian Nonparametric Density Estimate”, Wu, Y. and Hooker, G. [11], pointed out Entropy 2019, 21, 391; doi:10.3390/e21040391 1 www.mdpi.com/journal/entropy Entropy 2019, 21, 391 that in frequentist inference, minimizing the Hellinger distance (Beran et al. [12]) between a kernel density estimate and a parametric family produces estimators that are both robust to outliers and statistically efﬁcient when the parametric family contains the datagenerating distribution. In this paper the previous results are extended to the use of nonparametric Bayesian density estimators within disparity methods. They proposed two estimators: One replaces the kernel density estimator with the expected posterior density using a random histogram prior; the other transforms the posterior over densities into a posterior over parameters through minimizing the Hellinger distance for each density. They show that it is possible to adapt the mathematical machinery of efﬁcient inﬂuence functions from semiparametric models to demonstrate that both estimators introduced in this paper are efﬁcient in the sense of achieving the CramérRao lower bound. They further demonstrate a BernsteinvonMises result for the second estimator, indicating that its posterior is asymptotically Gaussian. In addition, the robustness properties of classical minimum Hellinger distance estimators continue to hold. In “Composite Likelihood Methods Based on Minimum Density Power Divergence Estimator”, E. Castilla, N. Martin, L. Pardo and K. Zografos [13] pointed out that the classical likelihood function requires exact speciﬁcation of the probability density function, but in most applications, the true distribution is unknown. In some cases, where the data distribution is available in an analytic form, the likelihood function is still mathematically intractable due to the complexity of the probability density function. There are many alternatives to the classical likelihood function; in this paper, they focus on the composite likelihood. Composite likelihood is an inference function derived by multiplying a collection of component likelihoods; the particular collection used is a conditional determined by the context. Therefore, the composite likelihood reduces the computational complexity, so that it is possible to deal with large datasets and very complex models even when the use of standard likelihood methods is not feasible. Asymptotic normality of the composite maximum likelihood estimator (CMLE) still holds with the Godambe information matrix to replace the expected information in the expression of the asymptotic variancecovariance matrix. This allows the construction of composite likelihood ratio test statistics, Waldtype test statistics, as well as scoretype statistics. A review of composite likelihood methods is given in Varin [14]. They mentioned at this point that CMLE, as well as the respective test statistics are seriously affected by the presence of outliers in the set of available data. The main purpose of this paper is to introduce a new robust family of estimators, namely, composite minimum density power divergence estimators (CMDPDE), as well as a new family of Waldtype test statistics based on the CMDPDE in order to get broad classes of robust estimators and test statistics. A simulation study is presented, in order to study the robustness of the CMDPDE, as well as the performance of the Waldtype test statistics based on CMDPDE. The paper “Composite Tests under Corrupted Data”, by M. Broniatowski, J. Jurecková, A. Kumar Moses and E. Miranda [15] investigate test procedures under corrupted data. They assume that the observations Zi are mismeasured, due to the presence√of measurement errors. Thus, instead of observing Zi for i = 1, ..., n, we observe Xi = Zi + δVi , with an unknown parameter δ and an unobservable random variable Vi . It is assumed that the random variables Zi are independent and identically distributed, as are the Xi and the Vi . The test procedure aims at deciding between two simple hypotheses pertaining to the density of the variable Zi , namely f 0 and g0 . In this setting, the density of the Vi is supposed to be known. The procedure which they propose aggregates likelihood ratios for a collection of values of δ. A new deﬁnition of leastfavorable hypotheses for the aggregate family of tests is presented, and a relation with the KullbackLeibler divergence between the sets f δ (δ) and gδ (δ) is presented. Finitesample lower bounds for the power of these tests are presented, both through analytical inequalities and through simulation under the leastfavorable hypotheses. Since no optimality holds for the aggregation of likelihood ratio tests, a similar procedure is proposed, replacing the individual likelihood ratio by some divergence based test statistics. It is shown and discussed that the resulting aggregated test may perform better than the aggregate likelihood ratio procedure. 2 Entropy 2019, 21, 391 The article “Convex Optimization via Symmetrical H´’older Divergence for a WLAN Indoor Positioning System”, by O. Abdullah [16], uses the Hölder divergence, which generalizes the idea of divergence in information geometry by smooth the nonmetric of statistical distances in a way that are not required to follow the law of indiscernibles. The inequality of logratio gap pseudodivergence is built to measure the statistical distance of two classes based on Hölder’s ordinary divergence. By experiment, the WiFi signal suffers from multimodal distribution; nevertheless, the Hölder divergence is considered the proper divergence to measure the dissimilarities between probability densities since the Hölder divergence is a projective divergence that does not need the distribution be normalized and allows the closed form expressions when the expansion family is an afﬁne natural space like multinomial distributions. Hölder divergences encompass both the skew Bhattacharyya divergences and CauchySchwarz divergence, Nielsen et al. [17], and can be symmetrized, and the symmetrized Hölder divergence outperformed the symmetrized CauchySchwarz divergence over the dataset of Gaussians. Both CauchySchwarz divergences are part of a projective divergence distance family with a closedform expression that does not need to be normalized when considering closedform expressions with an afﬁne and conic parameter space, such as multivariate or multinomial distributions. In the paper “Likelihood Ratio Testing under Measurement Errors”, M. Broniatowski, J. Jurecková and J. Kalina [18] consider the likelihood ratio test of a simple null hypothesis (with density f 0 ) against a simple alternative hypothesis (with density g0 ) in the situation that observations Xi are mismeasured √ due to the presence of measurement errors. Thus instead of Xi for i = 1, ..., n, we observe Zi = Xi + δVi with unobservable parameter δ and unobservable random variable Vi . When we ignore the presence of measurement errors and perform the original test, the probability of type I error becomes different from the nominal value, but the test is still the most powerful among all tests on the modiﬁed level. Further, they derive the minimax test of some families of misspeciﬁed hypotheses and alternatives. The paper “Minimum Penalized φDivergence Estimation under Model Misspeciﬁcation”, by M. V. AlbaFernández, M. D. JiménezGamero and F. J. ArizaLópez [19], focuses on the consequences of assuming a wrong model for multinomial data when using minimum penalized φdivergence, also known as minimum penalized disparity estimators, to estimate the model parameters. These estimators are shown to converge to a welldeﬁned limit. An application of the results obtained shows that a parametric bootstrap consistently estimates the null distribution of a certain class of test statistics for model misspeciﬁcation detection. An illustrative application to the accuracy assessment of the thematic quality in a global land cover map is included. In “NonQuadratic Distances in Model Assessment”, M. Markatou and Y. Chen [20] consider that as a natural way to measure model adequacy is by using statistical distances as loss functions. A related fundamental question is how to construct loss functions that are scientiﬁcally and statistically meaningful. In this paper, they investigate nonquadratic distances and their role in assessing the adequacy of a model and/or ability to perform model selection. They ﬁrst present the deﬁnition of a statistical distance and its associated properties. Three popular distances, total variation, the mixture index of ﬁt and the KullbackLeibler distance, are studied in detail, with the aim of understanding their properties and potential interpretations that can offer insight into their performance as measures of model misspeciﬁcation. A small simulation study exempliﬁes the performance of these measures and their application to different scientiﬁc ﬁelds is brieﬂy discussed. In “φDivergence in Contingency Table Analysis”, M. Kateri [21] presents a review about the role of φdivergence measures, see Pardo [2], in modelling association in twoway contingency tables, and illustrated it for the special case of uniform association in ordinal contingency tables. This is targeted at pointing out the potential of this modelling approach and the generated families of models. Throughout this paper a multinomial sampling scheme is assumed. For the models considered here, the other two classical sampling schemes for contingency tables (independent Poisson and product multinomial) are inferentially equivalent. Furthermore, for ease of presentation, we restricted here 3 Entropy 2019, 21, 391 to twoway tables. The proposed models extend straightforwardly to multiway tables. For two or higherdimensional tables, the subset of models that are linear in their parameters (i.e., multiplicative RowColumn (RC) and RC(M)type terms are excluded) belong to the family of homogeneous linear predictor models, Goodman [22] and can thus be ﬁtted using the Rpackage mph. In “Robust and Sparse Regression via γDivergence”, T. Kawashima and H. Fujisawa [23] study robust and sparse regression based on the γdivergence. They showed desirable robust properties under both homogeneous and heterogeneous contamination. In particular, they presented the Pythagorean relation for the regression case, although it was not shown in Kanamori and Fujisawa, [24]. In most of the robust and sparse regression methods, it is difﬁcult to obtain the efﬁcient estimation algorithm, because the objective function is nonconvex and nondifferentiable. Nonetheless, they succeeded to propose the efﬁcient estimation algorithm, which has a monotone decreasing property of the objective function by using the Majorization–Minimization algorithm (MMalgorithm). The numerical experiments and real data analyses suggested that their method was superior to comparative robust and sparse linear regression methods in terms of both accuracy and computational costs. However, in numerical experiments, a few results of performance measure “true negative rate (TNR)” were a little less than the best results. Therefore, if more sparsity of coefﬁcients is needed, other sparse penalties, e.g., the Smoothly Clipped Absolute Deviations (SCAD), see Fan et al. [25] and the Minimax Concave Penalty (MCP), see Zhang [26], can also be useful. The manuscript “RobustBregman Divergence (BD) Estimation and Inference for General Partially Linear Models”, by C. Zhang and Z. Zhang [27], proposes a class of “robustBregman divergence (BD)” estimators of both the parametric and nonparametric components in the general partially linear model (GPLM), which allows the distribution of the response variable to be partially speciﬁed, without being fully known. Using the localpolynomial function estimation method, they proposed a computationallyefﬁcient procedure for obtaining “robustBD” estimators and established the consistency and asymptotic normality of the “robustBD” estimator of the parametric component β0 . For inference procedures of β0 in the GPLM, they show that the Waldtype test statistic, Wn , constructed from the “robustBD” estimators is asymptotically distribution free under the null, whereas the likelihood ratiotype test statistic, Λn , is not. This provides an insight into the distinction from the asymptotic equivalence (Fan and Huang, [28]) between Wn and Λn in the partially linear model constructed from proﬁle leastsquares estimators using the nonrobust quadratic loss. Numerical examples illustrate the computational effectiveness of the proposed “robustBD” estimators and robust Waldtype test in the appearance of outlying observations. In “Robust Estimation for the Single Index Model Using Pseudodistances”, A. Toma and C. Fulga [29] consider minimum pseudodistance estimators for the parameters of the single index model (model to reduce the number of parameters in portfolios), see Sharpe [30], and using them they construct new robust optimal portfolios. When outliers or atypical observations are present in the data set, the new portfolio optimization method based on robust minimum pseudodistance estimates yields better results than the classical single index method based on maximum likelihood estimates, in the sense that it leads to larger returns for smaller risks. In literature, there exist various methods for robust estimation in regression models. In the present paper, they proposed the method based on the minimum pseudodistance approach, which suppose to solve a simple optimization problem. In addition, from a theoretical point of view, these estimators have attractive properties, such as being redescending robust, consistent, equivariant and asymptotically normally distributed. The comparison with other known robust estimators of the regression parameters, such as the least median of squares estimators, the Sestimators or the minimum density power divergence estimators, shows that the minimum pseudodistance estimators represent an attractive alternative that may be considered in other applications too. They study properties of the estimators, such as, consistency, asymptotic normality, robustness and equivariance and illustrate the beneﬁts of the proposed portfolio optimization method through examples for real ﬁnancial data. 4 Entropy 2019, 21, 391 The paper “Robust Inference after Random Projections via Hellinger Distance for Locationscale Family”, by L. Li, A. N. Vidyashankar, G. Diao and E. Ahmed [31], proposes Hellinger distance based methods to obtain robust estimates for mean and variance in a locationscale model that takes into account (i) storage issues, (ii) potential model misspeciﬁcations, and (iii) presence of aberrant outliers. These issueswhich are more likely to occur when dealing with massive amounts of dataif not appropriately accounted in the methodological development, can lead to inaccurate inference and misleading conclusions. On the other hand, incorporating them in the existing methodology may not be feasible due to a computational burden. Our extensive simulations show the usefulness of the methodology and hence can be applied in a variety of scientiﬁc settings. Several theoretical and practical questions concerning robustness in a big data setting arise. The paper “Robustness Property of RobustBD WaldType Test for VaryingDimensional General Linear Models” by X. Guo and C. Zhang [32], aims to demonstrate the robustness property of the robustBD Waldtype test in Zhang et al. [33]. Nevertheless, it is a nontrivial task to address this issue. Although the local stability for the Waldtype tests have been established for the Mestimators, see Heritier and Ronchetti, [34], generalized method of moment estimators, Ronchetti and Trojan, [35], minimum density power divergence estimator, Basu et al. [36] and general Mestimators under random censoring, Ghosh et al. [3], their results for ﬁnitedimensional settings are not directly applicable to our situations with a diverging number of parameters. Under certain regularity conditions, we provide rigorous theoretical derivations for robust testing based on the Waldtype test statistics. The essential results are approximations of the asymptotic level and power under contaminated distributions of the data in a small neighborhood of the null and alternative hypotheses, respectively. The manuscript “Robust Relative Error Estimation” by K. Hirose and H. Masuda [37], presents a relative error estimation procedure that is robust against outliers. The proposed procedure is based on the γlikelihood function, which is constructed by γcross entropy, Fujisawa and Eguch, [38]. They showed that the proposed method has the redescending property, a desirable property in robust statistics literature. The asymptotic normality of the corresponding estimator together with a simple consistent estimator of the asymptotic covariance matrix are derived, which allows the construction of approximate conﬁdence sets. Besides the theoretical results, they have constructed an efﬁcient algorithm, in which we minimize a convex loss function at each iteration. The proposed algorithm monotonically decreases the objective function at each iteration. Conﬂicts of Interest: The author declares no conﬂict of interest. References 1. Basu, A.; Shioya, H.; Park, C. Statistical Inference: The Minimum Distance Approach; Chapman and Hall/CRC: Boca Raton, FL, USA, 2011. 2. Pardo, L. Statistical Inference Based on Divergence Measures; Chapman and Hall/CRC: Boca Raton, FL, USA, 2006. 3. Ghosh, A.; Basu, A.; Pardo, L. Robust Waldtype tests under random censoring. ArXiv 2017, arXiv:1708.09695. 4. Basu, A.; Mandal, A.; Martín, N.; Pardo, L. A Robust WaldType Test for Testing the Equality of Two Means from LogNormal Samples. Methodol. Comput. Appl. Probab. 2019, 21, 85–107. [CrossRef] 5. Basu, A.; Mandal, A.; Martin, N.; Pardo, L. Robust tests for the equality of two normal means based on the density power divergence. Metrika 2015, 78, 611–634. [CrossRef] 6. Basu, A.; Ghosh, A.; Mandal, A.; Martín, N.; Pardo, L. A Waldtype test statistic for testing linear hypothesis in logistic regression models based on minimum density power divergence estimator. Electron. J. Stat. 2017, 11 , 2741–2772. [CrossRef] 7. Castilla, E.; Ghosh, A.; Martín, N.; Pardo, L. New robust statistical procedures for polytomous logistic regression models. Biometrics 2019, in press, doi:10.1111/biom.12890. [CrossRef] 8. Martín, N.; Pardo, L.; Zografos, K. On divergence tests for composite hypotheses under composite likelihood. Stat. Pap. 2019, in press. [CrossRef] 5 Entropy 2019, 21, 391 9. Ghosh, A.; Basu, A. A Generalized Relative (α, β)Entropy: Geometric Properties and Applications to Robust Statistical Inference. Entropy 2018, 20, 347. [CrossRef] 10. Maji, A.; Ghosh, A.; Basu, A. The Logarithmic Super Divergence and Asymptotic Inference Properties. AStA Adv. Stat. Anal. 2016, 100, 99–131. [CrossRef] 11. Wu, Y.; Hooker, G. Asymptotic Properties for Methods Combining the Minimum Hellinger Distance Estimate and the Bayesian Nonparametric Density Estimate. Entropy 2018, 20, 955. [CrossRef] 12. Beran, R. Minimum Hellinger Distance Estimates for Parametric Models. Ann. Stat. 1977, 5, 445–463. [CrossRef] 13. Castilla, E.; Martín, N.; Pardo, L.; Zografos, K. Composite Likelihood Methods Based on Minimum Density Power Divergence Estimator. Entropy 2018, 20, 18. [CrossRef] 14. Varin, C.; Reid, N.; Firth, D. An overview of composite likelihood methods. Stat. Sin. 2011, 21, 4–42. 15. Broniatowski, M.; Jurečková, J.; Moses, A.K.; Miranda, E. Composite Tests under Corrupted Data. Entropy 2019, 21, 63. [CrossRef] 16. Abdullah, O. Convex Optimization via Symmetrical Hölder Divergence for a WLAN Indoor Positioning System. Entropy 2018, 20, 639. [CrossRef] 17. Nielsen, F.; Sun, K.; MarchandMaillet, S. kMeans Clustering with Hölder Divergences. In Proceedings of the International Conference on Geometric Science of Information, Paris, France, 7–9 November 2017. 18. Broniatowski, M.; Jurečková, J.; Kalina, J. Likelihood Ratio Testing under Measurement Errors. Entropy 2018, 20, 966. [CrossRef] 19. AlbaFernández, M.V.; JiménezGamero, M.D.; ArizaLópez, F.J. Minimum Penalized φDivergence Estimation under Model Misspeciﬁcation. Entropy 2018, 20, 329. [CrossRef] 20. Markatou, M.; Chen, Y. NonQuadratic Distances in Model Assessment. Entropy 2018, 20, 464. [CrossRef] 21. Kateri, M. φDivergence in Contingency Table Analysis. Entropy 2018, 20, 324. [CrossRef] 22. Goodman, L.A. Association models and canonical correlation in the analysis of crossclassiﬁcations having ordered categories. J. Am. Stat. Assoc. 1981, 76, 320–334. 23. Kawashima, T.; Fujisawa, H. Robust and Sparse Regression via γDivergence. Entropy 2017, 19, 608. [CrossRef] 24. Kanamori, T.; Fujisawa, H. Robust estimation under heavy contamination using unnormalized models. Biometrika 2015, 102, 559–572. [CrossRef] 25. Fan, J.; Li, R. Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties. J. Am. Stat. Assoc. 2001, 96, 1348–1360. [CrossRef] 26. Zhang, C.H. Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 2010, 38, 894–942. [CrossRef] 27. Zhang, C.; Zhang, Z. RobustBD Estimation and Inference for General Partially Linear Models. Entropy 2017, 19, 625. [CrossRef] 28. Fan, J.; Huang, T. Proﬁle likelihood inferences on semiparametric varyingcoefﬁcient partially linear models. Bernoulli 2005, 11, 1031–1057. [CrossRef] 29. Toma, A.; Fulga, C. Robust Estimation for the Single Index Model Using Pseudodistances. Entropy 2018, 20, 374. [CrossRef] 30. Sharpe, W.F. A simpliﬁed model to portfolio analysis. Manag. Sci. 1963, 9, 277–293. [CrossRef] 31. Li, L.; Vidyashankar, A.N.; Diao, G; Ahmed, E. Robust Inference after Random Projections via Hellinger Distance for Locationscale Family. Entropy 2019, 21, 348. [CrossRef] 32. Guo, X.; Zhang, C. Robustness Property of RobustBD WaldType Test for VaryingDimensional General Linear Models. Entropy 2018, 20, 168. [CrossRef] 33. Zhang, C.M.; Guo, X.; Cheng, C.; Zhang, Z.J. RobustBD estimation and inference for varyingdimensional general linear models. Stat. Sin. 2012, 24, 653–673. [CrossRef] 34. Heritier, S.; Ronchetti, E. Robust boundedinﬂuence tests in general parametric models. J. Am. Stat. Assoc. 1994, 89, 897–904. [CrossRef] 35. Ronchetti, E.; Trojani, F. Robust inference with GMM estimators. J. Econom. 2001, 101, 37–69. [CrossRef] 36. Basu, A.; Ghosh, A.; Martin, N.; Pardo, L. Robust Waldtype tests for nonhomogeneous observations based on minimum density power divergence estimator. Metrika 2018, 81, 493–522. [CrossRef] 6 Entropy 2019, 21, 391 37. Hirose, K.; Masuda, H. Robust Relative Error Estimation. Entropy 2018, 20, 632. [CrossRef] 38. Fujisawa, H.; Eguchi, S. Robust parameter estimation with a small bias against heavy contamination. J. Multivar. Anal. 2008, 99, 2053–2081. [CrossRef] c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). 7 entropy Article A Generalized Relative (α, β)Entropy: Geometric Properties and Applications to Robust Statistical Inference Abhik Ghosh and Ayanendranath Basu * Indian Statistical Institute, Kolkata 700108, India; abhianik@gmail.com * Correspondence: ayanbasu@isical.ac.in; Tel.: +913325752806 Received: 30 March 2018; Accepted: 1 May 2018; Published: 6 May 2018 Abstract: Entropy and relative entropy measures play a crucial role in mathematical information theory. The relative entropies are also widely used in statistics under the name of divergence measures which link these two ﬁelds of science through the minimum divergence principle. Divergence measures are popular among statisticians as many of the corresponding minimum divergence methods lead to robust inference in the presence of outliers in the observed data; examples include the φdivergence, the density power divergence, the logarithmic density power divergence and the recently developed family of logarithmic super divergence (LSD). In this paper, we will present an alternative information theoretic formulation of the LSD measures as a twoparameter generalization of the relative αentropy, which we refer to as the general (α, β)entropy. We explore its relation with various other entropies and divergences, which also generates a twoparameter extension of Renyi entropy measure as a byproduct. This paper is primarily focused on the geometric properties of the relative (α, β)entropy or the LSD measures; we prove their continuity and convexity in both the arguments along with an extended Pythagorean relation under a powertransformation of the domain space. We also derive a set of sufﬁcient conditions under which the forward and the reverse projections of the relative (α, β)entropy exist and are unique. Finally, we brieﬂy discuss the potential applications of the relative (α, β)entropy or the LSD measures in statistical inference, in particular, for robust parameter estimation and hypothesis testing. Our results on the reverse projection of the relative (α, β)entropy establish, for the ﬁrst time, the existence and uniqueness of the minimum LSD estimators. Numerical illustrations are also provided for the problem of estimating the binomial parameter. Keywords: relative entropy; logarithmic super divergence; robustness; minimum divergence inference; generalized renyi entropy 1. Introduction Decision making under uncertainty is the backbone of modern information science. The works of C. E. Shannon and the development of his famous entropy measure [1–3] represent the early mathematical foundations of information theory. The Shannon entropy and the corresponding relative entropy, commonly known as the KullbackLeibler divergence (KLD), has helped to link information theory simultaneously with probability [4–8] and statistics [9–13]. If P and Q are two probability measures on a measurable space (Ω, A) and have absolutely continuous densities p and q, respectively, with respect to a common dominating σﬁnite measure μ, then the Shannon entropy of P is deﬁned as E ( P) = − p log( p)dμ, (1) Entropy 2018, 20, 347; doi:10.3390/e20050347 8 www.mdpi.com/journal/entropy Entropy 2018, 20, 347 and the KLD measure between P and Q is given by p RE ( P, Q) = p log dμ. (2) q In statistics, the minimization of the KLD measure produces the most likely approximation as given by the maximum likelihood principle; the latter, in turn, has a direct equivalence to the (Shannon) entropy maximization criterion in information theory. For example, if Ω is ﬁnite and μ is the counting measure, it is easy to see that RE ( P, U ) = log Ω − E ( P), where U is the uniform measure on Ω. Minimization of this relative entropy, or equivalently maximization of the Shannon entropy, with respect to P within a suitable convex set E, generates the most probable distribution for an independent identically distributed ﬁnite source having true marginal probability in E with noninformative (uniform) prior probability of guessing [14,15]. In general, with a ﬁnite source, RE ( P, Q) denotes the penalty in expected compressed length if the compressor assumes a mismatched probability Q [16,17]. The corresponding general minimizer of RE ( P, Q) given Q, namely its forward projection, and other geometric properties of RE ( P, Q) are well studied in the literature; see [18–29] among others. Although the maximum entropy or the minimum divergence criterion based on the classical Shannon entropy E ( P) and the KLD measure RE ( P, Q) is still widely used in major (probabilistic) decision making problems in information science and statistics [30–43], there also exist many different useful generalizations of these quantities to address eminent issues in quantum statistical physics, complex codings, statistical robustness and many other topics of interest. For example, if we consider the standardized cumulant of compression length in place of the expected compression length in Shannon’s theory, the optimum distribution turns out to be the maximizer of a generalization of the Shannon entropy [44,45] which is given by 1 Eα ( P) = log pα dμ , α > 0, α = 1 (3) 1−α provided p ∈ Lα (μ), the complete vector space of functions for which the αth power of their absolute values are μintegrable. This general entropy functional is popular by the name Renyi entropy of order α [46] and covers many important entropy measures like Hartley entropy at α → 0 (for ﬁnite source), Shannon entropy at α → 1, collision entropy at α = 2 and the minentropy at α → ∞. The corresponding Renyi divergence measure is given by 1 Dα ( P, Q) = log pα q1−α dμ , α > 0, α = 1, (4) α−1 whenever p, q ∈ Lα (μ) and coincides with the classical KLD measure at α → 1. The Renyi entropy and the Renyi divergence are widely used in recent complex physical and statistical problems; see, for example, [47–56]. Other nonlogarithmic extensions of Shannon entropy include the classical f entropies [57], the Tsallis entropy [58] as well as the more recent generalized (α, β, γ)entropy [59,60] among many others; the corresponding divergences and the minimum divergence criteria are widely used in critical information theoretic and statistical problems; see [57,59–70] for details. We have noted that there is a direct information theoretic connection of KLD to the Shannon entropy under mismatched guessing by minimizing the expected compressed length. However, such a connection does not exist between the Renyi entropy Eα (P) and the Renyi divergence Dα (P, Q) as recently noted by [17,71]. Herein, it has been shown that, for a finite source with marginal distribution P and a (prior) mismatched compressor distribution Q, the penalty in the normalized cumulant of compression length is not Dα (P, Q); rather it is given by D1/α (Pα , Qα ) where Pα and Qα are defined by dPα pα dQα qα = pα = α , = qα = α . (5) dμ p dμ dμ q dμ 9 Entropy 2018, 20, 347 The new quantity D1/α ( Pα , Qα ) also gives a measure of discrimination (i.e., is a divergence) between the probability distributions P and Q and coincides with the KLD at α → 1. This functional is referred to as the relative αentropy in the terminology of [72] and has the simpler form RE α ( P, Q) := D1/α ( Pα , Qα ) (6) α 1 = log pqα−1 dμ − log pα dμ + log qα dμ, α > 0, α = 1. 1−α 1−α The geometric properties of this relative αentropy along with its forward and reverse projections have been studied recently [16,73]; see Section 2.1 for some details. This quantity had, however, already been proposed earlier as a statistical divergence, although for α ≥ 1 only, by [74] while developing a robust estimation procedure following the generalized methodofmoments approach of [75]. Later authors referred to the divergence proposed in [74] as the logarithmic density power divergence (LDPD) measure. The advantages of the minimum LDPD estimator in terms of robustness against outliers in data have been studied by, among other, [66,74]. Fujisawa [76], Fujisawa and Eguchi [77] have also used the same divergence measure with γ = (α − 1) ≥ 0 in different statistical problems and have referred to it as the γdivergence. Note that, the formulation in (6) extends the deﬁnition of the divergence over the 0 < α < 1 region as well. Motivated by the substantial advantages of the minimum LDPD inference in terms of statistical robustness against outlying observations, Maji et al. [78,79] have recently developed a twoparameter generalization of the LDPD family, namely the logarithmic super divergence (LSD) family, given by 1 1+τ 1 LSD τ,γ ( P, Q) = log p1+τ dμ − log p A q B dμ + log q1+τ dμ, B AB A (7) with A = 1 + γ(1 − τ ), B = 1 + τ − A, τ ≥ 0, γ ∈ R. This rich superfamily of divergences contain many important divergence measures including the LDPD at γ = 0 and the KullbackLeibler divergence at τ = γ → 0; this family also contains a transformation of Renyi divergence at τ = 0 which has been referred to as the logarithmic powerdivergence family by [80]. As shown in [78,79], the statistical inference based on some of the new members of this LSD family, outside the existing ones including the LDPD, provide much better tradeoff between the robustness and efﬁciency of the corresponding minimum divergence estimators. The statistical beneﬁts of the LSD family over the LDPD family raise a natural question: is it possible to translate this robustness advantage of the LSD family of divergences to the information theoretic context, through the development of a corresponding generalization of the relative αentropy in (6)? In this paper, we partly answer this question by deﬁning an independent information theoretic generalization of the relative αentropy measure coinciding with the LSD measure. We will refer to this new generalized relative entropy measure as the “Relative (α, β)entropy” and study its properties for different values of α > 0 and β ∈ R. In particular, this new formulation will extend the scope of the LSD measure for −1 < τ < 0 as well and generate several interesting new divergence and entropy measures. We also study the geometric properties of all members of the relative (α, β)entropy family, or equivalently the LSD measures, including their continuity in both the arguments and a Pythagoreantype relation. The related forward projection problem, i.e., the minimization of the relative (α, β)entropy in its ﬁrst argument, is also studied extensively. In summary, the main objective of the present paper is to study the geometric properties of the LSD measure through the new information theoretic or entropic formulation (or the relative (α, β)entropy). Our results indeed generalize the properties of the relative αentropy from [16,73]. The speciﬁc and signiﬁcant contributions of the paper can be summarized as follows. 1. We present a two parameter extension of the relative αentropy measure in (6) motivated by the logarithmic Sdivergence measures. These divergence measures are known to generate more robust statistical inference compared to the LDPD measures related to the relative αentropy. 10 Entropy 2018, 20, 347 2. In the new formulation of the relative (α, β)entropy, the LSD measures are linked with several important information theoretic divergences and entropy measures like the ones named after Renyi. A new divergence family is discovered corresponding to α → 0 case (properly standardized) for the ﬁnite measure cases. 3. As a byproduct of our new formulation, we get a new twoparameter generalization of the Renyi entropy measure, which we refer to as the Generalized Renyi entropy (GRE). This opens up a new area of research to examine the detailed properties of GRE and its use in complex problems in statistical physics and information theory. In this paper, we show that this new GRE satisﬁes the basic entropic characteristics, i.e., it is zero when the argument probability is degenerate and is maximum when the probability is uniform. 4. Here we provide a detailed geometric analysis of the robust LSD measure, or equivalently the relative (α, β)entropy in our new formulation. In particular, we show their continuity or lower semicontinuity with respect to the ﬁrst argument depending on the values of the tuning parameters α and β. Also, its lower semicontinuity with respect to the second argument is proved. 5. We also study the convexity of the LSD measures (or the relative (α, β)entropies) with respect to its argument densities. The relative αentropy (i.e, the relative (α, β)entropy at β = 1) is known to be quasiconvex [16] only in its ﬁrst argument. Here, we will show that, for general α > 0 and β = 1, the relative (α, β)entropies are not quasiconvex on the space of densities, but they are always quasiconvex with respect to both the arguments on a suitably (power) transformed space of densities. Such convexity results in the second argument were unavailable in the literature even for the relative αentropy, which we will introduce in this paper through a transformation of space. 6. Like the relative αentropy, but unlike the relative entropy in (2), our new relative (α, β)entropy also does not satisfy the data processing inequalities. However, we prove an extended Pythagorean relation for the relative (α, β)entropy which makes it reasonable to treat them as “squared distances” and talk about their projections. 7. The forward projection of a relative entropy or a suitable divergence, i.e., their minimization with respect to the ﬁrst argument, is very important for both statistical physics and information theory. This is indeed equivalent to the maximum entropy principle and is also related to the Gibbs conditioning principle. In this paper, we will examine the conditions under which such a forward projection of the relative (α, β)entropy (or, LSD) exists and is unique. 8. Finally, for completeness, we brieﬂy present the application of the LSD measure or the relative (α, β)entropy measure in robust statistical inference in the spirit of [78,79] but now with extended range of tuning parameters. It uses the reverse projection principle; a result on the existence of the minimum LSD functional is ﬁrst presented with the new formulation of this paper. Numerical illustrations are provided for the binomial model, where we additionally study their properties for the extended tuning parameter range α ∈ (0, 1) as well as for some new divergence families (related to α = 0). Brief indications of the potential use of these divergences in testing of statistical hypotheses are also provided. Although we are primarily discussing the logarithmic entropies like the Renyi entropy and its generalizations in this paper, it is important to point out that nonlogarithmic entropies including the fentropy and the Tsallis entropy are also very useful in several applications with real systems. Recently, several complex physical and social systems have been observed to follow the theory developed from such nonlogarithmic, nonadditive entropies instead of the classical additive Shannon entropy. In particular, the Tsallis entropy has led to the development of the nonextensive statistical mechanics [61,64] to solve several critical issues in modern physics. Important areas of application include, but certainly are not limited to, the motion of cold atoms in dissipative optical lattices [81,82], the magnetic ﬁeld ﬂuctuations in the solar wind and related qtriplet [83], the distribution of velocity 11 Entropy 2018, 20, 347 in driven dissipative dusty plasma [84], spin glass relaxation [85], the interaction of trapped ion with a classical buffer gas [86], different high energy collisional experiments [87–89], derivation of the black hole entropy [90], along with water engineering [63], text mining [65] and many others. Therefore, it is also important to investigate the possible generalizations and manipulations of such nonlogarithmic entropies both from mathematical and application point of view. However, as our primary interest here is in logarithmic entropies, we have, to keep the focus clear, otherwise avoided the description and development of nonlogarithmic entropies in this paper. Although there are many applications of extended and general nonadditive entropy and divergence measures, there are also some criticisms of these nonadditive measures that should be kept in mind. It is of course possible to employ such quantities simply as new descriptors of the complexity of systems, but at the same time, it is known that the minimization of a generalized divergence (or maximization of the corresponding entropy) under constraints in order to determine an optimal probability assignment leads to inconsistencies for information measures other than the KullbackLeibler divergence. See, for instance [91–96], among others. So, one needs to be very careful in discriminating the application of the newly introduced entropies and divergence measures for the purposes of inference under given information, from the ones where it is used as a measure of complexity. In this respect, we would like to emphasize that, the main advantage of our twoparameter extended family of LSD or relative (α, β)entropy measures in parametric statistical inference is in their strong robustness property against possible contamination (generally manifested through outliers) in the sample data. The classical additive Shannon entropy and KullbackLeibler divergence produce nonrobust inference even under a small proportion of data contamination, but the extremely high robustness of the LSD has been investigated in detail, with both theoretical and empirical justiﬁcations, by [78,79]; in this respect, we will present some numerical illustrations in Section 5.2. Another important issue could be to decide whether to stop at the twoparameter level for information measures or to extend it to threeparameters, fourparameters, etc. It is not an easy question to answer. However, we have seen that many members of the twoparameter family of LSD measures generate highly robust inference along with a desirable tradeoff between efﬁciency under pure data and robustness under contaminated data. Therefore a twoparameter system appears to work well in practice. Since it is a known principle that one “should not multiply entities beyond necessity”, we will, for the sake of parsimony, restrict ourselves to the second level of generalization for robust statistical inference, at least until there is further convincing evidence that the next higher level of generalization can produce a signiﬁcant improvement. 2. The Relative (α, β)Entropy Measure 2.1. Deﬁnition: An Extension of the Relative αEntropy In order to motivate the development of our generalized relative (α, β)entropy measure, let us ﬁrst brieﬂy describe an alternative formulation of the relative αentropy following [16]. Consider the mathematical setup of Section 1 with α > 0 and assume that the space Lα (μ) is equipped with the norm 1/α (  f α dμ) if α ≥ 1, f ∈ Lα (μ),  f α = α (8)  f  dμ if 0 < α < 1, f ∈ Lα (μ), and the corresponding metric dα ( g, f ) =  g − f α for g, f ∈ Lα (μ). Then, the relative αentropy between two distributions P and Q is obtained as a function of the CressieRead power divergence measure [97], deﬁned below in (11), between the escort measures Pα and Qα deﬁned in (5). Note that the disparity family or the φdivergence family [18,98–103] between P and Q is deﬁned as p Dφ ( P, Q) = qφ dμ, (9) q 12 Entropy 2018, 20, 347 for a continuous convex function φ on [0, ∞) satisfying φ(0) = 0 and with the usual convention 0φ(0/0) = 0. We consider the φfunction given by φ(u) = φλ (u) = sign(λ(λ + 1)) uλ+1 − 1 , λ ∈ R, u ≥ 0, (10) with the convention that, for any u > 0, 0φλ (u/0) = 0 if λ < 0 and 0φλ (u/0) = ∞ if λ > 0. The corresponding φdivergence has the form λ +1 p Dλ ( P, Q) = Dφλ ( P, Q) = sign(λ(λ + 1)) q − 1 dμ, (11) q which is just a positive multiple of the CressieRead power divergence with the multiplicative constant being λ(1 + λ); when this constant is present, the case λ = 0 leads to the KLD measure in a limiting sense. Note that, our φfunction in (10) differs slightly from the one used by [16] in that we use sign(λ(λ + 1)) in place of sign(λ) there; this is to make the divergence in (11) nonnegative for all λ ∈ R ([16] considered only λ > −1) which will be needed to deﬁne our generalized relative entropy. Then, given an α > 0, [16,17] set λ = α−1 − 1(> −1) and show that the relative αentropy of P with respect to Q can be obtained as μ 1 RE α ( P, Q) = RE α ( P, Q) = log [sign(λ) Dλ ( Pα , Qα ) + 1] . (12) λ It is straightforward to see that the above formulation (12) coincides with the deﬁnition given in (6). We often suppress the superscript μ whenever the underlying measure is clear from the context; in most applications in information theory and statistics it is either counting measure or the Lebesgue measure depending on whether the distribution is discrete or continuous. We can now change the tuning parameters in the formulation given by (12) suitably as to arrive at the more general form of the LSD family in (7). For this purpose, let us ﬁx α > 0, β ∈ R and assume that p, q ∈ Lα (μ) are the μdensities of P and Q, respectively. Instead of considering the reparametrization λ = α−1 − 1 as above, we now consider the twoparameter reparametrization λ = βα−1 − 1 ∈ R. Note that, the feasible range of λ, in order to make α > 0, now clearly depends on β β through α = 1+λ > 0; whenever β > 0 we have −1 < λ < ∞ and if β < 0 we need −∞ < λ < −1. We have already taken care of this dependence through the modiﬁed φ function deﬁned in (10) which ensures that Dλ (·, ·) is nonnegative for all λ ∈ R. So we can again use the relation as in (12), after suitable standardization due to the additional parameter β, to deﬁne a new generalized relative entropy measure as given in the following deﬁnition. β β Deﬁnition 1 (Relative (α, β)entropy). Given any α > 0 and β ∈ R, put λ = α − 1 (i.e., α = 1+λ ). Then, the relative (α, β)entropy of P with respect to Q is deﬁned as μ 1 RE α,β ( P, Q) = RE α,β ( P, Q) = log [sign( βλ) Dλ ( Pα , Qα ) + 1] . (13) βλ The cases β = 0 and λ = 0 (i.e, β = α) are deﬁned in limiting sense; see Equations (15) and (16) below. A straightforward simpliﬁcation gives a simpler form of this new relative (α, β)entropy which coincides with the LSD measure as follows. 1 α 1 RE α,β ( P, Q) = log pα dμ − log p β qα− β dμ + log qα dμ, (14) α−β β(α − β) β = LSD α−1, β−1 ( P, Q). 2− α 13 Entropy 2018, 20, 347 Note that, it coincides with the relative αentropy RE α ( P, Q) at the choice β = 1. For the limiting cases, it leads to the forms α log(q/p)qα dμ 1 p dμ RE α,0 ( P, Q) = + log , (15) qα dμ α qα dμ log( p/q) pα dμ 1 qα dμ RE α,α ( P, Q) = + log α . (16) α p dμ α p dμ By the divergence property of Dλ (·, ·), all the relative (α, β)entropies are nonnegative and valid statistical divergences. Note that, in view of (14), the formulation (13) extends the scope of LSD measure, deﬁned in (7), for τ ∈ (−1, 0). Proposition 1. For any α > 0 and β ∈ R, RE α,β ( P, Q) ≥ 0 for all probability measures P and Q, whenever it is deﬁned. Further, RE α,β ( P, Q) = 0 if and only in P = Q[μ]. Also, it is important to identify the cases where the relative (α, β)entropy is not ﬁnitely deﬁned, which can be obtained from the deﬁnition and convention related to Dλ divergence; these are summarized in the following proposition. Proposition 2. For any α > 0, β ∈ R and distributions P, Q having μdensities in Lα (μ), the relative (α, β)entropy RE α,β ( P, Q) is a ﬁnite positive number except for the following three cases: 1. P is not absolutely continuous with respect to Q and α < β, in which case RE α,β ( P, Q) = +∞. 2. P is mutually singular to Q and α > β, in which case also RE α,β ( P, Q) = +∞. 3. 0 < β < α and Dλ ( Pα , Qα ) ≥ 1, in which case also RE α,β ( P, Q) is undeﬁned. The above two propositions completely characterize the values and existence of our new relative (α, β)entropy measure. In the next subsection, we will now explore its relation with other existing entropies and divergence measures; along the way we will get some new ones as byproducts of our generalized relative entropy formulation. 2.2. Relations with Different Existing or New Entropies and Divergences The relative (α, β)entropy measures form a large family containing several existing relative entropies and divergences. Its relation with some popular ones are summarized in the following proposition; the proof is straightforward from deﬁnitions and hence omitted. Proposition 3. For α > 0, β ∈ R and distributions P, Q, the following results hold (whenever the relevant integrals and divergences are deﬁned ﬁnitely, even in limiting sense). 1. RE 1,1 ( P, Q) = RE ( P, Q), the KLD measure. 2. RE α,1 ( P, Q) = RE α ( P, Q), the relative αentropy. 3. RE 1,β ( P, Q) = β1 D β ( P, Q), a scaled Renyi divergence, which also coincides with the logarithmic power divergence measure of [80]. 4. RE α,β ( P, Q) = β1 D β/α ( Pα , Qα ), where Pα and Qα are as deﬁned in (5). Remark 1. Note that, items 3 and 4 in Proposition 3 indicate a possible extension of the Renyi divergence measure over negative values of the tuning parameter β as follows: 1 q D β∗ ( P, Q) = D ( P, Q), β ∈ R\{0}, D0∗ ( P, Q) = q log dμ. β β p 14 Entropy 2018, 20, 347 Note that this modiﬁed Renyi divergence also coincides with the KLD measure at β = 1. Statistical applications of this divergence family have been studied by [80]. However, not all the members of the family of relative (α, β)entropies are distinct or symmetric. For example, RE α,0 ( P, Q) = RE α,α ( Q, P) for any α > 0. The following proposition characterizes all such identities. Proposition 4. For α > 0, β ∈ R and distributions P, Q, the relative (α, β)entropy RE α,β ( P, Q) is symmetric if and only if β = α2 . In general, we have RE α, α −γ ( P, Q) = RE α, α +γ ( Q, P) for any α > 0, γ ∈ R. 2 2 Recall that the KLD measure is linked to the Shannon entropy and the relative αentropy is linked with the Renyi entropy when the prior mismatched probability is uniform over the ﬁnite space. To derive such a relation for our general relative (α, β)entropy, let us assume μ(Ω) < ∞ and let U denote the uniform probability measure on Ω. Then, we get 1 RE α,β ( P, U ) = log μ(Ω) − Eα,β ( P) , β = 0 (17) β where the functional Eα,β ( P) is given in Deﬁnition 2 below and coincides with the Renyi entropy at β = 1. Thus, it can be used to deﬁne a twoparameter generalization of the Renyi entropy as follows. Deﬁnition 2 (Generalized Renyi Entropy). For any probability measure P over a measurable space Ω, we deﬁne the generalized Renyi entropy (GRE) of order (α, β) as β 1 ( pα dμ) Eα,β ( P) = log α , α > 0, β ∈ R, β = 0, α; (18) β−α p β dμ log( p) pα dμ 1 Eα,α ( P) = − α + log pα dμ , α > 0. (19) p dμ α Note that, at β = 1, we have Eα,1 ( P) = Eα ( P), the usual Renyi entropy measure of order α. The GRE is a new entropy to the best of our knowledge, and does not belong to the general class of entropy functionals as given in [104] which covers many existing entropies (including most, if not all, classical entropies). The following property of the functional Eα,β ( P) is easy to verify and justiﬁes its use as a new entropy functional. To keep the focus of the present paper clear on the relative (α, β)entropy, further properties of the GRE will be explored in our future work. Theorem 1 (Entropic characteristics of GRE). For any probability measure P over a ﬁnite measure space Ω, we have 0 ≤ Eα,β ( P) ≤ log μ(Ω) for all α > 0 and β ∈ R\{0}. The two extremes are attained as follows. 1. Eα,β ( P) = 0 if P is degenerate at a point in Ω (no uncertainty). 2. Eα,β ( P) = log μ(Ω) if P is uniform over Ω (maximum uncertainty). Example 1 (Normal Distribution). Consider distributions Pi from the most common class of multivariate (sdimensional) normal distributions having mean μi ∈ Rs and variance matrix Σi for i = 1, 2. It is known that the Shannon and the Renyi entropies of P1 are, respectively, given by s s 1 E ( P1 ) = + log(2π ) + log Σ1 , 2 2 2 s log α s 1 Eα ( P1 ) = + log(2π ) + log Σ1 , α > 0, α = 1. 2α−1 2 2 15 Entropy 2018, 20, 347 With the new entropy measure, GRE, the entropy of the normal distribution P1 can be seen to have the form s (α log β − β log α) s 1 Eα,β ( P1 ) = + log(2π ) + log Σ1 , α > 0, β ∈ R\{0, α}, 2 ( β − α) 2 2 s s 1 Eα,α ( P1 ) = (1 − log α) + log(2π ) + log Σ1 , α > 0. 2 2 2 Interestingly, the GRE of a normal distribution is effectively the same as its Shannon entropy or Renyi entropy up to an additive constant. However, similar characteristic does not hold between the relative entropy (KLD) and relative (α, β)entropy. The KLD measure between two normal distributions P1 and P2 is given by 1 1 1  Σ2  s RE ( P1 , P2 ) = Trace(Σ2−1 Σ1 ) + (μ2 − μ1 ) T Σ2−1 (μ2 − μ1 ) + log − , 2 2 2  Σ1  2 whereas the general relative (α, β)entropy, with α > 0 and β ∈ R\{0, α}, has the form α RE α,β ( P1 , P2 ) = (μ − μ1 ) T [ βΣ2 + (α − β)Σ1 ]−1 (μ2 − μ1 ) 2 2 1  Σ2  β  Σ1  α − β sα log α + log α − . 2β( β − α)  βΣ2 + (α − β)Σ1  2β(α − β) Note that the relative (α, β)entropy gives a more general divergence measure which utilizes different weights for the variance (or precision) matrix of the two normal distributions. Example 2 (Exponential Distribution). Consider the exponential distribution P having density pθ ( x ) = θe−θx I ( x ≥ 0) with θ > 0. This distribution is very useful in lifetime modeling and reliability engineering; it is also the maximum entropy distribution of a nonnegative random variable with ﬁxed mean. The Shannon and the Renyi entropies of P are, respectively, given by log α E ( P) = 1 − log θ, and Eα ( P) = − log θ, α > 0, α = 1. α−1 A simple calculation leads to the following form of the our new GRE measure of the exponential distribution P. (α log β − β log α) Eα,β ( P) = − log θ, α > 0, β ∈ R\{0, α}, ( β − α) Eα,α ( P) = (1 − log α) − log θ, α > 0. Once again, the new GRE is effectively the same as the Shannon entropy or the Renyi entropy, up to an additive constant, for the exponential distribution as well. Further, if P1 and P2 are two exponential distributions with parameters θ1 and θ2 , respectively, the relative entropy (KLD) and the relative (α, β)entropy between them are given by θ2 RE ( P1 , P2 ) = + log θ1 − log θ2 − 1, θ1 α 1 1 α log α RE α,β ( P1 , P2 ) = log [ βθ1 + (α − β)θ2 ] − log θ1 − log θ2 − , β(α − β) α−β β β(α − β) for α > 0 and β ∈ R\{0, α}. Clearly, the contributions of both the distribution is weighted differently by β and (α − β) in their relative (α, β)entropy measure. Before concluding this section, we study the nature of our relative (α, β)entropy as α → 0. For this purpose, we restrict ourselves to the case of ﬁnite measure spaces with μ(Ω) < ∞. It is again straightforward to note that lim RE α,β ( P, Q) = 0 for any β ∈ R and any distributions P and Q on Ω. α →0 16 Entropy 2018, 20, 347 However, if we take the limit after scaling the relative entropy measure by α we get a nondegenerate divergence measure as follows. β 1 1 p β p RE ∗β ( P, Q) = lim RE α,β ( P, Q) = 2 log dμ − log dμ − log μ(Ω) , α ↓0 α β q μ(Ω) q for β ∈ R\{0}, and 2 1 1 1 RE 0∗ ( P, Q) = lim RE α,0 ( P, Q) = {log ( p/q)}2 dμ − log ( p/q) dμ . α ↓0 α 2μ(Ω) μ(Ω) These interesting relative entropy measures again deﬁne a subfamily of valid statistical divergences, from its construction. The particular member at β = 1 is linked to the LDPD (or the γdivergence) with tuning parameter −1 and can be thought of as a logarithmic extension of the famous Itakura–Saito divergence [105] given by p p D IS ( P, Q) = dμ − log dμ − μ(Ω). (20) q q This Itakura–Saitodivergence has been successfully applied to nonnegative matrix factorization in different applications [106] which can be extended by using the new divergence family RE ∗β ( P, Q) in future works. 3. Geometry of the Relative (α, β)Entropy 3.1. Continuity We start the exploration of the geometric properties of the relative (α, β)entropy with its continuity over the functional space Lα (μ). In the following, we interchangeably use the notation RE α,β ( p, q) and Dλ ( p, q) to denote RE α,β ( P, Q) and Dλ ( P, Q), respectively. Our results generalize the corresponding properties of the relative αentropy from [16,73] to our relative (α, β)entropy or equivalent LSD measure. Proposition 5. For a given q ∈ Lα (μ), consider the function p → RE α,β ( p, q) from p ∈ Lα (μ) to [0, ∞]. This function is lower semicontinuous in Lα (μ) for any α > 0, β ∈ R. Additionally, it is continuous in Lα (μ) when α > β > 0 and the relative entropy is ﬁnitely deﬁned. Proof. First let us consider any α > 0 and take pn → p in Lα (μ). Then,  pn α →  pα . Also,  pαn − pα  ≤  pn α +  pα and hence a general version of the dominated convergence theorem yields pαn → pα in L1 (μ). Thus, we get pαn pn,α := → pα in L1 (μ). (21) pαn dμ Further, following ([107], Lemma 1), we know that the function h → φλ (h)dν is lower semicontinuous in L1 (ν) for any λ ∈ R and any probability measure ν on (Ω, A). Taking ν = Qα , we get from (21) that pn,α /qα → pα /qα in L1 (ν). Therefore, the above lower semicontinuity result along with (9) implies that lim inf Dλ ( pn,α , qα ) ≥ Dλ ( pα , qα ) ≥ 0, λ ∈ R. (22) n→∞ 17 Entropy 2018, 20, 347 Now, note that the function ψ(u) = 1ρ log(sign(ρ)u + 1) is continuous and increasing on [0, ∞) for ρ > 0 and on [0, 1) for ρ < 0. Thus, combining (22) with the deﬁnition of the relative (α, β)entropy in (13), we get that lim inf RE α,β ( pn , q) ≥ RE α,β ( p, q), (23) n→∞ i.e., the function p → RE α,β ( p, q) is lower semicontinuous. Finally, consider the case α > β > 0. Note that the dual space of Lα/β (μ) is L α (μ) since α− β α− β q α > β > 0. Also, for q ∈ Lα (μ), we have q ∈ L α (μ), the dual space of the Banach space α α− β Lα/β (μ). Therefore, the function T : Lα/β (μ) → R deﬁned by α− β q T (h) = h dμ, h ∈ Lα/β (μ), qα is a bounded linear functional and hence continuous. Now, take pn → p in Lα (μ) so that  pn α → β β  pα as n → ∞. Therefore,  ppn →  pp in Lα (μ) implying  ppn n α α →  pp in Lα/β (μ). n α α Hence, by the continuity of T on Lα/β (μ), we get β β pn p T →T , as n → ∞.  pn α  pα However, from (14), we get β β α pn α p RE α,β ( pn , q) = log T → log T = RE α,β ( p, q). (24) β( β − α)  pn α β( β − α)  pα This proves the continuity of RE α,β ( p, q) in its ﬁrst argument when α > β > 0. Remark 2. Whenever Ω is ﬁnite (discrete) equipped with the counting measure μ, all integrals in the deﬁnition of RE α,β ( P, Q) become ﬁnite sums and any limit can be taken inside these ﬁnite sums. Thus, whenever deﬁned ﬁnitely, the function p → RE α,β ( p, q) is always continuous in this case. Remark 3. For a general inﬁnite space Ω, the function p → RE α,β ( p, q) is not necessarily continuous for the cases α < β. This can be seen by using the same counterexample as given in Remark 3 of [16]. However, it is yet to be veriﬁed if this function can be continuous for β < 0 cases. Proposition 6. For a given p ∈ Lα (μ), consider the function q → RE α,β ( p, q) from q ∈ Lα (μ) to [0, ∞]. This function is lower semicontinuous in Lα (μ) for any α > 0 and β ∈ R. Proof. Fix an α > 0 and β ∈ R, which in turn ﬁxes a λ ∈ R. Note that, the relative (α, β)entropy measure can be reexpressed from (13) as 1 RE α,β ( p, q) = log sign( βλ) D−(λ+1) (qα , pα ) + 1 . (25) βλ Now, consider a sequence qn → q in Lα (μ) and proceed as in the proof of Proposition 5 using ([107], Lemma 1) to obtain lim inf D−(λ+1) (qn,α , pα ) ≥ D−(λ+1) (qα , pα ) ≥ 0, λ ∈ R. (26) n→∞ 18 Entropy 2018, 20, 347 Now, whenever D−(λ+1) (qα , pα ) = 1 with βλ < 0 or D−(λ+1) (qα , pα ) = ∞ with βλ > 0, we get from (25) and (26) that lim inf RE α,β ( p, qn ) = RE α,β ( p, q) = +∞. (27) n→∞ In all other cases, we consider the function ψ(u) = 1ρ log(sign(ρ)u + 1) as in the proof of Proposition 5. This function is continuous and increasing whenever the corresponding relative entropy is ﬁnitely deﬁned for all tuning parameter values; on [0, ∞) for ρ > 0 and on [0, 1) for ρ < 0. Hence, again combining (26) with (25) through the function ψ, we conclude that lim inf RE α,β ( p, qn ) ≥ RE α,β ( p, q). (28) n→∞ Therefore, the function q → RE α,β ( p, q) is also lower semicontinuous. Remark 4. As in Remark 2, whenever Ω is ﬁnite (discrete) and is equipped with the counting measure μ, the function q → RE α,β ( p, q) is continuous in Lα (μ) for any ﬁxed p ∈ Lα (μ), α > 0 and β ∈ R. 3.2. Convexity It has been shown in [16] that the relative αentropy (i.e., RE α,1 ( p, q)) is neither convex nor biconvex, but it is quasiconvex in p. For general β = 1, however, the relative (α, β)entropy RE α,β ( p, q) is not even quasiconvex in p ∈ Lα (μ); rather it is quasiconvex on the βpower transformed space of densities, Lα (μ) β = p β : p ∈ Lα (μ) , as described in the following theorem. Note that, for α, β > 0, Lα (μ) β = Lα/β (μ). Here we deﬁne the lower level set Bα,β (q, r ) = p : RE α,β ( p, q) ≤ r and its powertransformed set Bα,β (q, r ) β = p β : p ∈ Bα,β (q, r ) , for any q ∈ Lα (μ) and r > 0. Theorem 2. For any given α > 0, β ∈ R and q ∈ Lα (μ), the sets Bα,β (q, r ) β are convex for all r > 0. Therefore, the function p β → RE α,β ( p, q) is quasiconvex on Lα (μ) β . Proof. Note that, at β = 1, our theorem coincides with Proposition 5 of [16]; so we will prove the result for the case β = 1. Fix α, r > 0, a real β ∈ / {1, α}, q ∈ Lα (μ), and p0 , p1 ∈ Bα,β (q, r ). β β β β β Then p0 , p1 ∈ Bα,β (q, r ) β . For τ ∈ [0, 1], we consider pτ = τ p1 + τ̄ p0 with τ̄ = 1 − τ. We need to show β that pτ ∈ Bα,β (q, r ) β , i.e., RE α,β ( pτ , q) ≤ r. Now, from (14), we have β α− β β/α 1 p q 1 pα RE α,β ( p, q) = log dμ = log dQα . (29) βλ  pα qα βλ qα β β Since p0 , p1 ∈ Bα,β (q, r ) β , we have β α− β pτ q sign( βλ) dμ ≤ sign( βλ)erβλ , for τ = 0, 1. (30)  pτ α qα For any τ ∈ (0, 1), we get β α− β β β α− β pτ q τ p1 + τ̄ p0 q sign( βλ) dμ = sign( βλ) dμ, [by deﬁnition of pτ ]  pτ α qα β  pτ α qα β β τ  p  1 α + τ̄  p  0 α ≤ sign( βλ)erβλ β , [by (30)].  pτ α (31) 19 Entropy 2018, 20, 347 Now, using the extended Minkowski’s inequalities from Lemma 1, given below, along with (31) and noting that βλ = β( β − α)/α, we get that β α− β pτ q sign( βλ) dμ ≤ sign( βλ)erβλ .  pτ α qα Therefore, by (29) and the fact that 1ρ log(sign(ρ)u) is increasing in u, we ﬁnally get RE α,β ( pτ , q) ≤ r. This proves the result for α = β. The case β = α can be proved in a similar manner and is left as an exercise to the readers. Lemma 1 (Extended Minkowski’s inequality). Fix α > 0, a real β ∈ / {1, α}, p0 , p1 ∈ Lα (μ), and τ ∈ [0, 1]. β β β Deﬁne pτ = τ p1 + τ̄ p0 with τ̄ = 1 − τ. Then we have the following inequalities: β β β  pτ α ≥ τ  p1 α + τ̄  p0 α , if β( β − α) > 0, (32) β β β  pτ α ≤ τ  p1 α + τ̄  p0 α , if β( β − α) < 0. (33) Proof. It follows by using the Jensen’s inequality and the convexity of the function x β/α . Next, note in view of Proposition 4 that, for any p, q ∈ Lα (μ), RE α,β ( p, q) = RE α,α− β (q, p). Using this result along with the above theorem, we also get the quasiconvexity of the relative (α, β)entropy RE α,β ( p, q) in q over a different power transformed space of densities. This leads to the following theorem. Theorem 3. For any given α > 0, β ∈ R and p ∈ Lα (μ), the function qα− β → RE α,β ( p, q) is quasiconvex on Lα (μ)α− β . In particular, for the choice β = α − 1, the function q → RE α,β ( p, q) is quasiconvex on Lα (μ). Remark 5. Note that, at α = β = 1, the RE 1,1 ( p, q) coincides with the KLD measure (or relative entropy) which is quasiconvex in both the arguments p and q on Lα (μ). 3.3. Extended Pythagorean Relation Motivated by the quasiconvexity of RE α,β ( p, q) on Lα (μ) β , we now present a Pythagoreantype result for the general relative (α, β)entropy over the powertransformed space. It generalizes the corresponding result for relative αentropy [16]; the proof is similar to that in [16] with necessary modiﬁcations due to the transformation of the domain space. Theorem 4 (Pythagorean Property). Fix an α > 0, β ∈ R with β = α and p0 , p1 , q ∈ Lα (μ). Deﬁne β β β pτ ∈ Lα (μ) by pτ = τ p1 + τ̄ p0 for τ ∈ [0, 1] and τ̄ = 1 − τ. (i) Suppose RE α,β ( p0 , q) and RE α,β ( p1 , q) are ﬁnite. Then, RE α,β ( pτ , q) ≥ RE α,β ( p0 , q) for all τ ∈ [0, 1], β β i.e., the backtransformation of line segment joining p1 and p0 on Lα (μ) β to Lα (μ) does not intersect Bα,β (q, RE α,β ( p0 , q)), if and only if RE α,β ( p1 , q) ≥ RE α,β ( p1 , p0 ) + RE α,β ( p0 , q). (34) (ii) Suppose RE α,β ( pτ , q) is ﬁnite for some ﬁxed τ ∈ (0, 1). Then, the backtransformation of line segment β β joining p1 and p0 on Lα (μ) β to Lα (μ) does not intersect Bα,β (q, RE α,β ( pτ , q)) if and only if RE α,β ( p1 , q) = RE α,β ( p1 , pτ ) + RE α,β ( pτ , q), (35) and RE α,β ( p0 , q) = RE α,β ( p0 , pτ ) + RE α,β ( pτ , q). (36) 20 Entropy 2018, 20, 347 α Proof of Part (i). Let Pτ,α to be the probability measure having μdensity pτ,α = pατ for τ ∈ [0, 1]. pτ dμ Also note that, with λ = β/α − 1, we have β p Dλ ( Pα , Qα ) = sign( βλ) (qα )−λ dμ − 1 , for p, q ∈ Lα (μ). (37)  pα Thus, (34) is equivalent to the statement β β β β sign( βλ) p0 α p1 (qα )−λ dμ ≥ sign( βλ) p1 ( p0,α )−λ dμ · p0 (qα )−λ dμ. (38) and we have β pτ s(τ ) Dλ ( Pτ,α , Qα ) = sign( βλ) (qα )−λ dμ − 1 = sign( βλ) , (39)  pτ α t(τ ) β β where s(τ ) = pτ (qα )−λ dμ and t(τ ) =  pτ α . Now consider the two implications separately. Only if statement: Now, let us assume that RE α,β ( pτ , q) ≥ RE α,β ( p0 , q) for all τ ∈ (0, 1). Then, we get τ [ Dλ ( Pτ,α , Qα ) − Dλ ( P0,α , Qα )] ≥ 0 for all τ ∈ (0, 1). Letting τ ↓ 0, we get that 1 ∂ Dλ ( Pτ,α , Qα ) ≥ 0. (40) ∂τ τ =0 In order to ﬁnd the derivative of Dλ ( Pτ,α , Qα ), we ﬁrst note that s ( τ ) − s (0) 1 β β β β = pτ (qα )−λ dμ − p0 (qα )−λ dμ = ( p1 − p0 ) (qα )−λ dμ, τ τ and hence s ( τ ) − s (0) β β s (0) = lim = ( p1 − p0 ) (qα )−λ dμ. (41) τ ↓0 τ Further, using a simple modiﬁcation of the techniques in the proof of ([16], Theorem 9), it is easy to verify that the derivative of t(τ ) with respect to τ exists and is given by ( β−α) α α− β β β t (τ ) = pατ dμ pτ ( p1 − p0 )dμ. Hence we get ( β−α) α α− β β β β β t (0) = p0α dμ p0 ( p1 − p0 )dμ = p1 ( p0,α )−λ dμ −  p0 α . (42) Therefore, the derivative of Dλ ( Pτ,α , Qα ) = sign( βλ)s(τ )/t(τ ) exists and is given by sign( βλ) [t(0)s (0) − t (0)s(0)] /t(0)2 . Therefore, using (40), we get that sign( βλ)t(0)s (0) ≥ sign( βλ)t (0)s(0), (43) which implies (38) after substituting the values from (41) and (42). If statement: Now, let us assume that (34)—or equivalently (38)—holds true. Further, as in the derivation of (38), we can start from the trivial statement RE α,β ( p0 , q) = RE α,β ( p0 , p0 ) + RE α,β ( p0 , q), 21 Entropy 2018, 20, 347 to deduce β β β β sign( βλ) p0 α p0 (qα )−λ dμ = sign( βλ) p0 ( p0,α )−λ dμ · p0 (qα )−λ dμ. (44) Now, multiply (38) by τ and (44) by τ̄, and add to get β β β β sign( βλ) p0 α pτ (qα )−λ dμ ≥ sign( βλ) pτ ( p0,α )−λ dμ · p0 (qα )−λ dμ. In view of (37), this implies that RE α,β ( pτ , q) ≥ RE α,β ( pτ , p0 ) + RE α,β ( p0 , q) ≥ RE α,β ( p0 , q). This proves the if statement of Part (i) completing the proof. Proof of Part (ii). Note that the if statement follows directly from Part (i). To prove the only if statement, we ﬁrst show that RE α,β ( p1 , q) and RE α,β ( p0 , q) are ﬁnite since β β RE α,β ( pτ , q) is ﬁnite. For this purpose, we note that p1 ≤ τ −1 pτ by the deﬁnition of pτ and hence ( p1 /q) β ≤ τ −1 ( pτ /q) β . Therefore, we get β/α β β β β β β p1,α p1 q 1 pτ q 1 pτ,α  pτ  = ≤ = . (45) qα q  p1  τ q  p1  τ qα  p1  Integration with respect to Qα and using (29), we get RE α,β ( p1 , q) ≤ RE α,β ( pτ , q) + c < ∞, where c is a constant. Similarly one can also show that RE α,β ( p0 , q) < ∞. Therefore, we can apply Part (i) to conclude that RE α,β ( p1 , q) ≥ RE α,β ( p1 , pτ ) + RE α,β ( pτ , q), and RE α,β ( p0 , q) ≥ RE α,β ( p0 , pτ ) + RE α,β ( pτ , q). (46) These relations imply that β β β β sign( βλ) pτ α p1 (qα )−λ dμ ≥ sign( βλ) p1 ( pτ,α )−λ dμ · pτ (qα )−λ dμ, (47) β β β β and sign( βλ) pτ α p0 (qα )−λ dμ ≥ sign( βλ) p0 ( pτ,α )−λ dμ · pτ (qα )−λ dμ. (48) The proof of the above results proceed in a manner analogous to the proof of (38). Now, if either of the inequalities in (46) is strict, the corresponding inequality in (47) or (48) will also be strict. Then, multiplying (47) and (48) by τ and τ̄, respectively, and adding them we get (44) with a strict inequality (in place of an equality), which is a contradiction. Hence, both inequalities in (46) must be equalities implying (35) and (36). This completes the proof. Note that, at β = 1, the above theorem coincides with Theorem 9 of [16]. However, for general α, β as well, the above extended Pythagorean relation for the relative (α, β)entropy suggests that it behaves “like" a squared distance (although with a nonlinear space transformation). So, one can meaningfully deﬁne its projection on to a suitable set which we will explore in the following sections. 4. The Forward Projection of Relative (α, β)Entropy The forward projection, i.e., minimization with respect to the ﬁrst argument given a ﬁxed second argument, leads to the important maximum entropy principle of information theory; it also relates to the Gibbs conditioning principle from statistical physics [16]. Let us now formally deﬁne and study the forward projection of the relative (α, β)entropy. Let S∗ denote the set of probability measure on (Ω, A) and let the set of corresponding μdensities be denoted by S = { p = dP/dμ : P ∈ S∗ }. 22 Entropy 2018, 20, 347 Deﬁnition 3 (Forward (α, β)Projection). Fix Q ∈ S∗ having μdensity q ∈ Lα (μ). Let E ⊂ S with RE α,β ( p, q) < ∞ for some p ∈ E. Then, p∗ ∈ E is called the forward projection of the relative (α, β)entropy or simply the forward (α, β)projection (or forward LSD projection) of q on E if it satisﬁes the relation RE α,β ( p∗ , q) = inf RE α,β ( p, q). (49) p∈E Note that we must assume that, E ⊂ Lα (μ) so that the above relative (α, β)entropy is ﬁnitely deﬁned for p ∈ E. We ﬁrst prove the uniqueness of the forward (α, β)projection from the Pythagorean property, whenever it exists. The following theorem describe the connection of the forward (α, β)projection with Pythagorean relation; the proof is same as that of ([16], Theorem 10) using Theorem 4 and hence omitted for brevity. Theorem 5. Consider the set E ⊂ S such that Eβ is convex and ﬁx q ∈ Lα (μ). Then, p∗ ∈ E ∩ Bα,β (q, ∞) is a forward (α, β)projection of q on E if and only if every p ∈ E ∩ Bα,β (q, ∞) satisﬁes RE α,β ( p, q) ≥ RE α,β ( p, p∗ ) + RE α,β ( p∗ , q). (50) Further, if ( p∗ ) β is an algebraic inner point of Eβ , i.e., for every p ∈ E there exists p ∈ E and τ ∈ (0, 1) such that ( p∗ ) β = τ p β + (1 − τ )( p ) β , then every p ∈ E satisﬁes RE α,β ( p, q) < ∞ and RE α,β ( p, q) = RE α,β ( p, p∗ ) + RE α,β ( p∗ , q), and RE α,β ( p , q) = RE α,β ( p , p∗ ) + RE α,β ( p∗ , q). Corollary 1 (Uniqueness of Forward (α, β)Projection). Consider the set E ⊂ S such that Eβ is convex and ﬁx q ∈ Lα (μ). If a forward (α, β)projection of q on E exists, it must be unique a.s.[μ]. Proof. Suppose p1∗ and p2∗ are two forward (α, β)projection of q on E. Then, by deﬁnition, RE α,β ( p1∗ , q) = RE α,β ( p2∗ , q) < ∞. Applying Theorem 5 with p∗ = p1∗ and p = p2∗ , we get RE α,β ( p2∗ , q) ≥ RE α,β ( p2∗ , p1∗ ) + RE α,β ( p1∗ , q). Hence RE α,β ( p2∗ , p1∗ ) ≤ 0 or RE α,β ( p2∗ , p1∗ ) = 0 by nonnegativity of relative entropy, which further implies that p1∗ = p2∗ a.s.[μ] by Proposition 1. Next we will show the existence of the forward (α, β)projection under suitable conditions. We need to use an extended Apollonius Theorem for the φdivergence measure Dλ used in the deﬁnition (13) of the relative (α, β)entropy. Such a result is proved in [16] for the special case α(1 + λ) = 1; the following lemma extends it for the general case α(1 + λ) = β ∈ R. Lemma 2. Fix p0 , p1 , q ∈ Lα (μ), τ ∈ [0, 1] and α(1 + λ) = β ∈ R with α > 0 and deﬁne r satisfying τ β 1− τ β β p + β p β  p1 α 1  p0 α 0 r = τ 1− τ . (51) β + β  p1 α  p0 α Let p j,α = pαj / pαj dμ for j = 0, 1, and similarly qα and rα . Then, if β( β − α) > 0 we have τDλ ( p1,α , qα ) + (1 − τ ) Dλ ( p0,α , qα ) ≥ τDλ ( p1,α , rα ) + (1 − τ ) Dλ ( p0,α , rα ) + Dλ (rα , qα ), (52) but the inequality gets reversed if β( β − α) < 0. 23 Entropy 2018, 20, 347 Proof. By (37), we get τDλ ( p1,α , qα ) + (1 − τ ) Dλ ( p0,α , qα ) − τDλ ( p1,α , rα ) − (1 − τ ) Dλ ( p0,α , rα ) β β p1 p0 = sign( βλ)τ (qα )−λ − (rα )−λ dμ + sign( βλ)(1 − τ ) (qα )−λ − (rα )−λ dμ  p1 α  p0 α β β τ 1−τ r = sign( βλ)r α + (qα )−λ − (rα )−λ dμ β  p1 α β  p0 α r α β τ 1−τ = sign( βλ)r α β + β Dλ ( R α , Q α ) .  p1 α  p0 α Then the Lemma follows by an application of the extended Minkowski’s inequalities (32) and (33) from Lemma 1. We now present the sufﬁcient conditions for the existence of the forward (α, β)projection in the following theorem. Theorem 6 (Existence of Forward (α, β)Projection). Fix α > 0 and β ∈ R with β = α and q ∈ Lα (μ). Given any set E ⊂ S for which Eβ is convex and closed and RE α,β ( p, q) < ∞ for some p ∈ E, a forward (α, β)projection of q on E always exists (and it is unique by Corollary 1). Proof. We prove it separately for the cases βλ > 0 and βλ < 0, extending the arguments from [16]. The case βλ = 0 can be obtained from these two cases by standard limiting arguments and hence omitted for brevity. The Case βλ > 0: Consider a sequence { pn } ⊂ E such that Dλ ( pn,α , qα ) < ∞ for each n and Dλ ( pn,α , qα ) → inf Dλ ( pα , qα ) as n → ∞. Then, by Lemma 2 applied to pm and pn with τ = 1/2, we get p∈E 1 1 1 1 D ( pm,α , qα ) + Dλ ( pn,α , qα ) ≥ Dλ ( pm,α , rm,n,α ) + Dλ ( pn,α , rm,n,α ) + Dλ (rm,n,α , qα ), (53) 2 λ 2 2 2 where rm,n is deﬁned by τ β 1− τ β β pm + β pn β  pm α  pn α rm,n = τ 1− τ . (54) β + β  pm α  pn α Note that, since Eβ is convex, rm,n ∈ Eβ and so rm,n ∈ E. Also, using the nonnegativity of divergence, (53) leads to 1 1 1 1 0≤ D ( pm,α , rm,n,α ) + Dλ ( pn,α , rm,n,α ) ≤ Dλ ( pm,α , qα ) + Dλ ( pn,α , qα ) − Dλ (rm,n,α , qα ). (55) 2 λ 2 2 2 Taking limit as m, n → ∞, one can see that 12 Dλ ( pm,α , qα ) + 12 Dλ ( pn,α , qα ) − Dλ (rm,n,α , qα ) → 0 and hence [ Dλ ( pm,α , rm,n,α ) + Dλ ( pn,α , rm,n,α )] → 0. Thus, Dλ ( pm,α , rm,n,α ) → 0 as m, n → ∞ by nonnegativity. This along with a generalization of Pinker’s inequality for φdivergence ([100], Theorem 1) gives lim  pm,α − rm,n,α  T = 0, (56) m,n→∞ 24 Entropy 2018, 20, 347 whenever λ(1 + λ) > 0 (which is true since βλ > 0); here  ·  T denotes the total variation norm. Now, by triangle inequality  pm,α − pn,α  T ≤  pm,α − rm,n,α  T +  pn,α − rm,n,α  T → 0, as m, n → ∞. Thus, { pn,α } is Cauchy in L1 (μ) and hence converges to some g ∈ L1 (μ), i.e., lim  pn,α − gdμ = 0, (57) n→∞ and g is a probability density with respect to μ since each pn is so. Also, (57) implies that pn,α → g in [μ]measure and hence p1/α n,α → g 1/α in L ( μ ) by an application of generalized dominated α convergence theorem. Next, as in the proof of ([16], Theorem 8), we can show that  pn α is bounded and hence  pn α → c for some c > 0, possibly working with a subsequence if needed. Thus we have pn =  pn α p1/α 1/α in L ( μ ). However, since E β is closed, we have E is closed and hence cg1/α = p∗ n,α → cg α for some p ∈ E. Further, since gdμ = 1, we must have c =  p∗ α and hence g = p∗α . Since pn → p∗ ∗ and p∗ ∈ E, Proposition 5 implies that RE α,β ( p∗ , q) ≤ lim inf RE α,β ( pn , q) = inf RE α,β ( p, q) ≤ RE α,β ( p∗ , q), n→∞ p∈E where the second equality follows by continuity of the function f (u) = ( βλ)−1 log(sign( βλ)u + 1), deﬁnitions of pn sequence and (13). Hence, we must have RE α,β ( p∗ , q) = inf RE α,β ( p, q), i.e., p∗ is a p∈E forward (α, β)projection of q on E. The Case βλ < 0: Note that, in this case, we must have 0 < β < α, since α > 0. Then, using (29), we can see that β α− β 1 p q inf RE α,β ( p, q) = log sup dμ p∈E βλ p∈E  pα qα 1 = log sup hgdμ , (58) βλ h ∈E α− β q where g = qα ∈L α (μ) and α− β β = p E s : p ∈ E, s ∈ [0, 1] ⊂ Lα/β (μ).  pα Now, since Eβ and hence E is closed, one can show that E is also closed; see, e.g., the proof of ([16], β β is also convex. For take s1 Theorem 8). Next, we will show that E p1 and s0 ∈E p0 ∈E  p   p  1 α 0 α for some s0 , s1 ∈ [0, 1] and p0 , p1 ∈ E, and take any τ ∈ [0, 1]. Note that β β β p1 p0 pτ τs1 + (1 − τ ) s0 = sτ ,  p1 α  p0 α  pτ α where β β p1 p0 β τs1  p1 α + (1 − τ ) s0  p0 α τs1 (1 − τ ) s0 β pτ = (1− τ ) s0 , and sτ = β + β  pτ α . τs1 β + β  p1 α  p0 α  p1 α  p0 α 25 Entropy 2018, 20, 347 However, by convexity of Eβ , pτ ∈ E and also 0 ≤ sτ ≤ 1 by the extended Minkowski inequality (33). β p Therefore, sτ  p τ and hence E ∈E is convex. τ α Finally, since 0 < β < α, Lα/β (μ) is a reﬂexive Banach space and hence the closed and convex ⊂ Lα/β (μ) is also closed in the weak topology. So, the unit ball is compact in the weak topology by E the BanachAlaoglu theorem and hence its closed subset E is also weakly compact. However, since g belongs to the dual space of Lα/β (μ), the linear functional h → hgdμ is continuous in weak topology and also increasing in s. Hence its supremum over E is attained at s = 1 and some p∗ ∈ E, which is the required forward (α, β)projection. Before concluding this section, we will present one example of the forward (α, β)projection onto a transformedlinear family of distributions. Example 3 (An example of the forward (α, β)projection). Fix α > 0, β ∈ R\{0, α} and q ∈ Lα (μ) related to the measure Q. Consider measurable functions f i : Ω → R for i ∈ I, an index set, and the family of distributions L∗β = P ∈ S∗ : f γ dPβ = 0 ⊂ S∗ . Let us denote the corresponding μdensity set by Lβ = p= dP dμ : P ∈ L∗β . We assume that, L∗β is nonempty, every P ∈ L∗β is absolute continuous with respect to μ and Lβ ⊂ Lα (μ). Then, p∗ is the forward (α, β)projection of q on Lβ if and only if there exists a function g in the L1 ( Q β )closure of the linear space spanned by { f i : i ∈ I } and a subset N ⊂ Ω such that, for every P ∈ L∗β P( N ) = 0 if α < β, c N qα− β dPβ ≤ Ω\ N gdPβ if α > β, ∗ α ( p ) dμ with c = ( p∗ ) β qα− β dμ and p∗ satisﬁes p∗ ( x )α− β = cq( x )α− β + g( x ), if x ∈ / N, p∗ ( x ) = 0, if x ∈ N. The proof follows by extending the arguments of the proof of ([16], Theorem 11) and hence it is left as an exercise to the readers. Remark 6. Note that, at the special case β = 1, L1∗ is a linear family of distributions and the above example coincides with ([16], Theorem 11) on the forward projection of relative αentropy on L1∗ . However, it is still an open question to derive the forward (α, β)projection on L1∗ . 5. Statistical Applications: The Minimum Relative Entropy Inference 5.1. The Reverse Projection and Parametric Estimation As in the case of the forward projection of a relative entropy measure, we can also deﬁne the reverse projection by minimizing it with respect to the second argument over a convex set E keeping the ﬁrst argument ﬁxed. More formally, we use the following deﬁnition. Deﬁnition 4 (Reverse (α, β)Projection). Fix p ∈ Lα (μ) and let E ⊂ S with RE α,β ( p, q) < ∞ for some q ∈ E. Then, q∗ ∈ E is called the reverse projection of the relative (α, β)entropy or simply the reverse (α, β)projection (or reverse LSD projection) of p on E if it satisﬁes the relation RE α,β ( p, q∗ ) = inf RE α,β ( p, q). (59) q∈E 26 Entropy 2018, 20, 347 We can get sufﬁcient conditions for the existence and uniqueness of the reverse (α, β)projection directly from Theorem 6 and the fact that RE α,β ( p, q) = RE α,α− β (q, p); this is presented in the following theorem. Theorem 7 (Existence and Uniqueness of Reverse (α, β)Projection). Fix α > 0 and β ∈ R with β = α and p ∈ Lα (μ). Given any set E ⊂ S for which Eα− β is convex and closed and RE α,β ( p, q) < ∞ for some q ∈ E, a reverse (α, β)projection of p on E exists and is unique. The reverse projection is mostly used in statistical inference where we ﬁx the ﬁrst argument of a relative entropy measure (or divergence measure) at the empirical data distribution and minimize the relative entropy with respect to the model family of distributions in its second argument. The resulting estimator, commonly known as the minimum distance or minimum divergence estimator, yields the reverse projection of the observed data distribution on the family of model distributions with respect to the relative entropy or divergence under consideration. This approach was initially studied by [9–13] to obtain the popular maximum likelihood estimator as the reverse projection with respect to the relative entropy in (2). More recently, this approach has become widely popular, but with more general relative entropies or divergence measures, to obtain robust estimators against possible contamination in the observed data. Let us describe it more rigorously in the following for our relative (α, β)entropy. Suppose we have independent and identically distributed data X1 , . . . , Xn from a true distribution G having density g with respect to some common dominating measure μ. We model g by a parametric model family of μdensities F = { f θ : θ ∈ Θ ⊆ R p }, where it is assumed that both g and f θ have the same support independent of θ. Our objective is to infer about the unknown parameter θ. In minimum divergence inference, an estimator of θ is obtained by minimizing the divergence measure between (an estimate of) g and f θ with respect to θ ∈ Θ. Maji et al. [78] have considered the LSD (or equivalently the relative (α, β)entropy) as the divergence under consideration and deﬁned the corresponding minimum divergence functional at G, say T α,β ( G ), through the relation RE α,β g, f T α,β (G) = min RE α,β ( g, f θ ), (60) θ∈Θ whenever the minimum exists. We will refer to T α,β ( G ) as the minimum relative (α, β)entropy (MRE) functional, or the minimum LSD functional in the language of [78,79]. Note that, if g ∈ F , i.e., g = f θ0 for some θ0 ∈ Θ, then we must have T α,β ( G ) = θ0 . If g ∈ / F , we call T α,β ( G ) as the “best ﬁtting parameter" value, since f T α,β (G) is the closest model element to g in the LSD sense. In fact, for g ∈ / F , T α,β ( G ) is nothing but the reverse (α, β)projection of the true density g on the model family F , which exists and is unique under the sufﬁcient conditions of Theorem 7. Therefore, under identiﬁability of the model family F we get the existence and uniqueness of the MRE functional, which is presented in the following corollary. Although this estimator was ﬁrst introduced by [78] in terms of the LSD, the results concerning the existence of the estimate were not provided. Corollary 2 (Existence and Uniqueness of the MRE Functional). Consider the above parametric estimation problem with g ∈ Lα (μ) and F ⊂ Lα (μ). Fix α > 0 and β ∈ R with β = α and assume that the model family F is identiﬁable in θ. 1. Suppose g = f θ0 for some θ0 ∈ Θ. Then the unique MRE functional is given by T α,β ( G ) = θ0 . 2. Suppose g ∈ / F . If F α− β is convex and closed and RE α,β ( g, f θ ) < ∞ for some θ ∈ Θ, the MRE functional T α,β ( G ) exists and is unique. 27 Entropy 2018, 20, 347 Further, under standard differentiability assumptions, we can obtain the estimating equation of the MRE functional T α,β ( G ) as given by α− β β α− β f θα uθ dμ fθ g dμ = f θ g β uθ dμ f θα dμ , (61) ∂ where uθ ( x ) = ∂θ ln f θ ( x ). It is important to note that, at β = α = 1, the MRE functional T 1,1 ( G ) coincides with the maximum likelihood functional since RE 1,1 = RE , the KLD measure. Based on the estimating Equation (61), Maji et al. [78] extensively studied the theoretical robustness properties of the MRE functional against grosserror contamination in data through the higher order inﬂuence function analysis. The classical ﬁrst order inﬂuence function was seen to be inadequate for this purpose; it becomes independent of β at the model but the reallife performance of the MRE functional critically depends on both α and β [78,79] as we will also see in Section 5.2. In practice, however, the true data generating density is not known and so we need to use some empirical estimate in place of g and the resulting value of the MRE functional is called the minimum relative (α, β)entropy estimator (MREE) or the minimum LSD estimator in the terminology of [78,79]. Note that, when the data are discrete and μ is the counting measure, one can use a simple estimate of g given by the relative frequencies rn ( x ) = n1 ∑in=1 I ( Xi = x ), where I ( A) is the indicator function of the event A; the corresponding MREE is then obtained by solving (61) with g( x ) replaced by rn ( x ) and integrals replaced by sums over the discrete support. Asymptotic properties of this MREE under discrete models are wellstudied by [78,79] for the tuning parameters α ≥ 1 and β ∈ R; the same line of argument can be used to extend them also for the cases α ∈ (0, 1) in a straightforward manner. However, in case of continuous data, there is no such simple estimator available to use in place of g unless β = 1. When β = 1, the estimating Equation (61) depends on g through the terms f θα−1 gdμ = f θα−1 dG and f θα−1 uθ gdμ = f θα−1 uθ dG; so we can simply use the empirical distribution function Gn in place of G and solve the resulting equation to obtain the corresponding MREE. However, for β = 1, we must use a nonparametric kernel estimator gn of g in (61) to obtain the MREE under continuous models; this leads to complications including bandwidth selection while deriving the asymptotics of the resulting MREE. One possible approach to avoid such complications is to use the smoothed model technique, which has been applied in [108] for the case of minimum φdivergence estimators. Another alternative approach has been discussed in [109,110]. However, the detailed analyses of the MREE under the continuous model, in either of the above approaches, are yet to be studied so far. 5.2. Numerical Illustration: Binomial Model Let us now present numerical illustrations under the common binomial model to study the ﬁnite sample performance of the MREEs. Along with the known properties of the MREE at α ≥ 1 (i.e., the minimum LSD estimators with τ ≥ 0 from [78,79]), here we will additionally explore their properties in case of α ∈ (0, 1) and for the new divergences RE ∗β ( P, Q) related to α = 0. Suppose X1 , . . . , Xn are random observations from a true density g having support χ = {0, 1, 2, . . . , m} for some positive integer m. We model g by the Binomial(m, θ) densities f θ ( x ) = (nx)θ x (1 − θ )m− x for x ∈ χ and θ ∈ [0, 1]. Here an estimate g of g is given by the relative frequency g( x ) = rn ( x ). For any α > 0 and β ∈ R, the relative (α, β)entropy between g and f θ is given by α m αx m 1 n θ 1 RE α,β ( g, f θ ) = log ∑ x (1 − θ )mα + log ∑ rn ( x )α β x =0 1−θ α−β x =0 m α− β (α− β) x α n θ − log ∑ (1 − θ ) m ( α − β ) r n ( x ) β , β(α − β) x =0 x 1−θ 28 Entropy 2018, 20, 347 which can be minimized with respect to θ ∈ [0, 1] to obtain the corresponding MREE of θ. Note that, it is also the solution of the estimating Equation (61) with g( x ) replaced by the relative frequency rn ( x ). However, in this example, uθ ( x ) = θx(− mθ 1− θ ) and hence the MREE estimating equation simpliﬁes to αx (α− β) x n α θ n α− β θ x =0 ( x ) ( x − mθ ) 1−θ ∑m x =0 ( x − mθ )( x ) ∑m 1− θ rn ( x ) β αx = (α− β) x . (62) n α θ n α− β ∑m x =0 ( x ) θ 1− θ ∑m x =0 ( ) x 1− θ rn ( x ) β We can numerically solve the above estimating equation over θ ∈ [0, 1], or equivalently over the transformed parameter p := 1−θ θ ∈ [0, ∞], to obtain the corresponding MREE (i.e., the minimum LSD estimator). We simulate random sample of size n from a binomial population with true parameter θ0 = 0.1 with m = 10 and numerically compute the MREE. Repeating this exercise 1000 times, we can obtain an empirical estimate of the bias and the mean squared error (MSE) of the MREE of 10θ (since θ is very small in magnitude). Tables 1 and 2 present these values for sample sizes n = 20, 50, 100 and different values of tuning parameters α > 0 and β > 0; their existences are guaranteed by Corollary 2. Note that the choice α = 1 = β gives the maximum likelihood estimator whereas β = 1 only yields the minimum LDPD estimator with parameter α. Next, in order to study the robustness, we contaminate 10% of each sample by random observations from a distant binomial distribution with parameters θ = 0.9 and m = 10 and repeat the above simulation exercise; the resulting bias and MSE for the contaminated samples are given in Tables 3 and 4. Our observations from these tables can be summarized as follows. • Under pure data with no contamination, the maximum likelihood estimator (the MREE at α = 1 = β) has the least bias and MSE as expected, which further decrease as sample size increases. • As we move away from α = 1 and β = 1 in either direction, the MSEs of the corresponding MREEs under pure data increase slightly; but as long as the tuning parameters remain within a reasonable window of the (1, 1) point and neither component is very close to zero, this loss in efﬁciency is not very signiﬁcant. • When α or β approaches zero, the MREEs become somewhat unstable generating comparatively larger MSE values. This is probably due to the presence of inliers under the discrete binomial model. Note that, the relative (α, β)entropy measures with β ≤ 0 are not ﬁnitely deﬁned for the binomial model if there is just only one empty cell present in the data. • Under contamination, the bias and MSE of the maximum likelihood estimator increase signiﬁcantly but many MREEs remains stable. In particular, the MREEs with β ≥ α and the MREEs with β close to zero are nonrobust against data contamination. Many of the remaining members of the MREE family provide signiﬁcantly improved robust estimators. • In the entire simulation, the combination (α = 1, β = 0.7) appears to provide the most stable results. In Table 4, the best results are available along a tubular region which moves from the top lefthand to the bottom righthand of the table subject to the conditions that α > β and none of them are very close to zero. • Based on our numerical experiments, the optimum range of values of α, β providing the most robust minimum relative (α, β)estimators are α = 0.9, 1, 0.5 ≤ β ≤ 0.7 and 1 < α ≤ 1.5, 0.5 ≤ β < 1. Note that this range includes the estimators based on the logarithmic power divergence measure as well as the new LSD measures with α < 1. • Many of the MREEs, which belong to the optimum range mentioned in the last item and are close to the combination α = 1 = β, generally also provide the best tradeoff between efﬁciency under pure data and robustness under contaminated data. 29 Entropy 2018, 20, 347 In summary, many MREEs provide highly robust estimators under data contamination along with only a very small loss in efﬁciency under pure data. These numerical ﬁndings about the ﬁnite sample behavior of the MREEs under the binomial model and the corresponding optimum range of tuning parameters, for the subclass with α ≥ 1, are consistent with the ﬁndings of [78,79] who used a Poisson model. Additionally, our illustrations shed lights on the properties of the MREEs at α < 1 as well and show that some MREEs in this range, e.g., at α = 0.9 and β = 0.5, also yield optimum estimators in terms of the dual goal of high robustness and high efﬁciency. Table 1. Bias of the MREE for different α, β and sample sizes n under pure data. α β 0.3 0.5 0.7 0.9 1 1.1 1.3 1.5 1.7 2 n = 20 0.1 −0.210 −0.416 −0.397 −0.311 −0.277 −0.227 −0.130 0.021 0.024 0.122 0.3 2.218 −0.273 −0.229 −0.160 −0.141 −0.115 −0.096 −0.068 −0.036 0.034 0.5 −0.127 0.001 −0.125 −0.088 −0.082 −0.069 −0.058 −0.042 −0.032 −0.019 0.7 −0.093 −0.110 −0.010 −0.046 −0.044 −0.029 −0.023 −0.031 −0.023 −0.020 0.9 −0.066 −0.056 −0.028 −0.001 −0.015 −0.002 0.008 0.000 −0.006 −0.013 1 −0.041 −0.045 −0.017 0.005 −0.002 0.011 0.014 0.012 0.008 −0.003 1.3 −0.035 −0.013 0.023 0.036 0.030 0.039 0.088 0.039 0.035 0.021 1.5 −0.003 0.012 0.048 0.053 0.047 0.058 0.053 0.170 0.048 0.035 1.7 0.012 0.028 0.058 0.067 0.061 0.070 0.070 0.058 0.269 0.045 2 0.008 0.049 0.078 0.084 0.078 0.086 0.087 0.078 0.069 0.444 n = 50 0.1 −0.085 −0.301 −0.254 −0.183 −0.156 −0.106 −0.002 0.114 0.292 0.245 0.3 1.829 −0.176 −0.150 −0.078 −0.066 −0.042 −0.045 −0.014 0.005 0.030 0.5 −0.056 0.099 −0.054 −0.037 −0.033 −0.026 −0.019 −0.009 −0.007 −0.005 0.7 −0.009 −0.059 0.035 −0.012 −0.013 −0.005 −0.002 −0.009 −0.002 0.006 0.9 −0.031 −0.031 −0.009 0.012 0.002 0.013 0.021 0.015 0.008 0.004 1 0.014 −0.023 0.000 0.011 0.009 0.019 0.022 0.020 0.018 0.004 1.3 0.002 −0.004 0.022 0.034 0.027 0.030 0.084 0.034 0.035 0.028 1.5 0.009 0.023 0.038 0.044 0.037 0.042 0.034 0.174 0.040 0.032 1.7 0.028 0.029 0.049 0.054 0.047 0.050 0.047 0.036 0.277 0.039 2 0.040 0.051 0.065 0.068 0.059 0.063 0.060 0.051 0.041 0.464 n = 100 0.1 −0.028 −0.216 −0.175 −0.113 −0.103 −0.063 0.036 0.169 0.452 0.349 0.3 1.874 −0.135 −0.125 −0.052 −0.044 −0.022 −0.038 −0.023 0.009 0.024 0.5 −0.002 0.146 −0.034 −0.026 −0.025 −0.021 −0.019 0.001 −0.008 −0.009 0.7 0.000 −0.042 0.045 −0.009 −0.013 −0.009 0.000 −0.009 −0.008 −0.001 0.9 0.007 −0.025 −0.015 0.001 −0.004 0.005 0.009 0.013 −0.001 −0.003 1 0.014 −0.010 −0.007 −0.001 −0.001 0.005 0.009 0.014 0.010 0.009 1.3 0.036 0.010 0.006 0.015 0.010 0.010 0.065 0.012 0.019 0.014 1.5 0.041 0.023 0.018 0.022 0.017 0.018 0.006 0.158 0.016 0.015 1.7 0.052 0.027 0.028 0.032 0.024 0.025 0.016 0.009 0.267 0.019 2 0.056 0.043 0.042 0.043 0.033 0.034 0.023 0.020 0.013 0.454 30 Entropy 2018, 20, 347 Table 2. MSE of the MREE for different α, β and sample sizes n under pure data. α β 0.3 0.5 0.7 0.9 1 1.1 1.3 1.5 1.7 2 n = 20 0.1 0.347 0.251 0.222 0.145 0.122 0.106 0.098 0.242 0.206 0.240 0.3 7.506 0.147 0.100 0.069 0.063 0.059 0.059 0.062 0.098 0.169 0.5 0.238 0.076 0.067 0.051 0.049 0.047 0.050 0.055 0.064 0.101 0.7 0.177 0.091 0.056 0.045 0.044 0.043 0.045 0.055 0.056 0.071 0.9 0.163 0.085 0.061 0.045 0.042 0.043 0.047 0.053 0.058 0.064 1 0.171 0.085 0.064 0.045 0.042 0.045 0.048 0.053 0.058 0.063 1.3 0.148 0.082 0.065 0.052 0.046 0.046 0.061 0.055 0.058 0.065 1.5 0.146 0.085 0.069 0.056 0.050 0.050 0.051 0.087 0.061 0.065 1.7 0.150 0.085 0.070 0.060 0.053 0.055 0.055 0.056 0.134 0.066 2 0.132 0.091 0.076 0.065 0.059 0.060 0.060 0.060 0.061 0.265 n = 50 0.1 0.334 0.170 0.118 0.066 0.044 0.037 0.067 0.195 0.401 0.275 0.3 5.050 0.093 0.051 0.026 0.021 0.020 0.024 0.027 0.035 0.050 0.5 0.196 0.059 0.030 0.018 0.017 0.018 0.021 0.026 0.030 0.037 0.7 0.191 0.053 0.031 0.018 0.016 0.017 0.023 0.025 0.028 0.035 0.9 0.131 0.050 0.029 0.019 0.016 0.018 0.022 0.025 0.028 0.029 1 0.154 0.044 0.031 0.018 0.017 0.020 0.022 0.024 0.027 0.031 1.3 0.112 0.046 0.029 0.023 0.018 0.018 0.033 0.028 0.029 0.031 1.5 0.108 0.049 0.033 0.024 0.020 0.022 0.022 0.059 0.031 0.031 1.7 0.119 0.049 0.036 0.026 0.022 0.023 0.025 0.025 0.108 0.033 2 0.108 0.053 0.040 0.030 0.025 0.026 0.028 0.029 0.028 0.249 n = 100 0.1 0.295 0.139 0.085 0.038 0.022 0.022 0.068 0.201 0.583 0.403 0.3 4.770 0.075 0.039 0.016 0.011 0.011 0.017 0.019 0.023 0.035 0.5 0.189 0.061 0.022 0.011 0.009 0.012 0.016 0.017 0.022 0.023 0.7 0.141 0.038 0.024 0.010 0.009 0.010 0.014 0.017 0.018 0.021 0.9 0.123 0.035 0.021 0.011 0.009 0.011 0.012 0.015 0.019 0.021 1 0.122 0.036 0.019 0.010 0.009 0.011 0.013 0.016 0.017 0.020 1.3 0.114 0.035 0.019 0.012 0.009 0.010 0.021 0.016 0.017 0.019 1.5 0.105 0.037 0.019 0.012 0.010 0.011 0.012 0.045 0.017 0.020 1.7 0.097 0.034 0.021 0.014 0.011 0.012 0.014 0.014 0.092 0.020 2 0.088 0.039 0.023 0.016 0.012 0.013 0.013 0.016 0.016 0.227 Table 3. Bias of the MREE for different α, β and sample sizes n under contaminated data. α β 0.3 0.5 0.7 0.9 1 1.1 1.3 1.5 1.7 2 n = 20 0.1 −0.104 −0.382 −0.340 −0.243 −0.131 −0.071 0.090 0.188 0.295 0.379 0.3 3.287 −0.157 −0.187 −0.135 −0.113 −0.091 −0.045 0.013 0.107 0.237 0.5 2.691 1.483 −0.024 −0.067 −0.069 −0.043 −0.031 −0.010 −0.003 0.051 0.7 3.004 2.546 1.168 0.036 −0.017 −0.008 0.003 0.006 0.005 0.010 0.9 3.133 2.889 2.319 0.917 0.222 0.058 0.019 0.023 0.017 0.022 1 3.183 2.986 2.558 1.619 0.805 0.214 0.039 0.030 0.031 0.019 1.3 3.239 3.121 2.902 2.550 2.262 1.872 0.613 0.077 0.049 0.040 1.5 3.255 3.170 3.012 2.775 2.606 2.396 1.676 0.571 0.069 0.051 1.7 3.271 3.194 3.071 2.903 2.790 2.661 2.256 1.489 0.578 0.057 2 3.289 3.216 3.122 3.012 2.942 2.865 2.649 2.305 1.690 0.682 31
Enter the password to open this PDF file:











