BazEkon - The Main Library of the Cracow University of Economics

BazEkon home page

Main menu

Veček Niki (University of Maribor, Slovenia), Mernik Marjan (University of Maribor, Slovenia), Črepinšek Matej (University of Maribor, Slovenia), Hrnčič Dejan (University of Maribor, Slovenia)
A Comparison between Different Chess Rating Systems for Ranking Evolutionary Algorithms
Annals of Computer Science and Information Systems, 2014, vol. 2, s. 511 - 518, rys., tab., bibliogr. 31 poz.
Algorytmy, Ranking, Eksperyment badawczy
Algorithms, Ranking, Scientific experiment
Chess Rating System for Evolutionary algorithms (CRS4EAs) is a novel method for comparing evolutionary algorithms which evaluates and ranks algorithms regarding the formula from the Glicko-2 chess rating system. It was empirically shown that CRS4EAs can be compared to the standard method for comparing algorithms - null hypothesis significance testing. The following paper examines the applications of chess rating systems beyond Glicko-2. The results of 15 evolutionary algorithms on 20 minimisation problems obtained using the Glicko-2 system were empirically compared to the Elo rating system, Chessmetrics rating system, and German Evaluation Number (DWZ). The results of the experiment showed that Glicko-2 is the most appropriate choice for evaluating and ranking evolutionary algorithms. Whilst other three systems' benefits were mainly the simple formulae, the ratings in Glicko-2 are proven to be more reliable, the detected significant differences are supported by confidence intervals, the inflation or deflation of ratings is easily detected, and the weight of individual results is set dynamically.(original abstract)
Full text
  1. Brest J., Greiner S., Boskovic B., Mernik M., Zumer V. Self-adapting control parameters in differential evolution: A comparative study on numerical benchmark problems. IEEE Transactions on Evolutionary Computation, 10(6):646-657, 2006.
  2. Cohen J. The earth is round (p < .05). American psychologist, 49(12):997-1003, 1994.
  3. Crepinsek M., Liu S. H., Mernik L. A Note on Teaching-Learning-Based Optimization Algorithm. Information Sciences, 212:79-93, 2012.
  4. Das S., Suganthan P. N. Differential evolution: A survey of the stateof-the-art. IEEE Transactions on Evolutionary Computation, 15(1):4-31, 2011.
  5. Deutscher Schachbund [Online]. Available:
  6. Dowell M., Jarratt P. A modified regula falsi method for computing the root of an equation. BIT Numerical Mathematics, 11(2):168-174, 1971.
  7. Dyba T., Kampenes V. B., Sjoberg D. I. A systematic review of statistical power in software engineering experiments. Information and Software Technology, 48(8):745-755, 2006.
  8. Elo A. E. The rating of chessplayers, past and present (Vol. 3). Batsford, 1978.
  9. Epitropakis M. G., Plagianakos V. P., Vrahatis M. N.. Balancing the exploration and exploitation capabilities of the differential evolution algorithm. IEEE World Congress on Computational Intelligence 2008, 2686-2693, 2008.
  10. Evolutionary Algorithms Rating System (Github) [Online]. Available:
  11. Evolutionary Algorithms Rating System [Online]. Available:
  12. Friedman M. A comparison of alternative tests of significance for the problem of m rankings. Annals of Mathematical Statistics, 11:86-92, 1940.
  13. Friedman M. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32:675-701, 1937.
  14. Gill J. The insignificance of null hypothesis significance testing. Political Research Quarterly, 52(3):647-674, 1999.
  15. Glickman M. E. A comprehensive guide to chess ratings. American Chess Journal, 3:59-102, 1995.
  16. Glickman M. E. Dynamic paired comparison models with stochastic variances. Journal of Applied Statistics, 28(6):673-689, 2001.
  17. Glickman M. E. Example of the Glicko-2 system. Boston University, 2012.
  18. Glickman M. E. Parameter estimation in large dynamic paired comparison experiments. Journal of the Royal Statistical Society: Series C (Applied Statistics), 48(3):377-394, 1999.
  19. Glickman M. E. The glicko system. Boston University, 1995.
  20. Hansen N. The CMA Evolution Strategy: A Comparing Review. Towards a new evolutionary computation, Springer, 75-102, 2006.
  21. Hooper D, Whyld K. The Oxford Companion to Chess. Oxford University Press, 1992.
  22. Kitchenham B. A., Pfleeger S. L., Pickard L. M., Jones P. W., Hoaglin D. C., El Emam K., Rosenberg J. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721-734, 2002.
  23. Neyman J., Pearson E. On the problem of the most efficient test of statistical hypothesis. Philosophical Transaction of the Royal Society of London - Series A, 231:289-337, 1933.
  24. Rao R. V., Savsani V. J., Vakharia D. P. Teaching-Learning-Based Optimization: An optimization method for continuous non-linear large scale problems. Information Sciences, 183(1):1-15, 2012.
  25. Rastrigin L .A. The convergence of the random search method in the extremal control of a many-parameter system. Automation and Remote Control, 24(10):1337-1342, 1963.
  26. Rechenberg I. Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Frommann-Holzboog, 1973.
  27. Sonas J., Februar 2014.
  28. Tang K., Li X., Suganthan P. N., Yang z., Weise T. Benchmark Functions for the CEC2010 Special Session and Competition on Large-Scale Global Optimization. Nature Inspired Computation and Applications Laboratory, 2009.
  29. Tvrdik J. Adaptive differential evolution: application to nonlinear regression. In Proceedings of the International Multiconference on Computer Science and Information Technology, 193-202, 2007.
  30. Vecek N., Mernik M., Crepinsek M. A Chess Rating System for Evolutionary Algorithms - A New Method for the Comparison and Ranking of Evolutionary Algorithms. Information Sciences, 277:656-679, 2014.
  31. Zaharie D. A comparative analysis of crossover variants in differential evolution. Proceedings of IMCSIT, 171-181, 2007.
Cited by
Share on Facebook Share on Twitter Share on Google+ Share on Pinterest Share on LinkedIn Wyślij znajomemu