An international contest on symbol recognition will be held during the 5th IAPR workshop on graphics recognition (GREC’2003) in Barcelona, July 2003. The idea is to compare various methods for recognizing linear symbols, i.e. symbols made of lines, arcs, and simple geometric primitives. Because of the large variety of possible symbols and of application domains, this first announcement aims primarily at explaining the organizers’ views on the way the contest should be run, to get feedback from possible participants.
domains: the idea is to have
symbols from various domains (electronics, architecture …) and to let contestants
choose to participate in all domains or only in some of them.
data format: the symbols to be
recognized will be available in 3 formats – vector format (in a format inspired
by the VEC format of previous contests), scanned binary image, scanned grey-level
Ground truth: models for each symbol to be recognized will be made available as perfect symbols in VEC format, “perfect” scans at 600 dpi, both grey-level and binary, and a number of degraded versions (see below).
Segmentation: basically, the symbols will be pre-segmented, i.e. disconnected from their context. An optional step will be to provide a few complete drawings, containing a number of different symbols, to assess the methods’ ability to segment or extract and recognize non-segmented symbols.
Degradation: the “perfect” vector symbols will be progressively degraded using a vector deformation model. At each level of vector degradation, scans will be performed at 600 dpi, 300 dpi, 200 dpi, 100 dpi and maybe even lower resolutions. In addition, photocopies, folding and other physical degradations will be applied to printed symbols before new scans are performed. When invariance to rotation makes sense, rotated instances of symbols will also be provided.
Scalability: for each application domain chosen for the contest, there will be at least two sets of possible symbols, one being a subset of the other, to test the scalability of recognition methods when the number of symbols to recognize increases.
Training: all models appearing in the actual contest, and a number of degraded instances of these models, will be made available to the contestants sufficiently early (a couple of months before) to allow for all training to be performed in advance of the contest.
The contest itself: on the day of the contest, each participant will run his/her methods on her/his own laptop computer. There may be a special day – one day before the workshop – for running all the tests.
Performance criteria to be measured: we think of assessing the methods’ robustness to noise, their scalability (discrimination potential) when the number of symbols increases, their complexity (simply measured as the increase of computation time when noise increases or when the number of models increases), and, optionally, their ability to segment symbols which are not disconnected. In this first edition of the contest, we will limit ourselves to assessing discrete responses of symbol recognition methods (correct recognition, wrong recognition, reject) and not confidence values or scores.
Presentation of results: the idea is not to have as outcome something like “the winner is…” but rather a number of tables, where the various criteria (robustness, scalability, complexity, segmentation) are presented, with a different table for each application domain, and for each type of input data (vector, binary image, grey level image), and the possibility to compete only in a subset of all the possible combinations. Ultimately, the result obtained at the workshop should be the starting point for a thorough collaboration between the teams involved, and the organizers of the contest, resulting in a deeper understanding of the characteristics, pros and cons of various approaches to symbol recognition, to be published in one or several journal articles (a special issue of one of our reference journals could be one of the possibilities).
More information, declaration of interest: to help us define precisely the application domains and the precise scope of the contest, we need your input! Symbol recognition has a very wide scope, and we do not pretend having thought of all possible areas and methods. Potential participants are kindly requested to pre-register by filling out a detailed questionnaire, available at the feedback section of this site, before October 31, 2002.