Judging system and forms

When we decided to create the International Chocolate Awards in 2011, it was important to us to develop a judging system that was fair and transparent and easy to use. We wanted out new awards to be judged in a way that would be seen to be well thought out and credible.

With this in mind, we started to build our International advisory committees with independent representatives from around the chocolate world. Some of the first work of the committees has been to create a brand new scoring system that is based on opinion, but using a controlled and fair system that has been developed with extensive testing in five different countries.

We wanted a system that went beyond the traditional ‘marks out of 10’ approach, for example as used for Seventy% bar reviews. This can be fine for considered reviews made over a few days, but not for judging many samples together in a competition, especially when there is a mix of levels of chocolate knowledge among the judges. ( We think a mixed level is a good thing, because it helps represent wider consumer tastes as well as the views of chocolate experts.)

You can download our judging forms at the end of this page.

Managed subjectivity

We have developed a new system for the Awards that we like to call ‘managed subjectivity’. This idea accepts that chocolate judging is a matter of taste, and that personal sensorial aspects such as flavour are paramount, but uses a scoring system that helps guide judges on exactly what mark means what. This is all based on the work and suggestions of Seventy% reviewer Alex Rast, using ideas originally developed for the Seventy% review section.

The system identifies key areas in the assessment of a chocolate sample and assigns marks out of five for each of those. For example, for filled chocolates, we consider:

Execution
Formulation
Interpretation
Taste

Each of these receives a mark out of five, with a weighting towards a 100% total. We don’t just leave judges to each make their own and probably different interpretation of say, what ‘4’ means though. We give guidance statements for each mark that help judges quickly and consistently decide which mark is appropriate.

The idea is that all judges will have more or less the same meaning for a mark of ‘4’ and that once judges are used to the system, they can quickly mark each product.

Award

In addition to the total score out of 100, we also ask judges to tell us whether a product deserves an award, again on a scale of 1-5. This gives a second, straight-forward evaluation of the product, and is useful for the Grand Jury when considering which products should receive an award.

Feedback

Another important element of the form is that it gives us a way of providing entrants with feedback from the judges, both positive and negative. This has been a common request from entrants, and was something that you told us was wanted in our online survey.

It can be difficult though for judges to write constructive comments about every sample. It’s all too easy to write negative remarks in the heat of the moment, but these are not helpful for entrants and could be seen as offensive or derogatory.

Our answer then is to provide fixed feedback statements on the entry form that address technical aspects of the sample judging. These just require a judge to become familiar with the statements, then tick the appropriate letters.

Check total

As a final statistical check, we also ask judges to give a quick overal ‘mark out of 10’. This mark does NOT count towards the total judging, but gives us a way to check for statistical variation or errors. So if a judge gave low section marks but a high check total, we’d know to check for misunderstandings or errors.

Selection

One of our main concerns has been the danger overloading the palates of the judges. After too many samples, our ability to fairly and correctly test samples goes down as our palates become clogged with sugar and tannins. This is not fair either to entrants or to the judges who are asked to make decisions when they may be physically or mentally tired, as well as have a tired palate.

One method we are using to reduce this is to begin the judging with a first round of ‘auditions’. In this ‘Selection Round’, all samples are tested by a small panel who vote on whether or not the sample has a chance to win an award. If a majority vote yes, the sample goes through to main judging. If the majority reject the sample, the product does not go through. If there is doubt or a borderline decision, then the sample is let through.

The exact level at which samples will be let through will depend on the number of samples that can be judged in the later rounds, but generally a vote of over 50% allows samples through, and we expect to remove about on third of the total number of samples during this round.

New in 2013, we also give judges a list of feedback statements to choose from for each entry, as described for the Main round feedback above. Because we don’t believe that a sample should be rejected without a good reason, from this year, Selection round judges must give at least one reason taken from a standard list if they decide to reject a sample, eg for technical reasons.

Palate calibration

We’ve also been experimenting to find out just exactly how many sample a judge can take before their palate begins to significantly change. We’ve run a system of tasting four different couvertures before judging begins, then retesting on of these every 5 or 6 samples to see how much our palates have changed.

From this, we’ve consistently concluded that palates really begin to go off after 15-20 samples. Because of this, we try to limit the number of samples any judge tastes in one session to 15, with a maximum of 20. Sessions last 1.5 hours, with a 1 hour break for judges staying for more than one session. We’ll also randomise the sample order for different judges, so the affects of palate ‘wear’ don’t unfairly disadvantage some entries.

This system has worked so well that we’re actually introducing it into the judging sessions. All judges will try three couverture chocolates together at the start of each session, then repeat with a single chocolate about every 5 samples. This will be done with discussion and will help the judges to warm up, then have a personal awareness of how their palate is changing.

Palate cleansing

In another innovation, we’ve introduced soupy polenta as our palate cleanser of choice. After extensive testing, it’s been shown to be the best for defending the palate against build up of tannins and sugar. Thanks again to Alex Rast for that one! We’ll also have the traditional choices of water, bread, apples, crackers etc available during the judging sessions.

Testing

Part of our ethos for the awards has been not to have a set of rules or a judging method drawn up behind closed doors and without a proper test until the first day of judging. To this end, we’ve carried out a full set of test, trials, betas and improvements to our forms and system. We’ve had tests in London, New York, Vancouver, Sweden and Germany, followed by our first live test at the Italian Chocolate Awards in April.

Our Independent Advisory Committed and Grand Jury have been working together to develop the system, bringing an unprecedented and international range of expertise to help with our development. We’ve also had help from bloggers, societies and chefs and other experts. Thanks to all those involved so far.

There is always more to learn, and we welcome your input as we develop. We’re confident though we’ve come up with a good workable system, which takes a big leap forward in fair and consistent judging.

Judging forms

As part of our commitment to transparency, we’re publishing our judging forms so that both entrants and the wider public can see our approach to judging.

You can view and download the latest versions of our judging forms here. Please note that the forms are constantly tweaked and refined according to rule changes and judges and entrants feedback. The forms used in previous competitions may differ from the current versions.

2014 – English

2014 – Italian

2014 – German

Please note that all the forms are copyright the International Chocolate Awards and must not be reproduced, copied or used in any form of comptition without written permission from the Awards organisers.

Professional