We’re always trying to improve and refine the judging system of the International Chocolate Awards, based on our experience as we run more competitions, on the feedback we receive from judges and entrants and from the results of our annual review surveys.

2013 sees a number of improvements, particularly for the selection round. We’ve also been refining and formalising the Grand Jury process to ensure that it’s is as fair as possible and reflects the Main round scores, while allowing the Grand Jury to work in an effective way as the final arbiters of technical issues and the competition prizes.

Improved selection round feedback

One of our aims with the International Chocolate Awards is to help support upcoming chocolate companies and to help them to improve through constructive feedback. It’s difficult though, when hundreds of samples are being judged in some of the larger competitions, to find ways to capture the opinions and notes of our judges while they are tasting.

We always try to limit the number of samples tried in our main two-hour judging sessions to a maximum of 15-20 samples per judge – with around 15 to 18 being the ideal. Even so, it would be difficult for any judge to keep writing positive and useful comments about each sample, especially if they are not technically trained in every area of chocolate production and tasting. (We aim to include a wide range of judges with good palates, including sommeliers, chefs, journalists and others, so not all of our judges are chocolate specialists.)

Main round feedback

Since our first competition, we have given our judges a series of pre-written statements identifying a range of positive and negative qualities about each sample, so the judges can quickly circle the letter corresponding to each statement. This lets us reliably input the statements into our judging system and then the feedback is (anonymously) output through the secure part of our website for each entrant to review.

In 2012 this system worked well for products that made it through to the Main judging round.  Each judge is given one judging form for each product they try, which includes the sample scoring section and also space for an extensive list of feedback statements. We feel that this gave quite comprehensive positive and negative feedback, which is far more controlled and far beyond any other competition.

Selection

However, if a product was unlucky enough to not make it through to the Main round of judging, we gave only limited feedback in broad areas where the product either had problems or in areas where it could improve. An example of the areas for filled chocolates would be ‘Execution, Formulation, Interpretation and Taste’. So an entry might be rejected for ‘Formulation’ and ‘Taste’, but have ‘Execution’ and ‘Interpretation’ marked as areas of potential.

We kept things very simple, because the Selection round is meant to be a quick-fire round judged by more experienced judges. The aim is to reduce the numbers before we get to the Main round judging in each competition. This is always a trade-off – we want to give each sample the best possible chance and the maximum judging time, so it’s important that judges are not overwhelmed by the number of samples in the Main round.

During Selection, entries are rejected if they have technical flaws, contamination, are spoiled or are very unlikely to have a chance of winning. If there is any doubt for an individual judge, they can vote ‘Further judging’, so the sample can go through. Each sample gets a single line on a form for up to 18 samples, with space just to mention the criteria areas mentioned above.

We’ve recognised though that this limited level of feedback might not be terribly useful, especially if you are a new or aspiring company and most or all of your entries didn’t get past Selection. This was one of the points we covered in our 2013 Entrant Survey, and your feedback on our feedback was that we could do better!

We’re always trying to improve our system, so we’ve come up with an improved Selection form for 2013 to address these issues.

Better Selection feedback – no rejection without a reason

For the main competitions in 2013, we have a new Selection form in use. There’s still one line per entry on each judge’s form and the same votes for ‘Reject’, ‘Further judging’ and ‘Award contender’.

For 2013 though, we’ve added an extensive list of technical reasons listing possible problems with a product. We’ve also added a new rule – no Selection round judge may reject a sample without giving a reason. This is done by simply marking the relevant letter in a box on each line. This also takes away any temptation for a judge to skip over giving any feedback for a particular sample.

We’ve actually allowed space for up to three reasons, but there must be at least one. (We feel three is more than enough and gives an entrant good guidance on what needs improving to create a great product.) The possible reasons are tailored to each class of product – plain bars, milk bars, flavoured bars, filled chocolates and spreads.

We’ve also included a short but improved list of ‘Areas of Potential’. These are less important, as a product with good potential will go through to the Main round, but we are still encouraging Selection judges to add these, and if they vote ‘Further judging’, they must give at least one ‘Areas of Potential’ reason.

We feel confident that the new version of the Selection form will give a much more useful level of feedback for this year and that it also gives the Selection judges much better guidance on the exact reasons for why they might reject an entry.

Our judging system is completely open and transparent and all the forms can be downloaded from our Judges Systems and Forms page. (We do maintain copyright on the forms though and ask that they are not used without permission.)

(Note: due to time constraints we were only able to implement this system in time for the European Semi-final, the Israeli and Italian competitions for 2013 used the previous system. All future competitions will use the new system, which we will continue to refine and improve.)

Grand Jury

The Grand Jury round has been one of the areas that has had the most development since we began the International Chocolate Awards at the start of 2012 in Italy. Our Grand Jury members from last year will remember the 12.30am finish at the first European Semi-final in May – to get through everything we were still tasting and (sometimes hotly) debating the merits of each entry well past the time allotted for our communal end of Awards celebration meal.

We’ve learnt a lot since then and the Grand Jury session is now very structured and coordinated with its own set of rules.

First we determine which of the highest scoring entries from the main round are contenders for Gold and Silver awards or if not, can be considered for Nominations to compete in the World Final. This is done be establishing a statistical ‘break point’ based on analysis of the Main round scores.

Each Grand Juror is then given a form for the current category being considered that lists the entries in score order, the scores and a column to vote in.

The potential winners are then presented with a break down of the scores and positions For the first time in the competition, discussion while tasting is encouraged. This is to be sure that any technical points or problems are fully identified. In larger categories, each product will be presented by a Grand Juror who will act as its advocate, pointing out its strong points or any problems, which could be technical or relating to the qualification rules or categories.

Blind tasting

As with the rest of the competition, only the judges’ anonymised description is used, the tasting is still completely blind until the end of the competition.

On a side note, we accept that it’s inevitable that some judges will always recognise some products, but we believe credible judges will not be swayed by this and we always try to include a range of Grand Jurors from different countries and experiences. Knowledge of the subject is essential, and this inevitably – and should – mean some recognition of products. This is especially true when it comes to plain origin bars – any decent taster should be able to recognise different cacao origins and manufacturing styles right away.

Voting

Once the tasting is finished, the Grand Jury votes in private. The votes are then collated and averaged and the prizes are proposed, still blind at this point. Ideally this is done during the Grand Jury session, but occasionally this is not possible and final details have to be ironed out by email. (Or if the prize announcements are delayed for a later ceremony.)

It’s important that the Main round scores are respected, given all the hard work put in and the range of judges present at the Main judging. So the Grand Jury is not allowed to ‘move’ the medal position, as determined by the Main round scores, by more than one place unless there is a technical reason for doing so. So a Gold can go to a Silver by voting, but not down to a Nomination or no prize, and the same applies in the other direction.

Grand Jury members may not vote an entry up or down just because they personally like or dislike it (or recognise the product).

The Main round scores can only be overridden by more than one place if a breach of rules is found or if a technical flaw is found. Examples of this could include: mislabeling, ingredient problems, mould or other off notes detected in plain bars, poor execution not noticed in main judging, artificial flavouring detected, use of poor or off tasting chocolate, being overly sweet or bland. These are just a few examples.

Ideally these types of problems will all have been caught at earlier stages, but the Grand Jury round is where full consideration of the top entries can be made.

The Grand Jury’s role is to make a thorough examination of the potential winners and to ensure that all prize winners live up to the high standards set by the International Chocolate Awards.

Grand Jury rule changes

So to sum up the improvements we’ve made at the end of 2012 and early 2013:

• Grand Jury round is now judged completely blind
• Personal judging forms are given out with the potential winners and scores
• Voting is completely private (compiled by Grand Jury head)
• Grand Jury cannot change the prize position by more than one place unless a technical or rule error is found
• Individual Grand Jurors must give a technical or rule based reason on their form if they move an entry by more than one position
• Based on the results of the 2012 Entrants Survey, all the names and companies of all entries that made it into the Grand Jury round will be published on the Awards website, not just the prize winners. (This does not though include and entries disqualified for technical or rule qualification reasons.)

The Grand Jury rules can be downloaded and read as a PDF here: International Chocolate Awards – Grand Jury round rules

Judges registration

In another innovation for 2013, we’ve now introduced an online registration process for all judges. Existing and potential judges now have to sign up and complete the registration form, which includes questions asking for detailed information about the judge’s background, employment, current level of knowledge and anything that might constitute a potential conflict of interest.

Before being asked to choose their judging sessions for a competition, applications are first checked and approved. This system also lets us manage communication with judges in a better way and also to manage the judging sessions. Judges should see more improvements coming from this in the coming year.

As part of our policy for a transparent and fair Awards, we will also now publish the names of judges for each competition, unless they choose to opt out.

Other changes coming out of our 2012 review surveys included new white chocolate categories and a new category for ‘mixed’ chocolate types. We’ll continue to refine and improve our process and adapt to changes in the chocolate market. We welcome your feedback either through our year end surveys or you can contact us through our online form here.