How the scoring is done in challenge problem?
During the contest, the master judge returns the score for only a certain percentage (20% is the current number) of the total uploaded test files. However, the test case judge runs for all the test files and computes the scores for all test files. The master judge saves them internally. It does not show this score to the user. Also, if there is any error (WA, TLE, RTE) in the solution against any test file, including the files for which the score is not shown, the judge should return the error to the user.
After the contest is over, the total score averaged over all the test files (including the remaining 80%), is shown to the user after doing a quick rejudge for the problem (which will only copy the pre competed score and not judge all the solutions once again).
What is the change?
We have simplified the process for this 80%-20% scoring in the challenge problem. Now we just need to pass the test file number from the "auxiliary field" to the master judge. The master judge will return score for only those files. Scores for other files will be computed but not shown during the contest.
Previously, we used to do a hard rejudge of all the AC submissions on challenge problem due to which many of the solutions turned out to be TLE due to high load and resource contention on the judge. Now onwards, we will be saving the score of all the testcases (during the contest) in a temporary table and will swap the scores from this temporary table (after the contest). Thus saving us time and the TLE issue mentioned above will be addressed.
Once the contest is over, just by clicking Quick Rejudge button, the scores will get updated.
How to configure a Challenge problem from now on?
There are two phases now:
- Before/During the contest
- After the contest
The settings mentioned below need to be configured before/during the contest:
- Problem Type: You can select the type of the problem from classic or interactive.
- Master Judge:The master judge to be used should be 1004 (available in the dropdown) for challenge problem henceforth (we have modified and used the existing judge written by Sergey Kulik). After this change, problem setters won't have to take a pain of writing a custom master judge. :)
- Problem test sequence: Use this field as usual to define the number of test files to be included for the solution to run. (This should have all the test files that you have added) Auxiliary data passed to masterjudge: Put the serial number of test cases which you want to return score during the contest.
- Eg. If you have test cases 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and you want only cases 1, 4 and 5 to be considered to show score during the contest, just put 1 4 5 in this field. Please don't get confused with the format of the data you put in Auxiliary data field for subtask type problem. You may still need to write a custom judge for test files if your problem needs.
After the contest is over, the test files' numbers mentioned in Auxiliary data passed to masterjudge should be removed. This should be done before moving the problems to practice section. This is done so that in the practice section, all the users get the score on the entire test data.
After this, as mentioned above we will need to a "Quick rejudge" for all the solutions which will copy the precomputed scores on all test files, to the main score stream and on completion of this process, the users will be able to see the final scores.
PS: Number of submissions allowed in a challenge problem in CodeChef's Long Challenge is 200