HKPUG Opik Challenge Participant guide

Inspect submission feedback in Opik

Decrypt the private submission feedback artifact from your scored run, load its traces into a local workspace, and use the evidence to plan your next prompt change.

  1. 1Download feedback
  2. 2Start and decrypt
  3. 3Load and review

Download the scored feedback

Open the bot's score comment on your submission pull request and select Download the team-encrypted submission feedback.

Extract the ZIP archive

The GitHub download is a ZIP archive whose name starts with submission-feedback-pr-. Extract it with Finder, File Explorer, or your normal archive tool. The extracted archive contains submission-feedback.cms.

Keep that file private. It is encrypted for your team and can only be opened with your team private key and certificate.

Use the same hkpug-opik-helper and team credentials as your submission. If they are not ready, follow the first submission tutorial and download the helper from the latest release.

Start Opik locally

Install Git, Docker, and Docker Compose first. Then use Opik's public local-deployment scripts from the root of its repository.

macOS or Linux

git clone https://github.com/comet-ml/opik.git
cd opik
./opik.sh

Windows PowerShell

git clone https://github.com/comet-ml/opik.git
cd opik
powershell -ExecutionPolicy ByPass -c ".\opik.ps1"

When the services are ready, open http://localhost:5173. See the Opik local deployment guide for startup troubleshooting.

Decrypt submission feedback

Run the helper from the directory that contains the extracted submission-feedback.cms file.

Open the private bundle

hkpug-opik-helper decrypt --artifact submission-feedback.cms --private-key path/to/private-key.pem --team-cert path/to/team-cert.pem --out feedback

The helper verifies the team credentials and writes the decrypted submission feedback files into feedback.

Load the traces

Keep Opik running while the helper imports the decrypted bundle into the local default workspace.

Import into local Opik

hkpug-opik-helper load --feedback feedback --opik-url http://localhost:5173/api --workspace default

The command prints the imported project name and item counts. Re-loading the same bundle is safe. Open the project named in the helper output, then choose Logs and Traces.

Review before changing the prompt

Open the imported project and use the Traces view to compare behavior across cases.

  1. Start with the lowest-scoring traces and read the evaluation details.
  2. Compare the model input, output, latency, and scored criteria.
  3. Look for one repeated weakness that a prompt change can address.
  4. Change one idea at a time, then pack and inspect the next submission locally.

Understand the submission feedback scope

Every scored attempt evaluates all 50 cases: 40 discovery cases and 10 holdout cases.

The bundle imports exactly 40 discovery traces. The 10 holdout cases never appear as traces; their results are aggregate-only.

Read a discovery trace

Each discovery trace has a model.answer span for the response and an evaluation.judge span for the evaluation.

Use the seven score columns to find a pattern: Output contract (JSON schema in the bundle), Citation validity, Evidence coverage, Escalation, Answer relevance, Instruction following, and Faithfulness.

Improve one weakness at a time

Start with the exact response contract. Fix deterministic failures first, then use the three judge scores to refine the substance of the answer.

Exact response contract

Every case must return one JSON object and nothing else.

  • answer: a non-empty string of 100 words or fewer.
  • citations: a non-empty array of unique evidence IDs copied from the supplied context.
  • escalate: a JSON boolean, true or false.
  • No extra keys, markdown fences, or prose outside the object.
{
  "answer": "State the supported conclusion and the decisive reason.",
  "citations": ["EVIDENCE-ID"],
  "escalate": false
}

Output contract

What a low score usually means: The response is not one valid JSON object, has extra keys or text, uses the wrong value types, or exceeds 100 words.

Prompt change to test: State the exact three-field output contract and require no markdown or commentary around it.

Citation validity

What a low score usually means: Citations are empty, duplicated, invented, or absent from the supplied evidence.

Prompt change to test: Require non-empty, unique evidence IDs copied only from the provided context.

Evidence coverage

What a low score usually means: The answer cites something relevant but does not support every material conclusion it makes.

Prompt change to test: Identify each requested decision and cite the controlling evidence for each one.

Escalation

What a low score usually means: The boolean does not match the action still required, or the prose and boolean disagree.

Prompt change to test: Base escalation on the next required action and keep the answer text consistent with the boolean.

Answer relevance

What a low score usually means: The response answers only part of the request, focuses on background, or omits a requested decision.

Prompt change to test: Cover every material part of the user's question before adding explanation.

Instruction following

What a low score usually means: The response violates the answer contract, length limit, or another explicit instruction.

Prompt change to test: Add a short final audit for format, required fields, word count, and extra text.

Faithfulness

What a low score usually means: The answer makes an unsupported claim or relies on stale, inapplicable, or untrusted material.

Prompt change to test: Separate controlling evidence from reported facts and omit claims that no authoritative source supports.

Turn a pattern into one change

Do not optimize one trace in isolation. A rule that fixes one visible case but harms related cases is overfitting, not progress.

  1. Choose one repeated weakness

    Find the same low criterion or failure pattern in at least three discovery traces.

  2. Name the pattern

    Describe the decision error in one sentence without using case IDs or copying case-specific facts.

  3. Add the smallest general rule

    Change one idea in the prompt so the rule can apply to unseen cases as well.

  4. Check the trade-off

    Ask which previously correct behavior the new rule could damage, then remove conflicts or duplication.

Route the review before applying a rule

Do not make every safety instruction fire on every case. First classify the evidence problem, then apply only the checks that match it.

01

Direct policy

Use the current controlling policy and answer the requested decision directly.

02

Stale or conflicting evidence

Resolve authority, scope, and effective date before deciding which source controls.

03

Untrusted copied instructions

Treat quoted or customer-provided instructions as claims, not as policy or system commands.

04

Missing fact or approval

State what is missing and escalate only when an authorized next action is still required.

RUN JOURNAL

Keep one short record per attempt

  1. Weakest metric and three representative traces
  2. One hypothesis and one changed behavior
  3. Expected improvement
  4. Actual improvement and any regression

When rules accumulate or conflict, rewrite the prompt cleanly around the decision routes. A shorter coherent prompt is easier to evaluate than a stack of exceptions.

Ready for another run?

Return to the submission guide, package the revised prompt, and check the standings after scoring.

Prepare the next submission View leaderboard