Software Engineering

Implementing features, repairing defects, and maintaining repositories: tests, migrations, dependency upgrades, and build or CI fixes inside realistic project workspaces.

81 tasks2 subcategories81 general tasksArtifact-based oracle scoring

Mean Score

57.34

general suite

Score ≥ 0.80

39.37%

R0.8

Full Credit

22.75%

R1

Averaged over all 81 tasks in this category, each evaluated under the 70 general model-harness configurations (7 harnesses × 10 models). Open a task to inspect its instruction, package files, and the trajectory of every run.

Implementation and Repair

67 tasks

Repair monorepo affected-build planning without stale cache reuse

Bisect and Repair an Invoice Eligibility Regression

Repair a cross-platform archive extraction preflight

Repair typed layered configuration without leaking secrets

Retire a checkout feature flag without breaking historical replay

Implement a backward-compatible batch quote contract

Repair HTTP cache revalidation across validators and Vary variants

Recover an interrupted wheel installation transaction safely

Recover OAuth refresh-token rotation after concurrent requests

Reconcile distributed rate-limit leases without oversubscribing capacity

Repair deterministic workspace dependency resolution across peers and platforms

Repair consumer-driven API schema compatibility analysis

Repair an order-independent integration-test runner with bounded quarantine

Repair locale fallback, plural selection, and message interpolation

Repair an HTTP range download without corrupting the staged artifact

Reconcile event-time windows after late and replayed events

Repair destructive CLI confirmation across TTY and piped input

Repair a reproducible release-archive manifest planner

Repair a Git submodule state-recovery planner without losing local work

Repair an atomic cross-ecosystem release-version synchronization planner

Repair a collision-safe Git case-rename recovery planner

Repair an idempotent SQLite migration transaction-recovery planner

Recover a checksummed multipart CI artifact upload safely

Repair Python wheel selection across ABI and platform tags

Repair HTTP retries across unknown non-idempotent outcomes

Repair recursive structured-log redaction without destroying diagnostics

Repair a file watcher across atomic saves, duplicate events, and inode replacement

Repair an asyncio worker pool with bounded backpressure and cancellation cleanup

Repair a versioned JSON importer without losing explicit state

Repair bounded TTL/LRU cache eviction without destructive failed writes

Repair incremental JSONL recovery across partial records and rotation

Repair webhook verification across a signing-key overlap window

Repair transactional plugin activation and stale-hook cleanup

Repair recurring schedules across DST gaps folds and timezone changes

Repair a Unicode identifier migration without silent account merges

Repair CSV dialect and encoding intake

Repair an HTTP cookie jar across scope expiry and replacement rules

Repair distributed trace propagation across malformed and external requests

Repair MIME multipart decoding and attachment selection

Repair a transactional outbox relay after ambiguous delivery outcomes

Repair normalized GraphQL cache rollback across out-of-order mutations

Repair WebAuthn sign-counter and passkey risk reconciliation

Repair GitHub Actions matrix DAG rerun planning

Repair GitHub Actions OIDC workload trust authorization

Repair an HTTP/2 GOAWAY request retry planner

Repair gRPC deadline budget and cancellation propagation

Repair an offline Sigstore bundle release-policy verifier

Repair an HTTP Early-Data replay safety planner

Repair directional OpenAPI schema compatibility analysis

Repair OAuth DPoP binding and replay policy

Repair JWKS cache rotation and key selection

Repair connection-pool reset and fair reuse

Repair an incremental build dependency plan

Repair a safe Git worktree prune plan

Recover an idempotent message inbox

Repair HTTP content negotiation and Vary caching

Recover API pagination after cursor invalidation

Repair retention compaction under legal hold

Repair shell pipeline status and retry handling

Repair Git LFS pointer and object reconciliation

Repair mill DAG scheduling with priority-stable ties

Repair mill cron next-fire on a 30-day clock

Repair JSON Pointer resolution with RFC 6901 escapes

user service registration repair

stateful python rpc service

node record processing contract

weekly operations brief generator

Repository Maintenance

14 tasks

Safely integrate a PR whose final diff hides unsafe history

Prepare a production-safe slug library dependency upgrade

Review a bulk-export state-machine change

Repair a split migration graph and prove replay safety

Reconcile CODEOWNER coverage after a path move

Reconcile a monorepo lock graph under security and runtime constraints

Re-evaluate a protected merge queue after a late commit

Repair a migrated multi-file JSON Schema bundle

Repair a leaking Docker build context

Repair a split Python namespace package

Admit an offline patchset through checks, CODEOWNERS, and rebase

Record flaky tests only after three real failures, ignoring infra errors

Excise a leaked commit while keeping later legitimate work

safe skill archive packaging