Give frontier agents more than questions to answer. Give them work to do. Deaimer builds custom environments where models use tools, make decisions and learn from outcomes you can verify.
Custom task design · Independent validation · Versioned delivery
Most AI agent training runs waste GPU time on rollouts that teach the model nothing. This article covers three ways to get more learning per GPU-hour, plus a simple metric to track it.
Can AI-generated worlds teach robots when their physics is wrong? This research examines failure fidelity, action-conditioned world models, and sim-to-real transfer—and proposes a reproducible framework for evaluating RL environments.
Harder training tasks do not automatically produce transferable AI skills. Learn how RL curricula can overfit to generators, and explore a proposed framework for testing surface, structural, and mechanism-level generalization
An AI agent can master a convincing simulation and still learn the wrong behavior. Explore causal fidelity in RL environments and a proposed audit of state transitions, observations, resets, and delayed consequences
A passing reward does not always mean an AI agent completed the intended task. Explore how reward hacking, weak verification, and misleading feedback affect AI agent training—and why robust RL environments and verifiable rewards matter for reliable AI systems.
The path toward more general AI depends on the experiences agents learn from. Explore how diverse tasks, custom environments, and domain expertise support adaptation and transfer - and how Deaimer contributes through datasets and executable workflows.
Explore how Termetheus turns custom terminal tasks into executable learning environments for AI agents, supporting post-training, domain-specific skills, and evaluation on unseen problems.
Start with the capability you need to improve. We design the task, workspace, available tools and evaluation around the work your agent should be able to complete.
Building, debugging, and maintaining complex software.
A versioned repository with a defined starting commit, runnable dependencies, development tools and a clean reset procedure. Tasks can require changes across modules while respecting existing APIs and project conventions.
Run task-specific tests and regression checks against the final patch. Reference solutions establish feasibility, while negative controls help catch graders that accept incomplete fixes or superficial changes.
Vary repository size, the amount of context provided, tooling constraints and the number of dependent changes. Inspect patches and execution traces to understand how the agent reached its result.
Completing tasks across desktop applications, browsers, and tools.
A controlled browser or desktop workspace with specified applications, accounts, permissions and initial state. Define the interaction interface, available tools and any actions the agent must avoid before the task starts.
Check final application state, saved files and required field values. Use action logs and explicit review rubrics for steps that cannot be judged from the final screen alone, including permissions and prohibited actions.
Introduce longer workflows, partial information, recoverable errors or changes in layout. Keep the desired outcome explicit so the evaluation measures task completion rather than a single prescribed click sequence.
Synthesizing information, analyzing data, and producing deliverables.
A bounded set of documents, data tables and reference material, paired with the tools needed to search, analyze and produce deliverables. Specify the audience, output format and source requirements in the task brief.
Evaluate factual grounding, numerical correctness, source attribution and coverage of required points. Combine executable checks on structured outputs with documented rubrics for reasoning and written deliverables.
Vary source volume, conflicting evidence, data quality and the number of deliverables. Include unsupported claims, missing evidence and incorrect calculations in failure analysis rather than relying on writing style alone.
Following instructions, respecting constraints, and exercising sound judgment.
A task with explicit instruction priorities, permitted tools, action boundaries and success criteria. Scenarios can include untrusted content, conflicting requests or decisions that require clarification or approval.
Check both whether the legitimate objective was achieved and whether the agent respected the constraints. Use action traces, state checks and behavior-specific rubrics to distinguish safe completion, justified refusal and unnecessary refusal.
Vary ambiguity, attack placement, tool permissions and the cost of a mistaken action. Record the tested threat assumptions and limitations; success on a scoped scenario is evidence about that scenario, not a general safety guarantee.
Understanding, synthesizing, and extending scientific research.
A documented research question with relevant literature, datasets, computational tools and fixed execution requirements. Define expected artifacts, numerical tolerances and available resources before running the experiment.
Check numerical results, analysis code, generated artifacts and whether conclusions follow from the evidence. Use repeat runs, reference calculations and expert rubrics where scientific interpretation cannot be reduced to an exact match.
Vary experimental complexity, incomplete methods, noisy data and resource limits. Inspect whether the agent reports uncertainty, preserves reproducibility and distinguishes observed results from unsupported extrapolation.
The task is only one part of the environment. The surrounding setup, feedback and evidence determine whether a result is useful for training or evaluation.
task specificationGoal, tools & constraintsexecution environmentSetup, dependencies & resetreference solutionA demonstrated success pathverifier & rubricOutcome checks & feedbackvalidation evidenceControls, trials & findingsversion & run guideReproduce & integrateIllustrative package structure. Delivery formats and required artifacts are agreed with your team during scoping.
Containerized setups where appropriate, pinned dependencies, input assets and a reset procedure establish a repeatable starting point. Tool access and execution limits match the intended task.
Task instructions and acceptance criteria connect to automated checks or structured rubrics. Reward signals are designed around observable outcomes, with grading assumptions documented.
Reference solutions and example trajectories establish feasibility. Validation records show what was checked, which failure modes were tested and what limitations remain.
Agree on schemas, execution interfaces and reporting requirements before production. Versioned assets and run instructions support integration into your own training or evaluation pipeline.
A solvable task can still have a weak grader. A correct grader can still sit inside an unstable environment. We review the complete system, with distinct checks for each failure mode.
A working reference solution establishes that the task can be completed within the intended tools, permissions and resource limits.
Known-failing runs, partial solutions and negative controls test whether the verifier distinguishes actual success from superficial completion.
A separate reviewer probes ambiguous instructions, exposed answers, shortcut solutions and opportunities to exploit the reward.
Repeated trials and clean resets help identify flaky tests, unstable dependencies and evaluation noise before delivery.
Task authors and reviewers work separately. Validation evidence makes acceptance a reviewable decision, with the tested assumptions and remaining limitations recorded alongside the task.
A demo task from our private collection: the task as an agent sees it, plus the runs, checks and reviews we deliver with it.
Demo task for illustration. Reference solutions, verifiers and the full collection are shared under sample access or licensing.
Use custom environments to develop a capability, measure it privately or understand where it breaks. The objective shapes task coverage, grading and delivery.
Task families with verifiable outcomes and feedback for reinforcement learning workflows. Scope variation, difficulty and reward design around the behaviors you want agents to learn.
Held-out task suites for internal comparisons. Use consistent starting states, grading rules and repeated trials to examine success, reliability and failure patterns.
Adversarial conditions, untrusted inputs and constrained actions reveal how agents behave when workflows become difficult. Review trajectories as well as final outcomes.
Through third-party engagements, Deaimer's teams have produced and reviewed tasks associated with Terminal-Bench, Terminal-Bench Science and SWE-bench-style evaluations, covering environment construction, verifiers, reference solutions and model-run analysis.
Deaimer is an independent contributor to select third-party benchmark programs. Deaimer does not own or operate Terminal-Bench or SWE-bench, and this work does not imply an official partnership or endorsement.
A controlled pilot gives your team something concrete to inspect before committing to a larger task suite. Acceptance criteria come first; scale follows the evidence.
Agree on the target agent, task domains, available tools, difficulty, security boundaries and acceptance criteria.
Construct a small batch of tasks with environment setup, reference solutions and initial verifiers for your review.
Separate author and reviewer workflows test solvability, grading, shortcuts and model behavior across repeated trials.
Receive versioned task assets and validation evidence. Use pilot findings to refine coverage before expanding the suite.
Bring a capability goal or an existing task suite. We can help turn it into a concrete environment brief.
Yes. We scope the execution interface, tools, container requirements, task schema and grading outputs with your team before construction. Integration requirements and any constraints are agreed during the pilot.
Executable checks are appropriate when success can be verified directly. Knowledge work, computer use and alignment scenarios may also require structured rubrics or human review. We specify the grading method and its limitations for each task type.
Private evaluation suites and controlled access can be scoped for your engagement. Data handling, permitted access, delivery arrangements, licensing and any exclusivity requirements should be agreed before work begins.
Yes. An independent audit can review environment setup, reference solutions, verifier correctness, negative controls, reproducibility and potential reward exploits. Findings can guide targeted repairs or a broader task-suite revision.
Share the capability you want to train or measure, the tools your agent can use, a few representative tasks and how you define success. If you already have a harness, security requirements or a task schema, include those in the brief.
Tell us what your agent needs to do. We'll work with you to define a focused environment pilot, the success criteria and the evidence your team needs to evaluate it.