Skip to content

refactor(checks)!: new 'Interact' API#2270

Merged
mattbit merged 16 commits intomainfrom
refactor/interaction-api
Feb 27, 2026
Merged

refactor(checks)!: new 'Interact' API#2270
mattbit merged 16 commits intomainfrom
refactor/interaction-api

Conversation

@mattbit
Copy link
Member

@mattbit mattbit commented Feb 26, 2026

Major renaming and API change:

  • InteractionSpec is now simply InteractionInteract. This is the main dynamic interface. It extends InteractionSpec which is its abstract definition (with generate method).
  • The old Interaction becomes InteractionRecord to signal that it is an immutable realization of the Interaction Interaction remains a static representation of a single exchange, but it is now only used internally and not exposed. E.g. Scenario only accepts InteractionSpec/Interact objects and no Interaction.

Scenarios and testcases now only see InteractionSpec, i.e. the dynamic version. However, the Check logic receives a static Trace composed of Interactions which are guaranteed to be complete (no dynamic values, no mutable fields).

I also regrouped the interactions and trace objects under the checks.core.interaction submodule. Before, definitions were split across checks.core.interaction, checks.interaction, and checks.core.trace.

This replaces #2269 by maintaining the two distinct classes (dynamic vs frozen).

gemini-code-assist[bot]

This comment was marked as outdated.

Copy link
Member

@henchaves henchaves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we are still giving much attention to the usage of InteractionRecord as a Scenario component. We need to focus on using only Interaction wherever it's possible.

The manual instantiation of InteractionRecord should be limited for when creating a Trace:

trace = Trace(interactions=[InteractionRecord(inputs="test", outputs="result")])

Having scenarios with mixed InteractionRecord and Interaction is not nice, and it's kind of the same thing of having InteractionSpec/Interaction - so what is the goal of this PR?

# ❌ BAD
scenario = Scenario(sequence=[
   InteractionRecord(inputs="test", outputs="result"),
   Interaction(inputs="hello", outputs="word")
])

If you think it's appropriate, I would make only Interaction as a Scenario component (and Check, of course), and if needed, users could map InteractionRecord to Interaction using an API such as Scenario.from_records()

Overall, the implementation is well done. Did many tests with different scenarios and all of them worked well.

mattbit and others added 2 commits February 26, 2026 16:53
Co-authored-by: Henrique Chaves <44180294+henchaves@users.noreply.github.com>
Co-authored-by: Henrique Chaves <44180294+henchaves@users.noreply.github.com>
@mattbit mattbit changed the title refactor(checks)!: unify 'Interaction' API and rename frozen to 'InteractionRecord' refactor(checks)!: new 'Interact' API Feb 26, 2026
@mattbit
Copy link
Member Author

mattbit commented Feb 26, 2026

@henchaves should have addressed all comments, except for the union which didn't feel necessary at this stage.

@mattbit mattbit requested a review from henchaves February 26, 2026 19:22
@Giskard-AI Giskard-AI deleted a comment from gemini-code-assist bot Feb 26, 2026
@mattbit
Copy link
Member Author

mattbit commented Feb 26, 2026

/gemini review

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a significant and valuable refactoring of the interaction APIs, clarifying the distinction between dynamic specifications (Interact) and static records (Interaction). The code changes are well-executed and the new structure is more intuitive. However, the documentation (README, CODEMAP) and some docstrings have not been fully updated to reflect these changes, leading to inconsistencies and incorrect examples. I've pointed out several places where class names like Interact, InteractionSpec, and Interaction are used incorrectly. Addressing these documentation issues will be crucial for users to understand and adopt the new API.

@henchaves
Copy link
Member

/gemini review

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a major and beneficial refactoring of the interaction API, clarifying the distinction between dynamic interaction specifications (Interact) and static interaction records (Interaction). The renames from InteractionSpec to Interact and BaseInteractionSpec to InteractionSpec improve clarity. The code reorganization into the checks.core.interaction submodule is logical. The changes are extensive but appear to be consistently applied across the codebase, including documentation and tests. I've found a couple of minor issues in the documentation that should be addressed.

henchaves and others added 2 commits February 27, 2026 11:58
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Copy link
Member

@henchaves henchaves left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mattbit mattbit merged commit 7bb64bf into main Feb 27, 2026
17 checks passed
@mattbit mattbit deleted the refactor/interaction-api branch February 27, 2026 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

2 participants