Skip to content

RHIDP-17791: feat: add positive feedback categories #2875

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tisnik merged 3 commits into from
Oct 9, 2026
Merged

RHIDP-17791: feat: add positive feedback categories #2875

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
679449d
UIERFE-27: add predefined positive feedback categories
Jdubrick Oct 8, 2026
9fe57cb
UIERFE-27: simplify feedback category documentation
Jdubrick Oct 8, 2026
3e5ecb6
RHIDP-17791: fix CodeRabbit review comment on feedback category item …
Jdubrick Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 37 additions & 3 deletions docs/devel_doc/openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14349,7 +14349,14 @@
"anyOf": [
{
"items": {
"$ref": "#/components/schemas/FeedbackCategory"
"anyOf": [
{
"$ref": "#/components/schemas/FeedbackCategory"
},
{
"$ref": "#/components/schemas/PositiveFeedbackCategory"
}
]
},
"type": "array"
},
Expand All @@ -14358,11 +14365,16 @@
}
],
"title": "Categories",
"description": "List of feedback categories that describe issues with the LLM response (for negative feedback).",
"description": "List of positive or negative feedback categories.",
"examples": [
[
"incorrect",
"incomplete"
],
[
"helpful",
"accurate",
"resolved_issue"
]
]
}
Expand All @@ -14375,7 +14387,7 @@
"llm_response"
],
"title": "FeedbackRequest",
"description": "Model representing a feedback request.\n\nAttributes:\n conversation_id: The required conversation ID (UUID).\n user_question: The required user question.\n llm_response: The required LLM response.\n sentiment: The optional sentiment.\n user_feedback: The optional user feedback.\n categories: The optional list of feedback categories (multi-select for negative feedback).",
"description": "Model representing a feedback request.\n\nAttributes:\n conversation_id: The required conversation ID (UUID).\n user_question: The required user question.\n llm_response: The required LLM response.\n sentiment: The optional sentiment.\n user_feedback: The optional user feedback.\n categories: The optional list of positive or negative feedback categories.",
"examples": [
{
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
Expand All @@ -14393,6 +14405,15 @@
"sentiment": -1,
"user_question": "What is the capital of France?"
},
{
"categories": [
"helpful"
],
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
"llm_response": "The API is easy to use.",
"sentiment": 1,
"user_question": "How do I use the API?"
},
{
"categories": [
"incomplete",
Expand Down Expand Up @@ -18215,6 +18236,19 @@
"title": "PgvectorVectorStoreProviderConfig",
"description": "Storage config for a pgvector dynamic vector-store provider."
},
"PositiveFeedbackCategory": {
"type": "string",
"enum": [
"helpful",
"accurate",
"clear",
"relevant",
"actionable",
"resolved_issue"
],
"title": "PositiveFeedbackCategory",
"description": "Enum representing predefined categories for positive AI responses.\n\nThese categories cover qualities commonly associated with a useful answer."
},
"PostgreSQLDatabaseConfiguration": {
"properties": {
"host": {
Expand Down
12 changes: 7 additions & 5 deletions docs/devel_doc/openapi.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,7 @@ Lightspeed Core Stack (LCS) service API specification.
* [OpenIdConnectSecurityScheme](#openidconnectsecurityscheme)
* [PasswordOAuthFlow](#passwordoauthflow)
* [PostgreSQLDatabaseConfiguration](#postgresqldatabaseconfiguration)
* [PositiveFeedbackCategory](#positivefeedbackcategory)
* [PromptCreateRequest](#promptcreaterequest)
* [PromptDeleteResponse](#promptdeleteresponse)
* [PromptResourceResponse](#promptresourceresponse)
Expand Down Expand Up @@ -6403,9 +6404,6 @@ These categories help provide structured feedback about AI inference quality
when users provide negative feedback (thumbs down). Multiple categories can
be selected to provide comprehensive feedback about response issues.




## FeedbackRequest


Expand All @@ -6417,7 +6415,7 @@ Attributes:
llm_response: The required LLM response.
sentiment: The optional sentiment.
user_feedback: The optional user feedback.
categories: The optional list of feedback categories (multi-select for negative feedback).
categories: The optional list of positive or negative feedback categories.


| Field | Type | Description |
Expand All @@ -6427,7 +6425,7 @@ Attributes:
| llm_response | string | Response from LLM |
| sentiment | | User sentiment, if provided must be -1 or 1 |
| user_feedback | | Feedback on the LLM response. |
| categories | | List of feedback categories that describe issues with the LLM response (for negative feedback). |
| categories | array of FeedbackCategory or PositiveFeedbackCategory | List of positive or negative feedback categories. |


## FeedbackResponse
Expand Down Expand Up @@ -7797,6 +7795,10 @@ Useful resources:
| ca_cert_path | | Path to CA certificate |


## PositiveFeedbackCategory

Enum representing predefined categories for positive feedback.

## PromptCreateRequest


Expand Down
41 changes: 39 additions & 2 deletions docs/models/requests.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
},
"FeedbackRequest": {
"additionalProperties": false,
"description": "Model representing a feedback request.\n\nAttributes:\n conversation_id: The required conversation ID (UUID).\n user_question: The required user question.\n llm_response: The required LLM response.\n sentiment: The optional sentiment.\n user_feedback: The optional user feedback.\n categories: The optional list of feedback categories (multi-select for negative feedback).",
"description": "Model representing a feedback request.\n\nAttributes:\n conversation_id: The required conversation ID (UUID).\n user_question: The required user question.\n llm_response: The required LLM response.\n sentiment: The optional sentiment.\n user_feedback: The optional user feedback.\n categories: The optional list of positive or negative feedback categories.",
"examples": [
{
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
Expand All @@ -155,6 +155,15 @@
"sentiment": -1,
"user_question": "What is the capital of France?"
},
{
"categories": [
"helpful"
],
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
"llm_response": "The API is easy to use.",
"sentiment": 1,
"user_question": "How do I use the API?"
},
{
"categories": [
"incomplete",
Expand Down Expand Up @@ -215,13 +224,28 @@
},
"categories": {
"type": "array",
"items": {
"anyOf": [
{
"$ref": "`#/components/schemas/`FeedbackCategory"
},
{
"$ref": "`#/components/schemas/`PositiveFeedbackCategory"
}
]
},
"nullable": true,
"default": null,
"description": "List of feedback categories that describe issues with the LLM response (for negative feedback).",
"description": "List of positive or negative feedback categories.",
Comment thread
coderabbitai[bot] marked this conversation as resolved.
"examples": [
[
"incorrect",
"incomplete"
],
[
"helpful",
"accurate",
"resolved_issue"
]
],
"title": "Categories"
Expand Down Expand Up @@ -1850,6 +1874,19 @@
"title": "OpenAITopLogProb",
"type": "object"
},
"PositiveFeedbackCategory": {
"description": "Enum representing predefined categories for positive AI responses.\n\nThese categories cover qualities commonly associated with a useful answer.",
"enum": [
"helpful",
"accurate",
"clear",
"relevant",
"actionable",
"resolved_issue"
],
"title": "PositiveFeedbackCategory",
"type": "string"
},
"PromptCreateRequest": {
"additionalProperties": false,
"description": "Request body to create a stored prompt template in OGX.\n\nAttributes:\n prompt: Prompt text with variable placeholders.\n variables: Variable names allowed in the template.",
Expand Down
14 changes: 12 additions & 2 deletions docs/models/requests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Attributes:
llm_response: The required LLM response.
sentiment: The optional sentiment.
user_feedback: The optional user feedback.
categories: The optional list of feedback categories (multi-select for negative feedback).
categories: The optional list of positive or negative feedback categories.


| Field | Type | Description |
Expand All @@ -112,7 +112,7 @@ Attributes:
| llm_response | string | Response from LLM |
| sentiment | integer | User sentiment, if provided must be -1 or 1 |
| user_feedback | string | Feedback on the LLM response. |
| categories | array | List of feedback categories that describe issues with the LLM response (for negative feedback). |
| categories | array | List of positive or negative feedback categories. |


## FeedbackStatusUpdateRequest
Expand Down Expand Up @@ -839,6 +839,16 @@ The top log probability for a token from an OpenAI-compatible chat completion re
| logprob | number | The log probability of the token. |


## PositiveFeedbackCategory


Enum representing predefined categories for positive AI responses.

These categories cover qualities commonly associated with a useful answer.




## PromptCreateRequest


Expand Down
30 changes: 19 additions & 11 deletions src/models/api/requests/feedback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

from pydantic import BaseModel, Field, field_validator, model_validator

from models.common import FeedbackCategory
from models.common import FeedbackCategory, PositiveFeedbackCategory
from utils import suid


Expand All @@ -17,7 +17,7 @@ class FeedbackRequest(BaseModel):
llm_response: The required LLM response.
sentiment: The optional sentiment.
user_feedback: The optional user feedback.
categories: The optional list of feedback categories (multi-select for negative feedback).
categories: The optional list of positive or negative feedback categories.
"""

conversation_id: str = Field(
Expand Down Expand Up @@ -51,14 +51,14 @@ class FeedbackRequest(BaseModel):
examples=["I'm not satisfied with the response because it is too vague."],
)

# Optional list of predefined feedback categories for negative feedback
categories: Optional[list[FeedbackCategory]] = Field(
# Optional list of predefined feedback categories for positive and negative feedback
categories: Optional[list[FeedbackCategory | PositiveFeedbackCategory]] = Field(
default=None,
description=(
"List of feedback categories that describe issues with the LLM response "
"(for negative feedback)."
),
examples=[["incorrect", "incomplete"]],
description="List of positive or negative feedback categories.",
examples=[
["incorrect", "incomplete"],
["helpful", "accurate", "resolved_issue"],
],
)

# provides examples for /docs endpoint
Expand All @@ -80,6 +80,13 @@ class FeedbackRequest(BaseModel):
"sentiment": -1,
"categories": ["incorrect"],
},
{
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
"user_question": "How do I use the API?",
"llm_response": "The API is easy to use.",
"sentiment": 1,
"categories": ["helpful"],
},
{
"conversation_id": "12345678-abcd-0000-0123-456789abcdef",
"user_question": "How do I deploy a web app?",
Expand Down Expand Up @@ -135,8 +142,9 @@ def check_sentiment(cls, value: Optional[int]) -> Optional[int]:
@field_validator("categories")
@classmethod
def validate_categories(
cls, value: Optional[list[FeedbackCategory]]
) -> Optional[list[FeedbackCategory]]:
cls,
value: Optional[list[FeedbackCategory | PositiveFeedbackCategory]],
) -> Optional[list[FeedbackCategory | PositiveFeedbackCategory]]:
"""Normalize and deduplicate a feedback categories list.

Converts an empty list to None for consistency and removes duplicate
Expand Down
3 changes: 2 additions & 1 deletion src/models/common/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
ConversationTurn,
Message,
)
from models.common.feedback import FeedbackCategory
from models.common.feedback import FeedbackCategory, PositiveFeedbackCategory
from models.common.health import (
HealthStatus,
ProviderHealthStatus,
Expand Down Expand Up @@ -56,6 +56,7 @@
"MCPServerAuthInfo",
"MCPServerInfo",
"Message",
"PositiveFeedbackCategory",
"ProviderHealthStatus",
"RAGChunk",
"RAGContext",
Expand Down
14 changes: 14 additions & 0 deletions src/models/common/feedback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,17 @@ class FeedbackCategory(StrEnum):
OUTDATED_INFORMATION = "outdated_information" # "This information is from several years ago and no longer accurate" # pylint: disable=line-too-long
UNSAFE = "unsafe" # "This response could be harmful or dangerous if followed"
OTHER = "other" # "The response has issues not covered by other categories"


class PositiveFeedbackCategory(StrEnum):
"""Enum representing predefined categories for positive AI responses.

These categories cover qualities commonly associated with a useful answer.
"""

HELPFUL = "helpful" # "The response is useful"
ACCURATE = "accurate" # "The information provided is correct"
CLEAR = "clear" # "The response is easy to understand"
RELEVANT = "relevant" # "This answer addresses my question"
ACTIONABLE = "actionable" # "The response provides steps I can follow"
RESOLVED_ISSUE = "resolved_issue" # "This response solved my problem"
3 changes: 2 additions & 1 deletion tests/unit/app/endpoints/test_feedback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,13 +138,14 @@ async def test_assert_feedback_enabled_disabled_full_config_chain(
"feedback_request_data",
[
{**VALID_BASE, "sentiment": 1},
{**VALID_BASE, "sentiment": 1, "categories": ["helpful"]},
{
**VALID_BASE,
"sentiment": -1,
"categories": ["incorrect", "incomplete"],
},
],
ids=["no_categories", "with_negative_categories"],
ids=["no_categories", "with_positive_category", "with_negative_categories"],
)
@pytest.mark.asyncio
async def test_feedback_endpoint_handler(
Expand Down
Loading
Loading