Skip to content

Conversation

@sats-23
Copy link
Contributor

@sats-23 sats-23 commented Jan 27, 2026

-Implements critical inference metrics directly within the IGW framework, removing the dependency on the SLO predictor plugin for observability.
-Framework now natively tracks

Time to First Token (TTFT)
Time to Predict Output Token (TPOT)
Inter-Token Latency (ITL) ( request_itl_seconds )
Decode Duration ( request_decode_duration_seconds ): Total Duration - TTFT
End-to-End (E2E) latency ( request_e2e_latency_seconds )

-Added tests to validate metrics tracking

What type of PR is this?
/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #2163

Does this PR introduce a user-facing change?:

Metrics (TTFT, TPOT, ITL, Decode Duration, E2E latency) are directly exposed by IGW framework removing the dependency on SLO predictor plugin for basic metrics

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jan 27, 2026
@netlify
Copy link

netlify bot commented Jan 27, 2026

Deploy Preview for gateway-api-inference-extension ready!

Name Link
🔨 Latest commit c75d584
🔍 Latest deploy log https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6978a42705c77e000876074d
😎 Deploy Preview https://deploy-preview-2220--gateway-api-inference-extension.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@k8s-ci-robot k8s-ci-robot requested review from ahg-g and kfswain January 27, 2026 11:39
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sats-23
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jan 27, 2026
@k8s-ci-robot
Copy link
Contributor

Hi @sats-23. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

-Implements critical inference metrics directly within the IGW framework, removing the dependency on the SLO predictor plugin for observability.
-Framework now natively tracks Time to First Token (TTFT), Time to Predict Output Token (TPOT), Inter-Token Latency (ITL), Decode Duration and End-to-End (E2E) latency for all inference requests.
-Added tests to validate metrics tracking

Signed-off-by: Sathvik <Sathvik.S@ibm.com>
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

report TTFT, TPOT and other important metrics as part of IGW framework

2 participants