302-Initial changes for adding e2e tests for ai-services testing framework #257

Abhishek-Kushwaha1 · 2026-01-08T09:47:30Z

This PR expands the E2E test coverage for the ai-services CLI, focusing on the application lifecycle s flow and document ingestion using the ingest-docs pod.
The tests are written using Ginkgo/Gomega and follow Kubernetes-style E2E patterns.
Any test cases that are not yet implemented or are unstable are explicitly skipped using XContext and are not silently ignore.

Commands covered:
ai-services -help
ai-services bootstrap
ai-services bootstrap configure
ai-services bootstrap validate
ai-services application create
ai-services application ps
ai-services application stop
ai-services application delete

- Skips downloading of images during application creation if the image with given tag already exist locally. - Introduces a new `--image-pull-policy` flag which takes 3 values `Always`, `IfNotPresent` and `Never`. - If the flag is not set, then it uses `IfNotPresent` by default. - Deprecates `--skip-image-download` flag and moving `--image-pull-policy` flag can be used Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

…ework Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Signed-off-by: Abhishek Kushwaha <Abhishek.Kushwaha@ibm.com>

Shubhamag12

Added few comments
will review this PR in multiple iterations

Shubhamag12 · 2026-01-16T06:26:26Z

ai-services/tests/e2e/bootstrap/build.go

+			return bin, nil
+		}
+
+		return "", fmt.Errorf("AI_SERVICES_BIN=%s failed verification: %w", bin, err)


As per my understanding, if ai-serivces binary is not found in first step, we proceed for next step (to either lookup or build one).

If that is the case, should we return from here? or should we just log that :AI_SERVICES_BIN not set, will check temp bin, and proceed for next step

Please correct me if I am wrong

@Shubhamag12
If AI_SERVICES_BIN is set, we assume the user wants to use that binary and fail if it’s invalid.
We only look for or build another binary when AI_SERVICES_BIN is not set.
Considering your suggestion, we can add that log message and proceed further.

Shubhamag12 · 2026-01-16T06:31:08Z

ai-services/tests/e2e/bootstrap/build.go

+
+	fmt.Printf("[BOOTSTRAP] Successfully built and verified binary: %s\n", binPath)
+
+	return binPath, nil


small recommendation: If the build partially succeeds but verification fails, the invalid binary remains in testBinDir. Maybe a cleanup logic will do here.

Can be taken up in subsequent PRs

Shubhamag12 · 2026-01-16T06:46:22Z

ai-services/tests/e2e/cli/output.go

+			continue
+		}
+
+		parts := strings.Fields(line)


we should enable bound checking here, as the code will panic if in any case the output has less than 2 fields

Shubhamag12 · 2026-01-16T06:55:05Z

ai-services/tests/e2e/cli/runner.go

+// extractHostIP extracts the host IP from the CLI output using regex.
+func extractHostIP(output string) (string, error) {
+	const minMatchGroups = 2
+	re := regexp.MustCompile(`http[s]?://([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)`)


nit

This regex signifies: 999.999.999.999

we can use somewhat like this https?://(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}), and validate this ip using net.ParseIP

Shubhamag12 · 2026-01-16T06:56:28Z

ai-services/tests/e2e/cli/runner.go

+	backendPort string,
+	uiPort string,
+	opts CreateOptions,
+	_ []string,


can we remove this if not used?

Will be removing it in coming commits.

Shubhamag12 · 2026-01-19T06:46:18Z

ai-services/tests/e2e/podman/containers.go

+// VerifyContainers checks if application pods are healthy and their restart counts are zero.
+func VerifyContainers(appName string) error {
+	fmt.Println("[Podman] verifying containers for app:", appName)
+	res, _ := common.RunCommand("ai-services", "application", "ps", appName)


we should check for error as well

ok. will fix this

Shubhamag12 · 2026-01-19T07:18:07Z

ai-services/tests/e2e/podman/containers.go

+	for _, ctr := range pod.Containers {
+		ctrRes, err := common.RunCommand("podman", "inspect", ctr.Id)
+		if err != nil {
+			return 0, fmt.Errorf("failed to inspect container %s: %w", ctr.Name, err)
+		}
+		var ctrData []ContainerInspect
+		if err := json.Unmarshal([]byte(ctrRes), &ctrData); err != nil {
+			return 0, fmt.Errorf("failed to parse container inspect %s: %w", ctr.Name, err)
+		}
+		if len(ctrData) == 0 {
+			return 0, fmt.Errorf("no container inspect data for %s", ctr.Name)
+		}
+		totalRestarts += ctrData[0].State.RestartCount
+	}
+
+	return totalRestarts, nil
+}


we are making too many calls here.
1 for pod and N for pod containers (n being number of containers)

Suggested approach

Collect all container IDs

Inspect all containers in one call

Parse all container data at once

And counts restarts

Sample

Suggested change

for _, ctr := range pod.Containers {

ctrRes, err := common.RunCommand("podman", "inspect", ctr.Id)

if err != nil {

return 0, fmt.Errorf("failed to inspect container %s: %w", ctr.Name, err)

}

var ctrData []ContainerInspect

if err := json.Unmarshal([]byte(ctrRes), &ctrData); err != nil {

return 0, fmt.Errorf("failed to parse container inspect %s: %w", ctr.Name, err)

}

if len(ctrData) == 0 {

return 0, fmt.Errorf("no container inspect data for %s", ctr.Name)

}

totalRestarts += ctrData[0].State.RestartCount

}

return totalRestarts, nil

}

ctrIDs := make([]string, 0, len(pod.Containers))

for _, ctr := range pod.Containers {

ctrIDs = append(ctrIDs, ctr.Id)

}

args := append([]string{"inspect"}, ctrIDs...)

ctrRes, err := common.RunCommand("podman", args...)

if err != nil {

return 0, fmt.Errorf("failed to inspect containers in pod %s: %w", podName, err)

}

var allContainers []ContainerInspect

if err := json.Unmarshal([]byte(ctrRes), &allContainers); err != nil {

return 0, fmt.Errorf("failed to parse container inspect: %w", err)

}

totalRestarts := 0

for _, ctr := range allContainers {

totalRestarts += ctr.State.RestartCount

}

We are making 2 calls in total here

Let me know if I understood it correctly and is the suggestion apt.

Noted. will fix in next commit

Shubhamag12 · 2026-01-19T07:22:12Z

ai-services/tests/e2e/podman/containers.go

+	AppName      string
+	PodName      string
+	Status       string
+	RestartCount int


I see RestartCount not being used anywhere. We are feeding it into the structure here, but never reference it in later code. Should we remove it?

Ok. i will remove it

mayuka-c

Here are my initial comments. PTAL.
I will be reviewing in multiple rounds as it is quite big.

mayuka-c · 2026-01-09T06:40:34Z

ai-services/tests/e2e/rag/validate.go

@@ -0,0 +1,8 @@
+package rag
+
+import "log"


I see logs being used and in few places fmt. Can we have a single pkg being used?

mayuka-c · 2026-01-09T06:45:01Z

ai-services/tests/e2e/e2e_suite_test.go

+		})
+	})
+
+})


Add a new line

mayuka-c · 2026-01-09T06:49:21Z

ai-services/tests/e2e/httpclient/client_test.go

+		testServer = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			switch r.Method {
+			case "GET":
+				w.Write([]byte(`{"status":"ok"}`))


Please take care of all the linters

Yes, we have checked and resolved linter errors.

mayuka-c · 2026-01-19T08:26:34Z

ai-services/tests/e2e/e2e_suite_test.go

+				"5000", // backend port
+				"3000", //ui port
+				cli.CreateOptions{
+					SkipImageDownload: false,


This is been deprecated. So prefer to use --image-pull-policy here. Pls refer this PR: #230

mayuka-c · 2026-01-19T08:27:25Z

ai-services/tests/e2e/e2e_suite_test.go

+				cfg,
+				appName,
+				"rag",
+				"ui.port=3000,backend.port=5000",


By default ui port and backend port runs at 3000 and 5000 respectively. So I would prefer to specify a diff ports here and make sure its running,

Noted, will be testing it with different ports,

mayuka-c · 2026-01-19T08:29:28Z

ai-services/tests/e2e/podman/containers.go

+	expectedPodSuffixes := []string{
+		"vllm-server",
+		"milvus",
+		"clean-docs",
+		"ingest-docs",
+		"chat-bot",
+	}


This could probably be present outside the method

okay , What do you suggest as appropriate place ?

Probably can we have vars dir under e2e and keep it there. There are chances that this slice will be used even for others in future like upgrade and rollbacks

mayuka-c · 2026-01-19T08:30:45Z

ai-services/tests/e2e/httpclient/client_test.go

+	It("GET works", func() {
+		client := NewHTTPClient()
+		resp, err := client.Get("/test")
+		Expect(err).NotTo(HaveOccurred())
+		defer func() {
+			if cerr := resp.Body.Close(); cerr != nil {
+				fmt.Printf("[WARNING] failed to close response body: %v\n", cerr)
+			}
+		}()
+		body, err := io.ReadAll(resp.Body)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(string(body)).To(ContainSubstring("ok"))
+	})


May I know what are these API calls for?

Actually, while create the base structure of the e2e I created it. I will remove it in coming PR.

ai-services/tests/e2e/cli/runner.go

mayuka-c · 2026-01-20T02:49:10Z

ai-services/tests/e2e/podman/containers.go

+	ginkgo "github.com/onsi/ginkgo/v2"
+	gomega "github.com/onsi/gomega"


In few places I see implicit imports done using . and in few places I see named imports. Can we have convention followed everywhere?

Okay. For now I will make the changes by following the existing convention of implicit imports. If necessary, we can shift to named.

mayuka-c · 2026-01-20T02:50:27Z

ai-services/tests/e2e/httpclient/client.go

@@ -0,0 +1,64 @@
+package httpclient


May I know why we have httpClient package?.
I see the BaseURL says "AI_SERVICES_BASE_URL" but we are not running any API server

Actually, while create the base structure of the e2e I created it. I will remove it in coming PR.

ai-services/tests/e2e/common/logger.go

mayuka-c · 2026-01-20T02:53:31Z

ai-services/tests/e2e/common/json.go

+
+import (
+	"encoding/json"
+	"log"


Pls replace all the standard log pkg with our own internal logger pkg defined here: https://github.com/IBM/project-ai-services/blob/main/ai-services/internal/pkg/logger/logger.go

Do replace it everywhere, so that only 1 logger pkg

mayuka-c · 2026-01-20T03:01:10Z

ai-services/tests/e2e/cli/runner.go

+}
+
+// CreateAppWithHealthAndRAG creates an application, waits for health checks, and validates RAG endpoints.
+func CreateAppWithHealthAndRAG(


I feel this func is tied to RAG.
I feel this might not be the right way as CLI is agnostic to application. So better to have a seperate dir application for each app with CreateApp present under it.

Similarly wherever it is application specific can exists under it

I have renamed the function and added comment to make it explicit that this helper is RAG-specific and used only by RAG E2E tests.

mkumatag requested review from Shubhamag12 and mayuka-c January 9, 2026 04:43

Abhishek-Kushwaha1 force-pushed the e2e branch 4 times, most recently from 19b7750 to b000575 Compare January 13, 2026 07:27

mayuka-c and others added 21 commits January 13, 2026 08:23

Http client tests

f38b072

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding Put and delete method

34c0f3e

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding config and bootstrap base utilities for e2e test

584fdb1

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

bese structure for e2e test framework

219a722

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding bootstrap cli runner and validation

d0229b1

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding configure and validation commands tests

c13c2cc

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding fixes for validate and configure command

2f54a18

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding create command test code for e2e tests

dad5859

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Modifying create command validation output

3b19554

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Modifying health check code

2f0e6e0

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Modifying create command logic for e2e

b56d949

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Modifying health check code

ac834b4

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Add final code for create command and validating rag endpoints

f29c17a

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Add final code for create command and validating rag endpoints

4016b22

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Add tests for help command

623e6a7

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Added tests for app ps,stop,delete

b31f175

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding logic for pods verification , health check and restart count

8555b0d

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

302-Initial changes for adding e2e tests for ai-services testing fram…

061062a

…ework Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Fixing linter issues in the go code of e2e testing

7debbde

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Adding comments

e606e6b

Signed-off-by: Abhishek-Kushwaha <Abhishek.Kushwaha@ibm.com>

Abhishek-Kushwaha1 force-pushed the e2e branch from 92919de to e606e6b Compare January 13, 2026 08:24

Abhishek-Kushwaha1 added 2 commits January 13, 2026 13:54

Merge branch 'main' into e2e

9a4a450

Merge branch 'main' into e2e

a466f21

Merge branch 'main' into e2e

9e9997c

Signed-off-by: Abhishek Kushwaha <Abhishek.Kushwaha@ibm.com>

Shubhamag12 reviewed Jan 16, 2026

View reviewed changes

Shubhamag12 reviewed Jan 19, 2026

View reviewed changes

mayuka-c reviewed Jan 19, 2026

View reviewed changes

mayuka-c reviewed Jan 20, 2026

View reviewed changes


		fmt.Printf("[BOOTSTRAP] Successfully built and verified binary: %s\n", binPath)

		return binPath, nil

+              		})
+              	})
+              })

                
                    No newline at end of file

		ginkgo "github.com/onsi/ginkgo/v2"
		gomega "github.com/onsi/gomega"

302-Initial changes for adding e2e tests for ai-services testing framework #257

Are you sure you want to change the base?

302-Initial changes for adding e2e tests for ai-services testing framework #257

Uh oh!

Conversation

Abhishek-Kushwaha1 commented Jan 8, 2026 • edited by prachikurade1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shubhamag12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shubhamag12 Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayuka-c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AS-Prajna Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mayuka-c Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Abhishek-Kushwaha1 commented Jan 8, 2026 •

edited by prachikurade1

Loading

Shubhamag12 Jan 19, 2026 •

edited

Loading

AS-Prajna Jan 19, 2026 •

edited

Loading

mayuka-c Jan 20, 2026 •

edited

Loading

Abhishek-Kushwaha1 Jan 28, 2026 •

edited

Loading