Conversation
PR #790 moved build:x64 onto AlmaLinux 8, which took stresstest:x64 with it since it ran in the same image — runtime coverage at the shipped binary's glibc 2.17 floor went to zero. This follows up on the two items proposed there: - .stresstest_job now takes RUNTIME_IMAGE independently of BUILD_IMAGE, and stresstest:x64 points it at a new Oracle Linux 7 image so it goes back to running against glibc 2.17 while build:x64 stays on EL8. - adds a nightly functional:x64-el7-jdk{8,17,21} job matrix that runs the ddprof-test debug suite against the shipped x64 artifact on the same EL7 image, covering the libc-sensitive suites without multiplying the per-PR matrix. The new .gitlab/base/el7/Dockerfile is runtime-only (no compiler): both jobs consume the AlmaLinux-8-built .so via -Pskip-native -Pwith-libs. functional-tests.sh fetches the JDK under test from Adoptium into JAVA_TEST_HOME so Gradle itself keeps running on JDK 21 regardless of which JDK is under test. BUILD_IMAGE_X64_EL7 is a placeholder pin — the image still needs to be built and pushed via a real pipeline (REBUILD_IMAGES="x64-el7") before this can merge, same as #790's own image rebuild. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: daedd84e8c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # TODO: repin after running `REBUILD_IMAGES="x64-el7"` through rebuild-images.sh | ||
| # in a real pipeline (see .gitlab/base/el7/Dockerfile) — this digest is a | ||
| # placeholder and does not exist in the registry yet. | ||
| BUILD_IMAGE_X64_EL7: registry.ddbuild.io/ci/async-profiler-build:v0-x64-el7-base@sha256:0000000000000000000000000000000000000000000000000000000000000 |
There was a problem hiding this comment.
This value explicitly references a placeholder digest that does not exist in the registry, yet stresstest:x64 uses it on every normal pipeline and the new functional jobs use it on schedules. The runner will fail while pulling the image, before either suite can start, so the EL7 image must be built and this value replaced with its real tag and digest before merging.
Useful? React with 👍 / 👎.
| variables: | ||
| TARGET: linux-x64 | ||
| script: | ||
| - .gitlab/scripts/functional-tests.sh |
There was a problem hiding this comment.
When any scheduled functional job reaches this command after the image is repinned, it attempts to execute functional-tests.sh directly, but the new file is committed with mode 100644; the shell therefore exits with permission denied (status 126) without running the tests. Commit it as executable or invoke it as bash .gitlab/scripts/functional-tests.sh.
AGENTS.md reference: AGENTS.md:L460-L461
Useful? React with 👍 / 👎.
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The EL7 stress job cannot pull its image. The nightly functional jobs also cannot start the script or load the required native test library.
🤖 Datadog Autotest · Commit daedd84 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| # TODO: repin after running `REBUILD_IMAGES="x64-el7"` through rebuild-images.sh | ||
| # in a real pipeline (see .gitlab/base/el7/Dockerfile) — this digest is a | ||
| # placeholder and does not exist in the registry yet. | ||
| BUILD_IMAGE_X64_EL7: registry.ddbuild.io/ci/async-profiler-build:v0-x64-el7-base@sha256:0000000000000000000000000000000000000000000000000000000000000 |
There was a problem hiding this comment.
The required x64 stress job and all new functional jobs cannot run.
Assertion details
- Input: Any pipeline that creates stresstest:x64, or a scheduled functional job.
- Expected:
The runner pulls a valid EL7 image and starts the job. - Actual:
The runner cannot pull the image. The job stops before it starts.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
| variables: | ||
| TARGET: linux-x64 | ||
| script: | ||
| - .gitlab/scripts/functional-tests.sh |
There was a problem hiding this comment.
Every nightly functional job stops before it downloads a JDK or runs tests.
Assertion details
- Input: Any scheduled functional:x64-el7-jdk job.
- Expected:
The runner starts the functional test script. - Actual:
The runner gets a permission denied error because the file mode is 100644.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
| trap onexit EXIT | ||
|
|
||
| ./gradlew -Pddprof_version="$(get_version)" -Pskip-native -Pwith-libs="$(pwd)/libs" -PCI \ | ||
| ":ddprof-test:test${TEST_CONFIG}" --max-workers=1 --build-cache --stacktrace --info --no-watch-fs --no-daemon |
There was a problem hiding this comment.
The nightly functional suite fails after the script starts.
Assertion details
- Input: A functional test that loads libddproftest.so, such as a native thread or native memory test.
- Expected:
The job reuses the profiler artifact and supplies the required native test library. - Actual:
The skip-native option also skips the ddproftest link task. Tests that load ddproftest then fail.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
| trap onexit EXIT | ||
|
|
||
| ./gradlew -Pddprof_version="$(get_version)" -Pskip-native -Pwith-libs="$(pwd)/libs" -PCI \ | ||
| ":ddprof-test:test${TEST_CONFIG}" --max-workers=1 --build-cache --stacktrace --info --no-watch-fs --no-daemon |
There was a problem hiding this comment.
The nightly functional suite fails after the script starts.
Assertion details
- Input: A functional test that loads libddproftest.so, such as a native thread or native memory test.
- Expected:
The job reuses the profiler artifact and supplies the required native test library. - Actual:
The skip-native option also skips the ddproftest link task. Tests that load ddproftest then fail.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 2 Pipeline jobs failed
Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: 1b2b6d9 | Docs | View more details | Give us feedback! |
Keeps the 3-JDK sweep nightly but lets one JDK's worth of EL7 functional coverage run on every pipeline, so it isn't gated entirely on the nightly schedule. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI Test ResultsRun: #35367405162 | Commit:
Status Overview
Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled Summary: Total: 32 | Passed: 32 | Failed: 0 Updated: 2026-09-18 17:05:17 UTC |
Built and pushed successfully in pipeline 138399627, job 2056785023. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r probe stresstest:x64 failed with "No C++ compiler found": ConfigurationPresets. setupStandardConfigurations() calls PlatformUtils.findCompiler() unconditionally at project-configuration time, before -Pskip-native is consulted, so an image with no compiler at all fails to even configure the build. The stock EL7 gcc-c++ (4.8.5, base repo, no EPEL) satisfies that --version probe; -Pskip-native still gates every real compile/link task onlyIf, so it's never invoked to build anything. Verified the image still builds and g++ --version succeeds. This needs another `REBUILD_IMAGES="x64-el7"` run and a re-pin of BUILD_IMAGE_X64_EL7 before the fix takes effect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rebuilt in pipeline 138438605 with the gcc-c++ fix from the previous commit; pushed and signed successfully before the rebuild-images log cuts off (mid-way through the unrelated datadog-ci image build). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
functional:x64-el7-jdk8 failed with "Permission denied" — the script was committed as mode 100644 instead of 100755. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ContendedCallTraceStorageTest OOM'd with "GC overhead limit exceeded" on jdk8: it drives availableProcessors()*2 threads of CPU/allocation load against the standard -Xmx512m test heap, and the GitLab shared runner pod apparently gives it less headroom than GitHub's ubuntu-latest runners do. Adds an opt-in -PtestMaxHeap gradle property to ddprof-test (appended after the plugin's default -Xmx512m, so the last -Xmx wins) and passes -PtestMaxHeap=1536m from functional-tests.sh only. The shared default used by every other caller (GH Actions matrix, stresstest, etc.) is untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Several ddprof-test suites (NativeThread, DynamicNativeThread, ThreadEntryDetectionTest, ...) need ddprof-test-native's small JNI test helper on java.library.path regardless of which profiler binary is under test. Bare -Pskip-native disabled ALL native compilation project-wide, so those suites failed with UnsatisfiedLinkError/ NoClassDefFoundError on the EL7 functional job, which needs to skip only ddprof-lib's own build (substituted via -Pwith-libs) while still compiling that helper for real. PlatformUtils.isNativeSkipped() now supports -Pskip-native=<comma list of project names>, skipping only those; bare -Pskip-native (no value) keeps skipping everywhere, unchanged for every existing caller (stresstests.sh, build.sh, GH Actions). SimpleNativeLibPlugin (used by ddprof-test-native and malloc-shim) and ddprof-lib/build.gradle.kts's own skip-native check now go through it. functional-tests.sh passes -Pskip-native=ddprof-lib,malloc-shim (malloc-shim excluded too: it's unrelated to ddprof-test and its -std=c++17 flag doesn't compile under EL7's stock gcc 4.8.5). Verified locally: with -Pskip-native=ddprof-lib,malloc-shim, ddprof-test-native:linkLib actually compiles and links, while ddprof-lib:assembleDebugJar only runs copyExternalLibs (no native compile task in the graph at all). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that -Pskip-native=ddprof-lib,malloc-shim actually compiles ddprof-test-native for the EL7 functional job, nativealloc.c failed: error: 'for' loop initial declarations are only allowed in C99 mode EL7's stock gcc (4.8.5) still defaults to gnu89; every other Linux gcc/ clang this project builds on already defaults to a C99+ dialect, so this was latent until a compiler this old actually built the file. Adds an explicit -std=gnu99. Verified against the real EL7 toolchain (oraclelinux:7 + gcc 4.8.5): all three ddprof-test-native source files compile and link. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build:x64 (build.sh) only ever produces and ships the release-config
libjavaProfiler.so -- that IS the shipped artifact this job exists to
exercise. Running testDebug set ddprof_test.config=debug while the
loaded library was actually release-built, so debug-only assertions
failed outright instead of assumeTrue-skipping correctly, e.g.
JVMAccessTest.agentOnLoadVMFlagDetectionTest checking for
[TEST::INFO] log lines that only exist in debug builds
("initLibrary not found", "DebugNonSafepoints flag lookup log line
was not observed").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
functional:x64-el7-jdk8 has NativeSocketEvent count: 0 across every retry (zero, not just fewer-than-expected), suggesting the send/recv/ write/read hooks (libraryPatcher_linux.cpp's GOT patching) never actually get installed in this environment -- but that can't be told apart from "installed but never firing" without the debug-only hook-fire counters, which only exist in a debug-config build. build:x64-debug-diag builds a real debug-config libjavaProfiler.so on the same AlmaLinux 8 toolchain build:x64 uses (release configs don't carry NativeSocketSampler's #ifdef DEBUG counters/TEST_LOG calls). functional:x64-el7-debug-diag runs just the NativeSocket* suite against it on the EL7 runtime (via functional-tests.sh's new TESTS_FILTER -> -Ptests passthrough), so --info output shows whether send_hook/recv_hook/write_hook/read_hook ever get called at all. Both are marked TEMPORARY and should be removed once this is root-caused. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 participant
Summary
Follow-up to #790's "One coverage regression this introduces, and how to undo it" section.
Moving
build:x64to AlmaLinux 8 (#790) tookstresstest:x64with it, since it ran in the same image — runtime coverage at the shipped binary's actual glibc floor (2.17) went from thin to zero. This implements the two follow-ups #790 proposed:.stresstest_jobgets its own runtime image, independent of the image the artifact was built in.RUNTIME_IMAGEreplaces the reusedBUILD_IMAGEvariable in the job template;stresstest:x64now points it at a new Oracle Linux 7 image (glibc 2.17), whilebuild:x64stays on AlmaLinux 8 — matching ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790's own note that this "is better than the old arrangement, because it would exercise the el8-built artifact on 2.17 rather than a 2.17-built one." Every otherstresstest:*job keeps its existing image, just renamed to the new variable.functional:x64-el7-jdk{8,17,21}, runs theddprof-testdebug suite against the shipped x64 artifact on the same EL7 image — the "libc-sensitive suites" coverage ci: move the shipped binaries to AlmaLinux 8, get the test images off EOL Debian #790 called out.jdk17andjdk21are nightly-only (CI_PIPELINE_SOURCE == "schedule", same pattern as the existingchaos:buildjob), so the full 3-JDK sweep doesn't multiply the per-PR matrix.jdk8additionally runs on every pipeline — it's the oldest/most different from the JDK 21 that runs Gradle, so it's the one job worth the added per-PR cost, and it keeps EL7 functional coverage from being gated on the nightly schedule entirely.Details
.gitlab/base/el7/Dockerfile(new): Oracle Linux 7, runtime-only — no compiler. Both jobs consume the AlmaLinux-8-built.sovia-Pskip-native -Pwith-libs, so nothing here needs to compile anything. Built and smoke-tested locally with Docker (JDK install via SDKMAN,java -versionon glibc 2.17), then built and pushed for real in pipeline 138399627 (job 2056785023):registry.ddbuild.io/ci/async-profiler-build:v138399627-x64-el7-base@sha256:d324b9b6c3fc48c552185e8969d88c836fb1e17a0e5dc483ba04bbd4f4ed2f38, now pinned asBUILD_IMAGE_X64_EL7..gitlab/scripts/functional-tests.sh(new): fetches the JDK under test from Adoptium intoJAVA_TEST_HOME, so Gradle itself keeps running on JDK 21 (Gradle 9's minimum) regardless of which JDK the suite targets. Verified the Adoptium downloads and resultingjava -versionfor 8/11/17/21 from inside the EL7 image..gitlab/build-deploy/images.yml,rebuild-images.sh,check-image-updates.sh: wiredBASE_IMAGE_EL7/x64-el7through the same image-build, repin and staleness-check machinery as the existing images.Not done here
That same
rebuild-imagesrun also failed to rebuildx64-glibcandarm64(both pre-existing images, unrelated to this change) — SDKMAN's CDN flaked mid-download on both (Stop! The archive was corrupt and has been removed!). Their pins are untouched here; a retry ofrebuild-imagesshould fix those independently.Test plan
.gitlab/base/el7/Dockerfilebuilds locally (docker build) and the resulting image runsjava -versionagainst glibc 2.17javainside the EL7 image.gitlab/build-deploy/.gitlab-ci.ymlandimages.ymlYAML-validatedbash -n+shellcheckonfunctional-tests.shx64-el7image built and pushed via a real pipeline, digest pinnedstresstest:x64andfunctional:x64-el7-jdk*green in a real GitLab pipeline (both nightly and per-PR paths)🤖 Generated with Claude Code