Skip to content

feat(server): execute local REST Gremlin in process - #3388

Open
contrueCT wants to merge 5 commits into
apache:masterfrom
hugegraph:task/gremlin-embedded-rest
Open

contrueCT wants to merge 5 commits into
apache:masterfrom
hugegraph:task/gremlin-embedded-rest

Conversation

@contrueCT

@contrueCT contrueCT commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

Remove the internal HTTP hop for REST GET /gremlin and POST /gremlin when their target is the Gremlin Server co-started in the same process. Targets confirmed to belong to the co-started listener automatically reuse the existing executor when their configuration supports embedded execution; other targets and unsupported configurations retain HTTP forwarding. No execution-mode configuration option is required, and TLS listeners support local execution.

Builds on the merged Java 17 (#3261) and TinkerPop 3.8.1 (#3263) upgrades. Design and investigation.
Eligible embedded execution and HTTP forwarding

Main Changes

eligible local: REST -----------------------------------> GremlinExecutor
other targets:  REST -> GremlinClient -> Gremlin HTTP ---> GremlinExecutor
                                                          materialize / commit / serialize
  • Reuse the live graph manager, workers and initialized JSON serializer, with authentication context and transaction callbacks on the worker.
  • Match every resolved target address against the listener bind and port. A specific bind requires exact address matches; a wildcard bind accepts concrete local addresses in the same address family. Mismatched, mixed, cross-family and standalone-REST cases retain forwarding.
  • Support HTTP and HTTPS target URLs and TLS-enabled Gremlin listeners. Listener TLS configuration remains intact; local execution uses the REST entry point's authentication and transport without an internal TLS handshake.
  • Preserve request/error handling, blocking admission and timeout behavior. Custom Channelizers/authorizers, custom or mismatched authentication, and different or unresolvable authentication config files retain HTTP forwarding with a warning instead of introducing a startup rejection. Selection occurs before execution; scripts are never retried through another path after execution starts.
  • Add differential contract tests, automatic-selection and live TLS coverage. Update configuration docs and comparison runners; the RocksDB runner now requires a separate forward-default baseline build.

Readiness, new overload controls, authentication-cache consistency and global shutdown redesign remain separate tasks.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests.
  • Need tests and can be verified as follows:
    • Java 17: formatting, full clean compile and Apache RAT checks passed at e19099ef3.
    • 56 focused tests passed with zero failures or skips, including distinct same-port listeners, exact-bind and address-family matching, a working custom Channelizer through forwarding, authentication-file fallback, a real TLS listener, Memory/RocksDB transactions and the server-stop regression.
    • Python comparison runner syntax and documentation links checked.

HStore, full TinkerPop suites and real authentication-store integration remain unverified. Earlier remote RocksDB and HTTP comparisons are diagnostic evidence from earlier revisions; this change has not established production capacity or statistical performance acceptance.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects (Server startup/shutdown integration)
  • Nope

Documentation Status

  • Doc - TODO: required documentation is pending; complete it before merging.
  • Doc - Done: documentation is included here or linked below.
  • Doc - No Need: no user-visible documentation is affected.

Documentation: REST Gremlin execution, linked from the Server README and shipped configuration.

Embedded execution is selected only for a matching co-started listener and supported configuration. Specific binds require every resolved target address to match exactly; wildcard binds require concrete local addresses in the same address family. Unsupported Channelizer/authorizer/authentication settings or unresolved/mismatched authentication files use HTTP forwarding with a warning. Authentication-file matching applies when configured. Selection happens before execution, with no fallback retry after execution starts. Design illustration, not HStore, authentication-store, performance or RC acceptance.

- Add selectable forward and embedded REST executors with forward as default.
- Reuse the existing Gremlin workers, serializer, auth context and transactions.
- Integrate executor startup, shutdown and supported-topology validation.
- Add differential contract tests, usage docs and reproducible benchmarks.
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.35849% with 72 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.56%. Comparing base (c2ec570) to head (e19099e).
⚠️ Report is 40 commits behind head on master.

Files with missing lines Patch % Lines
...graph/api/gremlin/EmbeddedGremlinRestExecutor.java 78.30% 27 Missing and 14 partials ⚠️
...egraph/api/gremlin/GremlinRestExecutorFactory.java 88.88% 2 Missing and 5 partials ⚠️
...n/java/org/apache/hugegraph/server/RestServer.java 44.44% 4 Missing and 1 partial ⚠️
...ava/org/apache/hugegraph/dist/HugeGraphServer.java 50.00% 5 Missing ⚠️
...egraph/api/gremlin/ForwardGremlinRestExecutor.java 77.77% 2 Missing and 2 partials ⚠️
.../apache/hugegraph/api/gremlin/GremlinQueryAPI.java 50.00% 4 Missing ⚠️
...org/apache/hugegraph/server/ApplicationConfig.java 63.63% 3 Missing and 1 partial ⚠️
...a/org/apache/hugegraph/api/gremlin/GremlinAPI.java 80.00% 0 Missing and 1 partial ⚠️
...che/hugegraph/api/gremlin/GremlinRestExecutor.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3388      +/-   ##
============================================
+ Coverage     55.93%   56.56%   +0.62%     
- Complexity     2277     6782    +4505     
============================================
  Files           774      779       +5     
  Lines         68534    72750    +4216     
  Branches       9333    10736    +1403     
============================================
+ Hits          38334    41150    +2816     
- Misses        26634    27885    +1251     
- Partials       3566     3715     +149     
Flag Coverage Δ *Carryforward flag
commons 96.37% <ø> (ø) Carriedforward from a2f98d5
hstore 35.81% <ø> (+<0.01%) ⬆️ Carriedforward from a2f98d5
pd 49.15% <ø> (-0.03%) ⬇️ Carriedforward from a2f98d5
server_hbase 41.83% <ø> (-0.07%) ⬇️ Carriedforward from a2f98d5
server_memory 56.34% <77.35%> (+2.01%) ⬆️
server_rocksdb 42.55% <48.42%> (+0.21%) ⬆️
store 31.93% <ø> (ø) Carriedforward from a2f98d5

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Initialize the executor mock used by the isolated server stop fixture.
- Assert admission shutdown precedes Gremlin and REST shutdown.
- Validate the timeout regression with 48 Java 17 regression tests.
@contrueCT
contrueCT marked this pull request as ready for review October 27, 2026 16:50
imbajin
imbajin previously approved these changes Oct 28, 2026

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Embedded REST execution preserves worker-side context, transaction completion and serialization behavior, with explicit configuration guards. Evidence: Exact-head source review, comparison with pinned TinkerPop 3.8.1 behavior, passing server memory/RocksDB and macOS ARM CI; local runtime tests were not rerun. Score: 8.4/10.

- Resolve every configured Gremlin target address before enabling embedded mode.
- Reject mixed loopback and non-loopback DNS answers.
- Match the selected listener bind independent of target answer order.
- Add deterministic IPv4, IPv6, wildcard and mixed-address regression tests.
- Remove the REST execution-mode option and select embedded execution for local listener targets.
- Support local interfaces, host aliases and TLS listeners while preserving remote forwarding.
- Cover automatic selection, DNS address checks and a live TLS listener with regression tests.
- Update configuration docs and the baseline/candidate benchmark runner.
@contrueCT contrueCT changed the title feat(server): add embedded REST Gremlin execution feat(server): execute local REST Gremlin in process Oct 28, 2026

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Local target detection can route requests to a different Gremlin server than the configured endpoint. Evidence: The address checks accept distinct local bind and target addresses at the same port; existing tests also accept an IPv6 target for an IPv4-only bind.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: in-process execution reuses the Gremlin workers correctly, but local target detection accepts endpoints this listener does not own, and unsupported local configurations now fail startup with no way to keep HTTP forwarding.

- Match every target address against the listener bind and handle wildcard address families conservatively.
- Keep HTTP forwarding with a warning for unsupported local execution configurations.
- Cover distinct same-port listeners, custom Channelizer startup, authentication-file fallback and TLS behavior.
- Compare error response semantics independently of Groovy cache-dependent stack frames and document routing rules.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants