Repository navigation
Use LowMemory in the IP Intelligence examples built for the Enterprise file - #73
Draft
jwrosewell wants to merge 1 commit into
Draft
jwrosewell wants to merge 1 commit into
jwrosewell wants to merge 1 commit into
Conversation
…e file The Enterprise IP Intelligence data file is over 6 GB, and MaxPerformance loads all of it into memory, so the examples that need it, or that are built around it, now use the LowMemory profile, which reads the file from disk as it is needed. - SuspiciousOnPrem needs properties only the Enterprise file has, and moves from MaxPerformance to LowMemory. - UpdateDataFile downloads the Enterprise file and took the engine default, and now names LowMemory. - MetricsOnPrem and CompareOnPrem default to LowMemory and take a profile as an optional fourth argument, so MaxPerformance can be chosen for speed where there is memory for the whole file. Their existing run methods keep their signatures and use the default. - OfflineProcessing runs on the Lite file and keeps MaxPerformance. Its comment described the low memory profile directly above the MaxPerformance line, and now says what the code does and why. The examples README says which examples use LowMemory and why.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Enterprise IP Intelligence data file is over 6 GB (6,488,178,052 bytes for 7 November 2026), and the MaxPerformance profile loads the whole file into memory. Several IP Intelligence console examples that only do anything useful with the Enterprise file used MaxPerformance, so someone running them as written needs more than 6 GB of free memory before the first lookup.
The rule this change follows is that an IP Intelligence example built for the Enterprise file, or one that has to be given it, uses the LowMemory profile, which reads the data file from disk as it is needed. Examples that load every property from the free Lite data file stay as they are. The Metrics and Compare examples default to LowMemory and take an argument that chooses MaxPerformance for speed. Each comment says why.
Change
All paths are under
examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/.SuspiciousOnPrem.javaENTERPRISE_PROPERTIES), and with any other file it says which and stops, so it is only ever run on the Enterprise file.UpdateDataFile.javaMetricsOnPrem.javaCompareOnPrem.javaOfflineProcessing.javaFor Metrics and Compare:
DEFAULT_PERFORMANCE_PROFILEconstant holds LowMemory, and its documentation gives the trade-off. For Metrics it quotes the measurement the .NET version of the example, which uses the same native library, records in its own comment, being about 166 profiles a second with LowMemory against 2,013 with MaxPerformance (.NETMetrics-Console). I have not measured the Java example, because MaxPerformance on the Enterprise file needs over 6 GB of free memory, more than I could safely use on the machine I used.mainreads the profile from a fourth argument, after the three each example already takes, withConstants.PerformanceProfiles.valueOf, in the same style as those arguments. The .NET example takes its profile as its sixth argument (line 1440).runmethods, which the example tests call, keep their signatures and use the default. A newrunoverload takes the profile. SoMetricsOnPremTestandCompareOnPremTestnow run on LowMemory, which also spares CI from loading the whole Enterprise file, for example for the Metrics test, which reads ten ranges.The examples README now says which examples use LowMemory and why, and that Metrics and Compare take a profile as their fourth argument.
Not changed are
OfflineProcessing's profile, thegetting-started.apiweb example,PerformanceBenchmark, and every device detection example.Checks
mvn -B -pl console -am test-compileinexamples/ip-intelligencecompiled the examples' shared and console modules, main and test sources, with Maven 3.9.16 on Java 21 against the published packages (BUILD SUCCESS).What remains
Metrics-Consolestill defaults to MaxPerformance, so the two languages now differ on that example's default.