Skip to content

Use LowMemory in the IP Intelligence examples built for the Enterprise file - #73

Draft
jwrosewell wants to merge 1 commit into
rcfrom
fix/ipi-examples-low-memory
Draft

jwrosewell wants to merge 1 commit into
rcfrom
fix/ipi-examples-low-memory

Conversation

@jwrosewell

Copy link
Copy Markdown
Contributor

Problem

The Enterprise IP Intelligence data file is over 6 GB (6,488,178,052 bytes for 7 November 2026), and the MaxPerformance profile loads the whole file into memory. Several IP Intelligence console examples that only do anything useful with the Enterprise file used MaxPerformance, so someone running them as written needs more than 6 GB of free memory before the first lookup.

The rule this change follows is that an IP Intelligence example built for the Enterprise file, or one that has to be given it, uses the LowMemory profile, which reads the data file from disk as it is needed. Examples that load every property from the free Lite data file stay as they are. The Metrics and Compare examples default to LowMemory and take an argument that chooses MaxPerformance for speed. Each comment says why.

Change

All paths are under examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/.

Example Before After Why
SuspiciousOnPrem.java MaxPerformance (line 165 on rc) LowMemory It judges an address with properties only the Enterprise file has (ENTERPRISE_PROPERTIES), and with any other file it says which and stops, so it is only ever run on the Enterprise file.
UpdateDataFile.java No profile, so the engine default LowMemory, named explicitly It downloads the Enterprise file with a licence key and is built around that file. Naming the profile also keeps it the same whatever the engine default is.
MetricsOnPrem.java MaxPerformance (line 365 on rc) LowMemory by default, or the profile named as the fourth argument It needs the Enterprise file's location properties. It reads the values of every profile in the file, which is where MaxPerformance helps most, so the argument lets it be chosen where there is memory for it.
CompareOnPrem.java MaxPerformance (line 235 on rc) LowMemory by default, or the profile named as the fourth argument It needs the Enterprise file's location properties, and MaxPerformance makes each lookup faster where there is memory for the whole file.
OfflineProcessing.java MaxPerformance, under a comment describing the low memory profile (lines 181 to 187 on rc) MaxPerformance, unchanged It runs on the free Lite data file, so it keeps its profile. The comment now says what the code does and why, and the commented-out alternatives list LowMemory in place of a second copy of the MaxPerformance line.

For Metrics and Compare:

  • A DEFAULT_PERFORMANCE_PROFILE constant holds LowMemory, and its documentation gives the trade-off. For Metrics it quotes the measurement the .NET version of the example, which uses the same native library, records in its own comment, being about 166 profiles a second with LowMemory against 2,013 with MaxPerformance (.NET Metrics-Console). I have not measured the Java example, because MaxPerformance on the Enterprise file needs over 6 GB of free memory, more than I could safely use on the machine I used.
  • main reads the profile from a fourth argument, after the three each example already takes, with Constants.PerformanceProfiles.valueOf, in the same style as those arguments. The .NET example takes its profile as its sixth argument (line 1440).
  • The existing run methods, which the example tests call, keep their signatures and use the default. A new run overload takes the profile. So MetricsOnPremTest and CompareOnPremTest now run on LowMemory, which also spares CI from loading the whole Enterprise file, for example for the Metrics test, which reads ten ranges.
  • The class documentation mentions the argument.

The examples README now says which examples use LowMemory and why, and that Metrics and Compare take a profile as their fourth argument.

Not changed are OfflineProcessing's profile, the getting-started.api web example, PerformanceBenchmark, and every device detection example.

Checks

  • mvn -B -pl console -am test-compile in examples/ip-intelligence compiled the examples' shared and console modules, main and test sources, with Maven 3.9.16 on Java 21 against the published packages (BUILD SUCCESS).
  • I did not run the examples. The Metrics, Compare and Suspicious examples need the Enterprise data file, and their tests skip without it.
  • The added text has no em dashes, and every changed file keeps its line endings.

What remains

  • CI runs the example tests with the Enterprise data file where it has one. Those tests now build the engine with LowMemory.
  • The .NET Metrics-Console still defaults to MaxPerformance, so the two languages now differ on that example's default.

…e file

The Enterprise IP Intelligence data file is over 6 GB, and MaxPerformance
loads all of it into memory, so the examples that need it, or that are
built around it, now use the LowMemory profile, which reads the file
from disk as it is needed.

- SuspiciousOnPrem needs properties only the Enterprise file has, and
  moves from MaxPerformance to LowMemory.
- UpdateDataFile downloads the Enterprise file and took the engine
  default, and now names LowMemory.
- MetricsOnPrem and CompareOnPrem default to LowMemory and take a
  profile as an optional fourth argument, so MaxPerformance can be
  chosen for speed where there is memory for the whole file. Their
  existing run methods keep their signatures and use the default.
- OfflineProcessing runs on the Lite file and keeps MaxPerformance. Its
  comment described the low memory profile directly above the
  MaxPerformance line, and now says what the code does and why.

The examples README says which examples use LowMemory and why.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant