Create a 'data' directory and mount it in the agent.

.gitignore

@@ -10,6 +10,10 @@ rulesets/zeek/
 app/static/engine-configs/snort/
 app/static/engine-configs/suricata/
 
+# don't track the contents of the data directory
+data/**
+!data/.gitkeep
+
 # don't include docker-compose
 docker-compose.yml
 #Dockerfile*

README.rst

@@ -69,6 +69,7 @@ Contents
    -  `Non-Docker Sensors <#non-docker-sensors>`__
 
 -  `Adding Sensor Configs <#adding-sensor-configs>`__
+-  `Agent Data Directory <#agent-data-directory>`__
 -  `Logging and Debugging <#logging-and-debugging>`__
 -  `Flowsynth WebUI <#flowsynth-webui>`__
 -  `Zeek <#zeek>`__
@@ -411,7 +412,7 @@ number of user-configurable options:
         size for the EVE log is 512MB; see note about 512MB limit for
         'Other logs'.
    -  | **Other logs (Alert Debug, HTTP, TLS, DNS, etc.)**
-      | *Suricata only*.  This will return other logs generated by the
+      | *Suricata only*.  This will return other logs generated by the
         engine that can be useful for analysis and debugging.
         Depending on the version
         of Suricata running on the agent, some logs may not be supported.
@@ -431,12 +432,12 @@ number of user-configurable options:
            application layer protocols, etc.
       -  | **Alert Debug**
          | Detailed information on what particular rules matched on for
-           each alert.  Useful for seeing why an alert fired and/or
+           each alert.  Useful for seeing why an alert fired and/or
            troubleshooting false positives.
       -  | **HTTP Log**
          | A log of HTTP requests and responses, showing time, IPs and
            ports, HTTP method, URI, HTTP version, Host, User-Agent,
-           Referer, response code, response size, etc.  By default, each
+           Referer, response code, response size, etc.  By default, each
            line represents the HTTP request and response all in one.
       -  | **DNS Log**
          | A log of DNS requests and responses as provided by Suricata.
@@ -464,10 +465,10 @@ number of user-configurable options:
    -  | **Fast pattern info**
 
       -  *Suricata only*. Return fast pattern data about the submitted
-         rules.  The Dalton Suricata agent will return a file (displayed
+         rules.  The Dalton Suricata agent will return a file (displayed
          in the "Fast Pattern" tab) with details on what the engine is
-         using for the fast pattern match.  To generate this, Suricata
-         must do two runs – one to generate the fast pattern info and
+         using for the fast pattern match.  To generate this, Suricata
+         must do two runs – one to generate the fast pattern info and
          one to actually run the submitted job so this will approximately
          double the job run time. Unless fast pattern info is needed for
          some reason, there isn't a need to check this. Fast pattern
@@ -526,8 +527,8 @@ presents the results from the job run in a tabulated interface:
      shown.
 -  | **Other logs**
    | If other logs are returned by the agent they will each be displayed
-     in their own tab if they are non-empty.  ``Engine Stats`` and ``Packet
-     Stats`` are always returned for Suricata jobs.  See discussion in the
+     in their own tab if they are non-empty.  ``Engine Stats`` and ``Packet
+     Stats`` are always returned for Suricata jobs.  See discussion in the
      above "Configuration Options" discussion for more details.
 
 Job Queue
@@ -538,7 +539,7 @@ a quasi-unique sixteen byte Job ID, which is based on the job's runtime
 parameters. Each recent Job ID is included on the 'Queue' page as a
 hyperlink for easy access. Queued jobs will be cleared out periodically 
 if an agent has not picked them up; this should not happen unless
-all agents are down or are unreasonably backlogged.  There is additional
+all agents are down or are unreasonably backlogged.  There is additional
 logic in the Dalton controller to respond appropriately when jobs have
 timed out or have been interrupted; this should happen rarely, if ever.
 
@@ -588,7 +589,7 @@ Job API
 -------
 
 The Dalton controller provides a RESTful API to retrieve data about
-submitted jobs.  API responses use JSON or the raw ("RAW") data, and
+submitted jobs.  API responses use JSON or the raw ("RAW") data, and
 the data returned in the values is, 
 in most cases, just the raw text that is displayed in the Dalton web interface.
 
@@ -608,16 +609,16 @@ For requests, ``<jobid>`` is the Job ID and::
 A JSON API request returns JSON with three root elements:
 
 -  | **data**
-   | The requested data.  If the key is invalid for the
+   | The requested data.  If the key is invalid for the
      job, then an error is returned, along with an error message stating
      as such. If there is no data for the requested Job ID and key, then
      this ``data`` parameter value is an empty string and ``error`` is set
      to false..
 
 -  | **error**
    | [true\|false] depending if the API request generated an error. This is
-     not returned as a quoted string.  \ **This** **indicates an error with
-     the API request, not an error running the job**.  Errors running the job
+     not returned as a quoted string.  \ **This** **indicates an error with
+     the API request, not an error running the job**.  Errors running the job
      can be found by querying for the 'error' key (see above).
 
 -  | **error_msg**
@@ -656,7 +657,7 @@ in the Content-Disposition header that prompt browsers to download/save the file
 
 -  **all** - Returns data from all keys (except for "all" of course).
 
--  **debug** - Debug data from the job.  This is the same as what is
+-  **debug** - Debug data from the job.  This is the same as what is
    displayed in the "Debug" tab in the job results page.
 
 -  **dns\_log** - A line based log of DNS requests and responses (Suricata only).
@@ -667,10 +668,10 @@ in the Content-Disposition header that prompt browsers to download/save the file
    engine (Suricata only).  This is the same as what is displayed in
    the "Engine Stats" tab in the job results page.
 
--  **error** - Error data from the job.  This is the same as what is
+-  **error** - Error data from the job.  This is the same as what is
    displayed in the "Error" tab in the job results page.
 
--  **eve** - EVE JSON output from the job (Suricata only).  This is the same as what is
+-  **eve** - EVE JSON output from the job (Suricata only).  This is the same as what is
    displayed in the "EVE JSON" tab in the job results page.
 
 -  **fast\_pattern** - Fast pattern details for the submitted rules (Suricata only).
@@ -680,8 +681,8 @@ in the Content-Disposition header that prompt browsers to download/save the file
 -  **http\_log** - A line based log of HTTP requests (Suricata only).  This is the
    same as what is displayed in the "HTTP Log" tab in the job results page.
 
--  **ids** - IDS Engine output from the job.  This is the same as what
-   is displayed in the "IDS Engine" tab in the job results page.  
+-  **ids** - IDS Engine output from the job.  This is the same as what
+   is displayed in the "IDS Engine" tab in the job results page.
    For Snort Agents, engine statistics output at the end of the job 
    run are populated here.
 
@@ -700,15 +701,15 @@ in the Content-Disposition header that prompt browsers to download/save the file
    same as what is displayed in the "Engine Stats" tab in the job results page.
 
 -  **perf** - Performance data from the job (if the job generated
-   performance data).   This is the same as what is displayed in the
+   performance data).   This is the same as what is displayed in the
    "Performance" tab in the job results page.
 
 -  **start\_time** - The time (epoch) the job was requested by a Dalton
-   agent.  This is returned as a string.
+   agent.  This is returned as a string.
 
--  **statcode** - Status code of the job.  This is a number returned as
-   a string.  If a job doesn't exist, the API will return an error (see
-   below) instead of an "Invalid" statcode.  Here is how to interpret
+-  **statcode** - Status code of the job.  This is a number returned as
+   a string.  If a job doesn't exist, the API will return an error (see
+   below) instead of an "Invalid" statcode.  Here is how to interpret
    the status code:
 
    +-------+-------------+
@@ -727,12 +728,12 @@ in the Content-Disposition header that prompt browsers to download/save the file
    |   4   |   Timeout   |
    +-------+-------------+
 
--  **status** - A string corresponding to the current status of a job. 
+-  **status** - A string corresponding to the current status of a job.
    This is used in the Dalton Controller web UI and is what is displayed
    in the browser when a job is submitted via the web interface to
-   inform the user of the current progress/state of the job.  When a job
+   inform the user of the current progress/state of the job.  When a job
    is done, this will actually be a hyperlink saying "Click here to view
-   your results".  Unless there is a specific use case, 'statcode' is 
+   your results".  Unless there is a specific use case, 'statcode' is
    usually used instead of 'status' for determining the status of a job.
 
 -  **submission\_time** - The time (formatted as "%b %d %H:%M:%S") the
@@ -747,7 +748,7 @@ in the Content-Disposition header that prompt browsers to download/save the file
    for example, ``suricata/rust_4.1.5``.
 
 -  **time** - The time in seconds the job took to run, as reported by
-   the Dalton Agent (this includes job download time by the agent). 
+   the Dalton Agent (this includes job download time by the agent).
    This is returned as a string and is the same as the "Processing Time"
    displayed in the job results page.
 
@@ -939,13 +940,13 @@ An example script can be found in ``api/examples/submit_job.py``.
 Teapot Jobs
 ===========
 
-Dalton has the concept and capability of what is called a "teapot" job. 
+Dalton has the concept and capability of what is called a "teapot" job.
 A teapot job is one that is short lived in the Redis database and
 (usually) on disk.
 
 Teapot jobs are useful when submitting large number of jobs and/or jobs
 where the results are immediately processed and there isn't a need to
-keep them around after that.  Often this is utilized in the programmatic
+keep them around after that.  Often this is utilized in the programmatic
 submission of jobs combined with using the `Dalton API <#dalton-api>`__
 to automatically and/or quickly process the results.
 
@@ -958,7 +959,7 @@ Teapot jobs differ from regular jobs in a few main ways:
    Teapot job expire timeouts are  configured with the ``teapot_redis_expire`` 
    option in ``dalton.conf``.
 -  Teapot jobs are submitted using the 'teapotJob' POST parameter (with
-   any value).  This parameter is not set or available when submitting
+   any value).  This parameter is not set or available when submitting
    jobs via the Dalton web UI.
 -  Teapot jobs have a job id ("JID") that starts with 'teapot\_'.
 -  The submission of a teapot job results in the JID being returned
@@ -1217,11 +1218,26 @@ default ``.conf`` file be run thru  the ``clean_snort_config.py``
 script located in the ``engine-configs/`` directory::
 
     Usage:
-    
+
     python clean_snort_config.py <in-file> <out-file>
 
 
 
+Agent Data Directory
+=====================
+
+The ``data`` directory at the root of the Dalton project is bind-mounted
+read-only into the Suricata and Snort Agent containers at
+``/opt/dalton-agent/data``. Processes running in an Agent container (e.g.
+Suricata or Snort itself) can read files under this path, but nothing in
+the container can write to it.
+
+This is intended as a generic drop location for any data you want to be
+accessed by the Agent.  An example might be a Suricata variables file,
+that could then be referenced with an ``include:`` directive from a
+sensor's engine config.
+
+
 Logging and Debugging
 =====================
 
@@ -1473,4 +1489,3 @@ Contributors
 Feedback including bug reports, suggestions, improvements, questions,
 etc. is welcome.
 
-

docker-compose.yml

@@ -85,6 +85,8 @@ services:
     container_name: suricata-current
     environment:
       - AGENT_DEBUG=${AGENT_DEBUG}
+    volumes:
+      - ./data:/opt/dalton-agent/data:ro
     restart: always
 
 # Suricata 7.0.14 from source, with Rust support
@@ -101,6 +103,8 @@ services:
     container_name: suricata-7.0.14
     environment:
       - AGENT_DEBUG=${AGENT_DEBUG}
+    volumes:
+      - ./data:/opt/dalton-agent/data:ro
     restart: always
 
 # Suricata 6.0.20 from source, with Rust support
@@ -293,6 +297,8 @@ services:
     container_name: snort-2.9.20
     environment:
       - AGENT_DEBUG=${AGENT_DEBUG}
+    volumes:
+      - ./data:/opt/dalton-agent/data:ro
     restart: always
 
 # Snort 2.9.18 from source