diff --git a/.gitignore b/.gitignore index e7783fed..8ea03447 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,10 @@ rulesets/zeek/ app/static/engine-configs/snort/ app/static/engine-configs/suricata/ +# don't track the contents of the data directory +data/** +!data/.gitkeep + # don't include docker-compose docker-compose.yml #Dockerfile* diff --git a/README.rst b/README.rst index 25e76caa..7c6d8f7f 100644 --- a/README.rst +++ b/README.rst @@ -69,6 +69,7 @@ Contents - `Non-Docker Sensors <#non-docker-sensors>`__ - `Adding Sensor Configs <#adding-sensor-configs>`__ +- `Agent Data Directory <#agent-data-directory>`__ - `Logging and Debugging <#logging-and-debugging>`__ - `Flowsynth WebUI <#flowsynth-webui>`__ - `Zeek <#zeek>`__ @@ -411,7 +412,7 @@ number of user-configurable options: size for the EVE log is 512MB; see note about 512MB limit for 'Other logs'. - | **Other logs (Alert Debug, HTTP, TLS, DNS, etc.)** - | *Suricata only*.  This will return other logs generated by the + | *Suricata only*. This will return other logs generated by the engine that can be useful for analysis and debugging. Depending on the version of Suricata running on the agent, some logs may not be supported. @@ -431,12 +432,12 @@ number of user-configurable options: application layer protocols, etc. - | **Alert Debug** | Detailed information on what particular rules matched on for - each alert.  Useful for seeing why an alert fired and/or + each alert. Useful for seeing why an alert fired and/or troubleshooting false positives. - | **HTTP Log** | A log of HTTP requests and responses, showing time, IPs and ports, HTTP method, URI, HTTP version, Host, User-Agent, - Referer, response code, response size, etc.  By default, each + Referer, response code, response size, etc. By default, each line represents the HTTP request and response all in one. - | **DNS Log** | A log of DNS requests and responses as provided by Suricata. @@ -464,10 +465,10 @@ number of user-configurable options: - | **Fast pattern info** - *Suricata only*. Return fast pattern data about the submitted - rules.  The Dalton Suricata agent will return a file (displayed + rules. The Dalton Suricata agent will return a file (displayed in the "Fast Pattern" tab) with details on what the engine is - using for the fast pattern match.  To generate this, Suricata - must do two runs – one to generate the fast pattern info and + using for the fast pattern match. To generate this, Suricata + must do two runs – one to generate the fast pattern info and one to actually run the submitted job so this will approximately double the job run time. Unless fast pattern info is needed for some reason, there isn't a need to check this. Fast pattern @@ -526,8 +527,8 @@ presents the results from the job run in a tabulated interface: shown. - | **Other logs** | If other logs are returned by the agent they will each be displayed - in their own tab if they are non-empty.  ``Engine Stats`` and ``Packet - Stats`` are always returned for Suricata jobs.  See discussion in the + in their own tab if they are non-empty. ``Engine Stats`` and ``Packet + Stats`` are always returned for Suricata jobs. See discussion in the above "Configuration Options" discussion for more details. Job Queue @@ -538,7 +539,7 @@ a quasi-unique sixteen byte Job ID, which is based on the job's runtime parameters. Each recent Job ID is included on the 'Queue' page as a hyperlink for easy access. Queued jobs will be cleared out periodically if an agent has not picked them up; this should not happen unless -all agents are down or are unreasonably backlogged.  There is additional +all agents are down or are unreasonably backlogged. There is additional logic in the Dalton controller to respond appropriately when jobs have timed out or have been interrupted; this should happen rarely, if ever. @@ -588,7 +589,7 @@ Job API ------- The Dalton controller provides a RESTful API to retrieve data about -submitted jobs.  API responses use JSON or the raw ("RAW") data, and +submitted jobs. API responses use JSON or the raw ("RAW") data, and the data returned in the values is, in most cases, just the raw text that is displayed in the Dalton web interface. @@ -608,7 +609,7 @@ For requests, ```` is the Job ID and:: A JSON API request returns JSON with three root elements: - | **data** - | The requested data.  If the key is invalid for the + | The requested data. If the key is invalid for the job, then an error is returned, along with an error message stating as such. If there is no data for the requested Job ID and key, then this ``data`` parameter value is an empty string and ``error`` is set @@ -616,8 +617,8 @@ A JSON API request returns JSON with three root elements: - | **error** | [true\|false] depending if the API request generated an error. This is - not returned as a quoted string.  \ **This** **indicates an error with - the API request, not an error running the job**.  Errors running the job + not returned as a quoted string. \ **This** **indicates an error with + the API request, not an error running the job**. Errors running the job can be found by querying for the 'error' key (see above). - | **error_msg** @@ -656,7 +657,7 @@ in the Content-Disposition header that prompt browsers to download/save the file - **all** - Returns data from all keys (except for "all" of course). -- **debug** - Debug data from the job.  This is the same as what is +- **debug** - Debug data from the job. This is the same as what is displayed in the "Debug" tab in the job results page. - **dns\_log** - A line based log of DNS requests and responses (Suricata only). @@ -667,10 +668,10 @@ in the Content-Disposition header that prompt browsers to download/save the file engine (Suricata only). This is the same as what is displayed in the "Engine Stats" tab in the job results page. -- **error** - Error data from the job.  This is the same as what is +- **error** - Error data from the job. This is the same as what is displayed in the "Error" tab in the job results page. -- **eve** - EVE JSON output from the job (Suricata only).  This is the same as what is +- **eve** - EVE JSON output from the job (Suricata only). This is the same as what is displayed in the "EVE JSON" tab in the job results page. - **fast\_pattern** - Fast pattern details for the submitted rules (Suricata only). @@ -680,8 +681,8 @@ in the Content-Disposition header that prompt browsers to download/save the file - **http\_log** - A line based log of HTTP requests (Suricata only). This is the same as what is displayed in the "HTTP Log" tab in the job results page. -- **ids** - IDS Engine output from the job.  This is the same as what - is displayed in the "IDS Engine" tab in the job results page.  +- **ids** - IDS Engine output from the job. This is the same as what + is displayed in the "IDS Engine" tab in the job results page. For Snort Agents, engine statistics output at the end of the job run are populated here. @@ -700,15 +701,15 @@ in the Content-Disposition header that prompt browsers to download/save the file same as what is displayed in the "Engine Stats" tab in the job results page. - **perf** - Performance data from the job (if the job generated - performance data).   This is the same as what is displayed in the + performance data). This is the same as what is displayed in the "Performance" tab in the job results page. - **start\_time** - The time (epoch) the job was requested by a Dalton - agent.  This is returned as a string. + agent. This is returned as a string. -- **statcode** - Status code of the job.  This is a number returned as - a string.  If a job doesn't exist, the API will return an error (see - below) instead of an "Invalid" statcode.  Here is how to interpret +- **statcode** - Status code of the job. This is a number returned as + a string. If a job doesn't exist, the API will return an error (see + below) instead of an "Invalid" statcode. Here is how to interpret the status code: +-------+-------------+ @@ -727,12 +728,12 @@ in the Content-Disposition header that prompt browsers to download/save the file | 4 | Timeout | +-------+-------------+ -- **status** - A string corresponding to the current status of a job.  +- **status** - A string corresponding to the current status of a job. This is used in the Dalton Controller web UI and is what is displayed in the browser when a job is submitted via the web interface to - inform the user of the current progress/state of the job.  When a job + inform the user of the current progress/state of the job. When a job is done, this will actually be a hyperlink saying "Click here to view - your results".  Unless there is a specific use case, 'statcode' is + your results". Unless there is a specific use case, 'statcode' is usually used instead of 'status' for determining the status of a job. - **submission\_time** - The time (formatted as "%b %d %H:%M:%S") the @@ -747,7 +748,7 @@ in the Content-Disposition header that prompt browsers to download/save the file for example, ``suricata/rust_4.1.5``. - **time** - The time in seconds the job took to run, as reported by - the Dalton Agent (this includes job download time by the agent).  + the Dalton Agent (this includes job download time by the agent). This is returned as a string and is the same as the "Processing Time" displayed in the job results page. @@ -939,13 +940,13 @@ An example script can be found in ``api/examples/submit_job.py``. Teapot Jobs =========== -Dalton has the concept and capability of what is called a "teapot" job.  +Dalton has the concept and capability of what is called a "teapot" job. A teapot job is one that is short lived in the Redis database and (usually) on disk. Teapot jobs are useful when submitting large number of jobs and/or jobs where the results are immediately processed and there isn't a need to -keep them around after that.  Often this is utilized in the programmatic +keep them around after that. Often this is utilized in the programmatic submission of jobs combined with using the `Dalton API <#dalton-api>`__ to automatically and/or quickly process the results. @@ -958,7 +959,7 @@ Teapot jobs differ from regular jobs in a few main ways: Teapot job expire timeouts are configured with the ``teapot_redis_expire`` option in ``dalton.conf``. - Teapot jobs are submitted using the 'teapotJob' POST parameter (with - any value).  This parameter is not set or available when submitting + any value). This parameter is not set or available when submitting jobs via the Dalton web UI. - Teapot jobs have a job id ("JID") that starts with 'teapot\_'. - The submission of a teapot job results in the JID being returned @@ -1217,11 +1218,26 @@ default ``.conf`` file be run thru the ``clean_snort_config.py`` script located in the ``engine-configs/`` directory:: Usage: - + python clean_snort_config.py +Agent Data Directory +===================== + +The ``data`` directory at the root of the Dalton project is bind-mounted +read-only into the Suricata and Snort Agent containers at +``/opt/dalton-agent/data``. Processes running in an Agent container (e.g. +Suricata or Snort itself) can read files under this path, but nothing in +the container can write to it. + +This is intended as a generic drop location for any data you want to be +accessed by the Agent. An example might be a Suricata variables file, +that could then be referenced with an ``include:`` directive from a +sensor's engine config. + + Logging and Debugging ===================== @@ -1473,4 +1489,3 @@ Contributors Feedback including bug reports, suggestions, improvements, questions, etc. is welcome. -  diff --git a/data/.gitkeep b/data/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/docker-compose.yml b/docker-compose.yml index 766893ae..77b41a53 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -85,6 +85,8 @@ services: container_name: suricata-current environment: - AGENT_DEBUG=${AGENT_DEBUG} + volumes: + - ./data:/opt/dalton-agent/data:ro restart: always # Suricata 7.0.14 from source, with Rust support @@ -101,6 +103,8 @@ services: container_name: suricata-7.0.14 environment: - AGENT_DEBUG=${AGENT_DEBUG} + volumes: + - ./data:/opt/dalton-agent/data:ro restart: always # Suricata 6.0.20 from source, with Rust support @@ -293,6 +297,8 @@ services: container_name: snort-2.9.20 environment: - AGENT_DEBUG=${AGENT_DEBUG} + volumes: + - ./data:/opt/dalton-agent/data:ro restart: always # Snort 2.9.18 from source