succulent

A lightweight framework for collecting and processing data from HTTP POST requests

🔍 Detailed Insights • 📦 Installation • 🐳 Container • 🚀 Usage • 🔧 Configuration • 📄 Cite us • 🔑 License • 🫂 Contributors

Do you ever find it challenging and tricky to send sensor measurements 📏, data 📊, or GPS positions from embedded devices 📱, microcontrollers, and smartwatches to a central server? 📡 Setting up the primary data collection scripts can be a time-consuming ⏳ process, involving selecting a protocol, framework, API, and testing them out. Moreover, these scripts are often tailored for specific tasks, making them difficult to adapt to different scenarios.

But fear not! Introducing succulent 🌵, a pure Python framework that simplifies the configuration, management, collection, and preprocessing of data collected via POST requests. This framework draws inspiration from real-world data collection challenges in smart agriculture 🧠🌿, specifically plant monitoring using ESP32 devices. The main goal behind succulent is to streamline the process of configuring various data parameters and provide a range of useful functions for data transformations. By leveraging succulent, you can set up your entire data collection endpoint within minutes, freeing you from the hassle of dealing with server-side scripts. 🚀🔧

• Free software: MIT license
• Documentation: https://succulent.readthedocs.io/en/latest
• Python versions: 3.8.x, 3.9.x, 3.10.x, 3.11.x, 3.12.x, 3.13.x
• Tested OS: Windows, Ubuntu, Fedora, Alpine, Arch, macOS. However, that does not mean it does not work on others

🔍 Detailed Insights

The current version of succulent comes packed with exciting features, including, but not limited to:

• Hassle-free generation of request URLs for seamless data collection 🌐
• Effortless data retrieval from POST requests 📥
• Versatile data storage options, such as CSV, JSON, SQLite, XML, and even images 🗂️📊🖼️
• Customisable boundaries for collected data, allowing you to set minimum and maximum thresholds ⚙️
• Web dashboard for exploring the data with charts, data quality and image gallery (optional) 📊

With succulent, the process of collecting, managing, and preprocessing data becomes a breeze, empowering you to focus on what truly matters—gaining valuable insights from your embedded devices, microcontrollers, and smartwatches. ⌚ So why waste precious time? ⏳ Dive into the world of succulent and unlock the true potential of your data! 💪📈

📦 Installation

pip

To install succulent with pip, use:

pip install succulent

Alpine Linux

To install succulent on Alpine Linux, use:

$ apk add py3-succulent

Arch Linux

To install succulent on Arch Linux, use an AUR helper:

$ yay -Syyu python-succulent

Fedora Linux

To install succulent on Fedora, use:

$ dnf install python3-succulent

🐳 Container

Create a docker-compose.yml file with the following content in the root directory:

version: '3.8'

services:
  app:
    image: codeberg.org/firefly-cpp/succulent:v6
    ports:
      - "8080:8080"
    volumes:
      - ./run.py:/succulent-app/run.py
      - ./configuration.yml:/succulent-app/configuration.yml
    environment:
      - GUNICORN_WORKERS=2

Next create a configuration.yml file in the root directory. Here's an example of a configuration file:

data:
  - name: 'temperature'
  - name: 'humidity'
  - name: 'light'
results:
  - enable: true
  - export: true
timestamp: true

More information regarding the configuration file and its settings can be found in the configuration section.

Then create a Python file named run.py with the following content in the root directory:

from succulent.api import SucculentAPI

api = SucculentAPI(config='configuration.yml', format='csv')

# Flask app instance, called by gunicorn
app = api.app

Once you have set up the configuration file and the Python file, build the Docker image with the following command:

docker compose build

Finally, run the Docker container with the following command:

docker compose up

🚀 Usage

Example

from succulent.api import SucculentAPI
api = SucculentAPI(host='0.0.0.0', port=8080, config='configuration.yml', format='csv')
api.start()

🔧 Configuration

Data collection

In the root directory, create a configuration.yml file and define the following:

data:
  - name: # Measure name
    min:  # Minimum value (optional)
    max:  # Maximum value (optional)

To collect images, create a configuration.yml file in the root directory and define the following:

data:
  - key: # Key in POST request
    allowed_extensions: ['jpg', 'png'] # Optional, see below

Only images are accepted. By default, the allowed extensions are jpg, jpeg, png, gif, bmp, webp, tif and tiff; use allowed_extensions to narrow or extend this list. For these image types, the content of the file is checked as well, so a file renamed to .jpg that is not a JPEG image is rejected. Other extensions you allow (e.g. heic) are checked by extension only. A file without an extension (e.g. from an ESP32-CAM) is accepted if its content is an allowed image, and the matching extension is added. Rejected uploads return a 400 response with the reason. Images are stored as <timestamp>_<filename>; an image with the same name in the same second gets a counter (e.g. _1) instead of overwriting the earlier one.

To store data collection timestamps, define the following setting in the configuration.yml file in the root directory:

timestamp: true # false by default

To access the URL for data collection, send a GET request (or navigate) to http://localhost:8080/measure.

To store data, send a POST request to /measure with the values in one of the following ways:

• query parameters, e.g. POST /measure?temperature=21.5&humidity=40 (works with every format),
• a JSON object body, e.g. {"temperature": 21.5, "humidity": 40} (works with every format),
• an XML body (with format='xml'), where each child of the root element is one value; the name of the root element is not checked:

  curl -X POST http://localhost:8080/measure -H "Content-Type: application/xml" -d '<data><temperature>21.5</temperature><humidity>40</humidity></data>'

A malformed XML body or a JSON body that is not an object (or not valid JSON) is rejected with a 400 response. All errors are returned as JSON with a message field, e.g. {"message": "Invalid JSON: the request body could not be parsed."}. With format='xml', column names must be valid XML element names: letters, digits, _, - or ., starting with a letter or _, without spaces, and not starting with xml.

To restrict access to the collected data, define the following setting in the configuration.yml file in the root directory:

password: 'password' # Password for data access

The password can be sent with any request (/measure, /data and /export) in one of the following ways:

• Authorization: Bearer password header (recommended),
• X-API-Key: password header,
• "password": "password" field in a JSON body or <password>password</password> element in an XML body (/measure only),
• ?password=password query parameter (supported for backward compatibility).

Prefer one of the headers, since URLs (including query parameters) are often written to server and proxy logs. For example:

curl -X POST http://localhost:8080/measure -H "Authorization: Bearer password" -H "Content-Type: application/json" -d '{"temperature": 21.5}'

Data access

To access data via the Succulent API, define the following setting in the configuration.yml file in the root directory:

results:
  - enable: true # false by default

To access the collected data, send a GET request (or navigate) to http://localhost:8080/data. To access password-protected data, send the password as described above (e.g. the Authorization: Bearer password header).

Data export

To export the data, enable the export option in the configuration file:

results:
  - export: true # false by default

To export the data, send a GET request (or navigate) to http://localhost:8080/export. To export password-protected data, send the password as described above. The data will be downloaded in the format specified in the configuration file.

Data exploration

succulent includes an optional web dashboard for monitoring the collected data in a browser. To enable it, define the following setting in the configuration.yml file in the root directory:

explore:
  - enable: true # false by default
  - refresh: 10  # auto-refresh interval in seconds (optional, 0 disables it)

Then navigate to http://localhost:8080/explore. The dashboard shows:

• the latest value of each column, with its position between min and max,
• data quality: the number of measurements, missing values, values outside min/max and the typical interval between measurements,
• a chart for each numeric column, with min and max as dashed lines (long ranges are averaged so the page stays fast),
• the latest measurements in a table,
• for the image format, a gallery of the uploaded images.

With timestamp: true, you can choose the time range (1 h, 24 h, 7 d, 30 d or all). If export is enabled, the data can be downloaded as CSV from the dashboard.

If a password is set, the dashboard asks for it once and keeps the session in a cookie, so the password never appears in the URL. Sessions last until the server restarts; to keep them across restarts, set a secret_key in the configuration. Scripts can read /explore/api/data with the Authorization: Bearer password or X-API-Key: password header instead.

The dashboard only reads the stored data; data collection works exactly as without it. It needs no internet access (no external scripts or fonts), so it also works on a local network, and values sent by devices are always shown as text.

📄 Cite us

Fister Jr., Iztok, and Tadej Lahovnik. Succulent. 0.4.0, doi:10.5281/zenodo.10402365.

🔑 License

This package is distributed under the MIT License. This license can be found online at http://www.opensource.org/licenses/MIT.

Disclaimer

This framework is provided as-is, and there are no guarantees that it fits your purposes or that it is bug-free. Use it at your own risk!

🫂 Contributors

Thanks goes to these wonderful people (emoji key):

Tadej Lahovnik 💻 🐛 🤔 📖 ✅

Ayan Das 💻 ⚠️

Iztok Fister Jr. 💻 🤔 🧑‍🏫

Oromion 🐛 📦

rhododendrom 🎨

Zala Lahovnik 📖

This project follows the all-contributors specification. Contributions of any kind welcome!