diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 94b2f9a..185f87b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -8,10 +8,6 @@ on: - 'Folderly/Frontend/**' - '.github/workflows/ci.yml' pull_request: - paths: - - 'Folderly/Backend/**' - - 'Folderly/Frontend/**' - - '.github/workflows/ci.yml' # A second push to the same branch makes the first one irrelevant. concurrency: diff --git a/Folderly/compose.yaml b/Folderly/compose.yaml new file mode 100644 index 0000000..ccedaa2 --- /dev/null +++ b/Folderly/compose.yaml @@ -0,0 +1,15 @@ +# The whole application locally, the way the host runs it. `include` resolves +# the API's paths and .env against Backend/, so nothing is repeated here. +include: + - Backend/docker-compose.yml + +services: + # No FOLDERLY_DOMAIN, so Caddy stays on plain HTTP. + web: + build: Frontend + ports: + - "127.0.0.1:8080:80" + volumes: + - ../deploy/Caddyfile:/etc/caddy/Caddyfile:ro + depends_on: + - app diff --git a/README.md b/README.md new file mode 100644 index 0000000..eb6d3a9 --- /dev/null +++ b/README.md @@ -0,0 +1,309 @@ +

Folderly

+ +

+ Self-hosted file storage with folder sharing - a small, honest Google Drive you run yourself. +

+ +

+ + CI/CD + + Python + Flask + Vue + Docker + License +

+ +--- + +## Overview + +Folderly gives a handful of people - a family, a team, a couple of machines - a private place to keep +files, and a way to hand a folder to someone else without handing over the account. It is a REST API +with a Vue single-page frontend, packaged as two Docker images small enough for a Raspberry Pi +and portable enough for anything else. + +Three decisions shape the whole project: + +- **Clients never see the storage layout.** Every path a client sends is relative to a _space_ - your + own files, where `/` is your root, or a folder someone shared with you, where `/` is that folder. + User ids and real paths never appear in the API. +- **A share is a role, not a password.** `read`, `add` or `edit`, granted per person per folder, + inherited downwards, and cumulative: a share further down a path can widen access, never narrow it. +- **Registration is open, access is not.** A new account can log in and manage itself but cannot + touch a single file until it redeems an activation code that the instance owner generates by hand. + +> [!NOTE] +> **Project status.** Complete and running. 124 backend tests and 89 frontend tests run in CI, +> alongside the linters and a smoke test against the freshly built image, before anything is +> published, a systemd timer on the host pulls what passed. + +## Features + +**Files and folders** + +- Create, rename, delete and browse folders, a listing carries per-entry sizes, extensions and + recursive counts, and a whole subtree can be summed at once. +- Upload, download, rename and delete files, up to 100 MB each. + +**Sharing** + +- Recipients reach a shared folder through an unguessable token, so their access survives the owner + renaming or moving it. +- `add` deliberately cannot overwrite: replacing a file destroys the old one as surely as deleting it. + +**Accounts and security** + +- JWT access and refresh tokens with a self-pruning revocation list, each bound to its account, so no + token survives a password change or outlives the account itself. +- Changing a password or deleting an account requires the current password, not just a valid token. +- Activation codes generated from the CLI, stored only as hashes and spendable once. + +**Operations** + +- Uploads refused with `507` once the disk passes a configurable fill level, with the incoming file + counted as already written. +- One error shape - `{code, status, message}` - for every failure in the API. + +## Tech stack + +| Layer | Technology | +| -------- | --------------------------------------------------------------------- | +| API | Python 3.14, Flask 3.1, flask-smorest, marshmallow, webargs | +| Auth | Flask-JWT-Extended, passlib (`pbkdf2_sha256`) | +| Database | SQLite via SQLAlchemy and Flask-SQLAlchemy | +| Serving | gunicorn with `gthread` workers | +| API docs | OpenAPI 3 and Swagger UI, served by the application | +| Frontend | Vue 3.5, Vite 8, TypeScript, Pinia, Vue Router, axios, Tailwind CSS 4 | +| Tests | pytest (unit and integration), Vitest, an HTTP smoke test | +| Tooling | ruff, oxlint, ESLint, Prettier, Docker, GitHub Actions, Caddy | + +## Repository layout + +``` +Folderly/ compose.yaml - both halves locally, as the host runs them +├── Backend/ Flask REST API +│ ├── models/ SQLAlchemy models +│ ├── resources/ One blueprint per endpoint group +│ ├── utils/ Access control, activation, disk, validation +│ └── tests/ pytest suite plus the HTTP smoke test +└── Frontend/ Vue 3 single-page app - stock create-vue layout under src/ +deploy/ Production compose file, Caddyfile, systemd units, backup script +docs/ Deployment guide and the screenshots below +scripts/ ci-local.ps1 - every CI gate, run locally +.github/workflows/ CI/CD pipeline +``` + +## Quick start + +### From the published image + +The image is published for `amd64` and `arm64` under one tag, so the same command works on a server +and on a Raspberry Pi. It is the API alone - for the interface as well, run the pair from source +below, or the published pair through the deployment guide. + +```bash +mkdir -p folderly/files folderly/db +# The image runs as UID 10001 and these are bind mounts, so ownership comes from +# the host - without this the database cannot be created and the container exits. +sudo chown -R 10001:10001 folderly/files folderly/db + +docker run -d --name folderly -p 5000:5000 \ + -e DISK_PATH=/data \ + -e DB_DIR=/db \ + -e JWT_SECRET_KEY="$(openssl rand -hex 32)" \ + -v "$PWD/folderly/files:/data" \ + -v "$PWD/folderly/db:/db" \ + ghcr.io/ultipro/folderly-backend:latest +``` + +> [!TIP] +> On Windows run this from PowerShell, with `-v "${PWD}\folderly\files:/data"` +> and without the `chown` - Docker Desktop keeps the files in its own virtual machine, where +> host ownership does not apply. Do not use Git Bash: it rewrites absolute paths, so the volumes +> land inside that machine and the data silently never reaches the directories you created. + +Registration is open, but a new account can do nothing until it redeems a code - so generate one +before you sign up, or you will be locked out of your own instance: + +```bash +docker exec folderly flask --app "app:init()" codes new --label "me" +``` + +The code is printed once, the database keeps only its hash. Now open , +call `POST /register`, then `POST /login`, paste the `access_token` into **Authorize**, and redeem the +code with `POST /user/activate`. + +### From source + +```bash +git clone https://github.com/UltiPro/Folderly.git +cd Folderly/Folderly # Backend/ and Frontend/ side by side +cp Backend/.env.example Backend/.env # fill in JWT_SECRET_KEY +docker compose up --build # http://localhost:8080 +``` + +That is both images, the pair production runs, from the same two Dockerfiles, so the whole +application answers on one origin with `/api` proxied to the API beside it. For the API alone, with +no SPA and no Caddy, run `docker compose up --build` from `Backend/` instead. + +The two halves below run it without Docker, which is what work on the frontend wants. + +**The API without Docker**, in a virtualenv. `DISK_PATH` and `DB_DIR` in `.env` are _container_ +paths, so point them at local directories first - real environment variables take precedence over +the file - and create them before starting: the application checks both rather than creating them, +since a missing directory almost always means a wrong setting. + +```powershell +pip install -r requirements.txt +mkdir folderly-data, folderly-db +$env:DISK_PATH = "$PWD\folderly-data" +$env:DB_DIR = "$PWD\folderly-db" +flask --app "app:init()" run +``` + +**The SPA with hot reload**, which needs the API already running. It expects it at +`http://127.0.0.1:5000`, copy `.env.example` to `.env.local` and set `VITE_API_BASE_URL` to point +it elsewhere, and whichever origin serves it must appear in the backend's `CORS_ORIGINS`. + +```bash +cd Folderly/Frontend +npm install +npm run dev # http://localhost:5173 +``` + +The same activation code is needed here, from a compose stack it comes from the directory the +stack was started in: + +```bash +docker compose exec app flask --app "app:init()" codes new --label "me" +``` + +## Configuration + +Read from the environment at startup. The first three are required - the application refuses to start +without them rather than failing later on someone's first upload. + +| Variable | Default | Purpose | +| -------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `DISK_PATH` | - | Root of the file storage. Must exist. | +| `DB_DIR` | - | Directory holding `folderly.db`. Must exist, and **must not be inside `DISK_PATH`** - that is the tree the API serves files from, where one path-handling bug could expose the database. | +| `JWT_SECRET_KEY` | - | Token signing key. Changing it logs everyone out. | +| `CORS_ORIGINS` | `*` | Comma-separated origins the frontend may call from. | +| `DISK_USAGE_LIMIT_PERCENT` | `85` | Fill level at which uploads start being refused with `507`. | +| `WEB_CONCURRENCY` | `2` | gunicorn worker processes. | +| `GUNICORN_THREADS` | `4` | Threads per worker, concurrency is workers x threads. | +| `GUNICORN_TIMEOUT` | `120` | Seconds before a stuck worker is killed. Slow uploads are not affected. | + +## API + +Swagger UI at `/swagger-ui`, the raw specification at `/openapi.json`. Reads take their arguments +from the query string, writes take a JSON body, uploads are `multipart/form-data`. + +| Group | Endpoints | +| ------- | ---------------------------------------------------------------- | +| Auth | `POST /register`, `POST /login`, `POST /refresh`, `POST /logout` | +| Account | `GET/PUT/DELETE /user`, `POST /user/activate` | +| Folders | `GET/POST/PUT/DELETE /folder`, `GET /folder/list` | +| Files | `POST/GET/PUT/DELETE /file`, `GET /file/download` | +| Access | `GET /folder/shared`, `GET/POST/DELETE /folder/share` | +| Disk | `GET /disk` | +| Health | `GET /hello` | + +## Development + +```bash +cd Folderly/Backend +pip install -r requirements-dev.txt + +pytest +ruff check . + +# Against a server that is already running, with a code from `codes new`: +python tests/smoke.py http://localhost:5000 K7M2-PX94-TB3H-QR8F +``` + +Every test builds its own application on a temporary `DISK_PATH` and `DB_DIR` and creates its accounts +through the API, so the suite needs no setup and touches no real data. `tests/smoke.py` is not +collected by pytest: it walks one account through a **running** server, which is the only way to +exercise gunicorn, the volumes and the image itself. + +From `Folderly/Frontend`: + +| Command | What it does | +| -------------------- | ---------------------------------------------------------- | +| `npm run dev` | Dev server with hot reload, on . | +| `npm run build` | Type-check, then build to `dist/`. | +| `npm run test:unit` | Vitest in watch mode (`npx vitest run` for a single pass). | +| `npm run lint` | oxlint + ESLint, both with `--fix`. | +| `npm run format` | Prettier over the frontend. | +| `npm run type-check` | `vue-tsc` on its own. | + +`npx vitest run` is 89 tests: the pure logic in `utils/`, the upload queue's concurrency limit and +weighted progress, and five specs covering traps that are invisible in the source. + +`scripts/ci-local.ps1` runs every gate CI runs - both linters and test suites, the type-check, and +the smoke test against a container it builds and throws away on port 5050 - and prints one +pass/fail table at the end. PowerShell only, and the smoke step needs Docker. + +## Deployment + +A push to `main` runs three gates in parallel - ruff and pytest, the frontend's type-check, unit +tests and linters, and a smoke test over HTTP against the freshly built image - and only then builds +for real: two images, `folderly-backend` and `folderly-web` (Caddy with the built SPA inside), each +on native `amd64` and `arm64` runners, published to GHCR as one manifest per image under `latest` +and `sha-`. Hosts follow `latest`, pinning a `sha-` tag is the rollback. + +**Nothing in the workflow names this account.** It publishes to +`ghcr.io//folderly-backend` and `-web`, so a fork builds its own pair under its +own login with no edit anywhere - or you can skip building and run the images this repository +publishes. + +`VITE_API_BASE_URL` is a **build** argument, not a runtime one - Vite inlines it into the +JavaScript, so the published image is fixed to the relative `/api`. That is what lets one image +serve any domain, and why the browser never makes a cross-origin request in production. + +**[Putting it on your own host](docs/deployment.md)** - Docker, the two files to fetch, the five +values to fill in, the update timer, backups, rollback, and the split-host arrangements. + +## Preview + +**The way in.** Registration is open to anyone, what it gets you is an account that can do nothing +until it redeems a code. + +![Sign in](docs/preview/01-sign-in.png) + +**Your own space.** Folders first, then files, each with its size and what it is. + +![The file browser](docs/preview/02-my-files.png) + +**Three uploads at a time**, each with its own progress, because the backend serves eight threads in +total. Navigating away mid-batch does not redirect the rest. + +![The upload queue](docs/preview/03-upload-queue.png) + +**A share is a role.** The owner invites by email and picks what that person may do, nobody else can +pass the access on. + +![The share dialog](docs/preview/04-share.png) + +**Previews never hand the file to the browser as a document.** Images and PDFs become object URLs, +text is read into a `
`, so an uploaded `.html` is shown as source rather than run.
+
+![The preview dialog](docs/preview/05-preview.png)
+
+**The same folder as the person it was shared with sees it:** the name alone - never where it sits in
+the owner's tree - a badge for their role, and a toolbar with nothing on it they are not allowed to
+do.
+
+![A shared folder, seen by its recipient](docs/preview/07-shared-with-me.png)
+
+**The theme is chosen, not sniffed**, and the browser chrome follows it.
+
+![The same space in the dark theme](docs/preview/06-my-files-dark.png)
+
+## License
+
+MIT - see [LICENSE](LICENSE). Run it, change it, redistribute it, build on it commercially if
+you like, keep the copyright notice, and note that it comes with no warranty of any kind.
diff --git a/deploy/docker-compose.prod.yml b/deploy/docker-compose.prod.yml
index 2d5392c..c27c06d 100644
--- a/deploy/docker-compose.prod.yml
+++ b/deploy/docker-compose.prod.yml
@@ -1,4 +1,4 @@
-# Lives at /srv/folderly next to its .env, see the README.
+# Lives at /srv/folderly next to its .env, see docs/deployment.md.
 
 services:
   app:
diff --git a/docs/deployment.md b/docs/deployment.md
new file mode 100644
index 0000000..fe37939
--- /dev/null
+++ b/docs/deployment.md
@@ -0,0 +1,188 @@
+# Deploying Folderly
+
+Everything needed to put Folderly on a host of your own. What the project is, and how to run it
+locally, is in the [README](../README.md).
+
+## One host, one domain
+
+You need a 64-bit Linux host, a domain pointing at it, and ports 80 and 443 reachable from the
+internet. A Raspberry Pi 4 is enough. Nothing has to be built and nothing has to be edited in the
+repository - every value that differs between installations lives in one `.env` file.
+
+**1. Docker**, if the host does not have it. Distribution packages are usually too old:
+
+```bash
+curl -fsSL https://get.docker.com | sudo sh
+sudo usermod -aG docker $USER      # log out and back in before the group applies
+```
+
+**2. Directories for the data.** `files/` is `DISK_PATH`, `db/` holds the SQLite database:
+
+```bash
+sudo mkdir -p /srv/folderly/files /srv/folderly/db
+sudo chown -R 10001:10001 /srv/folderly/files /srv/folderly/db
+```
+
+The `chown` is required, not tidiness. The API image runs as UID 10001 and these are bind mounts,
+so ownership comes from the host, without it `db.create_all()` cannot create the database at
+startup, gunicorn exits and the container restart-loops before serving a request. The UID is pinned
+in the Dockerfile precisely so this command is the same on every machine.
+
+**3. The two files that describe the stack:**
+
+```bash
+cd /srv/folderly
+BASE=https://raw.githubusercontent.com/UltiPro/Folderly/main/deploy
+sudo curl -fsSLO $BASE/docker-compose.prod.yml
+sudo curl -fsSLO $BASE/Caddyfile
+```
+
+`/srv/folderly` is a convention rather than a requirement, and the only host path involved: the
+container paths are fixed inside the images. Four files name it, though - the compose file's three
+bind mounts, both service units and the backup script - so putting the data elsewhere means
+changing it in all of them.
+
+**4. Configuration.** Copy the template and fill in five values:
+
+```bash
+sudo curl -fsSL $BASE/.env.example -o .env
+openssl rand -hex 32          # the JWT secret, generated on this host
+sudo nano .env
+sudo chmod 600 .env
+```
+
+| Variable                               | What to put there                                                                            |
+| -------------------------------------- | -------------------------------------------------------------------------------------------- |
+| `FOLDERLY_IMAGE`, `FOLDERLY_WEB_IMAGE` | `ghcr.io//...` for your own fork's builds, or `ultipro` to run this repository's |
+| `FOLDERLY_DOMAIN`                      | the domain Caddy answers on and obtains a certificate for                                    |
+| `CORS_ORIGINS`                         | `https://`                                                                 |
+| `JWT_SECRET_KEY`                       | the `openssl` output, never a key reused from development                                    |
+
+The remaining five have working defaults. Leave `FOLDERLY_DOMAIN` empty to serve plain HTTP on
+port 80 instead - then `CORS_ORIGINS` takes the `http://` form.
+
+**5. Start it:**
+
+```bash
+sudo docker compose -f docker-compose.prod.yml up -d
+sudo docker compose -f docker-compose.prod.yml logs -f web     # "certificate obtained successfully"
+sudo docker compose -f docker-compose.prod.yml ps              # app healthy, web running
+```
+
+Caddy obtains the certificate itself, and port 80 has to be open for that as well - that is how
+Let's Encrypt verifies the domain. Then, from anywhere:
+
+```
+https:///api/hello        ->  "Hello World!"
+https:///                 ->  the application
+https:///api/swagger-ui   ->  the API docs
+```
+
+**6. The first account.** Registration is open, but a new account is inactive and can do nothing
+until it redeems a code, so an instance needs one before anyone can use it:
+
+```bash
+sudo docker compose -f docker-compose.prod.yml exec app \
+  flask --app "app:init()" codes new --label "me"
+```
+
+That command prints the code, and it is the only time you see it - the database keeps nothing but
+its hash. Register in the browser, sign in, and enter the code on the activation screen.
+
+**7. Automatic updates**, which is what closes the CI/CD loop: a systemd timer pulls whatever CI
+last published and restarts only if the digest changed. Every connection is outgoing, so the host
+needs no inbound access for GitHub to reach it, and no credential of this network ever leaves it:
+
+```bash
+sudo curl -fsSL $BASE/folderly-update.service -o /etc/systemd/system/folderly-update.service
+sudo curl -fsSL $BASE/folderly-update.timer   -o /etc/systemd/system/folderly-update.timer
+sudo systemctl daemon-reload
+sudo systemctl enable --now folderly-update.timer
+systemctl list-timers folderly-update.timer
+```
+
+Five minutes is the default. To look less often on a host that does other things too, override the
+interval rather than editing the unit, so re-installing the file later does not undo it:
+
+```bash
+sudo mkdir -p /etc/systemd/system/folderly-update.timer.d
+sudo tee /etc/systemd/system/folderly-update.timer.d/override.conf >/dev/null <` from a build that worked, then
+`up -d` again. One tag covers both images, so a tested pair stays together. Go back to `latest`
+afterwards, or the timer holds the old version forever.
+
+**When it does not come up**, in order of likelihood: the `chown` in step 2 was skipped (the
+container restart-loops on "unable to open database file"), port 80 does not reach the host from
+the internet (no certificate, ACME errors in the web log), or `CORS_ORIGINS` has the wrong scheme
+(the SPA loads but every request fails). There is no `Flask-Migrate`, so a model change means
+deleting the database file and creating the accounts again, uploads on disk survive it.
+
+Two things worth knowing about the limits. The 85% fill threshold measures the whole filesystem,
+system included, so uploads are refused before a full disk can take down the OS, Docker and SQLite
+at once. And uploads are capped at 100 MB (`MAX_UPLOAD_BYTES`), which Caddy passes through untouched
+because it sets no body limit of its own.