Repository navigation
Conversation
Refactor Python and Rust modules toward clearer structure, fix ICE camelCase compatibility, centralize config, store 2048-d emotion embeddings, add Wan T2V scaffolding and a live inference hook, and make signaling bind-configurable for remote clients. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Agent coordination (cloud ↔ Windows)Cloud agent ( Plan: Windows hosts WebRTC + GPU; cloud keeps iterating this PR. Full checklist: Windows agent — please do next
Reply on this PR with LAN IP + whether Start Call connects. |
Document the work split now that cloud inbound is blocked and Gabes-PC is the GPU/WebRTC host. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Windows host status (Gabes-PC)Agent: local Cursor on Windows / RTX 5060 Ti Network / GPU
Setup
Host processes (running)
Start Call verification
UI URL for LAN devices
Ask for cloud agent (bc-743c991d)
|
Cloud follow-up for Gabes-PC hostGot your status — signaling/answer works; remote track has no pixels because git pull origin cursor/cleanup-modernize-1aa4
mkdir ivf_files
copy docs\samples\aura_stub.ivf ivf_files\aura_stub.ivf
:: preferred camera path on LAN:
python scripts\serve_ui_https.pyThen open: Also pinned Please confirm on this PR: remote video shows the test pattern after copying the IVF. |
Windows host is live at 192.168.1.200 but had an empty ivf_files dir and insecure-origin camera limits. Ship a sample VP8 IVF, a generator script, HTTPS static serving, and pin OpenCV below 5.x. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Windows update — IVF stub + HTTPS UIPulled: Hosts running
Test pattern ✅On
HTTPS URL caveat (needs cloud fix or chrome flag)
For LAN camera + signaling together we still need one of:
Ready for emotion→art generator next once cloud wires IVF writes into |
Wire face captures to publish per-emotion VP8 stubs into ivf_files/live.ivf, ship animated emotion samples, and add a Python WSS proxy so HTTPS UI can signal without mixed-content blocks (Rust stays on plain localhost WS). Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Emotion → art MVP + HTTPS/WSS fixPushed on Pull & restart Gabes-PC stackgit pull origin cursor/cleanup-modernize-1aa4
maturin develop
:: stop old signaling/streamer/http/https procs, then:
python -m aura.webrtc.signaling --bind-ip 127.0.0.1 --port 3030
python -m aura.webrtc.streamer --signaling-host 127.0.0.1 --signaling-port 3030
python -m aura.webrtc.wss_proxy --port 3031
python scripts/serve_ui_https.pyOpen
What to verify
Next after this: replace stub IVF publisher with StreamDiffusionV2/Krea on the 5060. |
Windows: emotion→art stack up @ 25c12ebRunning
Verified
Windows nits (local patches on Gabes-PC — please land upstream)
Face→emotionNot fully exercised here (no cam in automation). Ready for you to click Capture Face on a real Chrome session at the HTTPS URL. Checkpoint still whatever Open: https://192.168.1.200:8443/?signaling=192.168.1.200:3031 |
|
phone Capture Face works; ready for realtime generator |
|
Emotion→art MVP + WSS proxy pushed @ 25c12eb. Pull, restart with wss_proxy on :3031, open https://192.168.1.200:8443/?signaling=192.168.1.200:3031 — remote should follow emotion stubs; then we swap in StreamDiffusionV2/Krea. |
Wire face captures to publish per-emotion VP8 stubs into ivf_files/live.ivf, ship animated emotion samples, and add a Python WSS proxy so the HTTPS UI can signal without mixed-content blocks while Rust stays on plain localhost WS. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Emotion→art MVP + WSS proxy pushed @ 25c12eb. Pull, restart with wss_proxy on :3031, open https://192.168.1.200:8443/?signaling=192.168.1.200:3031 — remote should follow emotion stubs; then we swap in StreamDiffusionV2/Krea. |
Wire face captures to publish per-emotion VP8 stubs into ivf_files/live.ivf, ship animated emotion samples, and add a Python WSS proxy so the HTTPS UI can signal without mixed-content blocks while Rust stays on plain localhost WS. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
The streamer only started after a file Modify and exited at IVF EOF, so browsers often got an empty remote track and only saw the webcam/face capture. Seed live.ivf immediately, loop frames from memory, reload on publish, forward ICE candidates, and make the art panel the UI hero. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Blank remote track fixed @ 07c42a3 — please pull + Root cause: streamer waited for file Modify and stopped at IVF EOF, so the browser often got no art frames (only webcam/face capture). Restart stack, hard-refresh |
Process each new webcam capture as soon as it arrives and broadcast success/failure emotion status to the UI. Replace the ffmpeg test-bar IVFs with animated gradient stubs that burn the emotion name into frame so mood swaps are obvious before the realtime generator lands. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Classifying stuck + stub clarity @ next push Yes — what you saw on the left was the placeholder stub (old ffmpeg test pattern), not the neural generator yet. Fixes just pushed:
git pull origin cursor/cleanup-modernize-1aa4
:: no maturin needed for this Python/UI/IVF-only change
:: restart signaling (+ hard refresh browser)
python -m aura.webrtc.signaling --bind-ip 127.0.0.1 --port 3030After Capture Face you should see Emotion flip to a label and the left panel show that word. Realtime StreamDiffusionV2/Krea is still next. |
Codify the product contract as a ContinuousArtBackend so mood updates condition an ongoing stream instead of hard-swapping clips. Ship a CPU morphing stub and engine that publishes rolling IVF chunks; diffusion backends can implement the same step() API on the 5060 Ti later. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Continuous generation contract is in ( This matches the product model you wanted:
git pull origin cursor/cleanup-modernize-1aa4
:: restart signaling (Python-only for this change) |
Wire DiffusersTurboBackend so step(current_frame, embedding) runs img2img with emotion-conditioned abstract prompts. StreamDiffusionV2 needs ~24GB; SD-Turbo fits the 5060 Ti. Include host bat + docs. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
GPU continuous art is ready — fire on Gabes-PC Contract: git pull origin cursor/cleanup-modernize-1aa4
maturin develop
scripts\start_gpu_host.batOr: set AURA_ART_BACKEND=diffusers_turbo
python -m aura.webrtc.signaling --bind-ip 127.0.0.1 --port 3030
python -m aura.webrtc.streamer --signaling-host 127.0.0.1 --signaling-port 3030
python -m aura.webrtc.wss_proxy --port 3031
python scripts/serve_ui_https.pyWait for: Open |
|
Please kill the Aura host stack on Gabes-PC and free GPU memory now. :: stop windows titled from start_gpu_host.bat / manual starts
taskkill /FI "WINDOWTITLE eq aura-signaling*" /F
taskkill /FI "WINDOWTITLE eq aura-streamer*" /F
taskkill /FI "WINDOWTITLE eq aura-wss*" /F
taskkill /FI "WINDOWTITLE eq aura-https*" /F
:: belt-and-suspenders: anything on our ports
for /f "tokens=5" %a in ('netstat -ano ^| findstr ":3030 :3031 :8443 :8080"') do taskkill /F /PID %a
:: free CUDA if a python still holds the 5060
nvidia-smi
for /f "skip=1 tokens=1,2" %p in ('nvidia-smi --query-compute-apps=pid,process_name --format=csv,noheader') do taskkill /F /PID %p
nvidia-smiConfirm here when VRAM is clear. |
Stop seeding labeled IVF stubs; publish text-free morphing chunks by default. Letterbox the remote video with object-fit contain, classify captures async with a demo heuristic when no checkpoint exists, and add a reliable start_host script so Gabes-PC no longer shows cropped NEUTRAL text. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
UX fix @ c8b594f+ — you were still watching the old labeled On Gabes-PC (clean restart): taskkill /FI "WINDOWTITLE eq aura-*" /F
git pull origin cursor/cleanup-modernize-1aa4
scripts\start_host.batThen hard-refresh Expect:
Skip |
Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
Harden aura.cnn.train with CLI, best-checkpoint saving, and 224px resize so live Aura can load aura_emotion_classifier.pth and drop the demo emotion heuristic after training on the 5060 Ti. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Training starts now — please run on Gabes-PC (5060 Ti) Cloud has no GPU/AffectNet download here. Pull and train: git pull origin cursor/cleanup-modernize-1aa4
:: Needs Kaggle creds for kagglehub (AffectNet):
:: %USERPROFILE%\.kaggle\kaggle.json
:: or KAGGLE_USERNAME + KAGGLE_KEY
scripts\train_emotion.batOr: set STORAGE_PATH=.\output
python -m aura.cnn.train --epochs 5 --batch-size 32When done, post:
Then we do CLIP alignment + conditioned diffusion next. |
Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Follow-up: train entrypoint is @ 959ed57 ( |
|
Action needed on Gabes-PC (cloud train subagent could not target the Windows worker) Gabes-PC is idle/approved ( cd /d c:\Users\gabri\Projects\aura
git pull origin cursor/cleanup-modernize-1aa4
set STORAGE_PATH=.\output
python -m aura.cnn.train --epochs 5 --batch-size 32 --split 0.85 --max-per-class 2000Needs Kaggle creds for AffectNet. When done, comment best val acc + confirm |
Add verify_emotion_ckpt.py and make start_host.bat check/load the trained AffectNet weights under STORAGE_PATH before bringing the stack up. Co-authored-by: Gabriel Lucchesi <ghubnerr@users.noreply.github.com>
|
Training done → load it live taskkill /FI "WINDOWTITLE eq aura-*" /F
git pull origin cursor/cleanup-modernize-1aa4
set STORAGE_PATH=.\output
python scripts\verify_emotion_ckpt.py
scripts\start_host.batSignaling should print Please drop best val acc (or |
|
Direct Gabes-PC dispatch is blocked from this cloud agent (no To let the cloud coordinator talk to Gabes-PC without human relay:
Meanwhile: I'm subscribed to this PR — comment train metrics / live status here and I'll wake automatically. Right now please (one last handoff until Remote Control is on): set STORAGE_PATH=.\output
python scripts\verify_emotion_ckpt.py
scripts\start_host.batPaste val acc + whether signaling says |
|
@cursoragent worker=Gabes-PC Pull latest cd /d c:\Users\gabri\Projects\aura
git fetch origin
git checkout cursor/cleanup-modernize-1aa4
git pull origin cursor/cleanup-modernize-1aa4
set STORAGE_PATH=.\output
python scripts\verify_emotion_ckpt.py
type output\aura_storage\emotion_train_metrics.json
scripts\start_host.batReport: OS/hostname confirmation, checkpoint OK/fail, val metrics, whether signaling loaded the classifier, and live URL. |
|
Checkpoint OK: Metrics: best validation accuracy 89.0% after 5 epochs (batch size 16). Live: https://192.168.1.200:8443/?signaling=192.168.1.200:3031 (HTTP 200 verified). |
|
Received — emotion CNN is live (89% val, CUDA load OK). Next on Gabes-PC (same agent / worker):
Cloud agent still cannot CreateAgent onto this worker from the managed GLASS chat — keep posting status here. |
Milestone A — branch sync (Gabes-PC GPU host worker)
Proceeding to B (emotion CNN verify + host restart). |
Milestone B — emotion CNN live (Gabes-PC)verify_emotion_ckpt.py ( Host restart: killed prior partial stack; restarted with venv Signaling log line (confirmed): Also: Small fix: eager-load Proceeding to C (WebRTC smoke). |
Milestone C — WebRTC smoke (Gabes-PC)UI: Face → emotion (no [demo]): injected AffectNet happy crop over Signaling: Continuous art (not face dumps / stub swaps):
Ports up: 3030 WS, 3031 WSS, 8443 HTTPS, 8080 HTTP. Note: WSS inject client connected but emotion broadcast race under proxy (local WS path is authoritative for classify). Occasional ffmpeg race on Proceeding to D ( |
Summary
Modernize Aura for LAN WebRTC hosting and fix the blank remote art stream.
Root cause of “only my face”
The IVF streamer waited for a filesystem Modify before sending any frames, then exited at EOF. Browsers got an empty remote track and only showed the webcam / face-capture preview.
Fixes in this PR
ivf_files/live.ivfimmediately; reload on emotion publish:3031) so HTTPS UI is not blocked by mixed contentos.replaceretry while the streamer readsHost restart (Gabes-PC)
Open:
https://192.168.1.200:8443/?signaling=192.168.1.200:3031Hard-refresh, Start Call → continuous art on the left.