M3DPIS: Multimodal 3D Printing Intelligence System

Transform speech into printable 3D models using cutting-edge AI and contrastive learning.

🎯 Project Vision

Create an AI-powered system that enables natural language interaction with 3D printing workflows through multimodal contrastive learning. Users can generate and modify 3D printable models through conversational speech, with a professional, elegant web interface that handles everything from narration to printer-ready G-code.

🚀 What We've Built (Planning Phase)

1. Comprehensive Product Requirements Document (PRD)

Location: .taskmaster/docs/prd.txt

A detailed technical specification covering:

Phase 1: G-code dataset acquisition & preprocessing (ORNL Peregrine, open datasets)
Phase 2: MeshLib integration for G-code → 3D model reconstruction
Phase 3: VLM-based 3D model description (DeepSeek-VL, Qwen 2.5 VL)
Phase 4: Contrastive learning model training (CLIP-style for text/3D/gcode)
Phase 5: CLI tool with local speech-to-text (Voxtral, Whisper alternatives)

Key Innovation: Apply CLIP-style contrastive learning to create aligned embeddings across:

Natural language descriptions
3D mesh geometry
G-code toolpaths

2. Complete User Journey & UX Design

Location: .taskmaster/docs/ux-user-journey.md

A professional, elegant web application flow:

Speech Input → Model Generation → Visualization → Validation → Optimization → Slicing → Printing

10 Detailed Stages:

Welcome & Onboarding: First-time user experience with tutorials
Voice Input Interface: Pulsing mic button, real-time transcription, waveform viz
Model Generation: Animated progress with 4 stages
Model Visualization: PBR rendering, orbit controls, multiple view modes
Iterative Refinement: Voice commands for modifications, version history
Validation & Analysis: Automated printability checks, issue detection
Optimization: Smart optimization wizard (quality, material, supports, orientation)
Slicing Preparation: Layer-by-layer preview, toolpath visualization
Final Review & Export: Pre-flight checklist, direct printer integration
Print Monitoring: Live webcam, failure detection (stretch feature)

Tech Stack Recommendations:

Frontend: React + TypeScript + Vite
3D: Three.js + React Three Fiber
UI: Radix UI + Tailwind CSS
State: Zustand
Voice: Web Speech API / AssemblyAI

3. Development Workflow with Trae + Figma

Location: .taskmaster/docs/development-workflow-trae-figma.md

A cutting-edge design-to-development pipeline:

Figma → Trae Integration:

Design system in Figma (colors, typography, components)
Export design tokens as JSON
AI-powered code generation in Trae IDE
Automatic sync: Figma changes → React components

Key Workflow Features:

Voice commands in Trae (meta-experience!)
Screenshot → React component generation
Figma API integration for automation
Live Figma preview in split-screen
Design-code feedback loop

Example Workflow:

Designer creates component in Figma (4 hrs)
  ↓
Export & handoff to Trae
  ↓
Trae AI generates React component (5 min)
  ↓
Test & iterate
  ↓
Designer reviews → feedback
  ↓
Trae updates code instantly
  ↓
Ship to production

Total Time: 6 hours (vs. 2-3 days traditional)

4. Task Master Implementation Plan

Location: .taskmaster/tasks/tasks.json

30 AI-Generated Tasks organized by dependency:

✅ Task #1 (Next): Set up project repository and dev environment
20 High Priority Tasks: Core ML and frontend implementation
10 Medium Priority Tasks: Testing, docs, optimization

Dependency Metrics:

Tasks with no dependencies: 1
Tasks ready to work on: 1
Tasks blocked by dependencies: 29
Most depended-on task: #1 (12 dependents)

📁 Project Structure

mac3d/
├── .taskmaster/
│   ├── config.json                 # AI model configuration
│   ├── state.json                  # Project state
│   ├── tasks/
│   │   └── tasks.json              # 30 implementation tasks
│   ├── docs/
│   │   ├── prd.txt                 # Product Requirements Document
│   │   ├── ux-user-journey.md      # Complete UX design
│   │   └── development-workflow-trae-figma.md  # Dev workflow
│   ├── reports/                    # Task Master reports
│   └── templates/                  # PRD templates
├── src/                            # Source code (to be created)
├── public/                         # Static assets (to be created)
├── .gitignore
├── .env.example                    # API keys template
└── README.md                       # This file

🛠 Setup & Installation

Prerequisites

Node.js v18+ (for Task Master)
Python 3.10+ (for ML backend)
GPU (recommended): NVIDIA RTX 3060+ for inference, RTX 4090 for training

Task Master Setup (Completed ✅)

Task Master v0.32.2 is installed and initialized with:

Global npm package
Project configuration
MCP integrations for: Amp, Claude, Cursor, Gemini, Kiro, OpenCode, Kilo, Roo, VSCode, Windsurf, Zed
Shell aliases (if .bashrc exists)

Environment Variables

Create .env file with your API keys:

# Required
ANTHROPIC_API_KEY="sk-ant-api03-..."  # ✅ Configured

# Optional (for enhanced features)
OPENAI_API_KEY="sk-proj-..."          # ❌ Not set
PERPLEXITY_API_KEY="pplx-..."        # ❌ Not set
GOOGLE_API_KEY="..."                  # For Gemini VLM

📋 Getting Started

View Current Tasks

task-master list                     # View all 30 tasks
task-master next                     # See next task to work on
task-master show 1                   # View details of task #1

Start Working

# Mark task as in-progress
task-master set-status --id=1 --status=in-progress

# Expand task into subtasks for detailed breakdown
task-master expand --id=1 --num=5

# When complete
task-master set-status --id=1 --status=done

Analyze Complexity

# Analyze all tasks for complexity
task-master analyze-complexity

# View the complexity report
task-master complexity-report

# Expand complex tasks with research
task-master expand --id=2 --research --num=8

🎨 Design & Development Workflow

Phase 1: Design in Figma

Create Figma workspace: "Speech-to-3D-Print App"
Build design system (colors, typography, components)
Design key screens (landing, voice input, 3D viewer, optimization)
Create interactive prototype
Export design tokens

Phase 2: Develop in Trae IDE

Initialize React + TypeScript + Vite project
Import design tokens → Tailwind config

Use Trae AI to generate components from Figma:

"Build VoiceInputPanel component from Figma design [link]"

Implement 3D viewer with Three.js
Integrate backend ML API for model generation
Test and iterate

Phase 3: Backend ML Development

Acquire G-code datasets (ORNL Peregrine, Thingiverse)
Implement MeshLib G-code reconstruction
Generate 3D model descriptions with VLM
Train contrastive learning model (3D-CLIP-G)
Deploy inference API

Phase 4: Integration & Testing

Connect frontend to ML backend
Implement validation & optimization
Integrate slicer (CuraEngine WASM)
User testing & feedback
Polish and optimize

🏗 Technical Architecture

System Overview

┌─────────────────────────────────────────────────────────────┐
│                     Frontend (React + Three.js)              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Voice Input  │  │  3D Viewer   │  │ Optimization │      │
│  │   Module     │  │    Module    │  │    Wizard    │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────┬───────────────┬───────────────┬───────────────┘
              │               │               │
          Web Speech API   Three.js      CuraEngine WASM
              │               │               │
┌─────────────▼───────────────▼───────────────▼───────────────┐
│                    Backend API (FastAPI)                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  3D-CLIP-G   │  │   MeshLib    │  │ Validation   │      │
│  │    Model     │  │ Integration  │  │   Engine     │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└──────────────────────────────────────────────────────────────┘

Key Components

Frontend (Trae + Figma)

React + TypeScript for type safety
Three.js for 3D rendering
Tailwind CSS for styling
Zustand for state management
Web Speech API for voice input

ML Backend (Python)

PyTorch for deep learning
3D-CLIP-G contrastive model
DeepSeek-VL or Qwen 2.5 VL for descriptions
MeshLib (C++ with Python bindings) for G-code processing

Deployment

Frontend: Vercel / Netlify
Backend: AWS Lambda / Google Cloud Run (serverless)
Model: Hugging Face Inference Endpoints / Modal
Storage: AWS S3 for datasets and models

📊 Success Metrics

User Experience

Time to First Model: < 30 seconds from speech to preview
Semantic Accuracy: > 80% match between description and generated model
Print Success Rate: > 85% of generated models print successfully
User Satisfaction: > 4.5/5 stars

Technical Performance

Inference Speed: < 5 seconds per generation
Model Size: < 5GB (deployment-friendly)
Frontend Load Time: < 2 seconds (3D viewer ready)
Accessibility: WCAG 2.1 Level AA compliance

Development Efficiency (Trae + Figma)

Design-to-Code Time: < 1 hour per component
Code Generation: 80% AI-generated
Design-Implementation Match: 99%+ pixel-perfect

🗺 Roadmap

✅ Phase 0: Planning (COMPLETED)

Product Requirements Document
UX/UI Journey Design
Development Workflow (Trae + Figma)
Task Master implementation plan (30 tasks)

🚧 Phase 1: Foundation (Current)

Task #1: Set up project repository (NEXT)
Task #2: G-code dataset collection
Task #3: MeshLib integration
Task #5: VLM setup (DeepSeek-VL)

📅 Phase 2: ML Pipeline (Weeks 4-8)

Implement 3D-CLIP-G model architecture
Train contrastive learning model
Model evaluation and benchmarking

📅 Phase 3: Frontend (Weeks 7-10)

📅 Phase 4: Integration (Weeks 9-12)

Backend API integration
Validation & slicing
User testing
Polish & deployment

🔮 Future Enhancements

Multi-material support
AR preview (view model in real-world scale)
Collaborative editing
Community marketplace
Generative design optimization

🤝 Contributing

This is currently a solo project in the planning phase. Contributions will be welcome once the MVP is complete.

📄 License

MIT License - See LICENSE file for details

🙏 Acknowledgments

Task Master AI by @eyaltoledano - Project management
ORNL Peregrine Dataset - High-quality 3D printing data (2025)
MeshLib by MeshInspector - G-code processing
CLIP by OpenAI - Contrastive learning inspiration
Trae IDE - AI-powered development environment
Figma - Professional design tool

📞 Contact

Project Lead: [Your Name]
Email: [Your Email]
GitHub: [Your GitHub]
Twitter/X: [Your Handle]

Built with ❤️ and AI

Making 3D printing accessible to everyone, one voice command at a time.

Name		Name	Last commit message	Last commit date
.github		.github
.taskmaster		.taskmaster
backend		backend
.gitignore		.gitignore
DEVELOPMENT_ROADMAP.md		DEVELOPMENT_ROADMAP.md
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation