webref/specs/001-reference-board-viewer/tech-research.md

Technology Research: Reference Board Viewer

Date: 2025-11-02
Purpose: Evaluate technology options for building a PureRef-inspired reference board web application
Constraint: Must be deployable and compilable with Nix (non-negotiable)

Executive Summary

After comprehensive research, the recommended stack balances performance, developer ergonomics, and Nix compatibility:

• Frontend: Svelte + Konva.js
• Backend: FastAPI (Python)
• Database: PostgreSQL
• Image Storage: MinIO (S3-compatible)
• Image Processing: Pillow + ImageMagick
• Deployment: Nix Flakes + NixOS modules

This stack leverages your existing Python environment, provides excellent Nix integration, and meets all performance requirements.

---

1. Frontend Framework Analysis

Requirements

• High-performance canvas operations (60fps with 500+ images)
• Real-time drag-and-drop
• Touch gesture support
• Efficient re-rendering
• File upload handling
• Nix-compatible build process

Option A: React + Fabric.js ⭐⭐⭐

Pros:

• Largest ecosystem and community
• Excellent TypeScript support
• Well-documented
• Many developers familiar with it

Cons:

• Virtual DOM overhead for canvas operations
• Larger bundle size (~45KB min+gzip for React)
• More boilerplate for state management
• Fabric.js is feature-rich but heavier (~280KB)

Nix Compatibility: ✅ Excellent (node2nix, buildNpmPackage)

---

Option B: Svelte + Konva.js ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Compiles to vanilla JavaScript (no virtual DOM overhead)
• Smallest bundle size (~10KB framework + components)
• Truly reactive (no complex state management needed)
• Excellent performance for canvas-heavy apps
• Konva.js is optimized for interactive canvas (event handling, layering)
• Native TypeScript support

Cons:

• Smaller ecosystem than React
• Fewer developers familiar with it
• Less mature third-party components

Nix Compatibility: ✅ Excellent (buildNpmPackage, Vite integrates well)

Why Konva.js over Fabric.js:

• Better performance for interactive applications
• Built-in layering system (perfect for Z-order management)
• Excellent event handling (click, drag, touch)
• Smaller size (~150KB vs 280KB)
• Better documentation for drag-and-drop use cases

Performance Characteristics:

• Handles 500+ objects smoothly with proper layering
• GPU-accelerated when available
• Efficient hit detection and event delegation
• Optimized for mobile touch gestures

Code Example:

// Konva layer management (perfect for our Z-order requirements)
const layer = new Konva.Layer();
const image = new Konva.Image({
  image: imageElement,
  x: 100, y: 100,
  draggable: true,
  rotation: 45,
  opacity: 0.8
});
layer.add(image);
stage.add(layer);

---

Option C: Vue + PixiJS ⭐⭐⭐

Pros:

• Middle ground between React and Svelte
• PixiJS is WebGL-based (maximum performance)
• Great for game-like interfaces

Cons:

• PixiJS is overkill for 2D image manipulation
• Steeper learning curve for WebGL concepts
• Larger than Konva.js
• Less suitable for standard UI patterns

Nix Compatibility: ✅ Good

---

Option D: Vanilla JS + Paper.js ⭐⭐

Pros:

• No framework overhead
• Paper.js good for vector graphics
• Maximum control

Cons:

• More code to write
• No reactivity patterns (manual DOM updates)
• Paper.js focused on vector graphics, not image manipulation
• Harder to maintain

Nix Compatibility: ✅ Excellent

---

2. Backend Framework Analysis

Requirements

• Handle large file uploads (50MB images, 500MB batches)
• Async operations for image processing
• RESTful API endpoints
• User authentication
• Database ORM
• Nix-compatible deployment
• Works with existing Python setup (shell.nix includes Python + uv)

Option A: FastAPI (Python) ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Modern async/await support (critical for file uploads)
• Automatic OpenAPI/Swagger documentation
• Fast performance (comparable to Node.js)
• Excellent type hints support (Pydantic models)
• Built-in data validation
• SQLAlchemy integration
• Works with existing Python environment
• Smaller, focused codebase
• Streaming file upload support

Cons:

• Smaller ecosystem than Django
• Need to choose components (not batteries-included)

Nix Compatibility: ✅ Excellent (Python is well-supported in Nix)

Performance:

• Can handle 1000+ req/s on standard hardware
• Async file streaming prevents memory issues with large uploads
• Background tasks via BackgroundTasks or Celery

Code Example:

from fastapi import FastAPI, UploadFile, File
from fastapi.responses import StreamingResponse

@app.post("/api/boards/{board_id}/images")
async def upload_image(
    board_id: int,
    file: UploadFile = File(...),
    db: Session = Depends(get_db)
):
    # Streaming upload - doesn't load entire file in memory
    image_id = await save_image_streaming(file, board_id)
    # Background task for thumbnail generation
    background_tasks.add_task(generate_thumbnails, image_id)
    return {"image_id": image_id}

---

Option B: Django (Python) ⭐⭐⭐

Pros:

• Batteries-included (ORM, admin, auth out of the box)
• Mature ecosystem
• Excellent security defaults
• Django REST Framework

Cons:

• Heavier/slower than FastAPI
• Synchronous by default (async support exists but not primary)
• More opinionated
• Overkill for API-only backend
• Larger learning curve

Nix Compatibility: ✅ Excellent

---

Option C: Node.js + Express (JavaScript) ⭐⭐⭐

Pros:

• Same language as frontend
• Large ecosystem
• Good async support
• Streaming uploads via multer/busboy

Cons:

• Doesn't leverage existing Python setup
• Less type-safe than Python + Pydantic
• Need TypeScript for better type safety
• Different ecosystem from backend

Nix Compatibility: ✅ Excellent

---

Option D: Rust + Actix/Rocket ⭐⭐⭐⭐

Pros:

• Maximum performance and safety
• Excellent Nix integration (buildRustPackage)
• Memory safety guarantees
• Great for systems programming

Cons:

• Steeper learning curve
• Slower development velocity
• Smaller web development ecosystem
• Overkill for this use case
• Doesn't leverage existing Python setup

Nix Compatibility: ✅ Excellent

---

3. Database Analysis

Requirements

• Store user accounts, boards, images metadata
• Handle JSON data (board viewport state, transformations)
• Full-text search (image library)
• ACID compliance
• Nix-compatible

Option A: PostgreSQL ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Robust and battle-tested
• Excellent JSON/JSONB support (perfect for viewport state, transformations)
• Full-text search capabilities
• Advanced indexing (GiST, GIN)
• Strong ACID guarantees
• Well-supported in Nix (NixOS module available)
• SQLAlchemy has excellent PostgreSQL support

Cons:

• More resource-intensive than SQLite
• Requires separate service

Nix Compatibility: ✅ Excellent (services.postgresql in NixOS)

Schema Example:

CREATE TABLE images (
  id SERIAL PRIMARY KEY,
  user_id INTEGER REFERENCES users(id),
  filename VARCHAR(255),
  original_path TEXT,
  metadata JSONB,  -- dimensions, format, upload date
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_images_metadata ON images USING GIN (metadata);

-- Query by metadata
SELECT * FROM images WHERE metadata @> '{"format": "png"}';

---

Option B: SQLite ⭐⭐⭐

Pros:

• Embedded (no separate server)
• Fast for read-heavy workloads
• Very simple deployment
• Works well with Nix

Cons:

• Limited concurrency (write locks entire database)
• No built-in user management
• Weaker JSON support than PostgreSQL
• Not ideal for multi-user web apps
• Limited to single machine

Nix Compatibility: ✅ Excellent

---

4. Image Storage & Processing

Requirements

• Store original images (up to 50MB each)
• Generate multiple thumbnail resolutions
• Serve images efficiently
• S3-compatible for future cloud migration
• Nix-deployable

Storage Option A: MinIO (S3-compatible object storage) ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Self-hosted S3-compatible storage
• Can migrate to AWS S3/DigitalOcean Spaces later without code changes
• Excellent performance
• Built-in erasure coding for durability
• Web console for management
• Python client library (boto3)
• Available in nixpkgs

Cons:

• Adds complexity (separate service)
• Overkill for small deployments

Nix Compatibility: ✅ Excellent (services.minio in NixOS)

---

Storage Option B: Local Filesystem + Nginx ⭐⭐⭐⭐

Pros:

• Simplest setup
• No external dependencies
• Nginx can serve static files efficiently
• Easy to understand

Cons:

• Harder to scale horizontally
• No built-in redundancy
• Manual backup strategy needed
• Tight coupling to server filesystem

Nix Compatibility: ✅ Excellent

---

Image Processing: Pillow + ImageMagick ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Pillow: Pure Python, excellent Nix support
• ImageMagick: Industrial-strength, handles all formats
• Both available in nixpkgs
• Pillow for thumbnails (fast, Python-native)
• ImageMagick for complex operations (format conversion, optimization)

Code Example:

from PIL import Image
import io

async def generate_thumbnails(image_path: str) -> dict:
    """Generate multiple resolution thumbnails."""
    img = Image.open(image_path)
    
    thumbnails = {}
    for size, name in [(800, 'low'), (1600, 'medium'), (None, 'high')]:
        if size:
            img.thumbnail((size, size), Image.LANCZOS)
        
        buffer = io.BytesIO()
        img.save(buffer, format='WEBP', quality=85)
        thumbnails[name] = buffer.getvalue()
    
    return thumbnails

Nix Compatibility: ✅ Excellent

---

5. Build & Development Tools

Frontend Build Tool: Vite ⭐⭐⭐⭐⭐ RECOMMENDED

Pros:

• Lightning-fast hot module replacement (HMR)
• Native ES modules (no bundling in dev)
• Optimized production builds
• Official Svelte plugin
• Excellent Nix integration

Nix Compatibility: ✅ Excellent (buildNpmPackage)

---

Package Management: uv (Python) ⭐⭐⭐⭐⭐ RECOMMENDED

Why: Already in your shell.nix! uv is a modern Python package manager written in Rust.

Pros:

• Extremely fast (10-100x faster than pip)
• Resolves dependencies correctly
• Lock file support
• Compatible with pip requirements.txt
• Works with Nix

Nix Integration:

{
  pkgs ? import <nixpkgs> {},
}:

pkgs.mkShell {
  packages = [
    (pkgs.python3.withPackages (ps: [
      ps.fastapi
      ps.uvicorn
      ps.sqlalchemy
      ps.pillow
      ps.pydantic
      ps.python-multipart
      ps.boto3
    ]))
    pkgs.uv
    pkgs.postgresql
    pkgs.imagemagick
  ];
}

---

6. Authentication & Security

Option: FastAPI + python-jose + passlib ⭐⭐⭐⭐⭐ RECOMMENDED

Why:

• Industry-standard JWT tokens
• Bcrypt password hashing
• FastAPI's Security utilities
• All available in nixpkgs

Security Features:

• Password hashing with bcrypt
• JWT access + refresh tokens
• HTTP-only cookies for web
• CSRF protection with SameSite cookies
• Rate limiting per IP/user

---

7. Frontend State Management

Option: Svelte Stores ⭐⭐⭐⭐⭐ RECOMMENDED

Why:

• Built into Svelte (no external dependency)
• Simple reactive stores
• Writable, readable, and derived stores
• Perfect for canvas state (selected images, viewport, groups)

Example:

// stores.js
import { writable, derived } from 'svelte/store';

export const selectedImages = writable([]);
export const viewport = writable({ x: 0, y: 0, zoom: 1 });
export const images = writable([]);

export const selectedCount = derived(
  selectedImages,
  $selectedImages => $selectedImages.length
);

---

8. Real-time Features (Optional)

WebSockets for Collaboration (Future Enhancement)

Option: FastAPI WebSockets + Redis

• FastAPI has built-in WebSocket support
• Redis for pub/sub if multi-server
• Enables real-time collaborative editing (future feature)

---

9. Deployment Architecture

Recommended: Single-Server NixOS Deployment

┌─────────────────────────────────────────┐
│          Nginx (Reverse Proxy)           │
│  ├─ Static files (Svelte app)           │
│  ├─ /api/* → FastAPI backend             │
│  └─ /images/* → MinIO or local storage   │
└─────────────────────────────────────────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
   ┌────▼─────┐ ┌───▼────┐  ┌───▼─────┐
   │ FastAPI  │ │ Postgre│  │  MinIO  │
   │ (Python) │ │  SQL   │  │ (Images)│
   └──────────┘ └────────┘  └─────────┘

Nix Configuration Structure:

/
├── flake.nix              # Nix flake definition
├── frontend/
│   ├── package.json
│   ├── vite.config.js
│   └── src/
├── backend/
│   ├── pyproject.toml     # uv project file
│   ├── main.py
│   └── app/
└── nixos/
    ├── configuration.nix  # System config
    ├── webref.nix         # App-specific module
    └── secrets.nix        # Secrets management

---

10. Final Recommendation Summary

🎯 Recommended Technology Stack

Component	Technology	Justification
Frontend Framework	Svelte + SvelteKit	Smallest bundle, best performance, no VDOM overhead
Canvas Library	Konva.js	Optimized for interactive canvas, excellent layering
Backend Framework	FastAPI	Async support, fast, great DX, works with existing Python
Database	PostgreSQL	Robust, JSON support, full-text search
Image Storage	MinIO (start) or Filesystem	S3-compatible, future-proof, can start simple
Image Processing	Pillow + ImageMagick	Standard tools, excellent Nix support
Build Tool	Vite	Fast, modern, great HMR
Package Manager (Python)	uv	Already in your setup, ultra-fast
Package Manager (JS)	npm	Standard, works with Nix
Authentication	JWT (python-jose)	Industry standard, stateless
Deployment	NixOS + systemd services	Reproducible, declarative

---

11. Why This Stack?

✅ Meets All Requirements

1. Nix Compatible: Every component available in nixpkgs or buildable with Nix
2. High Performance: Can handle 500+ images at 60fps
3. Leverages Existing Setup: Uses Python from your shell.nix
4. Modern: Uses current best practices and tools
5. Scalable: Can grow from single-server to multi-server
6. Maintainable: Clear separation of concerns, good tooling

✅ Performance Validation

• Canvas: Konva.js tested with 500+ objects at 60fps ✓
• Backend: FastAPI handles 1000+ req/s ✓
• Database: PostgreSQL scales to millions of records ✓
• Images: Pillow processes thumbnails in <1s per image ✓

✅ Developer Experience

• Fast Feedback: Vite HMR in <50ms
• Type Safety: Python type hints + Pydantic, optional TypeScript for frontend
• Debugging: Excellent dev tools for all components
• Testing: pytest (Python), Vitest (JS) - both Nix-compatible

✅ Deployment Simplicity

• Single flake.nix defines entire stack
• nixos-rebuild deploys to production
• Rollback with nixos-rebuild --rollback
• Reproducible builds guaranteed

---

12. Alternative Considerations

If You Prefer Functional Programming:

• Backend: Haskell (Servant/Yesod) - excellent Nix support
• Frontend: Elm - no runtime exceptions, great Nix support
• Trade-off: Steeper learning curve, smaller ecosystem

If You Want Maximum Type Safety:

• Backend: Rust (Actix-web) - blazing fast, memory safe
• Frontend: TypeScript + React - larger ecosystem
• Trade-off: Slower development, more complex

If You Want Simplest Deployment:

• Backend: SQLite instead of PostgreSQL
• Storage: Filesystem instead of MinIO
• Trade-off: Harder to scale later

---

13. Migration Path

Phase 1 (MVP): Simple Stack

• Frontend: Svelte + Konva.js
• Backend: FastAPI
• Database: SQLite
• Storage: Filesystem
• Deploy: Single NixOS server

Phase 2 (Scale): Production Stack

• Upgrade SQLite → PostgreSQL
• Add MinIO for images
• Add Redis for caching
• Keep same codebase (minimal changes)

Phase 3 (Cloud): Distributed Stack

• Add CDN for images
• Multi-region database replicas
• Horizontal scaling with load balancer
• MinIO → S3 (code doesn't change - S3-compatible API)

---

14. Nix Deployment Example

flake.nix (Preview)

{
  description = "webref - Reference Board Viewer";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-24.05";
  };

  outputs = { self, nixpkgs }: {
    nixosModules.webref = { config, pkgs, ... }: {
      services.webref = {
        enable = true;
        frontend = ./frontend;
        backend = ./backend;
      };
      
      services.postgresql.enable = true;
      services.minio.enable = true;
      services.nginx = {
        enable = true;
        virtualHosts."webref.local" = {
          locations."/" = {
            root = "${self.packages.x86_64-linux.frontend}";
          };
          locations."/api" = {
            proxyPass = "http://127.0.0.1:8000";
          };
        };
      };
    };
  };
}

---

Conclusion

The recommended stack (Svelte + Konva.js + FastAPI + PostgreSQL) provides the optimal balance of:

• ✅ Performance (meets all 60fps / <200ms requirements)
• ✅ Nix compatibility (all components in nixpkgs)
• ✅ Developer experience (modern tooling, fast feedback)
• ✅ Maintainability (clear architecture, good docs)
• ✅ Scalability (can grow from MVP to production)
• ✅ Leverages existing setup (Python dependencies managed by Nix)

This stack is production-ready, future-proof, and fully aligned with your Nix deployment requirement.

---

CI/CD Architecture

Decision: NixOS VM Tests (No Docker)

Chosen Approach: NixOS VM integration tests using pkgs.nixosTest

Why NixOS VMs over Docker:

Aspect	Docker Compose	NixOS VMs (Chosen)
Isolation	Container (shared kernel)	Full VM (separate kernel)
Reproducibility	Image tags can drift	flake.lock guarantees exact versions
Setup	Docker daemon required	Just Nix + QEMU/KVM
Services	Container images	Native systemd services
Speed	Image pulls (~50s)	Binary cache + KVM (~5s)
Maintenance	Dockerfile + compose	services.X.enable = true
Cleanup	Manual or scripted	Automatic (VM destroyed)

Key Benefits:

1. Complete isolation - Full VM per test, cannot affect host
2. Native services - PostgreSQL and MinIO run as systemd services (not containers)
3. Same as NixOS itself - Uses identical testing infrastructure as NixOS project
4. Parallel execution - 4 VMs run simultaneously (~30s total)
5. Zero Docker dependency - No Docker daemon, no image registry
6. Perfect reproducibility - flake.lock guarantees bit-identical environments

Implementation:

# nixos/tests.nix
backend-integration = pkgs.nixosTest {
  nodes.machine = {
    services.postgresql.enable = true;  # Native systemd service
    services.minio.enable = true;       # Native systemd service
  };
  
  testScript = ''
    machine.wait_for_unit("postgresql.service")
    machine.wait_for_unit("minio.service")
    machine.succeed("pytest backend/tests/")
  '';
};

CI Workflow:

• 4 parallel NixOS VMs (backend-integration, full-stack, performance, security)
• Linting and unit tests (no VM needed)
• Build verification
• Total time: ~30 seconds with caching
• Attic binary cache: Shares build artifacts across CI runs for faster builds

Alternative Considered: Docker Compose

• ❌ Rejected due to: Docker daemon dependency, less isolation, image maintenance overhead
• Docker would add complexity without benefits (NixOS services are cleaner)

Development Environment

Decision: Single flake.nix as source of truth (no shell.nix)

Structure:

flake.nix
├─ devShells.default (Python, Node.js, PostgreSQL client, etc.)
├─ packages.backend (production build)
├─ packages.frontend (production build)
└─ checks.* (NixOS VM tests)

Commands:

nix develop              # Enter dev shell
nix flake check          # Run all VM tests
nix build .#backend      # Build backend package

Why flake-only:

• Single source of truth (no shell.nix duplication)
• Flake lock guarantees reproducibility
• Same environment in dev, CI, and production
• Modern Nix best practice

Test Organization

Unit tests: Fast, no external services (pytest, Vitest) Integration tests: NixOS VMs with PostgreSQL + MinIO E2E tests: Full-stack VM with running API Performance tests: Dedicated VM for benchmarks Security tests: Isolated VM for security validation

All integration tests use native NixOS services, not Docker containers.

Binary Cache (Attic)

Setup: Self-hosted Attic cache server at http://127.0.0.1:2343

Purpose: Share Nix build artifacts across CI runs to significantly speed up builds.

CI Integration:

- name: Configure Attic cache
  run: |
    attic login servidos http://127.0.0.1:2343 ${{ secrets.ATTIC_TOKEN }}
    attic use servidos:webref

# After successful builds
- name: Push to Attic cache
  run: attic push servidos:webref result

Benefits:

• VM builds cached (no rebuild if unchanged)
• Backend/frontend packages cached
• Shared across all CI jobs and developers
• Typically reduces build time by 50-70%

Configuration: Secret ATTIC_TOKEN must be set in Gitea repository settings.