# ONA Labs - Model Comparison Experiment: YOLO vs Faster R-CNN

**Date:** January 27, 2026  
**Status:** Experiment Complete - YOLO Recommended for Real-Time Use  
**Author:** ONA Labs Team

---

## Executive Summary

This document captures findings from the first object detection model comparison experiment in ONA Labs. We evaluated **YOLOv8** and **Faster R-CNN (ResNet50-FPN)** for real-time camera streaming detection.

**Conclusion:** YOLOv8 is significantly better suited for real-time detection. Faster R-CNN's initialization overhead and inference speed make it impractical for live camera streaming, though it may have use cases in batch processing or accuracy-critical offline analysis.

---

## Models Tested

### 1. YOLOv8 Nano (yolov8n.pt)
- **Architecture:** Single-shot detector, anchor-free
- **Backbone:** CSPDarknet with C2f modules
- **Framework:** Ultralytics
- **Size:** ~6.2 MB

### 2. Faster R-CNN (ResNet50-FPN)
- **Architecture:** Two-stage detector (Region Proposal Network + classifier)
- **Backbone:** ResNet-50 with Feature Pyramid Network
- **Framework:** PyTorch torchvision
- **Size:** ~160 MB

---

## Hardware Configuration

| Component | Specification |
|-----------|---------------|
| Device | Apple Silicon Mac (M-series) |
| Compute Backend | MPS (Metal Performance Shaders) |
| Camera | USB Webcam @ 30 FPS target |
| Memory | Shared GPU/CPU memory |

---

## Performance Metrics

### Initialization Time

| Model | Cold Start | Warmup | Total Init |
|-------|-----------|--------|------------|
| YOLOv8n | ~1.5s | ~0.5s | **~2s** |
| Faster R-CNN | ~3s | ~90-97s | **~95-100s** |

**Finding:** Faster R-CNN requires ~50x longer initialization due to:
1. Larger model weights loading
2. FPN (Feature Pyramid Network) initialization
3. Required warmup inference to JIT compile MPS kernels

### Inference Speed

| Model | Inference Time | FPS (theoretical) | FPS (observed) |
|-------|---------------|-------------------|----------------|
| YOLOv8n | 10-20ms | 50-100 FPS | 30-60 FPS |
| Faster R-CNN | 200-500ms | 2-5 FPS | 2-4 FPS |

**Finding:** YOLOv8 is 10-50x faster per inference.

### Memory Usage

| Model | GPU Memory | Peak Usage |
|-------|-----------|------------|
| YOLOv8n | ~200 MB | ~400 MB |
| Faster R-CNN | ~800 MB | ~1.2 GB |

---

## Technical Issues Encountered

### Issue 1: Faster R-CNN Warmup Blocking Event Loop

**Problem:** The Faster R-CNN warmup (first inference) blocks for ~90 seconds, freezing the entire FastAPI async event loop.

**Symptoms:**
- Dashboard UI frozen during model load
- WebSocket connections timing out
- Camera frames not being processed

**Solution Implemented:**
```python
# Run warmup in separate thread to not block async loop
await asyncio.to_thread(self._warmup_inference, warmup_image)
```

**Result:** Partial fix - warmup runs in background but detector unusable until complete.

---

### Issue 2: Race Condition - Detector Used Before Initialization

**Problem:** Multiple callers requesting FRCNN simultaneously would each create new instances, discarding previous loading progress.

**Symptoms:**
- "Starting Faster R-CNN initialization" appearing multiple times
- "Detector not initialized" errors during inference
- 90-second warmup restarting repeatedly

**Root Cause:**
```python
# Old code - checking existence, not initialization status
if self.frcnn_detector is None:  # WRONG
    self.frcnn_detector = FasterRCNNWrapper(...)

# Fixed code - checking initialization flag
if self.frcnn_detector and self.frcnn_detector._initialized:  # CORRECT
    print("Already initialized")
```

**Solution Implemented:**
1. Added `asyncio.Lock()` to prevent concurrent initialization
2. Check `_initialized` flag, not just object existence
3. Pre-load all models at server startup (final solution)

---

### Issue 3: Model Selection Not Persisting

**Problem:** User selects Faster R-CNN, clicks Start, but YOLO runs instead.

**Root Cause:** 
- Client sends mode change but not model selection
- Server defaults to YOLO if no model specified

**Solution:**
```javascript
// Client sends both mode AND model
ws.send(JSON.stringify({ 
    camera_id: cameraId, 
    mode: currentMode, 
    model: currentModel  // Added this
}));
```

---

### Issue 4: Session Tracking Not Capturing Model Switches

**Problem:** Session results showed all activity under one model even when user switched between YOLO and FRCNN.

**Solution:** Implemented segment-based session tracking:
```python
@dataclass
class ModeSegment:
    mode: str
    model: str
    start_time: float
    end_time: Optional[float]
    frame_count: int
    detection_count: int
    inference_samples: List[float]
```

---

## Architecture Comparison

### YOLOv8 (Recommended)
```
Input Image
    ↓
CSPDarknet Backbone (feature extraction)
    ↓
Neck (PANet feature fusion)
    ↓
Detection Head (direct bbox + class prediction)
    ↓
Output (all detections in single forward pass)
```

**Advantages:**
- Single forward pass = fast
- Anchor-free = simpler, more accurate small objects
- Optimized for edge devices

### Faster R-CNN
```
Input Image
    ↓
ResNet-50 Backbone
    ↓
Feature Pyramid Network (multi-scale features)
    ↓
Region Proposal Network (generates ~2000 proposals)
    ↓
ROI Pooling + Classification (per proposal)
    ↓
Output (filtered detections)
```

**Disadvantages for Real-Time:**
- Two-stage = inherently slower
- RPN generates thousands of proposals
- Each proposal requires separate classification

---

## Code Architecture Changes

### Before (On-Demand Loading)
```
User clicks Start → Load model → Wait 90s → Start streaming
```
- Long waits before inference can begin
- Race conditions with concurrent requests
- Timeout errors

### After (Pre-Loading at Startup)
```
Server starts → Load ALL models → Accept connections → Instant switching
```
- Models ready before server accepts traffic
- Switching between models is instant
- No race conditions

**Implementation:**
```python
@asynccontextmanager
async def lifespan(app: FastAPI):
    print("Loading models...")
    
    # YOLO loads in ~2s
    camera_state.yolo_detector = YOLOWrapper(...)
    await camera_state.yolo_detector.initialize()
    
    # FRCNN loads in ~95s
    camera_state.frcnn_detector = FasterRCNNWrapper(...)
    await camera_state.frcnn_detector.initialize()
    
    print("All models ready - starting server")
    yield
```

---

## Recommendations

### Recommendation 1: Use YOLOv8 for Real-Time Detection

- 10-20ms inference fits within 33ms frame budget (30 FPS)
- Fast initialization (~2s)
- Low memory footprint

### Recommendation 2: Consider Faster R-CNN for Offline Analysis

- Higher accuracy on small objects (in some benchmarks)
- Better for offline analysis where latency is not the primary constraint
- Not suitable for real-time streaming

### Recommendation 3: Pre-load Models at Startup for Controlled Experiments

- Eliminates loading delays during experiments
- Enables instant A/B switching
- More predictable operator experience

---

## Future Work

1. **Add more models for comparison:**
   - YOLOv8s/m/l (larger YOLO variants)
   - YOLO-NAS
   - RT-DETR (real-time transformer)
   - EfficientDet

2. **Implement proper benchmarking:**
   - mAP (mean Average Precision) calculation
   - Per-class accuracy comparison
   - Latency percentiles (p50, p95, p99)

3. **Optimize Faster R-CNN (if needed):**
   - TensorRT optimization
   - ONNX conversion
   - Quantization (INT8)

4. **Add comparison visualization:**
   - Side-by-side detection overlay
   - Confidence score comparison
   - Detection count over time graphs

---

## Files Modified

| File | Changes |
|------|---------|
| `dashboard/app.py` | Pre-load models at startup, simplified load_detector(), session tracking |
| `dashboard/templates/camera.html` | Model selection sync, removed preload REST calls |
| `experiments/object_detection/detectors/fasterrcnn_wrapper.py` | Added async warmup, _initialized flag |
| `experiments/object_detection/detectors/base.py` | Added _initialized property |

---

## Appendix: Faster R-CNN Warmup Analysis

The 90+ second warmup is caused by MPS (Metal Performance Shaders) JIT compilation:

```
First inference triggers:
1. Metal shader compilation (~30s)
2. Memory allocation optimization (~20s)  
3. Kernel fusion and optimization (~40s)
```

This is a one-time cost per model load, but makes Faster R-CNN impractical for:
- Quick experiments
- Development iteration
- Real-time applications

---

## Conclusion

**YOLOv8 wins for real-time detection** by a significant margin. The 50x speed advantage and 50x faster initialization make it the clear choice for ONA Labs' camera streaming use case.

Faster R-CNN implementation is preserved for:
- Future batch processing features
- Accuracy comparison experiments
- Educational/research purposes

**Next Step:** Focus on YOLOv8 optimizations and add more YOLO variants for comparison.