OPEN SOURCE

Pipeline & Tools

Open source annotation pipeline for egocentric household activity data

View on GitHub

How It Works

Raw GoPro Video

↓

Hand Pose

↓

Keypoints JSON

→

Arm Pose

↓

Arm JSON

→

Segmentation

↓

SAM2 Masks (PNG)

→

Depth

↓

Depth PNGs (16-bit)

→

Transcription

↓

Narration JSON

→

Quality Gate

↓

Quality Report

↓

Combined Full JSON

Clips scoring below 60 on the quality gate are moved to assets/videos/rejected/ — never deleted.

Tools

HAND TRACKING

HomeHands-Pose

21-point hand skeleton detection per frame. Detects left and right hand with confidence scores and pixel coordinates.

Input Raw .mp4 video

Output hand_pose.json per clip

Model MediaPipe Hand Landmarker

pip install mediapipe

ARM TRACKING

HomeHands-Arm

Wrist-to-elbow arm segment tracking with EMA smoothing and a temporal hold buffer to prevent flickering across frames.

Input Raw .mp4 video

Output arm_pose.json per clip

Model YOLOv8n-pose (Ultralytics)

pip install ultralytics

SEGMENTATION

HomeHands-Seg

Box-prompted SAM2 segmentation per detected hand: a tight mask around the hand itself, plus a wider mask around whatever active object it's holding.

Input Video + hand_pose.json

Output Binary mask PNGs per clip

Model SAM2.1 Hiera-Tiny (Meta AI)

pip install sam2

DEPTH

HomeHands-Depth

Per-frame metric depth in meters, fine-tuned for indoor scenes — not just a relative depth ordering.

Input Video + hand_pose.json

Output 16-bit depth PNGs per clip

Model Depth-Anything-V2-Metric-Indoor-Small

pip install transformers torch

AUDIO

HomeHands-Audio

Speech transcription from egocentric narration. Generates subtitles and timestamped narration JSON locally.

Input Raw .mp4 video with audio

Output .srt + narration.json + subtitled .mp4

Model OpenAI Whisper base

pip install openai-whisper

QUALITY CHECK

HomeHands-QC

Scores a clip 0-100 on resolution, hand detection rate, blur, brightness, and narration presence.

Input Raw .mp4 video

Output Quality report JSON (0-100 score)

Model MediaPipe Hand Landmarker + Whisper

pip install mediapipe openai-whisper opencv-python

QC GATE

HomeHands-Gate

Runs the quality check and quarantines clips scoring below 60 — moved to assets/videos/rejected/, never deleted, so they naturally drop out of the rest of the pipeline.

Input Raw .mp4 video

Output Quality report JSON, clip relocated if rejected

Threshold Score < 60

python pipeline/quality_gate.py assets/videos/

FULL PIPELINE

HomeHands-Pipeline

Master script that runs all modules automatically on every video in a folder. Produces one combined annotation JSON per clip.

Input Folder of .mp4 videos

Output Full annotation JSON per clip

Model All of the above

Run python pipeline/run_pipeline.py

python pipeline/run_pipeline.py

Quick Start

# 1. Clone the repository
git clone https://github.com/aneessaheba/Egocentric_Homes
cd Egocentric_Homes

# 2. Install dependencies
pip install -r requirements.txt
brew install ffmpeg

# 3. Add your videos
cp your_videos/*.mp4 assets/videos/

# 4. Run full pipeline
python pipeline/run_pipeline.py

# Output per video:
# Hand pose JSON      → assets/processed/hand_pose/
# Arm pose JSON       → assets/processed/arm_pose/
# Segmentation masks  → assets/processed/masks/
# Depth maps (16-bit) → assets/processed/depth/
# Narration JSON      → assets/processed/narrations/
# Quality reports     → assets/processed/quality/
#   (rejected clips moved to assets/videos/rejected/)
# Combined JSON       → assets/processed/annotations/

Models Used

Model	Task	Made by	Size	License
MediaPipe Hands	Hand tracking	Google	8 MB	Apache 2.0
SAM2.1 Hiera-Tiny	Segmentation	Meta AI	156 MB	Apache 2.0
YOLOv8n-pose	Arm tracking	Ultralytics	6.5 MB	AGPL-3.0
Depth-Anything-V2-Metric-Indoor-Small	Depth estimation	HKU / TikTok	95 MB	Apache 2.0
Whisper base	Transcription	OpenAI	145 MB	MIT
ffmpeg	Audio extraction	OSS	—	LGPL
OpenCV	Video processing	OSS	—	Apache 2.0

Output Format

{
  "clip_id": "HH_001",
  "filename": "WashingCup.mp4",
  "activity_category": "[from clip_metadata.csv, or null]",
  "activity_label": "[from clip_metadata.csv, or null]",
  "participant_id": "[from clip_metadata.csv, or null]",
  "home_id": "[from clip_metadata.csv, or null]",
  "room": "[from clip_metadata.csv, or null]",
  "camera_model": "[from clip_metadata.csv, or null]",
  "duration_sec": 76.109,
  "total_frames": 2281,
  "fps": 29.97,
  "resolution": "3840x2160",
  "quality_control": {
    "score": 100,
    "verdict": "✅ GOOD",
    "issues": []
  },
  "hand_detection_rate": 0.9527,
  "narrations": [
    {
      "id": 1,
      "start": 4.2,
      "end": 7.8,
      "text": "[transcribed narration text]"
    }
  ],
  "frames": [
    {
      "frame_id": 600,
      "timestamp_sec": 20.02,
      "hands": {
        "hands_found": 2,
        "wrists": [
          {
            "label": "left",
            "confidence": 0.9821,
            "px": 1620, "py": 1435,
            "keypoints": { "WRIST": { "px": 1620, "py": 1435, "x": 0.422, "y": 0.664, "z": 0.0 } /* + 20 more named landmarks */ }
          }
        ]
      },
      "arm": {
        "left_wrist": { "px": 1564, "py": 1451 },
        "left_elbow": { "px": 1324, "py": 2158 },
        "right_wrist": { "px": 2060, "py": 1294 },
        "right_elbow": null
      },
      "segmentation": {
        "method": "SAM2.1-Hiera-Tiny",
        "prompted_by": "hand_bbox",
        "masks": [
          { "category": "hand", "label": "left", "pixel_count": 107266, "coverage_pct": 1.2932, "mask_path": "assets/processed/masks/WashingCup/frame_000600_hand_left.png" },
          { "category": "active_object", "label": "near_left", "pixel_count": 162690, "coverage_pct": 1.9614, "mask_path": "assets/processed/masks/WashingCup/frame_000600_object_left.png" }
        ]
      },
      "depth": {
        "method": "Depth-Anything-V2-Metric-Indoor-Small",
        "depth_path": "assets/processed/depth/WashingCup/frame_000600.png",
        "depth_format": "png16",
        "depth_unit": "mm",
        "depth_scale": 1000,
        "min_depth_m": 0.7469, "max_depth_m": 2.6691, "avg_depth_m": 1.5871
      },
      "narration_id": null
    }
  ]
}

Every frame always has the same key set — hands, arm, segmentation, depth, narration_id — defaulting to null/empty when a given module wasn't run for that clip, never omitted.