A benchmarking project for understanding where inference time actually goes in small and medium AI systems.
What it measures
- cold-start versus warm-path latency
- impact of batch size on response time
- preprocessing and postprocessing overhead
- caching effectiveness for repeated query patterns
Why it matters
Many AI demos report a single latency number without explaining what is included. I wanted a repeatable harness that breaks the path down and makes tradeoffs visible.
Main takeaway so far
In many pipelines, model execution is only one part of the latency story. Data movement, serialization, and orchestration cost often deserve equal attention.