PyArrow

High-performance analytics, made Pythonic and powerful.

Performance Testing - NumPy versus PyArrow, One Year Later

A one-year follow-up to my NumPy versus PyArrow performance testing, using the Lahman baseball database to compare pandas default CSV reads with pandas read_csv using the PyArrow engine.

June 16, 2026

Performance Testing - NumPy versus PyArrow

This post, the first in a series on Fantasy Baseball, is going to start at two points. First, it's going to implement a scraper that will collect stats for an entire year of major league baseball. And then it's going to run some performance tests to see whether NumPy or PyArrow is faster at reading the CSV files generated. PyArrow is faster all the time...but it particularly shines when data sets get larger.

June 16, 2025

More about PyArrow

PyArrow is a powerful cross-language development platform for working with large datasets and in-memory analytics. It implements the Apache Arrow columnar memory format in Python, enabling efficient data processing and interchange between different big data systems. PyArrow provides robust tools for working with columnar data, offering high-performance reading and writing of various file formats including Parquet and CSV, seamless integration with pandas DataFrames, and native support for nested data structures. With features like zero-copy reads, memory-efficient data sharing, and SIMD vectorized operations, PyArrow is essential for data scientists and engineers working with large-scale data processing pipelines, especially in distributed computing environments.