More Performance Testing - NumPy versus PyArrow
A follow-up to the Lahman baseball benchmark using 18.2 GB of NYC Yellow Taxi CSV data. PyArrow was faster in all 48 file-level comparisons, but the real lesson is still about data shape, absolute time, and choosing the right tool for the work.
NumPy versus PyArrow - CSV versus Parquet
A CSV versus Parquet benchmark using the Lahman Baseball Database. Parquet reduced the dataset to 25.6% of its CSV size and was faster overall, but several tiny files were larger or slower after conversion.
NumPy versus PyArrow: File Size Matters
The second NumPy versus PyArrow benchmark groups Lahman Baseball Database CSV files into small, medium, and large buckets. PyArrow wins the medium files, but pandas default still wins several tiny files.
Performance Testing - NumPy versus PyArrow, One Year Later
A one-year follow-up to my NumPy versus PyArrow performance testing, using the Lahman baseball database to compare pandas default CSV reads with pandas read_csv using the PyArrow engine.
Performance Testing - NumPy versus PyArrow
This post, the first in a series on Fantasy Baseball, is going to start at two points. First, it's going to implement a scraper that will collect stats for an entire year of major league baseball. And then it's going to run some performance tests to see whether NumPy or PyArrow is faster at reading the CSV files generated. PyArrow is faster all the time...but it particularly shines when data sets get larger.