Instackup

A Python library that standardizes connections to BigQuery, Cloud Storage, Google Sheets, S3, Redshift and SQL databases. Created in 2019 and brought up to date in 2026 with Claude.

Status In progress
Categories Data engineering
Published on
Source code

Instackup is a Python library, published on PyPI, that standardizes and simplifies connections to the cloud services and databases used in data work: BigQuery, Google Cloud Storage, Google Sheets, Amazon S3, Redshift, PostgreSQL, MySQL and SQLite. The same code runs on a laptop, during development, and in production, in scheduled pipelines.

It's installed with the extras of the services in use:

pip install 'instackup[gcp,postgresql]'
from instackup import BigQueryTool, PostgreSQLTool

with PostgreSQLTool() as pg:
    orders = pg.query("SELECT * FROM orders WHERE created_at >= %s", params=("2026-01-01",))

BigQueryTool().upload(orders, dataset="analytics", table="orders", if_exists="replace")

The original project

I started Instackup in December 2019, when I was starting out in data engineering. The idea was to stop rewriting the same connection code in every script, and to have the same code run in development, on my computer, and in production, in scheduled pipelines. So I put that code in a library, with one class per service (BigQueryTool, S3Tool, RedShiftTool...) and the credentials of all of them in a single YAML file.

The first version went to PyPI in January 2020. Up to version 0.3.0, in February 2021, there were 18 versions, which added Google Cloud Storage, Google Sheets, the SQL databases and more than one account per service. After that, the library stayed as it was for more than five years.

Five years later

Five years is a long time in the Python ecosystem, and it showed:

  • The library required Python 3.6 and pinned old versions of its dependencies (like grpcio==1.34.0 and gspread==3.6.0), which made it impossible to install next to anything modern.
  • Its dependencies moved on: pandas 3 removed DataFrame.to_gbq, used to upload data to BigQuery; the BigQuery Data Transfer, gspread and Cloud Storage libraries changed their APIs; and oauth2client had been deprecated since 2017.
  • Credentials only came from a file on disk, with keys and passwords in plain text. That's not how production should work today, with secret managers and the cloud platforms' own identities (IAM roles, service accounts).
  • It created log files inside its own installation folder, which fails on read-only systems, like containers.
  • And its tests covered a single module, so there was no way of knowing what still worked.

Version 0.4.0

The revision kept the library's purpose and the names of its modules, classes and methods, so most existing scripts keep working, and rebuilt everything around them:

  • Current Python and dependencies. Python 3.11 to 3.15, pandas 3 and the current Google and AWS libraries. Each service became an optional extra: the basic install no longer brings every cloud SDK (it used to be around 100 packages), only pandas and PyYAML.
  • Credentials ready for production. They can come from environment variables, AWS Secrets Manager, GCP Secret Manager or a file that references environment variables instead of holding the secrets. When none is configured, the cloud tools use the platform's own identity (Google's Application Default Credentials and the AWS default credential chain).
  • About 30 bugs fixed. Some methods always failed, like renaming and deleting files on S3 or uploading and downloading folders on Cloud Storage, and three security issues were closed: SQL injection through a table name, shell injection in the Heroku tool and folder downloads that could write files outside the target folder.
  • Tests and automation. 420 tests, with fakes and moto in place of the cloud services, plus integration tests against real PostgreSQL and MySQL databases. GitHub Actions runs them on Python 3.11 to 3.15, on Linux, macOS and Windows, and publishing a release on GitHub sends the package to PyPI, with no stored token (Trusted Publishing).
  • Types and documentation. Type hints everywhere, checked with mypy, documentation for every module and an architecture document with the reasons behind each decision and the next steps.

Every change, including the few that break compatibility, is in the version log.

Revising it with Claude

The whole revision was made with Claude Code, Anthropic's coding agent, using the Claude Opus 5.5 model, which signs the commits as co-author. It's a single pull request, with 63 files, around 10,700 lines added and 3,600 removed, opened, merged and published to PyPI on the same day. It's the kind of work that, in a personal project, tends to stay on the "someday" list forever. What made the difference:

  • Old code, read in minutes. Code I wrote five years ago is almost someone else's code. Claude reads the whole library, its documentation and its history in minutes, which made it possible to change everything while keeping old scripts working.
  • Five years of changes at once. Instead of reading the changelogs of a dozen libraries, I had help that already knew what changed in pandas, in the Google and AWS SDKs and in gspread, and how to migrate each call.
  • The work that usually gets postponed. Tests, types, documentation and CI on three operating systems are what make a library trustworthy, and also what gets put off the most in a personal project. With Claude, they stop being the expensive part, and covering every method with tests is what brought out the bugs nobody had noticed.
  • Problems on other systems. The first CI run failed on macOS and Windows: the MySQL driver used an SSL constant that some Python builds don't have, pandas writes CSV files with different line endings on Windows, and Windows finds the user's folder through a different variable. Everything was fixed from the CI logs, in a second commit.
  • Decisions written down. Each choice, like Python 3.11 as the minimum version, the optional extras and compatibility with old scripts, is documented with its reasons, which makes reviewing a pull request this size much easier.

Claude doesn't replace knowing what the library is for: what to keep, what to break and what to leave for later are still my decisions. But it turned a revision that had waited five years into a single pull request, and now I can use Instackup in new projects again, with 420 tests behind it.

« Back to the projects