A practical guide on probability calibration — Suppose you have a binary classifier and two observations; the model scores them as 0.6 and 0.99, respectively. Is there a higher chance that the sample with the 0.99 score belongs to the positive class? For some models, this is true, but for others it might not. This blog post…

A step-by-step guide to deploy a Data Science platform on AWS with open-source software — In our previous post, we configured Amazon ECR to push a Docker image to AWS and configured an S3 bucket to write the output of our Data Science experiments. In this final post, we’ll show you how to use Ploomber and Soopervisor to create grids of experiments that you can…

A step-by-step guide to deploy a Data Science platform on AWS with open-source software — In our previous post, we saw how to configure AWS Batch and tested our infrastructure by executing a task that spinned up a container, waited for 3 seconds and shut down. In this post, we’ll leverage the existing infrastructure, but this time, we’ll execute a more interesting example. We’ll ship…

A step-by-step guide to deploy a Data Science platform on AWS with open-source software — In this series of tutorials, we’ll show you how to deploy a Data Science platform with AWS and open-source software. By the end of the series, you’ll be able to submit computational jobs to AWS scalable infrastructure with a single command.

A few things to make you 10x more productive with Jupyter. — The Jupyter Notebook is a web-based interactive computing platform, and it is usually the first tool we learn about in data science. Most of us start our learning journeys in Jupyter notebooks. They are great for learning, practicing, and experimenting. There are several reasons why the Jupyter notebook is a…

nbsnapshot is an open-source package that benchmarks notebook’s outputs to detect issues automatically — If you want to keep up-to-date with my content. Follow me on Medium or Twitter. Thanks for reading! Motivation When analyzing data in a Jupyter notebook, I unconsciously memorize “rules of thumb” to determine if my results are correct. For example, I might print some summary statistics and become skeptical of…

A step-by-step guide to going from a messy notebook to a pipeline running in Kubernetes — Notebooks are great for rapid iterations and prototyping but quickly get messy. After working on a notebook, my code becomes difficult to manage and unsuitable for deployment. In production, code organization is essential for maintainability (it’s much easier to improve and debug organized code than a long, messy notebook).

Develop scalable pipelines on Google Cloud using open-source software — This tutorial will show how you can use Google Cloud and Ploomber to develop a scalable and production-ready pipeline. We’ll use Google BigQuery (data warehouse) and Cloud Storage to show how we can transform big datasets with ease using SQL, plot the results with Python, and store the results in…

A step-by-step guide for contributing to an open-source project — As Ploomber becomes the go-to tool for building pipelines from Jupyter, we’ve seen an increasing interest from our community to contribute to the project. We’re extremely honored to have such a vibrant community willing to go the extra mile, especially because many of our contributors are first-time open-source contributors. To…