Installation on RStudio with the `sparklyr` package might be a relatively easy option with a spark_install() helper function if you only want to try Spark in R but I used DataGriff's more manual approach and then created a venv environment in Python. This might be a little challenging and I recommend leveraging ChatGPT to help you troubleshoot your setup. To get Spark setup I recommend the Apache Spark docs and DataGriff's explainer. To begin you'll need to install Spark on your machine This 3GB dataset is not too big for most computers but will quickly lead to trouble once your R or Python starts making multiple transformed versions of this dataframe throughout your workflow. We'll be working with a moderately large kaggle dataset of the complete Spotify song rankings from 2017-21. Our job today will be to set up a local instance of the computational engine Apache Spark and use it to perform some simple data analysis. An alternative to local DB's are computational engines like DASK, Flink and Spark. This is a solid option but we won't be demonstrating this approach today.

One option for working with large datasets is to set up a local database like DuckDB or SQLite so that you can compute on the data on disk, not only in memory. It's quite possible to learn how to do big data science on a small scale. Today we'll start learning how to work with big data on your local machine and RAM, without an expensive Snowflake or Databricks account.

Not starting this learning curve until the coding test for a job application is a bad strategy because the learning curve can be a bit steep. So rather than putting off learning about how to work with big data it is highly recommended you become passionate about the subject, if you are not already, right now. Even relatively small 3-8GB datasets can be enough to bog most local machines down. This can be a bit of an existential crisis for data scientists when we are given coding exercises for roles that involve working with large datasets. Both the tidyverse and pandas packages alone tend to run into severe computer memory limitations once the datasets get too big. As a data scientist if you haven't yet faced the challenge of big data you no doubt will very soon.

Python with rstudio