Run heavy jobs on AWS from your laptop
Work on your laptop as usual. When a script needs more than your laptop has, saturn run sends it to a machine with up to 32 cores and 244 GB of memory, streams the output back to your terminal, and records what it cost. The same script runs in both places, in R, Julia or Python.
- 1
Get an account. Ask an admin to add you. You get an email from AWS: set a password and register MFA.
- 2
Install once with one command, then each session:
$ saturn login $ saturn connect
- 3
Put your script in your folder in the shared workspace,
~/aws/workspace/<you>/, and try it locally. - 4
Run it on AWS. The path is relative to the workspace.
$ saturn run <you>/project/analysis.R
How it fits together
After saturn connect, your laptop sees the same files and data a job does, under the same names. That's why a script you try locally runs unchanged on AWS.
Your laptop
- Shared workspace~/aws/workspace
- Satellite data (read-only)~/aws/satellite
A job on AWS (Mumbai)
- Shared workspace/workspace
- Satellite data/vsis3/…
- Parquet datasetss3://…
Your code never uses these paths directly. It reads them from environment variables (see below), which point to the right place on each side.
Setup (once)
On a Mac or Linux, in a terminal:
$ curl -fsSL https://raw.githubusercontent.com/xKDR/saturn-v/main/install.sh | shThen open a new terminal. The installer:
- installs the tools saturn uses: the AWS command line, rclone (for the mounts) and AWS’s Session Manager plugin (for the MariaDB tunnel; see the appendix). On a Mac through Homebrew (install that first); on Linux into
~/.local, without sudo; - puts saturn in
~/.saturn-vand on your PATH; - adds two lines to your shell’s startup file, marked
# saturn-v.
It’s safe to run again. To read it first, replace | sh with | less.
- Linux: the mounts need FUSE (
sudo apt install fuse3); most desktops have it, and the installer says if it’s missing. - Updates: when a new version is out, saturn asks you to run
saturn updatebefore anything else (disconnect,stopanddocsstill work). It reconnects you with the new version if you were connected.
Each session
$ saturn login # opens your browser; lasts 7 days $ saturn connect
saturn connect does four things:
- mounts the shared workspace at
~/aws/workspace, and the satellite data (read-only) at~/aws/satellite; - opens a tunnel to the MariaDB database on
127.0.0.1:3306(see the appendix), and reopens it in the background if it drops; - creates your own folder,
~/aws/workspace/<your user name>; - writes the job's environment variables to
~/.config/saturn-v/env, which new terminals load;
Everything runs on your sign-in, the mounts included: AWS keys renew in the background for the whole week, and nothing secret is stored on your laptop. When the week is up, saturn asks you to run saturn login again.
saturn status shows what's connected. saturn disconnect undoes it all. In the terminal you're already in, eval "$(saturn connect)" also sets the variables there.
The workspace
Everyone shares one workspace, so keep to this layout:
| Folder | For |
|---|---|
<your user name>/ | Your own work. MY_WORKSPACE points here, on your laptop and in your jobs. |
shared/<project>/ | Work several people use. Agree on the folder name with them. |
db/, .julia/, .R/ | Used by saturn itself. Leave them alone. |
If two people save the same file, the last save wins. Anything a job writes appears on your laptop within a minute or two. Anything else at the top level shows up in the Monday summary as “not in anyone’s folder”.
The same code on your laptop and on AWS
These variables are set in every job, and on your laptop after saturn connect. Build paths and connections from them, never from fixed paths.
| Variable | In a job | On your laptop |
|---|---|---|
WORKSPACE | /workspace | ~/aws/workspace |
MY_WORKSPACE | /workspace/<you> | ~/aws/workspace/<you> |
SATELLITE | /vsis3/<bucket>, read straight from S3 | ~/aws/satellite |
Reading a satellite file
f <- file.path(Sys.getenv("SATELLITE"), "ntl/sl/rad/<file>.tif")
r <- terra::rast(f)
out <- file.path(Sys.getenv("MY_WORKSPACE"), "project", "results.rds")using Rasters, ArchGDAL r = Raster(joinpath(ENV["SATELLITE"], "ntl", "sl", "rad", "<file>.tif"); lazy=true) out = joinpath(ENV["MY_WORKSPACE"], "project", "results.csv")
import os, rasterio r = rasterio.open(os.path.join(os.environ["SATELLITE"], "ntl/sl/rad/<file>.tif")) out = os.path.join(os.environ["MY_WORKSPACE"], "project", "results.parquet")
Reading satellite files on your laptop downloads them, at about $0.11 per GB. Try things on a few files or a small area locally, and do the heavy reading in a job, where it's free.
Apps started from the Dock don't read your terminal's setup. In R, run readRenviron("~/.config/saturn-v/env") first.
Running a job
$ saturn run <you>/project/main.jl $ saturn run <you>/project/analysis.R $ saturn run <you>/project/analysis.py $ saturn run --size large <you>/project/main.jl # bigger machine $ saturn run -e START=2014-01 <you>/project/main.jl # extra variable for the job $ SPOT=1 saturn run <you>/project/main.jl # ~70% cheaper, may be interrupted
The job starts in about a minute and its output streams to your terminal. Press Ctrl-C to stop watching: the job keeps running, and your laptop can sleep or disconnect.
--size | Machine | About per hour |
|---|---|---|
small | 1 core, 4 GB | $0.06 |
medium (default) | 4 cores, 16 GB | $0.25 |
large | 8 cores, 60 GB | $0.62 |
xlarge | 16 cores, 120 GB | $1.24 |
max | 32 cores, 244 GB | $2.50 |
You pay per second, only while a job runs, with a one-minute minimum. A quick check job costs about half a cent.
The examples folder has a first script in each language. Copy one into your folder and run it.
Packages
Why a project lists its packages
On your laptop, a package you installed once is there for every script. A job starts on a fresh machine that has none of them. It needs a list of what to install, and it can only read that list from files in your project folder.
The list also pins versions. Packages change: a new release can rename a function, change a default or shift a result. With versions pinned, the job uses exactly what you tested on your laptop, and so does a colleague running your code, or you in a year when a referee asks for a rerun.
And each project gets its own list. Upgrading a package for one project doesn't quietly change another.
Each language keeps two kinds of information:
| What you use | Exact versions of everything | The job runs | |
|---|---|---|---|
| Julia | Project.toml | Manifest.toml | Pkg.instantiate() |
| R | (renv reads your code) | renv.lock | renv::restore() |
| Python | pyproject.toml, requirements.txt or the script header | uv.lock | uv run |
"Everything" includes the packages your packages depend on, which is why the version files are long. You never edit them by hand; the commands below write them.
Do you need renv? Not if your script only uses packages already in the R image: terra, sf, GDAL, the tidyverse, data.table, DBI and more. Add renv when you need anything else, or exact versions.
Start a project (once)
setwd("~/aws/workspace/<you>/project")
renv::init()This writes three things into the folder: renv.lock (the list), a renv/ folder, and an .Rprofile that switches renv on whenever R starts in this folder. R prints Project '…' loaded when it does.
The installed packages themselves are kept on your laptop, not in the shared workspace. saturn connect sets this up; in RStudio, run readRenviron("~/.config/saturn-v/env") before renv::init().
Add a package
install.packages("fixest")
renv::snapshot() # write it to renv.lockinstall.packages() installs it on your laptop only. renv::snapshot() is what puts it in the list the job reads. Forget it, and the job fails with there is no package called 'fixest'.
Check before running a job
renv::status()
It lists packages your code uses that aren't recorded, and recorded packages that aren't installed. "No issues found" means the job will get what you have.
Other things you'll do
renv::install("lrberge/fixest") # from GitHub instead of CRAN; then renv::snapshot()
renv::update(); renv::snapshot() # newer versions of everything
renv::remove("fixest"); renv::snapshot()
renv::restore() # install what renv.lock lists, e.g. in a colleague's projectR versions
renv.lock also records your R version, and the job uses the same one (4.5 or 4.6). If yours isn't available, saturn run says so: upgrade R, then renv::update() and renv::snapshot().
Julia always needs a project for a job: the image has Julia and nothing else.
Start a project (once)
$ cd ~/aws/workspace/<you>/project $ julia --project=. julia> ]add AWS Parquet2 DataFrames CSV
--project=. makes this folder the active project, so add writes Project.toml (the packages you use) and Manifest.toml (exact versions of everything). Keep both in the folder: the job installs from the Manifest.
Run with the project
$ julia --project=. main.jl # on your laptop $ saturn run <you>/project/main.jl # on AWS; finds the nearest Project.toml itself
A Project.toml in a folder above the script works too, so several scripts can share one project.
The usual mistake
Adding a package after starting plain julia (without --project) puts it in your personal environment, not the project's. It works on your laptop but the job fails with Package AWS not found in current path. Check which project is active, and what it has:
julia> ]status # first line shows the active Project.toml julia> ]activate . # switch to this folder's project if it's the wrong one
Other things you'll do
julia> ]update # newer versions of everything julia> ]rm CSV julia> ]add DataFrames@1.7 # a specific version julia> ]instantiate # install what Manifest.toml lists, e.g. in a colleague's project
Packages are cached in the workspace, so only a job's first run installs them. Julia uses every core the job has.
Python jobs run with uv, which installs your packages (and the Python version you ask for; 3.13 otherwise) in a few seconds at the start of every job. Install it on your laptop with brew install uv.
A single script: list packages in its header
$ uv add --script analysis.py pandas boto3 pyarrowThat writes this at the top of the script, which is all the job needs:
# /// script # dependencies = ["pandas", "boto3", "pyarrow"] # ///
Run it with uv run analysis.py on your laptop and saturn run <you>/analysis.py on AWS. This is the simplest choice for code in the workspace: uv keeps the installed packages in its own cache, not in your folder.
A project with several scripts
$ cd ~/aws/workspace/<you>/project $ uv init --bare $ uv add pandas boto3 pyarrow
This writes pyproject.toml (what you use) and uv.lock (exact versions); keep both. uv also makes a .venv folder here, which is slow on the shared workspace, so prefer the script header when you can. A requirements.txt next to the script works as well.
rasterio, geopandas and pyogrio come with their own GDAL, so nothing else is needed for the satellite data.
Data
This is where to get data. The datasets are proprietary, so they can only be queried, never copied: you send SQL, it runs in AWS under your name, and only the result comes back. Every query is recorded with who ran it. The same lines work on your laptop and in a job.
| Dataset | Tables | Split by |
|---|---|---|
hhd | CPHS: hh_income_monthly, hh_consumption_monthly, hh_consumption_weekly, member_income_monthly, people_of_india_wave, aspirational_wave, hh_char | year, state |
fld | Prowess (prowess_sa_*, prowess_ca_*), CapEx (capex_*), identity, ic_nse_daily: 127 tables | year, for three tables: prowess_sa_annual_expenditure, prowess_sa_annual_relatedparty_transaction, prowess_sa_daily_stocks |
maps | Boundaries: states, districts_2011, districts_states_hr, homogeneous_regions |
Table and column names are as in the MariaDB databases, in lower case.
Install the connector once
You don’t need DuckDB. Install these on your laptop once; jobs need them too:
install.packages(c("paws", "arrow"))If your project uses renv, start the script with library(paws) and library(arrow) and run renv::snapshot(). renv records only the packages your code names; without those lines, the job won’t have them.
$ julia --project=.
julia> ]add AWS Parquet2 CSV DataFrames DBInterfaceThis records them in the project’s Project.toml, which is where the job gets them: the Julia image has no packages of its own.
$ uv add --script analysis.py boto3 pyarrow pandasThe job installs what the script lists. In a project, add them to pyproject.toml or requirements.txt instead.
Connect and query
saturn’s helper connects you with one line. The same lines work on your laptop and in a job.
library(paws)
library(arrow)
source(Sys.getenv("SATURN_R"))
con <- saturn_db("hhd")
df <- DBI::dbGetQuery(con, "
SELECT year, AVG(tot_inc) AS avg_income
FROM hh_income_monthly
WHERE state = 'Karnataka' AND year >= 2022 AND response_status = 'Accepted'
GROUP BY year ORDER BY year")
DBI::dbListTables(con)include(ENV["SATURN_JL"])
con = saturn_db("hhd")
df = DBInterface.execute(con, """
SELECT * FROM member_income_monthly
WHERE state = 'Karnataka' AND year = 2023 LIMIT 10""") |> DataFrame
saturn_tables(con)import os
exec(open(os.environ["SATURN_PY"]).read())
con = saturn_db("hhd")
df = con.sql("""
SELECT * FROM member_income_monthly
WHERE state = 'Karnataka' AND year = 2023 LIMIT 10""").df() # .arrow(), .pl()
con.tables()Maps
The maps tables keep each boundary in a shape column, in MariaDB’s format: 4 bytes, then the standard WKB. Skip the 4 bytes. The coordinates are latitude and longitude (WGS 84).
With sf (install.packages("sf")):
source(Sys.getenv("SATURN_R"))
maps <- saturn_db("maps")
s <- DBI::dbGetQuery(maps, "SELECT st_nm, shape FROM states")
s$geometry <- sf::st_as_sfc(structure(lapply(s$shape, function(b) as.raw(b)[-(1:4)]), class = "WKB"), crs = 4326)
s <- sf::st_as_sf(s[c("st_nm", "geometry")])
plot(sf::st_geometry(s))Work with the shapes in SQL, which runs in AWS:
include(ENV["SATURN_JL"])
maps = saturn_db("maps")
df = DBInterface.execute(maps, """
SELECT st_nm FROM states
WHERE ST_Contains(ST_GeomFromBinary(substr(shape, 5)), ST_Point(77.59, 12.97))""") |> DataFrame # KarnatakaWith shapely (uv add shapely):
import os, shapely
exec(open(os.environ["SATURN_PY"]).read())
maps = saturn_db("maps")
df = maps.sql("SELECT st_nm, shape FROM states").df()
df["geometry"] = shapely.from_wkb([b[4:] for b in df["shape"]])In any language, SQL can work with the shapes directly: ST_GeomFromBinary(substr(shape, 5)) turns one into a geometry, for ST_Contains, ST_Intersects, ST_Area and the like.
The SQL
It’s Athena’s (Trino), close to what you know. The differences that come up:
- Strings in single quotes, names in double quotes. Column names come back in lower case (
tot_inc). yearis a whole number:year = 2023, not'2023'.- No
ILIKE: uselower(state) LIKE '%pradesh%'. Dates:date '2023-01-01',date_trunc('month', month_slot_date),year(month_slot_date). - dplyr’s lazy tables don’t work here: filter and summarise in SQL, then use dplyr on the result.
- Another dataset’s tables by name: connected to
hhd,SELECT company_name FROM fld.identity WHERE co_code = 11works.
Tips
- Filter on
yearandstatewhenever you can: those skip whole folders. - Do the heavy work in SQL (joins, aggregations): it runs in AWS on many machines, so only the answer travels. A query summarising all of India takes a few seconds.
- Large pulls are fine but slower: half a million rows by 44 columns takes about 5 seconds. Leave out
ORDER BYon big pulls and sort afterwards. - Each query reads at most 20 GB (all of
hhdis about 5 GB) and costs a fraction of a cent. - The data is a copy of the MariaDB databases as of 27 Oct 2026 (
hhd) and 29 Oct 2026 (fld,maps). For tables that aren't in the datasets, see the appendix.
Jobs and costs
$ saturn list # running jobs $ saturn logs <task-id> # follow a job's output again $ saturn stop <task-id> $ saturn costs # spending per month and person
When a job ends, a notice goes to the team's Zulip channel, tagging the person who ran it, with the script, the machine, the runtime and the cost. It comes from AWS, so it arrives even if your laptop is closed. Every Monday at 9:00 a summary of the past week follows, with jobs, failures, hours and spend per person, and how much each workspace folder stores, when it last changed and its last job.
When something's off
echo $WORKSPACE prints nothing
This terminal was opened before saturn connect. Open a new one, or run eval "$(saturn connect)". In RStudio, run readRenviron("~/.config/saturn-v/env").
The job doesn't have a package I installed
R: run renv::snapshot() after install.packages(), and check with renv::status(). Julia: start Julia with --project=. before ]add; ]status shows which project is active. Python: add it to the header, pyproject.toml or requirements.txt. See Packages.
"R 4.x is not available on AWS"
Your renv.lock was made with an R version AWS doesn't offer. Upgrade R on your laptop, then run renv::update() and renv::snapshot().
A job's output file isn't on my laptop yet
Files written by a job take a minute or two to appear in ~/aws/workspace.
saturn asks me to sign in
Your weekly sign-in expired. Run saturn login, then carry on.
Cheat sheet
saturn loginSign in (lasts 7 days)saturn connectMounts, tunnel, environmentsaturn statusWhat's connectedsaturn disconnectUndo connectsaturn run <path>Run a .R, .jl or .py on AWS--size largeBigger machine (small … max)SPOT=1 saturn run …~70% cheaper, may be interruptedsaturn listRunning jobssaturn logs <id>Follow a job againsaturn stop <id>Stop a jobsaturn costsSpending per month and personsaturn docsOpen this guideAppendix: the MariaDB database
Use the datasets for everyday work. The mayin-data MariaDB database (on Amazon RDS) is still there for tables that aren’t in the datasets.
Jobs and laptops share one login, aws_jobs, which can only read the mayin-data MariaDB databases (fld, hhd, maps, …). Save results as files in the workspace. The password reaches your code through DB_PASSWORD, so it never appears in a script.
| Variable | In a job | On your laptop |
|---|---|---|
DB_HOST, DB_PORT | the database | 127.0.0.1 and the tunnel port |
DB_USER, DB_PASSWORD | the shared, read-only aws_jobs login | |
con <- DBI::dbConnect(RMariaDB::MariaDB(),
host = Sys.getenv("DB_HOST"), port = as.integer(Sys.getenv("DB_PORT")),
user = Sys.getenv("DB_USER"), password = Sys.getenv("DB_PASSWORD"),
dbname = "fld", client.flag = RMariaDB::CLIENT_SSL)
df <- DBI::dbGetQuery(con, "SELECT co_code, company_name FROM identity LIMIT 10")
DBI::dbDisconnect(con)using MySQL, DBInterface, DataFrames
con = DBInterface.connect(MySQL.Connection, ENV["DB_HOST"], ENV["DB_USER"], ENV["DB_PASSWORD"];
port=parse(Int, ENV["DB_PORT"]), db="fld", ssl_mode=:required)
df = DBInterface.execute(con, "SELECT co_code, company_name FROM identity LIMIT 10") |> DataFrame
DBInterface.close!(con)# add pymysql and pandas to the script's packages
import os, pymysql, pandas as pd
con = pymysql.connect(host=os.environ["DB_HOST"], port=int(os.environ["DB_PORT"]),
user=os.environ["DB_USER"], password=os.environ["DB_PASSWORD"],
database="fld", ssl={"ssl": {}})
with con.cursor() as cur:
cur.execute("SELECT co_code, company_name FROM identity LIMIT 10")
df = pd.DataFrame(cur.fetchall(), columns=[c[0] for c in cur.description])
con.close()The database only accepts encrypted connections, which is what the SSL options above ask for. Other clients (DBeaver, TablePlus, mysql) work too: host 127.0.0.1, port 3306, user and password from ~/.config/saturn-v/env.
"Connections using insecure transport are prohibited"
Add the SSL option: client.flag = RMariaDB::CLIENT_SSL in R, ssl_mode=:required in Julia.
Port 3306 is already in use
Something on your laptop (a local MySQL, say) uses it. Connect with DB_LOCAL_PORT=13306 saturn connect; DB_PORT then says 13306.
The tunnel goes through a small server on AWS, which saturn connect starts if it’s stopped.