xKDR research computing

Run heavy jobs on AWS from your laptop

Work on your laptop as usual. When a script needs more than your laptop has, saturn run sends it to a machine with up to 32 cores and 244 GB of memory, streams the output back to your terminal, and records what it cost. The same script runs in both places, in R, Julia or Python.

  1. 1

    Get an account. Ask an admin to add you. You get an email from AWS: set a password and register MFA.

  2. 2

    Install once with one command, then each session:

    $ saturn login
    $ saturn connect
  3. 3

    Put your script in your folder in the shared workspace, ~/aws/workspace/<you>/, and try it locally.

  4. 4

    Run it on AWS. The path is relative to the workspace.

    $ saturn run <you>/project/analysis.R

How it fits together

After saturn connect, your laptop sees the same files and data a job does, under the same names. That's why a script you try locally runs unchanged on AWS.

Your code never uses these paths directly. It reads them from environment variables (see below), which point to the right place on each side.

Setup (once)

On a Mac or Linux, in a terminal:

$ curl -fsSL https://raw.githubusercontent.com/xKDR/saturn-v/main/install.sh | sh

Then open a new terminal. The installer:

  • installs the tools saturn uses: the AWS command line, rclone (for the mounts) and AWS’s Session Manager plugin (for the MariaDB tunnel; see the appendix). On a Mac through Homebrew (install that first); on Linux into ~/.local, without sudo;
  • puts saturn in ~/.saturn-v and on your PATH;
  • adds two lines to your shell’s startup file, marked # saturn-v.

It’s safe to run again. To read it first, replace | sh with | less.

  • Linux: the mounts need FUSE (sudo apt install fuse3); most desktops have it, and the installer says if it’s missing.
  • Updates: when a new version is out, saturn asks you to run saturn update before anything else (disconnect, stop and docs still work). It reconnects you with the new version if you were connected.

Each session

$ saturn login       # opens your browser; lasts 7 days
$ saturn connect

saturn connect does four things:

  • mounts the shared workspace at ~/aws/workspace, and the satellite data (read-only) at ~/aws/satellite;
  • opens a tunnel to the MariaDB database on 127.0.0.1:3306 (see the appendix), and reopens it in the background if it drops;
  • creates your own folder, ~/aws/workspace/<your user name>;
  • writes the job's environment variables to ~/.config/saturn-v/env, which new terminals load;

Everything runs on your sign-in, the mounts included: AWS keys renew in the background for the whole week, and nothing secret is stored on your laptop. When the week is up, saturn asks you to run saturn login again.

saturn status shows what's connected. saturn disconnect undoes it all. In the terminal you're already in, eval "$(saturn connect)" also sets the variables there.

The workspace

Everyone shares one workspace, so keep to this layout:

FolderFor
<your user name>/Your own work. MY_WORKSPACE points here, on your laptop and in your jobs.
shared/<project>/Work several people use. Agree on the folder name with them.
db/, .julia/, .R/Used by saturn itself. Leave them alone.

If two people save the same file, the last save wins. Anything a job writes appears on your laptop within a minute or two. Anything else at the top level shows up in the Monday summary as “not in anyone’s folder”.

The same code on your laptop and on AWS

These variables are set in every job, and on your laptop after saturn connect. Build paths and connections from them, never from fixed paths.

VariableIn a jobOn your laptop
WORKSPACE/workspace~/aws/workspace
MY_WORKSPACE/workspace/<you>~/aws/workspace/<you>
SATELLITE/vsis3/<bucket>, read straight from S3~/aws/satellite

Reading a satellite file

f <- file.path(Sys.getenv("SATELLITE"), "ntl/sl/rad/<file>.tif")
r <- terra::rast(f)

out <- file.path(Sys.getenv("MY_WORKSPACE"), "project", "results.rds")
Cost

Reading satellite files on your laptop downloads them, at about $0.11 per GB. Try things on a few files or a small area locally, and do the heavy reading in a job, where it's free.

RStudio

Apps started from the Dock don't read your terminal's setup. In R, run readRenviron("~/.config/saturn-v/env") first.

Running a job

$ saturn run <you>/project/main.jl
$ saturn run <you>/project/analysis.R
$ saturn run <you>/project/analysis.py
$ saturn run --size large <you>/project/main.jl              # bigger machine
$ saturn run -e START=2014-01 <you>/project/main.jl           # extra variable for the job
$ SPOT=1 saturn run <you>/project/main.jl                     # ~70% cheaper, may be interrupted

The job starts in about a minute and its output streams to your terminal. Press Ctrl-C to stop watching: the job keeps running, and your laptop can sleep or disconnect.

--sizeMachineAbout per hour
small1 core, 4 GB$0.06
medium (default)4 cores, 16 GB$0.25
large8 cores, 60 GB$0.62
xlarge16 cores, 120 GB$1.24
max32 cores, 244 GB$2.50

You pay per second, only while a job runs, with a one-minute minimum. A quick check job costs about half a cent.

The examples folder has a first script in each language. Copy one into your folder and run it.

Packages

Why a project lists its packages

On your laptop, a package you installed once is there for every script. A job starts on a fresh machine that has none of them. It needs a list of what to install, and it can only read that list from files in your project folder.

The list also pins versions. Packages change: a new release can rename a function, change a default or shift a result. With versions pinned, the job uses exactly what you tested on your laptop, and so does a colleague running your code, or you in a year when a referee asks for a rerun.

And each project gets its own list. Upgrading a package for one project doesn't quietly change another.

Each language keeps two kinds of information:

What you useExact versions of everythingThe job runs
JuliaProject.tomlManifest.tomlPkg.instantiate()
R(renv reads your code)renv.lockrenv::restore()
Pythonpyproject.toml, requirements.txt or the script headeruv.lockuv run

"Everything" includes the packages your packages depend on, which is why the version files are long. You never edit them by hand; the commands below write them.

Do you need renv? Not if your script only uses packages already in the R image: terra, sf, GDAL, the tidyverse, data.table, DBI and more. Add renv when you need anything else, or exact versions.

Start a project (once)

setwd("~/aws/workspace/<you>/project")
renv::init()

This writes three things into the folder: renv.lock (the list), a renv/ folder, and an .Rprofile that switches renv on whenever R starts in this folder. R prints Project '…' loaded when it does.

The installed packages themselves are kept on your laptop, not in the shared workspace. saturn connect sets this up; in RStudio, run readRenviron("~/.config/saturn-v/env") before renv::init().

Add a package

install.packages("fixest")
renv::snapshot()          # write it to renv.lock

install.packages() installs it on your laptop only. renv::snapshot() is what puts it in the list the job reads. Forget it, and the job fails with there is no package called 'fixest'.

Check before running a job

renv::status()

It lists packages your code uses that aren't recorded, and recorded packages that aren't installed. "No issues found" means the job will get what you have.

Other things you'll do

renv::install("lrberge/fixest")   # from GitHub instead of CRAN; then renv::snapshot()
renv::update(); renv::snapshot()  # newer versions of everything
renv::remove("fixest"); renv::snapshot()
renv::restore()                   # install what renv.lock lists, e.g. in a colleague's project

R versions

renv.lock also records your R version, and the job uses the same one (4.5 or 4.6). If yours isn't available, saturn run says so: upgrade R, then renv::update() and renv::snapshot().

Data

This is where to get data. The datasets are proprietary, so they can only be queried, never copied: you send SQL, it runs in AWS under your name, and only the result comes back. Every query is recorded with who ran it. The same lines work on your laptop and in a job.

DatasetTablesSplit by
hhdCPHS: hh_income_monthly, hh_consumption_monthly, hh_consumption_weekly, member_income_monthly, people_of_india_wave, aspirational_wave, hh_charyear, state
fldProwess (prowess_sa_*, prowess_ca_*), CapEx (capex_*), identity, ic_nse_daily: 127 tablesyear, for three tables: prowess_sa_annual_expenditure, prowess_sa_annual_relatedparty_transaction, prowess_sa_daily_stocks
mapsBoundaries: states, districts_2011, districts_states_hr, homogeneous_regions

Table and column names are as in the MariaDB databases, in lower case.

Install the connector once

You don’t need DuckDB. Install these on your laptop once; jobs need them too:

install.packages(c("paws", "arrow"))

If your project uses renv, start the script with library(paws) and library(arrow) and run renv::snapshot(). renv records only the packages your code names; without those lines, the job won’t have them.

Connect and query

saturn’s helper connects you with one line. The same lines work on your laptop and in a job.

library(paws)
library(arrow)
source(Sys.getenv("SATURN_R"))
con <- saturn_db("hhd")

df <- DBI::dbGetQuery(con, "
  SELECT year, AVG(tot_inc) AS avg_income
  FROM hh_income_monthly
  WHERE state = 'Karnataka' AND year >= 2022 AND response_status = 'Accepted'
  GROUP BY year ORDER BY year")

DBI::dbListTables(con)

Maps

The maps tables keep each boundary in a shape column, in MariaDB’s format: 4 bytes, then the standard WKB. Skip the 4 bytes. The coordinates are latitude and longitude (WGS 84).

With sf (install.packages("sf")):

source(Sys.getenv("SATURN_R"))
maps <- saturn_db("maps")

s <- DBI::dbGetQuery(maps, "SELECT st_nm, shape FROM states")
s$geometry <- sf::st_as_sfc(structure(lapply(s$shape, function(b) as.raw(b)[-(1:4)]), class = "WKB"), crs = 4326)
s <- sf::st_as_sf(s[c("st_nm", "geometry")])
plot(sf::st_geometry(s))

In any language, SQL can work with the shapes directly: ST_GeomFromBinary(substr(shape, 5)) turns one into a geometry, for ST_Contains, ST_Intersects, ST_Area and the like.

The SQL

It’s Athena’s (Trino), close to what you know. The differences that come up:

  • Strings in single quotes, names in double quotes. Column names come back in lower case (tot_inc).
  • year is a whole number: year = 2023, not '2023'.
  • No ILIKE: use lower(state) LIKE '%pradesh%'. Dates: date '2023-01-01', date_trunc('month', month_slot_date), year(month_slot_date).
  • dplyr’s lazy tables don’t work here: filter and summarise in SQL, then use dplyr on the result.
  • Another dataset’s tables by name: connected to hhd, SELECT company_name FROM fld.identity WHERE co_code = 11 works.

Tips

  • Filter on year and state whenever you can: those skip whole folders.
  • Do the heavy work in SQL (joins, aggregations): it runs in AWS on many machines, so only the answer travels. A query summarising all of India takes a few seconds.
  • Large pulls are fine but slower: half a million rows by 44 columns takes about 5 seconds. Leave out ORDER BY on big pulls and sort afterwards.
  • Each query reads at most 20 GB (all of hhd is about 5 GB) and costs a fraction of a cent.
  • The data is a copy of the MariaDB databases as of 27 Oct 2026 (hhd) and 29 Oct 2026 (fld, maps). For tables that aren't in the datasets, see the appendix.

Jobs and costs

$ saturn list                 # running jobs
$ saturn logs <task-id>       # follow a job's output again
$ saturn stop <task-id>
$ saturn costs                # spending per month and person

When a job ends, a notice goes to the team's Zulip channel, tagging the person who ran it, with the script, the machine, the runtime and the cost. It comes from AWS, so it arrives even if your laptop is closed. Every Monday at 9:00 a summary of the past week follows, with jobs, failures, hours and spend per person, and how much each workspace folder stores, when it last changed and its last job.

When something's off

echo $WORKSPACE prints nothing

This terminal was opened before saturn connect. Open a new one, or run eval "$(saturn connect)". In RStudio, run readRenviron("~/.config/saturn-v/env").

The job doesn't have a package I installed

R: run renv::snapshot() after install.packages(), and check with renv::status(). Julia: start Julia with --project=. before ]add; ]status shows which project is active. Python: add it to the header, pyproject.toml or requirements.txt. See Packages.

"R 4.x is not available on AWS"

Your renv.lock was made with an R version AWS doesn't offer. Upgrade R on your laptop, then run renv::update() and renv::snapshot().

A job's output file isn't on my laptop yet

Files written by a job take a minute or two to appear in ~/aws/workspace.

saturn asks me to sign in

Your weekly sign-in expired. Run saturn login, then carry on.

Cheat sheet

saturn loginSign in (lasts 7 days)
saturn connectMounts, tunnel, environment
saturn statusWhat's connected
saturn disconnectUndo connect
saturn run <path>Run a .R, .jl or .py on AWS
--size largeBigger machine (small … max)
SPOT=1 saturn run …~70% cheaper, may be interrupted
saturn listRunning jobs
saturn logs <id>Follow a job again
saturn stop <id>Stop a job
saturn costsSpending per month and person
saturn docsOpen this guide

Appendix: the MariaDB database

Use the datasets for everyday work. The mayin-data MariaDB database (on Amazon RDS) is still there for tables that aren’t in the datasets.

Jobs and laptops share one login, aws_jobs, which can only read the mayin-data MariaDB databases (fld, hhd, maps, …). Save results as files in the workspace. The password reaches your code through DB_PASSWORD, so it never appears in a script.

VariableIn a jobOn your laptop
DB_HOST, DB_PORTthe database127.0.0.1 and the tunnel port
DB_USER, DB_PASSWORDthe shared, read-only aws_jobs login
con <- DBI::dbConnect(RMariaDB::MariaDB(),
  host = Sys.getenv("DB_HOST"), port = as.integer(Sys.getenv("DB_PORT")),
  user = Sys.getenv("DB_USER"), password = Sys.getenv("DB_PASSWORD"),
  dbname = "fld", client.flag = RMariaDB::CLIENT_SSL)
df <- DBI::dbGetQuery(con, "SELECT co_code, company_name FROM identity LIMIT 10")
DBI::dbDisconnect(con)

The database only accepts encrypted connections, which is what the SSL options above ask for. Other clients (DBeaver, TablePlus, mysql) work too: host 127.0.0.1, port 3306, user and password from ~/.config/saturn-v/env.

"Connections using insecure transport are prohibited"

Add the SSL option: client.flag = RMariaDB::CLIENT_SSL in R, ssl_mode=:required in Julia.

Port 3306 is already in use

Something on your laptop (a local MySQL, say) uses it. Connect with DB_LOCAL_PORT=13306 saturn connect; DB_PORT then says 13306.

The tunnel goes through a small server on AWS, which saturn connect starts if it’s stopped.