Skip to content

Using SquarePeg#

SquarePeg lets you run a single command inside a container on CLIMB's Kubernetes infrastructure, with as many CPUs and as much memory as the command needs, straight from the terminal of a small JupyterLab environment.

SquarePeg is pre-installed and pre-configured in every JupyterLab environment, so there is nothing to set up.

This page covers using SquarePeg on CLIMB. For more technical documentation, including every command line option, the configuration format, how runs are carried out and clean-up behaviour, see the SquarePeg wiki.

This page covers the following sections:

This page assumes you already understand advanced workings of the terminal and containers. For more information on using the terminal, please see our terminal page.

Why should you use SquarePeg ?#

Your JupyterLab environment has a fixed amount of CPU and memory, chosen when you launched it. That is usually enough for exploring data, writing scripts and running small jobs. Sometimes, though, a single step needs far more than that. Common examples include:

  • Classifying reads with Kraken2 against a large database that must be loaded entirely into memory
  • Assembling a large or complex dataset
  • Building an index or database
  • Any multi-threaded tool that would run much faster with 16 or 32 cores

You could relaunch your environment with a bigger profile, but then those resources stay reserved (and count against your team's quota) for as long as the environment is running, even while you're just reading output or editing code. You could also write a Nextflow pipeline, but that's a lot of setup for a single command.

SquarePeg sits between these two options. Your JupyterLab environment stays small, and only the heavy command gets the large resources, only for as long as it runs:

JupyterLab environment (small) -> squarepeg run ... -> pod with the CPU/memory you asked for -> output streamed back -> pod deleted

SquarePeg or Nextflow?

Use SquarePeg for one-off commands or simple shell loops. Use Nextflow for multi-step pipelines that need caching, resuming and many tasks running in parallel.

CLIMB SquarePeg config defaults#

SquarePeg in your JupyterLab environment reads a CLIMB config file from /etc/squarepeg/config.yaml (set through the SQUAREPEG_CONFIG environment variable). It uses the credentials that are already set up for your team, so every pod it starts runs in your team's own space on the cluster and uses the same resource quota. The config sets the following defaults:

Setting Default
/shared/public Mounted read-only in every pod. This is where the shared databases are, such as the Kraken2 databases in /shared/public/db/kraken2.
/shared/team Mounted read-write in every pod. Keep your inputs and outputs here.
CPUs / memory 1 CPU and 8 GB unless you set --cpus / --memory
User Pods run as the same jovyan user (UID 1000, GID 100) as your JupyterLab environment, so files they write belong to you
Environment Your S3 credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL) and Onyx credentials (ONYX_DOMAIN, ONYX_TOKEN) are passed through to the container
Clean-up The pod is deleted when the command finishes. Leftover pods from earlier runs that crashed are also cleaned up automatically.
Start-up timeout 300 seconds to pull the image and start the container

Your home directory is not available inside the pod

The pod is separate from your JupyterLab environment and usually runs on a different node in the cluster. Your home directory (/home/jovyan) is attached only to your own environment, so the pod can't see it or any other files on your environment's local disk. Only /shared/team and /shared/public are mounted. All input files must be in /shared/team (or /shared/public), and all output must be written to /shared/team, or it will be lost when the pod is deleted.

You can see the complete resolved config at any time with:

squarepeg config

Your first run#

Open a terminal and run:

squarepeg run alpine echo "hello from a pod"

SquarePeg will:

  1. Print the Kubernetes context and namespace it is using
  2. Create a pod running the alpine image
  3. Wait for the pod to start (pulling the image if needed)
  4. Stream the container's output (hello from a pod) to your terminal
  5. Delete the pod
  6. Exit with the container's exit code

SquarePeg's own status messages are prefixed [squarepeg:<run-id>] and go to stderr, so they never get mixed into your command's output.

The container's stdout and stderr are combined

Everything your command prints, both its normal output (stdout) and its log or progress messages (stderr), arrives on SquarePeg's stdout as a single stream. This is a current limitation of Kubernetes, not of SquarePeg. SquarePeg reads output through the Kubernetes log API, which records both streams together and hands them back as one. Kubernetes has a feature to keep them separate, but it's still experimental, so it isn't enabled on CLIMB.

In practice, squarepeg run ... > output.txt captures your command's messages as well as its results, and 2> won't separate them. If you need a clean output file, have the tool write it directly with its own output option (such as Kraken2's --output), or redirect inside the pod with bash -c "... > /shared/team/output.txt" (see Running several commands in one pod).

The general form of the command is:

squarepeg run [SQUAREPEG OPTIONS] IMAGE [COMMAND...]

Info

Everything after the image name is passed to the container exactly as written, including anything that looks like a flag. All SquarePeg options must therefore come before the image name.

Worked example: Kraken2 with the PlusPF database#

Kraken2 is a taxonomic classifier commonly used in metagenomics. By default it loads its whole database into memory before classifying anything. The PlusPF database (archaea, bacteria, viral, plasmid, human, UniVec_Core, protozoa and fungi) is far too large to fit in a typical JupyterLab environment, which makes it a good example of a job to hand to SquarePeg.

1. Find the database and check how big it is#

Pre-built Kraken2 databases are kept in /shared/public/db/kraken2, and latest always points to the newest version. These are the Ben Langmead Kraken2 indexes. Check the size of the PlusPF hash file, because that is roughly how much memory Kraken2 will need:

du -sh /shared/public/db/kraken2/pluspf/latest/hash.k2d

Ask for this much memory plus some headroom. The examples below use 128 GB for a database of roughly 110 GB. Adjust this to match the size du reports for any database you wish to use.

2. Put your reads somewhere the pod can see them#

Your input files must be under /shared/team. For example:

mkdir -p /shared/team/kraken_demo/{reads,results}
cp ~/my_sample_R1.fastq.gz ~/my_sample_R2.fastq.gz /shared/team/kraken_demo/reads/

3. Pick a container image#

Any public container image with Kraken2 in it will work. Here we use the BioContainers image:

quay.io/biocontainers/kraken2:2.17.2--pl5321h3be2455_0

Tip

Use a specific version tag rather than latest, so that your analysis is reproducible.

4. Check the pod with a dry run#

Before using a lot of resources, you can add --dry-run to print the exact pod definition SquarePeg would create, without creating anything:

squarepeg run --dry-run \
    --cpus 16 --memory 128g \
    quay.io/biocontainers/kraken2:2.17.2--pl5321h3be2455_0 \
    kraken2 --version

Look for the resources section (your CPU and memory) and the volumeMounts section (/shared/public and /shared/team).

5. Run Kraken2#

Once you are happy with the resource requests, you can run the full squarepeg command:

squarepeg run \
    --cpus 16 \
    --memory 128g \
    -w /shared/team/kraken_demo \
    quay.io/biocontainers/kraken2:2.17.2--pl5321h3be2455_0 \ # Everything after this is passed directly to the container
    kraken2 \
        --db /shared/public/db/kraken2/pluspf/latest \
        --threads 16 \
        --paired --gzip-compressed \
        --report results/my_sample.kreport \
        --output results/my_sample.kraken \
        reads/my_sample_R1.fastq.gz reads/my_sample_R2.fastq.gz

Taking each part in turn:

Part What it does
squarepeg run Runs one command in a new pod
--cpus 16 Asks for 16 CPU cores for this pod
--memory 128g Asks for 128 GB of memory, enough to load PlusPF
-w /shared/team/kraken_demo Sets the working directory inside the container, so the relative paths reads/ and results/ work
quay.io/biocontainers/kraken2:... The container image. Everything after this goes to the container.
kraken2 --db ... The Kraken2 command itself, run inside the pod. The database is read straight from the read-only /shared/public mount.
--threads 16 A Kraken2 option. Keep it equal to --cpus so Kraken2 uses the cores you asked for.

Kraken2's progress messages will appear in your terminal as it runs. When it finishes, my_sample.kreport and my_sample.kraken will be in /shared/team/kraken_demo/results, and you can work with them from your JupyterLab environment as normal.

Warning

The resources you request count against your team's CPU and memory quota for as long as the pod runs. If the team has used up its quota, the pod can't be scheduled and you won't be able to launch new JupyterLab or Nextflow pods either. Only ask for what the command needs.

Important command line options#

These are the options you'll use most often. Remember that they all go before the image name. Run squarepeg run --help for the full list, or see the SquarePeg CLI reference.

Resources#

Option Example Description
--cpus --cpus 16 Number of CPU cores. Decimals are allowed (0.5). Default: 1.
-m, --memory --memory 128g Memory, in the same units as Docker (512m, 8g, 128g). Default: 8G.
--gpus --gpus 1 Number of whole GPUs. Default: none. See Using GPUs.

--cpus and --memory set both the Kubernetes request and limit. If your command goes over the memory limit, the pod is killed and SquarePeg exits with code 137 and reports OOMKilled. If that happens, rerun with more memory.

Using GPUs#

If your team has GPU quota, --gpus gives the pod its own NVIDIA A100 GPU(s). This means you can run a GPU tool from a standard, CPU-only JupyterLab environment, and the GPU is only reserved while the command runs. SquarePeg sets up everything else the pod needs to use the GPU. You don't need to change any config.

To check that you can get a GPU, run nvidia-smi in a CUDA image:

squarepeg run --gpus 1 nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

A real job looks like any other SquarePeg command, with --gpus added. The image must include GPU (CUDA) support:

squarepeg run --gpus 1 --cpus 8 --memory 32g -w /shared/team/project \
    MY_GPU_IMAGE my_gpu_tool --input data/ --output results/

A few things to know:

  • Only a number is accepted (--gpus 1 or --gpus count=1). Docker's --gpus all and --gpus device=0 forms are rejected, because Kubernetes chooses which GPUs to give you.
  • GPUs are scarce. If none is free, or your team's GPU quota is used up, the pod waits and SquarePeg prints a warning such as FailedScheduling: ... Insufficient nvidia.com/gpu. It keeps waiting until --timeout (300 seconds by default) in case a GPU becomes free. Press Ctrl+C to stop waiting.
  • Still ask for CPUs and memory. --gpus doesn't change the defaults of 1 CPU and 8 GB, and most GPU tools need more than that for loading data.

Files and directories#

Option Example Description
-w, --workdir -w /shared/team/project Working directory inside the container. Without it, the image's default is used (often /), so set it, or use absolute paths everywhere.

/shared/team and /shared/public are mounted in every pod automatically, so you shouldn't need any other mounts. Point your command at files under these paths and set -w to a directory in /shared/team.

Avoid -v / --volume

SquarePeg accepts Docker's -v option, but on CLIMB it rarely does what you would expect, so we recommend not using it. What -v does depends on what comes before the ::

  • A path, such as -v /home/jovyan/data:/data or -v ./data:/data: rejected with an error. In Docker, this shares a folder from the machine you're on. Your SquarePeg pod isn't running on that machine. Your home directory is a storage volume attached only to your own JupyterLab environment, so other pods can't see it, and the path would point at the node's own disk rather than your files. SquarePeg therefore refuses any -v source that starts with / or .. This includes paths under /shared/team, so you can't mount a subfolder such as -v /shared/team/project:/data at a different path either. Copy your files to /shared/team and use them at their real path.
  • Just a container path, such as -v /scratch: an empty temporary directory. It is created on the node's local disk and deleted with the pod, so anything written there is lost when the command finishes. It also uses the node's limited local disk, and if it fills that disk Kubernetes may evict your pod part-way through the run.
  • A name, such as -v shared-team:/data: a named volume. CLIMB's config defines two names: shared-team and shared-public. These mount the whole team or public share at the extra path you give (add :ro for read-only), alongside the usual /shared/team and /shared/public. This is only useful if a tool insists on a fixed path. Any other name silently gives you an empty temporary directory, exactly like -v /scratch. For example, a typo such as -v shared_team:/data doesn't produce an error: your command just sees an empty /data, and anything written there is lost. Unlike Docker named volumes, these temporary directories aren't kept between runs.

In almost every case, the simplest option is to use files at their real paths under /shared/team and /shared/public. Write temporary files to a directory under /shared/team too, and delete them when you're done.

Environment variables#

Option Example Description
-e, --env -e MY_VAR=value Sets an environment variable in the container. Repeatable.
-e (bare) -e MY_VAR Copies MY_VAR from your terminal into the container

Naming, inspecting and debugging#

Option Description
--name NAME Pod name (lowercase letters, digits and -, up to 63 characters)
--dry-run Prints the pod definition and exits without creating anything
--dry-run-server Sends the pod definition to the cluster for validation without creating it. This catches quota or permission problems before a real run.
--keep Keeps the pod after it finishes so you can inspect it with kubectl describe pod NAME / kubectl logs NAME. Delete it yourself afterwards (see Using kubectl directly).
--quiet Hides SquarePeg's own status messages and shows only the container's output
--timeout SECONDS How long to wait for the pod to start (default 300). Large images may need longer. This doesn't limit how long the command runs.
--entrypoint CMD Overrides the image's entrypoint, for example --entrypoint bash with -c "cmd1 && cmd2" to run several commands

Running several commands in one pod#

Because everything after the image name goes to the container, you can use a shell to chain commands or use pipes and redirection inside the pod:

squarepeg run --cpus 8 --memory 16g -w /shared/team/project \
    quay.io/biocontainers/kraken2:2.17.2--pl5321h3be2455_0 \
    bash -c "kraken2 --db /shared/public/db/kraken2/standard_16gb/latest --threads 8 reads.fq.gz > out.kraken && echo done"

Warning

Without bash -c "...", any > or | is run by your JupyterLab shell, not by the pod. For example, squarepeg run ... kraken2 ... > out.kraken writes the container's output to a file on your JupyterLab environment, but that file also contains Kraken2's progress messages, because Kubernetes combines stdout and stderr into one stream. It is also slower, since all the data is streamed back through your terminal. Write output files directly to /shared/team from inside the pod instead, either with the tool's own output options or with bash -c. Inside the pod, > and 2> work normally.

Tips for longer runs#

  • Write results to files, not to the terminal. Use the tool's output options (such as Kraken2's --output and --report) to write directly to /shared/team. This keeps your results separate from the tool's log messages and avoids streaming large outputs back through your terminal.
  • Run one sample after another with a loop. SquarePeg exits with the container's exit code, so it works in ordinary shell scripts:
cd /shared/team/kraken_demo
for r1 in reads/*_R1.fastq.gz; do
    sample=$(basename "$r1" _R1.fastq.gz)
    squarepeg run --cpus 16 --memory 128g -w /shared/team/kraken_demo \
        quay.io/biocontainers/kraken2:2.17.2--pl5321h3be2455_0 \
        kraken2 --db /shared/public/db/kraken2/pluspf/latest --threads 16 \
            --paired --gzip-compressed \
            --report "results/${sample}.kreport" --output "results/${sample}.kraken" \
            "reads/${sample}_R1.fastq.gz" "reads/${sample}_R2.fastq.gz" \
        || echo "${sample} failed"
done
  • Ctrl+C once stops the run, deletes the pod and exits. Pressing Ctrl+C a second time exits straight away and leaves the pod running (see Using kubectl directly).
  • SquarePeg also works from a Jupyter notebook cell with !squarepeg run ....

Troubleshooting#

Symptom What it means / what to do
Exit code 137 and OOMKilled The command used more memory than you asked for. Increase --memory.
Exit code 125 SquarePeg couldn't run the container. For example, the image name is wrong, the pull failed or the pod didn't start within --timeout. Read the message above it for the specific reason.
Exit code 2 A problem with the SquarePeg command itself, such as an unsupported flag or a bad value. Check that all SquarePeg options come before the image name.
Pod stays waiting and then times out with Insufficient cpu / Insufficient memory No node has enough free resources, or your team's quota is used up. Ask for less, or wait for other team pods to finish. kubectl get pods shows what's running.
No such file or directory for an input The file is probably in your home directory. Move it into /shared/team.
Output files are missing They were written somewhere other than /shared/team inside the pod and were deleted with it.

Cleaning up leftover pods#

Leftover pods of yours that have finished or are stuck (for example, because of a mistyped image name) are cleaned up automatically the next time you use squarepeg run, once they're more than 5 minutes old. To clean them up straight away, without starting a new run, use squarepeg clean. First, list what it would delete without deleting anything:

squarepeg clean --dry-run

Then run it without --dry-run to delete them:

squarepeg clean

squarepeg clean doesn't ask for confirmation, but it only ever deletes SquarePeg pods that have already finished or are stuck and can never start. It never touches a pod that is still running, or one started with --keep.

Pods you kept with --keep are never cleaned up automatically, and nor are pods left running after a second Ctrl+C, until they finish. You'll need to delete these yourself with kubectl. Read the warning below first.

Using kubectl directly#

ADVANCED USERS ONLY: kubectl acts on your whole team, not just you

The kubectl credentials in your JupyterLab environment belong to your team, not to you personally. With them you can see, and delete, every pod in your team's space: your teammates' JupyterLab environments, their Nextflow tasks and their SquarePeg runs, as well as your own. kubectl won't ask "are you sure?" and won't stop you deleting someone else's work, and a deleted pod can't be brought back. A mistyped name, or a command that matches more pods than you meant, can kill a colleague's long-running job or their whole JupyterLab environment, including any unsaved work.

SquarePeg is designed to only ever clean up SquarePeg pods. When you use kubectl yourself, there's no such safety net.

Commands that only read information are safe to use at any time:

kubectl get pods -l app.kubernetes.io/managed-by=squarepeg   # list SquarePeg pods (all team squarepeg pods)
kubectl describe pod POD_NAME                                 # details and events for one pod
kubectl logs POD_NAME                                         # output from one pod

Before you delete anything, check that the pod is yours. CLIMB tags every SquarePeg pod with the JupyterHub user who started it:

kubectl get pod POD_NAME -o jsonpath='{.metadata.annotations.user}'

This should print sp- followed by your own JupyterHub username. For example:

kubectl get pod POD_NAME -o jsonpath='{.metadata.annotations.user}'
sp-sajwilkinson.loman-labz

Only then delete it, by its exact name:

kubectl delete pod POD_NAME

Warning

Only ever delete pods one at a time, by exact name. Don't use kubectl delete with --all, label selectors (-l) or wildcards, because these can match your teammates' pods as well as your own. If you're not sure whether a pod is yours, ask your team before deleting it.

Further information#

The full SquarePeg documentation, including all options and advanced configuration, is on the SquarePeg wiki.