Outputs

Outputs are the files produced by your job.

Examples include:

• Trained models
• Checkpoints
• Evaluation results
• CSV files
• Images and plots
• Videos
• Reports
• Generated datasets

Output Directory

All files that should be preserved after a job completes must be written to:

/workspace/output

Anything written to this directory is automatically copied back to HPC storage when the job finishes.

Example:

from pathlib import Path

output_dir = Path("/workspace/output")
output_dir.mkdir(parents=True, exist_ok=True)

(output_dir / "results.txt").write_text(
    "Training completed successfully\n",
    encoding="utf-8",
)

Containers Are Temporary

Docker containers are non-persistent.

Any files written inside the container will be lost when the job finishes.

Save all results to:

/workspace/output

Files written to this directory are copied back to the HPC storage and remain available after the job completes.

Outputs Must Be Saved to the Correct Path

You are responsible for ensuring all of your code saves outputs to:

/workspace/output

If files are written somewhere else, they will be lost when the container is removed.

What Gets Saved

Files written inside:

/workspace/output

are preserved.

Example:

/workspace/output
├── model.pt
├── metrics.csv
└── figures
    └── reward_curve.png

These files will be available after the job completes.

What Does Not Get Saved

Files written elsewhere inside the container are temporary.

Example:

/app
/tmp
/home

Data written to these locations will be lost when the container exits.

Example:

Path("/tmp/results.txt").write_text("data")

This file will disappear when the job finishes.

Always save important files into:

/workspace/output

Recommended Output Structure

For larger projects, organise outputs into folders.

Example:

/workspace/output
├── checkpoints
├── figures
├── logs
├── models
└── results

Example:

from pathlib import Path

output_dir = Path("/workspace/output")

(output_dir / "checkpoints").mkdir(parents=True, exist_ok=True)
(output_dir / "models").mkdir(parents=True, exist_ok=True)
(output_dir / "results").mkdir(parents=True, exist_ok=True)

Output Structure

The directory structure created inside the container is preserved when the job finishes.

If you create a messy output structure, it will be preserved as-is.

Organise outputs into directories to make them easier to browse and analyse.

Code Responsibility

You are responsible for ensuring your code saves outputs to the correct path and handles any necessary directory creation.

The HPC system does not automatically create subdirectories inside:

/workspace/output

If you want to save files into subdirectories, your code needs to create those directories first.

Resumable Jobs and Checkpoints

Long-running jobs can optionally be submitted as resumable:

"resumable": true

This is useful for workloads such as machine-learning training that periodically save checkpoints.

A resumable job may need to restart after:

• an unexpected worker or machine interruption
• scheduler-controlled opportunistic preemption

The directory:

/workspace/output

is also the persistent workspace used for resumable job state.

Your application should therefore save any state required to continue execution somewhere under this directory.

For example:

from pathlib import Path

output_dir = Path("/workspace/output")
checkpoint_dir = output_dir / "checkpoints"
checkpoint_dir.mkdir(parents=True, exist_ok=True)

checkpoint_path = checkpoint_dir / "latest.pt"

if checkpoint_path.exists():
    print(f"Resuming from: {checkpoint_path}")
    # Load your model, optimiser, training step, etc.
else:
    print("Starting a new training run")

Your training loop should periodically update the checkpoint:

# Example only - save whatever state your framework requires.
torch.save(
    {
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),
        "step": step,
    },
    checkpoint_path,
)

Recovery After a Worker Interruption

If a worker or machine unexpectedly restarts, surviving local job files may be reused when a resumable job is restarted on that worker.

The scheduler runs the original Docker command again and the application can detect the existing checkpoint under:

/workspace/output

Recovery After Opportunistic Preemption

Long resumable jobs may also be preempted when they are using Opportunistic capacity and higher-priority work requires the worker.

For a scheduler-controlled resumable preemption:

Running Job
    ↓
Container Stopped
    ↓
/workspace/output Persisted to NAS
    ↓
Job Returned to Queue
    ↓
Job Scheduled Again
    ↓
Saved Output Restored
    ↓
Container Starts Again
    ↓
Application Loads Checkpoint

The resumed job does not need to return to the same worker.

If no local job state exists on the new worker, the scheduler restores the previously persisted output state from the NAS before starting the container.

Your application then sees the restored files under the same path:

/workspace/output

and is responsible for loading the appropriate checkpoint.

See Job Priorities and Scheduling Tiers for the rules governing when Opportunistic jobs may be preempted.

Save Checkpoints Under /workspace/output

Recovery only preserves resumable state stored under:

/workspace/output

A checkpoint saved somewhere such as:

/app/checkpoint.pt
/tmp/checkpoint.pt
/home/user/checkpoint.pt

is inside the temporary container filesystem and cannot be relied upon after the container is restarted.

Checkpoint Frequently Enough

A resumable job can only continue from state that has actually been saved.

For example, if a checkpoint is written every 30 minutes and the job is interrupted 29 minutes after the previous checkpoint, approximately 29 minutes of work may need to be repeated.

Choose a checkpoint frequency appropriate for the cost of your workload.

The Original Command Is Run Again

Recovery starts a new Docker container and executes the original job command again.

Your program should therefore be safe to start when output files from an earlier execution already exist.

Recovery Is Not Exactly-Once Execution

A worker may fail or a job may be preempted after some work has completed but before a newer checkpoint has been written.

A resumable job may therefore execute some work more than once.

Applications should rely on checkpoints and saved progress rather than assuming every training step or operation will execute exactly once.

Resumable Does Not Mean Automatic Application Recovery

The scheduler preserves and restores the /workspace/output workspace.

It does not understand model checkpoints or application state.

Your application must detect and load its own saved state when it starts.

Accessing Job Outputs from the NAS

Job outputs are stored on the CARES NAS and can be accessed directly from your computer.

After a job completes, outputs are available from the shared HPC storage under:

/cares-nas/hpc/outputs/<upi>/<job_id>

The scheduler preserves the directory structure created inside:

/workspace/output

For example:

/workspace/output
├── models
│   └── model.pt
└── results
    └── metrics.csv

will appear in the final job outputs exactly as written.

Private Access

Output directories are private to each user.

You cannot access outputs from other users' jobs.

---

Downloading Outputs (UI)

To download outputs, navigate to the outputs/<upi> folder at: http://130.216.238.2:5000. Click on the relevant job folder to download the outputs as a zip file.

Mounting the Output Directory (Linux)

Mounting the NAS locally is often more convenient than downloading files through a web browser, especially for large datasets, model checkpoints, videos, and experiment results.

Create a local mount point:

mkdir -p ~/hpc_outputs

Mount the NAS:

sudo mount -t cifs \
    //130.216.238.2/outputs/<upi> \
    ~/hpc_outputs \
    -o username=<upi>

Example:

sudo mount -t cifs \
    //130.216.238.2/outputs \
    ~/hpc_outputs/jsmith123 \
    -o username=jsmith123

You will be prompted for your NAS password.

---

Accessing Your Outputs

After mounting:

cd ~/hpc_outputs

Browse your output directory:

cd jsmith123

Example:

~/hpc_outputs/jsmith123/
├── job_001/
├── job_002/
├── job_003/
└── ...

---

Copying Results

Copy a job locally:

cp -r \
    ~/hpc_outputs/jsmith123/job_001 \
    ~/Downloads/

Copy an individual file:

cp \
    ~/hpc_outputs/jsmith123/job_001/results.csv \
    .

---

Unmounting

When finished:

sudo umount ~/hpc_outputs

---

Using rsync

For large results it is often more efficient to use rsync.

Example:

rsync -avP \
    ~/hpc_outputs/jsmith123/job_001/ \
    ./job_001/

This allows interrupted transfers to resume.

---

Recommended Workflow

Run Job
    ↓
Monitor Job
    ↓
Job Completes
    ↓
Mount NAS
    ↓
Copy Results Locally
    ↓
Analyse Results

For large machine learning experiments, model checkpoints, videos, and datasets, mounting the NAS is generally the easiest way to access your results.

Outputs vs Logs

Outputs and logs serve different purposes.

Type	Purpose
Logs	Progress updates, debugging information, status messages
Outputs	Models, checkpoints, CSV files, figures, reports, videos

Example:

print("Training complete")

appears in the logs.

Example:

torch.save(
    model.state_dict(),
    "/workspace/output/models/model.pt",
)

creates a saved output file.

Large Outputs

Large outputs are supported, however users should:

• Remove unnecessary files
• Avoid storing temporary data
• Compress outputs where appropriate
• Delete old outputs when no longer required

Common Mistakes

Outputs Missing

Files were written outside:

/workspace/output

Empty Output Directory

Verify files were actually created.

Example:

output_dir.mkdir(parents=True, exist_ok=True)

Only Logs Exist

Printing information does not create output files.

Example:

print("Training complete")

creates log messages but does not save any files.

Output Structure Is Messy

Organise outputs into directories:

checkpoints/
figures/
models/
results/

to make outputs easier to browse and analyse.