Run Python scripts

DPDispatcher can be used to directly run a single Python script:

dpdisp run script.py

The script must include inline script metadata compliant with PEP 723. An example of the script is shown below.

 1# /// script
 2# # dpdispatcher doesn't use `requires-python` and `dependencies`
 3# requires-python = ">=3"
 4# dependencies = [
 5# ]
 6# [tool.dpdispatcher]
 7# work_base = "./"
 8# forward_common_files=[]
 9# backward_common_files=[]
10# [tool.dpdispatcher.machine]
11# batch_type = "Shell"
12# local_root = "./"
13# context_type = "LazyLocalContext"
14# [tool.dpdispatcher.resources]
15# number_node = 1
16# cpu_per_node = 1
17# gpu_per_node = 0
18# group_size = 0
19# [[tool.dpdispatcher.task_list]]
20# # no need to contain the script filename
21# command = "python"
22# # can be a glob pattern
23# task_work_path = "./"
24# forward_files = []
25# backward_files = ["log"]
26# ///
27
28print("hello world!")

The PEP 723 metadata entries for tool.dpdispatcher are defined as follows:

pep723:
type: dict
argument path: pep723

PEP 723 metadata

work_base:
type: str, optional, default: ./
argument path: pep723/work_base

Base directory for the work

forward_common_files:
type: typing.List[str], optional, default: []
argument path: pep723/forward_common_files

Common files to forward to the remote machine

backward_common_files:
type: typing.List[str], optional, default: []
argument path: pep723/backward_common_files

Common files to backward from the remote machine

machine:
type: dict
argument path: pep723/machine

Machine configuration. See related documentation for details.

resources:
type: dict
argument path: pep723/resources

Resources configuration. See related documentation for details.

task_list:
type: list
argument path: pep723/task_list

List of tasks to execute.

This argument takes a list with each element containing the following:

command:
type: str, optional, default: python
argument path: pep723/task_list/command

Python interpreter or launcher. No need to contain the Python script filename.

task_work_path:
type: str
argument path: pep723/task_list/task_work_path

The dir where the command to be executed. Can be a glob pattern.

forward_files:
type: typing.List[str], optional, default: []
argument path: pep723/task_list/forward_files

The files to be uploaded in task_work_path before the task exectued.

backward_files:
type: typing.List[str], optional, default: []
argument path: pep723/task_list/backward_files

The files to be download to local_root in task_work_path after the task finished

outlog:
type: str | NoneType, optional, default: log
argument path: pep723/task_list/outlog

The out log file name. redirect from stdout

errlog:
type: str | NoneType, optional, default: err
argument path: pep723/task_list/errlog

The err log file name. redirect from stderr