cctools
|
Public Member Functions | |
__init__ (self, func, *args, **kwargs) | |
Creates a new python task. | |
submit_finalize (self) | |
Finalizes the task definition once the manager that will execute is run. | |
enable_temp_output (self) | |
Marks the output of this task to stay at the worker. | |
set_output_cache (self, cache=False) | |
Set the cache behavior for the output of the task. | |
output_file (self) | |
Returns the ndcctools.taskvine.file.File object that represents the output of this task. | |
output (self) | |
returns the result of a python task as a python variable | |
disable_output_serialization (self) | |
Disables serialization of results to disk when writing to a file for transmission. | |
![]() | |
clone (self) | |
Return a copy of this task. | |
set_command (self, command) | |
Set the command to be executed by the task. | |
set_library_required (self, library) | |
Set the name of the library at the worker that should execute the task's command. | |
get_library_required (self) | |
Get the name of the library at the worker that should execute the task's command. | |
needs_library (self, library) | |
Deprecated, see set_library_required. | |
set_library_provided (self, library_name) | |
Set the library name provided by this task. | |
get_libray_provided (self) | |
Get the name of the library at the worker that should execute the task's command. | |
provides_library (self, library) | |
Deprecated, see set_library_provided. | |
set_function_slots (self, nslots) | |
Set the number of concurrent functions a library can run. | |
set_function_exec_mode_from_string (self, exec_mode) | |
Set the execution mode of functions in a library. | |
set_scheduler (self, scheduler) | |
Set the worker selection scheduler for task. | |
set_tag (self, tag) | |
Attach a user defined logical name to the task. | |
set_category (self, name) | |
Label the task with the given category. | |
add_feature (self, name) | |
Label the task with the given user-defined feature. | |
add_input (self, file, remote_name, strict_input=False, mount_symlink=False) | |
Add any input object to a task. | |
add_output (self, file, remote_name, watch=False, failure_only=None, success_only=None) | |
Add any output object to a task. | |
set_snapshot_file (self, filename) | |
When monitoring, indicates a json-encoded file that instructs the monitor to take a snapshot of the task resources. | |
add_starch_package (self, file) | |
Add a Starch package as an execution context. | |
add_poncho_package (self, file) | |
Add a Poncho package as an execution context. | |
add_execution_context (self, f) | |
Adds an execution context to the task. | |
set_retries (self, max_retries) | |
Indicate the number of times the task should be retried. | |
set_max_forsaken (self, max_forsaken) | |
Indicate the number of times the task can be returned to the manager without being executed. | |
set_cores (self, cores) | |
Indicate the number of cores required by this task. | |
set_memory (self, memory) | |
Indicate the memory (in MB) required by this task. | |
set_disk (self, disk) | |
Indicate the disk space (in MB) required by this task. | |
set_gpus (self, gpus) | |
Indicate the number of GPUs required by this task. | |
set_priority (self, priority) | |
Indicate the the priority of this task (larger means better priority, default is 0). | |
set_env_var (self, name, value=None) | |
Set this environment variable before running the task. | |
set_monitor_output (self, directory) | |
Set a name for the resource summary output directory from the monitor. | |
tag (self) | |
Get the user-defined logical name for the task. | |
category (self) | |
Get the category name for the task. | |
command (self) | |
Get the shell command executed by the task. | |
state (self) | |
Get the state of the task. | |
std_output (self) | |
Get the standard output of the task. | |
id (self) | |
Get the task id number. | |
exit_code (self) | |
Get the exit code of the command executed by the task. | |
result (self) | |
Return a string that explains the result of a task. | |
completed (self) | |
Return True if task executed and its command terminated normally. | |
successful (self) | |
Return True if task executed successfully, (i.e. | |
get_metric (self, name) | |
Return various integer performance metrics about a completed task. | |
addrport (self) | |
Get the address and port of the host on which the task ran. | |
hostname (self) | |
Get the address and port of the host on which the task ran. | |
resources_measured (self) | |
Get the resources measured for the task execution if resource monitoring is enabled. | |
limits_exceeded (self) | |
Get the resources the task exceeded. | |
resources_requested (self) | |
Get the resources the task requested to run. | |
resources_allocated (self) | |
Get the resources allocated to the task in its latest attempt. | |
add_nopen (self, manager) | |
Adds inputs for nopen library and rules file and sets LD_PRELOAD. | |
Additional Inherited Members | |
![]() | |
![]() | |
_compute_library_name (self, library) | |
Compute the name of a given library. | |
![]() | |
![]() |
ndcctools.taskvine.task.PythonTask.__init__ | ( | self, | |
func, | |||
* | args, | ||
** | kwargs ) |
Creates a new python task.
self | Reference to the current python task object |
func | python function to be executed by task |
args | arguments used in function to be executed by task |
kwargs | keyword arguments used in function to be executed by task |
Reimplemented from ndcctools.taskvine.task.Task.
Reimplemented in ndcctools.taskvine.dask_executor.FunctionCallDask, ndcctools.taskvine.dask_executor.PythonTaskDask, ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.
ndcctools.taskvine.task.PythonTask.submit_finalize | ( | self | ) |
Finalizes the task definition once the manager that will execute is run.
This function is run by the manager before registering the task for execution.
self | Reference to the current python task object |
manager | Manager to which the task was submitted |
Reimplemented from ndcctools.taskvine.task.Task.
Reimplemented in ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.
ndcctools.taskvine.task.PythonTask.enable_temp_output | ( | self | ) |
Marks the output of this task to stay at the worker.
Functions that consume the output of this tasks have to add self.output_file as an input, and cloudpickle.load() it.
E.g.:
where fn_with_tmp may look something like this:
self | Reference to the current python task object |
ndcctools.taskvine.task.PythonTask.set_output_cache | ( | self, | |
cache = False ) |
Set the cache behavior for the output of the task.
cache | If True or 'workflow', cache the file at workers for reuse until the end of the workflow. If 'always', the file is cache until the end-of-life of the worker. Default is False (file is not cache). |
ndcctools.taskvine.task.PythonTask.output_file | ( | self | ) |
Returns the ndcctools.taskvine.file.File object that represents the output of this task.
ndcctools.taskvine.task.PythonTask.output | ( | self | ) |
returns the result of a python task as a python variable
self | reference to the current python task object |
Reimplemented from ndcctools.taskvine.task.Task.
Reimplemented in ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.
ndcctools.taskvine.task.PythonTask.disable_output_serialization | ( | self | ) |
Disables serialization of results to disk when writing to a file for transmission.
WARNING: Only do this if the function itself encodes the output in a way amenable for serialization.
self | Reference to the current python task object |