cctools
ndcctools.taskvine.task.PythonTask Class Reference
Inheritance diagram for ndcctools.taskvine.task.PythonTask:
ndcctools.taskvine.task.Task ndcctools.taskvine.dask_executor.PythonTaskDask ndcctools.taskvine.futures.FuturePythonTask ndcctools.taskvine.task.FunctionCall ndcctools.taskvine.dask_executor.FunctionCallDask ndcctools.taskvine.futures.FutureFunctionCall

Public Member Functions

 __init__ (self, func, *args, **kwargs)
 Creates a new python task.
 
 submit_finalize (self)
 Finalizes the task definition once the manager that will execute is run.
 
 enable_temp_output (self)
 Marks the output of this task to stay at the worker.
 
 set_output_cache (self, cache=False)
 Set the cache behavior for the output of the task.
 
 output_file (self)
 Returns the ndcctools.taskvine.file.File object that represents the output of this task.
 
 output (self)
 returns the result of a python task as a python variable
 
 disable_output_serialization (self)
 Disables serialization of results to disk when writing to a file for transmission.
 
- Public Member Functions inherited from ndcctools.taskvine.task.Task
 clone (self)
 Return a copy of this task.
 
 set_command (self, command)
 Set the command to be executed by the task.
 
 set_library_required (self, library)
 Set the name of the library at the worker that should execute the task's command.
 
 get_library_required (self)
 Get the name of the library at the worker that should execute the task's command.
 
 needs_library (self, library)
 Deprecated, see set_library_required.
 
 set_library_provided (self, library_name)
 Set the library name provided by this task.
 
 get_libray_provided (self)
 Get the name of the library at the worker that should execute the task's command.
 
 provides_library (self, library)
 Deprecated, see set_library_provided.
 
 set_function_slots (self, nslots)
 Set the number of concurrent functions a library can run.
 
 set_function_exec_mode_from_string (self, exec_mode)
 Set the execution mode of functions in a library.
 
 set_scheduler (self, scheduler)
 Set the worker selection scheduler for task.
 
 set_tag (self, tag)
 Attach a user defined logical name to the task.
 
 set_category (self, name)
 Label the task with the given category.
 
 add_feature (self, name)
 Label the task with the given user-defined feature.
 
 add_input (self, file, remote_name, strict_input=False, mount_symlink=False)
 Add any input object to a task.
 
 add_output (self, file, remote_name, watch=False, failure_only=None, success_only=None)
 Add any output object to a task.
 
 set_snapshot_file (self, filename)
 When monitoring, indicates a json-encoded file that instructs the monitor to take a snapshot of the task resources.
 
 add_starch_package (self, file)
 Add a Starch package as an execution context.
 
 add_poncho_package (self, file)
 Add a Poncho package as an execution context.
 
 add_execution_context (self, f)
 Adds an execution context to the task.
 
 set_retries (self, max_retries)
 Indicate the number of times the task should be retried.
 
 set_max_forsaken (self, max_forsaken)
 Indicate the number of times the task can be returned to the manager without being executed.
 
 set_cores (self, cores)
 Indicate the number of cores required by this task.
 
 set_memory (self, memory)
 Indicate the memory (in MB) required by this task.
 
 set_disk (self, disk)
 Indicate the disk space (in MB) required by this task.
 
 set_gpus (self, gpus)
 Indicate the number of GPUs required by this task.
 
 set_priority (self, priority)
 Indicate the the priority of this task (larger means better priority, default is 0).
 
 set_env_var (self, name, value=None)
 Set this environment variable before running the task.
 
 set_monitor_output (self, directory)
 Set a name for the resource summary output directory from the monitor.
 
 tag (self)
 Get the user-defined logical name for the task.
 
 category (self)
 Get the category name for the task.
 
 command (self)
 Get the shell command executed by the task.
 
 state (self)
 Get the state of the task.
 
 std_output (self)
 Get the standard output of the task.
 
 id (self)
 Get the task id number.
 
 exit_code (self)
 Get the exit code of the command executed by the task.
 
 result (self)
 Return a string that explains the result of a task.
 
 completed (self)
 Return True if task executed and its command terminated normally.
 
 successful (self)
 Return True if task executed successfully, (i.e.
 
 get_metric (self, name)
 Return various integer performance metrics about a completed task.
 
 addrport (self)
 Get the address and port of the host on which the task ran.
 
 hostname (self)
 Get the address and port of the host on which the task ran.
 
 resources_measured (self)
 Get the resources measured for the task execution if resource monitoring is enabled.
 
 limits_exceeded (self)
 Get the resources the task exceeded.
 
 resources_requested (self)
 Get the resources the task requested to run.
 
 resources_allocated (self)
 Get the resources allocated to the task in its latest attempt.
 
 add_nopen (self, manager)
 Adds inputs for nopen library and rules file and sets LD_PRELOAD.
 

Additional Inherited Members

- Data Fields inherited from ndcctools.taskvine.task.Task
- Protected Member Functions inherited from ndcctools.taskvine.task.Task
 _compute_library_name (self, library)
 Compute the name of a given library.
 
- Static Protected Member Functions inherited from ndcctools.taskvine.task.Task
- Protected Attributes inherited from ndcctools.taskvine.task.Task

Constructor & Destructor Documentation

◆ __init__()

ndcctools.taskvine.task.PythonTask.__init__ ( self,
func,
* args,
** kwargs )

Creates a new python task.

Parameters
selfReference to the current python task object
funcpython function to be executed by task
argsarguments used in function to be executed by task
kwargskeyword arguments used in function to be executed by task

Reimplemented from ndcctools.taskvine.task.Task.

Reimplemented in ndcctools.taskvine.dask_executor.FunctionCallDask, ndcctools.taskvine.dask_executor.PythonTaskDask, ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.

Member Function Documentation

◆ submit_finalize()

ndcctools.taskvine.task.PythonTask.submit_finalize ( self)

Finalizes the task definition once the manager that will execute is run.

This function is run by the manager before registering the task for execution.

Parameters
selfReference to the current python task object
managerManager to which the task was submitted

Reimplemented from ndcctools.taskvine.task.Task.

Reimplemented in ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.

◆ enable_temp_output()

ndcctools.taskvine.task.PythonTask.enable_temp_output ( self)

Marks the output of this task to stay at the worker.

Functions that consume the output of this tasks have to add self.output_file as an input, and cloudpickle.load() it.

E.g.:

ta = PythonTask(fn, ...)
ta.enable_temp_output()
tid = m.submit(ta)
t = m.wait(...)
if t.id == tid:
tb = PythonTask(fn_with_tmp, "ta_output.file")
tb.add_input(ta.output_file, "ta_output.file")
m.submit(tb)
TaskVine PythonTask object.

where fn_with_tmp may look something like this:

def fn_with_tmp(filename):
import cloudpickle
with open(filename) as f:
data = cloudpickle.load(f)
Parameters
selfReference to the current python task object

◆ set_output_cache()

ndcctools.taskvine.task.PythonTask.set_output_cache ( self,
cache = False )

Set the cache behavior for the output of the task.

Parameters
cacheIf True or 'workflow', cache the file at workers for reuse until the end of the workflow. If 'always', the file is cache until the end-of-life of the worker. Default is False (file is not cache).

◆ output_file()

ndcctools.taskvine.task.PythonTask.output_file ( self)

Returns the ndcctools.taskvine.file.File object that represents the output of this task.

◆ output()

ndcctools.taskvine.task.PythonTask.output ( self)

returns the result of a python task as a python variable

Parameters
selfreference to the current python task object

Reimplemented from ndcctools.taskvine.task.Task.

Reimplemented in ndcctools.taskvine.futures.FutureFunctionCall, ndcctools.taskvine.futures.FuturePythonTask, and ndcctools.taskvine.task.FunctionCall.

◆ disable_output_serialization()

ndcctools.taskvine.task.PythonTask.disable_output_serialization ( self)

Disables serialization of results to disk when writing to a file for transmission.

WARNING: Only do this if the function itself encodes the output in a way amenable for serialization.

Parameters
selfReference to the current python task object

The documentation for this class was generated from the following file: