horus-runtime
Core

Artifact

Artifact System

The Artifact System is part of the core execution mechanism of the Horus Runtime SDK.

What Is an Artifact?

An artifact represents a concrete unit of data produced or consumed by a task.

Every artifact is backed by a filesystem path. Regardless of what the artifact logically represents (a Python dict, a trained model, a dataset) the runtime always materializes it on disk as either a file or a directory. This is a fundamental invariant: the path field is not optional metadata, it is the canonical identity of the artifact: existence means the path exists on the target where the artifact lives (checked through the ArtifactStore, not by the artifact itself).

This constraint makes artifacts deterministic, cacheable, and transportable across machines without any special serialization protocol at the workflow level.

Examples of what artifacts can represent (and how they materialize):

  • Local files and folders (stored as-is)
  • Python dicts, lists, and other JSON-compatible objects (serialized to a .json file)
  • Arbitrary Python objects (serialized to a .pkl file via pickle)
  • Datasets and model checkpoints (stored as files or directories)
  • Remote objects such as S3 or HTTP resources (downloaded and cached locally before use)

Each artifact:

  • Has a stable ID. The id identifies the artifact within its task and is the handle that workflow edges wire together (a producer output to a consumer input). Tasks are not linked by matching ids — an explicit edge does the wiring — so a producer output and the consumer input it feeds may have different ids. Within a single task, input and output ids must be unique (see DAG planning and edges).
  • Has a path identifying its location on disk
  • Carries optional labels: a dict[str, str] of free-form metadata (labels: {subject: batch_017, role: measurement}). The runtime never interprets them and they play no part in caching; they exist for whatever reads the workflow afterwards, such as a UI grouping nodes or a lineage query.
  • Defines:
    • How to read its contents back into a Python object
    • How to write a Python object to its file representation
    • How to package / unpackage itself for transport, as portable shell commands (see pack_command / unpack_command below)

Filesystem lifecycle operations (existence checks, deletion, packaging) are not performed by the artifact itself. They run through the ArtifactStore against the target where the artifact physically lives, so they behave identically whether that is the orchestrator or a remote machine.

Defining Custom Artifacts

Artifacts inherit from BaseArtifact and are automatically registered using the runtime registry mechanism.

from horus_runtime.core.artifact.base import BaseArtifact

class MyArtifact(BaseArtifact[str]):
    kind = "my_artifact"
    kind_name = "My Artifact"
    kind_description = "A short, human-readable description of this kind."

    def read(self) -> str:
        with open(self.path) as f:
            return f.read()

    def write(self, value: str) -> None:
        with open(self.path, "w") as f:
            f.write(value)

BaseArtifact API and Required Implementations

BaseArtifact[T] is a generic, file-backed artifact abstraction:

  • Generic type: BaseArtifact[T] specifies the native Python type that read() returns and write() accepts.

  • Path normalization: Accepts both str and Path for path, always resolved to an absolute Path.

  • ID logic: Each artifact has a unique internal_id (UUID) and a user-friendly id (auto-set if not provided).

  • Event emission: Emits lifecycle events via the runtime event bus.

  • Required implementations:

    • read() -> T: Read and deserialize the artifact contents. Runs task-local, on the machine where the value is produced or consumed.
    • write(value: T) -> None: Write the native representation to disk (also task-local).
    • kind: str: Concrete discriminator value used for registry dispatch and type resolution.

  • Kind metadata (optional ClassVars):

    • kind_name: ClassVar[str]: short, human-readable name for the kind (e.g. "File"), used by client UIs, registries, and logging.
    • kind_description: ClassVar[str]: a longer description of the kind. Defaults to an empty string. For translatable text, prefer a plugin-scoped translator (see the SDK i18n guide).

  • Materialization hook (optional):

    • materialize() -> None: make the artifact's bytes available at path. The default is a no-op — a file-backed artifact assumes a producer task, or an upload, already wrote it. Value artifacts override it to write their inline value, which is what lets a workflow-owned root artifact run with no producer at all.

  • Transport hooks (override only when the artifact is not a single file):

    • pack_command(src, pkg) -> str | None: return a portable shell command that produces the single-file package pkg from the artifact materialized at src, or None when the artifact is already a single file (identity packaging — the base default).
    • unpack_command(pkg, dest) -> str | None: return a portable shell command that materializes the artifact at dest from the package pkg, or None for identity (the store simply moves the file into place).

    Both commands are executed on the target where the artifact lives (or is being materialized) by the ArtifactStore, so they must be POSIX-portable and reference only the given target-side paths. FileArtifact and other single-file artifacts inherit the identity defaults; FolderArtifact overrides them with tar commands.

ArtifactStore

An artifact is just a file-backed value description; where it lives, and how to check, delete, or transport it there, is owned by a target. The ArtifactStore (horus_runtime.core.artifact.store) is the mediator that binds an artifact to a target and performs its lifecycle operations through that target's filesystem primitives:

from horus_runtime.core.artifact.store import ArtifactStore

store = ArtifactStore(target)

await store.exists(artifact)                 # -> bool
await store.delete(artifact)                 # remove + emit delete event
package_path = await store.package(artifact) # build a single transferable file
await store.unpackage(artifact, package_path)

Because every operation runs through the target rather than local pathlib calls, the same code checks and moves an artifact whether it physically lives on the orchestrator or on a remote (e.g. SSH) machine. package() runs the artifact's pack_command on the target and returns the package path (or the artifact's own path unchanged for single-file, identity artifacts); unpackage() runs unpack_command on the destination, or simply moves the file into place for identity artifacts.

This package() is unrelated to the horus package CLI command, which zips a whole workflow for transport — see Packaging workflows.

ArtifactStore depends only on a small set of methods (path_on_target, path_exists, remove, run_command_sync, resolved_working_directory) that BaseTarget satisfies. See Target for those primitives.

Built-in Artifacts

The SDK provides the following artifact implementations:

FileArtifact

Reads and writes raw bytes. Suitable for any opaque file.

from horus_builtin.artifact.file import FileArtifact

FolderArtifact

Represents a directory. It overrides the transport hooks so a whole directory can move between targets as a single file: pack_command archives the folder's contents with tar czf … -C <src> . and unpack_command extracts them with tar xzf into a freshly recreated destination.

from horus_builtin.artifact.folder import FolderArtifact

JSONArtifact[T]

Serializes any JSON-compatible Python object (dict, list, str, etc.) to a .json file. The generic parameter T is used for type-checking only; no runtime validation is performed.

from horus_builtin.artifact.json import JSONArtifact

artifact = JSONArtifact[dict](path="/tmp/result.json")
artifact.write({"key": "value"})
data = artifact.read()  # dict, cast to T

PickleArtifact[T]

Serializes arbitrary Python objects using the pickle protocol.

from horus_builtin.artifact.pickle import PickleArtifact

artifact = PickleArtifact[list](path="/tmp/data.pkl")
artifact.write([1, 2, 3])
data = artifact.read()  # list

Warning: pickle is not secure against malformed or maliciously crafted data. Never unpickle data from untrusted sources.

Value artifacts

Most artifacts describe data that some task will produce. A value artifact describes data you author directly in the workflow — a sample count, a threshold, a feature flag — while still being a normal, file-backed artifact in every other respect.

ValueArtifact[T] adds one field to BaseArtifact[T]:

  • value: T — the artifact's authored value. It round-trips through model_dump, so a workflow document can carry a value whose bytes have not been written yet.

and overrides materialize() to write that value to path. The override is idempotent: once the file exists it is left alone, so an uploaded blob, or a value a producer task already wrote, is never clobbered by the declared one.

The graph does not distinguish the two cases. A NumberArtifact produced by a task is transferred to its consumer exactly like a NumberArtifact root whose value you typed. The only runtime difference is that a root value artifact has no producer, so transfer_artifacts calls materialize() on it the first time a consumer needs it — and copies the root's value onto the consumer's own input artifact, so that substitution sees the authored value rather than the field default.

NumberArtifact

An int | float, materialized as a JSON number. Defaults to 0.

from horus_builtin.artifact.number import NumberArtifact

samples = NumberArtifact(id="samples", path="samples.json", value=10)
samples.materialize()
samples.read()  # 10

BooleanArtifact

A bool, materialized as a JSON boolean (true / false). Defaults to False.

from horus_builtin.artifact.boolean import BooleanArtifact

verbose = BooleanArtifact(id="verbose", path="verbose.json", value=True)

StringArtifact

A str, materialized as plain text — the string as authored, with no quotes. Defaults to "".

from horus_builtin.artifact.string import StringArtifact

label = StringArtifact(id="label", path="label.txt", value="baseline")

ValueArtifact itself is abstract and is not registered as a kind; only number, boolean and string are. Subclass it to add your own — declare the concrete value type and the on-disk serialization through read/write, exactly as for any other artifact, and materialize() follows for free.

Value artifacts also change how $id renders in a command — see Artifact substitution.

Registering custom artifacts

To register and discover artifact plugins within the Horus runtime use the following entry point:

[project.entry-points."horus.artifact"]

For more details, refer to the AutoRegistry documentation.

Runtime SDK Overview

API reference for the Horus runtime SDK — artifacts, executors, runtimes, targets, tasks, and the plugin system.

Executor

Next Page

On this page

Artifact SystemWhat Is an Artifact?Defining Custom ArtifactsBaseArtifact API and Required ImplementationsArtifactStoreBuilt-in ArtifactsFileArtifactFolderArtifactJSONArtifact[T]PickleArtifact[T]Value artifactsNumberArtifactBooleanArtifactStringArtifactRegistering custom artifacts