ComponentSpec Reference Guide

The ComponentSpec format is the standard specification for defining reusable machine learning pipeline components. This guide covers all aspects of the ComponentSpec schema, from basic structure to advanced features.

Overview

A ComponentSpec defines a reusable component that can be executed as part of a machine learning pipeline. A container component follows this structure:

<ComponentSpec>:name: stringdescription: stringmetadata:annotations: Map[string, any]inputs: #InputSpec[]<InputSpec[]>:name: stringtype: TypeSpecTypedescription: stringdefault: stringoptional: booleanannotations: Map[string, any]outputs: #OutputSpec[]<OutputSpec[]>:name: stringtype: TypeSpecTypedescription: stringannotations: Map[string, any]implementation: #ImplementationType<oneOf>:<ContainerImplementation>:container: #ContainerSpecimage: StringOrPlaceholdercommand: #StringOrPlaceholder[]<oneOf>:string:<InputValuePlaceholder>:inputValue: string<InputPathPlaceholder>:inputPath: string<OutputPathPlaceholder>:outputPath: string<ConcatPlaceholder>:concat: StringOrPlaceholder[]<IfPlaceholder>:if:cond: IfConditionArgumentTypethen: StringOrPlaceholder[]else: StringOrPlaceholder[]args: StringOrPlaceholder[]env: Map[str, StringOrPlaceholder]<GraphImplementation>:graph: #GraphSpecoutputValues: Map string -> TaskOutputArgumenttasks: #Map string -> TaskSpec<TaskSpec>:componentRef: #ComponentReferencename: stringdigest: stringtag: stringurl: stringspec: ComponentSpecarguments: #Map string -> ArgumentType<ArgumentType>:<oneOf>:string:<GraphInputArgument>:graphInput:inputName: stringtype: TypeSpecType<TaskOutputArgument>:taskOutput:taskId: stringoutputName: stringtype: TypeSpecTypeisEnabled: PredicateTypeexecutionOptions: #ExecutionOptionsSpecretryStrategy: #RetryStrategySpecmaxRetries: integercachingStrategy: #CachingStrategySpecmaxCacheStaleness: stringannotations: Map[string, any]

Component Specification Schema Reference

This document describes all definitions in the Component Specification JSON Schema.

note

You can see original ComponentSpec in https://github.com/Cloud-Pipelines/component_spec_schema/blob/master/component_spec.json_schema.json

TypeSpecType

Defines the type system for component inputs and outputs.

Property	Type	Required	Description
-	string | object	-	Either a string type name or an object with nested type specifications

InputSpec

Describes the component input specification.

Property	Type	Required	Description
name	string	Yes	Name of the input
type	TypeSpecType	No	Type specification for the input
description	string	No	Description of the input
default	string	No	Default value for the input
optional	boolean	No	Whether the input is optional (default: false)
annotations	object	No	Additional metadata annotations

OutputSpec

Describes the component output specification.

Property	Type	Required	Description
name	string	Yes	Name of the output
type	TypeSpecType	No	Type specification for the output
description	string	No	Description of the output
annotations	object	No	Additional metadata annotations

InputValuePlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by the input argument value.

Property	Type	Required	Description
inputValue	string	Yes	Name of the input

InputPathPlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by a local file path pointing to a file containing the input argument value.

Property	Type	Required	Description
inputPath	string	Yes	Name of the input

OutputPathPlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by a local file path pointing to a file where the program should write its output data.

Property	Type	Required	Description
outputPath	string	Yes	Name of the output

StringOrPlaceholder

Union type that can be one of several placeholder types or a string.

Type Options
string
InputValuePlaceholder
InputPathPlaceholder
OutputPathPlaceholder
ConcatPlaceholder
IfPlaceholder

ConcatPlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by the concatenated values of its items.

Property	Type	Required	Description
concat	array of StringOrPlaceholder	Yes	Items to concatenate

IsPresentPlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by a boolean value specifying whether the caller has passed an argument for the specified optional input.

Property	Type	Required	Description
isPresent	string	No	Name of the input

IfConditionArgumentType

Union type for conditional arguments.

Type Options
IsPresentPlaceholder
boolean
string
InputValuePlaceholder

ListOfStringsOrPlaceholders

Array type definition.

Type	Description
array of StringOrPlaceholder	Array containing strings or placeholder objects

IfPlaceholder

Represents the command-line argument placeholder that will be replaced at run-time by a boolean value specifying whether the caller has passed an argument for the specified optional input.

Property	Type	Required	Description
if	object	Yes	Conditional structure
if.cond	IfConditionArgumentType	Yes	Condition to evaluate
if.then	ListOfStringsOrPlaceholders	Yes	Values to use when condition is true
if.else	ListOfStringsOrPlaceholders	No	Values to use when condition is false

ContainerSpec

Defines container execution specification.

Property	Type	Required	Description
image	StringOrPlaceholder	Yes	Docker image name
command	array of StringOrPlaceholder	No	Entrypoint array. Not executed within a shell. The docker image's ENTRYPOINT is used if this is not provided
args	array of StringOrPlaceholder	No	Arguments to the entrypoint. The docker image's CMD is used if this is not provided
env	object with StringOrPlaceholder values	No	List of environment variables to set in the container

ContainerImplementation

Represents the container component implementation.

Property	Type	Required	Description
container	ContainerSpec	Yes	Container specification

ImplementationType

Union type for component implementations.

Type Options
ContainerImplementation
GraphImplementation

MetadataSpec

Metadata specification for components.

Property	Type	Required	Description
annotations	object	No	Additional metadata annotations

ComponentSpec

Component specification. Describes the metadata (name, description, source), the interface (inputs and outputs) and the implementation of the component.

Property	Type	Required	Description
name	string	No	Component name
description	string	No	Component description
inputs	array of InputSpec	No	Component input specifications
outputs	array of OutputSpec	No	Component output specifications
implementation	ImplementationType	Yes	Component implementation
metadata	MetadataSpec	No	Component metadata

ComponentReference

Component reference. Contains information that can be used to locate and load a component by name, digest or URL.

Property	Type	Required	Description
name	string	No	Component name
digest	string	No	Component digest
tag	string	No	Component tag
url	string	No	Component URL
spec	ComponentSpec	No	Inline component specification

GraphInputArgument

Represents the component argument value that comes from the graph component input.

Property	Type	Required	Description
graphInput	object	Yes	References the input of the graph/pipeline
graphInput.inputName	string	Yes	Name of the graph input
graphInput.type	TypeSpecType	No	Type specification

TaskOutputArgument

Represents the component argument value that comes from the output of a sibling task.

Property	Type	Required	Description
taskOutput	object	Yes	References the output of a sibling task
taskOutput.taskId	string	Yes	ID of the task
taskOutput.outputName	string	Yes	Name of the output
taskOutput.type	TypeSpecType	No	Type specification

ArgumentType

Union type for task arguments.

Type Options
string
GraphInputArgument
TaskOutputArgument

TwoArgumentOperands

Pair of operands for a binary operation.

Property	Type	Required	Description
op1	ArgumentType	Yes	First operand
op2	ArgumentType	Yes	Second operand

TwoLogicalOperands

Pair of operands for a binary logical operation.

Property	Type	Required	Description
op1	PredicateType	Yes	First logical operand
op2	PredicateType	Yes	Second logical operand

PredicateType

Union type for predicate expressions.

Operator	Operand Type	Description
==	TwoArgumentOperands	Equality comparison
!=	TwoArgumentOperands	Inequality comparison
>	TwoArgumentOperands	Greater than comparison
>=	TwoArgumentOperands	Greater than or equal comparison
<	TwoArgumentOperands	Less than comparison
<=	TwoArgumentOperands	Less than or equal comparison
and	TwoLogicalOperands	Logical AND operation
or	TwoLogicalOperands	Logical OR operation
not	PredicateType	Logical NOT operation

RetryStrategySpec

Optional configuration that specifies how the task should be retried if it fails.

Property	Type	Required	Description
maxRetries	integer	No	Maximum number of retry attempts

CachingStrategySpec

Optional configuration that specifies how the task execution may be skipped if the output data exist in cache.

Property	Type	Required	Description
maxCacheStaleness	string (duration format)	No	Maximum age of cached data to consider valid

ExecutionOptionsSpec

Optional configuration that specifies how the task should be executed. Can be used to set some platform-specific options.

Property	Type	Required	Description
retryStrategy	RetryStrategySpec	No	Retry configuration
cachingStrategy	CachingStrategySpec	No	Caching configuration

TaskSpec

Task specification. Task is a configured component - a component supplied with arguments and other applied configuration changes.

Property	Type	Required	Description
componentRef	ComponentReference	Yes	Reference to the component
arguments	object with ArgumentType values	No	Arguments to pass to the component
isEnabled	PredicateType	No	Condition for enabling the task
executionOptions	ExecutionOptionsSpec	No	Execution configuration
annotations	object	No	Additional metadata annotations

GraphSpec

Describes the graph component implementation. It represents a graph of component tasks connected to the upstream sources of data using the argument specifications. It also describes the sources of graph output values.

Property	Type	Required	Description
tasks	object with TaskSpec values	Yes	Tasks in the graph
outputValues	object with TaskOutputArgument values	No	Output value mappings

GraphImplementation

Represents the graph component implementation.

Property	Type	Required	Description
graph	GraphSpec	Yes	Graph specification

PipelineRunSpec

The object that can be sent to the backend to start a new Run.

Property	Type	Required	Description
rootTask	TaskSpec	Yes	Main task to execute
onExitTask	TaskSpec	No	Task to execute on exit