initial commit: setup_pieline.sh

INSATLL.md

+PWD=`pwd`
+export PATH=$PATH:${PWD}/bin
\ No newline at end of file

README.md

+# IMAGE EXPORT
+
+important for image export:
+
+https://tuhrig.de/difference-between-save-and-export-in-docker/
+
+note to keep state after running drun
+
+docker ps -a to get all containers (also stoped ones)
+docker ps -a --last 1 --filter ancestor=${STEP_NAME}:latest --format {{.ID}}
+docker commit <container id> <image name>
+docker save --output <tar file> <image name> 
+
+## loading back
+
+docker load <tar file>
+docker run -it -v <artifacts folder>:/home/<user>/artifacts <image name> -c bash
+
+# INSTALL
+
+needs stoml (get binary fron git repo)
+
+need toml lib
+
+need docker, later k8 client
+
+# K8
+
+With kuberntes we could get rid of the run_pipeline scripts and use manifest instead.
+We need to use "kind: Jobs" (see doc: https://kubernetes.io/docs/concepts/workloads/controllers/job/)
+This would allow to run each step as a Job, either with single node or in parallel (not straight forward)
+With this mechanism one can require various resources and access the resources of the cluster.
+
+Need to make a prototype for it, and test various scenario:
+
+1. How to make a parallel Job (the constraint on the dev of the Step)
+2. How to use user defined images (image registry)
+3. How can we interact with the Jobs (Are all necessary feature there)
+4. How to get the results (artifacts, mounting volume...)
+5. How to archive the resulting images and conf together?
+
+
+# CONCEPT:
+
+in pipeline.toml you can find all the information to initialize the pipeline folder, and run the pipeline.
+- you can specify a hash for the archive if you don't want to build-it
+
+in pipeline folder you can find for each step all the source to build the image, build the archive and the runtime parameters
+
+in step-tree you can find the hierarchy of all runs, with first steps at upper layer and going deeper in the folder hierarchy.
+
+in runs you find the final results (final product) with everything for tracability and reusability.
+
+## BINS
+
+1. setup_pipeline.sh: make sure all step are in the pipeline folder, and checkout the provided commits, and generate hash for the runtime parameters and artifact-in
+2. build_pipeline.sh: make sure that all necessary images are present and generate their hash
+3. build_arch_volume.sh: make the arch volume for each step is build and generate the hash save the untar into archive/hash/
+4. run_step.sh: run a given step based on the hash of its parent step and the actual state of the pipeline/step folder and the hash of the archive
+5. When pipeline is done: the report form of the run is stored in runs under the hash of the md5sum of the ordered names (with hash) of the steps.
+
+## HASH magic
+
+We need to use SHA1 as a Hashing algo because it fits most our purpose:
+- sustainable
+- very low probability of collision
+- simple enough
+- plenty of implementations
+
+The run of each step produces an artifacts that will be named and stored in a folder named:
+
+`StepName - DepHash - StepHash - ContribHash`
+
+* StepName: The StepName is the name of the step in pipeline.toml
+* StepHash: the truncated (10 char) sha1 hash of: `BuildHash - RuntimeParamHash - ArchHash` where
+    * BuildHash is the sha1 hash of the build-image.tar
+    * RuntimeParamHash is the sha1 hash of the runtime-param.tar
+    * ArchHash is the sha1 hash of the archive.tar (may be several of them)
+
+ContribHash: is the truncated (10 char) sha1 hash of the Artifact-out volume (.tar)
+DepHash: is the truncated (10 char) sha1 hash of the sorted (ordered) full sha1 ContribHashes of all dependances
+
+Example:
+
+step-1:
+dep = []
+
+step-2:
+dep = [step-1]
+
+step-3:
+dep = [step-1]
+
+step-4:
+dep = [step-2, step-3]
+
+directory name of artifacts
+step-1_A_G_B
+step-2_B_H_C
+step-3_B_I_D
+step-4_E_J_F
+
+with hash(C_D) = E
+
+With this method one can continue a pipeline and
+or modify it witout rerunnning steps that were done already.
+
+examples:
+
+if now the hash of step-4 becomes K because
+the runtime parameters were changed the same pipeline will do the following:
+
+> run step-1_A_G_*
+already exists with artifact-out B
+> run step-2_B_H_*
+already exists with artifact-out C
+> run step-3_B_I_*
+already exists with artifact-out D
+> step-4 depends on step-3 and step-2 so hash(sorted(C, D)) = E
+> run step-3_E_K_*
+does not exists (different from step-3_E_J_*) so run it.
+artifact-out is G
+
+
+## Within the folder
+
+We need to provide the full-trace for each artifacts: from them one should be able to rebuild the
+folder name of all necessary artifacts.
+
+each step has dependances. each depandance is represented by an artifact volume with it's trace.
+The trace is a file with the list of steps already executed to end with the artifacts.
+
+artifact-step1:
+ - artifacts:
+   - artifact-init
+ - image: hash
+ - runtime-parameters: hash
+ - archive: hash
+
+artifact-step2:
+  - artifacts:
+    - artifact-step1
+  - image: hash
+  - runtime-parameters: hash
+  - archive: hash
+
+artifact-step3:
+  - artifacts:
+    - artifact-step1
+  - image: hash
+  - runtime-parameters: hash
+  - archive: hash
+
+artifact-step4:
+  - artifacts:
+    - artifact-step3
+    - artifact-step2
+  - image: hash
+  - runtime-parameters: hash
+  - archive: hash

bin/setup_pipeline.sh

+# READ pipeline config
+
+function checkout_commit() {
+    # args:
+    # PIPE_ROOT: The root path of the pipeline
+    # STEP_NAME: The name of the step to setup
+    # COMMIT: The commit to be used
+    PIPE_ROOT=$1
+    STEP_NAME=$2
+    COMMIT=$4
+
+    local STEPS_DIR=${PIPE_ROOT}/pipeline
+    local STEP_DIR=${STEPS_DIR}/${STEP_NAME}
+    
+    cd ${STEP_DIR}
+    git checkout ${COMMIT} > /dev/null 2>&1
+    CUR_COMMIT=`git rev-parse --short HEAD`
+    echo "The repo is now in commit: ${CUR_COMMIT}"
+
+    if [[ $? != 0 ]];
+    then
+	echo "ERROR: could not checkout the provided commit"
+	exit 1
+    fi;
+
+    cd ${PIPE_ROOT}
+
+}
+    
+function setup_step_from_git_repo() {
+    # args:
+    # PIPE_ROOT: The root path of the pipeline
+    # STEP_NAME: The name of the step to setup
+    # REPO_URL: The https url of the repo (must be public)
+    # COMMIT: The commit to be used
+    PIPE_ROOT=$1
+    STEP_NAME=$2
+    REPO_URL=$3
+    COMMIT=$4
+    
+    local STEPS_DIR="${PIPE_ROOT}/pipeline"
+    local STEP_DIR="${STEPS_DIR}/${STEP_NAME}"
+
+    echo ">>> Setup ${STEP_NAME} from ${REPO_URL}:${COMMIT}"
+
+    # If the step does not exists in the pipeline
+    if [ ! -d ${STEP_DIR} ];
+    then
+	cd ${STEPS_DIR}
+	git clone ${REPO_URL}
+	cd ${PIPE_ROOT}
+
+    fi;
+
+    # if git repo then Checkout commit
+    if [ -d ${STEP_DIR}/.git ];
+    then
+	checkout_commit ${PIPE_ROOT} ${STEP_NAME} ${COMMIT}
+
+    else
+	echo "ERROR: the source must be under Git version control"
+	
+    fi;
+
+}
+    
+
+PIPE_ROOT=`pwd`
+STEP_NAMES=(`./stoml ${PIPE_ROOT}/config.toml pipeline`)
+
+echo ">>> Steps: ${STEP_NAMES[@]}"
+
+for STEP_NAME in "${STEP_NAMES[@]}";
+do
+
+    TYPE=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.type`
+
+    case $TYPE in
+
+	git_repo)
+	    REPO_URL=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.repo_url`
+	    COMMIT=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.commit`
+
+	    setup_step_from_git_repo ${PIPE_ROOT} ${STEP_NAME} ${REPO_URL} ${COMMIT}
+
+	    ;;
+
+	*)
+	    echo "TYPE: ${TYPE} not supported."
+	    ;;
+    esac
+
+done;
+
+exit 0
+
+

config.toml

+pipeline = ["extract-features", "label-features", "select-meaningful-plates-and-features", "extract-features-properties"]
+
+[extract-features]
+
+type = "git_repo"
+repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/extract-features.git"
+commit = "1ccb3969"
+
+[label-features]
+
+type = "git_repo"
+repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/label-features.git"
+commit = "a4ce45b2"
+
+[select-meaningful-plates-and-features]
+
+type = "git_repo"
+repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/select-meaningful-plates-and-features.git"
+commit = "5fc5a7bc"
+
+[extract-features-properties]
+
+type = "git_repo"
+repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/extract-features-properties.git"
+commit = "e36591f9"
+
+

pipeline.toml

+[pipeline]
+
+order = ["extract-features",
+      	 "label-features",
+	 "select-meaningful-plates-and-features",
+	 "extract-features-properties"]
+
+[extract-features]
+
+src_git_repo = "..."
+src_commit = "..."
+
+arch_descr_git_repo = "..."
+arch_descr_commit = "..."
+

run_pipeline.sh

+#!/bin/bash
+
+# CHECK ARGS OPTIONS (TODO: serious options parser)
+RUN_HASH=$1
+
+
+# (RE)INIT the RUN
+if [[ ${RUN_HASH} != "" ]];
+then
+
+    if [[ -d run/${RUN_HASH} ]]; then
+
+	# GET STEP NAMES
+	STEP_NAMES=(`./stoml run/${RUN_HASH}/pipeline.toml pipeline`)
+
+    else
+
+	echo "run does not exists... --> EXIT"
+	exit 1
+
+    fi
+	
+else
+
+    # CREATE RUN DIRECTORY
+    HASH=`date | md5sum`
+    RUN_HASH="${HASH%% *}" # first element
+    mkdir run/${RUN_HASH}
+
+    # INITIALIZE RUN
+    cp -r artifacts run/${RUN_HASH}/
+    mkdir run/${RUN_HASH}/images
+
+    # GET STEP NAMES
+    STEP_NAMES=(`./stoml config.toml pipeline`)
+    
+fi
+
+echo ">>> run/${RUN_HASH}/"
+echo ">>> Steps: ${STEP_NAMES[@]}"
+
+# RUN THE REMAINING STEPS
+
+remaining_steps=("${STEP_NAMES[@]}") # copy into remaining_steps
+
+for STEP_NAME in "${STEP_NAMES[@]}";
+do
+    echo ">>> Start ${STEP_NAME}"
+
+    # Checkout the correct COMMIT HASH from Repo (TODO: read commit hash and checkout)
+    
+    # BUILD
+
+    BUILD_ENV_VARIABLES="--build-arg USER --build-arg UID=1000 --build-arg HOME"
+    HOST_ARTIFACTS_PATH="${PWD}/run/${RUN_HASH}/artifacts"
+
+    docker build ${BUILD_ENV_VARIABLES} -t ${STEP_NAME}:latest ./${STEP_NAME}
+
+    if [[ $? -ne 0 ]]; then
+	echo ">>> ERROR with build of STEP: ${STEP_NAME}"
+	exit 1
+    fi
+	
+    # EXECUTE
+
+    ## Get environement variables from config.toml
+    VAR_NAMES=`./stoml config.toml ${STEP_NAME}`
+
+    RUNTIME_ENV_VARIABLES=""
+
+    for VAR_NAME in $VAR_NAMES;
+    do
+	VAR_VALUE=`./stoml config.toml ${STEP_NAME}.${VAR_NAME}`
+	RUNTIME_ENV_VARIABLES="${RUNTIME_ENV_VARIABLES} --env ${VAR_NAME^^}=$VAR_VALUE"
+    done
+
+    echo ${RUNTIME_ENV_VARIABLES}
+
+    ## DEFINE further variables
+    DOCKER_ARTIFACTS_PATH=/home/${USER}/artifacts
+
+    ## execute the script
+    docker run -v ${HOST_ARTIFACTS_PATH}:${DOCKER_ARTIFACTS_PATH} ${RUNTIME_ENV_VARIABLES} ${STEP_NAME}:latest
+
+    ## if success save the image
+    if [[ $? -eq 0 ]]; then
+
+
+	## get the container id
+	CONTAINER_ID=`docker ps -a --last 1 --filter ancestor=${STEP_NAME}:latest --format {{.ID}}`
+
+	## Commit to a new image
+	docker commit $CONTAINER_ID ${STEP_NAME}:last_run
+
+	## Save the image to TAR
+	docker save --output run/${RUN_HASH}/images/${STEP_NAME}.tar ${STEP_NAME}:last_run
+	
+	## remove the step from pipeline.toml (set as DONE)
+	remaining_steps=("${remaining_steps[@]:1}")
+
+	## update pipeline.toml
+	size=${#remaining_steps[@]}
+
+	printf 'pipeline=['
+	printf '%s' "${remaining_steps[0]}" > run/${RUN_HASH}/pipeline.toml
+	if [[ size -gt 1 ]]; then
+	    printf ',%s' "${remaining_steps[@]:1}" >> run/${RUN_HASH}/pipeline.toml
+	fi
+	printf ']'
+
+    else
+	echo ">>> ERROR at STEP: ${STEP_NAME}"
+	exit 1
+    fi
+    
+done