Commit 8befc741 authored by Yori Fournier's avatar Yori Fournier
Browse files

initial commit: setup_pieline.sh

parents
PWD=`pwd`
export PATH=$PATH:${PWD}/bin
\ No newline at end of file
# IMAGE EXPORT
important for image export:
https://tuhrig.de/difference-between-save-and-export-in-docker/
note to keep state after running drun
docker ps -a to get all containers (also stoped ones)
docker ps -a --last 1 --filter ancestor=${STEP_NAME}:latest --format {{.ID}}
docker commit <container id> <image name>
docker save --output <tar file> <image name>
## loading back
docker load <tar file>
docker run -it -v <artifacts folder>:/home/<user>/artifacts <image name> -c bash
# INSTALL
needs stoml (get binary fron git repo)
need toml lib
need docker, later k8 client
# K8
With kuberntes we could get rid of the run_pipeline scripts and use manifest instead.
We need to use "kind: Jobs" (see doc: https://kubernetes.io/docs/concepts/workloads/controllers/job/)
This would allow to run each step as a Job, either with single node or in parallel (not straight forward)
With this mechanism one can require various resources and access the resources of the cluster.
Need to make a prototype for it, and test various scenario:
1. How to make a parallel Job (the constraint on the dev of the Step)
2. How to use user defined images (image registry)
3. How can we interact with the Jobs (Are all necessary feature there)
4. How to get the results (artifacts, mounting volume...)
5. How to archive the resulting images and conf together?
# CONCEPT:
in pipeline.toml you can find all the information to initialize the pipeline folder, and run the pipeline.
- you can specify a hash for the archive if you don't want to build-it
in pipeline folder you can find for each step all the source to build the image, build the archive and the runtime parameters
in step-tree you can find the hierarchy of all runs, with first steps at upper layer and going deeper in the folder hierarchy.
in runs you find the final results (final product) with everything for tracability and reusability.
## BINS
1. setup_pipeline.sh: make sure all step are in the pipeline folder, and checkout the provided commits, and generate hash for the runtime parameters and artifact-in
2. build_pipeline.sh: make sure that all necessary images are present and generate their hash
3. build_arch_volume.sh: make the arch volume for each step is build and generate the hash save the untar into archive/hash/
4. run_step.sh: run a given step based on the hash of its parent step and the actual state of the pipeline/step folder and the hash of the archive
5. When pipeline is done: the report form of the run is stored in runs under the hash of the md5sum of the ordered names (with hash) of the steps.
## HASH magic
We need to use SHA1 as a Hashing algo because it fits most our purpose:
- sustainable
- very low probability of collision
- simple enough
- plenty of implementations
The run of each step produces an artifacts that will be named and stored in a folder named:
`StepName - DepHash - StepHash - ContribHash`
* StepName: The StepName is the name of the step in pipeline.toml
* StepHash: the truncated (10 char) sha1 hash of: `BuildHash - RuntimeParamHash - ArchHash` where
* BuildHash is the sha1 hash of the build-image.tar
* RuntimeParamHash is the sha1 hash of the runtime-param.tar
* ArchHash is the sha1 hash of the archive.tar (may be several of them)
ContribHash: is the truncated (10 char) sha1 hash of the Artifact-out volume (.tar)
DepHash: is the truncated (10 char) sha1 hash of the sorted (ordered) full sha1 ContribHashes of all dependances
Example:
step-1:
dep = []
step-2:
dep = [step-1]
step-3:
dep = [step-1]
step-4:
dep = [step-2, step-3]
directory name of artifacts
step-1_A_G_B
step-2_B_H_C
step-3_B_I_D
step-4_E_J_F
with hash(C_D) = E
With this method one can continue a pipeline and
or modify it witout rerunnning steps that were done already.
examples:
if now the hash of step-4 becomes K because
the runtime parameters were changed the same pipeline will do the following:
> run step-1_A_G_*
already exists with artifact-out B
> run step-2_B_H_*
already exists with artifact-out C
> run step-3_B_I_*
already exists with artifact-out D
> step-4 depends on step-3 and step-2 so hash(sorted(C, D)) = E
> run step-3_E_K_*
does not exists (different from step-3_E_J_*) so run it.
artifact-out is G
## Within the folder
We need to provide the full-trace for each artifacts: from them one should be able to rebuild the
folder name of all necessary artifacts.
each step has dependances. each depandance is represented by an artifact volume with it's trace.
The trace is a file with the list of steps already executed to end with the artifacts.
artifact-step1:
- artifacts:
- artifact-init
- image: hash
- runtime-parameters: hash
- archive: hash
artifact-step2:
- artifacts:
- artifact-step1
- image: hash
- runtime-parameters: hash
- archive: hash
artifact-step3:
- artifacts:
- artifact-step1
- image: hash
- runtime-parameters: hash
- archive: hash
artifact-step4:
- artifacts:
- artifact-step3
- artifact-step2
- image: hash
- runtime-parameters: hash
- archive: hash
# READ pipeline config
function checkout_commit() {
# args:
# PIPE_ROOT: The root path of the pipeline
# STEP_NAME: The name of the step to setup
# COMMIT: The commit to be used
PIPE_ROOT=$1
STEP_NAME=$2
COMMIT=$4
local STEPS_DIR=${PIPE_ROOT}/pipeline
local STEP_DIR=${STEPS_DIR}/${STEP_NAME}
cd ${STEP_DIR}
git checkout ${COMMIT} > /dev/null 2>&1
CUR_COMMIT=`git rev-parse --short HEAD`
echo "The repo is now in commit: ${CUR_COMMIT}"
if [[ $? != 0 ]];
then
echo "ERROR: could not checkout the provided commit"
exit 1
fi;
cd ${PIPE_ROOT}
}
function setup_step_from_git_repo() {
# args:
# PIPE_ROOT: The root path of the pipeline
# STEP_NAME: The name of the step to setup
# REPO_URL: The https url of the repo (must be public)
# COMMIT: The commit to be used
PIPE_ROOT=$1
STEP_NAME=$2
REPO_URL=$3
COMMIT=$4
local STEPS_DIR="${PIPE_ROOT}/pipeline"
local STEP_DIR="${STEPS_DIR}/${STEP_NAME}"
echo ">>> Setup ${STEP_NAME} from ${REPO_URL}:${COMMIT}"
# If the step does not exists in the pipeline
if [ ! -d ${STEP_DIR} ];
then
cd ${STEPS_DIR}
git clone ${REPO_URL}
cd ${PIPE_ROOT}
fi;
# if git repo then Checkout commit
if [ -d ${STEP_DIR}/.git ];
then
checkout_commit ${PIPE_ROOT} ${STEP_NAME} ${COMMIT}
else
echo "ERROR: the source must be under Git version control"
fi;
}
PIPE_ROOT=`pwd`
STEP_NAMES=(`./stoml ${PIPE_ROOT}/config.toml pipeline`)
echo ">>> Steps: ${STEP_NAMES[@]}"
for STEP_NAME in "${STEP_NAMES[@]}";
do
TYPE=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.type`
case $TYPE in
git_repo)
REPO_URL=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.repo_url`
COMMIT=`./stoml ${PIPE_ROOT}/config.toml ${STEP_NAME}.commit`
setup_step_from_git_repo ${PIPE_ROOT} ${STEP_NAME} ${REPO_URL} ${COMMIT}
;;
*)
echo "TYPE: ${TYPE} not supported."
;;
esac
done;
exit 0
pipeline = ["extract-features", "label-features", "select-meaningful-plates-and-features", "extract-features-properties"]
[extract-features]
type = "git_repo"
repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/extract-features.git"
commit = "1ccb3969"
[label-features]
type = "git_repo"
repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/label-features.git"
commit = "a4ce45b2"
[select-meaningful-plates-and-features]
type = "git_repo"
repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/select-meaningful-plates-and-features.git"
commit = "5fc5a7bc"
[extract-features-properties]
type = "git_repo"
repo_url = "https://gitlab.aip.de/ml_solar_plates/modern_pipeline/extract-features-properties.git"
commit = "e36591f9"
[pipeline]
order = ["extract-features",
"label-features",
"select-meaningful-plates-and-features",
"extract-features-properties"]
[extract-features]
src_git_repo = "..."
src_commit = "..."
arch_descr_git_repo = "..."
arch_descr_commit = "..."
#!/bin/bash
# CHECK ARGS OPTIONS (TODO: serious options parser)
RUN_HASH=$1
# (RE)INIT the RUN
if [[ ${RUN_HASH} != "" ]];
then
if [[ -d run/${RUN_HASH} ]]; then
# GET STEP NAMES
STEP_NAMES=(`./stoml run/${RUN_HASH}/pipeline.toml pipeline`)
else
echo "run does not exists... --> EXIT"
exit 1
fi
else
# CREATE RUN DIRECTORY
HASH=`date | md5sum`
RUN_HASH="${HASH%% *}" # first element
mkdir run/${RUN_HASH}
# INITIALIZE RUN
cp -r artifacts run/${RUN_HASH}/
mkdir run/${RUN_HASH}/images
# GET STEP NAMES
STEP_NAMES=(`./stoml config.toml pipeline`)
fi
echo ">>> run/${RUN_HASH}/"
echo ">>> Steps: ${STEP_NAMES[@]}"
# RUN THE REMAINING STEPS
remaining_steps=("${STEP_NAMES[@]}") # copy into remaining_steps
for STEP_NAME in "${STEP_NAMES[@]}";
do
echo ">>> Start ${STEP_NAME}"
# Checkout the correct COMMIT HASH from Repo (TODO: read commit hash and checkout)
# BUILD
BUILD_ENV_VARIABLES="--build-arg USER --build-arg UID=1000 --build-arg HOME"
HOST_ARTIFACTS_PATH="${PWD}/run/${RUN_HASH}/artifacts"
docker build ${BUILD_ENV_VARIABLES} -t ${STEP_NAME}:latest ./${STEP_NAME}
if [[ $? -ne 0 ]]; then
echo ">>> ERROR with build of STEP: ${STEP_NAME}"
exit 1
fi
# EXECUTE
## Get environement variables from config.toml
VAR_NAMES=`./stoml config.toml ${STEP_NAME}`
RUNTIME_ENV_VARIABLES=""
for VAR_NAME in $VAR_NAMES;
do
VAR_VALUE=`./stoml config.toml ${STEP_NAME}.${VAR_NAME}`
RUNTIME_ENV_VARIABLES="${RUNTIME_ENV_VARIABLES} --env ${VAR_NAME^^}=$VAR_VALUE"
done
echo ${RUNTIME_ENV_VARIABLES}
## DEFINE further variables
DOCKER_ARTIFACTS_PATH=/home/${USER}/artifacts
## execute the script
docker run -v ${HOST_ARTIFACTS_PATH}:${DOCKER_ARTIFACTS_PATH} ${RUNTIME_ENV_VARIABLES} ${STEP_NAME}:latest
## if success save the image
if [[ $? -eq 0 ]]; then
## get the container id
CONTAINER_ID=`docker ps -a --last 1 --filter ancestor=${STEP_NAME}:latest --format {{.ID}}`
## Commit to a new image
docker commit $CONTAINER_ID ${STEP_NAME}:last_run
## Save the image to TAR
docker save --output run/${RUN_HASH}/images/${STEP_NAME}.tar ${STEP_NAME}:last_run
## remove the step from pipeline.toml (set as DONE)
remaining_steps=("${remaining_steps[@]:1}")
## update pipeline.toml
size=${#remaining_steps[@]}
printf 'pipeline=['
printf '%s' "${remaining_steps[0]}" > run/${RUN_HASH}/pipeline.toml
if [[ size -gt 1 ]]; then
printf ',%s' "${remaining_steps[@]:1}" >> run/${RUN_HASH}/pipeline.toml
fi
printf ']'
else
echo ">>> ERROR at STEP: ${STEP_NAME}"
exit 1
fi
done
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment