Conditional Execution in Apps¶

This tutorial extends the previously created “Hello World” workflow and app to demonstrate conditional execution in apps. The “Piped Execution in Apps” tutorial must be completed, and its resulting “Hello World!” workflow should be committed to a Git repo prior to beginning this tutorial.

Modify the “Hello World!” App¶

Clone the previous “Hello World!” app from the Git repo (be sure to replace the Git repo URL with the appropriate URL for your app):

cd ~/geneflow_work
git clone https://github.com/[USER]/hello-world-gf2.git

Note: if the hello-world-gf2 directory still exists with the contents of the previous app tutorial, do a git pull instead:

cd ~/geneflow_work/hello-world-gf2
git pull

Update App Metadata¶

Update the app metadata to reflect the new functionality:

vi ./app.yaml

Change the description field to indicate a change in the app; leave the git field as is (e.g., with [USER] replaced with your username, and github.com replaced with the appropriate Git service); update the version field to 0.4:

# Basic app information

# name: standard GeneFlow app name
name: hello-world-gf
# description: short description for the app
description: Updated hello world GeneFlow app demonstrating conditional execution
# git: link to the app's git repo
git: https://github.com/[USER]/hello-world-gf2.git
# version: must be incremented every time this file, or any file in the app
# project is modified
version: '0.4'

Update App Parameters¶

Add a string parameter named mode to the app to control the app’s execution. If this parameter is set to basic, then the app will simply output Hello World! to a file. However, if this parameter is set to any other value, the app will execute as defined in the previous tutorial, i.e., it will use pipes to count the number of words in the input file. Leave the output parameter as is.

parameters:
  mode:
    label: Mode
    description: Execution mode
    type: string
    required: false
    default: basic
    test_value: basic
  output:
    label: Output Text File
    description: Output text file
    type: File
    required: true
    test_value: output.txt

Update App Execution Commands¶

Update the app execution commands in the exec_methods section so that it checks the mode parameter and executes commands accordingly:

exec_methods:
- name: environment
  if:
  - in_path: cat
  - in_path: wc
  - in_path: echo
  exec:
  - if:
    - str_equal: ['${MODE}', 'basic']
    run: echo 'Hello World!'
    stdout: ${OUTPUT_FULL}
    else:
    - pipe:
      - run: cat ${FILE_FULL}
      - run: wc -w
        stdout: ${OUTPUT_FULL}

The modified exec_methods block first checks if three command-line utilities are available in the system path: echo, cat (which prints the contents of a file) and wc (which counts the number of words). If two or more items are defined in the if block, they are treated as an AND conditional, so echo, cat, and wc must all be available in the system path in order for the environment execution method to run. The if block checks the value of the mode parameter. If mode is set to basic, the app echos Hello World! to the output file, as in the original version of the “Hello World!” app. However, if the mode parameter is set to anything else, the app executes the piped command as defined in the previous “Piped Execution in Apps” tutorial.

Re-“Make” the App¶

Now that the app.yaml file has been updated with the new conditional execution block and a new parameter has been added, rebuild the app with the following commands:

cd ~/geneflow_work/hello-world-gf2
geneflow make-app .

This should re-generate the agave-app-def.json.j2, hello-world-gf2.sh, and test.sh scripts.

Test the App¶

Run the test.sh script to test the app. Note: this test uses the same test input file added in the previous tutorial:

cd ~/geneflow_work/hello-world-gf2/test
./test.sh

You should see output similar to the following:

CMD=/home/[USER]/geneflow_work/hello-world-gf/test/../assets/hello-world-gf.sh --file="/home/[USER]/geneflow_work/hello-world-gf/test/data/file.txt" --mode="basic" --output="output.txt" --exec_method="auto"
File: /home/[USER]/geneflow_work/hello-world-gf/test/data/file.txt
Mode: basic
Output: output.txt
Execution Method: auto
Detected Execution Method: environment
CMD=echo 'Hello World!'  >"/scicomp/home/ktr2/geneflow_work/hello-world-gf/test/output.txt"
Exit code: 0
Exit code: 0

The output.txt file should have been created in the test directory with the text Hello World!. View it with:

cat ./output.txt

And you should see the appropriate text:

Hello World!

When we test the workflow, we’ll execute it using the “advanced” condition.

Update the App README¶

Update the app README.rst file to reflect changes to the app:

cd ~/geneflow_work/hello-world-gf2
vi ./README.rst

Modify the file so it looks like the following:

Hello World! Updated GeneFlow App
=================================

Version: 0.4

This is a basic GeneFlow app that demostrates piping and conditional execution.

Inputs
------

1. file: Input text file.

Parameters
----------

1. mode: Mode of execution to test app conditions. 'basic' mode will pipe 'Hello World!' to a text file. Any value other than 'basic' will process the input file with a series of commands that counts the number of words in the input file.

2. output: Output text file where either 'Hello World!', or number of words in the input text file will be printed.

Commit and Tag the New App¶

Finally, commit the updated app to the Git repo and update its tag to reflect the new version number:

cd ~/geneflow_work/hello-world-gf2
git add -u
git commit -m "update hello world app with conditional execution"
git tag 0.4
git push --tags origin master

Modify the “Hello World!” Workflow¶

Now that the app has been updated, we can update the “Hello World!” workflow to use the new app.

Checkout the previous “Hello World!” workflow:

cd ~/geneflow_work
git clone https://github.com/[USER]/hello-world-workflow-gf2.git

Note: if the hello-world-workflow-gf2 directory still exists with the contents of the previous workflow tutorial, do a git pull instead:

cd ~/geneflow_work/hello-world-workflow-gf2
git pull

Update Workflow Metadata¶

Update the workflow metadata by incrementing the version number to 0.4:

vi ./workflow.yaml

The metadata section should have the line:

version: '0.4'

Update Apps¶

Update the “Apps” section by incrementing the version:

apps:
  hello-world:
    git: https://github.com/[USER]/hello-world-gf2.git
    version: '0.4'

Be sure to replace the “git” field with the correct Git repo to which you committed the “Hello World!” app.

Add a Workflow Parameter¶

In the workflow.yaml file, edit the parameters section so that it includes a mode parameter:

parameters:
  mode:
    label: Mode
    description: Execution mode
    type: string
    default: basic
    enable: true
    visible: true

This parameter will be passed on to the new app in the steps section.

Update Workflow Steps¶

Update the steps section so that the hello step passes the new mode parameter to the app:

steps:
   hello:
     app: hello-world
     depend: []
     template:
       file: ${workflow->file}
       mode: ${workflow->mode}
       output: output.txt

Update the Workflow README¶

Update the README.rst file to reflect the new functionality:

cd ~/geneflow_work/hello-world-workflow-gf2
vi ./README.rst

Modify the file so it looks like the following:

Hello World! One-Step GeneFlow Workflow
=======================================

Version: 0.4

This is a basic one-step GeneFlow workflow that demonstrates piping and conditional execution.

Inputs
------

1. file: Input text file.

Parameters
----------

1. mode: Mode of exeuction. 'basic' mopde will pipe 'Hello World!' to a text file. Any value other than 'basic' will process the input file with a series of commands that counts the number of words in the input file.

Commit and Tag the New Workflow¶

Commit the new version of the workflow to the Git repo:

cd ~/geneflow_work/hello-world-workflow-gf2
git add -u
git commit -m "updated hello world workflow"
git tag 0.4
git push --tags origin master

Install and Test the Workflow¶

Now that the workflow has been updated and committed to a Git repo, we can install and test it:

cd ~/geneflow_work
geneflow install-workflow -g https://github.com/[USER]/hello-world-workflow-gf2.git -c --make_apps ./test-workflow-0.4

This command installs the updated “Hello World!” one-step workflow, and its updated “Hello World!” app into the directory test-workflow-0.4. Remember to replace the Git URL with the URL to which you committed the workflow.

Test the workflow to validate its functionality:

geneflow run ./test-workflow-0.4 -o output --in.file=./test-workflow-0.4/data/test.txt --param.mode=basic

This command runs the workflow in the test-workflow-0.4 directory using the test data, mode of “basic”, and copies the output to the output directory.

Once complete, you should see a file called output.txt with contents of Hello World!:

cat ./output/geneflow-job-[JOB ID]/hello/output.txt

Be sure to replace [JOB ID] with the ID of the GeneFlow job. The job ID is a randomly generated string and ensures that workflow jobs do not overwrite existing job output. You should see the following text in the output.txt file:

Hello World!

Next, test the workflow with an alternate mode (i.e., other than “basic”):

geneflow run ./test-workflow-0.4 -o output --in.file=./test-workflow-0.4/data/test.txt --param.mode=advanced

This command runs the workflow in the test-workflow-0.4 directory using the test data, mode of “advanced”, and copies the output to the output directory.

Once complete, you should see a file called output.txt with contents of 4:

cat ./output/geneflow-job-[JOB ID]/hello/output.txt

Be sure to replace [JOB ID] with the ID of the GeneFlow job. You should see the following text in the output.txt file:

Summary¶

Congratulations! You updated the one-step GeneFlow workflow so that it demonstrates conditional execution in the app. The next tutorial will expand on this workflow by adding more steps.