Piped Execution in Apps

This tutorial extends the previously created “Hello World” workflow and app to demonstrate the use of Bash pipes. The “Basic Workflow and App Inputs” tutorial must be completed, and its resulting “Hello World!” workflow should be committed to a Git repo prior to beginning this tutorial.

Modify the “Hello World!” App

Clone the previous “Hello World!” app from the Git repo (be sure to replace the Git repo URL with the appropriate URL for your app):

cd ~/geneflow_work
git clone https://github.com/[USER]/hello-world-gf2.git

Note: if the hello-world-gf2 directory still exists with the contents of the previous app tutorial, do a git pull instead:

cd ~/geneflow_work/hello-world-gf2
git pull

Update App Metadata

Update the app metadata to reflect the new functionality:

vi ./app.yaml

Change the description field to indicate a change in the app; leave the git field as is (e.g., with [USER] replaced with your username, and github.com replaced with the appropriate Git service); update the version field to 0.3:

# Basic app information

# name: standard GeneFlow app name
name: hello-world-gf
# description: short description for the app
description: Updated hello world GeneFlow app demonstrating pipes
# git: link to the app's git repo
git: https://github.com/[USER]/hello-world-gf2.git
# version: must be incremented every time this file, or any file in the app
# project is modified
version: '0.3'

Update App Execution Commands

Update the app execution commands in the exec_methods section as follows:

- name: environment
  - in_path: cat
  - in_path: wc
  - pipe:
    - run: cat ${FILE_FULL}
    - run: wc -w
      stdout: ${OUTPUT_FULL}

The modified exec_methods block first checks if two command-line utilities are available in the system path: cat (which prints the contents of a file) and wc (which counts the number of words). If two or more items are defined in the if block, they are treated as an AND conditional, so both cat and wc must be available in the system path in order for the environment execution method to run. The pipe section includes a list of execution blocks. Each execution block is piped into the subsequent execution block, and the final block pipes its output to the item defined by the stdout field. Thus, the pipe section translates to the following Bash code: cat ${FILE_FULL} | wc -w > ${OUTPUT_FULL}

This command prints the contents of ${FILE_FULL}, pipes the result to wc, the word count utility, then outputs the word count result to ${OUTPUT_FULL}.

Re-“Make” the App

Now that the app.yaml file has been updated with the new piped execution block, rebuild the app with the following commands:

cd ~/geneflow_work/hello-world-gf2
geneflow make-app .

This should re-generate the agave-app-def.json.j2, hello-world-gf2.sh, and test.sh scripts.

Test the App

Run the test.sh script to test the app. Note: this test uses the same test input file added in the previous tutorial:

cd ~/geneflow_work/hello-world-gf2/test

You should see output similar to the following:

CMD=/home/[USER]/geneflow_work/hello-world-gf/test/../assets/hello-world-gf.sh --file="/home/[USER]/geneflow_work/hello-world-gf/test/data/file.txt" --output="output.txt" --exec_method="auto"
File: /home/[USER]/geneflow_work/hello-world-gf/test/data/file.txt
Output: output.txt
Execution Method: auto
Detected Execution Method: environment
CMD=cat /home/[USER]/geneflow_work/hello-world-gf/test/data/file.txt |wc -w  >"/scicomp/home/ktr2/geneflow_work/hello-world-gf/test/output.txt"
Exit code: 0
Exit code: 0

The output.txt file should have been created in the test directory with the text 4, which is the number of words in the test file (which has contents Hello World File Contents!. View it with:

cat ./output.txt

And you should see the number of words in the test file:


Update the App README

Update the app README.rst file to reflect changes to the app:

cd ~/geneflow_work/hello-world-gf2
vi ./README.rst

Modify the file so it looks like the following:

Hello World! Updated GeneFlow App

Version: 0.3

This is a basic GeneFlow app with an input that demonstrates pipes.


1. file: Input text file.


1. output: Output text file where the number of words in the input text file will be printed.

Commit and Tag the New App

Finally, commit the updated app to the Git repo and update its tag to reflect the new version number:

cd ~/geneflow_work/hello-world-gf2
git add -u
git commit -m "update hello world app with pipes"
git tag 0.3
git push --tags origin master

Modify the “Hello World!” Workflow

Now that the app has been updated, we can update the “Hello World!” workflow to use the new app.

Checkout the previous “Hello World!” workflow:

cd ~/geneflow_work
git clone https://github.com/[USER]/hello-world-workflow-gf2.git

Note: if the hello-world-workflow-gf2 directory still exists with the contents of the previous workflow tutorial, do a git pull instead:

cd ~/geneflow_work/hello-world-workflow-gf2
git pull

Update Workflow Metadata

Update the workflow metadata by incrementing the version number to 0.3:

vi ./workflow.yaml

The metadata section should have the line:

version: '0.3'

Update Apps

Update the “Apps” section by incrementing the version:

    git: https://github.com/[USERS]/hello-world-gf2.git
    version: '0.3'

Be sure to replace the “git” field with the correct Git repo to which you committed the “Hello World!” app.

Update the Workflow README

Update the README.rst file to reflect the new functionality:

cd ~/geneflow_work/hello-world-workflow-gf2
vi ./README.rst

Modify the file so it looks like the following:

Hello World! One-Step GeneFlow Workflow

Version: 0.3

This is a basic one-step GeneFlow workflow that demonstrates piping.


1. file: Input text file.



Commit and Tag the New Workflow

Commit the new version of the workflow to the Git repo:

cd ~/geneflow_work/hello-world-workflow-gf2
git add -u
git commit -m "updated hello world workflow"
git tag 0.3
git push --tags origin master

Install and Test the Workflow

Now that the workflow has been updated and committed to a Git repo, we can install and test it:

cd ~/geneflow_work
geneflow install-workflow -g https://github.com/[USER]/hello-world-workflow-gf2.git -c --make_apps ./test-workflow-0.3

This command installs the updated “Hello World!” one-step workflow, and its updated “Hello World!” app into the directory test-workflow-0.3. Remember to replace the Git URL with the URL to which you committed the workflow.

Test the workflow to validate its functionality:

geneflow run ./test-workflow-0.3 -o output --in.file=./test-workflow-0.3/data/test.txt

This command runs the workflow in the test-workflow-0.3 directory using the test data and copies the output to the output directory.

Once complete, you should see a file called output.txt with contents of 3:

cat ./output/geneflow-job-[JOB ID]/hello/output.txt

Be sure to replace [JOB ID] with the ID of the GeneFlow job. The job ID is a randomly generated string and ensures that workflow jobs do not overwrite existing job output. You should see the following text in the output.txt file:



Congratulations! You updated the one-step GeneFlow workflow so that it demonstrates piping the app. The next tutorial will expand on this workflow by introducing conditional execution in apps.