config: Set AUGUR_SEARCH_PATHS

[victorlin]

Description of proposed changes

To be used by various Augur commands and pathogen workflows, starting with augur subsample in the measles repo.

Related issue(s)

Used in nextstrain/measles#90

Checklist

• Checks pass N/A
• If adding a script, add an entry for it in the README. N/A

[jameshadfield]

For workflows which may want to customise this, e.g. add in workflow.basedir a workflow could do something like the following?

import: shared/config.smk

os.environ["AUGUR_SEARCH_PATHS"] = os.environ["AUGUR_SEARCH_PATHS"] + ":" + workflow.basedir

[victorlin]

Two ways to customize:

1. Override defaults by setting AUGUR_SEARCH_PATHS before the import
2. Extend defaults by prepending or appending after the import (e.g. your snippet)

[jameshadfield]

This won't exist for lots of workflows. I'm not sure what's best -- exclude it (and expect the workflow to update the env variable) or fall back to using workflow.basedir by default?

[jameshadfield]

Another direction is to wrap this entire if/else block in a function set_search_paths and expect all workflows to call it. An extra step for workflows, but given the diversity of workflows we have I think potentially a better one.

[victorlin]

We can add workflow.basedir as a search path:

search_paths = [
    # User analysis directory
    os.getcwd(),

    # Workflow defaults folder
    os.path.join(workflow.basedir, "defaults"),

    # Workflow root (contains Snakefile)
    workflow.basedir,
]

We can also go further and include the pathogen repo root to cover avian-flu:

search_paths = [ … ]

repo_root = Path(workflow.basedir) / ".."
if (repo_root / "nextstrain-pathogen.yaml").is_file():
    search_paths.append(repo_root)

These defaults and their order should cover most if not all repos. Outlier repos can customize per #68 (comment).

[joverlee521]

+1 for adding workflow root and repo root to the default search paths set here.

[victorlin]

Done, also updated to use pathlib:

shared/snakemake/config.smk

Lines 26 to 42 in cd3def6

	search_paths = [
	# User analysis directory
	Path.cwd(),
	# Workflow defaults folder
	Path(workflow.basedir) / "defaults",
	# Workflow root (contains Snakefile)
	Path(workflow.basedir),
	]
	repo_root = Path(workflow.basedir) / ".."
	if (repo_root / "nextstrain-pathogen.yaml").is_file():
	search_paths.extend([
	# Pathogen repo root
	repo_root,
	])

[victorlin]

Here's a draft of what that could look like: nextstrain/avian-flu@dadbde0...290e6a7.

I don't think it should block this PR, but that's what I have in mind. We could discuss at the next dev chat.