A Structured Approach
Key Components:
__init__.py
)DBScan1D example:
There has been heated discussions about why there are so many ways to do things.
Designed for people (generally not codes)
GUIs
TUIs
Command line
-Make a library
-Define command line entry points in pyproject.toml
-Install the library
This Real Python article is great for building command line tools.
Web services
Purposes
project-template # Project folder
├── conf # Configuration files
├── data # Local project data
├── docs # Documentation
├── logs # Logs of pipeline runs
├── notebooks # Exploratory Jupyter notebooks
├── pyproject.toml # Identifies the project root
├── README.md # README.md explaining your project
├── setup.cfg # Configuration options for testing and linting
└── src # Source code for pipelines
from prefect import flow, task
from typing import List
import httpx
@task(retries=3)
def get_stars(repo: str):
url = f"https://api.github.com/repos/{repo}"
count = httpx.get(url).json()["stargazers_count"]
print(f"{repo} has {count} stars!")
@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
for repo in repos:
get_stars(repo)
# run the flow!
github_stars(["PrefectHQ/Prefect"])
Simple Project
├── a010_download_data.py
├── a020_preprocess_seismograms.py
├── a030_detect_earthquakes.py
├── a040_plot_detections.py
├── local.py
├── utils.py
├── inputs/
├── outputs/
├──── a010_raw_data
├──── a020_preprocessed_seismograms
├──── ...
├── environment.yml
├── README.md
└── test_*.py...
How do you organize your projects?