Skip to content

Data Ingestion Pipeline

# The following packages are required: soflow, soload, soler, stout

# Update the path based on which blueprint you want to run
python ..path/soflow/soflow/jobs/blueprints/run_script.py -e Prod_v01xx -m True -b ..path/soflow/soflow/jobs/blueprints/daily_blueprint.json

# OR if you want to run a predefined pipeline...
# Call the bash script and provide the run environment as a parameter
/...path/soflow/soflow/jobs/bash_scripts/daily_run.sh Prod_v01xx

Scheduled Pipeline Runs: PROD_v01xx

  • Daily:
    • script: soflow/soflow/jobs/bash_scripts/daily_run.sh
    • Schedule/Frequency: Every Mon - Sat, 6:00 AM EST
    • Processes:
      • SHE soflow/soflow/jobs/blueprints/she_blueprint.json AND
      • Census soflow/soflow/jobs/blueprints/census_blueprint.json THEN
        • OpenMRS update & Core update soflow/soflow/jobs/blueprints/openmrs_core_update_blueprint.json THEN
        • Soler update soflow/soflow/jobs/blueprints/soler_update_blueprint.json THEN
        • HASPublic update soflow/soflow/jobs/blueprints/openmrs_create_connected_table_blueprint.json
  • Weekly:
    • script: soflow/soflow/jobs/bash_scripts/weekly_run.sh
    • Schedule/Frequency: Ever Sunday except the first Sunday of the month, 6:00 AM EST
    • Processes:
      • SHE soflow/soflow/jobs/blueprints/she_blueprint.json AND
      • Census soflow/soflow/jobs/blueprints/census_blueprint.json THEN
        • OpenMRS update soflow/soflow/jobs/blueprints/openmrs_update_blueprint.json THEN
        • Spawn and reload Core soflow/soflow/jobs/blueprints/core_full_blueprint.json THEN
        • Spawn and full run of SolerProgram & SolerPerson soflow/soflow/jobs/blueprints/soler_full_blueprint.json THEN
        • HASPublic update soflow/soflow/jobs/blueprints/openmrs_create_connected_table_blueprint.json
  • Monthly:
    • script: soflow/soflow/jobs/bash_scripts/monthly_run.sh
    • Schedule/Frequency: First Sunday of the month, 6:00 AM EST
    • Processes:
      • SHE soflow/soflow/jobs/blueprints/she_blueprint.json AND
      • Census soflow/soflow/jobs/blueprints/census_blueprint.json THEN
        • Spawn and reload OpenMRS/HAS soflow/soflow/jobs/blueprints/openmrs_nuke_blueprint.json THEN
        • Spawn and reload Core soflow/soflow/jobs/blueprints/core_full_blueprint.json THEN
        • Spawn and full run of SolerProgram & SolerPerson soflow/soflow/jobs/blueprints/soler_full_blueprint.json THEN
        • HASPublic update soflow/soflow/jobs/blueprints/openmrs_create_connected_table_blueprint.json