AzkabanCLI

A lightweight Azkaban client providing:

  • A command line interface to run jobs, upload projects, and more.

    $ azkaban --project=my_project upload archive.zip
    Project my_project successfully uploaded (id: 1, size: 205kB, version: 1).
    Details at https://azkaban.server.url/manager?project=my_project
    
    $ azkaban --project=my_project run my_flow
    Flow my_flow successfully submitted (execution id: 48).
    Details at https://azkaban.server.url/executor?execid=48
    
  • A convenient and extensible way to build project configuration files.

    from azkaban import PigJob, Project
    from getpass import getuser
    
    PROJECT = Project('sample', root=__file__)
    
    # default options for all jobs
    DEFAULTS = {
      'user.to.proxy': getuser(),
      'param': {
        'input_root': 'sample_dir/',
        'n_reducers': 20,
      },
      'jvm.args.mapred': {
        'max.split.size': 2684354560,
        'min.split.size': 2684354560,
      },
    }
    
    # list of pig job options
    OPTIONS = [
      {'pig.script': 'first.pig'},
      {'pig.script': 'second.pig', 'dependencies': 'first.pig'},
      {'pig.script': 'third.pig', 'param': {'foo': 48}},
      {'pig.script': 'fourth.pig', 'dependencies': 'second.pig,third.pig'},
    ]
    
    for option in OPTIONS:
      PROJECT.add_job(option['pig.script'], PigJob(DEFAULTS, option))
    

Indices and tables