Core Library#

PyPiVersion image Pytest CodeStyle

Main Documentation: https://orgua.github.io/shepherd

Source Code: https://github.com/orgua/shepherd-datalib

Main Project: https://github.com/orgua/shepherd

shepherd-core is designed as a library and bundles data-models and file-access-routines for the shepherd-testbed, that are used by several codebases.

For postprocessing shepherds .h5-files usage of shepherd_data is recommended.

Features#

  • read and write shepherds hdf5-files

  • create, read, write and convert experiments for the testbed

    • all required data-models are included

  • simulate the virtual source, including virtual harvesters (and virtual converter as a whole)

  • connect and query the testbed via a webclient (TestbedClient in alpha-stage)

    • offline usage defaults to static demo-fixtures loaded from yaml-files in the model-directories

  • work with target-firmwares

    • embed, modify, verify, convert

    • Note: working with ELF-files requires external dependencies, see Installation-Chapter

  • decode waveforms (gpio-state & timestamp) to UART

  • create an inventory (for deployed versions of software, hardware)

See official documentation or example scripts for more details and usage. Most functionality is showcased in both. The extra-directory holds data-generators relevant for the testbed. Notably is a trafficbench-experiment that’s used to derive the link-matrix of the testbed-nodes.

Config-Models in Detail#

These pydantic data-models are used throughout all shepherd interfaces. Users can create an experiment, include their own content and feed it to the testbed.

  • orchestration /data-models with focus on remote shepherd-testbed

  • classes of sub-models

    • /base: base-classes, configuration and -functionality for all models

    • /testbed: meta-data representation of all testbed-components

    • /content: reusable user-defined meta-data for fw, h5 and vsrc-definitions

    • /experiment: configuration-models including sub-systems

    • /task: digestible configs for shepherd-herd or -sheep

    • behavior controlled by ShpModel and content-model

  • a basic database is available as fixtures through a tb_client

    • fixtures selectable by name & ID

    • fixtures support inheritance

  • the models support

    • auto-completion with neutral / sensible values

    • complex and custom datatypes (i.e. PositiveInt, lists-checks on length)

    • checking of inputs and type-casting

    • generate their own schema (for web-forms)

    • pre-validation

    • store to & load from yaml with typecheck through wrapper

    • documentation

  • experiment-definition is designed securely

    • types are limited in size (str)

    • exposes no internal paths

  • experiments can be transformed to task-sets (TestbedTasks.from_xp())

Compatibility#

OS

PyVersion

Comment

Ubuntu

3.8 - 3.12

Windows

3.8 - 3.12

no support for elf and hex-conversions yet

MacOS

3.8 - 3.12

hex-conversion missing

Notes:

  • hex-conversion needs a working and accessible objcopy

  • elf-supports needs

    • shepherd-core[elf] installs pwntools-elf-only

    • most elf-features also still utilize hex-conversion

Installation#

The Library is available via PyPI and can be installed with

  pip install shepherd-core -U

  # or for the full experience (includes core)
  pip install shepherd-data -U

For bleeding-edge-features or dev-work it is possible to install directly from GitHub-Sources (here dev-branch):

pip install git+https://github.com/orgua/shepherd-datalib.git@dev#subdirectory=shepherd_core -U

If you are working with .elf-files (embedding into experiments) you make “objcopy” accessible to python. In Ubuntu, you can either install build-essential or binutils-$ARCH with arch being msp430 or arm-none-eabi for the nRF52.

  sudo apt install build-essential

For more advanced work with .elf-files (modify value of symbols / target-ID) you should install

  pip install shepherd-core[elf]

and also make sure the prereqs for the pwntools are met.

For creating an inventory of the host-system you should install

  pip install shepherd-core[inventory]

Unittests#

To run the testbench, follow these steps:

  1. Navigate your host-shell into the package-folder and

  2. install dependencies

  3. run the testbench (~ 320 tests):

cd shepherd-datalib/shepherd_core
pip3 install ./[tests]
pytest