Beaker's Hardware Inventory System

class: center, middle

# Beaker's Hardware Inventory System
      ### LCA 2015
      ##  Nick Coghlan, Amit Saha
      ### Red Hat Inc.

---

# Agenda
      
      0. What is Beaker?
      
      1. Beaker's hardware inventory

2. How is the inventory created?

3. Replacing smolt with lshw

# Slides

* https://amitksaha.fedorapeople.org/lca2015/slides.html

---

# What is Beaker?

* Full stack integration testing system

* Supports running tests on Fedora, CentOS, Red Hat Enterprise Linux, Docker hosts such as Atomic images

* Run tests on bare metal, preconfigured VM, dynamically created VM, Docker container

* A test can specify the hardware (architecture, memory, disk space, devices ..) and distro

* LCA 2014 [talk](https://www.youtube.com/watch?v=tjUjdBm-Mqw) by Nick Coghlan
        * ([Many improvements](https://beaker-project.org/docs/whats-new/) since then!)

* Learn [more](https://beaker-project.org)

---
      
      # Beaker's hardware inventory

* Hardware database maintaining record of the hardware for each system in Beaker

* Significantly more detailed than typical server inventories
      
      * Supported architectures, memory, CPU details, disk, hypervisor, devices, etc.

---

# Making use of the inventory data

* Example: testing support for Intel Celeron processors

* Job XML specification

* ```<family op="=" value="15"/><model_name op="like" value="%Intel%"/>```

* can get longer, difficult to type in/remember

* A friendlier way - *pre-defined host filters*

* ``bkr workflow-simple --host-filter INTEL__FAM15_CELERON ..``

---
      
      # How is the inventory created?

* Run a Beaker job with the [/distributon/inventory](https://git.beaker-project.org/cgit/beaker-core-tasks/tree/inventory) task
      
      * Makes use of [beaker-system-scan](https://git.beaker-project.org/cgit/beaker-system-scan/)
      
      * The system is provisioned, the task is run and the inventory data is sent to Beaker's database

```
      Data:
      {'Arch': ['ppc64'],
      'Cpu': {'CpuFlags': [],
      'cores': 0,
      'family': 0,
      'model': 4915712,
      'modelName': 'POWER7 (architected), altivec supported',
      'processors': 4,
      'sockets': 0,
      'speed': 4116.0,
      'stepping': 0,
      'vendor': 'IBM'},
      ..
         ```

---
      
      # beaker-system-scan

* Mostly Python, can be used outside Beaker (``-d`` for debug mode)

```
      # beaker-system-scan -d
       Data:
       {'Arch': ['ppc64'],
        ..
      
       'model': 4915712,
       'modelName': 'POWER7 (architected), altivec supported',
       'processors': 4,
       ..
       'Disk': {'Disks': [{'model': 'AIX VDASD',
       'phys_sector_size': 512,
       'sector_size': 512,
       'size': '429496729600'}]},
       'Numa': {'nodes': 2},
       'memory': 15780,
       ..
      
       ```

* Uses ``python-linux-procfs``, ``libparted``, and
      
        * The ``master`` branch uses [smolt](http://fedoraproject.org/wiki/Smolt)
        * The ``lshw`` branch uses [lshw](http://www.ezix.org/project/wiki/HardwareLiSter)
     
      ---
      
      # Replacing smolt with lshw

* ``smolt`` has been [retired](https://fedoraproject.org/wiki/Smolt_retirement) for a while

* ``beaker-system-scan`` doesn't work on RHEL/CentOS 7+ and Fedora

* Hence, for example, ``aarch64`` hardware cannot be inventoried

* [Plan](https://beaker-project.org/dev/proposals/inventory-lshw-migration.html)

* Chose not to fork or otherwise take responsibility for ``smolt`` maintenance

* Use ``lshw`` instead in ``beaker-system-scan``

* ``lshw -xml``
      
        * Parse the XML data for the relevant information
      
      * ``lshw`` based task supports: ``ia64, i686, x86_64, s390x, ppc64, arm, aarch64``

---

# Replacing smolt with lshw: dev cycle

* Beaker project maintains a [lshw fork](https://git.beaker-project.org/cgit/lshw)
      
      * Maintaining a fork helps the Beaker team in trying out fixes and enhancements quickly

* Work with upstream maintainer to merge patches after we've tested them

* Enhanced lshw's functionality on ``ppc64, s390x, arm, aarch64`` hardware ([PR](https://github.com/lyonel/lshw/pull/2/commits))

* Automated test suite improvements to help ensure changes made on x86_64
        don't break other architectures (incomplete, not upstream yet)

---

# Replacing smolt with lshw: Comparison

* Are we missing any important information that ``smolt`` gave us (and ``lshw`` doesn't?)
      
      * Are we collecting the wrong information?

* Compare the data that ``smolt`` and ``lshw`` gives us on all supported architectures
      
          * ``arm`` and ``aarch64`` comparison not possible (no ``smolt`` supported distro)

* Will review some sample results for a ``ppc64`` KVM guest

---

# Replacing smolt with lshw: Comparison

![Alt text](comparison_cpu.png "CPU data comparison")

---

# Replacing smolt with lshw: Comparison

![Alt text](comparison_devices.png "Devices comparison")

---

# Replacing smolt with lshw: current status

* ``beaker-system-scan`` with ``lshw`` looks good overall

* ``lshw`` doesn't *seem* to do well w.r.t retrieving devices on non-x86 archs
      
      * Thorough comparison still remains to be done

* We expect some fixes will be required in both ``beaker-system-scan`` and ``lshw``

---

# Learn more
      
      * Design proposal: [Migration to lshw for inventory task](https://beaker-project.org/dev/proposals/inventory-lshw-migration.html)
      
      * [Beaker developers mailing list](https://lists.fedorahosted.org/mailman/listinfo/beaker-devel)

---

# People involved

* James de Vries (past intern) started off the replacement
      
      * Current Beaker development team (Amit Saha, Dan Callaghan, Matt Jia)

* Lyonel Vincent (lshw maintainer)

# Slides

* https://amitksaha.fedorapeople.org/lca2015/slides.html