Learning Yocto

0.0.0

preliminary (some mature, much in progress)


Content
  • Domain and Tool Landscape

  • Concepts, Workflow and Build System

  • Dive into the deep – coming soon


build with Sphinx, presented with Reveal.js
Copyright © 2018‒2021 Li-Pro.Net and individual contributors. — All rights reserved. — CC-BY-SA-3.0 — Release: 0.0.0-75-g2094a49
Creative Commons License: Attribution-ShareAlike 3.0 Unported

What for who?

Involved Components:

  • Git – Repo and Manifest

  • Yocto – Terms and Workflow

  • Bitbake – Layer and Recipies

Audience:

  • System Architects

  • Hard- and Software Developers

  • Integrators and Testers

What you should know?

  • Revision Controll Systems

  • Configuration and Build Tools

  • Distribution and Packaging

  • Scripting

Revision Controll Systems

  • Git

  • Subversion

  • Mercurial

  • Principals of work

  • Collaboration work

Configuration and Build Tools

  • Kconfig + GNU Make

  • Autotools and CMake

  • Qmake

  • Ninja

  • Source Code Patching

  • Triad: configmakeinstall

  • Canadian Cross: buildhosttarget

Distribution and Packaging

  • Debian Packages: *.deb

  • Red Hat Packages: *.rpm

  • Open Packages: *.opkg / *.ipkg

  • System Root: SYSROOT

  • Destination Directory: DESTDIR

  • Package Repository (Server)

Scripting

  • Bash

  • Python

  • Perl (rarely)

Challenge

_images/challenge.svg

Example: UI Toolkits

_images/Free_and_open-source-software_display_servers_and_UI_toolkits.svg

English Wikipedia: Linux#Desktop

Collaboration

_images/collaboration.svg

Market

→ that are:

  • Customer and User

  • Retailer and Service

→ which are constantly looking for:

  • Improvements and Bug Fixes (updates)

  • Features and Extensions (back ports)

  • Documentation and Tests

Vendor and Contractor

→ all participants and partners:

  • Engineers

  • Business Owners

  • Compliance Offices

→ have to ensure results with:

  • protected release base line: Software Configuration Management Plan

  • build and test: Continuous Integration

  • DISTRIBUTION: Continuous Deployment / Delivery

Community

→ what is their role ❓

  • best efforts to COOPERATE with and MUTUAL ASSISTANCE the business

  • 🈚 free of charge: SCALE, BE FAST, pay later

  • 🈺 open for business: SHARE WITH ALL, no protection, survival of the fittest

  • 🈴 passing grade: BEST FOR ALL, acceptance

  • secret: kick out the Troll

Business

→ where and who am I ❓

  • mostly I am a tiny candle in front of the biggest star of all

  • leads to: relation of 🕯 candle / ⭐ star

  • ask you: what is MY BUSINESS 🕯 candle

  • ask you: what is THE COMMUNITYstar

my experience of the last 20 years:

1040 % 🕯 candle / 9060 %star

Domains

→ the COMMUNITY – there are we all ‼

  • Market with its Suppliers and Vendors

  • Contractors and their Principals

→ we should take over the WORKFLOWS

  • Infrastructure

  • Frameworks

  • Toolboxes

Intentions

from the perspective of:

  • Architecture

  • Life Cycle

Architecture

1 – Product Management

which hard- and software features we have, and need for a specific product → feature sets

2 – Feature Management

how the distribution looks for a given feature set → composing

3 – Distribution Management

which packages will needed for a given distribution / composing → flavor

Life Cycle

4 – Package Management

how to build and deploy the given list of packages and their dependencies

5 – Deployment Management

what have to share and how to distribute all the resulting artefacts

6 – QA Management (Test, Integration)

how to ensure accuracy of all the development, in parts and altogether

Decision

excerpts and quotes from [Str15,OMM15]. — © 2015 The Linux Foundation.

Embedded Linux Landscape

  • Android

  • Baserock

  • Buildroot

  • OpenEmbedded

  • OpenWrt

  • The Yocto Project

  • Commercial Distributions

Reflection

  • BuildrootThe Yocto Project

Android

https://source.android.com/

  • Great for systems with ARM-based SoCs and touch screens

  • Build system and development tools by Google

  • Java based system runtime environment on target

  • Stable and widely accepted Java API for final applications

  • Very limited adaptability for vendors, contributors or customers

Baserock

https://www.baserock.org/

  • Targeted for embedded appliances

  • Slightly pushed by the GENIVI Alliance

  • Native Builds for x86, x86_64, ARMv7

  • Based on Morph workflow tool

  • Maintained by Codethink Ltd.

Last releases in 2016. → https://wiki.baserock.org/releases/

Buildroot

https://buildroot.org/

  • Targeted for complete Linux root filesystem build

  • Generate a cross-compilation toolchain

  • Simplifies and automates the building

  • uClibc or musl target library

  • BusyBox command line utility applications

  • Based on Kconfig and GNU Make build tool

  • Promoted by french non-profit Buildroot Association

OpenEmbedded

https://www.openembedded.org/

  • Created by merging OpenZaurus with contributions from Familiar Linux, OpenSIMPad into a common code base

  • Focusing on broad hardware and architectures

  • Based on BitBake build engine

  • Technology code base for the Yocto Project and the Ångström Distribution

The Ångström Distribution was widely used on TI-based embedded boards like the BeagleBoard and PandaBoard and uses the Yocto Project build environment. Last releases in 2017 (2020). → https://github.com/Angstrom-distribution

OpenWrt

https://www.openwrt.org/

  • Debuted as open source OS for embedded devices routing network traffic

  • Originally created from Linksys GPL sources for their WRT54G residential gateway

  • Targeted thousands of different hardware

  • Buildroot-based build environment

  • Headless operation with web UI

  • Member of Software Freedom Conservancy (like Git, Inkscape, or Qemu)

Reflection 1/2

https://lwn.net/Articles/682540/ (ELCE-2016, ELCE-2018, Youtube)

Buildroot

Yocto Project

Minimal Size:

2.2 MB

4.9 MB

Build Time:

15-45 minutes

1-2 hours (at least)

Concept:

single Makefile

layered classes

Complexity:

being simple by design, only patchable

multiple aspects of composing and QA

Composing:

multiple Kconfig file

feature driven

State Cache:

compiler only

on task level (states)

Reflection 2/2

Buildroot

Yocto Project

Rebuild:

full (will be improved)

partial per task / on changes

Meta/Configure:

fixed

sharable

Output:

no packages or SDK

images, packages, SDK, licenses, manifests

Reliance:

no QA workflow

documented QA and RM

3rd Party:

rare, some few

many (growing)

Decision:

Buildroot

Yocto Project

About

About the Yocto Project®

The Yocto Project is not an Embedded Linux Distribution.

It creates a custom one for You!

The Yocto Project is not Single Open Source Project.

It is an Ecosystem.

The Yocto Project combines the convenience of a ready-to-run Linux Distribution with the flexibility of a custom Linux operating system stack. [Str15,OMM15] — © 2015 The Linux Foundation and Yocto Project.

Governance Structure

Ecosystem and Showcase

  • based on meritocracy (leadership by demonstrated achievement)

  • managed by its chief architect, Richard Purdie, a Linux Foundation fellow

  • remain independent of any one of its member organizations

The Yocto Project is a lab workgroup of the Linux Foundation, who owns its trademark.

Achievement and Standing

  • open source collaboration project

  • founded in 2010 as a collaboration among many hardware manufacturers, open-source operating systems vendors and electronics companies

  • help you create custom Linux-based systems for embedded products

  • regardless of the hardware architecture

  • provides templates, tools and methods (Best-Practice)

Offer and Support

  • resources and information for new and experienced users

  • core system component recipes by OE

  • example code to demonstrate its capabilities

  • include the Yocto Project Linux kernel

  • covers several build profiles across multiple architectures: ARM, PPC, x86, and more

  • BSP layers for customer or vendor specific platform support

  • BSP layers follow a predetermined and standardized format

Yocto Project Provides

_images/yp-provides.svg
  • embedded tools

  • best practices

  • reference implementation

Yocto Project Community

_images/yp-community.svg
  • Linux Foundation: chief architect

  • OpenEmbedded: technologies and software

  • silicon vendors and electronics companies

Terms, Concepts, Idioms

Yocto Project Terms

slightly extended

Terms overall

  1. Git Repo (repo): The collection manager that Google has built on top of Git to help manage multiple Git repositories, does the uploads to Gerrit and automates the Android system image development workflow. The repo command is an executable Python script that you can put anywhere in your path. → https://gerrit.googlesource.com/git-repo/

  2. BitBake: The Task executor and scheduler used by the Build System to build Images. The bitbake command is part of an fancy Python and Bash environment around the BitBake project. → BitBake, or Yocto Components and ToolsBitBake, or BitBake User Manual

  3. Metadata: The files that BitBake parses when building an Images. In general, Metadata provides Recipes, Classes and Configuration Files. → Metadata, or BitBake User ManualOverviewIntroduction

  4. OE-Core: A core set of Metadata originating with OpenEmbedded that is shared between OE and the Yocto Project. → OpenEmbedded-Core (OE-Core), or Yocto Components and ToolsOE-Core, or OE Wikipedia

  5. Poky: The reference distribution of the Yocto Project. It contains the Build System (BitBake and OE-Core) as well as a set of Metadata to get you started building your own distro. → Poky, or Yocto Components and ToolsPoky, or Yocto Reference Distribution

Terms internal

  1. Layer: A collection of Recipes representing the core, a Board Support Package, an application stack, or any other 3rd party contributions. Layers are hierarchical in their ability to override previous specifications. → Layer, or in Concept

  2. Image: An artifact of the BitBake build process given a collection of Recipes and related Metadata. Images are the binary output that run on specific target architecture. → Image, or in Concept, or in Reference

  3. Package: Refers to a Recipe’s packaged output produced by BitBake (i.e. a “baked recipe”). A Package is generally the compiled binaries produced from the Recipe’s sources. You “bake” any “bin” by running it through BitBake. → Package

  4. Package Groups: Arbitrary group of software Recipes. You use Package Groups to hold Recipes that, when built, usually accomplish a single Task. For example, Package Groups could contain the Recipes for a company’s proprietary or value-add software. → Package Groups

  5. Task: A unit of execution for BitBake (e.g. do_fetch, do_patch, do_compile, and so forth). → Task, or in Reference

Metadata

  1. Classes: Files with the .bbclass file extension that provided for logic encapsulation and inheritance so that commonly used patterns can be defined once and then easily used in multiple Recipes. → Classes, or in Concept, or in Reference

  2. Recipes: Files with the .bb file extension have a set of instructions for building Packages or Package Groups or Images and describes where you get source code, which patches to apply, how to configure the source, how to compile it and so on. It also describe dependencies for libraries or for other Recipes. Recipes represent the logical unit of execution. → Recipe, or in Concept

  3. Append Files: Files with the .bbappend file extension that append build information to a Recipe. The Build System respects every Append file to have a corresponding Recipe (.bb) file. → Append Files

  4. Configuration Files: Configuration information in various → Configuration File, or in Concept

Features

Features provide a mechanism for:

  1. working out which Packages or Package Groups should be included in the generated Image, typically configured in Image Recipes and their related ClassesImage Features

  2. distributions Configuration Files can select which features they want to support → Distro Features

  3. machine Configuration Files specifies the hardware functions that will be available for a given machine → Machine Features

  4. backward and inter-layer compatibility for special (historical) cases → Feature Backfilling

_images/features.svg

Idioms

  1. Build System: The build system specific to the Yocto Project based on the project Poky, which uses BitBake as the Task executor.

  2. Source Directory: Refers to the directory structure created as a result of creating a local copy of the Poky and all other Git repositories as referred by the Git Repo manifest. It contains BitBake, Documentation, Metadata and other files that all support the Yocto Project. Consequently, you must have the Source Directory in place on your development system in order to do any development using the Yocto Project. → Source Directory

  3. Build Directory: This term refers to the area used by the Build System for downloads, builds and deployments. → Build Directory

  4. Cross-Development Toolchain: A collection of cross-compilers, linkers, debuggers, libraries and utilities that run on one architecture and allow you to develop software for a different, or targeted, architecture. → Cross-Development Toolchain

  5. Board Support Package (BSP): A group of drivers, definitions, and other components that provide support for a specific hardware configuration. → Board Support Package (BSP), or Yocto Project Board Support Package Developer’s Guide and Yocto Project Linux Kernel Development Manual

Conclusion

the following applies and is unavoidable

Git Repo: determines the code base line

Layers: are essential and able to overlay

Classes: inheritable but not extensible

Recipes: extensible but not inheritable

Features: determines the final product

Git Repo Manifest

PREFACE

A product firmware stack will developed locally and pushed to different in-house or external repositories at irregular intervals. A tool to unifies all that repositories as necessary, performs uploads and automates the pull and sync process will be needed.

The repo tool and a meta Git repository called manifest or such as in our case platform repo make it possible to get all the code needed to build your own Yocto source and build directory.

Android Tools

excerpts and quotes from https://source.android.com/setup/develop#repo — © 2008–2021 Google LLC.

Git

handle projects that are distributed over multiple Git repositories

Repo

unifies Git repositories, performs uploads to Gerrit, and automates the workflow

Gerrit

code review system for projects that use Git

Platform Manifest

https://github.com/lipro-yocto/lpn-central-repo


<?xml version="1.0" encoding="UTF-8"?>
<manifest>

  <default sync-j="4" revision="master"/>

  <remote fetch="https://git.yoctoproject.org/git" name="yocto"/>
  <remote fetch="https://github.com/openembedded" name="oe"/>
  <remote fetch="https://github.com/meta-qt5" name="qt5"/>
  <remote fetch="https://github.com/lipro-yocto" name="lpy"/>

  <project remote="yocto" name="poky" path="sources/poky"/>
  <project remote="yocto" name="meta-mingw" path="sources/yocto/meta-mingw"/>
  <project remote="oe" name="meta-openembedded" path="sources/openembedded"/>
  <project remote="qt5" name="meta-qt5" path="sources/qt/meta-qt5" />

  <project remote="lpy" name="lpn-central" path="sources/central">
    <copyfile dest="setup-environment" src="scripts/setup-environment"/>
  </project>

</manifest>

https://gerrit.googlesource.com/git-repo/+/master/docs

Manifest Format, Configuration, Hooks

Platform Pipeline

_images/lpn-central-bsp-pipeline.svg

Build System

OpenEmbedded Build System Concepts

In general, the build’s workflow consists of several functional areas.

_images/areas.svg

Configuration Area

_images/user-configuration-area.svg

User Configuration

Setup values you can use to control the build process.

Metadata Area

_images/metadata-layers-area.svg

Metadata Layers

Various layers that provide software, machine and distro metadata.

Source Area

_images/source-files-area.svg

Sources

Upstream releases, local projects, and SCMS.

Package Area

_images/package-feeds-area.svg

Package Feeds

Directories containing output packages (RPM, DEB or IPK), which are subsequently used in the construction of an image or SDK.

BitBake Tool Area

_images/bitbake-tool-area.svg

BitBake Tool

Processes under the control of BitBake. This block expands on how BitBake fetches source, applies patches, completes compilation, analyzes output for package generation, creates and tests packages, generates images and generates cross-development tools.

Images Area

_images/images-area.svg

Images

Images produced by the build system.

SDK Area

_images/sdk-area.svg

Application Development SDK

Cross-development tools which are produced with the build system.

User Configuration

_images/user-configuration.png

Through user configuration you can tell BitBake the target architecture for which you are building the image, where to store downloaded source and other build properties.

bblayers.conf

https://github.com/lipro-yocto/lpn-central → meta-cedi/conf


CEDI_BBLAYERS_CONF_VERSION = "1"

BBPATH = "${TOPDIR}"
BSPDIR := "${@os.path.abspath('##OEROOT##/../..')}"
BBFILES ?= ""

BBLAYERS = " \
  ${BSPDIR}/sources/poky/meta \
  ${BSPDIR}/sources/poky/meta-poky \
  ${BSPDIR}/sources/poky/meta-yocto-bsp \
  \
  ${BSPDIR}/sources/yocto/meta-mingw \
  \
  ${BSPDIR}/sources/openembedded/meta-oe \
  ${BSPDIR}/sources/openembedded/meta-python \
  ${BSPDIR}/sources/openembedded/meta-initramfs \
  ${BSPDIR}/sources/openembedded/meta-filesystems \
  ${BSPDIR}/sources/openembedded/meta-networking \
  ${BSPDIR}/sources/openembedded/meta-multimedia \
  \
  ${BSPDIR}/sources/qt/meta-qt5 \
  \
  ${BSPDIR}/sources/central/meta \
  ${BSPDIR}/sources/central/meta-cedi \
  ${BSPDIR}/sources/central/meta-lpn-bsp \
"

The bblayers.conf file tells BitBake what layers you want considered during the build.

local.conf

https://github.com/lipro-yocto/lpn-central → meta-cedi/conf

The local.conf file provides many basic variables that define a build environment. Here is a list of a few:

Target Machine Selection: MACHINE ??= "qemux86-64"

Target Distribution Selection: DISTRO ?= "cedi"

Target Packaging Format: PACKAGE_CLASSES ?= "package_rpm"

Download Directory: DL_DIR ?= "${TOPDIR}/downloads"

Shared State Cache Directory: SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

Build Output: TMPDIR = "${TOPDIR}/tmp"

Parallelism Options: BB_NUMBER_THREADS ?= "4"

Host SDK Machine Selection: SDKMACHINE ?= "x86_64"

The setup-environment script have to use to generate the content of the local.conf file.

setup-environment

https://github.com/lipro-yocto/lpn-central → scripts

The setup-environment script provides many basic options that setup a build environment with the correct User Configuration. Here is a short synopsis:


DISTRO="cedi" \
MACHINE="helium" \
SDKMACHINE="x86_64-mingw32" \
BUILD_DIR="build-cedi-helium" \
source setup-environment

The setup-environment calls the oe-init-build-env script implicitly from the correct place in the Poky Source Directory.

To get a detailed online help use:


source setup-environment --help

Metadata Layers

_images/layer-input-landscape.png

These layers provide Metadata for the software, machine and policy.

Meta Layer Types

Software Layers: Metadata .bb + Patches

user/vendor/community supplied recipe files, patches and append files

Distribution Layers: Policy Configuration

top-level or general policies for Images or SDKs being built

BSP Layers: Machine Configuration

specific to a particular target architecture

Meta Layer Stack

_images/lpn-central-bsp-layer.svg

Metadata Layer Priorities (BBFILE_PRIORITY_LayerName) – remember:

_images/lpn-central-bsp-pipeline.svg

Meta Layer Machines

https://github.com/lipro-yocto/lpn-centralmeta-lpn-bsp/conf/machine

The meta-lpn-bsp layer provides machine configurations for all the specific evaluation hardware based on QEMU or real target architecture. The following target machines are supported:

hydrogen: QEMU x86 machine based on qemux86

helium: QEMU x86-64 machine based on qemux86-64

hydrogen.conf


#@TYPE: Machine
#@NAME: QEMU x86 machine
#@DESCRIPTION: Machine configuration for running an x86 system on QEMU (based on qemux86).
#@MAINTAINER: Stephan Linz <linz@li-pro.net>

require conf/machine/include/qemux86.inc
require conf/machine/include/lpn-emu.inc

qemux86.inc


# derived from qemux86
require conf/machine/qemux86.conf
MACHINEOVERRIDES_prepend = "qemux86:"

# derived this machine back to qemux86 for the Linux kernel
KMACHINE = "qemux86"

lpn-emu.inc


# is a Li-Pro.Net emulated system
require conf/machine/include/lpn.inc
MACHINEOVERRIDES_append = ":lpnemu"

lpn.inc


# is a Li-Pro.Net system
MACHINEOVERRIDES_append = ":lpn"

helium.conf


#@TYPE: Machine
#@NAME: QEMU x86-64 machine
#@DESCRIPTION: Machine configuration for running an x86-64 system on QEMU (based on qemux86-64).
#@MAINTAINER: Stephan Linz <linz@li-pro.net>

require conf/machine/include/qemux86-64.inc
require conf/machine/include/lpn-emu.inc

qemux86-64.inc


# derived from qemux86-64
require conf/machine/qemux86-64.conf
MACHINEOVERRIDES_prepend = "qemux86-64:"

# remap derived machine back to qemux86-64 for the Linux kernel
KMACHINE = "qemux86-64"

lpn-emu.inc


# is a Li-Pro.Net emulated system
require conf/machine/include/lpn.inc
MACHINEOVERRIDES_append = ":lpnemu"

lpn.inc


# is a Li-Pro.Net system
MACHINEOVERRIDES_append = ":lpn"

Meta Layer Distros

https://github.com/lipro-yocto/lpn-centralmeta-cedi/conf/distro

The meta-cedi layer provides policy configurations for all the specific evaluation distributions based on Poky. The following distributions are supported:

cedi: Central Distro (Li-Pro.Net Demonstration) based on poky

cedi.conf


require ${COREBASE}/meta-poky/conf/distro/poky.conf

DISTROOVERRIDES = "poky:${@d.getVar('DISTRO') or ''}"

DISTRO = "cedi"
DISTRO_NAME = "Central Distro (Li-Pro.Net Demonstration)"
DISTRO_VERSION_append = "+r1.0"
DISTRO_CODENAME_append = "-cedi"
TARGET_VENDOR = "-cedi"
SDK_VENDOR = "-cedisdk"

MAINTAINER = "Cedi <cedi@li-pro.net>"

LOCALCONF_VERSION = "1"

# Override poky pre-settings (remove).
POKY_DEFAULT_DISTRO_FEATURES = ""
POKY_DEFAULT_EXTRA_RDEPENDS = ""
POKY_DEFAULT_EXTRA_RRECOMMENDS = ""

# Setup Cedi defaults.
CEDI_DEFAULT_DISTRO_FEATURES = "largefile opengl ptest multiarch wayland vulkan"
CEDI_DEFAULT_EXTRA_RDEPENDS = "packagegroup-core-boot"
CEDI_DEFAULT_EXTRA_RRECOMMENDS = "kernel-module-af-packet"

DISTRO_FEATURES = "${DISTRO_FEATURES_DEFAULT} ${CEDI_DEFAULT_DISTRO_FEATURES}"
DISTRO_EXTRA_RDEPENDS += " ${CEDI_DEFAULT_EXTRA_RDEPENDS}"
DISTRO_EXTRA_RRECOMMENDS += " ${CEDI_DEFAULT_EXTRA_RRECOMMENDS}"

# Use systemd for system initialization
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_login_manager = "shadow-base"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "sysvinit"
VIRTUAL-RUNTIME_initscripts = "systemd-compat-units"

# Add Cedi sanity bbclass.
INHERIT += "cedi-sanity"

Meta Layer Images

https://github.com/lipro-yocto/lpn-centralmeta/recipes-core/images

The meta (central core) layer provides recipes and classes for all the specific evaluation and demonstration images and package groups based on the customized image class central-image.bbclass. The following images are supported:

central-image-minimal: A small image just capable of allowing an Central device to boot.

central-dev-image: A developer image just capable of allowing an Central device to boot.

central-debug-image: A debugging image just capable of allowing an Central device to boot.

central-image: A product image capable of allowing an Central device to boot and provides full feature support.

central-image.bb


require central-image-base.bb

SUMMARY = "A product image capable of allowing an Central \
device to boot and provides full feature support."

DESCRIPTION = "A product image capable of allowing an Central \
device to boot in graphical mode and provides full feature support."

LICENSE = "MIT"
PR = "r0"

IMAGE_FEATURES += " \
    ${@bb.utils.contains('DISTRO_FEATURES', 'x11', 'x11-base', '', d)} \
"

inherit central-image

central-image-base.bb


SUMMARY = "A console-only image that fully supports the target device \
hardware provided by Central."

LICENSE = "MIT"
PR = "r0"

IMAGE_FEATURES += " \
    package-management \
    splash \
"

inherit central-image

central-image.bbclass


# Common code for generating Central reference images

LIC_FILES_CHKSUM = "\
    file://${CENTRALCOREBASE}/LICENSE;md5=4d183b8707e22082e5a8c5ad268e5149 \
"

# - qt5-sdk          - Qt5/X11 SDK and demo applications
# - tools-cross      - tools usable for basic cross development
FEATURE_PACKAGES_qt5-sdk = "\
    packagegroup-qt5-toolchain-target \
    packagegroup-qt5-qtcreator-debug \
"
FEATURE_PACKAGES_tools-cross = ""
# Provides the Central specific features 'qt5-sdk' and 'tools-cross'.

CENTRAL_IMAGE_BASE_INSTALL = '\
    packagegroup-central-boot \
    packagegroup-base-central \
    \
    ${@bb.utils.contains("IMAGE_FEATURES", "tools-cross", \
                         "packagegroup-central-tools-cross", "", d)} \
    ${@bb.utils.contains("IMAGE_FEATURES", "tools-debug", \
                         "packagegroup-central-tools-debug", "", d)} \
    ${@bb.utils.contains("IMAGE_FEATURES", "tools-profile", \
                         "packagegroup-central-tools-profile", "", d)} \
    ${@bb.utils.contains("IMAGE_FEATURES", "tools-testapps", \
                         "packagegroup-central-tools-testapps", "", d)} \
    ${@bb.utils.contains("IMAGE_FEATURES", "tools-sdk", \
                         "packagegroup-central-sdk", "", d)} \
    \
    ${CENTRAL_IMAGE_EXTRA_INSTALL} \
    '

CENTRAL_IMAGE_EXTRA_INSTALL ?= ""

CORE_IMAGE_EXTRA_INSTALL += "${CENTRAL_IMAGE_BASE_INSTALL}"

inherit core-image central-image-version

central-image-version.bbclass


# Common code for generating Central version file

LIC_FILES_CHKSUM = "\
    file://${CENTRALCOREBASE}/LICENSE;md5=4d183b8707e22082e5a8c5ad268e5149 \
"

CENTRAL_VERSION_FILE = "${IMAGE_ROOTFS}${sysconfdir}/central_version"

write_central_version() {
        cat > ${CENTRAL_VERSION_FILE} <<EOF
[build information]
vendor-id=LPN
manufacturer-name=Li-Pro.Net
device-variant=${MACHINE}
purpose=${IMAGE_BASENAME}
feature=${IMAGE_FEATURES}
build-number=${BUILD_NUMBER}
EOF
}

ROOTFS_POSTPROCESS_COMMAND += "write_central_version;"

packagegroup-central-boot.bb


SUMMMARY = "Central Minimal Boot Requirements"
DESCRIPTION = "The minimal set of packages required to boot a Central system"
LICENSE = "MIT"
PR = "r0"

# For backwards compatibility after rename
RPROVIDES_packagegroup-central-boot = "packagegroup-boot-central"
RREPLACES_packagegroup-central-boot = "packagegroup-boot-central"
RCONFLICTS_packagegroup-central-boot = "packagegroup-boot-central"

# packages which content depend on MACHINE_FEATURES need to be MACHINE_ARCH
PACKAGE_ARCH = "${MACHINE_ARCH}"

inherit packagegroup

# Set by the machine configuration with packages essential
# for Central device bootup.
MACHINE_ESSENTIAL_EXTRA_CENTRAL_RDEPENDS ?= ""
MACHINE_ESSENTIAL_EXTRA_CENTRAL_RRECOMMENDS ?= ""

RDEPENDS_${PN} = " \
    ${MACHINE_ESSENTIAL_EXTRA_CENTRAL_RDEPENDS} \
"

RRECOMMENDS_${PN} = " \
    ${MACHINE_ESSENTIAL_EXTRA_CENTRAL_RRECOMMENDS} \
"

packagegroup-central-tools-testapps.bb


SUMMMARY = "Central Testing tools/applications"
DESCRIPTION = "The testing set of packages required for a Central system"
LICENSE = "MIT"
PR = "r0"

inherit packagegroup

# Event device test tools
EVTOOLS = "evtest"

# Simple memmory access tools
MEMTOOLS = "devmem2 libuio"

# Simple serial bus tools
SERBUSTOOLS = "i2c-tools spitools"

# Requires Serial to work
USE_SERIAL = "picocom serial-forward"

RDEPENDS_${PN} = " \
    ${EVTOOLS} \
    ${MEMTOOLS} \
    ${SERBUSTOOLS} \
    ${@bb.utils.contains('MACHINE_FEATURES', 'serial', '${USE_SERIAL}', '',d)} \
"

RRECOMMENDS_${PN} = " \
"

Source Files

_images/source-input.png

In order for the build system to create an image or any target, it must be able to access source files. The Yocto Project represents source files over the Source Material using Upstream Project Releases, Local Projects, and SCMs.

Upstream Project Releases

(almost all recipes)

  • exist anywhere in the form of an archived file (e.g. tarball or ZIP file)

  • correspond to individual recipes, PN and PV match with a upstream archive file

  • archive file can be for any released product that can be built using a recipe

Example

The tslib_1.11.bb recipe points to the following XZ compressed tarball file: https://github.com/kergoth/tslib/releases/download/1.11/tslib-1.11.tar.xz

Local Projects

(no preference for this)

  • custom bits of software the user provides

  • these bits reside somewhere local to a project; perhaps a directory into which the user checks in items

  • use the externalsrc.bbclass class in recipes to include a local projects

Note

Community does not use local projects! Never seen!

Source Code Manager (optional)

(almost vendor or product recipes)

  • get source files through an SCMS such as Git or Subversion or Mercurial

  • repository is cloned and/or checked out

Example

The meta/conf/layer.conf file may tells BitBake how to reach special (perhaps hidden) Git, Subversion, or Mercurial servers:


CENTRAL_GIT ?= "git://10.20.30.40/services/git"
CENTRAL_SVN ?= "svn://10.20.30.40/services/svn"
CENTRAL_HG ?= "hg://10.20.30.40/services/hg"

Source Mirror(s)

(almost all recipes by policy)

The meta-poky/conf/distro/poky.conf file as required by the meta-cedi/conf/distro/cedi.conf file provides the two kinds of mirrors:

PREMIRRORS: the pre-mirrors

check before looking Upstream Project Releases

MIRRORS: the regular mirrors

alternative location should the Upstream Project Releases not available

Package Feeds

_images/package-feeds.png

When the build system generates an image or an SDK, it gets the packages from a package feed area, an intermediary step in the build process. Before placing the packages into package feeds, the build process validates them.

BitBake Tool

http://git.yoctoproject.org/cgit/cgit.cgi/pokymeta/classes/*.bbclass

1. Source Fetching tasks: do_fetchdo_unpack

2. Patch Application tasks: do_patch

3. Configure / Compile / Install tasks: do_configuredo_compiledo_install

4. Analysis / Splitting Packages tasks: do_package/datado_package_write_*do_populate_sysroot

5. Image Generation tasks: do_rootfsdo_image/_complete

6. SDK Generation tasks: do_populate_sdk/_ext

Inherit

tasks by classes:

cmake

base

package/group

image

populate_sdk_base

do_fetch, do_unpack

X

X

do_patch

X

X

do_configure, do_compile, do_install

X

X

do_package/data, do_package_write_*, do_populate_sysroot

X

X

do_rootfs, do_image/_complete

X

populate_sdk/_ext

X

Fetching

_images/source-fetching.png

The first stages of building a recipe are to fetch and unpack the source code. The do_fetch and do_unpack tasks fetch the source files and unpack them into the Build Directory. BitBake uses the SRC_URI variable to point to source files regardless of their location.

Fetching in Recipe

https://github.com/lipro-yocto/meta-lpn-apps (dummy, not yet)

The meta-lpn-apps layer may provides private (closed) recipes. The following related SRC_URI may being used:

lib-utility_hg.bb: "${CENTRAL_HG};module=lib_utility;branch=default;rev=${SRCREV};protocol=ssh"

lib-crypto_hg.bb: "${CENTRAL_HG};module=lib_crypto;branch=default;rev=${SRCREV};protocol=ssh"

app-mcu-ctrl_hg.bb: "${CENTRAL_HG};module=app_mcu_ctrl;branch=default;rev=${SRCREV};protocol=ssh"

boost_1.75.0.bb: "https://dl.bintray.com/boostorg/release/${PV}/source/${BOOST_P}.tar.bz2"

For private recipes: SRCREV is set to SRCREV = "default" and CENTRAL_HG should comes from e.g. meta/conf/layer.conf.

Patching

_images/patching.png

Once source code is fetched and unpacked, BitBake locates patch files and applies them to the source files. The do_patch task processes recipes by using the SRC_URI variable to locate applicable patch files. BitBake finds and applies multiple patches for a single recipe.

Patching in Recipe

http://git.yoctoproject.org/cgit/cgit.cgi/poky
http://cgit.openembedded.org/meta-openembedded

The poky/meta and meta-openembedded/meta-oe layer provides support recipes. The following related SRC_URI being used:

boost_1.75.0.bb: SRC_URI += "file://boost-CVE-2012-2677.patch ..."

opencv_4.5.0.bb: SRC_URI = "git://github.com/opencv/opencv.git;name=opencv ... file://download.patch ..."

Files in the SRC_URI list variable with the .patch or .diff extension are automatically identified as a patch and applied if present beside the recipe.

Compilation

_images/configuration-compile-autoreconf.png

After source code is patched, BitBake executes tasks that configure and compile the source code. This step consists of three tasks: do_configure, do_compile and do_install. It starts at the earliest after all dependencies from the DEPENDS variable in the recipe have been resolved.

Compilation in Recipe

https://github.com/lipro-yocto/meta-lpn-apps (dummy, not yet)

The meta-lpn-apps layer may provides private (closed) recipes. The following related DEPENDS may being used:

lib-utility_hg.bb: "boost"

lib-crypto_hg.bb: "boost opencv"

app-mcu-ctrl_hg.bb: "boost opencv qtbase qtsvg lib-utility lib-crypto"

boost_1.75.0.bb: "boost-build-native zlib bzip2"

opencv_4.5.0.bb: "libtool swig-native bzip2 zlib glib-2.0 libwebp"

For the application: RDEPENDS may be set to additional dependent package names that have to be installed later into targets’ runtime environment, e.g. RDEPENDS = "app-observer-daemon".

Packaging

_images/analysis-for-package-splitting.png

After compilation, the build system analyzes the results and splits the output into packages. The do_package and do_packagedata tasks combine to analyze the files found in the installation directory: debugging symbols, looking at shared library dependencies and relationships of RDEPENDS in the recipe.

Packaging in Recipe

https://github.com/lipro-yocto/meta-lpn-apps (dummy, not yet)

The meta-lpn-apps layer may provides private (closed) recipes. The following related RDEPENDS may being used:

app-mcu-ctrl: "i2c-tools"

packagegroup-lpn-apps-toolchain-target: "boost-staticdev opencv-staticdev lib-utility-staticdev lib-crypto-staticdev"

The following related RDEPENDS may being used as policy; mostly controlled by common Image Features, e.g. IMAGE_FEATURES += "tools-testapps":

packagegroup-central-tools-testapps: "devregs spitools v4l-utils"

hostapd_%: "lpn-mcu-ssid"

Image Generation 1/2

_images/image-generation-1.png

Once packages are split and stored in the Package Feeds, the build system uses BitBake to generate the root filesystem image. The do_rootfs task creates the root filesystem for an image.

Image Generation 2/2

_images/image-generation-2.png

The final stages of the do_rootfs task handle post processing, includes creation of a manifest file and optimizations. After this, processing begins on the image through the do_image task and dynamically created tasks as needed by the image types; ends with do_image_complete.

SDK Generation

_images/sdk-generation.png

The build system uses BitBake to generate the Software Development Kit (SDK) installer script for both the Standard SDK (do_populate_sdk) and Extensible SDK (do_populate_sdk_ext). Like image generation, the SDK script process consists of several stages and depends on many variables.

Images

_images/images.png

The images produced by the build system are compressed forms of the root filesystem that are ready to boot on a target device. Images are written out to the deploy directory.

Image Types

written out to the Build Directory inside the tmp/deploy/images/machine/ folder for the types:

bootloaders

bootloader binary file, if applicable to the target machine

kernel-image

kernel binary file with specifyed naming scheme

kernel-modules (legacy purposes and can be suppressed)

tarball file with all the modules built for a kernel

root-filesystem-image

root filesystem image files for the target machine

symlinks

symbolic link that points to the most recently built file for each machine

Application Devel-SDK

_images/sdk.png

The SDK produced by the build system is the compressed form of a self-extracting installer, which consists of a cross-development toolchain, a set of libraries and headers and an SDK environment setup script. SDKs are written out to the deploy directory.

SDK Types

written out to the Build Directory inside the tmp/deploy/sdk/ folder for two different types:

Standard SDK: bitbake -c populate_sdk imagename

standard (legacy) SDK w/o any binding into the build system

Extensible SDK: bitbake -c populate_sdk_ext imagename

extensible (next) SDK with complete binding into the build system

Note

An Extensible SDK requires a shared infrastructure!

Final Closer Look

The Canadian Cross (reduced for host and target only)

_images/lpn-central-bsp-canadian-cross.svg

Read-The-Docs

  • Documentations

  • Presentations

  • Books

Documentations

Presentations

Books


Table 1 Books about and with Yocto

2016 (480 pages)
Embedded Linux Systems with the Yocto Project (ISBN: 978-0-13344-324-0)
978-0-13344-324-0
[Str16]

2017 (162 pages)
Embedded Linux Development using Yocto Projects - Second Edition (ISBN: 978-1-78847-046-9)
978-1-78847-046-9
[SA17]

2017 (478 pages)
Mastering Embedded Linux Programming - Second Edition (ISBN: 978-1-78728-328-2)
978-1-78728-328-2
[Sim17]

2018 (456 pages)
Embedded Linux Development Using Yocto Project Cookbook - Second Edition (ISBN: 978-1-78839-921-0)
978-1-78839-921-0
[Gon18]

2016 (989 pages)
Linux: Embedded Development - Learning Path (ISBN: 978-1-78712-445-5)
978-1-78712-445-5
[VGS16]

missing one?
Question Mark

2016 (214 pages)
Yocto for Raspberry Pi (ISBN: 978-1-78528-195-2)
978-1-78528-195-2
[TM15]

2015 (144 pages)
Using Yocto Project with BeagleBone Black (ISBN: 978-1-78528-973-6)
978-1-78528-973-6
[Sad15]

Thank You

TODO: complete chapter

_images/todo.jpg