virt-what/SOURCES/0009-Introduce-virt-what-cvm-program.patch

715 lines
20 KiB
Diff
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

From 772dfd3a966d766d4566fd048f8b0178f7f827e5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= <berrange@redhat.com>
Date: Fri, 26 May 2023 12:39:03 +0100
Subject: [PATCH] Introduce 'virt-what-cvm' program
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
The 'virt-what' program prints facts that reflect the hypervisor that
the guest is running under.
The new complementary 'virt-what-cvm' program prints facts that reflect
the confidential virtualization technology the guest is running under,
if any.
It is kept as a separate tool, rather than incorporating the facts into
'virt-what' output because it is considering a different aspect of the
virtualization. Furthermore there are specific security concerns around
the usage of facts reported by 'virt-what-cvm'.
The tool has been tested in a number of environments
* Azure confidential guest with AMD SEV-SNP (GA)
* Azure confidential guest with Intel TDX (technology preview)
* Fedora 37 QEMU/KVM guest with AMD SEV (GA)
* Fedora 37 QEMU/KVM guest with AMD SEV-ES (GA)
* Fedora 38 QEMU/KVM guest with AMD SEV-SNP + SVSM (devel snapshot)
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
(cherry picked from commit 22e33361e980ddefe08e2c68bf145943af8375f9)
---
.gitignore | 3 +
Makefile.am | 12 +-
configure.ac | 3 +
virt-what-cvm.c | 404 ++++++++++++++++++++++++++++++++++++++++++++++
virt-what-cvm.pod | 195 ++++++++++++++++++++++
5 files changed, 613 insertions(+), 4 deletions(-)
create mode 100644 virt-what-cvm.c
create mode 100644 virt-what-cvm.pod
diff --git a/.gitignore b/.gitignore
index 4833fd6be..ba897a162 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,5 +26,8 @@ Makefile.in
/test-driver
/virt-what
/virt-what-cpuid-helper
+/virt-what-cvm
+/virt-what-cvm.1
+/virt-what-cvm.txt
/virt-what.1
/virt-what.txt
diff --git a/Makefile.am b/Makefile.am
index 543513204..2050bef8d 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -24,20 +24,24 @@ EXTRA_DIST = .gitignore virt-what.in virt-what.pod
SUBDIRS = . tests
sbin_SCRIPTS = virt-what
+sbin_PROGRAMS = virt-what-cvm
libexec_PROGRAMS = virt-what-cpuid-helper
if HOST_CPU_IA64
libexec_PROGRAMS += virt-what-ia64-xen-rdtsc-test
endif
+virt_what_cvm_LDADD = $(TPM2_TSS_LIBS)
+virt_what_cvm_CFLAGS = $(TPM2_TSS_CFLAGS)
+
if HAVE_POD2MAN
-CLEANFILES += virt-what.1 virt-what.txt
-man_MANS = virt-what.1
+CLEANFILES += virt-what.1 virt-what-cvm.1 virt-what.txt virt-what-cvm.txt
+man_MANS = virt-what.1 virt-what-cvm.1
-virt-what.1: virt-what.pod
+%.1: %.pod
pod2man -c "Virtualization Support" --release "$(PACKAGE)-$(VERSION)" \
$? > $@
-virt-what.txt: virt-what.pod
+%.txt: %.pod
pod2text $? > $@
endif
diff --git a/configure.ac b/configure.ac
index 4dd2c9731..b1dadd64d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -32,6 +32,9 @@ dnl Architecture we are compiling for.
AC_CANONICAL_HOST
AM_CONDITIONAL([HOST_CPU_IA64], [ test "x$host_cpu" = "xia64" ])
+PKG_HAVE_DEFINE_WITH_MODULES(TPM2_TSS, tss2-esys, [tpm2-tss package])
+
+
dnl List of tests.
tests="\
alibaba-cloud-arm \
diff --git a/virt-what-cvm.c b/virt-what-cvm.c
new file mode 100644
index 000000000..407efb492
--- /dev/null
+++ b/virt-what-cvm.c
@@ -0,0 +1,404 @@
+/* virt-what-cvm-helper: Are we running inside confidential VM
+ * Copyright (C) 2023 Red Hat Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+ */
+
+#include "config.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <string.h>
+#include <stdbool.h>
+#include <fcntl.h>
+#include <unistd.h>
+#include <getopt.h>
+#ifdef HAVE_TPM2_TSS
+#include <tss2/tss2_esys.h>
+#include <assert.h>
+#endif
+
+static bool dodebug = false;
+
+#define debug(...) do { if (dodebug) fprintf(stderr, __VA_ARGS__); } while(0)
+
+/*
+ * AMD64 Architecture Programmers Manual Volume 3:
+ * General-Purpose and System Instructions.
+ * Chapter: E4.1 - Maximum Extended Function Number and Vendor String
+ * https://www.amd.com/system/files/TechDocs/24594.pdf
+ */
+#define CPUID_GET_HIGHEST_FUNCTION 0x80000000
+
+/*
+ * AMD64 Architecture Programmers Manual Volume 3:
+ * General-Purpose and System Instructions.
+ * Chapter: E4.17 - Encrypted Memory Capabilities
+ * https://www.amd.com/system/files/TechDocs/24594.pdf
+ */
+#define CPUID_AMD_GET_ENCRYPTED_MEMORY_CAPABILITIES 0x8000001f
+
+/*
+ * AMD64 Architecture Programmers Manual Volume 3:
+ * General-Purpose and System Instructions.
+ * Chapter: 15.34.10 - SEV_STATUS MSR
+ * https://www.amd.com/system/files/TechDocs/24593.pdf
+ */
+#define MSR_AMD64_SEV 0xc0010131
+
+/*
+ * Intel® TDX Module v1.5 Base Architecture Specification
+ * Chapter: 11.2
+ * https://www.intel.com/content/www/us/en/content-details/733575/intel-tdx-module-v1-5-base-architecture-specification.html
+ */
+
+#define CPUID_INTEL_TDX_ENUMERATION 0x21
+
+
+#define CPUID_SIG_AMD "AuthenticAMD"
+#define CPUID_SIG_INTEL "GenuineIntel"
+#define CPUID_SIG_INTEL_TDX "IntelTDX "
+
+/*
+ * This TPM NV data format is not explicitly documented anywhere,
+ * but the header definition is present in code at:
+ *
+ * https://github.com/kinvolk/azure-cvm-tooling/blob/main/az-snp-vtpm/src/hcl.rs
+ */
+#define TPM_AZURE_HCLA_REPORT_INDEX 0x01400001
+
+struct TPMAzureHCLAHeader {
+ uint32_t signature;
+ uint32_t version;
+ uint32_t report_len;
+ uint32_t report_type;
+ uint32_t unknown[4];
+};
+
+/* The bytes for "HCLA" */
+#define TPM_AZURE_HCLA_SIGNATURE 0x414C4348
+#define TPM_AZURE_HCLA_VERSION 0x1
+#define TPM_AZURE_HCLA_REPORT_TYPE_SNP 0x2
+
+#if defined(__x86_64__)
+
+#ifdef HAVE_TPM2_TSS
+static char *
+tpm_nvread(uint32_t nvindex, size_t *retlen)
+{
+ TSS2_RC rc;
+ ESYS_CONTEXT *ctx = NULL;
+ ESYS_TR primary = ESYS_TR_NONE;
+ ESYS_TR session = ESYS_TR_NONE;
+ ESYS_TR nvobj = ESYS_TR_NONE;
+ TPM2B_NV_PUBLIC *pubData = NULL;
+ TPMT_SYM_DEF sym = {
+ .algorithm = TPM2_ALG_AES,
+ .keyBits = { .aes = 128 },
+ .mode = { .aes = TPM2_ALG_CFB }
+ };
+ char *ret;
+ size_t retwant;
+
+ rc = Esys_Initialize(&ctx, NULL, NULL);
+ if (rc != TSS2_RC_SUCCESS)
+ return NULL;
+
+ rc = Esys_Startup(ctx, TPM2_SU_CLEAR);
+ debug("tpm startup %d\n", rc);
+ if (rc != TSS2_RC_SUCCESS)
+ goto error;
+
+ rc = Esys_StartAuthSession(ctx, ESYS_TR_NONE, ESYS_TR_NONE,
+ ESYS_TR_NONE, ESYS_TR_NONE, ESYS_TR_NONE,
+ NULL, 0,
+ &sym, TPM2_ALG_SHA256, &session);
+ debug("tpm auth session %d\n", rc);
+ if (rc != TSS2_RC_SUCCESS)
+ goto error;
+
+ rc = Esys_TR_FromTPMPublic(ctx, nvindex, ESYS_TR_NONE,
+ ESYS_TR_NONE, ESYS_TR_NONE, &nvobj);
+ debug("tpm from public %d\n", rc);
+ if (rc != TSS2_RC_SUCCESS)
+ goto error;
+
+ rc = Esys_NV_ReadPublic(ctx, nvobj, ESYS_TR_NONE,
+ ESYS_TR_NONE, ESYS_TR_NONE,
+ &pubData, NULL);
+ debug("tpm read public %d\n", rc);
+ if (rc != TPM2_RC_SUCCESS)
+ goto error;
+
+ retwant = pubData->nvPublic.dataSize;
+ free(pubData);
+ *retlen = 0;
+ ret = malloc(retwant);
+ assert(ret);
+ while (*retlen < retwant) {
+ size_t want = retwant - *retlen;
+ TPM2B_MAX_NV_BUFFER *data = NULL;
+ if (want > 1024)
+ want = 1024;
+ rc = Esys_NV_Read(ctx, ESYS_TR_RH_OWNER, nvobj, session, ESYS_TR_NONE, ESYS_TR_NONE,
+ want, *retlen, &data);
+ debug("tpm nv read %d\n", rc);
+ if (rc != TPM2_RC_SUCCESS) {
+ free(ret);
+ goto error;
+ }
+
+ memcpy(ret + *retlen, data->buffer, data->size);
+ *retlen += data->size;
+ free(data);
+ }
+
+ return ret;
+
+ error:
+ if (nvobj != ESYS_TR_NONE)
+ Esys_FlushContext(ctx, nvobj);
+ if (session != ESYS_TR_NONE)
+ Esys_FlushContext(ctx, session);
+ if (primary != ESYS_TR_NONE)
+ Esys_FlushContext(ctx, primary);
+ Esys_Finalize(&ctx);
+ *retlen = 0;
+ return NULL;
+}
+#else /* ! HAVE_TPM2_TSS */
+static char *
+tpm_nvread(uint32_t nvindex, size_t *retlen)
+{
+ return NULL;
+}
+#endif /* ! HAVE_TPM2_TSS */
+
+/* Copied from the Linux kernel definition in
+ * arch/x86/include/asm/processor.h
+ */
+static inline void
+cpuid (uint32_t *eax, uint32_t *ebx, uint32_t *ecx, uint32_t *edx)
+{
+ debug("CPUID func %x %x\n", *eax, *ecx);
+ asm volatile ("cpuid"
+ : "=a" (*eax), "=b" (*ebx), "=c" (*ecx), "=d" (*edx)
+ : "0" (*eax), "2" (*ecx)
+ : "memory");
+ debug("CPUID result %x %x %x %x\n", *eax, *ebx, *ecx, *edx);
+}
+
+
+static uint32_t
+cpuid_leaf (uint32_t eax, char *sig)
+{
+ uint32_t *sig32 = (uint32_t *) sig;
+
+ cpuid (&eax, &sig32[0], &sig32[2], &sig32[1]);
+ sig[12] = 0; /* \0-terminate the string to make string comparison possible */
+ debug("CPUID sig %s\n", sig);
+ return eax;
+}
+
+#define MSR_DEVICE "/dev/cpu/0/msr"
+
+static uint64_t
+msr (off_t index)
+{
+ uint64_t ret;
+ int fd = open (MSR_DEVICE, O_RDONLY);
+ if (fd < 0) {
+ debug ("Cannot open MSR device %s", MSR_DEVICE);
+ return 0;
+ }
+
+ if (pread (fd, &ret, sizeof(ret), index) != sizeof(ret))
+ ret = 0;
+
+ close (fd);
+
+ debug ("MSR %llx result %llx\n", (unsigned long long)index,
+ (unsigned long long)ret);
+ return ret;
+}
+
+bool
+cpu_sig_amd_azure (void)
+{
+ size_t datalen = 0;
+ char *data = tpm_nvread(TPM_AZURE_HCLA_REPORT_INDEX, &datalen);
+ struct TPMAzureHCLAHeader *header = (struct TPMAzureHCLAHeader *)data;
+ bool ret;
+
+ if (!data)
+ return false;
+
+ if (datalen < sizeof(struct TPMAzureHCLAHeader)) {
+ debug ("TPM data len is too small to be an Azure HCLA report");
+ return false;
+ }
+
+ debug ("Azure TPM HCLA report header sig %x ver %x type %x\n",
+ header->signature, header->version, header->report_type);
+
+ ret = (header->signature == TPM_AZURE_HCLA_SIGNATURE &&
+ header->version == TPM_AZURE_HCLA_VERSION &&
+ header->report_type == TPM_AZURE_HCLA_REPORT_TYPE_SNP);
+ debug ("Azure TPM HCLA report present ? %d\n", ret);
+
+ free(data);
+ return ret;
+}
+
+static void
+cpu_sig_amd (void)
+{
+ uint32_t eax, ebx, ecx, edx;
+ uint64_t msrval;
+
+ eax = CPUID_GET_HIGHEST_FUNCTION;
+ ebx = ecx = edx = 0;
+
+ cpuid (&eax, &ebx, &ecx, &edx);
+
+ if (eax < CPUID_AMD_GET_ENCRYPTED_MEMORY_CAPABILITIES)
+ return;
+
+ eax = CPUID_AMD_GET_ENCRYPTED_MEMORY_CAPABILITIES;
+ ebx = ecx = edx = 0;
+
+ cpuid (&eax, &ebx, &ecx, &edx);
+
+ /* bit 1 == CPU supports SEV feature
+ *
+ * Note, Azure blocks this CPUID leaf from its SEV-SNP
+ * guests, so we must fallback to probing the TPM which
+ * exposes a SEV-SNP attestation report as evidence.
+ */
+ if (!(eax & (1 << 1))) {
+ debug ("No sev in CPUID, try azure TPM NV\n");
+
+ if (cpu_sig_amd_azure()) {
+ puts ("amd-sev-snp");
+ puts ("azure-hcl");
+ } else {
+ debug("No azure TPM NV\n");
+ }
+ return;
+ }
+
+ msrval = msr (MSR_AMD64_SEV);
+
+ /* Test reverse order, since the SEV-SNP bit implies
+ * the SEV-ES bit, which implies the SEV bit */
+ if (msrval & (1 << 2)) {
+ puts ("amd-sev-snp");
+ } else if (msrval & (1 << 1)) {
+ puts ("amd-sev-es");
+ } else if (msrval & (1 << 0)) {
+ puts ("amd-sev");
+ }
+}
+
+static void
+cpu_sig_intel (void)
+{
+ uint32_t eax, ebx, ecx, edx;
+ char sig[13];
+
+ eax = CPUID_GET_HIGHEST_FUNCTION;
+ ebx = ecx = edx = 0;
+
+ cpuid (&eax, &ebx, &ecx, &edx);
+ debug ("CPUID max function: %x %x %x %x\n", eax, ebx, ecx,edx);
+
+ if (eax < CPUID_INTEL_TDX_ENUMERATION)
+ return;
+
+ memset (sig, 0, sizeof sig);
+ cpuid_leaf (CPUID_INTEL_TDX_ENUMERATION, sig);
+
+ if (memcmp (sig, CPUID_SIG_INTEL_TDX, sizeof(sig)) == 0)
+ puts ("intel-tdx");
+}
+
+static void
+cpu_sig (void)
+{
+ char sig[13];
+
+ memset (sig, 0, sizeof sig);
+ cpuid_leaf (0, sig);
+
+ if (memcmp (sig, CPUID_SIG_AMD, sizeof(sig)) == 0)
+ cpu_sig_amd ();
+ else if (memcmp (sig, CPUID_SIG_INTEL, sizeof(sig)) == 0)
+ cpu_sig_intel ();
+}
+
+#else /* !x86_64 */
+
+static void
+cpu_sig (void)
+{
+ /* nothing for other architectures */
+}
+
+#endif
+
+int
+main(int argc, char **argv)
+{
+ int c;
+
+ while (true) {
+ int option_index = 0;
+ static struct option long_options[] = {
+ {"debug", no_argument, 0, 'd' },
+ {"version", no_argument, 0, 'v' },
+ {"help", no_argument, 0, 'h'},
+ {0, 0, 0, 0 }
+ };
+
+ c = getopt_long(argc, argv, "dvh",
+ long_options, &option_index);
+ if (c == -1)
+ break;
+
+ switch (c) {
+ case 'd':
+ dodebug = true;
+ break;
+ case 'v':
+ fprintf(stdout, "%s\n", PACKAGE_VERSION);
+ exit(EXIT_SUCCESS);
+ break;
+ case 'h':
+ default: /* '?' */
+ fprintf(c == 'h' ? stdout : stderr,
+ "Usage: %s [--debug|-d] [--help|-h] [--version|-v]\n",
+ argv[0]);
+ exit(c == 'h' ? EXIT_SUCCESS : EXIT_FAILURE);
+ }
+ }
+
+ if (!dodebug)
+ setenv("TSS2_LOG", "all+none", 1);
+
+ cpu_sig ();
+
+ exit(EXIT_SUCCESS);
+}
diff --git a/virt-what-cvm.pod b/virt-what-cvm.pod
new file mode 100644
index 000000000..12cfc6a96
--- /dev/null
+++ b/virt-what-cvm.pod
@@ -0,0 +1,195 @@
+=encoding utf8
+
+=head1 NAME
+
+virt-what-cvm - detect if we are running in a confidential virtual machine
+
+=head1 SUMMARY
+
+virt-what-cvm [options]
+
+=head1 DESCRIPTION
+
+C<virt-what-cvm> is a tool which can be used to detect if the program
+is running in a confidential virtual machine.
+
+The program prints out a list of "facts" about the confidential virtual
+machine, derived from heuristics. One fact is printed per line.
+
+If nothing is printed and the script exits with code 0 (no error),
+then it can mean I<either> that the program is running on bare-metal
+I<or> the program is running inside a non-confidential virtual machine,
+I<or> inside a type of confidential virtual machine which we don't know
+about or cannot detect.
+
+=head1 FACTS
+
+=over 4
+
+=item B<amd-sev>
+
+This is a confidential guest running with AMD SEV technology
+
+Status: tested on Fedora 37 QEMU+KVM
+
+=item B<amd-sev-es>
+
+This is a confidential guest running with AMD SEV-ES technology
+
+Status: tested on Fedora 37 QEMU+KVM
+
+=item B<amd-sev-snp>
+
+This is a confidential guest running with AMD SEV-SNP technology
+
+Status: tested on Microsoft Azure SEV-SNP CVM
+
+Status: tested on Fedora 38 QEMU+KVM SEV-SNP (devel snapshot)
+
+=item B<intel-tdx>
+
+This is a confidential guest running with Intel TDX technology
+
+Status: tested on Microsoft Azure TDX CVM (preview)
+
+=item B<azure-hcl>
+
+This is a confidential guest running unenlightened under the
+Azure HCL (Host Compatibility Layer). This will be paired with
+B<amd-sev-snp>.
+
+Status: tested on Microsoft Azure SEV-SNP CVM
+
+=back
+
+=head1 EXIT STATUS
+
+Programs that use or wrap C<virt-what-cvm> should check that the exit
+status is 0 before they attempt to parse the output of the command.
+
+A non-zero exit status indicates some error, for example, an
+unrecognized command line argument. If the exit status is non-zero
+then the output "facts" (if any were printed) cannot be guaranteed and
+should be ignored.
+
+The exit status does I<not> have anything to do with whether the
+program is running on baremetal or under confidential virtualization,
+nor with whether C<virt-what-cvm> managed detection "correctly" (which
+is basically unknowable given the large variety of virtualization
+systems out there)
+
+=head1 RUNNING VIRT-WHAT-CVM FROM OTHER PROGRAMS
+
+C<virt-what-cvm> is designed so that you can easily run it from
+other programs or wrap it up in a library.
+
+Your program should check the exit status (see the section above).
+
+=head1 IMPORTANT NOTE
+
+This program detects whether it is likely to be running within a known
+confidential VM, but does I<NOT> prove that the environment is trustworthy.
+To attain trust in the environment requires an attestation report for the
+virtual machine, which is then verified by an already trusted 3rd party.
+
+The hardware features that this program relies on to establish facts
+about the confidential virtualization environment, are those features
+whose behaviour will be proved by verification of an attestation report.
+
+This program I<MAY> have false positives. ie it may report that it is a
+confidential VM when it is in fact a non-confidential VM faking it.
+
+This program I<SHOULD NOT> have false negatives. ie it should not fail to
+report existance of a confidential VM. Caveat that this only applies to
+environments which have been explicitly tested.
+
+If this program does print a fact, this can be used for enabling or
+disabling use of certain features, according to whether they are
+appropriate for a confidential environment. None the less, the VM
+I<MUST NOT> be trusted until an attestation report is verified.
+
+As a protection against false negatives from this tool, environments
+requiring high assurance should take one or more of these measures:
+
+ * The facts reported by this program I<SHOULD> should be measured
+ into one of the TPM PCRs
+ * The attestation report I<SHOULD> cover the facts reported by
+ this program
+ * The attestation report I<SHOULD> should cover the enablement
+ status of any features affected by decisions involving facts
+ reported by this tool
+
+=head1 SEE ALSO
+
+L<http://people.redhat.com/~rjones/virt-what/>,
+L<https://github.com/Azure/confidential-computing-cvm-guest-attestation>,
+L<https://virtee.io/>
+
+=head1 AUTHORS
+
+Daniel P. Berrangé <berrange @ redhat . com>
+
+=head1 COPYRIGHT
+
+(C) Copyright 2023 Red Hat Inc.,
+L<http://people.redhat.com/~rjones/virt-what/>
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+
+=head1 REPORTING BUGS
+
+Bugs can be viewed on the Red Hat Bugzilla page:
+L<https://bugzilla.redhat.com/>.
+
+If you find a bug in virt-what-cvm, please follow these steps to report it:
+
+=over 4
+
+=item 1. Check for existing bug reports
+
+Go to L<https://bugzilla.redhat.com/> and search for similar bugs.
+Someone may already have reported the same bug, and they may even
+have fixed it.
+
+=item 2. Capture debug and error messages
+
+Run
+
+ virt-what-cvm -d > virt-what-cvm.log 2>&1
+
+and keep I<virt-what-cvm.log>. It may contain error messages which you
+should submit with your bug report.
+
+=item 3. Get version of virt-what-cvm.
+
+Run
+
+ virt-what-cvm --version
+
+=item 4. Submit a bug report.
+
+Go to L<https://bugzilla.redhat.com/> and enter a new bug.
+Please describe the problem in as much detail as possible.
+
+Remember to include the version numbers (step 3) and the debug
+messages file (step 2) and as much other detail as possible.
+
+=item 5. Assign the bug to rjones @ redhat.com
+
+Assign or reassign the bug to B<rjones @ redhat.com> (without the
+spaces). You can also send me an email with the bug number if you
+want a faster response.
+
+=back
--
2.43.0