The utility for building of AlmaLinux distributions (repos, ISO images).
Go to file
Adam Williamson 4f69f6c242 pkgset: optimize cache check (saves 20 minutes)
The pkgset phase takes around 35 minutes in current composes.
Around 20 minutes of that is spent creating these per-arch
subsets of the global package set. In a rather roundabout way
(see #1794 ), I figured out that almost all of this time is
spent in this cache check, which is broken for a subtle reason.

Python's `in` keyword works by first attempting to call the
container's magic `__contains__` method. If the container does
not implement `__contains__`, it falls back to iteration - it
tries to iterate over the container until it either hits what
it's looking for, or runs out. (If the container implements
neither, you get an error).

The FileCache instance's `file_cache` is a plain Python dict.
dicts have a very efficient `__contains__` implementation, so
doing `foo in (somedict)` is basically always very fast no matter
how huge the dict is. FileCache itself, though, implements
`__iter__` by returning an iterator over the `file_cache` dict's
keys, but it does *not* implement `__contains__`. So when we do
`foo in self.file_cache`, Python has to iterate over every key
in the dict until it hits foo or runs out. This is massively
slower than `foo in self.file_cache.file_cache`, which uses the
efficient `__contains__` method.

Because these package sets are so huge, and we're looping over
*one* huge set and checking each package from it against the cache
of another, increasingly huge, set, this effect becomes massive.
To make it even worse, I ran a few tests where I added a debug log
if we ever hit the cache, and it looks like we never actually do -
so every check has to iterate through the entire dict.

We could probably remove this entirely, but changing it to check
the dict instead of the FileCache instance makes it just about as
fast as taking it out, so I figured let's go with that in case
there's some unusual scenario in which the cache does work here.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
(cherry picked from commit c8fe99b1aa5a9a9b941b7515cda367d24829dedf)
2025-09-29 18:15:21 +03:00
contrib Add config file for cleaning up cache files 2023-11-10 16:51:51 +02:00
doc Update phase diagram 2025-09-29 18:15:20 +03:00
pungi pkgset: optimize cache check (saves 20 minutes) 2025-09-29 18:15:21 +03:00
pungi_utils Fix formatting of long line 2024-08-30 13:39:51 +03:00
share Allow setting <kojitag/> in <modules/> in variants.xml to get the modules from this Koji tag. 2018-03-21 14:33:45 +01:00
tests Install dnf4 into test image 2025-09-29 18:15:21 +03:00
.gitignore nomacboot option for livemedia koji tasks 2022-03-23 09:36:51 +01:00
1860.patch Release 4.10.1 2025-09-29 18:14:47 +03:00
AUTHORS extra-files: Write a metadata file enumerating extra files 2016-09-07 13:02:48 +02:00
COPYING Remove FSF address from comments 2016-09-23 10:26:43 +02:00
git-changelog git-changelog: Fix running on Python 3 2019-07-02 15:05:25 +02:00
GPL Update GPL to latest version from https://www.gnu.org/licenses/gpl-2.0.txt 2015-06-25 07:50:03 -04:00
Makefile Use pytest instead of nosetests 2020-07-29 14:57:16 +08:00
MANIFEST.in Default bztar format for sdist command 2023-11-10 16:50:33 +02:00
pungi.spec Release 4.10.1 2025-09-29 18:14:47 +03:00
README.md Correct irc network name & add matrix room 2021-10-25 07:26:18 +00:00
requirements.txt Clean up requirements 2024-08-30 13:40:02 +03:00
setup.cfg Default bztar format for sdist command 2023-11-10 16:50:33 +02:00
setup.py Release 4.7.0 2024-08-30 13:40:54 +03:00
sources Release 4.10.1 2025-09-29 18:14:47 +03:00
test-requirements.txt Install unittest2 only on python 2 2024-08-30 13:39:26 +03:00
TODO Drop buildinstall method 2024-08-30 13:30:35 +03:00
tox.ini Clean up requirements 2024-08-30 13:40:02 +03:00

Pungi

Pungi is a distribution compose tool.

Composes are release snapshots that contain release deliverables such as:

  • installation trees
    • RPMs
    • repodata
    • comps
  • (bootable) ISOs
  • kickstart trees
    • anaconda images
    • images for PXE boot

Tool overview

Pungi consists of multiple separate executables backed by a common library.

The main entry-point is the pungi-koji script. It loads the compose configuration and kicks off the process. Composing itself is done in phases. Each phase is responsible for generating some artifacts on disk and updating the compose object that is threaded through all the phases.

Pungi itself does not actually do that much. Most of the actual work is delegated to separate executables. Pungi just makes sure that all the commands are invoked in the appropriate order and with correct arguments. It also moves the artifacts to correct locations.