TranslateProject/sources/tech/20210416 Use the DNF local plugin to speed up your home lab.md

[#]: subject: (Use the DNF local plugin to speed up your home lab)
[#]: via: (https://fedoramagazine.org/use-the-dnf-local-plugin-to-speed-up-your-home-lab/)
[#]: author: (Brad Smith https://fedoramagazine.org/author/buckaroogeek/)
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )

Use the DNF local plugin to speed up your home lab
======

![][1]

Photo by [Sven Hornburg][2] on [Unsplash][3]

### Introduction

If you are a Fedora Linux enthusiast or a developer working with multiple instances of Fedora Linux then you might benefit from the [DNF local][4] plugin. An example of someone who would benefit from the DNF local plugin would be an enthusiast who is running a cluster of Raspberry Pis. Another example would be someone running several virtual machines managed by Vagrant. The DNF local plugin reduces the time required for DNF transactions. It accomplishes this by transparently creating and managing a local RPM repository. Because accessing files on a local file system is significantly faster than downloading them repeatedly, multiple Fedora Linux machines will see a significant performance improvement when running _dnf_ with the DNF local plugin enabled.

I recently started using this plugin after reading a tip from Glenn Johnson (aka glennzo) in [a 2018 fedoraforum.org post][5]. While working on a Raspberry Pi based Kubernetes cluster running Fedora Linux and also on several container-based services, I winced with every DNF update on each Pi or each container that downloaded a duplicate set of rpms across my expensive internet connection. In order to improve this situation, I searched for a solution that would cache rpms for local reuse. I wanted something that would not require any changes to repository configuration files on every machine. I also wanted it to continue to use the network of Fedora Linux mirrors. I didn’t want to use a single mirror for all updates.

### Prior art

An internet search yields two common solutions that eliminate or reduce repeat downloads of the same RPM set – create a private Fedora Linux mirror or set up a caching proxy.

Fedora provides guidance on setting up a [private mirror][6]. A mirror requires a lot of bandwidth and disk space and significant work to maintain. A full private mirror would be too expensive and it would be overkill for my purposes.

The most common solution I found online was to implement a caching proxy using Squid. I had two concerns with this type of solution. First, I would need to edit repository definitions stored in _/etc/yum.repo.d_ on each virtual and physical machine or container to use the same mirror. Second, I would need to use _http_ and not _https_ connections which would introduce a security risk.

After reading Glenn’s 2018 post on the DNF local plugin, I searched for additional information but could not find much of anything besides the sparse documentation for the plugin on the DNF documentation web site. This article is intended to raise awareness of this plugin.

### About the DNF local plugin

The [online documentation][4] provides a succinct description of the plugin: “Automatically copy all downloaded packages to a repository on the local filesystem and generating repo metadata”. The magic happens when there are two or more Fedora Linux machines configured to use the plugin and to share the same local repository. These machines can be virtual machines or containers running on a host and all sharing the host filesystem, or separate physical hardware on a local area network sharing the file system using a network-based file system sharing technology. The plugin, once configured, handles everything else transparently. Continue to use _dnf_ as before. _dnf_ will check the plugin repository for rpms, then proceed to download from a mirror if not found. The plugin will then cache all rpms in the local repository regardless of their upstream source – an official Fedora Linux repository or a third-party RPM repository – and make them available for the next run of _dnf_.

### Install and configure the DNF local plugin

Install the plugin using _dnf_. The _createrepo_c_ packages will be installed as a dependency. The latter is used, if needed, to create the local repository.

```
sudo dnf install python3-dnf-plugin-local
```

The plugin configuration file is stored at /_etc/dnf/plugins/local.conf_. An example copy of the file is provided below. The only change required is to set the _repodir_ option. The _repodir_ option defines where on the local filesystem the plugin will keep the RPM repository.

```
[main]
enabled = true
# Path to the local repository.
# repodir = /var/lib/dnf/plugins/local

# Createrepo options. See man createrepo_c
[createrepo]
# This option lets you disable createrepo command. This could be useful
# for large repositories where metadata is priodically generated by cron
# for example. This also has the side effect of only copying the packages
# to the local repo directory.
enabled = true

# If you want to speedup createrepo with the --cachedir option. Eg.
# cachedir = /tmp/createrepo-local-plugin-cachedir

# quiet = true

# verbose = false
```

Change _repodir_ to the filesystem directory where you want the RPM repository stored. For example, change _repodir_ to _/srv/repodir_ as shown below.

```
...
# Path to the local repository.
# repodir = /var/lib/dnf/plugins/local
repodir = /srv/repodir
...
```

Finally, create the directory if it does not already exist. If this directory does not exist, _dnf_ will display some errors when it first attempts to access the directory. The plugin will create the directory, if necessary, despite the initial errors.

```
sudo mkdir -p /srv/repodir
```

Repeat this process on any virtual machine or container that you want to share the local repository. See the use cases below for more information. An alternative configuration using NFS (network file system) is also provided below.

### How to use the DNF local plugin

After you have installed the plugin, you do not need to change how you use _dnf_. The plugin will cause a few additional steps to run transparently behind the scenes whenever _dnf_ is called. After _dnf_ determines which rpms to update or install, the plugin will try to retrieve them from the local repository before trying to download them from a mirror. After _dnf_ has successfully completed the requested updates, the plugin will copy any rpms downloaded from a mirror to the local repository and then update the local repository’s metadata. The downloaded rpms will then be available in the local repository for the next _dnf_ client.

There are two points to be aware of. First, benefits from the local repository only occur if multiple machines share the same architecture (for example, x86_64 or aarch64). Virtual machines and containers running on a host will usually share the same architecture as the host. But if there is only one aarch64 device and one x86_64 device there is little real benefit to a shared local repository unless one of the devices is constantly reset and updated which is common when developing with a virtual machine or container. Second, I have not explored how robust the local repository is to multiple _dnf_ clients updating the repository metadata concurrently. I therefore run _dnf_ from multiple machines serially rather than in parallel. This may not be a real concern but I want to be cautious.

The use cases outlined below assume that work is being done on Fedora Workstation. Other desktop environments can work as well but may take a little extra effort. I created a GitHub repository with examples to help with each use case. Click the _Code_ button at <https://github.com/buckaroogeek/dnf-local-plugin-examples> to clone the repository or to download a zip file.

#### Use case 1: networked physical machines

The simplest use case is two or more Fedora Linux computers on the same network. Install the DNF local plugin on each Fedora Linux machine and configure the plugin to use a repository on a network-aware file system. There are many network-aware file systems to choose from. Which file system you will use will probably be influenced by the existing devices on your network.

For example, I have a small Synology Network Attached Storage device (NAS) on my home network. The web admin interface for the Synology makes it very easy to set up a NFS server and export a file system share to other devices on the network. NFS is a shared file system that is well supported on Fedora Linux. I created a share on my NAS named _nfs-dnf_ and exported it to all the Fedora Linux machines on my network. For the sake of simplicity, I am omitting the details of the security settings. However, please keep in mind that security is always important even on your own local network. If you would like more information about NFS, the online Red Hat Enable Sysadmin magazine has an [informative post][7] that covers both client and server configurations on Red Hat Enterprise Linux. They translate well to Fedora Linux.

I configured the NFS client on each of my Fedora Linux machines using the steps shown below. In the below example, _quga.lan_ is the hostname of my NAS device.

Install the NFS client on each Fedora Linux machine.

```
$ sudo dnf install nfs-utils
```

Get the list of exports from the NFS server:

```
$ showmount -e quga.lan
Export list for quga.lan:
/volume1/nfs-dnf  pi*.lan
```

Create a local directory to be used as a mount point on the Fedora Linux client:

```
$ sudo mkdir -p /srv/repodir
```

Mount the remote file system on the local directory. See _man mount_ for more information and options.

```
$ sudo mount -t nfs -o vers=4 quga.lan:/nfs-dnf /srv/repodir
```

The DNF local plugin will now work until as long as the client remains up. If you want the NFS export to be automatically mounted when the client is rebooted, then you must to edit /_etc/fstab_ as demonstrated below. I recommend making a backup of _/etc/fstab_ before editing it. You can substitute _vi_ with _nano_ or another editor of your choice if you prefer.

```
$ sudo vi /etc/fstab
```

Append the following line at the bottom of _/etc/fstab_, then save and exit.

```
quga.lan:/volume1/nfs-dnf /srv/repodir nfs defaults,timeo=900,retrans=5,_netdev 0 0
```

Finally, notify systemd that it should rescan _/etc/fstab_ by issuing the following command.

```
$ sudo systemctl daemon-reload
```

NFS works across the network and, like all network traffic, may be blocked by firewalls on the client machines. Use _firewall-cmd_ to allow NFS-related network traffic through each Fedora Linux machine’s firewall.

```
$ sudo firewall-cmd --permanent --zone=public --allow-service=nfs
```

As you can imagine, replicating these steps correctly on multiple Fedora Linux machines can be challenging and tedious. Ansible automation solves this problem.

In the _rpi-example_ directory of the github repository I’ve included an example Ansible playbook (_configure.yaml_) that installs and configures both the DNF plugin and the NFS client on all Fedora Linux machines on my network. There is also a playbook (_update.yaml)_ that runs a DNF update across all devices. See this [recent post in Fedora Magazine][8] for more information about Ansible.

To use the provided Ansible examples, first update the inventory file (_inventory_) to include the list of Fedora Linux machines on your network that you want to managed. Next, install two Ansible roles in the roles subdirectory (or another suitable location).

```
$ ansible-galaxy install --roles-path ./roles -r requirements.yaml
```

Run the _configure.yaml_ playbook to install and configure the plugin and NFS client on all hosts defined in the inventory file. The role that installs and configures the NFS client does so via _/etc/fstab_ but also takes it a step further by creating an automount for the NFS share in systemd. The automount is configured to mount the share only when needed and then to automatically unmount. This saves network bandwidth and CPU cycles which can be important for low power devices like a Raspberry Pi. See the github repository for the role and for more information.

```
$ ansible-playbook -i inventory configure.yaml
```

Finally, Ansible can be configured to execute _dnf update_ on all the systems serially by using the _update.yaml_ playbook.

```
$ ansible-playbook -i inventory update.yaml
```

Ansible and other automation tools such as Puppet, Salt, or Chef can be big time savers when working with multiple virtual or physical machines that share many characteristics.

#### Use case 2: virtual machines running on the same host

Fedora Linux has excellent built-in support for virtual machines. The Fedora Project also provides [Fedora Cloud][9] base images for use as virtual machines. [Vagrant][10] is a tool for managing virtual machines. Fedora Magazine has [instructions][11] on how to set up and configure Vagrant. Add the following line in your _.bashrc_ (or other comparable shell configuration file) to inform Vagrant to use _libvirt_ automatically on your workstation instead of the default VirtualBox.

```
export VAGRANT_DEFAULT_PROVIDER=libvirt
```

In your project directory initialize Vagrant and the Fedora Cloud image (use 34-cloud-base for Fedora Linux 34 when available):

```
$ vagrant init fedora/33-cloud-base
```

This creates a Vagrant file in the project directory. Edit the Vagrant file to look like the example below. DNF will likely fail with the default memory settings for _libvirt_. So the example Vagrant file below provides additional memory to the virtual machine. The example below also shares the host _/srv/repodir_ with the virtual machine. The shared directory will have the same path in the virtual machine – _/srv/repodir_. The Vagrant file can be downloaded from [github][12].

```
# -*- mode: ruby -*-
# vi: set ft=ruby :

# define repo directory; same name on host and vm
REPO_DIR = "/srv/repodir"

Vagrant.configure("2") do |config|

  config.vm.box = "fedora/33-cloud-base"

  config.vm.provider :libvirt do |v|
    v.memory = 2048
  #  v.cpus = 2
  end

  # share the local repository with the vm at the same location
  config.vm.synced_folder REPO_DIR, REPO_DIR

  # ansible provisioner - commented out by default
  # the ansible role is installed into a path defined by
  # ansible.galaxy_roles-path below. The extra_vars are ansible
  # variables passed to the playbook.
  #
#  config.vm.provision "ansible" do |ansible|
#    ansible.verbose = "v"
#    ansible.playbook = "ansible/playbook.yaml"
#    ansible.extra_vars = {
#      repo_dir: REPO_DIR,
#      dnf_update: false
#    }
#    ansible.galaxy_role_file = "ansible/requirements.yaml"
#    ansible.galaxy_roles_path = "ansible/roles"
#  end
end
```

Once you have Vagrant managing a Fedora Linux virtual machine, you can install the plugin manually. SSH into the virtual machine:

```
$ vagrant ssh
```

When you are at a command prompt in the virtual machine, repeat the steps from the _Install and configure the DNF local plugin_ section above. The Vagrant configuration file should have already made _/srv/repodir_ from the host available in the virtual machine at the same path.

If you are working with several virtual machines or repeatedly re-initiating a new virtual machine then some simple automation becomes useful. As with the network example above, I use ansible to automate this process.

In the [vagrant-example directory][12] on github, you will see an _ansible_ subdirectory. Edit the Vagrant file and remove the comment marks under the _ansible provisioner_ section. Make sure the _ansible_ directory and its contents (_playbook.yaml_, _requirements.yaml_) are in the project directory.

After you’ve uncommented the lines, the _ansible provisioner_ section in the Vagrant file should look similar to the following:

```
....
  # ansible provisioner
  # the ansible role is installed into a path defined by
  # ansible.galaxy_roles-path below. The extra_vars are ansible
  # variables passed to the playbook.
  #
  config.vm.provision "ansible" do |ansible|
    ansible.verbose = "v"
    ansible.playbook = "ansible/playbook.yaml"
    ansible.extra_vars = {
      repo_dir: REPO_DIR,
      dnf_update: false
    }
    ansible.galaxy_role_file = "ansible/requirements.yaml"
    ansible.galaxy_roles_path = "ansible/roles"
  end
....
```

Ansible must be installed (_sudo dnf install ansible_). Note that there are significant changes to how Ansible is packaged beginning with Fedora Linux 34 (use _sudo dnf install ansible-base ansible-collections*_).

If you run Vagrant now (or reprovision: _vagrant provision_), Ansible will automatically download an Ansible role that installs the DNF local plugin. It will then use the downloaded role in a playbook. You can _vagrant ssh_ into the virtual machine to verify that the plugin is installed and to verify that rpms are coming from the DNF local repository instead of a mirror.

#### Use case 3: container builds

Container images are a common way to distribute and run applications. If you are a developer or enthusiast using Fedora Linux containers as a foundation for applications or services, you will likely use _dnf_ to update the container during the development/build process. Application development is iterative and can result in repeated executions of _dnf_ pulling the same RPM set from Fedora Linux mirrors. If you cache these rpms locally then you can speed up the container build process by retrieving them from the local cache instead of re-downloading them over the network each time. One way to accomplish this is to create a custom Fedora Linux container image with the DNF local plugin installed and configured to use a local repository on the host workstation. Fedora Linux offers _podman_ and _buildah_ for managing the container build, run and test life cycle. See the Fedora Magazine post [_How to build Fedora container images_][13] for more about managing containers on Fedora Linux.

Note that the _fedora_minimal_ container uses _microdnf_ by default which does not support plugins. The _fedora_ container, however, uses _dnf_.

A script that uses _buildah_ and _podman_ to create a custom Fedora Linux image named _myFedora_ is provided below. The script creates a mount point for the local repository at _/srv/repodir_. The below script is also available in the [_container-example_][14] directory of the github repository. It is named _base-image-build.sh_.

```
#!/bin/bash
set -x

# bash script that creates a 'myfedora' image from fedora:latest.
# Adds dnf-local-plugin, points plugin to /srv/repodir for local
# repository and creates an external mount point for /srv/repodir
# that can be used with a -v switch in podman/docker

# custom image name
custom_name=myfedora

# scratch conf file name
tmp_name=local.conf

# location of plugin config file
configuration_name=/etc/dnf/plugins/local.conf

# location of repodir on container
container_repodir=/srv/repodir

# create scratch plugin conf file for container
# using repodir location as set in container_repodir
cat <<EOF > "$tmp_name"
[main]
enabled = true
repodir = $container_repodir
[createrepo]
enabled = true
# If you want to speedup createrepo with the --cachedir option. Eg.
# cachedir = /tmp/createrepo-local-plugin-cachedir
# quiet = true
# verbose = false
EOF

# pull registry.fedoraproject.org/fedora:latest
podman pull registry.fedoraproject.org/fedora:latest

#start the build
mkdev=$(buildah from fedora:latest)

# tag author
buildah config --author "$USER" "$mkdev"

# install dnf-local-plugin, clean
# do not run update as local repo is not operational
buildah run "$mkdev" -- dnf --nodocs -y install python3-dnf-plugin-local createrepo_c
buildah run "$mkdev" -- dnf -y clean all

# create the repo dir
buildah run "$mkdev" -- mkdir -p "$container_repodir"

# copy the scratch plugin conf file from host
buildah copy "$mkdev" "$tmp_name" "$configuration_name"

# mark container repodir as a mount point for host volume
buildah config --volume "$container_repodir" "$mkdev"

# create myfedora image
buildah commit "$mkdev" "localhost/$custom_name:latest"

# clean up working image
buildah rm "$mkdev"

# remove scratch file
rm $tmp_name
```

Given normal security controls for containers, you usually run this script with _sudo_ and when you use the _myFedora_ image in your development process.

```
$ sudo ./base_image_build.sh
```

To list the images stored locally and see both _fedora:latest_ and _myfedora:latest_ run:

```
$ sudo podman images
```

To run the _myFedora_ image as a container and get a bash prompt in the container run:

```
$ sudo podman run -ti -v /srv/repodir:/srv/repodir:Z myfedora /bin/bash
```

Podman also allows you to run containers rootless (as an unprivileged user). Run the script without _sudo_ to create the _myfedora_ image and store it in the unprivileged user’s image repository:

```
$ ./base-image-build.sh
```

In order to run the _myfedora_ image as a rootless container on a Fedora Linux host, an additional flag is needed. Without the extra flag, SELinux will block access to _/srv/repodir_ on the host.

```
$ podman run --security-opt label=disable -ti -v /srv/repodir:/srv/repodir:Z myfedora /bin/bash
```

By using this custom image as the base for your Fedora Linux containers, the iterative building and development of applications or services on them will be faster.

**Bonus Points** – for even better _dnf_ performance, Dan Walsh describes how to share _dnf_ metadata between a host and container using a file overlay (see <https://www>. redhat.com/sysadmin/speeding-container-buildah). This technique will work in combination with a shared local repository only if the host and the container use the same local repository. The _dnf_ metadata cache includes metadata for the local repository under the name __dnf_local_.

I have created a container file that uses _buildah_ to do a _dnf_ update on a _fedora:latest_ image. I’ve also created a container file to repeat the process using a _myfedora_ image. There are 53 MB and 111 rpms in the _dnf_ update. The only difference between the images is that _myfedora_ has the DNF local plugin installed. Using the local repository cut the elapse time by more than half in this example and saves 53MB of internet bandwidth consumption.

With the _fedora:latest_ image the command and elapsed time is:

```
# sudo time -f "Elapsed Time: %E" buildah bud -v /var/cache/dnf:/var/cache/dnf:O - f Containerfile.3 .
128 Elapsed Time: 0:48.06
```

With the _myfedora_ image the command and elapsed time is less than half of the base run. The **:Z** on the **-v** volume below is required when running the container on a SELinux-enabled host.

```
# sudo time -f "Elapsed Time: %E" buildah bud -v /var/cache/dnf:/var/cache/dnf:O -v /srv/repodir:/srv/repodir:Z -f Containerfile.4 .
133 Elapsed Time: 0:19.75
```

### Repository management

The local repository will accumulate files over time. Among the files will be many versions of rpms that change frequently. The kernel rpms are one such example. A system upgrade (for example upgrading from Fedora Linux 33 to Fedora Linux 34) will copy many rpms into the local repository. The _dnf repomanage_ command can be used to remove outdated rpm archives. I have not used the plugin long enough to explore this. The interested and knowledgeable reader is welcome to write an article about the _dnf repomanage_ command for Fedora Magazine.

Finally, I keep the _x86_64_ rpms for my workstation, virtual machines and containers in a local repository that is separate from the _aarch64_ local repository for the Raspberry Pis and (future) containers hosting my Kubernetes cluster. I have separated them for reasons of convenience and happenstance. A single repository location should work across all architectures.

### An important note about Fedora Linux system upgrades

Glenn Johnson has more than four years experience with the DNF local plugin. On occasion he has experienced problems when upgrading to a new release of Fedora Linux with the DNF local plugin enabled. Glenn strongly recommends that the _enabled_ attribute in the plugin configuration file _/etc/dnf/plugins/local.conf_ be set to **false** before upgrading your systems to a new Fedora Linux release. After the system upgrade, re-enable the plugin. Glenn also recommends using a separate local repository for each Fedora Linux release. For example, a NFS server might export _/volume1/dnf-repo/33_ for Fedora Linux 33 systems only. Glenn hangs out on fedoraforum.org – an independent online resource for Fedora Linux users.

### Summary

The DNF local plugin has been beneficial to my ongoing work with a Fedora Linux based Kubernetes cluster. The containers and virtual machines running on my Fedora Linux desktop have also benefited. I appreciate how it supplements the existing DNF process and does not dictate any changes to how I update my systems or how I work with containers and virtual machines. I also appreciate not having to download the same set of rpms multiple times which saves me money, frees up bandwidth, and reduces the load on the Fedora Linux mirror hosts. Give it a try and see if the plugin will help in your situation!

Thanks to Glenn Johnson for his post on the DNF local plugin which started this journey, and for his helpful reviews of this post.

--------------------------------------------------------------------------------

via: https://fedoramagazine.org/use-the-dnf-local-plugin-to-speed-up-your-home-lab/

作者：[Brad Smith][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)

本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出

[a]: https://fedoramagazine.org/author/buckaroogeek/
[b]: https://github.com/lujun9972
[1]: https://fedoramagazine.org/wp-content/uploads/2021/04/dnf-local-816x345.jpg
[2]: https://unsplash.com/@_s9h8_?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
[3]: https://unsplash.com/?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText
[4]: https://dnf-plugins-core.readthedocs.io/en/latest/local.html
[5]: https://forums.fedoraforum.org/showthread.php?318812-dnf-plugin-local
[6]: https://fedoraproject.org/wiki/%20Infrastructure/Mirroring#How_can_someone_make_a_private_mirror
[7]: https://www.redhat.com/sysadmin/nfs-server-client
[8]: https://fedoramagazine.org/using-ansible-setup-workstation/
[9]: https://alt.fedoraproject.org/cloud/
[10]: https://vagrantup.com/
[11]: https://fedoramagazine.org/vagrant-qemukvm-fedora-devops-sysadmin/
[12]: https://github.com/buckaroogeek/dnf-local-plugin-examples/tree/main/vagrant-example
[13]: https://fedoramagazine.org/how-to-build-fedora-container-images/
[14]: https://github.com/buckaroogeek/dnf-local-plugin-examples/tree/main/container-example