mirror of
https://github.com/LCTT/TranslateProject.git
synced 2025-03-21 02:10:11 +08:00
Merge remote-tracking branch 'LCTT/master'
This commit is contained in:
Xingyu Wang
commit
88000682a6
266
published/20180612 Systemd Services- Reacting to Change.md
Normal file
266
published/20180612 Systemd Services- Reacting to Change.md
Normal file
@ -0,0 +1,266 @@
|
||||
Systemd 服务：响应变化
|
||||
======
|
||||
|
||||
|
||||
|
||||
[我有一个这样的电脑棒][1]（图1），我把它用作通用服务器。它很小且安静，由于它是基于 x86 架构的，因此我为我的打印机安装驱动没有任何问题，而且这就是它大多数时候干的事：与客厅的共享打印机和扫描仪通信。
|
||||
|
||||
![ComputeStick][3]
|
||||
|
||||
*一个英特尔电脑棒。欧元硬币大小。*
|
||||
|
||||
大多数时候它都是闲置的，尤其是当我们外出时，因此我认为用它作监视系统是个好主意。该设备没有自带的摄像头，也不需要一直监视。我也不想手动启动图像捕获，因为这样就意味着在出门前必须通过 SSH 登录，并在 shell 中编写命令来启动该进程。
|
||||
|
||||
因此，我以为应该这么做：拿一个 USB 摄像头，然后只需插入它即可自动启动监视系统。如果这个电脑棒重启后发现连接了摄像头也启动监视系统就更加分了。
|
||||
|
||||
在先前的文章中，我们看到 systemd 服务既可以[手动启动或停止][5]，也可以[在满足某些条件时启动或停止][6]。这些条件不限于操作系统在启动或关机时序中达到某种状态，还可以在你插入新硬件或文件系统发生变化时进行。你可以通过将 Udev 规则与 systemd 服务结合起来实现。
|
||||
|
||||
### 有 Udev 支持的热插拔
|
||||
|
||||
Udev 规则位于 `/etc/udev/rules` 目录中，通常是由导致一个<ruby>动作<rt>action</rt></ruby>的<ruby>条件<rt>conditions</rt></ruby>和<ruby>赋值<rt>assignments</rt></ruby>的单行语句来描述。
|
||||
|
||||
有点神秘。让我们再解释一次：
|
||||
|
||||
通常，在 Udev 规则中，你会告诉 systemd 当设备连接时需要查看什么信息。例如，你可能想检查刚插入的设备的品牌和型号是否与你让 Udev 等待的设备的品牌和型号相对应。这些就是前面提到的“条件”。
|
||||
|
||||
然后，你可能想要更改一些内容，以便以后可以方便使用该设备。例如，更改设备的读写权限：如果插入 USB 打印机，你会希望用户能够从打印机读取信息（用户的打印应用程序需要知道其模型、制造商，以及是否准备好接受打印作业）并向其写入内容，即发送要打印的内容。更改设备的读写权限是通过你之前阅读的“赋值” 之一完成的。
|
||||
|
||||
最后，你可能希望系统在满足上述条件时执行某些动作，例如在插入某个外部硬盘时启动备份程序以复制重要文件。这就是上面提到的“动作”的例子。
|
||||
|
||||
了解这些之后, 来看看以下几点:
|
||||
|
||||
```
|
||||
ACTION=="add", SUBSYSTEM=="video4linux", ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="e207",
|
||||
SYMLINK+="mywebcam", TAG+="systemd", MODE="0666", ENV{SYSTEMD_WANTS}="webcam.service"
|
||||
```
|
||||
|
||||
规则的第一部分,
|
||||
|
||||
```
|
||||
ACTION=="add", SUBSYSTEM=="video4linux", ATTRS{idVendor}=="03f0",
|
||||
ATTRS{idProduct}=="e207" [etc... ]
|
||||
```
|
||||
|
||||
表明了执行你想让系统执行的其他动作之前设备必须满足的条件。设备必须被添加到（`ACTION=="add"`）机器上，并且必须添加到 `video4linux` 子系统中。为了确保仅在插入正确的设备时才应用该规则，你必须确保 Udev 正确识别设备的制造商（`ATTRS{idVendor}=="03f0"`）和型号（`ATTRS{idProduct}=="e207"`）。
|
||||
|
||||
在本例中，我们讨论的是这个设备（图2）：
|
||||
|
||||
![webcam][8]
|
||||
|
||||
*这个试验使用的是 HP 的摄像头。*
|
||||
|
||||
注意怎样用 `==` 来表示这是一个逻辑操作。你应该像这样阅读上面的简要规则：
|
||||
|
||||
> 如果添加了一个设备并且该设备由 video4linux 子系统控制，而且该设备的制造商编码是 03f0，型号是 e207，那么...
|
||||
|
||||
但是，你从哪里获取的这些信息？你在哪里找到触发事件的动作、制造商、型号等？你可要使用多个来源。你可以通过将摄像头插入机器并运行 `lsusb` 来获得 `IdVendor` 和 `idProduct` ：
|
||||
|
||||
```
|
||||
lsusb
|
||||
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
|
||||
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
|
||||
Bus 003 Device 003: ID 03f0:e207 Hewlett-Packard
|
||||
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
Bus 001 Device 003: ID 04f2:b1bb Chicony Electronics Co., Ltd
|
||||
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
|
||||
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
```
|
||||
|
||||
我用的摄像头是 HP 的，你在上面的列表中只能看到一个 HP 设备。`ID` 提供了制造商和型号，它们以冒号（`:`）分隔。如果你有同一制造商的多个设备，不确定哪个是哪个设备，请拔下摄像头，再次运行 `lsusb` , 看看少了什么。
|
||||
|
||||
或者...
|
||||
|
||||
拔下摄像头，等待几秒钟，运行命令 `udevadmin monitor --environment` ，然后重新插入摄像头。当你使用的是HP摄像头时，你将看到：
|
||||
|
||||
```
|
||||
udevadmin monitor --environment
|
||||
UDEV [35776.495221] add /devices/pci0000:00/0000:00:1c.3/0000:04:00.0
|
||||
/usb3/3-1/3-1:1.0/input/input21/event11 (input)
|
||||
.MM_USBIFNUM=00
|
||||
ACTION=add
|
||||
BACKSPACE=guess
|
||||
DEVLINKS=/dev/input/by-path/pci-0000:04:00.0-usb-0:1:1.0-event
|
||||
/dev/input/by-id/usb-Hewlett_Packard_HP_Webcam_HD_2300-event-if00
|
||||
DEVNAME=/dev/input/event11
|
||||
DEVPATH=/devices/pci0000:00/0000:00:1c.3/0000:04:00.0/
|
||||
usb3/3-1/3-1:1.0/input/input21/event11
|
||||
ID_BUS=usb
|
||||
ID_INPUT=1
|
||||
ID_INPUT_KEY=1
|
||||
ID_MODEL=HP_Webcam_HD_2300
|
||||
ID_MODEL_ENC=HPx20Webcamx20HDx202300
|
||||
ID_MODEL_ID=e207
|
||||
ID_PATH=pci-0000:04:00.0-usb-0:1:1.0
|
||||
ID_PATH_TAG=pci-0000_04_00_0-usb-0_1_1_0
|
||||
ID_REVISION=1020
|
||||
ID_SERIAL=Hewlett_Packard_HP_Webcam_HD_2300
|
||||
ID_TYPE=video
|
||||
ID_USB_DRIVER=uvcvideo
|
||||
ID_USB_INTERFACES=:0e0100:0e0200:010100:010200:030000:
|
||||
ID_USB_INTERFACE_NUM=00
|
||||
ID_VENDOR=Hewlett_Packard
|
||||
ID_VENDOR_ENC=Hewlettx20Packard
|
||||
ID_VENDOR_ID=03f0
|
||||
LIBINPUT_DEVICE_GROUP=3/3f0/e207:usb-0000:04:00.0-1/button
|
||||
MAJOR=13
|
||||
MINOR=75
|
||||
SEQNUM=3162
|
||||
SUBSYSTEM=input
|
||||
USEC_INITIALIZED=35776495065
|
||||
XKBLAYOUT=es
|
||||
XKBMODEL=pc105
|
||||
XKBOPTIONS=
|
||||
XKBVARIANT=
|
||||
```
|
||||
|
||||
可能看起来有很多信息要处理，但是，看一下这个：列表前面的 `ACTION` 字段, 它告诉你刚刚发生了什么事件，即一个设备被添加到系统中。你还可以在其中几行中看到设备名称的拼写，因此可以非常确定它就是你要找的设备。输出里还显示了制造商的ID（`ID_VENDOR_ID = 03f0`）和型号（`ID_VENDOR_ID = 03f0`）。
|
||||
|
||||
这为你提供了规则条件部分需要的四个值中的三个。你可能也会想到它还给了你第四个，因为还有一行这样写道：
|
||||
|
||||
```
|
||||
SUBSYSTEM=input
|
||||
```
|
||||
|
||||
小心！尽管 USB 摄像头确实是提供输入的设备（键盘和鼠标也是），但它也属于 usb 子系统和其他几个子系统。这意味着你的摄像头被添加到了多个子系统，并且看起来像多个设备。如果你选择了错误的子系统，那么你的规则可能无法按你期望的那样工作，或者根本无法工作。
|
||||
|
||||
因此，第三件事就是检查网络摄像头被添加到的所有子系统，并选择正确的那个。为此，请再次拔下摄像头，然后运行：
|
||||
|
||||
```
|
||||
ls /dev/video*
|
||||
```
|
||||
|
||||
这将向你显示连接到本机的所有视频设备。如果你使用的是笔记本，大多数笔记本都带有内置摄像头，它可能会显示为 `/dev/video0` 。重新插入摄像头，然后再次运行 `ls /dev/video*`。
|
||||
|
||||
现在，你应该看到多一个视频设备（可能是`/dev/video1`）。
|
||||
|
||||
现在，你可以通过运行 `udevadm info -a /dev/video1` 找出它所属的所有子系统：
|
||||
|
||||
```
|
||||
udevadm info -a /dev/video1
|
||||
|
||||
Udevadm info starts with the device specified by the devpath and then
|
||||
walks up the chain of parent devices. It prints for every device
|
||||
found, all possible attributes in the udev rules key format.
|
||||
A rule to match, can be composed by the attributes of the device
|
||||
and the attributes from one single parent device.
|
||||
|
||||
looking at device '/devices/pci0000:00/0000:00:1c.3/0000:04:00.0
|
||||
/usb3/3-1/3-1:1.0/video4linux/video1':
|
||||
KERNEL=="video1"
|
||||
SUBSYSTEM=="video4linux"
|
||||
DRIVER==""
|
||||
ATTR{dev_debug}=="0"
|
||||
ATTR{index}=="0"
|
||||
ATTR{name}=="HP Webcam HD 2300: HP Webcam HD"
|
||||
|
||||
[etc...]
|
||||
```
|
||||
|
||||
输出持续了相当长的时间，但是你感兴趣的只是开头的部分：`SUBSYSTEM =="video4linux"`。你可以将这行文本直接复制粘贴到你的规则中。输出的其余部分（为简洁未显示）为你提供了更多的信息，例如制造商和型号 ID，同样是以你可以复制粘贴到你的规则中的格式。
|
||||
|
||||
现在，你有了识别设备的方式吗，并明确了什么事件应该触发该动作，该对设备进行修改了。
|
||||
|
||||
规则的下一部分，`SYMLINK+="mywebcam", TAG+="systemd", MODE="0666"` 告诉 Udev 做三件事：首先，你要创建设备的符号链接（例如 `/dev/video1` 到 `/dev/mywebcam`。这是因为你无法预测系统默认情况下会把那个设备叫什么。当你拥有内置摄像头并热插拔一个新的时，内置摄像头通常为 `/dev/video0`，而外部摄像头通常为 `/dev/video1`。但是，如果你在插入外部 USB 摄像头的情况下重启计算机，则可能会相反，内部摄像头可能会变成 `/dev/video1` ，而外部摄像头会变成 `/dev/video0`。这想告诉你的是，尽管你的图像捕获脚本（稍后将看到）总是需要指向外部摄像头设备，但是你不能依赖它是 `/dev/video0` 或 `/dev/video1`。为了解决这个问题，你告诉 Udev 创建一个符号链接，该链接在设备被添加到 `video4linux` 子系统的那一刻起就不会再变，你将使你的脚本指向该链接。
|
||||
|
||||
第二件事就是将 `systemd` 添加到与此规则关联的 Udev 标记列表中。这告诉 Udev，该规则触发的动作将由 systemd 管理，即它将是某种 systemd 服务。
|
||||
|
||||
注意在这个两种情况下是如何使用 `+=` 运算符的。这会将值添加到列表中，这意味着你可以向 `SYMLINK` 和 `TAG` 添加多个值。
|
||||
|
||||
另一方面，`MODE` 值只能包含一个值（因此，你可以使用简单的 `=` 赋值运算符）。`MODE` 的作用是告诉 Udev 谁可以读或写该设备。如果你熟悉 `chmod`（你读到此文, 应该会熟悉），你就也会熟悉[如何用数字表示权限][9]。这就是它的含义：`0666` 的含义是 “向所有人授予对设备的读写权限”。
|
||||
|
||||
最后， `ENV{SYSTEMD_WANTS}="webcam.service"` 告诉 Udev 要运行什么 systemd 服务。
|
||||
|
||||
将此规则保存到 `/etc/udev/rules.d` 目录名为 `90-webcam.rules`（或类似的名称）的文件中，你可以通过重启机器或运行以下命令来加载它：
|
||||
|
||||
```
|
||||
sudo udevadm control --reload-rules && udevadm trigger
|
||||
```
|
||||
|
||||
### 最后是服务
|
||||
|
||||
Udev 规则触发的服务非常简单：
|
||||
|
||||
```
|
||||
# webcam.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/home/[user name]/bin/checkimage.sh
|
||||
```
|
||||
|
||||
基本上，它只是运行存储在你个人 `bin/` 中的 `checkimage.sh` 脚本并将其放到后台。[这是你在先前的文章中看过的内容][5]。它看起来似乎很小，但那只是因为它是被 Udev 规则调用的，你刚刚创建了一种特殊的 systemd 单元，称为 `device` 单元。 恭喜。
|
||||
|
||||
至于 `webcam.service` 调用的 `checkimage.sh` 脚本，有几种方法从摄像头抓取图像并将其与前一个图像进行比较以检查变化（这是 `checkimage.sh` 所做的事），但这是我的方法：
|
||||
|
||||
```
|
||||
#!/bin/bash
|
||||
# This is the checkimage.sh script
|
||||
|
||||
mplayer -vo png -frames 1 tv:// -tv driver=v4l2:width=640:height=480:device=
|
||||
/dev/mywebcam &>/dev/null
|
||||
mv 00000001.png /home/[user name]/monitor/monitor.png
|
||||
|
||||
while true
|
||||
do
|
||||
mplayer -vo png -frames 1 tv:// -tv driver=v4l2:width=640:height=480:device=/dev/mywebcam &>/dev/null
|
||||
mv 00000001.png /home/[user name]/monitor/temp.png
|
||||
|
||||
imagediff=`compare -metric mae /home/[user name]/monitor/monitor.png /home/[user name]
|
||||
/monitor/temp.png /home/[user name]/monitor/diff.png 2>&1 > /dev/null | cut -f 1 -d " "`
|
||||
if [ `echo "$imagediff > 700.0" | bc` -eq 1 ]
|
||||
then
|
||||
mv /home/[user name]/monitor/temp.png /home/[user name]/monitor/monitor.png
|
||||
fi
|
||||
|
||||
sleep 0.5
|
||||
done
|
||||
```
|
||||
|
||||
首先使用[MPlayer][10]从摄像头抓取一帧（`00000001.png`）。注意，我们怎样将 `mplayer` 指向 Udev 规则中创建的 `mywebcam` 符号链接，而不是指向 `video0` 或 `video1`。然后，将图像传输到主目录中的 `monitor/` 目录。然后执行一个无限循环，一次又一次地执行相同的操作，但还使用了[Image Magick 的 compare 工具][11]来查看最后捕获的图像与 `monitor/` 目录中已有的图像之间是否存在差异。
|
||||
|
||||
如果图像不同，则表示摄像头的镜框里某些东西动了。该脚本将新图像覆盖原始图像，并继续比较以等待更多变动。
|
||||
|
||||
### 插线
|
||||
|
||||
所有东西准备好后，当你插入摄像头后，你的 Udev 规则将被触发并启动 `webcam.service`。 `webcam.service` 将在后台执行 `checkimage.sh` ，而 `checkimage.sh` 将开始每半秒拍一次照。你会感觉到，因为摄像头的 LED 在每次拍照时将开始闪。
|
||||
|
||||
与往常一样，如果出现问题，请运行：
|
||||
|
||||
```
|
||||
systemctl status webcam.service
|
||||
```
|
||||
|
||||
检查你的服务和脚本正在做什么。
|
||||
|
||||
### 接下来
|
||||
|
||||
你可能想知道：为什么要覆盖原始图像？当然，系统检测到任何动静，你都想知道发生了什么，对吗？你是对的，但是如你在下一部分中将看到的那样，将它们保持原样，并使用另一种类型的 systemd 单元处理图像将更好，更清晰和更简单。
|
||||
|
||||
请期待下一篇。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.linux.com/blog/intro-to-linux/2018/6/systemd-services-reacting-change
|
||||
|
||||
作者：[Paul Brown][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[messon007](https://github.com/messon007)
|
||||
校对：[wxy](https://github.com/wxy)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://www.linux.com/users/bro66
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://www.intel.com/content/www/us/en/products/boards-kits/compute-stick/stk1a32sc.html
|
||||
[2]: https://www.linux.com/files/images/fig01png
|
||||
[3]: https://lcom.static.linuxfound.org/sites/lcom/files/fig01.png
|
||||
[4]: https://www.linux.com/licenses/category/used-permission
|
||||
[5]: https://linux.cn/article-9700-1.html
|
||||
[6]: https://linux.cn/article-9703-1.html
|
||||
[7]: https://www.linux.com/files/images/fig02png
|
||||
[8]: https://www.linux.com/sites/lcom/files/styles/floated_images/public/fig02.png?itok=esFv4BdM (webcam)
|
||||
[9]: https://chmod-calculator.com/
|
||||
[10]: https://mplayerhq.hu/design7/news.html
|
||||
[11]: https://www.imagemagick.org/script/compare.php
|
||||
[12]: https://training.linuxfoundation.org/linux-courses/system-administration-training/introduction-to-linux
|
||||
@ -1,5 +1,5 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: (MjSeven)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: translator: ( guevaraya)
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
|
||||
@ -1,112 +0,0 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: (geekpi)
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Fixing “Unable to parse package file /var/lib/apt/lists” Error in Ubuntu and Other Linux Distributions)
|
||||
[#]: via: (https://itsfoss.com/unable-to-parse-package-file/)
|
||||
[#]: author: (Abhishek Prakash https://itsfoss.com/author/abhishek/)
|
||||
|
||||
Fixing “Unable to parse package file /var/lib/apt/lists” Error in Ubuntu and Other Linux Distributions
|
||||
======
|
||||
|
||||
I have discussed a number of [Ubuntu update errors][1] in the past. If you [use the command line to update Ubuntu][2], you might run into some ‘errors’.
|
||||
|
||||
Some of these ‘errors’ are basically built-in features to prevent unwarranted changes to your system. I am not going into those details in this quick tutorial.
|
||||
|
||||
In this quick tip, I’ll show you how to tackle the following error that you could encounter while updating your system or installing new software:
|
||||
|
||||
**Reading package lists… Error!
|
||||
E: Unable to parse package file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease
|
||||
E: The package lists or status file could not be parsed or opened.**
|
||||
|
||||
A similar error can be encountered in Debian:
|
||||
|
||||
**E: Unable to parse package file /var/lib/apt/extended_states (1)**
|
||||
|
||||
There is absolutely no need to panic even thought it says ‘**The package cache file is corrupted**‘. This is really easy to ‘fix’.
|
||||
|
||||
### Handling “Unable to parse package file” error in Ubuntu and Debian-based Linux distributions
|
||||
|
||||
![][3]
|
||||
|
||||
Here’s what you need to do. Take a closer look at the name and path of the file the [Ubuntu][4] is complaining about.
|
||||
|
||||
Reading package lists… Error!
|
||||
**E: Unable to parse package file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease**
|
||||
E: The package lists or status file could not be parsed or opened.
|
||||
|
||||
For example, in the above error, it was complaining about /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease
|
||||
|
||||
This gives you the idea that something is not right with this file. Now all you need to do is to remove this file and regenerate the cache.
|
||||
|
||||
```
|
||||
sudo rm <file_that_is_not_parsed>
|
||||
```
|
||||
|
||||
So in my case, I could use this command: **sudo rm /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease** and then rebuild the cache with sudo apt update command.
|
||||
|
||||
#### Step by step for beginners
|
||||
|
||||
If you are familiar with Linux commands, you may know how to do delete the file with its absolute path. For novice users, let me guide you to safely delete the file.
|
||||
|
||||
First, you should go to the directory where the file is stored:
|
||||
|
||||
```
|
||||
cd /var/lib/apt/lists/
|
||||
```
|
||||
|
||||
Now delete the file which is not being parsed:
|
||||
|
||||
```
|
||||
sudo rm archive.ubuntu.com_ubuntu_dists_bionic_InRelease
|
||||
```
|
||||
|
||||
Now if you run the update again, the apt cache will be regenerated.
|
||||
|
||||
```
|
||||
sudo apt update
|
||||
```
|
||||
|
||||
#### Too many files cannot be parsed?
|
||||
|
||||
This is fine if you have one or two files that are not being parsed while updating the system. But if the system complains about ten or twenty such files, removing them one by one is too tiring.
|
||||
|
||||
What you can do in such a case to remove the entire cache and then generate it again:
|
||||
|
||||
```
|
||||
sudo rm -r /var/lib/apt/lists/*
|
||||
sudo apt update
|
||||
```
|
||||
|
||||
#### Explanation of how it fixed your problem
|
||||
|
||||
The /var/lib/apt is the directory where files and data related to the apt package manager are stored. The /var/lib/apt/lists is the directory which is used for storing information for each package resource specified in your system’s sources.list.
|
||||
|
||||
In slightly non complicated terms, this /var/lib/apt/lists stores the package information cache. When you want to install or update a program, your system checks in this directory for the information on the said package. If it finds the detail on the package, then it goes to remote repository and actually download the program or its update.
|
||||
|
||||
When you run the “sudo apt update”, it builds the cache. This is why even when you remove everything in the /var/lib/apt/lists directory, running the update will build a fresh cache.
|
||||
|
||||
This is how it handles the issue of file not being parsed. Your system complained about a particular package or repository information that somehow got corrupted (either a failed download or manual change to sources.list). Removing that file (or everything) and rebuilding the cache solves the issue.
|
||||
|
||||
#### Still facing error?
|
||||
|
||||
This should fix the issue for you. But if the problem still persists or if you have some other related issue, let me know in the comment section and I’ll try to help you out.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://itsfoss.com/unable-to-parse-package-file/
|
||||
|
||||
作者：[Abhishek Prakash][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://itsfoss.com/author/abhishek/
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://itsfoss.com/ubuntu-update-error/
|
||||
[2]: https://itsfoss.com/update-ubuntu/
|
||||
[3]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2020/05/Unable-to-parse-package-file.png?ssl=1
|
||||
[4]: https://ubuntu.com/
|
||||
@ -1,164 +0,0 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Ensmallening Go binaries by prohibiting comparisons)
|
||||
[#]: via: (https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons)
|
||||
[#]: author: (Dave Cheney https://dave.cheney.net/author/davecheney)
|
||||
|
||||
Ensmallening Go binaries by prohibiting comparisons
|
||||
======
|
||||
|
||||
Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is to detect functions which are not referenced by a program–say they are part of a library of which only a subset of functionality is used–and remove them from the final output. Yet, the adage mo’ types, mo’ binary holds true for the majority of Go programs.
|
||||
|
||||
In this post I’ll dig into what equality, in the context of a Go program, means and why changes [like this][1] have a measurable impact on the size of a Go program.
|
||||
|
||||
### Defining equality between two values
|
||||
|
||||
The Go spec defines the concepts of assignability and equality. Assignabiity is the act of assigning a value to an identifier. Not everything which is declared can be assigned, for example constants and functions. Equality is the act of comparing two identifies by asking _are their contents the same?_
|
||||
|
||||
Being a strongly typed language, the notion of sameness is fundamentally rooted in the identifier’s type. Two things can only be the same if they are of the same type. Beyond that, the type of the values defines how they are compared.
|
||||
|
||||
For example, integers are compared arithmetically. For pointer types, equality is determining if the addresses they point too are the same. Reference types like maps and channels, like pointers, are considered to be the same if they have the same address.
|
||||
|
||||
These are all examples of bitwise equality, that is, if the bit patterns of the memory that value occupies are the same, those values are equal. This is known as memcmp, short for memory comparison, as equality is defined by comparing the contents of two areas of memory.
|
||||
|
||||
Hold on to this idea, I’ll come back to in a second.
|
||||
|
||||
### Struct equality
|
||||
|
||||
Beyond scalar types like integers, floats, and pointers is the realm of compound types; structs. All structs are laid out in memory in program order, thus this declaration:
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a, b, c, d int64
|
||||
}
|
||||
```
|
||||
|
||||
will consume 32 bytes of memory; 8 bytes for `a`, then 8 bytes for `b`, and so on. The spec says that _struct values are comparable if all their fields are comparable_. Thus two structs are equal iff each of their fields are equal.
|
||||
|
||||
```
|
||||
a := S{1, 2, 3, 4}
|
||||
b := S{1, 2, 3, 4}
|
||||
fmt.Println(a == b) // prints true
|
||||
```
|
||||
|
||||
Under the hood the compiler uses memcmp to compare the 32 bytes of `a` and `b`.
|
||||
|
||||
### Padding and alignment
|
||||
|
||||
However the simplistic bitwise comparison strategy will fail in situations like this:
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a byte
|
||||
b uint64
|
||||
c int16
|
||||
d uint32
|
||||
}
|
||||
|
||||
func main()
|
||||
a := S{1, 2, 3, 4}
|
||||
b := S{1, 2, 3, 4}
|
||||
fmt.Println(a == b) // prints true
|
||||
}
|
||||
```
|
||||
|
||||
The code compiles, the comparison is still true, but under the hood the compiler cannot rely on comparing the bit patterns of `a` and `b` because the structure contains _padding_.
|
||||
|
||||
Go requires each field in a struct to be naturally aligned. 2 byte values must start on an even address, four byte values on an address divisible by 4, and so on[1][2]. The compiler inserts padding to ensure the fields are _aligned_ to according to their type and the underlying platform. In effect, after padding, this is what the compiler sees[2][3]:
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a byte
|
||||
_ [7]byte // padding
|
||||
b uint64
|
||||
c int16
|
||||
_ [2]int16 // padding
|
||||
d uint32
|
||||
}
|
||||
```
|
||||
|
||||
Padding exists to ensure the correct field alignments, and while it does take up space in memory, the contents of those padding bytes are unknown. You might assume that, being Go, the padding bytes are always zero, but it turns out that’s not the case–the contents of padding bytes are simply not defined. Because they’re not defined to always be a certain value, doing a bitwise comparison may return false because the nine bytes of padding spread throughout the 24 bytes of `S` are may not be the same.
|
||||
|
||||
The Go compiler solves this problem by generating what is known as an equality function. In this case `S`‘s equality function knows how to compare two values of type `S` by comparing only the fields in the function while skipping over the padding.
|
||||
|
||||
### Type algorithms
|
||||
|
||||
Phew, that was a lot of setup to illustrate why, for each type defined in a Go program, the compiler may generate several supporting functions, known inside the compiler as the type’s algorithms. In addition to the equality function the compiler will generate a hash function if the type is used as a map key. Like the equality function, the hash function must consider factors like padding when computing its result to ensure it remains stable.
|
||||
|
||||
It turns out that it can be hard, and sometimes non obvious, to intuit when the compiler will generate these functions–it’s more than you’d expect–and it can be hard for the linker to eliminate the ones that are not needed as reflection often causes the linker to be more conservative when trimming types.
|
||||
|
||||
### Reducing binary size by prohibiting comparisons
|
||||
|
||||
Now we’re at a point to explain Brad’s change. By adding an incomparable field [3][4] to the type, the resulting struct is by extension incomparable, thus forcing the compiler to elide the generation of eq and hash algorithms, short circuiting the linkers elimination of those types and, in practice, reducing the size of the final binary. As an example of this technique, this program:
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
func main() {
|
||||
type t struct {
|
||||
// _ [0][]byte uncomment to prevent comparison
|
||||
a byte
|
||||
b uint16
|
||||
c int32
|
||||
d uint64
|
||||
}
|
||||
var a t
|
||||
fmt.Println(a)
|
||||
}
|
||||
```
|
||||
|
||||
when compiled with Go 1.14.2 (darwin/amd64), decreased from 2174088 to 2174056, a saving of 32 bytes. In isolation this 32 byte saving may seem like small beer, but consider that equality and hash functions can be generated for every type in the transitive closure of your program and all its dependencies, and the size of these functions varies depending on the size of the type and its complexity, prohibiting them can have a sizeable impact on the final binary over and above the old saw of `-ldflags="-s -w"`.
|
||||
|
||||
The bottom line, if you don’t wish to make your types comparable, a hack like this enforces it at the source level while contributing to a small reduction in the size of your binary.
|
||||
|
||||
* * *
|
||||
|
||||
Addendum: thanks to Brad’s prodding, Go 1.15 already has a bunch of improvements by [Cherry Zhang][5] and [Keith Randall][6] that fix the most egregious of the failures to eliminate unnecessary equality and hash functions (although I suspect it was also to avoid the proliferation of this class of CLs).
|
||||
|
||||
1. On 32bit platforms `int64` and `uint64` values may not be 8 byte aligned as the natural alignment of the platform is 4 bytes. See [issue 599][7] for the gory details.[][8]
|
||||
2. 32 bit platforms would see `_ [3]byte` padding between the declaration of `a` and `b`. See previous.[][9]
|
||||
3. Brad used `[0]func()`, but any type that the spec limits or prohibits comparisons on will do. By declaring the array has zero elements the type has no impact on the size or alignment of the struct.[][10]
|
||||
|
||||
|
||||
|
||||
#### Related posts:
|
||||
|
||||
1. [How the Go runtime implements maps efficiently (without generics)][11]
|
||||
2. [The empty struct][12]
|
||||
3. [Padding is hard][13]
|
||||
4. [Typed nils in Go 2][14]
|
||||
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons
|
||||
|
||||
作者：[Dave Cheney][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://dave.cheney.net/author/davecheney
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://github.com/golang/net/commit/e0ff5e5a1de5b859e2d48a2830d7933b3ab5b75f
|
||||
[2]: tmp.uBLyaVR1Hm#easy-footnote-bottom-1-4116 (On 32bit platforms <code>int64</code> and <code>uint64</code> values may not be 8 byte aligned as the natural alignment of the platform is 4 bytes. See <a href="https://github.com/golang/go/issues/599">issue 599</a> for the gory details.)
|
||||
[3]: tmp.uBLyaVR1Hm#easy-footnote-bottom-2-4116 (32 bit platforms would see <code>_ [3]byte</code> padding between the declaration of <code>a</code> and <code>b</code>. See previous.)
|
||||
[4]: tmp.uBLyaVR1Hm#easy-footnote-bottom-3-4116 (Brad used <code>[0]func()</code>, but any type that the spec limits or prohibits comparisons on will do. By declaring the array has zero elements the type has no impact on the size or alignment of the struct.)
|
||||
[5]: https://go-review.googlesource.com/c/go/+/231397
|
||||
[6]: https://go-review.googlesource.com/c/go/+/191198
|
||||
[7]: https://github.com/golang/go/issues/599
|
||||
[8]: tmp.uBLyaVR1Hm#easy-footnote-1-4116
|
||||
[9]: tmp.uBLyaVR1Hm#easy-footnote-2-4116
|
||||
[10]: tmp.uBLyaVR1Hm#easy-footnote-3-4116
|
||||
[11]: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implements-maps-efficiently-without-generics (How the Go runtime implements maps efficiently (without generics))
|
||||
[12]: https://dave.cheney.net/2014/03/25/the-empty-struct (The empty struct)
|
||||
[13]: https://dave.cheney.net/2015/10/09/padding-is-hard (Padding is hard)
|
||||
[14]: https://dave.cheney.net/2017/08/09/typed-nils-in-go-2 (Typed nils in Go 2)
|
||||
@ -1,5 +1,5 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: translator: (geekpi)
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
|
||||
@ -0,0 +1,144 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (How I track my home's energy consumption with open source)
|
||||
[#]: via: (https://opensource.com/article/20/5/energy-monitoring)
|
||||
[#]: author: (Stephan Avenwedde https://opensource.com/users/hansic99)
|
||||
|
||||
How I track my home's energy consumption with open source
|
||||
======
|
||||
These open source components help you find ways to save money and
|
||||
conserve resources.
|
||||
![lightbulb drawing outline][1]
|
||||
|
||||
An important step towards optimizing energy consumption is knowing your actual consumption. My house was built during the oil crisis in the 1970s, and due to the lack of a natural gas connection, the builders decided to use electricity to do all of the heating (water and home heating). This is not unusual for this area of Germany, and it remains an appropriate solution in countries that depend highly on nuclear power.
|
||||
|
||||
Electricity prices here are quite high (around € 0.28/kWh), so I decided to monitor my home's energy consumption to get a feel for areas where I could save some energy.
|
||||
|
||||
I used to work for a company that sold energy-monitoring systems for industrial customers. While this company mostly used proprietary software, you can set up a similar smart monitoring and logging solution for your home based on open source components. This article will show you how.
|
||||
|
||||
In Germany, the grid operator owns the electricity meter. The grid operator is obliged to provide an interface on its metering device to enable the customer to access the meter reading. Here is the metering device on my home:
|
||||
|
||||
![Actaris ACE3000 electricity meter][2]
|
||||
|
||||
Actaris ACE3000 Type 110 (dry contact located behind the marked cover)
|
||||
|
||||
Generally, almost every metering device has at least a [dry contact][3]—as my electricity meter does—that you can use to log metering. As you can see, my electricity meter has two counters: The upper one is for the day tariff (6am to 10pm), and the lower one is for the night tariff (10pm to 6am). The night tariff is a bit cheaper. Two-tariff meters are usually found only in houses with electric heating.
|
||||
|
||||
### Design
|
||||
|
||||
A reliable energy-monitoring solution for private use should meet the following requirements:
|
||||
|
||||
* Logging of metering impulses (dry contact)
|
||||
* 24/7 operation
|
||||
* Energy-saving operation
|
||||
* Visualization of consumption data
|
||||
* Long-term recording of consumption data
|
||||
* Connectivity (e.g., Ethernet, USB, WiFi, etc.)
|
||||
* Affordability
|
||||
|
||||
|
||||
|
||||
I choose the Siemens SIMATIC IOT2020 as my hardware platform. This industrial-proven device is based on an Intel Quark x86 CPU, has programmable interrupts, and is compatible with many Arduino shields.
|
||||
|
||||
![Siemens SIMATIC IOT2020][4]
|
||||
|
||||
Siemens SIMATIC IOT2020
|
||||
|
||||
The Siemens device comes without an SD card and, therefore, without an operating system. Luckily, you can find a current Yocto-based Linux OS image and instructions on how to flash the SD card in the [Siemens forum][5].
|
||||
|
||||
In addition to the hardware platform, you also need some accessories. The following materials list shows the minimum components you need. Each item includes links to the parts I purchased, so you can get a sense of the project's costs.
|
||||
|
||||
#### Materials list
|
||||
|
||||
* [Siemens SIMATIC IoT2020 unit][6]
|
||||
* [Siemens I/O Shield for SIMATIC IoT2000 series][7]
|
||||
* [microSD card][8] (2GB or more)
|
||||
* [CSL 300Mbit USB-WLAN adapter][9]
|
||||
* 24V power supply (I used a 2.1A [TDK-Lambda DRB50-24-1][10], which I already owned). You could use a less expensive power supply with less power: the SIMATIC IOT2020 has a maximum current of 1.4A, and the dry contact needs an additional 0.1A (24V / 220Ω).
|
||||
* 5 terminal blocks ([Weidmueller WDU 2.5mm][11])
|
||||
* 2 terminal cross-connecting bridges ([Weidmueller WQV][12])
|
||||
* [DIN rail][13] (~300 mm)
|
||||
* [220Ω / 3W resistor][14]
|
||||
* Wire
|
||||
|
||||
|
||||
|
||||
Here is the assembled result:
|
||||
|
||||
![Mounted and hooked up energy logger][15]
|
||||
|
||||
Energy logger mounted and hooked up
|
||||
|
||||
Unfortunately, I didn't have enough space at the rear wall of the cabinet; therefore, the DIN rail with the mounted parts lies on the ground.
|
||||
|
||||
The connections between the meter and the Siemens device look like this:
|
||||
|
||||
![Wiring between meter and energy logger][16]
|
||||
|
||||
### How it works
|
||||
|
||||
A dry contact is a current interface. When the electricity meter triggers, a current of 0.1A starts flowing between **s0+** and **s0-**. On **DI0**, the voltage rises to 24V and triggers an interrupt. When the electricity meter disconnects **s0+** and **s0-**, **DI0** is grounded over the resistor.
|
||||
|
||||
On my device, the contact closes 1,000 times per kWh (this value varies between metering devices).
|
||||
|
||||
To count these peaks reliably, I created [a C program][17] that registers an interrupt service routine on the DI0 input and counts upwards in memory. Once a minute, the values from memory are written to an [SQLite][18] database.
|
||||
|
||||
The overall meter reading is also written to a text file and can be preset with a starting value. This acts as a copy of the overall metering value of the meter in the cabinet.
|
||||
|
||||
![Energy logger architecture][19]
|
||||
|
||||
Energy logger architecture
|
||||
|
||||
The data is visualized using [Node-RED][20], and I can access overviews, like the daily consumption dashboard below, over a web-based GUI.
|
||||
|
||||
![Node-RED based GUI][21]
|
||||
|
||||
Daily overview in the Node-RED GUI
|
||||
|
||||
For the daily overview, I calculate the hourly costs based on the consumption data (the large bar chart). On the top-left of the dashboard you can see the actual power; below that is the daily consumption (energy and costs). The water heater for the shower causes the large peak in the bar chart.
|
||||
|
||||
### A reliable system
|
||||
|
||||
Aside from a lost timestamp during a power failure (the real-time clock in the Siemens device is not backed by a battery by default), everything has been working fine for more than one-and-a-half years.
|
||||
|
||||
If you can set up the whole Linux system completely from the command line, you'll get a reliable and flexible system with the ability to link interrupt service routines to the I/O level.
|
||||
|
||||
Because the I/O Shield runs on standard control voltage (24V), you can extend its functionality with the whole range of standardized industrial components (e.g., relays, sensors, actors, etc.). And, due to its open architecture, this system can be extended easily and applied to other applications, like for monitoring gas or water consumption or as a weather station, a simple controller for tasks, and more.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/20/5/energy-monitoring
|
||||
|
||||
作者：[Stephan Avenwedde][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/hansic99
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/Collaboration%20for%20health%20innovation.png?itok=s4O5EX2w (lightbulb drawing outline)
|
||||
[2]: https://opensource.com/sites/default/files/uploads/openenergylogger_1_electricity_meter.jpg (Actaris ACE3000 electricity meter)
|
||||
[3]: https://en.wikipedia.org/wiki/Dry_contact
|
||||
[4]: https://opensource.com/sites/default/files/uploads/openenergylogger_2_siemens_device.jpg (Siemens SIMATIC IOT2020)
|
||||
[5]: https://support.industry.siemens.com/tf/ww/en/posts/new-example-image-version-online/189090/?page=0&pageSize=10
|
||||
[6]: https://de.rs-online.com/web/p/products/1244037
|
||||
[7]: https://de.rs-online.com/web/p/products/1354133
|
||||
[8]: https://de.rs-online.com/web/p/micro-sd-karten/7582584/
|
||||
[9]: https://www.amazon.de/300Mbit-WLAN-Adapter-Hochleistungs-Antennen-Dual-Band/dp/B00LLIOT34
|
||||
[10]: https://de.rs-online.com/web/p/products/8153133
|
||||
[11]: https://de.rs-online.com/web/p/din-schienenklemmen-ohne-sicherung/0425190/
|
||||
[12]: https://de.rs-online.com/web/p/din-schienenklemmen-zubehor/0202574/
|
||||
[13]: https://de.rs-online.com/web/p/din-schienen/2835729/
|
||||
[14]: https://de.rs-online.com/web/p/widerstande-durchsteckmontage/2142673/
|
||||
[15]: https://opensource.com/sites/default/files/uploads/openenergylogger_3_assembled_device.jpg (Mounted and hooked up energy logger)
|
||||
[16]: https://opensource.com/sites/default/files/uploads/openenergylogger_4_wiring.png (Wiring between meter and energy logger)
|
||||
[17]: https://github.com/hANSIc99/OpenEnergyLogger
|
||||
[18]: https://www.sqlite.org/index.html
|
||||
[19]: https://opensource.com/sites/default/files/uploads/openenergylogger_5_architecure.png (Energy logger architecture)
|
||||
[20]: https://nodered.org/
|
||||
[21]: https://opensource.com/sites/default/files/uploads/openenergylogger_6_dashboard.png (Node-RED based GUI)
|
||||
@ -0,0 +1,240 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Modify a disk image to create a Raspberry Pi-based homelab)
|
||||
[#]: via: (https://opensource.com/article/20/5/disk-image-raspberry-pi)
|
||||
[#]: author: (Chris Collins https://opensource.com/users/clcollins)
|
||||
|
||||
Modify a disk image to create a Raspberry Pi-based homelab
|
||||
======
|
||||
Create a "private cloud at home" with a Raspberry Pi or other
|
||||
single-board computer.
|
||||
![Science lab with beakers][1]
|
||||
|
||||
Building a [homelab][2] can be a fun way to entertain yourself while learning new concepts and experimenting with new technologies. Thanks to the popularity of single-board computers (SBCs), led by the [Raspberry Pi][3], it is easier than ever to build a multi-computer lab right from the comfort of your home. Creating a "private cloud at home" is also a great way to get exposure to cloud-native technologies for considerably less money than trying to replicate the same setup with a major cloud provider.
|
||||
|
||||
This article explains how to modify a disk image for the Raspberry Pi or another SBC, pre-configure the host for SSH (secure shell), and disable the service that forces interaction for configuration on first boot. This is a great way to make your devices "boot and go," similar to cloud instances. Later, you can do more specialized, in-depth configurations using automated processes over an SSH connection.
|
||||
|
||||
Also, as you add more Pis to your lab, modifying disk images lets you just write the image to an SD card, drop it into the Pi, and go!
|
||||
|
||||
![Multiple Raspberry Pi computers, a switch, and a power bank][4]
|
||||
|
||||
### Decompress and mount the image
|
||||
|
||||
For this project, you need to modify a server disk image. I used the [Fedora Server 31 ARM][5] image during testing. After you download the disk image and [verify its checksum][6], you need to decompress and mount it to a location on the host computer's file system so you can modify it as needed.
|
||||
|
||||
You can use the **[xz][7]** command to decompress the Fedora Server image by using the **\--decompress** argument:
|
||||
|
||||
|
||||
```
|
||||
`xz --decompress Fedora-Server-armhfp-X-y.z-sda.raw.xz`
|
||||
```
|
||||
|
||||
This leaves you with a raw, decompressed disk image (which automatically replaces the **.xz** compressed file). This raw disk image is just what it sounds like: a file containing all the data that would be on a formatted and installed disk. That includes partition information, the boot partition, the root partition, and any other partitions. You need to mount the partition you intend to work in, but to do that, you need information about where that partition starts in the disk image and the size of the sectors on the disk, so you can mount the file at the right sector.
|
||||
|
||||
Luckily, you can use the [**fdisk**][8] command on a disk image just as easily as on a real disk and use the **\--list** or **-l** argument to view the list of partitions and their information:
|
||||
|
||||
|
||||
```
|
||||
# Use fdisk to list the partitions in the raw image:
|
||||
$ fdisk -l Fedora-Server-armhfp-31-1.9-sda.raw
|
||||
Disk Fedora-Server-armhfp-X-y.z-sda.raw: 3.2 GiB, 3242196992 bytes, 6332416 sectors
|
||||
Units: sectors of 1 * 512 = 512 bytes
|
||||
Sector size (logical/physical): 512 bytes / 512 bytes
|
||||
I/O size (minimum/optimal): 512 bytes / 512 bytes
|
||||
Disklabel type: dos
|
||||
Disk identifier: 0xdaad9f57
|
||||
|
||||
Device Boot Start End Sectors Size Id Type
|
||||
Fedora-Server-armhfp-X-y.z-sda.raw1 8192 163839 155648 76M c W95 F
|
||||
Fedora-Server-armhfp-X-y.z-sda.raw2 * 163840 1163263 999424 488M 83 Linux
|
||||
Fedora-Server-armhfp-X-y.z-sda.raw3 1163264 6047743 4884480 2.3G 83 Linux
|
||||
```
|
||||
|
||||
All the information you need is available in this output. Line 3 indicates the sector size, both logical and physical: (512 bytes / 512 bytes).
|
||||
|
||||
The list of devices shows the partitions inside the raw disk image. The first one, **Fedora-Server-armhfp-X-y.z-sda.raw1** is no doubt the bootloader partition because it is the first, small (only 76MB), and type W95 FAT32 (LBA), as identified by the Id "c," a FAT32 partition for booting off the SD card.
|
||||
|
||||
The second partition is not very large either, just 488MB. This partition is a Linux native-type partition (Id 83), and it probably is the Linux boot partition containing the kernel and [initramfs][9].
|
||||
|
||||
The third partition is what you probably want: it is 2.3GB, so it should have the majority of the distribution on it, and it is a Linux-native partition type, which is expected. This should contain the partition and data you want to modify.
|
||||
|
||||
The third partition starts on sector 1163264 (indicated by the "Start" column in the output of **fdisk**), so your mount offset is **595591168**, calculated by multiplying the sector size (512) by the start sector (1163264) (i.e., **512 * 1163264**). This means you need to mount the file with an offset of 595591168 to be in the right place at the mount point.
|
||||
|
||||
ARMed (see what I did there?) with this information, you can now mount the third partition to a directory in your homedir:
|
||||
|
||||
|
||||
```
|
||||
$ mkdir ~/mnt
|
||||
$ sudo mount -o loop,offset=595591168 Fedora-Server-armhfp-X-y.z-sda.raw ~/mnt
|
||||
$ ls ~/mnt
|
||||
```
|
||||
|
||||
### Work directly within the disk image
|
||||
|
||||
Once the disk image has been decompressed and mounted to a spot on the host computer, it is time to start modifying the image to suit your needs. In my opinion, the easiest way to make changes to the image is to use **chroot** to change the working root of your session to that of the mounted image. There's a tricky bit, though.
|
||||
|
||||
When you change root, your session will use the binaries from the new root. Unless you are doing all of this from an ARM system, the architecture of the decompressed disk image will not be the same as the host system you are using. Even inside the **chroot**, the host system will not be able to make use of binaries with a different architecture. At least, not natively.
|
||||
|
||||
Luckily, there is a solution: **qemu-user-static**. From the [Debian Wiki][10]:
|
||||
|
||||
> "[qemu-user-static] provides the user mode emulation binaries, built statically. In this mode QEMU can launch Linux processes compiled for one CPU on another CPU… If binfmt-support package is installed, qemu-user-static package will register binary formats which the provided emulators can handle, so that it will be possible to run foreign binaries directly."
|
||||
|
||||
This is exactly what you need to be able to work in the non-native architecture inside your chroot. If the host system is Fedora, install the **qemu-user-static** package with DNF, and restart **systemd-binfmt.service**:
|
||||
|
||||
|
||||
```
|
||||
# Enable non-native arch chroot with DNF, adding new binary format information
|
||||
# Output suppressed for brevity
|
||||
$ dnf install qemu-user-static
|
||||
$ systemctl restart systemd-binfmt.service
|
||||
```
|
||||
|
||||
With this, you should be able to change root to the mounted disk image and run the **uname** command to verify that everything is working:
|
||||
|
||||
|
||||
```
|
||||
sudo chroot ~/mnt/ /usr/bin/uname -a -r
|
||||
Linux marvin 5.5.16-200.fc31.x86_64 #1 SMP Wed Apr 8 16:43:33 UTC 2020 armv7l armv7l armv7l GNU/Linux
|
||||
```
|
||||
|
||||
Running **uname** from within the changed root shows **armv7l** in the output—the architecture of the raw disk image—and not the host machine. Everything is working as expected, and you can continue on to modify the image.
|
||||
|
||||
### Modify the disk image
|
||||
|
||||
Now that you can change directly into the ARM-based disk image and work in that environment, you can begin to make changes to the image itself. You want to set up the image so it can be booted and immediately accessed without having to do any additional setup directly on the Raspberry Pi. To do this, you need to install and enable sshd (the OpenSSH daemon) and add the authorized keys for SSH access.
|
||||
|
||||
And to make this behave more like a cloud environment and realize the dream of a private cloud at home, add a local user, give that user **sudo** rights, and (to be just like the cloud heavyweights) allow that user to use **sudo** without a password.
|
||||
|
||||
So, your to-do list is:
|
||||
|
||||
* Install and enable SSHD (SSHD is already installed and enabled in the Fedora ARM image, but you may need to do this manually for your distribution)
|
||||
* Set up a local user
|
||||
* Allow the local user to use sudo (without a password, optional)
|
||||
* Add authorized keys
|
||||
* Allow root to SSH with the authorized keys (optional)
|
||||
|
||||
|
||||
|
||||
I use the GitHub feature that allows you to upload your public SSH keys and make them available at **[https://github.com/<your_github_username>.keys][11]**. I find this to be a convenient way to distribute public keys, although I am paranoid enough that I always check that the downloaded keys match what I am expecting. If you don't want to use this method, you can copy your public keys into the **chroot** directly from your host computer, or you can host your keys on a web server that you control in order to use the same workflow.
|
||||
|
||||
To start modifying the disk image, **chroot** into the mounted disk image again, this time starting a shell so multiple commands can be run:
|
||||
|
||||
|
||||
```
|
||||
# Output of these commands (if any) are omitted for brevity
|
||||
$ sudo chroot ~/mnt /bin/bash
|
||||
|
||||
# Install openssh-server and enable it (already done on Fedora)
|
||||
$ dnf install -y openssh-server
|
||||
$ systemctl enable sshd.service
|
||||
|
||||
# Allow root to SSH with your authorized keys
|
||||
$ mkdir /root/.ssh
|
||||
|
||||
# Download, or otherwise add to the authorized_keys file, your public keys
|
||||
# Replace the URL with the path to your own public keys
|
||||
$ curl <https://github.com/clcollins.keys> -o /root/.ssh/authorized_keys
|
||||
$ chmod 700 /root/.ssh
|
||||
$ chmod 600 /root/.ssh/authorized_keys
|
||||
|
||||
# Add a local user, and put them in the wheel group
|
||||
# Change the group and user to whatever you desire
|
||||
groupadd chris
|
||||
useradd -g chris -G wheel -m -u 1000 chris
|
||||
|
||||
# Download or add your authorized keys
|
||||
# Change the homedir and URL as you've done above
|
||||
mkdir /home/chris/.ssh
|
||||
curl <https://github.com/clcollins.keys> -o /home/chris/.ssh/authorized_keys
|
||||
chmod 700 /home/chris/.ssh
|
||||
chmod 600 /home/chris/.ssh/authorized_keys
|
||||
chown -R chris.chris /home/chris/.ssh/
|
||||
|
||||
# Allow the wheel group (with your local user) to use suso without a password
|
||||
echo "%wheel ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers.d/91-wheel-nopasswd
|
||||
```
|
||||
|
||||
This is all that generally needs to be done to set up SSH into a Raspberry Pi or other single-board computer on first boot. However, each distribution has its own quirks. For example, Rasbian already includes a local user, **pi**, and does not use the **wheel** group. So for Raspbian, it may be better to use the existing user or to delete the **pi** user and replace it with another.
|
||||
|
||||
In the case of Fedora ARM, the image prompts you to finish setup on first boot. This defeats the purpose of the changes you made above, especially since it blocks startup entirely until setup is complete. Your goal is to make the Raspberry Pi function like part of a private cloud infrastructure, and that workflow includes setting up the host remotely via SSH when it starts up. Disable the initial setup, controlled by the service **initial-setup.service**:
|
||||
|
||||
|
||||
```
|
||||
# Disable the initial-setup.service for both the multi-user and graphical targets
|
||||
unlink /etc/systemd/system/multi-user.target.wants/initial-setup.service
|
||||
unlink /etc/systemd/system/graphical.target.wants/initial-setup.service
|
||||
```
|
||||
|
||||
While you are in the change root, you can make any other changes you might want for your systems or just leave it at that and follow the cloud-native workflow of configuration over SSH after first boot.
|
||||
|
||||
### Recompress and install the modified image
|
||||
|
||||
With these changes to your system completed, all that is left is to recompress the disk image and install it on an SD card for your Raspberry Pi.
|
||||
|
||||
Make sure to exit the chroot, then unmount the disk image:
|
||||
|
||||
|
||||
```
|
||||
`$ sudo umount ~/mnt/`
|
||||
```
|
||||
|
||||
Just as you decompressed the image initially, you can use the **xz** command again to compress the image. By using the **\--keep** argument, **xz** will leave the raw image rather than cleaning it up. While this uses more disk space, leaving the uncompressed image allows you to make incremental changes to the images you are working with without needing to decompress them each time. This is great for saving time while testing and tweaking images to your liking:
|
||||
|
||||
|
||||
```
|
||||
# Compress the raw disk image to a .xz file, but keep the raw disk image
|
||||
xz --compress Fedora-Server-armhfp-31-1.9-sda.raw --keep
|
||||
```
|
||||
|
||||
The compression takes a while, so take this time to stand up, stretch, and get your blood flowing again.
|
||||
|
||||
Once the compression is done, the new, modified disk image can be copied to an SD card to use with a Raspberry Pi. The standard **dd** method to copy the image to the SD card works fine, but I like to use Fedora's **arm-image-installer** because of the options it provides when working with unedited images. It also works great for edited images and is a little more user-friendly than the **dd** command.
|
||||
|
||||
Make sure to check which disk the SD card is on and use that for the **\--media** argument:
|
||||
|
||||
|
||||
```
|
||||
# Use arm-image-installer to copy the modified disk image to the SD card
|
||||
arm-image-installer --image=Fedora-Server-armhfp-X-y.z-sda.raw.xz --target=rpi3 --media=/dev/sdc --norootpass --resizefs -y
|
||||
```
|
||||
|
||||
Now you are all set with a new, modified Fedora Server ARM image for Raspberry Pis or other single board computers, ready to boot and immediately SSH into with your modifications. This method can also be used to make other changes, and you can use it with other distributions' raw disk images if you prefer them to Fedora. This is a good base to start building a private-cloud-at-home homelab. In future articles, I will guide you through setting up a homelab using cloud technologies and automation.
|
||||
|
||||
### Further reading
|
||||
|
||||
A lot of research went into learning how to do the things in this article. Two of the most helpful sources I found for learning how to customize disk images and work with non-native architectures are listed below. They were extremely helpful in rounding the corner from "I have no idea what I'm doing" to "OK, I can do this!"
|
||||
|
||||
* [How to modify a raw disk image of your custom Linux distro][12]
|
||||
* [Using DNF wiki][13]
|
||||
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/20/5/disk-image-raspberry-pi
|
||||
|
||||
作者：[Chris Collins][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/clcollins
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/science_experiment_beaker_lab.png?itok=plKWRhlU (Science lab with beakers)
|
||||
[2]: https://opensource.com/article/19/3/home-lab
|
||||
[3]: https://opensource.com/resources/raspberry-pi
|
||||
[4]: https://opensource.com/sites/default/files/uploads/raspberrypi_homelab.jpg (Multiple Raspberry Pi computers, a switch, and a power bank)
|
||||
[5]: https://arm.fedoraproject.org/
|
||||
[6]: https://arm.fedoraproject.org/verify.html
|
||||
[7]: https://tukaani.org/xz/
|
||||
[8]: https://en.wikipedia.org/wiki/Fdisk
|
||||
[9]: https://wiki.debian.org/initramfs
|
||||
[10]: https://wiki.debian.org/RaspberryPi/qemu-user-static
|
||||
[11]: https://github.com/%3Cyour_github_username%3E.keys
|
||||
[12]: https://www.linux.com/news/how-modify-raw-disk-image-your-custom-linux-distro/
|
||||
[13]: https://wiki.mageia.org/en/Using_DNF#Setting_up_a_container_for_a_non-native_architectur
|
||||
@ -0,0 +1,269 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Start using systemd as a troubleshooting tool)
|
||||
[#]: via: (https://opensource.com/article/20/5/systemd-troubleshooting-tool)
|
||||
[#]: author: (David Both https://opensource.com/users/dboth)
|
||||
|
||||
Start using systemd as a troubleshooting tool
|
||||
======
|
||||
While systemd is not really a troubleshooting tool, the information in
|
||||
its output points the way toward solving problems.
|
||||
![Magnifying glass on code][1]
|
||||
|
||||
No one would really consider systemd to be a troubleshooting tool, but when I encountered a problem on my webserver, my growing knowledge of systemd and some of its features helped me locate and circumvent the problem.
|
||||
|
||||
The problem was that my server, yorktown, which provides name services, DHCP, NTP, HTTPD, and SendMail email services for my home office network, failed to start the Apache HTTPD daemon during normal startup. I had to start it manually after I realized that it was not running. The problem had been going on for some time, and I recently got around to trying to fix it.
|
||||
|
||||
Some of you will say that systemd itself is the cause of this problem, and, based on what I know now, I agree with you. However, I had similar types of problems with SystemV. (In the [first article][2] in this series, I looked at the controversy around systemd as a replacement for the old SystemV init program and startup scripts. If you're interested in learning more about systemd, read the [second][3] and [third][4] articles, too.) No software is perfect, and neither systemd nor SystemV is an exception, but systemd provides far more information for problem-solving than SystemV ever offered.
|
||||
|
||||
### Determining the problem
|
||||
|
||||
The first step to finding the source of this problem is to determine the httpd service's status:
|
||||
|
||||
|
||||
```
|
||||
[root@yorktown ~]# systemctl status httpd
|
||||
● httpd.service - The Apache HTTP Server
|
||||
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
|
||||
Active: failed (Result: exit-code) since Thu 2020-04-16 11:54:37 EDT; 15min ago
|
||||
Docs: man:httpd.service(8)
|
||||
Process: 1101 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
|
||||
Main PID: 1101 (code=exited, status=1/FAILURE)
|
||||
Status: "Reading configuration..."
|
||||
CPU: 60ms
|
||||
|
||||
Apr 16 11:54:35 yorktown.both.org systemd[1]: Starting The Apache HTTP Server...
|
||||
Apr 16 11:54:37 yorktown.both.org httpd[1101]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address 192.168.0.52:80
|
||||
Apr 16 11:54:37 yorktown.both.org httpd[1101]: no listening sockets available, shutting down
|
||||
Apr 16 11:54:37 yorktown.both.org httpd[1101]: AH00015: Unable to open logs
|
||||
Apr 16 11:54:37 yorktown.both.org systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
|
||||
Apr 16 11:54:37 yorktown.both.org systemd[1]: httpd.service: Failed with result 'exit-code'.
|
||||
Apr 16 11:54:37 yorktown.both.org systemd[1]: Failed to start The Apache HTTP Server.
|
||||
[root@yorktown ~]#
|
||||
```
|
||||
|
||||
This status information is one of the systemd features that I find much more useful than anything SystemV offers. The amount of helpful information here leads me easily to a logical conclusion that takes me in the right direction. All I ever got from the old **chkconfig** command is whether or not the service is running and the process ID (PID) if it is. That is not very helpful.
|
||||
|
||||
The key entry in this status report shows that HTTPD cannot bind to the IP address, which means it cannot accept incoming requests. This indicates that the network is not starting fast enough to be ready for the HTTPD service to bind to the IP address because the IP address has not yet been set. This is not supposed to happen, so I explored my network service systemd startup configuration files; all appeared to be correct with the right "after" and "requires" statements. Here is the **/lib/systemd/system/httpd.service** file from my server:
|
||||
|
||||
|
||||
```
|
||||
# Modifying this file in-place is not recommended, because changes
|
||||
# will be overwritten during package upgrades. To customize the
|
||||
# behaviour, run "systemctl edit httpd" to create an override unit.
|
||||
|
||||
# For example, to pass additional options (such as -D definitions) to
|
||||
# the httpd binary at startup, create an override unit (as is done by
|
||||
# systemctl edit) and enter the following:
|
||||
|
||||
# [Service]
|
||||
# Environment=OPTIONS=-DMY_DEFINE
|
||||
|
||||
[Unit]
|
||||
Description=The Apache HTTP Server
|
||||
Wants=httpd-init.service
|
||||
After=network.target remote-fs.target nss-lookup.target httpd-init.service
|
||||
Documentation=man:httpd.service(8)
|
||||
|
||||
[Service]
|
||||
Type=notify
|
||||
Environment=LANG=C
|
||||
|
||||
ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND
|
||||
ExecReload=/usr/sbin/httpd $OPTIONS -k graceful
|
||||
# Send SIGWINCH for graceful stop
|
||||
KillSignal=SIGWINCH
|
||||
KillMode=mixed
|
||||
PrivateTmp=true
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
The **httpd.service** unit file explicitly specifies that it should load after the **network.target** and the **httpd-init.service** (among others). I tried to find all of these services using the **systemctl list-units** command and searching for them in the resulting data stream. All were present and should have ensured that the httpd service did not load before the network IP address was set.
|
||||
|
||||
### First solution
|
||||
|
||||
A bit of searching on the internet confirmed that others had encountered similar problems with httpd and other services. This appears to happen because one of the required services indicates to systemd that it has finished its startup—but it actually spins off a child process that has not finished. After a bit more searching, I came up with a circumvention.
|
||||
|
||||
I could not figure out why the IP address was taking so long to be assigned to the network interface card. So, I thought that if I could delay the start of the HTTPD service by a reasonable amount of time, the IP address would be assigned by that time.
|
||||
|
||||
Fortunately, the **/lib/systemd/system/httpd.service** file above provides some direction. Although it says not to alter it, it does indicate how to proceed: Use the command **systemctl edit httpd**, which automatically creates a new file (**/etc/systemd/system/httpd.service.d/override.conf**) and opens the [GNU Nano][5] editor. (If you are not familiar with Nano, be sure to look at the hints at the bottom of the Nano interface.)
|
||||
|
||||
Add the following text to the new file and save it:
|
||||
|
||||
|
||||
```
|
||||
[root@yorktown ~]# cd /etc/systemd/system/httpd.service.d/
|
||||
[root@yorktown httpd.service.d]# ll
|
||||
total 4
|
||||
-rw-r--r-- 1 root root 243 Apr 16 11:43 override.conf
|
||||
[root@yorktown httpd.service.d]# cat override.conf
|
||||
# Trying to delay the startup of httpd so that the network is
|
||||
# fully up and running so that httpd can bind to the correct
|
||||
# IP address
|
||||
#
|
||||
# By David Both, 2020-04-16
|
||||
|
||||
[Service]
|
||||
ExecStartPre=/bin/sleep 30
|
||||
```
|
||||
|
||||
The **[Service]** section of this override file contains a single line that delays the start of the HTTPD service by 30 seconds. The following status command shows the service status during the wait time:
|
||||
|
||||
|
||||
```
|
||||
[root@yorktown ~]# systemctl status httpd
|
||||
● httpd.service - The Apache HTTP Server
|
||||
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
|
||||
Drop-In: /etc/systemd/system/httpd.service.d
|
||||
└─override.conf
|
||||
/usr/lib/systemd/system/httpd.service.d
|
||||
└─php-fpm.conf
|
||||
Active: activating (start-pre) since Thu 2020-04-16 12:14:29 EDT; 28s ago
|
||||
Docs: man:httpd.service(8)
|
||||
Cntrl PID: 1102 (sleep)
|
||||
Tasks: 1 (limit: 38363)
|
||||
Memory: 260.0K
|
||||
CPU: 2ms
|
||||
CGroup: /system.slice/httpd.service
|
||||
└─1102 /bin/sleep 30
|
||||
|
||||
Apr 16 12:14:29 yorktown.both.org systemd[1]: Starting The Apache HTTP Server...
|
||||
Apr 16 12:15:01 yorktown.both.org systemd[1]: Started The Apache HTTP Server.
|
||||
[root@yorktown ~]#
|
||||
```
|
||||
|
||||
And this command shows the status of the HTTPD service after the 30-second delay expires. The service is up and running correctly:
|
||||
|
||||
|
||||
```
|
||||
[root@yorktown ~]# systemctl status httpd
|
||||
● httpd.service - The Apache HTTP Server
|
||||
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
|
||||
Drop-In: /etc/systemd/system/httpd.service.d
|
||||
└─override.conf
|
||||
/usr/lib/systemd/system/httpd.service.d
|
||||
└─php-fpm.conf
|
||||
Active: active (running) since Thu 2020-04-16 12:15:01 EDT; 1min 18s ago
|
||||
Docs: man:httpd.service(8)
|
||||
Process: 1102 ExecStartPre=/bin/sleep 30 (code=exited, status=0/SUCCESS)
|
||||
Main PID: 1567 (httpd)
|
||||
Status: "Total requests: 0; Idle/Busy workers 100/0;Requests/sec: 0; Bytes served/sec: 0 B/sec"
|
||||
Tasks: 213 (limit: 38363)
|
||||
Memory: 21.8M
|
||||
CPU: 82ms
|
||||
CGroup: /system.slice/httpd.service
|
||||
├─1567 /usr/sbin/httpd -DFOREGROUND
|
||||
├─1569 /usr/sbin/httpd -DFOREGROUND
|
||||
├─1570 /usr/sbin/httpd -DFOREGROUND
|
||||
├─1571 /usr/sbin/httpd -DFOREGROUND
|
||||
└─1572 /usr/sbin/httpd -DFOREGROUND
|
||||
|
||||
Apr 16 12:14:29 yorktown.both.org systemd[1]: Starting The Apache HTTP Server...
|
||||
Apr 16 12:15:01 yorktown.both.org systemd[1]: Started The Apache HTTP Server.
|
||||
```
|
||||
|
||||
I could have experimented to see if a shorter delay would work as well, but my system is not that critical, so I decided not to. It works reliably as it is, so I am happy.
|
||||
|
||||
Because I gathered all this information, I reported it to Red Hat Bugzilla as Bug [1825554][6]. I believe that it is much more productive to report bugs than it is to complain about them.
|
||||
|
||||
### The better solution
|
||||
|
||||
A couple of days after reporting this as a bug, I received a response indicating that systemd is just the manager, and if httpd needs to be ordered after some requirements are met, it needs to be expressed in the unit file. The response pointed me to the **httpd.service** man page. I wish I had found this earlier because it is a better solution than the one I came up with. This solution is explicitly targeted to the prerequisite target unit rather than a somewhat random delay.
|
||||
|
||||
From the [**httpd.service** man page][7]:
|
||||
|
||||
> **Starting the service at boot time**
|
||||
>
|
||||
> The httpd.service and httpd.socket units are _disabled_ by default. To start the httpd service at boot time, run: **systemctl enable httpd.service**. In the default configuration, the httpd daemon will accept connections on port 80 (and, if mod_ssl is installed, TLS connections on port 443) for any configured IPv4 or IPv6 address.
|
||||
>
|
||||
> If httpd is configured to depend on any specific IP address (for example, with a "Listen" directive) which may only become available during start-up, or if httpd depends on other services (such as a database daemon), the service _must_ be configured to ensure correct start-up ordering.
|
||||
>
|
||||
> For example, to ensure httpd is only running after all configured network interfaces are configured, create a drop-in file (as described above) with the following section:
|
||||
>
|
||||
> [Unit]
|
||||
> After=network-online.target
|
||||
> Wants=network-online.target
|
||||
|
||||
I still think this is a bug because it is quite common—at least in my experience—to use a **Listen** directive in the **httpd.conf** configuration file. I have always used **Listen** directives, even on hosts with only a single IP address, and it is clearly necessary on hosts with multiple network interface cards (NICs) and internet protocol (IP) addresses. Adding the lines above to the **/usr/lib/systemd/system/httpd.service** default file would not cause problems for configurations that do not use a **Listen** directive and would prevent this problem for those that do.
|
||||
|
||||
In the meantime, I will use the suggested solution.
|
||||
|
||||
### Next steps
|
||||
|
||||
This article describes a problem I had with starting the Apache HTTPD service on my server. It leads you through the problem determination steps I took and shows how I used systemd to assist. I also covered the circumvention I implemented using systemd and the better solution that followed from my bug report.
|
||||
|
||||
As I mentioned at the start, it is very likely that this is the result of a problem with systemd, specifically the configuration for httpd startup. Nevertheless, systemd provided me with the tools to locate the likely source of the problem and to formulate and implement a circumvention. Neither solution really resolves the problem to my satisfaction. For now, the root cause of the problem still exists and must be fixed. If that is simply adding the recommended lines to the **/usr/lib/systemd/system/httpd.service** file, that would work for me.
|
||||
|
||||
One of the things I discovered during this is process is that I need to learn more about defining the sequences in which things start. I will explore that in my next article, the fifth in this series.
|
||||
|
||||
### Resources
|
||||
|
||||
There is a great deal of information about systemd available on the internet, but much is terse, obtuse, or even misleading. In addition to the resources mentioned in this article, the following webpages offer more detailed and reliable information about systemd startup.
|
||||
|
||||
* The Fedora Project has a good, practical [guide][8] [to systemd][8]. It has pretty much everything you need to know in order to configure, manage, and maintain a Fedora computer using systemd.
|
||||
* The Fedora Project also has a good [cheat sheet][9] that cross-references the old SystemV commands to comparable systemd ones.
|
||||
* For detailed technical information about systemd and the reasons for creating it, check out [Freedesktop.org][10]'s [description of systemd][11].
|
||||
* [Linux.com][12]'s "More systemd fun" offers more advanced systemd [information and tips][13].
|
||||
|
||||
|
||||
|
||||
There is also a series of deeply technical articles for Linux sysadmins by Lennart Poettering, the designer and primary developer of systemd. These articles were written between April 2010 and September 2011, but they are just as relevant now as they were then. Much of everything else good that has been written about systemd and its ecosystem is based on these papers.
|
||||
|
||||
* [Rethinking PID 1][14]
|
||||
* [systemd for Administrators, Part I][15]
|
||||
* [systemd for Administrators, Part II][16]
|
||||
* [systemd for Administrators, Part III][17]
|
||||
* [systemd for Administrators, Part IV][18]
|
||||
* [systemd for Administrators, Part V][19]
|
||||
* [systemd for Administrators, Part VI][20]
|
||||
* [systemd for Administrators, Part VII][21]
|
||||
* [systemd for Administrators, Part VIII][22]
|
||||
* [systemd for Administrators, Part IX][23]
|
||||
* [systemd for Administrators, Part X][24]
|
||||
* [systemd for Administrators, Part XI][25]
|
||||
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/20/5/systemd-troubleshooting-tool
|
||||
|
||||
作者：[David Both][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/dboth
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/find-file-linux-code_magnifying_glass_zero.png?itok=E2HoPDg0 (Magnifying glass on code)
|
||||
[2]: https://opensource.com/article/20/4/systemd
|
||||
[3]: https://opensource.com/article/20/4/systemd-startup
|
||||
[4]: https://opensource.com/article/20/4/understanding-and-using-systemd-units
|
||||
[5]: https://www.nano-editor.org/
|
||||
[6]: https://bugzilla.redhat.com/show_bug.cgi?id=1825554
|
||||
[7]: https://www.mankier.com/8/httpd.service#Description-Starting_the_service_at_boot_time
|
||||
[8]: https://docs.fedoraproject.org/en-US/quick-docs/understanding-and-administering-systemd/index.html
|
||||
[9]: https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
|
||||
[10]: http://Freedesktop.org
|
||||
[11]: http://www.freedesktop.org/wiki/Software/systemd
|
||||
[12]: http://Linux.com
|
||||
[13]: https://www.linux.com/training-tutorials/more-systemd-fun-blame-game-and-stopping-services-prejudice/
|
||||
[14]: http://0pointer.de/blog/projects/systemd.html
|
||||
[15]: http://0pointer.de/blog/projects/systemd-for-admins-1.html
|
||||
[16]: http://0pointer.de/blog/projects/systemd-for-admins-2.html
|
||||
[17]: http://0pointer.de/blog/projects/systemd-for-admins-3.html
|
||||
[18]: http://0pointer.de/blog/projects/systemd-for-admins-4.html
|
||||
[19]: http://0pointer.de/blog/projects/three-levels-of-off.html
|
||||
[20]: http://0pointer.de/blog/projects/changing-roots
|
||||
[21]: http://0pointer.de/blog/projects/blame-game.html
|
||||
[22]: http://0pointer.de/blog/projects/the-new-configuration-files.html
|
||||
[23]: http://0pointer.de/blog/projects/on-etc-sysinit.html
|
||||
[24]: http://0pointer.de/blog/projects/instances.html
|
||||
[25]: http://0pointer.de/blog/projects/inetd.html
|
||||
@ -0,0 +1,201 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: ( )
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Tips and tricks for optimizing container builds)
|
||||
[#]: via: (https://opensource.com/article/20/5/optimize-container-builds)
|
||||
[#]: author: (Ravi Chandran https://opensource.com/users/ravichandran)
|
||||
|
||||
Tips and tricks for optimizing container builds
|
||||
======
|
||||
Try these techniques to minimize the number and length of your container
|
||||
build iterations.
|
||||
![Toolbox drawing of a container][1]
|
||||
|
||||
How many iterations does it take to get a container configuration just right? And how long does each iteration take? Well, if you answered "too many times and too long," then my experiences are similar to yours. On the surface, creating a configuration file seems like a straightforward exercise: implement the same steps in a configuration file that you would perform if you were installing the system by hand. Unfortunately, I've found that it usually doesn't quite work that way, and a few "tricks" are handy for such DevOps exercises.
|
||||
|
||||
In this article, I'll share some techniques I've found that help minimize the number and length of iterations. In addition, I'll outline a few good practices beyond the [standard ones][2].
|
||||
|
||||
In the [tutorial repository][3] from my previous article about [containerizing build systems][4], I've added a folder called **/tutorial2_docker_tricks** with an example covering some of the tricks that I'll walk through in this post. If you want to follow along and you have Git installed, you can pull it locally with:
|
||||
|
||||
|
||||
```
|
||||
`$ git clone https://github.com/ravi-chandran/dockerize-tutorial`
|
||||
```
|
||||
|
||||
The tutorial has been tested with Docker Desktop Edition, although it should work with any compatible Linux container system (like [Podman][5]).
|
||||
|
||||
### Save time on container image build iterations
|
||||
|
||||
If the Dockerfile involves downloading and installing a 5GB file, each iteration of **docker image build** could take a lot of time even with good network speeds. And forgetting to include one item to be installed can mean rebuilding all the layers after that point.
|
||||
|
||||
One way around that challenge is to use a local HTTP server to avoid downloading large files from the internet multiple times during **docker image build** iterations. To illustrate this by example, say you need to create a container image with Anaconda 3 under Ubuntu 18.04. The Anaconda 3 installer is a ~0.5GB file, so this will be the "large" file for this example.
|
||||
|
||||
Note that you don't want to use the **COPY** instruction, as it creates a new layer. You should also delete the large installer after using it to minimize the container image size. You could use [multi-stage builds][6], but I've found the following approach sufficient and quite effective.
|
||||
|
||||
The basic idea is to use a Python-based HTTP server locally to serve the large file(s) and have the Dockerfile **wget** the large file(s) from this local server. Let's explore the details of how to set this up effectively. As a reminder, you can access the [full example][7].
|
||||
|
||||
The necessary contents of the folder **tutorial2_docker_tricks/** in this example repository are:
|
||||
|
||||
|
||||
```
|
||||
tutorial2_docker_tricks/
|
||||
├── build_docker_image.sh # builds the docker image
|
||||
├── run_container.sh # instantiates a container from the image
|
||||
├── install_anaconda.dockerfile # Dockerfile for creating our target docker image
|
||||
├── .dockerignore # used to ignore contents of the installer/ folder from the docker context
|
||||
├── installer # folder with all our large files required for creating the docker image
|
||||
│ └── Anaconda3-2019.10-Linux-x86_64.sh # from <https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86\_64.sh>
|
||||
└── workdir # example folder used as a volume in the running container
|
||||
```
|
||||
|
||||
The key steps of the approach are:
|
||||
|
||||
* Place the large file(s) in the **installer/** folder. In this example, I have the large Anaconda installer file **Anaconda3-2019.10-Linux-x86_64.sh**. You won't find this file if you clone my [Git repository][8] because only you, as the container image creator, need this source file. The end users of the image don't. [Download the installer][9] to follow along with the example.
|
||||
* Create the **.dockerignore** file and have it ignore the **installer/** folder to avoid Docker copying all the large files into the build context.
|
||||
* In a terminal, **cd** into the **tutorial2_docker_tricks/** folder and execute the build script as **./build_docker_image.sh**.
|
||||
* In **build_docker_image.sh**, start the Python HTTP server to serve any files from the **installer/** folder: [code] cd installer
|
||||
python3 -m http.server --bind 10.0.2.15 8888 &
|
||||
cd ..
|
||||
```
|
||||
* If you're wondering about the strange internet protocol (IP) address, I'm working with a VirtualBox Linux VM, and **10.0.2.15** shows up as the address of the Ethernet adapter when I run **ifconfig**. This IP seems to be the convention used by VirtualBox. If your setup is different, you'll need to update this IP address to match your environment and then update **build_docker_image.sh** and **install_anaconda.dockerfile**. The server's port number is set to **8888** for this example. Note that the IP and port numbers could be passed in as build arguments, but I've hard-coded them for brevity.
|
||||
* Since the HTTP server is set to run in the background, stop the server near the end of the script with the **kill -9** command using an [elegant approach][10] I found: [code]`kill -9 `ps -ef | grep http.server | grep 8888 | awk '{print $2}'`
|
||||
```
|
||||
* Note that this same **kill -9** is also used earlier in the script (before starting the HTTP server). In general, when I iterate on any build script that I might deliberately interrupt, this ensures a clean start of the HTTP server each time.
|
||||
* In the [Dockerfile][11], there is a **RUN wget** instruction that downloads the Anaconda installer from the local HTTP server. It also deletes the installer file and cleans up after the installation. Most importantly, all these actions are performed within the same layer to keep the image size to a minimum: [code] # install Anaconda by downloading the installer via the local http server
|
||||
ARG ANACONDA
|
||||
RUN wget --no-proxy <http://10.0.2.15:8888/${ANACONDA}> -O ~/anaconda.sh \
|
||||
&& /bin/bash ~/anaconda.sh -b -p /opt/conda \
|
||||
&& rm ~/anaconda.sh \
|
||||
&& rm -fr /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*
|
||||
```
|
||||
* This file runs the wrapper script, **anaconda.sh**, and cleans up large files by removing them with **rm**.
|
||||
* After the build is complete, you should see an image **anaconda_ubuntu1804:v1**. (You can list the images with **docker image ls**.)
|
||||
* You can instantiate a container from this image using **./run_container.sh** at the terminal while in the folder **tutorial2_docker_tricks/**. You can verify that Anaconda is installed with: [code] $ ./run_container.sh
|
||||
$ python --version
|
||||
Python 3.7.5
|
||||
$ conda --version
|
||||
conda 4.8.0
|
||||
$ anaconda --version
|
||||
anaconda Command line client (version 1.7.2)
|
||||
```
|
||||
* You'll note that **run_container.sh** sets up a volume **workdir**. In this example repository, the folder **workdir/** is empty. This is a convention I use to set up a volume where I can have my Python and other scripts that are independent of the container image.
|
||||
|
||||
|
||||
|
||||
### Minimize container image size
|
||||
|
||||
Each **RUN** command is equivalent to executing a new shell, and each **RUN** command creates a layer. The naive approach of mimicking installation instructions with separate **RUN** commands may eventually break at one or more interdependent steps. If it happens to work, it will typically result in a larger image. Chaining multiple installation steps in one **RUN** command and including the **autoremove**, **autoclean**, and **rm** commands (as in the example below) is useful to minimize the size of each layer. Some of these steps may not be needed, depending on what's being installed. However, since these steps take an insignificant amount of time, I always throw them in for good measure at the end of **RUN** commands invoking **apt-get**:
|
||||
|
||||
|
||||
```
|
||||
RUN apt-get update \
|
||||
&& DEBIAN_FRONTEND=noninteractive \
|
||||
apt-get -y --quiet --no-install-recommends install \
|
||||
# list of packages being installed go here \
|
||||
&& apt-get -y autoremove \
|
||||
&& apt-get clean autoclean \
|
||||
&& rm -fr /var/lib/apt/lists/{apt,dpkg,cache,log} /tmp/* /var/tmp/*
|
||||
```
|
||||
|
||||
Also, ensure that you have a **.dockerignore** file in place to ignore items that don't need to be sent to the Docker build context (such as the Anaconda installer file in the earlier example).
|
||||
|
||||
### Organize the build tool I/O
|
||||
|
||||
For software build systems, the build inputs and outputs—all the scripts that configure and invoke the tools—should be outside the image and the eventually running container. The container itself should remain stateless so that different users will have identical results with it. I covered this extensively in my [previous article][4] but wanted to emphasize it because it's been a useful convention for my work. These inputs and outputs are best accessed by setting up container volumes.
|
||||
|
||||
I've had to use a container image that provides data in the form of source code and large pre-built binaries. As a software developer, I was expected to edit the code in the container. This was problematic, because containers are by default stateless: they don't save data within the container, because they're designed to be disposable. But I worked on it, and at the end of each day, I stopped the container and had to be careful not to remove it, because the state had to be maintained so I could continue work the next day. The disadvantage of this approach was that there would be a divergence of development state had there been more than one person working on the project. The value of having identical build systems across developers is somewhat lost with this approach.
|
||||
|
||||
### Generate output as non-root user
|
||||
|
||||
An important aspect of I/O concerns the ownership of the output files generated when running the tools in the container. By default, since Docker runs as **root**, the output files would be owned by **root**, which is unpleasant. You typically want to work as a non-root user. Changing the ownership after the build output is generated can be done with scripts, but it is an additional and unnecessary step. It's best to set the [**USER**][12] argument in the Dockerfile at the earliest point possible:
|
||||
|
||||
|
||||
```
|
||||
ARG USERNAME
|
||||
# other commands...
|
||||
USER ${USERNAME}
|
||||
```
|
||||
|
||||
The **USERNAME** can be passed in as a build argument (**\--build-arg**) when executing the **docker image build**. You can see an example of this in the example [Dockerfile][11] and corresponding [build script][13].
|
||||
|
||||
Some portions of the tools may also need to be installed as a non-root user. So the sequence of installations in the Dockerfile may need to be different from the way it's done if you are installing manually and directly under Linux.
|
||||
|
||||
### Non-interactive installation
|
||||
|
||||
Interactivity is the opposite of container automation. I've found the
|
||||
|
||||
|
||||
```
|
||||
`DEBIAN_FRONTEND=noninteractive apt-get -y --quiet --no-install-recommends`
|
||||
```
|
||||
|
||||
options for the **apt-get install** instruction (as in the example above) necessary to prevent the installer from opening dialog boxes. Note that these options should be used as part of the **RUN** instruction. The **DEBIAN_FRONTEND=noninteractive** should not be set as an environment variable (**ENV**) in the Dockerfile, as this [FAQ explains][14], as it will be inherited by the containers.
|
||||
|
||||
### Log your build and run output
|
||||
|
||||
Debugging why a build failed is a common task, and logs are a great way to do this. Save a TypeScript of everything that happened during the container image build or container run session using the **tee** utility in a Bash script. In other words, add **|& tee $BASH_SOURCE.log** to the end of the **docker image build** and the **docker image run** commands in your scripts. See the examples in the [image build][13] and [container run][15] scripts.
|
||||
|
||||
What this **tee**-ing technique does is generate a file with the same name as the Bash script but with a **.log** extension appended to it so that you know which script it originated from. Everything you see printed to the terminal when running the script will get logged to this file with a similar name.
|
||||
|
||||
This is especially valuable for users of your container images to report issues to you when something doesn't work. You can ask them to send you the log file to help diagnose the issue. Many tools generate so much output that it easily overwhelms the default size of the terminal's buffer. Relying only on the terminal's buffer capacity to copy-paste error messages may not be sufficient for diagnosing issues because earlier errors may have been lost.
|
||||
|
||||
I've found this to be useful, even in the container image-building scripts, especially when using the Python-based HTTP server discussed above. The server generates so many lines during a download that it typically overwhelms the terminal's buffer.
|
||||
|
||||
### Deal with proxies elegantly
|
||||
|
||||
In my work environment, proxies are required to reach the internet for downloading the resources in **RUN apt-get** and **RUN wget** commands. The proxies are typically inferred from the environment variables **http_proxy** or **https_proxy**. While **ENV** commands can be used to hard-code such proxy settings in the Dockerfile, there are multiple issues with using **ENV** for proxies directly.
|
||||
|
||||
If you are the only one who will ever build the container, then perhaps this will work. But the Dockerfile couldn't be used by someone else at a different location with a different proxy setting. Another issue is that the IT department could change the proxy at some point, resulting in a Dockerfile that won't work any longer. Furthermore, the Dockerfile is a precise document specifying a configuration-controlled system, and every change will be scrutinized by quality assurance.
|
||||
|
||||
One simple approach to avoid hard-coding the proxy is to pass your local proxy setting as a build argument in the **docker image build** command:
|
||||
|
||||
|
||||
```
|
||||
docker image build \
|
||||
--build-arg MY_PROXY=<http://my\_local\_proxy.proxy.com:xx>
|
||||
```
|
||||
|
||||
And then, in the Dockerfile, set the environment variables based on the build argument. In the example shown here, you can still set a default proxy value that can be overridden by the build argument above:
|
||||
|
||||
|
||||
```
|
||||
# set a default proxy
|
||||
ARG MY_PROXY=MY_PROXY=<http://my\_default\_proxy.proxy.com:nn/>
|
||||
ENV http_proxy=$MY_PROXY
|
||||
ENV https_proxy=$MY_PROXY
|
||||
```
|
||||
|
||||
### Summary
|
||||
|
||||
These techniques have helped me significantly reduce the time it takes to create container images and debug them when they go wrong. I continue to be on the lookout for additional best practices to add to my list. I hope you find the above techniques useful.
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/20/5/optimize-container-builds
|
||||
|
||||
作者：[Ravi Chandran][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://opensource.com/users/ravichandran
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/toolbox-learn-draw-container-yearbook.png?itok=xDbwz1pP (Toolbox drawing of a container)
|
||||
[2]: https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
|
||||
[3]: https://github.com/ravi-chandran/dockerize-tutorial
|
||||
[4]: https://opensource.com/article/20/4/how-containerize-build-system
|
||||
[5]: https://podman.io/getting-started/installation
|
||||
[6]: https://docs.docker.com/develop/develop-images/multistage-build/
|
||||
[7]: https://github.com/ravi-chandran/dockerize-tutorial/blob/master/tutorial2_docker_tricks/
|
||||
[8]: https://github.com/ravi-chandran/dockerize-tutorial/
|
||||
[9]: https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh
|
||||
[10]: https://stackoverflow.com/a/37214138
|
||||
[11]: https://github.com/ravi-chandran/dockerize-tutorial/blob/master/tutorial2_docker_tricks/install_anaconda.dockerfile
|
||||
[12]: https://docs.docker.com/engine/reference/builder/#user
|
||||
[13]: https://github.com/ravi-chandran/dockerize-tutorial/blob/master/tutorial2_docker_tricks/build_docker_image.sh
|
||||
[14]: https://docs.docker.com/engine/faq/
|
||||
[15]: https://github.com/ravi-chandran/dockerize-tutorial/blob/master/tutorial2_docker_tricks/run_container.sh
|
||||
@ -1,273 +0,0 @@
|
||||
Systemd服务：响应变更
|
||||
======
|
||||
|
||||
|
||||
|
||||
[我有一个这样的电脑棒][1]（图1），并将其用作通用服务器。它很小且安静，由于它是基于x86架构，因此我为我的打印机安装驱动没有任何问题，而且这就是它大多数时候干的事：与客厅的共享打印机和扫描仪通信。
|
||||
|
||||
![ComputeStick][3]
|
||||
|
||||
一个英特尔电脑棒。欧元硬币大小。
|
||||
|
||||
[Used with permission][4]
|
||||
|
||||
大多数时候，它都闲着，尤其是当我们外出时，因此我认为用它作监视系统是个好主意。该设备没有自带的摄像头，也不需要一直监视。我也不想手动启动图像捕获，因为这样就意味着在出门前必须通过SSH登录，并在shell中编写命令来启动该进程。
|
||||
|
||||
因此，我以为应该这么做：抓住USB摄像头，然后只需插入它即可自动启动监视系统。如果Stick重启后发现连接了摄像头也启动监视系统就更加分了。
|
||||
|
||||
在先前的文章中，我们看到[systemd服务可以手动启动或停止][5]或[在满足某些条件时][6]。这些条件不限于操作系统在启动或关机时序中达到某种状态，还可以在您插入新硬件或文件系统发生变化时进行。您可以通过将Udev规则与systemd服务结合起来实现。
|
||||
|
||||
### 有Udev(支持)的热插拔
|
||||
|
||||
Udev规则位于 _/etc/udev/rules_ 目录中，通常是由导致一个 _动作(action)_ 的 _条件(conditions)_ 和 _赋值(assignments)_ 的单行语句来描述。
|
||||
|
||||
有点神秘。让我们再试一次：
|
||||
|
||||
通常，在Udev规则中，您告诉systemd当连接一个设备时需要查看什么信息。例如，您可能想检查刚插入的设备的品牌和型号是否与您让Udev等待的设备的品牌和型号相对应。这些就是前面提到的条件。
|
||||
|
||||
然后，您可能想要更改一些内容，以便以后可以轻松使用该设备。例如，更改设备的读写权限：如果插入USB打印机，您将希望用户能够从打印机读取信息（用户的打印应用程序需要知道其模型，制造商，以及是否准备好接受打印作业）并向其写入内容，即发送要打印的内容。更改设备的读写权限是通过您之前阅读的 _赋值(assignments)_ 之一完成的。
|
||||
|
||||
最后，您可能希望系统在满足上述条件时执行某些动作，例如在插入某个外接硬盘时启动备份程序以复制重要文件。这就是上面提到的 _动作(action)_ 的例子。
|
||||
|
||||
了解这些之后, 来看看以下几点:
|
||||
|
||||
```
|
||||
ACTION=="add", SUBSYSTEM=="video4linux", ATTRS{idVendor}=="03f0", ATTRS{idProduct}=="e207",
|
||||
SYMLINK+="mywebcam", TAG+="systemd", MODE="0666", ENV{SYSTEMD_WANTS}="webcam.service"
|
||||
```
|
||||
|
||||
规则的第一部分,
|
||||
|
||||
```
|
||||
ACTION=="add", SUBSYSTEM=="video4linux", ATTRS{idVendor}=="03f0",
|
||||
ATTRS{idProduct}=="e207" [etc... ]
|
||||
```
|
||||
|
||||
表明了执行您想让系统执行的其他动作之前设备必须满足的条件。设备必须被添加到（`ACTION=="add"`）机器上，并且必须添加到 `video4linux` 子系统中。为了确保仅在插入正确的设备时才应用该规则，您必须确保Udev正确识别设备的制造商（`ATTRS{idVendor}=="03f0"`）和型号（`ATTRS{idProduct}=="e207"`）。
|
||||
|
||||
在本例中，我们讨论的是这个设备（图2）：
|
||||
|
||||
![webcam][8]
|
||||
|
||||
这个试验使用的是HP的摄像头。
|
||||
|
||||
[Used with permission][4]
|
||||
|
||||
注意怎样用 `==` 来表示这是一个逻辑操作。您应该像这样阅读上面的简要规则：
|
||||
|
||||
```
|
||||
如果添加了一个设备并且该设备由video4linux子系统控制
|
||||
而且该设备的制造商是03f0，型号是e207，那么...
|
||||
```
|
||||
|
||||
但是，您从哪里获取的这些信息？ 您在哪里找到触发事件的动作，制造商，模型等等？您可能必须使用多个来源。您可以通过将摄像头插入机器并运行 `lsusb` 来获得 `IdVendor` 和 `idProduct` ：
|
||||
|
||||
```
|
||||
lsusb
|
||||
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
|
||||
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
|
||||
Bus 003 Device 003: ID 03f0:e207 Hewlett-Packard
|
||||
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
Bus 001 Device 003: ID 04f2:b1bb Chicony Electronics Co., Ltd
|
||||
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
|
||||
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
|
||||
```
|
||||
|
||||
我用的摄像头是HP的，您在上面的列表中只能看到一个HP设备。 `ID` 告诉了制造商和型号，它们以冒号（ `:` ）分隔。如果您有同一制造商的多个设备，不确定哪个是哪个设备，请拔下摄像头，再次运行 `lsusb` , 看看少了什么。
|
||||
|
||||
或者...
|
||||
|
||||
拔下摄像头，等待几秒钟，运行命令 `udevadmin monitor --environment` ，然后重新插入摄像头。当您使用的是HP摄像头时，您将看到：
|
||||
|
||||
```
|
||||
udevadmin monitor --environment
|
||||
UDEV [35776.495221] add /devices/pci0000:00/0000:00:1c.3/0000:04:00.0
|
||||
/usb3/3-1/3-1:1.0/input/input21/event11 (input)
|
||||
.MM_USBIFNUM=00
|
||||
ACTION=add
|
||||
BACKSPACE=guess
|
||||
DEVLINKS=/dev/input/by-path/pci-0000:04:00.0-usb-0:1:1.0-event
|
||||
/dev/input/by-id/usb-Hewlett_Packard_HP_Webcam_HD_2300-event-if00
|
||||
DEVNAME=/dev/input/event11
|
||||
DEVPATH=/devices/pci0000:00/0000:00:1c.3/0000:04:00.0/
|
||||
usb3/3-1/3-1:1.0/input/input21/event11
|
||||
ID_BUS=usb
|
||||
ID_INPUT=1
|
||||
ID_INPUT_KEY=1
|
||||
ID_MODEL=HP_Webcam_HD_2300
|
||||
ID_MODEL_ENC=HP\x20Webcam\x20HD\x202300
|
||||
ID_MODEL_ID=e207
|
||||
ID_PATH=pci-0000:04:00.0-usb-0:1:1.0
|
||||
ID_PATH_TAG=pci-0000_04_00_0-usb-0_1_1_0
|
||||
ID_REVISION=1020
|
||||
ID_SERIAL=Hewlett_Packard_HP_Webcam_HD_2300
|
||||
ID_TYPE=video
|
||||
ID_USB_DRIVER=uvcvideo
|
||||
ID_USB_INTERFACES=:0e0100:0e0200:010100:010200:030000:
|
||||
ID_USB_INTERFACE_NUM=00
|
||||
ID_VENDOR=Hewlett_Packard
|
||||
ID_VENDOR_ENC=Hewlett\x20Packard
|
||||
ID_VENDOR_ID=03f0
|
||||
LIBINPUT_DEVICE_GROUP=3/3f0/e207:usb-0000:04:00.0-1/button
|
||||
MAJOR=13
|
||||
MINOR=75
|
||||
SEQNUM=3162
|
||||
SUBSYSTEM=input
|
||||
USEC_INITIALIZED=35776495065
|
||||
XKBLAYOUT=es
|
||||
XKBMODEL=pc105
|
||||
XKBOPTIONS=
|
||||
XKBVARIANT=
|
||||
```
|
||||
|
||||
可能看起来有很多信息要处理，但是，看一下这个：列表前面的 `ACTION` 字段, 它告诉您刚刚发生了什么事件，即一个设备被添加到系统中。您还可以在几行中看到设备名称的拼写，因此可以非常确定它就是您要找的设备。输出里还显示了制造商的ID（`ID_VENDOR_ID = 03f0`）和型号（`ID_VENDOR_ID = 03f0`）。
|
||||
|
||||
这为您提供了规则条件部分需要的四个值中的三个。您可能也会想到它还给了您第四个，因为还有一行这样写道：
|
||||
|
||||
```
|
||||
SUBSYSTEM=input
|
||||
```
|
||||
|
||||
小心！尽管USB摄像头确实是提供输入的设备（键盘和鼠标也是），但它也属于 _usb_ 子系统和其他几个子系统。这意味着您的摄像头被添加到了多个子系统，并且看起来像多个设备。如果您选择了错误的子系统，那么您的规则可能无法按您期望的那样工作，或者根本无法工作。
|
||||
|
||||
因此，您必须检查的第三件事是摄像头添加到的所有子系统，并选择正确的那个。为此，请再次拔下摄像头，然后运行：
|
||||
|
||||
```
|
||||
ls /dev/video*
|
||||
```
|
||||
|
||||
这将向您显示连接到本机的所有视频设备。如果您使用的是笔记本，大多数笔记本都带有内置摄像头，它可能会显示为 `/dev/video0` 。重新插入摄像头，然后再次运行 `ls /dev/video*` 。
|
||||
|
||||
现在，您应该看到多一个视频设备（可能是`/dev/video1`）。
|
||||
|
||||
现在，您可以通过运行`udevadm info -a /dev/video1`找出它所属的所有子系统：
|
||||
|
||||
```
|
||||
udevadm info -a /dev/video1
|
||||
|
||||
Udevadm info starts with the device specified by the devpath and then
|
||||
walks up the chain of parent devices. It prints for every device
|
||||
found, all possible attributes in the udev rules key format.
|
||||
A rule to match, can be composed by the attributes of the device
|
||||
and the attributes from one single parent device.
|
||||
|
||||
looking at device '/devices/pci0000:00/0000:00:1c.3/0000:04:00.0
|
||||
/usb3/3-1/3-1:1.0/video4linux/video1':
|
||||
KERNEL=="video1"
|
||||
SUBSYSTEM=="video4linux"
|
||||
DRIVER==""
|
||||
ATTR{dev_debug}=="0"
|
||||
ATTR{index}=="0"
|
||||
ATTR{name}=="HP Webcam HD 2300: HP Webcam HD"
|
||||
|
||||
[etc...]
|
||||
```
|
||||
|
||||
输出持续了一段时间，但是您感兴趣的只是开始：`SUBSYSTEM =="video4linux"`。您可以将这行按文本复制并粘贴到规则中。输出的其余部分（为简便起见未显示）为您提供了更多的块，例如制造商和模型ID，您也可以以同样的格式复制并粘贴到规则中。
|
||||
|
||||
现在，您有了识别设备的方式并明确了什么事件应该触发该动作，该对设备进行修改了。
|
||||
|
||||
规则的下一部分，`SYMLINK+="mywebcam", TAG+="systemd", MODE="0666"`告诉Udev做三件事：首先，您要创建设备的符号链接（例如 _/dev/video1_ 到 _/dev/mywebcam_ 。这是因为您无法预测系统默认情况下会把那个设备叫什么。当您拥有内置摄像头并热插拔一个新的时，内置摄像头通常为 _/dev/video0_ ，而外部摄像头通常为 _/dev/video1_ 。但是，如果您在插入外部USB摄像头的情况下重启计算机，则可能会相反，内部摄像头可能会变成 _/dev/video1_ ，而外部摄像头会变成 _/dev/video0_ 。这想告诉您的是，尽管您的图像捕获脚本（稍后将看到）总是需要指向外部摄像头设备，但是您不能依赖它是 _/dev/video0_ 或 _/dev/video1_ 。为了解决这个问题，您告诉Udev创建一个符号链接，该链接在设备被添加到 _video4linux_ 子系统的那一刻起就不会再变，您将使您的脚本指向该链接。
|
||||
|
||||
您要做的第二件事是将 `"systemd"` 添加到与此规则关联的Udev标记列表中。这告诉Udev，该规则触发的动作将由systemd管理，即它将是某种systemd服务。
|
||||
|
||||
注意在两种情况下该如何使用 `+=` 运算符。这会将值添加到列表中，这意味着您可以向 `SYMLINK` 和 `TAG` 添加多个值。
|
||||
|
||||
另一方面，`MODE` 值只能包含一个值（因此，您可以使用简单的 `=` 赋值运算符）。`MODE` 的作用是告诉Udev谁可以读或写该设备。如果您熟悉 `chmod`（您读到此文, 应该会熟悉），您就也会熟悉[如何用数字表示权限][9]。这就是它的含义： `0666` 的含义是 “ _向所有人授予对设备的读写权限_ ”。
|
||||
|
||||
最后， `ENV{SYSTEMD_WANTS}="webcam.service"` 告诉Udev要运行什么systemd服务。
|
||||
|
||||
将此规则保存到 _/etc/udev/rules.d_ 目录名为 _90-webcam.rules_ (或类似的名称)的文件中，您可以通过重启机器或运行以下命令来加载它：
|
||||
|
||||
```
|
||||
sudo udevadm control --reload-rules && udevadm trigger
|
||||
```
|
||||
|
||||
## 最后描述服务
|
||||
|
||||
Udev规则触发的服务非常简单：
|
||||
```
|
||||
# webcam.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/home/[user name]/bin/checkimage.sh
|
||||
```
|
||||
|
||||
基本上，它只是运行存储在您个人 _bin/_ 中的 _checkimage.sh_ 脚本并将其放到后台。 [这是您在先前的部分中看过的内容][5]。 它看起来似乎很小，但那只是因为它是被Udev规则调用的，您刚刚创建了一种特殊的systemd单元，称为 _device_ 单元。 恭喜。
|
||||
|
||||
至于 _webcam.service_ 调用的 _checkimage.sh_ 脚本，有几种方法从摄像头抓取图像并将其与前一个图像进行比较以检查变化（这是 _checkimage.sh_ 所做的事），但这是我的方法：
|
||||
|
||||
```
|
||||
#!/bin/bash
|
||||
# This is the checkimage.sh script
|
||||
|
||||
mplayer -vo png -frames 1 tv:// -tv driver=v4l2:width=640:height=480:device=
|
||||
/dev/mywebcam &>/dev/null
|
||||
mv 00000001.png /home/[user name]/monitor/monitor.png
|
||||
|
||||
while true
|
||||
do
|
||||
mplayer -vo png -frames 1 tv:// -tv driver=v4l2:width=640:height=480:device=/dev/mywebcam &>/dev/null
|
||||
mv 00000001.png /home/[user name]/monitor/temp.png
|
||||
|
||||
imagediff=`compare -metric mae /home/[user name]/monitor/monitor.png /home/[user name]
|
||||
/monitor/temp.png /home/[user name]/monitor/diff.png 2>&1 > /dev/null | cut -f 1 -d " "`
|
||||
if [ `echo "$imagediff > 700.0" | bc` -eq 1 ]
|
||||
then
|
||||
mv /home/[user name]/monitor/temp.png /home/[user name]/monitor/monitor.png
|
||||
fi
|
||||
|
||||
sleep 0.5
|
||||
done
|
||||
```
|
||||
|
||||
首先使用[MPlayer][10]从摄像头抓取一帧（_00000001.png_）。注意，我们怎样将 `mplayer` 指向Udev规则中创建的 `mywebcam` 符号链接，而不是指向 `video0` 或 `video1` 。然后，将图像传输到主目录中的 _monitor/_ 目录。然后执行一个无限循环，一次又一次地执行相同的操作，但还使用了[Image Magick的_compare_工具][11]来查看最后捕获的图像与 _monitor/_ 目录中已有的图像之间是否存在差异。
|
||||
|
||||
如果图像不同，则表示摄像头的镜框里某些东西动了。该脚本将新图像覆盖原始图像，并继续比较以等待更多变动。
|
||||
|
||||
### 插线
|
||||
|
||||
所有东西准备好后，当您插入摄像头后，您的Udev规则将被触发并启动 _webcam.service_ 。 _webcam.service_ 将在后台执行 _checkimage.sh_ ，而 _checkimage.sh_ 将开始每半秒拍一次照。您会感觉到，因为摄像头的LED在每次拍照时将开始闪。
|
||||
|
||||
与往常一样，如果出现问题，请运行
|
||||
|
||||
```
|
||||
systemctl status webcam.service
|
||||
```
|
||||
|
||||
检查您的服务和脚本正在做什么。
|
||||
|
||||
### 接下来
|
||||
|
||||
您可能想知道：为什么要覆盖原始图像？ 当然，系统检测到任何动静，您都想知道发生了什么，对吗？您是对的，但是如您在下一部分中将看到的那样，将它们保持原样，并使用另一种类型的systemd单元处理图像将更好，更清晰和更简单。
|
||||
|
||||
请稍等。
|
||||
|
||||
通过Linux基金会和edX的免费["Linux简介"][12]课程了解有关Linux的更多信息。
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://www.linux.com/blog/intro-to-linux/2018/6/systemd-services-reacting-change
|
||||
|
||||
作者：[Paul Brown][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[messon007](https://github.com/messon007)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://www.linux.com/users/bro66
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://www.intel.com/content/www/us/en/products/boards-kits/compute-stick/stk1a32sc.html
|
||||
[2]: https://www.linux.com/files/images/fig01png
|
||||
[3]: https://www.linux.com/sites/lcom/files/styles/floated_images/public/fig01.png?itok=cfEHN5f1 (ComputeStick)
|
||||
[4]: https://www.linux.com/licenses/category/used-permission
|
||||
[5]: https://www.linux.com/blog/learn/intro-to-linux/2018/5/writing-systemd-services-fun-and-profit
|
||||
[6]: https://www.linux.com/blog/learn/2018/5/systemd-services-beyond-starting-and-stopping
|
||||
[7]: https://www.linux.com/files/images/fig02png
|
||||
[8]: https://www.linux.com/sites/lcom/files/styles/floated_images/public/fig02.png?itok=esFv4BdM (webcam)
|
||||
[9]: https://chmod-calculator.com/
|
||||
[10]: https://mplayerhq.hu/design7/news.html
|
||||
[11]: https://www.imagemagick.org/script/compare.php
|
||||
[12]: https://training.linuxfoundation.org/linux-courses/system-administration-training/introduction-to-linux
|
||||
@ -0,0 +1,112 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: (geekpi)
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Fixing “Unable to parse package file /var/lib/apt/lists” Error in Ubuntu and Other Linux Distributions)
|
||||
[#]: via: (https://itsfoss.com/unable-to-parse-package-file/)
|
||||
[#]: author: (Abhishek Prakash https://itsfoss.com/author/abhishek/)
|
||||
|
||||
修复 Ubuntu 和其他 Linux 发行版中的 “Unable to parse package file /var/lib/apt/lists” 错误
|
||||
======
|
||||
|
||||
过去，我已经讨论了许多 [Ubuntu 更新错误][1]。如果你[使用命令行更新 Ubuntu][2]，那可能会遇到一些“错误”。
|
||||
|
||||
其中一些“错误”基本上是内置功能，可防止对系统进行不必要的更改。在本教程中，我不会涉及那些细节。
|
||||
|
||||
在本文中，我将向你展示如何解决在更新系统或安装新软件时可能遇到的以下错误：
|
||||
|
||||
**Reading package lists… Error!
|
||||
E: Unable to parse package file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease
|
||||
E: The package lists or status file could not be parsed or opened.**
|
||||
|
||||
在 Debian 中可能会遇到类似的错误：
|
||||
|
||||
**E: Unable to parse package file /var/lib/apt/extended_states (1)**
|
||||
|
||||
即使遇到 “**The package cache file is corrupted**” 也完全不必惊慌。这真的很容易“修复”。
|
||||
|
||||
### 在基于 Ubuntu 和 Debian 的 Linux 发行版中处理 “Unable to parse package file” 错误
|
||||
|
||||
![][3]
|
||||
|
||||
以下是你需要做的。仔细查看 [Ubuntu][4] 报错文件的名称和路径。
|
||||
|
||||
Reading package lists… Error!
|
||||
**E: Unable to parse package file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease**
|
||||
E: The package lists or status file could not be parsed or opened.
|
||||
|
||||
例如，上面的错误是在报 /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease 文件错误。
|
||||
|
||||
这让你想到这个文件不正确。现在，你需要做的就是删除该文件并重新生成缓存。
|
||||
|
||||
```
|
||||
sudo rm <file_that_is_not_parsed>
|
||||
```
|
||||
|
||||
因此，这里我可以使用以下命令：**sudo rm /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic_InRelease**，然后使用 sudo apt update 命令重建缓存。
|
||||
|
||||
#### 给初学者的分步指导
|
||||
|
||||
如果你熟悉 Linux 命令，那么可能知道如何使用绝对路径删除文件。对于新手用户，让我指导你安全删除文件。
|
||||
|
||||
首先，你应该进入文件目录：
|
||||
|
||||
```
|
||||
cd /var/lib/apt/lists/
|
||||
```
|
||||
|
||||
现在删除无法解析的文件：
|
||||
|
||||
```
|
||||
sudo rm archive.ubuntu.com_ubuntu_dists_bionic_InRelease
|
||||
```
|
||||
|
||||
现在，如果你再次运行更新，将重新生成 apt 缓存。
|
||||
|
||||
```
|
||||
sudo apt update
|
||||
```
|
||||
|
||||
#### 有很多文件无法解析？
|
||||
|
||||
如果你在更新系统时有一个或两个文件无法解析，那么问题不大。但是，如果系统报错有十个或二十个此类文件，那么一一删除它们就太累了。
|
||||
|
||||
在这种情况下，你可以执行以下操作来删除整个缓存，然后再次生成它：
|
||||
|
||||
```
|
||||
sudo rm -r /var/lib/apt/lists/*
|
||||
sudo apt update
|
||||
```
|
||||
|
||||
#### 解释这为何能解决问题
|
||||
|
||||
/var/lib/apt 是与 apt 软件包管理器相关的文件和数据的存储目录。/var/lib/apt/lists 是用于保存系统 source.list 中指定的每个软件包资源信息的目录。
|
||||
|
||||
简单点来说，/var/lib/apt/lists 保存软件包信息缓存。当你要安装或更新程序时，系统会在此目录中检查该软件包中的信息。如果找到了该包的详细信息，那么它将进入远程仓库并实际下载程序或其更新。
|
||||
|
||||
当你运行 “sudo apt update” 时，它将构建缓存。这就是为什么即使删除 /var/lib/apt/lists 目录中的所有内容，运行更新也会建立新的缓存的原因。
|
||||
|
||||
这就是处理文件无法解析问题的方式。你的系统报某个软件包或仓库信息以某种方式损坏（下载失败或手动更改 sources.list）。删除该文件（或所有文件）并重建缓存即可解决此问题。
|
||||
|
||||
#### 仍然有错误？
|
||||
|
||||
这应该能解决你的问题。但是，如果问题仍然存在，或者你还有其他相关问题，请在评论栏告诉我，我将尽力帮助你。
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://itsfoss.com/unable-to-parse-package-file/
|
||||
|
||||
作者：[Abhishek Prakash][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[geekpi](https://github.com/geekpi)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://itsfoss.com/author/abhishek/
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://itsfoss.com/ubuntu-update-error/
|
||||
[2]: https://itsfoss.com/update-ubuntu/
|
||||
[3]: https://i1.wp.com/itsfoss.com/wp-content/uploads/2020/05/Unable-to-parse-package-file.png?ssl=1
|
||||
[4]: https://ubuntu.com/
|
||||
@ -0,0 +1,164 @@
|
||||
[#]: collector: (lujun9972)
|
||||
[#]: translator: (lxbwolf)
|
||||
[#]: reviewer: ( )
|
||||
[#]: publisher: ( )
|
||||
[#]: url: ( )
|
||||
[#]: subject: (Ensmallening Go binaries by prohibiting comparisons)
|
||||
[#]: via: (https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons)
|
||||
[#]: author: (Dave Cheney https://dave.cheney.net/author/davecheney)
|
||||
|
||||
通过禁止比较让 Go 二进制文件变小
|
||||
======
|
||||
|
||||
大家常规的认知是，Go 程序中声明的类型越多，生成的二进制文件就越大。这个符合直觉，毕竟如果你写的代码不去操作定义的类型，那么定义一堆类型就没有意义了。然而，链接器的部分工作就是检测程序没有引用的函数，比如仅仅有某个功能的子功能使用的库中的某些函数，然后把他们从最后的编译产出中删除。常言道，“类型越多，二进制文件越大“，对于多数 Go 程序还是正确的。
|
||||
|
||||
本文中我会深入讲解在 Go 程序的上下文中相等的意义，以及为什么[像这样][1]的修改会对 Go 程序的大小有重大的影响。
|
||||
|
||||
### 定义两个值相等
|
||||
|
||||
Go 的语法定义了赋值和相等的概念。赋值是把一个值赋给一个标识符的行为。并不是所有声明的标识符都可以被赋值，如常量和函数就不可以。相等是通过检查标识符的内容是否相等来比较两个标识符的行为。
|
||||
|
||||
作为强类型语言，“相同”的概念从根源上被植入标识符的类型中。两个标识符只有是相同类型的前提下，才有可能相同。除此之外，值的类型定义了如何比较该类型的两个值。
|
||||
|
||||
例如，整型是用算数方法进行比较的。对于指针类型，是否相等是指他们指向的地址是否相同。map 和 channel 等引用类型，跟指针类似，如果它们有相同的地址，那么就认为它们是相同的。
|
||||
|
||||
上面都是按位比较相等的例子，即值占用的位模式内存是相同的，那么这些值就相等。这个就是 memcmp，全称为 memory comparison，相等是通过比较两个内存区域的内容来定义的。
|
||||
|
||||
记住这个思路，我会很快回来的。
|
||||
|
||||
### 结构体相等
|
||||
|
||||
除了整型、浮点型和指针等标量类型，还有复合类型；结构体。所有的结构体以程序中的顺序被排列在内存中。因此下面这个声明
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a, b, c, d int64
|
||||
}
|
||||
```
|
||||
|
||||
会占用 32 字节的内存空间；`a` 占用 8 个字节，`b` 占用 8 个字节，以此类推。Go 的规则说如果结构体所有的字段都是可以比较的，那么结构体的值就是可以比较的。因此如果两个结构体所有的字段都相等，那么它们就相等。
|
||||
|
||||
```
|
||||
a := S{1, 2, 3, 4}
|
||||
b := S{1, 2, 3, 4}
|
||||
fmt.Println(a == b) // prints true
|
||||
```
|
||||
|
||||
编译器在底层使用 memcmp 来比较 `a` 的 32 个字节和 `b` 的 32 个字节。
|
||||
|
||||
### 填充和对齐
|
||||
|
||||
然而，在下面的场景下过分简单化的按位比较的策略会返回错误的结果：
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a byte
|
||||
b uint64
|
||||
c int16
|
||||
d uint32
|
||||
}
|
||||
|
||||
func main()
|
||||
a := S{1, 2, 3, 4}
|
||||
b := S{1, 2, 3, 4}
|
||||
fmt.Println(a == b) // prints true
|
||||
}
|
||||
```
|
||||
|
||||
编译代码后，这个比较表达式的结果还是 true，但是编译器在底层并不能仅依赖比较 `a` 和 `b` 的位模式，因为结构体有*填充*。
|
||||
|
||||
Go 要求结构体的所有字段都对齐。2 字节的值必须从偶数地址开始，4 字节的值必须从 4 的倍数地址开始，以此类推[1][2]。编译器根据字段的类型和底层平台加入了填充来确保字段都*对齐*。在填充之后，编译器实际上看到的是[2][3]：
|
||||
|
||||
```
|
||||
type S struct {
|
||||
a byte
|
||||
_ [7]byte // padding
|
||||
b uint64
|
||||
c int16
|
||||
_ [2]int16 // padding
|
||||
d uint32
|
||||
}
|
||||
```
|
||||
|
||||
填充的存在保证了字段正确对齐，而填充确实占用了内存空间，但是填充字节的内容是未知的。你可能会认为在 Go 中 填充字节都是 0，但实际上并不是 — 填充字节的内容是未定义的。由于它们并不是被定义为某个确定的值，因此按位比较会因为分布在 `s` 的 24 字节中的 9 个填充字节不一样而返回错误结果。
|
||||
|
||||
Go 通过生成相等函数来解决这个问题。在这个例子中，`s` 的相等函数只比较函数中的字段略过填充部分，这样就能正确比较类型 `s` 的两个值。
|
||||
|
||||
### 类型算法
|
||||
|
||||
嚄，需要做很多准备工作才能解释原因，对于 Go 程序中定义的每种类型，编译器都会生成几个支持它的函数，编译器内部把它们识别为类型的算法。如果类型是一个 map 的 key，那么除相等函数外，编译器还会生成一个哈希函数。为了维持稳定，哈希函数在计算结果时也会像相等函数一样考虑诸如填充等因素。
|
||||
|
||||
凭直觉判断编译器什么时候生成这些函数实际上很难，有时并不明显，（因为）这超出了你的预期，而且链接器也很难消除没有被使用的类型，因为反射往往导致链接器在裁剪类型时变得更保守。
|
||||
|
||||
### 通过禁止比较来减小二进制文件的大小
|
||||
|
||||
现在，我们能解释 Brad 的修改了。向类型添加一个不可比较的字段[3][4]，结构体也随之变成不可比较的，从而强制编译器不再生成相等和哈希算法、规避了链接器对那些类型的消除、在实际应用中减小了生成的二进制文件的大小。作为这项技术的一个例子，下面的程序：
|
||||
|
||||
```
|
||||
package main
|
||||
|
||||
import "fmt"
|
||||
|
||||
func main() {
|
||||
type t struct {
|
||||
// _ [0][]byte uncomment to prevent comparison
|
||||
a byte
|
||||
b uint16
|
||||
c int32
|
||||
d uint64
|
||||
}
|
||||
var a t
|
||||
fmt.Println(a)
|
||||
}
|
||||
```
|
||||
|
||||
用 Go 1.14.2（darwin/amd64）编译，大小从 2174088 降到了 2174056，节省了 32 字节。单独看节省的这 32 字节似乎微不足道，但是考虑到你的程序中每个类型及其传递闭包都会生成相等和哈希函数，还有他们的依赖，这些函数的大小随类型大小和复杂度的不同而不同，禁止它们会大大减小最终的二进制文件的大小，效果比之前使用 `-ldflags="-s -w"` 还要好。
|
||||
|
||||
最后总结一下，如果你不想把类型定义为可比较的，像这样的入侵可以在源码层级强制实现，而且会使生成的二进制文件变小。
|
||||
|
||||
* * *
|
||||
|
||||
附录：在 Brad 的推动下，[Cherry Zhang][5] 和 [Keith Randall][6] 已经在 Go 1.15 做了大量的改进来修复最恶名昭彰的消除无用相等和哈希函数失败（虽然我猜想这也是为了避免这类 CL 的扩散）。
|
||||
|
||||
1. 在 32 位平台上 `int64` 和 `unit64` 的值可能不是按 8 字节对齐的，因为平台原生的是以 4 字节对齐的。查看 [issue 599][7] 了解内部详细信息[][8]。
|
||||
2. 32 位平台会在 `a` 和 `b` 的声明中填充 `_ [3]byte`。查看前面的内容[][9]。
|
||||
3. Brad 使用的是`[0]func()`，但是所有能限制和禁止比较的类型都可以。添加了一个有 0 个元素的数组的声明后，结构体的大小和对齐不会受影响。[][10]
|
||||
|
||||
|
||||
|
||||
#### 相关文章：
|
||||
|
||||
1. [Go 运行时如何高效地实现 map（不使用泛型）][11]
|
||||
2. [空结构体][12]
|
||||
3. [填充很难][13]
|
||||
4. [Go 中有类型的 nil 2][14]
|
||||
|
||||
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://dave.cheney.net/2020/05/09/ensmallening-go-binaries-by-prohibiting-comparisons
|
||||
|
||||
作者：[Dave Cheney][a]
|
||||
选题：[lujun9972][b]
|
||||
译者：[lxbwolf](https://github.com/lxbwolf)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]: https://dave.cheney.net/author/davecheney
|
||||
[b]: https://github.com/lujun9972
|
||||
[1]: https://github.com/golang/net/commit/e0ff5e5a1de5b859e2d48a2830d7933b3ab5b75f
|
||||
[2]: tmp.uBLyaVR1Hm#easy-footnote-bottom-1-4116 (On 32bit platforms <code>int64</code> and <code>uint64</code> values may not be 8 byte aligned as the natural alignment of the platform is 4 bytes. See <a href="https://github.com/golang/go/issues/599">issue 599</a> for the gory details.)
|
||||
[3]: tmp.uBLyaVR1Hm#easy-footnote-bottom-2-4116 (32 bit platforms would see <code>_ [3]byte</code> padding between the declaration of <code>a</code> and <code>b</code>. See previous.)
|
||||
[4]: tmp.uBLyaVR1Hm#easy-footnote-bottom-3-4116 (Brad used <code>[0]func()</code>, but any type that the spec limits or prohibits comparisons on will do. By declaring the array has zero elements the type has no impact on the size or alignment of the struct.)
|
||||
[5]: https://go-review.googlesource.com/c/go/+/231397
|
||||
[6]: https://go-review.googlesource.com/c/go/+/191198
|
||||
[7]: https://github.com/golang/go/issues/599
|
||||
[8]: tmp.uBLyaVR1Hm#easy-footnote-1-4116
|
||||
[9]: tmp.uBLyaVR1Hm#easy-footnote-2-4116
|
||||
[10]: tmp.uBLyaVR1Hm#easy-footnote-3-4116
|
||||
[11]: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implements-maps-efficiently-without-generics (How the Go runtime implements maps efficiently (without generics))
|
||||
[12]: https://dave.cheney.net/2014/03/25/the-empty-struct (The empty struct)
|
||||
[13]: https://dave.cheney.net/2015/10/09/padding-is-hard (Padding is hard)
|
||||
[14]: https://dave.cheney.net/2017/08/09/typed-nils-in-go-2 (Typed nils in Go 2)
|
||||
Loading…
Reference in New Issue
Block a user