document/TranslateProject
2
0
Fork 0
mirror of https://github.com/LCTT/TranslateProject.git synced 2024-12-29 21:41:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'LCTT/master'

Browse Source
This commit is contained in:
Xingyu Wang 2020-08-09 20:05:50 +08:00
commit 5f0823c5cb
10 changed files with 1714 additions and 664 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

627
published/20200318 Share data between C and Python with this messaging library.md Normal file
View File

@ -0,0 +1,627 @@
[#]: collector: (lujun9972)
[#]: translator: (silentdawn-zz)
[#]: reviewer: (wxy)
[#]: publisher: (wxy)
[#]: url: (https://linux.cn/article-12499-1.html)
[#]: subject: (Share data between C and Python with this messaging library)
[#]: via: (https://opensource.com/article/20/3/zeromq-c-python)
[#]: author: (Cristiano L. Fontana https://opensource.com/users/cristianofontana)
使用 ZeroMQ 消息库在 C 和 Python 间共享数据
======
> ZeroMQ 是一个快速灵活的消息库，用于数据收集和不同编程语言间的数据共享。
![](https://img.linux.net.cn/data/attachment/album/202008/08/202106uale11l1qf11slzw.jpg)
作为软件工程师，我有多次在要求完成指定任务时感到浑身一冷的经历。其中有一次，我必须在一些新的硬件基础设施和云基础设施之间写一个接口，这些硬件需要 C 语言，而云基础设施主要是用 Python。
实现的方式之一是 [用 C 写扩展模块][2]，Python 支持 C 扩展的调用。快速浏览文档后发现，这需要编写大量的 C 代码。这样做的话，在有些情况下效果还不错，但不是我喜欢的方式。另一种方式就是将两个任务放在不同的进程中，并使用 [ZeroMQ 消息库][3] 在两者之间交换消息。
在发现 ZeroMQ 之前，遇到这种类型的情况时，我选择了编写扩展的方式。这种方式不算太差，但非常费时费力。如今，为了避免那些问题，我将一个系统细分为独立的进程，通过 [通信套接字][4] 发送消息来交换信息。这样，不同的编程语言可以共存，每个进程也变简单了，同时也容易调试。
ZeroMQ 提供了一个更简单的过程：
1. 编写一小段 C 代码，从硬件读取数据，然后把发现的东西作为消息发送出去。
2. 使用 Python 编写接口，实现新旧基础设施之间的对接。
[Pieter Hintjens][5] 是 ZeroMQ 项目发起者之一，他是个拥有 [有趣视角和作品][6] 的非凡人物。
### 准备
本教程中，需要：
* 一个 C 编译器（例如 [GCC][7] 或 [Clang][8]）
* [libzmq 库][9]
* [Python 3][10]
* [ZeroMQ 的 Python 封装][11]
Fedora 系统上的安装方法：
```
$ dnf install clang zeromq zeromq-devel python3 python3-zmq
```
Debian 和 Ubuntu 系统上的安装方法：
```
$ apt-get install clang libzmq5 libzmq3-dev python3 python3-zmq
```
如果有问题，参考对应项目的安装指南（上面附有链接）。
### 编写硬件接口库
因为这里针对的是个设想的场景，本教程虚构了包含两个函数的操作库：
* `fancyhw_init()` 用来初始化（设想的）硬件
* `fancyhw_read_val()` 用于返回从硬件读取的数据
将库的完整代码保存到文件 `libfancyhw.h` 中:
```
#ifndef LIBFANCYHW_H
#define LIBFANCYHW_H
#include <stdlib.h>
#include <stdint.h>
// This is the fictitious hardware interfacing library
void fancyhw_init(unsigned int init_param)
{
srand(init_param);
}
int16_t fancyhw_read_val(void)
{
return (int16_t)rand();
}
#endif
```
这个库可以模拟你要在不同语言实现的组件间交换的数据，中间有个随机数发生器。
### 设计 C 接口
下面从包含管理数据传输的库开始，逐步实现 C 接口。
#### 需要的库
开始先加载必要的库（每个库的作用见代码注释）：
```
// For printf()
#include <stdio.h>
// For EXIT_*
#include <stdlib.h>
// For memcpy()
#include <string.h>
// For sleep()
#include <unistd.h>
#include <zmq.h>
#include "libfancyhw.h"
```
#### 必要的参数
定义 `main` 函数和后续过程中必要的参数：
```
int main(void)
{
    const unsigned int INIT_PARAM = 12345;
    const unsigned int REPETITIONS = 10;
    const unsigned int PACKET_SIZE = 16;
    const char *TOPIC = "fancyhw_data";
    ...
```
#### 初始化
所有的库都需要初始化。虚构的那个只需要一个参数：
```
fancyhw_init(INIT_PARAM);
```
ZeroMQ 库需要实打实的初始化。首先，定义对象 `context`，它是用来管理全部的套接字的：
```
void *context = zmq_ctx_new();
if (!context)
{
printf("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
```
之后定义用来发送数据的套接字。ZeroMQ 支持若干种套接字，各有其用。使用 `publish` 套接字（也叫 `PUB` 套接字），可以复制消息并分发到多个接收端。这使得你可以让多个接收端接收同一个消息。没有接收者的消息将被丢弃（即不会入消息队列）。用法如下：
```
void *data_socket = zmq_socket(context, ZMQ_PUB);
```
套接字需要绑定到一个具体的地址，这样客户端就知道要连接哪里了。本例中，使用了 [TCP 传输层][15]（当然也有 [其它选项][16]，但 TCP 是不错的默认选择）：
```
const int rb = zmq_bind(data_socket, "tcp://*:5555");
if (rb != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
```
下一步, 计算一些后续要用到的值。 注意下面代码中的 `TOPIC`，因为 `PUB` 套接字发送的消息需要绑定一个主题。主题用于供接收者过滤消息：
```
const size_t topic_size = strlen(TOPIC);
const size_t envelope_size = topic_size + 1 + PACKET_SIZE * sizeof(int16_t);
printf("Topic: %s; topic size: %zu; Envelope size: %zu\n", TOPIC, topic_size, envelope_size);
```
#### 发送消息
启动一个发送消息的循环，循环 `REPETITIONS` 次：
```
for (unsigned int i = 0; i < REPETITIONS; i++)
{
...
```
发送消息前，先填充一个长度为 `PACKET_SIZE` 的缓冲区。本库提供的是 16 个位的有符号整数。因为 C 语言中 `int` 类型占用空间大小与平台相关，不是确定的值，所以要使用指定宽度的 `int` 变量：
```
int16_t buffer[PACKET_SIZE];
for (unsigned int j = 0; j < PACKET_SIZE; j++)
{
buffer[j] = fancyhw_read_val();
}
printf("Read %u data values\n", PACKET_SIZE);
```
消息的准备和发送的第一步是创建 ZeroMQ 消息，为消息分配必要的内存空间。空白的消息是用于封装要发送的数据的：
```
zmq_msg_t envelope;
const int rmi = zmq_msg_init_size(&envelope, envelope_size);
if (rmi != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_msg_init_size(): %s\n", zmq_strerror(errno));
zmq_msg_close(&envelope);
break;
}
```
现在内存空间已分配，数据保存在 ZeroMQ 消息 “信封”中。函数 `zmq_msg_data()` 返回一个指向封装数据缓存区顶端的指针。第一部分是主题，之后是一个空格，最后是二进制数。主题和二进制数据之间的分隔符采用空格字符。需要遍历缓存区的话，使用类型转换和 [指针算法][18]。（感谢 C 语言，让事情变得直截了当。）做法如下：
```
memcpy(zmq_msg_data(&envelope), TOPIC, topic_size);
memcpy((void*)((char*)zmq_msg_data(&envelope) + topic_size), " ", 1);
memcpy((void*)((char*)zmq_msg_data(&envelope) + 1 + topic_size), buffer, PACKET_SIZE * sizeof(int16_t))
```
通过 `data_socket` 发送消息：
```
const size_t rs = zmq_msg_send(&envelope, data_socket, 0);
if (rs != envelope_size)
{
printf("ERROR: ZeroMQ error occurred during zmq_msg_send(): %s\n", zmq_strerror(errno));
zmq_msg_close(&envelope);
break;
}
```
使用数据之前要先解除封装：
```
zmq_msg_close(&envelope);
printf("Message sent; i: %u, topic: %s\n", i, TOPIC);
```
#### 清理
C 语言不提供 [垃圾收集][20] 功能，用完之后记得要自己扫尾。发送消息之后结束程序之前，需要运行扫尾代码，释放分配的内存:
```
const int rc = zmq_close(data_socket);
if (rc != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_close(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
const int rd = zmq_ctx_destroy(context);
if (rd != 0)
{
printf("Error occurred during zmq_ctx_destroy(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
```
#### 完整 C 代码
保存下面完整的接口代码到本地名为 `hw_interface.c` 的文件：
```
// For printf()
#include <stdio.h>
// For EXIT_*
#include <stdlib.h>
// For memcpy()
#include <string.h>
// For sleep()
#include <unistd.h>
#include <zmq.h>
#include "libfancyhw.h"
int main(void)
{
const unsigned int INIT_PARAM = 12345;
const unsigned int REPETITIONS = 10;
const unsigned int PACKET_SIZE = 16;
const char *TOPIC = "fancyhw_data";
fancyhw_init(INIT_PARAM);
void *context = zmq_ctx_new();
if (!context)
{
printf("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
void *data_socket = zmq_socket(context, ZMQ_PUB);
const int rb = zmq_bind(data_socket, "tcp://*:5555");
if (rb != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
const size_t topic_size = strlen(TOPIC);
const size_t envelope_size = topic_size + 1 + PACKET_SIZE * sizeof(int16_t);
printf("Topic: %s; topic size: %zu; Envelope size: %zu\n", TOPIC, topic_size, envelope_size);
for (unsigned int i = 0; i < REPETITIONS; i++)
{
int16_t buffer[PACKET_SIZE];
for (unsigned int j = 0; j < PACKET_SIZE; j++)
{
buffer[j] = fancyhw_read_val();
}
printf("Read %u data values\n", PACKET_SIZE);
zmq_msg_t envelope;
const int rmi = zmq_msg_init_size(&envelope, envelope_size);
if (rmi != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_msg_init_size(): %s\n", zmq_strerror(errno));
zmq_msg_close(&envelope);
break;
}
memcpy(zmq_msg_data(&envelope), TOPIC, topic_size);
memcpy((void*)((char*)zmq_msg_data(&envelope) + topic_size), " ", 1);
memcpy((void*)((char*)zmq_msg_data(&envelope) + 1 + topic_size), buffer, PACKET_SIZE * sizeof(int16_t));
const size_t rs = zmq_msg_send(&envelope, data_socket, 0);
if (rs != envelope_size)
{
printf("ERROR: ZeroMQ error occurred during zmq_msg_send(): %s\n", zmq_strerror(errno));
zmq_msg_close(&envelope);
break;
}
zmq_msg_close(&envelope);
printf("Message sent; i: %u, topic: %s\n", i, TOPIC);
sleep(1);
}
const int rc = zmq_close(data_socket);
if (rc != 0)
{
printf("ERROR: ZeroMQ error occurred during zmq_close(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
const int rd = zmq_ctx_destroy(context);
if (rd != 0)
{
printf("Error occurred during zmq_ctx_destroy(): %s\n", zmq_strerror(errno));
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
```
用如下命令编译：
```
$ clang -std=c99 -I. hw_interface.c -lzmq -o hw_interface
```
如果没有编译错误，你就可以运行这个接口了。贴心的是，ZeroMQ `PUB` 套接字可以在没有任何应用发送或接受数据的状态下运行，这简化了使用复杂度，因为这样不限制进程启动的次序。
运行该接口：
```
$ ./hw_interface
Topic: fancyhw_data; topic size: 12; Envelope size: 45
Read 16 data values
Message sent; i: 0, topic: fancyhw_data
Read 16 data values
Message sent; i: 1, topic: fancyhw_data
Read 16 data values
...
...
```
输出显示数据已经通过 ZeroMQ 完成发送，现在要做的是让一个程序去读数据。
### 编写 Python 数据处理器
现在已经准备好从 C 程序向 Python 应用传送数据了。
#### 库
需要两个库帮助实现数据传输。首先是 ZeroMQ 的 Python 封装：
```
$ python3 -m pip install zmq
```
另一个就是 [struct 库][21]，用于解码二进制数据。这个库是 Python 标准库的一部分，所以不需要使用 `pip` 命令安装。
Python 程序的第一部分是导入这些库：
```
import zmq
import struct
```
#### 重要参数
使用 ZeroMQ 时，只能向常量 `TOPIC` 定义相同的接收端发送消息：
```
topic = "fancyhw_data".encode('ascii')
print("Reading messages with topic: {}".format(topic))
```
#### 初始化
下一步，初始化上下文和套接字。使用 `subscribe` 套接字（也称为 `SUB` 套接字），它是 `PUB` 套接字的天生伴侣。这个套接字发送时也需要匹配主题。
```
with zmq.Context() as context:
    socket = context.socket(zmq.SUB)
    socket.connect("tcp://127.0.0.1:5555")
    socket.setsockopt(zmq.SUBSCRIBE, topic)
    i = 0
    ...
```
#### 接收消息
启动一个无限循环，等待接收发送到 `SUB` 套接字的新消息。这个循环会在你按下 `Ctrl+C` 组合键或者内部发生错误时终止：
```
    try:
        while True:
            ... # we will fill this in next
    except KeyboardInterrupt:
        socket.close()
    except Exception as error:
        print("ERROR: {}".format(error))
        socket.close()
```
这个循环等待 `recv()` 方法获取的新消息，然后将接收到的内容从第一个空格字符处分割开，从而得到主题：
```
binary_topic, data_buffer = socket.recv().split(b' ', 1)
```
#### 解码消息
Python 此时尚不知道主题是个字符串，使用标准 ASCII 编解码器进行解码：
```
topic = binary_topic.decode(encoding = 'ascii')
print("Message {:d}:".format(i))
print("\ttopic: '{}'".format(topic))
```
下一步就是使用 `struct` 库读取二进制数据，它可以将二进制数据段转换为明确的数值。首先，计算数据包中数值的组数。本例中使用的 16 个位的有符号整数对应的是 `struct` [格式字符][22] 中的 `h`：
```
packet_size = len(data_buffer) // struct.calcsize("h")
print("\tpacket size: {:d}".format(packet_size))
```
知道数据包中有多少组数据后，就可以通过构建一个包含数据组数和数据类型的字符串，来定义格式了（比如“`16h`”）：
```
struct_format = "{:d}h".format(packet_size)
```
将二进制数据串转换为可直接打印的一系列数字：
```
data = struct.unpack(struct_format, data_buffer)
print("\tdata: {}".format(data))
```
#### 完整 Python 代码
下面是 Python 实现的完整的接收端：
```
#! /usr/bin/env python3
import zmq
import struct
topic = "fancyhw_data".encode('ascii')
print("Reading messages with topic: {}".format(topic))
with zmq.Context() as context:
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:5555")
socket.setsockopt(zmq.SUBSCRIBE, topic)
i = 0
try:
while True:
binary_topic, data_buffer = socket.recv().split(b' ', 1)
topic = binary_topic.decode(encoding = 'ascii')
print("Message {:d}:".format(i))
print("\ttopic: '{}'".format(topic))
packet_size = len(data_buffer) // struct.calcsize("h")
print("\tpacket size: {:d}".format(packet_size))
struct_format = "{:d}h".format(packet_size)
data = struct.unpack(struct_format, data_buffer)
print("\tdata: {}".format(data))
i += 1
except KeyboardInterrupt:
socket.close()
except Exception as error:
print("ERROR: {}".format(error))
socket.close()
```
将上面的内容保存到名为 `online_analysis.py` 的文件。Python 代码不需要编译，你可以直接运行它。
运行输出如下：
```
$ ./online_analysis.py
Reading messages with topic: b'fancyhw_data'
Message 0:
        topic: 'fancyhw_data'
        packet size: 16
        data: (20946, -23616, 9865, 31416, -15911, -10845, -5332, 25662, 10955, -32501, -18717, -24490, -16511, -28861, 24205, 26568)
Message 1:
        topic: 'fancyhw_data'
        packet size: 16
        data: (12505, 31355, 14083, -19654, -9141, 14532, -25591, 31203, 10428, -25564, -732, -7979, 9529, -27982, 29610, 30475)
...
...
```
### 小结
本教程介绍了一种新方式，实现从基于 C 的硬件接口收集数据，并分发到基于 Python 的基础设施的功能。借此可以获取数据供后续分析，或者转送到任意数量的接收端去。它采用了一个消息库实现数据在发送者和处理者之间的传送，来取代同样功能规模庞大的软件。
本教程还引出了我称之为“软件粒度”的概念，换言之，就是将软件细分为更小的部分。这种做法的优点之一就是，使得同时采用不同的编程语言实现最简接口作为不同部分之间沟通的组件成为可能。
实践中，这种设计使得软件工程师能以更独立、合作更高效的方式做事。不同的团队可以专注于数据分析的不同方面，可以选择自己中意的实现工具。这种做法的另一个优点是实现了零代价的并行，因为所有的进程都可以并行运行。[ZeroMQ 消息库][3] 是个令人赞叹的软件，使用它可以让工作大大简化。
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/3/zeromq-c-python
作者：[Cristiano L. Fontana][a]
选题：[lujun9972][b]
译者：[silentdawn-zz](https://github.com/译者ID)
校对：[wxy](https://github.com/wxy)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/cristianofontana
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/email_chat_communication_message.png?itok=LKjiLnQu (Chat via email)
[2]: https://docs.python.org/3/extending/extending.html
[3]: https://zeromq.org/
[4]: https://en.wikipedia.org/wiki/Network_socket
[5]: https://en.wikipedia.org/wiki/Pieter_Hintjens
[6]: http://hintjens.com/
[7]: https://gcc.gnu.org/
[8]: https://clang.llvm.org/
[9]: https://github.com/zeromq/libzmq#installation-of-binary-packages-
[10]: https://www.python.org/downloads/
[11]: https://zeromq.org/languages/python/
[12]: http://www.opengroup.org/onlinepubs/009695399/functions/srand.html
[13]: http://www.opengroup.org/onlinepubs/009695399/functions/rand.html
[14]: http://www.opengroup.org/onlinepubs/009695399/functions/printf.html
[15]: https://en.wikipedia.org/wiki/Transmission_Control_Protocol
[16]: http://zguide.zeromq.org/page:all#Plugging-Sockets-into-the-Topology
[17]: http://www.opengroup.org/onlinepubs/009695399/functions/strlen.html
[18]: https://en.wikipedia.org/wiki/Pointer_%28computer_programming%29%23C_and_C++
[19]: http://www.opengroup.org/onlinepubs/009695399/functions/memcpy.html
[20]: https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
[21]: https://docs.python.org/3/library/struct.html
[22]: https://docs.python.org/3/library/struct.html#format-characters

53
sources/talk/20200807 Meeting for the first time after 26 years of open source collaboration.md Normal file
View File

@ -0,0 +1,53 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Meeting for the first time after 26 years of open source collaboration)
[#]: via: (https://opensource.com/article/20/8/meet-open-source-collaborators)
[#]: author: (Jim Hall https://opensource.com/users/jim-hall)
Meeting for the first time after 26 years of open source collaboration
======
Open source is great because people from anywhere can get involved, but
it can be lonely. Take advantage of the proliferation of video
conferencing tools available to make connections with your fellow
developers.
![Two people chatting via a video conference app][1]
Collaborating on an open source software project is inherently an online experience. For me, almost all of my interaction has been via email. I'll send someone a patch, and they'll review it and reply to me. Or a user will file a bug, and I'll respond to it via the bug tracker. More commonly, developers in the open source community will discuss ideas via the email list.
Over the years, I've only interacted on projects electronically, and have only met a few people in person, usually in settings unrelated to the project. Sometimes it's at conferences—we'll recognize each other's names, and realize we're working on the same open source project. I really enjoy those connections, but they're rare.
In 1994, I created the [FreeDOS Project][2], an open source implementation of the DOS operating system. In the last 26 years, I've met only a handful of folks from that project in person. I met one developer when I attended a work conference in his city, and we got dinner together. Another developer took time out of his vacation tour of the Midwest to visit and have a drink. I crossed paths with another developer at an open source conference. And I spoke with Pat, who wrote our first kernel, on the phone several times, but we never met in person.
A few weeks ago, I decided to change that dynamic. We live in an era of ubiquitous video conferencing, even more so in the last few months, as many people have shifted to remote work due to the pandemic. Even though open source work happens virtually, independent of geography, we can still build relationships and put names to faces. Zoom, BlueJeans, Google Hangouts, or Jitsi Meet are the new way to connect while maintaining social distance.
On June 29, 2020, to celebrate the 26th anniversary of FreeDOS, we held our first online get-together. It was an opportunity to finally actually meet the people I'd been working with for the last 20+ years.
And it was awesome! At our peak, we had over 16 people on the video meeting. I loved getting to see everyone and talk "in-person." We had almost an hour and a half of engaging conversations with folks from all over the world. Thanks to everyone who participated!
It's meetings like this when I remember open source is more than just writing code; it's about building a community. And while I've always done well with our virtual community that communicates via email, I really appreciated getting to talk to people without the filter of email—making that real-time connection meant a lot to me.
This virtual meet-up went so well that I'd love to do it again. In fact, I'm planning another virtual get-together later this summer. I'm hoping to focus the conversation on that one, maybe make it a "working session" to nail down some things for the upcoming FreeDOS 1.3 release. Stay tuned for more details!
![Screenshot of zoom call participants with names and faces blurred][3]
*Names and faces blurred for privacy
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/8/meet-open-source-collaborators
作者：[Jim Hall][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/jim-hall
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/chat_video_conference_talk_team.png?itok=t2_7fEH0 (Two people chatting via a video conference app)
[2]: https://www.freedos.org/
[3]: https://opensource.com/sites/default/files/uploads/screenshot_from_2020-06-29_09-29-49.jpg (Screenshot of zoom call participants with names and faces blurred)

59
sources/talk/20200807 Microsoft uses AI to boost its reuse, recycling of server parts.md Normal file
View File

@ -0,0 +1,59 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Microsoft uses AI to boost its reuse, recycling of server parts)
[#]: via: (https://www.networkworld.com/article/3570451/microsoft-uses-ai-to-boost-its-reuse-recycling-of-server-parts.html)
[#]: author: (Andy Patrizio https://www.networkworld.com/author/Andy-Patrizio/)
Microsoft uses AI to boost its reuse, recycling of server parts
======
Get ready to hear the term 'circular' a lot more in reference to data center gear.
Monsitj / Getty Images
Microsoft is bringing artificial intelligence to the task of sorting through millions of servers to determine what can be recycled and where.
The new initiative calls for the building of so-called Circular Centers at Microsoft data centers around the world, where AI algorithms will be used to sort through parts from decommissioned servers or other hardware and figure out which parts can be reused on the campus.
**READ MORE:** [How to decommission a data center][1]
Microsoft says it has more than three million servers and related hardware in its data centers, and that a server's average lifespan is about five years. Plus, Microsoft is expanding globally, so its server numbers should increase.
Circular Centers are all about quickly sorting through the inventory rather than tying up overworked staff. Microsoft plans to increase its reuse of server parts by 90% by 2025. "Using machine learning, we will process servers and hardware that are being decommissioned onsite," wrote Brad Smith, president of Microsoft, in a [blog post][2] announcing the initiative. "We'll sort the pieces that can be reused and repurposed by us, our customers, or sold."
Smith notes that today there is no consistent data about the quantity, quality and type of waste, where it is generated, and where it goes. Data about construction and demolition waste, for example, is inconsistent and needs a standardized methodology, better transparency and higher quality.
"Without more accurate data, it's nearly impossible to understand the impact of operational decisions, what goals to set, and how to assess progress, as well as an industry standard for waste footprint methodology," he wrote.
A Circular Center pilot in an Amsterdam data center reduced downtime and increased the availability of server and network parts for its own reuse and buy-back by suppliers, according to Microsoft. It also reduced the cost of transporting and shipping servers and hardware to processing facilities, which lowered carbon emissions.
The term "circular economy" is catching on in tech. It's based on recycling of server hardware, putting equipment that is a few years old but still quite usable back in service somewhere else. ITRenew, a reseller of used hyperscaler servers [that I profiled][3] a few months back, is big on the term.
The first Microsoft Circular Centers will be built at new, major data-center campuses or regions, the company says. It plans to eventually add these centers to campuses that already exist.
Microsoft has an expressed goal of being "carbon negative" by 2030, and this is just one of several projects. Recently Microsoft announced it had conducted a test at its system developer's lab in Salt Lake City where a 250kW hydrogen fuel cell system powered a row of server racks for 48 hours straight, something the company says has never been done before.
"It is the largest computer backup power system that we know that is running on hydrogen and it has run the longest continuous test," Mark Monroe, a principal infrastructure engineer, wrote in a Microsoft [blog post][4]. He says hydrogen fuel cells have plummeted so much in recent years that they are now a viable alternative to diesel-powered backup generators but much cleaner burning.
Join the Network World communities on [Facebook][5] and [LinkedIn][6] to comment on topics that are top of mind.
--------------------------------------------------------------------------------
via: https://www.networkworld.com/article/3570451/microsoft-uses-ai-to-boost-its-reuse-recycling-of-server-parts.html
作者：[Andy Patrizio][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://www.networkworld.com/author/Andy-Patrizio/
[b]: https://github.com/lujun9972
[1]: https://www.networkworld.com/article/3439917/how-to-decommission-a-data-center.html
[2]: https://blogs.microsoft.com/blog/2020/08/04/microsoft-direct-operations-products-and-packaging-to-be-zero-waste-by-2030/
[3]: https://www.networkworld.com/article/3543810/for-sale-used-low-mileage-hyperscaler-servers.html
[4]: https://news.microsoft.com/innovation-stories/hydrogen-datacenters/
[5]: https://www.facebook.com/NetworkWorld/
[6]: https://www.linkedin.com/company/network-world

79
sources/talk/20200808 Matrix encrypted chat rolls out across Germany, Project ACRN-s new IoT release, and more open source news.md Normal file
View File

@ -0,0 +1,79 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Matrix encrypted chat rolls out across Germany, Project ACRN's new IoT release, and more open source news)
[#]: via: (https://opensource.com/article/20/8/news-aug-8)
[#]: author: (Lauren Maffeo https://opensource.com/users/lmaffeo)
Matrix encrypted chat rolls out across Germany, Project ACRN's new IoT release, and more open source news
======
Catch up on the biggest open source headlines from the past two weeks.
![][1]
In this week’s edition of our open source news roundup, an open source microfluidics pump, Germany rolls out an encrypted messaging platform based on Matrix, and more open source news.
### A Reddit user shares his open source microfluidics pump
As COVID-19 limits access to chemistry and biology labs, one open source user made his own solution. Josh Maceachern shared the model in a thread [on Reddit][2], sharing his microfluidics pump for less than $300 in total. 
According to the creator, the pump runs on only 3D-printed and common components for less than $300 total. An [article by Hackaday][3] adds that the pump delivers up to 15 mL/min with accuracy to 0.1uL/min. That's enough power for users to do their own genetics or biology projects at home.
Along with the design files for folks to build their own pumps, Maceachern said on Reddit that he's releasing the pump under the CC BY 4.0 license. He gave his blessing for anyone to use it, asking that they attribute him.
### Germany prepares for the world's largest deployment of Matrix-based open source software 
When [Dataport][4] rolls out Element collaboration software this fall, they'll deploy it for half a million users in the German states of Schlesweig-Holstein and Hamburg. According to Element in [this article][5], it will be "...the biggest single messaging and collaboration implementation in the world."
In an era of videoconferencing dominated by big brands, Germany's choice of a fully open source tool is noteworthy. It's also no accident that this deployment will occur in Europe: Element shared [in a blog post][6] that it's devoted to giving users more control over their data, rather than seeing it stored on overseas systems.
The post also cited "increasingly anti-encryption" sentiment in the U.S. as another reason to offer end-to-end encryption for European users. Element's forthcoming rollout in Germany will allow users to host their own data and messages. Dataport hopes to [deploy Element][7] as a matrix-based solution by September, in time for the start of term in Germany.
### Project ACRN release Version 2.0 of its IoT hypervisor
ACRN is an open source hypervisor supported by the Linux Foundation's [Automotive Grade Linux][8]. The project says its lightweight hypervisor supports IoT functions like graphics, imaging, and audio on less than 40k lines of code. After the initial focus on automotive safety, the Linux Foundation's latest release of [Project ACRN][9] is more focused on IoT applications by balancing safety features with more general purpose Virtual Machines (VMs). 
[The Project's v2.0][10] expands support to Kata containers and OpenStack. It also supports partition mode and sharing mode in tandem.
In other news:
* [Open wearables group buttons down on datasets][11]
* [Microsoft joins Open Source Security Foundation][12]
* [Uniting for better open-source security: The Open Source Security Foundation][13]
* [China unveils AI open-source platform 'Dubhe'][14]
* [System76 reveals an open source PC surprise for AMD Ryzen fans][15]
Thanks, as always, to Opensource.com staff members and [Correspondents][16] for their help this week.
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/8/news-aug-8
作者：[Lauren Maffeo][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/lmaffeo
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/weekly_news_roundup_tv.png?itok=tibLvjBd
[2]: https://www.reddit.com/r/functionalprint/comments/hsevpl/opensource_multichannel_microfluidicsosmm_pump/
[3]: https://hackaday.com/2020/07/29/an-open-source-microfluidic-pump-for-your-science-needs/
[4]: https://www.dataport.de/who-we-are/
[5]: https://www.zdnet.com/article/open-source-is-behind-this-massive-messaging-and-collaboration-project-for-500000-people/
[6]: https://element.io/blog/element-brings-matrix-to-europe/
[7]: https://www.dataport.de/was-wir-bewegen/portfolio/projekt-phoenix/
[8]: https://www.automotivelinux.org/
[9]: https://github.com/projectacrn
[10]: https://projectacrn.github.io/2.0/
[11]: https://www.computerweekly.com/blog/Open-Source-Insider/Open-wearables-group-buttons-down-on-datasets
[12]: https://www.microsoft.com/security/blog/2020/08/03/microsoft-open-source-security-foundation-founding-member-securing-open-source-software/
[13]: https://www.zdnet.com/article/uniting-for-better-open-source-security-the-open-source-security-foundation/#ftag=RSSbaffb68
[14]: https://news.cgtn.com/news/2020-08-03/China-unveils-AI-open-source-platform-Dubhe--SE3UpfEX5e/index.html
[15]: https://www.forbes.com/sites/jasonevangelho/2020/07/26/system76-reveals-an-open-source-surprise-for-amd-ryzen-fans/#4566bf106e4e
[16]: https://opensource.com/correspondent-program

4
sources/tech/20190325 Reducing sysadmin toil with Kubernetes controllers.md
View File

@ -1,5 +1,5 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: translator: (chen-ni)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
@ -140,7 +140,7 @@ via: https://opensource.com/article/19/3/reducing-sysadmin-toil-kubernetes-contr
作者：[Paul Czarkowski][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
译者：[chen-ni](https://github.com/chen-ni)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出

542
sources/tech/20200807 An advanced guide to NLP analysis with Python and NLTK.md Normal file
View File

@ -0,0 +1,542 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (An advanced guide to NLP analysis with Python and NLTK)
[#]: via: (https://opensource.com/article/20/8/nlp-python-nltk)
[#]: author: (Girish Managoli https://opensource.com/users/gammay)
An advanced guide to NLP analysis with Python and NLTK
======
Get deeper into the foundational concepts behind natural language
processing.
![Brain on a computer screen][1]
In my [previous article][2], I introduced natural language processing (NLP) and the Natural Language Toolkit ([NLTK][3]), the NLP toolkit created at the University of Pennsylvania. I demonstrated how to parse text and define stopwords in Python and introduced the concept of a corpus, a dataset of text that aids in text processing with out-of-the-box data. In this article, I'll continue utilizing datasets to compare and analyze natural language.
The fundamental building blocks covered in this article are:
* WordNet and synsets
* Similarity comparison
* Tree and treebank
* Named entity recognition
### WordNet and synsets
[WordNet][4] is a large lexical database corpus in NLTK. WordNet maintains cognitive synonyms (commonly called synsets) of words correlated by nouns, verbs, adjectives, adverbs, synonyms, antonyms, and more.
WordNet is a very useful tool for text analysis. It is available for many languages (Chinese, English, Japanese, Russian, Spanish, and more), under many licenses (ranging from open source to commercial). The first WordNet was created by Princeton University for English under an MIT-like license.
A word is typically associated with multiple synsets based on its meanings and parts of speech. Each synset usually provides these attributes:
**Attribute** | **Definition** | **Example**
---|---|---
Name | Name of the synset | Example: The word "code" has five synsets with names `code.n.01`, `code.n.02`, `code.n.03`, `code.v.01`, `code.v.02`
POS | Part of speech of the word for this synset | The word "code" has three synsets in noun form and two in verb form
Definition | Definition of the word (in POS) | One of the definitions of "code" in verb form is: "(computer science) the symbolic arrangement of data or instructions in a computer program"
Examples | Examples of word's use | One of the examples of "code": "We should encode the message for security reasons"
Lemmas | Other word synsets this word+POC is related to (not strictly synonyms, but can be considered so); lemmas are related to other lemmas, not to words directly | Lemmas of `code.v.02` (as in "convert ordinary language into code") are `code.v.02.encipher`, `code.v.02.cipher`, `code.v.02.cypher`, `code.v.02.encrypt`, `code.v.02.inscribe`, `code.v.02.write_in_code`
Antonyms | Opposites | Antonym of lemma `encode.v.01.encode` is `decode.v.01.decode`
Hypernym | A broad category that other words fall under | A hypernym of `code.v.01` (as in "Code the pieces with numbers so that you can identify them later") is `tag.v.01`
Meronym | A word that is part of (or subordinate to) a broad category | A meronym of "computer" is "chip"
Holonym | The relationship between a parent word and its subordinate parts | A hyponym of "window" is "computer screen"
There are several other attributes, which you can find in the `nltk/corpus/reader/wordnet.py` source file in `<your python install>/Lib/site-packages`.
Some code may help this make more sense.
This helper function:
```
def synset_info(synset):
    print("Name", synset.name())
    print("POS:", synset.pos())
    print("Definition:", synset.definition())
    print("Examples:", synset.examples())
    print("Lemmas:", synset.lemmas())
    print("Antonyms:", [lemma.antonyms() for lemma in synset.lemmas() if len(lemma.antonyms()) &gt; 0])
    print("Hypernyms:", synset.hypernyms())
    print("Instance Hypernyms:", synset.instance_hypernyms())
    print("Part Holonyms:", synset.part_holonyms())
    print("Part Meronyms:", synset.part_meronyms())
    print()
[/code] [code]`synsets = wordnet.synsets('code')`
```
shows this:
```
5 synsets:
Name code.n.01
POS: n
Definition: a set of rules or principles or laws (especially written ones)
Examples: []
Lemmas: [Lemma('code.n.01.code'), Lemma('code.n.01.codification')]
Antonyms: []
Hypernyms: [Synset('written_communication.n.01')]
Instance Hpernyms: []
Part Holonyms: []
Part Meronyms: []
...
Name code.n.03
POS: n
Definition: (computer science) the symbolic arrangement of data or instructions in a computer program or the set of such instructions
Examples: []
Lemmas: [Lemma('code.n.03.code'), Lemma('code.n.03.computer_code')]
Antonyms: []
Hypernyms: [Synset('coding_system.n.01')]
Instance Hpernyms: []
Part Holonyms: []
Part Meronyms: []
...
Name code.v.02
POS: v
Definition: convert ordinary language into code
Examples: ['We should encode the message for security reasons']
Lemmas: [Lemma('code.v.02.code'), Lemma('code.v.02.encipher'), Lemma('code.v.02.cipher'), Lemma('code.v.02.cypher'), Lemma('code.v.02.encrypt'), Lemma('code.v.02.inscribe'), Lemma('code.v.02.write_in_code')]
Antonyms: []
Hypernyms: [Synset('encode.v.01')]
Instance Hpernyms: []
Part Holonyms: []
Part Meronyms: []
```
Synsets and lemmas follow a tree structure you can visualize:
```
def hypernyms(synset):
    return synset.hypernyms()
synsets = wordnet.synsets('soccer')
for synset in synsets:
    print(synset.name() + " tree:")
    pprint(synset.tree(rel=hypernyms))
    print()
[/code] [code]
code.n.01 tree:
[Synset('code.n.01'),
 [Synset('written_communication.n.01'),
   ...
code.n.02 tree:
[Synset('code.n.02'),
 [Synset('coding_system.n.01'),
   ...
code.n.03 tree:
[Synset('code.n.03'),
   ...
code.v.01 tree:
[Synset('code.v.01'),
 [Synset('tag.v.01'),
   ...
code.v.02 tree:
[Synset('code.v.02'),
 [Synset('encode.v.01'),
   ...
```
WordNet does not cover all words and their information (there are about 170,000 words in English today and about 155,000 in the latest version of WordNet), but it's a good starting point. After you learn the concepts of this building block, if you find it inadequate for your needs, you can migrate to another. Or, you can build your own WordNet!
#### Try it yourself
Using the Python libraries, download Wikipedia's page on [open source][5] and list the synsets and lemmas of all the words.
### Similarity comparison
Similarity comparison is a building block that identifies similarities between two pieces of text. It has many applications in search engines, chatbots, and more.
For example, are the words "football" and "soccer" related?
```
syn1 = wordnet.synsets('football')
syn2 = wordnet.synsets('soccer')
# A word may have multiple synsets, so need to compare each synset of word1 with synset of word2
for s1 in syn1:
    for s2 in syn2:
        print("Path similarity of: ")
        print(s1, '(', s1.pos(), ')', '[', s1.definition(), ']')
        print(s2, '(', s2.pos(), ')', '[', s2.definition(), ']')
        print("   is", s1.path_similarity(s2))
        print()
[/code] [code]
Path similarity of:
Synset('football.n.01') ( n ) [ any of various games played with a ball (round or oval) in which two teams try to kick or carry or propel the ball into each other's goal ]
Synset('soccer.n.01') ( n ) [ a football game in which two teams of 11 players try to kick or head a ball into the opponents' goal ]
   is 0.5
Path similarity of:
Synset('football.n.02') ( n ) [ the inflated oblong ball used in playing American football ]
Synset('soccer.n.01') ( n ) [ a football game in which two teams of 11 players try to kick or head a ball into the opponents' goal ]
   is 0.05
```
The highest path similarity score of the words is 0.5, indicating they are closely related.
What about "code" and "bug"? Similarity scores for these words used in computer science are:
```
Path similarity of:
Synset('code.n.01') ( n ) [ a set of rules or principles or laws (especially written ones) ]
Synset('bug.n.02') ( n ) [ a fault or defect in a computer program, system, or machine ]
   is 0.1111111111111111
...
Path similarity of:
Synset('code.n.02') ( n ) [ a coding system used for transmitting messages requiring brevity or secrecy ]
Synset('bug.n.02') ( n ) [ a fault or defect in a computer program, system, or machine ]
   is 0.09090909090909091
...
Path similarity of:
Synset('code.n.03') ( n ) [ (computer science) the symbolic arrangement of data or instructions in a computer program or the set of such instructions ]
Synset('bug.n.02') ( n ) [ a fault or defect in a computer program, system, or machine ]
   is 0.09090909090909091
```
These are the highest similarity scores, which indicates they are related.
NLTK provides several similarity scorers, such as:
* path_similarity
* lch_similarity
* wup_similarity
* res_similarity
* jcn_similarity
* lin_similarity
See the Similarity section of the [WordNet Interface][6] page to determine the appropriate one for your application.
#### Try it yourself
Using Python libraries, start from the Wikipedia [Category: Lists of computer terms][7] page and prepare a list of terminologies, then see how the words correlate.
### Tree and treebank
With NLTK, you can represent a text's structure in tree form to help with text analysis.
Here is an example:
A simple text pre-processed and part-of-speech (POS)-tagged:
```
import nltk
text = "I love open source"
# Tokenize to words
words = nltk.tokenize.word_tokenize(text)
# POS tag the words
words_tagged = nltk.pos_tag(words)
```
You must define a grammar to convert the text to a tree structure. This example uses a simple grammar based on the [Penn Treebank tags][8].
```
# A simple grammar to create tree
grammar = "NP: {&lt;JJ&gt;&lt;NN&gt;}"
```
Next, use the grammar to create a tree:
```
# Create tree
parser = nltk.RegexpParser(grammar)
tree = parser.parse(words_tagged)
pprint(tree)
```
This produces:
```
`Tree('S', [('I', 'PRP'), ('love', 'VBP'), Tree('NP', [('open', 'JJ'), ('source', 'NN')])])`
```
You can see it better graphically.
```
`tree.draw()`
```
![NLTK Tree][9]
(Girish Managoli, [CC BY-SA 4.0][10])
This structure helps explain the text's meaning correctly. As an example, identify the [subject][11] in this text:
```
subject_tags = ["NN", "NNS", "NP", "NNP", "NNPS", "PRP", "PRP$"]
def subject(sentence_tree):
    for tagged_word in sentence_tree:
        # A crude logic for this case -  first word with these tags is considered subject
        if tagged_word[1] in subject_tags:
            return tagged_word[0]
print("Subject:", subject(tree))
```
It shows "I" is the subject:
```
`Subject: I`
```
This is a basic text analysis building block that is applicable to larger applications. For example, when a user says, "Book a flight for my mom, Jane, to NY from London on January 1st," a chatbot using this block can interpret the request as:
**Action**: Book
**What**: Flight
**Traveler**: Jane
**From**: London
**To**: New York
**Date**: 1 Jan (of the next year)
A treebank refers to a corpus with pre-tagged trees. Open source, conditional free-for-use, and commercial treebanks are available for many languages. The most commonly used one for English is Penn Treebank, extracted from the _Wall Street Journal_, a subset of which is included in NLTK. Some ways of using a treebank:
```
words = nltk.corpus.treebank.words()
print(len(words), "words:")
print(words)
tagged_sents = nltk.corpus.treebank.tagged_sents()
print(len(tagged_sents), "sentences:")
print(tagged_sents)
[/code] [code]
100676 words:
['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', ...]
3914 sentences:
[[('Pierre', 'NNP'), ('Vinken', 'NNP'), (',', ','), ('61', 'CD'), ('years', 'NNS'), ('old', 'JJ'), (',', ','), ('will', 'MD'), ('join', 'VB'), ('the', 'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive', 'JJ'), ('director', 'NN'), ...]
```
See tags in a sentence:
```
sent0 = tagged_sents[0]
pprint(sent0)
[/code] [code]
[('Pierre', 'NNP'),
 ('Vinken', 'NNP'),
 (',', ','),
 ('61', 'CD'),
 ('years', 'NNS'),
...
```
Create a grammar to convert this to a tree:
```
grammar = '''
    Subject: {&lt;NNP&gt;&lt;NNP&gt;}
    SubjectInfo: {&lt;CD&gt;&lt;NNS&gt;&lt;JJ&gt;}
    Action: {&lt;MD&gt;&lt;VB&gt;}
    Object: {&lt;DT&gt;&lt;NN&gt;}
    Stopwords: {&lt;IN&gt;&lt;DT&gt;}
    ObjectInfo: {&lt;JJ&gt;&lt;NN&gt;}
    When: {&lt;NNP&gt;&lt;CD&gt;}
'''
parser = nltk.RegexpParser(grammar)
tree = parser.parse(sent0)
print(tree)
[/code] [code]
(S
  (Subject Pierre/NNP Vinken/NNP)
  ,/,
  (SubjectInfo 61/CD years/NNS old/JJ)
  ,/,
  (Action will/MD join/VB)
  (Object the/DT board/NN)
  as/IN
  a/DT
  (ObjectInfo nonexecutive/JJ director/NN)
  (Subject Nov./NNP)
  29/CD
  ./.)
```
See it graphically:
```
`tree.draw()`
```
![NLP Treebank image][12]
(Girish Managoli, [CC BY-SA 4.0][10])
The concept of trees and treebanks is a powerful building block for text analysis.
#### Try it yourself
Using the Python libraries, download Wikipedia's page on [open source][5] and represent the text in a presentable view.
### Named entity recognition
Text, whether spoken or written, contains important data. One of text processing's primary goals is extracting this key data. This is needed in almost all applications, such as an airline chatbot that books tickets or a question-answering bot. NLTK provides a named entity recognition feature for this.
Here's a code example:
```
`sentence = 'Peterson first suggested the name "open source" at Palo Alto, California'`
```
See if name and place are recognized in this sentence. Pre-process as usual:
```
import nltk
words = nltk.word_tokenize(sentence)
pos_tagged = nltk.pos_tag(words)
```
Run the named-entity tagger:
```
ne_tagged = nltk.ne_chunk(pos_tagged)
print("NE tagged text:")
print(ne_tagged)
print()
[/code] [code]
NE tagged text:
(S
  (PERSON Peterson/NNP)
  first/RB
  suggested/VBD
  the/DT
  name/NN
  ``/``
  open/JJ
  source/NN
  ''/''
  at/IN
  (FACILITY Palo/NNP Alto/NNP)
  ,/,
  (GPE California/NNP))
```
Name tags were added; extract only the named entities from this tree:
```
print("Recognized named entities:")
for ne in ne_tagged:
    if hasattr(ne, "label"):
        print(ne.label(), ne[0:])
[/code] [code]
Recognized named entities:
PERSON [('Peterson', 'NNP')]
FACILITY [('Palo', 'NNP'), ('Alto', 'NNP')]
GPE [('California', 'NNP')]
```
See it graphically:
```
`ne_tagged.draw()`
```
![NLTK Treebank tree][13]
(Girish Managoli, [CC BY-SA 4.0][10])
NLTK's built-in named-entity tagger, using PENN's [Automatic Content Extraction][14] (ACE) program, detects common entities such as ORGANIZATION, PERSON, LOCATION, FACILITY, and GPE (geopolitical entity).
NLTK can use other taggers, such as the [Stanford Named Entity Recognizer][15]. This trained tagger is built in Java, but NLTK provides an interface to work with it (See [nltk.parse.stanford][16] or [nltk.tag.stanford][17]).
#### Try it yourself
Using the Python libraries, download Wikipedia's page on [open source][5] and identify people who had an influence on open source and where and when they contributed.
### Advanced exercise
If you're ready for it, try building this superstructure using the building blocks discussed in these articles.
Using Python libraries, download Wikipedia's [Category: Computer science page][18] and:
* Identify the most-occurring unigrams, bigrams, and trigrams and publish it as a list of keywords or technologies that students and engineers need to be aware of in this domain.
* Show the names, technologies, dates, and places that matter in this field graphically. This can be a nice infographic.
* Create a search engine. Does your search engine perform better than Wikipedia's search?
### What's next?
NLP is a quintessential pillar in application building. NLTK is a classic, rich, and powerful kit that provides the bricks and mortar to build practically appealing, purposeful applications for the real world.
In this series of articles, I explained what NLP makes possible using NLTK as an example. NLP and NLTK have a lot more to offer. This series is an inception point to help get you started.
If your needs grow beyond NLTK's capabilities, you could train new models or add capabilities to it. New NLP libraries that build on NLTK are coming up, and machine learning is being used extensively in language processing.
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/8/nlp-python-nltk
作者：[Girish Managoli][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/gammay
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/brain_computer_solve_fix_tool.png?itok=okq8joti (Brain on a computer screen)
[2]: https://opensource.com/article/20/8/intro-python-nltk
[3]: http://www.nltk.org/
[4]: https://en.wikipedia.org/wiki/WordNet
[5]: https://en.wikipedia.org/wiki/Open_source
[6]: https://www.nltk.org/howto/wordnet.html
[7]: https://en.wikipedia.org/wiki/Category:Lists_of_computer_terms
[8]: https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
[9]: https://opensource.com/sites/default/files/uploads/nltk-tree.jpg (NLTK Tree)
[10]: https://creativecommons.org/licenses/by-sa/4.0/
[11]: https://en.wikipedia.org/wiki/Subject_(grammar)
[12]: https://opensource.com/sites/default/files/uploads/nltk-treebank.jpg (NLP Treebank image)
[13]: https://opensource.com/sites/default/files/uploads/nltk-treebank-2a.jpg (NLTK Treebank tree)
[14]: https://www.ldc.upenn.edu/collaborations/past-projects/ace
[15]: https://nlp.stanford.edu/software/CRF-NER.html
[16]: https://www.nltk.org/_modules/nltk/parse/stanford.html
[17]: https://www.nltk.org/_modules/nltk/tag/stanford.html
[18]: https://en.wikipedia.org/wiki/Category:Computer_science

121
sources/tech/20200807 install Fedora on a Raspberry Pi 3.md Normal file
View File

@ -0,0 +1,121 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (install Fedora on a Raspberry Pi 3)
[#]: via: (https://fedoramagazine.org/install-fedora-on-a-raspberry-pi/)
[#]: author: (Nick Hardiman https://fedoramagazine.org/author/nickhardiman/)
install Fedora on a Raspberry Pi 3
======
![][1]
Fire up a Raspberry Pi with Fedora.
The [Raspberry Pi Foundation][2] has produced quite a few models over the years. This procedure was tested on third generation Pis – a [Model B v1.2][3], and a [Model B+][4] (the older [2][5] and the new [4][6] weren’t tested). These are the credit-card size Pis that have been around a few years.
### get hardware
You do need a few hardware items, including the Pi. You don’t need any [HaT (Hardware Attached on Top)][7] boards or USB antennas. If you have used your Pi in the past, you probably have all these items.
* **current network**. Perhaps this is your home lab.
* **ethernet cable**. This connects the current network to the Raspberry Pi
* **Raspberry Pi 3**, model B or B+.
* **power supply**
* **micro-SD card**, 8GB or larger.
* **keyboard** and **video monitor**.
The keyboard and video monitor together make up the local console. It’s possible – though complicated – to get by without a console, such as setting up an automated install then connecting over the network. A local console makes it easy to answer the configuration questions during Fedora’s first boot. Also, a mistake during AP configuration may break the network, locking out remote users.
### download Fedora Minimal
* Find Fedora’s [alternate architecture images][8].
* Download the [ARM® aarch64 Architecture image][9].
The _Fedora Minimal_ image, one of [Fedora’s alt downloads][10], has all the core packages and network packages required (well, nearly – check out _dnsmasq_ below). The image contains a ready-made file system, with over 400 packages already installed. This minimal image does not include popular packages like a development environment, Internet service or desktop. These types of software aren’t required for this work, and may well use too much memory if you install them.
The _Fedora Minimal_ raw image fits on a small SD card and runs in less than 1 GB of memory (these old Pis have 1GB RAM).
The name of the downloaded file is something like _Fedora-Minimal-32-1.6.aarch64.raw.xz_. The file is compressed and is about 700MB in size. When the file is uncompressed, it’s 5GB. It’s an ext4 file system that’s mostly empty – about 1GB is used and 4GB is empty. All that empty space is the reason the compressed download is so much smaller than the uncompressed raw image.
### copy to the micro-SD card
* Copy the image to a micro-SD card.
This can be a more complex than it sounds, and a painful experience. Finding a [good micro-SD card][11] takes work. Then there’s the challenge of physically attaching the card to your computer.Perhaps your laptop has a full SD card slot and you need a card adapter, or perhaps you need a USB adapter. Then, when it comes to copying, the OS may either help or get in your way. You may have luck with [Fedora Media Writer][12], or with these Linux commands.
```
unxz ./Fedora-Minimal-32-1.6.aarch64.raw.xz
dd if=./Fedora-Minimal-32-1.6.aarch64.raw of=/dev/mmcblk0 bs=8M status=progress oflag=direct
```
### set up Fedora
* Connect the Pi, power cable, network cable and micro-SD card.
* Hit the power.
* See the colored box as the graphics chip powers up.
* Wait for the [anaconda installer][13] to start.
* Answer anaconda’s setup questions.
Initial configuration of the OS takes a few minutes – a couple minutes waiting for boot-up, and a couple to fill out the spokes of anaconda’s text-based installer. In the examples below, the user is named **nick** and is an administrator (a member of the _wheel_ group).
Congratulations! Your Fedora Pi is up and operational.
### update software
* Update packages with `dnf update`.
* Reboot the machine with `systemctl reboot`.
Over the years, a lot of people have put a lot of work into making the Raspberry Pi devices work. Use the latest software to make sure you get the benefit of their hard work. If you skip this step, you may find some things just don’t work.
The update downloads and installs about a hundred packages. Since the storage is a micro-SD card, writing new software is a slow process. This is what using computing storage felt like in the 1990s.
### things to play with
There are a few other things that can be set up at this point, if you want to play around. It’s all optional. Try things like this.
* Replace the _localhost_ hostname with the command `sudo hostnamectl set-hostname raspi`.
* Find the IP address with `ip addr`.
* Try an SSH login, or even set up key-based login with `ssh-copy-id`.
* Power down with `systemctl poweroff`.
--------------------------------------------------------------------------------
via: https://fedoramagazine.org/install-fedora-on-a-raspberry-pi/
作者：[Nick Hardiman][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://fedoramagazine.org/author/nickhardiman/
[b]: https://github.com/lujun9972
[1]: https://fedoramagazine.org/wp-content/uploads/2020/07/fedora-on-rpi-816x346.png
[2]: https://www.raspberrypi.org/about/
[3]: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
[4]: https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus/
[5]: https://www.raspberrypi.org/products/raspberry-pi-2-model-b/
[6]: https://www.raspberrypi.org/products/raspberry-pi-4-model-b/
[7]: https://www.raspberrypi.org/blog/introducing-raspberry-pi-hats/
[8]: https://alt.fedoraproject.org/alt/
[9]: https://download.fedoraproject.org/pub/fedora-secondary/releases/32/Spins/aarch64/images/Fedora-Minimal-32-1.6.aarch64.raw.xz
[10]: https://alt.fedoraproject.org/
[11]: https://www.jeffgeerling.com/blog/2019/raspberry-pi-microsd-card-performance-comparison-2019
[12]: https://fedoramagazine.org/make-fedora-usb-stick/
[13]: https://fedoraproject.org/wiki/Anaconda

121
sources/tech/20200808 An attempt to make a font look more handwritten.md Normal file
View File

@ -0,0 +1,121 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (An attempt to make a font look more handwritten)
[#]: via: (https://jvns.ca/blog/2020/08/08/handwritten-font/)
[#]: author: (Julia Evans https://jvns.ca/)
An attempt to make a font look more handwritten
======
I’m actually not super happy with the results of this experiment, but I wanted to share it anyway because it was very easy and fun to play with fonts. And somebody asked me how to do it and I told her I’d write a blog post about it :)
### background: the original handwritten font
Some background: I have a font of my handwriting that I’ve been use in my zines for a couple of years. I made it using a delightful app called [iFontMaker][1]. They pitch themselves on their website as “You can create your handmade typeface in less than 5 minutes just with your fingers”. In my experience the ‘5 minutes” part is pretty accurate – I might have spent more like 15 minutes. I’m skeptical of the “just your fingers” claim – I used an Apple Pencil, which has much better accuracy. But it is extremely easy to make a TTF font of your handwriting with the app and if you happen to already have an Apple Pencil and iPad I think it’s a fun way to spend $7.99.
Here’s what my font looks like. The “CONNECT” text on the left is my actual handwriting, and the paragraph on the right is the font. There are actually 2 fonts – there’s a regular font and a handwritten “monospace” font. (which actually isn’t monospace in practice, I haven’t figured out how to make a actual monospace font in iFontMaker)
![][2]
### the goal: have more character variation in the font
In the screenshot above, it’s pretty obvious that it’s a font and not actual handwriting. It’s easiest to see this when you have two of the same letter next to each other, like in “HTTP’.
So I thought it might be fun to use some OpenType features to somehow introduce a little more variation into this font, like maybe the two Ts could be different. I didn’t know how to do this though!
### idea from Tristan Hume: use OpenType!
Then I was at !!Con 2020 in May (all the [talk recordings are here!][3]) and saw this talk by Tristan Hume about using OpenType to place commas in big numbers by using a special font. His talk and blog post are both great so here are a bunch of links – the live demo is maybe the fastest way to see his results.
* a live demo: [Numderline Test][4]
* the blog post: [Commas in big numbers everywhere: An OpenType adventure][5]
* the talk: [!!Con 2020 - Using font shaping to put commas in big numbers EVERYWHERE!! by Tristan Hume][6]
* the github repo: <https://github.com/trishume/numderline/blob/master/patcher.py>
### the main idea: OpenType lets you replace characters based on context
I started out being extremely confused about what OpenType even is. I still don’t know much, but I learned that you can write extremely simple OpenType rules to change how a font looks, and you don’t even have to really understand anything about fonts.
Here’s an example rule:
```
sub a' b by other_a;
```
What `sub a' b by other_a;` means is: If an `a` glyph is before a `b`, then replace the `a` with the glyph `other_a`.
So this means I can make `ab` appear different from `ac` in the font. It’s not random the way handwriting is, but it does introduce a little bit of variation.
### OpenType reference documentation: awesome
The best documentation I found for OpenType was this [OpenType™ Feature File Specification][7] reference. There are a lot of examples of cool things you can do in there, like replace “ffi” with a ligature.
### how to apply these rules: `fonttools`
Adding new OpenType rules to a font is extremely easy. There’s a Python library called `fonttools`, and these 5 lines of code will apply a list of OpenType rules (in `rules.fea`) to the font file `input.ttf`.
```
from fontTools.ttLib import TTFont
from fontTools.feaLib.builder import addOpenTypeFeatures
ft_font = TTFont('input.ttf')
addOpenTypeFeatures(ft_font, 'rules.fea', tables=['GSUB'])
ft_font.save('output.ttf')
```
`fontTools` also provides a couple of command line tools called `ttx` and `fonttools`. `ttx` converts a TTF font into an XML file, which was useful to me because I wanted to rename some glyphs in my font but did not understand anything about fonts. So I just converted my font into an XML file, used `sed` to rename the glyphs, and then used `ttx` again to convert the XML file back into a `ttf`.
`fonttools merge` let me merge my 3 handwriting fonts into 1 so that I had all the glyphs I needed in 1 file.
### the code
I put my extremely hacky code for doing this in a repository called [font-mixer][8]. It’s like 33 lines of code and I think it’s pretty straightforward. (it’s all in `run.sh` and `combine.py`)
### the results
Here’s a small sample the old font and the new font. I don’t think the new font “feels” that much more like handwriting – there’s a little more variation, but it still doesn’t compare to actual handwritten text (at the bottom).
It feels a little uncanny valley to me, like it’s obviously still a font but it’s pretending to be something else.
![][9]
And here’s a sample of the same text actually written by hand:
![][10]
It’s possible that the results would be better if I was more careful about how I made the 2 other handwriting fonts I mixed the original font with.
### it’s cool that it’s so easy to add opentype rules!
Mostly what was delightful to me here is that it’s so easy to add OpenType rules to change how fonts work, like you can pretty easily make a font where the word “the” is always replaced with “teh” (typos all the time!).
I still don’t know how to make a more realistic handwriting font though :). I’m still using the old one (without the extra variations) and I’m pretty happy with it.
--------------------------------------------------------------------------------
via: https://jvns.ca/blog/2020/08/08/handwritten-font/
作者：[Julia Evans][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://jvns.ca/
[b]: https://github.com/lujun9972
[1]: https://2ttf.com/
[2]: https://jvns.ca/images/font-sample-connect.png
[3]: http://bangbangcon.com/recordings.html
[4]: https://thume.ca/numderline/
[5]: https://blog.janestreet.com/commas-in-big-numbers-everywhere/
[6]: https://www.youtube.com/watch?v=Biqm9ndNyC8
[7]: https://adobe-type-tools.github.io/afdko/OpenTypeFeatureFileSpecification.html
[8]: https://github.com/jvns/font-mixer/
[9]: https://jvns.ca/images/font-mixer-comparison.png
[10]: https://jvns.ca/images/handwriting-sample.jpeg

110
sources/tech/20200809 Open Source Drawing App Pinta Sees New Release After 5 Years. Here-s How to Get it.md Normal file
View File

@ -0,0 +1,110 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Open Source Drawing App Pinta Sees New Release After 5 Years. Here’s How to Get it!)
[#]: via: (https://itsfoss.com/pinta-new-release/)
[#]: author: (Abhishek Prakash https://itsfoss.com/author/abhishek/)
Open Source Drawing App Pinta Sees New Release After 5 Years. Here’s How to Get it!
======
_**Brief: Open source painting and drawing application has a new release after more than 5 years. The new release fixes numerous bugs and adds new features.**_
[Pinta][1] is an [open source drawing application for Linux][2], Windows and macOS. You can use it for freehand drawing/sketching. You may also use it to add arrows, boxes, text etc on an existing image.
![][3]
Pinta version 1.7 was released a few days ago after a gap of almost five years. Let’s see what do we have in the new version.
### New features in Pinta 1.7
Here are the new features the latest version of Pinta brings:
* Tab view to switch between images
* Addition of a Smooth Erase tool
* Drag and drop URL to download and open the image in Pinta for editing
* The Pencil tool can switch between different blend modes
* ‘Move Selected’ tool can be scaled by holding Ctrl
* The Rectangle Select tool now shows different arrow cursors at each corner of the selection
* Performance improvements when interacting with selections, particularly for large images
There are numerous bug fixes as well and this should improve the overall Pinta experience. You can learn about more changes in the [official release note][4].
### Installing Pinta 1.7 on Ubuntu and other Linux distributions
For Ubuntu and Ubuntu-based Linux distributions, there is an [official PPA available][5]. If you are using Ubuntu 18.04 or 20.04, you can use this PPA.
Open a terminal and use the following command to add the new repository. You’ll be asked to enter your password. You probably already know that when you type password in terminal, nothing is displayed. Just type the password and press enter.
```
sudo add-apt-repository ppa:pinta-maintainers/pinta-stable
```
Not required on Ubuntu 18.04 and higher version anymore but some other distributions may need to update the cache:
```
sudo apt update
```
Now install the latest version of Pinta using this command.
```
sudo apt install pinta
```
Good thing here is that if you had Pinta 1.6 installed previously, it gets updated to the new version.
#### Removing Pinta
To remove Pinta installed via PPA, use this command:
```
sudo apt remove pinta
```
You should also [delete the PPA][6]:
```
sudo add-apt-repository -r ppa:pinta-maintainers/pinta-stable
```
#### Installing Pinta on other distributions
When I last checked Pinta 1.7 wasn’t available for Fedora or AUR yet. You may wait for some time or download the source code and try it on your own.
[Download Pinta 1.7][7]
Pinta team has also refreshed its user guide with detailed documentation. If you are new to Pinta or if you want to use it extensively, you may refer to [this user guide][8] for reference.
**Conclusion**
I am happy to see a new release of Pinta. It was my go-to tool for editing images to add arrows, boxes and text for tutorials on It’s FOSS. I use [Shutter][9] for this purpose these days but with this new release, I might perhaps switch to Pinta again.
What about you? Do you use Pinta or have you used it in the past? Are you looking forward to use the new version?
--------------------------------------------------------------------------------
via: https://itsfoss.com/pinta-new-release/
作者：[Abhishek Prakash][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://itsfoss.com/author/abhishek/
[b]: https://github.com/lujun9972
[1]: https://www.pinta-project.com/
[2]: https://itsfoss.com/open-source-paint-apps/
[3]: https://i2.wp.com/itsfoss.com/wp-content/uploads/2020/08/pinta-1-7.png?resize=800%2C517&ssl=1
[4]: https://www.pinta-project.com/releases/1-7
[5]: https://launchpad.net/~pinta-maintainers/+archive/ubuntu/pinta-stable
[6]: https://itsfoss.com/how-to-remove-or-delete-ppas-quick-tip/
[7]: https://www.pinta-project.com/releases/
[8]: https://www.pinta-project.com/user-guide/
[9]: https://itsfoss.com/install-shutter-ubuntu/

662
translated/tech/20200318 Share data between C and Python with this messaging library.md
View File

@ -1,662 +0,0 @@
[#]: collector: (lujun9972)
[#]: translator: (silentdawn-zz)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Share data between C and Python with this messaging library)
[#]: via: (https://opensource.com/article/20/3/zeromq-c-python)
[#]: author: (Cristiano L. Fontana https://opensource.com/users/cristianofontana)
使用 ZeroMQ 消息库在 C 和 Python 间共享数据
======
ZeroMQ 是一个快速灵活的消息库，用于数据收集和不同语言间的数据共享。
![Chat via email][1]
作为软件工程师，我有多次在要求完成指定任务时感到毛骨悚然的经历。其中一次是要写个接口，用于控制一些新的底层硬件间的对接，接口实现形式分别是 C 和 云底层组件，而后者主要是 Python。
实现的方式之一是 [用 C 写扩展模块][2]，Python 支持 C 扩展的调用。快速浏览文档后发现，这需要编写大量的 C 代码。这样做的话，在有些情况下效果还不错，但不是我想要的方式。另一种方式就是将两个任务放在不同的进程中，以使用 [ZeroMQ 消息库][3] 在进程间交换信息的方式实现数据的共享。
在引入 ZeroMQ 之前，我选择了编写扩展的方式，试图解决这个场景的需求。这种方式不算太差，但非常复杂和费时。现在为了避免那些问题，我细分出一个系统作为独立的进程运行，专门用于交换通过 [通信套接字][4] 发送的消息所承载的数据。这样，不同的编程语言可以共存，每个进程也变简单了，同时也容易调试。
ZeroMQ 提供了更简单的进程：
1. 编写一小组 C 代码，从硬件读取数据，发送到它能作用到的消息中。
2. 使用 Python 编写接口，实现新旧基础组件的对接。
[Pieter Hintjens][5] 是 ZeroMQ 项目发起者之一，他是个拥有 [有趣视角和作品][6] 的非凡人物。
### 准备
本教程中，需要：
* 一个 C 编译器（例如 [GCC][7] 或 [Clang][8]）
* [**libzmq** 库][9]
* [Python 3][10]
* [ZeroMQ 的 Python 封装][11]
Fedora 系统上的安装方法：
```
$ dnf install clang zeromq zeromq-devel python3 python3-zmq
```
Debian 和 Ubuntu 系统上的安装方法：
```
$ apt-get install clang libzmq5 libzmq3-dev python3 python3-zmq
```
如果有问题，参考对应项目的安装指南（上面附有链接）。
### 编写硬件接口库
因为这里针对的是个设想的场景，本教程虚构了包含两个函数的操作库：
* **fancyhw_init()** 用来初始化（设想的）硬件
* **fancyhw_read_val()** 用于返回从硬件读取的数据
将库的完整代码保存到文件 **libfancyhw.h** 中:
```
#ifndef LIBFANCYHW_H
#define LIBFANCYHW_H
#include &lt;stdlib.h&gt;
#include &lt;stdint.h&gt;
// This is the fictitious hardware interfacing library
void fancyhw_init(unsigned int init_param)
{
    [srand][12](init_param);
}
int16_t fancyhw_read_val(void)
{
    return (int16_t)[rand][13]();
}
#endif
```
这个库可以模拟你要在不同语言实现的组件间交换的数据，中间有劳随机数发生器。
### 设计 C 接口
下面从包含管理数据传输的库开始，逐步实现 C 接口。
#### 需要的库
开始先加载必要的库（每个库的作用见代码注释）：
```
// For printf()
#include &lt;stdio.h&gt;
// For EXIT_*
#include &lt;stdlib.h&gt;
// For memcpy()
#include &lt;string.h&gt;
// For sleep()
#include &lt;unistd.h&gt;
#include &lt;zmq.h&gt;
#include "libfancyhw.h"
```
#### 必要的参数
定义 **main** 函数和后续过程中必要的参数：
```
int main(void)
{
    const unsigned int INIT_PARAM = 12345;
    const unsigned int REPETITIONS = 10;
    const unsigned int PACKET_SIZE = 16;
    const char *TOPIC = "fancyhw_data";
    ...
```
#### 初始化
所有的库都需要初始化。虚构的那个只需要一个参数：
```
fancyhw_init(INIT_PARAM);
```
ZeroMQ 库需要实打实的初始化。首先，定义对象 **context**，它是用来管理全部的套接字的：
```
void *context = zmq_ctx_new();
if (!context)
{
    [printf][14]("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
    return EXIT_FAILURE;
}
```
之后定义用来发送数据的套接字。ZeroMQ 支持若干种套接字，各有其用。使用 **publish** 套接字（也叫 **PUB** 套接字），可以复制消息并分发到多个接收端。这使得你可以让多个接收端接收同一个消息。没有接收者的消息将被丢弃（即不会入消息队列）。用法如下：
```
void *data_socket = zmq_socket(context, ZMQ_PUB);
```
套接字需要绑定到一个具体的地址，这样客户端就知道要连接哪里了。本例中，使用了 [TCP 传输层][15]（当然也有 [其它选项][16]，但 TCP 是不错的默认选择）：
```
const int rb = zmq_bind(data_socket, "tcp://*:5555");
if (rb != 0)
{
    [printf][14]("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
    return EXIT_FAILURE;
}
```
下一步, 计算一些后续要用到的值。 注意下面代码中的 **TOPIC**，因为 **PUB** 套接字发送的消息需要绑定一个 topic。topic 用于供接收者过滤消息：
```
const size_t topic_size = [strlen][17](TOPIC);
const size_t envelope_size = topic_size + 1 + PACKET_SIZE * sizeof(int16_t);
[printf][14]("Topic: %s; topic size: %zu; Envelope size: %zu\n", TOPIC, topic_size, envelope_size);
```
#### 发送消息
启动一个 **重复** 发送消息的循环：
```
for (unsigned int i = 0; i &lt; REPETITIONS; i++)
{
    ...
```
发送消息前，先填充一个长度为 **PACKET_SIZE** 的缓冲区。本库提供的是 16 bits 有符号整数。因为 C 语言中 **int** 类型占用空间大小与平台相关，不是确定的值，所以要使用指定宽度的 **int** 变量：
```
int16_t buffer[PACKET_SIZE];
for (unsigned int j = 0; j &lt; PACKET_SIZE; j++)
{
    buffer[j] = fancyhw_read_val();
}
[printf][14]("Read %u data values\n", PACKET_SIZE);
```
消息的准备和发送中，第一步是创建 ZeroMQ 消息，为消息分配必要的内存空间。空白的消息是用于封装要发送的数据的：
```
zmq_msg_t envelope;
const int rmi = zmq_msg_init_size(&amp;envelope, envelope_size);
if (rmi != 0)
{
    [printf][14]("ERROR: ZeroMQ error occurred during zmq_msg_init_size(): %s\n", zmq_strerror(errno));
    zmq_msg_close(&amp;envelope);
    break;
}
```
现在内存空间已分配，数据保存在 ZeroMQ 消息“envelope”中。函数 **zmq_msg_data()** 返回一个指向封装数据缓存区顶端的指针。第一部分是 topic，之后是一个空格，最后是二进制数。topic 和二进制数据之间的分隔符采用空格字符。需要遍历缓存区的话，使用 cast 和 [指针算法][18]。（感谢 C 语言，让事情变得直截了当。）做法如下：
```
[memcpy][19](zmq_msg_data(&amp;envelope), TOPIC, topic_size);
[memcpy][19]((void*)((char*)zmq_msg_data(&amp;envelope) + topic_size), " ", 1);
[memcpy][19]((void*)((char*)zmq_msg_data(&amp;envelope) + 1 + topic_size), buffer, PACKET_SIZE * sizeof(int16_t));
```
通过 **data_socket** 发送消息：
```
const size_t rs = zmq_msg_send(&amp;envelope, data_socket, 0);
if (rs != envelope_size)
{
    [printf][14]("ERROR: ZeroMQ error occurred during zmq_msg_send(): %s\n", zmq_strerror(errno));
    zmq_msg_close(&amp;envelope);
    break;
}
```
使用数据之前要先解取封装：
```
zmq_msg_close(&amp;envelope);
[printf][14]("Message sent; i: %u, topic: %s\n", i, TOPIC);
```
#### 清场
C 语言不提供 [垃圾收集][20] 功能，用完之后记得要自己扫尾。发送消息之后结束程序之前，需要运行扫尾代码，释放分配的内存:
```
const int rc = zmq_close(data_socket);
if (rc != 0)
{
    [printf][14]("ERROR: ZeroMQ error occurred during zmq_close(): %s\n", zmq_strerror(errno));
    return EXIT_FAILURE;
}
const int rd = zmq_ctx_destroy(context);
if (rd != 0)
{
    [printf][14]("Error occurred during zmq_ctx_destroy(): %s\n", zmq_strerror(errno));
    return EXIT_FAILURE;
}
return EXIT_SUCCESS;
```
#### 完整 C 代码
保存下面完整的接口代码到本地名为 **hw_interface.c** 的文件：
```
// For printf()
#include &lt;stdio.h&gt;
// For EXIT_*
#include &lt;stdlib.h&gt;
// For memcpy()
#include &lt;string.h&gt;
// For sleep()
#include &lt;unistd.h&gt;
#include &lt;zmq.h&gt;
#include "libfancyhw.h"
int main(void)
{
    const unsigned int INIT_PARAM = 12345;
    const unsigned int REPETITIONS = 10;
    const unsigned int PACKET_SIZE = 16;
    const char *TOPIC = "fancyhw_data";
    fancyhw_init(INIT_PARAM);
    void *context = zmq_ctx_new();
    if (!context)
    {
        [printf][14]("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
        return EXIT_FAILURE;
    }
    void *data_socket = zmq_socket(context, ZMQ_PUB);
    const int rb = zmq_bind(data_socket, "tcp://*:5555");
    if (rb != 0)
    {
        [printf][14]("ERROR: ZeroMQ error occurred during zmq_ctx_new(): %s\n", zmq_strerror(errno));
        return EXIT_FAILURE;
    }
    const size_t topic_size = [strlen][17](TOPIC);
    const size_t envelope_size = topic_size + 1 + PACKET_SIZE * sizeof(int16_t);
    [printf][14]("Topic: %s; topic size: %zu; Envelope size: %zu\n", TOPIC, topic_size, envelope_size);
    for (unsigned int i = 0; i &lt; REPETITIONS; i++)
    {
        int16_t buffer[PACKET_SIZE];
        for (unsigned int j = 0; j &lt; PACKET_SIZE; j++)
        {
            buffer[j] = fancyhw_read_val();
        }
        [printf][14]("Read %u data values\n", PACKET_SIZE);
        zmq_msg_t envelope;
   
        const int rmi = zmq_msg_init_size(&amp;envelope, envelope_size);
        if (rmi != 0)
        {
            [printf][14]("ERROR: ZeroMQ error occurred during zmq_msg_init_size(): %s\n", zmq_strerror(errno));
   
            zmq_msg_close(&amp;envelope);
   
            break;
        }
       
        [memcpy][19](zmq_msg_data(&amp;envelope), TOPIC, topic_size);
        [memcpy][19]((void*)((char*)zmq_msg_data(&amp;envelope) + topic_size), " ", 1);
        [memcpy][19]((void*)((char*)zmq_msg_data(&amp;envelope) + 1 + topic_size), buffer, PACKET_SIZE * sizeof(int16_t));
   
        const size_t rs = zmq_msg_send(&amp;envelope, data_socket, 0);
        if (rs != envelope_size)
        {
            [printf][14]("ERROR: ZeroMQ error occurred during zmq_msg_send(): %s\n", zmq_strerror(errno));
   
            zmq_msg_close(&amp;envelope);
   
            break;
        }
   
        zmq_msg_close(&amp;envelope);
        [printf][14]("Message sent; i: %u, topic: %s\n", i, TOPIC);
        sleep(1);
    }
    const int rc = zmq_close(data_socket);
    if (rc != 0)
    {
        [printf][14]("ERROR: ZeroMQ error occurred during zmq_close(): %s\n", zmq_strerror(errno));
        return EXIT_FAILURE;
    }
    const int rd = zmq_ctx_destroy(context);
    if (rd != 0)
    {
        [printf][14]("Error occurred during zmq_ctx_destroy(): %s\n", zmq_strerror(errno));
        return EXIT_FAILURE;
    }
    return EXIT_SUCCESS;
}
```
用如下命令编译：
```
$ clang -std=c99 -I. hw_interface.c -lzmq -o hw_interface
```
如果没有编译错误，你就可以运行这个接口了。贴心的是，ZeroMQ **PUB** 套接字可以在没有任何应用发送或接受数据的状态下运行，这简化了使用复杂度，因为这样不限制进程启动的次序。
运行该接口：
```
$ ./hw_interface
Topic: fancyhw_data; topic size: 12; Envelope size: 45
Read 16 data values
Message sent; i: 0, topic: fancyhw_data
Read 16 data values
Message sent; i: 1, topic: fancyhw_data
Read 16 data values
...
...
```
输出显示数据已经通过 ZeroMQ 完成发送，现在要做的是让一个程序去读数据。
### 编写 Python 数据处理器
现在已经准备好从 C 程序向 Python 应用发送数据了。
#### 库
需要两个库帮助实现数据传输。首先是 ZeroMQ 的 Python 封装：
```
$ python3 -m pip install zmq
```
另一个就是 [**struct** 库][21]，用于解码二进制数据。这个库是 Python 标准库的一部分，所以不需要使用 **pip** 命令安装。
Python 程序的第一部分是导入这些库：
```
import zmq
import struct
```
#### 重要参数
使用 ZeroMQ 时，只能向常量 **TOPIC** 定义相同的接收端发送消息：
```
topic = "fancyhw_data".encode('ascii')
print("Reading messages with topic: {}".format(topic))
```
#### 初始化
下一步，初始化上下文和套接字。使用 **subscribe** 套接字（也称为 **SUB** 套接字），它是 **PUB** 套接字的天生好友。这个套接字发送时也需要匹配 topic。
```
with zmq.Context() as context:
    socket = context.socket(zmq.SUB)
    socket.connect("tcp://127.0.0.1:5555")
    socket.setsockopt(zmq.SUBSCRIBE, topic)
    i = 0
    ...
```
#### 接收消息
启动一个无限循环，等待接收发送到 SUB 套接字的新消息。这个循环会在你按下 **Ctrl+C** 组合键或者内部发生错误时终止：
```
    try:
        while True:
            ... # we will fill this in next
    except KeyboardInterrupt:
        socket.close()
    except Exception as error:
        print("ERROR: {}".format(error))
        socket.close()
```
这个循环等待 **recv()** 方法获取的新消息，然后将接收到的内容从第一个空格字符处分割开，从而得到 topic：
```
binary_topic, data_buffer = socket.recv().split(b' ', 1)
```
#### 解码消息
Python 此时尚不知道 topic 是个字符串，使用标准 ASCII 编解码器进行解码：
```
topic = binary_topic.decode(encoding = 'ascii')
print("Message {:d}:".format(i))
print("\ttopic: '{}'".format(topic))
```
下一步就是使用 **struct** 库读取二进制数据，它可以将二进制数据段转换为明确的数值。首先，计算数据包中数值的组数。本例中使用的 16 bit 有符号整数对应的是 **struct** [格式字符][22] 中的“h”：
```
packet_size = len(data_buffer) // struct.calcsize("h")
print("\tpacket size: {:d}".format(packet_size))
```
知道数据包中有多少组数据后，就可以通过构建一个包含数据组数和数据类型的字符串，来定义格式了（比如“**16h**”）：
```
struct_format = "{:d}h".format(packet_size)
```
将二进制数据串转换为可直接打印的一系列数字：
```
data = struct.unpack(struct_format, data_buffer)
print("\tdata: {}".format(data))
```
#### 完整 Python 代码
下面是 Python 实现的完整的接收器：
```
#! /usr/bin/env python3
import zmq
import struct
topic = "fancyhw_data".encode('ascii')
print("Reading messages with topic: {}".format(topic))
with zmq.Context() as context:
    socket = context.socket(zmq.SUB)
    socket.connect("tcp://127.0.0.1:5555")
    socket.setsockopt(zmq.SUBSCRIBE, topic)
    i = 0
    try:
        while True:
            binary_topic, data_buffer = socket.recv().split(b' ', 1)
            topic = binary_topic.decode(encoding = 'ascii')
            print("Message {:d}:".format(i))
            print("\ttopic: '{}'".format(topic))
            packet_size = len(data_buffer) // struct.calcsize("h")
            print("\tpacket size: {:d}".format(packet_size))
            struct_format = "{:d}h".format(packet_size)
            data = struct.unpack(struct_format, data_buffer)
            print("\tdata: {}".format(data))
            i += 1
    except KeyboardInterrupt:
        socket.close()
    except Exception as error:
        print("ERROR: {}".format(error))
        socket.close()
```
将上面的内容保存到名为 **online_analysis.py** 的文件。Python 代码不需要编译，你可以直接运行它。
运行输出如下：
```
$ ./online_analysis.py
Reading messages with topic: b'fancyhw_data'
Message 0:
        topic: 'fancyhw_data'
        packet size: 16
        data: (20946, -23616, 9865, 31416, -15911, -10845, -5332, 25662, 10955, -32501, -18717, -24490, -16511, -28861, 24205, 26568)
Message 1:
        topic: 'fancyhw_data'
        packet size: 16
        data: (12505, 31355, 14083, -19654, -9141, 14532, -25591, 31203, 10428, -25564, -732, -7979, 9529, -27982, 29610, 30475)
...
...
```
### 小结
本教程介绍了一种新方式，实现从基于 C 的硬件接口收集数据，并分发到基于 Python 的基础组件的功能。借此可以获取数据供后续分析，或者转送到任意数量的接收端去。它采用了一个消息库实现数据在发送者和处理者之间的传送，来取代同样功能规模庞大的软件。
本教程还引出了我称之为“软件粒度”的概念，换言之，就是将软件细分为更小的部分。这种做法的优点之一就是，使得同时采用不同的编程语言实现最简接口作为不同部分之间沟通的组件成为可能。
实践中，这种设计使得软件工程师能以更独立、合作更高效的方式做事。不同的团队可以专注于数据分析的不同方面，可以选择自己中意的实现工具。这种做法的另一个优点是实现了零代价的并行，因为所有的进程都可以并行运行。[ZeroMQ 消息库][3] 是个令人赞叹的软件，使用它可以让工作大大简化。
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/3/zeromq-c-python
作者：[Cristiano L. Fontana][a]
选题：[lujun9972][b]
译者：[silentdawn-zz](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/cristianofontana
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/email_chat_communication_message.png?itok=LKjiLnQu (Chat via email)
[2]: https://docs.python.org/3/extending/extending.html
[3]: https://zeromq.org/
[4]: https://en.wikipedia.org/wiki/Network_socket
[5]: https://en.wikipedia.org/wiki/Pieter_Hintjens
[6]: http://hintjens.com/
[7]: https://gcc.gnu.org/
[8]: https://clang.llvm.org/
[9]: https://github.com/zeromq/libzmq#installation-of-binary-packages-
[10]: https://www.python.org/downloads/
[11]: https://zeromq.org/languages/python/
[12]: http://www.opengroup.org/onlinepubs/009695399/functions/srand.html
[13]: http://www.opengroup.org/onlinepubs/009695399/functions/rand.html
[14]: http://www.opengroup.org/onlinepubs/009695399/functions/printf.html
[15]: https://en.wikipedia.org/wiki/Transmission_Control_Protocol
[16]: http://zguide.zeromq.org/page:all#Plugging-Sockets-into-the-Topology
[17]: http://www.opengroup.org/onlinepubs/009695399/functions/strlen.html
[18]: https://en.wikipedia.org/wiki/Pointer_%28computer_programming%29%23C_and_C++
[19]: http://www.opengroup.org/onlinepubs/009695399/functions/memcpy.html
[20]: https://en.wikipedia.org/wiki/Garbage_collection_(computer_science)
[21]: https://docs.python.org/3/library/struct.html
[22]: https://docs.python.org/3/library/struct.html#format-characters