document/TranslateProject
2
0
Fork 0
mirror of https://github.com/LCTT/TranslateProject.git synced 2025-03-21 02:10:11 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'LCTT/master'

Browse Source
This commit is contained in:
Xingyu Wang 2021-01-06 11:14:13 +08:00
commit d9261f95dc
11 changed files with 1531 additions and 364 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

119
sources/talk/20210105 My Linux Story- How an influential security developer got started in open source.md Normal file
View File

@ -0,0 +1,119 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (My Linux Story: How an influential security developer got started in open source)
[#]: via: (https://opensource.com/article/21/1/lynis)
[#]: author: (Gaurav Kamathe https://opensource.com/users/gkamathe)
My Linux Story: How an influential security developer got started in open source
======
Michael Boelen shares how he strengthened Lynis by building a company
around the open source, security auditing and hardening tool.
![A lock on the side of a building][1]
Michael Boelen is very active in computer security. He is the author of the popular open source security tools Rootkit Hunter ([rkhunter][2]) and [Lynis][3], and he blogs about Linux security on [Linux Audit][4] and evaluates security tools on [Linux Security Expert][5]. He also formed a company named [CISOfy][6] around the Lynis tool to help organizations with security audits of multiple operating systems.
[Having used Lynis][7], I was eager to speak with Michael to learn about his views on Linux, security, and open source software. The following interview has been edited for clarity and conciseness.
**Gaurav: Michael, could you please introduce yourself?**
**Michael:** Hi there! My name is Michael Boelen, and I live in the southern part of The Netherlands with my wife, Debbie, and our son, Hugo. I was born in 1982 and got hooked on computers early when we got a used Commodore 64 from our neighbors. In the beginning, my brother and I played games like [Boulder Dash][8] on it. Then I discovered you could create your own programs in BASIC. It was a magical feeling to make something out of nothing. At the age of 10, my programs were simple, but the desire to create more had emerged.
Later on, I moved to a 286 with MS-DOS. It did not take long before it got infected with a computer virus. Some viruses were pretty funny, like characters that slowly dropped from the screen. Maybe this is the origin of my interest in malware and information security.
You could always find me behind the computer when I was not at school. We played games on the computers that followed, like a 150MHz Intel Pentium and later a 450MHz Intel Pentium III. When I was alone, I tried to figure out how things worked, including programming and learning how to build computers.
Moving quickly forward: I completed school a bit too easily due to circumstances like not having the right modules. For example, I dropped mathematics B because I didn't like it. A few years later, my mentor told me that I could not proceed to higher education due to missing that module. Oops… So I had to continue education below my level. It did not encourage me to study further after completing it, so I went on a job hunt. I got my first full-time job in 2002 at a small company that created a customer relationship management (CRM) solution with some additional web development. My direct manager saw my potential and gave me the freedom to research and optimize or replace existing services. I migrated an old system running Sendmail, created a new DNS server, and developed a custom web interface for both. I was doing DevOps when the word did not exist yet.
After my first full-time job, I switched to a consultancy company. This company served most of the more prominent companies in The Netherlands. It provided me with a few opportunities to work for multinationals like Philips, ASML, and a daughter company of Deutsche Telekom named T-Systems. Work included technical responsibilities for things like Unix administration and data storage. Next in line were roles such as security officer, problem manager, and service manager. That last role taught me more about financial planning, budgeting, servicing internal stakeholders, and dealing with deadlines.
It came time to take a bold step: say farewell to a well-paying job and start my own company. That happened in 2013. The idea was to create a valuable service that used the open source tool Lynis in its core. At the same time, it allowed me to do more development on the tool while allowing me to make a living. Fortunately, things worked out well for me and, more importantly, for the Lynis project.
**Gaurav: You have been associated with open source software and security for quite some time. What projects have you worked on?**
**Michael:** People may know me from two popular software projects named rkhunter and Lynis. I created rkhunter in 2003 to offer an alternative for chkrootkit. As the name implies, it finds traces of rootkits or other malware parts that can exist on a system.
The other tool is Lynis, a security tool to scan Unix-based systems to detect room for improvement when it comes to system hardening.
**Gaurav: How did you get started in open source?**
**Michael:** My real start with open source was in 2003 with rkhunter. I did not have enough shell scripting experience to help the chkrootkit project when I found it showed several false positives on a FreeBSD system. Instead, I decided it was a great opportunity to learn shell scripting and, at the same time, build a useful tool. As a bonus challenge, I thought it was a good idea to make it [POSIX][9] compatible. This way, it could run on Linux, but also on BSD, Solaris, and others.
**Gaurav: What does open source software mean to you?**
**Michael:** Open source is a way to express creativity in software while solving a problem. With the right license, it allows almost anyone to use the software, typically for free. That is also important, as not everyone has the luxury to pay for software or related services. The Dutch are known to be humble, outspoken, and in love with things being "gratis." This word is the same in Latin and means "for thanks" or "for nothing." While the F in FOSS does not refer to this type of free, I believe it is a powerful driver to bring the software into more people's hands. That is valuable in itself, as it can open the gates to more feedback, ideas, or even code improvements.
**Gaurav: What compelled you to develop Lynis?**
**Michael:** In 2007, while being between two jobs for a month, I wanted to start a new project that helped me in my own job of improving security aspects of Linux and Unix-based systems. Existing tooling seemed to be outdated, so I wanted a fresh start.
**Gaurav: How is Lynis different from other security tools available out there? What has been your experience promoting Lynis as an open source tool?**
**Michael:** There are many security tools available. I'm reviewing many of them as part of the Linux Security Expert project. After looking at hundreds of tools, I found that Lynis has something that many of them don't have. The difference is "simplicity," or making the tool as easy to use as possible. From this experience, I can also share that it is really hard to make a tool simple to use.
When it comes to promoting Lynis, I try to diversify things. Sometimes it is writing about it in a blog post or speaking at a conference. At the same time, the value of the tool itself does the real promotion. People tend to share the tool with others because they like it. I strongly believe in the value of promotion. It shows your creation to more people, resulting in increased usage and suggestions. So, if you are a tool author, don't skip this part. You might like to read my blog post _[How to promote your open source project][10]_.
**Gaurav: You also run a popular blog at Linux Audit. Does it complement your learning from Lynis or Linux security in general?**
**Michael:** Yes, the blog is popular and consists of hundreds of articles. Nowadays, I get less time to write, but I hope that will change soon. Blogging is a great way to dive into a difficult subject and turn it into something easier for the reader to understand. The principle of "simplicity" really applies here as well. The blog focuses on Linux security, including the use of Lynis, but also other relevant topics.
**Gaurav: You also formed a company named CISOfy around Lynis and to provide training. Please tell me about these.**
**Michael:** I'm very wary about companies using their open source software as a marketing instrument. That's not how Lynis and CISOfy relate. The Lynis project was there long before the company was founded (2007 and 2013, respectively). Instead of the software being closed, as with some other security companies, the company allowed me to do more active development on the tool and gave it a big boost. I believe that doing good for others returns the favor in unexpected ways.
And I'm also very strict when it comes to Lynis: It must remain open source. That is one of the primary reasons I founded the company.
To ensure that there's a clear line in the sand, we named the commercial part Lynis Enterprise. It is not an extended version of Lynis but a separate thing that actually requires the Lynis tool. This way, the company relies on the quality of the tool, which gives it a strong motivation to keep investing in Lynis. Everyone wins, as both the community and customers get a better tool in their hands.
Although textbooks say companies exist to make money and turn a profit, I discovered that a software company can also make a difference. For this reason, I decided several years ago that we should no longer use tools like Google Analytics, and we got rid of tracking technologies. We also limit the data that we collect and store to the bare minimum to conduct business. I dislike spam, so that is also the reason we don't reach out to companies or individuals. You only receive email from us if you requested it or do business with us. Right now, I want CISOfy to become even more proactive in things besides privacy. One is to support more sustainability initiatives, such as becoming CO2 neutral or even absorbing CO2. Things like using solar power, donating to causes that help with replanting of forests, switching a more sustainable bank, and so on.
**Gaurav: As Linux continues to grow, it will likely continue to be a target for malware authors. What do you think?**
**Michael:** We have already seen an increase, even though most attacks still focus on Windows, Android, and Apple-based systems. My guess is that targeting a Linux system has its own challenges, such as the variety in distributions and versions.
**Gaurav: How is ransomware different from regular malware, and how should organizations tackle this threat?**
**Michael:** It is a different threat actor with the primary focus on getting your data and holding it hostage. My suggestion is simple: [make backups and test your restores][11].
**Gaurav: People often want to work on Linux security but have no idea where to start. How do you suggest they get started?**
**Michael:** I would suggest implementing security measures on your personal systems or any test devices that you have available. If you don't know where to start, then Lynis might be a good inspiration for what can be done when it comes to Linux security. You also mentioned the Linux Audit blog, which might help readers get more familiar with the subject.
**Gaurav: Are you working on any interesting new projects or tools?**
**Michael:** Spare time is minimal currently. We recently moved to a new house and are still settling in. When my spare time increases, I will most likely start blogging again and reviewing security tools for the Linux Security Expert website.
**Gaurav: When not working on Lynis or security tooling, what do you do in your spare time?**
**Michael:** It depends a little bit on the season. I like a good walk and drinking a whisky. Then there are the chores at home and in the garden. My current focus is on cleaning up the crawl space under the house to add additional floor insulation. The attic also needs further insulation. Not an average spare-time project, but essential for the comfort level in the future.
### Find out more
If you're interested in learning how to secure your operating system, you will find a wealth of information on Michael's blogs. Also, open source enthusiasts should try Lynis to uncover weaknesses in their systems and learn how to mitigate them.
--------------------------------------------------------------------------------
via: https://opensource.com/article/21/1/lynis
作者：[Gaurav Kamathe][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/gkamathe
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/BUSINESS_3reasons.png?itok=k6F3-BqA (A lock on the side of a building)
[2]: http://rkhunter.sourceforge.net/
[3]: https://cisofy.com/lynis/
[4]: https://linux-audit.com/
[5]: https://linuxsecurity.expert/
[6]: https://cisofy.com/
[7]: https://opensource.com/article/20/5/linux-security-lynis
[8]: https://en.wikipedia.org/wiki/Boulder_Dash
[9]: https://opensource.com/article/19/7/what-posix-richard-stallman-explains
[10]: https://linux-audit.com/how-to-promote-your-open-source-project/
[11]: https://opensource.com/article/19/3/backup-solutions

290
sources/tech/20201222 Learn to use the Sed text editor.md
View File

@ -1,290 +0,0 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Learn to use the Sed text editor)
[#]: via: (https://opensource.com/article/20/12/sed)
[#]: author: (Seth Kenlon https://opensource.com/users/seth)
Learn to use the Sed text editor
======
Sed lacks the usual text boxes and instead writes directly on a file,
directed by user commands.
![Command line prompt][1]
Created for version 7 of AT&T’s original Unix operating system, the `sed` command has been included with probably every Unix and Linux OS since. The `sed` application is a _stream editor_, and unlike a text editor it doesn’t open a visual buffer into which a file’s data is loaded for processing. Instead, it operates on a file, line by line, according to either a command typed into a terminal or a series of commands in a script.
### Installing
If you’re using Linux, BSD, or macOS, then you already have GNU or BSD `sed` installed. These are two unique reimplementations of the original `sed` command, and while they’re similar, there can be minor differences. GNU `sed` is generally regarded to be the most feature-rich `sed` out there, and it’s widely available on any of these platforms.
If you can’t find GNU `sed` (often called `gsed` on non-Linux systems), then you can [download its source code from the GNU website][2]. The nice thing about having GNU `sed` installed is that it can be used for its extra functions, but it can also be constrained to conform to just the [POSIX][3] specifications of `sed`, should you require portability.
On Windows, you can [install][4] GNU `sed` with [Chocolatey][5].
### How Sed works
The `sed` application works on one line at a time. Because it has no visual display, it creates a pattern space—a space in memory containing the current line from the input stream (with any trailing newline character removed). Once the pattern space is populated, your instructions to `sed` are executed. Sometimes your commands are conditional, and other times they are absolute, so the results of these commands depend on how you’re using `sed`.
When the end of commands is reached, `sed` prints the contents of the pattern space to the output stream. The default output stream is **stdout**, but it can be redirected to a file or even back into the same file using the `--in-place=.bak` option.
Then the cycle begins again with the next input line.
The syntax for the `sed` command is:
```
`$ sed --options [optional SCRIPT] [INPUT FILE or STREAM]`
```
#### Finding what you want to edit
In a visual editor, you usually locate what you want to change in a text file without thinking much about it. Your eye (or screen reader) scans the text, finds the word you want to change or the place you want to insert or remove text, and then you just start typing. There is no interactive mode for `sed`, though, so you must tell it what conditions must be met for it to run specific commands.
For these examples, assume that a file called `example.txt` contains this text:
```
hello
world
This is line three.
Here is the final line.
```
#### Line number
Specifying a line number tells `sed` to operate only on that specific line in the file.
For instance, this command selects line 1 of a file and prints it. Because `sed`’s default action after processing is also to print a line to **stdout**, this has the effect of duplicating the first line:
```
$ sed ‘1p’ example.txt
hello
hello
world
This is line three.
Here is the final line.
```
You can specify line numbers in steps, too. For instance, `1~2` indicates that every 2 lines are selected ("select every second line starting with the first"). The instruction `1~3` means to select every third line after the first:
```
$ sed ‘1p’ example.txt
hello
hello
world
This is line three.
Here is the final line.
Here is the final line.
```
#### Line position
You can operate only on the last line of a file by using `$` as a selector:
```
$ sed ‘$p’ example.txt
hello
world
This is line three.
Here is the final line.
Here is the final line.
```
In GNU `sed`, you can select more than one line (`sed '1,$p'` prints the first and final line, for example).
#### Negation
Any selection by number or position, you can invert with the exclamation mark (`!`) character. This selects all lines _except_ the first line:
```
$ sed ‘1!p’ example.txt
hello
world
world
This is line three.
This is line three.
Here is the final line.
Here is the final line.
```
#### Pattern matching
You can think of a pattern match as a **find** operation in a word processor or a browser. You provide a word (a _pattern_), and the results are selected. The syntax for a pattern match is `/pattern/`.
```
$ sed ‘/hello/p’ example.txt
hello
hello
world
This is line three.
Here is the final line.
$ sed ‘/line/p’ example.txt
hello
world
This is line three.
This is line three.
Here is the final line.
Here is the final line.
```
### Editing with Sed
Once you’ve found what you want to edit, you can perform whatever action you want. You perform edits with `sed` with commands. Commands in `sed` are different from the `sed` command itself. If it helps, think of them as "actions" or "verbs" or "instructions."
Commands in `sed` are single letters, such as the `p` for **print** command used in previous examples. They can be difficult to recall at first, but as with everything, you get to know them with practice.
#### p for print
The `p` instruction prints whatever is currently held in pattern space.
#### d for delete
The `d` instruction deletes the pattern space.
```
$ sed ‘$d’ example.txt
hello
world
This is line three.
$ sed ‘1d’ example.txt
world
This is line three.
Here is the final line.
```
#### s for search and replace
The `s` command searches for a pattern and replaces it with something else. This is probably the most popular and casual use for `sed`, and it’s often the first (and sometimes the only) `sed` command a user learns. It’s almost certainly the most useful command for text editing.
```
$ sed ‘s/world/opensource.com/’
hello
opensource.com
This is line three.
Here is the final line.
```
There are special functions you can use in your replacement text, too. For instance, `\L` transforms the replacement text to lowercase and `\l` does the same for just the next character. There are others, which are listed in the `sed` documentation (you can view that with the `info sed` command).
The special character `&` in the replacement clause refers to the matched pattern:
```
$ sed ‘s/is/\U&/’ example.txt
hello
world
ThIS is line three.
Here IS the final line.
```
You can also pass special flags to affect how `s` processes what it finds. The `g` (for _global_, presumably) flag tells `s` to apply the replacement to all matches found on the line and not just the first match:
```
$ sed ‘s/is/\U&/g’ example.txt
hello
world
ThIS IS line three.
Here IS the final line.
```
Other important flags include a number to indicate which occurrence of a matched pattern to affect:
```
$ sed ‘s/is/\U&/2’ example.txt
hello
world
This IS line three.
Here is the final line.
```
The `w` flag, followed by a filename, writes a matched line to a file _only if_ a change is made:
```
$ sed ‘s/is/\U&/w sed.log’ example.txt
hello
world
ThIS is line three.
Here IS the final line.
$ cat sed.log
ThIS is line three.
Here IS the final line.
```
Flags can be combined:
```
$ sed ‘s/is/\U&/2w sed.log’ example.txt
hello
world
This IS line three.
Here is the final line.
$ cat sed.log
This IS line three.
```
### Scripts
There are lots of great sites out there with `sed` "one-liners." They give you task-oriented `sed` commands to solve common problems. However, learning `sed` for yourself enables you to write your own one-liners, and those can be tailored to your specific needs.
Scripts for `sed` can be written as lines in a terminal, or they can be saved to a file and executed with `sed` itself. I tend to write small scripts all as one command because I find myself rarely re-using `sed` commands in real life. When I write a `sed` script, it’s usually very specific to one file. For example, after writing the initial draft of this very article, I used `sed` to standardize the capitalization of "sed", and that’s a task I’ll probably never have to do again.
You can issue a series of distinct commands to `sed` separated by a semicolon (`;`).
```
$ sed ‘3t ; s/line/\U&/’ example.txt
hello
world
This is LINE three.
This is the final line.
```
### Scope changes with braces
You can also limit which results are affected with braces (`{}`). When you enclose `sed` commands in braces, they apply only to a specific selection. For example, the word "line" appears in two lines of the sample text. You can force `sed` to affect only the final line by declaring the required match condition (`$` to indicate the final line) and placing the `s` command you want to be performed in braces immediately thereafter:
```
$ sed ‘$ {s/line/\U&/}’ example.txt
hello
world
This is line three.
This is the final LINE.
```
### Learn Sed
You can do a lot more with `sed` than what’s explained in this article. I haven’t even gotten to branching (`b`), tests (`t`), the _hold_ space (`H`), and many other features. Like [`ed`][6], `sed` probably isn’t the text editor you’re going to use for document creation or even for every scripted task you need doing, but it is a powerful option you have as a POSIX user. Learning how `sed` commands are structured and how to write short scripts can make for quick changes to massive amounts of text. Read through the `info` pages of GNU `sed`, or the man pages of BSD `sed`, and find out what `sed` can do for you.
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/12/sed
作者：[Seth Kenlon][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/seth
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/command_line_prompt.png?itok=wbGiJ_yg (Command line prompt)
[2]: http://www.gnu.org/software/sed/
[3]: https://opensource.com/article/19/7/what-posix-richard-stallman-explains
[4]: https://chocolatey.org/packages/sed
[5]: https://opensource.com/article/20/3/chocolatey
[6]: https://opensource.com/article/20/12/gnu-ed

2
sources/tech/20201224 5 reasons to use the Atom text editor.md
View File

@ -1,5 +1,5 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )

73
sources/tech/20201230 Use the Markdown Editor app in Nextcloud.md
View File

@ -1,73 +0,0 @@
[#]: collector: (lujun9972)
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Use the Markdown Editor app in Nextcloud)
[#]: via: (https://opensource.com/article/20/12/nextcloud-markdown)
[#]: author: (Seth Kenlon https://opensource.com/users/seth)
Use the Markdown Editor app in Nextcloud
======
Nextcloud has one of the smoothest editors of the popular Markdown file
type, with lots of convenient and intuitive features.
![Digital images of a computer desktop][1]
The advantage of plain text is that there’s no extra computer-specific information cluttering up your otherwise human-readable writing. The good thing about computers is that they’re programmable, and so as long as we humans agree to follow very specific conventions when writing, we can program computers to interpret human-readable text as secret instructions. For instance, by surrounding a word with two asterisks, we not only give a visual cue to humans that a word is significant, but we can also program computers to display the word in **bold**.
This is exactly the theory and practice behind [Markdown][2], the popular plain text format that promises writers that as long as _they_ use specific plain text conventions, then their text will be rendered with a specific style.
Traditionally, that has meant that an author writes in plain text and doesn’t see the pretty styling until the text is fed to a converter application (originally `markdown.pl`), but the Markdown Editor app in Nextcloud changes that.
With Nextcloud’s Markdown Editor, you can type in plain text while seeing the style it renders. This is a gamechanger for authors who struggle to remember Markdown’s sometimes confusing notation (do the brackets become before or after the parentheses in a hyperlink?) or who just don’t like the look of plain text. And better yet, it runs in the (Next)cloud, so it’s available to you anywhere.
### Install
To use Nextcloud’s Markdown Editor, you must have an install of Nextcloud. The good news is, Nextcloud is surprisingly _easy_ to install. I’ve installed it on a Raspberry Pi, on a shared server, and even as a local app (which is silly, don’t do it). If you don’t trust your own abilities, you can even rely on [Turnkey Linux][3] to do the hard part for you, or else just purchase managed hosting directly from [Nextcloud.com][4]. Once you have Nextcloud installed, adding apps is simple. Click on your user icon in the top right corner of the Nextcloud interface and select **Apps**. Navigate to the **Office and Text** category and click to install and enable the **Markdown Editor**.
![Nextcloud app store showing Markdown Editor installer][5]
### Launching
After activation, the Markdown Editor gets associated with any file ending in .md in your Nextcloud files, so when you open a Markdown file, you launch Markdown Editor.
### Using Markdown Editor
The Markdown Editor contains a large text field for you to type into and a single toolbar along the top.
![Example markdown file ][6]
The toolbar contains the basic functions of a word processor—styling your text with bold, italics, and strike-through, creating headings and paragraphs, lists, and so on.
Many of these functions get invoked automatically as you type if you know Markdown. If you’re not familiar with Markdown, then the toolbar or the usual keyboard shortcuts (**Ctrl+B** for bold, **Ctrl+I** for italics, and so on) help you style your text.
The great thing about the way Markdown Editor works is that it truly is all things for all users: if you want to type in Markdown, then it accepts that and instantly converts it into visual styling, and if you don’t want to think about Markdown, then it generates the style for you when you use keyboard shortcuts or toolbar buttons. Either way, you never have to see the Markdown syntax, but you also never lose it. It’s the perfect compromise.
It’s an awfully smart editor, too. It offers to create a hyperlink for you when you select a word, it auto-converts Markdown quickly and smoothly, and it knows a few different "flavors" of Markdown syntax (Commonmark primarily, but also traditional Markdown, Github Markdown, and so on).
![black text on white background, word highlighted in blue to create an automatic link][7]
### Try Nextcloud
I’ve used a few Markdown preview applications, and Nextcloud’s Markdown Editor is one of the smoothest. It respects its user and does the bare minimum to display Markdown, so its conversion is fast and accurate. Because it’s an app in Nextcloud, you also get the benefit of having your work saved instantly, with version control, on your own private, open source cloud. Text editing doesn’t get much better than that.
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/12/nextcloud-markdown
作者：[Seth Kenlon][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/seth
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/computer_desk_home_laptop_browser.png?itok=Y3UVpY0l (Digital images of a computer desktop)
[2]: https://opensource.com/article/19/9/introduction-markdown
[3]: https://www.turnkeylinux.org/nextcloud
[4]: http://nextcloud.com
[5]: https://opensource.com/sites/default/files/uploads/nextcloud-app-install-31_days-markdown-opensource.jpg (Nextcloud app store showing Markdown Editor installer)
[6]: https://opensource.com/sites/default/files/uploads/nextcloud-markdown-31-days-opensource.jpg (Example markdown file )
[7]: https://opensource.com/sites/default/files/uploads/nextcloud-link-31_days_markdown-opensource.jpg (black text on white background, word highlighted in blue to create an automatic link)

83
sources/tech/20210101 Resize images using Python.md Normal file
View File

@ -0,0 +1,83 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Resize images using Python)
[#]: via: (https://opensource.com/life/15/2/resize-images-python)
[#]: author: (Dayo Ntwari https://opensource.com/users/dayontwari)
Resize images using Python
======
A quick explanation of how to resize images in Python while keeping the
same aspect ratio.
![Python in a tree][1]
I love [Python][2], and I've been learning it for a while now. Some time ago, I wrote a Python script where I needed to resize a bunch of images while at the same time keeping the aspect ratio (the proportions) intact. So I looked around and found [Pillow][3], a Python imaging library and "friendly fork" of an old library just called PIL. 
To install Pillow, use the `pip` module of Python:
```
`$ python3 -m pip install Pillow`
```
### Scaling by width
Here's a basic script to resize an image using the Pillow module:
```
from PIL import Image
basewidth = 300
img = Image.open('fullsized_image.jpg')
wpercent = (basewidth / float(img.size[0]))
hsize = int((float(img.size[1]) * float(wpercent)))
img = img.resize((basewidth, hsize), Image.ANTIALIAS)
img.save('resized_image.jpg')
```
These few lines of Python code resize an image (**fullsized_image.jpg**) using Pillow to a width of 300 pixels, which is set in the variable **basewidth** and a height proportional to the new width. The proportional height is calculated by determining what percentage 300 pixels is of the original width (**img.size[0]**) and then multiplying the original height (**img.size[1]**) by that percentage. The resulting height value is saved in the variable **hsize.**
You can change **basewidth** to any other number if you need a different width for your images. Also, notice I saved the resized image under a different name, **resized_image.jpg**, because I wanted to preserve the full-size image (**fullsized_image.jpg**) as well. You don't have to do this, of course. You can use the same filename to overwrite the full-size image with the resized image, if that is what you want.
### Scaling by height
If the height is fixed and the width proportionally variable, it's pretty much the same thing, you just need to switch things around a bit:
```
from PIL import Image
baseheight = 560
img = Image.open('fullsized_image.jpg')
hpercent = (baseheight / float(img.size[1]))
wsize = int((float(img.size[0]) * float(hpercent)))
img = img.resize((wsize, baseheight), Image.ANTIALIAS)
img.save('resized_image.jpg')
```
Notice **basewidth** is now **baseheight**, since height is fixed. In the third line, we are calculating the height percentage, so we need **img.size[1]** instead of **img.size[0]**. The size attribute is a tuple containing width and height in pixels; **size[0]** refers to the first tuple element, which is the width, and **size[1]** is the second element, which is height. Line 4 also has this switch between **size[0]** for width and **size[1]** for height.
_Originally published on Dayo Ntwari's [blog][4] and republished under Creative Commons with permission._
_This article was updated in January 2021 by the editor._
--------------------------------------------------------------------------------
via: https://opensource.com/life/15/2/resize-images-python
作者：[Dayo Ntwari][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/dayontwari
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/life-python.jpg?itok=F2PYP2wT (Python in a tree)
[2]: http://python.org/ (Python Programming Language – Official Website)
[3]: https://pypi.org/project/Pillow/ (Python Imaging Library)
[4]: https://dayontwari.wordpress.com/2015/01/06/how-to-resize-images-with-python/

249
sources/tech/20210104 Docker Compose- a nice way to set up a dev environment.md Normal file
View File

@ -0,0 +1,249 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Docker Compose: a nice way to set up a dev environment)
[#]: via: (https://jvns.ca/blog/2021/01/04/docker-compose-is-nice/)
[#]: author: (Julia Evans https://jvns.ca/)
Docker Compose: a nice way to set up a dev environment
======
Hello! Here is another post about [computer tools that I’ve appreciated][1]. This one is about Docker Compose!
This post is mostly just about how delighted I was that it does what it’s supposed to do and it seems to work and to be pretty straightforward to use. I’m also only talking about using Docker Compose for a dev environment here, not using it in production.
I’ve been thinking about this kind of personal dev environment setup more recently because I now do all my computing with a personal cloud budget of like $20/month instead of spending my time at work thinking about how to manage thousands of AWS servers.
I’m very happy about this because previous to trying Docker Compose I spent two days getting frustrated with trying to set up a dev environment with other tools and Docker Compose was a lot easier and simpler. And then I told my sister about my docker-compose experiences and she was like “I KNOW, DOCKER COMPOSE IS GREAT RIGHT?!?!” So I thought I’d write a blog post about it, and here we are.
### the problem: setting up a dev environment
Right now I’m working on a Ruby on Rails service (the backend for a sort of computer debugging game). On my production server, I have:
* a nginx proxy
* a Rails server
* a Go server (which proxies some SSH connections with [gotty][2])
* a Postgres database
Setting up the Rails server locally was pretty straightforward without resorting to containers (I just had to install Postgres and Ruby, fine, no big deal), but then I wanted send `/proxy/*` to the Go server and everything else to the Rails server, so I needed nginx too. And installing nginx on my laptop felt too messy to me.
So enter `docker-compose`!
### docker-compose lets you run a bunch of Docker containers
Docker Compose basically lets you run a bunch of Docker containers that can communicate with each other.
You configure all your containers in one file called `docker-compose.yml`. I’ve pasted my entire `docker-compose.yml` file here for my server because I found it to be really short and straightforward.
```
version: "3.3"
services:
db:
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: password # yes I set the password to 'password'
go_server:
# todo: use a smaller image at some point, we don't need all of ubuntu to run a static go binary
image: ubuntu
command: /app/go_proxy/server
volumes:
- .:/app
rails_server:
build: docker/rails
command: bash -c "rm -f tmp/pids/server.pid && source secrets.sh && bundle exec rails s -p 3000 -b '0.0.0.0'"
volumes:
- .:/app
web:
build: docker/nginx
ports:
- "8777:80" # this exposes port 8777 on my laptop
```
There are two kinds of containers here: for some of them I’m just using an existing image (`image: postgres` and `image: ubuntu`) without modifying it at all. And for some I needed to build a custom container image – `build: docker/rails` says to use `docker/rails/Dockerfile` to build a custom container.
I needed to give my Rails server access to some API keys and things, so `source secrets.sh` puts a bunch of secrets in environment variables. Maybe there’s a better way to manage secrets but it’s just me so this seemed fine.
### how to start everything: `docker-compose build` then `docker-compose up`
I’ve been starting my containers just by running `docker-compose build` to build the containers, then `docker-compose up` to run everything.
You can set `depends_on` in the yaml file to get a little more control over when things start in, but for my set of services the start order doesn’t matter, so I haven’t.
### the networking is easy to use
It’s important here that the containers be able to connect to each other. Docker Compose makes that super simple! If I have a Rails server running in my `rails_server` container on port 3000, then I can access that with `http://rails_server:3000`. So simple!
Here’s a snippet from my nginx configuration file with how I’m using that in practice (I removed a bunch of `proxy_set_header` lines to make it more clear)
```
location ~ /proxy.* {
proxy_pass http://go_server:8080;
}
location @app {
proxy_pass http://rails_server:3000;
}
```
Or here’s a snippet from my Rails project’s database configuration, where I use the name of the database container (`db`):
```
development:
<<: *default
database: myproject_development
host: db # <-------- this "magically" resolves to the database container's IP address
username: postgres
password: password
```
I got a bit curious about how `rails_server` was actually getting resolved to an IP address. It seems like Docker is running a DNS server somewhere on my computer to resolve these names. Here are some DNS queries where we can see that each container has its own IP address:
```
$ dig +short @127.0.0.11 rails_server
172.18.0.2
$ dig +short @127.0.0.11 db
172.18.0.3
$ dig +short @127.0.0.11 web
172.18.0.4
$ dig +short @127.0.0.11 go_server
172.18.0.5
```
### who’s running this DNS server?
I dug into how this DNS server is set up a very tiny bit.
I ran all these commands outside the container, because I didn’t have a lot of networking tools installed in the container.
**step 1**: find the PID of my Rails server with `ps aux | grep puma`
It’s 1837916. Cool.
**step 2**: find a UDP server running in the same network namespace as PID `1837916`
I did this by using `nsenter` to run `netstat` in the same network namespace as the `puma` process. (technically I guess you could run `netstat -tupn` to just show UDP servers, but my fingers only know how to type `netstat -tulpn` at this point)
```
$ sudo nsenter -n -t 1837916 netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:32847 0.0.0.0:* LISTEN 1333/dockerd
tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 1837916/puma 4.3.7
udp 0 0 127.0.0.11:59426 0.0.0.0:* 1333/dockerd
```
So there’s a UDP server running on port `59426`, run by `dockerd`! Maybe that’s the DNS server?
**step 3**: check that it’s a DNS server
We can use `dig` to make a DNS query to it:
```
$ sudo nsenter -n -t 1837916 dig +short @127.0.0.11 59426 rails_server
172.18.0.2
```
But – when we ran `dig` earlier, we weren’t making a DNS query to port 59426, we were querying port 53! What’s going on?
**step 4**: iptables
My first guess for “this server seems to be running on port X but I’m accessing it on port Y, what’s going on?” was “iptables”.
So I ran iptables-save in the container’s network namespace, and there we go:
```
$ sudo nsenter -n -t 1837916 iptables-save
.... redacted a bunch of output ....
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -m udp --sport 59426 -j SNAT --to-source :53
COMMIT
```
There’s an iptables rule that sends traffic on port 53 to 59426. Fun!
### it stores the database files in a temp directory
One nice thing about this is: instead of managing a Postgres installation on my laptop, I can just mount the Postgres container’s data directory at `./tmp/db`.
I like this because I really do not want to administer a Postgres installation on my laptop (I don’t really know how to configure Postgres), and conceptually I like having my dev database literally be in the same directory as the rest of my code.
### I can access the Rails console with `docker-compose exec rails_server rails console`
Managing Ruby versions is always a little tricky and even when I have it working, I always kind of worry I’m going to screw up my Ruby installation and have to spend like ten years fixing it.
With this setup, if I need access to the Rails console (a REPL with all my Rails code loaded), I can just run:
```
$ docker-compose exec rails_server rails console
Running via Spring preloader in process 597
Loading development environment (Rails 6.0.3.4)
irb(main):001:0>
```
Nice!
### small problem: no history in my Rails console
I ran into a problem though: I didn’t have any history in my Rails console anymore, because I was restarting the container all the time.
I figured out a pretty simple solution to this though: I added a `/root/.irbrc` to my container that changed the IRB history file’s location to be something that would persist between container restarts. It’s just one line:
```
IRB.conf[:HISTORY_FILE] = "/app/tmp/irb_history"
```
### I still don’t know how well it works in production
Right now my production setup for this project is still “I made a digitalocean droplet and edited a lot of files by hand”.
I think I’ll try to use docker-compose to run this thing in production. My guess is that it should work fine because this service is probably going to have at most like 2 users at a time and I can easily afford to have 60 seconds of downtime during a deploy if I want, but usually something goes wrong that I haven’t thought of.
A few notes from folks on Twitter about docker-compose in production:
* `docker-compose up` will only restart the containers that need restarting, which makes restarts faster
* there’s a small bash script [wait-for-it][3] that you can use to make a container wait for another service to be available
* You can have 2 docker-compose.yaml files: `docker-compose.yaml` for DEV, and `docker-compose-prod.yaml` for prod. I think I’ll use this to expose different nginx ports: 8999 in dev and 80 in prod.
* folks seemed to agree that docker-compose is fine in production if you have a small website running on 1 computer
* one person suggested that Docker Swarm might be better for a slightly more complicated production setup, but I haven’t tried that (or of course Kubernetes, but the whole point of Docker Compose is that it’s super simple and Kubernetes is certainly not simple :) )
Docker also seems to have a feature to [automatically deploy your docker-compose setup to ECS][4], which sounds cool in theory but I haven’t tried it.
### when doesn’t docker-compose work well?
I’ve heard that docker-compose doesn’t work well:
* when you have a very large number of microservices (a simple setup is best)
* when you’re trying to include data from a very large database (like putting hundreds of gigabytes of data on everyone’s laptop)
* on Mac computers, I’ve heard that Docker can be a lot slower than on Linux (presumably because of the extra VM). I don’t have a Mac so I haven’t run into this.
### that’s all!
I spent an entire day before this trying to configure a dev environment by using Puppet to provision a Vagrant virtual machine only to realize that VMs are kind of slow to start and that I don’t really like writing Puppet configuration (I know, huge surprise :)).
So it was nice to try Docker Compose and find that it was straightforward to get to work!
--------------------------------------------------------------------------------
via: https://jvns.ca/blog/2021/01/04/docker-compose-is-nice/
作者：[Julia Evans][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://jvns.ca/
[b]: https://github.com/lujun9972
[1]: https://jvns.ca/#cool-computer-tools---features---ideas
[2]: https://github.com/yudai/gotty/
[3]: https://github.com/vishnubob/wait-for-it
[4]: https://docs.docker.com/cloud/ecs-integration/

148
sources/tech/20210105 4 lines of code to improve your Ansible play.md Normal file
View File

@ -0,0 +1,148 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (4 lines of code to improve your Ansible play)
[#]: via: (https://opensource.com/article/21/1/improve-ansible-play)
[#]: author: (Jeff Warncia https://opensource.com/users/jeffwarncia)
4 lines of code to improve your Ansible play
======
With a tiny bit of effort, you can help the next person by not just
mapping the safe path but leaving warnings about the dangers.
![A person programming][1]
Out in the blogosphere, which sings the virtues of infrastructure-as-code, continuous integration/continuous delivery (CI/CD) pipelines, code reviews, and testing regimes, it is easy to forget that such a well-engineered ivory tower is an ideal, not a reality. Imperfect systems haunt us, but we must ship something.
There are few towers less ivory than those created by gluing together APIs in the course of system automation. It is a brittle world. The pressures are enormous to get it "working," get it shipped, and move on.
### A problem to solve
Imagine a simple feature request: Write some [Ansible][2] code to create several records in an external system to record some details of a VLAN. I recently had an itch to do some lab administration work to fulfill this task. The external system is a well-known internet protocol address management (IPAM) tool, but the roadblocks are the same for a more abstract configuration management database (CMDB) or a record unrelated to networking. In this example, my immediate desire for creating a record is that alone—the system is only for record-keeping.
If the goal was a hyper-compact, straightforward, and dumb macro, it might work out to be 100 lines of code. If I had the API memorized, I might be able to bang it out in an hour, with the code doing nothing more than expected and leaving nothing but the exact, finished artifact behind. Exactly perfect for its purpose and entirely useless for any future expansion.
Nowadays, I'd expect almost everyone to start this task with a role and several task files, prepared to expand to what will be a dozen or so create, read, update, and delete (CRUD) operations. Because I'm not one of the people who know this API, I may spend between several hours and several days just fooling around with it, figuring out its internal patterns and craftsmanship, bridging the gap between its capabilities and my desires encoded in the code.
While researching the API, I discovered that creating a VLAN record requires a parent object _reference_. This looks something like a path fragment with random characters in it. Maybe it's a hash, or maybe it really is random; I don't know. I imagine that many down in the mud with a looming deadline might just copy and paste this arbitrary string into Ansible and move on with their lives. Ignoring the implementation details of the role, the obvious play-level task would be:
```
\- name: "Create VLAN"
  include_role:
    name: otherthing
    tasks_from: vlan_create.yml
  vars:
    vlan_name: "lab-infra"
    vlan_tag: 100
    vlan_view_ref: "vlan_view/747f602d-0381"
```
Unfortunately, that `vlan_view_ref` identifier isn't available except via the API, so even moving it to an inventory or extra variable doesn't help very much. The playbook's user would need to have some unusual level of understanding of the system to find out the correct reference ID.
In my lab-building situation, I will redeploy this system of record frequently. Therefore, the parent ID will change from day to day, and I don't want to have to figure it out manually each time. So I definitely must search for the reference by name. No problem:
```
\- name: Get Lab vlan view reference
  include_role:
    name: otherthing
    tasks_from: search_for.yml
  vars:
    _resource: vlan_view
    _query: "name={{ vlan_parent_view_name }}"
```
Ultimately, it makes a REST call. This "returns" JSON, which I stuff into `_otherthing_search_result` by convention and for ease of access outside the role. The `search_for.yml` implementation is abstract, and it always returns a dict of zero or more results.
Most Ansible developers, as evidenced by nearly all the real-world Ansible code I've ever read, would proceed as if all is great and directly access the expected single result:
```
\- name: Remember our default vlan view ref
  set_fact:
    _thatthig_vlan_view_ref: "{{ _otherthing_search_result[0]._ref }}"
\- name: "Create VLAN"
  include_role:
    name: otherthing
    tasks_from: vlan_create.yml
  vars:
    vlan_name: "lab-infra"
    vlan_tag: 100
    vlan_view_ref: "{{ vlan_parent_view_name }}"
```
But sometimes `_thatthing_search_result[0]` will be undefined, so `_thatthig_vlan_view_ref` will be undefined. Most likely, this is because the code ran in a different real-world environment and someone forgot to update `{{ vlan_parent_view_name }}` either in the inventory or from the command line. Or, fair or foul, maybe someone went into the tool's graphical user interface (GUI) and deleted the record or changed its name or something.
I know what you're thinking.
_"Well, don't do that. This is a no dumb venue. Be less dumb."_
Maybe I am OK with this situation and retort: "Ansible will tell you quite correctly `The error was: list object has no element 0` and even cough up a line number. What more do you want?" As the developer, of course, I know what that means—I just wrote it. I just came off three days of fooling around with the API. My mind is fresh.
### Tomorrow is another story
But by tomorrow, I'll probably forget what a vlan view reference is, and I'll definitely forget what is on line 30. If it goes wrong in a month, even if you manage to find me, I'll have to take an afternoon to decode the API guide again to figure out what went wrong.
And what if I'm out the door? What if I've turned the code over to an operations team, maybe an intern running it through [Tower][3], hand-feeding `vlan_view_name` into a survey or such? That line 30 is the problem is of no help to them.
Add comments, you say! Well, yes. I could write some terse prose in the code to help the developer next week or next month. This does not help the person running the code and whose job just failed, and it certainly doesn't help the business get done whatever it needs done.
Remember, we are all-powerful in the moment. In writing code or skipping writing code, we do it from a position of strength and knowledge. We have taken hours, or even days, assimilating the documentation, the reality, the other bugs, the other problems, and we have left behind code, comments, and maybe even documentation. We write code that shares that success, and success is what our users want. But there was a lot of failure in that learning; we can leave that behind, too.
### Leave a message in code
`Error on line 30` helps no one. At the absolute minimum, I can handle the obvious error case with a better error message:
```
  - name: Fail if zero vlan views returned
     fail:
       msg: "Got 0 results from searching for VLAN view {{ vlan_parent_view_name }}. Please verify exists in otherthing, and is accessible by the service account."
     when: _otherthing_search_result | length == 0
```
In four lines of code (and zero additional thinking), I have specific, helpful advice to whoever comes next—that hapless operations team member, or more likely me in a month—about the real-world problem that really isn't about the code at all. This message allows anyone to discover a simple copy/paste error or that the system of record changed. No Ansible knowledge needed, no 3am page to a developer to "look at line 30."
But wait! There's more!
In learning about `otherthing`, I learned it was, well, kinda dumb in one critical respect. Many, if not all, of its record types have no uniqueness constraint, and several identical records may exist. A vlan view is defined as having a name, a start ID, and an end ID; other record types are similarly simple and obviously should be a unique tuple—based both on reality and the abstract concept of database normalization. But `otherthing` allows duplicate tuples, despite it never conceptually being possible.
In my lab, I'm happy enough to try and remember not to do that. In a corporate production environment, I might write a policy. Either way, experience tells me that the system will get corrupted, it will get corrupted at a bad time, and it might take a long time for these problems to become, well, a problem.
With `Error on line 30`, an otherwise reasonably experienced Ansible developer might recognize that as being "record not found" without knowing anything else, and this would be enough to fix the problem. But it's much, much worse to have `_thatthing_search_result[0]` only sometimes the correct `vlan_view_ref`—it allows the world to be broken, silently. And the error may manifest itself somewhere else entirely; perhaps a security audit six months from now will flag this as inconsistent record-keeping, and if there are several tools and manual access, it might take days or weeks to track down the fact that this particular code was at fault.
In several days of fumbling around with the API, I learned this. I wasn't looking for problems; if it was documented, I did not see it. So I come to the point of this essay. Rather than dismiss this impossible situation as being because its a lab, fixing it, and moving on, I took two minutes to leave behind _code_—not a comment, not a mental note, not documentation—but code that will always run, that covers this impossible situation:
```
  - name: Fail if &gt;1 views returned
     fail:
       msg: "Got {{ _otherthing_search_result | length }} results from searching for VLAN view {{ vlan_parent_view_name }}. Otherthing allows this, but is not handled by this code."
     when: _otherthing_search_result | length &gt; 1
```
I manually created the failure condition, so I could manually test this condition. I hope it will never run in real-world use, but I am confident it will.
If (when) that error happens in production, then someone can decide what to do. I'd expect them to fix the bad data. If it happens a lot, I'd hope they would track down the other broken system. If they demand this code be removed, and this code does the undefined and wrong thing, that is their prerogative and a place where I don't want to work. The code is imperfect, but it is complete. The work of a craftsman.
Automation in the real world is an iterative process that fights with and uses imperfect systems equally. It will never handle all the exceptional situations. It might not even handle all the normal situations. Working code that passes lint, code reviews, and acceptance tests is code that handles the safe and needed path. With a tiny bit of effort, you can help out the next person by not just mapping the safe path but leaving behind warnings to the dangers you found.
--------------------------------------------------------------------------------
via: https://opensource.com/article/21/1/improve-ansible-play
作者：[Jeff Warncia][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/jeffwarncia
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/computer_keyboard_laptop_development_code_woman.png?itok=vbYz6jjb (A person programming)
[2]: https://www.ansible.com/
[3]: https://www.ansible.com/products/tower

180
sources/tech/20210105 A few bytes here, a few there, pretty soon you-re talking real memory.md Normal file
View File

@ -0,0 +1,180 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (A few bytes here, a few there, pretty soon you’re talking real memory)
[#]: via: (https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory)
[#]: author: (Dave Cheney https://dave.cheney.net/author/davecheney)
A few bytes here, a few there, pretty soon you’re talking real memory
======
Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment.[1][1]
```
func BenchmarkSortStrings(b *testing.B) {
s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
b.ReportAllocs()
for i := 0; i < b.N; i++ {
sort.Strings(s)
}
}
```
A convenience wrapper around `sort.Sort(sort.StringSlice(s))`, `sort.Strings` sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each iteration of the benchmark causes one heap allocation. Why does this happen?
Interfaces, as all Go programmers should know, are implemented as a [two word structure][2]. Each interface value contains a field which holds the type of the interface’s contents, and a pointer to the interface’s contents.[2][3]
In pseudo Go code, an interface might look something like this:
```
type interface struct {
// the ordinal number for the type of the value
// assigned to the interface
type uintptr
// (usually) a pointer to the value assigned to
// the interface
data uintptr
}
```
`interface.data` can hold one machine word–8 bytes in most cases–but a `[]string` is 24 bytes; one word for the pointer to the slice’s underlying array; one word for the length; and one for the remaining capacity of the underlying array, so how does Go fit 24 bytes into 8? With the oldest trick in the book, indirection. `s`, a `[]string` is 24 bytes, but `*[]string`–a pointer to a slice of strings–is only 8.
### Escaping to the heap
To make the example a little more explicit, here’s the benchmark rewritten without the `sort.Strings` helper function:
```
func BenchmarkSortStrings(b *testing.B) {
s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
b.ReportAllocs()
for i := 0; i < b.N; i++ {
var ss sort.StringSlice = s
var si sort.Interface = ss // allocation
sort.Sort(si)
}
}
```
To make the interface magic work, the compiler rewrites the assignment as `var si sort.Interface = &ss`–the _address_ of `ss` is assigned to the interface value.[3][4] We now have a situation where the interface value holds a pointer to `ss`, but where does it point? Where in memory does `ss` live?
It appears that `ss` is moved to the heap, causing the allocation that the benchmark reports.
```
Total: 296.01MB 296.01MB (flat, cum) 99.66%
8 . . func BenchmarkSortStrings(b *testing.B) {
9 . . s := []string{"heart", "lungs", "brain", "kidneys", "pancreas"}
10 . . b.ReportAllocs()
11 . . for i := 0; i < b.N; i++ {
12 . . var ss sort.StringSlice = s
13 296.01MB 296.01MB var si sort.Interface = ss // allocation
14 . . sort.Sort(si)
15 . . }
16 . . }
```
The allocation occurs because the compiler currently cannot convince itself that `ss` outlives `si`. The general attitude amongst Go compiler hackers seems to be that [this could be improved][5], but that’s a discussion for another time. As it stands, `ss` is allocated on the heap. Thus the question becomes, how many bytes are allocated per iteration? Why don’t we ask the `testing` package.
```
% go test -bench=. sort_test.go
goos: darwin
goarch: amd64
cpu: Intel(R) Core(TM) i7-5650U CPU @ 2.20GHz
BenchmarkSortStrings-4 12591951 91.36 ns/op 24 B/op 1 allocs/op
PASS
ok command-line-arguments 1.260s
```
Using Go 1.16beta1, on amd64, 24 bytes are allocated per operation.[4][6] However, the previous Go version, on the same platform, consumes 32 bytes per operation
```
% go1.15 test -bench=. sort_test.go
goos: darwin
goarch: amd64
BenchmarkSortStrings-4 11453016 96.4 ns/op 32 B/op 1 allocs/op
PASS
ok command-line-arguments 1.225s
```
This brings us, circuitously, to the subject of this post, a nifty quality of life improvement coming in Go 1.16. But before I can talk about it, I need to discuss size classes.
### Size classes
To explain what a size class is, consider how a theoretical Go runtime could allocate 24 bytes on its heap. A simple way to do this would be keep track of all the memory allocated so far using a pointer to the last allocated byte on the heap. To allocate 24 bytes, the heap pointer is incremented by 24, and the previous value returned to the caller. As long as the code that asked for 24 bytes never writes beyond that mark this mechanism has no overhead. Sadly, in real life, memory allocators don’t just allocate memory, sometimes they have to free it.
Eventually the Go runtime will have to free those 24 bytes, but from the runtime’s point of view, the only thing it knows is the starting address it gave to the caller. It does not know how many bytes after that address were allocated. To permit deallocation, our hypothetical Go runtime allocator would have to record, for each allocation on the heap, its length. Where are the allocation for those lengths allocated? On the heap of course.
In our scenario, when the runtime wants to allocate memory, it could request a little more than it was asked for and use it to store the amount requested. For our slice example, when we asked for 24 bytes, we’d actually consume 24 bytes plus some overhead to store the number 24. How large is this overhead? It turns out the minimum amount that is practical is one word.[5][7]
To record a 24 byte allocation the overhead would 8 bytes. 25% is not great, but not terrible, and as the size of the allocation goes up, the overhead would become insignificant. However, what would happen if we wanted to store just one `byte` on the heap? The overhead would be eight times the amount of data we asked for! Is there are more efficient way to allocate small amounts on the heap?
Rather than storing the length alongside each allocation, what if all the thing of the same size were stored together? If all the 24 byte things are stored together, then the runtime would automatically know how large they are. All the runtime needs is a single bit to indicate if a 24 byte area is in use or not. In Go these areas are known as size classes, because all the things of the same size are stored together (think school class–all the students are the same grade–not a C++ class). When the runtime needs to allocate a small amount, it does so using the smallest size class that can accomodate the allocation.
### Unlimited size classes for all
Now we know how size classes work, the obvious question is, where are they stored? Not surprisingly, the memory for a size class comes from the heap. To minimise overhead, the runtime allocates a larger amount from the heap (usually a multiple of the system page size) then dedicates that space for allocations of a single size. But, there’s a problem.
The pattern of allocating a large area to store things of the same size works well[6][8] if there’s a fixed, preferably small, number of allocation sizes, but in a general purpose language programs can ask the runtime for an allocation of any size.[7][9]
For example, imagine asking the runtime for 9 bytes. 9 bytes is an uncommon size, so it’s likely that a new size class for things of size 9 would be required. As 9 byte things are uncommon, it’s likely the remainder of the allocation, 4kb or more, would be wasted. Because of this the set of possible size classes is fixed. If a size class of the exact amount is not available, the allocation is rounded up to the next size class. In our example 9 bytes might be allocated in the 12 byte size class. The overhead of 3 unused bytes is better than an entire size class allocation which goes mostly unused.
### All together now
This is the final piece of the puzzle. Go 1.15 did not have a 24 byte size class, so the heap allocation of `ss` was allocated in the 32 byte size class. Thanks to the work of Martin Möhrmann Go 1.16 has a 24 byte size class, which is a perfect fit for slice values assigned to interfaces.
1. This is not the correct way to benchmark a sort function because after the first iteration, the input is sorted. But I digress.[][10]
2. The accuracy of this statement depends on the version of Go in use. For example, Go 1.15 added the ability to [store some integers directly in the interface value,][11] saving the allocation and indirection. However, for the majority of values, if it wasn’t a pointer type already, its address is taken and stored in the interface value.[][12]
3. The compiler keeps track of this sleight of hand in the interface value’s type field so it remembers that the type assigned to `si` is `sort.StringSlice`, not `*sort.StringSlice`.[][13]
4. On 32 bit platforms this number is halved, [but we never look back][14].[][15]
5. If you were prepared to limit allocations to 4G, or maybe 64kb, you could use a smaller amount of memory to store the size of the allocation, but this would mean the first word of the allocation is not naturally aligned so in practice the savings of using less than a word to store the length header are undermined by padding.[][16]
6. Storing things of the same size together is also an effective strategy to combat fragmentation.[][17]
7. This isn’t a far fetched scenario, strings come in all shapes and sizes and generating a string of a size not previously seen before can be as simple as appending a space.[][18]
#### Related posts:
1. [I’m talking about Go at DevFest Siberia 2017][19]
2. [If aligned memory writes are atomic, why do we need the sync/atomic package?][20]
3. [A real serial console for your Raspberry Pi][21]
4. [Why is a Goroutine’s stack infinite ?][22]
--------------------------------------------------------------------------------
via: https://dave.cheney.net/2021/01/05/a-few-bytes-here-a-few-there-pretty-soon-youre-talking-real-memory
作者：[Dave Cheney][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://dave.cheney.net/author/davecheney
[b]: https://github.com/lujun9972
[1]: tmp.AZQSXjFgPm#easy-footnote-bottom-1-4231 (This is not the correct way to benchmark a sort function because after the first iteration, the input is sorted. But I digress.)
[2]: https://research.swtch.com/interfaces
[3]: tmp.AZQSXjFgPm#easy-footnote-bottom-2-4231 (The accuracy of this statement depends on the version of Go in use. For example, Go 1.15 added the ability to <a href="https://golang.org/doc/go1.15#runtime">store some integers directly in the interface value,</a> saving the allocation and indirection. However, for the majority of values, if it wasn’t a pointer type already, its address is taken and stored in the interface value.)
[4]: tmp.AZQSXjFgPm#easy-footnote-bottom-3-4231 (The compiler keeps track of this sleight of hand in the interface value’s type field so it remembers that the type assigned to <code>si</code> is <code>sort.StringSlice</code>, not <code>*sort.StringSlice</code>.)
[5]: https://github.com/golang/go/issues/23676
[6]: tmp.AZQSXjFgPm#easy-footnote-bottom-4-4231 (On 32 bit platforms this number is halved, <a href="https://www.tallengestore.com/products/i-never-look-back-darling-it-distracts-from-the-now-edna-mode-inspirational-quote-tallenge-motivational-poster-collection-large-art-prints">but we never look back</a>.)
[7]: tmp.AZQSXjFgPm#easy-footnote-bottom-5-4231 (If you were prepared to limit allocations to 4G, or maybe 64kb, you could use a smaller amount of memory to store the size of the allocation, but this would mean the first word of the allocation is not naturally aligned so in practice the savings of using less than a word to store the length header are undermined by padding.)
[8]: tmp.AZQSXjFgPm#easy-footnote-bottom-6-4231 (Storing things of the same size together is also an effective strategy to combat fragmentation.)
[9]: tmp.AZQSXjFgPm#easy-footnote-bottom-7-4231 (This isn’t a far fetched scenario, strings come in all shapes and sizes and generating a string of a size not previously seen before can be as simple as appending a space.)
[10]: tmp.AZQSXjFgPm#easy-footnote-1-4231
[11]: https://golang.org/doc/go1.15#runtime
[12]: tmp.AZQSXjFgPm#easy-footnote-2-4231
[13]: tmp.AZQSXjFgPm#easy-footnote-3-4231
[14]: https://www.tallengestore.com/products/i-never-look-back-darling-it-distracts-from-the-now-edna-mode-inspirational-quote-tallenge-motivational-poster-collection-large-art-prints
[15]: tmp.AZQSXjFgPm#easy-footnote-4-4231
[16]: tmp.AZQSXjFgPm#easy-footnote-5-4231
[17]: tmp.AZQSXjFgPm#easy-footnote-6-4231
[18]: tmp.AZQSXjFgPm#easy-footnote-7-4231
[19]: https://dave.cheney.net/2017/08/23/im-talking-about-go-at-devfest-siberia-2017 (I’m talking about Go at DevFest Siberia 2017)
[20]: https://dave.cheney.net/2018/01/06/if-aligned-memory-writes-are-atomic-why-do-we-need-the-sync-atomic-package (If aligned memory writes are atomic, why do we need the sync/atomic package?)
[21]: https://dave.cheney.net/2014/01/05/a-real-serial-console-for-your-raspberry-pi (A real serial console for your Raspberry Pi)
[22]: https://dave.cheney.net/2013/06/02/why-is-a-goroutines-stack-infinite (Why is a Goroutine’s stack infinite ?)

404
sources/tech/20210105 How to customize your voice assistant with the voice of your choice.md Normal file
View File

@ -0,0 +1,404 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (How to customize your voice assistant with the voice of your choice)
[#]: via: (https://opensource.com/article/21/1/customize-voice-assistant)
[#]: author: (Rich Lucente https://opensource.com/users/rlucente)
How to customize your voice assistant with the voice of your choice
======
The Nana and Poppy project enables a voice assistant to greet users with
their great-grandchildren's voices instead of a generic AI.
![radio communication signals][1]
It can be hard to find meaningful gifts for relatives that already have almost everything. My wife and I have given our parents "experiences" to try something novel, such as going to a themed restaurant or seeing a concert, but as our parents get older, it becomes more difficult. This year was no exception—until I thought about a way open source could give them something really special.
What if when they request help from an artificial intelligence (AI) voice assistant such as [Mycroft][2], my in-laws could get a special greeting? I looked at the existing voice assistant APIs to see if something like this was already available. There was something close, but not exactly what I was looking for. My idea was to record their great-grandchildren speaking a short greeting that would play whenever they push the button and before the conversation with the voice assistant begins. The greeting would be something like:
> "Good morning, Nana and Poppy. Today is December 25th. The time is 3:10 pm. The current temperature for Waynesboro is 47 degrees. The current temperature for Ocean City is 50 degrees."
When they press the button, my in-laws would hear their great-grandchildren reporting the date, time, and temperature for their home and their favorite vacation spot. To make this a reality, I had to solve a few problems.
### So many audio files…
The first problem was figuring out what phrases the voice assistant would need to say. Thinking about all the dates, times, and temperatures that I would need to cover, I arrived at a list of 79 phrases. I sent these instructions to my nieces:
> _Please record the kids saying each line below. Sorry there are so many. It's okay to do this in one setting with prompting them if it makes it easier. I can edit the audio files and deal with most formats, so none of that should be a problem. Just record using your phone in whatever way is easiest._
>
> _Make sure that the kids say each line clearly and loudly. There should be a slight pause between each line to make editing easier (prompting helps like "Repeat after me …"). That will make it easier for me to chop these up into individual sound files._
>
> _Whenever the button on the device is pushed, it will respond with a random grandchild saying the correct date/time/temperature, like:_
>
> _"Good afternoon, Nana and Poppy. Today is January third. The time is one oh four pm. The current temperature for Waynesboro is thirty degrees. The current temperature for Ocean City is thirty four degrees."_
>
> _PLEASE RECORD EACH CHILD SAYING THE FOLLOWING PHRASES WITH A SHORT PAUSE BETWEEN EACH ONE:_
Then I provided the following list of words for the children to record:
_Good
morning
afternoon
evening
night_
_Nana and Poppy_
_The time
Today 
The current temperature for
Waynesboro
Ocean City
is
and
degrees
minus_
_am
pm_
_January
February
March
April
May_
| _June
July
August
September
October
November
December
first
second
third
fourth
fifth
sixth
seventh
eighth
ninth
tenth
eleventh
twelfth
thirteenth
fourteenth
fifteenth
sixteenth
seventeenth
eighteenth
nineteenth
twentieth
thirtieth
oh_ | _one
two
three
four
five
six
seven
eight
nine
ten
eleven
twelve
thirteen
fourteen
fifteen
sixteen
seventeen
eighteen
nineteen
twenty
thirty
forty
fifty
sixty
seventy
eighty
ninety
hundred_
---|---|---
My nieces are doubly blessed with children under 10 years old and near-infinite patience. So, after a couple of months of prodding, I received a three-minute audio file for each child.
Now my problem was how to edit them. I needed to normalize the recordings, reduce noise, and chop them into audio clips for individual words and phrases. I also wanted to take advantage of lossless audio, and I decided to convert the tracks to Waveform Audio File Format ([WAV][3]). Audacity was just the open source tool to do all of that.
### Audacity to the rescue!
[Audacity][4] is a feature-rich open source sound-editing tool. The software's features and capabilities can be overwhelming, so I'll describe the workflow I followed to accomplish my goals. I make no claims to being an Audacity expert, but the steps I followed seemed to work pretty well. (Comments are always welcome on how to improve what I've done.)
Audacity has [downloads][5] for Linux, Windows, and macOS. I grabbed the most recent macOS binary and quickly installed it on my laptop. Launching Audacity opens an empty new project. I imported all of the children's audio files using the **Import** feature.
![Import audio files in Audacity][6]
(Rich Lucente, [CC BY-SA 4.0][7])
#### Normalizing audio files
Some of the children spoke louder than others, so the various audio files had different volume levels. I needed to normalize the audio tracks so that the greeting's volume would be the same regardless of which child was speaking. To normalize the volumes, I began by selecting all of the audio tracks after they were imported.
![Selecting all the audio tracks][8]
(Rich Lucente, [CC BY-SA 4.0][7])
To normalize the children's peaks and valleys, so one child wasn't louder than the other, I used Audacity's **Normalize** effect.
![Normalize effect][9]
(Rich Lucente, [CC BY-SA 4.0][7])
It's important to understand that the Normalize and Amplify effects do very different things. Normalize adjusts the highest peaks and lowest valleys for multiple tracks, so they are all similar, whereas Amplify exaggerates the existing peaks and valleys. If I had used Amplify instead of Normalize, the louder child would have become even louder. I used the default settings to normalize the two audio tracks.
![Normalize defaults][10]
(Rich Lucente, [CC BY-SA 4.0][7])
#### Remove background noise
Another thing I noticed is that there was noise between the spoken phrases on the tracks. Audacity has tooling to help reduce background noise and result in much cleaner audio. To reduce noise, select a sample of an audio track with background noise. I used the **View-&gt;Zoom** menu option to see the track's noise more easily.
![Background noise sample][11]
(Rich Lucente, [CC BY-SA 4.0][7])
To make sure I selected only the background noise, I listened to the selected audio clip using the **Play** button in the toolbar. Next, I selected **Effect-&gt;Noise Reduction**.
![Noise Reduction effect][12]
(Rich Lucente, [CC BY-SA 4.0][7])
Then I created a **Noise Profile** using step 1 in the **Noise Reduction** dialog.
![Get a Noise Profile from audio sample][13]
(Rich Lucente, [CC BY-SA 4.0][7])
Audacity characterizes the background noise in the audio sample so that it can be removed. To remove the background noise, I selected the entire audio track by pressing the small **Select** button to the left of the track.
![Select whole audio track button][14]
(Rich Lucente, [CC BY-SA 4.0][7])
I applied the **Noise Reduction** effect again, but this time I pressed **OK** in step 2 of the dialog. I accepted the default settings.
![Noise Reduction effect step 2][15]
(Rich Lucente, [CC BY-SA 4.0][7])
I repeated these steps for each child's audio track, so I had normalized audio tracks, and the background noise was characterized and removed.
#### Export the clips as WAV files
The remaining task was to zoom and scroll through each track and export the specific clips as separate audio files in WAV format. When working with one child's track, I needed to mute the other tracks using either the small **Mute** button to the left of each audio track or, since there were so many tracks, selecting the **Solo** button for the track I wanted to work with.
![Mute and Solo buttons for multiple tracks][16]
(Rich Lucente, [CC BY-SA 4.0][7])
Selecting each word and phrase can be tricky, but the ability to zoom into an audio track was my friend. I tried to set each audio clip's start and end to just before and just after the word or phrase being spoken. Before exporting any audio clips, I played the selected clip using the **Play** icon on the toolbar to make sure I got it all.
One interesting thing is how waveforms map to spoken words. The waveforms for "six" and "sixth" are incredibly similar, with the latter having a smaller audio waveform to the right for the "th" sound. I carefully tested each clip before exporting it to make sure I had captured the full word or phrase.
After selecting an audio clip for a word or phrase, I exported the selected audio using the **File-&gt;Export** menu.
![Exporting selected audio][17]
(Rich Lucente, [CC BY-SA 4.0][7])
I had to make sure to save each clip using the correct file name from the list of words and phrases. This is because the application I used to customize the voice assistant expects the file name to match an entry in the phrase list.
The expected file names for the audio clips (without the .wav extension) are listed below. Note the underscores within the phrases. If you're doing this project, adjust the bold file names to match your loved ones' nicknames and location preferences. You'll also have to make the same changes in the application source code.
_good
morning
afternoon
evening
night
**nana_and_poppy**
the_time
today
the_current_temperature_for
**waynesboro
ocean_city**
is
and
degrees
minus
am
pm
january
february
march
april
may
june
july
august
september
october_ | _november
december
first
second
third
fourth
fifth
sixth
seventh
eighth
ninth
tenth
eleventh
twelfth
thirteenth
fourteenth
fifteenth
sixteenth
seventeenth
eighteenth
nineteenth
twentieth
thirtieth
oh
one
two_ | _three
four
five
six
seven
eight
nine
ten
eleven
twelve
thirteen
fourteen
fifteen
sixteen
seventeen
eighteen
nineteen
twenty
thirty
forty
fifty
sixty
seventy
eighty
ninety
hundred_
---|---|---
This project's GitHub repository also includes a Bash script to run as a sanity check for any missing or misnamed files.
After choosing each clip's appropriate name, I saved the clip in the child's specific folder (child1, child2, etc.) as a WAV format file. I accepted the default export settings.
![Converting clip to WAV format][18]
(Rich Lucente, [CC BY-SA 4.0][7])
After exporting all the audio clips, I had a folder for each child that was fully populated with WAV files for the phrases above. This seems like a lot of work, but it took only about 90 minutes for each child, and I got way more efficient with each successive audio clip.
### Package the application
Now that I had the audio clips for the greeting, I needed to think about the application and how to package it. I also wanted an open source-friendly solution that was open to modification.
About two years ago, a colleague gave me a [Google AIY Voice Kit][19] that he grabbed from the clearance bin for just $10. It's a cleverly folded box containing a speaker, microphone, and custom circuit board. You supply a Raspberry Pi and quickly have a do-it-yourself Google voice assistant. These kits are available for purchase online and in electronics stores. This small box offered an easy way to package the project.
![Google AIY Voice Kit][20]
(Rich Lucente, [CC BY-SA 4.0][7])
### Customize the voice assistant
The Google kit includes a Python API and several Python modules. I followed the kit's instructions to get the initial configuration working. The [Google Assistant gRPC][21] software is open source under an Apache 2.0 license.
I adapted the Google Assistant gRPC demo to implement my application. The application's operation is fairly simple: First, it waits for the device's button to be pressed. The code then constructs four separate word lists for: 1. the greeting and date, 2. the current time, 3. the current temperature of the first location, and 4. the current temperature of the second location. The children's voices are randomly shuffled, and then each word list is used to play the audio clips corresponding to the child assigned to that list. (This is why it was important to strictly follow the naming convention for the audio clips.) The application then initiates a conversation with the Google Assistant API.
At first, I thought the code to gather weather data for the current temperature and convert numbers to words would be challenging. This proved not to be the case at all. In fact, existing open source Python modules made it all simple and intuitive.
There were two cases to be addressed for converting numbers to word lists: I needed to convert ordinal numbers to words (e.g., 1 and 2 to first and second), and I also needed to convert cardinal numbers to words (e.g., 28 to twenty-eight). The open source [inflect.py module][22] has functions that handle both cases quite easily.
```
import inflect
p = inflect.engine()
number = 23
# get a cardinal word list (e.g. ['twenty', 'three'])
print(p.number_to_words(number).replace('-', ' ').split(' '))
# get an ordinal word list (e.g. ['twenty', 'third'])
print(p.number_to_words(p.ordinal(number)).replace('-', ' ').split(' '))
```
The inflect engine returns string representations of the numbers with embedded hyphens (e.g., twenty-three) so that the code splits the strings into variable-length word lists by converting the hyphens to spaces and splitting the string into a list using a space as the delimiter.
The next problem to solve was getting the current temperature for the two locations. [Open Weather Map][23] offers a free-tier weather service that allows up to 60 calls a minute or 1 million calls a month, which is way more than this project needs. I signed up for the free-tier service and received an API key. It was very easy to access the service by using the open source Python wrapper module [PyOWM][24]. Here is a simplified code snippet:
```
import pyowm
# use the OpenWeatherMap API key to get a weather manager
owm = pyowm.OWM('YOUR-OPEN-WEATHER-MANAGER-API-KEY')
mgr = owm.weather_manager()
# convert location phrase to city for OWM API
# e.g. ocean_city becomes 'Ocean City, US'
location = 'ocean_city'
city = location.replace('_', ' ').title() + ', US'
# get current temperature in fahrenheit for location
observation = owm_mgr.weather_at_place(city)
temp = round(observation.weather.temperature('fahrenheit')['temp'])
```
### Wrapping it up with a bow
The full source code for the project is available in my [GitHub repository][25]. The project includes a systemd service unit file adapted from Google's demo to automatically start the application on device boot. The GitHub repository includes instructions to install the Python modules and configure the systemd service.
I created a [short video][26] of the result. Five custom voice assistants were distributed during the holidays: one each for the great grandparents and grandparents of each child. For some, these gifts brought tears of joy. The children's voices are absolutely adorable and these boxes capture a fleeting moment of childhood that can be enjoyed for a very long time.
--------------------------------------------------------------------------------
via: https://opensource.com/article/21/1/customize-voice-assistant
作者：[Rich Lucente][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/rlucente
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/sound-radio-noise-communication.png?itok=KMNn9QrZ (radio communication signals)
[2]: https://opensource.com/article/20/7/mycroft-voice-skill
[3]: https://en.wikipedia.org/wiki/WAV
[4]: https://www.audacityteam.org/
[5]: https://www.audacityteam.org/download/
[6]: https://opensource.com/sites/default/files/uploads/audacity1_importaudio.png (Import audio files in Audacity)
[7]: https://creativecommons.org/licenses/by-sa/4.0/
[8]: https://opensource.com/sites/default/files/uploads/audacity2_selectingtracks.png (Selecting all the audio tracks)
[9]: https://opensource.com/sites/default/files/uploads/audacity3_normalize.png (Normalize effect)
[10]: https://opensource.com/sites/default/files/uploads/audacity4_normalizedefaults.png (Normalize defaults)
[11]: https://opensource.com/sites/default/files/uploads/audacity5_backgroundnoise.png (Background noise sample)
[12]: https://opensource.com/sites/default/files/uploads/audacity6_noisereduction.png (Noise Reduction effect)
[13]: https://opensource.com/sites/default/files/uploads/audacity7_noiseprofile.png (Get a Noise Profile from audio sample)
[14]: https://opensource.com/sites/default/files/uploads/audacity8_selecttrack.png (Select whole audio track button)
[15]: https://opensource.com/sites/default/files/uploads/audacity9_noisereduction2.png (Noise Reduction effect step 2)
[16]: https://opensource.com/sites/default/files/uploads/audacity10_mutesolo.png (Mute and Solo buttons for multiple tracks)
[17]: https://opensource.com/sites/default/files/uploads/audacity11_exportaudio.png (Exporting selected audio)
[18]: https://opensource.com/sites/default/files/uploads/audacity12_convertwav.png (Converting clip to WAV format)
[19]: https://aiyprojects.withgoogle.com/voice/
[20]: https://opensource.com/sites/default/files/uploads/googleaiy.png (Google AIY Voice Kit)
[21]: https://pypi.org/project/google-assistant-grpc/
[22]: https://pypi.org/project/inflect
[23]: https://openweathermap.org/
[24]: https://pypi.org/project/pyowm
[25]: https://github.com/rlucente-se-jboss/nana-poppy-project
[26]: https://youtu.be/Co7rigJRNUM

275
translated/tech/20201222 Learn to use the Sed text editor.md Normal file
View File

@ -0,0 +1,275 @@
[#]: collector: (lujun9972)
[#]: translator: (wxy)
[#]: reviewer: (wxy)
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Learn to use the Sed text editor)
[#]: via: (https://opensource.com/article/20/12/sed)
[#]: author: (Seth Kenlon https://opensource.com/users/seth)
学习使用 Sed 文本编辑器
======
> Sed 缺少通常的文本框，而是按照用户的命令直接写入到文件上。
![命令行提示][1]
`sed` 命令是为 AT&T 最初的 Unix 操作系统第 7 版创建的，此后，可能每一个 Unix 和 Linux 操作系统都包含了它。`sed` 应用程序是一个 _流编辑器_，与文本编辑器不同的是，它不会打开一个视觉缓冲区，将文件的数据加载到其中进行处理。相反，它根据在终端输入的命令或脚本中的一系列命令，逐行对文件进行操作。
### 安装
如果你使用的是 Linux、BSD 或 macOS，那么你已经安装了 GNU 或 BSD 版的 `sed`。这是两个不同的原始 `sed` 命令的重新实现，虽然它们很相似，但也有一些小的区别。GNU `sed` 通常被认为是功能最丰富的 `sed`，而且它在这些平台上都可以广泛使用。
如果你找不到 GNU `sed`（在非 Linux 系统上通常被称为 `gsed`），那么你可以[从 GNU 网站上下载它的源代码][2]。安装 GNU `sed` 的好处是，可以使用它的额外功能，但它也可以被限制为只符合 `sed` 的 [POSIX][3] 规范，如果你需要移植性的话。
在 Windows 上，你可以用 [Chocolatey][5] 来[安装][4] GNU `sed`。
### Sed 如何工作
`sed` 应用程序一次只处理一行。因为它没有视觉显示，所以它在内存中创建了一个模式空间：一个包含输入流的当前行的空间（去掉任何尾部的换行符）。一旦填充了模式空间，你对 `sed` 的指令就会被执行。有时你的指令是有条件的，有时是无条件的，所以这些指令的结果取决于你如何使用 `sed`。
当命令结束时，`sed` 会将模式空间的内容打印到输出流中。默认的输出流是**标准输出**，但可以将其重定向到一个文件，甚至使用 `--in-place=.bak` 选项重定向到同一个文件中。
然后再从下一个输入行开始循环。
`sed`命令的语法是：
```
$ sed --options [optional SCRIPT] [INPUT FILE or STREAM]
```
#### 找到你要编辑的内容
在可视化编辑器中，你通常不需要考虑太多，就能在文本文件中找到你想要修改的内容。你的眼睛（或屏幕阅读器）会扫描文本，找到你想改变的单词或你想插入或删除文本的地方，然后你就可以开始输入了。而 `sed` 没有交互模式，所以你需要告诉它必须满足什么条件才能运行特定的命令。
在这些例子中，假设一个名为 `example.txt` 的文件包含了这样的文字：
```
hello
world
This is line three.
Here is the final line.
```
#### 行号
指定行号告诉 `sed` 只对文件中的那一行进行操作。
例如，下面这条命令选择文件的第 1 行并打印出来。因为 `sed` 在处理后的默认操作也是打印一行到**标准输出**，这样做的效果就是重复第一行：
```
$ sed '1p' example.txt
hello
hello
world
This is line three.
Here is the final line.
```
你也可以步进式指定行号。例如，`1~2` 表示每两行选择一行（“从第一行开始每两行选择一行”）。指令 `1~3` 表示从第一行开始，每三行选择一行：
```
$ sed '1p' example.txt
hello
hello
world
This is line three.
Here is the final line.
Here is the final line.
```
#### 行定位
你可以通过使用 `$` 作为选择器，只对文件的最后一行进行操作：
```
$ sed '$p' example.txt
hello
world
This is line three.
Here is the final line.
Here is the final line.
```
在 GNU `sed` 中，你可以选择多行（例如，`sed '1,$p'` 打印第一行和最后一行）。
#### 反转
任何数字或位置的选择，你都可以用感叹号（`!`）字符反转。下面这将选择除第一行以外的所有行：
```
$ sed '1!p' example.txt
hello
world
world
This is line three.
This is line three.
Here is the final line.
Here is the final line.
```
#### 模式匹配
你可以把模式匹配想象成文字处理器或浏览器中的**查找**操作。你提供一个词（一个 _模式_），然后选择了结果。模式匹配的语法是 `/pattern/`：
```
$ sed '/hello/p' example.txt
hello
hello
world
This is line three.
Here is the final line.
$ sed '/line/p' example.txt
hello
world
This is line three.
This is line three.
Here is the final line.
Here is the final line.
```
### 用 Sed 编辑
一旦你找到了你要编辑的内容，你就可以执行你想要的任何操作。你可以用 `sed` 中的命令来执行编辑。`sed` 中的命令不是 `sed` 命令本身。如果这样说有帮助的话，可以把它们看作是“动作”或“动词”或“指令”。
`sed` 中的命令是单个字母，例如前面例子中使用的**打印**命令的 `p`。它们一开始可能很难记忆，但和所有事情一样，你会随着练习而了解它们。
#### p 代表打印
`p` 指令打印当前模式空间中的任何内容。
#### d 用于删除
`d` 指令删除模式空间：
```
$ sed '$d' example.txt
hello
world
This is line three.
$ sed '1d' example.txt
world
This is line three.
Here is the final line.
```
#### s 用于搜索和替换
`s` 命令搜索一个模式并将其替换为其他东西。这可能是 `sed` 最流行和最随意的用法，而且它通常是用户学习的第一个（有时也是唯一的）`sed` 命令。几乎可以肯定它是文本编辑中最有用的命令：
```
$ sed 's/world/opensource.com/' example.txt
hello
opensource.com
This is line three.
Here is the final line.
```
在你的替换文本中，也可以使用一些特殊的功能。例如，`\L` 将替换文本转换为小写，`\l` 则只转换下一个字符。还有其他一些功能，列在 `sed` 文档中（你可以用 `info sed` 命令查看）。
替换子句中的特殊字符 `&` 指的是匹配到的模式：
```
$ sed 's/is/\U&/' example.txt
hello
world
ThIS is line three.
Here IS the final line.
```
你也可以通过特殊的标志来影响 `s` 如何处理它找到的内容。`g`（应该是指 _全局_）标志告诉 `s` 对行上找到的所有匹配项进行替换，而不仅仅是第一个匹配项：
```
$ sed 's/is/\U&/g' example.txt
hello
world
ThIS IS line three.
Here IS the final line.
```
其他重要的标志还包括用一个数字来表示要影响第几个出现的匹配模式：
```
$ sed 's/is/\U&/2' example.txt
hello
world
This IS line three.
Here is the final line.
```
`w` 标志，后面跟着一个文件名，_只有_在有变化的情况下，才会将匹配的行写入文件：
```
$ sed 's/is/\U&/w sed.log' example.txt
hello
world
ThIS is line three.
Here IS the final line.
$ cat sed.log
ThIS is line three.
Here IS the final line.
```
标志可以组合：
```
$ sed 's/is/\U&/2w sed.log' example.txt
hello
world
This IS line three.
Here is the final line.
$ cat sed.log
This IS line three.
```
### 脚本
有很多很棒的网站都有 `sed` “单行脚本”，它们给你提供了面向任务的 `sed` 命令来解决常见的问题。然而，自己学习 `sed` 可以让你写出自己的单行脚本，而且这些单行脚本可以根据你的具体需求来定制。
`sed` 的脚本可以在终端中写成一行，也可以保存到文件中，然后用 `sed` 本身执行。我倾向于把小脚本写成一个命令，因为我发现自己在现实生活中很少重复使用 `sed` 命令。当我写一个 `sed` 脚本时，通常都是针对一个文件的。例如，在写完这篇文章的初稿后，我用 `sed` 来规范 “sed” 的大小写，而这是我可能永远也不会再做的任务。
你可以向 `sed` 发出一系列不同的命令，用分号（`;`）分开。
```
$ sed '3t ; s/line/\U&/' example.txt
hello
world
This is LINE three.
This is the final line.
```
### 带括号的范围改变
你也可以用大括号（`{}`）限制哪些结果受到影响。当你将 `sed` 命令用大括号括起来时，它们只适用于特定的选择。例如，“line” 字出现在样本文本的两行中。你可以通过声明所需的匹配条件（`$` 表示最后一行），并将你希望执行的 `s` 命令放在紧随其后的括号中，强制 `sed` 只影响最后一行：
```
$ sed '$ {s/line/\U&/}' example.txt
hello
world
This is line three.
This is the final LINE.
```
### 学习 Sed
你可以用 `sed` 做的事情比本文所解释的多得多。我甚至还没有涉及到分支（`b`）、测试（`t`）、保留空格（`H`）和许多其他功能。就像 [ed][6] 一样，`sed` 可能不是你要用来创建文档的文本编辑器，甚至不是你需要做的每一个脚本任务中使用的文本编辑器，但它是你作为 POSIX 用户的一个强大的选择。学习 `sed` 命令的结构以及如何编写简短的脚本可以快速修改大量的文本。阅读 GNU `sed` 的`info` 页面，或者 BSD `sed` 的手册页，看看 `sed` 能为你做什么。
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/12/sed
作者：[Seth Kenlon][a]
选题：[lujun9972][b]
译者：[wxy](https://github.com/wxy)
校对：[wxy](https://github.com/wxy)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/seth
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/command_line_prompt.png?itok=wbGiJ_yg (Command line prompt)
[2]: http://www.gnu.org/software/sed/
[3]: https://opensource.com/article/19/7/what-posix-richard-stallman-explains
[4]: https://chocolatey.org/packages/sed
[5]: https://opensource.com/article/20/3/chocolatey
[6]: https://opensource.com/article/20/12/gnu-ed

72
translated/tech/20201230 Use the Markdown Editor app in Nextcloud.md Normal file
View File

@ -0,0 +1,72 @@
[#]: collector: (lujun9972)
[#]: translator: (geekpi)
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Use the Markdown Editor app in Nextcloud)
[#]: via: (https://opensource.com/article/20/12/nextcloud-markdown)
[#]: author: (Seth Kenlon https://opensource.com/users/seth)
使用 Nextcloud 中的 Markdown Editor 应用
======
Nextcloud 拥有流行的 Markdown 文件类型的最流畅的编辑器之一，它有很多方便和直观的功能。
![Digital images of a computer desktop][1]
纯文本的好处是，没有额外的针对计算机的信息杂乱无章地出现在原本供人类阅读的文字中。计算机的好处是它们是可编程的，因此只要我们人类同意在写作时遵循非常特定的惯例，我们就可以对计算机进行编程，将人类可读的文本解释为秘密指令。例如，我们在一个词的周围打上两个星号，不仅可以给人类一个视觉上的提示，说明这个词很重要，我们还可给计算机编程让它用**粗体**显示这个词。
这正是 [Markdown][2] 背后的理论和实践，这种流行的纯文本格式向作者承诺，只要_他们_使用特定的纯文本约定，那么他们的文本就会以特定的风格呈现。
传统中，这意味着作者用纯文本写作，直到文本被传给转换器应用（最初是 `markdown.pl`），才会看到漂亮的样式，但 Nextcloud 的 Markdown Editor 应用改变了这一点。
通过 Nextcloud 的 Markdown Editor，你可以一边输入纯文本，一边看到它渲染的样式。这对于那些努力记住 Markdown 有时令人困惑的符号（方括号是在小括号中的超链接之前还是之后？）而且更好的是，它运行在 Nextcloud 中，所以你可以在任何地方使用它。
### 安装
要使用 Nextcloud 的 Markdown Editor，你必须安装 Nextcloud。好消息是，Nextcloud 非常_容易_安装。我已经在树莓派、共享服务器、甚至作为一个本地应用安装了它（这是愚蠢的，不要这样做）。如果你不相信自己的能力，你甚至可以依靠 [Turnkey Linux][3] 来帮你完成这些难关，否则就直接从 [Nextcloud.com][4] 购买托管服务。在你安装 Nextcloud 后，添加应用就很简单了。点击 Nextcloud 界面右上角的用户图标，选择 **Apps**。找到 **Office and Text** 类别，点击安装并启用 **Markdown Editor**。
![Nextcloud app store showing Markdown Editor installer][5]
### 启动
激活后，Markdown Editor 会与 Nextcloud 文件中任何以 .md 结尾的文件相关联，当你打开一个 Markdown 文件时，你就会启动 Markdown Editor。
### 使用 Markdown Editor
Markdown Editor 包含了一个大的文本区域供你输入，以及一个沿着顶部的工具栏。
![Example markdown file ][6]
工具栏包含了文字处理器的基本功能：用粗体、斜体和删除线设计文本样式、创建标题和段落、列表等等。
如果你了解 Markdown，这些功能中的许多会在你输入时自动调用。如果你不熟悉 Markdown，那么工具栏或常用的键盘快捷键（**Ctrl+B** 表示粗体，**Ctrl+I** 表示斜体等等）可以帮助你设计文本的样式。
Markdown Editor 工作方式的最好的一点是，它真正做到了使人人满意：如果你想用 Markdown 打字，那么它就会接受，并立即将其转化为视觉样式；如果你不想考虑 Markdown，那么当你使用键盘快捷键或工具栏按钮时，它就会为你生成样式。无论哪种方式，你永远不用看到 Markdown 语法，但你也永远不会失去它。这是一个完美的折中方案。
它也是一个非常聪明的编辑器。当你选择一个单词时，它为你创建一个超链接，它能快速流畅地自动转换 Markdown，而且它知道一些不同”风格“的 Markdown 语法（主要是 Commonmark，但也有传统的 Markdown、Github Markdown，等等）。
![black text on white background, word highlighted in blue to create an automatic link][7]
### 尝试 Nextcloud
我用过几个 Markdown 预览应用，Nextcloud 的 Markdown Editor 是最流畅的一个。它尊重用户，并为显示 Markdown 做了最基本的工作，所以它的转换速度很快，也很准确。因为它是 Nextcloud 中的一个应用，你还可以获得这样的好处：你的作品可以即时保存在自己的私有的开源云上，并进行版本控制。没有比这更好的文本编辑器了。
--------------------------------------------------------------------------------
via: https://opensource.com/article/20/12/nextcloud-markdown
作者：[Seth Kenlon][a]
选题：[lujun9972][b]
译者：[geekpi](https://github.com/geekpi)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://opensource.com/users/seth
[b]: https://github.com/lujun9972
[1]: https://opensource.com/sites/default/files/styles/image-full-size/public/lead-images/computer_desk_home_laptop_browser.png?itok=Y3UVpY0l (Digital images of a computer desktop)
[2]: https://opensource.com/article/19/9/introduction-markdown
[3]: https://www.turnkeylinux.org/nextcloud
[4]: http://nextcloud.com
[5]: https://opensource.com/sites/default/files/uploads/nextcloud-app-install-31_days-markdown-opensource.jpg (Nextcloud app store showing Markdown Editor installer)
[6]: https://opensource.com/sites/default/files/uploads/nextcloud-markdown-31-days-opensource.jpg (Example markdown file )
[7]: https://opensource.com/sites/default/files/uploads/nextcloud-link-31_days_markdown-opensource.jpg (black text on white background, word highlighted in blue to create an automatic link)