mirror of
https://github.com/LCTT/TranslateProject.git
synced 2025-01-25 23:11:02 +08:00
20180204-3 选题
This commit is contained in:
Ezio
parent
fd73baaa74
commit
6110c0846d
@ -0,0 +1,153 @@
|
||||
Building a Linux-based HPC system on the Raspberry Pi with Ansible
|
||||
============================================================
|
||||
|
||||
### Create a high-performance computing cluster with low-cost hardware and open source software.
|
||||
|
||||
|
||||
Image by : opensource.com
|
||||
|
||||
In my [previous article for Opensource.com][14], I introduced the [OpenHPC][15] project, which aims to accelerate innovation in high-performance computing (HPC). This article goes a step further by using OpenHPC's capabilities to build a small HPC system. To call it an _HPC system_ might sound bigger than it is, so maybe it is better to say this is a system based on the [Cluster Building Recipes][16] published by the OpenHPC project.
|
||||
|
||||
The resulting cluster consists of two Raspberry Pi 3 systems acting as compute nodes and one virtual machine acting as the master node:
|
||||
|
||||
|
||||
|
||||
|
||||
My master node is running CentOS on x86_64 and my compute nodes are running a slightly modified CentOS on aarch64.
|
||||
|
||||
This is what the setup looks in real life:
|
||||
|
||||
|
||||
|
||||
|
||||
To set up my system like an HPC system, I followed some of the steps from OpenHPC's Cluster Building Recipes [install guide for CentOS 7.4/aarch64 + Warewulf + Slurm][17] (PDF). This recipe includes provisioning instructions using [Warewulf][18]; because I manually installed my three systems, I skipped the Warewulf parts and created an [Ansible playbook][19] for the steps I took.
|
||||
|
||||
|
||||
Once my cluster was set up by the [Ansible][26] playbooks, I could start to submit jobs to my resource manager. The resource manager, [Slurm][27] in my case, is the instance in the cluster that decides where and when my jobs are executed. One possibility to start a simple job on the cluster is:
|
||||
```
|
||||
[ohpc@centos01 ~]$ srun hostname
|
||||
calvin
|
||||
```
|
||||
|
||||
If I need more resources, I can tell Slurm that I want to run my command on eight CPUs:
|
||||
|
||||
```
|
||||
[ohpc@centos01 ~]$ srun -n 8 hostname
|
||||
hobbes
|
||||
hobbes
|
||||
hobbes
|
||||
hobbes
|
||||
calvin
|
||||
calvin
|
||||
calvin
|
||||
calvin
|
||||
```
|
||||
|
||||
In the first example, Slurm ran the specified command (`hostname`) on a single CPU, and in the second example Slurm ran the command on eight CPUs. One of my compute nodes is named `calvin` and the other is named `hobbes`; that can be seen in the output of the above commands. Each of the compute nodes is a Raspberry Pi 3 with four CPU cores.
|
||||
|
||||
Another way to submit jobs to my cluster is the command `sbatch`, which can be used to execute scripts with the output written to a file instead of my terminal.
|
||||
|
||||
```
|
||||
[ohpc@centos01 ~]$ cat script1.sh
|
||||
#!/bin/sh
|
||||
date
|
||||
hostname
|
||||
sleep 10
|
||||
date
|
||||
[ohpc@centos01 ~]$ sbatch script1.sh
|
||||
Submitted batch job 101
|
||||
```
|
||||
|
||||
This will create an output file called `slurm-101.out` with the following content:
|
||||
|
||||
```
|
||||
Mon 11 Dec 16:42:31 UTC 2017
|
||||
calvin
|
||||
Mon 11 Dec 16:42:41 UTC 2017
|
||||
```
|
||||
|
||||
To demonstrate the basic functionality of the resource manager, simple and serial command line tools are suitable—but a bit boring after doing all the work to set up an HPC-like system.
|
||||
|
||||
A more interesting application is running an [Open MPI][20] parallelized job on all available CPUs on the cluster. I'm using an application based on [Game of Life][21], which was used in a [video][22] called "Running Game of Life across multiple architectures with Red Hat Enterprise Linux." In addition to the previously used MPI-based Game of Life implementation, the version now running on my cluster colors the cells for each involved host differently. The following script starts the application interactively with a graphical output:
|
||||
|
||||
```
|
||||
$ cat life.mpi
|
||||
#!/bin/bash
|
||||
|
||||
module load gnu6 openmpi3
|
||||
|
||||
if [[ "$SLURM_PROCID" != "0" ]]; then
|
||||
exit
|
||||
fi
|
||||
|
||||
mpirun ./mpi_life -a -p -b
|
||||
```
|
||||
|
||||
I start the job with the following command, which tells Slurm to allocate eight CPUs for the job:
|
||||
|
||||
```
|
||||
$ srun -n 8 --x11 life.mpi
|
||||
```
|
||||
|
||||
For demonstration purposes, the job has a graphical interface that shows the current result of the calculation:
|
||||
|
||||
|
||||
|
||||
|
||||
The position of the red cells is calculated on one of the compute nodes, and the green cells are calculated on the other compute node. I can also tell the Game of Life program to color the cell for each used CPU (there are four per compute node) differently, which leads to the following output:
|
||||
|
||||
|
||||
|
||||
|
||||
Thanks to the installation recipes and the software packages provided by OpenHPC, I was able to set up two compute nodes and a master node in an HPC-type configuration. I can submit jobs to my resource manager, and I can use the software provided by OpenHPC to start MPI applications utilizing all my Raspberry Pis' CPUs.
|
||||
|
||||
* * *
|
||||
|
||||
_To learn more about using OpenHPC to build a Raspberry Pi cluster, please attend Adrian Reber's talks at [DevConf.cz 2018][10], January 26-28, in Brno, Czech Republic, and at the [CentOS Dojo 2018][11], on February 2, in Brussels._
|
||||
|
||||
### About the author
|
||||
|
||||
[][23] Adrian Reber - Adrian is a Senior Software Engineer at Red Hat and is migrating processes at least since 2010\. He started to migrate processes in a high performance computing environment and at some point he migrated so many processes that he got a PhD for that and since he joined Red Hat he started to migrate containers. Occasionally he still migrates single processes and is still interested in high performance computing topics.[More about me][12]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/how-build-hpc-system-raspberry-pi-and-openhpc
|
||||
|
||||
作者：[Adrian Reber ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/adrianreber
|
||||
[1]:https://opensource.com/resources/what-are-linux-containers?utm_campaign=containers&intcmp=70160000000h1s6AAA
|
||||
[2]:https://opensource.com/resources/what-docker?utm_campaign=containers&intcmp=70160000000h1s6AAA
|
||||
[3]:https://opensource.com/resources/what-is-kubernetes?utm_campaign=containers&intcmp=70160000000h1s6AAA
|
||||
[4]:https://developers.redhat.com/blog/2016/01/13/a-practical-introduction-to-docker-container-terminology/?utm_campaign=containers&intcmp=70160000000h1s6AAA
|
||||
[5]:https://opensource.com/file/384031
|
||||
[6]:https://opensource.com/file/384016
|
||||
[7]:https://opensource.com/file/384021
|
||||
[8]:https://opensource.com/file/384026
|
||||
[9]:https://opensource.com/article/18/1/how-build-hpc-system-raspberry-pi-and-openhpc?rate=l9n6B6qRcR20LJyXEoUoWEZ4mb2nDc9sFZ1YSPc60vE
|
||||
[10]:https://devconfcz2018.sched.com/event/DJYi/openhpc-introduction
|
||||
[11]:https://wiki.centos.org/Events/Dojo/Brussels2018
|
||||
[12]:https://opensource.com/users/adrianreber
|
||||
[13]:https://opensource.com/user/188446/feed
|
||||
[14]:https://opensource.com/article/17/11/openhpc
|
||||
[15]:https://openhpc.community/
|
||||
[16]:https://openhpc.community/downloads/
|
||||
[17]:https://github.com/openhpc/ohpc/releases/download/v1.3.3.GA/Install_guide-CentOS7-Warewulf-SLURM-1.3.3-aarch64.pdf
|
||||
[18]:https://en.wikipedia.org/wiki/Warewulf
|
||||
[19]:http://people.redhat.com/areber/openhpc/ansible/
|
||||
[20]:https://www.open-mpi.org/
|
||||
[21]:https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life
|
||||
[22]:https://www.youtube.com/watch?v=n8DvxMcOMXk
|
||||
[23]:https://opensource.com/users/adrianreber
|
||||
[24]:https://opensource.com/users/adrianreber
|
||||
[25]:https://opensource.com/users/adrianreber
|
||||
[26]:https://www.ansible.com/
|
||||
[27]:https://slurm.schedmd.com/
|
||||
[28]:https://opensource.com/tags/raspberry-pi
|
||||
[29]:https://opensource.com/tags/programming
|
||||
[30]:https://opensource.com/tags/linux
|
||||
[31]:https://opensource.com/tags/ansible
|
||||
@ -0,0 +1,106 @@
|
||||
An introduction to the Web::Simple Perl module, a minimalist web framework
|
||||
============================================================
|
||||
|
||||
### Perl module Web::Simple is easy to learn and packs a big enough punch for a variety of one-offs and smaller services.
|
||||
|
||||
|
||||
|
||||
Image credits : [You as a Machine][10]. Modified by Rikki Endsley. [CC BY-SA 2.0][11].
|
||||
|
||||
One of the more-prominent members of the Perl community is [Matt Trout][12], technical director at [Shadowcat Systems][13]. He's been building core tools for Perl applications for years, including being a co-maintaner of the [Catalyst][14] MVC (Model, View, Controller) web framework, creator of the [DBIx::Class][15] object-management system, and much more. In person, he's energetic, interesting, brilliant, and sometimes hard to keep up with. When Matt writes code…well, think of a runaway chainsaw, with the trigger taped down and the safety features disabled. He's off and running, and you never quite know what will come out. Two things are almost certain: the module will precisely fit the purpose Matt has in mind, and it will show up on CPAN for others to use.
|
||||
|
||||
|
||||
One of Matt's special-purpose modules is [Web::Simple][23]. Touted as "a quick and easy way to build simple web applications," it is a stripped-down, minimalist web framework, with an easy to learn interface. Web::Simple is not at all designed for a large-scale application; however, it may be ideal for a small tool that does one or two things in a lower-traffic environment. I can also envision it being used for rapid prototyping if you wanted to create quick wireframes of a new application for demonstrations.
|
||||
|
||||
### Installation, and a quick "Howdy!"
|
||||
|
||||
You can install the module using `cpan` or `cpanm`. Once you've got it installed, you're ready to write simple web apps without having to hassle with managing the connections or any of that—just your functionality. Here's a quick example:
|
||||
|
||||
```
|
||||
#!/usr/bin/perl
|
||||
package HelloReader;
|
||||
use Web::Simple;
|
||||
|
||||
sub dispatch_request {
|
||||
GET => sub {
|
||||
[ 200, [ 'Content-type', 'text/plain' ], [ 'Howdy, Opensource.com reader!' ] ]
|
||||
},
|
||||
'' => sub {
|
||||
[ 405, [ 'Content-type', 'text/plain' ], [ 'You cannot do that, friend. Sorry.' ] ]
|
||||
}
|
||||
}
|
||||
|
||||
HelloReader->run_if_script;
|
||||
```
|
||||
|
||||
There are a couple of things to notice right off. For one, I didn't `use strict` and `use warnings` like I usually would. Web::Simple imports those for you, so you don't have to. It also imports [Moo][16], a minimalist OO framework, so if you know Moo and want to use it here, you can! The heart of the system lies in the `dispatch_request`method, which you must define in your application. Each entry in the method is a match string, followed by a subroutine to respond if that string matches. The subroutine must return an array reference containing status, headers, and content of the reply to the request.
|
||||
|
||||
### Matching
|
||||
|
||||
The matching system in Web::Simple is powerful, allowing for complicated matches, passing parameters in a URL, query parameters, and extension matches, in pretty much any combination you want. As you can see in the example above, starting with a capital letter will match on the request method, and you can combine that with a path match easily:
|
||||
|
||||
```
|
||||
'GET + /person/*' => sub {
|
||||
my ($self, $person) = @_;
|
||||
# write some code to retrieve and display a person
|
||||
},
|
||||
'POST + /person/* + %*' => sub {
|
||||
my ($self, $person, $params) = @_;
|
||||
# write some code to modify a person, perhaps
|
||||
}
|
||||
```
|
||||
|
||||
In the latter case, the third part of the match indicates that we should pick up all the POST parameters and put them in a hashref called `$params` for use by the subroutine. Using `?` instead of `%` in that part of the match would pick up query parameters, as normally used in a GET request. There's also a useful exported subroutine called `redispatch_to`. This tool lets you redirect, without using a 3xx redirect; it's handled internally, invisible to the user. So:
|
||||
|
||||
```
|
||||
'GET + /some/url' => sub {
|
||||
redispatch_to '/some/other/url';
|
||||
}
|
||||
```
|
||||
|
||||
A GET request to `/some/url` would get handled as if it was sent to `/some/other/url`, without a redirect, and the user won't see a redirect in their browser.
|
||||
|
||||
I've just scratched the surface with this module. If you're looking for something production-ready for larger projects, you'll be better off with [Dancer][17] or [Catalyst][18]. But with its light weight and built-in Moo integration, Web::Simple packs a big enough punch for a variety of one-offs and smaller services.
|
||||
|
||||
### About the author
|
||||
|
||||
[][19] Ruth Holloway - Ruth Holloway has been a system administrator and software developer for a long, long time, getting her professional start on a VAX 11/780, way back when. She spent a lot of her career (so far) serving the technology needs of libraries, and has been a contributor since 2008 to the Koha open source library automation suite.Ruth is currently a Perl Developer at cPanel in Houston, and also serves as chief of staff for an obnoxious cat. In her copious free time, she occasionally reviews old romance... [more about Ruth Holloway][7][More about me][8]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/introduction-websimple-perl-module-minimalist-web-framework
|
||||
|
||||
作者：[Ruth Holloway ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/druthb
|
||||
[1]:https://opensource.com/tags/python?src=programming_resource_menu1
|
||||
[2]:https://opensource.com/tags/javascript?src=programming_resource_menu2
|
||||
[3]:https://opensource.com/tags/perl?src=programming_resource_menu3
|
||||
[4]:https://developers.redhat.com/?intcmp=7016000000127cYAAQ&src=programming_resource_menu4
|
||||
[5]:http://perldoc.perl.org/functions/package.html
|
||||
[6]:https://opensource.com/article/18/1/introduction-websimple-perl-module-minimalist-web-framework?rate=ICN35y076ElpInDKoMqp-sN6f4UVF-n2Qt6dL6lb3kM
|
||||
[7]:https://opensource.com/users/druthb
|
||||
[8]:https://opensource.com/users/druthb
|
||||
[9]:https://opensource.com/user/36051/feed
|
||||
[10]:https://www.flickr.com/photos/youasamachine/8025582590/in/photolist-decd6C-7pkccp-aBfN9m-8NEffu-3JDbWb-aqf5Tx-7Z9MTZ-rnYTRu-3MeuPx-3yYwA9-6bSLvd-irmvxW-5Asr4h-hdkfCA-gkjaSQ-azcgct-gdV5i4-8yWxCA-9G1qDn-5tousu-71V8U2-73D4PA-iWcrTB-dDrya8-7GPuxe-5pNb1C-qmnLwy-oTxwDW-3bFhjL-f5Zn5u-8Fjrua-bxcdE4-ddug5N-d78G4W-gsYrFA-ocrBbw-pbJJ5d-682rVJ-7q8CbF-7n7gDU-pdfgkJ-92QMx2-aAmM2y-9bAGK1-dcakkn-8rfyTz-aKuYvX-hqWSNP-9FKMkg-dyRPkY
|
||||
[11]:https://creativecommons.org/licenses/by/2.0/
|
||||
[12]:https://shadow.cat/resources/bios/matt_short/
|
||||
[13]:https://shadow.cat/
|
||||
[14]:https://metacpan.org/pod/Catalyst
|
||||
[15]:https://metacpan.org/pod/DBIx::Class
|
||||
[16]:https://metacpan.org/pod/Moo
|
||||
[17]:http://perldancer.org/
|
||||
[18]:http://www.catalystframework.org/
|
||||
[19]:https://opensource.com/users/druthb
|
||||
[20]:https://opensource.com/users/druthb
|
||||
[21]:https://opensource.com/users/druthb
|
||||
[22]:https://opensource.com/article/18/1/introduction-websimple-perl-module-minimalist-web-framework#comments
|
||||
[23]:https://metacpan.org/pod/Web::Simple
|
||||
[24]:https://opensource.com/tags/perl
|
||||
[25]:https://opensource.com/tags/programming
|
||||
[26]:https://opensource.com/tags/perl-column
|
||||
[27]:https://opensource.com/tags/web-development
|
||||
@ -0,0 +1,280 @@
|
||||
Running a Python application on Kubernetes
|
||||
============================================================
|
||||
|
||||
### This step-by-step tutorial takes you through the process of deploying a simple Python application on Kubernetes.
|
||||
|
||||
|
||||
Image by : opensource.com
|
||||
|
||||
Kubernetes is an open source platform that offers deployment, maintenance, and scaling features. It simplifies management of containerized Python applications while providing portability, extensibility, and self-healing capabilities.
|
||||
|
||||
Whether your Python applications are simple or more complex, Kubernetes lets you efficiently deploy and scale them, seamlessly rolling out new features while limiting resources to only those required.
|
||||
|
||||
In this article, I will describe the process of deploying a simple Python application to Kubernetes, including:
|
||||
|
||||
* Creating Python container images
|
||||
|
||||
* Publishing the container images to an image registry
|
||||
|
||||
* Working with persistent volume
|
||||
|
||||
* Deploying the Python application to Kubernetes
|
||||
|
||||
### Requirements
|
||||
|
||||
You will need Docker, kubectl, and this [source code][10].
|
||||
|
||||
Docker is an open platform to build and ship distributed applications. To install Docker, follow the [official documentation][11]. To verify that Docker runs your system:
|
||||
|
||||
```
|
||||
$ docker info
|
||||
Containers: 0
|
||||
Images: 289
|
||||
Storage Driver: aufs
|
||||
Root Dir: /var/lib/docker/aufs
|
||||
Dirs: 289
|
||||
Execution Driver: native-0.2
|
||||
Kernel Version: 3.16.0-4-amd64
|
||||
Operating System: Debian GNU/Linux 8 (jessie)
|
||||
WARNING: No memory limit support
|
||||
WARNING: No swap limit support
|
||||
```
|
||||
|
||||
kubectl is a command-line interface for executing commands against a Kubernetes cluster. Run the shell script below to install kubectl:
|
||||
|
||||
```
|
||||
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
|
||||
```
|
||||
|
||||
Deploying to Kubernetes requires a containerized application. Let's review containerizing Python applications.
|
||||
|
||||
### Containerization at a glance
|
||||
|
||||
Containerization involves enclosing an application in a container with its own operating system. This full machine virtualization option has the advantage of being able to run an application on any machine without concerns about dependencies.
|
||||
|
||||
Roman Gaponov's [article][12] serves as a reference. Let's start by creating a container image for our Python code.
|
||||
|
||||
### Create a Python container image
|
||||
|
||||
To create these images, we will use Docker, which enables us to deploy applications inside isolated Linux software containers. Docker is able to automatically build images using instructions from a Docker file.
|
||||
|
||||
This is a Docker file for our Python application:
|
||||
|
||||
```
|
||||
FROM python:3.6
|
||||
MAINTAINER XenonStack
|
||||
|
||||
# Creating Application Source Code Directory
|
||||
RUN mkdir -p /k8s_python_sample_code/src
|
||||
|
||||
# Setting Home Directory for containers
|
||||
WORKDIR /k8s_python_sample_code/src
|
||||
|
||||
# Installing python dependencies
|
||||
COPY requirements.txt /k8s_python_sample_code/src
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copying src code to Container
|
||||
COPY . /k8s_python_sample_code/src/app
|
||||
|
||||
# Application Environment variables
|
||||
ENV APP_ENV development
|
||||
|
||||
# Exposing Ports
|
||||
EXPOSE 5035
|
||||
|
||||
# Setting Persistent data
|
||||
VOLUME ["/app-data"]
|
||||
|
||||
# Running Python Application
|
||||
CMD ["python", "app.py"]
|
||||
```
|
||||
|
||||
This Docker file contains instructions to run our sample Python code. It uses the Python 3.5 development environment.
|
||||
|
||||
### Build a Python Docker image
|
||||
|
||||
We can now build the Docker image from these instructions using this command:
|
||||
|
||||
```
|
||||
docker build -t k8s_python_sample_code .
|
||||
```
|
||||
|
||||
This command creates a Docker image for our Python application.
|
||||
|
||||
### Publish the container images
|
||||
|
||||
We can publish our Python container image to different private/public cloud repositories, like Docker Hub, AWS ECR, Google Container Registry, etc. For this tutorial, we'll use Docker Hub.
|
||||
|
||||
Before publishing the image, we need to tag it to a version:
|
||||
|
||||
```
|
||||
docker tag k8s_python_sample_code:latest k8s_python_sample_code:0.1
|
||||
```
|
||||
|
||||
### Push the image to a cloud repository
|
||||
|
||||
Using a Docker registry other than Docker Hub to store images requires you to add that container registry to the local Docker daemon and Kubernetes Docker daemons. You can look up this information for the different cloud registries. We'll use Docker Hub in this example.
|
||||
|
||||
Execute this Docker command to push the image:
|
||||
|
||||
```
|
||||
docker push k8s_python_sample_code
|
||||
```
|
||||
|
||||
### Working with CephFS persistent storage
|
||||
|
||||
Kubernetes supports many persistent storage providers, including AWS EBS, CephFS, GlusterFS, Azure Disk, NFS, etc. I will cover Kubernetes persistence storage with CephFS.
|
||||
|
||||
To use CephFS for persistent data to Kubernetes containers, we will create two files:
|
||||
|
||||
persistent-volume.yml
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolume
|
||||
metadata:
|
||||
name: app-disk1
|
||||
namespace: k8s_python_sample_code
|
||||
spec:
|
||||
capacity:
|
||||
storage: 50Gi
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
cephfs:
|
||||
monitors:
|
||||
- "172.17.0.1:6789"
|
||||
user: admin
|
||||
secretRef:
|
||||
name: ceph-secret
|
||||
readOnly: false
|
||||
```
|
||||
|
||||
persistent_volume_claim.yaml
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: appclaim1
|
||||
namespace: k8s_python_sample_code
|
||||
spec:
|
||||
accessModes:
|
||||
- ReadWriteMany
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
```
|
||||
|
||||
We can now use kubectl to add the persistent volume and claim to the Kubernetes cluster:
|
||||
|
||||
```
|
||||
$ kubectl create -f persistent-volume.yml
|
||||
$ kubectl create -f persistent-volume-claim.yml
|
||||
```
|
||||
|
||||
We are now ready to deploy to Kubernetes.
|
||||
|
||||
### Deploy the application to Kubernetes
|
||||
|
||||
To manage the last mile of deploying the application to Kubernetes, we will create two important files: a service file and a deployment file.
|
||||
|
||||
Create a file and name it `k8s_python_sample_code.service.yml` with the following content:
|
||||
|
||||
```
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
labels:
|
||||
k8s-app: k8s_python_sample_code
|
||||
name: k8s_python_sample_code
|
||||
namespace: k8s_python_sample_code
|
||||
spec:
|
||||
type: NodePort
|
||||
ports:
|
||||
- port: 5035
|
||||
selector:
|
||||
k8s-app: k8s_python_sample_code
|
||||
```
|
||||
|
||||
Create a file and name it `k8s_python_sample_code.deployment.yml` with the following content:
|
||||
|
||||
```
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: k8s_python_sample_code
|
||||
namespace: k8s_python_sample_code
|
||||
spec:
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
k8s-app: k8s_python_sample_code
|
||||
spec:
|
||||
containers:
|
||||
- name: k8s_python_sample_code
|
||||
image: k8s_python_sample_code:0.1
|
||||
imagePullPolicy: "IfNotPresent"
|
||||
ports:
|
||||
- containerPort: 5035
|
||||
volumeMounts:
|
||||
- mountPath: /app-data
|
||||
name: k8s_python_sample_code
|
||||
volumes:
|
||||
- name: <name of application>
|
||||
persistentVolumeClaim:
|
||||
claimName: appclaim1
|
||||
```
|
||||
|
||||
Finally, use kubectl to deploy the application to Kubernetes:
|
||||
|
||||
```
|
||||
$ kubectl create -f k8s_python_sample_code.deployment.yml $ kubectl create -f k8s_python_sample_code.service.yml
|
||||
```
|
||||
|
||||
Your application was successfully deployed to Kubernetes.
|
||||
|
||||
You can verify whether your application is running by inspecting the running services:
|
||||
|
||||
```
|
||||
kubectl get services
|
||||
```
|
||||
|
||||
May Kubernetes free you from future deployment hassles!
|
||||
|
||||
_Want to learn more about Python? Nanjekye's book, [Python 2 and 3 Compatibility][7]offers clean ways to write code that will run on both Python 2 and 3, including detailed examples of how to convert existing Python 2-compatible code to code that will run reliably on both Python 2 and 3._
|
||||
|
||||
|
||||
### About the author
|
||||
|
||||
[][13] Joannah Nanjekye - Straight Outta 256 , I choose Results over Reasons, Passionate Aviator, Show me the code.[More about me][8]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/running-python-application-kubernetes
|
||||
|
||||
作者：[Joannah Nanjekye ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/nanjekyejoannah
|
||||
[1]:https://opensource.com/resources/python?intcmp=7016000000127cYAAQ
|
||||
[2]:https://opensource.com/resources/python/ides?intcmp=7016000000127cYAAQ
|
||||
[3]:https://opensource.com/resources/python/gui-frameworks?intcmp=7016000000127cYAAQ
|
||||
[4]:https://opensource.com/tags/python?intcmp=7016000000127cYAAQ
|
||||
[5]:https://developers.redhat.com/?intcmp=7016000000127cYAAQ
|
||||
[6]:https://opensource.com/article/18/1/running-python-application-kubernetes?rate=D9iKksKbd9q9vOVb92Mg-v0Iyqn0QVO5fbIERTbSHz4
|
||||
[7]:https://www.apress.com/gp/book/9781484229545
|
||||
[8]:https://opensource.com/users/nanjekyejoannah
|
||||
[9]:https://opensource.com/user/196386/feed
|
||||
[10]:https://github.com/jnanjekye/k8s_python_sample_code/tree/master
|
||||
[11]:https://docs.docker.com/engine/installation/
|
||||
[12]:https://hackernoon.com/docker-tutorial-getting-started-with-python-redis-and-nginx-81a9d740d091
|
||||
[13]:https://opensource.com/users/nanjekyejoannah
|
||||
[14]:https://opensource.com/users/nanjekyejoannah
|
||||
[15]:https://opensource.com/users/nanjekyejoannah
|
||||
[16]:https://opensource.com/tags/python
|
||||
[17]:https://opensource.com/tags/kubernetes
|
||||
140
sources/tech/20180128 Being open about data privacy.md
Normal file
140
sources/tech/20180128 Being open about data privacy.md
Normal file
@ -0,0 +1,140 @@
|
||||
Being open about data privacy
|
||||
============================================================
|
||||
|
||||
### Regulations including GDPR notwithstanding, data privacy is something that's important for pretty much everybody.
|
||||
|
||||
|
||||
Image by : opensource.com
|
||||
|
||||
Today is [Data Privacy Day][9], ("Data Protection Day" in Europe), and you might think that those of us in the open source world should think that all data should be free, [as information supposedly wants to be][10], but life's not that simple. That's for two main reasons:
|
||||
|
||||
1. Most of us (and not just in open source) believe there's at least some data about us that we might not feel happy sharing (I compiled an example list in [a post][2] I published a while ago).
|
||||
|
||||
2. Many of us working in open source actually work for commercial companies or other organisations subject to legal requirements around what they can share.
|
||||
|
||||
So actually, data privacy is something that's important for pretty much everybody.
|
||||
|
||||
It turns out that the starting point for what data people and governments believe should be available for organisations to use is somewhat different between the U.S. and Europe, with the former generally providing more latitude for entities—particularly, the more cynical might suggest, large commercial entities—to use data they've collected about us as they will. Europe, on the other hand, has historically taken a more restrictive view, and on the 25th of May, Europe's view arguably will have triumphed.
|
||||
|
||||
### The impact of GDPR
|
||||
|
||||
That's a rather sweeping statement, but the fact remains that this is the date on which a piece of legislation called the General Data Protection Regulation (GDPR), enacted by the European Union in 2016, becomes enforceable. The GDPR basically provides a stringent set of rules about how personal data can be stored, what it can be used for, who can see it, and how long it can be kept. It also describes what personal data is—and it's a pretty broad set of items, from your name and home address to your medical records and on through to your computer's IP address.
|
||||
|
||||
What is important about the GDPR, though, is that it doesn't apply just to European companies, but to any organisation processing data about EU citizens. If you're an Argentinian, Japanese, U.S., or Russian company and you're collecting data about an EU citizen, you're subject to it.
|
||||
|
||||
"Pah!" you may say,[1][11] "I'm not based in the EU: what can they do to me?" The answer is simple: If you want to continue doing any business in the EU, you'd better comply, because if you breach GDPR rules, you could be liable for up to four percent of your _global_ revenues. Yes, that's global revenues: not just revenues in a particular country in Europe or across the EU, not just profits, but _global revenues_ . Those are the sorts of numbers that should lead you to talk to your legal team, who will direct you to your exec team, who will almost immediately direct you to your IT group to make sure you're compliant in pretty short order.
|
||||
|
||||
This may seem like it's not particularly relevant to non-EU citizens, but it is. For most companies, it's going to be simpler and more efficient to implement the same protection measures for data associated with _all_ customers, partners, and employees they deal with, rather than just targeting specific measures at EU citizens. This has got to be a good thing.[2][12]
|
||||
|
||||
However, just because GDPR will soon be applied to organisations across the globe doesn't mean that everything's fine and dandy[3][13]: it's not. We give away information about ourselves all the time—and permission for companies to use it.
|
||||
|
||||
There's a telling (though disputed) saying: "If you're not paying, you're the product." What this suggests is that if you're not paying for a service, then somebody else is paying to use your data. Do you pay to use Facebook? Twitter? Gmail? How do you think they make their money? Well, partly through advertising, and some might argue that's a service they provide to you, but actually that's them using your data to get money from the advertisers. You're not really a customer of advertising—it's only once you buy something from the advertiser that you become their customer, but until you do, the relationship is between the the owner of the advertising platform and the advertiser.
|
||||
|
||||
Some of these services allow you to pay to reduce or remove advertising (Spotify is a good example), but on the other hand, advertising may be enabled even for services that you think you do pay for (Amazon is apparently working to allow adverts via Alexa, for instance). Unless we want to start paying to use all of these "free" services, we need to be aware of what we're giving up, and making some choices about what we expose and what we don't.
|
||||
|
||||
### Who's the customer?
|
||||
|
||||
There's another issue around data that should be exercising us, and it's a direct consequence of the amounts of data that are being generated. There are many organisations out there—including "public" ones like universities, hospitals, or government departments[4][14]—who generate enormous quantities of data all the time, and who just don't have the capacity to store it. It would be a different matter if this data didn't have long-term value, but it does, as the tools for handling Big Data are developing, and organisations are realising they can be mining this now and in the future.
|
||||
|
||||
The problem they face, though, as the amount of data increases and their capacity to store it fails to keep up, is what to do with it. _Luckily_ —and I use this word with a very heavy dose of irony,[5][15] big corporations are stepping in to help them. "Give us your data," they say, "and we'll host it for free. We'll even let you use the data you collected when you want to!" Sounds like a great deal, yes? A fantastic example of big corporations[6][16] taking a philanthropic stance and helping out public organisations that have collected all of that lovely data about us.
|
||||
|
||||
Sadly, philanthropy isn't the only reason. These hosting deals come with a price: in exchange for agreeing to host the data, these corporations get to sell access to it to third parties. And do you think the public organisations, or those whose data is collected, will get a say in who these third parties are or how they will use it? I'll leave this as an exercise for the reader.[7][17]
|
||||
|
||||
### Open and positive
|
||||
|
||||
It's not all bad news, however. There's a growing "open data" movement among governments to encourage departments to make much of their data available to the public and other bodies for free. In some cases, this is being specifically legislated. Many voluntary organisations—particularly those receiving public funding—are starting to do the same. There are glimmerings of interest even from commercial organisations. What's more, there are techniques becoming available, such as those around differential privacy and multi-party computation, that are beginning to allow us to mine data across data sets without revealing too much about individuals—a computing problem that has historically been much less tractable than you might otherwise expect.
|
||||
|
||||
What does this all mean to us? Well, I've written before on Opensource.com about the [commonwealth of open source][18], and I'm increasingly convinced that we need to look beyond just software to other areas: hardware, organisations, and, relevant to this discussion, data. Let's imagine that you're a company (A) that provides a service to another company, a customer (B).[8][19] There are four different types of data in play:
|
||||
|
||||
1. Data that's fully open: visible to A, B, and the rest of the world
|
||||
|
||||
2. Data that's known, shared, and confidential: visible to A and B, but nobody else
|
||||
|
||||
3. Data that's company-confidential: visible to A, but not B
|
||||
|
||||
4. Data that's customer-confidential: visible to B, but not A
|
||||
|
||||
First of all, maybe we should be a bit more open about data and default to putting it into bucket 1\. That data—on self-driving cars, voice recognition, mineral deposits, demographic statistics—could be enormously useful if it were available to everyone.[9][20]Also, wouldn't it be great if we could find ways to make the data in buckets 2, 3, and 4—or at least some of it—available in bucket 1, whilst still keeping the details confidential? That's the hope for some of these new techniques being researched. They're a way off, though, so don't get too excited, and in the meantime, start thinking about making more of your data open by default.
|
||||
|
||||
### Some concrete steps
|
||||
|
||||
So, what can we do around data privacy and being open? Here are a few concrete steps that occurred to me: please use the comments to contribute more.
|
||||
|
||||
* Check to see whether your organisation is taking GDPR seriously. If it isn't, push for it.
|
||||
|
||||
* Default to encrypting sensitive data (or hashing where appropriate), and deleting when it's no longer required—there's really no excuse for data to be in the clear to these days except for when it's actually being processed.
|
||||
|
||||
* Consider what information you disclose when you sign up to services, particularly social media.
|
||||
|
||||
* Discuss this with your non-technical friends.
|
||||
|
||||
* Educate your children, your friends' children, and their friends. Better yet, go and talk to their teachers about it and present something in their schools.
|
||||
|
||||
* Encourage the organisations you work for, volunteer for, or interact with to make data open by default. Rather than thinking, "why should I make this public?" start with "why _shouldn't_ I make this public?"
|
||||
|
||||
* Try accessing some of the open data sources out there. Mine it, create apps that use it, perform statistical analyses, draw pretty graphs,[10][3] make interesting music, but consider doing something with it. Tell the organisations that sourced it, thank them, and encourage them to do more.
|
||||
|
||||
* * *
|
||||
|
||||
1\. Though you probably won't, I admit.
|
||||
|
||||
2\. Assuming that you believe that your personal data should be protected.
|
||||
|
||||
3\. If you're wondering what "dandy" means, you're not alone at this point.
|
||||
|
||||
4\. Exactly how public these institutions seem to you will probably depend on where you live: [YMMV][21].
|
||||
|
||||
5\. And given that I'm British, that's a really very, very heavy dose.
|
||||
|
||||
6\. And they're likely to be big corporations: nobody else can afford all of that storage and the infrastructure to keep it available.
|
||||
|
||||
7\. No. The answer's "no."
|
||||
|
||||
8\. Although the example works for people, too. Oh, look: A could be Alice, B could be Bob…
|
||||
|
||||
9\. Not that we should be exposing personal data or data that actually needs to be confidential, of course—not that type of data.
|
||||
|
||||
10\. A friend of mine decided that it always seemed to rain when she picked her children up from school, so to avoid confirmation bias, she accessed rainfall information across the school year and created graphs that she shared on social media.
|
||||
|
||||
|
||||
|
||||
### About the author
|
||||
|
||||
[][22] Mike Bursell - I've been in and around Open Source since around 1997, and have been running (GNU) Linux as my main desktop at home and work since then: [not always easy][4]... I'm a security bod and architect, and am currently employed as Chief Security Architect for Red Hat. I have a blog - "[Alice, Eve & Bob][5]" - where I write (sometimes rather parenthetically) about security. I live in the UK and... [more about Mike Bursell][6][More about me][7]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/being-open-about-data-privacy
|
||||
|
||||
作者：[Mike Bursell ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/mikecamel
|
||||
[1]:https://opensource.com/article/18/1/being-open-about-data-privacy?rate=oQCDAM0DY-P97d3pEEW_yUgoCV1ZXhv8BHYTnJVeHMc
|
||||
[2]:https://aliceevebob.wordpress.com/2017/06/06/helping-our-governments-differently/
|
||||
[3]:https://opensource.com/article/18/1/being-open-about-data-privacy#10
|
||||
[4]:https://opensource.com/article/17/11/politics-linux-desktop
|
||||
[5]:https://aliceevebob.com/
|
||||
[6]:https://opensource.com/users/mikecamel
|
||||
[7]:https://opensource.com/users/mikecamel
|
||||
[8]:https://opensource.com/user/105961/feed
|
||||
[9]:https://en.wikipedia.org/wiki/Data_Privacy_Day
|
||||
[10]:https://en.wikipedia.org/wiki/Information_wants_to_be_free
|
||||
[11]:https://opensource.com/article/18/1/being-open-about-data-privacy#1
|
||||
[12]:https://opensource.com/article/18/1/being-open-about-data-privacy#2
|
||||
[13]:https://opensource.com/article/18/1/being-open-about-data-privacy#3
|
||||
[14]:https://opensource.com/article/18/1/being-open-about-data-privacy#4
|
||||
[15]:https://opensource.com/article/18/1/being-open-about-data-privacy#5
|
||||
[16]:https://opensource.com/article/18/1/being-open-about-data-privacy#6
|
||||
[17]:https://opensource.com/article/18/1/being-open-about-data-privacy#7
|
||||
[18]:https://opensource.com/article/17/11/commonwealth-open-source
|
||||
[19]:https://opensource.com/article/18/1/being-open-about-data-privacy#8
|
||||
[20]:https://opensource.com/article/18/1/being-open-about-data-privacy#9
|
||||
[21]:http://www.outpost9.com/reference/jargon/jargon_40.html#TAG2036
|
||||
[22]:https://opensource.com/users/mikecamel
|
||||
[23]:https://opensource.com/users/mikecamel
|
||||
[24]:https://opensource.com/users/mikecamel
|
||||
[25]:https://opensource.com/tags/open-data
|
||||
@ -0,0 +1,245 @@
|
||||
CopperheadOS: Security features, installing apps, and more
|
||||
============================================================
|
||||
|
||||
### Fly your open source flag proudly with Copperhead, a mobile OS that takes its FOSS commitment seriously.
|
||||
|
||||
|
||||
|
||||
Image by : Norebbo via [Flickr][15] (Original: [public domain][16]). Modified by Opensource.com. [CC BY-SA 4.0][17].
|
||||
|
||||
_Editor's note: CopperheadOS is [licensed][11] under the Creative Commons Attribution-NonCommercial-Shar<wbr>eAlike 4.0 license (userspace) and GPL2 license (kernel). It is also based on Android Open Source Project (AOSP)._
|
||||
|
||||
Several years ago, I made the decision to replace proprietary technologies (mainly Apple products) with technology that ran on free and open source software (FOSS). I can't say it was easy, but I now happily use FOSS for pretty much everything.
|
||||
|
||||
The hardest part involved my mobile handset. There are basically only two choices today for phones and tablets: Apple's iOS or Google's Android. Since Android is open source, it seemed the obvious choice, but I was frustrated by both the lack of open source applications on Android and the pervasiveness of Google on those devices.
|
||||
|
||||
So I entered the world of custom ROMs. These are projects that take the base [Android Open Source Project][18] (AOSP) and customize it. Almost all these projects allow you to install the standard Google applications as a separate package, called GApps, and you can have as much or as little Google presence on your phone as you like. GApps packages come in a number of flavors, from the full suite of apps that Google ships with its devices to a "pico" version that includes just the minimal amount of software needed to run the Google Play Store, and from there you can add what you like.
|
||||
|
||||
I started out using CyanogenMod, but when that project went in a direction I didn't like, I switched to OmniROM. I was quite happy with it, but still wondered what information I was sending to Google behind the scenes.
|
||||
|
||||
Then I found out about [CopperheadOS][19]. Copperhead is a version of AOSP that focuses on delivering the most secure Android experience possible. I've been using it for a year now and have been quite happy with it.
|
||||
|
||||
Unlike other custom ROMs that strive to add lots of new functionality, Copperhead runs a pretty vanilla version of AOSP. Also, while the first thing you usually do when playing with a custom ROM is to add root access to the device, not only does Copperhead prevent that, it also requires that you have a device that has verified boot, so there's no unlocking the bootloader. This is to prevent malicious code from getting access to the handset.
|
||||
|
||||
Copperhead starts with a hardened version of the AOSP baseline, including full encryption, and then adds a [ton of stuff][20] I can only pretend to understand. It also applies a number of kernel and Android patches before they are applied to the mainline Android releases.
|
||||
|
||||
### [copperos_extrapatches.png][1]
|
||||
|
||||
|
||||
|
||||
It has a couple of more obvious features that I like. If you use a PIN to unlock your device, there is an option to scramble the digits.
|
||||
|
||||
### [copperos_scrambleddigits.png][2]
|
||||
|
||||
|
||||
|
||||
This should prevent any casual shoulder-surfer from figuring out your PIN, although it can make it a bit more difficult to unlock your device while, say, driving (but no one should be using their handset in the car, right?).
|
||||
|
||||
Another issue it addresses involves tracking people by monitoring their WiFi MAC address. Most devices that use WiFi perform active scanning for wireless access points. This protocol includes the MAC address of the interface, and there are a number of ways people can use [mobile location analytics][21] to track your movement. Copperhead has an option to randomize your MAC address, which counters this process.
|
||||
|
||||
### [copperos_randommac.png][3]
|
||||
|
||||
|
||||
|
||||
### Installing apps
|
||||
|
||||
This all sounds pretty good, right? Well, here comes the hard part. While Android is open source, much of the Google code, including the [Google Play Store][22], is not. If you install the Play Store and the code necessary for it to work, you allow Google to install software without your permission. [Google Play's terms of service][23] says:
|
||||
|
||||
> "Google may update any Google app or any app you have downloaded from Google Play to a new version of such app, irrespective of any update settings that you may have selected within the Google Play app or your Device, if Google determines that the update will fix a critical security vulnerability related to the app."
|
||||
|
||||
This is not acceptable from a security standpoint, so you cannot install Google applications on a Copperhead device.
|
||||
|
||||
This took some getting used to, as I had come to rely on things such as Google Maps. The default application repository that ships with Copperhead is [F-Droid][24], which contains only FOSS applications. While I previously used many FOSS applications on Android, it took some effort to use _nothing but_ free software. I did find some ways to cheat this system, and I'll cover that below. First, here are some of the applications I've grown to love from F-Droid.
|
||||
|
||||
### F-Droid favorites
|
||||
|
||||
**K-9 Mail**
|
||||
|
||||
### [copperheados_k9mail.png][4]
|
||||
|
||||
|
||||
|
||||
Even before I started using Copperhead, I loved [K-9 Mail][25]. This is simply the best mobile email client I've found, period, and it is one of the first things I install on any new device. I even use it to access my Gmail account, via IMAP and SMTP.
|
||||
|
||||
**Open Camera**
|
||||
|
||||
### [copperheados_cameraapi.png][5]
|
||||
|
||||
|
||||
|
||||
Copperhead runs only on rather new hardware, and I was consistently disappointed in the quality of the pictures from its default camera application. Then I discovered [Open Camera][26]. A full-featured camera app, it allows you to enable an advanced API to take advantage of the camera hardware. The only thing I miss is the ability to take a panoramic photo.
|
||||
|
||||
**Amaze**
|
||||
|
||||
### [copperheados_amaze.png][6]
|
||||
|
||||
|
||||
|
||||
[Amaze][27] is one of the best file managers I've ever used, free or not. When I need to navigate the filesystem, Amaze is my go-to app.
|
||||
|
||||
**Vanilla Music**
|
||||
|
||||
### [copperheados_vanillamusic.png][7]
|
||||
|
||||
|
||||
|
||||
I was unhappy with the default music player, so I checked out a number of them on F-Droid and settled on [Vanilla Music][28]. It has an easy-to-use interface and interacts well with my Bluetooth devices.
|
||||
|
||||
**OCReader**
|
||||
|
||||
### [coperheados_ocreader.png][8]
|
||||
|
||||
|
||||
|
||||
I am a big fan of [Nextcloud][29], particularly [Nextcloud News][30], a replacement for the now-defunct [Google Reader][31]. While I can access my news feeds through a web browser, I really missed the ability to manage them through a dedicated app. Enter [OCReader][32]. While it stands for "ownCloud Reader," it works with Nextcloud, and I've had very few issues with it.
|
||||
|
||||
**Noise**
|
||||
|
||||
The SMS/MMS application of choice for most privacy advocates is [Signal][33] by [Open Whisper Systems][34]. Endorsed by [Edward Snowden][35], Signal allows for end-to-end encrypted messaging. If the person you are messaging is also on Signal, your messages will be sent, encrypted, over a data connection facilitated by centralized servers maintained by Open Whisper Systems. It also, until recently, relied on [Google Cloud Messaging][36] (GCM) for notifications, which requires Google Play Services.
|
||||
|
||||
The fact that Signal requires a centralized server bothered some people, so the default application on Copperhead is a fork of Signal called [Silence][37]. This application doesn't use a centralized server but does require that all parties be on Silence for encryption to work.
|
||||
|
||||
Well, no one I know uses Silence. At the moment you can't even get it from the Google Play Store in the U.S. due to a trademark issue, and there is no iOS client. An encrypted SMS client isn't very useful if you can't use it for encryption.
|
||||
|
||||
Enter [Noise][38]. Noise is another application maintained by Copperhead that is a fork of Signal that removes the need for GCM. While not available in the standard F-Droid repositories, Copperhead includes their own repository in the version of F-Droid they ship, which at the moment contains only the Noise application. This app will let you communicate securely with anyone else using Noise or Signal.
|
||||
|
||||
### F-Droid workarounds
|
||||
|
||||
**FFUpdater**
|
||||
|
||||
Copperhead ships with a hardened version of the Chromium web browser, but I am a Firefox fan. Unfortunately, [Firefox is no longer included][39] in the F-Droid repository. Apps on F-Droid are all built by the F-Droid maintainers, so the process for getting into F-Droid can be complicated. The [Compass app for OpenNMS][40] isn't in F-Droid because, at the moment, it does not support builds using the [Ionic Framework][41], which Compass uses.
|
||||
|
||||
Luckily, there is a simple workaround: Install the [FFUpdater][42] app on F-Droid. This allows me to install Firefox and keep it up to date through the browser itself.
|
||||
|
||||
**Amazon Appstore**
|
||||
|
||||
This brings me to a cool feature of Android 8, Oreo. In previous versions of Android, you had a single "known source" for software, usually the Google Play Store, and if you wanted to install software from another repository, you had to go to settings and allow "Install from Unknown Sources." I always had to remember to turn that off after an install to prevent malicious code from being able to install software on my device.
|
||||
|
||||
### [copperheados_sources.png][9]
|
||||
|
||||
|
||||
|
||||
With Oreo, you can permanently allow a specified application to install applications. For example, I use some applications from the [Amazon Appstore][43] (such as the Amazon Shopping and Kindle apps). When I download and install the Amazon Appstore Android package (APK), I am prompted to allow the application to install apps and then I'm not asked again. Of course, this can be turned on and off on a per-application basis.
|
||||
|
||||
The Amazon Appstore has a number of useful apps, such as [IMDB][44] and [eBay][45]. Many of them don't require Google Services, but some do. For example, if I install the [Skype][46] app via Amazon, it starts up, but then complains about the operating system. The American Airlines app would start, then complain about an expired certificate. (I contacted them and was told they were no longer maintaining the version in the Amazon Appstore and it would be removed.) In any case, I can pretty simply install a couple of applications I like without using Google Play.
|
||||
|
||||
**Google Play**
|
||||
|
||||
Well, what about those apps you love that don't use Google Play Services but are only available through the Google Play Store? There is yet another way to safely get those apps on your Copperhead device.
|
||||
|
||||
This does require some technical expertise and another device. On the second device, install the [TWRP][47] recovery application. This is usually a key first step in installing any custom ROM, and TWRP is supported on a large number of devices. You will also need the Android Debug Bridge ([ADB][48]) application from the [Android SDK][49], which can be downloaded at no cost.
|
||||
|
||||
On the second device, use the Google Play Store to install the applications you want. Then, reboot into recovery. You can mount the system partition via TWRP; plug the device into a computer via a USB cable and you should be able to see it via ADB. There is a system directory called `/data/app`, and in it you will find all the APK files for your applications. Copy those you want to your computer (I use the ADB `pull`command and copy over the whole directory).
|
||||
|
||||
Disconnect that phone and connect your Copperhead device. Enable the "Transfer files" option, and you should see the storage directory mounted on your computer. Copy over the APK files for the applications you want, then install them via the Amaze file manager (just navigate to the APK file and click on it).
|
||||
|
||||
Note that you can do this for any application, and it might even be possible to install Google Play Services this way on Copperhead, but that kind of defeats the purpose. I use this mainly to get the [Electric Sheep][50] screensaver and a guitar tuning app I like called [Cleartune][51]. Be aware that if you install TWRP, especially on a Google Pixel, security updates may not work, as they'll expect the stock recovery. In this case you can always use [fastboot][52] to access TWRP, but leave the default recovery in place.
|
||||
|
||||
### Must-have apps without a workaround
|
||||
|
||||
Unfortunately, there are still a couple of Google apps I find it hard to live without. Google Maps is probably the main Google application I use, and yes, while I know I'm giving up my location to Google, it has saved hours of my life by routing me around traffic issues. [OpenStreetMap][53] has an app available via F-Droid, but it doesn't have the real-time information that makes Google Maps so useful. I also use Skype on occasion, usually when I am out of the country and have only a data connection (i.e., through a hotel WiFi network). It lets me call home and other places at a very affordable price.
|
||||
|
||||
My workaround is to carry two phones. I know this isn't an option for most people, but it is the only one I've found for now. I use my Copperhead phone for anything personal (email, contacts, calendars, pictures, etc.) and my "Googlephone" for Maps, Skype, and various games.
|
||||
|
||||
My dream would be for someone to perfect a hypervisor on a handset. Then I could run Copperhead and stock Google Android on the same device. I don't think anyone has a strong business reason to do it, but I do hope it happens.
|
||||
|
||||
### Devices that support Copperhead
|
||||
|
||||
Before you rush out to install Copperhead, there are some hurdles you'll have to jump. First, it is supported on only a [limited number of handsets][54], almost all of them late-model Google devices. The logic behind this is simple: Google tends to release Android security updates for its devices quickly, and I've found that Copperhead is able to follow suit within a day, if not within hours. Second, like any open source project, it has limited resources and it is difficult to support even a fraction of the devices now available to end users. Finally, if you want to run Copperhead on handsets like the Pixel and Pixel XL, you'll either have to build from source or [buy a device][55] from Copperhead directly.
|
||||
|
||||
When I discovered Copperhead, I had a Nexus 6P, which (along with the Nexus 5X) is one of the supported devices. This allowed me to play with and get used to the operating system. I liked it so much that I donated some money to the project, but I kind of balked at the price they were asking for Pixel and Pixel XL handsets.
|
||||
|
||||
Recently, though, I ended up purchasing a Pixel XL directly from Copperhead. There were a couple of reasons. One, since all of the code is available on GitHub, I set out to do [my own build][56] for a Pixel device. That process (which I never completed) made me appreciate the amount of work Copperhead puts into its project. Two, there was an article on [Slashdot][57] discussing how people were selling devices with Copperhead pre-installed and using Copperhead's update servers. I didn't appreciate that very much. Finally, I support FOSS not only by being a vocal user but also with my wallet.
|
||||
|
||||
### Putting the "libre" back into free
|
||||
|
||||
Another thing I love about FOSS is that I have options. There is even a new option to Copperhead being developed called [Eelo][58]. Created by [Gaël Duval][59], the developer of Mandrake Linux, this is a privacy-based Android operating system based on [LineageOS][60] (the descendant of CyanogenMod). While it should be supported on more handsets than Copperhead is, it is still in the development stage, and Copperhead is very stable and mature. I am eager to check it out, though.
|
||||
|
||||
For the year I've used CopperheadOS, I've never felt safer when using a mobile device to connect to a network. I've found the open source replacements for my old apps to be more than adequate, if not better than the original apps. I've also rediscovered the browser. Where I used to have around three to four tabs open, I now have around 10, because I've found that I usually don't need to install an app to easily access a site's content.
|
||||
|
||||
With companies like Google and Apple trying more and more to insinuate themselves into the lives of their users, it is nice to have an option that puts the "libre" back into free.
|
||||
|
||||
|
||||
### About the author
|
||||
|
||||
[][61]
|
||||
|
||||
Tarus Balog - Having been kicked out of some of the best colleges and universities in the country, I managed after seven years to get a BSEE and entered the telecommunications industry. I always ended up working on projects where we were trying to get the phone switch to talk to PCs. This got me interested in the creation and management of large communication networks. So I moved into the data communications field (they were separate back then) and started working with commercial network management tools... [more about Tarus Balog][12][More about me][13]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/copperheados-delivers-mobile-freedom-privacy-and-security
|
||||
|
||||
作者：[Tarus Balog ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/sortova
|
||||
[1]:https://opensource.com/file/384496
|
||||
[2]:https://opensource.com/file/384501
|
||||
[3]:https://opensource.com/file/384506
|
||||
[4]:https://opensource.com/file/384491
|
||||
[5]:https://opensource.com/file/384486
|
||||
[6]:https://opensource.com/file/384481
|
||||
[7]:https://opensource.com/file/384476
|
||||
[8]:https://opensource.com/file/384471
|
||||
[9]:https://opensource.com/file/384466
|
||||
[10]:https://opensource.com/article/18/1/copperheados-delivers-mobile-freedom-privacy-and-security?rate=P32BmRpJF5bYEYTHo4mW3Hp4XRk34Eq3QqMDf2oOGnw
|
||||
[11]:https://copperhead.co/android/docs/building#redistribution
|
||||
[12]:https://opensource.com/users/sortova
|
||||
[13]:https://opensource.com/users/sortova
|
||||
[14]:https://opensource.com/user/11447/feed
|
||||
[15]:https://www.flickr.com/photos/mstable/17517955832
|
||||
[16]:https://creativecommons.org/publicdomain/mark/1.0/
|
||||
[17]:https://creativecommons.org/licenses/by-sa/4.0/
|
||||
[18]:https://en.wikipedia.org/wiki/Android_(operating_system)#AOSP
|
||||
[19]:https://copperhead.co/
|
||||
[20]:https://copperhead.co/android/docs/technical_overview
|
||||
[21]:https://en.wikipedia.org/wiki/Mobile_location_analytics
|
||||
[22]:https://en.wikipedia.org/wiki/Google_Play#Compatibility
|
||||
[23]:https://play.google.com/intl/en-us_us/about/play-terms.html
|
||||
[24]:https://en.wikipedia.org/wiki/F-Droid
|
||||
[25]:https://f-droid.org/en/packages/com.fsck.k9/
|
||||
[26]:https://f-droid.org/en/packages/net.sourceforge.opencamera/
|
||||
[27]:https://f-droid.org/en/packages/com.amaze.filemanager/
|
||||
[28]:https://f-droid.org/en/packages/ch.blinkenlights.android.vanilla/
|
||||
[29]:https://nextcloud.com/
|
||||
[30]:https://github.com/nextcloud/news
|
||||
[31]:https://en.wikipedia.org/wiki/Google_Reader
|
||||
[32]:https://f-droid.org/packages/email.schaal.ocreader/
|
||||
[33]:https://en.wikipedia.org/wiki/Signal_(software)
|
||||
[34]:https://en.wikipedia.org/wiki/Open_Whisper_Systems
|
||||
[35]:https://en.wikipedia.org/wiki/Edward_Snowden
|
||||
[36]:https://en.wikipedia.org/wiki/Google_Cloud_Messaging
|
||||
[37]:https://f-droid.org/en/packages/org.smssecure.smssecure/
|
||||
[38]:https://github.com/copperhead/Noise
|
||||
[39]:https://f-droid.org/wiki/page/org.mozilla.firefox
|
||||
[40]:https://compass.opennms.io/
|
||||
[41]:https://ionicframework.com/
|
||||
[42]:https://f-droid.org/en/packages/de.marmaro.krt.ffupdater/
|
||||
[43]:https://www.amazon.com/gp/feature.html?docId=1000626391
|
||||
[44]:https://www.imdb.com/
|
||||
[45]:https://www.ebay.com/
|
||||
[46]:https://www.skype.com/
|
||||
[47]:https://twrp.me/
|
||||
[48]:https://en.wikipedia.org/wiki/Android_software_development#ADB
|
||||
[49]:https://developer.android.com/studio/index.html
|
||||
[50]:https://play.google.com/store/apps/details?id=com.spotworks.electricsheep&hl=en
|
||||
[51]:https://play.google.com/store/apps/details?id=com.bitcount.cleartune&hl=en
|
||||
[52]:https://en.wikipedia.org/wiki/Android_software_development#Fastboot
|
||||
[53]:https://f-droid.org/packages/net.osmand.plus/
|
||||
[54]:https://copperhead.co/android/downloads
|
||||
[55]:https://copperhead.co/android/store
|
||||
[56]:https://copperhead.co/android/docs/building
|
||||
[57]:https://news.slashdot.org/story/17/11/12/024231/copperheados-fights-unlicensed-installations-on-nexus-phones
|
||||
[58]:https://eelo.io/
|
||||
[59]:https://en.wikipedia.org/wiki/Ga%C3%ABl_Duval
|
||||
[60]:https://en.wikipedia.org/wiki/LineageOS
|
||||
[61]:https://opensource.com/users/sortova
|
||||
[62]:https://opensource.com/users/sortova
|
||||
[63]:https://opensource.com/users/sortova
|
||||
[64]:https://opensource.com/article/18/1/copperheados-delivers-mobile-freedom-privacy-and-security#comments
|
||||
[65]:https://opensource.com/tags/mobile
|
||||
[66]:https://opensource.com/tags/android
|
||||
236
sources/tech/20180129 Parsing HTML with Python.md
Normal file
236
sources/tech/20180129 Parsing HTML with Python.md
Normal file
@ -0,0 +1,236 @@
|
||||
Parsing HTML with Python
|
||||
============================================================
|
||||
|
||||
### With a little scripting, cleaning up documentation and other large sets of HTML files can be easy. But first you need to parse them.
|
||||
|
||||
|
||||
|
||||
Image by :
|
||||
|
||||
[Jason Baker][4] for Opensource.com.
|
||||
|
||||
As a long-time member of the documentation team at Scribus, I keep up-to-date with the latest updates of the source so I can help make updates and additions to the documentation. When I recently did a "checkout" using Subversion on a computer I had just upgraded to Fedora 27, I was amazed at how long it took to download the documentation, which consists of HTML pages and associated images. I became concerned that the project's documentation seemed much larger than it should be and suspected that some of the content was "zombie" documentation—HTML files that aren't used anymore and images that have lost all references in the currently used HTML.
|
||||
|
||||
I decided to create a project for myself to figure this out. One way to do this is to search for existing image files that aren't used. If I could scan through all the HTML files for image references, then compare that list to the actual image files, chances are I would see a mismatch.
|
||||
|
||||
Here is a typical image tag:
|
||||
|
||||
```
|
||||
<img src="images/edit_shapes.png" ALT="Edit examples" ALIGN=left>
|
||||
```
|
||||
|
||||
I'm interested in the part between the first set of quotation marks, after `src=`. After some searching for a solution, I found a Python module called [BeautifulSoup][5]. The tasty part of the script I wrote looks like this:
|
||||
|
||||
```
|
||||
soup = BeautifulSoup(all_text, 'html.parser')
|
||||
match = soup.findAll("img")
|
||||
if len(match) > 0:
|
||||
for m in match:
|
||||
imagelist.append(str(m))
|
||||
```
|
||||
|
||||
We can use this `findAll` method to pluck out the image tags. Here is a tiny piece of the output:
|
||||
|
||||
```
|
||||
<img src="images/pdf-form-ht3.png"/><img src="images/pdf-form-ht4.png"/><img src="images/pdf-form-ht5.png"/><img src="images/pdf-form-ht6.png"/><img align="middle" alt="GSview - Advanced Options Panel" src="images/gsadv1.png" title="GSview - Advanced Options Panel"/><img align="middle" alt="Scribus External Tools Preferences" src="images/gsadv2.png" title="Scribus External Tools Preferences"/>
|
||||
```
|
||||
|
||||
So far, so good. I thought that the next step might be to just carve this down, but when I tried some string methods in the script, it returned errors about this being tags and not strings. I saved the output to a file and went through the process of editing in [KWrite][6]. One nice thing about KWrite is that you can do a "find & replace" using regular expressions (regex), so I could replace `<img` with `\n<img`, which made it easier to see how to carve this down from there. Another nice thing with KWrite is that, if you make an injudicious choice with regex, you can undo it.
|
||||
|
||||
But I thought, surely there is something better than this, so I turned to regex, or more specifically the `re` module for Python. The relevant part of this new script looks like this:
|
||||
|
||||
```
|
||||
match = re.findall(r'src="(.*)/>', all_text)
|
||||
if len(match)>0:
|
||||
for m in match:
|
||||
imagelist.append(m)
|
||||
```
|
||||
|
||||
And a tiny piece of its output looks like this:
|
||||
|
||||
```
|
||||
images/cmcanvas.png" title="Context Menu for the document canvas" alt="Context Menu for the document canvas" /></td></tr></table><br images/eps-imp1.png" title="EPS preview in a file dialog" alt="EPS preview in a file dialog" images/eps-imp5.png" title="Colors imported from an EPS file" alt="Colors imported from an EPS file" images/eps-imp4.png" title="EPS font substitution" alt="EPS font substitution" images/eps-imp2.png" title="EPS import progress" alt="EPS import progress" images/eps-imp3.png" title="Bitmap conversion failure" alt="Bitmap conversion failure"
|
||||
```
|
||||
|
||||
At first glance, it looks similar to the output above, and has the nice feature of trimming out parts of the image tag, but there are puzzling inclusions of table tags and other content. I think this relates to this regex expression `src="(.*)/>`, which is termed _greedy_ , meaning it doesn't necessarily stop at the first instance of `/>` it encounters. I should add that I also tried `src="(.*)"` which was really no better. Not being a _regexpert _ (just made this up), my searching around for various ideas to improve this didn't help.
|
||||
|
||||
After a series of other things, even trying out `HTML::Parser` with Perl, I finally tried to compare this to the situation of some scripts that I wrote for Scribus that analyze the contents of a text frame, character by character, then take some action. For my purposes, what I finally came up with improves on all these methods and requires no regex or HTML parser at all. Let's go back to that example `img` tag I showed.
|
||||
|
||||
```
|
||||
<img src="images/edit_shapes.png" ALT="Edit examples" ALIGN=left>
|
||||
```
|
||||
|
||||
I decided to home in on the `src=` piece. One way would be to wait for an occurrence of `s`, then see if the next character is `r`, the next `c`, and the next `=`. If so, bingo! Then what follows between two sets of double quotation marks is what I need. The problem with this is the structure it takes to hang onto these. One way of looking at a string of characters representing a line of HTML text would be:
|
||||
|
||||
```
|
||||
for c in all_text:
|
||||
```
|
||||
|
||||
But the logic was just too messy to hang onto the previous `c`, and the one before that, the one before that, and the one before that.
|
||||
|
||||
In the end, I decided to focus on the `=` and to use an indexing method whereby I could easily reference any prior or future character in the string. Here is the searching part:
|
||||
|
||||
```
|
||||
index = 3
|
||||
while index < linelength:
|
||||
if (all_text[index] == '='):
|
||||
if (all_text[index-3] == 's') and (all_text[index-2] == 'r') and
|
||||
(all_text[index-1] == 'c'):
|
||||
imagefound(all_text, imagelist, index)
|
||||
index += 1
|
||||
else:
|
||||
index += 1
|
||||
else:
|
||||
index += 1
|
||||
```
|
||||
|
||||
I start the search with the fourth character (indexing starts at 0), so I don't get an indexing error down below, and realistically, there will not be an equal sign before the fourth character of a line. The first test is to see if we find `=` as we're marching through the string, and if not, we march on. If we do see one, then we ask if the three previous characters were `s`, `r`, and `c`, in that order. If that happens, we call the function `imagefound`:
|
||||
|
||||
```
|
||||
def imagefound(all_text, imagelist, index):
|
||||
end = 0
|
||||
index += 2
|
||||
newimage = ''
|
||||
while end == 0:
|
||||
if (all_text[index] != '"'):
|
||||
newimage = newimage + all_text[index]
|
||||
index += 1
|
||||
else:
|
||||
newimage = newimage + '\n'
|
||||
imagelist.append(newimage)
|
||||
end = 1
|
||||
return
|
||||
```
|
||||
|
||||
We're sending the function the current index, which represents the `=`. We know the next character will be `"`, so we jump two characters and begin adding characters to a holding string named `newimage`, until we reach the following `"`, at which point we're done. We add the string plus a `newline` character to our list `imagelist` and `return`, keeping in mind there may be more image tags in this remaining string of HTML, so we're right back in the middle of our searching loop.
|
||||
|
||||
Here's what our output looks like now:
|
||||
|
||||
```
|
||||
images/text-frame-link.png
|
||||
images/text-frame-unlink.png
|
||||
images/gimpoptions1.png
|
||||
images/gimpoptions3.png
|
||||
images/gimpoptions2.png
|
||||
images/fontpref3.png
|
||||
images/font-subst.png
|
||||
images/fontpref2.png
|
||||
images/fontpref1.png
|
||||
images/dtp-studio.png
|
||||
```
|
||||
|
||||
Ahhh, much cleaner, and this only took a few seconds to run. I could have jumped seven more index spots to cut out the `images/` part, but I like having it there to make sure I haven't chopped off the first letter of the image filename, and this is so easy to edit out with KWrite—you don't even need regex. After doing that and saving the file, the next step was to run another script I wrote called `sortlist.py`:
|
||||
|
||||
```
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# sortlist.py
|
||||
|
||||
import os
|
||||
|
||||
imagelist = []
|
||||
for line in open('/tmp/imagelist_parse4.txt').xreadlines():
|
||||
imagelist.append(line)
|
||||
|
||||
imagelist.sort()
|
||||
|
||||
outfile = open('/tmp/imagelist_parse4_sorted.txt', 'w')
|
||||
outfile.writelines(imagelist)
|
||||
outfile.close()
|
||||
```
|
||||
|
||||
This pulls in the file contents as a list, sorts it, then saves it as another file. After that I could just do the following:
|
||||
|
||||
```
|
||||
ls /home/gregp/development/Scribus15x/doc/en/images/*.png > '/tmp/actual_images.txt'
|
||||
```
|
||||
|
||||
Then I need to run `sortlist.py` on that file too, since the method `ls` uses to sort is different from Python. I could have run a comparison script on these files, but I preferred to do this visually. In the end, I ended up with 42 images that had no HTML reference from the documentation.
|
||||
|
||||
Here is my parsing script in its entirety:
|
||||
|
||||
```
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# parseimg4.py
|
||||
|
||||
import os
|
||||
|
||||
def imagefound(all_text, imagelist, index):
|
||||
end = 0
|
||||
index += 2
|
||||
newimage = ''
|
||||
while end == 0:
|
||||
if (all_text[index] != '"'):
|
||||
newimage = newimage + all_text[index]
|
||||
index += 1
|
||||
else:
|
||||
newimage = newimage + '\n'
|
||||
imagelist.append(newimage)
|
||||
end = 1
|
||||
return
|
||||
|
||||
htmlnames = []
|
||||
imagelist = []
|
||||
tempstring = ''
|
||||
filenames = os.listdir('/home/gregp/development/Scribus15x/doc/en/')
|
||||
for name in filenames:
|
||||
if name.endswith('.html'):
|
||||
htmlnames.append(name)
|
||||
#print htmlnames
|
||||
for htmlfile in htmlnames:
|
||||
all_text = open('/home/gregp/development/Scribus15x/doc/en/' + htmlfile).read()
|
||||
linelength = len(all_text)
|
||||
index = 3
|
||||
while index < linelength:
|
||||
if (all_text[index] == '='):
|
||||
if (all_text[index-3] == 's') and (all_text[index-2] == 'r') and
|
||||
(all_text[index-1] == 'c'):
|
||||
imagefound(all_text, imagelist, index)
|
||||
index += 1
|
||||
else:
|
||||
index += 1
|
||||
else:
|
||||
index += 1
|
||||
|
||||
outfile = open('/tmp/imagelist_parse4.txt', 'w')
|
||||
outfile.writelines(imagelist)
|
||||
outfile.close()
|
||||
imageno = len(imagelist)
|
||||
print str(imageno) + " images were found and saved"
|
||||
```
|
||||
|
||||
Its name, `parseimg4.py`, doesn't really reflect the number of scripts I wrote along the way, with both minor and major rewrites, plus discards and starting over. Notice that I've hardcoded these directory and filenames, but it would be easy enough to generalize, asking for user input for these pieces of information. Also as they were working scripts, I sent the output to `/tmp`, so they disappear once I reboot my system.
|
||||
|
||||
This wasn't the end of the story, since the next question was: What about zombie HTML files? Any of these files that are not used might reference images not picked up by the previous method. We have a `menu.xml` file that serves as the table of contents for the online manual, but I also needed to consider that some files listed in the TOC might reference files not in the TOC, and yes, I did find some.
|
||||
|
||||
I'll conclude by saying that this was a simpler task than this image search, and it was greatly helped by the processes I had already developed.
|
||||
|
||||
|
||||
### About the author
|
||||
|
||||
[][7] Greg Pittman - Greg is a retired neurologist in Louisville, Kentucky, with a long-standing interest in computers and programming, beginning with Fortran IV in the 1960s. When Linux and open source software came along, it kindled a commitment to learning more, and eventually contributing. He is a member of the Scribus Team.[More about me][2]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/parsing-html-python
|
||||
|
||||
作者：[Greg Pittman][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/greg-p
|
||||
[1]:https://opensource.com/article/18/1/parsing-html-python?rate=etxcPw1UgZP1LUvuRV9gwwgA1SxTVVZlVhigvK6bElI
|
||||
[2]:https://opensource.com/users/greg-p
|
||||
[3]:https://opensource.com/user/30666/feed
|
||||
[4]:https://opensource.com/users/jason-baker
|
||||
[5]:https://www.crummy.com/software/BeautifulSoup/
|
||||
[6]:https://www.kde.org/applications/utilities/kwrite/
|
||||
[7]:https://opensource.com/users/greg-p
|
||||
[8]:https://opensource.com/users/greg-p
|
||||
[9]:https://opensource.com/users/greg-p
|
||||
[10]:https://opensource.com/article/18/1/parsing-html-python#comments
|
||||
[11]:https://opensource.com/tags/python
|
||||
[12]:https://opensource.com/tags/programming
|
||||
@ -0,0 +1,194 @@
|
||||
An introduction to the DomTerm terminal emulator for Linux
|
||||
============================================================
|
||||
|
||||
### Learn about DomTerm, a terminal emulator and multiplexer with HTML graphics and other unusual features.
|
||||
|
||||
|
||||
Image by :
|
||||
|
||||
[Jamie Cox][25]. Modified by Opensource.com. [CC BY 2.0.][26]
|
||||
|
||||
[DomTerm][27] is a modern terminal emulator that uses a browser engine as a "GUI toolkit." This enables some neat features, such as embeddable graphics and links, HTML rich text, and foldable (show/hide) commands. Otherwise it looks and feels like a feature-full, standalone terminal emulator, with excellent xterm compatibility (including mouse handling and 24-bit color), and appropriate "chrome" (menus). In addition, there is built-in support for session management and sub-windows (as in `tmux` and `GNU screen`), basic input editing (as in `readline`), and paging (as in `less`).
|
||||
|
||||
### [domterm1.png][10]
|
||||
|
||||
|
||||
|
||||
Image 1: The DomTerminal terminal emulator. _[View larger image.][1]_
|
||||
|
||||
Below we'll look more at these features. We'll assume you have `domterm` installed (skip to the end of this article if you need to get and build DomTerm). First, though, here's a quick overview of the technology.
|
||||
|
||||
### Frontend vs. backend
|
||||
|
||||
More Linux resources
|
||||
|
||||
* [What is Linux?][2]
|
||||
|
||||
* [What are Linux containers?][3]
|
||||
|
||||
* [Download Now: Linux commands cheat sheet][4]
|
||||
|
||||
* [Advanced Linux commands cheat sheet][5]
|
||||
|
||||
* [Our latest Linux articles][6]
|
||||
|
||||
Most of DomTerm is written in JavaScript and runs in a browser engine. This can be a desktop web browser, such as Chrome or Firefox (see [image 3][28]), or it can be an embedded browser. Using a general web browser works fine, but the user experience isn't as nice (as the menus are designed for general browsing, not for a terminal emulator), and the security model gets in the way, so using an embedded browser is nicer.
|
||||
|
||||
The following are currently supported:
|
||||
|
||||
* `qtdomterm`, which uses the Qt toolkit and `QtWebEngine`
|
||||
|
||||
* An `[Electron][11]` embedding (see [image 1][16])
|
||||
|
||||
* `atom-domterm` runs DomTerm as a package in the [Atom text editor][17] (which is also based on Electron) and integrates with the Atom pane system (see [image 2][18])
|
||||
|
||||
* A wrapper for JavaFX's `WebEngine`, which is useful for code written in Java (see [image 4][19])
|
||||
|
||||
* Previously, the preferred frontend used [Firefox-XUL][20], but Mozilla has since dropped XUL
|
||||
|
||||
### [dt-atom1.png][12]
|
||||
|
||||
|
||||
|
||||
Image 2: DomTerm terminal panes in Atom editor. _[View larger image.][7]_
|
||||
|
||||
Currently, the Electron frontend is probably the nicest option, closely followed by the Qt frontend. If you use Atom, `atom-domterm` works pretty well.
|
||||
|
||||
The backend server is written in C. It manages pseudo terminals (PTYs) and sessions. It is also an HTTP server that provides the JavaScript and other files to the frontend. The `domterm` command starts terminal jobs and performs other requests. If there is no server running, `domterm` daemonizes itself. Communication between the backend and the server is normally done using WebSockets (with [libwebsockets][29] on the server). However, the JavaFX embedding uses neither WebSockets nor the DomTerm server; instead Java applications communicate directly using the Java–JavaScript bridge.
|
||||
|
||||
### A solid xterm-compatible terminal emulator
|
||||
|
||||
DomTerm looks and feels like a modern terminal emulator. It handles mouse events, 24-bit color, Unicode, double-width (CJK) characters, and input methods. DomTerm does a very good job on the [vttest testsuite][30].
|
||||
|
||||
Unusual features include:
|
||||
|
||||
**Show/hide buttons ("folding"):** The little triangles (seen in [image 2][31] above) are buttons that hide/show the corresponding output. To create the buttons, just add certain [escape sequences][32] in the [prompt text][33].
|
||||
|
||||
**Mouse-click support for `readline` and similar input editors:** If you click in the (yellow) input area, DomTerm will send the right sequence of arrow-key keystrokes to the application. (This is enabled by escape sequences in the prompt; you can also force it using Alt+Click.)
|
||||
|
||||
**Style the terminal using CSS:** This is usually done in `~/.domterm/settings.ini`, which is automatically reloaded when saved. For example, in [image 2][34], terminal-specific background colors were set.
|
||||
|
||||
### A better REPL console
|
||||
|
||||
A classic terminal emulator works on rectangular grids of character cells. This works for a REPL (command shell), but it is not ideal. Here are some DomTerm features useful for REPLs that are not typically found in terminal emulators:
|
||||
|
||||
**A command can "print" an image, a graph, a mathematical formula, or a set of clickable links:** An application can send an escape sequence containing almost any HTML. (The HTML is scrubbed to remove JavaScript and other dangerous features.)
|
||||
|
||||
The [image 3][35] shows a fragment from a [`gnuplot`][36] session. Gnuplot (2.1 or later) supports `domterm` as a terminal type. Graphical output is converted to an [SVG image][37], which is then printed to the terminal. My blog post [Gnuplot display on DomTerm][38]provides more information on this.
|
||||
|
||||
### [dt-gnuplot.png][13]
|
||||
|
||||
|
||||
|
||||
Image 3: Gnuplot screenshot. _[View larger image.][8]_
|
||||
|
||||
The [Kawa][39] language has a library for creating and transforming [geometric picture values][40]. If you print such a picture value to a DomTerm terminal, the picture is converted to SVG and embedded in the output.
|
||||
|
||||
### [dt-kawa1.png][14]
|
||||
|
||||
|
||||
|
||||
Image 4: Computable geometry in Kawa. _[View larger image.][9]_
|
||||
|
||||
**Rich text in output:** Help messages are more readable and look nicer with HTML styling. The lower pane of [image 1][41] shows the ouput from `domterm help`. (The output is plaintext if not running under DomTerm.) Note the `PAUSED` message from the built-in pager.
|
||||
|
||||
**Error messages can include clickable links:** DomTerm recognizes the syntax `<var style="font-size: 1em; line-height: 1.5em;">filename</var>:<var style="font-size: 1em; line-height: 1.5em;">line</var>:<var style="font-size: 1em; line-height: 1.5em;">column:</var>` and turns it into a link that opens the file and line in a configurable text editor. (This works for relative filenames if you use `PROMPT_COMMAND` or similar to track directories.)
|
||||
|
||||
A compiler can detect that it is running under DomTerm and directly emit file links in an escape sequence. This is more robust than depending on DomTerm's pattern matching, as it handles spaces and other special characters, and it does not depend on directory tracking. In [image 4][42], you can see error messages from the [Kawa compiler][43]. Hovering over the file position causes it to be underlined, and the `file:`URL shows in the `atom-domterm` message area (bottom of the window). (When not using `atom-domterm`, such messages are shown in an overlay box, as seen for the `PAUSED` message in [image 1][44].)
|
||||
|
||||
The action when clicking on a link is configurable. The default action for a `file:` link with a `#position` suffix is to open the file in a text editor.
|
||||
|
||||
**Structured internal representation:** The following are all represented in the internal node structure: Commands, prompts, input lines, normal and error output, tabs, and preserving the structure if you "Save as HTML." The HTML file is compatible with XML, so you can use XML tools to search or transform the output. The command `domterm view-saved` opens a saved HTML file in a way that enables command folding (show/hide buttons are active) and reflow on window resize.
|
||||
|
||||
**Built-in Lisp-style pretty-printing:** You can include pretty-printing directives (e.g., grouping) in the output such that line breaks are recalculated on window resize. See my article [Dynamic pretty-printing in DomTerm][45] for a deeper discussion.
|
||||
|
||||
**Basic built-in line editing** with history (like `GNU readline`): This uses the browser's built-in editor, so it has great mouse and selection handling. You can switch between normal character-mode (most characters typed are sent directly to the process); or line-mode (regular characters are inserted while control characters cause editing actions, with Enter sending the edited line to the process). The default is automatic mode, where DomTerm switches between character-mode and line-mode depending on whether the PTY is in <q style="quotes: "“" "”" "‘" "’";">raw<q style="quotes: "“" "”" "‘" "’";"> or <q style="quotes: "“" "”" "‘" "’";">canonical</q> mode.</q></q>
|
||||
|
||||
**A built-in pager** (like a simplified `less`): Keyboard shortcuts will control scrolling. In "paging mode," the output pauses after each new screen (or single line, if you move forward line-by-line). The paging mode is unobtrusive and smart about user input, so you can (if you wish) run it without it interfering with interactive programs.
|
||||
|
||||
### Multiplexing and sessions
|
||||
|
||||
**Tabs and tiling:** Not only can you create multiple terminal tabs, you can also tile them. You can use either the mouse or a keyboard shortcut to move between panes and tabs as well as create new ones. They can be rearranged and resized with the mouse. This is implemented using the [GoldenLayout][46] JavaScript library. [Image 1][47]shows a window with two panes. The top one has two tabs, with one running [Midnight Commander][48]; the bottom pane shows `domterm help` output as HTML. However, on Atom we instead use its built-in draggable tiles and tabs; you can see this in [image 2][49].
|
||||
|
||||
**Detaching and reattaching to sessions:** DomTerm supports sessions arrangement, similar to `tmux` and GNU `screen`. You can even attach multiple windows or panes to the same session. This supports multi-user session sharing and remote connections. (For security, all sessions of the same server need to be able to read a Unix domain socket and a local file containing a random key. This restriction will be lifted when we have a good, safe remote-access story.)
|
||||
|
||||
**The ****`domterm`**** command** is also like `tmux` or GNU `screen` in that has multiple options for controlling or starting a server that manages one or more sessions. The major difference is that, if it's not already running under DomTerm, the `domterm` command creates a new top-level window, rather than running in the existing terminal.
|
||||
|
||||
The `domterm` command has a number of sub-commands, similar to `tmux` or `git`. Some sub-commands create windows or sessions. Others (such as "printing" an image) only work within an existing DomTerm session.
|
||||
|
||||
The command `domterm browse` opens a window or pane for browsing a specified URL, such as when browsing documentation.
|
||||
|
||||
### Getting and installing DomTerm
|
||||
|
||||
DomTerm is available from its [GitHub repository][50]. Currently, there are no prebuilt packages, but there are [detailed instructions][51]. All prerequisites are available on Fedora 27, which makes it especially easy to build.
|
||||
|
||||
### About the author
|
||||
|
||||
[][52] Per Bothner - Per has been involved with Open Source since before the term existed. He was an early employee of Cygnus (later purchased by Red Hat), which was the first company commercializing Free Software. There he worked on gcc, g++, libio (the precursor to GNU/Linux stdio), gdb, Kawa, and more. Later he worked in the Java group at Sun and Oracle. Per wrote the Emacs term mode. Currently, Per spends too much time on [Kawa][21] (a Scheme compiler... [more about Per Bothner][22][More about me][23]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/introduction-domterm-terminal-emulator
|
||||
|
||||
作者：[ Per Bothner][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/perbothner
|
||||
[1]:https://opensource.com/sites/default/files/u128651/domterm1.png
|
||||
[2]:https://opensource.com/resources/what-is-linux?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[3]:https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[4]:https://developers.redhat.com/promotions/linux-cheatsheet/?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[5]:https://developers.redhat.com/cheat-sheet/advanced-linux-commands-cheatsheet?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[6]:https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[7]:https://opensource.com/sites/default/files/images/dt-atom1.png
|
||||
[8]:https://opensource.com/sites/default/files/dt-gnuplot.png
|
||||
[9]:https://opensource.com/sites/default/files/dt-kawa1.png
|
||||
[10]:https://opensource.com/file/384931
|
||||
[11]:https://electronjs.org/
|
||||
[12]:https://opensource.com/file/385346
|
||||
[13]:https://opensource.com/file/385326
|
||||
[14]:https://opensource.com/file/385331
|
||||
[15]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator?rate=9HfSplqf1e4NohKkTld_881cH1hXTlSwU_2XKrnpTJQ
|
||||
[16]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image1
|
||||
[17]:https://atom.io/
|
||||
[18]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image2
|
||||
[19]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image4
|
||||
[20]:https://en.wikipedia.org/wiki/XUL
|
||||
[21]:https://www.gnu.org/software/kawa/
|
||||
[22]:https://opensource.com/users/perbothner
|
||||
[23]:https://opensource.com/users/perbothner
|
||||
[24]:https://opensource.com/user/205986/feed
|
||||
[25]:https://www.flickr.com/photos/15587432@N02/3281139507/
|
||||
[26]:http://creativecommons.org/licenses/by/2.0
|
||||
[27]:http://domterm.org/
|
||||
[28]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image3
|
||||
[29]:https://libwebsockets.org/
|
||||
[30]:http://invisible-island.net/vttest/
|
||||
[31]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image2
|
||||
[32]:http://domterm.org/Wire-byte-protocol.html
|
||||
[33]:http://domterm.org/Shell-prompts.html
|
||||
[34]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image2
|
||||
[35]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image3
|
||||
[36]:http://www.gnuplot.info/
|
||||
[37]:https://developer.mozilla.org/en-US/docs/Web/SVG
|
||||
[38]:http://per.bothner.com/blog/2016/gnuplot-in-domterm/
|
||||
[39]:https://www.gnu.org/software/kawa/
|
||||
[40]:https://www.gnu.org/software/kawa/Composable-pictures.html
|
||||
[41]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image1
|
||||
[42]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image4
|
||||
[43]:https://www.gnu.org/software/kawa/
|
||||
[44]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image1
|
||||
[45]:http://per.bothner.com/blog/2017/dynamic-prettyprinting/
|
||||
[46]:https://golden-layout.com/
|
||||
[47]:https://opensource.com/sites/default/files/u128651/domterm1.png
|
||||
[48]:https://midnight-commander.org/
|
||||
[49]:https://opensource.com/article/18/1/introduction-domterm-terminal-emulator#image2
|
||||
[50]:https://github.com/PerBothner/DomTerm
|
||||
[51]:http://domterm.org/Downloading-and-building.html
|
||||
[52]:https://opensource.com/users/perbothner
|
||||
[53]:https://opensource.com/users/perbothner
|
||||
[54]:https://opensource.com/users/perbothner
|
||||
[55]:https://opensource.com/tags/linux
|
||||
104
sources/tech/20180130 Refreshing old computers with Linux.md
Normal file
104
sources/tech/20180130 Refreshing old computers with Linux.md
Normal file
@ -0,0 +1,104 @@
|
||||
Refreshing old computers with Linux
|
||||
============================================================
|
||||
|
||||
### A middle school's Tech Stewardship program is now an elective class for science and technology students.
|
||||
|
||||
|
||||
|
||||
Image by : opensource.com
|
||||
|
||||
It's nearly impossible to enter a school these days without seeing an abundance of technology. Despite this influx of computers into education, funding inequity forces school systems to make difficult choices. Some educators see things as they are and wonder, "Why?" while others see problems as opportunities and think, "Why not?"
|
||||
|
||||
[Andrew Dobbie ][31]is one of those visionaries who saw his love of Linux and computer reimaging as a unique learning opportunity for his students.
|
||||
|
||||
Andrew teaches sixth grade at Centennial Senior Public School in Brampton, Ontario, Canada, and is a[Google Certified Innovator][16]. Andrew said, "Centennial Senior Public School hosts a special regional science & technology program that invites students from throughout the region to spend three years learning Ontario curriculum through the lens of science and technology." However, the school's students were in danger of falling prey to the digital divide that's exacerbated by hardware and software product lifecycles and inadequate funding.
|
||||
|
||||
|
||||
|
||||
Image courtesy of [Affordable Tech for All][6]
|
||||
|
||||
Although there was a school-wide need for access to computers in the classrooms, Andrew and his students discovered that dozens of old computers were being shipped out of the school because they were too old and slow to keep up with the latest proprietary operating systems or function on the school's network.
|
||||
|
||||
Andrew saw this problem as a unique learning opportunity for his students and created the [Tech Stewardship][17] program. He works in partnership with two other teachers, Mike Doiu and Neil Lyons, and some students, who "began experimenting with open source operating systems like [Lubuntu][18] and [CubLinux][19] to help develop a solution to our in-class computer problem," he says.
|
||||
|
||||
The sixth-grade students deployed the reimaged computers into classrooms throughout the school. When they exhausted the school's supply of surplus computers, they sourced more free computers from a local nonprofit organization called [Renewed Computer Technology Ontario][20]. In all, the Tech Stewardship program has provided more than 200 reimaged computers for students to use in classrooms throughout the school.
|
||||
|
||||
|
||||
|
||||
|
||||
Image courtesy of [Affordable Tech for All][7]
|
||||
|
||||
The Tech Stewardship program is now an elective class for the school's science and technology students in grades six, seven, and eight. Not only are the students learning about computer reimaging, they're also giving back to their local communities through this open source outreach program.
|
||||
|
||||
### A broad impact
|
||||
|
||||
The Tech Stewardship program is linked directly to the school's curriculum, especially in social studies by teaching the [United Nations' Sustainable Development Goals][21] (SDGs). The program is a member of [Teach SDGs][22], and Andrew serves as a Teach SDGs ambassador. Also, as a Google Certified Innovator, Andrew partners with Google and the [EdTechTeam][23], and Tech Stewardship has participated in Ontario's [Bring it Together][24] conference for educational technology.
|
||||
|
||||
Andrew's students also serve as mentors to their fellow students. In one instance, a group of girls taught a grade 3 class about effective use of Google Drive and helped these younger students to make the best use of their Linux computers. Andrew said, "outreach and extension of learning beyond the classroom at Centennial is a major goal of the Tech Stewardship program."
|
||||
|
||||
### What the students say
|
||||
|
||||
Linux and open source are an integral part of the program. A girl named Ashna says, "In grade 6, Mr. Dobbie had shown us how to reimage a computer into Linux to use it for educational purposes. Since then, we have been learning more and growing." Student Shradhaa says, "At the very beginning, we didn't even know how to reimage with Linux. Mr. Dobbie told us to write steps for how to reimage Linux devices, and using those steps we are trying to reimage the computers."
|
||||
|
||||
|
||||
|
||||
|
||||
Image courtesy of [Affordable Tech for All][8]
|
||||
|
||||
The students were quick to add that Tech Stewardship has become a portal for discussion about being advocates for the change they want to see in the world. Through their hands-on activity, students learn to support the United Nations Sustainable Development goals. They also learn lessons far beyond the curriculum itself. For example, a student named Areez says he has learned how to find other resources, including donations, that allow the project to expand, since the class work upfitting older computers doesn't produce an income stream.
|
||||
|
||||
Another student, Harini, thinks the Tech Stewardship program has demonstrated to other students what is possible and how one small initiative can change the world. After learning about the program, 40 other schools and individuals are reimaging computers with Linux. Harini says, "The more people who use them for educational purposes, the more outstanding the future will become since those educated people will lead out new, amazing lives with jobs."
|
||||
|
||||
Joshua, another student in the program, sees it this way: "I thought of it as just a fun experience, but as it went on, we continued learning and understanding how what we were doing was making such a big impact on the world!" Later, he says, "a school reached out to us and asked us if we could reimage some computers for them. We went and completed the task. Then it continued to grow, as people from Europe came to see how we were fixing broken computers and started doing it when they went back."
|
||||
|
||||
Andrew Dobbie is keen to share his experience with schools and interested individuals. You can contact him on [Twitter][25] or through his [website][26].
|
||||
|
||||
|
||||
### About the author
|
||||
|
||||
[][27] Don Watkins - Educator, education technology specialist, entrepreneur, open source advocate. M.A. in Educational Psychology, MSED in Educational Leadership, Linux system administrator, CCNA, virtualization using Virtual Box. Follow me at [@Don_Watkins .][13][More about me][14]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/new-linux-computers-classroom
|
||||
|
||||
作者：[Don Watkins ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/don-watkins
|
||||
[1]:https://opensource.com/resources/what-is-linux?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[2]:https://opensource.com/resources/what-are-linux-containers?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[3]:https://developers.redhat.com/promotions/linux-cheatsheet/?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[4]:https://developers.redhat.com/cheat-sheet/advanced-linux-commands-cheatsheet?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[5]:https://opensource.com/tags/linux?intcmp=70160000000h1jYAAQ&utm_source=intcallout&utm_campaign=linuxcontent
|
||||
[6]:https://photos.google.com/share/AF1QipPnm-q9OIQnrzDD4n7oWIBBIE7RQ6BI9lv486RaU5lKBrs88pq3gPKM8VAgY0prkw?key=cS1RdEZ3ZHdXLWp0bUwzMEk3UnFQRkUwbWl1dWhn
|
||||
[7]:https://photos.google.com/share/AF1QipPnm-q9OIQnrzDD4n7oWIBBIE7RQ6BI9lv486RaU5lKBrs88pq3gPKM8VAgY0prkw?key=cS1RdEZ3ZHdXLWp0bUwzMEk3UnFQRkUwbWl1dWhn
|
||||
[8]:https://photos.google.com/share/AF1QipPnm-q9OIQnrzDD4n7oWIBBIE7RQ6BI9lv486RaU5lKBrs88pq3gPKM8VAgY0prkw?key=cS1RdEZ3ZHdXLWp0bUwzMEk3UnFQRkUwbWl1dWhn
|
||||
[9]:https://opensource.com/file/384581
|
||||
[10]:https://opensource.com/file/384591
|
||||
[11]:https://opensource.com/file/384586
|
||||
[12]:https://opensource.com/article/18/1/new-linux-computers-classroom?rate=bK5X7pRc5y9TyY6jzOZeLDW6ehlWmNPXuP38DYsQ-6I
|
||||
[13]:https://twitter.com/Don_Watkins
|
||||
[14]:https://opensource.com/users/don-watkins
|
||||
[15]:https://opensource.com/user/15542/feed
|
||||
[16]:https://edutrainingcenter.withgoogle.com/certification_innovator
|
||||
[17]:https://sites.google.com/view/mrdobbie/tech-stewardship
|
||||
[18]:https://lubuntu.net/
|
||||
[19]:https://en.wikipedia.org/wiki/Cub_Linux
|
||||
[20]:http://www.rcto.ca/
|
||||
[21]:http://www.un.org/sustainabledevelopment/sustainable-development-goals/
|
||||
[22]:http://www.teachsdgs.org/
|
||||
[23]:https://www.edtechteam.com/team/
|
||||
[24]:http://bringittogether.ca/
|
||||
[25]:https://twitter.com/A_Dobbie11
|
||||
[26]:http://bit.ly/linuxresources
|
||||
[27]:https://opensource.com/users/don-watkins
|
||||
[28]:https://opensource.com/users/don-watkins
|
||||
[29]:https://opensource.com/users/don-watkins
|
||||
[30]:https://opensource.com/article/18/1/new-linux-computers-classroom#comments
|
||||
[31]:https://twitter.com/A_Dobbie11
|
||||
[32]:https://opensource.com/tags/education
|
||||
[33]:https://opensource.com/tags/linux
|
||||
@ -0,0 +1,176 @@
|
||||
Microservices vs. monolith: How to choose
|
||||
============================================================
|
||||
|
||||
### Both architectures have pros and cons, and the right decision depends on your organization's unique needs.
|
||||
|
||||
|
||||
|
||||
Image by :
|
||||
|
||||
Onasill ~ Bill Badzo on [Flickr][11]. [CC BY-NC-SA 2.0][12]. Modified by Opensource.com.
|
||||
|
||||
For many startups, conventional wisdom says to start with a monolith architecture over microservices. But are there exceptions to this?
|
||||
|
||||
The upcoming book, [_Microservices for Startups_][13] , explores the benefits and drawbacks of microservices, offering insights from dozens of CTOs.
|
||||
|
||||
While different CTOs take different approaches when starting new ventures, they agree that context and capability are key. If you're pondering whether your business would be best served by a monolith or microservices, consider the factors discussed below.
|
||||
|
||||
### Understanding the spectrum
|
||||
|
||||
More on Microservices
|
||||
|
||||
* [How to explain microservices to your CEO][1]
|
||||
|
||||
* [Free eBook: Microservices vs. service-oriented architecture][2]
|
||||
|
||||
* [Secured DevOps for microservices][3]
|
||||
|
||||
Let's first clarify what exactly we mean by “monolith” and “microservice.”
|
||||
|
||||
Microservices are an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery.
|
||||
|
||||
A monolithic application is built as a single, unified unit, and usually one massive code base. Often a monolith consists of three parts: a database, a client-side user interface (consisting of HTML pages and/or JavaScript running in a browser), and a server-side application.
|
||||
|
||||
“System architectures lie on a spectrum,” Zachary Crockett, CTO of [Particle][14], said in an interview. “When discussing microservices, people tend to focus on one end of that spectrum: many tiny applications passing too many messages to each other. At the other end of the spectrum, you have a giant monolith doing too many things. For any real system, there are many possible service-oriented architectures between those two extremes.”
|
||||
|
||||
Depending on your situation, there are good reasons to tend toward either a monolith or microservices.
|
||||
|
||||
"We want to use the best tool for each service." Julien Lemoine, CTO at Algolia
|
||||
|
||||
Contrary to what many people think, a monolith isn’t a dated architecture that's best left in the past. In certain circumstances, a monolith is ideal. I spoke to Steven Czerwinski, head of engineering at [Scaylr][15] and a former Google employee, to better understand this.
|
||||
|
||||
“Even though we had had positive experiences of using microservices at Google, we [at Scalyr] went [for a monolith] route because having one monolithic server means less work for us as two engineers,” he explained. (This was back in the early days of Scalyr.)
|
||||
|
||||
But if your team is experienced with microservices and you have a clear idea of the direction you’re going, microservices can be a great alternative.
|
||||
|
||||
Julien Lemoine, CTO at [Algolia][16], chimed in on this point: “We have always started with a microservices approach. The main goal was to be able to use different technology to build our service, for two big reasons:
|
||||
|
||||
* We want to use the best tool for each service. Our search API is highly optimized at the lowest level, and C++ is the perfect language for that. That said, using C++ for everything is a waste of productivity, especially to build a dashboard.
|
||||
|
||||
* We want the best talent, and using only one technology would limit our options. This is why we have different languages in the company.”
|
||||
|
||||
If your team is prepared, starting with microservices allows your organization to get used to the rhythm of developing in a microservice environment right from the start.
|
||||
|
||||
### Weighing the pros and cons
|
||||
|
||||
Before you decide which approach is best for your organization, it's important to consider the strengths and weaknesses of each.
|
||||
|
||||
### Monoliths
|
||||
|
||||
### Pros:
|
||||
|
||||
* **Fewer cross-cutting concerns:** Most apps have cross-cutting concerns, such as logging, rate limiting, and security features like audit trails and DOS protection. When everything is running through the same app, it’s easy to address those concerns by hooking up components.
|
||||
|
||||
* **Less operational overhead:** There’s only one application to set up for logging, monitoring, and testing. Also, it's generally less complex to deploy.
|
||||
|
||||
* **Performance:** A monolith architecture can offer performance advantages since shared-memory access is faster than inter-process communication (IPC).
|
||||
|
||||
### Cons:
|
||||
|
||||
* **Tightly coupled:** Monolithic app services tend to get tightly coupled and entangled as the application evolves, making it difficult to isolate services for purposes such as independent scaling or code maintainability.
|
||||
|
||||
* **Harder to understand:** Monolithic architectures are more difficult to understand because of dependencies, side effects, and other factors that are not obvious when you’re looking at a specific service or controller.
|
||||
|
||||
### Microservices
|
||||
|
||||
### Pros:
|
||||
|
||||
* **Better organization:** Microservice architectures are typically better organized, since each microservice has a specific job and is not concerned with the jobs of other components.
|
||||
|
||||
* **Decoupled:** Decoupled services are easier to recompose and reconfigure to serve different apps (for example, serving both web clients and the public API). They also allow fast, independent delivery of individual parts within a larger integrated system.
|
||||
|
||||
* **Performance:** Depending on how they're organized, microservices can offer performance advantages because you can isolate hot services and scale them independently of the rest of the app.
|
||||
|
||||
* **Fewer mistakes:** Microservices enable parallel development by establishing a strong boundary between different parts of your system. Doing this makes it more difficult to connect parts that shouldn’t be connected, for example, or couple too tightly those that need to be connected.
|
||||
|
||||
### Cons:
|
||||
|
||||
* **Cross-cutting concerns across each service:** As you build a new microservice architecture, you’re likely to discover cross-cutting concerns you may not have anticipated at design time. You’ll either need to incur the overhead of separate modules for each cross-cutting concern (i.e., testing), or encapsulate cross-cutting concerns in another service layer through which all traffic is routed. Eventually, even monolithic architectures tend to route traffic through an outer service layer for cross-cutting concerns, but with a monolithic architecture, it’s possible to delay the cost of that work until the project is more mature.
|
||||
|
||||
* **Higher operational overhead:** Microservices are frequently deployed on their own virtual machines or containers, causing a proliferation of VM wrangling. These tasks are frequently automated with container fleet management tools.
|
||||
|
||||
### Decision time
|
||||
|
||||
Once you understand the pros and cons of both approaches, how do you apply this information to your startup? Based on interviews with CTOs, here are three questions to guide your decision process:
|
||||
|
||||
**Are you in familiar territory?**
|
||||
|
||||
Diving directly into microservices is less risky if your team has previous domain experience (for example, in e-commerce) and knowledge concerning the needs of your customers. If you’re traveling down an unknown path, on the other hand, a monolith may be a safer option.
|
||||
|
||||
**Is your team prepared?**
|
||||
|
||||
Does your team have experience with microservices? If you quadruple the size of your team within the next year, will microservices offer the best environment? Evaluating the dimensions of your team is crucial to the success of your project.
|
||||
|
||||
**How’s your infrastructure?**
|
||||
|
||||
To make microservices work, you’ll need a cloud-based infrastructure.
|
||||
|
||||
David Strauss, CTO of [Pantheon][17], explained: “[Previously], you would want to start with a monolith because you wanted to deploy one database server. The idea of having to set up a database server for every single microservice and then scale out was a mammoth task. Only a huge, tech-savvy organization could do that. Today, with services like Google Cloud and Amazon AWS, you have many options for deploying tiny things without needing to own the persistence layer for each one.”
|
||||
|
||||
### Evaluate the business risk
|
||||
|
||||
As a tech-savvy startup with high ambitions, you might think microservices is the “right” way to go. But microservices can pose a business risk. Strauss explained, “A lot of teams overbuild their project initially. Everyone wants to think their startup will be the next unicorn, and they should therefore build everything with microservices or some other hyper-scalable infrastructure. But that's usually wrong.” In these cases, Strauss continued, the areas that they thought they needed to scale are often not the ones that actually should scale first, resulting in wasted time and effort.
|
||||
|
||||
### Situational awareness
|
||||
|
||||
Ultimately, context is key. Here are some tips from CTOs:
|
||||
|
||||
#### When to start with a monolith
|
||||
|
||||
* **Your team is at founding stage:** Your team is small—say, 2 to 5 members—and is unable to tackle a broader, high-overhead microservices architecture.
|
||||
|
||||
* **You’re building an unproven product or proof of concept:** If you're bringing a brand-new product to market, it will likely evolve over time, and a monolith is better-suited to allow for rapid product iteration. The same notion applies to a proof of concept, where your goal is to learn as much as possible as quickly as possible, even if you end up throwing it away.
|
||||
|
||||
* **You have no microservices experience:** Unless you can justify the risk of learning on the fly at an early stage, a monolith may be a safer approach for an inexperienced team.
|
||||
|
||||
#### When to start with microservices
|
||||
|
||||
* **You need quick, independent service delivery:** Microservices allow for fast, independent delivery of individual parts within a larger integrated system. Note that it can take some time to see service delivery gains with microservices compared to a monolith, depending on your team's size.
|
||||
|
||||
* **A piece of your platform needs to be extremely efficient:** If your business does intensive processing of petabytes of log volume, you’ll likely want to build that service out in an efficient language like C++, while your user dashboard may be built in [Ruby on Rails][5].
|
||||
|
||||
* **You plan to grow your team:** Starting with microservices gets your team used to developing in separate small services from the beginning, and teams that are separated by service boundaries are easier to scale as needed.
|
||||
|
||||
To decide whether a monolith or microservices is right for your organization, be honest and self-aware about your context and capabilities. This will help you find the best path to grow your business.
|
||||
|
||||
### Topics
|
||||
|
||||
[Microservices][21][DevOps][22]
|
||||
|
||||
### About the author
|
||||
|
||||
[][18] jakelumetta - Jake is the CEO of [ButterCMS, an API-first CMS][6]. He loves whipping up Butter puns and building tools that makes developers lives better. For more content like this, follow [@ButterCMS][7] on Twitter and [subscribe to our blog][8].[More about me][9]
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
via: https://opensource.com/article/18/1/how-choose-between-monolith-microservices
|
||||
|
||||
作者：[jakelumetta ][a]
|
||||
译者：[译者ID](https://github.com/译者ID)
|
||||
校对：[校对者ID](https://github.com/校对者ID)
|
||||
|
||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
|
||||
|
||||
[a]:https://opensource.com/users/jakelumetta
|
||||
[1]:https://blog.openshift.com/microservices-how-to-explain-them-to-your-ceo/?intcmp=7016000000127cYAAQ&src=microservices_resource_menu1
|
||||
[2]:https://www.openshift.com/promotions/microservices.html?intcmp=7016000000127cYAAQ&src=microservices_resource_menu2
|
||||
[3]:https://opensource.com/business/16/11/secured-devops-microservices?src=microservices_resource_menu3
|
||||
[4]:https://opensource.com/article/18/1/how-choose-between-monolith-microservices?rate=tSotlNvwc-Itch5fhYiIn5h0L8PcUGm_qGvqSVzu9w8
|
||||
[5]:http://rubyonrails.org/
|
||||
[6]:https://buttercms.com/
|
||||
[7]:https://twitter.com/ButterCMS
|
||||
[8]:https://buttercms.com/blog/
|
||||
[9]:https://opensource.com/users/jakelumetta
|
||||
[10]:https://opensource.com/user/205531/feed
|
||||
[11]:https://www.flickr.com/photos/onasill/16452059791/in/photolist-r4P7ci-r3xUqZ-JkWzgN-dUr8Mo-biVsvF-kA2Vot-qSLczk-nLvGTX-biVxwe-nJJmzt-omA1vW-gFtM5-8rsk8r-dk9uPv-5kja88-cv8YTq-eQqNJu-7NJiqd-pBUkk-pBUmQ-6z4dAw-pBULZ-vyM3V3-JruMsr-pBUiJ-eDrP5-7KCWsm-nsetSn-81M3EC-pBURh-HsVXuv-qjgBy-biVtvx-5KJ5zK-81F8xo-nGFQo3-nJr89v-8Mmi8L-81C9A6-qjgAW-564xeQ-ihmDuk-biVBNz-7C5VBr-eChMAV-JruMBe-8o4iKu-qjgwW-JhhFXn-pBUjw
|
||||
[12]:https://creativecommons.org/licenses/by-nc-sa/2.0/
|
||||
[13]:https://buttercms.com/books/microservices-for-startups/
|
||||
[14]:https://www.particle.io/Particle
|
||||
[15]:https://www.scalyr.com/
|
||||
[16]:https://www.algolia.com/
|
||||
[17]:https://pantheon.io/
|
||||
[18]:https://opensource.com/users/jakelumetta
|
||||
[19]:https://opensource.com/users/jakelumetta
|
||||
[20]:https://opensource.com/users/jakelumetta
|
||||
[21]:https://opensource.com/tags/microservices
|
||||
[22]:https://opensource.com/tags/devops
|
||||
Loading…
Reference in New Issue
Block a user