document/TranslateProject
2
0
Fork 0
mirror of https://github.com/LCTT/TranslateProject.git synced 2024-12-29 21:41:00 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #14414 from lujun9972/add-MjAxOTA3MDIgSnVweXRlciBhbmQgZGF0YSBzY2llbmNlIGluIEZlZG9yYS5tZAo=

Browse Source 
自动选题: 20190702 Jupyter and data science in Fedora
This commit is contained in:
Xingyu.Wang 2019-07-03 15:59:19 +08:00 committed by GitHub
commit fc53f822d0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 246 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

246
sources/tech/20190702 Jupyter and data science in Fedora.md Normal file
View File

@ -0,0 +1,246 @@
[#]: collector: (lujun9972)
[#]: translator: ( )
[#]: reviewer: ( )
[#]: publisher: ( )
[#]: url: ( )
[#]: subject: (Jupyter and data science in Fedora)
[#]: via: (https://fedoramagazine.org/jupyter-and-data-science-in-fedora/)
[#]: author: (Avi Alkalay https://fedoramagazine.org/author/aviram/)
Jupyter and data science in Fedora
======
![][1]
In the past, kings and leaders used oracles and magicians to help them predict the future — or at least get some good advice due to their supposed power to perceive hidden information. Nowadays, we live in a society obsessed with quantifying everything. So we have data scientists to do this job.
Data scientists use statistical models, numerical techniques and advanced algorithms that didn’t come from statistical disciplines, along with the data that exist on databases, to find, to infer, to predict data that doesn’t exist yet. Sometimes this data is about the future. That is why we do a lot of predictive analytics and prescriptive analytics.
Here are some questions to which data scientists help find answers:
1. Who are the students with high propensity to abandon the class? For each one, what are the reasons for leaving?
2. Which house has a price above or below the fair price? What is the fair price for a certain house?
3. What are the hidden groups that my clients classify themselves?
4. Which future problems this premature child will develop?
5. How many calls will I get in my call center tomorrow 11:43 AM?
6. My bank should or should not lend money to this customer?
Note how the answer to all these question is not sitting in any database waiting to be queried. These are all data that still doesn’t exist and has to be calculated. That is part of the job we data scientists do.
Throughout this article you’ll learn how to prepare a Fedora system as a Data Scientist’s development environment and also a production system. Most of the basic software is RPM-packaged, but the most advanced parts can only be installed, nowadays, with Python’s _pip_ tool.
### Jupyter — the IDE
Most modern data scientists use Python. And an important part of their work is EDA (exploratory data analysis). EDA is a manual and interactive process that retrieves data, explores its features, searches for correlations, and uses plotted graphics to visualize and understand how data is shaped and prototypes predictive models.
Jupyter is a web application perfect for this task. Jupyter works with Notebooks, documents that mix rich text including beautifully rendered math formulas (thanks to [mathjax][2]), blocks of code and code output, including graphics.
Notebook files have extension _.ipynb_, which means Interactive Python Notebook.
#### Setting up and running Jupyter
First, install essential packages for Jupyter ([using][3] _[sudo][3]_):
```
$ sudo dnf install python3-notebook mathjax sscg
```
You might want to install additional and optional Python modules commonly used by data scientists:
```
$ sudo dnf install python3-seaborn python3-lxml python3-basemap python3-scikit-image python3-scikit-learn python3-sympy python3-dask+dataframe python3-nltk
```
Set a password to log into Notebook web interface and avoid those long tokens. Run the following command anywhere on your terminal:
```
$ mkdir -p $HOME/.jupyter
$ jupyter notebook password
```
Now, type a password for yourself. This will create the file _$HOME/.jupyter/jupyter_notebook_config.json_ with your encrypted password.
Next, prepare for SSLby generating a self-signed HTTPS certificate for Jupyter’s web server:
```
$ cd $HOME/.jupyter; sscg
```
Finish configuring Jupyter by editing your _$HOME/.jupyter/jupyter_notebook_config.json_ file. Make it look like this:
```
{
"NotebookApp": {
"password": "sha1:abf58...87b",
"ip": "*",
"allow_origin": "*",
"allow_remote_access": true,
"open_browser": false,
"websocket_compression_options": {},
"certfile": "/home/aviram/.jupyter/service.pem",
"keyfile": "/home/aviram/.jupyter/service-key.pem",
"notebook_dir": "/home/aviram/Notebooks"
}
}
```
The parts in red must be changed to match your folders. Parts in blue were already there after you created your password. Parts in green are the crypto-related files generated by _sscg_.
Create a folder for your notebook files, as configured in the _notebook_dir_ setting above:
```
$ mkdir $HOME/Notebooks
```
Now you are all set. Just run Jupyter Notebook from anywhere on your system by typing:
```
$ jupyter notebook
```
Or add this line to your _$HOME/.bashrc_ file to create a shortcut command called _jn_:
```
alias jn='jupyter notebook'
```
After running the command _jn_, access _<https://your-fedora-host.com:8888>_ from any browser on the network to see the Jupyter user interface. You’ll need to use the password you set up earlier. Start typing some Python code and markup text. This is how it looks:
![Jupyter with a simple notebook][4]
In addition to the IPython environment, you’ll also get a web-based Unix terminal provided by _terminado_. Some people might find this useful, while others find this insecure. You can disable this feature in the config file.
### JupyterLab — the next generation of Jupyter
JupyterLab is the next generation of Jupyter, with a better interface and more control over your workspace. It’s currently not RPM-packaged for Fedora at the time of writing, but you can use _pip_ to get it installed easily:
```
$ pip3 install jupyterlab --user
$ jupyter serverextension enable --py jupyterlab
```
Then run your regular _jupiter notebook_ command or _jn_ alias. JupyterLab will be accessible from _<http://your-linux-host.com:8888/**lab>_**.
### Tools used by data scientists
In this section you can get to know some of these tools, and how to install them. Unless noted otherwise, the module is already packaged for Fedora and was installed as prerequisites for previous components.
#### **Numpy**
_Numpy_ is an advanced and C-optimized math library designed to work with large in-memory datasets. It provides advanced multidimensional matrix support and operations, including math functions as log(), exp(), trigonometry etc.
#### Pandas
In this author’s opinion, Python is THE platform for data science mostly because of Pandas. Built on top of numpy, Pandas makes easy the work of preparing and displaying data. You can think of it as a no-UI spreadsheet, but ready to work with much larger datasets. Pandas helps with data retrieval from a SQL database, CSV or other types of files, columns and rows manipulation, data filtering and, to some extent, data visualization with matplotlib.
#### Matplotlib
Matplotlib is a library to plot 2D and 3D data. It has great support for notations in graphics, labels and overlays
![matplotlib pair of graphics showing a cost function searching its optimal value through a gradient descent algorithm][5]
#### Seaborn
Built on top of matplotlib, Seaborn’s graphics are optimized for a more statistical comprehension of data. It automatically displays regression lines or Gauss curve approximations of plotted data.
![Linear regression visualised with SeaBorn][6]
#### [StatsModels][7]
StatsModels provides algorithms for statistical and econometrics data analysis such as linear and logistic regressions. Statsmodel is also home for the classical family of [time series algorithms][8] known as ARIMA.
![Normalized number of passengers across time \(blue\) and ARIMA-predicted number of passengers \(red\)][9]
#### Scikit-learn
The central piece of the machine-learning ecosystem, [scikit][10] provides predictor algorithms for [regression][11] (Elasticnet, Gradient Boosting, Random Forest etc) and [classification][11] and clustering (K-means, DBSCAN etc). It features a very well designed API. Scikit also has classes for advanced data manipulation, dataset split into train and test parts, dimensionality reduction and data pipeline preparation.
#### XGBoost
XGBoost is the most advanced regressor and classifier used nowadays. It’s not part of scikit-learn, but it adheres to scikit’s API. [XGBoost][12] is not packaged for Fedora and should be installed with pip. [XGBoost can be accelerated with your nVidia GPU][13], but not through its _pip_ package. You can get this if you compile it yourself against CUDA. Get it with:
```
$ pip3 install xgboost --user
```
#### Imbalanced Learn
[imbalanced-learn][14] provides ways for under-sampling and over-sampling data. It is useful in fraud detection scenarios where known fraud data is very small when compared to non-fraud data. In these cases data augmentation is needed for the known fraud data, to make it more relevant to train predictors. Install it with _pip_:
```
$ pip3 install imblearn --user
```
#### NLTK
The [Natural Language toolkit][15], or NLTK, helps you work with human language data for the purpose of building chatbots (just to cite an example).
#### SHAP
Machine learning algorithms are very good on predicting, but aren’t good at explaining why they made a prediction. [SHAP][16] solves that, by analyzing trained models.
![Where SHAP fits into the data analysis process][17]
Install it with _pip_:
```
$ pip3 install shap --user
```
#### [Keras][18]
Keras is a library for deep learning and neural networks. Install it with _pip_:
```
$ sudo dnf install python3-h5py
$ pip3 install keras --user
```
#### [TensorFlow][19]
TensorFlow is a popular neural networks builder. Install it with _pip_:
```
$ pip3 install tensorflow --user
```
* * *
_Photo courtesy of [FolsomNatural][20] on [Flickr][21] (CC BY-SA 2.0)._
--------------------------------------------------------------------------------
via: https://fedoramagazine.org/jupyter-and-data-science-in-fedora/
作者：[Avi Alkalay][a]
选题：[lujun9972][b]
译者：[译者ID](https://github.com/译者ID)
校对：[校对者ID](https://github.com/校对者ID)
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出
[a]: https://fedoramagazine.org/author/aviram/
[b]: https://github.com/lujun9972
[1]: https://fedoramagazine.org/wp-content/uploads/2019/06/jupyter-816x345.jpg
[2]: http://mathjax.org
[3]: https://fedoramagazine.org/howto-use-sudo/
[4]: https://avi.alkalay.net/articlefiles/2018/07/jupyter-fedora.png
[5]: https://fedoramagazine.org/wp-content/uploads/2019/06/gradient-descent-cost-function-optimization.png
[6]: https://seaborn.pydata.org/_images/regression_marginals.png
[7]: https://www.statsmodels.org/
[8]: https://www.statsmodels.org/stable/examples/index.html#stats
[9]: https://fedoramagazine.org/wp-content/uploads/2019/06/time-series.png
[10]: https://scikit-learn.org/stable/
[11]: https://scikit-learn.org/stable/supervised_learning.html#supervised-learning
[12]: https://xgboost.ai
[13]: https://xgboost.readthedocs.io/en/latest/gpu/index.html
[14]: https://imbalanced-learn.readthedocs.io
[15]: https://www.nltk.org
[16]: https://github.com/slundberg/shap
[17]: https://raw.githubusercontent.com/slundberg/shap/master/docs/artwork/shap_diagram.png
[18]: https://keras.io
[19]: https://www.tensorflow.org
[20]: https://www.flickr.com/photos/87249144@N08/
[21]: https://www.flickr.com/photos/87249144@N08/45871861611/