mirror of
https://github.com/LCTT/TranslateProject.git
synced 2025-01-25 23:11:02 +08:00
translated by ThomazL
This commit is contained in:
ThomazL
parent
cd0c5af250
commit
291096c965
@ -1,141 +0,0 @@
|
|||||||
>> Translating by ThomazL 11/04/2014
|
|
||||||
|
|
||||||
|
|
||||||
How to run SQL queries against Apache log files on Linux
|
|
||||||
================================================================================
|
|
||||||
One of the distinguishing features of Linux is that, under normal circumstances, you should be able to know what is happening and has happened on your system by analyzing one or more system logs. Indeed, system logs are the first resource a system administrator tends to look to while troubleshooting system or application issues. In this article, we will focus on the Apache access log files generated by Apache HTTP web server. We will explore an alternative way of analyzing Apache access logs using [asql][1], an open-source tool that allows one to run SQL queries against the logs in order to view the same information in a more friendly format.
|
|
||||||
|
|
||||||
### Background on Apache Logs ###
|
|
||||||
|
|
||||||
There are two kinds of Apache logs:
|
|
||||||
|
|
||||||
- **Access log**: Found at /var/log/apache2/access.log (for Debian) or /var/log/httpd/access_log (for Red Hat). Contains records of every request served by an Apache web server.
|
|
||||||
- **Error log**: Found at /var/log/apache2/error.log (for Debian) or /var/log/httpd/error_log (for Red Hat). Contains records of all error conditions reported by an Apache web server. Error conditions include, but are not limited to, 403 (Forbidden, usually returned after a valid request missing access credentials or insufficient read permissions), and 404 (Not found, returned when the requested resource does not exist).
|
|
||||||
|
|
||||||
Although the verbosity of Apache access log file can be customized through Apache's configuration files, we will assume the default format in this article, which is as follows:
|
|
||||||
|
|
||||||
Remote IP - Request date - Request type - Response code - Requested resource - Remote browser (may also include operating system)
|
|
||||||
|
|
||||||
So a typical Apache log entry looks like:
|
|
||||||
|
|
||||||
192.168.0.101 - - [22/Aug/2014:12:03:36 -0300] "GET /icons/unknown.gif HTTP/1.1" 200 519 "http://192.168.0.10/test/projects/read_json/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
|
|
||||||
|
|
||||||
But what about Apache error log? Since error log entries dealing with particular requests have corresponding entries in the access log (which you can customize), you can use the access log file to obtain more information about error conditions (refer to example 5 for more details).
|
|
||||||
|
|
||||||
That being said, please note that access log is a system-wide log file. To find the log files of virtual hosts, you may also need to check their corresponding configuration files (e.g., within /etc/apache2/sites-available/[virtual host name] on Debian).
|
|
||||||
|
|
||||||
### Installing asql on Linux ###
|
|
||||||
|
|
||||||
asql is written in Perl, and requires two Perl modules: a DBI driver for SQLite and GNU readline.
|
|
||||||
|
|
||||||
#### Install asql on Debian, Ubuntu or their derivatives ####
|
|
||||||
|
|
||||||
asql and its dependencies will automatically be installed with aptitude on Debian-based distributions.
|
|
||||||
|
|
||||||
# aptitude install asql
|
|
||||||
|
|
||||||
#### Install asql on Fedora, CentOS or RHEL ####
|
|
||||||
|
|
||||||
On CentOS or RHEL, you will need to enable [EPEL repository][2] first, and then run the commands below. On Fedora, proceed to the following commands directly.
|
|
||||||
|
|
||||||
# sudo yum install perl-DBD-SQLite perl-Term-ReadLine-Gnu
|
|
||||||
# wget http://www.steve.org.uk/Software/asql/asql-1.7.tar.gz
|
|
||||||
# tar xvfvz asql-1.7.tar.gz
|
|
||||||
# cd asql
|
|
||||||
# make install
|
|
||||||
|
|
||||||
### How Does asql Work? ###
|
|
||||||
|
|
||||||
As you can guess from the dependencies listed above, asql converts unstructured plain-text Apache log files into a structured SQLite database, which can be queried using standard SQL commands. This database can be populated with the contents of current and past log files - including compressed rotated logs such as access.log.X.gz. or access_log.old.
|
|
||||||
|
|
||||||
First, launch asql from the command line with the following command
|
|
||||||
|
|
||||||
# asql
|
|
||||||
|
|
||||||
You will be entering asql's built-in shell interface.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Let's type help to list the available commands in the asql shell:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
We will begin by loading all the access logs in asql, which can be done with:
|
|
||||||
|
|
||||||
asql> load </path/to/apache-access-logs>
|
|
||||||
|
|
||||||
In case of Debian, the following command will do:
|
|
||||||
|
|
||||||
asql> load /var/log/apache2/access.*
|
|
||||||
|
|
||||||
In case of CentOS/RHEL, use this command instead:
|
|
||||||
|
|
||||||
asql> load /var/log/httpd/access_log*
|
|
||||||
|
|
||||||
When asql finishes loading access logs, we can start querying the database. Note that the database created after loading is "temporary," meaning that if you exit the asql shell, the database will be lost. If you want to preserve the database, you have to save it to a file first. We will see how to do that later (refer to examples 3 and 4).
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The database contains a table named logs. The available fields in the logs table can be displayed using the show command:
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The .asql hidden file, which is stored in each user's home directory, records the history of the commands that were typed by the user in an asql shell. Thus, you can browse through it using the arrow keys, and repeat previous commands by just pressing ENTER when you find the right one.
|
|
||||||
|
|
||||||
### SQL Query Examples with asql ###
|
|
||||||
|
|
||||||
Here a few examples of running SQL queries against Apache log files with asql.
|
|
||||||
|
|
||||||
**Example 1**: Listing the request sources / dates and HTTP status codes returned during the month of October 2014.
|
|
||||||
|
|
||||||
SELECT source, date, status FROM logs WHERE date >= '2014-10-01T00:00:00' ORDER BY source;
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**Example 2**: Displaying the total size (in bytes) of requests served per client in descending order.
|
|
||||||
|
|
||||||
SELECT source,SUM(size) AS Number FROM logs GROUP BY source ORDER BY Number DESC;
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**Example 3**: Saving the database to [filename] in the current working directory.
|
|
||||||
|
|
||||||
save [filename]
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This allows us to avoid the need for waiting while the log parsing is performed with the load command as shown earlier.
|
|
||||||
|
|
||||||
**Example 4**: Restoring the database in a new asql session after exiting the current one.
|
|
||||||
|
|
||||||
restore [filename]
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
**Example 5**: Returning error conditions logged in the access file. In this example, we will display all the requests that returned a 403 (access forbidden) HTTP code.
|
|
||||||
|
|
||||||
SELECT source,date,status,request FROM logs WHERE status='403' ORDER BY date
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This goes to show that although asql only analyzes access logs, we can use the status field of a request to display requests with error conditions.
|
|
||||||
|
|
||||||
### Summary ###
|
|
||||||
|
|
||||||
We have seen how asql can help us analyze Apache logs and present the results in a user friendly output format. Although you could obtain similar results by using command line utilities such as cat in conjunction with grep, uniq, sort, and wc (to name a few), in comparison asql represents a Swiss army knife due to the fact that it allows us to use standard SQL syntax to filter the logs according to our needs.
|
|
||||||
|
|
||||||
Feel free to leave your questions or comments below. Hope it helps.
|
|
||||||
|
|
||||||
--------------------------------------------------------------------------------
|
|
||||||
|
|
||||||
via: http://xmodulo.com/sql-queries-apache-log-files-linux.html
|
|
||||||
|
|
||||||
作者：[Gabriel Cánepa][a]
|
|
||||||
译者：[译者ID](https://github.com/译者ID)
|
|
||||||
校对：[校对者ID](https://github.com/校对者ID)
|
|
||||||
|
|
||||||
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创翻译，[Linux中国](http://linux.cn/) 荣誉推出
|
|
||||||
|
|
||||||
[a]:http://xmodulo.com/author/gabriel
|
|
||||||
[1]:http://www.steve.org.uk/Software/asql/
|
|
||||||
[2]:http://xmodulo.com/how-to-set-up-epel-repository-on-centos.html
|
|
||||||
@ -0,0 +1,145 @@
|
|||||||
|
如何在 Linux 上针对 Apache 日志文件运行 SQL 查询语句
|
||||||
|
================================================
|
||||||
|
|
||||||
|
Linux 有一个显著的特点，在正常情况下，你可以通过分析系统日志来了解你的系统中发生了什么，或正在发生什么。的确，系统日志是系统管理员在解决系统和应用问题时最需要的第一手资源。我们将在这篇文章中着重讲解 Apache HTTP web server 生成的 Apache access 日志。
|
||||||
|
|
||||||
|
我们会通过另类的途径分析 Apache access 日志，我们使用的工具是 [asql][1]。asql 是一个开源的工具，它能够允许使用者针对日志运行 SQL 查询，从而通过更加友好的格式展现相同的信息。
|
||||||
|
|
||||||
|
### Apache 日志背景知识 ###
|
||||||
|
|
||||||
|
Apache 有两种日志：
|
||||||
|
|
||||||
|
- **Access log**：存放在路径 /var/log/apache2/access.log (Debian) 或者 /var/log/httpd/access_log (Red Hat)。Access Log 记录所有 Apache web server 执行的请求。
|
||||||
|
- **Error log**：存放在路径 /var/log/apache2/error.log (Debian) 或者 /var/log/httpd/error_log (Red Hat)。Error log 记录所有 Apache web server 报告的错误以及错误的情况。Error 情况包括（不限于）403（Forbidden，通常在有效请求丢失访问许可时被报告），404（Not found，在请求资源不存在时被报告）。
|
||||||
|
|
||||||
|
虽然管理员可以通过配置 Apache 的配置文件来自定义 Apache access log 的罗嗦程度，在这篇文章中，我们会使用默认的配置，如下：
|
||||||
|
|
||||||
|
Remote IP - Request date - Request type - Response code - Requested resource - Remote browser (may also include operating system)
|
||||||
|
|
||||||
|
因此一个典型的 Apache 日志条目就是下面这个样子：
|
||||||
|
|
||||||
|
192.168.0.101 - - [22/Aug/2014:12:03:36 -0300] "GET /icons/unknown.gif HTTP/1.1" 200 519 "http://192.168.0.10/test/projects/read_json/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0"
|
||||||
|
|
||||||
|
但是 Apache error log 又是怎么样的呢？因为 error log 条目主要记录 access log 中特殊的请求（你可以自定义），你可以通过 access log 来获得关于错误情况的更多信息（example 5 有更多细节）。
|
||||||
|
|
||||||
|
话在前头， access log 是系统级别的日志文件。要分析 virtual hosts 的日志文件，你需要检查它们相应的配置文件（e.g. 在 /etc/apache2/sites-available/[virtual host name] 里（Debian））。
|
||||||
|
|
||||||
|
### 在 Linux 上安装 asql ###
|
||||||
|
|
||||||
|
asql 由 Perl 编写，而且需求以下两个 Perl module：SQLite 的 DBI 驱动以及 GNU readline.
|
||||||
|
|
||||||
|
### 在 Debian, Ubuntu 以及其衍生发行版上安装 asql ###
|
||||||
|
|
||||||
|
使用基于 Debian 发行版上的 aptitude，asql 以及其依赖会被自动安装。
|
||||||
|
|
||||||
|
# aptitude install asql
|
||||||
|
|
||||||
|
### 在 Fedora，CentOS，RHEL 上安装 asql ###
|
||||||
|
|
||||||
|
在 CentOS 或 RHEL 上，你需要启用 [EPEL repository][2]，然后运行以下代码。在 Fedora 中，直接运行以下代码：
|
||||||
|
|
||||||
|
# sudo yum install perl-DBD-SQLite perl-Term-Readline-Gnu
|
||||||
|
# wget http://www.steve.org.uk/Software/asql/asql-1.7.tar.gz
|
||||||
|
# tar xvfvz asql-1.7.tar.gz
|
||||||
|
# cd asql
|
||||||
|
# make install
|
||||||
|
|
||||||
|
### asql 是如何工作的？ ###
|
||||||
|
|
||||||
|
从上面代码中的依赖中你就可以看出来，asql 转换未结构化的明文 Apache 日志成结构化的 SQLite 数据库信息。生成的 SQLite 数据库接受正常的 SQL 查询语句。数据库可以通过当前以及曾经的日志文件生成，其中包括压缩转换过的日志文件，类似 access.log.X.gz 或者 access_log.old。
|
||||||
|
|
||||||
|
首先，从命令行启动 asql：
|
||||||
|
|
||||||
|
# asql
|
||||||
|
|
||||||
|
你会进入 asql 内置的 shell 交互界面。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
输入 help 列表可执行的命令：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
首先在 asql 中加载所有的 access 日志：
|
||||||
|
|
||||||
|
asql > load <path/to/apache-access-logs>
|
||||||
|
|
||||||
|
比如在 Debian 下：
|
||||||
|
|
||||||
|
asql > load /var/log/apache2/access.*
|
||||||
|
|
||||||
|
在 CentOS/RHEL 下：
|
||||||
|
|
||||||
|
asql > load /var/log/httpd/access_log*
|
||||||
|
|
||||||
|
当 asql 完成对 access 日志的加载后，我们就可以开始数据库查询了。注意一下，加载后生成的数据库是 "temporary" 的，意思就是数据库会在你退出 asql 的时候被清除。如果你想要保留数据库，你必须先将其保存为一个文件。我们会在后面介绍如何这么做（参考 example 3 和 4）。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
生成的数据库有一个名为 logs 的表。输入下面的命令列出 logs 表中提供的域：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
名为 .asql 的隐藏文件，保存于用户的 home 目录下，记录用户在 asql shell 中输入的命令历史。因此你可以使用方向键浏览历史命令，按下 ENTER 来重复执行之前的命令。
|
||||||
|
|
||||||
|
### asql 上的示例 SQL 查询 ###
|
||||||
|
|
||||||
|
下面是几个使用 asql 针对 Apache 日志文件运行 SQL 查询的示例：
|
||||||
|
|
||||||
|
**Example 1**：列出在 2014 年 10 月中请求的来源 / 时间以及 HTTP 状态码。
|
||||||
|
|
||||||
|
SELECT source, date, status FROM logs WHERE date >= '2014-10-01T00:00:00' ORDER BY source;
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**Example 2**：从小到大显示单个客户端处理的请求大小（bytes）。
|
||||||
|
|
||||||
|
SELECT source, SUM(size), AS NUMBER FROM logs GROUP BY source ORDER BY Number DESC;
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**Example 3**：在当前目录中保存数据库为 [filename]。
|
||||||
|
|
||||||
|
save [filename]
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
这样做可以避免使用 load 命令对日志的语法分析所占用的处理时间。
|
||||||
|
|
||||||
|
**Example 4**：在重新启动 asql 后载入数据库。
|
||||||
|
|
||||||
|
restore [filename]
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**Example 5**：返回 access 日志中记录的 error 情况。在这个例子中，我们将显示所有返回 HTTP 状态码为 403（access forbidden）的请求。
|
||||||
|
|
||||||
|
SELECT source, date, status, request FROM logs WHERE status='403' ORDER BY date
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
这个例子想要表现的是：虽然 asql 只分析 access 日志，我们还是可以通过使用请求的状态域来显示有 error 情况的请求。
|
||||||
|
|
||||||
|
### 小结： ###
|
||||||
|
|
||||||
|
我们体验了 asql 如何帮助我们分析 Apache 日志文件，并将结果通过友好的格式输出。虽然你也可以通过使用命令行的工具（例如 cat 与 grep，uniq，sort，wc 等等之间的管道）来实现类似功能，与此比较起来 asql 展示了它如同瑞士军刀一般的强大功能，使我们在自己的需求下能够通过标准 SQL 查询语句来过滤日志。
|
||||||
|
|
||||||
|
希望这篇教程能帮助到你们。
|
||||||
|
|
||||||
|
请不要拘束地将评论文章，分享文章，提出疑问。
|
||||||
|
|
||||||
|
|
||||||
|
--------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
via: http://xmodulo.com/sql-queries-apache-log-files-linux.html
|
||||||
|
|
||||||
|
作者：[Gabriel Cánepa][a]
|
||||||
|
译者：[ThomazL](https://github.com/ThomazL)
|
||||||
|
校对：[校对者ID](https://github.com/校对者ID)
|
||||||
|
|
||||||
|
本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创翻译，[Linux中国](http://linux.cn/) 荣誉推出
|
||||||
|
|
||||||
|
[a]:http://xmodulo.com/author/gabriel
|
||||||
|
[1]:http://www.steve.org.uk/Software/asql/
|
||||||
|
[2]:http://xmodulo.com/how-to-set-up-epel-repository-on-centos.html
|
||||||
|
|
||||||
Loading…
Reference in New Issue
Block a user