Contains the Apache protocol REFERER

Analyze and interpret the Apache web server protocol

Apache web servers can generate many logs. These logs contain information such as the HTTP requests that Apache processed and responded to, as well as other activities that are specific to Apache. Analyzing the logs is an important part of managing Apache and making sure it is performing as expected.

This guide explains the various logging options available in Apache and how to interpret this log data. You will learn how to analyze the logs that Apache creates and how to configure the logging settings so that you get the most relevant data about Apache's activity.

In this tutorial you will learn:
  • Configure and understand Apache web server logging
  • What are Apache Log Levels?
  • Interpretation of the Apache log formatting and its meaning
  • What are the most common Apache log configuration files?
  • How to add forensic data to the logging configuration
Analyze and interpret the Apache web server protocol

Software requirements and conventions used

categoryRequirements, conventions or software version used
systemUbuntu, Debian, CentOS, RHEL, Fedora
softwareApache web server
OtherPrivileged access to your Linux system as root or via command.
Conventions# - Requires certain Linux commands with root privileges either directly as the root user or with the command
$ are executed - Requires certain Linux commands to be run as a regular non-privileged user

Apache log files and their location

Apache creates two different log files:
  • access.log stores information about all incoming connection requests to Apache. Every time a user visits your website, it is logged here. Each page that a user requests is also logged as a separate entry.
  • error.log stores information about errors Apache encounters throughout the process. Ideally, this file should remain relatively empty.
Standard Apache protocol configuration on Ubuntu Linux server
The location of the log files may vary depending on what version of Apache you are running and what Linux distribution they are on. Apache can also be configured to save these files to some other non-standard location.

However, by default you should be able to find the access and error logs in one of these directories:
  • / var / log / apache /
  • / var / log / apache2 /
  • / etc / httpd / logs /

Apache log formatting

Apache allows you to customize what information is logged and how each log entry is displayed. This is covered later in this tutorial.

The usual format that Apache uses to represent log entries is: "% h% l% u% t \"% r \ "%> s% O \"% {Referer} i \ "\"% {User- Agent} i \ "" How to interpret this formatting:
  • % H - The IP address of the client.
  • % l - This is the 'identd' on the client with which you will be identified. This field is usually empty and is shown as a hyphen.
  • % u - The user ID of the client if HTTP authentication was used. If not, the log entry shows nothing for that field.
  • % t - Timestamp of the log entry.
  • \% r \ - The request line from the client. This shows which HTTP method was used (e.g. GET or POST), which file was requested and which HTTP protocol was used.
  • %> s - The status code returned to the client. Codes of 4xx (like 404, page not found) indicate client errors, and codes of 5xx (like 500, internal server error) indicate server errors. Other numbers should indicate success (e.g. 200, OK) or something else like redirect (e.g. 301, permanently moved).
  • % O - The size of the requested file (including the header) in bytes.
  • "% {Referer} i" - The referring link, if applicable. Here's how the user navigated to your page (either via an internal or an external link).
  • "% {User-Agent} i" - Contains information about the web browser and the operating system of the connection client.
A typical entry in the access log looks something like this: 10.10.220.3 - - [17 / Dec / 2019: 23: 05: 32 -0500] "GET /products/index.php HTTP / 1.1" 200 5015 "http: / /example.com/products/index.php "" Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 79.0.3945.79 Safari / 537.36 "The error log is a bit simpler and easier to use interpret. A typical entry can look like this: [Mon Dec 16 06: 29: 16.613789 2019] [php7: error] [pid 2095] [client 10.10.244.61:24145] script '/var/www/html/settings.php' not found or unable to stat This is a great way to see how many errors your visitors are encountering and can point you to some dead links on your website. More importantly, you can be alerted to a lack of resources or potential server problems. The above example shows a page that has been requested but is missing.

Apache protocol configuration

Apache logging is highly customizable and can be customized using a few configuration files. On Ubuntu and Debian, the main configuration file for Apache logging is here:
  • /etc/apache2/apache2.conf
Because you can run multiple websites (labeled as) from a single Apache instance, you can also configure each website to have separate access and error logs. Configure this file to define how to name these separate log files and where to save them:
  • /etc/apache2/sites-available/000-default.conf
Under CentOS, RHEL and Fedora the two configuration files are in the following locations:
  • /etc/httpd/conf/httpd.conf
  • /etc/httpd/conf.d/ (additional VirtualHost configurations in this directory insert )

Protocol instructions

There are a few different directives that can be configured in these files, but these are the main ones to look at if you want to customize Apache's logging:
  • CustomLog - Defines where the access log file is saved.
  • ErrorLog - Defines where the error log file is saved.
  • LogLevel - Defines how severe an event must be to be logged (see below for more information).
  • LogFormat - Lay Determine how each entry in the access log should be formatted (see below for more information).
LogLevel is set by default. This means that at Warning conditions or more serious events are written to the error log. If your error log is filled with many harmless warning messages, you can nudge it until it only reports errors or more serious problems.

Other options are (in order of severity), and. Apache recommends using at least a critical level. For debugging purposes, it can be temporarily suspended LogLevel too, but be aware that you may end up with an unwieldy amount of error log entries.

With LogFormat you can customize how the entries appear in the access log. If you have the example entry in (from the Apache log formatting section above) to be a little confusing, you are not alone. Apache allows you to customize the format of log entries to make them more logical. You can also use this customization to exclude certain information that you may find irrelevant.

Apache logging modules

The logging configuration we've shown so far in this guide is related to the Apache module. To extend the logging functionality even further, you can load other logging modules into Apache. This can provide some more features that are not available with the default settings.

mod_log_forensic starts logging before a request (when the headers are first received) and logs again after the request. This means that two log entries are created for each request so that an administrator can measure the response times more precisely.

Define the location of your forensic log with the instruction. For example: CustomLog $ {APACHE_LOG_DIR} /forensic.log forensicmod_logio logs the number of bytes sent to and received from each request. It provides very accurate information as it also counts the data that is in the header and body of each request, as well as the additional data required for SSL / TLS encrypted connections.

Add the placeholders and to the directive to use the additional data from this module. Other modules exist; These are just two of the most useful.

Conclusion

In this article, we've seen how Apache parses and interprets the access and error logs. We also learned how to customize logging in Apache's configuration files to make the log data more relevant. With this knowledge, you can isolate problems and troubleshoot Apache problems faster.

Remember that Apache's logging functionality can be enhanced with other logging engines. However, this is only necessary in marginal cases in which advanced debugging is required.

Something like that