Source-linked AI summary
Analysis of Web Logs and Web User in Web Mining
L. K. Joshila Grace, V. Maheswari, Dhinaharan Nagamalai
TL;DR
Web usage mining analyzes log files to measure website utilization, including frequently visited sites and sites used for longer durations. The paper reviews web-log formats and processing for usage mining and proposes recording clicks, scrolling, and other user operations in extended logs. The paper describes access patterns as page nodes linked by user counts and says session-time differences can still support efficient web mining.
Problem
Web usage mining analyzes log files to measure website utilization, including frequently visited sites and sites used for longer durations.
Source: §2 · p. 2Method
The paper reviews web-log formats and processing for usage mining and proposes recording clicks, scrolling, and other user operations in extended logs.
Source: §4.2 · p. 4, §8 · p. 11Results
The paper describes access patterns as page nodes linked by user counts and says session-time differences can still support efficient web mining.
Source: §7.2 · p. 9, §8 · p. 11Takeaways & Limitations
The paper proposes analyzing an initial page opening and log-file utility to identify user interests and tailor subsequent search results.
Source: §9 · p. 11Because of space constraints, the paper discusses only a few status codes, although response status codes are entered in the log.
Source: 5xx Server Error · p. 7
Abstract
from arXiv · showhide
Log files contain information about User Name, IP Address, Time Stamp, Access Request, number of Bytes Transferred, Result Status, URL that Referred and User Agent. The log files are maintained by the web servers. By analysing these log files gives a neat idea about the user. This paper gives a detailed discussion about these log files, their formats, their creation, access procedures, their uses, various algorithms used and the additional parameters that can be used in the log files which in turn gives way to an effective mining. It also provides the idea of creating an extended log file and learning the user behaviour.
1. INTRODUCTION
Web servers deliver requested pages to browsers, while their logs record client activity. The paper describes log fields used to examine site usage and user behavior.
1. INTRODUCTION
- Web servers return requested page data to browsers, which format it for display to users.A server can deliver pages to multiple client computers simultaneously.Source: §1 · p. 1
2. CONTENTS OF A LOG FILE
- User identification may rely on a temporary ISP-assigned IP address, limiting unique identification; authenticated usernames can distinguish revisits.The paper contrasts IP-based identification with access using a username and password.Source: §2 · p. 1
- A log records requests for web resources and methods such as GET and POST.The paper also lists URL, timestamps, user agent, and other usage-related fields.Source: §2 · p. 2
- Log analysis can quantify site use through frequently visited pages and time spent on a site.The paper identifies these measures as targets of web usage mining.Source: §2 · p. 2
3. LOCATION OF A LOG FILE
The paper describes where web logs reside and how Apache server logs capture requests, errors, and client-reported information in different formats.
3. LOCATION OF A LOG FILE
- A Web log records information each time a user requests a site from a server, and the paper identifies three possible locations.These include server, proxy-server, and client-browser logs.Source: §3 · p. 2, §3.1 · p. 2, §3.2 · p. 3
3.2 Web Proxy Server Log files
- A proxy-mediated request may be logged under the proxy server’s information rather than the original user’s.Proxy servers keep separate logs for gathering user information.Source: §3.2 · p. 3
4 TYPES OF WEB SERVER LOGS
- Web server logs include error and access logs, while referrer and agent information may be separately enabled or added to an extended format.The access log records requests processed by the server; error entries record request errors.Source: §4 · p. 3, §4.1 · p. 4, §4.2 · p. 4
4.2.1 Common Log Format (CLF)
- The Common Log Format example records client address, identity fields, timestamp, request, server status, and returned-object size.The request field contains the client’s request, and the byte count excludes response headers.Source: §4.2.1 · p. 4
4.2.2 Combined Log Format
- The Combined Log Format adds the reported referring site and user-agent information to the common fields.The referrer is the site the client reports coming from, and the user agent is information the browser reports about itself.Source: §4.2.2 · p. 5
4.2.3 Multiple Access Logs
- Multiple Custom Log directives can create separate access, referrer, and agent logs, dividing basic, referrer, and browser information across files.The paper describes this setup as combining common and combined log formats.Source: §4.2.3 · p. 5
5 STATUS CODES SENT BY THE SERVER
The server sends a status code after processing a client request; the paper introduces status codes for Apache HTTP Server Version 1.3.
5 STATUS CODES SENT BY THE SERVER
- Apache HTTP Server Version 1.3 uses status codes sent after processing a client request.The status code is included with the response data and entered in the log file.Source: §5 · p. 5
1xx Info
The 1xx information group indicates that a request is being processed rather than describing a completed response.
1xx Info
- The 1xx information group includes requests received while processing continues, protocol switching, and processing status.The examples listed are 100 Continue, 101 Switching Protocols, and 102 Processing.Source: 1xx Info · p. 6
2xx Success
The section identifies successful HTTP responses and lists three success-status codes.
2xx Success
- HTTP_SUCCESS means an action was successfully received, understood, and accepted.Source: 2xx Success · p. 6
- 200 OK is identified as HTTP_OK.Source: 2xx Success · p. 6
- 201 Created and 202 Accepted are identified as HTTP_CREATED and HTTP_ACCEPTED, respectively.Source: 2xx Success · p. 6
3xx Redirect
The section describes redirection responses as requiring further client action and names three redirect status codes.
3xx Redirect
- HTTP_REDIRECT means the client must take additional action to complete the request.Source: 3xx Redirect · p. 6
- 301 Moved Permanently and 302 Found are identified as HTTP_MOVED_PERMANENTLY and HTTP_MOVED_TEMPORARILY.Source: 3xx Redirect · p. 6
- 304 Not Modified is identified as HTTP_NOT_MODIFIED.Source: 3xx Redirect · p. 6
4xx Client Error
The section distinguishes client errors from server errors and gives representative status codes, while noting that only a few codes are covered.
4xx Client Error
- HTTP_CLIENT_ERROR indicates a request with bad syntax or one that cannot be fulfilled.Source: 4xx Client Error · p. 6
- Examples of 4xx codes include 400 Bad Request, 401 Authorization Required, 402 Payment Required, 404 Not Found, and 405 Method Not Allowed.Source: 4xx Client Error · p. 6
5xx Server Error
- HTTP_SERVER_ERROR means the server failed to fulfill an apparently valid request; examples include 500, 501, 503, and 505.The listed examples are Internal Server Error, Method Not Implemented, Service Temporarily Unavailable, and HTTP Version Not Supported.Source: 5xx Server Error · pp. 6–7
- Only a few status codes are discussed due to space constraints, and response status codes are also entered in the log file.Source: 5xx Server Error · p. 7
6 OVERVIEW OF WEB MINING
Web mining applies data-mining techniques to web documents, extracting information for website features; the overview distinguishes content, structure, and usage mining.
6 OVERVIEW OF WEB MINING
- Web mining extracts information from the World Wide Web using data-mining practices and puts it into website features.Source: §6 · p. 7
6.1 Web Structure mining
- Web structure mining uses graph theory to analyze website connections and node structure, including hyperlinks and the tree-like structure of HTML or XHTML tags.Source: §6.1 · p. 7
6.2 Web Content mining
- Web content mining is described as directly mining document content or improving content search in tools such as search engines.Source: §6.2 · p. 7
6.3 Web Usage mining
- Web usage mining extracts browsing and navigation patterns from log-file session information, including frequent access and time spent on a site.The paper says these patterns can inform website structure design.Source: §6.3 · p. 8
7 USING LOG FILE DATA IN WEB USAGE MINING
Web usage mining processes log files through preprocessing, pattern discovery, and pattern analysis. The paper describes several discovery techniques for examining web-page paths, user navigation, and relationships among accessed items.
7.1 Preprocessing
- Preprocessing removes unwanted image, graphic, and multimedia entries, identifies sessions from timestamps or user IDs, and converts log data into mining-ready formats.The paper notes that raw log data cannot be used directly for mining.Source: §7.1 · p. 8
7.2 Pattern discovery
- Path analysis models web pages as graph nodes and links as edges, with Figure 5 annotating links by the number of users who used them.The graph distinguishes links between websites from links between pages within a website.Source: §7.2 · p. 9, Figure 5 · p. 9
- Discovery also includes association rules for correlations among items sharing a customer or session ID, sequential patterns for ordered navigation, and clustering or classification of similar item profiles.The sequential-pattern example gives 30% of users following A=>B=>C=>D=>E; classification can reveal average customer age for a product, involving personal information.Source: §7.2 · pp. 9–10
7.3 Pattern analysis
- Pattern analysis filters irrelevant or uninteresting discovered patterns and can use queries, OLAP data cubes, or visualization to identify trends.Related techniques include WEBMINER's site filter, mWAP, EXT-Prefixspan, and BC-WAPT, which respectively filter direct-link rules, reduce intermediate-tree reconstruction, reduce candidate generation, and avoid recursive WAP-tree reconstruction.Source: §7.3 · pp. 10–11
8 CREATION OF AN EXTENDED LOG FILE
The paper proposes extending web logs with details of user clicks and other page interactions. It notes that a page can be logged as used even when the user is working in another browser window.
8 CREATION OF AN EXTENDED LOG FILE
- Adding click details could make log contents more efficient for web usage mining.The paper also proposes recording time spent scrolling or performing other operations.Source: §8 · p. 11
- A server log may count a page as used after it is opened even when the user is working on another site in a different browser window.The paper treats the other site as idle while recognizing that the log still records the open page as in use.Source: §8 · p. 11
- The paper suggests that small differences in page-use time or session can still support efficient web mining.Source: §8 · p. 11
9 LEARNING USERS AREA OF INTEREST
The paper describes learning a user's area of interest from an initial page-opening attempt rather than repeatedly asking the user questions. It proposes using that interest to narrow subsequent search results.
9 LEARNING USERS AREA OF INTEREST
- The proposed process infers a user's area of interest from the first page-opening attempt, then mines the web again to return results tailored to that interest.The paper says this may minimize the list of options and make searching more effective.Source: §9 · p. 11
10 CONCLUSIONS
- The paper concludes by describing log-file contents and processing for web usage mining, including mechanisms and their disadvantages.Source: §10 · p. 11
- The paper identifies combining user-interest learning with log-file analysis as extended work.Source: §10 · p. 11