PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Preprocessing Methods for Web Usage Data

V. Chitraa, Dr. Antony Selvdoss Davamani

arXiv:1004.1257v1cs.IR

TL;DR

Noisy, ambiguous web logs must be prepared to mine user behavior and construct sessions efficiently. This paper surveys preprocessing work and briefly reviews pattern discovery, analysis, and applications, concluding that preprocessing is important before mining.

Abstract

from arXiv · show

World Wide Web is a huge repository of web pages and links. It provides abundance of information for the Internet users. The growth of web is tremendous as approximately one million pages are added daily. Users' accesses are recorded in web logs. Because of the tremendous usage of web, the web log files are growing at a faster rate and the size is becoming huge. Web data mining is the application of data mining techniques in web data. Web Usage Mining applies mining techniques in log data to extract the behavior of users which is used in various applications like personalized services, adaptive web sites, customer profiling, prefetching, creating attractive web sites etc., Web usage mining consists of three phases preprocessing, pattern discovery and pattern analysis. Web log data is usually noisy and ambiguous and preprocessing is an important process before mining. For discovering patterns sessions are to be constructed efficiently. This paper reviews existing work done in the preprocessing stage. A brief overview of various data mining techniques for discovering patterns, and pattern analysis are discussed. Finally a glimpse of various applications of web usage mining is also presented.

I. INTRODUCTION

The introduction frames web mining as applying data-mining techniques to web information, including documents, services, and usage logs. It links user-behavior knowledge to applications and distinguishes three types of web mining.

I. INTRODUCTION

  • Web mining applies data-mining techniques to retrieve, extract, and evaluate web information for knowledge discovery.The introduction defines data mining as automatically extracting unknown, useful, and understandable patterns from large databases.Source: §I · p. 1
  • Knowledge of users’ habits and interests supports enterprise strategies and applications such as e-commerce, personalization, website design, and recommender systems.Source: §I · p. 1
  • Web mining is divided into content mining, structure mining, and usage mining.Content mining concerns web contents or services; structure mining concerns hyperlinks; usage mining mines web-server log data.Source: §I · p. 1

A. Web Usage Mining

Web usage mining analyzes large web-log repositories for user-behavior patterns and website-usage statistics. The paper presents its stages and surveys preprocessing alongside pattern discovery, analysis, and applications.

A. Web Usage Mining

  • Web usage mining applies data-mining techniques to large web-log repositories to discover user-behavior patterns and website-usage statistics.The paper describes those outputs as useful for website design tasks.Source: A. Web Usage Mining · p. 1
  • The four stages are data collection, preprocessing, pattern discovery, and pattern analysis.Preprocessing includes data cleaning, user and session identification, path completion, and transaction identification.Source: A. Web Usage Mining · p. 1
  • The paper surveys preprocessing work and reviews data collection, pattern discovery, pattern analysis, and applications of web usage mining.Source: A. Web Usage Mining · p. 2

II. DATA COLLECTION

Data collection gathers relevant web data from server-side, client-side, proxy, or organizational sources. The section describes logging approaches and constraints that affect what user activity is recorded.

II. DATA COLLECTION

  • Data collection gathers relevant web data from server-side, client-side, proxy, or organizational databases.The paper cites for organizational databases containing business or consolidated web data.Source: §II · p. 2
  • Server logs record client requests as plain text, and some servers add referrer and user-agent fields.The listed common log format includes IP address, username, password, timestamp, URL, version, status code, and bytes sent.Source: §II · p. 2
  • Cookies track site visitors with a server-generated identifier, while users can disable browser acceptance for privacy or security.The paper cites for the cookie description.Source: §II · p. 2
  • Registration-form data can provide personal, demographic, and preference information, but may be unreliable when entries are incorrect or users neglect sites.Source: §II · p. 2
  • Client-side collection can address caching and session-identification problems, but modified browsers must be accepted by users and Java applets can add overhead.The paper also notes that client-side logs can include intentional browsing actions such as adding favorites or copying.Source: §II · p. 2
  • Proxy logs record page requests and responses and can capture traces from multiple clients to multiple web servers.The paper notes that proxy traces may characterize browsing behavior of anonymous users sharing a proxy server.Source: §II · p. 2

III. DATA PREPROCESSING

Preprocessing prepares heterogeneous, unstructured web data for pattern discovery by transforming raw click streams into user profiles. The paper identifies a range of preprocessing challenges and techniques.

III. DATA PREPROCESSING

  • Preprocessing is a prerequisite for discovering patterns because web information is heterogeneous and unstructured.Source: §III · p. 2
  • Preprocessing aims to transform raw click-stream data into a set of user profiles.Source: §III · p. 2
  • Preprocessing challenges have prompted varied algorithms and heuristics for merging, cleaning, and identifying users and sessions.Research also groups sessions and transactions to discover user-behavior patterns.Source: §III · p. 2

A. Data Cleaning

Data cleaning removes irrelevant log entries before analysis, including non-content requests and spider navigation references.

A. Data Cleaning

  • Data cleaning removes requests for irrelevant files such as JPEGs, GIFs, and sound files, as well as spider-navigation references.The paper notes an exception for art-gallery sites, where images may be important.Source: A. Data Cleaning · p. 2
  • Log entries with status codes below 200 or above 299 are removed.Source: A. Data Cleaning · p. 2
  • Because page requests also download graphics and scripts, the paper distinguishes these from the HTML page when cleaning logs.Source: A. Data Cleaning · p. 2

B. User Identification

User identification begins with IP addresses, but shared addresses and multiple browsers require additional signals from extended logs.

B. User Identification

  • Assigning a different user ID to each IP address is a simple identification method, but proxy users can share an address and one user can use multiple browsers.Source: B. User Identification · p. 2
  • With the same IP address, a different user agent indicates a new user; when both match, referrer URL and site topology are checked for a new user.An unreachable requested page is treated as evidence of a new user at that address.Source: B. User Identification · p. 3
  • Assigning HTML pages short expiration times can address caching by forcing browsers to retrieve pages from the server.Source: B. User Identification · p. 3

C. Session Identification

Session identification partitions one user's page sequence into visits, using time, navigation, or combined approaches; the reviewed methods range from heuristics to graph and optimization techniques.

C. Session Identification

  • A user session is a same-user set of pages from one visit, and sessionization partitions that user's clickstream into logical clusters.Transactions are subsets of sessions containing homogeneous pages [16].Source: C. Session Identification · p. 3
  • Time heuristics use either total session duration or single-page stay time; one described rule starts a new session when the interval exceeds 10 minutes.The paper cautions that these methods can be unreliable because of other user activity, network delays, page loading, and page content size,,.Source: C. Session Identification · p. 3
  • Navigation heuristics start a different session when a page is disconnected from the previous page, while referrer-based rules require the referrer URL within the same session.A missing referrer makes the page the first page of a new session.Source: C. Session Identification · p. 3
  • A combined method uses IP address, browser or operating system, and referrer signals before applying time heuristics to split separate visits by the same user.When referrer is empty, a gap of more than 10 seconds can mark a new session.Source: C. Session Identification · p. 3
  • Zhou's algorithm estimates each URL's duration from the next request, assigns the last URL the session's average duration, and uses a 30-minute default session.The paper describes it as suitable when sessions contain more URLs.Source: C. Session Identification · p. 3
  • Smart Miner forms candidate sessions with duration and page-stay rules, then refines them using topology and referrer constraints.Its simulator models dynamic user behavior, and the method also considers a 10-minute interval between consecutive pages.Source: C. Session Identification · p. 3
  • Integer programming groups linked registers sharing an IP address and agent into sessions, constructs all sessions simultaneously, and limits register assignments through binary variables.The paper reports experiments on raw registers and data filtered for embedded objects and errors.Source: C. Session Identification · p. 3
  • Graph-based methods model pages as vertices and links as edges, then use browsing traversals to discover weighted frequent patterns.One reviewed approach uses client-side interaction events, builds a base graph, and mines patterns meeting a minimum browsing-time threshold,; another combines log cleaning with session identification.Source: C. Session Identification · p. 4

D. Path Completion

Path completion addresses pages missing from logs, using browsing history and site topology to infer navigation paths and transactions.

D. Path Completion

  • When proxy servers or caching leave pages missing, the requested page is checked against the last page, recent history, and site topology to infer the source page [26].If several pages link to the requested page, the closest is added to the session.Source: D. Path Completion · p. 4
  • Reference Length separates auxiliary from content references using time between requests, assuming short stays are auxiliary and long stays are content.It assumes the final reference is auxiliary because it has no following request.Source: D. Path Completion · p. 4
  • Maximal Forward Reference treats pages visited before a backward reference as one transaction, with a new transaction beginning at that backward reference.Source: D. Path Completion · p. 4
  • Time Window forms transactions from IP address, user identification, and page durations up to a time limit; a large window can include all of a user's page references.It can also merge results from either of the preceding methods.Source: D. Path Completion · p. 4
  • A two-way hashed access-history structure searches visited pages to construct complete paths, and the authors report more accurate paths than Reference Length and Maximal Forward methods.The method starts at externally referred pages rather than the site root and uses a new session when the page is absent from current sessions.Source: D. Path Completion · p. 4

IV. PATTERN DISCOVERY AND ANALYSIS

Pattern discovery uses classification, association rules, clustering, and sequential patterns to extract different kinds of user behavior. Pattern analysis filters mined patterns and supports interpretation through queries or OLAP.

IV. PATTERN DISCOVERY AND ANALYSIS

  • Classification maps data items to predefined classes through supervised learning driven by labeled training instances.Decision trees, naive Bayesian classifiers, and Support Vector Machines are listed as examples.Source: §IV · p. 4
  • Apriori discovers the biggest frequent access-item sets from transaction databases as user access patterns.These sets are the focus of association-rule discovery.Source: §IV · p. 5
  • Clustering groups users with similar browsing patterns, supporting demographic inference, market segmentation, and personalized web content.Source: §IV · p. 5
  • Sequential patterns identify time-ordered inter-session item sequences that can help marketers predict future visits and place targeted advertisements.Source: §IV · p. 5
  • Pattern analysis filters uninteresting rules and transforms information into knowledge, using application-dependent methods such as SQL queries or data-cube OLAP.Source: §IV · p. 5

V WEB USAGE MINING APPLICATIONS

Web usage patterns support applications ranging from personalization and e-commerce to system improvement and website redesign. The section also emphasizes preprocessing and accurate session reconstruction as important to analysis and its applications.

V WEB USAGE MINING APPLICATIONS

  • User behavior patterns support personalization, recommendations, e-commerce customer management, system improvements, and website redesign.System improvements include web caching, load balancing, network transmission, and data distribution; sites may be relinked around customer behavior.Source: V WEB USAGE MINING APPLICATIONS · p. 5
  • Web-log volume is a preprocessing challenge, while access-log analysis helps understand behavior and improve web components and applications.Source: VI FUTURE · p. 5
  • Cleaning removes irrelevant records, reducing log size and improving analysis quality; accurate sessions are important for effective usage mining and personalization.Data preparation can also generate rules identifying directories for website improvement.Source: VI FUTURE · p. 5
  • The survey emphasizes session identification and path completion, and proposes graph-based session reconstruction and mining to obtain quality sessions for more accurate pattern analysis.Source: VII CONCLUSION · p. 5

AUTHORS PROFILE

The author profiles describe the researchers’ positions, experience, research interests, and publications.

AUTHORS PROFILE

  • V. Chitraa is a doctoral student at Manonmaniam Sundaranar University and a lecturer at CMS College of Science and Commerce.Source: AUTHORS PROFILE · p. 6
  • Chitraa’s research interests include database concepts, web usage mining, and clustering.Source: AUTHORS PROFILE · p. 6
  • Antony Selvadoss Davamani is a Reader at NGM College with about 22 years of teaching experience and has published about eight books and 16 papers.Source: AUTHORS PROFILE · p. 6
  • Davamani’s research interests include knowledge management, web mining, networks, mobile computing, and telecommunication.Source: AUTHORS PROFILE · p. 6
Loading 1004.1257v1…