Source-linked AI summary
FLIN: A Flexible Natural Language Interface for Web Navigation
Sahisnu Mazumder, Oriana Riva
TL;DR
UI-element-level learning is effective in controlled or restricted environments, but not in the real web. FLIN maps commands to concept-level actions and ranks candidate navigation instructions using action and parameter-value matches. FLIN adapted relatively well to unseen websites and achieved best overall performance, with comparable or higher action accuracy and parameter F1 in cross-website evaluation.
Problem
UI-element-level learning is effective in controlled or restricted environments, but not in the real web.
Source: §1 · p. 1Method
FLIN maps commands to concept-level actions and ranks candidate navigation instructions using action and parameter-value matches.
Source: §1 · pp. 1–2Results
FLIN adapted relatively well to unseen websites and achieved best overall performance, with comparable or higher action accuracy and parameter F1 in cross-website evaluation.
Source: §5.2 · p. 8, §6 · p. 9Takeaways & Limitations
The authors report using FLIN on restaurant, shopping, and hotel websites and state that its design can apply to more domains.
Source: §6 · p. 9The reported use covers restaurant, shopping, and hotel websites; the broader applicability is stated as a design possibility.
Source: §6 · p. 9
Abstract
from arXiv · showhide
AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling techniques cannot flexibly adapt to many different websites without being constantly re-trained. We propose FLIN, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation instruction (involving action and parameter values). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain.
1 Introduction
The paper motivates an NL interface that can handle changing, diverse websites without retraining, and proposes mapping commands to concept-level actions. FLIN ranks candidate navigation instructions using action and parameter-value matches; the paper also introduces a dataset spanning nine websites.
1 Introduction
- The paper frames the target interface as a component that turns commands into browser-executable navigation instructions, such as finding an Italian restaurant for 7pm.The broader assistant setting executes tasks through website interactions such as typing, selecting, clicking, and navigating.Source: §1 · p. 1
- FLIN maps commands to concept-level actions rather than low-level UI actions, which the paper argues can make interfaces more flexible across websites.The motivating setting includes websites that change over time and users who may choose among websites.Source: §1 · p. 1
- Websites within a domain may use different names and parameter sets for similar actions, and may also offer different action types.OpenTable’s “Let’s go” action supports up to four parameters, while Yelp’s “search” supports two; restaurant sites may differ in offering reservations versus food ordering.Source: §1 · pp. 1–2
- FLIN treats semantic parsing as ranking candidate navigation instructions by action relevance and parameter-value matches to the command.It scores available actions, extracts command phrases expressing parameter values, and combines the action and value-assignment scores to select an instruction.Source: §1 · p. 2
- The authors contribute a dataset from nine websites across restaurant, hotel, and shopping domains, with empirical results intended to verify generalizability.A rule-based Action Extractor gathers concept-level actions and parameters for data collection; its implementation and evaluation are out of scope.Source: §1 · p. 2
2 Related Work
Related work spans semantic parsing, web task execution, and intent-and-slot models, but the paper distinguishes FLIN by its treatment of changing action sets and cross-website adaptation. Prior approaches described here often assume fixed actions, low-level actions, APIs, or application-specific labels.
2 Related Work
- Prior semantic parsing and related NLP work spans databases, question answering, visual analysis, robot navigation, object manipulation, and UI automation, but assumes fixed, known action sets.The paper contrasts that assumption with its setting of variable and unknown action sets.Source: §2 · pp. 2–3
- Web task execution work uses demonstrations, reinforcement learning, sequence-to-sequence API mapping, or API-generated task flows, but does not generalize across websites as FLIN does.The cited methods use different problem settings or operate on low-level web actions or API calls.Source: §2 · p. 3
- Pasupat et al. (2018) match commands to low-level UI actions but do not predict parameter values, unlike FLIN.Their actions are unparameterized.Source: §2 · p. 3
- Intent-detection and slot-filling models use application-specific labels, do not assign values for executable paths, and lack webpage state and handling of overlapping intents.The paper notes that one query may map to different actions on different pages of the same website.Source: §2 · p. 3
3 Problem Formulation
The problem formulation represents a webpage through its available concept-level actions and their parameters, then defines the goal as producing a navigation instruction with a correct action and parameter assignments.
3 Problem Formulation
- A webpage’s action set contains concept-level actions, each defined by an action name and a set of parameters; each parameter has a name and a value domain.Parameter domains may be closed or open.Source: §3 · p. 3
- Closed-domain parameters have finite website-imposed values, while open-domain parameters are drawn in practice from words or phrases in the user command.Examples of closed-domain values include product colors, sizes, and restaurant reservation times.Source: §3 · p. 3
- The goal is to map a command issued on a webpage to a navigation instruction containing the correct action and up to all of that action’s parameter-value assignments.The instruction may include m assignments, where m is no greater than the number of parameters for the selected action.Source: §3 · p. 3
4 The FLIN Model
FLIN decomposes web-command parsing into action recognition and parameter recognition with value assignment, then combines action and parameter scores to predict a navigation instruction.
4 The FLIN Model
- FLIN divides parsing into recognizing the intended webpage action and identifying expressed parameters, extracting their mentions, and assigning values.The model includes Action Scoring, Parameter Mention Extraction, Parameter Value Scoring, and Inference.Source: §4 · pp. 3–4
4.1 Action Scoring
- Action Scoring represents commands and action-name/parameter pairs, then ranks positive pairs above randomly sampled negative pairs.The command and action representations are compared using cosine similarity.Source: §4.1 · pp. 4–5
4.2 Parameter Mention Extraction
- Parameter Mention Extraction treats the parameter as a question and the command as a passage, using fine-tuned BERT to extract a mention or identify none.When a parameter is absent, BERT is trained to return [CLS].Source: §4.2 · p. 5
4.3 Parameter Value Scoring
- Parameter value scoring compares a mention and domain value using word-level semantic, character-level, and lexical-level similarity; lexical matching can be partial, while semantic similarity captures lexically distant but semantically closer expressions.Source: §4.3 · p. 5
4.4 Inference
- Inference assigns closed-domain values only when their scores meet a tuned threshold, averages assignment confidences, and combines that score with action relevance to select an instruction.Open-domain mentions become values directly; parameters without a qualifying domain value are discarded.Source: §4.4 · p. 6
5 Evaluation
FLIN is evaluated across nine websites in three domains, with domain-specific cross-website testing and real-query evaluation. Results indicate adaptation to unseen websites while revealing challenges from varied action and parameter spaces.
5 Evaluation
- Evaluation used nine websites across Restaurants, Hotels, and Shopping, training one model per domain on one site and testing it on the other two; cross-domain testing was excluded because action and parameter semantics overlapped little.The evaluation therefore examines in-domain transfer across websites rather than transfer between domains.Source: §5 · p. 6
5.1 Experimental Setup
- WebNav contains 53,520 generated command–instruction pairs built from website actions and parameter templates; DialQueries adds mapped real-user queries and was used only for evaluation.WebNav’s Action Extractor outputs were manually inspected and corrected, and each website retained its own action and parameter scheme.Source: §5.1 · p. 7, Table 1 · p. 7
- The evaluation measured action accuracy, parameter F1, exact-match accuracy, and 100% precision accuracy, with incorrect or missing actions scored as zero for parameter and exact-match metrics.FLIN-sem and FLIN-lex ablated lexical and semantic similarity, respectively, from parameter-value scoring.Source: §5.1 · pp. 7–8
5.2 Performance Results
- 75.6% of OpenTable, 60.3% of Bookatable, and 82.4% of Yelp commands had correct, executable mappings by PA-100; FLIN achieved comparable or higher cross-site A-acc and P-F1 than its variants.Combining lexical and semantic similarity generally outperformed either FLIN-sem or FLIN-lex alone.Source: §5.2 · p. 8, Table 2 · p. 8
- Hotels were most challenging: A-acc ranged from 45.5% to 93.9%, while EMA ranged from 14.6% to 64.3%, partly because hotel commands contain more parameters.Shopping was harder than Restaurants because products span more varied entity types, actions, and parameters across websites.Source: §5.2 · pp. 8–9
- On DialQueries, FLIN achieved overall A-acc above 50% and PA-100 above 46% despite not being trained on that dataset.DialQueries contains real user queries mapped to website navigation instructions.Source: §5.1 · p. 7, §5.2 · p. 9, Table 3 · p. 9
5.3 Error Analysis
- Among sampled failures, 13% involved no predicted action and 29% involved action mispredictions, often when webpage actions had overlapping semantics; unseen closed-domain parameters were also common errors.Hotels and shopping sites had more diverse action and parameter spaces than restaurant sites, with examples including Hyatt special rates and eBay style filters absent from training sites.Source: §5.3 · p. 9
- Closed-domain value errors often arose from parameter-value morphological variations rarely observed in training; open-domain extraction errors involved overly generic parameter names, partially matching mentions, or multiple value formats.Source: §5.3 · p. 9
6 Conclusion
FLIN maps commands to concept-level actions using matching-based semantic parsing, aiming to work with new website UIs without retraining. It adapted well to new websites and delivered good performance.
6 Conclusion
- FLIN uses matching-based semantic parsing to map user commands to concept-level actions for new website UIs without retraining.Source: §6 · p. 9
- FLIN adapted well to new websites and delivered good performance.Source: §6 · p. 9
- FLIN was used on restaurant, shopping, and hotel websites, and its design can apply to more domains.Source: §6 · p. 9