From a4559c3aa2919d45771e6ee1a0627ada495c8cad Mon Sep 17 00:00:00 2001 From: Pavan Mandava Date: Sun, 11 Dec 2022 21:44:25 +0000 Subject: [PATCH] Updated ANALYSIS.md --- ANALYSIS.md | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/ANALYSIS.md b/ANALYSIS.md index 5e62e12..2e4c516 100644 --- a/ANALYSIS.md +++ b/ANALYSIS.md @@ -14,14 +14,33 @@ The belief state prediction task of SOLOIST utilizes *top-k* and *top-p* samplin | **user:** we need to find a guesthouse of moderate price.
**system:** do you have any special area you would like to stay?
or possibly a star request for the guesthouse?
**user:** i would like it to have a 3 star rating. | type = guesthouse
pricerange = moderate
stars = 3 | parking = yes
stars = 3 | + ### Prompt-based Methods -#### Value-based prompt +#### Value-based prompt & Inverse prompt +Value-based prompt utilizes the dialog history and value to generate corresponding slots. This approach doesn't rely on the ontology of the slots. While training, both value-based prompts and inverse prompts are used to compute the training loss. The inverse prompt mechanism helped complementing the value-based prompt in generating the correct slots. It's worth mentioning that there's a 5-10% drop (depending on the data split trained on) in the JGA score when inverse prompt mechanism is not applied during training. + +The experimental results show a significant difference in the performance between baseline SOLOIST and Prompt-based methods. Prompt-based methods significantly outperformed the baseline model under low-resource settings (*5-dpd*, *10-dpd* and *50-dpd*). + +#### destination vs departure & leave vs arrive +Under low-resource settings, the prompt-based model struggled while generate slots like *departure*|*destination* and *leave*|*arrive*. For many instances, it wrongly generated *destination* instead of *departure* and vice-versa. Below is one example where slots are wrongly generated. -#### destination vs departure +| Dialog History | True belief states | Generated belief states | +|-------------------------------------------------------------------------| ----- | ----- | +| **user:** I need to be picked up from pizza hut city centre after 04:30 | leave = 04:30
departure = pizza hut city centre | arrive = 04:30
destination = pizza hut city centre | -#### Duplicate values +#### Repeated values +Consider the following example: + +| Dialog History | True belief states | +| ----- | ----- | +| **user:** hi, can you help me find a 3 star place to stay?
**system:** Is there a particular area or price range you would like?
**user:** how about a place in the centre of town that is of type hotel
**system:** how long would you like to stay, and how many are in your party?
**user:** I'll be arriving saturday and staying for 3 nights. there are 3 of us.| area = centre
stars = 3
type = hotel
day = saturday
people = 3
stay = 3| + +The repeated value `3` in the above example can lead to ambiguity for value-based prompt while generating the slots. #### Multi-prompt methods -### Value Extraction \ No newline at end of file +#### JGA and JGA* Scores +Higher JGA* scores suggest the current methods of extracting value candidates need improvements. + +### Value Extraction