Train a separate model for each data split. Edit the [train_prompting.sh](prompt-learning/train_prompting.sh) file and add `--with_prompt_ensemble` for training with multiple prompt functions.
In the previous section, only a single *value-based* prompt is used at training and inference time. In this task, multiple *value-based* prompts are utilized at training and inference time to leverage the advantages of generation ability from different prompts. This task aims to train a single model with multiple prompts as it is much faster and more memory efficient than having to train a separate model for each prompt (and multiple models at inference time).
// TODO :: Add more README for training and generating.
Prompt Augmentation, also called *demonstration learning*, provides a few additional *answered prompts* that can demonstrate to the PLM, how the actual prompt slot can be answered. Sample selection of answered prompts are manually hand-picked. Experiments are performed on different sets of *answered prompts*.
**Training**
Edit the [test_prompting.sh](prompt-learning/test_prompting.sh) file and add `--with_answered_prompts` flag for generating slots with answered prompts.
A separate prompt ensemble model is trained for each data split to evaluate the performance of multi-prompt methods in low-resource scenarios. Edit the [train_prompting.sh](prompt-learning/train_prompting.sh) file to add `--with_prompt_ensemble` flag for training with multiple prompt functions.
Generate belief states by running the below script:
```shell
sh test_prompting.sh -m <tuned-prompt-model-path>
The probability of slot $s_{t}$ on multiple prompt functions are calculated using:
where $|K|$ represents the number of prompt functions, $f_{k}$ is the $k$-th prompt function, $\alpha_{k}$ is the weight of prompt $k$. The prompt weight $\alpha_{k}$ is set to `0.25` for all prompt functions. The loss $L$ for prompt ensemble training is calculated using the above function.
Run the training script as before after adding the `--with_prompt_ensemble` flag:
```shell
sh train_prompting.sh -d <data-split-name>
```
**Testing/Slot-generations**
While testing (slot-generation), a simple majority voting is used to pick the generated slot from different prompts. When there's no simple majority in the generated slots by multiple prompts, the slot with the highest probability is picked.
Script for generating belief states (slots) using prompt-ensemble remains the same:
(there's no need to add any extra flags here, the scripts checks if the model was trained on multiple prompts and uses ensemble prompts for generating)
```shell
sh test_prompting.sh -m <saved-model-path>
```
// TODO :: Add results
### Prompt Augmentation
Prompt Augmentation, also called *demonstration learning*, provides a few additional *answered prompts* that can demonstrate to the PLM, how the actual prompt slot can be answered. Sample selection of answered prompts are hand-crafted and hand-picked manually. Experiments are performed on different sets of *answered prompts*.
Edit the [test_prompting.sh](prompt-learning/test_prompting.sh) file and add `--with_answered_prompts` flag for generating slots with answered prompts.
```shell
sh test_prompting.sh -m <tuned-prompt-model-path>
```
### Results from multi-prompt methods
|data-split| JGA | JGA* |
|--|:--:|:--:|
| 5-dpd | 30.09 | 69.23 |
| 10-dpd | 42.84 | 86.99 |
| 50-dpd | 47.62 | 91.74 |
| 100-dpd | **48.08** | **92.87** |
| 125-dpd | 46.96 | 92.08 |
| 250-dpd | **48.08** | **92.87** |
> **Note:** All the generated output files for the above reported results are available in this repository. Check [outputs/multi-prompt](outputs/multi-prompt) directory to see the output JSON files for each data-split.
