Added output prediction files, fixed scripts & added JGA results in README.md

README.md

@@ -18,7 +18,7 @@ MultiWOZ 2.1 [dataset](https://github.com/budzianowski/multiwoz/blob/master/data
 
 In the above table, term "*dpd*" refers to "*dialogues per domain*". For example, *50-dpd* means *50 dialogues per each domain*. 
 
-All the training and testing data can be found under [/data/baseline/](data/baseline/) folder.
+All the training and testing data can be found under [/data/baseline/](data/baseline) folder.
 
 ## Environment Setup
 Python 3.6 is required for training the baseline model. `conda` is used for creating environments.
@@ -140,3 +140,12 @@ Edit the [evaluate.py](baseline/evaluate.py) to set the predictions output file
 ```shell
 python evaluate.py
 ```
+#### Preliminary results of baseline evaluation
+|data-split| JGA |
+|--|:--:|
+| 50-dpd | 28.64 |
+| 100-dpd | 33.11 |
+| 125-dpd | 35.79 |
+| 250-dpd | 40.38 |
+
+> Note: The above results might change based on further experiments  

baseline/evaluate.py

@@ -101,20 +101,25 @@ class BaselineDSTEvaluator:
 
         correctly_predicted, total_turns = 0, 0
         for truth, prediction in zip(true_states, prediction_states):
+            # print("Truth :: ", truth)
+            # print("Prediction :: ", prediction)
             total_turns += 1
 
             if set(truth.keys()) != set(prediction.keys()):
                 continue
 
+            has_wrong_slot_value = False
             for slot in truth:
                 if truth[slot] != prediction[slot]:
+                    has_wrong_slot_value = True
                     break
-            correctly_predicted += 1
+            if not has_wrong_slot_value:
+                correctly_predicted += 1
 
         print('Evaluation :: Joint Goal Accuracy = ', (correctly_predicted / total_turns) * 100)
 
 
-evaluator = BaselineDSTEvaluator('../data/outputs/experiment-20220831/125-dpd/checkpoint-90000/output_test.json',
+evaluator = BaselineDSTEvaluator('../outputs/baseline/experiment-20220829/50-dpd/checkpoint-55000/output_test.json',
                                  '../data/baseline/test/test.soloist.json')
 predicted_belief_states = evaluator.parse_prediction_belief_states()
 true_belief_states = evaluator.parse_true_belief_states()