diff --git a/README.md b/README.md index f1140af..08ffa20 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,74 @@ -# citation-analysis -Project repo for Computational Linguistics Team Laboratory at the University of Stuttgart +# Citation Intent Classification +Project repo for Computational Linguistics Team Lab at the University of Stuttgart. +## Introduction +This repository contains code and datasets for classifying citation intents in research papers. -### **Evaluation** -we plan to implement and use ***f1_score*** metric for evaluation of our classifier +We implemented 3 different classifiers and evaluated the results: -> F1 score is a weighted average of Precision and Recall(or Harmonic Mean between Precision and Recall). -> The formula for F1 Score is: -> F1 = 2 * (precision * recall) / (precision + recall) + - Perceptron Classifier - Baseline (Implemented from scratch) + - Feedforward Neural Network Classifier (using [PyTorch](https://pytorch.org/)) + - BiLSTM + Attention with ELMo Embeddings (using [AllenNLP](https://allennlp.org/) library) +This README documentation focuses on running the code base, training the models and predictions. For more information about our project work and detailed error analysis, check [this](https://www.overleaf.com/project/5f1b0e8a6d0fb80001ceb5eb) report. +For more information on the Citation Intent Classification in Scientific Publications, follow this [link](https://arxiv.org/pdf/1904.01608.pdf) to the original published paper and the [GitHub repo](https://github.com/allenai/scicite) + +## Environment & Setup +TODO + +## Data +We have 3 different intents/classes in the dataset: + + - background (background information) + - method (use of methods) + - result (comparing results) + +**Dataset Class distribution:** +| | background | method | result | +|:---|:---:|:---:|:---:| +| train | 4.8 K | 2.3 K | 1.1 K | +| test | 1 K | 0.6 K | 0.2 K | + +## Methods (Classification) +### 1) Perceptron Classifier (Baseline Classifier) +We implemented [Perceptron](https://en.wikipedia.org/wiki/Perceptron) as a baseline classifier, from scratch (including evaluation). Perceptron is an algorithm for supervised learning of classification. It's a Linear and a Binary Classifier, which means it can only decide whether or not an input feature belongs to some specific class and also it's only capable of learning linearly separable patterns. ```python -eval.metrics.f1_score(y_true, y_pred, labels, average) +class Perceptron: + def __init__(self, label: str, weights: dict, theta_bias: float): + def score(self, features: list): + def update_weights(self, features: list, learning_rate: float, penalize: bool, reward: bool): + +class MultiClassPerceptron: + def __init__(self, epochs: int = 5000, learning_rate: float = 1, random_state: int = 42) + def fit(self, X_train: list, labels: list) + def predict(self, X_test: list) ``` -#### Parameters: -**y_true** : 1-d array or list of gold class values -**y_pred** : 1-d array or list of estimated values returned by a classifier -**labels** : list of labels/classes +Since we have 3 different classes for Classification, we create a Perceptron object for each class. Each Perceptron has score and update functions. During training, for a set of input features it takes the score from the Perceptron for each label and assigns the label with max score(for all the data instances). It compares the assigned label with the true label and decides whether or not to update the weights (with some learning rate). + +Check the source [code](/blob/master/classifier/linear_model.py) for more details on the implementation of Perceptron Classifier. + +#### Running the Model + +> `(citation-env) [user@server citation-analysis]$ python -m testing.model_testing` + +[link](/blob/master/testing/model_testing.py) to the source code. All the Hyperparameters can be modified to experiment with. + +**Evaluation** +we used ***f1_score*** metric for evaluation of our baseline classifier. + +> F1 score is a weighted average of Precision and Recall(or Harmonic Mean between Precision and Recall). +> The formula for F1 Score is: +> F1 = 2 * (precision * recall) / (precision + recall) + +```python +eval.metrics.f1_score(y_true, y_pred, labels, average) +``` +**Parameters**: +**y_true** : 1-d array or list of gold class values +**y_pred** : 1-d array or list of estimated values returned by a classifier +**labels** : list of labels/classes **average**: string - [None, 'micro', 'macro'] If None, the scores for each class are returned. + +[Link](/blob/master/eval/metrics.py) to the metrics source code. + +### Results diff --git a/plots/confusion_matrix_plot.png b/plots/confusion_matrix_plot.png new file mode 100644 index 0000000..3db13a4 Binary files /dev/null and b/plots/confusion_matrix_plot.png differ diff --git a/Structural Scaffolds for Citation Intent Classification in Scientific Publications.pdf b/scicite_paper.pdf similarity index 100% rename from Structural Scaffolds for Citation Intent Classification in Scientific Publications.pdf rename to scicite_paper.pdf diff --git a/testing/model_testing.py b/testing/model_testing.py index 9bdeb8d..9b0ba9a 100644 --- a/testing/model_testing.py +++ b/testing/model_testing.py @@ -1,12 +1,12 @@ from classifier.linear_model import MultiClassPerceptron from utils.csv import read_csv_file -from eval.metrics import f1_score +from eval.metrics import f1_score, plot_confusion_matrix, get_confusion_matrix import utils.constants as const import os project_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -train_file_path = project_root+'/data/tsv/train.tsv' -test_file_path = project_root+'/data/tsv/test.tsv' +train_file_path = project_root + '/data/tsv/train.tsv' +test_file_path = project_root + '/data/tsv/test.tsv' # Read the training dataset X_train_inst = read_csv_file(train_file_path, '\t') @@ -21,12 +21,11 @@ X_test_inst = read_csv_file(test_file_path, '\t') epochs = 50 # create MultiClassPerceptron classifier object -clf = MultiClassPerceptron(epochs=epochs, learning_rate=0.5, random_state=101) +clf = MultiClassPerceptron(epochs=epochs, learning_rate=0.7, random_state=101) # train the model clf.fit(X_train=X_train_inst, labels=list(labels)) - # predict y_test = clf.predict(X_test_inst) @@ -40,3 +39,7 @@ f1_score_none = f1_score(y_true, y_test, labels, None) # Print F1 Score for result in f1_score_micro + f1_score_macro + f1_score_none: result.print_result() + +# plot confusion matrix +plot_path = project_root + '/plots/confusion_matrix_plot.png' +plot_confusion_matrix(get_confusion_matrix(y_true, y_test), 'Perceptron Classifier (Baseline)', plot_path)