Using some commonly available standards [1] and word of the crowd [2], following is a rough template of how to review a research paper and collect notes for future references.

First Pass

The first pass is to review the usefulness of the paper using its Title, Abstract, and Figures (atleast the key figures of the paper), primarily Figure 1 and 2.

Second Pass

As part of the second pass, review the Introduction, Conclusion, and the Figures carefully again and skim the rest. The intent of this pass is to create a Summary which captures the purpose of the paper and if possible what major questions are being answered.

Third Pass

Review the Related Work section if this paper needs a more in-depth analysis or answers one of the open problems that we are actively working on. We should attempt to identify the paper implementation code and the data used. The potential locations are Paperswithcode, Github, Huggingface or Kaggle and also review blogs for more concise explanations and examples.

Checkpoint - Note summary

The below is necessary for all papers being reviewed to make any decision on further introspection or file them away for future exploration.

1. Abstract: - problem | relevancy | solution | summary | objective | novelty | keywords
2. Figure 1 - visual summary of the main idea
3. Intro - relevancy | problem defn | solution
4. Conclusion: main outcome | future work
5. Data: dataset used for results, training, metrics
6. Tasks: Planned tasks or objectives
7. Results: baseline | benchmarks | improvements | comparison to other famous models
8. Utility: application for our problem
9. Future: Potential Improvements identified in the paper or we can think of.

Fourth Pass [Implementation Review]

Do this only for papers you would like to replicate/improve

• Model Architecture: Architecture description layers used and network structure
• Inputs & Outputs: Inputs | Outputs whether it is a probability, segmentation map, bounding boxes, and so on
• New or novel layers: new techniques or layers | code or the implementation probably focus on these novel layers
• Loss calculation: mathematical formula for how the loss was calculated | on what basis it was chosen
• Model Training: hyperparameter used, the batch size, and the model configurations
• Know what you did not understand - Highlight the points you did not understand | find references and resources that can help you

Fifth Pass [Replication]

Train the model on the paper data if it is available and try to replicate the results if it is possible. If not possible, apply the model on just a subset of the data or just for one epoch to make sure that the implemented model is working as expected and then you can apply it to your data.

Sixth Pass [Adoption]

Apply the same model as in the paper without any modifications to any other data set and capture the results. Attempt to modify or generalize it for the paper dataset and observe its results. Capture why it works or does not work? What issues do we run into?

The below is necessary for all papers being replicated and adapted. - Model Modifications: modifications, hyperparameter used, the batch size, and the model configurations - Techniques - Highlight the problems and techniques applied to fix them

References

[1] Stanford CS230: Deep Learning | Autumn 2018 | Lecture 8 - Career Advice / Reading Research Papers. www.youtube.com, https://www.youtube.com/watch?v=733m6qBH-jI. Accessed 26 Nov. 2022.

[2] Hosni, Youssef. “How to Read Machine Learning Papers Effectively.” Medium, 9 Oct. 2022, https://pub.towardsai.net/how-to-read-machine-learning-papers-effectively-9c2df7906516.