Listen to this story |
As the discipline of machine learning has grown, papers based on the area have flooded academics. Communicating ideas from a field as dynamic as AI/ML is also an art. And just like art, ML-based research paper writing also has a set of rules. A researcher at Carnegie Mellon, Zachary Lipton, spoke about the serious repercussions that shoddy research paper writing has on ML and AI. “Sloppy writing poses an existential threat to AI ethics research. There’s no clear thinking without clear writing. Core ML papers can survive careless prose by expressing ideas clearly in notation. When the important ideas are qualitative, bad writing is a death sentence,” Lipton says.
Source: Twitter
Overusing technical jargon
Abhi Dubey, a research scientist with MetaAI’s FAIR, or Facebook’s AI research group, discussed most of the problems that he had come across as a peer reviewer. The usual tendency with writing introductions to research papers, in general, is to fill it with a lot of jargon that is mostly dispensable. Sentences with junk verbiage like, ‘Deep learning and CNNs have revolutionised computer vision in the past decade’ are completely unnecessary.
Dubey’s advice to researchers is to arrive directly at their point.
Economist Paul Romer has also spoken about the notion that adding more mathematical equations to a research paper was proof of being technically strong, but they instead often end up confusing readers and leave a larger gap instead of bridging it. “Like mathematical theory, mathiness uses a mixture of words and symbols, but instead of making tight links, it leaves ample room for slippage between statements in natural language versus formal language,” Romer stated.
Clear Literature Review
He also noted that sometimes researchers treat the ‘Related Work’ section of a paper as a mere checklist. Ideally, the literature review of a paper must only mention the work that is most relevant to the paper’s subject and then demonstrate why it improves upon previous work done in that regard. Dubey mentions he has noticed researchers want to mention as much related work as possible, even ones that are only related tangentially. Related work is an important section of the research because it describes the important work done around it in the past. This is why it must be treated selectively and with respect.
Source: Twitter
Excess citations and notations
Researchers have also come to believe that the more jargon their paper introduces, the more it signals technical prowess. This, Dubey says, is not true and instead serves to make the paper difficult to read. He observed papers, on average, introducing five new acronyms in one research, with a new acronym for every module. He suggests that researchers limit themselves to one new acronym per paper.
Similarly, he also notes that most submissions he has seen had excessive notations with intricate symbols that were unnecessary. In this regard, too, Dubey says that notations must be consistently used throughout the paper and with some consideration. Simplicity was the best when it came to writing notations. As is the norm, the matrices can be written in bold and sets in calligraphic. He also advises against using too many footnotes.
Source: Twitter
Most papers on ML are also littered with citations that are unintelligible. The numeric citation style that is accepted in CVPR and ICCV is hard to understand for readers. Citing papers using the author’s name or repeating the algorithm is far more legible.
Zachary Lipton, along with Jacob Steinhardt, presented a paper at ICML 2018: Machine Learning Debates, called ‘Troubling Trends in Machine Learning Scholarship’ on certain harmful patterns that had appeared in studies recently. The paper noted that certain studies mixed speculation with explanation. On the contrary, the research also encourages researchers to be explicit about being uncertain when the experiment is uncertain.
In Yoshua Bengio’s paper, ‘Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade,’ he mentions, “Although such recommendations come…from years of experimentation and to some extent mathematical justification, they should be challenged. They constitute a good starting point. . . but very often have not been formally validated, leaving open many questions that can be answered either by theoretical analysis or by solid comparative experimental work”. This expressly shows the author’s doubts about the methods and possible restrictions too.
Source: Twitter
Need for reproducibility
In a study titled, ‘Ten ways to fool the masses with machine learning,’ researchers Fayyaz Minhas, Amina Asif and Asa Ben-Hur discuss the problems with conducting ML experiments and reporting them. Usually, the recurring problem with scientific papers is that there is a reproducibility crisis. In a survey conducted by Joelle Pineau at ICML, most researchers said that there is a marked need for reproducibility with ML research papers.
While there is an understanding in the community that ML software needs to be more openly available, many studies that are published do not mention the code or the software. If a study mentions all the details of its execution, including hyperparameter settings and preprocessing, it can be reproduced by other researchers without them having to start from scratch. Even if a method performs well with one dataset, it does not necessarily indicate that it can be reproduced just as well with other datasets.
Source: Research Paper
Dataset and hyperparameters
Also, datasets are mainly dependent upon the phenomenon that the experiment intends to study.
There are studies which use proxy datasets, which isn’t advisable.
If the existing datasets aren’t of good quality, then it is more beneficial if the researcher builds the datasets themselves.
Dubey refers to the same point but for a different reason. When a study shows a novel result, the paper should underline how it advances previous work in the area and how to fine-tune the algorithm, i.e., select the hyperparameter. The study must also clarify where and if the algorithm falls short. The shortcomings of the study are normally a part of any research paper and cannot be omitted, even more so when it comes to scientific papers. The researcher should also be honest enough to mention their inspirations for the paper.
Being the most important part of the paper, the conclusion should be insightful on its own and not repetitive of the abstract. The conclusion must mention what has been done in the specific area, how studies in the area can progress and what the possible hindrances may be.
