A collection of resources and information for concrete skills that are helpful when pursuing a PhD in computer science (specifically in ML/AI or related disciplines)
Under Construction Please check back later!
This is a link to Matt Might’s blog. See the section on “Graduate School” for specific articles relevant to this topic. A few to look at in particular:
This essay, by John Schulman, provides an end-to-end guide to pursuing ML research. Note the two excellent citations to other essays, also linked here. I have yet to read it in full.
This is a (transcribed) talk by Richard Hamming. I have yet to read it in full.
This blog post, by Michael Nielsen describes several principles of effective research. In comparison to some of the other resources here, these are more self-oriented, and are thus highly relevant to the intrapersonal skills section of this guide as well. I have yet to read this in full.
This blog post by [Tom Silver] touches on lots of components of starting ML/AI research projects, and is relevant here and in other sections of this module. I have yet to read it in full.
A useful twitter thread with some good pointers.
There are a number of distinct, valuable perspectives to keep in mind when thinking about designing new projects. Some will resonate differently with different people.
A really powerful strategy that is likely the one that (especially early PhD students) most people should most often employ is one I call “Paper Arithmetic”. In this perspective, you look at recent, well-regarded papers in your particular subfield, and try to find 3 things:
If you can find a set of 3 of these things that synergize well, that’s a paper. Why is this calld “Paper Arithmetic”? Because rather than starting form an idea, you start with existing papers, and the background set of knowledge/context, and try to find different combinations that yield promising research ideas.
My first paper during my PhD was entitled Semi-Supervised Biomedical Translation with Cycle Wasserstein Regression GANs. This paper, while it undeniably has many flaws, is a great example of Paper Arithmetic. Why? The core technical method is a direct combination of two other papers: the [Cycle GAN] paper for unsupervised image-image translation and the [Wassertsein GAN] paper for stabilizing generative learning. The technical hole left in these lines of research is ways that cycle GANs can be producitvely deployed in real-world learning contents on data modalities that are not images (for which successful use of GANs was much less prevalent), and the motivating problem was paired vs. unpaired data disparities in treatment effect estimation in healthcare contexts.
Especially in application contexts, what separates high-impact from low-impact papers is a deep understanding of the real problems you’re trying to solve and the real constraints on the needed solutions. Immersing yourself in the application area of interest until you appreciate those problems from an instinctive, habitual level is, in my opinion, the best way to gain this understanding. If you can’t gain this yourself (e.g., in a multi-discplinary context), find collaborators who can, and trust their judgement (though not blindly, and not without limit).
If you can provide a solution that (1) solves a real problem, (2) fits into the users’ real workflow, and (3) doesn’t introduce new, other problems or require new ways of doing things, then this is a great starting point to having real impact.
An important subset of this area is when you produce a paper that provides a key resource for your community. For example, at the time of writing this, my most cited paper is also my least technically interesting–namely, Clinical BERT a paper that provides a pre-trained clinical BERT model for use in clinical NLP tasks. This paper was well-timed but provided a resource that met a clear need in the community, and thus has been used many times in many downstream works. As this is a resource for the community you’re already in, you will (or should) naturally already have this understanding of the problems people face.
I highlight this not because it is typically a good strategy, but because it is an unfortunately common strategy and because it is a trap. This strategy of coming up with a new project is when you suddenly, out of the blue, have a great idea, and on further thinking about it, you convince yourself further and further that this beautiful castle in the sky with minimal grounding. Often, when you try to execute on ideas generated in this fashion, you’ll find three things
I can give so many examples of papers I’ve pursued in this vein, but the sad reality is that as of the time of this writing, very few of them are published.
However, this is not to say that great projects can’t come out of this sort of project ideation. How can you turn an initially gestalt-motivated idea into something more likely to succeed?
Under Construction
Under Construction
Under Construction
This guide is largely focused on hos to structure projects for publication and impact, and is very related to communication questions as well. It is thus also featured in the communication section of this guide, but it is very relevant here too!
This guide is a great starting point for writing technical papers. I particularly value their discussion on the introduction, which states that an effective introduction must focus on answering five key questions:
I highlight this resource here (as well as in the communications section) because you need to do research such that you can eventual frame it into a compelling paper/project. Therefore, thinking about how you would frame your project, even as you’re just beginning the work, can help you structure the project overall.
Under Construction
Under Construction
Under Construction