Via http://meaningness.com/metablog/how-to-think. All of the following are quotes from the article that, in general, appeal to my priors. All emphasis in original, which you should read in its entirety.

The implicit assumption is that the problem Bayesianism solves is most of rationality, and if I’m unimpressed with Bayesianism, I must advocate some other solution to that problem. I do have technical doubts about Bayesianism, but that’s not my point. Rather, I think that the problem Bayesianism addresses is a small and easy one.

- Bayesianism is a theory of probability.

- Probability is only a small part of epistemology.

- Probability is only a small part of rationality.

- Probability is a solved problem. It’s easy. The remaining controversies in the field are arcane and rarely have any practical consequence.

My answer to “If not Bayesianism, then what?” is: all of human intellectual effort.

* * *

Understanding informal reasoning is probably more important than understanding technical methods.

* * *

Many of the heuristics I collected for “How to think real good” were about how to take an unstructured, vague problem domain and get it to the point where formal methods become applicable. ... Finding a good formulation for a problem is often most of the work of solving it.

* * *

Suppose you want to understand the cause of manic depression. For every grain of sand in the universe, there is the hypothesis that this particular grain of sand is the sole cause of manic depression. Finding evidence to rule out each one individually is impractical. ... [T]here is an infinite list of logically possible causes. ... We can’t even imagine them all, much less evaluate the evidence for them. So:

Before applying any technical method, you have to already have a pretty good idea of what the form of the answer will be.

* * *

Choosing a good vocabulary, at the right level of description, is usually key to understanding.

* * *

1. A successful problem formulation has to make the distinctions that are used in the problem solution.

...

2. A successful problem formulation has to make the problem small enough that it’s easy to solve.

* * *

It’s important to understand that problem formulations are never right or wrong.

Truth does not apply to problem formulations; what matters is usefulness.

In fact,

All problem formulations are “false,” because they abstract away details of reality.

* * *

[I]f you don’t know the solution to a problem, how do you know whether your vocabulary makes the distinctions it needs? The answer is: you can’t be sure; but there are many heuristics that make finding a good formulation more likely. Here are two very general ones:

Work through several specific examples before trying to solve the general case. Looking at specific real-world details often gives an intuitive sense for what the relevant distinctions are.

Problem formulation and problem solution are mutually-recursive processes.

You need to go back and forth between trying to formulate the problem and trying to solve it.

* * *

If a problem seems too hard, the formulation is probably wrong. Drop your formal problem statement, go back to reality, and observe what is going on.

* * *

Learn from fields very different from your own. They each have ways of thinking that can be useful at surprising times. Just learning to think like an anthropologist, a psychologist, and a philosopher will beneficially stretch your mind.

...

If you only know one formal method of reasoning, you’ll try to apply it in places it doesn’t work.

* * *

- Figuring stuff out is way hard.

- There is no general method.

- Selecting and formulating problems is as important as solving them; these each require different cognitive skills.

- Problem formulation (vocabulary selection) requires careful, non-formal observation of the real world.

- A good problem formulation includes the relevant distinctions, and abstracts away irrelevant ones. This makes problem solution easy.

- Little formal tricks (like Bayesian statistics) may be useful, but any one of them is only a tiny part of what you need.

- Progress usually requires applying several methods. Learn as many different ones as possible.

- Meta-level knowledge of how a field works--which methods to apply to which sorts of problems, and how and why--is critical (and harder to get).