Thinking outside the 10-dimensional box

Not an expert in this field, but I'm guessing this is related to unintuitive nature of geometry in high-dimensional spaces. One rough example I can think of is the fact that the number of ways to move away from the origin increases exponentially (or even faster?) as the dimensionality goes up. There is _way_ more volume away from the origin than near it, I've seen this explained as something like "most of the volume of a high-dimensional orange is in the peel". One result of this is the fact that samples of a standard gaussian end up forming a "shell" as opposed to a "ball" that you would expect (this phenomenon is called the concentration of measure in general). Also, very roughly, high-dimensional objects have lots of corners, and these corners are also very sharp. I would guess that gradient descent would get stuck in these corners and have a hard time getting out. Some links related to this: - Spikey Spheres: [link] - Thinking outside the 10-dimensional box - 3blue1brown: ▶ this video - This is a fairly long talk about HMC, but it does talk about some problems that come up when sampling high-dimensional distributions: [link]

in “Gradient descent visualization” · View on HN

Yeah, I see now. “times i” is discrete 90 degree rotation itself, not just ‘i’, nor ‘b’. Thanks everyone for making that clear. This though shows that explanations via analogies or non-strict wording may confuse one rather than enlighten. I’m not good at math, but once understood to not search analogies or geometry in things. Instead it is better to “shut up and calculate”. Not sure if imagining something is required to manage it. It’s only our brain’s faulty quirk. ▶ this video

in “A Visual, Intuitive Guide to Imaginary Numbers (2007)” · View on HN

I would reiterate that simply because there are problems with naive visualization, we shouldn't discredit visual thinking. There are several key elements to effective visual thinking. The primary importance is to keep it grounded in proofs and theorems, so you know exactly what are your limitations. Often you can use a geometric argument on top of a few theorems and you get a very strong result intuitively, and then use this intuition with a tiny amount of algebra to prove it (which might take you forever to arrive from a purely algebraic perspective). Another key is that there are several ways of visualizing things. You can almost always transform a problem into an equivalent one that is easy to visualize (just need a little bit of care with the transformation, etc). --- For example, you can show functions form a vector space, visualizing some interesting algebraic properties about them, even if it constitutes an infinite-dimensional space. You can show several operators (such as d/dx) are linear, you can give it a norm, internal product, etc. This trick lets you use visual tools (and linear algebra tools) with arbitrary functions. You can visualize projection of a function into a subspace, or into some non-canonical basis -- yielding useful applications -- such as Fourier analysis. Fourier analysis itself is a fertile ground for visual thinking. You'll be finding trivial arguments for seemingly difficult decisions such as "Does this linear system have a bounded output for a

in “What is backpropagation and what is it doing? [video]” · View on HN