"overfitting is like the key thing that machine learning is about" - jeremy howard.
the most successful machine learners tend to generalize, not specialize, when it comes to accuracy. it's almost like we are only capable of training generalists. not in an absolute sense (there can be a very specialized application of a model) but in the local sense: learning too much about one particular dataset does not allow it to be flexible to new data. i wonder if that flexibility is a sort of balancing function that keeps its learning function in check (indirectly, since it is the modeler who can assess overfitting with an increase in loss for the validation set after an initial decrease), making sure that it doesn't specialize too much on this source of information. a good fit, then, is a learner who can innovate, create and maybe one day imagine new connections with the world around them. again from howard: "we don't want it to memorize our particular dataset."
the lack of interdisciplinary research, discourse and thinking in our society may have to do with overfitting. i understand the desire to gain mastery, but why not master interdisciplinary thinking? from my view, i think we have a culture that is biased against interdisciplinary thinking, because it may not be efficient in one lifetime. for example, the amount of skill and productivity i gain when i have a single focus on one project for months is incomparable to when i am learning or reading about 4 or 5 different subjects that are seemingly unrelated. i am also experiencing conflict with my own thinking on mastery and specialization. i love the idea of grinding away at the edges of your understanding of a subject so that you can start to experience frictionless-ness while being creative. not always, but enough times where the experience is bliss. but learning or creative processes never work linearly, even though our education and professional system is designed as such. for many of us, this path is golden, and should be pursue it with zest, but for others, that path is counterproductive or at minimum, inefficient.
i find myself in asynchronous loops of learning. i began learning about machine learning around 2017, aggressively pursued it for awhile, distanced myself from it for a few years, and came back to it full force with a much higher chance of maintaining a sustainable and consistent pace of learning toward better understanding. i experience these loops in my music---i aggressively pursued it for a year, then cooled off for a year, then hit it hard for 2 years, then pulled back a year, then went after it for a year until recently i have pulled back again. i used think that i was "quitting" and "restarting" my focus on something, but really i was just learning it in an asynchronous way. the negative aspect of this is feeling anxiety around "not specializing" or "i won't gain mastery in anything" and so on.
the advantage, which i've only deeply experienced recently, is the ability to think about subjects in new ways. i often mash together two seemingly parallel concepts and push that relationship to the brink until either it still makes sense (hooray! this might be profound!), or it breaks the relationship between the two when the true difference between them is realized (hooray! back to square one!). it's a rough ride, but it's very rewarding, at least intellectually.
i believe that there are certain truths about a subject or skill that you can only uncover after spending enough time with it. maybe that means you are overfitting, which honestly helps predict productivity. your mind wants to stop thinking about as much as it can, so it off-loads certain concepts as taken for granted and continues on with more powerful, complex concepts that are rooted in those pesky fundamentals. i don't have a reference or research on this, but i would assume that interdisciplinary thinkers are more likely to self-correct initial assumptions than otherwise. they may know less about a single field, but they may know more about analogous fields, or may be better able to articulate the language of one field into another.
learning more about trauma helped me better understand my music, learning about deep learning is giving me fun analogies to rethink my own mental architecture (see?) and the study of the collapse of complex societies is allowing me to better understand current events and future goals. it requires a lot of dedication to be interested in a wide variety of subjects, similar to the amount of dedication it takes to be interested in one. it's just different. the greatest minds have had high competency in more than one field. mathematics, art, engineering, music, all comes together when you allow yourself to be interdisciplinary. i want to find something analogous enough with many different fields so that i can focus on a single subject while gaining the pleasure of interdisciplinary thinking. in essence, when i see myself overfitting, i get new, different data to feed my mind in order to bend it back toward generalization.