Or suppose that the needles come in a variety of shapes and sizes and some tend to look a bit like a piece of hay. Suppose that in our haystack there are a few short, sharp pieces of hay that may look a little bit needle-like. Let's move this metaphor a little closer to reality, one step at a time. Any good Kaggler should be able to get 95% accuracy at least, surely? Maybe not quite so easy! so the decision boundary should be quite clear with a nice large margin. Needle-finding is simple: just create a training set with lots of examples of needles (labelled positive) and bits of hay (labelled negative), train your favourite model, and let it loose on the haystack! Despite being small, needles have quite a few distinguishing features compared to hay - colour, density, flexibility etc. And it's an exercise in stretching a metaphor to its limit to see what we learn along the way. But it applies to any task with very unbalanced data sets, such as rare event detection. This is a bit of a rant about one problem I've been working on for a while: claim matching. How to find a needle in a haystack using machine learning Jan 19, 2021 How to find a needle in a haystack using machine learningĪutomated fact checking - how far can we go? How NOT to use machine learning to diagnose Covid-19* ![]() TIL - Symbol Grounding in Large Language ModelsĬan ChatGPT help automated fact checking?īad information is not the same as bad writing Generative AI, hallucinations and illusions Misinformation from generative AI images and text The ultimate academic league table - How high is your university above sea level?
0 Comments
Leave a Reply. |