Benfords Law: So this proves we live in a simulation right?

[Tawpgun]

I'd advise everyone to watch the Netflix Series Connected and the episode on Digits. This is where I came across it.

Watch Connected | Netflix Official Site

Science journalist Latif Nasser investigates the surprising and intricate ways in which we are connected to each other, the world and the universe.

www.netflix.com

What is Benfords Law? Its a law that states when you take a set of numbers from pretty much anything, the leading digit of those numbers will more or less follow this same curved distribution.

Benford's law - Wikipedia

en.wikipedia.org

At first glance this doesn't make sense. You would assume it should all hover around 11% or 1 in 9. But seemingly EVERYTHING follows this same population. Some examples.

Take the populations of every country in the world. (numbers vary from the tens to the billions) It follows the distribution.
Take the front page of The Financial Times and pick out ALL the numbers on it. It follows.
Take the size of volcanoes around the world. It follows.
Take the distance between all members on ERA to the Statue of Liberty. It will likely follow.
Take the distance between stars in light years. It will follow. Convert those distances to miles. Still follows. Convert those distances to a made up unit like the length of my dog. It will follow.
Length of rivers. Death Rates. Covid Data. Election data. Tax numbers. It all follows.
Take the length of notes in a song. It will follow.

Seemingly every data set gathered in our world, no matter if its in log10 or 16 or whatever, no matter what unit it is even if its converted, it will follow. You ask 10000 people for a random number between 1 and 9999999... it will follow. The real world is supposed to be random but we see this pattern EVERYWHERE. Even in places without human influence.

Here is where it gets crazy though... If you manipulate that data... the law fails. So if you take all the numbers on a tax form, it will follow the law. If someone manipulates the numbers (tax fraud) the law will fail. Auditors literally use this law to see at a glance if someone is manipulating numbers. It's also been used to point out election fraud in Iran. It is used often to detect fraud or tampering etc. Interesting in elections, the law will follow if everyone votes for their first choice. The moment people start "playing politics" and voting strategically to prevent so and so from winning and whatnot, the law fails. People use to detect fake images/videos. Tax and accounting fraud. Scientific data fraud.

This guy explains it well and offers a somewhat intuitive explanation for why it behaves the way it does.


But even with this explanation its still wild to me. If you take a set of data that ranges a couple orders a magnitude (think populations of cities vs height of human adults which doesn't vary much) it will all fit. And when it doesn't it means its basically evidence someone tampered with the data. Even though the world is chaotic and random. People die, people are born, people move. The distribution of the leading digit in world city populations will still follow benfords law.

It's some Da Vinci Code shit.

 

[Browser]

There is a natural explanation for this, for sure. Im interested, will watch the documentary. Thanks.

 

[Randdalf]

I wonder if you built this into video games, say roguelike level generation, then it would feel more "natural".

 

[Uncle Classy]

I know of the phenomenon but don't know actually know any of the possible explanations. My best guess would be that it has to do with entropy, given that the law seems to affect all systems in nature. That doesn't necessarily imply that the laws of nature are constructed by an intelligence/we live in a simulation, though.

 

[demosthenes]

I love it.

When I discovered it could be use for general ledgers in accounting to help find Fraud it became a tool I used on every job.

edit: Just saw that the OP addressed this. Can confirm that it is very easy to use.

 

[Tawpgun]

Some other cool examples from the Netflix show.

Sports stats will follow.

University of Maryland iSchool Professor Jen Golbeck appears in episode 4 of the series, where she discusses her groundbreaking research on Benford's law, an observation about the leading digits of numbers found in datasets, and how it helped to reveal patterns in social media. However, through her research, she realized that some accounts weren't following the patterns. This led to the discovery of thousands of Russian bot accounts.

^^ Thats so nuts. So if you look at followers/friends of people, it will follow the pattern. Bots will break the pattern.

 

[Browser]

Some other cool examples from the Netflix show.

Sports stats will follow.

University of Maryland iSchool Professor Jen Golbeck appears in episode 4 of the series, where she discusses her groundbreaking research on Benford's law, an observation about the leading digits of numbers found in datasets, and how it helped to reveal patterns in social media. However, through her research, she realized that some accounts weren't following the patterns. This led to the discovery of thousands of Russian bot accounts.

^^ Thats so nuts. So if you look at followers/friends of people, it will follow the pattern. Bots will break the pattern.

Click to shrink...

Holy shit this is incredible.

 

[teruterubozu]

A "simulation" means different things to different folks, especially when it comes to explaining the nature of reality. In this case it seems you are saying there is an "intelligence" behind this phenomenon, a transcendental "other".

 

[demosthenes]

Some other cool examples from the Netflix show.

Sports stats will follow.

University of Maryland iSchool Professor Jen Golbeck appears in episode 4 of the series, where she discusses her groundbreaking research on Benford's law, an observation about the leading digits of numbers found in datasets, and how it helped to reveal patterns in social media. However, through her research, she realized that some accounts weren't following the patterns. This led to the discovery of thousands of Russian bot accounts.

^^ Thats so nuts. So if you look at followers/friends of people, it will follow the pattern. Bots will break the pattern.

Click to shrink...

What specifically about bot accounts?

And is the Netflix series just called "Digits"?

 

[Tawpgun]

A "simulation" means different things to different folks, especially when it comes to explaining the nature of reality. In this case it seems you are saying there is an "intelligence" behind this phenomenon, a transcendental "other".

Click to shrink...

tbh Im only half joking about the sim thing. I personally don't think we live in a sim. I also don't believe in a creator/higher intelligence. I find some spirituality with how amazing and connected the natural world already is and this number thing only adds to that.

 

[Tawpgun]

What specifically about bot accounts?

And is the Netflix series just called "Digits"?

Click to shrink...

It's called Connected. the episode is called digits.

I'd have to rewatch but I think she looked at the followers/followed accounts.

 

[Lumination]

My first instinct was similar to the raffle explanation. 1 is the "first" digit. So if you were counting something, anytime it would overflow into the next digit, 1 is first digit it would hit. So unless your overflow is quite large, 1 is most likely where it would stop, followed by 2 being second most likely, and so forth.

It has nothing to do with world deaths "happening" to follow the law, but more like the law is a result of how we've chosen to count and represent values in our counting system of choice. This is why the law only holds when your values span wider distributions (ie it doesn't work on human heights in meters because that's capped to like 1-3).

 

[Uhtred]

You ask 10000 people for a random number between 1 and 9999999... it will follow.

Click to shrink...

Heh, reminds me that Benford's law has been used to spot possible fraud because people assume random numbers must have an even distribution of digits. So they do this when they add fake figures to the books.

 

[Speely]

Watching this as soon as I get home. This is kinda blowing my mind, conceptually.

 

[Dennis8K]

God prefers 1.

Mystery solved.

 

[Tawpgun]

My first instinct was similar to the raffle explanation. 1 is the "first" digit. So if you were counting something, anytime it would overflow into the next digit, 1 is first digit it would hit. So unless your overflow is quite large, 1 is most likely where it would stop, followed by 2 being second most likely, and so forth.

It has nothing to do with world deaths "happening" to follow the law, but more like the law is a result of how we've chosen to count and represent values in our counting system of choice. This is why the law only holds when your values span wider distributions (ie it doesn't work on human heights in meters because that's capped to like 1-3).

Click to shrink...

but even in his explanation he notes the probability changes as you overflow. And it follows even if you take numbers at random. Kind of like the people choose a number between 1 and 9999999. Or taking a newspaper and recording all numbers on it, something seemingly random and pulling data from random things.

All the way to something like city populations which can vary like 91, 90024, 1003940, 43455 etc. Not only that, but the pattern still applies when you change the log. So our numbers/counting is done in log10. But a similar pattern appears when you change that. Obviously doesn't work as well in log2 but duh

 

[Bakercat]

 

[Browser]

Think about all other patterns nature creates we havent tapped on yet.

 

[Tawpgun]

Heh, reminds me that Benford's law has been used to spot possible fraud because people assume random numbers must have an even distribution of digits. So they do this when they add fake figures to the books.

Click to shrink...

This is also what I struggle to grasp. Its completley arbitrary if I made 1,000 dollars or if I made 996 dollars. Yet Benfords Law will detect when someone manipulates the data.

 

[Rodney McKay]

My first instinct was similar to the raffle explanation. 1 is the "first" digit. So if you were counting something, anytime it would overflow into the next digit, 1 is first digit it would hit. So unless your overflow is quite large, 1 is most likely where it would stop, followed by 2 being second most likely, and so forth.

It has nothing to do with world deaths "happening" to follow the law, but more like the law is a result of how we've chosen to count and represent values in our counting system of choice. This is why the law only holds when your values span wider distributions (ie it doesn't work on human heights in meters because that's capped to like 1-3).

Click to shrink...

Yeah, that's what my guess was before watching the video, and the guy explained it pretty well with that raffle example.

Doesn't mean it's not really cool, especially how we've found uses for it like OP mentioned (cheating at taxes, votes, etc).

 

[L.E.D.]

Of course 1 appears the most, it's the first number 9 rolls into when data is counting. Law doesn't work when numbers are randomized. Interesting to see it's application in dealing with fraud though, pretty nifty.

 

[Tawpgun]

Can Benfords Law explain why yellow team always loses in Fall Guys???

 

[Lumination]

but even in his explanation he notes the probability changes as you overflow. And it follows even if you take numbers at random. Kind of like the people choose a number between 1 and 9999999. Or taking a newspaper and recording all numbers on it, something seemingly random and pulling data from random things.

All the way to something like city populations which can vary like 91, 90024, 1003940, 43455 etc. Not only that, but the pattern still applies when you change the log. So our numbers/counting is done in log10. But a similar pattern appears when you change that. Obviously doesn't work as well in log2 but duh

Click to shrink...

I'm not a mathematician at all (just too many math theory courses for my own good), so I apologize if I struggle articulating what I'm thinking. This law will work on all things "enumeratable" (not the mathematical definition of countable). Any value that tends to grow and shrink. If you think about what figures are on a newspaper, it'll be financials, addresses, death tolls, etc. Take addresses. House numbers are defined by how far down the street the house is. So, it's likely to fall under, let's call it the overflow law. Financials are the same, we're counting money earned or lost. Deaths same.'

And this isn't related to the base that we count at ie base 10 vs 12. This is more about how we use a string of digits to represent enumeratable things in the first place.

The picking a random number one is a bit more of a mystery. Perhaps explained by people's preference for 1's over 9's?

Doesn't mean it's not really cool, especially how we've found uses for it like OP mentioned (cheating at taxes, votes, etc).

Click to shrink...

Oh no, it's totally COOL. I love Numberphile. I just want people to appreciate MATH and the science and wonder of magic and that's it's all explainable and not a conspiracy haha.

 

[Timeaisis]

$199 looks better than $200. It's really as simple as that and doesn't prove anything. Confounding variables abound.

 

[Mona]

if the symbol for 1 was a 2, then everything would start with a 2, checkmate

 

[Deleted member 13101]

This has to do with the smoothness of logarithmic traversal. Where you find smoothness, you can spot anomalies.

And there isn't as much variation in the universe as we think (the universe itself is rather "flat" in terms of gravity warping). And there is always the presence of 1, because the wave function collapses to one result the vast majority of the time.

A leading 1 is also an overflow of an ordinal number that cannot fit in the previous x number of digits, so it would be more common. In computer science, 1 as a leading digit or bit could represent a negative sign in front of some number, making its use even more widespread.

As far as exponents go, a 1 exponent is the identity of the number.

As far as Fall Guys goes, yellow attracts the most cones (receptors) in the eye (why school buses are predominantly yellow), so it gives you too much visual data interfering with your performance of running around, grabbing tails, pushing balls, transporting eggs, etc.

 

[1.21Gigawatts]

This just means that existence follows a base logic we call math.
We try to represent that logic with numbers and our brains are wired to see patterns in them and try to interpret them when they are just inherent to the math and devoid of any meaning.

It is no different than the pattern odd and even numbers make.

 

[Tawpgun]

As far as Fall Guys goes, yellow attracts the most cones (receptors) in the eye (why school buses are predominantly yellow), so it gives you too much visual data interfering with your performance of running around, grabbing tails, pushing balls, transporting eggs, etc.

Click to shrink...

By god you've solved it!

Here is the other wild thing from the netflix doc. Some professor when he found out about it said there's no way and found a random book about facts of australia. He turned to a random page. Pulled 10 numbers out. Could be airline distances, temperatures, populations, any numbered fact about australia you can think of. It still followed the pattern despite the only common thing tying the numbers together is that they are about Australia in some sense. Otherwise its pulled from seemingly random data sets.

 

[Nude_Tayne]

I feel like I have an intuitive understanding of it, I think. It only works for naturally growing numbers, and whenever you reach a certain order of magnitude the leading digit will always first be a 1, but may not reach a higher digit. And the higher the digit, the less likely it is to reach it. Is that basically it?

 

[the_wart]

I think the wikipedia article has a pretty clear explanation of it, though it's a bit buried. Sets of numbers that span many orders of magnitude, like income (line cook vs CEO) or populations (Luxembourg vs China) tend to involve exponential growth -- that is, the larger you are the faster you grow. If that's the case, then jumping from a number starting with 1 to a number starting with 2 is more difficult than jumping from 2 to 3, because by the time you've reached 2, then you've grown and so exponential growth will take you to 3 even faster.

This predicts that if you looked at numbers describing things that are designed to be of similar scales, you won't see Bedford's law apply.

 

[Tawpgun]

What specifically about bot accounts?

And is the Netflix series just called "Digits"?

Click to shrink...

To follow up, I rewatched.

They took the accounts and looked at all their friends/followers friends counts. So you might have 500 friends. Your 500 friends will have varying friend counts. You tally that data up, it follows benfords curve.

For some reason, it does NOT follow benfords curve when its a bot

 

[BearPawB]

I think the wikipedia article has a pretty clear explanation of it, though it's a bit buried. Sets of numbers that span many orders of magnitude, like income (line cook vs CEO) or populations (Luxembourg vs China) tend to involve exponential growth -- that is, the larger you are the faster you grow. If that's the case, then jumping from a number starting with 1 to a number starting with 2 is more difficult than jumping from 2 to 3, because by the time you've reached 2, then you've grown and so exponential growth will take you to 3 even faster.

This predicts that if you looked at numbers describing things that are designed to be of similar scales, you won't see Bedford's law apply.

Click to shrink...

thank you, this is the first explanation in this thread that I was able to digest

 

[RedMercury]

 

[Deleted member 2809]

if we lived in a simulation I would assume mathematicians and physicists would figure it out, not a video game forum member, no offense
Math has a lot of neat "laws" like that don't seem to make sense at first but there are not so hard explanations for them

 

[Akira86]

if we lived in a simulation I would assume mathematicians and physicists would figure it out, not a video game forum member, no offense
Math has a lot of neat "laws" like that don't seem to make sense at first but there are not so hard explanations for them

Click to shrink...

i keep hearing people like those saying things like "non-zero chance we live in a simulation" and i'm just like...ok well.

 

[Tawpgun]

if we lived in a simulation I would assume mathematicians and physicists would figure it out, not a video game forum member, no offense
Math has a lot of neat "laws" like that don't seem to make sense at first but there are not so hard explanations for them

Click to shrink...

The thread title is meant to be a little tongue in cheek. I don't think I'm some genius because I figured the universe after watching a netflix episode. Hence why I provided a video better explaining why it is why it is. (The netflix episode doesn't really do that)

It's still a wild law and crazy how applicable it is to situations like finding doctored images/deepfakes, russian bots, accounting fraud etc

 

[Lumination]

To follow up, I rewatched.

They took the accounts and looked at all their friends/followers friends counts. So you might have 500 friends. Your 500 friends will have varying friend counts. You tally that data up, it follows benfords curve.

For some reason, it does NOT follow benfords curve when its a bot

Click to shrink...

Because people add friends one at a time. When you start counting from 0, your stopping point will be under the overflow law. Bots probably just generate a (pseudo)random number of friends it wants to add and just adds them. Computer-generated numbers will be (sort of) truly random and thus does not fall under this law in the sense that they have no skewed overflow chance because they were never counting.

 

[Septimus Prime]

By god you've solved it!

Here is the other wild thing from the netflix doc. Some professor when he found out about it said there's no way and found a random book about facts of australia. He turned to a random page. Pulled 10 numbers out. Could be airline distances, temperatures, populations, any numbered fact about australia you can think of. It still followed the pattern despite the only common thing tying the numbers together is that they are about Australia in some sense. Otherwise its pulled from seemingly random data sets.

Click to shrink...

It even applies to this post.

 

[demosthenes]

To follow up, I rewatched.

They took the accounts and looked at all their friends/followers friends counts. So you might have 500 friends. Your 500 friends will have varying friend counts. You tally that data up, it follows benfords curve.

For some reason, it does NOT follow benfords curve when its a bot

Click to shrink...

Ah. Because someone sets up 500 accounts and friends all the same accounts. Every account probably has the same number so it's not random. Pretty cool way to use this.

 

[Coyote Starrk]

if we lived in a simulation I would assume mathematicians and physicists would figure it out, not a video game forum member, no offense
Math has a lot of neat "laws" like that don't seem to make sense at first but there are not so hard explanations for them

Click to shrink...

I feel like this is something a computer program would say to make me not believe I am in a simulation.

 

[Ramala]

I know of the phenomenon but don't know actually know any of the possible explanations. My best guess would be that it has to do with entropy, given that the law seems to affect all systems in nature. That doesn't necessarily imply that the laws of nature are constructed by an intelligence/we live in a simulation, though.

Click to shrink...

But it could suggest the laws of nature adhere to rules we just haven't figured out.

 

[Parthenios]

Does this law apply to non-base ten number systems, like binary and hexadecimal? Well, I guess 100% of binary numbers start with 1. So no, it wouldn't apply. So I bet it has something to do with how the number system is set up.

 

[zashga]

but even in his explanation he notes the probability changes as you overflow. And it follows even if you take numbers at random. Kind of like the people choose a number between 1 and 9999999. Or taking a newspaper and recording all numbers on it, something seemingly random and pulling data from random things.

All the way to something like city populations which can vary like 91, 90024, 1003940, 43455 etc. Not only that, but the pattern still applies when you change the log. So our numbers/counting is done in log10. But a similar pattern appears when you change that. Obviously doesn't work as well in log2 but duh

Click to shrink...

Well now hold on. Literally every binary number after zero starts with a one. How deep does this go?!

In all seriousness it makes sense to me that "natural" quantities in any base are most likely to start with a one. If you're counting up from 1 to 9 the leading digit changes with each number. When you hit 10, it doesn't change until 20. When you hit 100, it doesn't change until 200, and so on. Every time you reach another leading one, you've reached a larger shelf that takes longer to roll over to a leading two. It follows that any quantity trending upwards is most likely to fall on a shelf where its value starts with a one.

 

[CitizenVectron]

This is because in any countable set, numbers naturally start lower than higher, and the amount of any set will not be infinite. So there will always be a 1, but might not be a 99999999. Because of this, lower numbers will be over-represented. This is also why a completely random set of numbers will be evenly distributed, because there was no starting lower number that increased.

I don't see why this is controversial, it follows basic reasoning.

 

[Yeeeeeeeeeer]

i get the feeling that ppl (usually atheists) who subscribe to the simulation theory are trying to attach some meaning to life. to me it is their version of god. maybe believing in a deity is just inherent in humans...

 

[LinkStrikesBack]

I know of the phenomenon but don't know actually know any of the possible explanations. My best guess would be that it has to do with entropy, given that the law seems to affect all systems in nature. That doesn't necessarily imply that the laws of nature are constructed by an intelligence/we live in a simulation, though.

Click to shrink...

To be honest, my guess would be it's more simple than that.

Take the population of a country example, for instance. If you have a country, the population will depend on lots of things, area, resources, etc etc, but there's something even more fundamental to it in terms of the first digit I bet.

And that is to get to having a 9 as the first digit you have to pass through all the previous population numbers beforehand. When you move up a population order of magnitude, your first digit is inevitably a 1. And then it will be 1 for longer than it is 2 (because more people have more babies). 2 for longer than it is 3. So on. This means when you look at all the populations of all countries, it's be more shocking if it wasn't heavily biased to starting with 1.

 

[Lumination]

This is because in any countable set, numbers naturally start lower than higher, and the amount of any set will not be infinite. So there will always be a 1, but might not be a 99999999. Because of this, lower numbers will be over-represented. This is also why a completely random set of numbers will be evenly distributed, because there was no starting lower number that increased.

I don't see why this is controversial, it follows basic reasoning.

Click to shrink...

Well put. I also read it in your avatar's voice. 10/10.

 

[Eoin]

Take the distance between stars in light years. It will follow. Convert those distances to miles. Still follows. Convert those distances to a made up unit like the length of my dog. It will follow.

Click to shrink...

Since people are already discussing Benford's Law I'll just skip right over that to make a totally different point, which is that the length of your dog is no more of a "made up unit" than a mile or a light year.

 

[Tawpgun]

To be honest, my guess would be it's more simple than that.

Take the population of a country example, for instance. If you have a country, the population will depend on lots of things, area, resources, etc etc, but there's something even more fundamental to it in terms of the first digit I bet.

And that is to get to having a 9 as the first digit you have to pass through all the previous population numbers beforehand. When you move up a population order of magnitude, your first digit is inevitably a 1. And then it will be 1 for longer than it is 2 (because more people have more babies). 2 for longer than it is 3. So on. This means when you look at all the populations of all countries, it's be more shocking if it wasn't heavily biased to starting with 1.

Click to shrink...

Yep. It fits very perfectly for things that tend to grow exponentially. This video goes into it.

Benford's law (with Vi Hart, 2 of 2) (video) | Khan Academy

Vi Hart visits Khan Academy and talks about the mysteries of Benford's Law with Sal

www.khanacademy.org

I do generally understand that.

Whats odd is it can also be applies to things that aren't based on exponential growth/decline. Like the guy that pulled random numbers from different data sets in a book. The numbers shouldn't have a relationship to each other, but they followed this sequence. Things in the natural world follow it. It all comes down to the logarithmic scale.

Benford's Law

And I get that... a little. But it doesn't still mean its not a cool phenomenon that has amazing applications. Reading through explanations and then looking at more benford examples I seem to waver. I watched the netflix doc. This is magic. I watch the numberphile video. Oh no its actually intuitive. I read an example of benford in something that has nothing to do with growing series of numbers. Ok its back to magic. Someone explains maybe why. Oh ok. and so on.

 

[LukeOP]

$199 looks better than $200. It's really as simple as that and doesn't prove anything. Confounding variables abound.

Click to shrink...

Yep, 199 light years looks better than 200 light years, that's why stars are the distance they are from each other.

Also. Does this law only apply to base 10 or other base numbers as well?