Quantiles with Rscript
Working with Rscript without a strong statistics background can be a little daunting. Some of its functions seem to have pretty mysterious properties, and the documentation can be a challenge to grok without prior knowledge. One such function I’ve recently encountered is quantile.
The wikipedia article defines quantiles as:
cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way.
If you think of a dataset as a stream, a quantile is a point that divides the stream into a segment. There are several kinds, but perhaps the most common is the quartile. Quartiles are cutpoints that divide datasets into four parts. So, if you have a dataset of
[1, 2, 3, 4, 5, 6, 7, 8], the first, second, third, and fourth quartiles are
I ran into an issue where the code would process and filter out records from data that I was trying to seed for testing. The codebase was very opaque about the reason; all it would tell me was that it was trying to perform processing on an empty dataset. After some digging, I ran into code that looked something like this:
Playing with the repl didn’t make things much clearer. I’d create a dataset of one element, and
quantile would return the same value:
data = c(1) print(quantile(data, 0.98)) # 1
I then figured that, if a quantile’s a cutpoint in the data, it only makes sense that the 0.98th cutpoint would be the sole element in the data. So I added more datapoints; true enough, calling the
quantile function led to a different result:
data = c(1, 2, 3, 4, 5, 6, 7, 8) print(quantile(data, 0.98)) # 7.86
If a quantile is a cutpoint in the data, then there can only be one cutpoint in a dataset of size
1. With a much larger dataset, the quantile becomes something more distinct, since there are more datapoints to work with. So it turns out that all I had to do was seed more data.